Creating a coin auction database
Boosibri
Posts: 12,727 ✭✭✭✭✭
I shared this on the World Coins forum, but I thought it would also be of interest on the US forum.
One of the most rewarding aspects of the hobby for me is to build out the provenance chain for coins that I own, tracing them back through to various important (or unimportant) collections. Often when I am going to acquire a major coin, I will build a census of the extant examples to understand how the coin for sale compares to others and note key features which might be missed when looking at only a single coin. Much of what I collect has only a few (less than 20) examples extant, so census and provenance tracing is much easier than if the coins were more available.
The challenge with searching a coins provenance is that the process of looking through individual catalogs and books is manual and time consuming. I will often have a good instinct for which sales are most likely to have the coin but it is still a large effort. With the advent of the Newman Numismatic Portal and the ANS digitizing catalogs and posting them on Archive.org, the ability to search large amounts of catalogs has improved, but it still has its limitations.
The catalogs which are digitized have an OCR scan which translates the images to text. When you search within a catalog you are searching for the appearance of the search terms ANYWHERE within the catalog, or the exact phrase without deviation. This makes for many false positives and also false negatives and a ton of time sorting through them.
To solve this problem I decided to create a master database of all of the coins sold in my specialty with an entry for each lot versus the mass of OCR text for the catalog as a whole. I created the data tables and then created skills in Claude to extract catalogs in mass from various public sites. When in PDF, the AI creates an OCR text file, or when the OCR is available it parses the text into individual lots to then populate the database.
I then built a front end to allow for easy search functionality vs. having to learn and use SQL logic. The result thus far is a 260,000 lot database from >800 catalogs and growing. I am only ingesting into the database the Latin American lots given that is my specialty, with an exception for Greenland as I have an interest in the Danish Asiatic Piastre.
Here is a view of the front end and an example query return:
Comments
sweet
outstanding
Top 10 Cal Fractional Type Set
successful BST with Ankurj, BigAl, Bullsitter, CommemKing, DCW(7), DesertMoon, Downtown1974, Elmerfusterpuck, Joelewis, Mach1ne, Minuteman810430, Modcrewman, Nankraut, Nederveit2, Philographer(5), Proofcollection, Realgator, Silverpop, SurfinxHI, TomB and Yorkshireman(3)
wow