Building an Auction Catalog Database

Boosibri

As some of you know, one of my favorite things to do with the coins in my collection is to trace the provenance of the coin back to other notable collections. The process of that is usually searching through my library or through various online sources. The online sources, including the NNP are very tedious and frankly frustrating. The search functionality of the Newman Portal is only effective when you open each catalog individually. With the recent upgrade, the catalog listings are now inaccurate and difficult to use. For example, the Hans M.F. Schulman listings now have over 8900 listings including English sales from the 1700's.

I found in my searches that I relied upon the same 100 sales or so each time. I decided then to create my own searchable database by ingesting the catalogs into a Claude AI agent to then build and populate a SQL database. Each catalog hOCR is uploaded from Archive.org or a PDF is uploaded for the agent to OCR and include in the database. Many times the OCR is poor especially for older catalogs or annotated sales where the writing obscures the text. This is generally an issue for lot codes, but the AI can usually infer the lot code based on sequencing.

I'm up to about 50 catalogs in the database now. I'll upload all of the Henry Christensen, Schulman, Almanzar, Glendenings, Spink and other similar catalogs. These are all intended to supplement CoinArchives or ACSearch which have newer listings. The next effort will be to try and add the Virgil Brand inventory and invoices from the NNP into the dataset. The OCR on the NNP is a challenge so I would need to see if Claude and read a PDF output and assign the dataset correctly.

Key gaps due to a lack of a digital copy will be: Norweb/Christies, UBS/Swiss Bank Corp, Ponterio.

Here is a view of the output and a list of the catalogs ingested so far.

threefifty

Cool project Boosibri. I went into the NNP yesterday for the first time in a few months and was disappointed to see the upgrades have made it even harder to use, and it wasn't easy before.

Re: missing catalogs - there used to be a bunch of Ponterio catalogs in there before the update... I'll send an email to see if they can restore access.

Boosibri

@threefifty said:
Cool project Boosibri. I went into the NNP yesterday for the first time in a few months and was disappointed to see the upgrades have made it even harder to use, and it wasn't easy before.

Re: missing catalogs - there used to be a bunch of Ponterio catalogs in there before the update... I'll send an email to see if they can restore access.

They have some still but there are many missing

robp2

When you have finished the current holding, send me a list and I will see what I have spare. If I listed everything spare, I would lose the will to live.

Abuelo

Very impressive @Boosibri

SmEagle1795

Here's an organized list of many generally publicly available auction catalogs, usually hosted on BnF's Gallica or archive.org, which should provide a good seed set:

https://www.rnumis.com/auctions_top.php?elookback=0

Boosibri

@SmEagle1795 said:
Here's an organized list of many generally publicly available auction catalogs, usually hosted on BnF's Gallica or archive.org, which should provide a good seed set:

https://www.rnumis.com/auctions_top.php?elookback=0

Really helpful. Some additional sources and a very helpful structure. The Gallica user experience is what the NNP hopefully will be one day.

Nap

I like to have auction catalogs stored locally onto my computer. Most are from archive org or from the auctioneers’ web sites. Having these stored locally allows me to do text searches to find certain coins. However even though the scans are often good, the compressed pdf files at archive are typically of very poor quality, making the pictures useless. So this requires downloading the scanned image files from archive. Then I need to search through the image files to find the high res scan of the coin I’m seeking. There certainly needs to be a better way. I suppose I could just use the higher resolution scanned images to make my own PDF or other format files, eschewing compression for quality. I’m not that well versed in this sort of computer workflow so I probably am doing it the most inefficiently way possible.

Boosibri

End of the work day after having a Claude running in the background most of the day has got me to 108 catalogs and about 25% of the way through Ponterio and going.

MEJ7070

Now this is AI I can get behind. Cool project!

Boosibri

And update for those interested:
The process of building the database has two time consuming elements:
1. Retrieving the catalogs meant going manually to Archive.org or another source, catalog by catalog, downloading them and then putting them into Claude.
2. Claude reviewing the OCR and parsing the text into data tables.

Since I have started this project, I have been able to further automate this process.

#

In the case of the catalog sourcing, I connected Claude through Chrome and via the Chrome extension, an agent can act on my behalf and download all of the catalogs without anything but my systematic instructions. I was able to automate this for Archive.org, Aureo, and seemingly all sites in scope. This has saved hours of time.

#

Second, the parsing of catalogs is a skill in Claude now. Based on doing several catalog ingestions, the steps are consistent. I created a skill from these repetitive tasks which then is followed by multiple agents at a time vs. one acting and reasoning. In this way multiple agents can process the task faster and uses less tokens.

#

The project is yielding real benefit, and querying is fast. Less than 1 second to have accurate results from all catalogs. The biggest barrier which is unavoidable at this point is the poor OCR of many of the Newman/Archive catalogs. The text is jumbled in the OCR from a poor scan and the AI can infer what the text says but the source is bad. In most cases this impacts just the lot number given the location near the spine, but in other cases it is more pronounced.