I have added PCGS integration so users can easily scan their PCGS slabs and pull up
CoinFacts
TrueView images [if they exist]
Previous auction prices
The use case I thought of when building out this feature was for users who are at a coin show or coin shop and want to pull up info on a PCGS slab. What makes this more powerful than just using the PCGS CoinFacts app is that Numi can pull up slab info and remember it as context in your conversation thread. So users can follow up the conversation and ask follow-up questions like
"Has this coin been getting more or less valuable over time?"
"Do the TrueView images look correct for this slab?" [Essentially a counterfeit/authentication check]
"What are some interesting facts about this coin?"
In other fun news, I was invited by the Coin World Podcast team to discuss Numi and my thoughts on the future of Artificial Intelligence's impact on Numismatics
I revamped Numi to v1.33 to use the official American Numismatic Association's grading standards. My goal was to transform Numi to focus solely on technical grading, as technical grading is much more objective as compared to market grading.
Below are the results for coins ranging from G-4 to PR-70.
As you can see, the results were quite off. With Numi getting increasingly accurate as the coin moved to a better condition.
Numi struggles quite a bit with recognizing wear on a coin. Interestingly enough, this was the exact issue Compugrade experienced back in the 90's. Even though their algorithms were completely different than today's AI Large Language Models.
Given Numi's failure to achieve consistent results, I am putting the project on ice until the next OpenAI GPT model is released. I'm also interested in testing Numi using Google's Gemini Ultra model once that releases sometime next year. Once they do, I will revisit these tests and see how much Numi has improved.
Observations & Lessons Learned
Like what many online are reporting, GPT-4 has substantially degraded in performance in the past few months. I found myself constantly arguing with the AI to get it to follow the most my custom instructions
The model can follow custom instructions well for the first 8-10k tokens, but perception and logic fall off a cliff after that. Even with highly tailored instructions with clear steps to follow, Numi would veer off course when it had to process too much info. This was incredibly frustrating as GPT-4 Turbo claimed to have a token limit of 128k [i.e. ability to remember up to nearly 25k words]
The vision model needs a major boost. The Optical Character Reader was great at picking up texts on coins. I rarely had it misread the text. But often it would fail to pick up the mint mark on more worn coins. The vision model could generally tell when a coin was more worn, but it failed to apply a correct grade.
Numi was excellent at identifying non-ancient coins. And was quite often great at identifying tokens. But it struggled immensely with Ancient coins. The only ancient coins it could identify were coins in very high condition.
Overall Numi was a fun exciting project. I got to apply my AI knowledge to my hobby and I'm excited to see how future models perform. AI is a force to be reckoned with and it's going to be an invaluable resource for researchers and those trying to learn more about their coins. I left this project feeling that AI technical grading would eventually work. But I now have my doubts that it can overtake market grading in a manner that would be widely adopted.
I’m sorry to hear this after reading this interesting thread. Do you think that if you got some consistency in the photos it would’ve led to a different conclusion? For instance, if one created a jig which moved in a controlled manner in control lighting?
This might be an oddball question but here I go. Is there any way to get the AI to define its grading formula? If there was a scientific formula for grading coins that would solve a lot of problems.
Can you get the AI to produce this formula instead of producing coin grades?
The substantial truth doctrine is an important defense in defamation law that allows individuals to avoid liability if the gist of their statement was true.
Why would you want the AI grading formula if it is not accurate and thus not usable?
We already have the ANA Grading Standards book which has objective and simple criteria.
@yosclimber said:
Why would you want the AI grading formula if it is not accurate and thus not usable?
We already have the ANA Grading Standards book which has objective and simple criteria.
Currently there isn't a scientific formula for grading coins. If the AI can make an example other people might be able to refine it into something correct. My dad is an engineer and says "If you cannot write a scientific formula for it, you don't know shit about it."
Until we get a formula for coin grading, it will continue to be a subjective mess.
The substantial truth doctrine is an important defense in defamation law that allows individuals to avoid liability if the gist of their statement was true.
They give subjective descriptions of the grades and do not provide a formula.
The substantial truth doctrine is an important defense in defamation law that allows individuals to avoid liability if the gist of their statement was true.
I'll admit I am mostly thinking of circulated grades, like "3 letters of LIBERTY" for Indian cents and Barbers VG-8.
That is definitely a formula.
And for EF-45, AU-50, AU-55, there is a % luster formula.
It looks to me that shadows and toning throw the AI off with gold. Maybe the AI needs to see several images of the same coin taken when it is tipped and rotated as we did in the grading class. Maybe they need to program the computer to view a video of both sides of the coin while it is moved around. Seems to me a whole lot of trouble to get a grade that is just another opinion. The idea of taking a phone image of an ancient coin or an islamic coin and having AI ID it is a great one.
@yosclimber said:
Why would you want the AI grading formula if it is not accurate and thus not usable?
We already have the ANA Grading Standards book which has objective and simple criteria.
Currently there isn't a scientific formula for grading coins. If the AI can make an example other people might be able to refine it into something correct. My dad is an engineer and says "If you cannot write a scientific formula for it, you don't know shit about it."
Until we get a formula for coin grading, it will continue to be a subjective mess.
The grading system is not subjective because the system itself is not understood. We devised the system. A coin either has wear or it doesn’t, and it can have a certain amount of surface damage or not. Those are objective characteristics. The grading system is subjective because of the following:
-Every coin is different. No coin below 70 is in an exactly identical state. Every grade below 70 is a net grade. Thus an MS63 (as an example) is a range of conditions, not a single physical state. Every unique coin must be placed into a range, which means there is no simple formula that applies to all coins.
-There are differences in the way different series are approached. Early large cents are graded with leniency towards surface corrosion. Early gold is graded with leniency towards cleaning. Standards have also changed over time in various areas.
-Ultimately, graders are human. It is the grader’s task to place each coin into the proper grade (aka range of conditions) with respect to the standards of the the coin’s series - but humans are neither identical to one another nor perfectly consistent. And sometimes a coin seems to lie on the line between one grade and the next, or between a details grade and a net grade. Those coins will not necessarily, and perhaps should not, receive the same grade every time they are sent in, because even the most discerning and experienced humans will not always line up on them - and the end users, collectors, are also only human. If AI were to assign an identical grade to such a coin every time, that might be seen as problematic to those who lean towards the other of the two possible grades. Perhaps it is our treatment of assigned grades as objective in the marketplace, and not this sort of human variability, that is problematic.
A simple “formula” will not solve any of the above issues in any way that would be helpful for human graders, especially since any “formula” would simply be drawn from our own grading formulas that we have devised. If AI were to grade coins itself, it might be able to eliminate the problems associated with the human aspect of grading, but it is only as good as the inputs that it is given. It can only learn what an MS64 is based on our own ideas of what an MS64 is, and learn how to approach a certain series based on our (current) standards of how to approach that series.
Apologies for the long delay in replying. Been doing some traveling!
@Zwiggy said:
I’m sorry to hear this after reading this interesting thread. Do you think that if you got some consistency in the photos it would’ve led to a different conclusion? For instance, if one created a jig which moved in a controlled manner in control lighting?
I don't think consistency in photos would have led to more accurate grading by Numi. I took pains to make sure my photos had consistent lighting and it still led to inconsistent results at lower grades.
IMO, the next step to get AI grading off the ground would have to be a stronger visual AI model. Specifically, a model that can handle video input. Thankfully AI is advancing quickly. I'm excited for the day where I can get my hands on an AI model where I can pull up a camera and the AI can determine a coin's grade based on the user moving a camera around a coin. Telling me in real time what it thinks the grade is.
@RiveraFamilyCollect said:
This might be an oddball question but here I go. Is there any way to get the AI to define its grading formula? If there was a scientific formula for grading coins that would solve a lot of problems.
Can you get the AI to produce this formula instead of producing coin grades?
It is possible to have an AI define its grading formula in the sense that initially with Numi, I did ask it to draft up descriptive grading criteria based on the Sheldon Scale. However, I would stress that the AI is essentially hashing together all the info it was trained on which includes many opinions on the Sheldon Scale. It wasn't creating a new objective standard.
Comments
Numi v1.30 alpha is now live!
I have added PCGS integration so users can easily scan their PCGS slabs and pull up
The use case I thought of when building out this feature was for users who are at a coin show or coin shop and want to pull up info on a PCGS slab. What makes this more powerful than just using the PCGS CoinFacts app is that Numi can pull up slab info and remember it as context in your conversation thread. So users can follow up the conversation and ask follow-up questions like
Live Video Demo
In other fun news, I was invited by the Coin World Podcast team to discuss Numi and my thoughts on the future of Artificial Intelligence's impact on Numismatics
Spotify
Other Podcast Links
More updates to come
Try these if you can.
MS-60 to MS-62
I’ll post the results in a day after my GTG is over.
AI is pure evil, I would stay away from it.
God Bless, CRHer700
Do unto others what you expect to be done to you.
Dubbed a "Committee Secret Agent" by @mr1931S on 7/23/24
Update: Numi v1.33 Alpha [12/20/2023]
I revamped Numi to v1.33 to use the official American Numismatic Association's grading standards. My goal was to transform Numi to focus solely on technical grading, as technical grading is much more objective as compared to market grading.
Below are the results for coins ranging from G-4 to PR-70.
As you can see, the results were quite off. With Numi getting increasingly accurate as the coin moved to a better condition.
Numi struggles quite a bit with recognizing wear on a coin. Interestingly enough, this was the exact issue Compugrade experienced back in the 90's. Even though their algorithms were completely different than today's AI Large Language Models.
Given Numi's failure to achieve consistent results, I am putting the project on ice until the next OpenAI GPT model is released. I'm also interested in testing Numi using Google's Gemini Ultra model once that releases sometime next year. Once they do, I will revisit these tests and see how much Numi has improved.
Observations & Lessons Learned
Overall Numi was a fun exciting project. I got to apply my AI knowledge to my hobby and I'm excited to see how future models perform. AI is a force to be reckoned with and it's going to be an invaluable resource for researchers and those trying to learn more about their coins. I left this project feeling that AI technical grading would eventually work. But I now have my doubts that it can overtake market grading in a manner that would be widely adopted.
I’m sorry to hear this after reading this interesting thread. Do you think that if you got some consistency in the photos it would’ve led to a different conclusion? For instance, if one created a jig which moved in a controlled manner in control lighting?
This might be an oddball question but here I go. Is there any way to get the AI to define its grading formula? If there was a scientific formula for grading coins that would solve a lot of problems.
Can you get the AI to produce this formula instead of producing coin grades?
The substantial truth doctrine is an important defense in defamation law that allows individuals to avoid liability if the gist of their statement was true.
Why would you want the AI grading formula if it is not accurate and thus not usable?
We already have the ANA Grading Standards book which has objective and simple criteria.
Currently there isn't a scientific formula for grading coins. If the AI can make an example other people might be able to refine it into something correct. My dad is an engineer and says "If you cannot write a scientific formula for it, you don't know shit about it."
Until we get a formula for coin grading, it will continue to be a subjective mess.
The substantial truth doctrine is an important defense in defamation law that allows individuals to avoid liability if the gist of their statement was true.
What is not scientific about the ANA Grading Standards book?
It does not cover the higher MS grades. Is that what you meant?
How about the PCGS book?
They give subjective descriptions of the grades and do not provide a formula.
The substantial truth doctrine is an important defense in defamation law that allows individuals to avoid liability if the gist of their statement was true.
I'll admit I am mostly thinking of circulated grades, like "3 letters of LIBERTY" for Indian cents and Barbers VG-8.
That is definitely a formula.
And for EF-45, AU-50, AU-55, there is a % luster formula.
.
It looks to me that shadows and toning throw the AI off with gold. Maybe the AI needs to see several images of the same coin taken when it is tipped and rotated as we did in the grading class. Maybe they need to program the computer to view a video of both sides of the coin while it is moved around. Seems to me a whole lot of trouble to get a grade that is just another opinion. The idea of taking a phone image of an ancient coin or an islamic coin and having AI ID it is a great one.
The grading system is not subjective because the system itself is not understood. We devised the system. A coin either has wear or it doesn’t, and it can have a certain amount of surface damage or not. Those are objective characteristics. The grading system is subjective because of the following:
-Every coin is different. No coin below 70 is in an exactly identical state. Every grade below 70 is a net grade. Thus an MS63 (as an example) is a range of conditions, not a single physical state. Every unique coin must be placed into a range, which means there is no simple formula that applies to all coins.
-There are differences in the way different series are approached. Early large cents are graded with leniency towards surface corrosion. Early gold is graded with leniency towards cleaning. Standards have also changed over time in various areas.
-Ultimately, graders are human. It is the grader’s task to place each coin into the proper grade (aka range of conditions) with respect to the standards of the the coin’s series - but humans are neither identical to one another nor perfectly consistent. And sometimes a coin seems to lie on the line between one grade and the next, or between a details grade and a net grade. Those coins will not necessarily, and perhaps should not, receive the same grade every time they are sent in, because even the most discerning and experienced humans will not always line up on them - and the end users, collectors, are also only human. If AI were to assign an identical grade to such a coin every time, that might be seen as problematic to those who lean towards the other of the two possible grades. Perhaps it is our treatment of assigned grades as objective in the marketplace, and not this sort of human variability, that is problematic.
A simple “formula” will not solve any of the above issues in any way that would be helpful for human graders, especially since any “formula” would simply be drawn from our own grading formulas that we have devised. If AI were to grade coins itself, it might be able to eliminate the problems associated with the human aspect of grading, but it is only as good as the inputs that it is given. It can only learn what an MS64 is based on our own ideas of what an MS64 is, and learn how to approach a certain series based on our (current) standards of how to approach that series.
Gobrecht's Engraved Mature Head Large Cent Model
https://www.instagram.com/rexrarities/?hl=en
All the AI I need for grading a coin is between my ears.
Great spirits have always encountered violent opposition from mediocre minds.-Albert Einstein
Apologies for the long delay in replying. Been doing some traveling!
I don't think consistency in photos would have led to more accurate grading by Numi. I took pains to make sure my photos had consistent lighting and it still led to inconsistent results at lower grades.
IMO, the next step to get AI grading off the ground would have to be a stronger visual AI model. Specifically, a model that can handle video input. Thankfully AI is advancing quickly. I'm excited for the day where I can get my hands on an AI model where I can pull up a camera and the AI can determine a coin's grade based on the user moving a camera around a coin. Telling me in real time what it thinks the grade is.
It is possible to have an AI define its grading formula in the sense that initially with Numi, I did ask it to draft up descriptive grading criteria based on the Sheldon Scale. However, I would stress that the AI is essentially hashing together all the info it was trained on which includes many opinions on the Sheldon Scale. It wasn't creating a new objective standard.
Just received my coins back from PCGS. 1951 PF64
1950 PF64CAM
Was hoping for a tiny bit better….🤔
@knovak1976 Thanks for following up!
Numi had guessed PR-68 for the 1950 and MS-65 for the 1951. Off on the 1950, but close with the 1951!
I agree, I would have guessed higher for the 1950