Any experts on statistics out there?

MrEureka

Just musing on grading and have a question/problem for you. And before I start, let's be clear that I'm not trying to represent the precise way that things work at any TPG. I'm just looking for some perspective. That said...

Let's assume that all coins are graded by three graders at the TPG. Let's also assume that the final grade is any grade on which at least two graders agree, and that all coins not getting "two of a kind" will be automatically resubmitted "in house" until a match and the final grade is established. (Assume the graders never recognize resubmissions.) Let's also assume that each individual grader will average 80% accuracy, with accuracy meaning that he matched the final grade, and that each grader will come within 1 point of the final grade 97% of the time. Also, for the purpose of this exercise, assume that there is no such thing as a "bodybag". Every coin gets a grade.

The question is, how often will a coin that receives a grade on one submission receive the same grade on the second submission?

DieClash

I believe from the "given" data that there is a linear equation that will give you the probability within 80-95-99 percent degrees of confidence as to the outcomes of x-number of multiple submissions. However, I had that class in college and since I don't perform statistical analysis for a living I can't answer your question.

Anybody with Minitab software would probably be able to provide the answer.

Assuming each grader is averaging 80% accuracy, I would hazard a guess that the subsequent submissions of the same coin will have a much greater than 80% chance of landing the same grade. In fact, I'd be surprised if it didn't receive the same grade at least 9 out of 10 times. JMO

Cheers!



EDITED: for spelling

MrEureka

I'd be surprised if it didn't receive the same grade at least 9 out of 10 times.

Me too. But the difference between 92% and 98% is all the world.

DNADave

Makes me want to say something about normal distributions and standard deviations....or a|b type scenarios.

In short............beats me!

jonathanb

<< <i>Let's also assume that each individual grader will average 80% accuracy, with accuracy meaning that he matched the final grade, and that each grader will come within 1 point of the final grade 97% of the time. >>

I think those two statements contradict each other. Suppose that 100 MS-64s were submitted. Based on the first clause, he would grade 80 of them as MS-64, right? And the other 20 of them as something else? But based on the second clause, he would only grade 3 of them as MS-63 or MS-65? So he'd grade 17 of them as lower than 63 or higher than 65? That doesn't sound like what you meant...

mojorizn

Way too many years and beers since Professor Rahman's 8 AM Statistical Analysis class back in '88. Besides, I can't find the sigma or delta keys on my laptop!

Mojo

DNADave

<< <i>

<< <i>Let's also assume that each individual grader will average 80% accuracy, with accuracy meaning that he matched the final grade, and that each grader will come within 1 point of the final grade 97% of the time. >>

I think those two statements contradict each other. Suppose that 100 MS-64s were submitted. Based on the first clause, he would grade 80 of them as MS-64, right? And the other 20 of them as something else? But based on the second clause, he would only grade 3 of them as MS-63 or MS-65? So he'd grade 17 of them as lower than 63 or higher than 65? That doesn't sound like what you meant... >>




no, they could be accurate but not precise.

If you think of precision and accuracy in archery.... ten arrows all touching each other is very precise, if all ten are in the bullseye, they would be accurate too.

So an 80% accurate grader would have a loose group around the "acutal grade" which may or may not be the correct grade. Just a thought.

MrEureka

I think those two statements contradict each other. Suppose that 100 MS-64s were submitted. Based on the first clause, he would grade 80 of them as MS-64, right?

How would you know the grades before they're graded?


based on the second clause, he would only grade 3 of them as MS-63 or MS-65? So he'd grade 17 of them as lower than 63 or higher than 65? That doesn't sound like what you meant...

On the average submission of 100 coins that miraculously all end up graded 64, our individual grader would grade 3 of them less than 63 or more than 65.

AnonimoGaddiano

<< <i>Let's assume that all coins are graded by three graders at the TPG. Let's also assume that the final grade is any grade on which at least two graders agree, and that all coins not getting "two of a kind" will be automatically resubmitted "in house" until a match and the final grade is established. (Assume the graders never recognize resubmissions.) Let's also assume that each individual grader will average 80% accuracy, with accuracy meaning that he matched the final grade, and that each grader will come within 1 point of the final grade 97% of the time. Also, for the purpose of this exercise, assume that there is no such thing as a "bodybag". Every coin gets a grade.

The question is, how often will a coin that receives a grade on one submission receive the same grade on the second submission? >>


Paragraph 1 promulgates a series of meaningless assumptions and percentages with respect to the question asked in paragraph 2.
Defining accuracy as above "accuracy meaning that he matched the final grade" is problematic with respect the your final question.

RWB

The question is, how often will a coin that receives a grade on one submission receive the same grade on the second submission?

80% - each submission is a unique event not causally connected to any other.

AnonimoGaddiano

<< <i> The question is, how often will a coin that receives a grade on one submission receive the same grade on the second submission?

80% - each submission is a unique event not causally connected to any other. >>



Not with the definition of accuracy supplied originally. Causality is not the issue nor is stats as defined previously, rather I think this becomes a mathematical logic issue.

Longacre

I got a "C" in statistics in college.

AnonimoGaddiano

A retired statistician purchased a brass Aladdin's lamp at an antique shop one day. Being very proud of his purchase, he cradled the lamp with one arm against his chest and began his walk home. He had only walked a block when he was startled by a belch of smoke from the lamp and the appearance of a magic genie.
"Hello kind sir," said the genie. "I am here to grant you three wishes. Since you have toiled your entire life with numbers to benefit people in many different professions, the only provision is that these wishes must also benefit others. To insure that this happens, those three lawyers walking on the other side of the street will each receive DOUBLE what you receive."

Now the statistician recalled some bad experiences with lawyers but was still excited and agreed to the conditions. The genie smiled gleefully and asked the statistician for his first wish. The statistician thought only for a second and responded,"I would like a brand new red Ferrari automobile." Poof! A sparkling red Ferrrari appeared. He then looked across the street and saw six red Ferraris pop up, two for each lawyer.

The genie smiled broadly and asked the statistician for his second wish. With very little thought the satistician said "I would like a million dollars." Poof! A million dollars appeared in a gilded suitcase. He quickly glanced across the street and saw that each of the three lawyers received two gilded suitcases containing a million bucks each.

By this time, the statistician was becoming somewhat angry because he thought the lawyers were receiving more than their fair share. The genie then admonished him that he had only one last wish and should think very carefully about what he wanted. The statistician painfully puzzled over his last wish for several minutes. He finally replied,"You know all my life I have always wanted to be an organ donor so I hereby wish the donation of ONE of my kidneys to the local hospital! Poof! A kidney was donated .........

RWB

Not with the definition of accuracy supplied originally.

Multiple definitions and assumptions seem to be part of the original problem, hence, the simple answer.

Type2

I dont remember.

jonathanb

Ack, sorry. I misread the original conditions. To be revised...

RWB

Given the odds as set out in the original problem, a coin has a 99.29% chance of receiving the same grade on the second submission as it does on the first.

Does that presume there is no profit motive ?

jonathanb

<< <i>On the average submission of 100 coins that miraculously all end up graded 64, our individual grader would grade 3 of them less than 63 or more than 65. >>

Ah, ok, that is what you said the first time. Sorry about that!



<< <i>How would you know the grades before they're graded? >>

You wouldn't, but we're talking about statistics here, and the whole point of statistics is that you're supposed to get the same answer in either case. So let's assume something easy, since we know it won't change the answer.

In this scenario, each grader is 80% accurate. That means that the chance that all three of them will be accurate at the same time is (80/100) * (80/100) * (80/100), or 51.2% of the time.

Additionally, there is a (80/100) * (80/100) * (20/100) = 12.8% chance that the first two will be accurate, and the third won't be. Since there are three ways that two can be accurate and one not, that makes an additional 38.4% of getting the "right" grade on the first try.

In total, there is therefore an 89.6% chance that at least two of the graders will agree on the "right" grade on the first try.

I'm going to assume that when a grader is wrong, he has an equal chance of being too high or too low. That means there is a (97/100) * (20/100) / 2 = 9.7% chance that any grader is 1 point low, and a 9.7% chance that the same grader is 1 point too high.

That means that there is a (9.7/100) * (9.7/100) * (9.7/100) = 0.09% chance (less than 1/1000) that all three of them are 1 point too low at the same time, and an equal chance that they are all one point too high.

As above, there is a (9.7/100) * (9.7/100) * (90.3/100) * 3 = 2.5% chance that exactly two of the three agree on a point too low, while the third says something else.

That makes a total of 2.6% chance that the coin will end up with a grade that is a point low, and a similar 2.6% chance that it will end up with a grade that is a point too high.

It's not worth considering what happens if they are all two points too low or two points too high -- the odds are teensy.

According to the problem, grading is finalized when all three agree. That means that there is a total of 89.6 % + 2.6% + 2.6% = 94.8% chance that all three will agree right off the bat (combining the odds that they are "right" with the much smaller odds that they are all one point high or low), and a 5.2% chance that they have to try it again.

BUT, if they try again, the odds of them agreeing will be the same the next time also. They'll agree eventually. The odds that they will agree on the "right" grade are 89.6% + (89.6% * 5.2%) + (lots of smaller terms). In all, the odds are 94.8% that they will agree on the "right" grade, and only 2.6% each that they will be one point high or one point low.

We can now answer the original question:



<< <i>The question is, how often will a coin that receives a grade on one submission receive the same grade on the second submission? >>

The odds that the coin receives the "right" grade both times is (94.8%) * (94.8%), or 89.9%. The odds that it is the same one point too high or two low both times is (2.6%) * (2.6%) , or 0.07%.

Given the odds as set out in the original problem, a coin has a 89.9% + 0.07% + 0.07% = 90% chance of receiving the same grade on the second submission as it does on the first.

badger

O.K., I took a break from cleaning the kitchen. Here are the results. I'll post the work in a separate reply.

How often will a coin that receives a grade on one submission receive the same grade on the second?
Answer: Depends on if the coin received the Correct Grade or Incorrect grade on the first submission.

Correct Grade First Submission: 94.1% Chance of staying the same
Higher or Lower than Correct on First Submission: 2.9% Chance of staying the same


However, if you submit the coin in the holder, PCGS will not downgrade. So, it changes the results:

Correct Grade First Submission: 97.1% Chance of staying the same
Lower than Correct on First Submission: 2.9% Chance of staying the same
Higher than Correct on First Submission: 100% Chance of staying the same

I had to assume that the graders are within one grade (higher or lower) of correct all the time. The 97% assumption was getting too complicated and probably not material.

I arbitrarily set the likelihood of a miss to the high grade to be the same as a miss to the lower grade - 10%, 80%, 10%. I can rerun with different assumption if needed.

Also, had to correct for the No Grade, since 4.8% of the time the graders do not agree and restart the internal process.

Findings:
- Resubmitting in the holder is a good deal in this scenario.
- You should only resubmit raw if your grading skills are accurate.

I will post the details, but if anyone wants the Lotus file I would be happy for someone to check the work.
Dave

sinin1

nice work Jonathon


but I disagree because grading is not a digital exercise (61,62,63,...) but more of a linear one (61.1,61.2,61.3,....) expressed rounded to the next grading point on their scale (AU50,AU53,AU55,AU58,MS60,MS61)

which adds to the complexity of the question - and then the finalizer who can veto the quorum?


I think that to get your answer Andy, you should take 100 randomly graded coins and resubmit with a bulk submission

and do this every month
until you get a satisfactory answer to your question




For an example, if you had a nice buffalo nickel that maybe graded MS63 but is worth $5K more in MS64, would it be worth it to you to resubmit upto ??(lets say 20) times
if you had any chance of success


surely you would have more success if the coin is truly a MS63.9 on PCGS scale than a MS63.1

roadrunner

In this scenario, each grader is 80% accurate. That means that the chance that all three of them will be accurate at the same time is (80/100) * (80/100) * (80/100), or 51.2% of the time.

That almost seems to be in-line with what I see for the repeatabiilty of 19th century MS silver type coins. But shouldn't the more 80% accurate graders being added improve one's odds of getting the right grade? Wouldn't 20 graders at 80% accuracy tend to give a better and more consistent result than a single grader?

On the flip side of 80% accuracy is 20% inaccuracy. So the odds of all 3 being inaccurate is (.2)(.2)(.2) = 99.2%. I'd be tinkled pink if the typical coin came back the same grade >80% of the time.

I was having a discussion with a friend recently about MS70 AGE's. If you took all the AGE's graded MS70 and sent them all back for a raw regrading, what % of those would you expect to grade MS70 again? I felt the percentage would be much closer to the raw percentage of random coins that grade MS70 the first time through.

roadrunner

badger

Showing the work:

MrEureka

Defining accuracy as above "accuracy meaning that he matched the final grade" is problematic with respect the your final question.


By "final grade", I mean the final grade on a given submission event. In other words, the "final grade" is the grade assigned by the grading service. And if a coin is submitted twice, it will have two final grades, which may or may not be the same grade.

segoja

Well my statistics class was....here is the book, answer the following questions at the end of each chapter. Then in 3 weeks at our next class we'll cover any quesitons in the first hour of class.....no lie!!!! MBA program

So the correct answer is: each coin will get the same grade each time because the graders at PCGS are perfect!!! Maybe across the street the exmaple is more appropriate

MidLifeCrisis

Wow. I actually read through this thread. Now I have a headache.

TomB

I think some members might be reading your question a bit different than what you have actually written. One of those members might be me, but I will lay out what I have done here.

1) Each grader will match the grade assigned on the slab 80% of the time.
2) In-house iterations for grading purposes do not count in this 80% since, by definition, the 80% level is with respect to the grade assigned on the slab.
3) A grade of one point off, two points off or sixty points off of the final grade does not matter since each is worth an equal weight for the 20% of the time that each grader does not match the grade assigned on the slab.
4) Given the above points, we cannot have the scenario of each grader grading a coin one point too low or one point too high with respect to any individual, present grading event because the definition of an assigned grade is that at least two graders agree on the grade.

The above yields-

(0.8)*(0.8)*(0.8)=0.512 or 51.2% of the time all three graders will agree with the final grade assigned on the slab.
(0.8)*(0.8)*(0.2)=0.128 or 12.8% of the time that graders one and two will be accurate with respect to the final grade assigned on the slab.
(0.8)*(0.2)*(0.8)=0.128 or 12.8% of the time that graders one and three will be accurate with respect to the final grade assigned on the slab.
(0.2)*(0.8)*(0.8)=0.128 or 12.8% of the time that graders two and three will be accurate with respect to the final grade assigned on the slab.

This gives an accuracy rate of .512+.128+.128+.128 or 0.896 (89.6%) for any grading event. If I have read the question and parameters correctly, those who are assigning an accuracy rate of greater than 89.6% are doing so in error since you cannot by definition have two or more graders grade a coin too low as long as their independent grade assigments agree with on another since their agreement will define the grade and they will then be accurate in this event.

Of course, this does not address the possibility of resubmission, but I would think that the resubmission percentage would also be an 89.6% probability of matching the first grade as long as the same graders and other parameters are in place.

badger

Tomb,

I am trying to see if we can get our numbers to match.

I believe that there are outcomes where all three graders would not agree. That would cause an internal recycle until a grade was agreed. Each recycle has the same probabilities as the inital grading. The overall probabilities need to be normalized for the chance of a No Grade.

Multiplying the probabilities is approach to determine the chance of the outcome. I needed a flow diagram of all outcomes to make sure that I did not leave out potential outcomes.

What do you think about how to handle the No Grade?
Dave

BillJones

From the way I interpret the question each submission for grading would be an independent event. Therefore the probably of getting the same grade would be 80%, and the probably to getting within one point would be 97%.

If you are looking at it from the concept of a series of submissions, the picture is difference. If you were to looking for the odds of getting the same grade from two submissions, the probability would be 64% (80% x 80%) and the probably if getting the same grade within a point would be 94% (97% x 97%). The odds grow longer with additional submissions. It would be only 51% if you sent the coin in three times. (80% x 80% x 80%) and within a point 91% (97% x 97% x 97%).

TomB

Dave, internal iterations where all three graders come up with different grades and the coin is recycled into the system do not count or matter for this question as it is posed. Additionally, the no grade possibility does not count, either, since the initial question stated that the "final grade" is the outcome and in a no grade (or genuine holder) scenario there is no final grade and therefore this event falls outside of the parameters of the initial question and can be assumed to not occur.

badger

Right, No grade is not a result, but it does change the percentages of outcome for agreed results.

Dave

Type2

I just send it in and get it back with a nice pice of plastic and a blue label with a number on it.

MrEureka

internal iterations where all three graders come up with different grades and the coin is recycled into the system do not count or matter for this question as it is posed.

It seems to me that a coin that does not get a "match" or "final grade" on a single submission is, once it receives a final grade, less likely to match an earlier final grade. In practice, this means that coins that are more difficult to grade are less likely to get the same result when cracked and resubmitted. And I think this will impact the answer to the statistical question, but I'm not sure.

DNADave

<< <i> internal iterations where all three graders come up with different grades and the coin is recycled into the system do not count or matter for this question as it is posed.

It seems to me that a coin that does not get a "match" or "final grade" on a single submission is, once it receives a final grade, less likely to match an earlier final grade. In practice, this means that coins that are more difficult to grade are less likely to get the same result when cracked and resubmitted. And I think this will impact the answer to the statistical question, but I'm not sure. >>



yes, its a conditional probability. What is the probability of "B" happening, given that "A" has already happened.

...I think.

CoinJunkie

Mathematically speaking, the problem as posed is ill-formulated. I'd work on clarifying it, except that I think the
answer would be meaningless (for a number of reasons), even if the problem were made solvable.

lathmach

The following formula gives you the answer.
r = (c²+4h²)/(8h)




















Just not the answer to this question.
The formula is for using chordal drop in determining a radius.
But it's a real good formula, anyway.
Go ahead, use it all you want to.

Ray

TwoSides2aCoin

<< <i>The question is, how often will a coin that receives a grade on one submission receive the same grade on the second submission? >>


50% of the time it will come back the same grade and 100% of the time it will cost money to prove it.
These figures may vary slightly with each submitter.

LostSisler

My answer is, with so many humans involved, there will never be an actual consensus on grading.
Besides, setting the standards would require even more humans, and even when they tried, and printed the standards, many did not agree.
Even if a computer were to do it, the answers may be repeatable but the answers wouldn't matter if still no-one agreed with the standard!
I contend that there are too many unknown truths in the equation to have an answer.
I suggest asking Mark Salzburg or Ron Guth to see if any of this data is tracked in their systems and if they'd let anyone see it.
Just my two cents...