Social Science Statistics Blog

Interesting note. I'd like to read the paper but as a windows user I don't have good access to post script files and the PDF link gives a 404 error.

Could you update the link?

Sorry, Steve. It should work now.

It's an interesting idea, sort of like grading the whole experience at school on a curve but in a more sophisticated way. Not bad! I think I would like it.

My school addressed that by giving inverted grades (1 was the best and 5 was failing) and then not calculating GPAs, so you had no official GPA to put on a transcript. In a way, that led us to be freer in selecting hard courses, I think, for we'd get grades but they wouldn't make (much of) a difference in the final diploma or transcript, at least over a fairly wide range of grades.

Interestingly, they somehow figured out who should be in Phi Beta Kappa. I see they've since switched to a standard scheme and now calculate GPAs.

That's interesting, Bill. My undergraduate school didn't officially calculate GPA, but I remember that mainly as an annoyance, since grad school applications asked for GPA so that just meant you had to calculate it yourself. That would be even harder if the grades are inverted!

FYI, it appears that the measure failed to pass (though the vote was close: 34-31). Apparently most of the people who disliked it were students who were afraid they would be hurt in admissions to grad school, med school, etc., relative to other students in the country. I'm not sure they would be, since they could also report the "standard" GPA, but I do understand the concern.

Amy, I seem to recall being told that the inverted scale was a Princeton invention. I have no idea if that is or even was true. When doing a quick search, I did find Edward Felten's How Much Information is Conveyed by a Princeton Grade?, which is another mathematical analysis of grading systems, in case you're interested.

Very interesting paper, Amy.

By-department stratification is certainly annoying; at least those of us who take hard majors and classes seem to think so. On the other hand, those biases can be discounted by prospective employers or application reviewers. (e.g. how would you rate the work ethic credibility of a literature 4.0 vs. a physics 4.0. one of these properties may be more informative than the other. perhaps one could even measure this informativeness.) This is bad for departments with a low amount information about their difficulty (is "symbolic systems" fluffly semiotics or highly technical formal logic?), but for many areas I think a certain amount of learning has happened. I wonder how well the intuitive judgments of employers would match a statistical estimation model of aptitude that discounts course difficulty in a manner similar to this one. There is certainly a huge incentive for employers to figure out the underlying difficulty model, and decisions and learning feedback happen all the time through the hiring and employment process.

I like the Felton paper: it's the straightforward thing to do from an information-theoretic perspective, but still quite interesting to see the results. It also throws into stark relief some of the various important factors that play into assigning grades: informativeness is certainly one, but is not the only important thing.

And I definitely agree, Brendan, that employers attempt to informally weight applicants by difficulty of major. This is of limited use, however, not just for weird majors like "Symbolic Systems" that they may not have heard of, but also for classes within majors. For instance, getting a B+ in Philosophy 160 (an extremely difficult logic class at Stanford, for you non-Stanfordites out there) isn't half bad -- at least it wasn't when I was there -- but nobody outside of Stanford could know that; and simply knowing that it is a philosophy class would be rather uninformative.

So I agree with you that people already try to do this informal discouting. I'm quite curious how effective that is; if any school used the AI then, as you say, you could do an empirical comparison of the two. I'd guess that it's a decent first-pass approximation, but you miss a whole lot of very relevant information by virtue of not understanding all the arcane details of each class in each individual university.

Here's a question: what difference does it make? I recall hearing of one company trying to correlate GPA with performance on the job in that company, and there was supposedly no correlation. I wish I had the study to know something substantive; I guess I'm confident R^2 wasn't identically zero, but I don't know what it was.

Are there any public studies that show the relationship of GPA to performance in graduate school or on the job?

The Tennessee Higher Education Commission (THEC) has recommended lowering the minimum grade point average in order for college students to keep their state lottery scholarships.
Earlier this year, Gov. Phil Bredesen appointed a committee, led by Shelby County Mayor and THEC member AC Wharton Jr. to review the lottery scholarship program after THEC issued a report showing three out of four students were losing the scholarship before they graduate because their grades did not meet the minimum requirement.
At present in order to qualify for a scholarship, a student must have a high school GPA of 3.0, or score 21 on the ACT college entrance test, and to keep it, students must be enrolled full time, have a college GPA of at least 2.75 after their freshman year and a 3.0 GPA for the subsequent years.
The Higher Education Commission approved the committees report last week and has recommended to the lawmakers to drop the minimum GPA from 3 to 2.75 for the subsequent years of college.
The commission proposes to pay for these recommendations with some of the $33 million in lottery surplus money projected for this year. Dropping the minimum GPA would require an estimated $7 million and would result in an additional 1,600 students retaining their scholarships.
Another plan, is to establish a statewide early commitment program that would provide tuition benefit to low-income students who graduate from high school with a college preparatory curriculum, this would cost $4 million and would benefit students regardless of their high school GPA.
The commission has also recommended increasing the amount of the Hope Scholarships for students who take college prep courses in high school. The last increment was made last fall. Currently students attending a four-year approved school can receive a scholarship of $3800 per year and students attending a two-year approved school can receive $1900 per year.

Bill, that's a good question. I did a quick google search to see what kinds of studies there were trying to correlate GPA with later performance. Based on this very superficial look, I'd characterise the conclusion as highly variable depending on the nature of the later performance being assessed (e.g., different kinds of job vs. different sorts of grad school vs. later salaries) but that many studies find a low but significant effect of GPA. But again, it seems to be quite dependent on the performance: this is sensible, since one wouldn't expect GPA to predict (say) waitressing ability very well - things like enthusiasm, people skills, and organization under pressure would do that.

The real question to me is whether and to what extent exchanging GPA for AI (or something else) would change this. Would AI have greater predictive power? To the extent that it better represents some underlying "achievement" or "ability" factor that also matters in some jobs, one would imagine so... but does it really do this?

Also, I can think of many reasons that GPA might have a low(er) correlation than you might expect, for reasons that have nothing to do with the effectiveness of GPA at measuring important aspects of an individual's performance. For instance, if grad schools selectively pick people in part based on their undergraduate GPA, then they are effectively lopping off a lot of the variation in GPA: if everyone has an undergrad GPA of 3.9 or higher, then knowing their GPA might not tell you much about their performance. That doesn't necessarily mean that GPA doesn't measure something interesting, of course.

Amy, you raise interesting questions. I'm mostly interested because, while it seems to make sense that grades make a difference, at least some of the time, some of the data seems to indicate that GPA is of little predictive power. If that's the case, it seems like a shame (and maybe even damaging) to make decisions based on GPA, but it would be nice to know how to replace it in places where it has been used to make decisions.

I'll buy into the lower utility of GPA in positions where people's main (and not inconsequential) contribution is serving other people directly. The company I hinted at above had a significant number of so-called knowledge workers, though, and, while I can't remember for sure, I think their study may have been limited to folks in that type of work.

I suspect one problem is the variability of GPA: I hazard a guess that someone who gets a 3.0 at a very challenging school may have gotten a noticeably higher GPA at a less challenging school.

I sense another problem may be the selection you mentioned: grad schools probably don't accept many 2.0 students, so correlation between an undergraduate GPA and success at getting a PhD might be measured and thus valid only over the range from 3.5 to 4.0 (or whatever the range is). Even if there is low correlation over that range, it's conceivable that there would be significant correlation between UG GPA and PhD completion over GPA ranges of 2.0 to 4.0. Without data, though, it's conceivable (if unimaginable :-) ? ) that the lack of correlation applies over any GPA range.

It would be interesting to know some answers. I suspect ETS has some data for predicting performance in the first year(s) of college based on SATs and high school GPAs. I wonder if the SHRM has data for employment. I can't think of another umbrella organization that comes close to covering the entire field of employment as ETS comes close to covering the entire field of college entrance in the USA. Oops: I just checked the ACT site out of fairness when thinking about the SAT, and I discovered they also do employment skills testing.

I definitely think that making hiring decisions entirely (or even mostly) on the basis of GPA is probably in most cases a stupid thing to do. But in addition to the questions you raise about how to either improve the utility of GPA, or replace it with something better, there are a couple of other things we should keep in mind. (With the obvious caveat, of course, that I'm speaking mainly from intuition and personal experience here, since I don't know a great deal about the data in this area...)

1. I tend to be of the general philosophy that more information is usually better. Even if GPA isn't very predictive on its own, it could probably be quite informative when combined with other information... for instance, if a candidate has a low GPA but you can see that they were busy starting their own business or spending 30 hr/wk in lab publishing papers, then ignoring the GPA is sensible. But if a candidate has a low GPA for no apparent reason then that may very well be informative (about their work habits, if nothing else). In short, even if GPA isn't very predictive taken individually, as part of a package it may very well be: and studies that simply look at it in isolation may thereby be missing some of its true value.

2) As you say, there are many statistical factors that might make GPA less predictive than otherwise: selective admissions that artificially cut the bottom (or top) off of an applicant pool, and immense variability between schools, both in student body makeup and in how grades are calculated. One reason I like the AI is it seems to offer a way to overcome at least some of those factors (by dealing with the variability within a school in how grades are assigned). It would be really cool if you could come up with some kind of AI-like measure that would work across different schools: but the AI itself wouldn't work, since it is able to calculate the "weightings" by comparing performance of the same students in many different classes, and there are almost certainly not enough transfer students to do the same across colleges.

3) Whenever this topic comes up, there are inevitably a lot of people who opine that GPA is totally useless and should be eliminated entirely. I have some sympathy for that perspective: I found that, ironically, I worked a lot harder on classes once I got into grad school and the exact grade didn't matter, but what I learned did. And, yes, a lot of a grade does reflect things like doing the busywork, showing up, etc. But those things matter in the real world, too. And a good class -- which, ironically, usually ends up being one that grades the hardest -- will push you to think and do far more than simply memorize and regurgitate.

I guess what I'm saying is that, a priori, I'm inclined to think that GPA does reflect something useful, though it could certainly be improved; and certainly many other things matter as well. As for exactly what and how and when, this is where data (and some expertise on this topic, because I'm sure there is scads of data out there) would be very handy...

Re: the issue of where you might get data -- I second your thoughts about ETS and SHRM. I was thinking the military might keep records on that sort of thing, but it might be uninformative for many of the reasons we discussed: more restricted applicant pool, and they deliberately funnel people into different specialties (and as officers or not) based on (I think) GPA and ASVAB results. My quick google searches revealed mainly studies that were looking at a particular graduate program or employer, which is interesting, but of more limited utility.

An employer, or grad school, has to take the rigor of the major and the reputation of the institution to interpret the GPA. There is a similar problem ranking wines. A poor Petrus is better drinking than most of the output of Napa Valley, or Bordeaux for that matter.

One Australian guide to wines graded wineries on a scale of 1 to 7 and each wine and vintage on a scale of 1 to 7. This was surprisingly useful and easy to calibrate. Most wine ranking scales run from 0 to 100, generally correlated with sugar content (brix) these days. The difference between an 88 and an 89 is generally incomprehensible. A 6 from a grade 3 winery might, at times, be more appealing than a 5 at a grade 2 winery, because grade 2 and grade 3 wineries are generally trying to do different things.