On the front page of The Australian‘s print edition today a headline reads:
ANU pips Stanford
The internet headline was a little less counter-intuitive, but the story the same. It’s a reference to the 2008 Times Higher Education rankings which puts the ANU at 16th in the world, and Stanford at 17th.
Now the ANU is a perfectly respectable university. But the THE rankings have been widely criticised, and results like this will not help the case for the defence.
The biggest criticism of the THE is their heavy reliance on subjective measures. 40% of the ranking is based on academic peer review, which is done by emailing tens of thousands of academics with an internet survey. The response rate is typically poor, and the response quality doubtful. One potential benefit of rankings is that use of objective information can correct impressionistic views of universities, but this method tends to reinforce the latter. The Shanghai Jiao Tong rankings use purely objective measures (though the weightings are subjective; I don’t think there is any way to make these objective). But while the THE rankings are dubious on this measure, this isn’t what’s dividing the ANU and Stanford. They both get the maximum score of 100.
On citations per staff member, an objective measure, Stanford is well ahead of the ANU (100 to 74). The only ‘academic’ measure on which they fall well behind is ‘international faculty’, presumably those with degrees from other countries (or born in other countries?), where the ANU is well ahead, 99 to 26. Now I am not sure where their Australian data came from – I have never seen it published. But whatever its basis this seems to me to be a near-worthless indicator. Seven of the ten universities with perfect 100 scores are in small countries, suggesting that the key variable here is the size of the domestic academic market. Though one American university (CIT) does get a 100, the size and strength of the US higher education system will work against American universities needing to import staff.
The THE rankings, unlike the Shanghai Jiao Tong, do try to measure student-related factors. One of these is international students, where Australian universities do well, because they are financially desperate and have aggressively recruited fee-paying international students. That these students will come is a positive market sign. But for other universities that are not aggressively recruiting, their lower numbers are not a sign of failure. Sheer numbers should probably be adjusted for price as another market signal. The ANU is slightly ahead of Stanford on international students, 91 to 87.
The THE have an employer survey conducted like the academic survey, but to me with even more suspect results. Academics are at least likely to have a reasonable sense of how other universities perform in their own speciality, but few employers are likely to have graduate staff that enable global comparisons. Stanford is ahead of the ANU on this one, 100 to 93.
What seems to have sealed the ANU’s narrow victory – 92 to 91.2 overall – is their student:staff score, 82 to 67. But this seems to me to just be a mistake; on the data I can find the ANU has about twice as many students per staff member as Stanford, as you would expect when you compare a rich private university with one mired in dysfunctional price controls and quotas.
In the Australian context, the ANU is a good university. But there is a reason why on the Shanghai Jiao Tong rankings it is 59 and Stanford is 2, which is that Stanford is a much more successful research university. It’s very hard to make objective comparisons on the student experience, but there is nothing in this data to suggest the ANU is better.
Status anxieties are acute in academia, and so universities will use these rankings publicly while critiquing them privately. On research, the Shanghai Jiao Tong provides useful information. But for potential undergraduates trying to decide where to study neither ranking is very helpful. Research performance is of doubtful relevance, there are no internationally comparable and widely used student satisfaction tests, and graduate outcomes need to be controlled for student quality on university entry and the effects of local labour markets.