On the front page of The Australian‘s print edition today a headline reads:
ANU pips Stanford
The internet headline was a little less counter-intuitive, but the story the same. It’s a reference to the 2008 Times Higher Education rankings which puts the ANU at 16th in the world, and Stanford at 17th.
Now the ANU is a perfectly respectable university. But the THE rankings have been widely criticised, and results like this will not help the case for the defence.
The biggest criticism of the THE is their heavy reliance on subjective measures. 40% of the ranking is based on academic peer review, which is done by emailing tens of thousands of academics with an internet survey. The response rate is typically poor, and the response quality doubtful. One potential benefit of rankings is that use of objective information can correct impressionistic views of universities, but this method tends to reinforce the latter. The Shanghai Jiao Tong rankings use purely objective measures (though the weightings are subjective; I don’t think there is any way to make these objective). But while the THE rankings are dubious on this measure, this isn’t what’s dividing the ANU and Stanford. They both get the maximum score of 100.
On citations per staff member, an objective measure, Stanford is well ahead of the ANU (100 to 74). The only ‘academic’ measure on which they fall well behind is ‘international faculty’, presumably those with degrees from other countries (or born in other countries?), where the ANU is well ahead, 99 to 26. Now I am not sure where their Australian data came from – I have never seen it published. But whatever its basis this seems to me to be a near-worthless indicator. Seven of the ten universities with perfect 100 scores are in small countries, suggesting that the key variable here is the size of the domestic academic market. Though one American university (CIT) does get a 100, the size and strength of the US higher education system will work against American universities needing to import staff.
The THE rankings, unlike the Shanghai Jiao Tong, do try to measure student-related factors. One of these is international students, where Australian universities do well, because they are financially desperate and have aggressively recruited fee-paying international students. That these students will come is a positive market sign. But for other universities that are not aggressively recruiting, their lower numbers are not a sign of failure. Sheer numbers should probably be adjusted for price as another market signal. The ANU is slightly ahead of Stanford on international students, 91 to 87.
The THE have an employer survey conducted like the academic survey, but to me with even more suspect results. Academics are at least likely to have a reasonable sense of how other universities perform in their own speciality, but few employers are likely to have graduate staff that enable global comparisons. Stanford is ahead of the ANU on this one, 100 to 93.
What seems to have sealed the ANU’s narrow victory – 92 to 91.2 overall – is their student:staff score, 82 to 67. But this seems to me to just be a mistake; on the data I can find the ANU has about twice as many students per staff member as Stanford, as you would expect when you compare a rich private university with one mired in dysfunctional price controls and quotas.
In the Australian context, the ANU is a good university. But there is a reason why on the Shanghai Jiao Tong rankings it is 59 and Stanford is 2, which is that Stanford is a much more successful research university. It’s very hard to make objective comparisons on the student experience, but there is nothing in this data to suggest the ANU is better.
Status anxieties are acute in academia, and so universities will use these rankings publicly while critiquing them privately. On research, the Shanghai Jiao Tong provides useful information. But for potential undergraduates trying to decide where to study neither ranking is very helpful. Research performance is of doubtful relevance, there are no internationally comparable and widely used student satisfaction tests, and graduate outcomes need to be controlled for student quality on university entry and the effects of local labour markets.
Andrew Norton 
Andrew, you’re right about neither ranking being very helpful to potential students. The SJT ranking focuses, as you mention, purely on research output and quality, divided by full-time (or equivalent) staff at the institution. But the SJT rankings could be made relevant for potential students if enrollments per staff member were considered. I’m assuming, of course, that staff:student ratios are relevant for the quality of education a student will receive.
LikeLike
I think the THES survey is really trying to shorten its life expectancy with data like this. Whatever survey they’re using, they need to fix it before becoming a bigger joke than they already are.
Incidentally, I disagree with this statement depending on how you want to define specialty:
“Academics are at least likely to have a reasonable sense of how other universities perform in their own speciality”
If we’re talking about categories as broad as those typically used in these surveys (e.g., economics, psychology, engineering…), I doubt most people have much idea at all, let alone have the ability to perform any meaningful ranking. For example, I doubt I could name more than 4 people working in a single department in the same kind of department I’m in for any of the universities in my state, let alone Australia and the world (and I’d then have to go and look up even those people to know how well they have been doing in the past few years). So the best I can really do is give you figures based on heresay (e.g.,”I heard Melbourne XX was good” and “Monash YYY was bad”) and I doubt most other people are much different (ERA bureaucrats excluded). I would think that the best knowledge people have is in their particular area (I could probably name, list and order the 10 best people in my speciality area for example), but that doesn’t tell you much about particular departments.
LikeLike
And there we go again, someone else believes a university should be ranked on it’s “research” not on it’s ability to teach.
LikeLike
“And there we go again, someone else believes a university should be ranked on it’s “research” not on it’s ability to teach”
If you want teaching rankings, you can look at the CEQ, the good universities guide and a few other measures. The problem with them is that their reliability and validity is even worse than the research ones (they’re almost completely useless for individual courses — it’s actually very hard to measure “teaching” — and measuring it leads to all sorts of weird consequences in itself as people will compete to get the best teaching marks using dirty methods). With American universities, you could use the USA Today measures which are more quantifiable.
LikeLike
conrad
I suspect the average student picks their University the way I did, the closest to home, or if they couldn’t get in, the closest that will accept them. If that is the case are any of the rankings worth jack.
I wonder, if funding was reverse, that is poor universities received more money to increase their rating, would the rating change?
LikeLike
“I suspect the average student picks their University the way I did, the closest to home, or if they couldn’t get in, the closest that will accept them.”
That’s true only to some extent (mainly competition between lower rung universities — where I work, for example, we certainly have catchment areas where most students come from). Students often choose on name — that’s why Melbourne, for example, has the highest cut-off scores in Melbourne and VUT and the Catholic Uni the lowest. This is true even on essentially identical courses that are kept at a decent standard by outside organizations. It’s even more important for full paying students, many of whom will only have heard about you via these rankings. When I worked in China, for example, some students would simply apply based on these rankings, and take the highest offer. Alternatively, good marketing (and good courses that get known) mean that universities not on these lists can still compete in some areas for these students. Again, where I work, I think we have almost the highest proportion of OS students of any university in Melbourne, but we are not on any top lists (although the administration seems desperate to get there).
“I wonder, if funding was reversed”
Funding is identical for students in Australia, so there’s no reverse. The fact Melbourne gets more money than, say, Latrobe, is because they are good at getting various sources of competitive funding.
LikeLike
The big underlieing/underlying factor is that Murdoch owns the Times, his family is Australian and has/had money invested in Melbourne Uni and MU Private. If he can boost Australia and its universities (and his alma matter Oxford and his father’s LSE), it is in his vested interest.
“that’s why Melbourne, for example, has the highest cut-off scores in Melbourne and VUT and the Catholic Uni the lowest”
Check your facts before getting online Conrad.
I wont descend into insults as you are prone to when your brain fails, but see the the http://www.vtac.edu.au website. I am willing to agree with you that the South Eastern suburbs of Melbourne are possibly the the largest catchment and this contributes to Monash’s dominance.
Arts degree cut off scores, for example, often the largest course and faculty on a campus:
2008
Monash Clayton Arts 85.80
Melbourne Parkville Arts 85
Monash Clayton Arts (Global) 87.5
LikeLike
In other words, Monash is (probably) now the university the highest standards of entry in Australia at its main campus.
This will increase with the unpopularity of the Melbourne model. Monash’s continuation of double degrees and speeding up of student transition to research will prove effective with student’s wishing to enjoy student life but not wanting to be ripped off over 7 years or more at Melbourne where many cannot even find, let alone afford a place to live unless it is under a bridge.
I see Rudd is reducing immigration to Australia on employment grounds, but he wont reduce student visa’s will he?
LikeLike
Nice to see you’re alive Parkos — from a statement a few months ago, I believe ACU lets anyone into many courses now, so you can’t go lower than that (am I missing a university in Melbourne here — is there somewhere worse than ACU or VUT?)
“I am willing to agree with you that the South Eastern suburbs of Melbourne are possibly the the largest catchment and this contributes to Monash’s dominance”
Actually, both Deakin and Swinburne are located in similar sorts of catchment areas to Monash (and perhaps ACU also), but they have lower cut-offs for similar courses, which shows it isn’t just area. In addition, Monash is pain to get to from many places so they are especially handicapped by that (and Clayton is not exactly fun city either compared to Fitzroy or Hawthorn). Luckily their brand-name trumps this, unlike La Trobe, who I imagine are going bankrupt at least in part because of competition based on location. At least they have nice ducks on campus.
LikeLike
Sorry Parkos,
you’re correct on the upside — I was looking at the glass half-empty (what are the worst places), but you’re looking at it from the half-full perspective (what are the best places).
LikeLike
Parkos – It’s not actually possible to invest in Australian public universities themselves, and MU Private – which was closed down a couple of years ago – never had any shareholders other than the University of Melbourne.
LikeLike
Looking at the froth that has spilled over the glass..
A private US style liberal arts college out near Castlemaine/Daylesford might be a way to bring the concept of a university town to Australia and fritter away some cash/time. However, it would need a great public transport link to Melbourne like Berkley and Standford have to San Francisco.
At the middle level of the glass, I predict LaTrobe and the northern suburbs will be the next growth area/knowledge corridor. Particularly when the sea level rises.
Great African restaurants near VUT Footscray.
LikeLike
Point taken, yet the line between a private and a public university gets bit blurry with companies like Melbourne IT.
There is some history of Murdoch putting Monash back in the ranks, from Wikipedia:
” Bean’s influence grew as the war progressed and he lobbied (along with Keith Murdoch, father of Rupert Murdoch)unsuccessfully against the appointment of General John Monash to the command of the Australian Corps in 1918. He disliked Monash for not fitting his ideal of Australian manhood (Monash was of Jewish background) and his promotion of his men — he had earned Monash’s wrath for failing to publicise his brigade at Anzac — which Bean viewed as a penchant for self-promotion and wrote in his diary, “We do not want Australia represented by men mainly because of their ability, natural and inborn in Jews, to push themselves.” (embarrassingly anti-semitic today, it was a common prejudice at the time)”
LikeLike