The people who write Graduate Careers Australia’s starting salaries report must love their time series of graduate salaries as a percentage of average weekly earnings, because they keep highlighting it in their report and in their media release, even though they are coming close to admitting is is meaningless statistical junk.
I can see why they want to keep it – it goes back nearly 30 years, to 1977 (the data in the report released yesterday is for people employed in early 2006). On surprisingly few topics do we have consistent data going back that far. And for people considering the costs and benefits of university study, it is useful to know their likely earnings compared to the alternatives. But as the Graduate Salaries 2006 report says:
…it is important to note that average weekly earnings may be positively affected over time as more and more graduates enter the workforce. As their careers progress their salaries grow, overall average weekly earnings are pushed up.
The only thing wrong with that is the ‘may’. They have a table showing full-time workers with a diploma and above going from 19.7% in 1998 to 27.8% in 2006. Using only bachelor and above and all workers, ABS Education and Work shows an increase from 14.5% in 1994 to 23.9% in 2006. I’m not sure what proportion of workers had degrees in 1977, but we’ve gone from graduates having a small impact on male average weekly earnings to a large impact – nearly a quarter of all earners. As bachelor degree graduates typically earn half as much again as people with Year 12 qualifications only, the statistical effect is not trivial.
This isn’t the only problem. Right from the start, they compared median starting salaries with average weekly earnings. High-income earners mean that average income is usually higher than median income, so we are not comparing like with like. The starting salary time series is for graduates under 25, yet the workforce is ageing, with income levels that incorporate an increasing experience premium.
In 1977 most graduates were male, but now most are female – and since women consistently have lower starting salaries than men this will affect the overall median. The report does have a section on this – showing that female graduates have higher median starting salaries than female AWE – but doesn’t seem to grasp that this has implications for the headline indicator.
In short, median starting salaries as a percentage of male AWE no longer tells us anything useful – the finding that it is at 79.7% in 2006 compared to 100% in 1977 is of no value to would-be uni students. To the extent it tells us anything at all it says more about AWE than graduate salaries.
Unfortunately, given the prominence Graduate Careers Australia gives this now pointless number, the media picks up on it. The AWE comparison is mentioned in the first paragraph of the AFR‘s report this morning. The SMH held off until paragraph 4 and reported a couple of the caveats, which is good, but they should have omitted it all together.
One of the other reports released by Graduate Careers Australia yesterday, on student satisfaction, is full of detailed methodological discussion. It’s not exciting reading, but it is a model that may save the starting salaries publication from producing misleading data.
Andrew Norton 
Do you know where the GCA is getting its funding from? It seems to me there are a plethora of these funny little research organizations that often release these statsically woeful reports. The AIFS and NDARC come to mind. The latter two of these take millions from the government and produce absolute garbage (the lifestyles survey comes to mind too, although I can’t remember who does it). Are these guys another one of these groups?
LikeLike
The universities pay at least some of the costs, and I think DEST also contributes though I will have to check that. You can extract interesting data from the salaries report; it is just their contextualising that is bad.
Some of the AIFS stuff I have seen is quite good, what were you thinking about in particular?
LikeLike
The problem I have with lots of their (AIFS) reports is that they constantly ask simplistic questions and expect meaningful answers, and they’re often biased toward finding conclusions.
For instance, flipping through the latest report (which is on mother’s with health problems), here’s what I noticed. The sample is the first thing and the first problem — they exclude people that don’t get FTBs, probably the healthiest group — which undoubtedly biases the results — why? Is the data of rich people not important? Lone mother’s are oversampled (why?), and then the stats are re-corrected (why?).
The next thing it looks at is a self reported health measure. Health is a multidimensional concept, and ending up with a single score where the baseline is basically unknown to the respondants looks a lot like those happiness questions you talk about often. Are you measuring real health or hypochrondia? However, its worse, because there are far more decent health surveys out there where these things can be controlled. You can find out what health problems people actually have, how serious etc. But they’re not used, despite this being the most important idea of the survey. Why not use a reasonable survey that looks at different dimensions of health (which exist exist already, have high validity etc.) ? It’s not like they take more than 5 minutes to fill in.
The next table I look at is on mother status and health — again open to sample bias that could have been easily fixed. The scale is now dichotomized into something rather meaningless. I’m sure I have health problems that affect work now and then (I catch colds occasionaly, afterall), and it would have been good to eliminate (or at least control) these sorts of effects. What’s the baseline? How do non-mother’s compare?
I could go on and on, but I won’t.
LikeLike
Getting back to the Graduate Salaries report, I agree that male (total) AWE is not the best comparator – for anything really, except the earnings of males. I think people persist with it because it is one of the few indicators that actually goes back as far as 1977. But this is not really a good enough reason, to my mind. Surely it would be better to benchmark graduate earnings to an indicator based on adult full-time earnings and one that includes women as well as men.
As earnings indicators go, MTAWE is a fairly conservative one which has tended to grow more slowly than most alternatives. This is because the growth in part-time work among men (most of which is undertaken by full-time students, ironically) has had the effect of retarding growth in total earnings relative to growth in full-time earnings. To some extent, this may have counteracted the upward bias in the figures due to increasing educational attainment.
I also agree that on the face of it comparing a median figure with a mean seems nonsensical, but the problem is that reliable median figures are available from the ABS only every two years, which wouldn’t suit people who put out reports annually, would it? The biennial Employment Earnings and Hours survey also contains detailed information on earnings by occupation, which you would have thought would be quite useful for mapping graduate salaries against.
So while it might be necessary to fall back on comparisons with average earnings on an annual basis (but preferable to use an indicator that represents a more appropriate basis for comparison – such as average full-time adult ordinary time earnings), these should be supplemented by more sophisticated analysis using other data as they become available.
As Andrew correctly notes, over time one would expect the (median or mean) earnings of new graduates to fall relative to the (median or mean) earnings of the workforce in general, if the workforce is becoming more skilled over time. This is not in itself a bad thing – why would you necessarily expect a young person just starting out in their career to earn anything like average earnings?
LikeLike
The starting salaries report also reports on earnings for 20-24 year olds generally, not without its own problems but far more meaningful than comparisons with the general population. If GCA did not give so much emphasis to AWE this would probably get more media attention than it does.
LikeLike
Conrad, I think you are being just a bit hard on AIFS. The report you refer to is on “Employment aspirations of non-working mothers with long-term health problems”, not on the health of mothers in general. It seems to be trying to ask and answer the question of whether non-working mothers on income support (most of whom are single parents) are likely to be able to respond to the government’s welfare reform changes if they have health problems that they see as limiting their capacity to work or study.
It uses an existing data set that AIFS constructed in conjunction with the Commonwealth Department of Family and Community Services, who were I think responsible for drawing the survey sample from administrative data (hence the inclusion of mothers receiving FTB). While I agree that excluding mothers who receive no FTB biases the sample somewhat, it is not as much as you may think since non-working mothers in high income families (who get FTB Part B, but not FTB Part A) are included. So for the specific purpose of investigating the circumstances on non-working mothers, the data set should be OK.
I would guess that the oversampling of single parents would be because they are a group of particular interest to the Department and to ensure that there were sufficient within the survey population and relevant sub-groups to draw statistically valid inferences about differences between them and partnered mothers.
There are of course lots of problems with relying on self-reported health status and its relationship to work capacity, not least of which that it provides an easy “reason” for not being able to work. Nevertheless a lot of the findings in this report appear completely unremarkable – such as that single mothers were more likely than partnered ones to report long-term health problems, that people with long-term health problems were less likely to be working and more likely to be on income support, etc.
One of the interesting findings for me was that people who reported a work-affecting health problem who said they would like to be in paid work had the same average reservation wages as the other two groups examined – mothers with non-work-affecting health problems and mothers with no health problems. This was an unexpected finding for the researcher, who presumably thought that this group might have lower productivity, but may, of course, just reflect the fact that people on income support think that they need to earn a higher income to be ‘better off’.
LikeLike
BG: I think what you have to consider is whether (a) you want to do a study properly; or (b) spends months churning out some numbers from a pre-existing data-set which wasn’t designed to can’t answer your question. As far as I’m concerned (b) is a waste of time, especially given that health outcomes is a huge field, and it wouldn’t be hard to actually do it properely (i.e., you wouldn’t have to develop you own scales and so on — you just need to go out and collect the data). Furthermore, if you have a decent question to start with, you can formulate intelligent models based on previous work that actually might give you some idea of causality and different patterns in your data (i.e., all people who are sick and do not work are not neccesarily qualitatively similar). A decent social science student with a 4 year degree should be able to do this, and there are plenty of them.
Also, the fact that some of the findings are unremarkable should *not* be taken as useful , as this is open to self confirmation bias. If I did the study properly and found the opposite, then all this study has done is solidify the incorrect view. Some things are naturally confounded (like health and work — e.g., Depression and jobs status, pregnancy difficulty and working etc.), and reporting simplistic data from them simply leads to people believing what appears intuitively obvious, which is not neccesarily correct.
LikeLike
Conrad – I agree with you (I think) that much that passes for social science research is simply designed to confirm people’s existing beliefs – in this case, perhaps, that single parents as a group are disadvantaged in a variety of ways and are unlikely to benefit from welfare reform in the ways that the govt claims they will. Which means that the govt shouldn’t be so mean to them, etc
I have an innate suspicion of any research that relies too heavily on subjective data. My main problem with this research is that it attempts to reach conclusions about what is likely to happen to people entirely on the basis of what they themselves think will happen to them in the future. That is no substitute for knowing what actually does happen to them.
I’m not really trying to have an argument over the worth of this particular report, which is a relatively slight offering in the end. But AIFS is solely funded by government (through the aforementioned Department of Families, Communities, etc), so you can expect that a fair bit of the research they turn out will use existing data sets, since it is a pretty expensive exercise to go out and collect new large-scale data whenever you come up with a new research question.
LikeLike
Yes, I realize that AIFS is soley funded through the government. It seems to me it would be much more efficient (and you would get better research done) if they tendered the contracts, or in case AIFS are actually getting tenders (?) then the people giving them might actually like to think about what they are getting in return (although to be cynical, I doubt the people in the government departments would understand the results any better).
I could be wrong — I’ve certainly come across some of these government proposals for reseach that actually get no bidders when tendered.
LikeLike
I think, though I am not 100% sure, that general AIFS funding is not conditional on any specific agreement with FaCSIA about what they will research. They do, however, supplement that funding by tendering for other research (including from FaCSIA) on a competitive basis. I’m not really up on all the legal distinctions but I think AIFS is a statutory body established by Act of Parliament, which gives it a degree of independence from the dictates of government.
But generally I agree with you that you get better research done when organisations have to compete for the funding, rather than just having a cosy funding arrangement. And I couldn’t possibly express an opinion on the capacity of people in government departments to understand research, only to observe that there has been a fairly systematic down-grading of research capacity within (Commonwealth) government departments over the last decade or so. In my view, if you never do research yourself (and these days very few public servants do), your capacity to understand (and, need I say, direct and influence) research is going to be somewhat limited.
LikeLike