My Australian political identity survey closed this morning, and I will write several posts on the results over the next few days. But first some discussion of how respondents were solicited and the biases that might cause – some of which is in response to questions and criticisms since the survey began.
The first point is that this was not a normal public opinon survey – it was a survey of people willing to identify with with a philosophical political label. The difficulty this poses is in finding such people, who are likely to be a small percentage of the electorate. If a pollster was commissioned to do it I expect it would take tens of thousands of phone calls to produce the 1,200 responses I received in my online survey.
My method was to largely recruit via blogs. Apart from my own, there were links from Andrew Leigh, Andrew Carr, Australian Conservative, Andrew Bartlett, Institutional Economics, the f-rant, Sacha Blumen, Club Troppo, Thoughts on Freedom, Pollytics, Quadrant blog, and Catallaxy.
Blogs were supplemented with my personal networks, via email and my Facebook page.
The advantage of these methods over random sampling is that they are more likely to capture the people most engaged in advocating the ideologies under discussion. This method also takes us a little closer to the other major methodology for understanding political ideologies, which is examining the published views of their advocates.
With a bigger budget (I spent about $40 for the online survey), the results could be improved by advertising the survey to the members of relevant organisations and readers of relevant publications, and perhaps providing a non-online version. The number of conservative respondents was lower than I expected, so maybe internet survey recruitment methods are not good at finding conservatives.
The main other concern is whether these blogs cause an over-representation of a particular strand of thinking within a broader ideology. I’m pretty confident that this won’t be the case for the classical liberal-libertarian blogs. I’m not so sure about social democrats. When the only social democratic-leaning blogs linking to the survey were Andrew Leigh and Club Troppo I was a little concerned that their readers would be more economically literate than I would expect social democrats more generally to be. But there was a surge of social democrat responses when Crikey-sponsored blogs linked to me, so this probably fixes the problem. I wasn’t represented on a green blog, but a quick look at the results suggests that stereotypes will be confirmed, so there is no initial reason to believe that despite their relatively small number (92) they will be unrepresentative.
The biggest obvious problem with the results, however, is the gender balance – or lack thereof. Only 14% of respondents overall are female. Even the most ‘progressive’ of the respondent groups, the greens, are more than 80% male. The 2007 Australian Election Survey does show that women are less interested in politics than men, though not by anywhere near this margin, and have similar (low) rates of internet usage for political purposes as men. As I go through the various ideologies, I will see if there are obvious gender differences in the responses that mean the lack of women could be affecting the results.
Andrew Norton 
I was a little concerned that their readers would be more economically literate than I would expect social democrats more generally to be. But there was a surge of social democrat responses when Crikey-sponsored blogs linked to me, so this probably fixes the problem
Now that’s called damning with faint praise!
LikeLike
Where I work it’s survey city, and, excluding obviously blokey surveys, your under representation of females is actually the opposite problem that most surveys have (i.e., “neutral” style surveys tend to get more female responses), which I guess is interesting in itself.
LikeLike
Conrad – Most surveys work by phoning strangers, and women are both more likely to answer the phone and to be cooperative once they have answered.
What I need to know here is whether women are much less likely than men to think of politics in these ways, or whether for reasons I do not fully understand the very blokey nature of the blogosphere has produced these results.
In the AES I can’t see any significant male-female differences in strong party ID or positioning at the extremes of the left-right scale, further suggesting that it is a blogosphere issue.
LikeLike
I think you’ll find it’s true of the internet too. Women just seem to respond more (my guess would be at a ratio 3 to 1 for a seemingly neutral question about some sort of attitude). This is a constant problem for our students — it’s really hard to get guys to fill things in (obviously excluding this survey!!). My experience with that is over quite a broad range of topics (albeit all psychology related) — I have the thrill of listening to at least 30-40 talks a year on survey stuff collected from the web, and the imbalance is a constant complaint (trying to get old people is fun too). You will also get much higher response rates if the group you target feels like it is part of a smallish social group rather than the general public, which is probably true given the probable source of your hits (indeed, I’m surprised you got such huge numbers so quickly). My bet then is that given that most posters on most of the groups you targeted are male, I imagine this is the reason for the imbalance.
.
Funnily enough, if you hadn’t noticed it, there is also an article in the Age today saying that women rule the blogosphere (in the lifestyle section). So there is definitely something very blokey going on.
LikeLike
I hadn’t noticed that Age article – took a day off newspapers. Women don’t rule political blogs – relatively few female bloggers and commenters, which is presumably the source of my problem as you suggest.
I do have a problem with older respondents – only 4% were 65 or more, which may also explain a lower % of conservatives than I was anticipating.
LikeLike
Andrew, there’s no real issue about biases based on the data collection method is there, as, in my understanding, you’re not wanting to extrapolate your survey results to a wide population – i.e., you’re not taking this as giving really good quantitative information – yes?
Having over 1000 people take a survey is pretty good especially on a very limited budget.
Psephologist-type blogs such as pollbludger could be a fruitful source of respondents in the future.
LikeLike
You write in a very dry, arid manner. This in itself would be offputting to even many political women. You inadvertently disenfranchise yourself by your very nature.
LikeLike
SL – Yes, my style is very ‘male’ – partly my personality, partly an assessment that there are people emoting and exploding all over the blogosphere, and there should be a niche for calm and analytical. But my style doesn’t explain all the other blogs also failing to attract women, many of which linked direct to the survey without the intermediary of my dry prose. 1,200 is way higher than my readership, so they must have been doing most of the recruitment.
LikeLike
Very interesting.
A question asking where people heard about the poll might have given you some interesting information. Information often travels in un-predictable ways in cyberspace.
Its something to consider next time 🙂
LikeLike
I disagree with SL. I would describe Andrew’s writing style as measured. I think this blog is informative and balanced, with clearly outlined influences. For a beginning thinker like myself this kind of writing is very helpful. I am often exhausted by the irrational and emotive writing and comments in other blogs ( not SL’s I have to add) rather than educated.
LikeLike
Polls show that 56% identify as ALP or Greens supporters and you had two categories for these and was it 4 for political gradations to the right of social democracy? I see too that 30% of US Democrats “prefer socialism to capitalism” according to latest Rasmussen reports.
Your bias probably influenced the numbers who bothered to participate. I didn’t for this reason. Oh and I’m a woman.
LikeLike
SL – This wasn’t bias, it was intentional and explained right at the start of the survey: “The aim of this survey is to see what people willing to identify with various political labels actually believe. My main interest is in seeing where, in practice as opposed to strict theory, Australian classical liberals overlap with and differ from libertarians, conservatives and social democrats. However, people with other political philosophies are welcome to take the survey.”
As Conrad pointed out, the overall numbers participating were high, with the gender balance the only major counter-intuitive result in the broad categories, though I thought I would get more conservatives.
If you want to know what the broad left thinks you can run your own survey at modest cost from freeonlinesurveys.com (only free for small surveys).
LikeLike
Ok, but I fail to see the point of the survey particularly when it obviously failed to achieve what you had hoped for.
LikeLike
“Ok, but I fail to see the point of the survey particularly when it obviously failed to achieve what you had hoped for”
.
Actually, I’m quite interested in seeing the results, so I think I’ve missed what is supposed to have failed — and I agree with Sacha, simply because you have different numbers of participants in one group or another isn’t necessarily a problem (especially with big Ns). There are lots of things you can do to see if there is bias in any case — you can compare groups, you can look at reliability statistics within distributions, you can check validity by doing something like a factor analysis and seeing whether it agrees with results you would expect from other research (for example, if you got a social and economic factor, it would show that even within a limited sample, there is a decent enough distribution of responses to pick up some important distinctions amongst people), etc.
.
Also, people would get nowhere in science if everyone couldn’t see the point of things that fail. Lots of things fail, but people keep on plugging away. Sometimes you learn more from projects that fail than that succeed. That’s just the way life is.
LikeLike
There are enough respondents to draw some general conclusions. While I would have liked more women to participate, unless the women who did participate hold significantly different views from the men (I have not done this analysis yet) there is little reason to believe that this is a major issue. Also, we need to keep in mind that the vast majority of participants in real-world debate on the issues discussed are men, so a 50-50 gender divide could -if the sexes have different views – actually be problematic in describing what ‘classical liberalism’, ‘social democracy’ etc mean in practical terms.
LikeLike
“Also, we need to keep in mind that the vast majority of participants in real-world debate on the issues discussed are men” – your evidence for this is?
LikeLike
SL – How about we start with the Rudd Cabinet, in which 4/20 are women. The first woman won a state election in….2009. The political bloggers – many of whom do MSM as well – are overwhelmingly men. Outside ‘women’s issues’ – such as childcare and maternity leave – I very much doubt you could find a major public issue in which most major players are women.
I thought feminists were constantly complaining that they were ‘under-represented’, now you are saying this is not true and there is nothing to worry about!
LikeLike
Andrew, your assumption/presumption that political discussion only takes place in the Cabinet, parliament, the meetings of political parties, or on blogs, or perhaps in the msm in letter responses to opinion pieces, etc, which seems to form the extent of what you allow to be the “real [political] world” is your fatal error. People discuss all sorts of political issues – the full gamut – in all sorts of places outside of these forums which yes are male dominated and thus enormously unrepresentative. To think that these are the only places where politics are discussed is to not only miss but worse discount the existence of the opinions and political discussion of much larger numbers, including of course most women.
btw, I don’t have a blog, Martha, SL stands for socialist leftist (as an individual, not an organisational entity).
LikeLike
It’s true political discussion takes place privately, but I think we can safely assume that public debate, particularly from people with real power or influence, is more important in shaping policy.
LikeLike
In addition to the influence of private discussions amongst “people with real power or influence”, pointing to the importance of individuals having the ability to influence.
LikeLike
Just pointing out that ‘SL’ isn’t me; a couple of people seem to have become confused.
LikeLike
I’ve only just come across your blog (and disappointed at missing the survey) and results regarding gender balance are interesting but to me, unsurprising. Perhaps it comes from growing up watching the other little girls play with cabbage patch dolls while I engaged in warfare with the boys.
I’m only a uni student, so my political views are still developing, but I find your blog very interesting. I’ll be following it!
LikeLike