By a Newsnet reporter
On Sunday, details of a poll emerged in the media that suggested support for independence amongst 14 to 17 year olds was a mere 21 per cent.
In a further blow to the Yes campaign, those against independence was said to be running at over 60 per cent.
It was the main topic of discussion on Sunday's Politics Show and headlines followed. Delighted Unionist politicians issued press releases as their pro-independence counterparts grabbed hold of an aspect of the survey that revealed over two thirds of respondents admitted they could well change their mind.
Unusually for such polls, the methodology and questions were published immediately and at first glance it seemed all was as reported – a blow to the Yes campaign indeed.
However, closer examination of this survey revealed, if not a disturbing aspect of the survey, then certainly something that seemed to cast doubt on the reliability of the poll. The proportion of households that could, for want of a better description, be termed pro-Union, significantly outnumbered those households one would have said was pro-Independence.
The difference, which was over forty per cent in favour of pro-Union, was alarming.
To clarify, the system used in this poll was RDD (Random Digit Dialling) which is pretty much as it sounds with random telephone numbers being dialled. Before teenage respondents were asked their views, those of their parents or guardians were sought.
Of the 1018 households whose responses made up the survey, a whopping 594 had parents or guardians who said they would be voting 'No' in the 2014 referendum. By contrast only 178 households had parents or guardians who said they would be voting 'Yes'.
So households that could best be described as anti-independence made up 58.3% of the survey, whilst households who could be described as pro-independence made up a mere 17.5%. Undecideds constituted just over 24%.
We wondered about this apparent anomaly, so sent our questions to the contacts given on the published PDF. We asked about the makeup of the survey and what we felt was an over-representation of anti-independence households which could have led to an inflated result for those opposed to independence. We suggested that a more accurate survey would have ensured recent opinion polls by reputable organisations could have been used in order to eliminate such an obvious bias.
We are at present looking into the methodology used and would be grateful if you could confirm our initial belief that what can best be described as 'pro Union' households (if parent views are used) made up almost 60 per cent of respondents against around 20 per cent 'pro Independence' households.
This, to us, would potentially lead to an inflated support for No, given the survey found that 75% of No respondents took the same view as their parents.
If our figures are accurate, do you agree that the poll methodology ought to have mirrored more accurately, in terms of household view, recent polls which indicate the gap between Yes and No to be significantly less than the 35 to 40 per cent gap that your own sample appears to have assumed?
The reply was, to say the least, interesting:
The sample was a random sample of households with children aged 14-17, and the randomness ensures that the sample is likely to be broadly representative of the population of such households. There have been no published surveys of the views of adults in such households, and it cannot validly be inferred that the views of such adults would be the same as the views of adults in the population as a whole. To have biased the sample to do what you suggest - mirror recent polls of the whole adult population - would have been to have distorted the representativeness with respect to the young people who were our target population.
It seemed a strange response, that to have sought to ensure the ratio of households chosen did not exaggerate - one way or another - would have "distorted the representativeness of the survey".
We challenged the claim that the survey was representative:
Further to your latest response. We note your claim that the randomness of your methodology will produce a representative response.
Given that the parent replies very clearly did not produce anything like a representative response then we would ask you to justify this claim in relation to the results of the young person survey.
The response was even more interesting.
Randomness would produce an approximately representative response of the RELEVANT population, which here - as I said - is parents of people with 14-17 year old children, NOT all adults. Since we don't know the views of such parents from any other source, there is no basis for your claim that representativeness has not been achieved.
So, the survey was seeking to represent only the relevant section of the population, which was of course households in which lived teenagers aged 14 to 17. The views, according to the spokesperson, of these parents are not known.
Were the survey organisers claiming that households in Scotland with teenagers between the age of 14 and 17, even more likely to vote 'No' in the referendum? If so then it was as big a story as the young person's survey.
We pursued the issue:
Then you appear to conclude that households with teenage children are even more likely to back a No vote than current polling suggests.
58.3% No compared to 17.5% Yes.
Can you confirm this?
The response appeared to suggest a casual disregard for the views of the adults who had responded, the survey was not about adults.
All that can be said is that the adults whom the survey talked to had the distribution of views that you calculate from Table 10. Remember that there is no such thing as a 'household' view, since the adults in the household will differ in their views, just as around 40% of the teenagers differ from that of the adult who was interviewed in the survey. This was not a survey of parents, but a survey of teenagers. We are not making any claims at all about the views of 'households', or of 'parents': as I said in an earlier email, that is a population about whom not much specifically is known. A different research design would be required to be able to assess the views of parents (or indeed other adults in a household), and how they do or do not differ from each other.
We sent one final email
Thanks for taking the time to respond. May we ask whether publication of this survey was always intended and whether the BBC or indeed any other news outlet has sought clarification on the methodology?
Also, does your organisation accept that the parent responses in this survey are exceptional when compared to adult polling results published by recognised organisations such as Yougov, Ipsos Mori and Panelbase.
If so, are you at all concerned by this difference?
The response is published below:
Publication was always intended: that is a condition of grant from the ESRC, and in any case is standard practice by academics. Indeed, we are grateful that you acknowledge that we have placed details of the survey in the public domain. The BBC did of course discuss the methods with us. The methods were also scrutinised in the usual rigorous way by the ESRC's peer-review process before they awarded us the grant. I have already answered your second main point: the results differ from the views of all adults, but that is not the relevant population for comparison in any assessment of how representative the views or the sample is of 14-17-year-olds in general.
As I explained, and as you kindly and accurately reported, the relevant population for comparison is not all adults at all. You claim also that we have not considered weighting the results. You did not ask me that, and the answer is that we have indeed examined in detail the effects of weighting. We will be reporting the results of the weighted analysis at a seminar in Edinburgh on 5 June, which is free for anyone to attend. You can sign up at: http://www.aqmen.ac.uk/youngscotsurveyresults. This link was also made available on the BBC web site from the time of publication on Sunday 2 June. You would be very welcome to come. The main point we will be reporting there, however, is that no form of weighting makes much difference to the balance of views expressed by the young people, especially to the proportion intending at present to vote No.
The poll may or may not be accurate. Young people in Scotland may well be even less likely to support independence than older members of the electorate. We accept that households also may contain adults with differing views. However what is known is that those adults who answered the initial question were overwhelmingly anti-independence, and that those youngsters who gave 'No' as their answers were in agreement with this adult in three quarters of the responses.
There are many factors which could have led to these young respondents giving the answers they did. That they will have been influenced by their home environment seems the most obvious. For a survey to refuse to acknowledge the importance of implementing a weighting system that acknowledges this home influence is incredible.
What other variables are at play in this survey is anyone's guess – how many people refused to take part when confronted by a random telephone call, how many young people gave answers as their parent listened in or felt compelled to placate their parent?
There is another more worrying aspect of this survey that no-one seems concerned about. Teenagers are influenced by trends and a desire to be accepted as one of the crowd.
The publicity surrounding this poll will have the effect of telling these young people that anyone supporting 'Yes' is in the minority, and by a considerable margin. The desire to conform and join the 'popular' group, for some, may well prove irresistible.
It is also not beyond the realms of possibility that what we adults accept as the normal cut and thrust of tribal politics, morphs into bullying in the less mature environment inhabited by fourteen year olds now that they know the 'winner'.
At the time of writing, the PDF document containing the questions can be read here.
This survey was funded by the Economic and Social Research Council which is based in Swindon. According to its own website, it receives most of its funding through the Department for Business, Innovation and Skills, a department of the UK government.
"Apart from presenting the results themselves, the outcomes of the research will be used to develop teaching materials for schools across Scotland to have materials that are relevant for young people in informing the debate." the survey report concludes.
More work needs to be done, and quickly, in order to establish whether or not this poll is accurate. Sadly, even if more surveys are carried out that challenge this poll, then the damage has probably already been done.