Polls are like movie trailers: some people like them, other people hate them, but they are not the movie, explains pollster Greg Lyle.
Polls are, however, a part of elections – for better or for worse – and give information on what people think about the issues of the day, on which parties are rising or falling in popularity or on the popularity of individual candidates.
The Greater Vancouver Board of Trade and communications firm FleishmanHillard HighRoad recently commissioned the Mustel Group to conduct a poll, called VoteLocal, ahead of the Oct. 20 municipal elections.
The survey showed that housing affordability and transportation are top of mind for North Shore residents as they head to the voting booths in October to elect new municipal governments. But the poll also showed that 81 per cent of those surveyed on the North Shore were largely satisfied with the level of community engagement and consultation with both residents and businesses on projects and 63 per cent said municipal politicians are on the right track.
The poll surveyed what the most important issues were facing them, and 35 per cent said housing affordability and 23 per cent identified transportation. But a larger majority of North Shore residents, 62 per cent, said they thought their quality of life and affordability in the region had gone down in the last five years. Fifty-one per cent said they expected the situation to get worse – only 14 per cent expected it to get better.
In the VoteLocal poll, the sample size of North Shore residents was 60 members of the general population who responded as well as 13 businesses. Also participating were seven politicians, either sitting mayors or councillors or declared candidates in the upcoming election.
Regionally, the poll surveyed 533 adults in Metro Vancouver aged 18 and older, 184 members of the Greater Vancouver Board of Trade and 93 current mayors and councillors and candidates for the upcoming election.
Forty-eight per cent, almost half of the North Shore population surveyed, have considered moving away because of quality of life and affordability concerns. This mirrored the rest of the region where 46 per cent said they’ve considered relocating.
Eighty-eight per cent of those surveyed on the North Shore supported measures to curb speculation – 68 per cent supported a tax on empty homes and 65 per cent supported measures to restrict foreign capital in the B.C. housing market.
More than half of North Shore residents, 56 per cent, believe that too much housing has been built taking into consideration the impact on traffic congestion, while 22 per cent said not enough housing has been built, and 22 per cent said there’s the right balance.
On questions on taxes and spending, 59 per cent of respondents to the Mustel Group poll said they are receiving fairly good value for their property taxes, but 40 per cent said they were receiving poor value.
The methodology of the poll was released with the date, which indicated that 184 of the 810 respondents were members of the Greater Vancouver Board of Trade. So, this wouldn’t be a random sample of businesses, Lyle pointed out, rather it’s a sample of the type of businesses that join the board of trade.
Lyle, who is president of Innovative Research Group, cautions that polling gives the “what” information about issues and candidates, that is, data on how people are responding to issues or how likely they are to vote for a candidate or support a party, but where pollsters get into trouble is when they start extrapolating the “why” from the what and start giving an analysis of their findings.
“If you ask a pollster to speculate, they will, but it’s important for your readers to know when they’re talking about something that they specifically asked in the survey versus when they’re saying, based on their experience and knowledge, why they think a certain response has happened, or why they think a certain reaction has occurred,” Lyle said.
The total regional VoteLocal survey has a margin of error of +/-4 per cent. The North Shore results have a margin of error of +/-11 per cent in the most conservative case, according to the Mustel Group. This was an online poll and Evi Mustel of the Mustel Group explained in an email that: “As our panel is randomly selected (rather than opt-in) we are able to quote margin of error. This is sanctioned by the industry.”
As far as reporting on polls, the mistake many media outlets make is with tracking polls, which are intended to keep an eye on how popular candidates or parties are. Often a headline will read that one party has moved into the lead, Lyle explained, but looking at the methodology, often that lead is within this margin of error.
“Directionally, it might be significant that they’ve moved in a certain way, but you can’t make a lot out of it given that margin of error,” Lyle said.
The best way to poll people is to randomly recruit them using live caller interviews either by landline or cellphone. However, that’s expensive and is getting harder and harder to do as people are less willing to sit through a 30-minute telephone interview than they were 20 years ago, Lyle said.
Traditionally, in pre-internet days, people were called randomly on their landlines and, more recently, on their cellphones. This can be expensive, up to $30 per call. It’s cheaper to do an online poll, at a cost of about $5 per survey. Online polls are challenging because it’s harder to get a representative sample of the population.
When people are called on their landlines or cellphones, polling companies are able to pick people at random – almost everyone has an equal chance to be selected. Online polling is more self-selected because there are no rules on how email addresses are assigned.
A compromise is to recruit online panels – these are not randomly recruited respondents, but, even though the tool of randomness is lost, there are ways to make the sample reflect the general population. However, the sampling error (the margin of error) won’t work the same way.
“The sample that we get might not be randomly selected but it still looks like the population as we know it from things like the census,” Lyle said.
Most people who answer polls are older people, so a balancing tool used by a pollster is to “weight” the poll, that is, to give more weight to younger people and less to older people. The issue with weighting is that it exaggerates the results of the group who gave fewer answers.
An interactive-voice response poll – also known as “demon dialers” or “robocalls” – needs to be weighted for sample size because most people responding to the polls are older women who are often the first to pick up the phone at home.
A good IVR poll will collect the data and then weight it to make it look like a typical population.
“Often IVR polls won’t tell what the raw sample looks like and what the weighted sample looks like and when that happens, you have to ask serious questions about can you believe the poll or not,” Lyle said.
The best kind of poll is a live caller poll done with a random sample – “other polls can work but you need to read the fine print,” Lyle explained.
Voters should ask the following questions of polls: how well was the poll constructed and whether the sample was randomly chosen or voluntary.
“You’re trying to get a representative sample that doesn’t have any bias in it,” Lyle said.
Results can differ depending on the order of the questions, for example, if you ask about an unpopular leader first but then about a party that’s popular, the first question can skew the second one. The pollster should make available the questions and the order of questions they were asked.
Polls can be entertaining for political nerds, Lyle said, but the information can be actually useful if someone is considering voting for someone to counter-balance the popularity of another candidate.
“Sometimes in elections that have parties … it can be interesting to know how well a party is doing,” Lyle said. “In that case, if you’re trying to vote strategically, the polls are quite useful."
