3

Standard Error: The Polls in the 2019 Alberta Election and Beyond

Brooks DeCillia

The TV stations’ election calls, with their dramatic music and fancy animation declaring a United Conservative Party (UCP) majority government, came quickly after the polls closed on 16 April 2019. It was not even close. The UCP—a party that hadn’t even existed two years before election day—captured 55 per cent of the vote, ousting Rachel Notley’s New Democratic Party (NDP) after a single term in power. The nascent UCP, a party created when the old Progressive Conservative (PC) Association and the Wildrose Party merged, captured sixty-three of eighty-seven seats in the prairie province’s Legislative Assembly. The NDP only captured 33 per cent of the popular vote and twenty-four seats. Yet, you could be excused for thinking the results would be much closer if you only got information about the 2019 campaign from the Alberta news media’s reporting of polls. Nine public opinion polls released during the campaign—half the total released during the campaign—suggested a single-digit gap between the NDP and UCP. The news media narrative suggested a much closer race than what voters ultimately decided at the ballot boxes. The coming pages analyze the publicly released polls in the 2019 campaign and critically examine the news media’s reporting about them. This chapter also explores the short honeymoon the UCP government had with Alberta voters.

Critiques of public opinion polls are not new. French sociologist Pierre Bourdieu famously declared that “public opinion does not exist” in 1979. In a similar vein, German philosopher Jürgen Habermas charged that polls are used to manufacture public opinion, preventing a deliberative democracy.1 Scrutiny of polling is not only philosophical, but practical as well. Some high profile misses abroad and at home in the last decade have undermined confidence in the accuracy of polls. Pollsters in the UK underestimated the British public’s desire to Brexit and most US pollsters failed to see Donald Trump’s narrow path to electoral victory in 2016. Pollsters have produced some spectacularly bad predications in Alberta, too. Notably, during the 2017 municipal election in Calgary, Mainstreet Research, a national public opinion and market research firm, released three polls on behalf of Postmedia, which owns the Calgary Sun and Calgary Herald, that wrongly forecast that the incumbent—and popular—Calgary Mayor Naheed Nenshi would lose to a relatively unknown challenger. The polls upended the tone and tenure of Calgary’s municipal campaign.2 After the election that Nenshi won handily, Mainstreet Research admitted to “big polling failures.”3 Five years earlier in the 2012 provincial election, many polls were also off the mark, with several surveys during the campaign predicting the upstart Wildrose Party would sweep away the PC government that had ruled Alberta since Peter Lougheed came to power in 1971.4 While pollsters patted themselves on the back four years later for correctly predicting that a strong Orange Chinook, led by the NDP’s Rachel Notley, would blow away the formidable PC dynasty,5 an analysis of the 2019 polls found they were “only marginally better” when compared to polls about other provincial elections around the same time that were labelled “failures” and much worse than the error rate of the polls in the 2015 and 2019 Canadian federal elections.6

When it comes to public opinion polls, accuracy can be defined in several ways. Polling should not only be reduced to forecasting the winner in a political campaign. Polls come with caveats, including a margin of error and the assumption that they are a snapshot in time.7 Accurate polls also correctly gauge the difference between each party’s measured level of support and their actual level of support on election day, while not exceeding the polls’ stated margin of error. While all the publicly available polls released during the 2019 Alberta provincial election campaign predicted a UCP win, a close examination of all surveys show the polls displayed the “standard” or typical error (pun fully intended) seen historically in Alberta polling—the underestimation of conservative support.

As the number of public polls have grown, so too has the news media’s insatiable appetite for public opinion data. In 2019, Alberta’s news organizations chewed through the vote preference and leadership approval numbers, spitting out a constant stream of “horserace journalism.”8 While political journalism is obsessed with public opinion data, a recent study suggests journalists are incapable of comprehending the numbers.9 Dubbing the phenomenon the “Nate Silver Effect,” the research questions the news media’s traditional role as an independent “gatekeeper,” policing the standards and release of polling data. Defenders of political polls argue that the information is invaluable to the public, fuelling a lively democratic debate, and stressing that political parties won’t stop polling. If parties have the polling information, the argument goes, so, too, should the public. And polls do matter. They can affect elections. Some research even suggests that voters “jump on the bandwagon,” casting their ballots for the party or candidate that pollster predict will win.10

Data and Methods

To quantify the extent of the “horserace” news media narrative during the 2019 Alberta provincial election, I conducted a classic content analysis of the reportage of all public polls.11 An exhaustive corpus of every article in the mainstream news media or on political blogs about opinion polls during the four-week provincial election campaign was compiled. The online news archives Factiva and Infomart and news aggregator Google News were used to compile the comprehensive corpus of sixty-nine articles and posts to evaluate variables, including, among other things, (1) which party was in the lead, (2) if the race was static or dynamic, (3) how the poll was characterized, (4) and if polling methodology was included.12

This work’s analysis of polling accuracy relies on a list of polls released publicly during the four-week provincial campaign in 2019—19 March to 16 April.13 There are eighteen polls from a dozen companies in the data set, as detailed in Table 3.1. The polling firms used a range of methods from online panels, interactive voice response (also known as IVR, or robocalling), and traditional random telephone dialling by human interviewers to survey Albertans during the 2019 provincial election. IVR gauged public sentiment by randomly dialling numbers (land and cell phone lines) and eliciting responses to a pre-recorded voice. Online panels, on the other hand, surveyed eligible Alberta voters using their internet-based panels. These online panels usually consist of people who have agreed to complete surveys using the internet.

Recruitment for these online panels varies. Sixty per cent of Leger’s panel, for instance, was recruited randomly over the phone. EKOS’ methodological description emphasizes that its respondents to online/telephone research panel are “recruited by telephone using random digit dialling and are confirmed by live interviewers.”14 Nanos’ random telephone survey of 500 Albertans used a live operator to ask people about their vote intention. Janet Brown Opinion Research/Trend also used random dialling of land lines and cell phones to measure public opinion. Respondents were given a choice of being interviewed by a live telephone operator or completing an online survey later online. Sample sizes in all the 2019 election campaign polls ranged from about five hundred to more than fifteen hundred. Margins of error varied from a high of 4.4 percentage points to a low of 2.7 percentage points.15 While online surveys are technically convenience samples and not truly random samples, polls conducted through online panels do aim to be representative of the population. In turn, these firms often provide a credibility interval or an equivalent margin of error that approximates the range of values if the online panel data were drawn from a truly random probability sample of the same size.

The dataset complied for this research was used to evaluate several considerations, including whether 16

(1) the poll correctly identifies the winner;

(2) the poll’s stated margin of error correctly encapsulates the actual vote for each party;

(3) the poll’s stated margin of error correctly encapsulates the actual vote for the NDP and UCP; and

(4) the poll’s total absolute polling error.17

Table 3.1. Polling Summary by Firm

Note:

i Of note, some of the online panels recruit respondents using random telephone dialing. That is, respondents do not opt themselves into the panel. EKOS, for example, stresses its “panel offers exhaustive coverage of the Canadian population (i.e., internet, phone, cell phone), random recruitment.” In fact, most of the polling companies included in this research describe their online panels as representative of the Canadian population.

ii This category reports how respondents were interviewed. That is, how the polling firms asked vote intentions during the 2019 campaign. Recruitment for online panels is different—and varied. Some firms, such as EKOS, recruited their internet-based panel using random dialing. Leger reported recruiting 60 per cent of its panel randomly using the phone. Ipsos reported interviewing its respondents “online via the Ipsos I-Say Panel and non-panel sources.” Other firms that used online panels to gauge vote intentions during the 2019 provincial election did not detail in their news releases or public-facing documents how their online panels were recruited.

iii Ipsos’ survey combined a mixture of online interviews and random computer-assisted telephone interviewing (CATI), calling a mix of cell and landlines.

iv Innovative Research Group does not report a margin of error, noting that the firm’s representative online survey is “not a random probability-based sample.” The firm notes: “a margin of error cannot be calculated. Statements about margins of sampling error or population estimates do not apply to most online panels.” For comparison purposes, a probability sample of this poll’s size would have a margin of error +/- 4.3 percentage points at the 95 per cent confidence interval.

Sources: Table compiled by author.

How the Media Reported on the Polls

Towards the end of the campaign, some news media highlighted public opinion polls with a single digit spread between the UCP and NDP, suggesting the race was tightening.18 Global News, for example, relying on an Ipsos poll, suggested an eight-point spread between the UCP and NDP, was evidence of a “tightening” race with the “NDP gaining ground” seven days before the UCP crushed the New Democrats.19 As Figure 3.1 illustrates, the extent to which news media characterized the race as dynamic, in fact, grew at a statistically significant level from zero in the first week of the four-week campaign to 45 per cent in week two, peaking at 76 per cent in week three, before dropping off to 53 per cent in the final week. On election day, for example, the online news source DailyHive’s headline read “Mainstreet poll shows UCP and NDP within 2% of each other in YYC.”20 The UCP beat the NDP by 19 percentage points in Calgary in 2019. Only slightly more than a third of the news coverage (36 per cent) mentioned the poll’s methodology, including important information such as margin of error, sample size, field dates, etc.

How Accurate Were the Polls?

Table 3.2 outlines, for all eighteen public opinion polls during the 2019 election campaign, how accurate the surveys were in (1) estimating actual support on election day, (2) predicting the correct winner, (3) correctly anticipating each parties’ support within its stated margin of error, (4) accurately estimating the correct level of support for the NDP and UCP on election day within the poll’s stated margin of error, and (5) getting it right over all (or total absolute error). All the polling firms correctly predicted the UCP would win the election. No polls ever showed the NDP in the lead. The public opinion polls did not, as conventional wisdom holds, become more accurate close to election day. The total absolute error (15.3) for the eight polls taken within a week of the 16 April vote is the same as the total absolute error (15.3) associated with the ten polls conducted earlier in the campaign. Two of the most accurate polls—Forum Research’s first poll (3.8 total absolute error) and Janet Brown Opinion Research/Trend21 (7.3 total absolute error)—came early in the campaign. Forum Research’s second campaign poll, with a total absolute error of 9.1, completed its data collection the day before the 16 April vote. As Table 3.2 details, the total absolute error for all eighteen polls in the 2019 campaign was an average of 15.3.

Figure 3.1. A “Dynamic” Race by Campaign Week

Only two of the polls—Forum Research’s first poll and Janet Brown Opinion Research/Trend—correctly anticipated the level of support for the four main parties within the firm’s stated margin of error (criteria two). Most of the polls accurately predict the support for the Alberta Party and the Liberals within their stated margin of error but did not get it right when it comes to the only two parties that had a viable chance at forming government—the UCP and the NDP. Only Forum Research’s first poll and Janet Brown Opinion Research/Trend hit the mark when it came to predicting the UCP and NDP’s support within their firm’s stated margin of error (criteria three). It is particularly notable that all the polls, except Forum Research’s first survey, underestimated UCP support. On average, all the public opinion polls during the 2019 Alberta election campaign underestimated the conservative party’s support by about seven per cent.

While the total absolute error for all the Alberta election polls in 2019 shrunk from 18.7 points in 2015 to 15.3 points in 2019, all the polls—except for three—produced a total absolute error rate above ten. The total absolute error ranged from a low of 3.8 points in Forum Research’s first poll to 22.3 points in both the EKOS and the Research Co. public opinion surveys. As noted above, the total absolute error does not diminish closer to the election. The two most accurate polls—Janet Brown Opinion Research/Trend and Forum Research’s first poll—were in the field surveying Alberta voters in the last week of March and first week of April, well before the 16 April vote. By means of an example, research evaluating the 2015 Canadian federal election polling concluded that the public polls were fairly accurate, with an average total absolute error of 6.7 points.22 Four years later, the total absolute error for the final twelve polls in the federal election was 8.5 points.23 The fifteen polls in the 2021 federal election had a total absolute error of 9.1 points.24 For comparison, Table 3.3 details the notable total absolute error of public opinion polls in recent federal and provincial elections.25

Table 3.2. Polling Error in the 2019 Alberta Election Campaign

Note: i Representative online panels are not truly random probability samples. Margins of errors are not applicable. Most firms, as reported in this table, offer a margin of error based on a probability sample of a similar size. ii Innovative Research Group does not report a margin of error, noting that the firm’s representative online survey is “not a random probability-based sample.” The firm notes: “a margin of error cannot be calculated. Statements about margins of sampling error or population estimates do not apply to most online panels.” For comparison purposes, a probability sample of this poll’s size would have a margin of error +/- 4.3 percentage points at the 95 per cent confidence interval. Sources: Table compiled by author.

Table 3.3. Average Error in 2019 Alberta Election (by Time, Period) Comparison with Other Canadian Polls

Sources: Table compiled by author.

While the total absolute error for all the 2019 Alberta election campaign polls (15.3 points) was better than the 2015 campaign (18.7 points), it is not much better. Additionally, the average error rate in 2019 in Alberta was not much better than the total absolute error (17 points) in the 2013 British Columbia polling debacle that wrongly predicted the Liberals with Christy Clark were tracking to lose to the NDP in an election the governing party won handily. The 2019 Alberta election campaign polls all predicted the right winner, but they did not, for the most part, perform well when it came to estimating the final proportion of votes that both the NDP and UCP earned from voters in the prairie province. Of particular concern, most of the polls did not accurately capture the true extent of the UCP’s strong support amongst Alberta voters. Since the UCP’s commanding performance on election day in April 2019, however, the polls suggest the governing UCP has lost considerable support amongst Alberta voters.

Jason Kenney and the United Conservative Party’s Brief Honeymoon

Jason Kenney was at the height of his popularity on the day that he was sworn in as Alberta’s eighteenth premier. His honeymoon, according to the polls, was brief, and ended amidst the global pandemic. Kenney’s election win, it is worth stressing again, was impressive. The UCP’s 55 per cent of the popular vote bested Ed Stelmach’s landslide victory in 2008, where the PCs captured 53 per cent of ballots cast. Even before Kenney became premier, the former federal Conservative cabinet minister was not the most popular political leader in Alberta. Polling data during the 2019 election campaign suggested voters liked NDP leader Rachel Notley (even after she had been premier for four years) more than Kenney, in fact.26 Kenney’s April 2019 election victory glow faded less than a year after his election as Figure 3.2 illustrates. According to survey research conducted by Janet Brown Opinion Research/Trend almost a year after the UCP swept to power, the governing party had dropped twelve points in popular support.27

By the spring of 2021—a year into the government’s controversial handing of the pandemic—support for the governing UCP had dropped by another ten points, below the NDP.28 By the time Kenney announced his intention to step down as party leader in May of 2022, polls consistently showed the UCP trailing the NDP in public opinion. Kenney’s critics, in fact, frequently used the premier’s unpopularity to argue his continued leadership all but assured an NDP victory in the next provincial election expected in May of 2023. Former Wildrose Party leader Brian Jean—arguably, Kenney’s biggest critic—returned to politics in the winter of 2022, running successfully for the UCP in the Fort McMurray-Lac La Biche by-election on a “brass knuckles” promise to overthrow Kenney as leader of the UCP.29 Two months later, Kenney resigned as party leader after receiving a lukewarm endorsement of his leadership from UCP members, telling his party that the bare majority was not “adequate support to continue as leader.”30

Jason Kenney’s approval ratings also dropped significantly from a high of nearly 50 per cent in 2018 to below 20 per cent in the aftermath of his controversial handling of the pandemic in 2021. Figure 3.331 illustrates Kenney’s drop in voters’ estimations and NDP leader Rachel Notley’s higher favourability since the spring of 2020. By the time Kenney made his surprise announcement to step down as party leader, less than a third of Albertans approved his leadership, while nearly double that (54 per cent) of Albertans approved of opposition leader and former NDP premier Rachel Notley.

Figure 3.2. Vote Choice (United Conservative Party & New Democratic Party), 2018–2022

Figure 3.3. Approval of Provincial Leaders, 2018–2022

Discussion

All the 2019 polls got the winner right. But predicting which party was going to come out on top was hardly a high bar for polls to clear. At best, the 2019 polls were only marginally better than 2015. Importantly, the 2019 election campaign polls persisted in systematically underestimating conservative support in the province of Alberta. Brown and Santos, in their examination of the 2015 polls, highlight how conservative support typically came up short in that campaign’s survey. Pollsters, as Brown and Santos suggest, largely “got a pass” for underestimating PC’s support because that election ended the party’s more than four decades in power.32 With no single party dominating Alberta politics anymore33 polling accuracy is increasingly important. More competitive elections—and even a minority government, which Alberta has never had—are possible. Estimating party support is important. Yet, Alberta political surveys traditionally underestimate conservative support in the heartland of Canadian conservatism. It is increasing clear that any read of an Alberta provincial poll should proceed carefully and assume that no poll probably captures the true extent of conservative support in the prairie province.

In Alberta in 2019, neither the interview mode (how people were asked who they planned to vote for) nor the method of coming up with the sample of people (random digit dialing [RDD], online panels) guaranteed that the poll accurately captured voters’ intentions. Let’s put the three surveys that came close to predicting the UCP’s actual vote under the microscope. Forum Research’s first poll, which correctly pegged UCP support at 55 per cent, used IVR to ask voters who they planned to vote for in the provincial election. Janet Brown Opinion Research/Trend also came close, predicting UCP support at 53 per cent. Its sample was collected using RDD. Respondents could choose either to share their vote intention with a live telephone interviewer or receive an email to do the survey online. A vast majority (90 per cent) chose to talk with a human. Four days before the 16 April election, the Angus Reid Institute’s survey estimated UCP support at 52 per cent. This prediction came from an online survey panel. These three different polls achieved similar results despite using different methods. It is worth emphasizing that most of the polls did not predict the UCP’s commanding lead. But underestimating conservative support is not new—and it is not an Alberta anomaly either.

So-called shy Tories or reluctant Republicans perplex pollsters. Surveys around the world have failed to adequately gauge conservative support. At the national level in Canada, polls collectively underestimated Conservative support in Canada’s 2019 federal election.34 In the UK in 2015, most pre-election polls predicted a hung parliament. David Cameron’s Conservatives won a majority. In Australia four years later, horse race polls there also underestimated conservative support in that country’s federal election.35 Polls in 2016 and 2020 underestimated support for Donald Trump in the US presidential election. Polling experts have offered a few theories for why polls underestimate conservative voters, including (1) these voters refuse to participate in surveys; (2) conservative voters, fearing retribution for their views, hedge or lie about who they intend to vote for; (3) pollsters do not reach enough conservative-leaning voters (unrepresentative samples); (4) the people who take polls are different than the people who vote; and (5) the voter models used by polling firms are possibly flawed.

On top of not wanting to share how they vote, some experts have hypothesized that these elusive conservative voters simply slip the pollsters’ sampling net. The thinking is that these voters do not answer or hang up on pollsters, especially when IVR or robocall polls reach them. These voters simply mistrust pollsters. There is, of course, a long history of conservatives casting doubt on public opinion polls. President Richard Nixon often described a “silent majority” that pollsters were not hearing. Post-truth politics accelerated the already declining trust and cynicism that many voters, especially conservative ones, have in institutions.36 This overarching lack of trust in institutions has also tarnished polls.37 The mistrust is particularly pronounced amongst Republicans in the US. Donald Trump, after all, frequently challenges the accuracy of polls, calling the ones predicting his loss “fake.”38 There also may be an out-and-out difference between the people who participate in polls and the people who eschew them, resulting in a systematic bias. Robert Putnam in 2001, in fact, found that people who have low levels of trust in people and institutions are less likely to participate in phone surveys.39 Veteran pollster and Obama presidential campaign advisor David Shor echoes Putnam, noting that people who are more likely to participate in polls in the US are more agreeable, and have higher levels of trust, which, in turn, results in a partisan non-response bias being baked into polls.40

Polling leading up to the UK general election in 2015 systematically under-represented conservative supporters.41 The polling experts who reviewed what went wrong with the UK polls concluded the industry needs to shift its “emphasis away from quantity and towards quality” and to be “more imaginative and proactive” in their efforts to find elusive conservatives.42 The final report for the British Polling Council and the Market Research Society recommended pollsters work harder to recruit samples that mirror the makeup of the population.43 American pollster David Shor, on the other hand, is not so sure that traditional survey methods can overcome the partisan non-response that results in undercounting Republican support in the United States. Weighting results against census data, he contends, will not fix the problem. “There used to be a world,” he said in a 2020 interview with Vox, “where polling involved calling people, applying classical statistical adjustments, and putting most of the emphasis on interpretation.”44 Shor advocates getting more sophisticated by combining polling data with voter files and proprietary first-party data, and using machine learning to interpret the combination of data points. Shor’s solution may work in the United States, but in Canada, voter information does not include the party identification or voting history that Shor suggests incorporating into polling analysis.

Some Humility about Polling Results

Polls matter.45 They can shape public discourse, influence campaigns, and motivate parties and caucuses to overthrow their leaders. Voters—especially those looking to vote against an incumbent—sometimes turn to polls to see who or what party has the best chance of winning. Pollsters—and the news media—need to be much more mindful about the potential influence of polls. Transparency about the limits of polls is needed now more than ever, from both the pollsters and the news media that report the data. In the wake of the British polling failure in 2015, Sturgis urged the public—and the news media—to recognize that polls are not perfect. “Even if we move to the most expensive random survey that you can possibly imagine,” he told the Guardian, “there would still be a chance that you would get it wrong.” No pollster, of course, wants to get it wrong. But, after all, the probability theory on which polling rests suggests there is a chance it can happen from time to time. Poll aggregators and election forecasters also need to be interpreted with a critical eye. News organizations and poll aggregators frequently predict the outcome of races, even attaching probabilities to certain outcomes. There is a difference between polls and predictors.

The News Media’s Addiction to Polls

Every day, journalists assess the veracity of sources and information. They sort fact from fiction in an ocean of misinformation and disinformation. Journalists seek out the truth, guided by principles such as accuracy, fairness, balance, impartiality, and integrity. Yet, as the evidence presented in this chapter clearly shows, Alberta journalists did not train their usual skepticism on the public opinion polls during the 2019 provincial election campaign. The polls got a pass. A tightening horse race is a better story than UCP cruising to an expected easy victory. A tight race is a better narrative. Considerable evidence from journalism sociology highlights the news media’s proclivity to seek out and highlight tension and conflict.46 It makes for a better story. This bias, arguably, blinded Alberta journalists. Sure, campaigns can matter, but polls consistently—since at least a year ahead of the election—showed the UCP on track to win big.47 Some research suggests news organizations are incapable of comprehending the data.48 Perhaps, news executives need to build that expertise into their newsrooms. At a minimum, they need to do a better job of detailing the polls’ methodology they report. Maybe, having to think about the margin of error and the probability of incorrect estimations might spark some caution in political journalists’ minds. As well, it might help their audiences interpret the results more critically. In addition, journalists need to become more reflexive about Alberta’s history of flawed polls.

It is, indeed, remarkable that journalists were so uncritical of the campaign polls in 2019. Alberta journalists had been burned by bad polls in recent elections. Remember, rogue polls in Calgary’s municipal election with its “catastrophic polling failure” should have made journalists more skeptical of horse race survey data. CBC News in Calgary, of note, conceded it should have been more circumspect of Mainstreet Research’s perplexing polling numbers.49 As well, only seven year earlier, all the polls in the 2012 provincial election pointed towards a Wildrose Party win that never materialized. As this chapter makes clear, Alberta polls consistently underestimate conservative support. Journalists need to incorporate that knowledge into their reporting on polls.

The Unpopular United Conservative Party

Since the spring of 2020, polls have suggested an uncertain future for the UCP. As detailed above, the NDP overtook the governing UCP in public opinion surveys in late 2020. As well, Jason Kenney’s personal popularity plummeted alongside his party’s precipitous drop in public support. Many long-time political watchers blamed Kenney’s controversial handing of the devastating fourth wave of COVID-19. Dubbed the “Kenney effect,” analysts suggest Kenney’s personal unpopularity even hurt federal Conservative at the ballot box in the 2021 national election. The UCP leader faced down a caucus revolt just days after the federal vote.50 As Duane Bratt and Bruce Foster have highlighted, “big tent” conservative parties are “fragile,” and Canada’s political history is filled with right-wing parties splintering and merging.51 The pandemic exposed real and pronounced divisions in the UCP over how best to handle COVID-19. Duane Bratt argues convincingly that while Kenney’s underlying conservative ideology—and its emphasis on personal responsibility, individual freedom, and small government—underlies his controversial response to the pandemic, his worldview is, nevertheless, “out of touch with Alberta values.”52 No matter what his reasons, the response appears to have hurt Kenney and his party politically. In response, the opposition NDP attempted to frame Kenney’s handling of the pandemic as politically motivated and not aligned with the public health measures most Albertans support. Unlike the PC dynasty, the UCP faces a single and capable opposition party led by a former premier. In the truest sense of the concept, the opposition NDP are a government in waiting. Only four years ago, the New Democrats held power. Alberta politics is decidedly more competitive, and this viable alternative for voters has complicated Kenney’s political fortunes. Detractors in his own party, in fact, used Kenney’s vulnerability to attack him.

Kenney’s biggest threat turned out to be within his party. While the premier managed to stare down caucus critics such as MLAs Todd Loewen and Drew Barnes and fend off a full-fledged caucus revolt in September of 2021, he could not escape the wrath of his party’s members. The melodrama associated with the internal skirmishes and infighting also, arguably, tarnished the UCP’s image as the no-nonsense, hard-working, pro-business government that would stand up for everyday Albertans. Caucus revolts and bruising leadership battles, especially during a crisis-filled global pandemic, likely did not instill confidence in many Albertans’ minds. On top of that, the UCP seemingly stumbled from one crisis or gaffe to another. From 2020’s Alohagate, where UCP MLAs and staffers jetted off to international Christmas vacation destinations after telling Albertans to hunker down for the holiday, to Premier Kenney’s prolonged holiday absence as a fourth wave of COVID-19 surged in August of 2021, the UCP’s political communication was often tone deaf. Repairing that damage falls to the party’s new leader. It will not be easy.

The governing party faces many challenges. The UCP was largely elected on a promise to get Alberta’s economy cooking with oil and gas again. But the province still faces tough economic challenges and volatile forces outside its control.53 On top of that, health care—with the lingering effects of the pandemic and the political headache of clearing the backlog of delayed medical procedures and cancelled surgeries—will challenge the UCP. The NDP, with its history of being the first North American government to establish universal single-payer medical insurance in Saskatchewan in 1962, tends to own health care as an issue in many voters’ minds. The issue could play a big role in the coming 2023 provincial election. What comes next will be fascinating to watch. Gone, it seems, are the days of political dynasties in Alberta, making the need for accurate polls even more important in assessing what is important to Alberta voters and how they feel about the public policy politicians propose. Let’s hope during the coming 2023 election campaign pollsters do a better job of capturing a truly representative sample of Albertans, and journalists and consumers of polls approach the numbers with some caution and humility when interpreting them.

Notes

1 Pierre Bourdieu, “Public Opinion Does Not Exist,” in Communication and Class Struggle 1, eds. Armand Mattelart and Seth Siegelaub (New York: International General, 1979), 124–130; Jürgen Habermas, The Structural Transformation of the Public Sphere: An Inquiry into a Category of Bourgeois Society (Cambridge, MA: MIT Press, 1991).

2 Alyssa Julie, “Mainstreet Research Apologizes for ‘Catastrophic Polling Failure’ in 2017 Calgary election,” Global News, 20 October 2017, https://globalnews.ca/news/3816393/mainstreet-research-apologizes-for-catastrophic-polling-failure-in-2017-calgary-election/ (accessed 30 September 2021).

3 Brooks DeCillia, “Mainstreet to release findings of investigation into ‘big polling failures’ during Calgary election,” CBC News, 10 December 2017, https://www.cbc.ca/news/canada/calgary/mainstreet-polling-failures-investigation-calgary-election-1.4441063 (accessed 30 September 2021).

4 Polling failures at the provincial level are also notable in the 2013 British Columbia, the 2014 Quebec, and the 2014 Ontario provincial election campaigns.

5 Bruce Cheadle, “Pollsters relieved at getting it right in Alberta’s unlikely swing to the left,” Maclean’s, 6 May 2015, https://www.macleans.ca/politics/pollsters-relieved-at-getting-it-right-in-albertas-unlikely-swing-to-the-left/ (accessed 31 August 2021).

6 Janet Brown and John Santos, “Marginally Better: Polling in the 2015 Alberta Election,” in Orange Chinook: Politics in the New Alberta, eds. Duane Bratt, Keith Brownsey, Richard Sutherland, and David Taras (Calgary: University of Calgary Press, 2019), 79–101.

7 Random surveys often get reported with a 95 per cent confidence interval of an estimated population parameter. This represents the range that would contain the true mean value in the population 95 per cent of the time if that same sample design could be replicated in the population an infinite number of times at the same time. The “margin of error” is technically actually the “margin of sampling error,” and only covers error from sampling and no other methodological artefacts like field procedure, question wording, etc.

8 Elizabeth Goodyear-Grant, Antonia Maioni, and Stuart Soroka, “The Role of the Media: A Campaign Saved by a Horserace,” Policy Options 25 (2004): 86–91; J. Scott Matthews, Mark Pickup, and Fred Cutler, “The Mediated Horserace: Campaign Polls and Poll Reporting,” Canadian Journal of Political Science 45 (2012): 261–87.

9 Benjamin Toff, “The ‘Nate Silver Effect’ on Political Journalism: Gatecrashers, Gatekeepers, and Changing Newsroom Practices around Coverage of Public Opinion Polls,” Journalism 20, no. 7 (2019): 873–889.

10 Jens Olav Dahlgaard, Jonas Hedegaard Hansen, Kasper M. Hansen, and Martin V. Larsen, “How Are Voters Influenced by Opinion Polls? The Effect of Polls on Voting Behavior and Party Sympathy,” World Political Science 12, no. 2 (2016), 283–300.

11 A classic content analysis is a trusted means of quantify phenomena in news media. It effectively turns words into numbers to draw statistical inferences about the news media. See Klaus Krippendorff, “Content Analysis: An Introduction to Its Methodology,” (Thousand Oaks, CA: Sage, 2018) for an overview of the method and its benefits.

12 An independent double-checking produced an intercoder reliability above 80 per cent for all the variables coded in this study, confirming a valid coding process.

13 This study’s public opinion dataset (see Tables 3.1, 3.2, and 3.3) was compiled after the 2019 election, using polling information detailed in news releases and mainstream news organizations and “poll aggregators” such as ThreeHundredEight (www.threehundredeight.com) and CBC Poll Tracker (https://newsinteractives.cbc.ca/elections/poll-tracker/alberta/). This work was done independent of polling firms, the news media, or political bloggers.

14 EKOS, “Rachel Notley Closing Gap on Jason Kenney,” 31 March 2019, https://www.ekospolitics.com/index.php/2019/03/rachel-notley-closing-gap-on-jason-kenney/

15 This research uses the reported margin of error of each poll, as reported by the polling firm. It is likely that these margins of errors exclude undecideds and non-voters, meaning the final reported “vote intention” proportions are smaller than the total reported sample size of the poll. Accounting for the smaller sample size would increase the margin of error. As well, it is a common convention for polls to offer one global margin of error for the entire poll. Technically, the margin of error is different for each proportion in a poll.

16 This study does not offer an “index of accuracy” akin to Elizabeth A. Martin, Michael W. Traugott, and Courtney Kennedy, “A Review and Proposal for a New Measure of Poll Accuracy,” Public Opinion Quarterly 69 (2005): 342–69, which uses similar criteria in quantitative evaluations of polling accuracy.

17 Total absolute error is calculated by summing the absolute differences between each party’s level of support in each public poll and the actual vote totals on election day. This measure replicates Brown and Santos, “Marginally Better.” See also David Coletto, “Polling and the 2015 Federal Election,” in The Canadian Federal Election of 2015, ed. Jon H. Pammett and Christopher Dornan (Toronto: Dundurn, 2016), 305–26.

18 The preliminary data from this study was presented as a working paper at the Prairie Political Science Association in 2019. “Alberta’s ‘Shy Tories’: Why Public Polls Underestimate Conservative Voters,” with John Santos (presentation, Prairie Political Science Association (PPSA), Banff, AB, 13 September 2019).

19 “Race tightening in Alberta provincial election,” Global News, 9 April 2019, https://globalnews.ca/video/5149284/race-tightening-in-alberta-provincial-election/ (accessed 30 September 2021).

20 Chandler Walter, “Mainstreet poll shows UCP and NDP within 2% of each other in YYC,” Daily Hive, 16 April 2019, https://dailyhive.com/calgary/mainstreet-pollucp-ndp-close-calgary (accessed 30 September 2021).

21 Full disclosure: I have worked with Janet Brown in the past as a journalist on a political research project conducted by CBC News. Brooks DeCillia, “The ‘Road Ahead’ for Alberta Voters As CBC Surveys Motivations and Polarization in Lead Up to Next Election,” CBC News, 21 February 2018, https://www.cbc.ca/news/canada/calgary/road-ahead-election-alberta-survey-motivations-1.4544536.

22 Data from: David Coletto and Bryan Breguet, “The Accuracy of Public Polls in Provincial Elections,” Canadian Political Science Review 9 (2015): 41–54; and David Coletto, “Polling and the 2015 Federal Election,” in The Canadian Federal Election of 2015, ed. Jon H. Pammett and Christopher Dornan (Toronto: Dundurn, 2016), 305–26.

23 Data from “338Canada Pollster Ratings,” 338Canada, last updated 27 September 2021, https://338canada.com/pollster-ratings.htm#ca2019 (accessed 30 September 2021).

24 Data from “338Canada Pollster Ratings.”

25 As calculated by Brown and Santos, “Marginally Better”; and Coletto and Breguet, “The Accuracy of Public Polls.” Data for 2019 federal election came from “338Canada Pollster Ratings” and was calculated by the author.

26 Janet Brown, “Global Petroleum Show: Provincial Election Survey,” Janet Brown Opinion Research, 3 April 2019, http://planetjanet.ca/wp-content/uploads/2019/04/2019-04-02-NWPA-March-2019-Election-Poll-Report.pdf (accessed 29 September 2021).

27 Janet Brown, “Special Projects,” Janet Brown Opinion Research, http://planetjanet.ca/special-projects/ (accessed 2 June 2022).

28 Janet Brown, “Special Projects.”

29 Graham Thompson, “Brain Jean’s bylection win sets the stage for showdown with Alberta Premier Jason Kenney,” CBC News, 16 March 2022, https://www.cbc.ca/news/canada/edmonton/brian-jean-s-byelection-win-sets-the-stage-for-showdown-with-alberta-premier-jason-kenney-1.6386455 (access 1 June 2022).

30 Michelle Bellfontaine, “Alberta Premier Jason Kenney resigning as UCP leader despite narrow win in leadership review,” CBC News, 17 May 2022, https://www.cbc.ca/news/canada/edmonton/alberta-premier-jason-kenney-resigning-as-ucp-leader-despite-narrow-win-in-leadership-review-1.6457221 (accessed 29 May 2022).

31 Janet Brown, “Special Projects.”

32 Brown and Santos, “Marginally Better,” 2019.

33 Anthony M. Sayers and David K. Stewart, “Out of the Blue: Goodbye Tories, Hello Jason Kenney,” in Orange Chinook, 399–423.

34 Phillipe J. Fournier, “How polls keep underestimating the Conservative vote: 338Canada,” Maclean’s, 5 August 2020, https://www.macleans.ca/politics/ottawa/338canada-how-polls-keep-underestimating-the-conservative-vote/ (accessed 30 September 2021).

35 Ben Westcott, “Australia Election 2019,” CNN, 18 May 2019, https://edition.cnn.com/world/live-news/australia-election-day-2019/index.html (accessed July 22, 2022).

36 For this argument, see Emily Ekins, “Why Did Republicans Outperform the Polls Again? Two Theories,” FiveThirtyEight, 2 March 2020, https://fivethirtyeight.com/features/why-did-republicans-outperform-the-polls-again-two-theories/ (accessed 29 September 2021).

37 Ekins, “Why Did Republicans Outperform.”

38 “Survey: Many Americans don’t believe polls are accurate,” The Hill, 27 December2020, https://thehill.com/hilltv/what-americas-thinking/423023-a-majority-of-americans-are-skeptical-that-public-opinion-polls (accessed 29 September 2021).

39 Richard Putnam, “Social Capital: Measurement and Consequences,” Isuma: Canadian Journal of Policy Research 2 (2001): 41–51.

40 For Shor’s thoughts, read Dylan Mathews, “One pollster’s explanation for why the polls got it wrong,” Vox, 10 November 2020, https://www.vox.com/policy-and-politics/2020/11/10/21551766/election-polls-results-wrong-david-shor (accessed 29 September 2021).

41 Parts of these recommendations first appeared as commentary in 2019 in Policy Options. See Brooks DeCillia, “The underestimated conservative voter in Alberta polls,” Policy Options, 25 June 2019, https://policyoptions.irpp.org/magazines/june-2019/underestimated-conservative-voter-alberta-polls/ (accessed 30 September 2021).

42 Tom Clark and Frances Perraudin, “General election opinion poll failure down to not reaching Tory voters,” Guardian, 19 January 2016, https://www.theguardian.com/politics/2016/jan/19/general-election-opinion-poll-failure-down-to-not-reaching-tory-voters (29 September 2021).

43 Patrick Sturgis et al. “Report of the Inquiry into the 2015 British General Election Opinion Polls,” British Polling Council and the Market Research Society, March 2016, https://benjaminlauderdale.net/files/papers/PollingInquiryReport.pdf (accessed 29 September 2021).

44 For Shor’s thoughts, read Mathews, “One pollster’s explanation.”

45 Patricia Moy and Eike Mark Rinke, “Attitudinal and Behavioral Consequences of Published Opinion Polls,” in Opinion Polls and the Media, eds. Christina Holtz-Bacha, Jesper Strömbäck (London, UK: Palgrave Macmillan, 2012), 225–45.

46 See, for example, Pamela J. Shoemaker and Stephen D. Reese, Mediating the Message in the 21st Century: A Media Sociology Perspective (New York: Routledge, 2014).

47 See, for example, Brooks DeCillia, “United Conservative Party on track to win big in Alberta, says poll,” CBC News, 28 April 2018, https://www.cbc.ca/news/canada/calgary/road-ahead-poll-ucp-win-next-election-1.4636786 (accessed 30 September 2021).

48 Benjamin Toff, “The ‘Nate Silver Effect.’” s:

49 Brooks DeCillia, “Mainstreet to release findings of investigation into ‘big polling failures’ during Calgary election,” CBC News, 10 December 2017, https://www.cbc.ca/news/canada/calgary/mainstreet-polling-failures-investigation-calgary-election-1.4441063 (accessed 30 September 2021).

50 Drew Anderson and Elise von Scheel, “Jason Kenney survives caucus meeting with leadership review to come,” CBC News, 21 September 2021, https://www.cbc.ca/news/canada/calgary/jason-kenney-premier-alberta-leadership-review-1.6185362 (accessed 30 September 2021).

51 Duane Bratt and Bruce Foster, “The fragility of a ‘big tent’ conservative party,” CBC News, 25 July 2019, https://www.cbc.ca/news/canada/calgary/conservative-party-splitting-merging-western-canada-1.5220417 (accessed 30 September 2021).

52 Duane Bratt, “How Kenney’s political ideology is out of touch with Alberta values,” CBC News, 8 September 2021, https://www.cbc.ca/news/canada/calgary/kenney-political-ideology-1.6167303 (accessed 30 September 2021).

53 David Bell, “Alberta’s economic outlook looking up but ‘bumpy ride’ ahead, says economist,” CBC News, 19 May 2022, https://www.cbc.ca/news/canada/calgary/alberta-economic-outlook-rob-roach-1.6459790 (accessed 1 June 2022).