Democratic Innovations: Polis and the political process

Open Rights Group and Demos yesterday published an exciting new report that provides insight on public attitudes towards data driven campaigning.

The report covers the use of an exciting democratic innovation; a deliberative platform called Polis. Polis has been used in Taiwan to involve the public in policy making in an innovative, cost effective way.

The report found, amongst other things:

1. While the public by no means speaks with one voice on the issue, statements in support of stricter rules around data driven political campaigning gained the greatest levels of support.
2. Overwhelming majorities agreed political campaigns should have to obey the same rules when they are advertising online as they do in leaflets or on TV (88%), that greater transparency is needed around political funding (84%), that political campaigns should publish all advertising materials (81%) and that they should publish how much they are spending (79%).
3. For the majority of opponents of regulation, it is not a principled point about freedom of speech or faith in the political system. Rather, they believe campaigns have little impact on their vote, and hold politicians and authorities holding them to account alike in contempt.

However perhaps the most significant of all was the demonstration of a positive use case for technology to enhance, rather than harm, the democratic process.

Pascal Crowe, Data and Democracy Project Officer at ORG said:

“This report supports previous research on this topic by Open Rights Group. The public do not want to be targeted and have their rights abused by data driven political campaigns. While transparency is part of the solution, the public also just wants tight rules and heavier enforcement.

By contrast, Polis makes a positive case for using innovative technology in the democratic process. Rather than narrowing debate, encouraging polarisation and demeaning trust, this tool provides the opportunity to build public consensus in policy making.”

ORG and Demos will be publicly discussing Polis, its findings, and potential for democracy Monday 10th August 2020.

HTML Version

Polis and the political process: Mapping public attitudes regarding data driven political campaigning, and an exploration of Polis as a democratic innovation

Josh Smith

Toby O’Brien

Harry Carr

Pascal Crowe 

Matthew Rice

Open Access. Some rights reserved. As the publisher of this work, Demos wants to encourage the circulation of our work as widely as possible while retaining the copyright. We therefore have an open access policy which enables anyone to access our content online without charge. Anyone can download, save, perform or distribute this work in any format, including translation, without written permission. This is subject to the terms of the Demos licence found at the back of this publication. 

Its main conditions are: 

• Demos and the author(s) are credited
• This summary and the address www.demos.co.uk are displayed
• The text is not altered and is used in full
• The work is not resold
• A copy of the work or link to its use online is sent to Demos. 

You are welcome to ask for permission to use this work for purposes other than those covered by the licence. Demos gratefully acknowledges the work of Creative Commons in inspiring our approach to copyright. To find out more go to www.creativecommons.org

Contents

Executive summary

Introduction

Key findings

Polis: A participant’s eye view

The public’s view of data driven campaigns

Conclusions

The democratic potential of Polis

Methodology

Literature Review

Annex

Acknowledgements

We would like to thank the Joseph Rowntree Reform Trust for their donation to the project, Brendan Russell and Mathew Cudd at Dynata for their expert technical support, Asli Atay for her eagle-eyed review, and to the Demos and ORG teams for being wonderful.

All mistakes remain the authors’ own.

Executive summary

This project has two distinct aims: to explore the potential of the research tool Polis as a democratic innovation, and – using Polis – to explore public attitudes around data driven political campaigning. 

This report provides a detailed use case of Polis, documenting the experience as a researcher and as a participant, issues faced and lessons learnt – and an overview of the topic-based findings of the research itself.

• The public fall into three distinct groups in terms of their attitudes around data driven political campaigning: a pro-regulation group, a group who are distrustful of regulators and politicians alike, and a group who support fact verification in campaigning, but are split on wider regulation measures and more likely to pass on questions across the board.
• Support for the verification of claims made by politicians during campaigns drew the greatest degree of consensus and elicited the most enthusiastic response and creativity in terms of submitted statements.
• While the public by no means speaks with one voice on the issue, statements in support of stricter rules around data driven political campaigning, and for those responsible for infractions to be held to account, gained the greatest levels of support.
• For the majority of opponents of regulation, it is not a principled point about freedom of speech or faith in the political system. Rather, they believe campaigns have little impact on their vote, and hold politicians and authorities holding them to account alike in contempt.
• On both sides, support is most forthcoming where it is rallied in opposition to an emotively charged enemy – lying politicians for those in favour of greater regulation, red tape wielding bureaucrats and shadowy authorities seeking to control public life for those who would prefer less regulation.

Introduction

Public understanding of and attitudes around data driven political campaigning is a key area of interest for both Demos and the Open Right Group (ORG). Many, including the Electoral Commission and the Information Commissioners’ Office (ICO), see our electoral laws as having been left in the dust by the advent of digital advertising and other data driven campaigning techniques. The future of data driven political campaigning is intertwined with the fundamental issues of our age: polarisation and the rise of populism, declining faith in democracy, and regulation of hegemonic tech companies.

This report is about data driven political campaigning – but it is also an exploration of a methodological process. Demos has pioneered the use of Polis, a tool which allows respondents to interact with each other constructively: mapping out the lay of the land with regard to opinion on a given subject, identifying attributes that define and differentiate between different clusters of opinion, and crucially highlighting areas of consensus between otherwise disparate attitudinal groups.

In particular, Demos is the first organisation anywhere to conduct Polis using a nationally representative sample. This exciting democratic innovation provides a uniquely rich view of public attitudes around a given subject at an affordable and convenient price point and turnaround time – enabling a grounded theory study with citizens providing their verbatim views and able to react to views they would not otherwise be exposed to, at a scale where nationally and demographically representative inferences can be drawn from the results.

Below, we use the results of our week-long conversation to evaluate the use of Polis in the UK, in three steps. Firstly, we show what the conversation looks like from a participant’s perspective, and outline how Polis is designed to enable healthy, democratic debate. Secondly, we evaluate the process on the strength of its findings, exploring what the process can tell us about how the UK feels about political campaigns and their use of data. Finally, we assess the platform’s potential for use as a democratic innovation, reflecting on our experiences while running the survey, and the extent to which different groups were engaged in the process. This section also goes over some lessons learned, which we hope will be useful to anyone planning to follow in our footsteps.

Key Findings

• The public fall into three distinct groups in terms of their attitudes around data driven political campaigning: 

• A pro-regulation group in favour of stricter rules around fact-checking and transparency, the majority of whom are middle class and voted Remain; 
• A group who are distrustful of regulators and politicians alike, who would be equally relaxed about deregulation of political campaigning or for campaigning to be abolished entirely, are sceptical about the ability of campaigns to influence their votes, and who were most likely to be working class and have voted Leave; 
• And a young group, the majority of whom are aged 16-34, who joined the other groups in overwhelming support for measures to stop politicians misleading the public, but were split on wider regulation measures and more likely to pass on questions across the board. 

• There is support across the board for verification of claims made by political campaigns and for politicians to be held to account where they mislead the public; greater transparency around campaign advertising methods and funding; the maintenance of fixed spending limits; and a need for companies providing services to data driven campaign to do more – and for regulation requiring more of them.

• Areas of greater division included suggestions for the deregulation of political campaigning, which were most popular when framed as cutting red tape and as in opposition to authorities “controlling” politicians; the abolishment of political campaigning entirely; scepticism about the extent to which political campaigns affect people’s voting behaviour; and whether national politicians should be subject to international oversight.

Polis: a participant’s eye view

In order to conduct this conversation, we first recruited a nationally representative sample of 997 respondents. Each of these, after completing a short demographic survey, were directed to the page pictured below.

Fig. 1 – A participants-eye-view of the Polis interface 

This page presents each participant with the following:

1. A short, neutral introduction to the topic and a note explaining the voting process. 
2. A statement, with which they can vote to agree, disagree, or pass over if they are unsure. These statements are initially drawn from a ‘seed’ list compiled by researchers, but as the conversation progresses they will include statements submitted by other participants. After voting on a statement, it disappears, and a new one is presented.
3. A text box inviting them to input a short statement of their own, limited to 140 characters. This is then sent to a moderation queue. If it passes, it is added to the list of statements shown above and other participants are able to vote on it. 

That’s it. If a participant manages to vote on every statement available, they are given the option to enter their email and to be notified when more statements are added, meaning engaged participants can return to the conversation as it progresses, and cast votes on any new ideas.

The participant-solicited nature of this process is key to Polis’ deliberative and democratic promise. By asking participants to submit their own statements after reading those already contributed to the debate, the tool leaves space for them to raise issues they believe have been overlooked, or to provide a new inflection on subjects which are being discussed. Furthermore, these views are presented anonymously – voters are not told whether a statement was seeded by moderators, stated by experts, or contributed by their peers. Each is judged on its own merits. 

Critically, and at first glance unusually for a tool built for deliberation, Polis does not allow for dialogue. In contrast to the design of every major social media platform, participants cannot directly reply to one another, or leave comments under submitted responses. This avoids the problems of debate descending into trolling and abuse and allows people to provide their true views, even where they are controversial, without any fear of receiving intimidating or aggressive replies. By providing a means of removing the heat from political discourse, Polis may point to a way of improving the health of our online spaces.

Participant visualisation

Once a suitable number of people have voted within the discussion, participants are shown how their views compare to those of their peers. This is shown in the form of a graph, which splits participants into groups depending on their voting records. This graph also shows participants where they stand in relation to the larger group; the example below shows a user who sits right in the middle of the pack.

Fig. 2 – Visualising public opinion

This visualisation is relatively impenetrable itself, but the buttons at the base of the graph can be used to provide some useful context. For example, clicking ‘Majority Opinion’ will show participants a few opinions the majority agreed or disagreed with, and the percentage of the whole group who voted each way.

Fig. 3 – Consensus statements 

By clicking on the groups themselves, users can see which statements characterised that group – the views on which they tended to vote differently from the rest of the cohort. In the examples above, this suggests that the majority agree on the need for politicians to be honest, but there is a division between groups A and C on how they should be regulated; a division we will explore more fully below.

Fig 4 – Group A tended to disagree that regulation should be cut…

Fig 4.i …while group C were more sympathetic to unrestricted political speech

This visualisation is designed to help participants orient themselves in the conversation. It’s not perfect, and faces challenges common to technologies which groups people using maths – it’s difficult to read at first glance, and it’s not immediately clear what determines how groups are laid out in space. The aim of this view, however, is to show participants – at least, those engaged enough to explore it – not only how the debate is playing out and where consensus lies, but also to see where they stand in relation to their peers. As with the ability to contribute new statements, this functionality is designed to give participants a feeling of ownership; to clearly illustrate that there are others – fellow participants – who hold differing points of view on some, but not all, matters.

The public’s view of data driven campaigns

By the end of the conversation, Polis’ grouped map showed that participants fell into three distinct clusters. To work out whether certain views characterised these groups, we used two datasets available through Polis. 

The first was a standardised report, available from the platform’s ‘back end’ – an administration and reporting site which also allows researchers to create and seed new surveys, monitor those underway, and moderate comments. The second was a full data export, detailing every vote and comment left by each participant. As this table also contained an ID linking voters to the demographic survey they had taken before entering Polis, we were able to connect respondents to their basic demographic information and weight responses to the profile of the UK population. Responses were weighted by gender, age, region, social grade, household income, education, and past vote in the 2019 general election and the EU referendum. An overview of the weighted and unweighted demographic data can be found in the annex to this report.

Two caveats are important to make here:

• The demographics naturally fall out to make each statement similar to the overall sample, but not every participant responded to every statement, and responses are not weighted to be representative for every individual statement. (For this survey, young people tended to be recruited earlier in the fieldwork process and so were less likely to vote on statements submitted later – a key learning going forward is to avoid this by ensuring demographics are recruited at a steady rate throughout fieldwork.) 
• People tend to be more likely to “agree” to any given statement than they would be to “disagree” with the reverse statement. Statements from all perspectives are more likely to have higher levels of support than they would if asked in a format where respondents chose between opposing statements. Results should be understood in that context – results for each individual statement show respondents’ reactions when presented with the given view in isolation, which should be compared with results for alternative statements presented to respondents in the same way. 

Participants grouped together as follows:

Group A: The regulators – 404 participants (38%)

Group A tend to identify as having strong views on the subject of data-driven campaigning, and over half of them believe campaigns had an effect on their votes. They are the group most likely to be enthusiastic about regulation, supporting proposals for campaigns to publish advertising materials, details of spending and efforts to restrict ‘fake news’. Only 10% agree that the rules work well as they are. 

People in Group A are notably more likely to be middle class – 65% are ABC1 compared with 55% overall. Most of this group voted for Remain in the EU referendum (48%, compared with 34% voting Leave and 18% who did not vote).

Group B:  The undecideds – 171 participants (18%)

The smallest group in the survey, Group B tend not to rally around a single set of views, and were much more likely to vote ‘Pass / Unsure’ than their peers, choosing this option 30% of the time compared to 11% for both groups A and C.

On the divisive issues of regulation and scepticism about the efficacy of political campaigning this group tended to be evenly split, with the issues which did unite them, such as the need for free speech to be balanced with accountability and the importance of spending limits in campaigns, also remaining popular with other participants. 

This group tends to be younger – more than half (52%) are aged 16-34 compared with 32% overall. 

Group C: The wary – 341 participants (35%)

Group C tend to be distrustful of both politicians and regulators; they were the group most likely to agree with the statements “I don’t trust any of the politicians of their departments to keep my data safe nor use it for the right purposes” (70% agreed) and “I don’t trust the people who regulate campaigns to be unbiased” (81% agreed). 

They are supportive of both data privacy and deregulation of political campaigning, and are the group most likely to say they have no strong views on the subject, with half of the group agreeing to this statement.

People in Group C tend to be from more working class backgrounds – 57% are C2DE compared with 45% overall. This group were more likely to vote Leave than Remain – by 47% to 30%.

A further 81 participants (9%) were not placed into any group. 

Areas of consensus

A key strength of Polis is that it highlights areas of consensus, and incentivises the development of new ideas – or communication of ideas that would otherwise be obscured to politicians, policymakers and researchers – that unite people across different groups.

Below we have included some graphs which show how voting was distributed across groups. The first column shows the percentage of users overall who agreed, disagreed or passed on a statement. Columns to the right of this show how that voting breaks down between groups.

Point of consensus 1: The overriding importance of truth

One theme, which appeared multiple times in submitted statements and generated consensus across participants, was that of honesty in politics. 

Five statements advocated for external verification of political speech, arguing that figures, statements and advertisements used by politicians should be monitored to keep them truthful. A simple statement of this idea, “Facts used in political campaigns need to be verified, and the politician should be held accountable for it”, drew the highest percentage of ‘agree’ votes of any statement in the discussion, with 90% of participants agreeing and only 4% disagreeing.

This enthusiasm for verification of political claims stands in stark contrast to the current regulatory position. While the Advertising Standards Authority and Ofcom have the power to intervene in the case of factual claims regarding products, no regulatory body in the UK has the mandate to enforce accuracy within claims made during campaigns. This is by design; the reasoning being that political speech should be protected from regulatory intervention, and that it is the job of the media, and ultimately voters, to hold politicians to account. The importance of this mission was highlighted by statements which advocated for political claims to be reliable and verifiable, without explicitly saying they should be regulated or fact-checked.

Alongside this discussion were two popular statements on free speech, asserting its importance as a fundamental right, but cautioning that that right comes with responsibility. While all of the ideas cited here were overwhelmingly popular with groups A and C, these statements around freedom of speech were notable for being rallied around by the usually cautious and divided participants in Group B – a higher percentage of participants from this group agreed with these statements than with any others.

This abundance of statements on the same topic, and the degree of consensus they attained, indicate that the trustworthiness of politicians and the facts quoted in their campaigns is crucially important to the UK public. The question which remains unresolved here is what should be done to ensure it. 

Point of consensus 2: Political campaigns should publish their campaigning methods, funding and use of data

Transparency around campaigning methods proved unifying in two ways – in very high levels of support, and perversely in an absence of further ideas or comments. 

Overwhelming majorities – between eight and nine in ten respondents – agreed political campaigns should have to obey the same rules when they are advertising online as they do in leaflets or on TV (88%), that greater transparency is needed around political funding (84%), that political campaigns should publish all advertising materials (81%) and that they should publish how much they are spending (79%). 

Respondents in both groups A and C were similarly overwhelmingly in favour of all these measures; group B were less clear cut with larger numbers unsure or disagreeing, though still with plurality support in every case.

However, three of these four statements were written and pre-loaded by Demos. In contrast to the importance of truth and verification in political campaigning, respondents had little to say off their own bat with regard to transparency around data driven campaigning. This may suggest there is more public enthusiasm for ways to hold individual politicians to account directly than to deal with more systemic issues. While transparency and practical ways to maintain it in data driven political campaigning is uniformly desired, it is not something that sparks creativity or springs as readily to mind as holding lying politicians to the fire does for the average citizen.

Point of consensus 3: Spending limits should be maintained

Fixed spending limits enjoy clear majority support across every group – including the generally anti-regulation group C and the generally apathetic group B. The uniform popularity of this view is noteworthy given the context that existing electoral spending rules are much more difficult to enact in the context of data driven political campaigning, though we did not unpack the intricacies of this issue so the results cannot be said to speak to the desirability of reform in this area specifically.

Point of consensus 4: Companies providing services to data driven campaigns should be doing more/should be more tightly regulated

Radical legal change to outlaw data driven campaigning entirely has the support of the majority of respondents – some 61% think profiling people based on their online data should be illegal. 

Across the board, statements requiring companies who provide services to data driven campaigns to do more or be more tightly regulated garnered the support of well over half of people. Three in four think there should be better options to see what data companies use to cater advertising campaigns to you (74%) and that companies store too much data about people (77%); eight in ten simply think “Data should be kept more private” (82%) and nine in ten think social media companies should do more to stop the spread of fake news (88%).

All three groups saw majority agreement across most of these statements, though again group B were more divided than A and C.

An interesting issue that was not addressed in this survey was whether regulation should be of those who commission services for data driven political campaigns rather than those who supply them; while we talked about regulation of political campaigns and their providers separately, no respondents made a connection between these to consider regulating how campaigns commission services. While the fact that this was not brought up organically suggests it is not top of mind for the public, prompting for this topic may have produced an additional dimension to the conversation, and has potential to be a fruitful topic in future research.

Areas of division

Polis can also identify areas of division. The nature of the method lends itself to building consensus; unlike in a regular poll, the statements provided by respondents are not designed to be balanced, and people will naturally seek to write their statements in ways likely to garner agreement. However, disagreements do inevitably occur even in these circumstances, and provide invaluable insights into where different groups are at odds.  

Areas of division 1: Political campaigns should be deregulated, or abolished

Statements supportive of deregulation saw a wide range of reactions. A clear majority think there should be less red tape stopping politicians saying and doing what their voters want (56% agree, 25% disagree), while at the other end of the spectrum respondents oppose politicians being free to say what they like by 58% to 28%. A clear majority think the authorities shouldn’t control what politicians are allowed to say (52% to 33% disagreeing), but far more disagree than agree that there should be less regulation to make campaigning more efficient (48% disagree, 32% agree). This vast difference suggests phrasing is particularly important for these issues – public opinion appears to be malleable depending on the way the idea is put to them.

This set of questions shows key differences between groups A and C, with group A opposed to every one of these statements, and group C supportive.

Interestingly, we see the same patterns in these groups for what would appear to be far more stringent regulations – in abolishing political campaigning entirely. Group A oppose ending political campaigning (albeit narrowly), while group C support it. 

The policies invoked in these statements would see far greater control of what politicians are allowed to say and how they are allowed to be covered. How can support for this be squared with group C’s strong support for cutting red tape?

As we go into in the next section, the majority of people in group C are also convinced that political campaigns are ineffective at swaying voters, which may make sense of this – given that premise, it would make sense to both think that regulation of data driven campaigns is unnecessary bureaucracy, and support the abolishment of political campaigning wholesale.

Areas of division 2: Targeted campaigning doesn’t affect my vote

How people feel about data driven campaigning appears to depend heavily on the way it is phrased. The majority of people (55%) say they are not worried about targeted campaigns on the basis that they still have the ability to figure out their own opinion (22% disagree); 46% say political campaigns don’t influence their voting intentions (41% disagree), while public opinion is evenly split as to whether the scale of profiling using online data is too overwhelming to worry about it (35% disagree, 34% agree).

It is noteworthy that the statement highlighting people’s feelings about the issue shows the lowest level of concern – even people in group A who are opposed to the other statements in this selection were net in agreement. On the other hand, the statement belittling the issue in principle – on the curious basis of too much of it is happening, rather than on the basis of its emotive impact – was given the shortest shrift. 

This may suggest people are concerned about these issues in principle, when asked, but in their day to day lives are not spurred to feel worried about data driven campaigning and its impacts.

As noted above, a clear majority of people in group C support every statement in this batch, while group B are characteristically split.

Areas of division 3: Politicians should not be subject to international oversight

Groups A and C are again at odds regarding international oversight. Perhaps unsurprisingly given the breakdown of the groups by EU referendum vote, group C are particularly virulently opposed to politicians having to obey international institutions (70% support national politicians not obeying ‘higher political authorities’, while 58% of group A oppose). Overall, respondents were more likely to agree than disagree (42% to 34%). 

Conclusions 

Analysis using Polis provides a unique insight into the public’s thinking around data driven political campaigning, capturing the lay of the land in terms of how attitudes intersect, and displaying people’s thoughts on the topic in their own words.

In terms of the regulation of data driven political campaigning, it is holding politicians to account that grasps public attention. There is a widespread appetite for greater regulation in terms of transparency, the application of spending limits and regulation of service providers for data driven political campaigns, but these issues elicit less public enthusiasm than holding lying politicians to the fire.

Many who oppose regulation of data driven political campaigns appear to do so on the basis of a generalised apathy or nihilism: it is not that they think politicians and campaigners can be trusted, but that regulators are just as bad – and (in their view) it doesn’t really matter anyway as such campaigns have little impact on their vote. For the majority of opponents of regulation, it is not a principled point about freedom of speech or faith in the political system – most say they would also like to see the abolition of political campaigning altogether. Rather, it appears they see regulation as a waste of time and energy, and hold the political sphere as a whole in contempt – campaigners and those looking to hold them to account alike.

The most effective messaging for those opposed to greater regulation of data driven political campaigning is to portray it as “red tape” or as authorities looking to control what politicians do and say – these arguments have appeal outside of the core supporters of deregulation, whereas arguments on the basis of efficiency or faith that the system will punish liars are far less effective. Views around deregulation appear to be much more malleable than those in favour of greater regulation, differing far more depending on different phrasing and emphasis.

On both sides, support is most forthcoming where it is rallied in opposition to an emotively charged enemy – lying politicians for those in favour of greater regulation, red tape wielding bureaucrats and shadowy authorities seeking to control public life for those who would prefer less regulation.

In overall terms, however, calls for greater regulation enjoy a far greater degree of consensus among the general public than even the most popular framing of the arguments against regulation. The public does not speak with one voice on the issue, but the weight of public opinion is behind stricter rules around data driven political campaigning, and for those responsible for infractions to be held to account.

The democratic potential of Polis

Overall, we believe this pilot to have shown that Polis succeeded as a tool for surfacing areas of consensus and encouraging debate. The analysis above demonstrated that people were able to use it to raise novel ideas, and it was effective in surfacing areas of consensus around political campaigning, as well as disagreement, between groups. 

In this section, we will step away from the subject matter of our survey and look at Polis’ use as a tool to facilitate democratic debate in the UK, through an examination of those who participated most eagerly in the discussion, the ways in which our implementation might have affected the results, and lessons learned for those wishing to run similar projects in the future.

Not all participants were engaged in the conversation

Our discussion faced a challenge common to many deliberative exercises, especially those conducted remotely, in that participants often acted in ways which suggested they were not engaged with the discussion. The most telling signal of this was the number of ‘nonconstructive’ comments submitted; i.e. comments which were unintelligible, (“hjkjhkh”, “????? say WHAT” etc.) or judged to be entirely irrelevant (“hi”, “this is boring” and so on). Some of this was fairly light-hearted – more than one user reacted to the instruction that ‘statements should be clear and concise’ by submitting “Clear and concise” as a comment. Altogether, these ‘nonconstructive’ submissions accounted for over half (52%) of all statements. 

As the results showed, the subject matter likely had an effect here – almost 1 in 3 participants agreed with the statement, submitted by the first participant to take the poll, “I have no strong views on this subject.” Given this, 48% might seem an encouraging amount of good faith participation; after all, it requires more investment to type out a considered statement than it does to mash your forehead against a keyboard. These statements were moderated out, and they didn’t affect the experience of other participants – but ‘insincere’ votes were not controlled for. 

A poll which is designed to emulate a national conversation must include some people who don’t take it too seriously. Researchers considering this approach should be prepared to deal with disengagement – whether by increasing engagement, weighting or removing activity appropriately, or leaving it in as representative of a true public discussion.

Challenges in moderation

While ‘nonconstructive’ comments were usually easy to moderate, there were two other categories of submission which presented challenges to moderators, and may affect the democratic nature of the process. The first of these was whether to ‘correct’ statements with errors in spelling or syntax, or which were typed entirely in capital letters. We decided not to do this and included statements with errors as long as they could be understood, reasoning that mistakes might humanise the process for readers, encouraging people to submit their own statements. However, there is a possibility that doing so, affected the results – while we did not code for spelling errors, people may have been less likely to vote for misspelled or ‘shouted’ comments.

More challenging were comments which either repeated a point already made, or which were considered ‘off topic’. In removing with the latter, our worry was that including too many statements not related to campaigning (“SNP need ousted from ruining Scotland” and “END IMMIGRATION” were two examples here) may lead future commenters to concentrate on these more incendiary views, rather than expanding the debate around campaigning.

To address this, a loose framework was agreed between moderators, outlining the types of statements which were in scope for inclusion – but with the understanding that exceptions should be made to these rules as new comments were submitted. As it turned out, allowing for exceptions improved the quality of the debate, and some of the more interesting and divisive comments were made around the periphery of the topic.

Polis encourages all groups to speak, but not all are equally heard

Table 1, below shows how likely various demographics were to participate in this discussion, divided by age and social class. The table shows the average number of votes and ‘constructive’ comments submitted by each group, alongside those which were chosen by moderators to include in the discussion.

Table 1 – comments and votes submitted and accepted by demographic group

This table shows that, with the exception of slightly reduced engagement in comments and votes for 25-34 year-olds, and a slight increase for participants over the age of 65, age and social class seem not to have had a significant effect on the extent to which people were likely to participate in the conversation. To some extent, this will be linked to the fact that participants were offered a small financial incentive for taking part and were asked to vote on all available comments. However, encouragingly, the likelihood that participants would submit a comment, which was explicitly optional, was similar between groups.

In a moderated context, however, speaking is not the same as being heard. C2DE participants were less likely to have their submissions included in the conversation by moderators, and, despite being the least likely group to comment, those aged 25-34 were most likely to be included.

There are a number of reasons why a comment might not be accepted, and we explore the challenges faced and decisions taken in moderation below. One contributing factor here is that Demos’ recruiting partner targeted younger groups first – these early participants were more likely to have fewer submitted comments to vote on, and to be expressing statements which did not repeat a point already made in the conversation. It is not impossible that this balance reflects the prejudice of the moderation team – both of whom, it should be noted, fell within the 24-34 age group. 

It is also worth noting that younger participants relatedly were less likely to have the opportunity to vote on statements submitted later in the process; going forward it will be important to ensure different demographic groups are recruited at a steady rate throughout fieldwork. 

In short, Polis did not seem to present a barrier to demographic by age or social background – but careful recruitment and moderation are required to avoid imbalanced communication.

Lessons learned

• Unlike a traditional poll, early participants to Polis will have a different experience to those arriving later; demographics should be evenly spaced when inviting participants to the poll. 
• Further investigation is needed to check whether including spelling and grammatical errors affects voting.
• Efforts should be taken to minimise prejudice in moderation, ideally by selecting moderators from diverse backgrounds.
• A loose set of rules is useful for moderation in deciding which topics to include, but be prepared to make exceptions to these rules. 
• It is important to ensure different demographic groups are recruited at a steady rate throughout fieldwork (even if this means requiring more leniency in targeting for responses towards the end of fieldwork).

Methodology

To prepare for the survey, Demos first collaborated with the ORG to develop a small number of ‘seed statements’ which were used to kick-start the process. Seeding the discussion in this way ensures that the first participants have something to vote on before submitting their own thoughts, but also allows researchers to set the tone of the debate, provide a model for the types of statement which might be useful, and introduce key concepts. The statements were designed to clearly state a single opinion relevant to political campaigning, and were themselves tested through running a short internal poll with Demos staff. A full list of these seed statements, as well as those submitted by participants, is included in an annex to this report. Alongside this ORG and Demos articulated data protection compliance responsibilities and signed a joint controller agreement.

The survey ran for a single week in June 2020, with participants recruited using the survey platform Dynata. To form a nationally representative sample, targets for completions were set according to age, area of residence, gender and social class. Of 1215 participants who took part in the Polis conversation, a nationally representative sample of 997 successfully returned to the survey platform. The votes, but not the statements, of people who were not part of this nationally representative sample were not included in the analysis. 

This representative sample cast 33,714 votes overall, of which 4,580 (14%) were ‘Pass / unsure’. Overall, 659 comments were submitted. These were moderated by Demos researchers as the discussion was underway, with 35 (5%) comments accepted by moderators and presented to participants to vote on. 

Output data was then filtered to remove participants who did not successfully complete the survey, and were therefore not counted as part of the nationally representative sample. The resulting data was then analysed in Tableau.

Data Protection compliance

The method of operating democratic innovations like Polis presents some data protection compliance issues that will be important to address for organisations and participants going forward. Additionally, Demos’ effort to create a nationally representative sample through the collaboration with Dynata is an innovation but adds an additional consideration regarding compliance.

The Polis tool collects personal data with individuals requested to sign up via e-mail or social log-in to use the platform. The individual joins the Pol.is tool via Dynata a market research firm that builds up samples of a population to participate in surveys. Through Dynata, individuals are asked a series of background demographic questions (age, gender, region lived in, income) and also some questions which may result in the collection of special category data such as voting preferences.

Demos, through the Polis tool collects data on how a participant has voted on submitted statements. Participants are also able to submit their own statements, through a free-text form on the site. Those free-text comments are moderated to ensure personal data are not included in the comments.

Demos is a joint data controller for the data collected through the use of the Polis tool as they determine the means of the processing alongside Open Rights Group who commissioned Demos to carry out the survey and who set out the manner and purpose of the processing. For this reason it is important that when Demos is collaborating with other organisations in the future that they sign a joint controller agreement.

The lawful basis for the processing of personal data relied upon by the use of Polis is consent as the privacy policy sets out. The policy sets out that personal data is used to process information which the user directs Polis to process (such as answers to questions and submission of comments). Usefully, it separates out additional processes (such as further polls or surveys, processing not described at the time of collection, marketing campaigns,) and acknowledges that Polis would seek the additional consent of the user.

While Polis does not collect further demographic information, the addition of Dynata’s services requires consideration. The Dynata data processed by Demos is pseudonymous but it comes with an individual ID that allows for users answers to be tracked. The level of identifiability of this information is something to consider for Demos, although joint controllers in the role that ORG played meant no personal data was available, either through Polis or Dyanata, Demos’ role in running the Polis system and ability to match against demographics provided by Dyanata does raise the question as to whether Demos is a controller both with a commissioning organisation like ORG and a provider like Dynata requiring a separate and comprehensive data controller agreement. 

Literature Review from the Open Rights Group

Ways of “reimagining and deepening the role of citizens in governance processes” – ‘democratic innovations’ – are being increasingly explored (Elstub & Escobar: 2017:14). This is largely in response to the perceived polarisation of political debate and political apathy of recent years (Grasso: 2016; Guigni et al: 2013). Some proposed democratic innovations have focused on supply side factors such as providing sources of trusted information. One example of this is the ‘information hub’ proposed by The University College London’s (UCL) recent report ‘Doing Democracy Better’, although this has not yet seen a proof of concept (Palese & Renwick: 2019). Most innovations focus instead on citizen led deliberative politics. Modern conceptions of ideal political debate broadly divide into two camps. The first, communicative rationality (synonymous with ‘deliberative democracy’) builds on the work of Jorgen Habermas. This tradition models an idealised “public sphere” a space for citizens to reach a consensual public opinion free from, inter alia, the distorting effect of private interests (Habermas: 1974; 1994; 1998; 2005; 2006). 

This has been criticised however by opposing traditions such as agonistic pluralism, which argue Habermas’ bloodless rationalism fails to recognise the important role of dissent and contestation in political debate. At worst, it instrumentalises rational discussion to dismiss minority opinions and reinforcing hegemonic power structures with a patrician ‘calm down dear’ approach (Phillips: 1996; Mouffe: 2004; Young: 2001). Although the axioms of these two schools have been debated and qualified to incorporate perspectives such as minority opinion (Phillips: 1996; Fishkin: 2009; Fraser: 1990; Ratner: 2008) and postcolonial theory (Kapoor: 2002), democratic innovations tend to draw more on the model of deliberative democracy. In doing so they valorise practical efforts to utilise a common set of facts, respectful debate, and consensus based decision making. 

Deliberative democracy has seemingly undergone a process of increasing legitimation, particularly in the use of ‘deliberative mini-publics’ to recommend policy decisions to governments. Although mini-publics can be formulated in a number of ways and sizes they are essentially deliberative and representative, using a body of citizens to “reason together about an issue of public concern” (Farell et al: 2019; Setälä & Smith: 2018: 300). There is a high degree of procedural difference within mini publics, with deliberation being either rigidly structured or flexible and discursive according to the organisers’ norms, reflecting different anxieties about the outcomes of such a process (Bächtiger et al: 2010; Pickard: 1998; Mansbridge et al: 2006). 

Various formulations such as ‘deliberative polls’, ‘planning cells’ and ‘citizens assemblies’ have been tested to formulate policy recommendations since the 1970s (Smith & Wales: 1999). In the UK context, citizens assemblies have become popular, in particular amongst the devolved legislatures. Here they have attempted to address ‘wicked’ policy issues, specifically around various constitutional questions (Macnab: 2019). 

However, these deliberative mini publics have a number of limitations, both theoretical and practical. For example, the sample of citizens may not be representative of the general population (French & Laver: 2009; Kenyon: 2005). Similarly, an expert-centric process may be used to legitimate existing policy hegemonies and prevailing wisdoms. This, in turn, can minimise the attention paid to the lived experience of individuals and the impacts and outcomes of policies (Glasner: 2001; French & Laver: 2009). There can be issues with recruitment and engagement of participants, and there can be high rates of attrition during the process (French & Laver: 2009; Kenyon: 2005). This is potentially a reflection of the fact that mini publics typically involve moving participants from one part of the country to another, takes a significant amount of time, and may encounter political obstacles (Kenyon: 2005; Pickard: 1998). To an extent, these reflect another limitation – they are extremely expensive to design, moderate, and resource (O’Leary: 2009). Finally, it is possible that such democratic innovations can be co-opted as a public relations exercise, to legitimate pre-existing decisions (French & Laver: 2009). Equally, some issues are so intractable and fundamentally oppositional that deliberative debate will not produce useful consensus. What is recommended by the mini public may be rejected by the general public (O’Leary: 2019).  

The use of deliberative digital platforms – online mini publics – remedies some of these concerns. Transposing deliberation onto the internet avoids some of the cost and recruitment issues posed by traditional forms of deliberation by collapsing space. Explicitly digital forms of deliberation have been in use for some time; notably participatory budgeting is used in Madrid, Brazil and elsewhere to allow citizens to decide how to spend a portion of the municipal budget (de Sousa Santos: 2005; Ganuza & Baiocchi: 2012; Souza: 2001; Naryanan: 2019). The VTaiwan project has gone one step further and allowed Taiwanese citizens to co-create legislation, currently limited to digital issues (Naryanan: 2019; Megill: 2016). VTaiwan used Polis, a machine learning based open source system which allows people to exchange views and form consensus online. It is this technologically driven democratic innovation that ORG and Demos have sought to test. Significantly, this is the first time the tool has been paired with nationally representative sampling. This allows responses to be reflective of the public at large, including marginalised groups, rather than a self-selecting sample. This will ensure policies will benefit from the ideas and experiences of groups who would otherwise be unlikely to take part in the process, and make the findings more persuasive to policymakers and politicians.

This system attempts to ameliorate many issues related to tech driven interventions and deliberative mini publics more generally. For example, it allows for participants to respond to each other’s statements rather than having an expert lead the conversation, although a number of seed statements and human moderators frame the debate. Whilst the software highlights points of consensus however, it also notes areas of disagreement, protecting minority opinions. The tech driven elements of the program must be accounted for in data protection law; for example, the recent decision of the European Court of Justice means the use of US hosted servers for EU citizens’ data in the future couldwill be unlawful (Schrems II: 2020). Crucially though, the open source code mitigates against the co-option of the technology by private interests. This is perhaps the key benefit to such a method; not only does it enable a democratic event, but the innovation democratises the process of conducting the event itself. 

The key difficulty though, as with all democratic innovations, will be making the case for opinions generated here to become policy and legislation.

Bibliography:  

Bächtiger, A. (2018). Chapter 41. A Preface to Studying Deliberation Empirically. In A. Bächtiger, J. S. Dryzek, J. Mansbridge, & M. E. Warren (Eds.), The Oxford Handbook of Deliberative Democracy (pp. 657–662). New York: Oxford University Press.                                                                                                            

Elstub, S., & Escobar., O. (2017). A Typology of Democratic Innovations. Retrieved from: https://www.psa.ac.uk/sites/default/files/conference/papers/2017/A%20Typology%20of%20Democratic%20Innovations%20-%20Elstub%20and%20Escobar%202017.pdf

Farrell, D., Curato, N., Dryzek, J., Grönlund, K., Marien, K., Niemeyer, S., Pilet, J.B., Renwick, A., Rose, J., Setälä, M., & Suiter, J. (2019). Deliberative Mini Publics: Core Design Features. Centre for Deliberative Democracy and Global Governance.

Fishkin, J. S. (2009). 3. Competing Visions. In When the People Speak. Deliberative Democracy and Public Consultation (pp. 65–94). Oxford: Oxford University Press. 

Fraser, N. (1990). Rethinking the Public Sphere: A Contribution to the Critique of Actually Existing Democracy. Social Text, (25/26), 56–80. https://doi.org/10.2307/466240         

French, D., & Laver, M. (2009). Participation Bias, Durable Opinion Shifts and Sabotage through Withdrawal in Citizens’ Juries. Political Studies, 57(2), 422–450. https://doi.org/10.1111/j.1467-9248.2009.00785.x

Ganuza, E., & Baiocchi, G. (2012). The Power of Ambiguity: How Participatory Budgeting Travels the Globe. Journal of Public Deliberation, 8(2). Retrieved from http://www.publicdeliberation.net/jpd/vol8/iss2/art8

Glasner, P. (2001). Rights or rituals? Why juries can do more harm than good. PLA Notes 40: Deliberative Democracy and Citizen Empowerment, 43– 45.                       

Habermas, J. (1974). The Public Sphere: An Encyclopedia Article (1964) (S. Lennox & F. Lennox, Trans.). New German Critique, (3), 49–55. https://doi.org/10.2307/487737Habermas, J. (1994). Three Normative Models of Democracy. Constellations, 1(1), 1–10. https://doi.org/10.1111/j.1467-8675.1994.tb00001.x          

Habermas, J. (1994). Three Normative Models of Democracy. Constellations, 1(1), 1–10. https://doi.org/10.1111/j.1467-8675.1994.tb00001.x            

Habermas, J. (1998). Chapter 9 Three Normative Models of Democracy. In C. Cronin & P. De Greiff (Eds.), The Inclusion of the Other: Studies in Political Theory (pp. 239–252; By J. Habermas). Cambridge, Massachusetts: The MIT Press.                 

Habermas, J. (2005). Concluding Comments on Empirical Approaches to Deliberative Politics. Acta Politica, 40(3), 384–392. https://doi.org/10.1057/palgrave.ap.5500119

Habermas, J. (2006). Political Communication in Media Society: Does Democracy Still Enjoy an Epistemic Dimension? The Impact of Normative Theory on Empirical Research. Communication Theory, 16(4), 411–426. https://doi.org/10.1111/j.1468-2885.2006.00280.x

Kenyon, W. (2005). A Critical Review of Citizens’ Juries: How Useful are they in Facilitating Public Participation in the EU Water Framework Directive? Journal of Environmental Planning and Management, 48(3), 431–443. https://doi.org/10.1080/09640560500067558 Macnab, S. (2019). Scotland’s new Citizens’ Assembly to ‘change tone of debate.’ The Scotsman. Retrieved from https://www.scotsman.com/news/politics/scotland-s-new-citizens- assembly-to-change-tone-of-debate-1-4961512  

Mansbridge, J., Hartz-Karp, J., Amengual, M., & Gastil, J. (2006). Norms of Deliberation: An Inductive Study. Journal of Public Deliberation, 2(1), 1– 47.

Max Schrems Vs Facebook Ireland, [2020], C-311/18. Retrieved at: http://curia.europa.eu/juris/documents.jsf?num=C-311/18

Megill, C. (2016). pol.is in Taiwan. Retrieved from https://blog.pol.is/pol-is-in-taiwan-da7570d372b5

Mouffe, C. (2004). 3. Pluralism, Dissensus and Democratic Citizenship. In F. Inglis (Ed.), Education and the Good Society (pp. 42–53). Basingstoke; New York: Palgrave Macmillan.Narayanan , D. (2019). Technology and political will can create better governance. The Economist. Retrieved from https://www.economist.com/open-future/2019/03/22/technology-and-political-will-can-create-better-governance

Narayanan , D. (2019). Technology and political will can create better governance. The Economist. Retrieved from https://www.economist.com/open-future/2019/03/22/technology-and-political-will-can-create-better-governance

O’Leary, N. (2019). The myth of the citizens’ assembly. Politico. Retrieved from https://www.politico.eu/article/the-myth-of-the-citizens-assembly-democracy/

Palese, M., & Renwick. A. (2019). Doing Democracy Better. Retrieved from: https://www.ucl.ac.uk/constitution-unit/sites/constitution-unit/files/184_-_doing_democracy_better.pdf

Phillips, K. R. (1996). The spaces of public dissension: Reconsidering the public sphere. Communication Monographs, 63(3), 231–248. https://doi.org/10.1080/03637759609376391                   

Pickard, S. (1998). Citizenship and Consumerism in Health Care: A Critique of Citizens’ Juries. Social Policy and Administration, 32(3), 226–244. https://doi.org/10.1111/1467-9515.00100                                                                 

Ratner, R. S. (2008). 7 Communicative rationality in the Citizens’ Assembly and referendum process. In M. E. Warren & H. Pearse (Eds.), Designing Deliberative Democracy: The British Columbia Citizens’ Assembly (pp. 145–165). New York: Cambridge University Press.  

Setälä, M., & Smith, G. (2018). Chapter 18. Mini-Publics and Deliberative Democracy. In A. Bächtiger, J. S. Dryzek, J. Mansbridge, & M. E. Warren (Eds.), The Oxford Handbook of Deliberative Democracy (pp. 300–314). New York: Oxford University Press.      

Smith, G. (2009). Democratic Innovations: Designing Institutions for Citizen Participation. Cambridge University Press. Retrieved from: https://www.academia.edu/11782670/Democratic_Innovations_Designing_Institutions_for_Citizen_Participation?auto=download

Smith, G., & Wales, C. (1999). The Theory and Practice of Citizens’ Juries. Policy & Politics, 27(3), 295–308. https://doi.org/info:doi/10.1332/030557399782453118        

de Sousa Santos, B. (2005). 11. Participatory Budgeting in Porto Alegre: Towards a Redistributive Democracy. In B. de Sousa Santos (Ed.), Democratizing Democracy: Beyond the Liberal Democratic Canon (pp. 307–377). Retrieved from http://www.boaventuradesousasantos.pt/media/Chapter%2011.pdf

Souza, C. (2001). Participatory budgeting in Brazilian cities: Limits and possibilities in building democratic institutions. Environment & Urbanization, 13(1), 159–184.

Young, I. M. (2001). Activist Challenges to Deliberative Democracy. Political Theory, 29(5), 670–690. https://doi.org/10.1177/0090591701029005004

ANNEX

Demographic data before and after weighting

Seed comments – submitted by Demos