Open Rights Group

Home Data: a new direction Consultation Guidance REMOVING THE RIGHT TO HUMAN REVIEW

REMOVING THE RIGHT TO HUMAN REVIEW

1.5 AI and Machine Learning

The Government is considering the option to scrap Article 22 of the UK GDPR, which provides “the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects […] or similarly significantly affects” individuals.

In our answer to Q1.5.17, we explain how:

  • Article 22 has been instrumental in ensuring remedies against unfair automated decisions.
  • Article 22 should be seen as a stepping stone to build upon, to enact complimentary legislation that address harms in the field of AI.

The government is seeking further evidence on the potential need for legislative reform rather than making proposals at this stage

Q1.5.17. To what extent do you agree with the Taskforce on Innovation, Growth and Regulatory Reform’s recommendation that Article 22 of UK GDPR should be removed and solely automated decision making permitted where it meets a lawful ground in Article 6(1) (and Article 9-10 (as supplemented by Schedule 1 to the Data Protection Act 2018) where relevant) and subject to compliance with the rest of the data protection legislation?

We strongly disagree with “the Taskforce on Innovation, Growth and Regulatory Reform’s recommendation that Article 22 of UK GDPR should be removed” (Q1.5.17). It is particularly important that this provision is not removed.

Removing the right to human review would shift this burden from organisations to individuals. This is fundamentally unfair, as individuals would be asked to actively monitor and scrutinise life-changing decisions that are taken against them, by systems that are out of their control or understanding.

Article 22 has proven to be an effective safeguard, that protected individuals against some of the most egregious abuses. While it may not provide a comprehensive remedy for harms related to AI, this does not justifies its removal. Rather, specific legislation in the field of AI could be implemented to expand the protections offered by the UK GDPR.

Q1.5.17a. Please explain your answer, and provide supporting evidence where possible, including: (a) The benefits and risks of the Taskforce’s proposal to remove Article 22 and permit solely automated decision making where (i) it meets a lawful ground in Article 6(1) (and, Articles 9 and 10, as supplemented by Schedule 1 to the Data Protection Act 2018) in relation to sensitive personal data, where relevant) and subject to compliance with the rest of the data protection legislation. (b) Any additional safeguards that should be in place for solely automated processing of personal data, given that removal of Article 22 would remove the safeguards currently listed in Article 22 (3) and (4)

Scrapping article 22 of the UK GDPR would impose an unbearable burden on individuals. The transparency, accountability and fairness of automated systems depend on the organisations that implement them. However, removing the right to human review would shift this burden from organisations to individuals. This is fundamentally unfair, as individuals would be asked to actively monitor and scrutinise life-changing decisions that are taken against them, by systems that are out of their control or understanding. In turn, it would mark a fundamental departure from the principle that technological progress should be rooted in respect for human dignity and “designed to serve mankind”.

Furthermore, the effectiveness of Article 22 of the UK GDPR in providing remedies for individuals who were harmed by automated systems has been invaluable.

  • Article 22 allows workers to stand up to abusive practices such as robo-firings,1 wage theft,2 or racists facial recognition systems.3
  • Article 22 stood in the way of the A-level debacle. The Norwegian Data Protection Authority ordered the correction of pupils grades by relying on Article 22.4 Soon afterwards, the UK Government scrapped their own algorithmic grades, facing a judicial challenge on the same grounds.5 This saved an entire generation of UK students from unfair grading that would have impacted the rest of their lives.

Finally, while we recognise that this does not provide comprehensive nor sufficient protection from the harms associated with algorithmic decision making, this does not constitute a reason to remove Article 22. On the contrary, it calls for a dedicated legal regime that introduces further and stronger protections against the harms associated with AI, thus complimenting existing standards under the UK GDPR.

1ADCU, Dutch & UK courts order Uber to reinstate ‘robo-fired’ drivers and pay compensation. Available at: https://www.adcu.org.uk/news-posts/uber-to-reinstate-robo-fired-drivers-and-pay-compensation

2ADCU, Gig economy workers score historic digital rights victory against Uber and Ola Cabs. Available at: https://www.adcu.org.uk/news-posts/gig-economy-workers-score-historic-digital-rights-victory-against-uber-and-ola-cabs

3ADCU, ADCU initiates legal action against Uber’s workplace use of racially discriminatory facial recognition systems. Available at: https://www.adcu.org.uk/news-posts/adcu-initiates-legal-action-against-ubers-workplace-use-of-racially-discriminatory-facial-recognition-systems

4Datatilsynet, The Norwegian DPA intends to order rectification of IB grades. Available at: https://www.datatilsynet.no/en/news/2020/the-norwegian-dpa-intends-to-order-rectification-of-ib-grades/

5Foxglove, We put a stop to the A Level grading algorithm! Available at: https://www.foxglove.org.uk/2020/08/17/we-put-a-stop-to-the-a-level-grading-algorithm/