Join 9ine for the first of our ‘Turing Trials Walk-throughs’ where we take you through Scenario 1, discussing the risks, issues and safeguards associated with automated essay grading. 

In our previous blog, we announced the launch of ‘Turing Trials Walk-Throughs’, where between now and the end of the year in a series of blogs, we will take you through each of the ten scenarios we currently have for Turing Trials, to discuss some of the risks, issues and safeguards you will need to consider for each. If you haven’t downloaded it already, Turing Trials is available to download for free from our website. In this blog, we will take you through Scenario 1, which concerns the use of AI for automated easy grading (AEG) in schools. 

Turing Trials Scenario 1 - Automated Essay Grading Using AI 

The Scenario: ‘A school is using an EdTech Vendor’s AI system to automatically grade student essays, but forgot to use an ethics by design process. A student’s (whose first language isn’t English) essay is automatically flagged as AI-generated and the student receives a fail. The student says that they didn’t use AI but the school upholds the decision without a review.’

This Scenario may be presented to a school because the student (or their parent or guardian) makes a complaint about the student receiving a fail. It could also be discovered as part of an inspection or an investigation by a regulator. Before we delve into the risks, issues and safeguards associated with implementing AI for automated essay grading, let’s understand a little more about what automated essay grading and ethics by design are. 

What is Automated Essay Grading? 

Automated essay grading (AEG) systems are one of the most discussed applications surrounding the use of AI in education. These tools promise to save teachers valuable time, provide instant feedback to students, and standardise evaluation. However, their implementation requires a careful consideration regarding accuracy, fairness and the very nature of learning. AEG systems use natural language processing and machine learning algorithms to evaluate written work. These systems are trained on large datasets of previously graded essays, learning to identify characteristics associated with different score levels. The primary appeal for schools is efficiency, as teachers spend a significant amount of time grading. An AEG system can reduce this burden, freeing up educators to focus on lesson planning, one-on-one student support, and other high-impact activities.

What is ethics by design?

‘Ethics by Design’ in AI refers to an approach where ethical principles and human values are built into the design, development, and deployment of AI systems from the very beginning, rather than being added as an afterthought or addressed only when problems arise. Typical questions that form part of an ethics by design process include: 

• Why are we using this AI system? Who benefits and who might be harmed? 
• Who are the groups that will be affected by using the AI system?
• Which school principles and policies does the AI system and the data used with it need to adhere to, and does it?
• How will we monitor that the AI system and the data used with it adheres to these principles and policies as well as any legal requirements that the school is under? 
• How will we ensure that we have human oversight of the AI system, making sure that we have a ‘human in the loop’ and individuals responsible for the AI system if it causes harm?  

Using an ethics by design process helps to ensure that use of the AI system aligns with ethical values like fairness, accountability and respect for human rights, taking into consideration the cultural and contextual factors that shape what ‘ethical’ means to your school.

What ‘Issues’ does this Scenario present? 

Turing Trials currently has fifteen Issues cards, and it is the role of the group playing to discuss what they think the top three Issues associated with this Scenario are. At 9ine, we know that schools don’t have endless time and endless budgets to investigate and present all possible issues to your leadership teams, which is why you can only select three! Ultimately it is the role of The Investigator to select the final three that are played in the game. 

There is no ‘right’ answer in Turing Trials, but it is important for the group to discuss and justify which Issues they think that this Scenario presents and why. It’s also the aim of Turing Trials that groups constantly cycle through and remind themselves of the various Issues that can be associated with the use of AI, even if they are not relevant to the Scenario being played. Some of the Issues that might be highlighted as part of this Scenario (remember that you can only select three) are: 

• Bias and Discrimination: The use of AI may have led to bias or discrimination in relation to an individual or group. Bias is a significant concern with any AI system, and AEG systems are no exception. AEG systems are trained on existing data which can contain inherent biases. If the training data primarily consists of essays from a specific demographic group, the AI system may inadvertently penalise writing styles that are common to underrepresented students, including those for whom English is not their first language. These systems can also enforce a standardised view of ‘good writing’, which can stifle diverse voices and writing styles, failing to recognise cultural variations in rhetoric and expression. This Scenario clearly states that ‘A student’s (whose first language isn’t English) essay is automatically flagged as AI-generated and the student receives a fail’, so whilst it is not confirmed that there is bias in the AI system, the fact that it has flagged a student’s essay as AI-generated, and their first language isn’t English, indicates that this is an Issue which warrants further investigation.

• Lack of Human Intervention: There has been a lack of human intervention in the decisions made by AI about an individual. AI is inherently flawed, not because it can never work well, but because it carries built-in limitations and biases that can never entirely be removed. These include: 

• • Data bias (which can lead to unfairness); 

• • Lack of understanding of meaning or intent (AI recognises patterns and correlations, not causation or truth); and 

• • Context limitations, meaning that when the context changes (different population, culture, or language), the performance of an AI often drops sharply

The fundamental question when using an AEG system is whether the AI can accurately assess the quality of a student’s writing. Modern systems are increasingly sophisticated, but they are not infallible, and whilst they excel at identifying surface-level features like grammar and spelling, they struggle to evaluate deeper elements such as creativity, critical thinking, nuance and originality. This means that a well structured essay with flawless grammar might score highly even if its arguments are weak or uninspired. This is why it is important to have human intervention in the decisions made by AI about an individual, particularly where there are serious consequences or an individual contests the outcome. This Scenario states that ‘The student says that they didn’t use AI but the school upholds the decision without a review.’ In this Scenario, despite the inherent flaws in AEG systems and the fact that the student has said that they didn’t use AI, the school has upheld the AI system’s decision without intervention from a human. 

• Process Not Followed: A process has not been followed e.g. vendor management, ethics, privacy or security by design, meaning that risks may not have been identified and mitigated. Processes provide a structured, consistent and repeatable way of doing something. They help to ensure consistency, efficiency, quality control, accountability, improvement, risk reduction and compliance. This Scenario clearly states that ‘A school is using an EdTech Vendor’s AI system to automatically grade student essays, but forgot to use an ethics by design process.’ We have already discussed the importance of following an ethics by design process when using AI systems, to ensure that the system meets the principles and policies of the school, and to ensure that the system is used safely, securely and compliantly. Using this process could have identified the Issues of bias and discrimination (which could have been mitigated), or highlighted that there was a lack of human intervention relating to the decisions that the AI system is making, allowing the school to ensure that appropriate human intervention was made when using the AEG system. 
• Lack of Training/Awareness: An individual has acted in a way that indicates that they are not aware of the risks of AI or how to use an AI system. Having an appropriate level of AI Literacy in your school means ensuring that your staff have the appropriate skills, understanding, technical knowledge and experience of AI for their role, and for the AI systems which they operate. This is a legal requirement in some countries (such as in the European Union under the EU AI Act). Whilst a lack of training and awareness has not been specifically called out in this Scenario, if the appropriate individuals had received and participated in appropriate training and awareness raising activities on AI, it would be less likely that an ethics by design process would not have been used or followed, and that a lack of human intervention would have occurred.  

What Safeguards might a school use for this Scenario?

Similar to the Issues cards, Turing Trials has fourteen Safeguards cards, and it is also the role of the group to discuss which three Safeguards they want to put in place to respond to the Issues which The Investigator has highlighted. It is ultimately the role of The Guardian to select the final three that are played in the game (again, schools don’t have endless time and endless budgets to remediate all possible risks and issues at once, which is why you can only select three!), There is no ‘right’ answer, but it is important for the group to discuss which Safeguards they think are the most important to put in place for this Scenario (whilst reminding themselves of many of the Safeguards that are available to the school when mitigating risks and issues relating to AI). 

The Safeguards cards are deliberately designed to each mitigate at least one of the Issues cards, but as there is no ‘right’ answer, The Guardian does not have to select the three Safeguards which match the Issues selected by The Investigator. Some of the Safeguards that might be highlighted as part of this Scenario are: 

• Bias/Discrimination Countering: The school takes steps to reduce bias in the AI system including any discriminatory effects of using it e.g. by making training data more diverse or representative. Schools might use bias detection and mitigation methods in the datasets used to train the AI system to ensure that the data used is inclusive and applicable to the population of the school. For example, the school might ensure that the previously graded essays used to train the AI system are sufficiently representative of students who do not have English as their first language. This will help to reduce the possibility of bias and discrimination, and the risk that this alone was what led to the AEG system automatically flagging the student’s essay as AI-generated.
• Training/Awareness Activities: A school provides training and awareness raising activities to relevant individuals, including on how AI systems work, should be used and what the limitations and potential risks or harms are of using AI. Ensuring that the school community has the appropriate level of AI literacy through training and awareness activities may have helped to ensure that an ethics by design process was followed and that staff were empowered to be ‘humans in the loop’, so that they would understand that this decision should have been reviewed and that they would have the skills and knowledge to be able to do this. Whether this is done through awareness raising activities like Turing Trials, or more formalised AI training such as Academy LMS, ensuring the appropriate level of AI literacy at your school is a safeguard which is the starting point for mitigating all Issues relating to the use of AI. 
• Human in the Loop: The school makes sure that a human reviews a decision that the AI system has made. As well as providing training and awareness activities to empower individuals to be ‘humans in the loop’, schools will also have to actively put humans ‘in the loop’. This means that they will need to define the staff member’s role clearly in relation to the AEG system, including how and where they should intervene, which decisions the AEG system should make automatically (versus the ones which should be reviewed by a human), and who is accountable if something goes wrong. In this Scenario having a ‘human in the loop’ would have ensured that the school would have considered whether the decision should have been reviewed, clearly designated the individual that was responsible for doing this and who was accountable for the AI system if there was found to be bias in it. 
• Repeat or Complete a Process: Make sure that the school puts the AI system through a relevant process e.g. ethics by design, privacy by design, vendor management etc. By forgetting to use an ethics by design process, the school may not have highlighted various Issues associated with the AEG system before it was used, which if mitigated, could have prevented this Scenario from arising. Completing this process would help the school to identify the concrete Issues (which may be more than three) with the AEG system before it was deployed so that the Safeguards could have been put in place to mitigate them.
• Explainable AI: The school makes sure that the AI system can provide clear and understandable explanations for its operations, decisions and outputs so that individuals can comprehend and explain how the AI system reached a particular conclusion. For any AEG system to be effective, its feedback must be understandable and actionable. As many AI systems operate as ‘black boxes’, meaning that their decision-making processes are not transparent. Whilst this Scenario does not explicitly state that the student did not receive a meaningful explanation as to why their essay was automatically flagged as AI-generated and they received a fail, The Guardian may decide that this is a Safeguard that they want to play, to ensure that students always receive feedback when an AEG system is used, to explain to them how and why their essay has been given a particular grade. The Guardian may also want to choose this safeguard because if the ‘Human in the Loop’ Safeguards card is played because Explainable AI will mean that the human in the loop has the ability to review the decision effectively. 

Identifying the Risk Level and making a Decision 

As the game unfolds, at different points it will be the role of the Risk Analyst to assess the level of risk that the Scenario presents based on the Issues and Safeguards that have been selected, and decide whether this presents a high, low or medium risk to the school. Turing Trials deliberately does not specify what defines each level of risk, as this will differ between schools and the groups that are playing, but you may want to consider what would impact your Risk Level decisions (for example, would it make a difference if it was more than just one student that it had been impacted in this Scenario?) At the end of the game, The Narrator and Decision Maker will need to make the decision on whether they would accept the Risk Level of the Scenario with the Issues and Safeguards highlighted on behalf of the school. What decision do you think you would make and why? 

What else do schools need to consider and how else can 9ine help? 

There is no doubt that AEG brings many opportunities to schools, but it needs to be implemented safely, securely and compliantly. These are just some of the risks, issues and safeguards that might be considered and selected when playing this Scenario in Turing Trials, but there are endless combinations you can make. The best way to see what your staff and students would choose is to run a session and see for yourself! 

As you play through, you may realise that your school needs more support with the risks and issues associated with the use of AEG (and AI in general), and with the safeguards that you need to put in place to mitigate them. At 9ine we offer a number of solutions that can help: 

• Vendor Management: Another key issue which schools will also need to consider when using an AEG system is that they are likely to be acquiring these from a third party vendor, meaning that they will need to vet the vendor for compliance. When schools upload student work onto a third party’s AEG system, it could contain personal data belonging to the student or others. These essays are also the intellectual property of the student and potentially the school. This means that schools will need to: ensure that there are safeguards in place with the vendor to protect any personal data that is shared with them; obtain contractual guarantees that the vendor is compliant with data privacy laws and regulations; and ensure that the vendor provides guarantees that they will not use the personal data or any intellectual property which the school provides them with for their own purposes. This vetting takes time and effort, which is where Vendor Management, 9ine’s centralised system to assess and monitor the compliance of all your EdTech vendors supports you. It provides a traffic-light dashboard for each app or platform, showing independent assessments of its safeguarding, data privacy, AI, and cybersecurity risks. This intelligence saves schools hundreds of hours of manual review and helps ensure you’re only using EdTech that meets required standards or that you have safeguards and mitigations in place. If inspectors ask how your school manages AI risks, you can literally show them the Vendor Management dashboard, demonstrating a rigorous, ongoing evaluation of each tool’s compliance. Vendor Management lets you easily identify risks and take action – whether that means engaging a vendor for improvements, configuring the tool safely, or ultimately choosing an alternative AEG system
• Academy LMS: Having the appropriate Training and Awareness Activities is both a key Issue and Safeguard in Turing Trials. If you think your school needs to improve its AI literacy, 9ine’s on-demand training and certification platform enables schools to enrol individual staff members or entire groups in comprehensive training courses, modules, and assessments, featuring in-built quizzes for knowledge checks. Our AI Pathway is your school's learning partner for AI ethics and governance. With over 20 differentiated course levels you can enrol all staff in an Introductory course to AI, then for those staff with a greater responsibility, enrol them in Intermediate and Advanced courses. There’s also Specialist courses for AI in Safeguarding, Child Protection and Technology. Schools can also subscribe to learning pathways in Privacy, Cyber, Tech Operations and Risk Management. Alternatively, schools can purchase courses on a per person and a per course basis. We are currently offering free trials for up to three members of a school’s leadership team, so contact us if you would like to take advantage of this, or have any questions on Academy LMS. 
• AI and Privacy Academy: 9ine’s certified monthly training programme for risk professionals and education privacy teams. It equips Data Protection Officers (DPOs) or anyone else at your school who are responsible for data protection and privacy in handling data breaches, subject access requests, and international data transfers confidently. The new enrollment intake will commence in November 2025, and cover nine live sessions ending in April 2026. The program offers interactive sessions, group scenarios, and comprehensive resources.

Join us next week for the Turing Trials Walk-Through of Scenario 2, which looks into the risks, issues and safeguards associated with using facial recognition for attendance monitoring. If you want receive these blogs direct to your inbox, you can sign up to our monthly newsletter here.

9ine company overview

9ine equips schools to stay safe, secure and compliant. We give schools access to all the expertise they need to meet their technology, cyber, data privacy, governance, risk & compliance needs - in one simple to use platform. For additional information, please visit www.9ine.com or follow us on LinkedIn @9ine.