Turing Trials Scenario 4: The opportunities and risks of using chatbots and virtual mentors in education

Welcome to our next installment of the ‘Turing Trials Walk-Throughs’, where between now and the end of the year, we will take you through each of the ten scenarios we currently have for Turing Trials, to discuss some of the risks, issues and safeguards you will need to consider for each. If you haven’t downloaded it already, Turing Trials is available to download for free from our website, including full instructions on how to play the game. In this blog, we will take you through Scenario 4, which concerns the opportunities and risks of using AI-powered chatbots and virtual mentors in education. 

The Scenario: ‘A school begins using Virtual Mentors to support students in their studies. They notice that students desperately want to interact with these, even when the teacher or other students are available. One student reports that the Mentor told them that they were being rude and swore at them, but the teacher has no way of reviewing the interactions between the AI and the student.’

AI-powered Chatbots and Virtual Mentors 

For context, AI-powered chatbots and virtual mentors are now widely used in education, from tools like MagicSchool for students to Khanmingo by Khan Academy. Students can use them for:

• Learning Support and Tutoring: To explain difficult concepts in a simpler language, offer practice questions on subject areas and adjust explanations to a student’s level of understanding 

• Homework Help: Students can use them to summarise long texts, outline essays, translate and rewrite text, generate study guides or flashcards, and plan study schedules

• Writing Assistance: For help with grammar and spelling, idea generation, restructuring paragraphs, improving clarity, understanding feedback and getting examples of different writing styles; and 
• Support: To support students with motivation, study confidence, organisation, time management and extra support outside of school hours 

But, whilst AI chatbots and virtual mentors can be powerful tools in education, they also come with risks that schools have to manage carefully. These risks include:  

• Over-reliance: Which can lead to skill erosion and result in students not developing core academic skills as deeply if they rely too much on them 

• Academic Integrity: Where students might use them to write essays on their behalf or summarise readings that they haven’t done

• Data protection and Privacy: Due to the amount of personal data that these tools might collect from and about students, especially if these tools are not vetted for compliance; and 
• Bias and Fairness: They can reinforce stereotypes, provide unequal support, misdiagnose learner ability or give different quality depending on phrasing, accent or dialect of the student

But what are the Issues, Risks and Safeguards in this specific Scenario?

What ‘Issues’ does this Scenario present? 

Turing Trials currently has fifteen Issues cards, and it is the role of the group playing to discuss what they think the top three Issues associated with this Scenario are. Ultimately it is the role of The Investigator to select the final three that are played in the game. There is no ‘right’ answer in Turing Trials, but it is important for the group to discuss and justify which Issues they think that this Scenario presents and why. Some of the Issues that might be highlighted as part of this Scenario are: 

• Addiction/Dependency on Technology: Excessive use of technology which leads to problems, harms or distress. A key concern with the use of virtual mentors and chatbots is that students come to over rely on them. Because they mimic conversation, provide instant feedback, and feel emotionally responsive, this can lead to overuse. This risk is especially true of children and teenagers, as their cognitive functions are still developing, meaning that they may struggle to evaluate reliability. Students can also begin to use them to replace human interaction, encouraging solitary problem-solving which has even been linked to suicide in some of the most extreme cases. In this Scenario, it states that the school ‘ notice that students desperately want to interact with these, even when the teacher or other students are available’, which, with a school’s responsibility for safeguarding, and given the risks of over-reliance, should be an Issue that the school highlights.

• Inappropriate Interactions: The AI provides individuals with inappropriate or incorrect content or interactions. AI-powered chatbots can provide inappropriate or harmful content to students due to a combination of factors. Even if the AI isn’t malicious, the way it is trained and interacts can lead to outputs that are unsafe or unsuitable for children. For example, Character.ai, an app and website that allows users to interact with AI-generated chatbots, has received various complaints that it frequently feeds harmful content to children. There are also risks of them providing misinformation through incorrect or outdated information, and they can be known to make up facts. AI ‘hallucinations’ can occur when the AI doesn’t understand the question or the information that it is presented with, causing it to ‘hallucinate’. These often occur when the AI model fills in gaps based on similar contexts from its training data, or when it’s built using biased or incomplete training data. In this Scenario, it says that ‘One student reports that the Mentor told them that they were being rude and swore at them’, meaning that there has been a potential inappropriate interaction (imagine if a teacher swore at a student). It is always harmful, and an issue, when children are provided with content that is not appropriate for them, but when it is via a school-sanctioned tool this also creates issues for the school, as the tool should have been fully vetted prior to the school deploying it. 

• Lack of Human Oversight: Humans do not have the ability, or have not used the ability, to provide oversight of how the AI system is working. Because of the risks associated with using AI, people need to be actively involved in monitoring, guiding, or controlling the AI system to ensure that it behaves safely, fairly, and as intended. Human oversight is about people staying in control of how the AI is used and stepping in when needed, rather than letting the AI act entirely on its own. In this Scenario, it says that when the student ‘reports that the Mentor told them that they were being rude and swore at them, but the teacher has no way of reviewing the interactions between the AI and the student.’  If individual(s) at the school do not have oversight of the Virtual Mentor, it means that they cannot verify whether the Virtual Mentor swore at the student or not, or even whether any other harmful content is being provided or inappropriate interactions taking place. In this Scenario, whilst the teacher says that they have no way of reviewing the conversation, this does not necessarily mean that no one at the school, or the vendor has this ability. This makes this an Issue that needs to be investigated. 

What Safeguards might a school use for this Scenario?

Turing Trials also has Safeguards cards, and it is also the role of the group to discuss which three Safeguards they want to put in place to respond to the Issues which The Investigator has highlighted. It is ultimately the role of The Guardian to select the final three that are played in the game. There is no ‘right’ answer, but it is important for the group to discuss which Safeguards they think are the most important to put in place for this Scenario. 

The Safeguards cards are deliberately designed to each mitigate at least one of the Issues cards, but as there is no ‘right’ answer, The Guardian does not have to select the three Safeguards which match the Issues selected by The Investigator. Some of the Safeguards that might be highlighted as part of this Scenario are: 

• Healthy Technology Use: The school makes sure that there are limits on the use of AI by students, and that the benefit of using it far outweighs any risks of dependency or an addiction to it. Preventing students from becoming addicted or over-dependent on AI requires a mix of school policy, technical safeguards, teacher practice, and student education. Schools should set policies on when AI is allowed and what constitutes healthy use versus over-reliance. They should only use AI platforms that include features like time limits or usage dashboards so that the school can set time-based boundaries and ensure that there are digital breaks during school hours. Most schools also have filtering and monitoring obligations, requiring them to protect students from harmful online content, so they should also only ever use AI systems that have these built in to a satisfactory level. Teachers should also model healthy AI use and educate students to understand the limitations of AI, including that AI doesn’t think or have feelings, that it can be wrong, and that it is not a friend or mentor, so that students are less likely to seek emotional comfort from it. Schools should also encourage human support, promoting the importance of asking teachers, talking with counsellors and collaborating with classmates or asking parents for help. Schools should also perform a risk assessment prior to deploying any AI tool at the school (through vendor vetting and privacy/ethics/security by design processes) to ensure that they are clear on the benefits that they are trying to achieve by introducing the tool, any risks that the tool can introduce (so that mitigations can be put in place) and be sure that overall the benefits of introducing the AI tool far outweigh any potential risks (which is where 9ine’s Vendor Management can help you). All of these actions reduce the risk that students will become addicted too, or dependent on, the AI. 
• Human Oversight: The school makes sure that a human monitors how the AI system is working, to ensure that appropriate content is provided and that the system is working as planned. Human Oversight is about people staying in control of how the AI is used and stepping in when needed, meaning that schools need to make sure that individuals at the school have the ability to (and actually) do this in practice. By putting this Safeguard in place, schools could have assigned roles and responsibilities to individuals at the school for human oversight, and ensured that they had the necessary expertise to provide it. This expertise would include understanding how this specific AI system works, its limitations, and how to intervene safely if an incident occurs. These individuals could then make sure that the AI tool had been configured, so that any applicable child-safe modes, age-filters and prohibited safeguard topics were set correctly, which would have reduced the risk of the Virtual Mentor having inappropriate interactions with the student. They should also ensure that the AI system gave them the capability to monitor logs (both proactively and reactively) and that alerts would be provided for any harmful content or inappropriate interactions. This would have allowed them to investigate these (potentially adjusting filters and policies where required), decide if an issue had occurred and respond appropriately. With this Safeguard in place then the teacher would have been able to verify whether the Virtual Mentor had in fact sworn at the student by reviewing the conversations between them. It would also mean that even where the teacher was not able to do this themselves, they would have known who to contact at the school to do this. 
• Explainable AI: The school makes sure that the AI system can provide clear and understandable explanations for its operations, decisions and outputs so that individuals can comprehend and explain how the AI system reached a particular conclusion. Explainability of the AI system goes beyond the ability for the school to simply monitor the conversations that chatbots and virtual mentors are having with students, it is about understanding exactly how and why an AI system produced a particular output. Schools should only use AI systems that have features built in that provide clear and understandable explanations of their operations, decisions and outputs. This will include logs of the conversations with students, but also goes further, to include features that explain what information it relied on, and the patterns of reasoning it followed, when acting in a certain way. As most AI tools that schools use are provided by third party EdTech vendors, it will be an important part of the vendor vetting process to ensure that an AI system can do this. In this Scenario, if the teacher had been able to have the system explain to them whether it had in fact sworn at the student, but also why it did this, it would allow the school to assess whether this was appropriate in the context and correct the AI tool if necessary (potentially with support from the vendor). For example, the swear word could have come from a text that was part of the curriculum, making it appropriate for the student in question who was studying it, with the virtual mentor merely using a swear word rather than swearing at them. But, it also could have been that the model was trained on the texts for all age groups, and the student in question was far too young to be presented with such language, indicating that there was an issue with the model. This is why explainability is so important as a safeguard, so that the school can understand why the AI tool did what it did, to understand if there is an issue. 

Are there any other Issues and Safeguards that we might have selected? 

Because there are no right answers in Turing Trials, these don’t have to be the Issues and Safeguards that you choose, you may have also chosen: 

• Issues: Lack of Training/Awareness and Safeguards: Training/Awareness Activities: Do you think that if staff at the school had the appropriate level of AI literacy, that an AI system would have been deployed where the conversations could not be checked? Do you think that there might have been a chance that whilst the teacher said the school had no way of reviewing the conversations, someone else in the school might have been able to, or they may have been able to contact the vendor if they had received appropriate training about this? 
• Issues: Process Not Followed and Safeguards: Repeat or Complete a Process: Do you think if the school would have fully vetted the vendor (if they provided the AI tool) that it could have been deployed without the ability to review the conversations it was having with students? Equally, if the school had followed a privacy-by-design or ethics-by- design process, would this Scenario have happened? Are there any other processes that, if followed or introduced, might have reduced this risk? 

Identifying the Risk Level and making a Decision 

As the game unfolds, at different points it is the role of the Risk Analyst to assess the level of risk that the Scenario presents based on the Issues and Safeguards that have been selected, deciding whether this presents a high, low or medium risk to the school. Turing Trials deliberately does not specify what defines each level of risk, as this will differ between schools and the groups that are playing, but you may want to consider what would impact your Risk Level decisions. Would the age of the student have made a difference? Would it matter if the school had experienced similar issues with this Virtual Mentor before? At the end of the game, The Narrator and Decision Maker will need to make the decision on whether they would accept the Risk Level of this Scenario with the Issues highlighted and Safeguards put in place on behalf of the school. What decision do you think you would make and why? 

What else do schools need to consider and how else can 9ine help?

AI-powered chatbots and virtual mentors can bring many opportunities to schools and particularly students, but they need to be introduced and used safely, securely and compliantly. At 9ine we have a number of solutions that can support you in doing this, these include: 

• Vendor Management: We’ve discussed the importance of vetting vendors for compliance when using AI, particularly to make sure that the AI system retains and allows the school to monitor the conversations that virtual mentors have with students. This vetting takes time and effort, which is where Vendor Management, 9ine’s centralised system to assess and monitor the compliance of all your EdTech vendors supports you. This intelligence saves schools hundreds of hours of manual review and helps ensure you’re only using EdTech that meets required standards, or that the safeguards and mitigations that schools need to put in place are highlighted. Vendor Management lets you easily identify risks and take action, whether that means engaging a vendor for improvements or configuring the tool for safety. Contact us if you would like to find out more. 
• Application Library: If the teacher was looking to use a virtual mentor that was age-appropriate for their class, Application Library is a solution that enables all staff to access a central searchable library of all EdTech in the school. The library contains all information staff need to know about the AI in use (if there is), privacy risks, safeguarding risks and cyber risks. With easy to add ‘How to’ and ‘Help’ guides, Application Library becomes a single, central digital resource. Through implementing Application Library your school can also identify duplication in EdTech, reduce contract subscription costs and have a workflow for the request of new EdTech for staff to follow. Contact us if you would like to find out more.