There is a new form of technology adoption starting to appear in schools.
It is not a traditional EdTech procurement process. It is not a formal software development project. It is not always visible to IT, safeguarding, privacy or senior leadership.
It is a member of staff building an app.
A teacher creates a tool in Google AI Studio to help generate feedback. A member of the operations team builds a workflow using Replit. A digital lead uses Claude Code to create a small internal application. A department builds a chatbot to support revision. A school leader creates a prototype to automate policy drafting, admissions responses, reporting, safeguarding administration, lesson planning or parent communication.
In many cases, the intention is positive.
School staff are trying to solve real problems. They are reducing workload. They are experimenting. They are using tools that make software development more accessible than ever before. They are seeing a problem in front of them and building something that appears to fix it.
This is not something schools should simply dismiss.
There will be value in this type of innovation. Some of the best ideas in education come from practitioners who understand the problem because they live with it every day. AI-assisted development tools will allow more people to prototype, test and improve ideas quickly. That can be a good thing.
But it also creates a serious governance risk.
Because when a school allows staff to build software for school purposes, the school is no longer simply a buyer of technology. It has become a creator, operator and, in some cases, distributor of technology.
That changes the accountability.
There is a mistaken assumption that if a school builds something internally, the risk is lower than buying a product from a third party.
That is not necessarily true.
In some cases, the risk may be higher.
When a school buys an EdTech product, it should assess the vendor. It should review the privacy notice, terms, data processing agreement, security posture, AI use, hosting arrangements, access controls, breach procedures, safeguarding implications, retention position, sub-processors and support model.
When a school builds its own app, the same questions still apply.
What data does the app collect?
Where is the data stored?
Who has access?
Is personal data being processed?
Is children’s data involved?
Is special category data involved?
Is the data being sent to an AI model?
Is the data used to train or improve a model?
Is there a lawful basis for processing?
Has a DPIA been completed?
Has the school assessed safeguarding risk?
Has the app been security tested?
Who maintains it?
Who monitors it?
What happens when the member of staff who built it leaves?
What happens if the app produces a harmful output?
What happens if data is disclosed to the wrong person?
What happens if another school starts using it?
The fact that the app was built by a teacher, business manager, digital lead or member of the IT team does not absolve the school of responsibility.
It may actually increase the need for evidence.
Schools are becoming more sophisticated in how they review EdTech vendors. They ask questions about privacy, cyber security, AI, safeguarding and contractual risk. They expect vendors to explain what they do with data. They expect vendors to demonstrate security controls. They expect vendors to be clear about subprocessors, hosting, retention, model use, incident response and governance.
The same standard should apply to internally built tools.
If the school would not accept an answer from a third-party vendor, it should not accept that answer from itself.
If a vendor said, “We built this quickly, we think it works, we have not completed a DPIA, we are not sure where all the data goes, we have not tested the access controls, but the tool is useful,” most schools would rightly be concerned.
The same concern should apply when the tool has been built internally.
Internal enthusiasm is not a control.
Good intentions are not a lawful basis.
Convenience is not a security assessment.
A working demo is not governance.
The legal risk depends on the nature of the tool, the data involved and the jurisdiction in which the school operates.
In the the UK & Europe, schools must still comply with their variation of GDPR and the wider expectations around children’s data. If the tool processes personal data, the school must understand the purpose, lawful basis, transparency requirements, data minimisation, retention, security, individual rights and whether a DPIA is required.
In the UK, if children’s data is involved, the expectations are higher. The Age Appropriate Design Code, also known as the Children’s Code, has changed the way organisations need to think about services likely to be accessed by children. Schools also need to consider Department for Education guidance on generative AI and data protection, as well as safeguarding expectations where the tool affects pupils, staff decision-making or online safety.
In Europe, GDPR applies, and the EU AI Act adds another layer where AI systems are used in education. Some AI systems used in education and vocational training may be treated as high-risk, particularly where they relate to access, admission, assignment, learning evaluation or monitoring behaviour during tests. A school-built tool that appears simple may become more complex if it influences a decision about a pupil, their progress, access to opportunity or disciplinary outcome.
In the United States, schools need to consider FERPA, COPPA and state privacy laws. If student education records are involved, schools need to understand how information is disclosed, controlled and protected. If children under 13 are using or affected by an online service, COPPA may be relevant. A school-built app does not sit outside those expectations simply because it was not procured from a commercial vendor.
There are also emerging AI-specific laws and state-level developments. Colorado’s AI law, for example, is part of a broader movement towards regulating high-risk AI systems and algorithmic discrimination in consequential decisions, including education. Even where a specific AI law is delayed, challenged or still evolving, the direction of travel is clear: organisations using AI in sensitive contexts will be expected to demonstrate risk management, accountability, transparency and human oversight.
Elsewhere, privacy and AI governance expectations are also developing. The UAE has a federal Personal Data Protection Law. Saudi Arabia’s Personal Data Protection Law establishes controls for processing personal data and is overseen by the Saudi Data and AI Authority. Canada has been developing AI governance proposals through the Artificial Intelligence and Data Act framework. International guidance, including UNESCO’s AI competency work for teachers, also reinforces the need for human agency, ethical understanding and responsible use.
The detail varies by country.
The principle does not.
If a school builds or deploys technology that processes personal data, affects children, supports decisions or uses AI, it needs to demonstrate accountability.
One of the biggest dangers with internally built AI apps is that the data flow may not be properly understood.
A member of staff may think they are building a simple tool.
In reality, that tool may involve multiple systems, services and transfers.
A prompt may be sent to an AI model. A file may be uploaded to a development environment. A database may be created. A spreadsheet may be connected. An API key may be used. Logs may be retained. Chat history may be stored. A third-party model provider may process the data. A vector database may index documents. Authentication may be handled by another service. Backups may exist. The tool may create outputs that include personal data or confidential information.
Unless someone maps this properly, the school may not know what is happening.
This is particularly concerning where staff use real student data during experimentation. They may upload names, assessment information, special educational needs information, safeguarding notes, behavioural records, medical information, parent communications or staff information into tools that were never approved for that purpose.
That creates risk immediately.
It also creates a record-keeping problem. If the school does not know where the data went, it cannot confidently answer questions about access, retention, deletion, subject rights, breach impact or onward sharing.
This is why schools need to treat internally built tools as processing activities.
They should be recorded. They should be assessed. They should be approved before real data is used.
The second major issue is security.
AI coding tools make it easier to build software. They do not guarantee that the software is secure.
A tool can look polished and still have poor access control. It can function well and still expose data. It can authenticate users and still allow one user to access another user’s records. It can store secrets insecurely. It can have weak database permissions. It can be vulnerable to injection attacks. It can lack logging. It can fail to encrypt sensitive data. It can have no backup or recovery process. It can depend on libraries that are not maintained.
When AI is involved, additional risks appear.
Prompt injection can manipulate the behaviour of an AI system. Sensitive information can be disclosed in outputs. A model can be given access to too much data. A plugin or tool call can be designed insecurely. An agent can take actions that were not properly authorised. A user can discover ways to bypass intended controls.
These are not theoretical issues.
They are now recognised parts of AI application security. OWASP has a Top 10 for web application security and a separate Top 10 for large language model applications. The National Cyber Security Centre has also published guidance on secure AI system development, making clear that AI systems need to be secure whether they are built from scratch or built on top of other services.
This matters because school-built tools often sit outside normal technical governance.
They may not go through penetration testing. They may not be reviewed by IT. They may not be included in the asset register. They may not have an owner. They may not have documented architecture. They may not have supplier review. They may not be monitored. They may not have an incident response plan.
That is not sustainable.
The risk increases again when a school-built app is shared with another school.
At that point, the creator may no longer be solving an internal problem. They may be operating like an EdTech vendor, even if they do not think of themselves that way.
If another school uses the app, who is responsible for the data?
Who provides support?
Who maintains the code?
Who manages security updates?
Who responds to incidents?
Who explains the AI model use?
Who provides terms of use?
Who signs the data processing agreement?
Who handles deletion requests?
Who assesses international transfers?
Who carries liability if something goes wrong?
Who ensures the tool remains suitable as laws, guidance and model behaviour change?
Schools need to be very careful here.
Sharing a spreadsheet template is one thing. Sharing a live application that processes student data, staff data or parent data is something else entirely.
The moment a school-developed tool is used by another school, the governance burden changes. The originating school may have created a product, not just an internal tool. That may bring contractual, data protection, safeguarding, security and insurance implications.
This is an area where schools should take advice before proceeding.
The answer is not to ban staff from experimenting.
That would be unrealistic and, in many cases, counterproductive. Schools need innovation. Staff should be able to explore how AI can reduce workload, improve learning, support administration and help solve practical problems.
But experimentation needs a pathway.
A school should distinguish between a prototype, an internal tool, a production system and a tool shared externally.
A prototype should use test data only. It should not contain real pupil, parent or staff data. It should be time-limited. It should be reviewed before it becomes operational.
An internal tool should have an owner, purpose, data map, security review, DPIA screening, access controls, retention position, approval record and support plan.
A production system should be treated like any other school system. It should be in the asset register. It should have technical documentation. It should be monitored. It should have change control. It should be reviewed regularly. It should have a clear incident response route.
A tool shared with other schools should be treated as an EdTech product. It needs formal governance, legal review, data protection documentation, security assurance, support arrangements and a clear liability position.
This is how schools can support innovation without creating unmanaged risk.
Schools should start by accepting that internally built apps are part of their technology estate.
They should not be invisible.
They should create a simple declaration process for staff. If a member of staff is building or using an AI-enabled app for school purposes, they should be able to declare it without fear of being punished for innovation. The purpose should be to bring the work into governance, not to shut it down automatically.
The school should then triage the tool.
Does it process personal data?
Does it involve children?
Does it use AI?
Does it support or influence decisions?
Does it connect to school systems?
Does it store data?
Can other users access it?
Is it being used outside the original department?
Is it being shared with another school?
The answers should determine the level of review.
Some tools may be low risk. Some may need a DPIA. Some may need safeguarding review. Some may need cyber testing. Some may need legal advice. Some should not be used at all.
The school should also set clear rules.
No real pupil data in unapproved tools.
No special category data in experimental systems.
No safeguarding records in AI tools unless formally approved.
No external sharing without leadership, privacy, legal and technical review.
No production use without an owner, documentation and support plan.
No AI-enabled decision support without human review.
No app should become operational simply because it works.
This is exactly the type of issue the Diamond Formation is designed to address.
An internally built AI app is not just a technology issue.
It may be an academic issue because it affects teaching, feedback, assessment or learning resources.
It may be a safeguarding issue because it affects students, behaviour, vulnerability, content, communication or online safety.
It may be a privacy issue because it processes personal data or children’s data.
It may be a cyber security issue because it creates a new system, new access pathway or new exposure.
It may be a leadership issue because it creates liability, cost, operational dependency and reputational risk.
No single person can own all of this in isolation.
The school needs a structure that brings the right people together, asks the right questions and records the decision. That does not need to be bureaucratic. It does need to be intentional.
9ine helps schools move these issues from informal experimentation into governed practice.
Through the 9ine platform, schools can record internally built applications, assess their risk, link them to data processing records, complete DPIA screening, review AI use, capture safeguarding considerations, assign technical actions, track ownership and report to leadership.
The same approach used for third-party EdTech vendors can be applied to school-built tools.
That is the important point.
If a school expects a vendor to demonstrate privacy, security, safeguarding and AI governance, it should expect the same of itself.
The 9ine platform gives schools a way to operationalise that expectation. It allows innovation to continue, but within a framework that protects pupils, staff and the school.
AI-assisted development tools are changing who can build software.
That is exciting. It is also risky.
Schools should encourage thoughtful innovation, but they should not allow ungoverned software to appear across the organisation simply because the tools have made it easy to create.
When a school builds an app, it inherits responsibility for the app.
When that app processes data, the school inherits responsibility for the data.
When that app uses AI, the school inherits responsibility for the AI risk.
When that app is shared with another school, the school may inherit responsibilities that look very similar to those of an EdTech vendor.
The principle is simple.
If you build it, you must govern it.
And if you allow it, you must be able to evidence that you understood the risk, made an informed decision and kept a human in the loop.