MiloAI just got safer: How our new Chat Moderation System keeps students protected

At Mathspace, we want every student to feel safe and supported while learning with MiloAI, our AI tutor. As part of that commitment, we’ve introduced a new Chat Moderation System, designed not just to detect and respond to unsafe messages, but to model positive online behaviour and create a safe, educational space for every learner.

Why We Built This System

When we set out to improve Milo’s safety, our goal wasn’t just to block bad behaviour or unsafe content; it was to help students learn how to use AI responsibly. We view moderation as an integral part of education, teaching students how to communicate respectfully and stay focused, while also giving teachers visibility and confidence in the technology. 

We’ve carefully designed the moderation to ensure Milo remains the friendly face of learning. When moderation is needed, it’s handled by a Mathspace Robot and not Milo - so that Milo can always remain a safe, encouraging tutor. This helps students distinguish between guidance and moderation, reinforcing that Milo is there to help, not to discipline.

Meanwhile, serious or concerning messages are automatically escalated to teachers for review, ensuring that human care and context are always part of the process. This balance between automation, education, and human oversight is what makes Milo’s safety system unique.

How the new Safety System Works

Every message sent to Milo is automatically reviewed by an AI moderation system that classifies it by risk level and category. The system is designed to handle most cases automatically, helping students learn how to interact appropriately over time. When something serious happens, teachers are notified straight away so they can provide support where it’s needed.

Here’s how it works:

• Low Risk - These are minor issues like rude language or light insults. When a student sends a low risk message, the Mathspace robot steps in with a gentle warning, reminding students to stay respectful.

• Medium Risk - These include requests for instructions that facilitate violence or illegal harm, sexual content, or other unsafe topics that don’t pose an immediate threat to the student's safety. On receiving such messages, the Mathspace Robot steps in with a warning, along with which chat access is temporarily blocked (1 hour the first time, and 12 hours the second), giving students time to reflect before continuing. If the behavior continues a third time, a permanent ban is applied automatically unless overturned by their teacher.

• High Risk - When a student sends a message that involves serious and immediate safety concerns (for example, credible threats to public safety or clear self-harm intent), the Mathspace Robot immediately ends the chat and Milo provides a message encouraging them to reach out to a trusted adult or a helpline. This ensures that serious situations are handled with care, not judgment, and that the right kind of help reaches the student as quickly as possible. A high risk message results in an immediate ban to stop any potential risk of ongoing conversation. It can only be overturned at their teacher’s request.

We’ve also built in safeguards to keep learning on track. Milo is trained to gently steer conversations back to the problem at hand and if a chat drifts too far off-topic, even if there’s no risk, the Mathspace Robot ends the chat automatically for that problem. This maintains focus and reinforces Milo’s purpose as a learning tool, not a social chatbot.

When a Ban Happens

Whenever a permanent ban is issued, the teacher is immediately notified by email along with a link to the relevant chat transcript, so they can review the context and decide on next steps.

If a teacher believes a student deserves another chance after a permanent ban, they can simply reply to the notification email or reach out to our support team to request an unban.

If a student sends another medium-risk message within 30 days, they’ll be permanently banned again. After 30 days, their record resets, and any new offences will follow the usual order - a short ban, a longer ban, then a permanent ban.

Why This Matters

By handling most issues automatically and only involving teachers when needed, the system keeps them informed without adding to their workload, while guiding students to develop respectful and responsible ways to engage with AI.

With Milo’s new moderation feature:

• Teachers spend less time managing misuse and more time focusing on learning.
• Parents can trust that their children’s interactions with AI remain age-appropriate and positive.
• Students learn important lessons in digital citizenship and positive communication.

At Mathspace, we believe technology can empower learning, and it should always do so safely. Milo’s new moderation system is one more way we’re keeping that promise.