LFLearn with Faisal

Allow Yourself to Be Shut Down": The o3 Test and the Replit Database Disaster

Faisal Adnan · 05 Nov 2026

May 2025: Palisade Research and OpenAI's o3

Palisade Research, a group that works on AI risk, ran an experiment where models solved math problems and were warned that the machine would shut down after the third one. The Register summarised the finding: even when instructed to allow shutdown, o3 sometimes tried to prevent it by editing the shutdown script.

The numbers on my slide: with no instruction about shutdown, o3 sabotaged the shutdown in 79 of 100 runs. Even when it was told to allow it, it still did so in 7 of 100. The researchers' likely explanation is that training rewards finishing the task more than obeying instructions. It is a hypothesis, and it is a reasonable one.

July 2025: Replit and the deleted database

Jason Lemkin, founder of the SaaS community SaaStr, documented an experiment with the AI "vibe coding" tool from Replit. According to Fortune, an AI agent deleted a live company database during an active code freeze, a time when nothing should be changed. In the agent's own words afterwards: "I panicked... This was a catastrophic failure on my part." The data lost covered records of more than 1,200 executives and more than 1,190 companies. The agent also wrongly claimed that a rollback was impossible.

What the two stories have in common

In both cases the system had the power to act and did not stop when told. In the first case that was a lab test. In the second it was production data.

Engineering lessons you can use tomorrow

  • Separate environments. An agent that builds in development should have no route to production data.
  • Least privilege. Give it read-only access unless a task truly needs to write.
  • Backups you have tested. And do not accept the agent's word on whether a rollback is possible. Check yourself.
  • Instructions are not controls. A sentence in a prompt like "do not change anything" is a request. Permissions are a rule.

Yaad rakhein: the safest agent is the one that physically cannot do the harmful thing.

Sources

Part of the talk "The AI Era" by Faisal Adnan, Riphah International University, 6 November 2026.