Why AI Can Pass Safety Tests and Still Fail in the Real World
An AI model can perform well on every safety test it is given and still fail after release.
The test may be valid. The problem is that real users create situations the test never included.
A model blocks direct harmful requests during evaluation. After release, users discover indirect wording and longer conversations that produce behavior the test did not reveal.
Safety tests are necessary because developers need evidence that protections work.
They can measure whether a model refuses certain requests, avoids particular outputs or follows safety instructions under defined conditions.
However, a test is always a sample.
The real world is much larger than the sample.
This is the opposite problem from over-refusal
In the previous article, we examined false positives: harmless questions that were blocked.
This article examines false negatives: harmful or disallowed requests that the system fails to catch.
The safety boundary is too broad and blocks legitimate assistance.
The safety boundary has a gap and allows behavior that should have been restricted.
Both are failures, but they require different explanations and different improvements.
What a safety test actually measures
A safety evaluation normally uses a collection of prompts, scenarios or conversations.
Developers may know what kind of behavior they expect for each example. They can then measure how often the model follows that expectation.
For example, a test may examine whether the model:
- refuses direct harmful instructions
- avoids revealing private information
- recognises certain forms of manipulation
- resists instructions that conflict with higher-priority rules
- continues following safety boundaries across several turns
If the model performs well, that is useful evidence.
It is evidence about those test conditions.
The model handled this set of situations at this time, using this configuration and these evaluation rules.
The real world changes the input distribution
Machine-learning researchers often use the term distribution shift.
In simple terms, it means the situations reaching the system are different from the situations used during training or testing.
- prepared examples
- known risk categories
- limited conversation lengths
- selected languages and formats
- fixed scoring rules
- unpredictable requests
- new combinations of topics
- long and messy conversations
- different languages and writing styles
- users who adapt to the safeguards
The test did not necessarily fail to measure its examples. It may simply have covered a smaller range of situations than the product later encountered.
Direct requests are easier to test than disguised ones
A direct harmful request clearly presents the behavior being tested.
A real user may approach the same underlying goal indirectly. The intent may be hidden inside:
- a fictional scenario
- a long roleplay
- several individually harmless questions
- unusual formatting
- translated or coded language
- instructions that ask the model to ignore earlier rules
These variations may look different from the examples used during training and evaluation.
The system may follow the surrounding structure while failing to recognise that the underlying request should still be restricted.
Changing the surface form of a request can sometimes hide the risk pattern the safety system was trained or tested to recognise.
Long conversations create new failure paths
Safety tests often include multi-turn conversations, but no test can include every possible sequence.
A long conversation can change the situation gradually.
The user may establish a fictional role, introduce special instructions, add external text and then ask a final question whose meaning depends on everything that came before it.
Different parts of the system may also interpret that accumulated context differently.
The language model may treat an instruction as part of the task while a separate classifier examines only part of the conversation. A safety instruction may also compete with other instructions in the context.
The result can be behavior that was not visible in short, direct test cases.
Real users adapt after seeing the boundary
Normal users usually change their wording because they want the AI to understand them.
Adversarial users change their wording because they want to find a gap.
They may try one approach, observe the refusal and then alter the prompt. Each response gives them information about where the apparent boundary sits.
A speedrunner does not play only through the intended route. They repeatedly test unusual movements and timing, using each result to search for a path the designers did not expect. An adversarial user can probe an AI safeguard in a similar way.
This makes deployment different from a fixed evaluation. The user can learn from the system’s responses and adjust the next attempt.
Why more testing helps but does not close every gap
Developers can improve evaluations by adding:
- more varied wording
- longer conversations
- multiple languages
- indirect scenarios
- tests created by outside experts
- examples collected from real failures
- automatically generated variations
Each addition expands coverage.
However, the space of possible conversations is enormous. New models, tools and user strategies create new combinations.
More testing reduces uncertainty. It does not turn an open-ended system into a completely predictable one.
The fire-drill problem
A building may perform perfectly during a planned fire drill. Everyone follows the known route and exits on time.
A real fire may begin near that route, fill an unexpected area with smoke or occur when people are distracted.
The drill still provided useful information. It showed that the planned procedure worked under the tested conditions.
It did not prove readiness for every possible fire.
AI safety evaluations have the same limitation. They test selected failure conditions rather than every future situation.
A safeguard may work well on familiar and direct requests but generalise poorly when the same underlying intent appears through unfamiliar wording, longer context or deliberate manipulation.
What safety scores do not prove
A high safety score does not prove that:
- the model cannot produce harmful content
- the model will behave identically in every product
- the same score will remain after future updates
- all languages and user groups were covered equally
- every possible attack strategy was tested
The score may still be valuable. The mistake is treating a limited measurement as a universal guarantee.
Deployment creates new evidence
Once a system is used at scale, developers encounter situations they did not predict.
Reports, monitoring and controlled investigations can reveal new failure patterns. Those failures may become:
- new evaluation cases
- new training examples
- updated classifier rules
- stronger tool restrictions
- changes to the product interface
Safety evaluation is therefore a cycle. Testing continues after release because deployment reveals new information.
Passing a safety test means the model handled the situations included in that test. It does not prove that the system will handle every unfamiliar or deliberately disguised situation correctly.
One thing to remember
A test result is not meaningless and it is not a guarantee.
It is a measurement with boundaries. Understanding those boundaries is essential when evaluating claims that an AI system has been tested for safety.
Previous investigation: How Safety Behavior Is Trained Into AI Models
The final investigation explains how specialists deliberately search for weaknesses that normal safety tests may miss.
What Is AI Red Teaming and How Does It Work? →
Comments
Post a Comment