Designing for Imperfect AI

Much of the AI safety debate asks how to make models more reliable, aligned and controllable. In aerospace, we are used to asking a complementary question: how do we design the surrounding system so that a component can fail without the whole system becoming unsafe?

Aircraft are full of imperfect components. Sensors fail, software contains defects and humans make mistakes. Safety comes from preventing those individual failures from becoming unacceptable aircraft-level outcomes.

That gives us a useful starting point for AI. We can work with imperfect models if we design the system around their imperfections.

Start with the consequence

Consider an AI application that proposes a structural repair. The important failure condition is not simply that the AI produces an incorrect answer. It is that an inadequate repair is approved and released.

If the AI proposes a bad repair and an independent analysis catches the error, the model has failed but the safety architecture has worked.

If the same AI can modify controlled engineering data and release the repair directly, the identical model error has a very different consequence.

The safety problem therefore depends on the whole path from model output to final outcome. The objective is to keep an AI error from becoming a system failure.

Humans and AI fail differently

This also makes the discussion around humans in the loop more interesting.

A human does not necessarily improve safety because human judgement is inherently more reliable. The human can improve safety because humans and AI often fail differently.

But the architecture has to preserve genuine independence. If an engineer receives a polished recommendation from an AI that is correct almost all of the time, reads it quickly and clicks approve, the nominally independent review may become strongly influenced by the AI output.

“Human in the loop” is therefore too weak as a safety requirement. What matters is whether the checking path can fail differently from the path that produced the original result.

The same issue appears when one AI checks another. Two agents may look like two barriers while sharing the same model, data, assumptions or reasoning tendencies.

Aerospace scales assurance with consequence

In aircraft development, we increase assurance as the safety consequence becomes more severe.

For AI applications, the same reasoning is useful. A model summarising an internal meeting and the same model influencing a flight-control requirement do not create the same assurance problem.

This allows us to work with imperfect AI today. A model does not need to be trustworthy enough to autonomously release a structural repair before it can help prepare one.

We can limit its authority and build stronger barriers around consequential actions.

But this addresses only one part of AI safety.

Capability can change the safety argument

Today’s models can make ordinary mistakes such as hallucinations or incorrect reasoning. System architecture can prevent many of those mistakes from becoming consequential.

More capable systems create another challenge: they may acquire capabilities that weaken the assumptions behind the safeguards themselves.

This is where frontier AI safety adds a different approach. Google DeepMind, OpenAI and Anthropic all use capability thresholds to identify when models may require stronger safeguards. These thresholds can signal when an AI system’s safety architecture should be reviewed.

A capability threshold asks: What can this AI now do that it could not do before, and does that change the risk?

What if the AI can challenge the safeguards?

There is a deeper difference between AI and most components we are used to.

A hydraulic pump does not study its monitoring system and search for a way around it.

A sufficiently capable AI could be different. Suppose an AI understands the workflow surrounding its output. It knows which artefacts a human reviewer will inspect, which checks are automated and which tools it can access.

A monitoring system only provides protection while the AI cannot modify or reliably evade it.

This is why frontier AI safety research considers problems such as loss of control, sabotage, misalignment and the possibility that safeguards or oversight could be undermined.

We may eventually need to design around a component that understands the protection built around it.

Checks may need to run outside the AI’s environment, permissions may need to constrain what it can influence, and critical actions may need authorization through a path the AI cannot control.

A barrier only works while the system it constrains cannot also defeat that barrier.

A model becoming capable of an action does not mean we have to give it the authority to perform that action. We should increase AI authority only as fast as we can demonstrate that the surrounding safety architecture remains effective.

Bringing the two perspectives together

Aerospace starts with consequence: what can go wrong, and which barriers prevent that outcome?

Frontier AI safety adds another question: what new capabilities could weaken those barriers?

For AI-enabled engineering systems, we need both.

Engineering has always built dependable systems from imperfect components. We can do the same with AI.

But as AI becomes more capable, the safety architecture has to evolve with it.

References

• Google DeepMind, “Frontier safety at Google DeepMind”
• OpenAI, “Our updated Preparedness Framework”
• Anthropic, “Responsible Scaling Policy”

AI helped to research and write this article.