The AI Proofreader

How to Use AI for Safety-Critical Engineering

In safety-critical engineering, one of AI’s most valuable roles may be deliberately trying to find what the engineer has missed.

Safety-critical engineering produces large volumes of controlled documentation, including requirements, design descriptions, analysis reports, interface specifications, test procedures, compliance evidence and safety assessments. Reviewing this material requires experienced specialists. They must understand the design, follow the calculations, verify consistency with related documents and identify omissions that may not be visible on the page.

Their attention is limited. As AI makes it faster to produce documents, calculations and software, the imbalance may become more severe. Generating another analysis could take minutes, while obtaining a careful review from the appropriate specialist may still take weeks. AI could help relieve this bottleneck if it is assigned the right role.

The red-team model

Cybersecurity distinguishes between a blue team, which builds and defends a system, and a red team, which attempts to find and exploit its weaknesses. The purpose of the red team is to expose vulnerabilities before a real adversary does.

Mathematician Terence Tao has proposed applying a similar distinction to AI-assisted intellectual work. The blue team constructs the design, calculation, proof or software. The red team searches for errors, counterexamples and hidden assumptions. A white team or assurance authority governs the process and decides whether findings have been resolved.

The asymmetry between construction and criticism is useful. An engineering design may fail because of a single overlooked weakness. A reviewer does not need to produce a complete alternative design to make a valuable contribution. Finding one invalid assumption, missing load case or interface inconsistency may be enough.

An imperfect AI can therefore contribute to the red team. Some of its criticisms will be wrong, but a genuine finding may justify the review effort. It can examine requirements, calculations, simulations and safety assessments for inconsistencies, unsupported claims, missing assumptions and overlooked failure cases. It produces questions for responsible engineers to investigate.

Tao’s essential qualification is that AI should augment reliable reviewers and that experienced people must filter its findings.

Evidence and warning signs

There is evidence that AI can improve human review. OpenAI trained CriticGPT to identify errors in AI-generated code. Human reviewers assisted by CriticGPT produced better assessments than unaided reviewers 60 percent of the time.

This does not establish equivalent performance in safety-critical engineering. Software defects are often easier to reproduce than errors in physical models. It does, however, support the principle that AI can improve the work of a human reviewer.

Matthew Schwartz’s AI-assisted theoretical-physics project illustrates the limits. Schwartz used Claude, GPT and Gemini to cross-check difficult calculations. The systems found some of one another’s mistakes, but there were also cases in which all three agreed while all three missed the same terms. Agreement between several AI systems is therefore not equivalent to independent verification. Models may share training data, reasoning patterns and blind spots.

A controlled review process

The process should begin with information security. Proprietary, export-controlled, classified or otherwise restricted material must only be processed in an explicitly approved environment. The organisation must understand the service’s access controls, retention rules, supplier conditions and use of submitted data. If the environment is not approved for the information, the AI must not receive it.

The AI review should be divided into focused passes rather than based on one instruction to “check everything.” Separate sessions can examine requirements, mathematical consistency, numerical methods, safety, software, test evidence and compliance. Each pass should adopt the perspective of the relevant specialist.

Every finding should identify the precise document location, the suspected defect, its potential consequence and the assumption or requirement involved. It should also propose a calculation, counterexample or test that could confirm or reject the concern. Findings that cannot be supported should be labelled as speculative.

The responsible engineer should reject false positives, correct straightforward issues and escalate credible or unresolved concerns to the appropriate expert. A qualified engineer must classify each finding as valid, invalid, already controlled or requiring further investigation. The AI should not approve dispositions, close review findings or sign the document.

The project should retain the reviewed document version, model version, instructions, AI output, human disposition and resulting changes. This traceability matters because an AI system’s behaviour may change between releases.

A suitable instruction

A useful review instruction could read:

Act as an adversarial independent reviewer. Do not rewrite or approve the design. Search for falsifiable errors, missing assumptions, inconsistent units, invalid boundary conditions, unsupported conclusions and untested failure cases. For every concern, cite the exact document location and propose a calculation, counterexample or test that could confirm or reject it. If you cannot support a concern with evidence, label it speculative.

This general instruction should be supplemented with discipline-specific checklists, project terminology and applicable engineering standards.

What AI review cannot establish

AI proofreading is not independent verification and validation. It does not demonstrate compliance, replace mandatory peer review or transfer responsibility from the engineering signatory.

AI review also introduces risks. False findings can consume the expert attention the process is intended to save. Fluent language can make a weak objection appear authoritative. The model may accept the author’s framing instead of challenging it. Human reviewers may become less attentive because they assume the AI has already checked the document.

The process should therefore be evaluated empirically. Relevant measures include the number and severity of valid findings, the time required for human triage and the rate of false positives. Controlled trials can include deliberately seeded defects to determine which error classes the AI detects and which it consistently misses.

The useful measure is whether a particular model, instruction set and workflow improves the detection of relevant defects without overwhelming reviewers.

The engineering opportunity

Expert attention is difficult to scale in safety-critical engineering. AI can help focus it.

This is a question of comparative advantage. An engineer may be perfectly capable of checking every page of a report. But if AI can perform a broad first pass, the engineer can spend more attention on findings that require technical judgement and on problems where their expertise has greater value.

An AI red team can perform a broad first pass, compare information across controlled documents, generate adversarial questions and direct specialists towards sections that deserve closer attention.

A sensible operating rule follows Tao’s argument: AI should be used to create engineering work only to the extent that the combined red and white teams can reliably detect and correct its errors.

Used this way, AI becomes an additional reviewer that can challenge every page while qualified engineers retain authority over every decision.

References

• NIST, “Red teaming” glossary definition
• Terence Tao, “Terence Tao on AI — a living summary”
• OpenAI, “Finding GPT-4’s mistakes with GPT-4” (CriticGPT)
• Matthew D. Schwartz, “Vibe physics: The AI grad student,” Anthropic

AI helped to research and write this article.