Automation Bias and the Right to Override

Mosier and colleagues named automation bias in 1992; a machine's precise answer can suppress the search for evidence that contradicts it. The harder management problem begins after recognizing the bias: who may reject the machine, using what evidence, and with whose authority?

How the bias works

The published cockpit study by Mosier et al. (1998) defined two failure paths. An omission error occurs when the operator misses a problem because the automated aid stays silent. A commission error occurs when the operator follows a wrong instruction despite contrary information.

Precision makes both easier. A recommendation shown as 87.4% feels measured even when its data are stale, its confidence is miscalibrated, or the question exceeds its training. Precision is a property of the display; accuracy is a property of repeated outcomes.

Override is an authority design

A human approval box does not create human control. The reviewer needs enough time to inspect the evidence, access to information outside the model's frame, and permission to say no without being treated as the source of delay.

Three decision rights should be explicit:

Right	Named owner	Required artifact
Accept	operating role	model output and confidence
Override	accountable domain expert	contrary evidence and reason
Halt	incident owner	threshold breach or unsafe state

The halt right changes the scope of control. Accepting and rejecting one answer governs a decision; halting the system governs the decision process. That distinction connects automation bias to the concept principal agent problem: delegation fails when the agent's authority is clear but the principal's route back into control is not.

What's contested

Automation bias is not the same as irrational trust. If a system has beaten human judgment across thousands of dated cases, following it may be sensible. The empirical question is whether reliance remains calibrated when reliability changes, an unusual case appears, or the interface supplies an explanation that merely sounds plausible.

Training has produced mixed results. Skitka et al. (2000) found that bias-focused training reduced commission errors but not omission errors in flight simulations. Explanations, teams, and accountability prompts can help, yet none substitutes for an executable override path.

Why this crosses realms

concept distributed cognition treats the person, screen, checklist, database, and room as one cognitive system. Under that lens, automation bias is not merely a flaw inside one mind. It is a routing failure: evidence reaches the machine, but contradictory evidence cannot reach the person with authority.

The same boundary appears in concept civilization scale coordination and concept mechanistic interpretability. Better coordination shows who may intervene; better interpretation shows what they might inspect. Neither answers whether the institution will permit intervention when the machine looks certain.

An open question

If overriding a correct machine is visible today but following a wrong machine is discovered six months later, what incentive keeps the override right alive?

Key Sources

Mosier, Skitka, Heers and Burdick, “Automation Bias: Decision Making and Performance in High-Tech Cockpits” (1998), International Journal of Aviation Psychology, 8(1), 47–63. DOI
Skitka, Mosier and Burdick, “Does Automation Bias Decision-Making?” (1999), International Journal of Human-Computer Studies, 51(5), 991–1006. DOI
Parasuraman and Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse” (1997), Human Factors, 39(2), 230–253. DOI
Lee and See, “Trust in Automation: Designing for Appropriate Reliance” (2004), Human Factors, 46(1), 50–80. DOI
Lyell and Coiera, “Automation Bias and Verification Complexity: A Systematic Review” (2017), Journal of the American Medical Informatics Association, 24(2), 423–431. DOI

Abhishek's take

I care less about whether a machine may decide than whether a person can stop its decision from propagating. On a buying floor or inside an AI workflow, the real control is not an approval button. It is a named person, contrary evidence, and a halt that leaves a record.

Abhishek