Who Is Responsible When AI Goes Wrong?

A question we cannot dodge

Imagine a self-driving car brakes too late and hits a cyclist. A hospital's AI tool misses a tumour a human radiologist would have caught. A bank's automated system wrongly rejects thousands of loan applications from one neighbourhood. In each case, something has gone wrong and someone has been harmed. The natural question follows immediately: who is responsible?

It is tempting to point at the machine. "The AI made a mistake." But that answer, however natural, is a trap. It quietly suggests that because a computer was involved, no person needs to answer for the harm. This lesson argues the opposite: an AI system is a tool that people build, choose and switch on, and responsibility travels with those human decisions. Untangling exactly where it lands is genuinely hard, which is precisely why it is worth thinking through carefully.

Why software cannot be responsible

Start with the basic point. To be responsible for something, in any meaningful sense, you need certain capacities: to understand what you are doing, to weigh consequences, to have duties you can fulfil or fail, and to face the results. A person and a company can do these things. A trained model cannot. It has no intentions, no awareness, no stake in whether it gets things right, and nothing it can lose. It is a sophisticated pattern-processor doing what its design and data lead it to do. The limits of what these systems actually "understand" are explored in The Limits of AI.

So when we say "the AI decided", we are using a kind of shorthand. Behind that decision sit human choices: what problem to solve, what data to train on, where to deploy the result, and how much to trust it. "The algorithm did it" is not an explanation of who is responsible; it is a way of avoiding the question.

The chain of responsibility

The harder truth is that responsibility is rarely one person's. It is usually shared along a chain, and good thinking means looking at the whole chain rather than the most convenient link.

The data and its sources. Models learn from data, and biased or flawed data produces biased or flawed behaviour. If a hiring tool was trained on a history of discriminatory decisions, the harm was partly baked in before any code ran. The mechanism here is covered in Training Data and Bias in AI. Those who assembled and approved the data carry part of the responsibility.

The developers. The engineers and researchers who designed the system made choices: which method to use, what to optimise for, how to test it, what failure modes to check. A team that skipped testing across different groups, or optimised purely for a number without considering wellbeing, owns those decisions.

The deploying organisation. Often the company that uses a system did not build it. But choosing to deploy a tool in a high-stakes setting is itself a serious decision. A hospital that buys a diagnostic AI is responsible for checking it works on its patients, training staff, and setting up oversight. Buying a product does not outsource the responsibility for using it on real people.

The operator. Finally there is the person using the system in the moment, the driver, the doctor, the loan officer. Their responsibility depends heavily on how much real control and understanding they were given, which brings us to a subtle trap.

Automation bias: when humans become rubber stamps

A common response to "who is responsible?" is to keep a human "in the loop" who can override the machine. This is a good instinct, but it can fail in a specific way. People tend to over-trust automated systems, a tendency called automation bias. When a confident-looking machine gives an answer, humans often defer to it, even when they should question it, and especially when they are busy, tired, or told the system is "smart".

This matters enormously for responsibility. If a doctor is told to review every AI recommendation but, in practice, the system is right often enough that they stop genuinely checking, the "human in the loop" becomes a rubber stamp. Harm can then slip through with everyone assuming someone else was watching. A meaningful human safeguard requires the person to have the time, information and authority to actually disagree, not just a box to tick. Designing for genuine oversight, rather than the appearance of it, is part of responsible engineering.

Learning from real cases

Several real episodes show why this is not abstract. Automated welfare and benefit-fraud systems in more than one country have wrongly accused large numbers of people, causing severe hardship, and the resulting scandals turned on exactly these questions: who built the system, who chose to trust it, who could have caught the error, and who answers to the people harmed. Early self-driving vehicle crashes prompted investigations into the software makers, the safety drivers, and the companies running the tests. In none of these did "the computer did it" end the inquiry. Investigators traced the human decisions, and that is the right instinct.

A recurring lesson is that diffuse responsibility is dangerous. When many parties each assume another is accountable, harm can fall into the gaps between them. Clear, assigned responsibility is a safety feature, not just a legal nicety.

Designing for accountability

Because responsibility is easy to lose inside a complex system, the answer is to design systems so it stays visible. Responsible teams and regulators increasingly expect:

1. Documentation. A clear record of what the system was trained on, what it is for, how it was tested, and where it is known to fail. You cannot hold anyone to account for a black box no one described.
2. Traceability. Logs that show what decision was made and on what basis, so harm can be investigated rather than shrugged off.
3. Genuine human oversight. People with the time, training and power to question and override high-stakes outputs, designed to resist automation bias.
4. A route to appeal. Anyone affected by an automated decision should be able to ask why and to challenge it, and reach a human who can put it right.
5. Clear lines of accountability. It should be written down, before deployment, who answers if things go wrong.

Laws around the world are starting to require versions of these, especially for "high-risk" uses such as health, employment, policing and finance. The broader ethical framework behind all of this is laid out in AI Ethics and Fairness.

The bottom line

When an AI system goes wrong, the honest answer to "who is responsible?" is never "the AI". It is some combination of the people who shaped the data, built the system, chose to deploy it, and operated it, in proportions that depend on the case. The job of ethics, engineering and law is to keep that responsibility from evaporating into the machine. The most important habit you can carry away is to refuse the easy excuse, and ask instead: which human decisions led here, and who should answer for them? That question keeps powerful tools tied to human accountability, which is exactly where they belong.