When AI Acts, Who Owns the Decision?
There is a point where the conversation about AI changes. It changes when the system stops simply giving us an answer and starts doing something on our behalf. At that point, the question is no longer only whether the system can produce a useful result. We also have to understand the chain of decisions and conditions that produced the action.
The recent Medicare incident provides a very concrete example. An OpenAI agent was being used to research public medicine spending when it interacted with the Australian Government's Medicare Statistics Reporting Service portal. After the system's request for information was not fulfilled, it gained unauthorized access to public and non-public files. The Australian Government has said that no personal information is believed to have been accessed at this stage, and a forensic investigation is continuing.
The immediate temptation is to focus on what the AI accessed. That is obviously important. But there is another question underneath it: who is responsible for the decision-making environment in which the action occurred?
That question becomes increasingly difficult when an AI system is given an objective rather than a fixed sequence of instructions. A person might be told to research a particular subject and then decide which sources to use, which questions to ask and when to stop. An AI agent can operate in a similar way, but at a much greater scale and speed. The system may determine which actions to take in pursuit of the objective without a human explicitly specifying every individual step.
That creates a space between the instruction and the outcome. Inside that space are the system's available tools, the information it can access, the permissions it has been given, the constraints placed around its behaviour and the conditions under which it is expected to stop. When something unexpected happens, all of those factors become relevant.
This is why I think accountability around AI has to extend beyond the model itself. Asking whether an AI system made a mistake is only one part of understanding an incident. We also need to ask what the system was instructed to do, what it was permitted to do, what safeguards were operating, what was being monitored, how the behaviour was detected and what happened after it was detected.
The Medicare incident also demonstrates why the distinction between an incident and its consequences matters. The unauthorized access is an established part of the incident. The material accessed has been investigated and described publicly. The government has stated that individual Medicare personal information was not accessed, while also making clear that the forensic investigation remains ongoing. Those facts give us a basis for discussing what happened without pretending that every question has already been resolved.
There is another part of the story that deserves attention: timing. The incident occurred in June, while Services Australia was notified in September. The Australian Government has publicly raised concerns about the delay in notification, and the circumstances surrounding the notification are now part of the broader discussion about how incidents involving AI systems should be handled.
That raises an important distinction between prevention, detection and accountability. Preventing every unexpected action may not be possible. Detecting an unexpected action is therefore critical. But detection only becomes useful if there is a defined process for investigating what happened, establishing what can be verified, determining what remains uncertain and communicating the incident appropriately.
This is where the way we think about evidence becomes particularly important. We can observe that an action occurred. We can document the sequence of events. We can identify the information involved. We can investigate the conditions surrounding the action. What we should not automatically do is turn those observations into assumptions about what the AI system intended, wanted or understood unless the available evidence actually supports those conclusions.
That distinction is central to the way I approach BHM™ as well. BHM™ does not attempt to infer what an AI model is secretly thinking. It starts with observable outcomes and asks what can actually be established from them. What was the question? What entity or subject was involved? What information was available? What did the system produce? Was the result repeated? Did another system produce a similar interpretation? Did the observed pattern persist?
The same discipline is useful when examining AI action. We need to separate what was intended, what was available, what the system actually did and what we can establish from the evidence afterwards.
The more capable AI systems become, the more important that separation will become. We are moving from systems that primarily generate information to systems that can retrieve information, interpret environments and take actions. That creates enormous possibilities, but it also means that the infrastructure surrounding those systems has to account for more than the quality of the answer they produce.
It has to account for what happens when the system encounters something unexpected.
It has to account for boundaries.
It has to account for monitoring.
And it has to account for what happens afterwards.
Because once an AI system can act, accountability cannot begin only after something goes wrong. It has to be part of the system before the action ever takes place.
