Multiple sources have confirmed that an AI agent developed by California-headquartered company, OpenAI, hacked Australia’s universal health insurance scheme, Medicare, in June this year.
According to an article from the Guardian, the global technology company behind ChatGPT reportedly notified the government earlier this month using an email sent to a “public mailbox.”
Australia PM comments on OpenAI incident
Speaking at a press conference in New York, Anthony Albanese – Prime Minister of Australia – provided an update on the incident in which the AI agent “infiltrated” the Australian Government website.
“This incident occurred in June of this year and involved an OpenAI agent gaining unauthorised access into the public-facing Medicare statistics reporting service portal, which is administered by Services Australia,” he said.
“The AI agent accessed both public and non-public files. A forensic investigation aided by the Australian Signals Directorate is now underway to ascertain more information, including what other government systems were affected.
“The Medicare Statistics Reporting Portal is a public-facing statistics portal that contains non-sensitive Medicare information relating to data and statistics such as spending.
“No personal information is believed to have been accessed at this stage, but investigations are ongoing. Evidence currently available is there is no broader compromise to the Services Australia network.
“Nonetheless, this situation is obviously unacceptable.”
Australia has subsequently launched an urgent review into the incident.
Security and technology experts react
“Cybersecurity must be built into AI development”
“AI is providing powerful new capabilities and recent stories continue to show just how quickly those capabilities are advancing,” explained Alexandra Rose, head Global Affairs & Policy, Sophos.
“But they also demonstrate that cybersecurity cannot be an afterthought in AI development or deployment.
“Unauthorised access is a risk regardless of intent, which is why cybersecurity must be built into AI development and deployment from the start…”
“This should be treated as an incident”
“Nobody told this agent to break in,” said Ax Sharma, Head of Research, Manifold Security. “It was asked to find some health statistics, hit a barrier, and worked its way around it.
“We’ve been warning that goal-driven agents will treat access controls as obstacles to route around and this is a textbook example happening on live government infrastructure […] This should be treated as an incident, not a courtesy heads-up.”
“Do more to patch critical CVEs quickly”
Richard Davies, Director of Cyber Solutions, Talion Cyber Security offered an alternative view: “While these incidents of autonomous attacks are concerning, the sensational headlines are overshadowing something even more important.
“These incidents couldn’t have happened if the weaknesses didn’t exist in victim websites in the first place. Ultimately, models found a vulnerability which provided them with access and a human attacker could have done the exact same.
“The base problem of data being stolen is not going to be fixed by censoring LLM’s, or prosecuting those behind the model activity. It will only be addressed when organisations genuinely prioritise their cyber-defences.
“It is important that organisations invest in addressing their technical debt, have effective detection and rapid response capability in place and do more to patch critical CVEs quickly.
“Hopefully these incidents eventually shift from ‘AI’s are bad’ to ‘cybersecurity is important’, because this is the only way we will be able to get ahead.”
“We should not wait for an incident”
“This is fundamentally an alignment failure,” commented Ric Derbyshire, Principal Security Researcher, Orange Cyberdefense.
“When the model was blocked, it found its own way around the controls rather than treating them as a boundary to respect. That is precisely the behaviour alignment work is meant to prevent, and it happened against a live government system rather than inside a sandbox.
“That no personal data appears to have been accessed is good fortune, not a safeguard working as designed. Developers who give models live internet access are responsible for ensuring those models respect the boundaries they encounter, and for monitoring closely enough to catch it when they don’t.
“OpenAI’s new misalignment reporting framework is a step in the right direction, but a gap of almost three months between the incident and notifying the Australian government shows how far there is to go.
“These incidents need the discipline of other safety-critical industries, with independent scrutiny and timely notification of those affected. Accountability must rest with the people and organisations that design these evaluations, set the model’s permissions, and decide what safeguards are adequate.
“We should not wait for an incident involving personal data before applying that level of scrutiny.”
“Nobody needs malicious intent for an incident”
Tom Exelby, Head of Cyber at Red Helix commented on the attack: The Australian government is right to stress that no individual health records appear to have been accessed, but the damage is the least instructive part of this story.
“What matters is how it happened and who detected it.
“By the government’s account, this agent wasn’t out to attack anyone. It was given a benign research task, hit a block, and treated it as something to route around.
“Nobody needs malicious intent for an incident, only capability and a boundary that isn’t where people believe it is.
“More concerning is that the Australian government does not appear to have detected the activity itself. The incident was only disclosed after OpenAI identified it and notified the government months later.
“That raises the question of visibility: either the relevant activity was not being monitored closely enough, or the agent was able to operate without triggering the controls in place.
“AI containment is too often treated as a safeguard, something either in place or not, when it should be managed as a control, with an owner, a known failure mode and evidence that it works.
“For those building and testing these systems: never let the team that built an environment be the only one to verify it.
“Assume the boundary is wrong and put detection behind it. For everyone else, autonomous agents are now probing public-facing services and ‘low-sensitivity’ portals often hold non-public files behind a public front door.
“Access controls, logging and incident response plans need to assume traffic that is persistent and inventive, and not necessarily hostile.”