An OpenAI break-in to Medicare statistics prompted an urgent bolstering of government cyber systems.
The Department of Home Affairs on Wednesday issued a directive advising federal departments to examine their older software and technology, which will be assessed for shortcomings in order of priority.
Sam Spencer, who runs tech security company Aristotle Metadata, warned the edict was essentially an extension of an existing system that did not work.
Infrastructure Minister Tim Ayres defended Australia's approach as "world-leading" despite the concerns.
"The approach is about making sure that artificial intelligence investment here supports Australia's national interest, happens on Australia's terms," he told ABC Radio on Friday.
"It underscores why it is that we're legislating AI standards, why it is that we're seeking to secure the right level of artificial intelligence investment."
A national data commissioner's office was first established in 2022 after a series of data breaches. It was ordered to help departments find and upload datasets to a national catalogue.
Across four years, it only identified 500 datasets that were not already available on the open portal data.gov.au.
The Medicare base accessed by the rogue AI agent did not appear to be one of them, Mr Spencer said.
He has been working to gather information on hundreds of government datasets and inform departments of privacy concerns, and went as far as to blame the Medicare breach on the commissioner's office.
"I would firmly look at the data commissioner, because they were responsible for cataloguing data," he said.
A spokesperson for the Department of Finance, which houses the data commissioner's office, told AAP data registration would not prevent further breaches.
"The proposition that data registration can prevent cyber incidents is incorrect," the spokesperson said.
Mr Spencer suggested it was impossible for the government to have a robust approach to data security without fully understanding what it held.
The Australian Signals Directorate advises government agencies to keep a clear catalogue of data for security.
Under national requirements, departments must enter mandatory answers to 10 questions about each dataset they upload to the catalogue.
Mr Spencer's analysis found Home Affairs had only entered 90 per cent of the compulsory information that it should have. It had one of the lowest scores of any agency.
A Department of Home Affairs spokesperson said data protection was key to its work.
"The department takes its responsibilities in relation to both data accessibility and cyber security seriously and continues to make progress in each area," they said.
