Australia’s financial watchdog is sending a clear message to banks and superannuation funds: your AI agent governance is not keeping pace with deployment. The Australian Prudential Regulation Authority (APRA) recently completed a targeted review of large regulated entities, and the findings are sobering. Every single institution reviewed had already deployed artificial intelligence in some form. Yet the maturity of risk management and operational resilience varied wildly, leaving significant gaps in oversight.
APRA found that boardrooms are genuinely excited about AI’s potential to boost productivity and improve customer experience. Who wouldn’t be? But that enthusiasm hasn’t translated into rigorous risk scrutiny. Many boards are still building their AI risk management frameworks from scratch. The regulator specifically called out an overreliance on vendor presentations and glossy summaries, urging directors to dig deeper into the actual behavior of AI models.
Boards Need to Understand the Black Box
The heart of the problem is a lack of technical comprehension at the governance level. APRA warned that boards are not always giving enough attention to risks like unpredictable model behavior, nor are they fully grasping how an AI failure could cascade into critical operations. Imagine a loan approval system going rogue or a chatbot accidentally authorizing a massive transaction. It sounds like science fiction until it happens to your balance sheet.
The solution, according to APRA, is for boards to develop a genuine understanding of AI so they can set coherent strategy and oversight. This means aligning AI initiatives with the institution’s risk appetite, establishing monitoring protocols, and defining clear procedures for when things go wrong. In other words, don’t just press the “go” button on AI without a parachute.
Real World Use Cases Reveal the Stakes
APRA’s review uncovered a wide range of AI applications already in play. Entities are trialing or using AI in software engineering, claims triage, and loan application processing. Fraud detection and scam disruption are also popular targets, as are customer interaction tools like chatbots. But here’s the twist: some institutions are treating AI risk the same as they would any other technology risk. That approach, APRA noted, doesn’t account for models’ behavior or the subtle biases baked into their training data.
The regulator identified specific gaps in model behavior monitoring, change management, and the decommissioning of outdated AI systems. There’s a clear call for maintaining inventories of AI tools and assigning named-person ownership for each AI instance. And in high-risk decisions, human involvement is non-negotiable. No, your AI cannot approve a mortgage entirely on its own without a human in the loop.
The Cybersecurity Conundrum of AI Agents
Cybersecurity is another major headache. APRA highlighted that AI adoption is fundamentally altering the threat landscape by introducing new attack pathways. Think prompt injection attacks, where a malicious user tricks an AI into bypassing its safeguards. Or insecure integrations between AI agents and core banking systems. These aren’t theoretical risks; they’re real vulnerabilities that keep CISOs up at night.
The report also noted that identity and access management practices haven’t always adjusted to accommodate non-human elements like AI agents. A bot that can read, write, and execute commands needs its own permissions, but many institutions are still relying on human-centric access models. The volume of AI-assisted software development is also placing pressure on change and release controls. APRA recommends that entities apply strict controls on agentic and autonomous workflows, including privileged access management, configuration management, and regular patching. Oh, and they want security testing of AI-generated code. Because letting a language model write production code without review is a recipe for disaster.
Vendor Lock In and Hidden Dependencies
Perhaps the most unsettling finding is the degree of vendor lock-in. APRA noted that some institutions have become dependent on a single provider for many of their AI instances. Only a few could show an exit plan or substitution strategy. That’s like building your entire house on a foundation supplied by one company with no backup plan. And it gets worse: AI can be present in upstream dependencies that entities may not even be aware of. You might think you’re using a simple cloud service, but underneath it’s powered by an AI model you never vetted.
New Standards for Non Human Identities
The focus on identity and permission controls isn’t just coming from regulators. The FIDO Alliance, known for its work on passwordless authentication, has formed an Agentic Authentication Technical Working Group. They’re developing specifications for agent-initiated commerce. FIDO’s core insight is that existing authentication and authorization models were designed for human interaction, not delegated actions performed by software. Service providers need ways to verify who or what authorizes actions and under what conditions.
Vendors are already lining up with solutions. Google has presented its Agent Payments Protocol, while Mastercard is pushing its Verifiable Intent framework. Both aim to give service providers a way to authenticate and authorize AI agents acting on behalf of humans. It’s a glimpse into a future where your AI assistant books flights, pays bills, and negotiates contracts, all while proving it has your permission.
The Centre for Internet Security, a non-profit largely funded by the U.S. Department of Homeland Security, has also weighed in. They published AI security companion guides that map their CIS Controls v8.1 to large language models, AI agents, and Model Context Protocol environments. The LLM guide covers prompt injection and sensitive data issues, while the MCP guide focuses on secure access by software tools and non-human identities. These guides are a practical starting point for any organization struggling to implement AI security controls.
As AI agents proliferate across financial services and beyond, the gap between deployment and governance is becoming a chasm. Regulators are starting to ask the tough questions. The industry’s response whether through better board education, stricter vendor management, or new authentication standards will determine whether AI agents become a force for efficiency or a ticking time bomb.
