by Maple & North
The Word That Gets Dropped Somewhere Around Month Six
Every AI agent starts life described as a "copilot." That word does real work in a sales deck: it implies a human is still flying the plane, the agent just hands over checklists and watches the instruments. The trouble is that the word rarely survives contact with production.
Picture a mid-sized Canadian financial services firm — the kind with a few hundred branches, a legacy core banking system, and a mandate to cut call-centre costs. It deploys an agent to handle routine account inquiries and initiate low-value transfers on customer request. For the first few months, that's exactly what it does: a narrow, supervised task, reviewed by a human before anything moves. Then the scope creeps, the way scope always creeps. The agent starts resolving disputed transactions under a threshold. Then it starts flagging accounts for fraud review on its own judgment, without a human in the loop for every call. Nobody signed a memo authorizing this transition. It happened because the agent worked, nobody objected, and removing a review step saved time every single day.
This is the actual shape of the risk. It is not a single dramatic failure — a rogue agent wiring money to the wrong account. It's the quiet reclassification of a tool from "copilot" to "autopilot" with no corresponding update to who is accountable when it's wrong. That reclassification is the real subject of AI agent risk management, and most Canadian organizations don't have a model for it yet.
Why "Just Add Oversight" Isn't an Answer
The reflexive response to that scenario is "keep a human in the loop." It's true, and it's also close to useless as governance advice, because it doesn't say which loop, at which decision point, with what authority to override.
The U.S. National Institute of Standards and Technology's AI Risk Management Framework is one of the few widely cited references that tries to be specific about this. Its guidance on human-AI interaction argues that human factors specialists — not just engineers — need to be embedded in system design so that AI behaviour stays anchored to user intent and to the norms of the people actually affected by a decision, not just the norms encoded by whoever built the model. The practical translation for a Canadian security or operations leader: oversight is a design discipline, not a checkbox added after deployment. If nobody deliberately designed what the agent is allowed to decide alone versus what it must escalate, the boundary will be set by default — usually by whichever engineer configured the confidence threshold, and usually in the direction of less friction, because friction is what gets measured as a cost.
The National Academies' work on human and organizational factors in AI risk management makes a related point that's easy to skip past: the risk isn't only technical performance, it's the interplay between the AI system and the organization's existing culture. An agent dropped into a firm with weak escalation habits will inherit those weak habits. It won't invent good judgment that the organization didn't already have.
What Accountability Actually Requires
The word "accountability" gets used loosely enough in AI discourse that it's worth being precise about what it means operationally. The Carnegie Council's framing is useful here: accountability means that when an AI system produces a bad outcome, responsibility can actually be assigned to a specific, liable party — not diffused across a vendor contract, an IT department, and a business unit until nobody is answerable. A system that can't produce that assignment isn't accountable, no matter how good its documentation looks.
Turning that principle into practice means an organization needs, at minimum, a defined structure of roles, responsibilities, and controls attached to each agent — something closer to a RACI matrix than a policy statement. Concretely, for the financial services scenario above, that means naming who owns the decision to expand the agent's authority beyond its original scope, who is accountable if a fraud flag freezes a legitimate customer's account, and who has the standing to shut the agent off. If those three names aren't written down somewhere before the agent goes live, they will be improvised after an incident, under pressure, with a customer or a regulator waiting for an answer.
That improvisation is exactly the failure mode research on human factors in AI-enabled systems has already documented in other high-stakes domains. Work on AI-enabled clinical tools — a sector where the consequences of ambiguity are measured in patient outcomes rather than account freezes — has identified recurring, practical risk categories: unclear definitions of who the system is designed for, outputs that don't support the user's actual comprehension or trust, weak integration into existing workflow, insufficient training that leaves staff unable to catch the system's mistakes, no safe fallback when the system fails, and inadequate monitoring once it's in real-world use. None of that research is about banking. But the categories translate directly, because they're describing organizational failure patterns, not clinical ones: a system deployed without a clear boundary of who it serves, how its outputs should be read, what happens when it's wrong, and who is watching it after launch.
The Counterargument Worth Taking Seriously
There's a reasonable objection to all of this: that formal accountability frameworks are premature bureaucracy for technology that is still changing every quarter, and that heavy governance slows down the productivity gains AI agents are supposed to deliver. This isn't a strawman — it reflects a genuine tradeoff. A firm that requires a documented RACI matrix, a full audit trail, and a human sign-off for every expansion of an agent's authority will move slower than a competitor that doesn't. In a market where the copilot-to-autopilot drift is happening precisely because it's more efficient, that speed cost is real.
But the objection assumes the only cost of governance is friction, and the only cost of skipping it is a delay. That's backwards. The cost of skipping it is that when something does go wrong — and the base rate for autonomous systems making occasional bad decisions is not zero — the organization discovers, in real time, during an incident, that nobody actually agreed in advance who answers for it. The friction of governance is scheduled and controllable. The friction of improvised accountability during a live incident is neither.
The honest position isn't "never let agents gain autonomy without a full framework." It's that the framework should scale with the authority being granted, not with the org chart's appetite for paperwork. A chatbot answering FAQ queries doesn't need a RACI matrix. An agent that can freeze accounts or move money does. The mistake most organizations make isn't refusing governance — it's never noticing the moment the agent crossed from one category into the other.
The Judgment Call
The defensible position here is narrower than "govern your AI agents," which is advice nobody disagrees with and nobody acts on. It's this: the trigger for building an accountability structure should not be the agent's job title — copilot, assistant, automation — it should be the specific moment its decisions become difficult or costly to reverse without a human noticing first. That moment is identifiable in advance, before deployment, if someone is willing to ask the boring question: what does this agent get to do that we can't easily undo? Financial transfers, account freezes, access grants, and escalation decisions all fail that test. Routine lookups and draft summaries mostly don't.
For Canadian organizations moving agents from advisory roles into operational ones, that single test — reversibility, not autonomy level — is a more useful governance trigger than any maturity model or capability tier. Autopilot doesn't announce itself. It arrives one small authority expansion at a time, each one individually reasonable, until the aggregate is a system nobody remembers approving. The choice organizations actually face isn't whether to build a risk model. It's whether they build it before that moment or reconstruct it, under scrutiny, after.
What Leaders Should Do Security and technology leaders should treat AI systems and autonomous agents as governed identities, not informal tools. In practical terms, they should:
- Inventory critical ai dependencies.
- Assign an accountable owner for each agent or ai-enabled workflow.
- Document access controls, authentication, logging, approval boundaries, and revocation paths.
- Test fallback options and incident response playbooks.
- Include ai availability, jurisdictional risk, and auditability in procurement.