The model you didn't vet
A Canadian credit union wants to add an AI-powered fraud-detection layer to its transaction monitoring. The vendor demo is convincing: the model catches patterns a rules engine misses, and it plugs into the existing case management system in an afternoon. Procurement runs its usual checklist — data residency, contract terms, SOC 2 report — and signs off.
What that checklist doesn't ask is what's underneath the model. Whose training pipeline built it. Which open-source libraries the vendor's inference stack depends on. Whether the model itself was fine-tuned on a dataset the vendor licensed from a third party, who licensed it from a fourth. Whether an update pushed next quarter changes the model's behaviour on cases nobody re-tested. None of that shows up in a SOC 2 report, because SOC 2 was built for a world where "the vendor" meant a piece of software with a version number, not a model with a training history, a fine-tuning lineage, and a dependency tree that updates itself.
This is the gap. Canadian organizations have spent two decades building supply chain security around code — vendor risk assessments, patch management, software bills of materials. AI systems arrived wearing the same procurement paperwork but carrying a fundamentally different risk profile, and most governance programs haven't caught up.
The number that should reframe the conversation
CISA puts the global cost of software supply chain attacks at $45.8 billion in 2023, projected to exceed $80.6 billion by 2026. That figure covers traditional software — compromised dependencies, poisoned updates, exploited build pipelines. It does not include AI-specific incidents, because the accounting hasn't caught up to the risk. But the mechanism is the same one that produced those numbers: organizations trust a component they didn't build and can't fully inspect, and that trust becomes the attack surface.
The conclusion worth drawing isn't "AI is dangerous." It's narrower and more useful: an AI model is a software dependency with an unusually opaque bill of materials, and the frameworks built to manage software dependency risk — NIST's Cyber Supply Chain Risk Management guidance and the Secure Software Development Framework — apply to it whether or not your organization has updated its vendor risk questionnaire to say so. CISA's own guidance for software customers, not just software builders, already covers procurement, testing, deployment, and patching, and calls out the need for software bills of materials. Nothing in that guidance excludes a model. Most Canadian procurement processes just haven't extended it there yet.
Where the credit union's problem actually lives
Go back to the fraud-detection deployment. Six months in, the vendor pushes a model update to improve detection accuracy. It works — false negatives drop. It also quietly changes which transaction types get flagged for manual review, and nobody at the credit union tested the new version against last quarter's confirmed fraud cases before it went live. Three weeks later, a pattern of legitimate but unusual transactions from a newcomer-heavy branch starts getting auto-flagged at a rate that looks a lot like discriminatory impact, and nobody can say which model version caused it or why, because the change log lives with the vendor, not the customer.
That's not a hypothetical vendor failure. It's what happens when an organization treats a model update the way it treats a software patch — as routine, low-risk, and someone else's job to validate — when in fact a model update can change decision logic in ways a traditional patch rarely does. A code patch usually fixes a defined bug. A model update can shift behaviour across thousands of edge cases simultaneously, in ways the vendor itself may not have fully characterized before release.
This is where the "AI supply chain" framing earns its keep. The credit union didn't fail because it chose a bad vendor. It failed because it applied software-era assumptions — patches are safe, vendors test before release, contracts cover liability — to a component that doesn't behave like software used to.
The counterargument, and where it breaks
The obvious pushback: our models are built in-house, so this doesn't apply to us. That argument has a real point buried in it — an internally built model does remove one layer of opacity, since you own the training data and the fine-tuning decisions. But it doesn't remove the dependency chain underneath. In-house models are still built on open-source libraries, third-party base models, licensed datasets, and cloud infrastructure the organization doesn't control end to end. The 2023 open-source software supply chain attacks that inflated CISA's cost figures didn't discriminate between companies that "built their own" software and companies that bought it off the shelf — a poisoned dependency several layers down affects both. The same is true of an in-house model built on a fine-tuned open-weight base: the vulnerability doesn't care who signed the contract.
The honest version of the counterargument, then, isn't "we're exempt." It's "we have fewer blind spots, but not zero" — which changes the governance posture from optional to proportional, not from mandatory to unnecessary.
What Canadian governance frameworks actually require, once you look
Canada's federal government has already had to work through a version of this problem for its own automated decision-making systems, under the Directive on Automated Decision-Making and its list of qualified AI suppliers. That framework requires agencies to document what a system does, assess its risk tier, and maintain accountability for outcomes — not just for the initial deployment, but on an ongoing basis as the system is used. Private-sector organizations don't have to comply with that directive, but its underlying logic transfers cleanly: risk assessment is not a one-time procurement gate, it's a standing obligation that has to survive vendor updates, model retraining, and staff turnover.
Vendor governance platforms — OneTrust, Databricks, Rubrik, and others building AI governance tooling — have converged on a similar structural answer, independent of Canadian regulation: inventory every AI use case, rank it by risk, and apply proportionally tighter controls to the high-risk tier rather than treating every deployment identically. That's sound practice, though it's worth naming plainly that this convergence also serves these vendors' commercial interest in selling governance platforms. The advice holds up on its own merits regardless of who's giving it, but a Canadian security leader should treat any single vendor's framework as a starting checklist, not a finished governance program.
The gap between knowing this and acting on it
Here is the uncomfortable part. Most of what fixes this problem isn't new technology — it's extending vendor risk management practices Canadian organizations already have to a category of vendor they haven't been applying them to. Ask for a model's training data provenance the way you'd ask for a software bill of materials. Require advance notice and a re-validation window before model updates go live in production, not after. Assign a named owner for each AI dependency who is accountable for testing changes, not just approving the initial contract. Build a fallback path for when a model update misbehaves or a vendor's model becomes unavailable, and actually test that fallback before you need it.
None of this is exotic. It's the same discipline Canadian organizations already apply, unevenly, to traditional software vendors — extended to a category of dependency that changes its own behaviour more often, more invisibly, and with less warning than a code patch ever did. The organizations that will get burned aren't the ones using risky AI. They're the ones that assumed their existing vendor risk process already covered it, and never checked.
What Leaders Should Do Security and technology leaders should treat AI systems and autonomous agents as governed identities, not informal tools. In practical terms, they should:
- Inventory critical ai dependencies.
- Assign an accountable owner for each agent or ai-enabled workflow.
- Document access controls, authentication, logging, approval boundaries, and revocation paths.
- Test fallback options and incident response playbooks.
- Include ai availability, jurisdictional risk, and auditability in procurement.