So… who actually signed off on this??

Scroll on linkedin for five minutes and you would think every engineering team on earth has handed the keys to a swarm of agents and walked away.

And somewhere in the middle of all of that, someone comes in and clicks tab+enter.

So the question remains: who exactly is looking at this?

Not “looking at it” in the corporate governance sense. Not in the “we have a human-in-the-loop” way every AI company says on a landing page. I mean actually looking. Who understands what changed?

I catch myself feeling great because a green check showed up and it told me that all 4/4 merge checks pass. But what exactly happened? is the part I keep getting stuck on.

I have been spoiled, I use these agents as a part of my normal workflow now. I think agents are going to become a normal part of how software gets built. 

But the more work I hand off, the more important the review layer becomes. And right now, I am not convinced that layer is keeping up.

It is not just me either. The industry seems to be arriving at the same anxiety, just with a more polished word for it: governance

“Governance” feels like a word from the corporate slop dictionary. I hear it and immediately picture massive checklists, lawyers doing risk assessments, compliance dashboards, and boring approval meetings where everyone pretends the process itself equals safety.

But the idea underneath is about answering one simple question:

What is this system allowed to do, and what has to be true before we trust it

That is it. So why should I care?

If my whole goal as an engineer is to ship as quickly as possible, why would I want another layer of process slowing me down?

Because AI has changed what “shipping fast” even means.

Models are getting smarter, and the pace at which I can build now is blazing fast. These tools are no longer just autocomplete or a chatbot sitting off to the side. They are inside the workflow. They help write code, generate tests, explain bugs, summarize diffs, suggest architecture, and move through unfamiliar systems. Which is great.

But it also means the model is no longer just producing text. It is operating inside a system of permissions. That is why governance matters. Not because I want to slow engineering down.

Because the faster I can ship, the more important it becomes to know what I am trusting, what the system was allowed to do, and where the actual human judgment happened.

That is what makes agentic development different from normal AI-assisted development. The risk is not just that the model might hallucinate. The risk is that it might act.

A lot of AI risk comes from giving a probabilistic system too much agency too early.

Imagine letting gradient descent decide who moves on to the next round of hiring. Or who gets offered what price on a website. Or which customer support answer becomes the official answer. Or which code change is safe enough to ship.

That sounds awful, right? It is.

This is why governance cannot just ask, “Is the model good?”

If an agent can read everything, edit everything, call everything, and retry actions without approval, then you have not built an assistant. You have built an unbounded actor inside your engineering environment.

That might sound dramatic, but it is really just security architecture.

Agents need identities. They need scoped access. They need least privilege. They need approval boundaries. They need logs. They need owners. They need expiration. They need a way to be shut off. But a lot of AI adoption is happening through tools that are optimized for speed first and governance second. The default product story is, “Look how much faster you can move.” The safety story is often, “Trust!” The phrase “human-in-the-loop” gets thrown around so much that it is starting to lose meaning.

A human who clicks approve on something they do not understand is not oversight. That is simply ceremony. Everyone participated. But no one may have actually understood. That is the failure mode I worry about.

A lot of AI safety conversations focus on the model, the model aligned? Does it hallucinate? Is it biased? Can it be jailbroken? Is it safe enough for deployment?

Those questions matter. But for most companies building AI into products, the safety problem is bigger than the model.A model can stay the same while the system becomes much riskier. 

A support chatbot can become a liability if users rely on the wrong answer. A recruiting model can become discriminatory if it learns from biased historical patterns or proxy variables. The risk lives in the workflow.

So the safety layer has to govern the workflow too. This is where I think the current market still feels early.

Companies can buy pieces of AI governance. Technically, governance happened. But what exactly happened in the workflow is a fragmented question no one can answer. If governance only happens once, it is not governance. It is a launch ritual.

The obvious warning sign

The Air Canada chatbot case is such a clean example because it cuts through all the abstraction. The chatbot gave a customer wrong information about a bereavement fare. The customer relied on it. Air Canada tried to argue that the chatbot was separate from the company’s responsibility. That argument did not work. And honestly, product teams should not need a tribunal to understand this.

If the AI is in the product, its output is product behavior.

You cannot separate the chatbot from the company. You cannot separate the agent from the workflow. You cannot separate the generated recommendation from the decision system around it.The same logic applies to engineering agents.

The incident review will not end with, “but the bot said it looked fine.” Responsibility does not disappear because the work was generated. The goal is not to make AI development slow. The goal is to make trust proportional to risk.Vibes are not going to be good enough when these systems start touching real users, real data, real infrastructure, and real decisions.