Autonomy Without Modularity Is Just Risk at Scale

At NEARCON this week, we spent a lot of time talking about autonomous systems in practice. Adoption curves. Enterprise readiness. Governance. Observability. Good conversations, all of them.

But there was one principle I didn't get enough time to emphasize on stage, and I think it's the most important architectural idea of the agent era: modularity. And more specifically, how autonomy at scale without modularity doesn't reduce risk; it amplifies it.

Most agent systems today are built as blobs.

The same system handles planning, tool use, execution, memory, spending, and permissions. That architecture works fine in a demo, where stakes are relatively low. But even in low-stakes environments, when something goes wrong, everything is entangled and the costs can surprise you.

We've already seen the early warning signs.

A multi-agent system looped for 11 days and burned roughly $47,000 in API costs. No hack. No breach. Just two agents repeatedly asking each other for clarification, both convinced they were making progress. That was a control-layer failure, not an intelligence failure, and it came with a very expensive bill.

We've seen attackers slip malicious instructions into an agent's input, and because the agent had broader tool access than it needed, it ended up leaking data it had no business touching. We've also seen denial-of-wallet style incidents, where recursive calls rack up inference spend because the budget logic lived inside the same execution loop that was already malfunctioning.

In every one of these cases, the underlying problem was the same: the agentic architecture didn't separate power from control.

When the planner misfires, it can still execute. When execution loops, it can still spend. When memory is polluted, it still influences decisions. When budget controls live inside the same logic path as the agent, a bug can effectively disable its own brakes.

That's fragility dressed up as autonomy.

Autonomy means delegation, and delegation at scale requires separation of powers.

In any well-run company, strategy doesn't approve its own expenses. Engineering doesn't audit its own compliance. Finance doesn't live inside the product org. These separations exist for good reasons that took decades of hard lessons to learn.

Yet many agent systems are architected exactly that way.

If we're serious about enterprise autonomy, the stack has to be modular. At minimum, that means clearly separated layers: a planner that defines intent, a router that selects tools, executors that perform actions, memory that is bounded and inspectable, a budget and policy layer that enforces limits, and a kill switch capable of independent revocation.

The critical constraint is that the budget controls and the kill switch cannot live inside the same execution path as the agent. If the agent breaks, the brakes have to still work.

We are entering the agent swarm era, and the math changes fast.

Enterprises won't deploy one agent. They'll deploy hundreds.

Procurement agents negotiating vendor contracts. Finance agents reconciling invoices. Compliance agents reviewing documentation. Monitoring agents responding to operational alerts. When autonomy scales horizontally, architectural flaws scale exponentially. A single runaway agent is a bug but a runaway swarm is a systemic event.

The incidents we've seen so far are small previews. They happened in experimental systems with limited scope. Enterprise deployment multiplies the impact of every one of those failure modes.

Capability is advancing quickly. Control infrastructure is not. That gap is where most enterprise deployments will struggle.

This is exactly why I'm proud of what we're building at ampersend.

Not because it's flashy, but because it focuses on the control layer: portable policy enforcement, enforceable budget limits, real-time revocation, all sitting outside the intelligence layer. It's the infrastructure piece most people skip.

And as a mom, building this work hits differently for me. We are shaping systems that will operate at scale in the world our kids inherit. Autonomy is not going back in the box. If we don't design modular, enforceable control layers now, we'll be retrofitting safety at highway speed.

The future of autonomy won't be defined by who built the smartest agent. It will be defined by who built the safest architecture around it.