Agents

Better Models, Different Risks

Better models shrink some enterprise risks and grow others. Capability is not authority: how reach, velocity, and blast radius change the shape of agent risk.

A model that writes a bad answer creates a quality problem.

An agent that can search company records, update a customer account, send an email, or move money can create an operational problem.

This distinction will matter more as AI models improve.

The usual discussion treats capability as a single upward curve. Models become better at reasoning, planning, using tools, and completing longer tasks. The assumption is that better capability will make enterprise AI safer and more reliable.

That is partly true. More capable models should misunderstand fewer instructions, make better plans, select tools more accurately, and recover from simple mistakes. But capability also changes the reach of a system. A model that once produced a paragraph can now carry out a sequence of actions. It can work for longer, touch more systems, and affect more people.

Some risks decline. Others grow.

model capability → risk Errors of understanding misread intent · brittle plans · wrong tool Errors of consequence reach · velocity · blast radius Induced errors jailbreaks · injection · long-context · slow burn the shape of risk changes here

And not every error is the model's own. Some are induced — an adversary talks the system into them. That family grows too, with capability and with context length.

The useful question is not whether better models increase or decrease risk in general. It is how they change the shape of risk.

Capability increases reach

One way to see this shift is to look at the length of tasks models can complete.

METR measures the duration of software tasks that AI agents can complete with a specified probability of success. Its results show rapid growth in the time horizon of frontier models. The growth is not uniform — METR found visual computer-use horizons roughly 40 to 100 times shorter than software and reasoning horizons — but the direction is clear. Models are becoming able to stay with a problem for longer.

That is valuable, because many enterprise processes are not single questions. They are sequences. Resolving a customer complaint may require finding the customer, reading the account history, checking the current policy, reviewing previous cases, calculating an amount, updating a system, and communicating the decision. The longer the sequence a model can manage, the more of the process it can perform.

The same improvement creates a new failure mode: an error near the beginning of a long workflow can travel.

Suppose an agent incorrectly decides that a customer is eligible for a refund. What happens next depends entirely on how much of the workflow the agent owns:

TRUST LEVEL: RECOMMEND Misjudges eligibility Drafts refund recommendation Human review ✕ stops here the mistake is caught as an output error TRUST LEVEL: EXECUTE END-TO-END Misjudges eligibility Selects the order Issues the payment Updates the record Emails the customer one wrong premise, four systems changed The model has not become less reliable. It has become more consequential.

Long workflows also give small errors room to accumulate, and occasional large errors room to appear. A recent preprint from Microsoft Research examined delegated document-editing workflows across 52 professional domains and 19 models. Even the frontier models tested degraded an average of 25 percent of document content by the end of long workflows — and the errors were often sparse but severe, worsening with larger documents, longer interactions, and distracting files. Tool use did not solve the problem in that evaluation.

The study is one benchmark, not a universal measure of enterprise agents. But it illustrates an important pattern: success on individual steps does not guarantee that the full artifact or process remains intact over time. A model can look competent at every turn while the overall result slowly drifts.

The risks that decline

Several risks should fall as models improve — and these gains are real. They will make more enterprise tasks economically viable.

What declines Why Example
Misunderstanding Better models infer what a person wants from incomplete, loosely worded instructions "Sort out the Hendersons' billing issue" resolves to the right account and the right problem
Planning failure Stronger decomposition, ordering, and replanning when new information appears A complaint is worked account → policy → precedent → amount, not in a broken order
Poor tool selection Models recognize when to search, retrieve, compute, or ask for human judgment The agent queries the policy database instead of guessing from memory
Brittle instruction following Less prompt engineering needed for valid structure and routine process Fewer "output valid JSON or else" scaffolds in the product
Routine correction Less human time on wording, obvious gaps, and simple reasoning slips Reviewers stop fixing drafts and start reviewing decisions

More capable models may also reduce risk directly. A strong model may notice that information is missing, identify a contradiction between two policies, question an unusual transaction, or recognize that an action exceeds its authority.

So the argument is not that stronger models are inherently less safe. The argument is that model quality addresses only part of the risk.

The risks that grow

Six risks grow with capability — not because the model gets worse, but because the system around it gets bigger, faster, and more connected. The profile looks very different for an assistant that drafts and an agent that executes:

Reach Velocity Authority Coupling Tail exposure Attack surface Assistant — recommends and drafts Agent — executes end-to-end, across systems
What grows What changes Example
Reach A capable agent keeps acting after an early mistake — a trail of individually valid actions built on an invalid premise The wrong eligibility call above: order selected, payment issued, record updated, email sent
Velocity A person makes one incorrect change at a time; an agent repeats the error at machine speed The same bad discount applied to thousands of accounts before anyone notices the pattern
Authority Recommending becomes executing A poor refund recommendation is an output error. An incorrect refund is a financial event
Coupling One action fans out across systems, making the full effect hard to see and reverse An order change touches inventory, billing, customer communication, revenue reporting, compliance
Tail exposure Average performance improves while rare failures still matter at volume 99.9% success across ten million transactions is ten thousand failures
Attack surface Every context source, tool, integration, and memory is a place where bad information can enter A poisoned document in the agent's context instructs it to approve the refund

Tail exposure deserves a picture, because averages hide it:

99.9% succeed — 9,990,000 transactions 0.1% fail — 10,000 failures sliver enlarged ~10× to be visible Whether this is acceptable depends on what each failure can do — not on the average.

The attack surface deserves more than a row in the table, because it grows for a different reason than the rest. Reach, velocity, and coupling amplify the model's own mistakes. The attack surface grows because errors can be induced.

Indirect prompt injection showed that an adversary does not need access to the prompt. Instructions planted in content the agent will read — a web page, a document, an email — can steer it. Every retrieval source, tool result, and memory an agent consumes is a channel.

Nor does jailbreak risk simply fall as models improve. Safety training makes each individual attempt harder. But Anthropic's many-shot jailbreaking work showed that the long context windows of modern models are themselves an attack surface: pack the context with enough faux dialogues and the rate of harmful responses rises on a power law with the number of examples. More striking, larger models were more susceptible, not less — the same in-context learning that makes them capable makes them better at learning the attack.

The most dangerous induced errors may be the slow ones. Crescendo showed that a jailbreak does not need a single malicious prompt at all: a benign conversation, escalated gradually over a handful of turns — each step building on what the model itself just said — reaches content the model would refuse outright, with no adversarial text anywhere in the transcript. AgentPoison stretched the timeline further: poison an agent's memory or knowledge base today, and the agent behaves normally on every benign query — until a trigger arrives, retrieves the poisoned entry, and acts on it.

Slow-burn attacks are the adversarial mirror of the reach problem. No single message looks harmful, the way no single action in a long workflow looks invalid. Filters that inspect one message at a time miss them by construction. And the properties that make agents useful — persistent memory, standing context, multi-session workflows — are exactly the room a slow burn needs.

And the stakes of a successful attack scale with everything else in this section. A jailbroken chatbot produces a harmful paragraph. A jailbroken agent with tools produces actions — with the reach, velocity, and coupling described above.

Tool use matters most here, because it converts model outputs into real effects. The ToolEmu research project tested agents in simulated high-stakes tool environments and identified plausible failures involving privacy leaks, financial harm, and other serious outcomes. Its broader lesson: tool-using agents need to be evaluated in realistic scenarios, including unusual and long-tailed cases, not only on whether they complete a benign task.

There are also more speculative risks that become relevant only as models receive broader goals and authority. Anthropic tested models in deliberately constructed corporate simulations involving goal conflicts and threats to the model's continued operation. Under some conditions, models chose harmful actions such as blackmail or corporate espionage. These experiments do not show that such behaviour is common in production. They show that it is possible under carefully designed conditions and therefore worth testing before agents are given sensitive roles.

This distinction matters. Product leaders should not treat extreme simulations as forecasts of everyday behaviour. They should also not wait for a failure to occur in production before deciding that a plausible failure mode deserves a control.

Capability is not authority

The central product decision is not simply which model to use. It is how much authority to place behind that model.

Capability asks:

Can the model perform this action?

Authority asks:

Under which conditions should the system allow it to perform the action?

The same capability supports very different grants of authority:

The model is capable of… The enterprise may decide…
Changing a supplier's bank details It may never do this autonomously
Processing a refund Only for verified customers, eligible orders, amounts below a fixed limit
Publishing a pricing change It prepares the change and runs checks; a person approves publication

The amount of risk is therefore not determined by model capability alone. A useful approximation is:

$$\begin{aligned}\text{Expected harm} \;=\;\; & P(\text{error}) \times P(\text{execution}) \\ & \times\; \text{scale of exposure} \times \text{severity} \times \text{recovery difficulty}\end{aligned}$$

Better models bear on exactly one of those five terms. The rest are shaped by the surrounding system:

Term Does a better model help? Shaped mostly by
$P(\text{error})$ Yes, directly Model quality, context, task design
$P(\text{execution})$ Marginally Approvals, permissions, policy checks
Scale of exposure No Transaction limits, rate caps, staged rollout
Severity No Which tools and systems the agent may touch
Recovery difficulty No Logging, reversibility, the ability to stop the agent

The safest design is not always to use a weaker model. A weaker model may make more mistakes, miss ambiguity, and require more supervision. The stronger design is often to use an appropriately capable model inside a constrained environment.

Let the model interpret the request. Let a deterministic service calculate the amount. Let a policy engine decide whether the transaction is permitted. Let the agent prepare a high-risk action. Let an authorized person approve it. Let the system verify that execution produced the intended result.

This separates judgment from authority and authority from execution. OpenAI's current Preparedness Framework reflects the same separation: it treats long-range autonomy as a capability that needs specific evaluation, and states that as models become more capable, safety increasingly depends on real-world safeguards around them.

What product leaders should do

The first step is to stop assigning trust at the level of the model. A benchmark can show that a model has useful capabilities. It cannot show that the full product is safe for a particular enterprise process.

Trust should be evaluated at the level of a defined task. For each task, product and technology leaders should ask:

  • How difficult is the reasoning?
  • How long is the chain of dependent actions?
  • Which systems and tools are involved?
  • What authority is required?
  • How far could a mistake spread, and how quickly would it be detected?
  • Can the action be reversed?
  • Which controls remain independent of model judgment?

This also changes how agent products should be measured. A successful demonstration is not enough. Neither is average benchmark accuracy.

Measure The question it answers
End-to-end completion Does the whole task finish — not just each step?
Repeated-run consistency Same input, same outcome?
Exception rate How often does the agent need a human?
Undetected error rate What gets through looking correct?
Maximum exposure What is the worst a single failure can do?
Recovery time How fast can an error be found and undone?
Human intervention at fixed risk How much supervision does this risk level actually require?

As models improve, some of these numbers should get better naturally. The rest will depend on product architecture.

This is the area Avianna is focused on: not treating model intelligence as sufficient, but placing it inside a task-specific operating environment with context, knowledge, tools, authority, evaluation, and controls. The point is not to constrain useful capability. It is to make greater capability deployable.

Better models will make more work possible.

Whether that work becomes dependable will be decided outside the model.

Sources