Multi-agent systems: when they help, and when one good agent wins

At 11:47 p.m., I was at my kitchen table watching an AI workflow retry the same failed task through three different model calls. The system had four agents, a router, a reviewer, and enough logs to make the whole thing look impressive. It still produced the wrong answer.

Multi-agent systems help when work can be split into genuinely separate jobs with clear handoffs. They hurt when you stack agents to hide a weak prompt, poor data, or an unresolved product decision. In many production cases, one well-scoped AI agent with strong tools, tests, and permissions will beat a small team of agents on cost, speed, and reliability.

I've built two live multi-model AI platforms and shipped large technical programs across Verizon, HP, and InterContinental Hotels. The pattern is familiar: people reach for more moving parts before they've proved that one part works.

The default mistake is adding agents too early

The industry repeats a simple idea: difficult work needs a team of AI agents. That sounds reasonable because difficult human work often needs several specialists. But an AI agent is not a human employee with judgment, context, and a stable sense of responsibility. It is a model operating inside a loop of instructions, tools, memory, and constraints.

Adding another agent adds another place for the system to lose information. A planner produces a vague task. A researcher pulls the wrong document. A writer trusts the researcher without checking the source. A reviewer approves the answer because its own context is incomplete. The final output looks polished while the original error survives every handoff.

Anthropic made a useful distinction in its December 2024 guide, "Building effective agents." It separated workflows, where code controls a fixed sequence, from agents, where the model decides how to use tools and proceed. That distinction matters more than the label multi-agent systems. A fixed workflow is often easier to test than a cluster of agents negotiating their next move.

So the deeper lesson is simple: use more agents only when the work has real separation. If one model can inspect the request, call a tool, check its result, and answer within a bounded loop, another agent may just add latency.

A single agent usually wins when:

• The task has one main goal and one owner.
• The available tools share the same context.
• The answer needs a short chain of actions.
• You can judge success with a clear test.
• A human can review the result before it causes harm.

That is not an argument against agent orchestration. It is an argument for earning it.

What one good agent can handle

A good single agent is not a chatbot with a system prompt. It has a defined job, a limited tool set, useful context, and a way to verify what it did. The agent might answer a billing question, look up a hotel reservation, classify an inbound lead, or prepare a draft from approved records.

Consider a support agent handling a refund request on a Saturday night with no human in the loop. It needs access to the order record, the refund policy, and perhaps a payment tool. It should verify the order number, check eligibility, calculate the amount, and either issue the refund or route the case to a person. That is a sequence, but it is still one job with one decision boundary.

A second agent does not automatically improve this flow. The system might ask a policy agent to interpret the refund rules, a customer agent to read the request, a finance agent to calculate the amount, and a supervisor agent to approve it. Now you're managing four prompts, four contexts, four tool permissions, and several handoffs. The extra structure is worthwhile only if those boundaries solve a real problem.

When I build an AI product, I start with a single-agent baseline. I measure:

• Task completion rate on a fixed test set.
• Correct tool selection and tool arguments.
• Time to a final answer.
• Cost per completed task.
• Rate of escalation to a human.

This baseline gives the team something to beat. Without it, a multi-agent demo can win by looking busy.

One agent also makes diagnosis less painful. If a result is wrong, you can inspect the request, retrieved context, tool call, model response, and final decision in one trace. That matters when you are handling customer data or making a decision that affects money.

The mistake I keep watching teams make is treating a longer chain as deeper reasoning. Sometimes it is. Often it is just a longer chain.

When multiple agents actually help

Multi-agent systems make sense when the work has independent domains, separate permissions, or tasks that can run at the same time. The division must be visible in the business process, not invented inside a diagram.

Take a procurement review for a 500-room hotel in Chicago. A request for new kitchen equipment may require a contract search, a price comparison, a safety check, and a budget calculation. Those jobs use different records and may belong to different teams. A contract agent searches approved agreements. A pricing agent compares vendor quotes. A compliance agent flags missing certifications. A finance agent calculates the effect on the department budget.

That design has a reason to exist. Each agent has a narrower role, a smaller context, and a different permission boundary. The outputs combine into a review packet for a human buyer. No agent needs to pretend it understands the entire procurement process.

Parallel execution is another valid reason. Imagine an incident response system checking a service health dashboard, recent deploys, database metrics, and customer reports at the same time. Separate workers gather evidence in parallel while a coordinator assembles the timeline. If the system waits for each check in sequence, the response takes longer for no gain.

Multiple agents can also help when the system needs adversarial review. One agent drafts a proposed answer. Another checks citations against source documents. A third tests whether the answer followed policy. This is useful only when the reviewers have different instructions and access to independent evidence. Asking the same model to critique its own output in a new wrapper is not independent review.

A practical test: remove one agent and ask what breaks. If the answer is "the diagram becomes less impressive," remove it. If removing it eliminates a distinct permission boundary, parallel task, or specialist evaluation, the role may be justified.

Agent orchestration is a control problem

Agent orchestration is often described as routing messages between models. In production, it is closer to running a control system. You decide which work can happen, which tools each worker may use, what evidence must travel between steps, and what happens when a worker fails.

Frameworks like LangGraph, Microsoft AutoGen, CrewAI, and the OpenAI Agents SDK can help you express these flows. None of them removes the design responsibility. A framework can make a bad architecture easier to deploy, which is a mixed blessing (I have built a few of those by accident).

I define a control surface for every agent:

• Allowed tools and exact argument types.
• Data the agent may read.
• Actions that require approval.
• Maximum turns, retries, and spend.
• Conditions for stopping or escalating.
• Evidence required in the output.

The handoff contract matters just as much. Do not pass a paragraph called "research." Pass a structured object with claims, source locations, confidence, unresolved questions, and timestamps. A downstream agent should not have to guess which sentence is a fact and which sentence is a suggestion.

This is where a Handoff Contract earns its place. It is a machine-readable agreement between agents about what enters and leaves a step. Once the contract is explicit, you can test it without running the entire system.

Use durable state when the work can pause or fail. LangGraph, for example, supports graph-based state and checkpoints, which helps a workflow resume after interruption. That does not make the workflow correct. It means you can recover without pretending the failed step never happened.

Build an Orchestration Budget for every request. Track model calls, wall-clock time, tokens, retries, tool calls, and human review. A system that saves three minutes of analyst time but burns 20 model calls to do it may be a poor product even when the answer is correct.

What I'm telling you is that orchestration is not the number of agents. It is the set of rules that keeps work bounded and observable.

Measure the system, not the demo

A multi-agent demo usually shows the happy path. Production begins when a tool times out, a document is stale, a model returns malformed JSON, or a user changes the request halfway through the process.

For every agent role, create tests before tuning the prose. A research worker should be tested on source selection and citation accuracy. A routing worker should be tested on ambiguous requests. A reviewer should be tested on known failures, not just clean outputs.

Your evaluation set should include cases like:

• A refund request with two matching orders.
• A contract clause that conflicts with a newer policy.
• A customer asking for an action the agent is not allowed to perform.
• A tool returning an empty result or a 500 error.
• Two agents producing conflicting facts.

Use traces to find the failure location. OpenTelemetry can carry spans across model calls and tools. LangSmith and similar platforms help inspect agent runs. Your own Postgres tables can work too. The tool matters less than recording the full path.

Watch the failure modes that compound as you add agents. If each handoff has a 95% chance of preserving the required information, six handoffs preserve the full chain only about 74% of the time. That multiplication is not a complete reliability model, but it shows why a little uncertainty becomes a large operational problem fast.

Cost grows the same way. A single user request may trigger a planner, three workers, a merger, and a reviewer. If each step retries twice, the bill and the latency can expand without any change in user value. Instrument retries separately from normal calls. A retry is not free recovery, it is another chance to produce a wrong result.

Test for disagreement, too. If two agents disagree, the system should expose the conflict or resolve it with evidence. Quietly selecting the last response is not orchestration. It is a coin toss with extra software.

The Eval Loop should run against the single-agent baseline and the multi-agent design. Keep the more complex system only when it improves a measured outcome: accuracy, completion rate, review time, or policy compliance.

How hiring leaders should evaluate AI agent designs

When I interview an applied-AI leader, I do not start by asking whether they have built a multi-agent system. I ask what the system was responsible for, where it failed, and why the team chose more than one agent.

A strong candidate can draw the boundary between model behavior and software behavior. They can say which decisions belong in code, which belong to the model, and which must stay with a person. They can explain how permissions work when an agent delegates to another agent.

Ask for a concrete production scenario. "Design an agent system for a support operation in Dallas at midnight, with an unavailable billing API." Then listen for the fallback path. A serious design will mention cached policy, idempotent actions, escalation, audit records, and a limit on retries. A weak design will mention a manager agent.

Useful interview questions include:

• What does one agent do well enough that you would not split it?
• What exact failure requires a second agent?
• Which data crosses the handoff, and in what format?
• How do you stop a loop between agents?
• What action needs human approval?
• How do you compare this design with a single-agent baseline?
• What metric would make you remove an agent?

Look for operational evidence. The candidate should be comfortable discussing traces, tool errors, test sets, spend per task, and escalation rates. They should know that model choice is only one part of the system.

Here's the part most people miss: every agent needs an owner at its boundary. My experience across long technical programs taught me to distrust diagrams where no one is accountable at the seams. The same rule applies here. Every agent needs a job, a budget, a permission set, an exit condition, and someone who will investigate its failures.

The strongest applied-AI leaders are not the people who add agents fastest. They are the people who can remove three agents and explain why the system is now safer.

A practical decision process

Start with the user request, not the architecture. Write down the result the user needs and the action the system may take. If you cannot state those plainly, you are not ready to select an agent pattern.

Then map the work as a sequence of decisions and tools. Mark places where the system needs a separate permission, an independent source, or parallel execution. Those marks are candidates for agent boundaries, not automatic requirements.

Run a single-agent version with strict tools and a small evaluation set. Give it enough time to show its real limits. If it fails because the context is too broad, try better retrieval or a narrower prompt before adding another worker.

Split the system only when you can name the benefit. Faster parallel research, isolation of sensitive data, independent review, a specialist tool interface. Write that benefit beside the agent in the design document.

Then set hard limits:

• Maximum total model calls per request.
• Maximum time before escalation.
• Maximum spend per completed task.
• Maximum retries for each tool.
• Maximum delegation depth.

Deploy with traces and a rollback path. Review failures weekly. Remove agents that do not improve the measured result.

For a hiring team, this process is also a practical work sample. Give candidates a real workflow and ask them to produce two designs: one agent and several agents. The difference between those designs will tell you more than a list of framework names.

One good agent is not the cheap version of a multi-agent system. It is often the better architecture. Use multiple agents when the work has real boundaries, then make those boundaries visible in permissions, contracts, tests, and costs.

The model is not the product. The control system around the model is the product. So your next move is this: take one live AI workflow, remove every agent you cannot justify, and measure what remains before you add anything back.

FAQ

When should I use a multi-agent system?

Use one when the work has separate domains, permissions, independent evidence, or tasks that can run in parallel. If the system has one goal and one shared context, start with one agent and a bounded tool loop.

Are multi-agent systems more accurate than single-agent systems?

Not by default. They can improve accuracy when agents perform genuinely independent research or review, but every handoff adds another failure point. Compare the design against a single-agent baseline with the same test cases.

Which framework is best for agent orchestration?

There is no universal winner. LangGraph is useful for explicit stateful graphs, AutoGen and CrewAI support multi-agent patterns, and the OpenAI Agents SDK provides tools for agent-based applications. Choose based on tracing, state management, tool control, and team familiarity rather than framework popularity.

How do I control the cost of multi-agent systems?

Set a maximum model-call count, retry limit, delegation depth, and spend per task. Record tokens, model names, tool calls, and retries for every run. Then remove any agent that does not improve accuracy, completion rate, response time, or policy compliance.

What should I ask an AI product leader about multi-agent systems?

Ask why the system needs more than one agent, what crosses each handoff, how failures are tested, and what would cause the team to remove an agent. Strong candidates will discuss permissions, structured outputs, evaluation data, human escalation, and production traces.