Building a Multi-Agent Orchestrator: 12 Specialists, 1 Leader

The first time I asked a single AI to "research this topic, write a report, generate an illustration, and review the result," I got a passable draft that did all four jobs badly. It hallucinated sources, the prose was generic, the image was off-topic, and the self-review was a polite "looks good to me."

That was the day I stopped believing in a one-prompt-fits-all model. A real team does not work that way. A real team has specialists.

The Orchestrator pattern is the simplest way to give an AI the structure of a small team: one leader that talks to the user, and a roster of specialists the leader can call when a job matches their skill.

What the Orchestrator Actually Does

The Orchestrator is the agent that sits in front of the user. I never chat with specialists directly. I send a message to the Orchestrator, and it does six things:

1. Analyzes the task
2. Breaks it into subtasks
3. Picks the right specialist for each subtask
4. Delegates with clear, scoped instructions
5. Validates the result from each specialist
6. Composes a single, coherent final answer

The hard part is step 3. Most failures I have seen in multi-agent systems come from a smart leader making bad delegation calls. The fix is to keep the specialist roster small and the responsibilities narrow.

The Workflow at a Glance

User Task
   ↓
Orchestrator (Leader)
   ↓
┌──────────────┬──────────────┬──────────────┐
│  Researcher  │  Coder/Ops   │  Writer      │
│  (cari info) │  (eksekusi)  │  (susun)     │
└──────────────┴──────────────┴──────────────┘
   ↓
Final Report + Memory Store

The user sees one message in, one message out. The leader's prompt is the only place where multi-agent logic lives. Every specialist is a dumb expert with a narrow job.

The Specialist Roster

I keep twelve specialists running. Twelve is the magic number for me: small enough to maintain discipline, large enough to cover the kinds of work I actually do.

Analysis and reasoning. reasoning-agent handles deep analysis, complex debugging, and system planning. It uses Claude for the heavy thinking. predictor-hand does predictions, forecasting, and trend analysis.

Research and data. researcher-hand is my workhorse for information gathering, web search, and source validation. collector-hand aggregates and monitors data over time. clip-hand does image analysis when I need to extract meaning from a picture.

Development and technical. lead-hand coordinates project work. trader-hand runs trading algorithms and financial analysis. browser-hand automates the browser for scraping and testing. infisical-sync-hand keeps secrets and config in sync.

Creative and content. image-generator produces visual assets, illustrations, and UI mockups. twitter-hand drafts social posts.

Business and planning. career-orchestrator handles long-horizon career planning.

Main coordination. The orchestrator itself, which is also the primary Telegram interface. It runs the kimi-k2.6 model and is the only agent that should ever talk to a human.

Total: twelve essential agents, all running, no redundant variants, no crashed zombie agents from earlier experiments.

Telegram as the Front Door

The Orchestrator doubles as the primary Telegram interface. I have a bot called @<your-bot-username> that drops every incoming message into the Orchestrator, waits for the final answer, and sends it back formatted. Auto-delegate is enabled, so I never have to tell the bot "use the researcher for this part." The Orchestrator figures it out.

The workflow is simple:

User sends message to @<your-bot-username>
   ↓
Orchestrator (Primary Interface)
   ↓
Delegates to specialist agents
   ↓
Synthesizes final response
   ↓
Sends structured answer via Telegram

This means I can trigger a multi-agent pipeline from my phone, on a bus, with the same effort as sending a chat. The interface cost is one message.

How to Use It

There are three ways to talk to the Orchestrator.

Via SSH, for serious work where I want to see the streaming output:

ssh -i "<your-ssh-key>" root@<your-vps-ip>
/root/.openfang/bin/openfang-wrapper chat orchestrator

Via HTTP API, for scripts and integrations:

curl -X POST http://127.0.0.1:4200/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/kimi-k2.6",
    "messages": [{"role": "user", "content": "Your task here"}],
    "max_tokens": 2000
  }'

Via Telegram, for everything else. This is what I use 80% of the time.

What a Real Conversation Looks Like

When I send the Orchestrator the task "build a sales dashboard," the response traces through the full delegation pattern. It first writes an ## Understanding block summarizing what it thinks I want, then a ## Plan block listing the delegation sequence, then it actually executes the delegations and reports each result, and finally a ## Final Answer block.

For a dashboard task, the plan looks like this:

1. researcher-hand finds sales dashboard examples and identifies data formats
2. analyst examines the data structure and picks five key metrics
3. coder builds the dashboard with React and Chart.js
4. image-generator produces icons and visual elements

The user gets a single coherent answer, not four disjoint chat messages.

For "write an article about AI agents," the plan is similar but uses writer instead of coder, and adds a reasoning-agent review pass at the end.

Best Practices That Saved Me Pain

Do give clear, specific tasks. Constraints like timeline, budget, or technology stack dramatically improve delegation quality.

Do trust the Orchestrator to pick the specialist. The moment I started second-guessing its delegation choices, my throughput dropped. It is better at routing than I am.

Do review the final answer. Multi-agent systems are not magic. The Orchestrator is good, but it can still pick the wrong specialist or feed it bad instructions. A two-minute human review catches most problems.

Do not give vague tasks. "Make something cool" is a recipe for the Orchestrator to thrash between specialists and produce a Frankenstein output.

Do not interfere with the delegation process. If you want to skip a specialist, say so explicitly. Otherwise let it run.

Do not expect one agent to do everything. That is the architectural anti-pattern we are explicitly leaving behind.

The 6-Layer Fallback Chain

A subtle but important detail: the Orchestrator's primary provider is a BluesMinds endpoint running moonshotai/kimi-k2.6. But I do not rely on a single key. I have six layers of fallback: five BluesMinds keys plus a final Sumopod fallback. If the primary key rate-limits, the next layer picks up automatically. If three keys fail in a row, the system degrades to Sumopod.

The point is not to chase 100% uptime. The point is that a 2am outage does not become a 4am outage because I am asleep while keys rotate.

What I'd Do Differently

If I were starting fresh, I would resist the urge to spin up twenty specialists. My first version had thirty agents, half of them overlapping. The cleanup pass was the most valuable weekend of the whole project.

I would also encode the delegation rules in the Orchestrator's system prompt from day one, with explicit "if the user asks for X, delegate to specialist Y" rules. The model is good at following instructions. It is less good at inventing a routing table on the fly.

The architecture I would recommend to anyone: one Orchestrator, ten to twelve narrow specialists, a single primary provider with manual fallback, and one human-facing channel. That is enough. Adding more agents or channels does not improve output; it adds integration surface to maintain.