How to Build a Multi-Agent AI System with API Streaming: Complete Tutorial
OpenAI's Astra model just proved that multi-agent AI systems can solve problems no single agent — human or AI — has ever cracked. But you don't need to wait for Astra's public release to start building multi-agent applications. In this tutorial, we'll build a complete multi-agent system using the OpenAI-compatible API format, with real-time streaming, task delegation, and result aggregation.
By the end, you'll have a working system where multiple AI agents collaborate to break down complex tasks, work on sub-problems in parallel, and combine their results — all with live streaming output.
Prerequisites
- Python 3.10+
- An API key from Qubax AI (gives you access to 100+ models through one API)
- Basic familiarity with Python and async programming
Step 1: Set Up Your Environment
First, let's install the required packages and set up our project:
# Create project directory
mkdir multi-agent-system && cd multi-agent-system
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install httpx asyncio pydanticCreate a .env file with your API credentials:
# .env
QUBAX_API_KEY=your_api_key_here
QUBAX_BASE_URL=https://api.qubax.ai/v1Step 2: Build the Streaming Client
The foundation of any multi-agent system is the ability to stream responses from your LLM provider. Streaming is essential because:
- It provides real-time feedback (users see progress immediately)
- It allows you to process partial outputs (useful for agent reasoning)
- It's more memory-efficient than buffering full responses
Here's our streaming client:
import os
import json
import asyncio
from typing import AsyncGenerator
from dataclasses import dataclass
from dotenv import load_dotenv
import httpx
load_dotenv()
@dataclass
class StreamChunk:
content: str
role: str = "assistant"
class LLMClient:
"""Streaming LLM client compatible with OpenAI API format."""
def __init__(self):
self.api_key = os.getenv("QUBAX_API_KEY")
self.base_url = os.getenv("QUBAX_BASE_URL", "https://api.qubax.ai/v1")
self.client = httpx.AsyncClient(
timeout=httpx.Timeout(300.0, connect=10.0),
headers={"Authorization": f"Bearer {self.api_key}"}
)
async def stream_completion(
self,
messages: list[dict],
model: str = "gpt-5.6-sol",
temperature: float = 0.7,
max_tokens: int = 4096
) -> AsyncGenerator[StreamChunk, None]:
"""Stream a completion from the LLM, yielding chunks as they arrive."""
payload = {
"model": model,
"messages": messages,
"temperature": temperature,
"max_tokens": max_tokens,
"stream": True
}
async with self.client.stream(
"POST",
f"{self.base_url}/chat/completions",
json=payload,
headers={"Content-Type": "application/json"}
) as response:
response.raise_for_status()
async for line in response.aiter_lines():
if not line or not line.startswith("data: "):
continue
if line.strip() == "data: [DONE]":
break
try:
data = json.loads(line[6:])
delta = data["choices"][0]["delta"]
if "content" in delta and delta["content"]:
yield StreamChunk(content=delta["content"])
except json.JSONDecodeError:
continue
async def complete(
self,
messages: list[dict],
model: str = "gpt-5.6-sol",
temperature: float = 0.7,
) -> str:
"""Non-streaming completion — collects all chunks."""
full_response = []
async for chunk in self.stream_completion(messages, model, temperature):
full_response.append(chunk.content)
return "".join(full_response)Step 3: Define the Agent Class
Now let's create our agent abstraction. Each agent has a role, a system prompt, and can work on tasks independently:
from enum import Enum
from typing import Optional
class AgentRole(Enum):
ORCHESTRATOR = "orchestrator"
RESEARCHER = "researcher"
CODER = "coder"
REVIEWER = "reviewer"
WRITER = "writer"
class Agent:
"""A single AI agent with a specific role and capabilities."""
def __init__(
self,
name: str,
role: AgentRole,
model: str = "gpt-5.6-sol",
system_prompt: Optional[str] = None
):
self.name = name
self.role = role
self.model = model
self.client = LLMClient()
self.conversation: list[dict] = []
if system_prompt:
self.conversation.append({
"role": "system",
"content": system_prompt
})
else:
self.conversation.append({
"role": "system",
"content": self._default_system_prompt()
})
def _default_system_prompt(self) -> str:
role_descriptions = {
AgentRole.ORCHESTRATOR: "You are the orchestrator. Break down complex tasks into sub-tasks, delegate to specialist agents, and synthesize their results into a coherent final answer.",
AgentRole.RESEARCHER: "You are a research agent. Gather information, analyze data, and provide structured findings. Cite sources when possible.",
AgentRole.CODER: "You are a coding agent. Write clean, well-documented code. Include error handling and follow best practices.",
AgentRole.REVIEWER: "You are a code reviewer. Check for bugs, security issues, and improvements. Provide specific, actionable feedback.",
AgentRole.WRITER: "You are a technical writer. Transform technical outputs into clear, readable documentation.",
}
return role_descriptions.get(self.role, "You are a helpful AI agent.")
async def execute(self, task: str, stream: bool = True) -> str:
"""Execute a task and optionally stream the response."""
self.conversation.append({"role": "user", "content": task})
if stream:
print(f"\n[{self.name}] Thinking...", flush=True)
full_response = []
async for chunk in self.client.stream_completion(
self.conversation, model=self.model
):
print(chunk.content, end="", flush=True)
full_response.append(chunk.content)
response = "".join(full_response)
else:
response = await self.client.complete(
self.conversation, model=self.model
)
print(f"\n[{self.name}] Done.")
self.conversation.append({"role": "assistant", "content": response})
return responseStep 4: Build the Multi-Agent Orchestrator
This is where the magic happens. The orchestrator breaks down a complex task, assigns sub-tasks to specialist agents, runs them in parallel, and combines the results — exactly the pattern that Astra uses:
import asyncio
from dataclasses import dataclass, field
from typing import Any
@dataclass
class Task:
id: str
description: str
agent_role: AgentRole
dependencies: list[str] = field(default_factory=list)
result: str = ""
@dataclass
class TaskPlan:
tasks: list[Task]
def get_task(self, task_id: str) -> Optional[Task]:
for task in self.tasks:
if task.id == task_id:
return task
return None
def get_ready_tasks(self) -> list[Task]:
"""Return tasks whose dependencies are all completed."""
return [
task for task in self.tasks
if not task.result
and all(self.get_task(dep).result for dep in task.dependencies)
]
class MultiAgentOrchestrator:
"""Coordinates multiple agents working on a complex task."""
def __init__(self, model: str = "gpt-5.6-sol"):
self.model = model
self.client = LLMClient()
self.agents = {}
self.results = {}
def register_agent(self, agent: Agent):
self.agents[agent.role] = agent
async def plan_task(self, complex_task: str) -> TaskPlan:
"""Use the orchestrator agent to break down a task into sub-tasks."""
planning_prompt = f"""Analyze this complex task and break it into sub-tasks.
For each sub-task, assign one of these roles: researcher, coder, reviewer, writer.
Define dependencies between tasks where appropriate.
Return a JSON array with this format:
[
{{
"id": "task_1",
"description": "Clear description of what to do",
"agent_role": "researcher",
"dependencies": []
}}
]
Task: {complex_task}
"""
response = await self.client.complete(
[{"role": "user", "content": planning_prompt}],
model=self.model,
temperature=0.3
)
# Parse the JSON response
import re
json_match = re.search(r'\[.*?\]', response, re.DOTALL)
if json_match:
tasks_data = json.loads(json_match.group())
tasks = [
Task(
id=t["id"],
description=t["description"],
agent_role=AgentRole(t["agent_role"]),
dependencies=t.get("dependencies", [])
)
for t in tasks_data
]
return TaskPlan(tasks=tasks)
raise ValueError("Failed to parse task plan from LLM response")
async def execute_task(self, task: Task) -> str:
"""Execute a single task with the appropriate agent."""
agent = self.agents.get(task.agent_role)
if not agent:
# Fall back to orchestrator
agent = self.agents[AgentRole.ORCHESTRATOR]
# Include context from dependency results
context_parts = []
for dep_id in task.dependencies:
if dep_id in self.results:
context_parts.append(
f"Results from {dep_id}:\n{self.results[dep_id]}"
)
full_task = task.description
if context_parts:
full_task = (
"Context from previous tasks:\n\n"
+ "\n\n".join(context_parts)
+ f"\n\nYour task: {task.description}"
)
result = await agent.execute(full_task, stream=True)
self.results[task.id] = result
task.result = result
return result
async def run(self, complex_task: str) -> dict[str, str]:
"""Run the full multi-agent pipeline."""
print(f"=== Multi-Agent System Starting ===")
print(f"Task: {complex_task}\n")
# Step 1: Plan
print(">> Orchestrator is planning the task...")
plan = await self.plan_task(complex_task)
print(f">> Plan created with {len(plan.tasks)} tasks\n")
# Step 2: Execute tasks (respecting dependencies)
completed = set()
while len(completed) < len(plan.tasks):
ready = [
t for t in plan.tasks
if all(d in completed for d in t.dependencies)
and t.id not in completed
]
if not ready:
break
# Run ready tasks in parallel
tasks_to_run = [self.execute_task(t) for t in ready]
results = await asyncio.gather(*tasks_to_run)
for i, result in enumerate(results):
completed.add(ready[i].id)
return self.resultsStep 5: Put It All Together
Finally, let's wire everything up and run our multi-agent system:
async def main():
# Create the orchestrator
orchestrator = MultiAgentOrchestrator(model="gpt-5.6-sol")
# Register specialist agents
orchestrator.register_agent(Agent(
name="Coordinator",
role=AgentRole.ORCHESTRATOR,
model="gpt-5.6-sol"
))
orchestrator.register_agent(Agent(
name="Research Bot",
role=AgentRole.RESEARCHER,
model="claude-sonnet-4-20250514"
))
orchestrator.register_agent(Agent(
name="Code Bot",
role=AgentRole.CODER,
model="gpt-5.6-sol"
))
orchestrator.register_agent(Agent(
name="Review Bot",
role=AgentRole.REVIEWER,
model="claude-opus-5"
))
orchestrator.register_agent(Agent(
name="Docs Bot",
role=AgentRole.WRITER,
model="deepseek-v4-pro"
))
# Define the task
task = """Build a Python REST API for a todo application with:
- CRUD endpoints for tasks
- JWT authentication
- Input validation
- Complete documentation"""
# Run the multi-agent system
results = await orchestrator.run(task)
print("\n\n=== Final Results ===")
for task_id, result in results.items():
print(f"\n--- {task_id} ---")
print(result[:500] + "..." if len(result) > 500 else result)
if __name__ == "__main__":
asyncio.run(main())Best Practices for Multi-Agent Systems
Based on the patterns that make Astra and similar systems work:
1. Use the Right Model for Each Agent
Different models excel at different tasks. In our example, we used:
- GPT-5.6 Sol for orchestration and coding (strong reasoning)
- Claude Sonnet 4 for research (excellent analysis)
- Claude Opus 5 for code review (high accuracy)
- DeepSeek V4 Pro for documentation (cost-effective writing)
You can explore all these models on the Qubax AI models page.
2. Implement Error Handling and Retries
async def execute_with_retry(agent, task, max_retries=3):
for attempt in range(max_retries):
try:
return await agent.execute(task)
except Exception as e:
if attempt == max_retries - 1:
raise
await asyncio.sleep(2 ** attempt)3. Add Checkpointing
For long-running tasks, save progress so you can resume if something fails:
import json
from pathlib import Path
def save_checkpoint(orchestrator, path="checkpoints/"):
Path(path).mkdir(exist_ok=True)
for task_id, result in orchestrator.results.items():
with open(f"{path}/{task_id}.json", "w") as f:
json.dump({"task_id": task_id, "result": result}, f)4. Set Cost Limits
Multi-agent systems can potentially run forever (remember Astra?) — always set cost and token limits:
MAX_TOTAL_TOKENS = 500_000 # ~$1-2 depending on model
total_tokens = 0
async def bounded_execute(agent, task):
global total_tokens
if total_tokens >= MAX_TOTAL_TOKENS:
raise RuntimeError("Token budget exceeded")
result = await agent.execute(task)
# Track tokens from response headers or usage data
return resultCommon Pitfalls to Avoid
- Agent feedback loops — agents passing the same information back and forth without progress. Solution: track what information has already been shared
- Context window overflow — long conversations exceeding model limits. Solution: use context compression techniques or summarize prior steps
- Race conditions — parallel agents overwriting shared state. Solution: use proper async locks and result isolation
- Runaway costs — agents iterating endlessly without converging. Solution: set hard limits on iterations and total tokens
Key Takeaways
- Multi-agent systems unlock new capabilities — coordinating multiple specialized agents achieves results no single agent could match
- Streaming is essential — real-time feedback keeps users informed and enables reactive systems
- Model diversity matters — using different models for different roles optimizes both cost and quality
- Task planning is the hard part — a good orchestrator prompt is worth more than any individual agent improvement
Frequently Asked Questions
What is a multi-agent AI system?
A multi-agent AI system uses multiple AI agents, each with specialized roles, that collaborate on complex tasks. One agent (the orchestrator) breaks down the task, assigns sub-tasks to specialist agents, and combines their results — similar to how a team of humans works together.
How much does it cost to run a multi-agent system?
Costs depend on:
- Which models you use (GPT-5.6 Sol is more expensive but more capable; DeepSeek V4 Flash is cheaper)
- How many agents and iterations you need
- Length of conversations
A typical multi-agent run for a medium-complexity task might cost $0.50-$5.00 in API tokens. You can check current pricing on the Qubax AI models page.
Which AI model is best for multi-agent orchestration?
For orchestration, you want strong reasoning and planning. GPT-5.6 Sol and Claude Opus 5 are excellent choices. For specialist roles, you can optimize cost by using less expensive models like DeepSeek V4 Flash for simpler tasks.
Can I run agents in different programming languages?
Yes — the API is language-agnostic. You can build agents in Python, JavaScript, Go, or any language with HTTP client support. Our example uses Python with the Qubax AI API, which is compatible with the OpenAI API format.
How do I prevent infinite loops in multi-agent systems?
Set hard limits on:
- Maximum number of iterations per agent
- Total token budget for the entire run
- Maximum conversation length per agent
- Timeout values for each task
Track state between agents and detect when the same information is being repeated.
Is this the same pattern OpenAI's Astra uses?
The core concept is similar — multiple agents coordinated over extended periods to solve hard problems. Astra adds more sophisticated self-correction, longer time horizons, and formalized verification. But the fundamental multi-agent + streaming pattern is the same one you can build today.
Ready to build multi-agent systems with frontier AI models? [Get started with Qubax AI](https://qubax.ai/docs) — unified API access to GPT-5.6, Claude, Gemini, DeepSeek, and 100+ more models with transparent per-token pricing.