Back to blog
Tutorial·11 min read·2062 words

How to Build a Multi-Agent AI System with API Streaming: Complete Tutorial

Learn to build a multi-agent AI system with real-time streaming, task delegation, and parallel execution — the same pattern OpenAI's Astra uses to solve complex problems. Complete Python tutorial with code examples.

How to Build a Multi-Agent AI System with API Streaming: Complete Tutorial — illustration

How to Build a Multi-Agent AI System with API Streaming: Complete Tutorial

OpenAI's Astra model just proved that multi-agent AI systems can solve problems no single agent — human or AI — has ever cracked. But you don't need to wait for Astra's public release to start building multi-agent applications. In this tutorial, we'll build a complete multi-agent system using the OpenAI-compatible API format, with real-time streaming, task delegation, and result aggregation.

By the end, you'll have a working system where multiple AI agents collaborate to break down complex tasks, work on sub-problems in parallel, and combine their results — all with live streaming output.

Prerequisites

  • Python 3.10+
  • An API key from Qubax AI (gives you access to 100+ models through one API)
  • Basic familiarity with Python and async programming

Step 1: Set Up Your Environment

First, let's install the required packages and set up our project:

bash
# Create project directory
mkdir multi-agent-system && cd multi-agent-system

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install httpx asyncio pydantic

Create a .env file with your API credentials:

bash
# .env
QUBAX_API_KEY=your_api_key_here
QUBAX_BASE_URL=https://api.qubax.ai/v1

Step 2: Build the Streaming Client

The foundation of any multi-agent system is the ability to stream responses from your LLM provider. Streaming is essential because:

  1. It provides real-time feedback (users see progress immediately)
  2. It allows you to process partial outputs (useful for agent reasoning)
  3. It's more memory-efficient than buffering full responses

Here's our streaming client:

python
import os
import json
import asyncio
from typing import AsyncGenerator
from dataclasses import dataclass
from dotenv import load_dotenv

import httpx

load_dotenv()

@dataclass
class StreamChunk:
    content: str
    role: str = "assistant"

class LLMClient:
    """Streaming LLM client compatible with OpenAI API format."""

    def __init__(self):
        self.api_key = os.getenv("QUBAX_API_KEY")
        self.base_url = os.getenv("QUBAX_BASE_URL", "https://api.qubax.ai/v1")
        self.client = httpx.AsyncClient(
            timeout=httpx.Timeout(300.0, connect=10.0),
            headers={"Authorization": f"Bearer {self.api_key}"}
        )

    async def stream_completion(
        self,
        messages: list[dict],
        model: str = "gpt-5.6-sol",
        temperature: float = 0.7,
        max_tokens: int = 4096
    ) -> AsyncGenerator[StreamChunk, None]:
        """Stream a completion from the LLM, yielding chunks as they arrive."""

        payload = {
            "model": model,
            "messages": messages,
            "temperature": temperature,
            "max_tokens": max_tokens,
            "stream": True
        }

        async with self.client.stream(
            "POST",
            f"{self.base_url}/chat/completions",
            json=payload,
            headers={"Content-Type": "application/json"}
        ) as response:
            response.raise_for_status()
            async for line in response.aiter_lines():
                if not line or not line.startswith("data: "):
                    continue
                if line.strip() == "data: [DONE]":
                    break
                try:
                    data = json.loads(line[6:])
                    delta = data["choices"][0]["delta"]
                    if "content" in delta and delta["content"]:
                        yield StreamChunk(content=delta["content"])
                except json.JSONDecodeError:
                    continue

    async def complete(
        self,
        messages: list[dict],
        model: str = "gpt-5.6-sol",
        temperature: float = 0.7,
    ) -> str:
        """Non-streaming completion — collects all chunks."""
        full_response = []
        async for chunk in self.stream_completion(messages, model, temperature):
            full_response.append(chunk.content)
        return "".join(full_response)

Step 3: Define the Agent Class

Now let's create our agent abstraction. Each agent has a role, a system prompt, and can work on tasks independently:

python
from enum import Enum
from typing import Optional

class AgentRole(Enum):
    ORCHESTRATOR = "orchestrator"
    RESEARCHER = "researcher"
    CODER = "coder"
    REVIEWER = "reviewer"
    WRITER = "writer"

class Agent:
    """A single AI agent with a specific role and capabilities."""

    def __init__(
        self,
        name: str,
        role: AgentRole,
        model: str = "gpt-5.6-sol",
        system_prompt: Optional[str] = None
    ):
        self.name = name
        self.role = role
        self.model = model
        self.client = LLMClient()
        self.conversation: list[dict] = []

        if system_prompt:
            self.conversation.append({
                "role": "system",
                "content": system_prompt
            })
        else:
            self.conversation.append({
                "role": "system",
                "content": self._default_system_prompt()
            })

    def _default_system_prompt(self) -> str:
        role_descriptions = {
            AgentRole.ORCHESTRATOR: "You are the orchestrator. Break down complex tasks into sub-tasks, delegate to specialist agents, and synthesize their results into a coherent final answer.",
            AgentRole.RESEARCHER: "You are a research agent. Gather information, analyze data, and provide structured findings. Cite sources when possible.",
            AgentRole.CODER: "You are a coding agent. Write clean, well-documented code. Include error handling and follow best practices.",
            AgentRole.REVIEWER: "You are a code reviewer. Check for bugs, security issues, and improvements. Provide specific, actionable feedback.",
            AgentRole.WRITER: "You are a technical writer. Transform technical outputs into clear, readable documentation.",
        }
        return role_descriptions.get(self.role, "You are a helpful AI agent.")

    async def execute(self, task: str, stream: bool = True) -> str:
        """Execute a task and optionally stream the response."""
        self.conversation.append({"role": "user", "content": task})

        if stream:
            print(f"\n[{self.name}] Thinking...", flush=True)
            full_response = []
            async for chunk in self.client.stream_completion(
                self.conversation, model=self.model
            ):
                print(chunk.content, end="", flush=True)
                full_response.append(chunk.content)
            response = "".join(full_response)
        else:
            response = await self.client.complete(
                self.conversation, model=self.model
            )
            print(f"\n[{self.name}] Done.")

        self.conversation.append({"role": "assistant", "content": response})
        return response

Step 4: Build the Multi-Agent Orchestrator

This is where the magic happens. The orchestrator breaks down a complex task, assigns sub-tasks to specialist agents, runs them in parallel, and combines the results — exactly the pattern that Astra uses:

python
import asyncio
from dataclasses import dataclass, field
from typing import Any

@dataclass
class Task:
    id: str
    description: str
    agent_role: AgentRole
    dependencies: list[str] = field(default_factory=list)
    result: str = ""

@dataclass
class TaskPlan:
    tasks: list[Task]

    def get_task(self, task_id: str) -> Optional[Task]:
        for task in self.tasks:
            if task.id == task_id:
                return task
        return None

    def get_ready_tasks(self) -> list[Task]:
        """Return tasks whose dependencies are all completed."""
        return [
            task for task in self.tasks
            if not task.result
            and all(self.get_task(dep).result for dep in task.dependencies)
        ]

class MultiAgentOrchestrator:
    """Coordinates multiple agents working on a complex task."""

    def __init__(self, model: str = "gpt-5.6-sol"):
        self.model = model
        self.client = LLMClient()
        self.agents = {}
        self.results = {}

    def register_agent(self, agent: Agent):
        self.agents[agent.role] = agent

    async def plan_task(self, complex_task: str) -> TaskPlan:
        """Use the orchestrator agent to break down a task into sub-tasks."""
        planning_prompt = f"""Analyze this complex task and break it into sub-tasks.
For each sub-task, assign one of these roles: researcher, coder, reviewer, writer.
Define dependencies between tasks where appropriate.

Return a JSON array with this format:
[
  {{
    "id": "task_1",
    "description": "Clear description of what to do",
    "agent_role": "researcher",
    "dependencies": []
  }}
]

Task: {complex_task}
"""
        response = await self.client.complete(
            [{"role": "user", "content": planning_prompt}],
            model=self.model,
            temperature=0.3
        )

        # Parse the JSON response
        import re
        json_match = re.search(r'\[.*?\]', response, re.DOTALL)
        if json_match:
            tasks_data = json.loads(json_match.group())
            tasks = [
                Task(
                    id=t["id"],
                    description=t["description"],
                    agent_role=AgentRole(t["agent_role"]),
                    dependencies=t.get("dependencies", [])
                )
                for t in tasks_data
            ]
            return TaskPlan(tasks=tasks)
        raise ValueError("Failed to parse task plan from LLM response")

    async def execute_task(self, task: Task) -> str:
        """Execute a single task with the appropriate agent."""
        agent = self.agents.get(task.agent_role)
        if not agent:
            # Fall back to orchestrator
            agent = self.agents[AgentRole.ORCHESTRATOR]

        # Include context from dependency results
        context_parts = []
        for dep_id in task.dependencies:
            if dep_id in self.results:
                context_parts.append(
                    f"Results from {dep_id}:\n{self.results[dep_id]}"
                )

        full_task = task.description
        if context_parts:
            full_task = (
                "Context from previous tasks:\n\n"
                + "\n\n".join(context_parts)
                + f"\n\nYour task: {task.description}"
            )

        result = await agent.execute(full_task, stream=True)
        self.results[task.id] = result
        task.result = result
        return result

    async def run(self, complex_task: str) -> dict[str, str]:
        """Run the full multi-agent pipeline."""
        print(f"=== Multi-Agent System Starting ===")
        print(f"Task: {complex_task}\n")

        # Step 1: Plan
        print(">> Orchestrator is planning the task...")
        plan = await self.plan_task(complex_task)
        print(f">> Plan created with {len(plan.tasks)} tasks\n")

        # Step 2: Execute tasks (respecting dependencies)
        completed = set()
        while len(completed) < len(plan.tasks):
            ready = [
                t for t in plan.tasks
                if all(d in completed for d in t.dependencies)
                and t.id not in completed
            ]
            if not ready:
                break

            # Run ready tasks in parallel
            tasks_to_run = [self.execute_task(t) for t in ready]
            results = await asyncio.gather(*tasks_to_run)

            for i, result in enumerate(results):
                completed.add(ready[i].id)

        return self.results

Step 5: Put It All Together

Finally, let's wire everything up and run our multi-agent system:

python
async def main():
    # Create the orchestrator
    orchestrator = MultiAgentOrchestrator(model="gpt-5.6-sol")

    # Register specialist agents
    orchestrator.register_agent(Agent(
        name="Coordinator",
        role=AgentRole.ORCHESTRATOR,
        model="gpt-5.6-sol"
    ))
    orchestrator.register_agent(Agent(
        name="Research Bot",
        role=AgentRole.RESEARCHER,
        model="claude-sonnet-4-20250514"
    ))
    orchestrator.register_agent(Agent(
        name="Code Bot",
        role=AgentRole.CODER,
        model="gpt-5.6-sol"
    ))
    orchestrator.register_agent(Agent(
        name="Review Bot",
        role=AgentRole.REVIEWER,
        model="claude-opus-5"
    ))
    orchestrator.register_agent(Agent(
        name="Docs Bot",
        role=AgentRole.WRITER,
        model="deepseek-v4-pro"
    ))

    # Define the task
    task = """Build a Python REST API for a todo application with:
    - CRUD endpoints for tasks
    - JWT authentication
    - Input validation
    - Complete documentation"""

    # Run the multi-agent system
    results = await orchestrator.run(task)

    print("\n\n=== Final Results ===")
    for task_id, result in results.items():
        print(f"\n--- {task_id} ---")
        print(result[:500] + "..." if len(result) > 500 else result)

if __name__ == "__main__":
    asyncio.run(main())

Best Practices for Multi-Agent Systems

Based on the patterns that make Astra and similar systems work:

1. Use the Right Model for Each Agent

Different models excel at different tasks. In our example, we used:

  • GPT-5.6 Sol for orchestration and coding (strong reasoning)
  • Claude Sonnet 4 for research (excellent analysis)
  • Claude Opus 5 for code review (high accuracy)
  • DeepSeek V4 Pro for documentation (cost-effective writing)

You can explore all these models on the Qubax AI models page.

2. Implement Error Handling and Retries

python
async def execute_with_retry(agent, task, max_retries=3):
    for attempt in range(max_retries):
        try:
            return await agent.execute(task)
        except Exception as e:
            if attempt == max_retries - 1:
                raise
            await asyncio.sleep(2 ** attempt)

3. Add Checkpointing

For long-running tasks, save progress so you can resume if something fails:

python
import json
from pathlib import Path

def save_checkpoint(orchestrator, path="checkpoints/"):
    Path(path).mkdir(exist_ok=True)
    for task_id, result in orchestrator.results.items():
        with open(f"{path}/{task_id}.json", "w") as f:
            json.dump({"task_id": task_id, "result": result}, f)

4. Set Cost Limits

Multi-agent systems can potentially run forever (remember Astra?) — always set cost and token limits:

python
MAX_TOTAL_TOKENS = 500_000  # ~$1-2 depending on model
total_tokens = 0

async def bounded_execute(agent, task):
    global total_tokens
    if total_tokens >= MAX_TOTAL_TOKENS:
        raise RuntimeError("Token budget exceeded")
    result = await agent.execute(task)
    # Track tokens from response headers or usage data
    return result

Common Pitfalls to Avoid

  • Agent feedback loops — agents passing the same information back and forth without progress. Solution: track what information has already been shared
  • Context window overflow — long conversations exceeding model limits. Solution: use context compression techniques or summarize prior steps
  • Race conditions — parallel agents overwriting shared state. Solution: use proper async locks and result isolation
  • Runaway costs — agents iterating endlessly without converging. Solution: set hard limits on iterations and total tokens

Key Takeaways

  1. Multi-agent systems unlock new capabilities — coordinating multiple specialized agents achieves results no single agent could match
  2. Streaming is essential — real-time feedback keeps users informed and enables reactive systems
  3. Model diversity matters — using different models for different roles optimizes both cost and quality
  4. Task planning is the hard part — a good orchestrator prompt is worth more than any individual agent improvement

Frequently Asked Questions

What is a multi-agent AI system?

A multi-agent AI system uses multiple AI agents, each with specialized roles, that collaborate on complex tasks. One agent (the orchestrator) breaks down the task, assigns sub-tasks to specialist agents, and combines their results — similar to how a team of humans works together.

How much does it cost to run a multi-agent system?

Costs depend on:

  • Which models you use (GPT-5.6 Sol is more expensive but more capable; DeepSeek V4 Flash is cheaper)
  • How many agents and iterations you need
  • Length of conversations

A typical multi-agent run for a medium-complexity task might cost $0.50-$5.00 in API tokens. You can check current pricing on the Qubax AI models page.

Which AI model is best for multi-agent orchestration?

For orchestration, you want strong reasoning and planning. GPT-5.6 Sol and Claude Opus 5 are excellent choices. For specialist roles, you can optimize cost by using less expensive models like DeepSeek V4 Flash for simpler tasks.

Can I run agents in different programming languages?

Yes — the API is language-agnostic. You can build agents in Python, JavaScript, Go, or any language with HTTP client support. Our example uses Python with the Qubax AI API, which is compatible with the OpenAI API format.

How do I prevent infinite loops in multi-agent systems?

Set hard limits on:

  • Maximum number of iterations per agent
  • Total token budget for the entire run
  • Maximum conversation length per agent
  • Timeout values for each task

Track state between agents and detect when the same information is being repeated.

Is this the same pattern OpenAI's Astra uses?

The core concept is similar — multiple agents coordinated over extended periods to solve hard problems. Astra adds more sophisticated self-correction, longer time horizons, and formalized verification. But the fundamental multi-agent + streaming pattern is the same one you can build today.


Ready to build multi-agent systems with frontier AI models? [Get started with Qubax AI](https://qubax.ai/docs) — unified API access to GPT-5.6, Claude, Gemini, DeepSeek, and 100+ more models with transparent per-token pricing.

Article tags

#multi-agent#streaming#api#python#tutorial
Share:Post on XTelegramLinkedInYHacker NewsReddit
Qubax AI

Qubax AI

AI Models at up to 99% off · Pay with crypto

Access GPT, Claude, Gemini, GLM & 340+ models through one OpenAI-compatible API. Up to 99% off. Pay with 200+ cryptocurrencies. Get $1 free credits — no credit card needed.

Related articles