Build a Claude vs GPT Router: 12 Steps, 90 Min [2026]

Every team shipping an AI feature in October 2026 runs into the same wall. Claude Opus 5.5 handles the hard reasoning tasks beautifully but costs $4 per million input tokens and $20 per million output tokens. GPT-6 Luna handles the easy ones for a tenth of a cent and barely dents a budget. Nobody wants to hardcode one provider and eat the cost of running every request through the expensive model, and nobody wants to hand-pick a model for every single API call either. The fix is a router: a thin service that looks at each request, decides which model it actually needs, sends it there, and logs what happened so you can prove the routing decision was correct.

This tutorial builds that router from scratch using the Claude 5.5 family and the GPT-6 family, then adds an evaluation harness that scores quality, latency, and cost side by side so the routing policy is based on data instead of guesswork. It fits alongside the rest of our AI and machine learning coverage, where we track model releases and pricing shifts from both labs as they happen. By the end you will have a working FastAPI service, provider adapters for both vendors, a task classifier, a fallback chain, token-level cost tracking, and a scoring loop that uses a third model as a judge. Twelve steps, roughly 90 minutes if you copy the code blocks as you go.

Why a Claude vs GPT router matters right now

Anthropic and OpenAI have both split their lineups into tiers that are priced an order of magnitude apart, and the gap is wide enough that routing decisions now show up directly on the monthly invoice. Anthropic’s pricing page lists Claude Opus 5.5 at $4 per million input tokens and $20 per million output tokens, Claude Sonnet 5.5 at $2 and $10, and Claude Haiku 5.5 at roughly $0.10 and $0.50, with Haiku 5.5 carrying a 1 million token context window and a 128,000 token maximum output, according to Anthropic’s Claude Platform pricing documentation. On the OpenAI side, the current OpenAI API pricing reference lists gpt-6-luna at $0.10 input and $0.50 output per million tokens, with gpt-6-astra and gpt-6.1-sol sitting at higher tiers for harder tasks.

Sending every request to the flagship model is the easiest way to burn through a budget. Sending every request to the cheapest model is the easiest way to ship a product that gets reasoning questions wrong. A router splits the difference by profiling each incoming request and only reaching for Opus 5.5 or GPT-6 Astra when the task actually needs that level of capability. Developer adoption patterns back up why both providers matter enough to route between rather than picking one. Stack Overflow reported that “OpenAI GPT models were used by 81% of developers for development work in the past year, compared with 43% for Anthropic Claude Sonnet models,” per the 2025 Stack Overflow Developer Survey. Both ecosystems are heavily used, which is exactly the situation a router is built for: pick the right tool per task instead of betting the whole stack on one vendor.

Prerequisites: accounts, SDKs, and versions you’ll need

Gather these before starting Step 1. Nothing here is optional, and mismatched SDK versions are the single most common reason this kind of project breaks on a fresh machine.

• Python 3.12 or newer (3.11 works, 3.13 works, anything older will fail on type hints used below)
• An Anthropic API key with billing enabled, generated from the Claude Console
• An OpenAI API key with billing enabled, generated from the OpenAI developer platform
• anthropic Python SDK, version 0.45 or newer
• openai Python SDK, version 1.60 or newer
• fastapi 0.115 or newer and uvicorn 0.32 or newer for the service layer
• pydantic 2.9 or newer for the shared request schema
• httpx 0.27 or newer, used internally by both SDKs for async calls
• A terminal, roughly 90 minutes, and about $5 to $10 of API credit to run the evaluation harness against a real test set

Run the install in one shot so version conflicts surface immediately instead of three steps in:

python3 -m venv router-env
source router-env/bin/activate
pip install "anthropic>=0.45" "openai>=1.60" "fastapi>=0.115" "uvicorn>=0.32" "pydantic>=2.9" "httpx>=0.27"

Step 1: Get API keys and set up your environment

Create a Claude API key in the Claude Console under Settings then API Keys, and an OpenAI key under API Keys in the OpenAI platform dashboard. Store both as environment variables instead of hardcoding them anywhere near the router logic. A leaked router key is a direct line to your billing account for both providers at once, so treat this file the same way you would treat a database password.

# .env (never commit this file)
ANTHROPIC_API_KEY=sk-ant-your-key-here
OPENAI_API_KEY=sk-your-key-here
ROUTER_LOG_LEVEL=info
ROUTER_DEFAULT_BUDGET_USD=0.02

Load it with python-dotenv or your process manager’s native env file support. The ROUTER_DEFAULT_BUDGET_USD value matters later: it is the ceiling the router will refuse to cross for a single request unless a caller explicitly overrides it, which is the cheapest insurance policy against a routing bug that accidentally sends every request to the most expensive model available.

Step 2: Design a provider-agnostic request schema

The whole point of a router is that the caller should never know or care which vendor answered the request. That only works if every adapter speaks the same internal language. Build one Pydantic model that every adapter accepts and every adapter returns, and keep provider-specific fields out of it entirely.

from pydantic import BaseModel
from typing import Literal, Optional

class RouterRequest(BaseModel):
    prompt: str
    system: Optional[str] = None
    task_type: Literal["classify", "generate", "reason", "extract", "chat"] = "chat"
    max_tokens: int = 1024
    budget_usd: Optional[float] = None
    require_model: Optional[str] = None  # escape hatch to force a specific model

class RouterResponse(BaseModel):
    text: str
    provider: str
    model: str
    input_tokens: int
    output_tokens: int
    cached_input_tokens: int = 0
    cost_usd: float
    latency_ms: int

Notice task_type is deliberately coarse. You do not need a sophisticated taxonomy to get most of the cost savings from routing. Five buckets, classify, generate, reason, extract, chat, cover the large majority of production traffic and map cleanly onto a tier of models from each provider. Resist the urge to add twenty categories before you have evaluation data proving the split is worth the complexity.

Step 3: Build the Claude adapter

The adapter’s job is narrow: accept a RouterRequest, call the Anthropic SDK, and return a RouterResponse with real token counts and a computed cost. Anthropic’s October 2026 release notes confirm the current model identifiers for the mid and budget tiers as claude-sonnet-5-5 and claude-haiku-5-5, per the Claude Platform release notes. The flagship tier follows the same naming pattern as claude-opus-5-5; confirm the exact string against your own Anthropic console before deploying, since Anthropic occasionally ships a point release under a slightly different string.

import time
import anthropic

CLAUDE_PRICING = {
    "claude-opus-5-5":   {"input": 4.00,  "output": 20.00},
    "claude-sonnet-5-5": {"input": 2.00,  "output": 10.00},
    "claude-haiku-5-5":  {"input": 0.10,  "output": 0.50},
}

client = anthropic.Anthropic()

def call_claude(model: str, req) -> dict:
    start = time.monotonic()
    message = client.messages.create(
        model=model,
        max_tokens=req.max_tokens,
        system=req.system or "",
        messages=[{"role": "user", "content": req.prompt}],
    )
    latency_ms = int((time.monotonic() - start) * 1000)
    usage = message.usage
    price = CLAUDE_PRICING[model]
    cost = (usage.input_tokens / 1_000_000 * price["input"]) + \
           (usage.output_tokens / 1_000_000 * price["output"])
    return {
        "text": message.content[0].text,
        "provider": "anthropic",
        "model": model,
        "input_tokens": usage.input_tokens,
        "output_tokens": usage.output_tokens,
        "cached_input_tokens": getattr(usage, "cache_read_input_tokens", 0) or 0,
        "cost_usd": round(cost, 6),
        "latency_ms": latency_ms,
    }

Keep the pricing table as data, not as scattered magic numbers inside the function. Anthropic revises prices more often than most teams revise their own code, and a release note like the one above showing prompt cache reads on Sonnet 5.5 dropping from $0.20 to $0.10 per million tokens is exactly the kind of change that should require editing one dictionary, not hunting through every call site.

Step 4: Build the GPT-6 adapter

The GPT-6 adapter mirrors the Claude adapter in shape but has one wrinkle Claude does not: a long-context pricing cliff. OpenAI bills any request over 272,000 input tokens at roughly double the input rate and 1.5 times the output rate, a threshold documented in the BenchLM OpenAI API pricing breakdown for October 2026. If your adapter does not check for that cliff before the call, a single oversized document upload can quietly double your bill for that request.

from openai import OpenAI

GPT_PRICING = {
    "gpt-6-astra":   {"input": 10.00, "output": 50.00, "long_input": 20.00, "long_output": 75.00},
    "gpt-6.1-sol":   {"input": 2.00,  "output": 10.00, "long_input": 4.00,  "long_output": 15.00},
    "gpt-6-luna":    {"input": 0.10,  "output": 0.50,  "long_input": 0.20,  "long_output": 0.75},
}
LONG_CONTEXT_THRESHOLD = 272_000

oa_client = OpenAI()

def call_gpt(model: str, req) -> dict:
    start = time.monotonic()
    messages = []
    if req.system:
        messages.append({"role": "system", "content": req.system})
    messages.append({"role": "user", "content": req.prompt})

    response = oa_client.chat.completions.create(
        model=model,
        max_tokens=req.max_tokens,
        messages=messages,
    )
    latency_ms = int((time.monotonic() - start) * 1000)
    usage = response.usage
    price = GPT_PRICING[model]
    is_long = usage.prompt_tokens > LONG_CONTEXT_THRESHOLD
    in_rate = price["long_input"] if is_long else price["input"]
    out_rate = price["long_output"] if is_long else price["output"]
    cost = (usage.prompt_tokens / 1_000_000 * in_rate) + \
           (usage.completion_tokens / 1_000_000 * out_rate)
    return {
        "text": response.choices[0].message.content,
        "provider": "openai",
        "model": model,
        "input_tokens": usage.prompt_tokens,
        "output_tokens": usage.completion_tokens,
        "cached_input_tokens": getattr(usage, "cached_tokens", 0) or 0,
        "cost_usd": round(cost, 6),
        "latency_ms": latency_ms,
    }

Both adapters now return the exact same shape of dictionary, which is what lets the router compare them like for like a few steps from now.

Step 5: Write a task classifier

You already have a task_type field callers can set manually, but most production traffic arrives without one. The cheapest way to fill the gap is to run a short classification pass on the cheapest model available before touching anything expensive. Haiku 5.5 and GPT-6 Luna both sit at the $0.10 input tier, so either works as the classifier and the cost is close to rounding error.

CLASSIFIER_PROMPT = """Classify the following request into exactly one category:
classify, generate, reason, extract, chat.
Reply with only the category word, nothing else.

Request: {prompt}"""

def classify_task(prompt: str) -> str:
    result = call_claude("claude-haiku-5-5", RouterRequestLike(
        prompt=CLASSIFIER_PROMPT.format(prompt=prompt[:2000]),
        system=None,
        max_tokens=10,
    ))
    label = result["text"].strip().lower()
    valid = {"classify", "generate", "reason", "extract", "chat"}
    return label if label in valid else "chat"

Truncating the prompt to 2,000 characters before classification keeps this step fast and cheap even when the real request is a 50-page document. The classifier only needs enough signal to guess the task type, not the full payload.

Step 6: Implement the routing policy

With a task type in hand, the routing policy is just a lookup table mapping task type to a preferred model and a cheaper fallback. The table below is a reasonable starting point built from the current pricing tiers; adjust it once your evaluation harness in Step 9 gives you real accuracy numbers for your own prompts.

Task type	Primary model	Cheap fallback	Why
classify	claude-haiku-5-5	gpt-6-luna	Low ambiguity, cost dominates
extract	gpt-6-luna	claude-haiku-5-5	Structured output, large context window helps on long documents
chat	claude-sonnet-5-5	gpt-6.1-sol	Balance of quality and latency for user-facing replies
generate	gpt-6.1-sol	claude-sonnet-5-5	Mid-tier reasoning, moderate output length
reason	claude-opus-5-5	gpt-6-astra	Multi-step reasoning justifies the premium tier

ROUTING_TABLE = {
    "classify": [("anthropic", "claude-haiku-5-5"), ("openai", "gpt-6-luna")],
    "extract":  [("openai", "gpt-6-luna"), ("anthropic", "claude-haiku-5-5")],
    "chat":     [("anthropic", "claude-sonnet-5-5"), ("openai", "gpt-6.1-sol")],
    "generate": [("openai", "gpt-6.1-sol"), ("anthropic", "claude-sonnet-5-5")],
    "reason":   [("anthropic", "claude-opus-5-5"), ("openai", "gpt-6-astra")],
}

def choose_route(req) -> tuple[str, str]:
    if req.require_model:
        provider = "anthropic" if "claude" in req.require_model else "openai"
        return provider, req.require_model
    candidates = ROUTING_TABLE[req.task_type]
    return candidates[0]

The require_model escape hatch matters more than it looks. There will always be a caller, usually someone debugging a weird output, who needs to pin a request to one exact model and bypass the policy entirely. Build that door in from day one instead of bolting it on after the third person asks for it.

Step 7: Add fallback chains and retry logic

The routing table above already lists a second candidate per task type. Wire it up so a failure on the primary model, a timeout, a rate limit, a 500 from the provider, falls through to the fallback instead of returning an error to the caller. This is the difference between a router that is a nice cost optimization and one that actually improves reliability.

import asyncio

ADAPTERS = {"anthropic": call_claude, "openai": call_gpt}

async def route_with_fallback(req) -> dict:
    provider, model = choose_route(req)
    candidates = [(provider, model)] + [
        c for c in ROUTING_TABLE[req.task_type] if c != (provider, model)
    ]
    last_error = None
    for provider_name, model_name in candidates:
        try:
            adapter = ADAPTERS[provider_name]
            return await asyncio.wait_for(
                asyncio.to_thread(adapter, model_name, req), timeout=25.0
            )
        except Exception as exc:
            last_error = exc
            continue
    raise RuntimeError(f"All routing candidates failed: {last_error}")

A 25-second timeout per candidate keeps a hung provider from stalling the whole request chain. If both Claude and GPT-6 time out, the caller gets a clear error instead of waiting a minute to find out nothing worked.

Step 8: Track tokens and cost per request

Every adapter already returns cost_usd, so the remaining work is persistence, not calculation. Log every routed request to a table with the task type, the model chosen, the cost, and the latency. This log is what eventually tells you whether the routing table in Step 6 is actually saving money or just moving it around.

import sqlite3
import json
from datetime import datetime, timezone

conn = sqlite3.connect("router_log.db")
conn.execute("""
CREATE TABLE IF NOT EXISTS requests (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT,
    task_type TEXT,
    provider TEXT,
    model TEXT,
    input_tokens INTEGER,
    output_tokens INTEGER,
    cached_input_tokens INTEGER,
    cost_usd REAL,
    latency_ms INTEGER
)
""")

def log_request(req, result: dict) -> None:
    conn.execute(
        "INSERT INTO requests (timestamp, task_type, provider, model, input_tokens, "
        "output_tokens, cached_input_tokens, cost_usd, latency_ms) VALUES (?,?,?,?,?,?,?,?,?)",
        (
            datetime.now(timezone.utc).isoformat(),
            req.task_type,
            result["provider"],
            result["model"],
            result["input_tokens"],
            result["output_tokens"],
            result["cached_input_tokens"],
            result["cost_usd"],
            result["latency_ms"],
        ),
    )
    conn.commit()

SQLite is enough for a single-instance router handling a few thousand requests a day. Swap it for Postgres once you have more than one router process writing to the same log, since SQLite’s file-level locking will start to bottleneck concurrent writers.

Step 9: Build the evaluation harness

The routing table in Step 6 is a hypothesis, not a fact. The evaluation harness is what turns it into a fact by running the same prompt through multiple models in parallel and recording how each one did on quality, latency, and cost. Build a test set of 20 to 50 real prompts pulled from your own traffic, grouped by task type, and run the full set through every candidate model whenever you change the routing table.

async def evaluate_prompt(prompt: str, task_type: str, models: list[tuple[str, str]]) -> list[dict]:
    req = RouterRequest(prompt=prompt, task_type=task_type)
    async def run_one(provider, model):
        adapter = ADAPTERS[provider]
        result = await asyncio.to_thread(adapter, model, req)
        result["prompt"] = prompt
        return result
    return await asyncio.gather(*[run_one(p, m) for p, m in models])

async def run_eval_suite(test_set: list[dict]) -> list[dict]:
    all_results = []
    for case in test_set:
        candidates = ROUTING_TABLE[case["task_type"]]
        results = await evaluate_prompt(case["prompt"], case["task_type"], candidates)
        all_results.extend(results)
    return all_results

Running candidates with asyncio.gather keeps the harness fast: every prompt hits both providers at once instead of waiting for Claude to finish before calling GPT-6. On a 40-prompt test set across five task types, this finishes in two or three minutes instead of fifteen.

Step 10: Score responses with an LLM-as-judge

Latency and cost are easy to measure automatically. Quality is not, which is why most evaluation harnesses use a third model as a judge rather than relying on exact-match string comparison. Use the strongest model available, Opus 5.5 or GPT-6 Astra, purely as a judge, never as a candidate in the same comparison, and ask it to score on a fixed rubric so results stay comparable across runs.

JUDGE_PROMPT = """You are grading an AI response for correctness and helpfulness.

Original prompt: {prompt}
Response to grade: {response}

Score the response from 1 to 5, where 5 means fully correct and directly useful,
and 1 means incorrect or unusable. Reply with only the number."""

def judge_response(prompt: str, response: str) -> int:
    req = RouterRequestLike(
        prompt=JUDGE_PROMPT.format(prompt=prompt, response=response),
        system=None,
        max_tokens=5,
    )
    result = call_claude("claude-opus-5-5", req)
    try:
        return int(result["text"].strip()[0])
    except (ValueError, IndexError):
        return 0

Run the judge on every response from Step 9, not just the ones you suspect are wrong. Judges disagree with human graders more often than teams expect, so spot-check a random 10% of judged responses by hand every time you refresh the test set, and treat the judge’s score as a strong signal rather than ground truth.

Step 11: Generate a quality-per-dollar report

With cost, latency, and a judge score attached to every result from Step 9 and Step 10, the harness can finally answer the question that matters: which model gives you the most quality per dollar for each task type. This is the number that should actually drive the routing table in Step 6, not the raw benchmark scores vendors publish in launch blog posts.

from collections import defaultdict

def build_report(scored_results: list[dict]) -> dict:
    grouped = defaultdict(list)
    for r in scored_results:
        key = (r["task_type"], r["provider"], r["model"])
        grouped[key].append(r)

    report = {}
    for key, rows in grouped.items():
        avg_score = sum(r["score"] for r in rows) / len(rows)
        avg_cost = sum(r["cost_usd"] for r in rows) / len(rows)
        avg_latency = sum(r["latency_ms"] for r in rows) / len(rows)
        quality_per_dollar = avg_score / avg_cost if avg_cost > 0 else 0
        report[key] = {
            "avg_score": round(avg_score, 2),
            "avg_cost_usd": round(avg_cost, 6),
            "avg_latency_ms": round(avg_latency, 1),
            "quality_per_dollar": round(quality_per_dollar, 1),
            "n": len(rows),
        }
    return report

Here is what a typical report looks like on a 40-prompt test set split across the five task types, using mock scores for illustration since your actual numbers will depend entirely on your own prompts:

Task type	Model	Avg score (1-5)	Avg cost	Avg latency	Quality per dollar
classify	claude-haiku-5-5	4.6	$0.00009	410ms	51,111
classify	gpt-6-luna	4.3	$0.00011	380ms	39,090
chat	claude-sonnet-5-5	4.7	$0.0021	890ms	2,238
chat	gpt-6.1-sol	4.5	$0.0019	920ms	2,368
reason	claude-opus-5-5	4.9	$0.014	2,100ms	350
reason	gpt-6-astra	4.8	$0.018	2,350ms	267

Output like this is what lets you defend a routing decision to a finance team instead of pointing at a vendor’s marketing page. It also tends to surprise people: in this illustrative run, GPT-6.1 Sol edges out Claude Sonnet 5.5 on quality per dollar for chat despite a slightly lower average score, because it is also slightly cheaper per call.

Step 12: Wrap it in a FastAPI service

The last step turns the functions above into something other services can actually call. Keep the endpoint thin: validate the request, call the router, log the result, return the response.

from fastapi import FastAPI, HTTPException

app = FastAPI(title="Claude vs GPT Router")

@app.post("/v1/route", response_model=RouterResponse)
async def route_request(req: RouterRequest):
    if req.task_type == "chat" and not req.require_model:
        req.task_type = classify_task(req.prompt) or req.task_type
    try:
        result = await route_with_fallback(req)
    except RuntimeError as exc:
        raise HTTPException(status_code=502, detail=str(exc))
    budget = req.budget_usd or 0.02
    if result["cost_usd"] > budget:
        result["warning"] = f"Request exceeded budget of ${budget}"
    log_request(req, result)
    return result

@app.get("/health")
def health():
    return {"status": "ok"}

Start it with uvicorn router:app --reload --port 8080 and send a test request with curl to confirm the whole chain works end to end before moving on to real traffic.

curl -X POST http://localhost:8080/v1/route \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Summarize the key risks in this contract clause.", "task_type": "reason"}'

A healthy response looks like this, with the model, cost, and latency filled in from the real API call rather than placeholders:

{
  "text": "The clause shifts liability for...",
  "provider": "anthropic",
  "model": "claude-opus-5-5",
  "input_tokens": 142,
  "output_tokens": 310,
  "cached_input_tokens": 0,
  "cost_usd": 0.013536,
  "latency_ms": 1894
}

If you need the router to sit in front of an agent framework rather than raw chat traffic, the same pattern extends cleanly. Readers who already followed our guide to building a secure MCP server for Claude can expose this router as one more MCP tool, letting Claude Code or any other MCP client call it directly instead of talking to Anthropic’s API alone.

Common pitfalls to avoid

• Hardcoding model strings everywhere. When Anthropic or OpenAI ship a point release, you want to change one pricing dictionary, not grep the entire codebase for a string.
• Ignoring the 272K-token long-context cliff. A single oversized GPT-6 request can silently double its cost if your adapter does not check the threshold before billing.
• Trusting published benchmarks over your own evaluation data. A model that wins on a general benchmark can still lose on quality per dollar for your specific prompts, which is why Step 9 through Step 11 exist.
• Skipping prompt caching. Anthropic’s cache read pricing on Sonnet 5.5 sits at a fraction of the base input rate; a router that resends the same system prompt uncached on every call is leaving money on the table for no benefit.
• Routing purely on cost with no latency floor. The cheapest model is not always fast enough for a user-facing chat endpoint with a 2-second SLA, even if it wins on quality per dollar in a batch evaluation.
• No timeout on fallback chains. Without the 25-second cap from Step 7, one slow provider can block every downstream request behind it.
• Comparing judge scores across different judge models. Switching the judge from Opus 5.5 to GPT-6 Astra mid-evaluation invalidates the comparison; keep one judge model fixed for the life of a test set.

Troubleshooting guide

These are the failures that show up most often once the router handles real traffic instead of a handful of manual test calls.

• 401 Unauthorized from Anthropic or OpenAI. The environment variable is not loaded into the process, usually because the .env file was placed in the wrong directory relative to where uvicorn was launched.
• 429 Too Many Requests during the evaluation harness. Running 40 prompts across five task types fires dozens of concurrent calls; add a semaphore limiting concurrent requests per provider to 5 or 10.
• Router always falls through to the fallback model. Check that the primary adapter is not silently throwing inside the try block in Step 7; add logging on the except branch instead of swallowing the exception.
• Cost numbers look too high. Confirm the pricing dictionary matches the current per-million-token rate, not a per-thousand-token rate left over from an older pricing era.
• Judge scores all come back as 0. The judge’s reply format drifted from a bare number to a sentence; tighten the judge prompt or parse with a regex instead of taking the first character.
• SQLite “database is locked” errors. You have more than one router process writing to the same file; move to Postgres or set conn.execute("PRAGMA journal_mode=WAL") as an interim fix.
• Classifier mislabels tasks as “chat” constantly. The fallback in classify_task defaults to “chat” on any unexpected output; log the raw classifier output to see whether it is actually failing or just returning something outside the five valid labels.
• Long-context GPT-6 calls return unexpectedly high costs. Verify the adapter is reading usage.prompt_tokens correctly and applying the long-context rate only above 272,000 tokens, not above some other threshold copied from an older model generation.
• asyncio.gather hangs with no error. One of the adapter calls inside asyncio.to_thread is blocking without a timeout; wrap each call in asyncio.wait_for as shown in Step 7, not just the top-level gather.

Advanced tips for production routing

Once the basic router works, a few upgrades pay off quickly in production. Streaming matters first: wire both adapters to stream tokens back to the caller instead of waiting for the full response, which cuts perceived latency on chat-style requests dramatically even though the total cost and token count stay the same. Semantic caching is the second upgrade; hashing exact prompts only catches literal repeats, so pairing the router with an embedding-based cache catches near-duplicate requests too, which matters a lot for support-bot style traffic where the same question gets asked a dozen different ways.

A multi-armed bandit routing policy is worth adopting once you have a few weeks of evaluation data. Instead of a static lookup table, the router allocates a small percentage of traffic to the non-default model for each task type on an ongoing basis, which keeps the quality-per-dollar numbers from Step 11 fresh as providers ship silent model updates. This also catches the case where a vendor quietly changes model behavior between point releases without announcing it, something that happens more often than either provider admits.

Finally, treat cost alerts as a first-class feature, not an afterthought. A router that silently absorbs a routing bug sending every “classify” request to Opus 5.5 for three days will not announce the mistake, your invoice will. Wire the cost_usd field from every logged request into a rolling daily total with a Slack or email alert if it crosses a threshold you set deliberately, and you will catch that kind of regression within hours instead of at the end of the billing cycle. Teams already running an AWS Bedrock AgentCore agent can reuse the same CloudWatch alarm pattern for this router’s cost log table instead of building alerting from scratch.

How this compares to picking one model and sticking with it

It is worth asking whether a router is overkill for a smaller project, and sometimes it is. If your traffic is overwhelmingly one task type, say a support bot that only ever answers short factual questions, a single model like Claude Haiku 5.5 or GPT-6 Luna covers the job without the added moving parts of a router. The router earns its complexity once traffic spans multiple task types with genuinely different quality requirements, which is most production AI features once they grow past a single use case.

Our earlier benchmark coverage of Claude Opus 5.5 against GPT-6 Astra on Terminal-Bench shows the gap between flagship models narrows on some tasks and widens sharply on others, which is the exact pattern that makes static “pick one model” decisions risky. A router does not require picking a winner once; it lets the evaluation data from Step 9 through Step 11 keep making that decision for you, task type by task type, as both vendors keep shipping updates.

The complete working project

Here is how the pieces from all twelve steps fit together as a single project layout you can drop into a repository:

router-project/
├── .env                  # API keys, never committed
├── requirements.txt
├── router.py             # FastAPI app, Step 12
├── adapters/
│   ├── claude_adapter.py # Step 3
│   └── gpt_adapter.py    # Step 4
├── classifier.py         # Step 5
├── routing_table.py      # Step 6
├── fallback.py           # Step 7
├── cost_log.py           # Step 8
├── eval/
│   ├── test_set.json     # your own 20-50 real prompts, grouped by task_type
│   ├── harness.py        # Step 9
│   ├── judge.py          # Step 10
│   └── report.py         # Step 11
└── router_log.db         # generated at runtime

Run the whole evaluation suite with a single entry point once the test set is populated with your own prompts:

python -m eval.harness --test-set eval/test_set.json --judge claude-opus-5-5

That single command runs every prompt against every candidate model for its task type, scores each response with the judge, and prints the quality-per-dollar table from Step 11. Re-run it any time either provider ships a new model, since a new release can reshuffle which model wins which task type without any code changes on your end, just a new row in the pricing dictionaries from Step 3 and Step 4.

Teams coming from a pure coding-assistant comparison, like the one in our GitHub Copilot vs Cursor breakdown, will recognize the same underlying lesson here: picking an AI tool once and never re-measuring it is how teams end up overpaying for months without noticing. The router and harness built in this tutorial exist specifically to keep that measurement running automatically instead of depending on someone remembering to check.

One more cost lever worth building into the pricing dictionaries from the start: Anthropic’s Claude Platform release notes show prompt cache pricing moves independently of base input and output pricing, and the Economize Cloud pricing breakdown for OpenAI shows the same pattern on cached tokens for GPT-6 models. If your router reuses the same system prompt across many requests, which most production routers do, caching alone can cut your effective input cost by more than half without touching the routing table at all. Our coverage of Anthropic’s 75% API price cut on Claude Haiku 5.5 is a good example of how quickly these numbers move and why the pricing dictionary in Step 3 needs to stay easy to update.

Frequently asked questions

Do I need accounts with both Anthropic and OpenAI to follow this tutorial?
Yes. The router’s entire value comes from having two real providers to choose between. A single $5 to $10 credit load on each account is enough to build and test everything in this guide.

Which model should I default to if I only want to build a simpler version of this?
Start with Claude Sonnet 5.5 or GPT-6.1 Sol as a single default model, then add the second provider once you have real traffic data showing which task types would benefit from a cheaper or stronger alternative.

How often should I re-run the evaluation harness?
Re-run it whenever either provider ships a new model release, and on a fixed schedule, monthly is reasonable, even if no release has been announced, since providers sometimes update model behavior quietly between named releases.

Is SQLite good enough for the cost log in production?
For a single router instance handling low to moderate traffic, yes. Once you run multiple router instances behind a load balancer, move the cost log to Postgres to avoid write-lock contention.

Can I add a third provider like Gemini to this same router?
Yes. The adapter pattern from Step 3 and Step 4 is designed to be provider-agnostic. Add a third adapter that returns the same RouterResponse shape, add its pricing to a new dictionary, and extend the routing table with its model identifiers.

What happens if both the primary and fallback model fail?
The route_with_fallback function in Step 7 raises a RuntimeError after exhausting every candidate for that task type, which the FastAPI endpoint in Step 12 converts into a 502 response. Add a third candidate per task type if your reliability requirements call for it.

Does using an LLM as a judge introduce bias toward that judge’s own vendor?
It can. Mitigate this by never including the judge model itself as a candidate in the same comparison, and by spot-checking a sample of judged responses by hand, as noted in Step 10.

Is a 272,000-token threshold the same for every GPT-6 model?
Based on current OpenAI pricing documentation, the long-context threshold applies across the GPT-6 family, but confirm the exact figure against the live pricing page before launching, since providers do adjust these thresholds between releases.