Free Best Router

Overview

One OpenAI-compatible endpoint that automatically discovers, ranks, and routes to the best available free AI model.

Every free-model provider exposes a different catalog, a different API contract, and a different failure mode. A consumer that wants "the best answer I can get right now, from the free tier, without manual failover" has no good primitive. Free Best Router is that primitive.

What I built

• A Node.js (≥22) OpenAI-compatible router that exposes a single /v1/chat/completions endpoint.
• A discovery layer that fetches each provider's free model catalog at startup and normalizes it.
• A health-check warm-up at boot and continuous runtime scoring using capability, reliability, latency, and context fit.
• A Wilson lower bound on success rate with Bayesian shrinkage for under-sampled candidates.
• Per-failure-type cooldowns (8 min for 429, 1 h for 404) with exponential backoff and per-type caps.
• Smart exploration: a small fraction of requests probe non-incumbent healthy candidates so the winner does not ossify.
• Bounded failover: up to 4 attempts per request with a hard wall-time cap, returning 502 / 429 with next_eligible_model and Retry-After when the pool is empty.
• DeepSeek Harness first-class integration via OpenAI-compatible provider config.
• 52 deterministic unit tests, GitHub Actions CI, zero telemetry, self-hosted, MIT-licensed.

Why it matters

It removes the single biggest operational tax of using free LLMs at scale: the manual failover loop. Once you have one endpoint that picks the best currently-healthy model and degrades safely, the rest of your application stack stops caring about provider churn.

Problem

Every free-model provider exposes a different catalog, a different API contract, and a different failure mode. A consumer that wants "the best answer I can get right now, from the free tier, without manual failover" has no good primitive. Free Best Router is that primitive.

Architecture

Your App
   │
   ▼
POST /v1/chat/completions   (OpenAI-compatible)
   │
   ▼
Free Best Router
   │
   ├──▶ catalog discovery         (startup + /_refresh)
   ├──▶ warm-up probe             (boot)
   ├──▶ runtime scoring           (capability × reliability × latency × ctx)
   ├──▶ Wilson lower bound        (success rate, time-decayed)
   ├──▶ exploration gate          (5% non-incumbent probes)
   ├──▶ bounded failover          (≤4 attempts, 180s wall time)
   └──▶ per-failure cooldowns     (429=8m, 404=1h, exponential)

        │   │
   ┌────┘   └────────────┐
   ▼                    ▼
OpenRouter          OpenCode / Zen
Groq                Cerebras
Mistral             DeepSeek
Local Ollama        Local LM Studio

Engineering decisions

• OpenAI-compatible on the input, OpenAI-compatible on the output — zero lock-in for callers.
• Free-only by default. Paid models are opt-in, never the default.
• Wilson lower bound over raw success rate — prevents brand-new or rarely-used models from winning on a small sample.
• Exploration is bounded and observable (x-free-router-explored response header).
• Every completion returns x-free-router-model so the caller knows which underlying model was actually picked.
• Zero telemetry. The router does not phone home.
• 52 deterministic unit tests instead of mocks against a real provider — the test surface is the routing logic itself.
• Self-hostable. Provider keys stay on the machine that runs the router.

What changed during development

• A first version used a simple weighted score. Under low traffic, brand-new models with no real evidence would briefly win on capability alone. Wilson lower bound fixed that.
• A first version tried to remember failures globally. With many providers and many model IDs, that grew unmanageable. The new version is per-model-per-failure-type with explicit cooldowns.
• A first version attempted streaming passthrough naively and lost tool_calls and finish_reason. The new version preserves both end-to-end.

Technical implementation

• Plain Node.js ≥22, ESM modules, minimal runtime surface.
• Provider modules each behind a small adapter that exposes the same shape.
• In-memory catalog + per-model health / score / cooldown state, refreshed on demand.
• OpenAI-compatible request and response schema, with custom response headers (x-free-router-*) for observability.
• Streaming SSE passthrough with finish_reason and tool_calls captured from upstream.
• GitHub Actions CI running the deterministic test suite.
• MIT-licensed, npm-installable, ready to drop into DeepSeek Harness via a small settings.yaml snippet.

Product implications

• One stable interface in front of a churning ecosystem is a product surface in its own right.
• Calling out which model actually answered is not optional — it is the difference between a debuggable system and a black box.
• A small test surface that exercises routing logic deterministically is more valuable than a large test surface that depends on live provider availability.

Lessons learned

• Abstractions earn their keep by absorbing change. The router absorbs provider churn; the caller does not have to.
• A score is only meaningful if you can defend it under low data. Use a confidence interval, not a point estimate.
• Failover has to be bounded. Without a wall-time cap, a stall in one provider cascades into a stall in your service.
• Telemetry is a product decision, not an engineering default. Shipping with zero telemetry is a feature when the caller is the operator.

Current state

v0.1.0 released. Public, MIT-licensed, 52 deterministic unit tests, GitHub Actions CI green, DeepSeek Harness first-class integration shipped. Featured on the DSH "Show Your Plugins!" board.