One place to manage model calls
What happens between asking an AI a question and paying for its answer? A walk through a small model gateway.
An LLM gateway with cost controls.
Python / FastAPI / Postgres
Includes fairq →The fair scheduler I pulled into its own Python library.
Notes & checks
Cost logging, caching, routing, limits, and streaming behind an OpenAI-compatible API.
fairq: small-tenant p50 went from 32 s with FIFO to 1.15 s with fair scheduling, at the same 31.9 req/s. Standalone simulated workload, separate from the gateway HTTP path.
Mock gateway load test: 0.73 ms p50. Simulated calls with an in-process cache and log store. Excludes network and Postgres overhead.
The problem
If several apps call a language model directly, each app has to handle retries, costs, and busy periods. infergate is a gateway: a service that sits between those apps and the model provider. It accepts chat requests in an OpenAI-compatible format.
How it works
- App requestQueue + limits
- Cache or modelCheap, then fallback
- AnswerRecord usage
Give callers a turn
Requests enter a fair queue before the other checks. A caller is a tenant, meaning one user of a shared service. The queue gives tenants turns, while concurrency limits cap how many requests can run together. Even cached requests take this path.
Reuse an answer or choose a model
For a non-streaming request, the gateway checks the caller's spend limit and an exact-match cache, a store of previous answers. With an auto route configured, a miss tries the cheaper model first. Errors, limits, or a short or refusal answer can trigger one fallback to the stronger model.
Return the answer and record usage
Each model attempt gets a cost-log entry in Postgres. Successful cacheable answers get an expiry time. Streaming sends chunks as they arrive and skips the cache; retries and fallback stop once delivery starts.
Decisions & tradeoffs
Simple checks are easy to explain
The routing quality check looks at answer length and refusal prefixes. It can catch an empty answer, but it cannot tell whether an answer is correct.
Limits have boundaries
Spend checks use logged usage, so an attempt can push the total past a cap. Concurrency and queue limits are per process. The mock load test uses fake calls and an in-memory store, so its timings do not include a network or Postgres.
Try it yourself
From a fresh infergate checkout, with Python 3.12+ and uv installed, run the small offline load test. No database server or model account is needed.
uv sync
uv run --no-env-file python scripts/loadtest.py \
--requests 20 --concurrency 3 --latency-scale 0The JSON output describes simulated calls and costs. The zero delay is for checking behavior, not measuring a real provider's speed. Follow the README for the separate HTTP demo.
Sources & scope
Based on the local README, request handler, and mock load-test source. No live-model performance claim is made here.