Let the small queue have a turn
Sharing workers fairly when one caller arrives with a much bigger pile of work.
Fair scheduling for shared workloads.
Python
Notes & checks
The fair scheduler extracted from infergate into its own Python library.
Simulated workload: small-tenant p50 went from 32 s with FIFO to 1.15 s with fair scheduling, at the same 31.9 req/s. Standalone simulation, separate from the gateway HTTP path.
The problem
A single first-in, first-out queue is simple: whoever arrives first goes first. But a caller that submits a large burst can put everyone else behind it. fairq is a small in-memory scheduling library extracted from infergate. It chooses work; it does not run it.
How it works
- Caller queuesFIFO within each
- Credit + turnsEstimated token cost
- Free workerDispatch one request
Keep a line for each tenant
A tenant is one caller sharing the workers. Requests stay in arrival order within that tenant's queue. Each request carries an estimated token cost, where tokens are the pieces of text a language model processes.
Give each line spending credit
On each visit, a tenant receives a fixed allowance multiplied by its weight. The scheduler dispatches whole requests it can afford. This is weighted deficit round robin: take turns and carry unused credit forward.
Carry credit, not idle advantage
A large request can accumulate enough credit over multiple rounds. When a tenant's queue becomes empty, it loses leftover credit. New work rejoins at the back, so being idle does not build a stockpile of turns.
Decisions & tradeoffs
Fairness changes who waits
The included burst simulation gets small tenants through sooner by making the noisy tenant wait longer. It does not create more worker capacity. The benchmark is deterministic simulated time, not a wall-clock throughput test.
Small library, explicit boundaries
There is no locking or persistence; use one dispatcher per queue. Running work is not interrupted. Inaccurate cost estimates and long requests can still affect waiting time.
Try it yourself
With uv installed, run these from a fresh fairq checkout. uv can install the required Python 3.12. No services or model calls are used.
uv sync
uv run --no-env-file python -m fairq.demo
uv run --no-env-file python -m fairq.benchCompare the FIFO dispatch order with the fair order. The benchmark's completion times and rates are simulated. Try changing the weights in the small Python example in the README and observe which caller gets more token service.
Sources & scope
Based on the local README and queue implementation. Benchmark discussion refers only to the standalone simulation.