An LLM server whose scheduler knows how sure it is. It predicts every step's latency, bounds the prediction with an online conformal guarantee, and keeps users inside their latency targets when the hardware slows down mid-run.
Continuous batching runs prompt processing (prefill) and token generation (decode) in the same GPU step. Each prompt token you add makes the step slower for everyone already generating. Each one you hold back delays somebody's first token. vLLM and TensorRT-LLM settle this with a fixed token budget per step and a hand-tuned safety margin.
The right budget depends on the model, the accelerator, the SLO and the traffic, and it moves. A roofline analysis of Llama-3-8B puts the free prefill headroom at about 200 tokens on an A100 and about 600 on an L4. At run time, co-tenants and throttling change step latency by tens of percent. A scheduler that packs steps up to its latency prediction is exactly the scheduler that breaks when the prediction goes stale.
Seven features with physical units: GEMM tokens, prefill attention FLOPs, decode KV bytes, padding, per-sequence cost. Non-negative least squares. 7.3% error, and 4.4% on step sizes it never saw.
After each step it records actual ÷ predicted and updates its bound with adaptive conformal inference. The long-run miss rate converges to 10% for any sequence of ratios, drift included.
Binary search for the largest prompt chunk whose bounded latency still meets the time-per-token SLO. A dot product per probe, microseconds per decision.
The exact algorithm for minimising missed deadlines on one machine decides which prompts to serve first. Hopeless prompts wait, so savable ones make it.
Requests per second a 4-core CPU sustains with at least 90% of requests meeting TTFT 2 s and TPOT 100 ms. One run each; an earlier run measured the fixed budget at 5.36.
Capacity as a share of the best policy, minimum over Poisson at three SLOs, bursty arrivals and the Azure chat and code production traces. The simulator matches the real engine within 3.3 points on average.
| Scheduler | In drift | Overall |
|---|---|---|
| Fixed budget 128 | 47.7% | 71.5% |
| Fixed 128 + deadline order | 48.6% | 72.0% |
| Pacer, offline calibration | 37.6% | 66.0% |
| Pacer, online calibration | 85.3% | 92.0% |
| Model | Random split | Extrapolating |
|---|---|---|
| Roofline + 2 scalars | 50.2% | 81.0% |
| Gradient-boosted trees | 17.9% | 50.8% |
| Linear + tree residual | 9.9% | 5.0% |
| Physics-shaped linear | 7.3% | 4.4% |
Mean absolute percentage error. Extrapolation trains on steps of at most 512 tokens and tests on larger ones.
Pacer with offline calibration had the best steady-state capacity and the worst result under drift: 37.6%, below every static baseline. It fills each step right up to the bound it believes, so once the machine slows, every step overshoots. Static budgets survive by accident, because they leave slack nobody planned. Online calibration is what turns the model-predictive idea from fragile into robust.
Spending banked time-per-token slack on larger prompt chunks never helped across six scenarios and sometimes hurt, so it is off by default and reported as a negative result.