More tokens per second, per dollar, on any hardware.

A small studio that tunes serving stacks, quantizes models without losing quality, and moves inference onto new accelerators. Every engagement starts and ends with a benchmark you can run yourself.

built formodel API providersteams serving their own modelsdecentralized compute networksaccelerator vendors

serveServing tuned to your model and your traffic, on vLLM or SGLang

quantFP8, INT4 and FP4 deployments, with an eval gate on every change

portPorts to AMD Instinct and newer accelerators, pinned per SKU

planCost per token measured on your model, not taken from a vendor chart

yoursYou keep the harness, the configs and the raw data

Cost model

live · edit any value

Most inference bills are set by three numbers: tokens per second per GPU, how busy the GPUs are, and the hourly rate. We work on the first two and help you negotiate the third with data. This is what a throughput gain is worth at your volume.

Cost / 1M output tokens: —
GPUs required: —
Monthly compute: —
Monthly saving: —

Accelerator

Hourly rate $ / GPU·h

Output throughput / GPU tok / s

Output volume tok / day

Utilisation %

Speed-up from tuning ×

2.00

rate Indicative on-demand list prices. Edit to match your contract.

tok/s Example value. Use your own measured number — it moves with model, context length and batch size.

monthly GPUs × rate × 730 h. Stepped because you rent whole GPUs.

What we do

U1–U4 · engage separately or together

Serving optimisation

vLLM and SGLang set up for your model and traffic shape: batching and scheduling, KV-cache sizing and offload, tensor and expert parallel layout, prefix caching, speculative decoding. We profile where the time goes before changing anything.

typical 3–6 weeks

Quantization with eval gates

FP8, INT4 and FP4, each change gated by an evaluation suite so quality loss is measured, not assumed. We also check that the quantized kernels on your hardware produce what the reference does, which is where most of the surprises are.

typical 2–5 weeks

Hardware portability

Your stack on AMD Instinct and newer accelerators. One image per SKU, pinned to an exact driver, framework, compiler and engine version. Output checks against your current hardware, and CI that fails when a point release changes your tensors.

typical 4–8 weeks per platform

Benchmarking and capacity planning

Cost per token across the hardware you could buy or rent, measured on your model. A capacity model you can keep updating, and a plain answer on whether a new accelerator is worth it for your workload.

typical 2–3 weeks

Signal path. U1–U3 can be engaged separately; U4 closes the loop with measured cost rather than a vendor's number.

Platforms

NVIDIA is the reference every other number is measured against. These are the platforms we build for, port to and evaluate.

Vendor	Device	Type	Software path

Notes from the work

each cites a public source

AN-001 One tolerance for every chip is no tolerance Widening a verification threshold to admit new hardware also lets quantized models through.

Any system that checks inference by recomputing it, whether a proof-of-compute network or a buyer auditing a provider, has to accept some numerical noise. Two honest runs of the same weights on different GPUs never match bit for bit. The question is how much noise to allow.

Published validation data from an open-source decentralized inference network makes the point. Honest FP8 recomputes of Qwen3-235B across A100, H100 and B200 differ by about 0.07 mean L2 on the compared vector. Swap FP8 weights for INT4 and it rises to about 0.24. Calibrate the threshold on honest data and the two separate. Widen it to cover more models and hardware and the swap passes. Drag the plot.

Per-nonce L2, prover vs validatordrag to move τ

threshold τ 0.174

Honest recomputes rejected: 2.02%
INT4 swaps caught: 73.0%

Workable trade-off.

Gamma fits to published mean ± σ; reproduces the published 2.0% honest mismatch at τ = 0.174. Modelled, not raw samples.

Recommendation. Calibrate a tolerance per hardware class instead of one global number. New silicon gets its own measured bound and nobody else's gets looser.

AN-002 Pin four versions, not one On new hardware the dangerous failure is a model that loads, runs at normal speed and outputs nonsense.

Outside CUDA, the driver, PyTorch, the Triton compiler and the serving engine are tightly coupled. A point release in any of them can change kernel selection and with it the numbers the model produces.

vLLM issue #36337 is a public example: a 4-bit MoE model on four AMD MI350X cards under ROCm 7.2 loaded normally, generated at 7–9 tokens per second, and produced incoherent text. The suspected cause was a change between two ROCm point releases in how packed 4-bit weights were dequantized. Nothing crashed, so a load check would have passed it.

driver: rocm 7.2.x
framework: torch 2.10
compiler: triton 3.6
engine: vllm 0.25

Recommendation. Pin all four per SKU, move them together, and gate every image on an output comparison against reference hardware, not on whether the model loads.

Source: github.com/vllm-project/vllm/issues/36337. The versions above illustrate a pin; they are not a recommendation.

AN-003 Determinism costs about a third Bit-identical outputs are available in vLLM today. Most workloads shouldn't pay for them.

Run the same prompt twice at temperature zero and you can get different answers, because the batch it lands in changes the order of floating-point reductions. Batch-invariant kernels fix that. vLLM ships them behind a flag, and its maintainers measured the cost in compiled mode:

RTX 4090 throughput: −29.9%
RTX 4090 latency: +73.3%
H20 throughput: −35.0%
H20 latency: +110.1%

Most of the added time is one persistent matmul kernel. Support is documented for NVIDIA and Intel XPU, with AMD listed as future work.

Recommendation. Decide per workload. RL rollouts and some verification setups need exact reproducibility. Chat and batch generation almost never do.

Source: github.com/vllm-project/vllm/issues/27433.

Start a conversation

Tell us the model, the hardware, the traffic and what hurts. An engineer reads it and usually replies with questions about your current numbers.

hi@wayfaster.org