Start a project

AI compilers

Why your inference bill is a compiler problem

LLM inference is usually limited by memory bandwidth, not raw compute. That makes fusion, quantisation and kernel work the biggest levers on cost and latency.

6 min readNexaFusion

When an AI feature gets expensive, the first instinct is to look at the model: use a smaller one, or a cheaper provider. Sometimes that is right. But if you run models yourself, a large share of the cost is often in how the model runs — and that is a compiler and kernel problem.

Bytes, not FLOPs

Generating text with a large language model happens one token at a time. For each new token, the model’s weights have to be read from GPU memory. At small batch sizes, the arithmetic per byte read is low, so the GPU spends much of its time waiting on memory rather than computing. In roofline terms, decoding is usually memory-bandwidth bound.

That changes what “optimisation” means. Making the maths faster doesn’t help much. Moving fewer bytes does.

The levers that matter

Batching

Serving many requests together reuses each weight read across more tokens. Continuous batching — adding and removing requests from a running batch — is now standard in serious serving stacks.

Quantisation

Storing weights in 8-bit or 4-bit formats instead of 16-bit cuts the bytes moved per token. Done carefully, with evaluation, quality loss can be small. Done carelessly, it quietly degrades your product — which is why evals come first.

Kernel fusion

Every time an intermediate result is written to GPU memory and read back, you pay for bandwidth. Fusing operations keeps data in fast on-chip memory. FlashAttention is the best-known example: it computes attention without ever materialising the full attention matrix in GPU memory.

KV-cache management

The attention cache grows with context length and concurrency. Paging and sharing it efficiently lets you serve more users per GPU.

Speculative decoding

A small draft model proposes several tokens and the large model verifies them in one pass, turning sequential work into parallel work.

Where compilers come in

Modern ML compilers automate much of this. torch.compile, XLA and MLIR-based stacks capture the model graph, fuse operations and choose memory layouts. MLIR lets compilers lower a model step by step through dialects — from high-level tensor operations down to hardware instructions — which is also how new accelerators get supported. Triton lets engineers write custom GPU kernels at the level of tiles rather than threads, for the hot operations a general compiler doesn’t handle well.

A sane process

  1. Measure the right thing: cost per thousand tokens and p95 latency at realistic load — not a single-request benchmark.
  2. Profile: framework profilers and Nsight Systems / Nsight Compute show where time goes, kernel by kernel.
  3. Classify: compute-bound, memory-bound or launch-overhead-bound? Each needs a different fix.
  4. Fix the top items: serving configuration first, then compilation, then custom kernels for what remains.
  5. Guard the gains: benchmarks in CI so the next model or library upgrade doesn’t erase them.

Teams that work this way often find substantial savings on the same hardware — and they understand why, which means the savings last.

More about AI compilers & performance →