Why your inference bill is a compiler problem
LLM inference is usually limited by memory bandwidth, not raw compute. That makes fusion, quantisation and kernel work the biggest levers on cost and latency.
Service
Compiler passes, custom kernels and profiling from the model graph down to the silicon — so inference and training run faster on the hardware you already pay for.
Trace from framework to kernel to find whether you are compute-, memory- or launch-bound.
Fusion, layout and quantisation passes with torch.compile, XLA or MLIR-based stacks.
Triton or CUDA kernels for the operations that dominate your profile.
Batching, KV-cache management and speculative decoding for LLM inference.
MLIR / LLVM lowering for new hardware targets.
Benchmarks and performance budgets in CI so gains don’t quietly erode.