Together AI Details Continual Compiler That Speeds LLM Inference Up to 2.6x

Together AI researchers introduce Together Compiler (TKCC), an agent-driven loop that continually re-optimizes inference engines as models and traffic change. Its Tiered Persistent Engine separates engine launch overhead from forward-pass computation, and a cost model selects parallelism configurations to seed optimization. On the InferenceX benchmark, the authors report up to 2.6x higher throughput than the best open-source engines on recent frontier models and up to 1.7x over hand-written megakernels on Llama 1B, with the system already serving trillions of tokens in production.

Why it matters: Engineers running LLM serving stacks get a concrete design for keeping inference kernels and engine configurations tuned as models and traffic shift, rather than relying on one-off manual optimization.