Inference optimisation is a set of techniques to make LLM inference faster, cheaper and more efficient. It’s about reducing latency, improving throughput, and lowering hardware costs without hurting model quality [in](Inference optimization is a set of techniques to make LLM inference faster, cheaper, and more efficient. It’s about reducing latency, improving throughput, and lowering hardware costs without hurting model quality.) source
Some common strategies include Continuous Batching, KV Cache management, Speculative decoding, Quantisation, Prefix Caching, Multi-GPU distribution/Parallelism. We will learn about all this later and I’ll update the notes link once those notes are created