Inference: Prefill and Decode

No conversation about inference can happen without these two important operations, i.e Prefill and Decode. If you have not yet read my previous blog on Inference: core loop of an inference engine you should go check it out. It sets up the base for understanding what does a core loop of inference does at it very core.

Intuition behind needing these operations: You see in a GPU which is a graphics processing unit, cores are at a premium, the cores are what do the heavy operations inside, read the matrix multiplications i.e the very core operation inside an LLM. Keeping the internal architecture of the LLM for a different discussion i want you to understand that Prefill is an expensive operation it is the operation that uses a lot of the GPU to run the prompt tokens through the model using the existing weights to compute the hidden states its attention outputs and most importantly, keys and values for the KV cache

Decode on the other hand is an operation that is sequential because it is generate for every token and hence, its less parallel and usually uses one token at a time and is often more memory-bound rather than compute bound

There is an important pre-req that i have left out of this discussion is the KV cache, but I feel I would be doing injustice to this entire concept if I explained that earlier. We would need to spend some time understanding the internal workings of a an LLM first

Why cant they be run concurrently ? For a single request, prefill must happen before decode because decode depends on the prompt’s computed state and across many requests, inference engines may interleave prefill and decode, but doing so efficiently is hard because the two phases stress the GPU differently and therefore we need to

Diagram: