ML Serving, Inference, and vLLM

Serving is where model behavior meets latency, cost, GPU memory, API compatibility, traffic shape, and rollback. For LLMs, inference has two very different phases: prefill processes the prompt and builds KV cache, while decode generates new tokens one step at a time. A system can have healthy GPUs and still fail user expectations if queue time, time to first token, inter-token latency, or context limits are wrong.

Command Examples

python -c "import torch; print(torch.cuda.is_available())"
nvidia-smi
curl -s http://localhost:8000/v1/models
curl -s http://localhost:8000/metrics | grep 'vllm:'

Example output and meaning:

These checks prove accelerator visibility, API reachability, and metrics exposure. They do not prove capacity, quality, safety, or cost.

Plain Inference vs vLLM Inference

Inference is the act of using trained weights to produce predictions or generated tokens. “vLLM inference” is not a different kind of model intelligence; it is inference executed through the vLLM runtime, scheduler, memory manager, kernels, API server, and metrics surface. The model, tokenizer, prompt, sampling settings, and adapters still define the behavior contract. vLLM changes how requests are packed onto hardware and observed in production.

Debugging starts at different layers. If a single deterministic prompt gives the wrong answer in both PyTorch/Hugging Face and vLLM, suspect model weights, tokenizer, prompt, adapter, or sampling config. If the answer is good in a single-call test but production vLLM traffic has high tail latency, suspect scheduler pressure, request shape mix, KV cache, batching, or hardware saturation.

Serving Mental Model

flowchart LR
  Client[Client / SDK] --> Gateway[API gateway / auth / rate limits]
  Gateway --> Router[Model router]
  Router --> Queue[Admission queue]
  Queue --> Prefill[Prefill prompt tokens]
  Prefill --> KV[KV cache]
  KV --> Decode[Decode output tokens]
  Decode --> Stream[Streaming response]
  Stream --> Client
  Router --> Metrics[Metrics and traces]

Core Inference Metrics

Batching, Streaming, and KV Cache

Traditional static batching waits to collect requests, then runs them together. LLM serving often uses continuous batching: new requests enter the active batch as other requests finish. This improves GPU utilization, but it means request latency depends on token lengths, scheduling, and memory pressure.

Prefill vs Decode

KV-Cache Deep Dive

The KV cache is the stored key and value tensors produced by transformer attention layers during autoregressive inference. It exists because decoder-only LLMs generate one token at a time. Each new token needs to attend to previous tokens, but the keys and values for those previous tokens do not change once they have been computed. Caching them avoids rerunning the full previous context on every decode step.

In a self-attention layer, each token representation is projected into query, key, and value tensors:

• query (Q) asks what the current position should attend to,
• key (K) describes what each previous position offers for matching,
• value (V) is the information read when attention selects that position.

For the next generated token, the model only needs a new query for the current position plus the cached keys and values for prior positions. The model appends the new token’s keys and values to the cache, then repeats the process for the next token.

Without a KV cache, generating token 1, token 2, token 3, and so on would repeatedly recompute attention state for the same earlier context. With a KV cache, the system pays the prompt prefill cost once, then each decode step extends the cached state. This is why KV cache is central to LLM serving performance.

Approximate KV-cache memory per active sequence:

layers * active_tokens * 2(K,V) * kv_heads * head_dim * bytes_per_value

For a rough fp16 multi-head attention example with 32 layers, 32 KV heads, and head dimension 128:

32 layers * 8192 tokens * 2 * 32 heads * 128 dim * 2 bytes
~= 4 GiB for one long active sequence

Grouped-query attention and multi-query attention reduce this by using fewer KV heads than query heads, but long context and high concurrency can still dominate memory. Allocator overhead, block size, padding, fragmentation, prefix sharing, quantized KV cache, and engine-specific layout also affect the real number.

Operationally, treat KV cache as a first-class capacity resource, not a hidden implementation detail. A serving stack can have model weights loaded and GPU utilization below 100 percent but still reject or delay requests because KV cache is full. Track prompt-token distribution, output-token distribution, active sequences, queue time, time to first token, inter-token latency, vllm:kv_cache_usage_perc, and vllm:num_preemptions together.

Common mistakes:

• confusing KV cache with application response caching; KV cache stores internal attention tensors, not final answers,
• assuming it persists conversation memory across requests; clients still need to send the conversation history unless the serving layer explicitly supports reusable prefixes,
• treating high GPU utilization as proof of healthy serving; KV-cache pressure can cause tail latency before GPU compute saturates,
• increasing --max-model-len without reducing concurrency or adding memory,
• enabling prefix caching without checking whether prompts actually share stable token prefixes,
• changing tokenizer, chat template, model, LoRA adapter, or sampling setup without retesting cache reuse and output compatibility.

PagedAttention Deep Dive

PagedAttention is vLLM’s KV-cache memory-management technique for transformer inference. The name is an operating-system analogy: instead of requiring each request’s KV cache to live in one large contiguous GPU allocation, vLLM splits the cache into fixed-size blocks and maps a request’s logical token positions to physical blocks through a block table. The attention kernel follows that table when reading past keys and values.

The problem it solves is not “how does attention work mathematically.” It solves the serving-layer memory problem created by autoregressive decoding:

• every active sequence needs KV cache for previously processed tokens,
• request lengths vary widely,
• prompts and outputs grow over time,
• requests finish at different moments,
• reserving a worst-case contiguous buffer wastes memory,
• fragmented free memory can prevent admitting new requests even when total free memory looks adequate.

PagedAttention makes KV allocation more like paged virtual memory. A request receives blocks as it needs them during prefill and decode. Its logical context can span many physical blocks that are not adjacent in GPU memory. When a request finishes, its blocks can be returned to the free pool. This lowers internal fragmentation and lets the scheduler keep more useful work resident on the GPU.

Why it matters for inference:

Important limits:

• PagedAttention does not change model weights, tokenizer behavior, sampling semantics, or output quality by itself.
• It does not make attention compute free; long context still increases prefill work and decode memory bandwidth.
• It primarily improves KV-cache placement and sharing, not the model’s reasoning ability.
• Very high --max-model-len values still reserve capacity expectations and can reduce achievable concurrency.
• Prefix caching and PagedAttention are related but different: PagedAttention manages KV blocks; prefix caching decides when repeated prompt prefixes can reuse existing KV.
• The right tuning is workload-specific. Watch vllm:kv_cache_usage_perc, vllm:num_preemptions, queue time, TTFT, inter-token latency, prompt-token histograms, and output-token histograms together.

vLLM Runbook

vLLM is an LLM inference and serving engine focused on high-throughput serving, PagedAttention KV-cache management, continuous batching, OpenAI-compatible APIs, prefix caching, quantization, speculative decoding, parallelism, and production metrics.

Minimal local server:

vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key token-abc123

OpenAI-compatible request:

curl http://localhost:8000/v1/chat/completions \
  -H 'Authorization: Bearer token-abc123' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "NousResearch/Meta-Llama-3-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain prefill vs decode."}],
    "temperature": 0.2,
    "max_tokens": 256
  }'

Common vLLM serving controls:

vLLM Tuning Matrix

Deployment Patterns

Inference Optimization

Serving Incident Flow

1. Identify the failing route, model ID, adapter, runtime version, and request shape.
2. Split the symptom into queue, prefill, decode, streaming, API, or quality.
3. Compare prompt tokens, output tokens, TTFT, inter-token latency, and total latency.
4. Check KV cache utilization, waiting requests, preemptions, and GPU memory.
5. Compare canary and baseline metrics by tenant and prompt length.
6. Roll back model, runtime, quantization, or generation config if release-linked.
7. Add the request shape to serving load tests and eval gates.

Study Cards

References

• vLLM documentation
• vLLM OpenAI-Compatible Server
• vLLM Production Metrics
• vLLM Automatic Prefix Caching
• vLLM Speculative Decoding
• Efficient Memory Management for Large Language Model Serving with PagedAttention

nvidia-smi dmon

# gpu   sm  mem  enc  dec  mclk  pclk
# Idx    %    %    %    %   MHz   MHz
  0     72   88    0    0  1593  1410

curl -sS http://localhost:8000/metrics | grep vllm

vllm:num_requests_waiting{model_name="llama"} 14
vllm:gpu_cache_usage_perc{model_name="llama"} 0.91
vllm:time_to_first_token_seconds_bucket{le="1.0"} 248

kubectl top pod -l app=vllm

NAME            CPU(cores)   MEMORY(bytes)
vllm-0          920m         38Gi