vLLM in Production — PagedAttention, Continuous Batching, and OpenAI-Compatible LLM Serving

Why Naive LLM Serving Wastes Your GPU

Serving a large language model looks simple until you watch the GPU meter. You load a 13-billion-parameter model onto an expensive accelerator, send a handful of concurrent requests, and throughput collapses. The card is barely warm, yet you are out of memory. The bottleneck is almost never raw compute — it is how the key-value cache is managed.

vLLM is a serving engine built to fix exactly that. It started as a research project at UC Berkeley and is now a broadly adopted, open-source inference server. Its headline idea, PagedAttention, borrows a trick from operating systems to stop the KV cache from wasting most of your GPU memory.

This guide walks the memory problem, the paging solution, and then gets hands-on: offline batch inference, the OpenAI-compatible server, tensor parallelism for models that outgrow one card, quantization, and structured outputs. If you are weighing hosted GPUs against self-hosting, our note on serverless ML on Modal is a useful companion.

The KV Cache and the 60–80% Memory Tax

When a transformer generates text, each new token attends to every previous token. To avoid recomputing attention keys and values on every step, the model caches them — the KV cache. That cache grows with sequence length and is stored in GPU memory alongside the model weights.

The cache is large and, worse, its size is unknown up front: you do not know how long a response will be until it finishes. The original vLLM announcement reports the KV cache can reach up to 1.7 GB per sequence for LLaMA-13B.

Traditional serving systems allocate one contiguous block of memory per request, sized for the maximum possible length. Most requests never fill it, so the reserved-but-unused space is dead weight. The vLLM team measured the damage: existing systems “waste 60% – 80% of memory due to fragmentation and over-reservation.” That is why the GPU runs out of room while sitting nearly idle.

PagedAttention — Virtual Memory for the KV Cache

PagedAttention is described as “inspired by the classic idea of virtual memory and paging in operating systems.” Instead of one contiguous allocation per request, it splits each sequence’s KV cache into fixed-size blocks that can live anywhere in GPU memory, non-contiguously.

The analogy is exact: blocks act like pages, tokens like bytes, and sequences like processes. A per-sequence block table maps logical blocks to physical blocks, so the engine grows a sequence one block at a time as tokens arrive. There is no need to guess the final length or reserve for it.

The payoff is dramatic. Because waste is confined to the last partially-filled block of each sequence, vLLM reports “near-optimal memory usage, with a mere waste of under 4%.” Recovering that memory is what lets the engine keep more sequences resident at once.

Continuous Batching Keeps the GPU Full

Static batching waits for a whole batch to finish before starting the next, so one long generation stalls every short one beside it. vLLM instead schedules at the iteration level: as soon as any sequence emits its end-of-sequence token and leaves, a queued request takes its slot in the very next step.

Freeing memory efficiently is what makes this practical. Because PagedAttention keeps waste under 4%, the scheduler can batch more sequences together, push GPU utilization up, and raise throughput. The two features reinforce each other — paging supplies the free memory, continuous batching spends it.

The measured effect is large. Against Hugging Face Transformers, the announcement reports up to 24x higher throughput; against Hugging Face Text Generation Inference, up to 3.5x. Those figures come from a specific benchmark — LLaMA-7B on an NVIDIA A10G and LLaMA-13B on an A100 40GB with ShareGPT traffic — so treat them as directional and measure your own workload.

Offline Inference — LLM and SamplingParams

The fastest way to feel the engine is offline batch inference: load a model once and generate over a list of prompts. The quickstart installs vLLM into a fresh virtual environment with uv, letting the tool pick the right PyTorch backend automatically.

# Create an isolated environment and install vLLM
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

The offline API is two classes: LLM loads the model, and SamplingParams configures decoding. A single generate call runs the whole list through continuous batching for you.

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="facebook/opt-125m")

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated: {generated_text!r}")

The OpenAI-Compatible Server

In production you want a long-lived HTTP service, not a script. The vllm serve command starts a server that implements the OpenAI REST API, so any existing OpenAI client or framework can point at it with a changed base URL. It listens on http://localhost:8000 by default.

# Start an OpenAI-compatible server
vllm serve Qwen/Qwen2.5-1.5B-Instruct

# List the served models
curl http://localhost:8000/v1/models

Both classic completion and chat endpoints are available. The model field must match the served model’s identifier — the same string you passed to vllm serve.

# Chat completion over HTTP
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Explain PagedAttention in one sentence."}
        ]
    }'

Authentication is optional. Pass --api-key on the command line, or set the VLLM_API_KEY environment variable, and the server will require a matching bearer token. This matters once the endpoint is reachable beyond localhost.

Querying With the OpenAI Python Client

Because the API matches OpenAI’s, the official openai package works unchanged. You only override base_url and supply a placeholder key when auth is disabled. This is what makes migration from a hosted API to a self-hosted one nearly free at the call site.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="-",  # any value when the server runs without --api-key
)

# Discover the served model id at runtime
model = client.models.list().data[0].id

completion = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "user", "content": "Summarize continuous batching."}
    ],
)
print(completion.choices[0].message.content)

The same substitution works from LangChain, LlamaIndex, or your own service layer. If you are wiring vLLM behind an agent runtime, the tool-calling and streaming semantics also mirror OpenAI’s — see our guide on building AI agents that actually work for how those pieces fit into an orchestration loop.

Scaling Up — Tensor Parallelism and Engine Args

A 70B model does not fit on one card. vLLM shards the model across GPUs with tensor parallelism, set by --tensor-parallel-size (alias -tp), which defaults to 1. For multi-node deployments, combine it with --pipeline-parallel-size (default 1).

# Shard one model across four GPUs on a single node
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90

Two engine args deserve attention. --gpu-memory-utilization is the fraction of each GPU vLLM may claim, defaulting to 0.92; the rest of that memory becomes KV-cache blocks, so lowering it shrinks your effective batch size. --max-model-len caps the context window; if unset it is derived from the model config, and setting it lower frees memory for more concurrent sequences.

Quantization to Fit Bigger Models

Quantization stores weights in fewer bits, cutting memory and often improving throughput at some accuracy cost. vLLM selects a method through --quantization (alias -q), which has no default and is otherwise inferred from the model’s own config. The precision of activations and unquantized weights is controlled separately by --dtype, which defaults to auto.

# Serve a pre-quantized checkpoint; dtype stays auto
vllm serve TheBloke/Llama-2-13B-AWQ \
    --quantization awq \
    --dtype auto

The cleanest path is a checkpoint already published in a supported format, so no conversion step is needed at deploy time. Because accuracy trade-offs are model- and task-specific, validate a quantized model on your own evaluation set before trusting it — the same discipline we describe in LLM evaluation in production.

Structured Outputs — Constrained Decoding

When a downstream system needs valid JSON or one of a fixed set of labels, hoping the model complies is not a plan. vLLM constrains generation so the output must match a schema, a regex, a set of choices, or a grammar. You pass the constraint through extra_body on the OpenAI client.

completion = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
    ],
    extra_body={"structured_outputs": {"choice": ["positive", "negative"]}},
)
print(completion.choices[0].message.content)  # -> positive

The structured_outputs object also accepts json for a JSON schema, regex for a pattern, and grammar for a context-free grammar. The backend defaults to auto, picking an implementation for you.

Constrained decoding pairs naturally with the patterns in our prompt engineering for enterprise guide, where structured outputs and tool use are the load-bearing parts of a reliable pipeline.

Operating vLLM in Production

The server exposes Prometheus metrics for request latency, tokens per second, and — most importantly — KV-cache utilization and the number of running versus waiting sequences. Those two scheduler metrics tell you whether you are memory-bound or under-loaded, which is the first question to answer when tuning.

Watch time-to-first-token and inter-token latency separately from throughput; a batch size that maximizes tokens per second can quietly ruin the interactive experience. Feeding these signals into a broader monitoring stack is the subject of our note on LLM observability and monitoring.

Production Checklist

• Pin the version. Flags and defaults shift between releases — the guided_* removal in v0.12.0 is one example. Pin an exact vLLM version and re-test on upgrade.
• Size the KV-cache budget. Tune --gpu-memory-utilization and --max-model-len together against real prompt and output lengths, not the model maximum.
• Benchmark your workload. The 24x-vs-HF and 3.5x-vs-TGI figures come from one ShareGPT benchmark; measure with your own traffic before promising numbers.
• Match parallelism to hardware. Use --tensor-parallel-size within a node and add --pipeline-parallel-size only when crossing nodes.
• Constrain critical outputs. Use structured_outputs for anything a downstream system parses, and validate quantized checkpoints on your eval set.
• Protect the endpoint. Set --api-key or VLLM_API_KEY and put the server behind a gateway before it leaves localhost.
• Watch scheduler metrics. Alert on KV-cache utilization and the waiting-sequence count to catch saturation before latency spikes.

vLLM’s bet is that most LLM serving pain is a memory-management problem, not a compute problem — and that treating the KV cache like OS virtual memory recovers the throughput a naive server throws away. If your GPUs are expensive and your utilization is embarrassing, that trade is worth a proof of concept.

For fine-tuning the open models you deploy with vLLM — LoRA adapters, QLoRA quantisation, and PEFT-based domain adaptation — see the fine-tuning open models guide. When vLLM inference results need to be logged and compared across model versions, integrate with MLflow experiment tracking to store per-request latency, token counts, and output samples alongside the model artifact. For Kubernetes deployments of vLLM that autoscale GPU nodes on request queue depth, see Karpenter autoscaling — GPU node pools with spot instances reduce inference serving costs by 60–70% compared to on-demand provisioning.