How Much VRAM Do You Need to Run an LLM Locally?
· updated · by Andergrove Software
Running an open model locally, with Ollama, LM Studio, llama.cpp or vLLM, comes down to one question: will it fit on my GPU? Get it right and the model runs fast, entirely on the card. Get it wrong and it either refuses to load or spills into system RAM and crawls.
VRAM use is mostly predictable from a few numbers, which is what makes this answerable rather than trial and error. Here is where the memory actually goes, what quantization saves you, and a rough sizing table. Plug your own model into the LLM GPU Memory (VRAM) Calculator for an exact estimate.
The three things that eat VRAM
When you load a model for inference, the card's memory goes to three things:
- Model weights. The parameters themselves, and by far the biggest chunk for short prompts. Size depends on the parameter count and the precision (more on that below).
- The KV cache. As the model generates, it caches a key and value vector for every token in the context, on every layer. This grows with your context length, so a long prompt or a long chat can use gigabytes here on its own.
- Overhead. Activations, the CUDA context, framework buffers, and fragmentation. Budget roughly 1 to 2 GB on top of everything else.
Total VRAM is weights plus KV cache plus overhead. The weights are fixed once you pick a model and precision. The KV cache is the part people forget.
Quantization: the size dial
A model's weights can be stored at different precisions, and that choice scales the weight memory almost linearly:
| Precision | Bytes per parameter | 7B model | 13B model | 70B model |
|---|---|---|---|---|
| FP16 / BF16 (full) | 2 | ~14 GB | ~26 GB | ~140 GB |
| 8-bit (Q8) | ~1 | ~7 GB | ~13 GB | ~70 GB |
| 4-bit (Q4) | ~0.5 | ~3.5 GB | ~6.5 GB | ~35 GB |
The short version: 4-bit quantization roughly quarters the weight memory versus full precision, which is what makes big models runnable on consumer cards at all. The cost is a small quality loss. It's usually minor at 8-bit and modest at 4-bit, and often well worth the memory you get back. Below 4-bit (3-bit, 2-bit) the quality drop becomes noticeable.
A handy mental formula: weight VRAM is roughly parameters (in billions) times bytes-per-param. A 13B model at 4-bit is 13 × 0.5, so about 6.5 GB of weights.
Decoding GGUF quant names: Q4_K_M and friends
If you download models for llama.cpp, Ollama or LM Studio, you'll meet names like
Q4_K_M, Q5_K_S or Q6_K. The pattern: the number after
Q is the approximate bits per weight (the size dial), K marks the
newer "k-quant" scheme that spends a few extra bits on the layers that hurt most when
compressed, and the final letter is the size/quality trade within that bit level —
Small, Medium, Large.
- Q4_K_M is the community default for a reason: close to Q5 quality at close to Q4 size.
- Q5_K_M / Q6_K are the "I have a little headroom" picks — noticeably closer to full quality, ~25–50% more memory than Q4.
- Q3 and below are last resorts for squeezing a model onto a too-small card; expect visible degradation, especially on reasoning and code.
- Q8_0 is nearly lossless and rarely worth it for inference — if you can afford Q8 on an N-billion model, you can usually afford Q4/Q5 on a bigger model, which is almost always the better trade.
That last point generalises: a bigger model at 4-bit usually beats a smaller model at 8-bit for the same memory. Parameter count buys more capability than precision does.
Why context length costs memory too
The KV cache is the sneaky one. Its size grows with the context window you actually use:
KV cache ≈ 2 × layers × context length × key/value dimension × bytes-per-valueThe exact figure depends on the model's architecture (number of layers and key/value heads), but the shape is what matters. Double the context, double the cache. On a large model with a long context, the KV cache can rival or exceed the weights. That's why a model that loads fine on a short prompt can run out of memory deep into a long conversation, and why "how much VRAM" has no single answer until you fix a context length.
To make it concrete, here is the FP16 KV cache for two typical modern architectures (both using grouped-query attention, which most current open models do):
| Context used | Typical 8B-class model | Typical 70B-class model |
|---|---|---|
| 2k tokens | ~0.25 GB | ~0.6 GB |
| 8k tokens | ~1 GB | ~2.5 GB |
| 32k tokens | ~4 GB | ~10 GB |
| 128k tokens | ~16 GB | ~40 GB |
Two takeaways. First, "supports 128k context" does not mean you can afford 128k — at full stretch the cache alone can outweigh the quantized model. Second, older models without grouped-query attention cache every attention head and can use several times these figures; it's one of the quiet reasons newer models run long chats so much better. Most runtimes also let you quantize the KV cache itself to 8-bit, roughly halving these numbers at little cost.
Mixture-of-experts models: active ≠ resident
Mixture-of-experts (MoE) models advertise a small "active parameters" number — only a few experts run per token — but every expert must still sit in memory, because any of them might be routed to on the next token. For VRAM sizing, use the total parameter count, not the active one. The active count predicts speed, not fit. An MoE with 8×7B experts needs the memory of a ~47B model even though each token only exercises a fraction of it.
"Will it fit?" by GPU
Rough guidance for inference at 4-bit, leaving headroom for a reasonable context. Treat these as starting points, not guarantees, since the exact fit depends on the model and your context length:
| Your GPU VRAM | Comfortable | Tight / short context |
|---|---|---|
| 8 GB | 7 to 8B at Q4 | 13B at Q4 |
| 12 GB | 13B at Q4 | 14 to 20B at Q4 |
| 16 GB | 13B at Q8, 20B at Q4 | 32B at Q4 |
| 24 GB | 32 to 34B at Q4 | 70B at low-bit + short context |
| 48 GB+ | 70B at Q4 | larger / longer context |
If you're picking hardware, 24 GB (an RTX 3090 or 4090, for example) is the sweet spot for hobbyist local LLMs. It comfortably runs the popular mid-size models with room for context.
Apple Silicon and unified memory
Macs play by different rules: there is no separate VRAM, just one pool of unified memory shared between CPU and GPU. That has made higher-RAM Macs genuinely popular local-LLM machines — a 64 GB machine can hold a 4-bit 70B-class model that no single consumer GPU can. Two caveats: by default macOS only lets the GPU claim a portion of total RAM (roughly two-thirds to three-quarters, depending on the machine), so budget against that number, not the sticker RAM; and memory bandwidth, not capacity, sets the token speed — a big model may fit and still generate at a leisurely pace compared to a discrete GPU.
Three worked examples
- 8 GB card, 8B-class model at Q4_K_M: ~4.5 GB weights + ~1 GB KV at 8k context + ~1.5 GB overhead ≈ 7 GB. Fits, with the desktop's own display memory as the margin. This is why 7–8B at 4-bit is the standard recommendation for 8 GB GPUs.
- 24 GB card, 32B-class model at Q4_K_M: ~18 GB weights + ~2 GB KV at 16k + ~1.5 GB overhead ≈ 21.5 GB. Comfortable, with room to push context further or step up to Q5 at a shorter context.
- 24 GB card, 70B-class model at Q4: ~35 GB of weights alone — it does not fit, full stop. Your options: offload roughly half the layers to CPU (expect single-digit tokens per second), drop to a heavily degraded 2-bit quant, add a second card, or run the 32B-class model instead, which will usually be both faster and smarter than a crippled 70B.
When it doesn't fit: CPU offloading
If a model is slightly too big, runtimes like llama.cpp can offload some layers to system RAM and run them on the CPU. It works, but every offloaded layer is much slower than GPU compute, so tokens-per-second drops sharply as soon as you spill out of VRAM. A model that's 90% on the GPU is usually fine. One that's half offloaded often isn't worth it. The goal is almost always to fit the whole model, plus your working context, in VRAM.
Mistakes that waste an afternoon
- Reading the download size as the memory cost. The GGUF file size is close to the weight memory, but it excludes the KV cache and overhead — the two things that actually push you over the edge.
- Sizing for the model's maximum context. Some runtimes pre-allocate the KV cache for whatever context limit is configured. If loading fails or memory looks bloated, check the context setting before blaming the model.
- Leaving no headroom. Your OS and browser hold VRAM too — a "16 GB" card on a desktop with monitors attached realistically offers 14–15 GB. Fit to that.
- Assuming quantization ruins quality. For chat and general use, a good 4-bit k-quant is much closer to the original than people expect; run your own prompts side by side before paying for more VRAM.
Putting it together
To size a local LLM:
- Weights: parameters times bytes-per-param for your chosen precision (Q4 is about 0.5).
- KV cache: scale with the context length you actually need.
- Overhead: add 1 to 2 GB.
- Compare the total to your card's VRAM, and leave a little headroom.
Rather than do that by hand for every model and quantization level, drop the parameter count, precision and context into the VRAM Calculator. It adds up the weights, KV cache and overhead and tells you whether it fits your GPU. Pair it with the LLM Token Counter when you're working out how long your prompts, and therefore your KV cache, really are.