Home › Developer & IT › LLM VRAM Calculator

LLM VRAM Calculator

Estimates the GPU memory needed to run a transformer language model for inference from its parameter count and bits per weight (weights), layer count, hidden size, context length and batch size (KV cache), plus a percentage overhead for activations and the CUDA context, and lists which common GPU sizes it fits.

Parameters (billion)Model size in billions of parameters (7 for a 7B model). Bits per weight (optional)Weight precision: 16 for FP16/BF16, 8 for INT8/FP8, 4 for 4-bit quantization (GPTQ/AWQ/Q4), 32 for FP32; fractional values such as 4.5 (Q4_K_M) are allowed. Context length (tokens) (optional)Tokens held in the KV cache per sequence (prompt + generated). Batch size (optional)Concurrent sequences. Transformer layers (optional)Number of decoder layers (needed for the KV cache). Hidden size (optional)Model dimension d_model; for grouped-query attention use num_kv_heads × head_dim instead. KV cache bits (optional)Precision of the cached keys and values (16 for FP16, 8 for FP8/INT8 cache). Overhead (%) (optional)Extra memory for activations, CUDA context and allocator fragmentation, as a percentage of weights + KV cache.

When to use

You want to know whether a model of N billion parameters fits a given GPU at 4-, 8- or 16-bit precision, or how much the KV cache grows with context length.

Do not use when: You are sizing training memory (optimizer states and gradients need roughly 3–4× the weight memory in mixed precision) or the model uses mixture-of-experts offloading; the estimate ignores framework-specific allocators.

Formula

weights_gb = parameters_billions × bits_per_weight / 8; kv_cache_gb = 2 × num_layers × hidden_size × context_tokens × batch_size × kv_bits / 8 / 10⁹; overhead_gb = (weights_gb + kv_cache_gb) × overhead_percent / 100; total_gb = weights_gb + kv_cache_gb + overhead_gb

Standard inference sizing: weights scale with parameters × precision, the KV cache stores one key and one value vector per layer per token (multi-head attention; for GQA models pass num_kv_heads × head_dim as hidden_size). Embedding/vocabulary matrices are included in the parameter count; quantized models keep some tensors at higher precision, so real files are 5–15 % larger than the ideal figure.

Inputs

Parameter	Type	Unit	Required	Description
`parameters_billions`	number	billion	yes	Model size in billions of parameters (7 for a 7B model). Range: > 0, ≤ 100000
`bits_per_weight`	number		default 16	Weight precision: 16 for FP16/BF16, 8 for INT8/FP8, 4 for 4-bit quantization (GPTQ/AWQ/Q4), 32 for FP32; fractional values such as 4.5 (Q4_K_M) are allowed. Range: ≥ 1, ≤ 64
`context_tokens`	integer	tokens	default 4096	Tokens held in the KV cache per sequence (prompt + generated). Range: ≥ 1, ≤ 10000000
`batch_size`	integer		default 1	Concurrent sequences. Range: ≥ 1, ≤ 4096
`num_layers`	integer		no	Number of decoder layers (needed for the KV cache). Range: ≥ 1, ≤ 1000
`hidden_size`	integer		no	Model dimension d_model; for grouped-query attention use num_kv_heads × head_dim instead. Range: ≥ 1, ≤ 100000
`kv_bits`	number		default 16	Precision of the cached keys and values (16 for FP16, 8 for FP8/INT8 cache). Range: ≥ 1, ≤ 32
`overhead_percent`	number	%	default 20	Extra memory for activations, CUDA context and allocator fragmentation, as a percentage of weights + KV cache. Range: ≥ 0, ≤ 500

Outputs

Output	Type	Unit	Description
`weights_gb`	number	GB	parameters × bits_per_weight / 8, in gigabytes (10⁹ bytes).
`kv_cache_gb`	number	GB	2 × layers × hidden_size × context_tokens × batch_size × kv_bits / 8; 0 when layers or hidden size are not given.
`overhead_gb`	number	GB	(weights + KV cache) × overhead_percent / 100.
`total_gb`	number	GB	weights + KV cache + overhead, gigabytes (10⁹ bytes).
`total_gib`	number	GiB	Total in gibibytes (2³⁰ bytes), the unit GPU memory is actually specified in.
`kv_cache_per_token_kb`	number	kB	2 × layers × hidden_size × kv_bits / 8 per token, in kilobytes.
`fits_gpu`	string		Which of the common 8 / 12 / 16 / 24 / 32 / 48 / 80 GB (GiB) single cards can hold the total.
`gpus_80gb_needed`	integer		ceil(total_gib / 80): number of 80 GB accelerators for tensor-parallel serving, ignoring communication buffers.

Example

7B at FP16 without KV cache detail: {"parameters_billions":7,"bits_per_weight":16} → {"weights_gb":14,"kv_cache_gb":0,"overhead_gb":2.8,"total_gb":16.8,"total_gib":15.646,"fits_gpu":"16 GB, 24 GB, 32 GB, 48 GB, 80 GB"}

70B at 4-bit, 80 layers, hidden 8192, 4096 context: {"parameters_billions":70,"bits_per_weight":4,"num_layers":80,"hidden_size":8192,"context_tokens":4096,"batch_size":1} → {"weights_gb":35,"kv_cache_gb":10.737,"overhead_gb":9.147,"total_gb":54.885,"total_gib":51.116,"gpus_80gb_needed":1}

GET https://tttkmbb.com/api/v1/calculate/llm-vram-estimate?parameters_billions=7&bits_per_weight=16

Machine access

API: GET https://tttkmbb.com/api/v1/calculate/llm-vram-estimate (query parameters) or POST with a JSON body {"inputs": {...}}
Schema: https://tttkmbb.com/api/v1/calculators/llm-vram-estimate · Markdown: https://tttkmbb.com/developer/llm-vram-estimate.md · JSON definition: https://tttkmbb.com/developer/llm-vram-estimate.json
MCP: server https://tttkmbb.com/mcp, tool run_calculator with calculator_id="llm-vram-estimate"
OpenAPI operationId: estimate_llm_vram
Freshness: static. Authentication: none. Rate limit: fair use (see rate limits).

Sources

FAQ

Why is the KV cache so large for a 70B model?

Every token keeps a key and a value vector in every layer: 2 × 80 layers × 8192 × 2 bytes ≈ 2.6 MB per token, so 4,096 tokens need 10.7 GB. Grouped-query attention (Llama 3 70B: 8 KV heads × 128 = 1024) cuts this eightfold; pass that value as hidden_size.

Does a 7B model fit in 8 GB?

At 4 bits the weights are 3.5 GB and the total about 4.2 GB, so yes; at FP16 it needs 14 GB of weights alone.

What about training?

Full fine-tuning in mixed precision needs about 16 bytes per parameter (weights, gradients, Adam moments) plus activations; this calculator covers inference only.

Related calculators

LLM Token Cost Calculator — Cost of the same model served through a token-priced API.
Data Storage Converter — Convert between GB and GiB.