# LLM VRAM Calculator

> Estimates the GPU memory needed to run a transformer language model for inference from its parameter count and bits per weight (weights), layer count, hidden size, context length and batch size (KV cache), plus a percentage overhead for activations and the CUDA context, and lists which common GPU sizes it fits.

- Calculator id: `llm-vram-estimate` · Category: Developer & IT (`developer`) · Tool name: `estimate_llm_vram`
- Canonical page: https://tttkmbb.com/developer/llm-vram-estimate · This document: https://tttkmbb.com/developer/llm-vram-estimate.md · JSON definition: https://tttkmbb.com/developer/llm-vram-estimate.json

## Purpose

Estimates the GPU memory needed to run a transformer language model for inference from its parameter count and bits per weight (weights), layer count, hidden size, context length and batch size (KV cache), plus a percentage overhead for activations and the CUDA context, and lists which common GPU sizes it fits.

**Use when:** You want to know whether a model of N billion parameters fits a given GPU at 4-, 8- or 16-bit precision, or how much the KV cache grows with context length.

**Do not use when:** You are sizing training memory (optimizer states and gradients need roughly 3–4× the weight memory in mixed precision) or the model uses mixture-of-experts offloading; the estimate ignores framework-specific allocators.

## Input

| Parameter | Type | Unit | Required | Description |
| --- | --- | --- | --- | --- |
| `parameters_billions` | number | billion | required | Model size in billions of parameters (7 for a 7B model). (> 0, max 100000) |
| `bits_per_weight` | number |  | optional, default 16 | Weight precision: 16 for FP16/BF16, 8 for INT8/FP8, 4 for 4-bit quantization (GPTQ/AWQ/Q4), 32 for FP32; fractional values such as 4.5 (Q4_K_M) are allowed. (min 1, max 64) |
| `context_tokens` | integer | tokens | optional, default 4096 | Tokens held in the KV cache per sequence (prompt + generated). (min 1, max 10000000) |
| `batch_size` | integer |  | optional, default 1 | Concurrent sequences. (min 1, max 4096) |
| `num_layers` | integer |  | optional | Number of decoder layers (needed for the KV cache). (min 1, max 1000) |
| `hidden_size` | integer |  | optional | Model dimension d_model; for grouped-query attention use num_kv_heads × head_dim instead. (min 1, max 100000) |
| `kv_bits` | number |  | optional, default 16 | Precision of the cached keys and values (16 for FP16, 8 for FP8/INT8 cache). (min 1, max 32) |
| `overhead_percent` | number | % | optional, default 20 | Extra memory for activations, CUDA context and allocator fragmentation, as a percentage of weights + KV cache. (min 0, max 500) |

## Output

| Field | Type | Unit | Description |
| --- | --- | --- | --- |
| `weights_gb` | number | GB | parameters × bits_per_weight / 8, in gigabytes (10⁹ bytes). |
| `kv_cache_gb` | number | GB | 2 × layers × hidden_size × context_tokens × batch_size × kv_bits / 8; 0 when layers or hidden size are not given. |
| `overhead_gb` | number | GB | (weights + KV cache) × overhead_percent / 100. |
| `total_gb` | number | GB | weights + KV cache + overhead, gigabytes (10⁹ bytes). |
| `total_gib` | number | GiB | Total in gibibytes (2³⁰ bytes), the unit GPU memory is actually specified in. |
| `kv_cache_per_token_kb` | number | kB | 2 × layers × hidden_size × kv_bits / 8 per token, in kilobytes. |
| `fits_gpu` | string |  | Which of the common 8 / 12 / 16 / 24 / 32 / 48 / 80 GB (GiB) single cards can hold the total. |
| `gpus_80gb_needed` | integer |  | ceil(total_gib / 80): number of 80 GB accelerators for tensor-parallel serving, ignoring communication buffers. |

## Formula

`weights_gb = parameters_billions × bits_per_weight / 8; kv_cache_gb = 2 × num_layers × hidden_size × context_tokens × batch_size × kv_bits / 8 / 10⁹; overhead_gb = (weights_gb + kv_cache_gb) × overhead_percent / 100; total_gb = weights_gb + kv_cache_gb + overhead_gb`

Standard inference sizing: weights scale with parameters × precision, the KV cache stores one key and one value vector per layer per token (multi-head attention; for GQA models pass num_kv_heads × head_dim as hidden_size). Embedding/vocabulary matrices are included in the parameter count; quantized models keep some tensors at higher precision, so real files are 5–15 % larger than the ideal figure.

## Data Sources

- Pope et al. (2022) Efficiently Scaling Transformer Inference (arXiv:2211.05102) — https://arxiv.org/abs/2211.05102 (peer_reviewed, retrieved 2026-09-24)
- Kwon et al. (2023) Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.06180) — https://arxiv.org/abs/2309.06180 (peer_reviewed, retrieved 2026-09-24)

Data freshness: `static`. Deterministic formula with fixed constants; results never go stale. Inputs supplied by the caller determine the output.

## API

- `GET https://tttkmbb.com/api/v1/calculate/llm-vram-estimate?parameters_billions=…`
- `POST https://tttkmbb.com/api/v1/calculate/llm-vram-estimate` with JSON body `{"inputs": {…}}`
- Response: unified envelope (`success`, `request`, `result.values`, `result.units`, `sources`, `freshness`, `timestamp`, `next_actions`, `links`); see https://tttkmbb.com/docs/response-format.md
- Schema: https://tttkmbb.com/api/v1/calculators/llm-vram-estimate · OpenAPI operationId `estimate_llm_vram` in https://tttkmbb.com/openapi.json
- Authentication: none. Rate limit: fair use, see https://tttkmbb.com/docs/rate-limits.md.

## MCP

- Server: `https://tttkmbb.com/mcp` (Streamable HTTP, JSON-RPC 2.0, no auth)
- Tool:  `run_calculator` with `{"calculator_id": "llm-vram-estimate", "inputs": {…}}`

## Example

- 7B at FP16 without KV cache detail: inputs `{"parameters_billions":7,"bits_per_weight":16}` → `{"weights_gb":14,"kv_cache_gb":0,"overhead_gb":2.8,"total_gb":16.8,"total_gib":15.646,"fits_gpu":"16 GB, 24 GB, 32 GB, 48 GB, 80 GB"}`
- 70B at 4-bit, 80 layers, hidden 8192, 4096 context: inputs `{"parameters_billions":70,"bits_per_weight":4,"num_layers":80,"hidden_size":8192,"context_tokens":4096,"batch_size":1}` → `{"weights_gb":35,"kv_cache_gb":10.737,"overhead_gb":9.147,"total_gb":54.885,"total_gib":51.116,"gpus_80gb_needed":1}`

```
GET https://tttkmbb.com/api/v1/calculate/llm-vram-estimate?parameters_billions=7&bits_per_weight=16
```

## Limitations

You are sizing training memory (optimizer states and gradients need roughly 3–4× the weight memory in mixed precision) or the model uses mixture-of-experts offloading; the estimate ignores framework-specific allocators. Standard inference sizing: weights scale with parameters × precision, the KV cache stores one key and one value vector per layer per token (multi-head attention; for GQA models pass num_kv_heads × head_dim as hidden_size). Embedding/vocabulary matrices are included in the parameter count; quantized models keep some tensors at higher precision, so real files are 5–15 % larger than the ideal figure. All values are computed from the formula above; no measurement or live data is involved.

## FAQ

**Why is the KV cache so large for a 70B model?**

Every token keeps a key and a value vector in every layer: 2 × 80 layers × 8192 × 2 bytes ≈ 2.6 MB per token, so 4,096 tokens need 10.7 GB. Grouped-query attention (Llama 3 70B: 8 KV heads × 128 = 1024) cuts this eightfold; pass that value as hidden_size.

**Does a 7B model fit in 8 GB?**

At 4 bits the weights are 3.5 GB and the total about 4.2 GB, so yes; at FP16 it needs 14 GB of weights alone.

**What about training?**

Full fine-tuning in mixed precision needs about 16 bytes per parameter (weights, gradients, Adam moments) plus activations; this calculator covers inference only.

## Related

- [LLM Token Cost Calculator](https://tttkmbb.com/developer/llm-token-cost.md) — Cost of the same model served through a token-priced API.
- [Data Storage Converter](https://tttkmbb.com/conversion/data-storage.md) — Convert between GB and GiB.