Self-hosting an LLM means running inference on hardware the operator controls rather than via a third-party API, which changes the cost, data control, and privacy profile. Whether a model runs at all depends on memory.
LLM Compatibility Calculator
The calculator estimates the VRAM or unified memory a model needs to run locally, based on the model, its precision, the context length, and the target hardware. It returns whether a configuration fits, the memory split across weights, KV cache, and overhead, and the models a given GPU or Mac can run. The quantization formats and precision widths follow the Hugging Face Transformers documentation. 1
See our self-hosted LLM VRAM calculation methodology for the full math behind these estimates.
Hardware for self-hosting: GPUs and Apple Silicon
Two numbers decide the hardware. Capacity is the fit gate, and decode is memory-bandwidth-bound, so tokens per second scale roughly with a card’s GB/s.
The rows above sample each tier. The calculator carries 34 cards in all, adding the A100, L40S, RTX A6000, AMD Radeon RX 7900 XTX, and the AMD Instinct line. On 24 to 32 GB, a 70B model needs heavy quantization plus short context, or two cards. A 671B model needs 640 GB (8×80 GB) as the standard minimum. The RTX PRO 6000 is the top workstation-class single card, and its street price climbed to about $13,000 by mid-2026 from an $8,565 MSRP.2
Apple Silicon and unified memory: On Apple Silicon, the CPU and GPU share one memory pool, and the GPU addresses about 75% of total RAM by default through Metal’s recommendedMaxWorkingSetSize, a little lower on smaller Macs. The cap is raisable with sudo sysctl iogpu.wired_limit_mb=, leaving 8 to 16 GB for macOS. 3 4 A 128 GB Mac exposes about 96 GB to the GPU. The M3 Ultra, at up to 512 GB, exposes about 384 GB, enough for a 400B-class model at 4-bit with KV headroom on a single device with no multi-GPU replication tax. 5 That single-device path for large models is Apple’s edge, though AMD Strix Halo (128 GB) and NVIDIA DGX Spark (128 GB) now offer cheap large unified memory as well. 6
LLM serving engines
The serving-engine choice splits by workload rather than by a single best tool. Production-serving engines use paged attention and continuous batching for many concurrent users on datacenter GPUs, while local-first runtimes target one user on a desktop, laptop, or single GPU. The gap appears under load. At 64 concurrent requests, vLLM served about 44 times more tokens per second than llama.cpp, whose time-to-first-token blew past 3 minutes. For a single user the two are comparable. 7
The calculator models eight engines across the two camps:
Within the production tier, SGLang’s RadixAttention reuses cross-request prefixes, TensorRT-LLM freezes its activation budget when the engine is built, and LMDeploy serves W4A16 and MXFP4 cost-efficiently for InternLM, Qwen, and DeepSeek. 8 On the local side, ExLlamaV2 with TabbyAPI dials a fractional average bit-width to fill a card exactly, and MLX uses Apple unified memory so total RAM is the budget. Hugging Face TGI is the lone exit. It moved to maintenance mode in December 2025 and its GitHub repository was archived (read-only) on 21 March 2026, with users pointed to vLLM, SGLang, and llama.cpp. 9 10
Most people meet these engines through end-user apps that wrap them. Ollama, LM Studio, and AnythingLLM all run on llama.cpp (LM Studio also on MLX), adding one-command models, an OpenAI-compatible localhost API, and, for AnythingLLM, document RAG over PDFs and codebases. By GitHub stars as a rough adoption proxy on 2026-07-11, Ollama has 175,925 and vLLM 85,979, both open source, and AnythingLLM 63,120. LM Studio’s 5,053 comes from its open-source CLI repo (lmstudio-ai/lms) rather than the closed-source app, so it understates real desktop adoption. 11 12 13 14
An interactive landscape component maps these tools to use cases. Integrations and wide compatibility go to Ollama, developers and high performance to vLLM, local RAG applications to AnythingLLM, and beginner-friendly experimentation to LM Studio.
Open-source large language models
Open-weight models publish their architecture and weight files, so anyone can download, modify, and run them, usually from Hugging Face. The mid-2026 self-hostable frontier has moved well past the Qwen2.5 and Llama-3 era:
The total and active columns are the load-bearing distinction. Total parameters decide the memory and the number of GPUs, and active parameters decide the compute. DeepSeek-V4-Flash and GLM-5.2 arrived as open weights in 2026, Gemma 4 relicensed the family under Apache 2.0, and GPT-OSS ships MoE experts in native MXFP4. 15 16 17 18 19
Proprietary flagships (OpenAI’s GPT-5.6, Google’s Gemini 3.1 Pro, Anthropic’s Claude Opus 4.8, xAI’s Grok 4.5) cannot be downloaded or self-hosted; they are API-only, usually behind an OpenAI-compatible endpoint. A local-only deployment gives up whatever those models do better on a given task. 20 21 22
Quantization and MoE sizing
Quantization in 2026 is less about rounding weights after training and more about checkpoints that ship low-bit by design, alongside architecture choices that cut the cache before quantization enters.
Native low-bit checkpoints: Frontier open models increasingly ship quantization-aware. GPT-OSS ships its MoE experts in MXFP4 (E2M1 with a shared 8-bit scale, 4.25 bits per parameter), so the 120B fits in about 63 GB, one 80 GB GPU, where a naive BF16 estimate would demand about 234 GB across three GPUs. DeepSeek-V3 and R1 ship native FP8 at about 1 byte per parameter. The weights should be sized on the checkpoint’s actual dtype, not the parameter count. 23 24
The GGUF effective-bit trap: GGUF files mix precision, with per-block FP16 scales and un-quantized embedding and output tensors, so the name is a floor rather than the size. Q4_K_M is about 4.9 effective bits (0.61 bytes per parameter), not 4.0. Q8_0 is 8.5 bits (1.06 bytes per parameter), not 8.0. A calculator keyed on the bit number under-sizes weights by about 20% at Q4 and 6% at Q8. IQ-quants such as IQ4_XS (about 4.25 bits) now match Q4_K_M quality at a smaller size. 25
KV-cache quantization: Lowering the KV dtype scales the whole cache linearly and is independent of weight precision. FP8 (e4m3) KV halves it (Llama-3-8B at 128k, batch 1, from 16.0 GiB BF16 to 8.0 GiB FP8) at broadly free quality. INT4 quarters it but needs an evaluation. In vLLM it is one flag (--kv-cache-dtype fp8). 26
Attention as a memory saver: Multi-head Latent Attention (MLA, used by DeepSeek) caches a single shared low-rank latent (about 576 elements per token per layer) instead of per-head keys and values, roughly 30 times smaller than the nominal 128-head read. That is what lets a 671B model serve 128k context with about 8.6 GiB of KV. Sliding-window and local-global attention (Gemma 2 and 3, GPT-OSS, Llama 4) cap most layers at a fixed window instead of full context, cutting long-context KV by 10 to 40 times against a naive all-global estimate. These are fixed per architecture family, not tunable parameters. 27
Offloading and sharding: When weights exceed GPU memory, offloading moves inactive parts, such as unused MoE experts, between GPU memory and slower system RAM, trading tokens per second for the ability to run at all. 28 Sharding splits one model across several devices or memory tiers, which is how a 671B model spans an 8-GPU node. Both extend the reach of fixed hardware at a bandwidth cost.
Self-hosting benefits and tradeoffs
The case for self-hosting is data control, cost at volume, and configuration freedom. The case against it is hardware cost, operational burden, and the proprietary-model gap.
Data residency and compliance: Cross-border data transfer is the compliance driver. Under GDPR, sending personal data outside the EU can trigger legal safeguards, contractual obligations, or restrictions. The EU AI Act adds a second layer, but it is phasing in rather than fully in force. As of mid-2026, the prohibitions (applicable since 2 February 2025) and the general-purpose AI (GPAI) model obligations (since 2 August 2025) already apply, while the high-risk requirements around risk management, auditability, and governance are deferred, to 2 December 2027 for Annex III standalone high-risk systems and 2 August 2028 for Annex I embedded systems under the 2025-26 Digital Omnibus package. 29 30 31 Running inference in-jurisdiction, on a controlled network, keeps sensitive data out of a third party’s hands, which is the sovereign-AI case for finance, healthcare, and the public sector.
Cost and control: Self-hosting starts expensive, with consumer GPUs or a small server, but local inference can undercut recurring API fees for teams generating high request volumes. It also removes vendor lock-in and imposed limits on context window, inference settings, and integration, and gives direct access to the weights for fine-tuning on private data.
The tradeoffs: GPU memory is the binding limit, and some workloads still need 16 to 48 GB of VRAM, out of reach for smaller teams. Deployment adds dependency management, CUDA and kernel troubleshooting, monitoring, and updates that a cloud provider would otherwise handle. Performance is the operator’s responsibility, from batching and sharding to hardware utilization. And the strongest proprietary models stay API-only, so a local deployment accepts a capability gap on the tasks where they lead.
Add as preferred sourceSelf-hosted LLM VRAM calculation methodology
A model’s memory footprint is the sum of four terms that are computed independently and added, not a single figure scaled off the parameter count:
VRAM_total = Weights + KV cache + Activations + Overhead
The most common estimation error collapses the last three terms into a flat 20% markup on the weights. That is close at one model size and wrong at the others, because the three terms scale differently. 32
Weights: Weight memory is parameters times bytes-per-parameter. BF16 and FP16 store 2.0 bytes per parameter, and FP8 and INT8 store about 1.0. The trap is 4-bit, which is not 0.5 bytes per parameter. Real grouped INT4 (GPTQ, AWQ) lands at 0.52 to 0.55, and GGUF Q4_K_M is about 0.61 bytes per parameter (4.9 effective bits, not 4.0). GGUF Q8_0 is 8.5 bits (1.06 bytes per parameter), not 8.0. Assuming 0.5 undercounts a 70B model by about 8 GB, which flips a fit verdict. 33
Mixture-of-experts sizes every expert, not the active ones: All expert weights stay resident in VRAM even though a few fire per token. Mixtral-8x7B needs about 28 GB at Q4 for the full 46.7B parameters, not the 13B active. Total parameters set the memory and the number of GPUs, and active parameters set the compute. A calculator that sizes on active parameters falsely fits large MoE models.
KV cache: The key-value cache is 2 x n_layers x n_kv_heads x head_dim x seq_len x batch x bytes_per_element. Most current models use grouped-query attention (GQA), where many query heads share a few KV heads, so n_kv_heads pairs are cached, which shrinks the cache 4 times (Llama-3-8B) to 8 times (Llama-3-70B) against full multi-head attention. Both head_dim and n_kv_heads are read from each model’s config.json rather than derived from the hidden size and head count, since families like Gemma (head_dim 256) and Qwen3 (128) would otherwise be mis-sized by about 2 times. The cache grows linearly with both context length and batch size, and at long context it rivals or exceeds the weights. Llama-3-8B caches 128 KiB per token, so at 128k context and batch 1 an FP16 KV cache is 16.0 GiB, equal to the 16 GB of BF16 weights. An FP8 KV cache halves that. 34
Overhead is a floor, not a percentage: Framework overhead is a fixed per-GPU floor (CUDA context and kernels, about 1 to 2 GB per GPU) plus a small bounded fraction, not a share of the weights. A flat 20% under-counts small models, where a CUDA context alone can exceed 20% of a 6 GB model, and over-counts large ones, where a 140 GB model does not need 28 GB of context. The piecewise floor is the accurate model, and flat 20% works as a quick estimate. 35
Activations and the serving engine: The activation term is transient. During decode, one token at a time, it is small and folds into the overhead floor, but prefill processes the whole prompt at once, so its activation peak scales with batch size and prompt length. The floor itself depends on the engine. Paged servers (vLLM, SGLang) reserve a fixed share of each card through a memory-utilization setting, about 10% at the 0.90 default, and fit the weights and KV inside the rest. Non-paged runtimes (llama.cpp, Ollama) hold a fixed compute buffer of about 1 to 2 GB instead. Sizing both the same way is off by a few GB.
Multiple GPUs do not divide cleanly: Two 24 GB cards are not 48 GB of usable space. Under tensor parallelism the weights and KV cache split across the N cards (W/N and KV/N), but the activations, the CUDA context, and the NCCL communication buffers are replicated on every card, so the summed footprint is larger than a single-device estimate of the same total. That replication tax is why a 70B needing 43 GB of weights fits on two 24 GB cards with a short context, and why the safe check is the per-GPU total rather than the total divided by the GPU count.
The four terms interact at the fit boundary. A configuration reads as Fits when the required memory is at or below 90% of usable, Tight in the top 10% of capacity, and Won’t-fit above usable. Paged servers already hold back that 10% through the utilization setting. The examples below use Llama-3 on common hardware, with llama.cpp on the consumer cards:
At 128k context the KV cache equals the weights, so context length, not the parameter count, decides the fit. The last row fits because the Q4 weights (~405 GB) sit under 640 GB at 8k context. MLA’s payoff is at long context, where it keeps a 671B model’s KV small enough to serve 128k.
Further reading
- LLM Quantization: BF16 vs FP8 vs INT4
- LLM Inference Engines: vLLM vs LMDeploy vs SGLang
- GPU Concurrency Benchmark: H100 vs H200 vs B200 vs MI300X
- Multi-GPU Benchmark: B200 vs H200 vs H100 vs MI300X
- Top 60+ Cloud GPU Providers
- Cloud LLM vs Local LLMs
- LLM Pricing: Top 15+ Providers Compared
- LLM Fine-Tuning Guide for Enterprises
FAQs
A self-hosted LLM is a large language model used for LLM applications that runs entirely on hardware you control (like your personal computer or private server) rather than relying on a third-party cloud service.
Techniques include using frameworks like llama.cpp, libraries like Hugging Face transformers, user-friendly apps (Ollama, LM Studio), model quantization (e.g., GGUF, GPTQ) to reduce resource needs, model parallelism to distribute large models across multiple devices, and optimized inference engines (like vLLM).
Yes, tools like vLLM, Ollama, and LM Studio can run local servers capable of handling multiple (often concurrent) requests. This is similar to how cloud APIs operate, often using batching for efficiency.
No, you don’t need external access permission or API keys from a provider for self-hosted llm. Since you host it yourself, you have direct access; you might optionally set up your own authentication for your local server if needed.
Cite this research
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sari2026,
author = {Sarı, Ekrem},
title = {{LLM VRAM Calculator for Self-Hosting}},
year = {2026},
month = jul,
howpublished = {\url{https://aimultiple.com/self-hosted-llm}},
note = {AIMultiple. Retrieved July 12, 2026}
}Reference Links
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.