Self-Hosted LLM Memory Calculator

not found Self-Hosted LLM Memory Calculator

Model

Precision

Context length (tokens in the KV cache)

8,192

Hardware

Number of GPUs

Serving framework (overhead)

KV cache / Batch

Memory breakdown