LLM Parameters: Types, Scaling, Fine-Tuning Guide
Parameter count is one of the most cited specifications of a large language model, but it does not determine how well a model performs or how much it costs to run. The chart below compares 31 models by their LLM Index score and their average cost per task across seven benchmarks.
Four models form the Pareto line, where no other model offers a higher score at a lower cost: GPT-6 Luna, Xiaomi’s MiMo-V2.6-Pro, GPT-6.1 Sol, and Claude Opus 5.5. The chart points to three patterns:
- Diminishing returns at the top: Claude Opus 5.5 has the highest score, but its gain over GPT-6.1 Sol is small relative to the difference in cost. The last few points of performance cost the most.
- Higher cost does not guarantee a higher score: Most models sit well below the Pareto line, and several cost more per task than GPT-6.1 Sol while scoring lower. Price reflects provider pricing and the number of tokens a model generates.
- Open-weight models compete on efficiency: MiMo-V2.6-Pro is the only open-weight model on the Pareto line. It reaches a mid-range score at one of the lowest costs in the comparison, despite having over a trillion total parameters. Because it uses a mixture-of-experts architecture, a fraction of those parameters process each token, which keeps inference cheap.
Definitions and types of LLM parameters
A model parameter is a variable that a machine learning system trains on its own, distinct from values set by human operators. In neural network implementation, parameters are tensors registered with automatic gradient computation enabled, allowing optimization algorithms to update their values during backpropagation.
Parameters include weights, which scale input signals, biases, which offset activations, and embeddings. They provide learned vector representations of discrete tokens.1
Trainable parameters vs hyperparameters
Weights, biases, and embeddings
Weights and biases
Weights and biases constitute the learnable components of linear transformations. In a fully connected layer, the weight matrix has shape (out_features, in_features) and scales the input, while the bias vector, shape (out_features), shifts the activation function to allow non-zero outputs when inputs are zero.2
Inside a transformer, each attention sublayer contains query, key, value, and output projection matrices (W_q, W_k, W_v, W_o), and common implementations such as PyTorch’s MultiheadAttention add optional bias terms to all four projections.3
Embeddings
Embeddings are the third parameter category: a lookup table storing a learned vector for each token in the vocabulary, initialized from a normal distribution and trained via backpropagation like other weights.4
The original transformer architecture shares the same weight matrix between input embeddings, output embeddings, and the pre-softmax linear transformation, scaling weights by the square root of the model dimension.
How parameters work inside LLMs
A large language model operates by holding trained parameter values fixed during inference while applying them to transform input token embeddings into output probability distributions.
The transformer architecture consists of stacked layers, each containing multi-head self-attention and feed-forward blocks parameterized by weight matrices and bias vectors.5
Before training, weight initialization draws values from distributions designed to maintain stable forward and backward signal variance across layers.
How LLM parameters are learned during training
Training proceeds via cycles of forward propagation, loss calculation, and backpropagation. The forward pass computes intermediate activations and final outputs by applying the weight matrices sequentially.
Backpropagation applies the chain rule of calculus to compute the gradient of the loss with respect to every parameter, propagating error signals from output to input layers. Gradient descent then updates each parameter in the direction that reduces loss, and this iterative process continues until convergence.
Inference-time generation parameters
During inference, trained weights remain frozen while sampling hyperparameters control token selection. Temperature scales the logits before softmax, ranging from 0 to 2; higher values increase randomness, and lower values increase determinacy.6
Top-p (nucleus sampling) restricts selection to the smallest set of tokens whose cumulative probability mass exceeds the specified threshold, typically between 0 and 1. Top-k sampling restricts selection to the k highest-probability tokens.
Frequency penalty and presence penalty range from -2 to 2, with positive values reducing the probability of tokens based on their existing frequency in the generated text or their presence in the text so far, respectively.
Recent models have deprecated temperature and top-p controls, rejecting non-default values or treating them as no-ops:
Anthropic’s Claude Opus 4.7 and later models return a 400 error when temperature, top-p, or top-k is set to anything other than its default, and Anthropic recommends steering output through prompting instead.7
OpenAI’s reasoning models, including o1, o3, o4-mini, and GPT-5, also reject non-default values. They accept temperature at its default of 1 and do not support custom top-p, frequency penalty, or presence penalty settings.8
Google’s Gemini 4 Argon takes the no-op route: requests that include temperature, top-p, or top-k still go through, but the model ignores those values.9
Parameter count and model scale
OpenAI’s GPT, Anthropic’s Claude, and Google’s Gemini do not disclose parameter counts, so published figures for those models are outside estimates. Open-weight models publish their sizes, and they account for a large share of real-world usage: DeepSeek led OpenRouter with 43% of text requests in the week at the start of November 2026.10
Most current open-weight flagships use a mixture-of-experts (MoE) architecture, which splits the feed-forward layers into many smaller expert networks and routes each token to a few of them. These models therefore report two figures:
- Total parameters: Every expert stored in memory.
- Active parameters: The experts that process a given token.
Kimi K3, for example, has 2.8 trillion total parameters but activates 104 billion per token, using 16 of its 896 experts.11
Compute per token scales with the active count, while memory and storage requirements scale with the total. This is why gpt-oss-120b, with 117 billion total and 5.1 billion active parameters, fits on a single 80 GB GPU.12
Scaling laws for parameters, data, and compute
Scaling laws are empirical relationships that link model performance to parameter count, training data, and compute. Kaplan et al. found that loss falls as a power law as each of these three factors grows, provided the others scale with it. At a fixed compute budget, total parameter count and training tokens mattered more than model shape or depth.13
Compute-optimal training
The Chinchilla study revised this picture by showing that many large models were undertrained for their size. Under a fixed compute budget, parameters and training tokens should grow in roughly equal proportion, which works out to about 20 training tokens per parameter.14
Earlier models such as GPT-3, Gopher, and MT-NLG were trained on close to one token per parameter. Recent models go far past the Chinchilla ratio: Llama 3 8B was trained on roughly 1,900 tokens per parameter. Read LLM scaling laws to learn more.
Training beyond the compute-optimal point makes sense once you count inference costs. Sardana et al. optimized the combined cost of training and inference over a model’s lifetime, rather than training compute alone. They found that in high-usage settings, smaller models trained on more data can match larger models at a lower total cost.15
Beyond parameter count
Later research adds variables that the original laws left out. Bian et al. extended the Chinchilla framework with architectural factors, including hidden size, the split of parameters between MLP and attention layers, and grouped-query attention. They trained over 200 models from 80 million to 3 billion parameters and showed that these choices shift the trade-off between accuracy and inference cost in predictable ways.16
Techniques for efficient parameter management
Parameter-efficient fine-tuning and quantization reduce the memory and compute costs associated with large parameter counts. These techniques enable fine-tuning and inference of models that would otherwise exceed hardware constraints.
Parameter-efficient fine-tuning (LoRA and variants)
Low-Rank Adaptation (LoRA) freezes pretrained weight matrices and injects trainable rank-decomposition matrices alongside them. For a weight matrix of dimensions d × k, LoRA adds matrices A and B with inner dimension r << d, k, reducing trainable parameters from d × k to r × (d + k).17
Applied to GPT-3 175B, LoRA reduces trainable parameters by 10,000 times and cuts GPU memory requirements by a factor of 3 compared to full fine-tuning with Adam, while matching or exceeding full fine-tuning quality on benchmarks. For example, Llama-3.2-1B fine-tuning trains 0.04% of parameters.18
LoRA-XS further reduces storage requirements by over 100 times relative to standard LoRA on 7-billion-parameter models by inserting trainable matrices between frozen low-rank decompositions.19
Quantization of model parameters
Quantization reduces the bit-width used to store parameter values, typically from 16-bit floating point to 4-bit or 8-bit representations.
QLoRA combines 4-bit NormalFloat quantization of the frozen base model with LoRA fine-tuning, reducing the memory required to fine-tune a 65-billion-parameter model from over 780GB to under 48GB while matching 16-bit full fine-tuning performance.20
GPTQ quantizes weights to 3 or 4 bits using approximate second-order information, processing a 175-billion-parameter model in approximately 4 GPU-hours with negligible perplexity increase and enabling single-GPU inference.21
GPTQ achieves end-to-end inference speedups of approximately 3.25x on NVIDIA A100 GPUs and 4.5x on NVIDIA A6000 GPUs relative to FP16. PyTorch INT8 quantization reduces model size by 4x and memory bandwidth by 2-4x, with geometric-mean inference speedups of 2.97x on x86 CPUs relative to FP32.22
Apple’s on-device foundation model employs quantization-aware training to compress weights to 2 bits per weight and embeddings to 4 bits, recovering quality through low-rank adapters atop the quantized base.23
Add as preferred sourceCite this research
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{dilmegani2026,
author = {Dilmegani, Cem and Ermut, Sıla},
title = {{LLM Parameters: Types, Scaling, Fine-Tuning Guide}},
year = {2026},
month = oct,
howpublished = {\url{https://aimultiple.com/llm-parameters}},
note = {AIMultiple. Retrieved October 14, 2026}
}Results and timestamps of 34 data points. Download the summary data shown in this article's charts and tables as a ZIP file containing 3 CSV files.
Want the granular data behind it? Join Premium
Reference Links
Cem's work at AIMultiple has been cited by leading global publications including Business Insider, Forbes, Morning Brew, and Washington Post, global firms like Deloitte and HPE, NGOs like World Economic Forum, and supranational organizations like European Commission. [1], [2], [3], [4], [5]
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
She previously worked as a recruiter in project management and consulting firms. Sıla holds a Master of Science degree in Social Psychology and a Bachelor of Arts degree in International Relations.
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.