What Q4, Q5 and Q8 mean, and how much memory a model needs

Quantization stores a model's weights with fewer bits, so the file is smaller and needs less memory. The rough size of any model file is its parameter count times the bits per weight, divided by 8.

  • GGUF formats
  • The size formula, worked
  • 14 GB A 7B model at F16, by the formula
  • 7.4 GB The same model at Q8_0
  • 3.9 GB The same model at Q4_0

What quantization is, and the file you download

A quantized model keeps its shape and its parameter count; each number just takes less space.

Fewer bits per weight

A model is billions of numbers, called weights. Most models are released with each weight stored in 16 bits (F16 or BF16). Quantizing rounds groups of weights to a small set of levels and stores one scale per group, so the runner can expand them back while it computes.

GGUF, the file you download

GGUF is the model file format of llama.cpp. One file holds the quantized weights plus what the runner needs to use them: the architecture, the tokenizer and usually the chat template. Ollama can import it too. The quantization type is normally in the file name, for example model-Q4_K_M.gguf.

Where the half bit comes from

A Q4_0 block stores 32 weights at 4 bits (128 bits) plus a 16-bit scale: 144 bits for 32 weights, which is 4.5 bits each. Q8_0 works the same way: 256 plus 16 bits for 32 weights is 8.5.

F16, Q8_0, Q6_K, Q5 and Q4 side by side

Bits per weight for each format, from the block layouts in llama.cpp and its own table for an 8B model.

F16 and BF16: 16 bits

The unquantized size. BF16 is also 16 bits, with a wider number range and less precision than F16.

Q8_0: 8.5 bits

8-bit values in blocks of 32 weights, plus one 16-bit scale per block. The closest to the original of the common formats.

Q6_K: 6.5625 bits

A 6-bit K format, in super-blocks of 256 weights.

Q5_K_M: about 5.7 bits

Mostly 5-bit K blocks, with some tensors kept at Q6_K. 5.70 bits per weight in llama.cpp's own table for Llama 3.1 8B. The older Q5_0 and Q5_1 use blocks of 32: 5.5 and 6 bits.

Q4_K_M: about 4.9 bits

Mostly 4-bit K blocks, with some tensors kept at Q6_K. 4.89 bits per weight in llama.cpp's own table for Llama 3.1 8B.

Q4_0: 4.5 bits

The older, simpler 4-bit format: blocks of 32 weights plus one 16-bit scale.

What K, S, M, L and IQ mean

K formats group weights into super-blocks of 256 with their own small scales, which keeps more accuracy at the same size. S, M and L say how many of the tensors get a higher-precision type: M is the middle option. IQ formats (i-quants) are another family, used mostly for the smallest files, down to under 2 bits per weight.

Work out the file size from the parameter count

Size in bytes is about the parameters times the bits per weight, divided by 8.

A 7B model

Worked for 7 billion parameters. Estimates from the formula, not measured files.

  • F16: 7,000,000,000 x 16 / 8 = 14.0 GB
  • Q8_0: 7,000,000,000 x 8.5 / 8 = 7.4 GB
  • Q4_0: 7,000,000,000 x 4.5 / 8 = 3.9 GB

A 70B model

The same sum for 70 billion parameters. Again estimates, not measured files.

  • F16: about 140 GB
  • Q8_0: about 74 GB
  • Q4_0: about 39 GB

Why the real file differs

  • "7B" is a rounded name. The model card gives the exact parameter count
  • Q4_K_M and Q5_K_M keep some tensors at higher precision, so they come out larger than the plain sum
  • Some apps show GiB (1,073,741,824 bytes), others GB (1,000,000,000 bytes). 3.9 GB is about 3.7 GiB

For a real model

Read the size on its download page, and note the date you read it.

From file size to the memory you need

The file is only the first part of what a model needs to run.

The weights

About the file size. The runner loads the whole file, so this part is fixed by the format you pick.

The context cache

The KV cache grows with the context window. A long context can need gigabytes on its own.

Buffers and everything else

Working buffers for the runner while it computes, and room for the rest of your computer: the system, the browser, the app itself.

The 4 to 5 GB rule of thumb

A common rule of thumb puts a 7B model at Q4 at around 4 to 5 GB of memory: the 3.9 GB of weights plus a little room. Treat it as a floor. A longer context, a larger format or other open apps push it up.

Mixture-of-experts models

MoE models, with names like "30B-A3B", use only some of their weights for each token, but every weight still has to be loaded. Size the memory by the total parameter count, not the active one.

What to buy

For the computer side, graphics memory, RAM and what each size of model needs, see the buyer's view.

Quality at fewer bits

Rounding weights adds error: smallest at 8 bits, growing as the bits drop, and faster below 4 bits.

What it changes

A lower-bit format needs less memory and less data to read per token. A higher-bit format stays closer to the original model. How much the difference shows depends on the model and the task, so test the formats you are choosing between on your own prompts.

When to change it

  • It fits with room to spare: go up to Q6_K or Q8_0
  • It is a tight fit: Q4_K_M is the usual middle ground
  • It doesn't fit even at Q4: try a smaller model and a 2- or 3-bit file of the big one on your own work

Weights are not the context cache

Quantizing the weights is not the same as quantizing the context cache. Some runners can store the cache at 8 or 4 bits to save memory; that also changes what the model answers.

Quantization in Pineido Studio

Quantization is not a setting in Pineido Studio: each catalog entry is one fixed file.

One fixed file per model

Studio picks the file for each model. There is no quantization menu to get wrong.

What the catalog uses

Of the catalog entries that name a quantization, Q4_K_M is the most common (92), then Q8_0 (26) and Q4_0 (11), plus a few others. Most entries don't name a format at all.

Sized to your memory

When Studio sizes the context window on Auto, it checks that the weights plus the KV cache fit in the graphics card's memory after reserves; without a card, in the usable share of RAM; on Apple Silicon, in the unified memory.

The context cache stays f16

Studio keeps the context cache at f16 on purpose, because a smaller cache type changes what the model answers.

No models included

Pineido Studio picks the file for each model and sizes the context to the memory your computer has. No models are included: you pick one and download it.

What people ask us first

How much memory does a 7B model need at Q4?

The weights come to about 3.9 GB by the formula (7 billion x 4.5 bits / 8). Add the context cache and room for the rest of your computer: 4 to 5 GB is a floor, not a target.

Is Q8 worth it over Q4?

If it fits with room for the context, Q8_0 stays closer to the original model. If it only fits at Q4, Q4_K_M is the usual choice.

What is the difference between Q4_0 and Q4_K_M?

Both are 4-bit. Q4_K_M uses the newer K blocks and keeps some tensors at 6 bits, so it is slightly larger and usually keeps more quality.

What does the M in Q4_K_M mean?

Medium: how many tensors get the higher-precision type. S is fewer, L is more.

Can I choose the quantization in Pineido Studio?

No. Each catalog entry is one fixed file that Studio picks per model.

14 GB A 7B model at F16, by the formula

Want a local model set up for your work?

Tell us what the model should do and which computers you have.

For one computer, Pineido Studio does this on a USB stick.

You leave with

  1. Tell us the work and the computers
  2. We test models on your own prompts

ByFounder and Imaginator

Pineido, based in Strumica, North Macedonia, is a software company that builds websites, online shops, web and mobile apps, business automation and AI for companies at home and abroad. Work ships in 30-day sprints at a fixed price, and the client owns the code.

Published