Where your memory goes when you run a model locally

Your memory holds two big things: the model's weights, and a cache of the conversation that grows with every token of context. If both fit on the graphics card the model runs there; if they don't, part of it runs on the processor, and that part sets a slower pace.

  • Context, RAM and VRAM
  • GPU offload explained
  • 2× The cache when you double the context
  • f16 llama.cpp's default context cache type

How much the model can see at once

The context window is how many tokens the model can see at once.

Everything counts

The instructions, the conversation so far, any text you paste or attach, and the reply being written. When a conversation outgrows the window, the oldest part has to make way, and the model no longer sees it.

The model's maximum

Every model has a maximum it was built for, given on its model card and stored in the GGUF file. You can run it with a smaller window to save memory.

What it changes, and when to change it

A larger window lets the model work with longer documents and remember more of a long chat. It doesn't make the model smarter, and it costs memory and speed. When to change it: raise it when the model loses track of the start of a long chat; lower it when the model won't load or runs out of memory.

Why a longer context needs more memory

While it writes, the model keeps a key and a value for every token in the context, in every layer.

  1. The size of the cache

    That store is the KV cache, so the model doesn't recompute the whole conversation for each new word. Roughly: KV cache = 2 x layers x KV heads x head size x bytes per value x tokens of context.

  2. It grows in a straight line

    The first four numbers are fixed by the model's design. The last one is yours: double the context, double the cache. Models that share keys and values across attention heads (grouped-query attention) have fewer KV heads and a smaller cache.

  3. Set aside when the model loads

    llama.cpp, and runners built on it, size the cache for the window you set when the model loads. So a large window costs memory even while the chat is still short.

  4. Smaller cache types

    llama.cpp stores the cache at 16 bits (f16) by default. Some runners can store it at 8 or 4 bits to save memory. That changes what the model answers.

Why long chats get slower

Two things slow down as the context grows, and starting a new chat resets both.

Reading the prompt

Before the first word of a reply, the runner has to process every token the model hasn't seen yet. A long pasted document means a longer wait for the first word. llama.cpp's server keeps that work between turns by default, so a follow-up question only processes the new part.

Writing each word

Every new token looks back at all the tokens in the context, so each step does more work as the chat gets longer.

RAM, graphics memory and Apple unified memory

Where the model sits decides how fast it runs and how big it can be.

Graphics memory (VRAM)

It sits on the graphics card. A model that fits there runs fastest. Its size is the hard limit.

System memory (RAM)

What the processor uses. A model runs from RAM when there is no suitable card or when it doesn't fit on the card. Writing each token mostly means reading through the weights, so memory speed matters more here than processor cores.

Unified memory on Apple Silicon

One pool shared by the processor and the graphics part of the chip. The graphics side can use a large share of it, but not all: macOS and your other apps keep some. That is why a Mac with a lot of memory can hold models that wouldn't fit on a typical graphics card.

What to buy

For which card or how much RAM to get for the models you want to run, see the buyer's view.

Layers on the card, layers on the processor

A model is a stack of layers, and GPU offload puts some of them in graphics memory and leaves the rest in RAM.

All, some or none

  • All layers on the card: the fastest case
  • Some layers on the card: every token goes through both parts, and the processor part sets the pace
  • No layers on the card: the processor does everything

The setting by name

In llama.cpp it is -ngl; in Ollama it is the num_gpu option. Put too many layers on the card and the model fails to load, or runs out of memory once the context fills.

What it changes, and when to change it

Speed, as long as everything fits. Lower the layer count if the model won't load on the card; raise it while there is graphics memory to spare. The context cache needs room on the card too.

Other speed settings you may meet

These change speed and memory use, not what the model writes.

Threads

How many processor threads the runner uses. llama.cpp picks a number by default.

Batch size

How many prompt tokens are handled in one step. A larger batch takes more working memory.

Memory mapping and locking

Mapping (mmap) lets the system read the model file from disk as it is needed. Locking (mlock) keeps the model in RAM, so the system can't move it out to disk.

Flash attention

A way of computing attention that moves less data through memory. It matters most on long contexts. llama.cpp sets it to auto by default.

Speculative decoding

A small draft model guesses a few tokens ahead, and the main model checks them all in one pass. It needs a draft model that matches the main one, and memory for both.

Context and memory in Pineido Studio

Studio calls the context window Memory and sizes most of this for you.

Memory (context window)

It defaults to "Auto (largest that fits)", and you can pick 2k, 4k, 8k, 16k, 32k, 64k or 128k from a dropdown. A change applies on the next message, and that first reply is slower while the model reloads. A size you pick by hand replaces Auto.

How Auto sizes the window

Auto sizes the context to what the conversation needs: an 8k working window, more when the model or the task needs it. It is always kept within what this computer can hold (the model's own maximum; weights plus memory cache in the graphics card, or in RAM without one, or unified memory on Apple Silicon; every parallel lane and anything else already running beside it), within what it can read in about 100 seconds at the speed measured here, and under any limit learned from earlier overflows. Pick a size yourself to override it.

Run the models on

In Settings > Performance, under "Where the AI runs": "Automatic (recommended)", the default, "Processor only (slower, works everywhere)", and the graphics cards your computer has. A change applies at the next start.

Set for you: layers and threads

  • GPU layers: none without a graphics card, all of them if the whole model fits the card, otherwise the number measured to fit
  • Threads: the physical cores, or one fewer on a processor without hyper-threading
  • Flash attention: auto with a graphics card, off on the processor

Set for you: batch and memory mapping

  • Batch with a card: 2048, 1024 or 512, by the card memory left after the model (6 GB or more, 3 GB or more, less)
  • Batch on the processor only: 512, 256 or 128, by cores (16 or more, 8 or more, fewer)
  • The model file is memory-mapped when it doesn't fit comfortably, and locked in RAM when it runs on the processor and fits

The context cache stays f16

Studio keeps the context cache at f16 on purpose, because a smaller cache type changes what the model answers.

Speculative decoding and Speed

Speculative decoding is in Settings > Performance and off by default. The app calls it lossless; it needs a matched draft model and enough graphics memory. Speed, on the same screen, is a read-out of drive speed, not a setting.

A context that fits

Pineido Studio does this sizing on every computer it runs on, so you start from a context that fits. No models are included: you pick one and download it.

What you see, and what to try

Most memory and speed trouble comes from a context or a model file that is too big for where it runs.

It won't load, or stops with an out-of-memory error

The weights plus the context don't fit. Try a smaller context, a smaller file, fewer GPU layers, or close other apps.

The first reply after a settings change is slow

The model is reloading. Wait for that reply; the next ones are normal.

Replies get slower as the chat grows

There is more context to read and look back over. Start a new chat for a new topic.

It forgets the start of a long chat

The context window is full. Use a larger window, if memory allows.

Slow on a computer with a graphics card

The model runs partly or fully on the processor. Check which device it runs on and how many layers are on the card. In Studio, check that "Run the models on" is Automatic or the card, then restart.

You set Memory by hand and the model struggles

In Studio, go back to Auto: a size you pick by hand replaces Auto, even when it is too big.

2× The cache when you double the context

Want a local model set up for your work?

Tell us what the model should do and which computers you have.

For one computer, Pineido Studio does this on a USB stick.

You leave with

  1. Tell us the work and the computers
  2. We test models on your own prompts

Direct answers before you commit

How much memory does a 32k context need?

It depends on the model's layers, KV heads and head size; the formula on this page gives it. The model card or the runner's load log tells you those numbers.

Does a bigger context window make the model smarter?

No. It lets the model see more text at once, and it costs memory and speed.

Is graphics memory or RAM more important?

If the model fits on the card, graphics memory. If it doesn't, the amount and speed of RAM decide what runs and how fast.

Why can't I set GPU layers in Pineido Studio?

Studio sets them from what fits on the card. The one choice you make is where the models run: "Run the models on".

Why does Pineido Studio keep the context cache at f16?

A smaller cache would save memory but change what the model answers, so Studio keeps the full one.

ByFounder and Imaginator

Pineido, based in Strumica, North Macedonia, is a software company that builds websites, online shops, web and mobile apps, business automation and AI for companies at home and abroad. Work ships in 30-day sprints at a fixed price, and the client owns the code.

Published