All, some or none
- All layers on the card: the fastest case
- Some layers on the card: every token goes through both parts, and the processor part sets the pace
- No layers on the card: the processor does everything
Your memory holds two big things: the model's weights, and a cache of the conversation that grows with every token of context. If both fit on the graphics card the model runs there; if they don't, part of it runs on the processor, and that part sets a slower pace.
The context window is how many tokens the model can see at once.
The instructions, the conversation so far, any text you paste or attach, and the reply being written. When a conversation outgrows the window, the oldest part has to make way, and the model no longer sees it.
Every model has a maximum it was built for, given on its model card and stored in the GGUF file. You can run it with a smaller window to save memory.
A larger window lets the model work with longer documents and remember more of a long chat. It doesn't make the model smarter, and it costs memory and speed. When to change it: raise it when the model loses track of the start of a long chat; lower it when the model won't load or runs out of memory.
While it writes, the model keeps a key and a value for every token in the context, in every layer.
That store is the KV cache, so the model doesn't recompute the whole conversation for each new word. Roughly: KV cache = 2 x layers x KV heads x head size x bytes per value x tokens of context.
The first four numbers are fixed by the model's design. The last one is yours: double the context, double the cache. Models that share keys and values across attention heads (grouped-query attention) have fewer KV heads and a smaller cache.
llama.cpp, and runners built on it, size the cache for the window you set when the model loads. So a large window costs memory even while the chat is still short.
llama.cpp stores the cache at 16 bits (f16) by default. Some runners can store it at 8 or 4 bits to save memory. That changes what the model answers.
Two things slow down as the context grows, and starting a new chat resets both.
Before the first word of a reply, the runner has to process every token the model hasn't seen yet. A long pasted document means a longer wait for the first word. llama.cpp's server keeps that work between turns by default, so a follow-up question only processes the new part.
Every new token looks back at all the tokens in the context, so each step does more work as the chat gets longer.
Where the model sits decides how fast it runs and how big it can be.
It sits on the graphics card. A model that fits there runs fastest. Its size is the hard limit.
What the processor uses. A model runs from RAM when there is no suitable card or when it doesn't fit on the card. Writing each token mostly means reading through the weights, so memory speed matters more here than processor cores.
One pool shared by the processor and the graphics part of the chip. The graphics side can use a large share of it, but not all: macOS and your other apps keep some. That is why a Mac with a lot of memory can hold models that wouldn't fit on a typical graphics card.
For which card or how much RAM to get for the models you want to run, see the buyer's view.
A model is a stack of layers, and GPU offload puts some of them in graphics memory and leaves the rest in RAM.
In llama.cpp it is -ngl; in Ollama it is the num_gpu option. Put too many layers on the card and the model fails to load, or runs out of memory once the context fills.
Speed, as long as everything fits. Lower the layer count if the model won't load on the card; raise it while there is graphics memory to spare. The context cache needs room on the card too.
These change speed and memory use, not what the model writes.
How many processor threads the runner uses. llama.cpp picks a number by default.
How many prompt tokens are handled in one step. A larger batch takes more working memory.
Mapping (mmap) lets the system read the model file from disk as it is needed. Locking (mlock) keeps the model in RAM, so the system can't move it out to disk.
A way of computing attention that moves less data through memory. It matters most on long contexts. llama.cpp sets it to auto by default.
A small draft model guesses a few tokens ahead, and the main model checks them all in one pass. It needs a draft model that matches the main one, and memory for both.
Studio calls the context window Memory and sizes most of this for you.
It defaults to "Auto (largest that fits)", and you can pick 2k, 4k, 8k, 16k, 32k, 64k or 128k from a dropdown. A change applies on the next message, and that first reply is slower while the model reloads. A size you pick by hand replaces Auto.
Auto sizes the context to what the conversation needs: an 8k working window, more when the model or the task needs it. It is always kept within what this computer can hold (the model's own maximum; weights plus memory cache in the graphics card, or in RAM without one, or unified memory on Apple Silicon; every parallel lane and anything else already running beside it), within what it can read in about 100 seconds at the speed measured here, and under any limit learned from earlier overflows. Pick a size yourself to override it.
In Settings > Performance, under "Where the AI runs": "Automatic (recommended)", the default, "Processor only (slower, works everywhere)", and the graphics cards your computer has. A change applies at the next start.
Studio keeps the context cache at f16 on purpose, because a smaller cache type changes what the model answers.
Speculative decoding is in Settings > Performance and off by default. The app calls it lossless; it needs a matched draft model and enough graphics memory. Speed, on the same screen, is a read-out of drive speed, not a setting.
Pineido Studio does this sizing on every computer it runs on, so you start from a context that fits. No models are included: you pick one and download it.
Most memory and speed trouble comes from a context or a model file that is too big for where it runs.
The weights plus the context don't fit. Try a smaller context, a smaller file, fewer GPU layers, or close other apps.
The model is reloading. Wait for that reply; the next ones are normal.
There is more context to read and look back over. Start a new chat for a new topic.
The context window is full. Use a larger window, if memory allows.
The model runs partly or fully on the processor. Check which device it runs on and how many layers are on the card. In Studio, check that "Run the models on" is Automatic or the card, then restart.
In Studio, go back to Auto: a size you pick by hand replaces Auto, even when it is too big.
2× The cache when you double the context
Tell us what the model should do and which computers you have.
For one computer, Pineido Studio does this on a USB stick.
It depends on the model's layers, KV heads and head size; the formula on this page gives it. The model card or the runner's load log tells you those numbers.
No. It lets the model see more text at once, and it costs memory and speed.
If the model fits on the card, graphics memory. If it doesn't, the amount and speed of RAM decide what runs and how fast.
Studio sets them from what fits on the card. The one choice you make is where the models run: "Run the models on".
A smaller cache would save memory but change what the model answers, so Studio keeps the full one.
ByKostadin StambolievFounder and Imaginator
Pineido, based in Strumica, North Macedonia, is a software company that builds websites, online shops, web and mobile apps, business automation and AI for companies at home and abroad. Work ships in 30-day sprints at a fixed price, and the client owns the code.
Published