Hardware Requirements for a Local LLM: CPU, RAM, and GPU Guide

What it really takes to run an LLM on your own hardware, from a 3B model on a laptop CPU to a 70B model on a workstation. The numbers below are the ones that decide whether your setup is fast, slow, or out of memory.

9 min read8 sectionsNovember 9, 2026

Kostadin Stamboliev

Founder and Imaginator

LinkedIn

Founder and Imaginator at Pineido. Practitioner across IT consulting, app and web development, technical SEO, and AI / automation. Ships production code in 30-day sprints with fixed scope and clear ownership.

The one number that decides everything: memory

Before you compare GPUs, understand the bottleneck. Generating each token means streaming the model's entire active weight set through the processor, so local LLM speed is governed by memory bandwidth and memory capacity, not raw compute. Capacity decides whether the model loads at all; bandwidth decides how fast it runs once loaded. If a model does not fit in memory, no amount of GPU horsepower helps.

This is why the practical sizing rule is built around memory. At full FP16 precision a model needs about 2 GB per billion parameters, so a 7B model is roughly 14 GB and a 70B model is roughly 140 GB. Almost nobody runs local models at FP16. Quantization compresses the weights, and at the popular Q4_K_M level the footprint drops to roughly 0.5 to 0.7 GB per billion parameters: about 4 to 5 GB for a 7B model and around 40 GB for a 70B model. Add the KV cache (the running memory of your conversation), which grows with context length, plus a bit of runtime overhead, and you have your real requirement.

The second factor, bandwidth, explains why a $300 GPU can be faster than a $2,000 CPU for the same model. An RTX 4090 moves roughly 1,000 GB/s, an Apple M4 Max around 546 GB/s, and typical dual-channel DDR5 system RAM somewhere near 70 to 90 GB/s. That gap is most of the reason GPU inference feels instant and CPU inference feels like dial-up.

Quantization: how to shrink a model without breaking it

Quantization stores each weight using fewer bits. FP16 uses 16 bits per weight, Q8 uses about 8, and Q4 uses about 4. The most-recommended setting across the local LLM community is Q4_K_M, a 4-bit variant that, with its block metadata, lands closer to 4.8 effective bits per weight. It retains roughly 95 percent of full-precision quality while cutting memory close to 4x, which is why it is the default that almost everyone starts with.

The practical guidance is straightforward. For everyday chat, drafting, and summarizing, Q4_K_M is the right choice. For code generation and harder reasoning, step up to Q5_K_M, Q6_K, or Q8_0 if you have the VRAM, because quality losses show up first on logic and code. Below Q4, degradation becomes noticeable, so Q3 and Q2 are last resorts for squeezing an oversized model onto undersized hardware. A useful corollary: a larger model at heavier quantization often beats a smaller model at light quantization. A 14B model at Q4 will usually outperform a 7B model at Q8 if both fit your memory budget.

The standard file format for these quantized weights is GGUF, the single-file format used by llama.cpp and Ollama. The same .gguf file runs across Windows, Linux, and macOS and across CUDA, Metal, and CPU backends, so you can download one file and move it between machines without conversion.

VRAM and RAM by model size: the working table

Here is what each model tier needs at Q4_K_M, including modest context headroom. A 3B to 4B model (Phi-4 Mini, Llama 3.2 3B, Gemma 3 4B, Qwen 3 4B) needs about 2 to 4 GB and runs on almost anything, including CPU-only laptops with 8 to 16 GB of RAM. A 7B to 8B model (Mistral 7B, Llama 3.1 8B, Qwen 3 8B) needs about 6 to 8 GB and is comfortable on any 8 GB or larger GPU. A 13B to 14B model needs about 9 to 12 GB and fits a 12 GB card. A 30B to 34B model needs roughly 18 to 24 GB and is the natural fit for a single 24 GB card.

The jump everyone underestimates is 70B. At Q4_K_M a 70B model wants roughly 40 to 48 GB once you include a usable context window, which means it does not fit on a single 24 GB card. You need two 24 GB GPUs pooled (48 GB total), a single 48 GB workstation card, or a large unified-memory Mac. A 405B model is a different universe: roughly 240 GB even at 4-bit, around 486 GB at FP8, which is datacenter territory (think 8x A100 80 GB or 8x H100), plus a terabyte-plus of system RAM and fast NVMe for the weights.

Do not forget the KV cache, the line item that quietly blows budgets. For an 8B model the cache grows from a fraction of a GB at 2K context to roughly 5 GB at 32K and around 20 GB at 128K. On a 70B model a 128K context can demand 40 GB-plus of cache on top of the weights. If you plan to use long contexts, size for the cache, not just the model. Quantizing the KV cache to 8-bit roughly halves it at a small quality cost.

GPU options and what they actually cost in 2026

For pure inference, VRAM per dollar is the metric that matters, and the used NVIDIA RTX 3090 remains the value king. With 24 GB of VRAM at roughly $600 to $850 used, it runs 13B models at high quantization, 30B-class models at Q4, and pairs cleanly into a dual-card 48 GB rig for 70B. It is also the last consumer card with NVLink, which gives two-card setups faster GPU-to-GPU communication. The catch: no warranty, possible mining history, and you may need to manage VRAM thermals.

The RTX 4090, around $1,600 to $2,200, is the best all-rounder. Same 24 GB but meaningfully faster, with strong performance for image generation and light fine-tuning on top of inference, and it carries a warranty. The RTX 5090, roughly $3,000 to $4,000 amid ongoing memory-chip shortages, is the only consumer card with 32 GB and the highest bandwidth of the three, which makes it the single-card pick for 32B models at Q4 with long context and for mixture-of-experts models. Note that even 32 GB does not cleanly fit a 70B at Q4.

Real-world generation speeds put the tradeoffs in perspective. An 8B model on a single 4090 runs north of 100 tokens per second. A 70B model at Q4_K_M will not load on a single 24 GB card; on dual 3090s it runs around 16 tokens per second, and on dual 4090s around 19. Those are usable, conversational speeds, with the dual rig drawing close to 900 watts under load.

Apple Silicon: the quiet 70B workhorse

Apple's unified memory architecture is the easiest single-box path to a 70B model. Because the CPU and GPU share one memory pool, a Mac with 128 GB of unified memory can load models that would otherwise require two NVLinked GPUs or a datacenter card. An M4 Max delivers roughly 546 GB/s of bandwidth and supports up to 128 GB, while the M3 Ultra goes to 192 GB at around 800 GB/s, which is the true high-end Apple part for AI work since there is no M4 Ultra.

In practice an M4 Max runs Llama 3.3 70B at Q4_K_M somewhere in the 15 to 28 tokens-per-second range depending on backend and context. Use Apple's MLX framework rather than the default llama.cpp backend; MLX is consistently 30 to 50 percent faster on Apple Silicon. One firm recommendation: for serious 70B work choose 128 GB, not 64 GB. A 70B Q4 model technically fits in 64 GB, but macOS memory pressure spikes and the system swaps to SSD, which tanks speed and stresses the drive. The 128 GB configuration leaves headroom for your browser, editor, and the model at once.

The tradeoff versus NVIDIA is speed for models that fit a GPU's VRAM, where a 4090 is roughly twice the bandwidth of an M4 Max. Apple wins on what fits in a single quiet machine that sips power; NVIDIA wins on raw throughput for models that fit on the card.

CPU-only and hybrid offloading: the budget path

You can run a local LLM with no GPU at all. On a modern CPU with dual-channel DDR5, a 3B to 4B model at Q4 runs around 8 to 25 tokens per second, fast enough for real interactive use. Phi-4 Mini (3.8B) is the standout here: about a 2.3 GB footprint, roughly 12 tokens per second on a modern CPU, and benchmark scores that match Llama 3.1 8B on general knowledge. A 7B to 8B model on CPU drops to roughly 3 to 8 tokens per second, which is fine for batch jobs but sluggish for chat. Above 8B, CPU-only stops being practical.

Two CPU details move the needle. Memory speed matters: going from DDR5-4800 to DDR5-6000 has been shown to lift 7B to 8B generation speed by 20 percent or more, purely from bandwidth. And dual-channel matters: two RAM sticks effectively double bandwidth versus one, so populate both channels. The simple RAM rule for CPU inference is the GGUF file size plus about 500 MB of overhead equals your minimum.

When a model is slightly too big for your VRAM, hybrid offloading keeps some layers on the GPU and spills the rest to system RAM. This is slower than full-GPU but faster than CPU-only, and crucially it does not change output quality at all; the same weights do the same math, just stored in different places. The speed cost depends on your PCIe bandwidth, so on a discrete GPU expect a real penalty per offloaded layer. It is a pragmatic stopgap, not a substitute for fitting the model in VRAM.

What we measured: one 4B model on an 8 GB card

Rules of thumb break down at the edge, so here is one model measured end to end on our own test machine: Intel Core i7-7700, 32 GB RAM, NVIDIA GeForce GTX 1080 with 8 GB, llama.cpp llama-server (CUDA 12 build, flash attention on), 9 November 2026. The model is a 4-billion-parameter hybrid at Q4_K_M, a 3.3 GB file. Its language weights took 2,514 MiB on the card, and the 497 MiB token-embedding table stayed in system RAM, as llama.cpp does by default.

With every layer on the card and a 16,384-token cache shared by two conversations, the card also held 512 MiB of KV cache, 100.5 MiB of recurrent state and 72 MiB of compute buffers, plus 223 MiB of scratch for the vision encoder. It read prompts at 750 to 892 tokens per second and wrote at 24.9 to 37.4 tokens per second.

Minutes later the same model was relaunched with a 12,288-token window per conversation while other models shared the card. The KV cache grew to 768 MiB and 3 of its 33 layers no longer fitted, so they ran on the processor. It then read prompts at 110 tokens per second and wrote at 5.47. Moving 9 percent of the layers cost 78 to 85 percent of the generation speed: a partial offload is a pipeline, and its slowest stage, weights read from system RAM by four CPU threads, sets the pace of every token. The practical rule: when a model only just fits, shrink the context window or pick the next smaller quantization before you let layers spill.

Cost tiers: pick the smallest rig that fits your workload

Match hardware to the model you actually need, not the largest one you can imagine. Entry tier, roughly $0 to $400: a laptop or desktop you already own, running 3B to 8B models on CPU or an 8 GB GPU. This covers private drafting, summarizing, classification, and simple coding help, and it is enough to confirm whether local AI fits your workflow at all. Enthusiast tier, roughly $700 to $2,200: a single used RTX 3090 or a new RTX 4090 with 24 GB, comfortably running 13B to 34B models for daily coding, RAG over your own documents, and agentic scripts.

Prosumer tier, roughly $2,000 to $5,000: dual 24 GB GPUs (48 GB pooled) or a 128 GB M4 Max Mac, the entry point for 70B-class models at conversational speed. This is the tier where a local model genuinely competes with a hosted API for many tasks. Anything beyond, the 405B-class frontier, is effectively cloud-only for almost everyone, and for nearly all real use cases a well-chosen 70B model matches or beats a 405B model while costing a fraction to run.

This sizing exercise is exactly where Pineido's local AI setup work starts. We scope the smallest rig that meets your privacy and performance requirements, pick the model and quantization to match, and stand up the stack so it survives contact with real use. If you are weighing local versus hosted AI for sensitive data, our local AI setup and pocket AI sprints turn this math into a working system in 30 days, with a fractional CTO engagement available when the decision touches your broader infrastructure roadmap.

Get operations insights weekly

Join service-business operators who get one actionable insight on ops, automation, and growth every week.

Checking that you're human. You can send in a moment.

No spam. Unsubscribe anytime.

You’re subscribed! Look for our next issue in your inbox.

Ready to ship?

Tell us where you are stuck. We will review it and send a focused recommendation within one business day.

Frequently asked questions

How much VRAM do I need to run a local LLM?

At the common Q4_K_M quantization, plan on roughly 0.5 to 0.7 GB per billion parameters plus headroom for context. That means about 6 to 8 GB for a 7B to 8B model, 9 to 12 GB for a 13B, 18 to 24 GB for a 30B, and roughly 40 to 48 GB for a 70B. Long contexts add KV cache on top, so size up if you plan to use 32K tokens or more.

Can I run a local LLM without a GPU?

Yes, for smaller models. A modern CPU with dual-channel DDR5 runs 3B to 4B models at roughly 8 to 25 tokens per second, which is usable for chat. A 7B to 8B model drops to about 3 to 8 tokens per second, fine for batch work but slow for interactive use. Above 8B, CPU-only inference becomes impractical and you want a GPU or an Apple Silicon Mac.

What hardware runs a 70B model locally?

A 70B model at Q4_K_M needs roughly 40 to 48 GB of memory, so it will not fit on a single 24 GB GPU. The three practical paths are two pooled 24 GB GPUs (dual RTX 3090 or 4090 for 48 GB), a single 48 GB workstation card, or an Apple Silicon Mac with 128 GB of unified memory. Expect roughly 15 to 28 tokens per second depending on the setup.

Is the RTX 3090 still worth buying for local AI in 2026?

For inference, yes. At roughly $600 to $850 used it offers the best VRAM-per-dollar of any consumer card, with 24 GB and the only consumer NVLink support, which makes dual-card 48 GB rigs for 70B models cheaper than any alternative. The tradeoffs are no warranty, possible mining history, and slightly slower speed than a 4090. Buy the 4090 instead if you also want warranty coverage, image generation, or light fine-tuning.

Why is memory bandwidth more important than the CPU or GPU clock speed?

Generating each token requires streaming the model's active weights through the processor, so token speed is limited by how fast memory can be read, not by raw compute. This is why an RTX 4090 at about 1,000 GB/s feels instant while system RAM at 70 to 90 GB/s feels slow on the same model, and why faster DDR5 alone can lift CPU inference speed by 20 percent or more.

Do I really need a 405B model, or is a 70B enough?

For nearly every practical use case a well-chosen 70B model is enough and often matches or beats a 405B model on benchmarks while costing a fraction to run. A 405B model needs roughly 240 GB even at 4-bit, which is datacenter hardware, so it is effectively cloud-only. Reserve it for specific research that genuinely requires maximum model size.