free tool · no signup · runs in your browser
What GPU do I need to run this model?
Pick a model and we rank every GPU by whether it can run it — the minimum card that runs it well, the cards that run it with trade-offs, and the ones that won't. Each row shows the best quantization, the memory it needs, and an estimated tokens-per-second, all from each model's real architecture.
Minimum GPU to run Llama 3.1 8B well
RTX 3070 (8GB)
NVIDIA · 8 GB · Q6_K · ~38 tok/s
Any card with at least 8 GB of memory runs Llama 3.1 8B fully in VRAM at Q6_K — fast and everyday-usable.
Runs great · 47
| GPU | Memory | Best quant | Needs | Est. speed |
|---|---|---|---|---|
| RTX 3070 (8GB) | 8 GB | Q6_K | 7.9 GB | 38 tok/s |
| RTX 2070 Super (8GB) | 8 GB | Q6_K | 7.9 GB | 38 tok/s |
| GTX 1070 Ti (8GB) | 8 GB | Q6_K | 7.9 GB | 22 tok/s |
| Intel Arc A750 (8GB) | 8 GB | Q6_K | 7.9 GB | 43 tok/s |
| RTX 3080 (10GB) | 10 GB | Q8_0 | 9.7 GB | 52 tok/s |
| Intel Arc B570 (10GB) | 10 GB | Q8_0 | 9.7 GB | 26 tok/s |
| RTX 2080 Ti (11GB) | 11 GB | Q8_0 | 9.7 GB | 42 tok/s |
| GTX 1080 Ti (11GB) | 11 GB | Q8_0 | 9.7 GB | 33 tok/s |
| RTX 5070 (12GB) | 12 GB | Q8_0 | 9.7 GB | 46 tok/s |
| RTX 4070 (12GB) | 12 GB | Q8_0 | 9.7 GB | 34 tok/s |
| RTX 3060 (12GB) | 12 GB | Q8_0 | 9.7 GB | 24 tok/s |
| Intel Arc B580 (12GB) | 12 GB | Q8_0 | 9.7 GB | 31 tok/s |
| RTX 5080 (16GB) | 16 GB | Q8_0 | 9.7 GB | 65 tok/s |
| RTX 5070 Ti (16GB) | 16 GB | Q8_0 | 9.7 GB | 61 tok/s |
| RTX 5060 Ti (16GB) | 16 GB | Q8_0 | 9.7 GB | 30 tok/s |
| RTX 4080 (16GB) | 16 GB | Q8_0 | 9.7 GB | 49 tok/s |
| RTX 4070 Ti Super (16GB) | 16 GB | Q8_0 | 9.7 GB | 46 tok/s |
| RTX 4060 Ti (16GB) | 16 GB | Q8_0 | 9.7 GB | 20 tok/s |
| AMD RX 9070 XT (16GB) | 16 GB | Q8_0 | 9.7 GB | 43 tok/s |
| AMD RX 7800 XT (16GB) | 16 GB | Q8_0 | 9.7 GB | 42 tok/s |
| AMD RX 7600 XT (16GB) | 16 GB | Q8_0 | 9.7 GB | 20 tok/s |
| AMD RX 6900 XT (16GB) | 16 GB | Q8_0 | 9.7 GB | 35 tok/s |
| AMD RX 6800 XT (16GB) | 16 GB | Q8_0 | 9.7 GB | 35 tok/s |
| Intel Arc A770 (16GB) | 16 GB | Q8_0 | 9.7 GB | 38 tok/s |
| AMD RX 7900 XT (20GB) | 20 GB | Q8_0 | 9.7 GB | 54 tok/s |
| RTX 4090 (24GB) | 24 GB | Q8_0 | 9.7 GB | 68 tok/s |
| RTX 3090 (24GB) | 24 GB | Q8_0 | 9.7 GB | 63 tok/s |
| NVIDIA L4 (24GB) | 24 GB | Q8_0 | 9.7 GB | 20 tok/s |
| AMD RX 7900 XTX (24GB) | 24 GB | Q8_0 | 9.7 GB | 65 tok/s |
| RTX 5090 (32GB) | 32 GB | Q8_0 | 9.7 GB | 122 tok/s |
| NVIDIA A100 (40GB) | 40 GB | Q8_0 | 9.7 GB | 105 tok/s |
| RTX 6000 Ada (48GB) | 48 GB | Q8_0 | 9.7 GB | 65 tok/s |
| NVIDIA L40S (48GB) | 48 GB | Q8_0 | 9.7 GB | 59 tok/s |
| Apple M4 Pro (24-64GB) | 48 GB | Q8_0 | 9.7 GB | 19 tok/s |
| Ryzen AI Max+ 395 (32-128GB) | 64 GB | Q8_0 | 9.7 GB | 17 tok/s |
| Apple M4 Max (36-128GB) | 64 GB | Q8_0 | 9.7 GB | 37 tok/s |
| Apple M3 Max (36-128GB) | 64 GB | Q8_0 | 9.7 GB | 27 tok/s |
| Apple M2 Max (32-96GB) | 64 GB | Q8_0 | 9.7 GB | 27 tok/s |
| Apple M1 Max (32-64GB) | 64 GB | Q8_0 | 9.7 GB | 27 tok/s |
| NVIDIA A100 (80GB) | 80 GB | Q8_0 | 9.7 GB | 138 tok/s |
| NVIDIA H100 SXM (80GB) | 80 GB | Q8_0 | 9.7 GB | 227 tok/s |
| Apple M2 Ultra (64-192GB) | 128 GB | Q8_0 | 9.7 GB | 54 tok/s |
| Apple M1 Ultra (64-128GB) | 128 GB | Q8_0 | 9.7 GB | 54 tok/s |
| NVIDIA H200 (141GB) | 141 GB | Q8_0 | 9.7 GB | 325 tok/s |
| NVIDIA B200 (192GB) | 192 GB | Q8_0 | 9.7 GB | 542 tok/s |
| AMD MI300X (192GB) | 192 GB | Q8_0 | 9.7 GB | 359 tok/s |
| Apple M3 Ultra (96-512GB) | 256 GB | Q8_0 | 9.7 GB | 56 tok/s |
Runs with trade-offs · 7
| GPU | Memory | Best quant | Needs | Est. speed |
|---|---|---|---|---|
| Apple M3 (8-24GB) | 16 GB | Q8_0 | 9.7 GB | 6.8 tok/s |
| Apple M2 (8-24GB) | 16 GB | Q8_0 | 9.7 GB | 6.8 tok/s |
| Apple M1 (8-16GB) | 16 GB | Q8_0 | 9.7 GB | 4.6 tok/s |
| Apple M4 (16-32GB) | 24 GB | Q8_0 | 9.7 GB | 8.1 tok/s |
| Apple M2 Pro (16-32GB) | 32 GB | Q8_0 | 9.7 GB | 14 tok/s |
| Apple M1 Pro (16-32GB) | 32 GB | Q8_0 | 9.7 GB | 14 tok/s |
| Apple M3 Pro (18-36GB) | 36 GB | Q8_0 | 9.7 GB | 10 tok/s |
How to choose a GPU for a local LLM
Two numbers decide whether a model runs on a given GPU: memory size (does it fit?) and memory bandwidth (how fast does it generate?). This tool is the model-first companion to our Can I Run It? tool: instead of picking a GPU and seeing models, you pick a model and see every GPU ranked.
Does it fit?
A model needs room for its weights (parameters × bytes-per-parameter, set by the quantization) plus a KV cache that grows with context length, plus a little runtime overhead. If that total exceeds the card's memory, layers spill into system RAM ("CPU offload"), which is far slower — or the model won't load at all. We pick the best quantization that fits each card and report the tier accordingly.
How fast will it be?
For one user generating a token at a time, inference is memory-bandwidth bound. A rough estimate is tokens/sec ≈ memory bandwidth ÷ bytes read per token. A 24 GB 4090 at ~1 TB/s generates roughly twice as fast as a 12 GB 4070 at ~504 GB/s on the same model. Mixture-of-experts models read only their active parameters per token, so they punch above their total size.
Honest caveats
These are estimates from published specs, not measured benchmarks. Real speeds vary with your llama.cpp build, the exact GGUF, context length, batching, flash attention, and thermals. Use the verdict to decide what to try — then confirm by running it. When you do, Wide Area Intelligence turns that GPU into an OpenAI-compatible endpoint with automatic cloud failover for the models it can't hold. Free for 2 nodes.
Frequently asked questions
- What GPU do I need to run a 70B model like Llama 3 70B?
- A 70B model at Q4_K_M needs roughly 40 GB just for weights, plus KV cache and overhead. In practice that means a 48 GB workstation card, two 24 GB cards (RTX 3090/4090) pooled, or Apple Silicon with 64–128 GB of unified memory (which runs it, but slowly). Select the exact model above to see every card that fits and its estimated speed.
- Can I run a 7B or 8B model on a gaming GPU?
- Yes. A 7B–8B model at Q4_K_M is about 5 GB of weights, so any GPU with 8 GB of VRAM or more runs it comfortably, and 12 GB cards handle it with a generous context window. These are the easiest models to run locally and a great starting point.
- How is the estimated tokens/sec calculated?
- Local single-user LLM speed is memory-bandwidth bound: tokens/sec ≈ usable memory bandwidth ÷ bytes read per generated token. We use each GPU's real bandwidth and each model's active parameters (so mixture-of-experts models, which read only a fraction of their weights per token, are scored correctly). These are estimates from hardware specs, not benchmarks — treat them as a ballpark.
- What if no GPU I own can run the model I want?
- That's the normal case for the biggest frontier-class models. Wide Area Intelligence lets you keep every model your hardware can handle local and free, then fail over to a cloud model for the few requests that need more — all behind one OpenAI-compatible endpoint, so your code never changes.
related tools
Ready to use that hardware? Turn your GPU into an OpenAI-compatible endpoint — free for 2 nodes.
/// wide area ai
These numbers are theory. Your GPU is real — put it on the network.
Wide Area Intelligence turns any machine with a GPU into an OpenAI-compatible endpoint — routed, cached, and failed over automatically. Free for 2 nodes.
Start routing — free →