Can I Run It? — Local AI edition

How "Can I run it?" works for local AI

Running a large language model on your own GPU comes down to two questions: does it fit, and how fast does it generate. This tool answers both from the specs of the hardware you select — no download, no install, no signup.

First, fit. A model needs room for its weights (parameters × bytes per parameter, set by the quantization) plus a KV cachethat grows with your context window, plus a little runtime overhead. If that total is larger than your GPU memory, layers spill into system RAM ("CPU offload"), which is far slower. If it doesn't fit even in VRAM plus RAM, the model won't run at all.

Why memory bandwidth decides your tokens/sec

For a single user generating one token at a time, local LLM inference is almost entirely memory-bandwidth bound, not compute bound. Every generated token requires reading the active model weights from memory once. So a rough speed estimate is:

tokens/sec ≈ memory bandwidth ÷ bytes read per token

A 7B model at Q4 reads roughly 4 GB per token-step. An RTX 4070 moves about 504 GB/s, and real-world efficiency on GPUs is around 65% of theoretical — so you land in the tens of tokens per second. Mixture-of-experts models read only their active params per token, so they punch above their total size. CPU and unified-memory systems run at lower efficiency, which the estimates account for.

What the tiers mean

Does the RTX 4070 / 4090 / Apple Silicon matter most?

For fit, it's the memory sizethat gates you: 12 GB cards comfortably run 7B–13B models, 24 GB cards reach 30B-class, and Apple Silicon with 64–128 GB of unified memory can hold genuinely large models (slowly). For speed, it's the bandwidth: a 4090 at ~1 TB/s generates roughly twice as fast as a 4070 on the same model.

Honest caveats

These are estimates, not benchmarks. Real speeds vary with your llama.cpp build, the exact GGUF quant, your context length, batching, flash attention, thermals, and background load. Treat the tokens/sec figures as a ballpark — useful for deciding what to try, not a guarantee. The honest way to know your number is to run the model.

When you do, Wide Area Intelligence turns that GPU into an OpenAI-compatible endpoint: deploy any Hugging Face GGUF to a node in one click, set your context window, and route requests with automatic cloud failover. Free for 2 nodes.