Loading benchmark data…
EyesInAI·Loading live benchmark data
What hardware runs what models — a quick VRAM guide plus hands-on notes and reviews.
| GPU / Platform | Memory | What it runs (4-bit) |
|---|---|---|
| RTX 3060 / 4060 | 8–12 GB | Up to ~8B in 4-bit (Llama-3 8B, Gemma 7B, Phi-3) |
| RTX 4070 / 4080 | 12–16 GB | ~13B comfortably, ~30B with offloading |
| RTX 4090 / 3090 | 24 GB | ~30B in 4-bit (Gemma-3 27B, Qwen 32B); 70B needs offload |
| A100 / H100 | 40–80 GB | 70B in 4-bit on one card; larger with multi-GPU |
| Apple Silicon (unified) | 32–192 GB | Memory = headroom: 64GB runs 70B in 4-bit via MLX/Ollama |
Rule of thumb for 4-bit quantization: ~0.6 GB of memory per billion parameters, plus context overhead.
AMD's Ryzen AI Max+ 395 (codenamed Strix Halo) is an APU where the CPU and GPU share a single 128GB LPDDR5X-8000 pool — no separate VRAM. On Linux, 110GB of that pool is addressable by the GPU; Windows gets up to 96GB via AMD Variable Graphics Memory. That ceiling is what matters: an RTX 5090 caps out at 32GB, a 4090 at 24GB. This chip gives a mini PC more than triple either — enough to load a full 235B-parameter open-weight model from a single unified address space. The demo model is Qwen3-235B-A22B, a Mixture-of-Experts: only ~22B parameters activate per token, which is why the math works. Loading it requires Q3 quantization (~101GB). Real-world token generation on the GMKtec EVO-X2 (the most widely available implementation) lands around 11 tokens/second — usable for local development, not cloud-competitive. The bandwidth ceiling is the honest constraint: 256 GB/s versus ~1,008 GB/s on a desktop RTX 4090 or ~800 GB/s on an Apple M2 Ultra. Token speed scales with bandwidth. AMD's own first-party device (Ryzen AI Halo mini PC) opened pre-orders in June 2026 at $3,999. The GMKtec EVO-X2 uses the same chip starting around $1,499 (64GB) to $1,800 (128GB). The "$1,499 kills the $4,000 Nvidia box" headline circulating online conflates the two: Lisa Su's CES 2026 demo used AMD's own unit, while the accessible price point lives in the third-party market. One common misconception: the chip's 50-TOPS NPU rating does not apply to LLM inference. As of mid-2026, Ollama, llama.cpp, and LM Studio route LLM workloads to the iGPU (Radeon 8060S, 40 RDNA 3.5 CUs), not the NPU. The NPU handles fixed-function tasks like video upscaling and image classification. Don't choose a config based on NPU TOPS for language model work. Worth considering if you need 70B–235B models running locally for privacy, compliance, or offline use, and you're not running high-concurrency inference where memory bandwidth becomes the hard limit. At 11 tok/s it's a developer and low-traffic deployment tool, not a production serving node. What actually fits on a 128GB unit (110GB usable on Linux): Qwen3-235B-A22B Q3 (~101GB, ~11 tok/s, MoE) is the headline; Llama 3.3 70B Q8 (~75GB, ~6–8 tok/s) is likely the best quality/speed tradeoff; Qwen2.5 72B Q8, Mistral Large 2 123B Q4 (~73GB), DeepSeek-R1-Distill 70B Q8, Mixtral 8x22B Q4 (~88GB) and Command R+ 104B all fit; anything ≤34B runs fast (15–30 tok/s). What does NOT fit: full DeepSeek V3/V4 and R1 (671B MoE, ~168GB even at Q2), Llama 3.1 405B at any useful quant, and the big cluster models (Kimi K2.6, Qwen3.5 397B, GLM-5.1) which need 500GB+. AMD's CES 2026 stage demo claimed roughly 3× an RTX 5080 on DeepSeek R1 — that figure is AMD's own benchmark and has not been independently replicated at scale. Everything else here is corroborated across multiple independent teardowns. https://www.amd.com/en/products/processors/laptop/ryzen-ai-max/395.html
The DGX Spark is NVIDIA's desktop-class local-inference machine: a GB10 Grace Blackwell superchip with 128GB of unified CPU/GPU memory, running an Ubuntu-based DGX OS. Pricing starts around $3,000 for the base unit and ~$4,000 for the Founders Edition, with OEM configs (e.g. Dell) closer to $4,750. The spec that actually governs its behavior is memory bandwidth: ~273 GB/s. Token generation is bandwidth-bound, and 273 GB/s sits only marginally above a high-end consumer integrated chip and roughly an order of magnitude below a datacenter GPU — an H100 is ~3.35 TB/s. So the Spark is not where you go for fast tokens. Its real advantage is that the 128GB of unified memory lets you hold models that won't fit on a typical discrete GPU. It's a capacity play, not a throughput play. NVFP4 is the precision that makes that capacity usable, and NVFP4 requires the Blackwell generation. Rough capacity: one Spark handles local inference up to ~200B parameters; two units linked over the built-in ConnectX interconnect reach ~405B. A post-launch software update (TensorRT-LLM optimizations plus speculative decoding) claimed up to ~2.5x throughput gains over the launch firmware. Reach for it for development and prototyping against large open-weight models in a CUDA-native environment, or for privacy-bound local inference where fitting the model matters more than raw speed. Skip it for anything latency-sensitive at production scale — a single datacenter GPU's bandwidth and batching win decisively there. https://www.nvidia.com/en-us/products/workstations/dgx-spark/
Unified memory lets M-series Macs run 70B models in 4-bit comfortably via Ollama/MLX. Slower than a 4090 but huge memory headroom.
Runs most ~30B models in 4-bit quant (gemma-3-27b, qwen-32b). Sweet spot for local. 70B needs offloading or two cards.