01Most inference never needed a GPU
Independent analysis of the Hugging Face ecosystem shows the models people actually run are small — small enough to fit comfortably on a modern CPU once quantized.
02When to choose CPU vs GPU for inference
This list is GPU-skeptical, not GPU-hostile. Here's the honest split — the full decision flowchart and "when you actually want a GPU" section live in the repo.
| Dimension | Lean CPU | Lean GPU |
|---|---|---|
| Workload | Inference | Training, fine-tuning |
| Model size | ≤ 13B (quantized) | 70B+ dense |
| Throughput | Single-digit req/s | Hundreds req/s, batched |
| Latency SLA | 100 ms – 2 s TTFT ok | < 50 ms TTFT |
| Context length | ≤ 8K tokens | 32K+ prefill-heavy |
| Deployment | Edge, serverless, SBC | Dedicated cluster |
| Cost / availability | Ubiquitous, no VRAM cap | 5–20× more, VRAM ceiling |
03CPU-native runtimes & inference engines
Engines built for CPU as a primary target — not GPU frameworks that fall back to a slow CPU path.
llama.cpp
C/C++ LLM inference designed for CPU from day one; GGUF, x86/ARM/RISC-V.
ONNX Runtime (CPU EP)
Production-grade CPU execution provider with operator fusion and INT8.
OpenVINO
Intel's toolkit targeting x86 CPU as first-class hardware.
ncnn
Dependency-free mobile/embedded inference for ARM and x86.
ExecuTorch
PyTorch on-device runtime; ARM CPU (XNNPACK) is the primary target.
llamafile
Single-file LLM executables that run on CPU with no install.
See all runtimes, plus a format/arch/OS comparison table, in the repo.
04What's inside
Plus companion guides: a cost calculator, green-inference power math, benchmark methodology, and troubleshooting.
05Contribute
Know a CPU-native runtime, a real benchmark, or a broken link? Contributions are welcome — there are one-click issue templates for suggesting a resource, reporting a stale link, or disputing a benchmark figure.