Best GPUs for Running AI Locally: What Really Matters
How to choose a GPU for local AI: why VRAM matters, how model size and quantization affect memory, consumer vs pro cards, Apple Silicon and budget tiers.
How to Run AI Models Locally: A Beginner’s Guide
How to run AI language models on your own computer: why you might, hardware needs, beginner-friendly tools, choosing and quantizing models, and limits.
Key takeaways
- Local models keep your data on your device and work offline, with no per-use fees.
- Memory is the key constraint: the model must fit in your GPU’s VRAM or your computer’s RAM.
- Quantized models trade a little quality for much smaller size, making local AI practical on normal hardware.
On this page
You don’t need a data centre to use AI. Open models that run on a laptop or desktop have improved dramatically, and easy tools now let you download and chat with one in minutes. Here’s what you need to know to get started.
Why run AI locally?
- Privacy: prompts and documents stay on your device, with no cloud privacy settings to manage.
- Offline use: no internet connection needed once the model is downloaded.
- No usage fees: after the hardware, running the model is free.
- Control: choose, customise and combine models as you like.
The trade-offs: local models are usually less capable than the largest cloud models, they need decent hardware, and you are responsible for setup and updates.
What hardware do you need?
The key question is memory. The model has to fit in your graphics card’s memory (VRAM) for fast performance, or in your system RAM for slower performance. Our GPU buying guide for local AI explains what to prioritise.
| Setup | What it can typically run |
|---|---|
| Laptop with 8–16 GB RAM | Small models (a few billion parameters), slower responses |
| Apple Silicon Mac with 16–32 GB unified memory | Small to mid-sized models, often with good speed |
| PC with a GPU with 8–12 GB VRAM | Small to mid-sized quantized models at good speed |
| PC with a GPU with 24 GB+ VRAM | Larger quantized models comfortably |
These are rough guides: speed and what fits depend on the model, its size and how it’s compressed.
Understanding model sizes and quantization
Model size is measured in parameters, for example 7 billion (7B). More parameters generally means more capable, but also more memory.
Quantization compresses a model by storing its numbers at lower precision, for example 4-bit instead of 16-bit. A 4-bit quantized model needs roughly a quarter of the memory of the full-precision version, with a modest drop in quality. That is what makes local AI practical on ordinary computers.
Beginner-friendly tools
- Ollama: a simple tool for downloading and running models from the command line, with a local API other apps can use.
- LM Studio: a desktop app with a friendly interface for browsing, downloading and chatting with models.
- llama.cpp: the open-source engine behind many local AI tools, for those who want more control.
Getting started in four steps
- Install a tool such as Ollama or LM Studio.
- Choose a small model first, a well-known open model in the 3B to 8B range.
- Download a quantized version that fits your memory.
- Start chatting, then try summarising a document or drafting text to test quality.
Choosing a model
- General chat and writing: well-known instruction-tuned open models.
- Coding: models trained specifically for code.
- Small and fast: compact models for older hardware.
- Check the licence: some open models restrict commercial use.
Start small, test on your real tasks and move up in size if you need better answers.
Serving a local model to a team
Running a model for several people on a shared machine raises new problems: slow responses when many people ask at once and repeated work for identical questions. Techniques like caching repeated responses help a lot; Backend Architect’s guide to caching patterns explains the approaches engineers use. To let the model answer from company documents, add retrieval-augmented generation (RAG).
Limitations to expect
- Smaller models make more mistakes and can be confidently wrong.
- Context windows may be shorter than cloud models’.
- Performance drops sharply if the model doesn’t fit in fast memory.
Frequently asked questions
Is running AI locally really private?
Your prompts stay on your device when you run a model locally, but check each tool’s settings, since some offer optional cloud features or telemetry.
Can I run a local model without a GPU?
Yes. Small models run on CPU and RAM alone, but more slowly. Apple Silicon Macs perform well thanks to unified memory.
Are local models as good as ChatGPT or Claude?
The largest cloud models are generally more capable, but local models are good enough for many tasks, especially with careful prompting.
Sources
Every article is edited by a human and checked against our editorial policy. Spotted a mistake? Tell us.
Keep reading
Prompt Engineering: A Practical Guide With 10 Templates
Practical prompt engineering: the six elements of a strong prompt, techniques that reliably improve answers and ten copy-ready templates for everyday work.
How to Spot AI-Generated Images, Video and Voice Clones
How to spot deepfakes and AI-generated media: visual tells in images and video, signs of voice clones, verification tools and how to protect yourself.