Run AI on Your Own Hardware
The moment you paste a contract, a customer list, or source code into a cloud chatbot, it's on someone else's server. Local AI tools run models entirely on your machine — your data never leaves. This is the fastest-growing category in privacy, and it's all open source.
The honest take on local AI
The open-source tools below have no affiliate programs — they're open source, so nobody pays us to recommend them. We list them on merit because they're genuinely the right answer for a specific, growing problem: how to use AI without handing your data to a third party. If that's your threat model, this page is the setup you need. (One exception below — Proton's Lumo — earns us a commission, and it's flagged as such.) Hardware requirements: a modern laptop can run small models (7B) acceptably; for serious work you want a desktop with 16GB+ RAM and a decent GPU.
The Setup
Three pieces: a runtime to run models, an interface to talk to them, and a document tool when you need retrieval over your own files.
One command installs and runs Llama, Mistral, Qwen, and hundreds of open models on macOS, Linux, or Windows. Built-in OpenAI-compatible API means every AI tool you already use can point at it. It's how we run our own private assistant stack — zero cloud, zero account. If you try one local AI tool, make it this one.
Visit Ollama ↗If Ollama is the terminal-first option, LM Studio is the GUI-first one. Browse, download, and chat with models without touching a command line. Excellent for testing different models side by side to find what fits your hardware. Same local-only privacy story — everything runs on your machine.
Visit LM Studio ↗Point it at a folder of PDFs, contracts, or notes and ask questions with full retrieval — all local, all private. The reference implementation for private RAG (retrieval-augmented generation). Setup is more involved than Ollama, so it's for the technically comfortable — but it's the pattern every "chat with your documents" product is built on, done without the cloud middleman.
View on GitHub ↗A drop-in, OpenAI-compatible API server that runs fully self-hosted — text, audio, image, and embedding models. If you're building an internal tool and want to swap the OpenAI endpoint for a local one without rewriting code, this is the cleanest path. Docker-friendly, works on CPU-only boxes (slower, but works), and the most complete self-hosted API surface of the group.
Visit LocalAI ↗Sometimes local hardware isn't enough — you need a frontier model, or you're on a laptop that can't run one. That's where Lumo fits: Proton's AI assistant with the same zero-access architecture as Proton Mail. Every conversation is end-to-end encrypted, never logged, never used for training, and hosted on European infrastructure. It can't read your data any more than it can read your inbox — the privacy property is identical to the local tools above, just without the hardware requirement. Also includes an image model and web search with citations.
🔒 The privacy difference
With cloud AI, your prompts are the product input — logged, retained, and often eligible for training use. With local AI, the model weights live on your disk, inference runs on your CPU/GPU, and the only copy of your data is the one you control. For anyone handling PII, PHI, or proprietary material, that's not a trade-off — it's the whole point. This is the same reasoning that drives our broader work on local-first data handling.
Want the short list, monthly?
New finds, updated reviews, and the occasional take on what the security industry gets wrong.
Subscribe Free