Oscar Crowe Artificial Intelligence Local LLMs on Cheap Hardware: What Actually Runs on a Raspberry Pi

Local LLMs on Cheap Hardware: What Actually Runs on a Raspberry Pi

A Raspberry Pi 5 can run a genuinely useful large language model, but only if you accept a narrow definition of useful. With the right quantized model, an $80 computer with 8GB of memory will summarize documents, extract fields from forms, classify support messages, and draft short replies at a few words per second. It will not replace a hosted frontier model, and it will not write your strategy. For a small business that needs private, always-on, low-volume AI, that trade is increasingly worth making.

What a Raspberry Pi 5 Actually Has

The Pi 5 is a quad-core ARM processor running at 2.4GHz with up to 8GB of RAM. That is the whole story, and it explains both the possibilities and the limits. There is no neural accelerator. The graphics chip is not built for the matrix math that LLMs need. Inference happens on the CPU, which means speed is measured in single-digit tokens per second for anything but the smallest models.

Memory is the harder constraint than speed. A model must fit in RAM alongside the operating system and anything else you run. An 8GB Pi gives you roughly 7GB for the model, and after quantization that is enough for a 3-billion-parameter model comfortably and a 7-billion-parameter model only with aggressive compression. The 4GB model runs out of room quickly.

Quantization in Plain English

Quantization is the trick that makes local models possible. A model stores its knowledge as numbers, and by default those numbers are stored with more precision than they need. Quantization rounds them to fewer bits. A 7-billion-parameter model at full precision needs around 28GB. At four bits it needs around 4GB. You lose some quality, and the art is choosing the level of rounding that keeps the model useful.

The common formats you will see are Q4_K_M, Q5_K_M, and Q8. The Q4 versions are the sweet spot on a Pi: small enough to fit, usually only a modest drop in accuracy. The Q8 versions keep more quality but cost too much memory. On a Pi 5, start with Q4_K_M for a 3B model and Q4 or even Q3 for a 7B.

The Models That Fit

Several families are worth trying. Llama 3.2 3B is a solid generalist. Qwen 2.5 3B punches above its size in many benchmarks and handles multilingual work well. Phi-3 mini at 3.8B is strong at following instructions. Gemma 2 2B is small and fast. Mistral 7B runs, but it is tight on an 8GB machine and slow. At the tiny end, SmolLM2 1.7B and TinyLlama 1.1B are fast enough for classification and extraction, though they will not hold a long conversation.

Pick one 3B model and one 1.5B model and test both on your actual tasks before you commit.

Realistic Speed and Quality

On a Pi 5 with 8GB, expect roughly 5 to 10 tokens per second from a well-quantized 3B model. A 7B model drops to about 2 to 4. For comparison, a hosted API returns a full paragraph in a second or two. The Pi is not competing on speed. It competes on cost, privacy, and the fact that it never goes offline.

Quality at 3B is good for narrow, structured work. Extracting a name, a date, and an amount from an invoice works well. Summarizing a one-page document works well. Classifying a message as billing, technical, or sales works well. Open-ended reasoning, long documents, and anything that needs broad world knowledge will feel the ceiling. The honest rule: if the answer is in the text in front of the model, 3B does fine. If the model has to know something it was never told, it will guess.

What a Small Business Can Actually Do

The realistic use cases are private and repetitive. A local model can read incoming email and route it to the right person. It can extract line items from supplier invoices and drop them into a spreadsheet. It can redact names and account numbers from documents before anything leaves the building. It can power a small knowledge-base bot that answers from your own handbook for a handful of staff, with no per-token bill.

These jobs share a shape. The input is short. The pattern is fixed. The volume is low. A Pi sitting under a desk can do them all day for the cost of a few watts of electricity.

The Limits You Should Respect

The Pi is not a server for the whole company. Ten simultaneous users will swamp it. A 50-page contract will exhaust the context and slow generation to a crawl. Anything requiring up-to-date facts will fail because a local model cannot search the web unless you add that pipeline yourself. And the model you can run locally is a fraction of the capability of what you can rent by the token.

Treat the Pi as a private appliance for specific jobs, not as a general AI platform.

Getting Started Without Wasting a Weekend

Install Ollama or llama.cpp on the Pi, download a 3B model in Q4_K_M, and run it. Then benchmark it on twenty of your own real prompts, not on a public leaderboard. Measure the tasks you actually care about. If the 3B model clears your bar, you are done. If it does not, a 7B might, and if that fails you have learned something worth knowing before you spend money on hosted APIs.

The local AI community is large and the tooling is mature. You can go from a blank SD card to a working assistant in an afternoon, which is exactly why now is a good time to try.

If you want help picking the right hardware, model, and setup for your own workload, talk to us at Crowe Endeavors and we will build a benchmark from your own data.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Post