Running AI Models on a Raspberry Pi 5 - Part 1
One fine lazy weekend, I was reading about something called Edge AI - typically means that the AI runs on the end machine than the remote cloud server. As much as it sounded tempting, I knew that you were crazy if you said that I could run a 8b+ param LLM on a mid range phone. Then I looked at my Pi 5 that I haven’t used in quite a while. I haven't written a tech article in a real while and then thought -
“Why not waste my Saturday trying to run models on this?”
I had to clean up a few stuff to make room for the models that I’ll be downloading. I was using my Pi for retro gaming and as a media streaming hub. Hated to see the NFS and FIFA ROMS go, but I'll download them back again.
After several consultations with ChatGPT and some googling (Yeah, I still google and also ignore Gemini’s summaries) to see people have got a few models working on Ollama on a Pi 5. But I was determined to see how far can I push a 8GB RAM and a quad core CPU.
At this point, I had no idea what I read in most of the pages - attention, KV cache, MoE, quantization, prefill. I’ve never heard of them - all Latin. One of my mentors at my previous org used to say - “You don’t just know anything in software unless you dig in and learn to look - past the abstractions”. By the end, I knew what these terms meant.
With hopes high, I then installed llama.cpp and decided to build for my chip rather than picking a built binary.
The CPU build was quick - I built it with defaults for ARM and did not play much with the CMake flags (some other day) but I wanted a Vulkan build that could use the Pi’s GPU to squeeze out more tokens. Despite people’s warnings and failed attempts stating that Vulkan works out poorly for Pi and llama cpp - I went ahead (Some dumb heads just don’t listen).
I got it to successfully build anyway, I had to delete some code that used the latest vulkan headers. Pulled a 0.8B Qwen3.5 model from huggingface and waited for it load only to be greeted by the GOATed C error - segmentation fault. I then realized that the GPU used a shared memory that was too small for matrix multiplication and lacked a few more computational abilities while being very good at graphics. Gave up at this point (Fortunately, I wasn't as stubborn as I used to be this time, being an optimist doesn't always help).
I started running the llama bench with a few models of different architectures to see if they indeed make a difference. Let’s dive into the technical details:
| Component | Setup |
|---|---|
| Board | Raspberry Pi 5 |
| RAM | 8 GB |
| CPU | 4× ARM Cortex-A76, up to 2.4 GHz |
| CPU ISA features | ARM64 / NEON / dot-product support |
| Runtime | llama.cpp |
| llama.cpp build | 9b0a2ce85 / build 10442 |
| Primary quantization | Q4_K_M unless otherwise specified |
- Qwen3.5-0.8B - a small hybrid model using Gated DeltaNet plus periodic full attention.
- Gemma 3 1B - a small model using a mix of local sliding-window and global attention.
- Llama 3.2 1B - a conventional dense autoregressive Transformer with grouped-query attention.
- OLMoE-1B-7B - a Mixture-of-Experts model with roughly 6.9B total parameters but only a fraction active for each token.
I tested them for their speed of:
- Prompt processing / prefill (
pp): ingesting many input tokens at once. - Token generation / decode (
tg): autoregressively producing one new token at a time.
More CPU cores ≠ faster generation
Prefill scales well with more cores. Generation does not. Two threads were often faster than three or four.
🧠Me: I paid for four cores, I’m going to use four cores.
llama.cpp: That’s not how memory bandwidth works, chief.
A reasonable interpretation is that prompt processing has enough arithmetic work to scale across cores, while token-by-token decode becomes increasingly constrained by memory bandwidth, cache behavior, and sync overhead.
Quantization is not just "fewer bits = faster"
This was another fascinating thing to learn. A FP32 (usually the default while training) value takes 4 bytes, for a model with 800M parameters which should take roughly 3.2 GB just to store the weights. But the model was just 500-700 MB. Apparently quantization converts those FP32 to INT4 which takes 0.5 bytes/weight, thus bringing the model size down significantly. I still have no idea on how we manage to retain LLM quality.
I kept Qwen3.5-0.8B fixed and compared Q4_K_M, Q6_K, and Q8_0.
| Quant | Size | pp512 @4t | tg128 @2t | PPL100 ↓ |
|---|---|---|---|---|
| Q4_K_M | 492.61 MiB | 92.77 | 17.52 ★ | 19.4128 ± 0.3715 |
| Q6_K | 590.12 MiB | 84.17 | 14.01 | 18.9349 ± 0.3631 |
| Q8_0 | 763.78 MiB | 111.84 ★ | 12.04 | 18.6712 ± 0.3575 ★ |
Perplexity improved as precision increased, but the quality gain was modest relative to the storage and decode-speed cost. For this Pi, Q4_K_M looked like a strong interactive operating point rather than a universally best quant.
Context length can completely reverse which model is “faster”
This experiment measured token generation after filling the model with increasing amounts of context.
| Context | Qwen3.5 0.8B | Gemma 3 1B | Llama 3.2 1B |
|---|---|---|---|
| 0 | 15.97 ★ | 12.83 | 15.30 |
| 512 | 15.02 ★ | 12.29 | 12.38 |
| 2,048 | 12.12 ★ | 11.98 | 6.78 |
| 4,096 | 9.75 | 11.46 ★ | 3.72 |
| 8,192 | 6.89 | 10.44 ★ | 2.12 |
At empty context, Gemma looks like the slowest model.
At 8K context, Gemma is almost five times faster than Llama and around 1.5x faster than Qwen.
💬 Llama at empty context: "I am speed."
Llama after 8K tokens of conversation: "I was fast before you told me your life story."
Why the curves differ?
Each of the models tend to solve the history problem differently.
A conventional full-attention layer still has to compare the current query against a growing history. GQA (Group Query Attenttion) reduces the amount of KV state, but it does not remove the need to operate over the context.
Gemma’s local attention changes that scaling behavior more fundamentally: most layers work over a bounded recent window, while only periodic global layers see the broader history.
Qwen’s DeltaNet-heavy hybrid lands between those two extremes.
Oh my my, I have cursed LLMs and their poor context for so long. But understanding that attention is costly because standard transformer models scale quadratically (O(n²)) with context length makes you feel empathy for them. It’s like you finally see why someone is the way they are.
Memory growth tells the same architecture story
| Context | Qwen RSS | Gemma RSS | Llama RSS |
|---|---|---|---|
| 0 | ~1106 MiB | ~1297 MiB | ~1613 MiB |
| 512 | ~1184 MiB | ~1371 MiB | ~1700 MiB |
| 2,048 | ~1222 MiB | ~1391 MiB | ~1798 MiB |
| 4,096 | ~1273 MiB | ~1409 MiB | ~1928 MiB |
| 8,192 | ~1373 MiB | ~1445 MiB | ~2188 MiB |
From empty context to 8K:
| Model | Extra Peak RSS | RSS / Context Token | Decode Slowdown |
|---|---|---|---|
| Gemma 3 | ~149 MiB | ~19 KiB/token | ~19% |
| Qwen3.5 | ~267 MiB | ~33 KiB/token | ~57% |
| Llama 3.2 | ~575 MiB | ~72 KiB/token | ~86% |
The strongest example is Gemma vs Llama: their Q4_K_M GGUF files are almost identical in size - around 762 MiB - yet their 8K runtime behavior is radically different.
A runtime data-layout change produced a 2.47x prompt speedup
For Llama 3.2 1B I compared normal Q4_0 execution with llama.cpp’s ARM repacking enabled and disabled. The model, prompt, machine, quantized weights, and thread count were otherwise unchanged.
| Mode | Prompt | Generation |
|---|---|---|
| ARM repacking ON ⚡ | 67.8 t/s | 15.7 t/s |
| ARM repacking OFF | 27.4 t/s | 12.0 t/s |
That is approximately:
- 2.47x prompt throughput (+147%).
- 1.31x generation throughput (+31%).
Nothing about the neural network changed. Just by optimizing how and where the weights are stored and arranged in memory speeded up the inference significantly.
MoE separates compute cost from model-storage cost
OLMoE was the first model in the project that clearly belongs in a larger-model class and I was excited to try that out. I also tried Granite MoE later and that was also impressive.
The GGUF contained:
- 6.92B total parameters
- 3.92 GiB model file
- llama.cpp classification:
olmoe A1.7B
Yet its short-context generation speed was striking:
| Threads | pp128 | tg64 |
|---|---|---|
| 1 | 15.76 | 11.50 |
| 2 ★ | 31.26 | 16.25 ★ |
| 3 | 43.72 | 15.14 |
| 4 | 49.77 ★ | 14.39 |
Despite having about 5.6x the total parameters of Llama 3.2 1B, OLMoE produced roughly the same short-context decode throughput.
That is sparse activation becoming visible on the Pi. The inactive experts are not zero weights. They are normal learned parameter sets that simply are not selected for a particular token.
Here's the catch, there's always one: memory
On the cached OLMoE benchmark, peak RSS reached about 7.23 GiB on an 8 GB Pi.
MoE saves compute. It does not make the inactive experts disappear from storage.
I could barely keep conversing with the model. The speed kept going downhill as I kept chatting. The Pi started heating up and fans actively ran but it still kept running.
Quality reminder: parameter count and throughput are not intelligence metrics
I asked OLMoE and Granite a couple of questions and it wasn’t that horrible as I would have expected although it did hallucinate at times. An eval experiment would be nice for another day.
Where next?
- Understand inference runtimes - try LiteRT, ONNX
- Use the Hailo-8L that's been lying on my table for quite a while
- See if I get a model to use my Pi's GPU via Vulkan
- Eventually build a tiny runtime or kernel experiment: quantized MatMul, KV/state handling, and hardware-specific packing - a very ambitious experiment that could easily be my final year project!
Comments
Post a Comment