Generative AI at the Edge
Running Small Language Models with Arduino UNO Q and llama.cpp
Preface
A few years ago, the idea that a language model could run on a board the size of an Arduino would have been a joke. The models were too large, the boards too small, and the gap between them measured in orders of magnitude. That gap has closed faster than almost anyone predicted — not because the boards became servers, but because the models learned to be small.
This book is about what becomes possible on the other side of that closure.
The Arduino UNO Q sits at an interesting point in the hardware landscape. It is not the most powerful single-board computer you can buy, and it is not trying to be. What makes it worth a book is the pairing: a Linux-capable Qualcomm processor that can load and run a quantized language model, bolted to a real-time microcontroller that can act on what the model concludes — both on one board, in the UNO form factor, connected by Bridge. A model can look at an image, decide something about the world, and make a motor turn, with nothing in between and no network involved.
Generative AI at the Edge: Running Small Language Models with Arduino UNO Q and llama.cpp is the official textbook for IESTI05 — Edge AI Engineering at the Federal University of Itajubá (UNIFEI) in Brazil. It is the generative-AI half of a two-book sequence; its companion, EdgeML Made Easy, covers fixed-function AI on the Raspberry Pi. Both are open, and both are written to be useful well outside a classroom — for the self-taught engineer, the maker, the professor building a curriculum from scratch with no budget.
This is not a book of concepts illustrated by code. It is a book of code, explained. You will compile llama.cpp on the board and wait while it builds. You will download weights, pick a quantization, and discover what the difference actually feels like in tokens per second. You will give a small model vision, then tools, then a servo. You will train a classifier on photographs you took yourself and watch an LED respond to it.
You will also hit walls. The model will generate forever and ignore your stop token. The container’s loopback address will not be the board’s loopback address. Reasoning mode will make the answer worse, not better. I have left all of that in, because those hours are where the engineering actually lives, and a tutorial that shows only the clean path is teaching you to be surprised later.
Beyond the technical material, there is a broader point. Intelligence that runs locally — on cheap hardware, offline, with no per-token bill and no data leaving the device — is not merely a technical preference. For a clinic without reliable connectivity, a farm without a data plan, or a school without a cloud budget, it is the difference between having access to this technology and not. That is much of why this book exists, and why it is free.
Whether you are a student meeting edge AI for the first time, a professional adding a new capability, or someone who simply wants to see how far a small model can be pushed on small hardware — welcome. Every expert was a beginner, and the interesting part of this field is that it is still young enough that nobody is very far ahead.
Happy coding, and may your models always converge.
Prof. Marcelo Rovai
September, 2026