About this Book

Generative AI at the Edge: Running Small Language Models with Arduino UNO Q and llama.cpp is a project-based book. Every chapter ends with something running on the board, and the concepts are introduced where they are needed rather than in a theory section you are expected to absorb first.

It is part of the AIEng4D (formerly TinyML4D) initiative, which works to make embedded machine learning education available everywhere — particularly in the developing world, where affordable, offline-capable, locally-run AI is not a convenience but a requirement.

Key Features

  1. Everything runs locally. No cloud inference, no API keys, no hosted endpoints. Models are downloaded, quantized, and executed on your own hardware — your PC in the opening chapter, the UNO Q itself from there on.
  2. The dual-brain architecture throughout. Projects use both processors — Python and models on the Linux MPU, real-time actuation on the MCU — connected through Bridge RPC. This is the capability that distinguishes the UNO Q from a general-purpose SBC.
  3. Honest performance numbers. Each chapter reports measured tokens per second, memory footprint, and latency on real hardware, including the cases where the result is disappointing.
  4. Open tooling. llama.cpp, GGUF models from Hugging Face, Edge Impulse Studio, and Arduino’s own App Lab and arduino-app-cli — all free, all installable, none requiring a subscription to keep working.
  5. Failure modes included. Runaway generation, container networking surprises, unreliable tool calls, and thermal throttling are documented alongside the fixes, because you will meet them.
  6. Terminal-first, IDE-optional. The entire book can be worked through over SSH from any machine. VS Code Remote-SSH is shown for those who prefer it, but is never required.

Structure and Organization

The book has two parts, and the order is deliberate: generative AI comes first because it is the focus of this book.

Part 1 — Generative AI (Proactive)

Chapter What it covers
SLMs at the Edge Local inference on hardware you already own: quantization, GGUF, RAM budgeting, and llama.cpp / Ollama / LM Studio on a PC
Arduino UNO Q Setup Flashing Linux, ADB, SSH, Wi-Fi, arduino-app-cli, pre-installed examples
Generative AI at the Edge Building llama.cpp from source, running Qwen3.5 from the CLI and a local llama-server, and calling it from Python with the openai client
Multimodal AI at the Edge Vision projectors, image understanding, model selection, reasoning-mode pitfalls
GenAI Meets the Real World Dual-brain dengue risk classifier — llama-server as a systemd service, Flask, Bridge to the MCU
Agentic AI at the Edge Native tool calling — the model decides which tools to run and in what order, starting hardware-free (system info, files, a calculator, the LED matrix) and extending to real sensors and actuators

Part 2 — Fixed Function AI (Reactive)

Chapter What it covers
Image Classification Pre-installed Brick, custom model in Edge Impulse, App Lab deployment, live camera, MCU actuation
Object Detection Custom YOLO-based detection with the same App Lab and Bridge workflow

Each chapter follows a consistent shape: what the chapter builds and why, hardware and software requirements, step-by-step implementation with complete code, measured performance, troubleshooting, exercises, and a resources table.

On Part 2, and the Companion Book

If you have worked through EdgeML Made Easy — the companion volume built around the Raspberry Pi — you have already trained image classifiers and object detectors, and you already understand what these models do. Part 2 does not repeat that.

What Part 2 teaches is the UNO Q deployment path, which is genuinely different and is not covered in the other book:

  • App Lab and Bricks — Arduino’s application framework, where a model, a web UI, and a sketch are packaged as one deployable app
  • arduino-app-cli — creating, building, running, and managing those apps entirely from the command line
  • Bridge-driven actuation — inference results on the Linux side triggering physical output on the MCU in the same application

In other words: the task is familiar, the toolchain is new. If you are coming to this book without the Raspberry Pi background, Part 2 still works as a complete introduction — the vision concepts are explained from the ground up.

Why This Board, in a Classroom

The UNO Q does not replace the microcontroller boards or the Raspberry Pi in the IESTI05 curriculum — it complements them by occupying a middle ground that turns out to be pedagogically useful:

  • One project spans two worlds. Students deploy an ML model in Python and control physical hardware from an Arduino sketch simultaneously, on one board. The MCU/SBC boundary stops being an abstraction described in lecture and becomes something they wire across.
  • Real Linux skills, on an Arduino. The board runs standard Debian, so SSH, package management, virtual environments, systemd, and Git are practiced for real — and all of it transfers directly to Raspberry Pi, cloud servers, and production edge devices.
  • A low step up, not a fresh start. Students who met Arduino in an introductory course recognize the form factor, the pin layout, and the shield compatibility immediately. What they are adding is Linux, not beginning again from nothing.
  • Generative AI becomes reachable. The UNO Q is the first board in this course’s lineup that can run local SLMs — previously a Raspberry Pi 5-only capability — while remaining a capable fixed-function board in its own right. Buy the 4 GB variant for Part 1. The 2 GB board manages only the smallest models (SmolLM2-360M and below), and even a 0.8B model is marginal on it once the OS has its share — fine as a demonstration, not as a base for real applications.
  • First-class Edge Impulse support, with pre-loaded models for image classification, object detection, keyword spotting, and anomaly detection: the same task set taught across the rest of the course.
  • Affordable to deploy in quantity. At roughly $60 for the 4 GB variant — the one a generative-AI course needs — the UNO Q costs about what a Raspberry Pi 4 does, while including the MCU subsystem, Wi-Fi, Bluetooth, and an LED matrix out of the box, with no HATs or accessories required for the basic projects.

Prerequisites

  • Basic Python programming
  • Comfort with Linux command-line basics over SSH
  • An elementary understanding of machine learning concepts
  • Arduino sketch familiarity — helpful for the MCU chapters, not required

Hardware You Will Need

Item Notes
Arduino UNO Q Get the 4 GB variant. Part 1 requires it in practice. The 2 GB board is fine for all of Part 2 (vision), but too tight for serious generative work
Data-capable USB-C cable Charge-only cables will not enumerate over ADB
Powered USB hub with PD Required whenever a webcam and power share the single USB-C port
USB webcam For the multimodal and vision chapters
Breadboard, LEDs, jumper wires For the Bridge actuation projects

Optional but useful: a microSD-free setup means everything lives on eMMC, so a second UNO Q is the cheapest form of backup if you are teaching with these materials.

License and Use

These materials were created for IESTI05 — Edge AI Machine Learning System Engineering at UNIFEI and are released under the GNU General Public License 3.0. You are free to use, adapt, and redistribute them, including for teaching, provided derivative work carries the same license. If you adopt this book in a course, I would genuinely like to hear about it.