Desk Calendar Agent
You send the agent a message like "lunch with John Appleseed on Friday at noon." A 3B parameter model running on the Raspberry Pi reads that into a title, a date and a time, creates the event through the Google Calendar API, and the e-ink panel updates. An 80%-accurate model, with zero destructive actions ever reaching the calendar.
Overview
There are two ways to reach the agent, both in the web UI: type the request, or hold the record button and speak it, which whisper.cpp transcribes on the Pi and shows back to you before the agent acts on it. Either way the request goes through the same path. The 7.5 inch e-ink panel shows the day, week, or month, and three physical buttons on the device switch between them.
The model is Qwen2.5 3B, quantized to 4-bit and served by Ollama on the Pi. Nothing about the reasoning leaves the device, and neither does your voice, since whisper.cpp transcribes locally too. The only thing that goes out is the Google Calendar API call that makes the edit.
The device
A Raspberry Pi 5 and a Waveshare 7.5 inch e-ink panel, 800x480 at one bit per pixel. The panel runs off its own driver HAT over a 9-pin cable, which leaves the GPIO header free for the three view buttons on the breadboard. Those switch straight to ground and lean on the Pi's internal pull-ups, so there are no resistors on the board.
The interface
The web UI is a chat log: you send a request, the agent says what it did, and the week view under it redraws. FastAPI serves it off the Pi, and Tailscale makes it reachable from anywhere. Tailscale also gives it a real HTTPS certificate, which is what makes voice input work at all, since browsers only allow microphone access on a secure origin.
Full demo
A 40-second run: a request typed into the web UI, the 3B model working it out on the Pi, and the e-ink panel redrawing with the new event.
The accuracy problem
Problem: the 3B model picks the right tool 80% of the time. That is not good enough for something that can delete appointments off your real calendar. Qwen2.5 7B scores better, but it ran 5x slower on a laptop, before even moving to the Pi's much weaker processor, and the 14B does not fit at all: 9 GB at 4-bit against the Pi's 8 GB of RAM. So the accuracy had to come from code instead of a bigger model, which is the better trade here anyway. The checks run instantly and cost nothing, while a bigger model would cost seconds on every single message.
Solution: at first I tried prompt engineering, but every problem I fixed just turned into a different one, so the overall failure rate barely moved. What actually worked was moving the decisions the model kept getting wrong out of the model entirely. It still picks the tool and names the event, which it is good at, and Python checks everything else before anything irreversible runs.
Validation layer
After testing I found five decisions the model kept getting wrong, so I moved all five out of it:
- Relative dates. The model cannot see today's date, so "next Friday" is resolved in Python.
- Ambiguous references. It acted on the first matching event instead of asking which one I meant.
- Negation. It read "don't delete my dentist appointment" as a delete instruction.
- Verbs inside event names. "Remind me to cancel the gym" rewrote an existing gym event.
- Missing details. It invented a 09:00 start time and silently dropped "every week".
Across 462 labelled cases the model proposed 54 destructive actions it should not have. The validation layer caught all 54, so none of them reached the calendar.
Numbers
Accuracy across 462 labelled cases, 3 samples each:
| Model | Tool choice | Arguments | Destructive FPs |
|---|---|---|---|
| qwen2.5:3b, local, on device | 79.8% | 90.6% | 0 |
| claude-sonnet-5, hosted, as a ceiling | 96.1% | 96.7% | 0 |
Claude Sonnet is in the table as a ceiling, not as a fallback, and it was scored on a 51 case subset rather than the full 462. No app module on the live agent path even imports the hosted client, and a test fails if one ever does.
Speed, measured on the Pi:
- Warm turn: ~60 s
- Cold turn, the first after a reboot: ~100 s
- Speech transcription: ~2.2× realtime (a 3-second clip takes ~6 s)
- Temperature, idle and under load: 33 °C and 57 °C, no throttling; swap used: 0 B
Choosing the model
I tested five models on the same 50 cases, one sample each:
| Model | Tool accuracy | Destructive FPs | Wall time |
|---|---|---|---|
| qwen2.5:7b | 88% | 2 | 263 s |
| qwen2.5:3b | 78% | 0 | 54 s |
| llama3.2:3b | 66% | 1 | 73 s |
| qwen3:4b | 46% | 0 | 2075 s |
| hermes3:3b | 20% | 0 | 75 s |
The 7B scored higher and was still the wrong pick. It proposed two destructive actions the 3B never did, and it was the quickest of the five to reach for a destructive tool on its own.
Hardware constraints
Refresh is expensive. A full e-ink refresh takes about four seconds and visibly flashes, and the panel only has a finite number of them in it. So the display hashes the image it is about to draw and redraws only when that hash changes. The hash ignores the minute on purpose, since otherwise the clock alone would force a refresh on every poll.
Presses outrun the panel. Button presses overwrite instead of queueing. A refresh takes four seconds, and more presses arrive during it.
Prompt caching. Prefilling the 2.2k-token system prompt costs about 100 seconds on the Pi, a cache hit about 6. Ollama caches by longest common prefix, so the timestamp rides on the user turn. Anywhere in the system prompt and every message pays the prefill again.
The panel is the last link. The display is output only, so the whole agent is testable with no hardware attached. The e-ink path is the last thing in the chain and nothing upstream depends on it, which means the model, the validation layer and the calendar client all run and get tested on a laptop.
Testing it the way it actually runs
A whole class of bug cannot appear until the system runs the way it really runs, and the only defence is a harness shaped like the real thing rather than shaped like whatever is convenient to test.
Every manual test of the display was a single render, and every one of them passed. On the device the
display is a loop, and only the first refresh ever worked. The panel's sleep() ends in
SPI.close(), and the connection was only opened once when the object was built, so every
refresh after the first had nothing to write to. The fix was to open and close the connection inside
each refresh instead of once at startup.
Known limitations
Speed. A warm turn takes about 50 seconds, and that is what it costs to run the model on hardware I own rather than someone else's.
Long requests time out in the browser. A turn can outlast the browser, so the page sometimes shows a network error for something that actually worked. The event still gets created and the turn still gets logged, so it is a reporting problem rather than a lost request. The fix is to acknowledge the message right away and poll for the result instead of keeping one request open the whole time.
Tech stack
About 8,100 lines of Python and 590 tests, none of which need the hardware attached.