JASPER — Self-Hosted LLM Agent Backend
JASPER is a personal assistant I run on my own hardware and reach from my phone. The point of building it was not the assistant. It was to understand what actually goes into an agent that can take actions on your behalf without becoming something you have to supervise.
The loop
The core is a multi-turn tool-use loop against the Anthropic API. The model receives a system prompt assembled from current context, decides whether it needs a tool, and the orchestrator executes the call, feeds the result back into the conversation, and lets the model continue. That cycle repeats until the model stops asking for tools or hits a bounded iteration limit.
The bound matters. An agent that can call tools in a loop can also loop forever on a task it cannot finish, so the ceiling is a deliberate design decision rather than an oversight.
Thirteen tools, one handler table
The agent can create and search notes, create and update and complete tasks, set and list reminders, store and search long-term memories, and read the current time. Each tool is declared with a JSON schema describing its parameters, and each maps to a Python function backed by SQLite.
Every model-selected action routes through a fixed handler table rather than dynamic dispatch. If the model names a tool that is not in the table, the dispatcher returns an error back to the model instead of trying to execute anything. Unrecognized or hallucinated calls fail closed and degrade into a retry.
That is the piece I would defend hardest. The interesting failure mode with a tool-using agent is not that it refuses to act, it is that it acts on something you never gave it. Making the dispatcher the narrow waist means the set of things JASPER can do is a list I wrote, not a string the model produced.
Memory that survives a restart
Conversations persist to the database as they happen, and on startup the server resumes the active conversation and reloads its recent history. Restarting the backend no longer wipes context, which was the single most annoying property of the earlier version.
The live context window is bounded to the most recent turns rather than the full history, so a long-running conversation does not grow without limit.
Separately, every exchange gets mined in the background for durable facts worth keeping. That extraction runs as an async task rather than inline, so writing to memory never adds latency to the response you are waiting on.
What is not built
No voice layer yet. The architecture document plans Whisper for speech-to-text and Piper for speech output, and there is a Docker config waiting for it, but today I interact with JASPER as text over Tailscale from my phone.
No test suite yet either, which is the next thing I am fixing. An agent whose main safety property is "unknown tools fail closed" should have a test that proves it.
Stack
Python, FastAPI, SQLite, the Anthropic API with tool use, async task handling, Docker, Tailscale for network access.