Can you build your own personal AI assistant?

The idea

If you write code, this looks easy now. An API key, a few hundred lines, a vector store, and you have something that answers questions about your notes. You show a friend. Then you use it for two weeks and quietly stop.

It happens often enough to be a pattern, and the pattern is worth knowing before you spend a month on it. The hard parts of a personal assistant are not the parts that are fun to build.

What is genuinely solved

Be fair about how much has been handed to you. Language understanding is done for this purpose. Retrieval over a pile of documents is a well-trodden path. Running something capable on a laptop is realistic. Speech in and out is a commodity. None of these are where the project dies.

Four walls, roughly in this order

• Ingestion. Your life is not in one place, and the places it lives in do not want to hand it over. Calendar, mail, messages, notes, photos, health, documents, banking. Different API, different auth dance, different rate limits, different idea of what an update is. Keeping ten sources fresh is an operational job, not a weekend.
• Identity resolution. The same person is a phone number here, an email there, a display name in a third place and a nickname in your notes. Until those are one entity, you cannot answer the simplest useful question, which is what is going on with this person.
• Memory that is not a log. Storing everything is trivial. Deciding what is worth keeping, what replaced what, what expired, and what was true in March and is wrong now, is the actual product. Skip it and retrieval quality drops as your corpus grows, which is the opposite of what you were hoping for.
• Units and edges. This one surprised us. A short swim is pace per hundred metres, not per kilometre. A ride is average speed, not running pace. Sleep charts belong in the timezone where you slept, not the one your browser is in. Nobody writes a blog post about timezone handling, and it is exactly the sort of thing that makes an assistant feel wrong.

It does not crash, it drifts

The usual failure is quiet. A connector stops refreshing and nothing tells you. The answers get subtly stale. You cannot tell whether an answer is wrong or the data is old, so you start checking every response. Once you are checking, the assistant costs more attention than it saves, and you are done.

The honest summary is that a personal assistant is an infrastructure product wearing a chat interface. The chat part is the last five percent.

Build, or build on top

Building the whole thing is right when the value is in the building: you are learning, researching, chasing an approach nobody has productised, or your requirements are so specific no product will serve them.

Using something existing is right when you want the outcome rather than the artefact, and when you want it still working in six months without you maintaining it.

There is a third option people miss. Build on a context layer that already handles ingestion, identity and memory, and spend your time on the part you care about. That is what the Souvéa personal API is for: your own context, reachable from whatever you build, including from tools like Cursor or Claude Code. Some of the people using it are not building an assistant at all. They are building one narrow thing that happens to need to know who they are.

If you are building it anyway

A few things worth doing from the first commit, because retrofitting them hurts.

• Keep raw source data separate from derived memory, so you can rebuild the memory layer without re-ingesting everything.
• Timestamp and version it all. Personal context is mostly a story about what changed.
• Make deletion real and testable on day one, not a to-do.
• Log what the assistant used to produce each answer, so you can debug a wrong answer instead of guessing.
• Test on a boring week. Assistants that only shine during a crisis do not get used.

Continued

The ceiling is high. An assistant that actually knows your context is a different category from a chatbot, and very few people have one yet. The ceiling just happens to sit on four floors of unglamorous plumbing. Decide which of those floors you want to own.