Vector search is useful. It is not the whole retrieval problem.
In an agent memory system, the question is not only "what is semantically similar?" The better question is "what context should this agent be allowed to use for this task?"
That is a judgment layer.
Job
Retrieval selects working context from memory.
It should narrow the field before the model reasons. Otherwise the model inherits every ambiguity in the archive and turns it into confident prose.
Inputs
The retrieval layer needs:
- the user request
- the skill being invoked
- the project or domain
- permission boundaries
- time sensitivity
- required source types
- memory status
- known contradictions
The same note may be useful for one task and unsafe for another. A private scratch note might help the owner think. It should not become source material for a shared output.
Processing
Retrieval should combine semantic search with filters:
- freshness: is this still current?
- source: where did it come from?
- status: active, draft, superseded, archived
- confidence: reviewed or unreviewed
- scope: personal, project, team, public
- fit: does this kind of source match this task?
- contradiction: does another record disagree?
The system should also retrieve less when the task is narrow. More context is not always more intelligence. Sometimes it is just more surface area for error.
Output
The output is a context packet, not a pile of chunks.
A context packet should include the selected records, citations or source trails, exclusions when important, and a short reason each record was included.
That reason is not decoration. It lets the human reviewer see whether the system understood the task.
Review Question
The review question is: if the answer is wrong, can I inspect what context caused it?
If not, retrieval is functioning as vibes.
Failure Mode
The failure mode is relatedness masquerading as relevance.
The model finds notes that sound close, then writes an answer from them. The answer may be fluent. It may even be plausible. But if the retrieved context was stale, private, unreviewed, or wrong for the task, the system has already lost.