The Ultimate Guide for a Working Agent Memory

I tried countless systems, developed many on my own, and saw that AI always saves the most primitive stuff in its memory and makes a mess of everything. Then it hallucinates even further, saying the memory is great when it's ass. For months, I've gone through this cycle until I found something that works. In this article, I will share the exact steps with you so that you can finally have a working agent memory, too. Let's dig deep: According to my measurements, 84% of what typical AI learning systems capture is primitive operational telemetry. "Bash then Edit sequence used." "Tool timeout 41% of the time." -> ABSOLUTE GARBAGE STUFF

The fix was counterintuitive: use Obsidian-style markdown files as the memory backbone, not just background databases. SO THAT YOU CAN READ WHAT IS STORED.

1. Most Agent Memory is Actually Noise #

Every time you start a new conversation with an AI agent, you start from zero. Your context, your preferences, the decisions you already made, the rabbit holes you already explored... gone.

But the deeper problem isn't just forgetting. It's when AI systems do remember, they remember the wrong things. Most agent memory systems proudly store operational telemetry that no human would ever want to see again. > Tool sequences > timing metrics > file modification logs. The system feels productive because it's writing things down, and if you ask your agent, it has a "memory," but it's writing down garbage.

Quality-gated memory isn't easy to create, it's actually pretty hard, and it is the area AI hallucinates the most on. Because that distinction between what's worth remembering vs. what's noise, isn't something Ai can decipher by itself, it needs YOU, to see what's going on in the background. And that is why you need Obsidian Vaults.

2. The Layered Memory Stack #

Agent memory isn't one thing. It's distinct layers that flow into each other.

Layer What It Stores Format Quality Gate Here is a simple formula to have a working memory, that is human readable:

1. MEMORY.md Session-to-session developer knowledge Markdown Human-curated   
2. Research Packets Atomic knowledge units and structured MD 5-level promotion ladder  
3. Doctrine Synthesis Consolidated higher-order insights (JSONL Best-of-N eval selection)   
4. Conversation Memory Episodic turn history similarity 
5. Obsidian Vault Human-readable dashboard Markdown pages and Watchtower auto-generation

The key insight: each layer has a quality gate. Memory without quality gates is just a log file. The gates are what make the difference between a system that degrades over time and one that compounds.

True agent memory has three foundational types working across these layers: > Short-term memory is the active conversation context. > Long-term memory is accumulated knowledge about your projects, preferences, and patterns. > Episodic memory is specific experiences and outcomes that inform future decisions. When these three types work together, your AI agent stops being a stateless tool and starts becoming an evolving collaborator.

3. Research Packets for Agents #

This is where it gets interesting. Instead of storing knowledge as blobs of text, we structure it as atomic knowledge units, one insight per file, with mandatory structure:

For example, on the Startup Research Packets, I used it the atomic structures and packets this way: / General Info about the Packet - packet_id: startup-research-pg-do-things-that-dont-scale - promotion_status: promoted - doctrine_richness_score: 9 - doctrine_tags: doctrine:user_truth_and_distribution

## Claim Early startups usually grow because founders manually create the first pockets of traction instead of waiting for demand to appear on its own. ## Mechanism Founder-led recruiting, support, setup, and direct observation shorten the distance between product assumptions and real user behavior. ## Boundary The lesson weakens if the team mistakes temporary manual effort for the long-term business model. ## Contradiction Some products break out with less visible founder labor because demand is unusually strong or distribution is already embedded.

Every packet has a Claim (what), Mechanism (why), Boundary (when it breaks), and Contradiction (where tension exists). This forces the system to capture nuanced knowledge and INTELLIGENCE. The packets live in docs/research-packets/ as plain markdown files, readable in Obsidian, greppable from the command line, diffable in git.

The Promotion Ladder: Not all packets are equal. They earn trust through a 5-level ladder, I use this especially while researching for improved intelligence/mastery.

Level Status Description Count 1. exploratory Raw extraction, unvalidate some 2. candidate_doctrine Shows promise, needs eval 3. promoted Survived evaluation, put them in the active pool 4 .benchmark_grounded, passed a certain benchmark 5. realworld_validated Passed real-world evaluation (you can skip this, but if you wanna make sure that an intelligence is worth keeping, worth to look into what's the end memories/intelligences kept and whether they are gud)

Only packets at the promoted or higher level enter the advisory pool when using memory for intelligence. This is the quality gate that prevents knowledge bloat.

4. Memory Consolidation and the Obsidian Vault #

Individual packets are like episodic memories. But the real power comes from consolidation, like how human brains consolidate memories during sleep.

Our synthesis layer groups packets by doctrine tags, then runs an LLM to produce a meta-doctrine: a higher-order insight that emerges from the cluster.

Everything surfaces into an Obsidian vault. The structure maps directly to Obsidian concepts:

The vault structure:

obsidian-vault/
  00-Intent/              -- What you're trying to learn
  05-Runtime/             -- Runtime artifacts
  06-References/
    research-packets/     -- 213 atomic knowledge files
    research-sources/     -- Near-source notes
    beliefs/              -- Benchmark runs + contradiction tracking
    realworld-eval/       -- Eval documentation
  domain/
    Home.md               -- Central dashboard (chip stats, run counts)
    Research Coverage.md  -- Doctrine depth by area
    Research Frontier.md  -- What to learn next
    Promotion Bridge.md   -- Benchmark -> doctrine promotion
    Loop Progression.md   -- Flywheel state

5. The Quality Filter: Primitive vs. Valuable #

This is the most important lesson in the entire system. Not what to remember, but what to forget.

Primitive (Don't Store) Valuable (Always Store) These are primitive stuff: - "Bash -> Edit sequence used" - "Health=300 for better game balance" - "Tool timeout 41% of the time" - "User prefers small iterative fixes" - "Read before Edit pattern" - "Lobster model needs baseY offset" - "Edit success rate: 85%"

The test is simple: would a human find this useful to know next time?

We use three tiers of learning value to sort what gets stored:

Tier 1: Operational (LOW, don't promote). Tool usage counts, timing metrics, file modification patterns. Store for diagnostics only, never pollute cognitive memory.

Tier 2: Behavioral (MEDIUM, selective). User preferences, workflow patterns, and communication style.

Tier 3: Cognitive (HIGH, always promote). Domain insights, design decisions with rationale, mistakes with lessons. -> This is what your memory should primarily contain.

6. Feedback Loops #

Memory without feedback degrades. We run three loops to keep it honest:

Packets (UCB1 Multi-Armed Bandit). Each packet is an "arm." When it's used in advisory and the user rates the response, that reward flows back to adjust future selection. Good packets get boosted, bad ones get deprioritized. Based on the MBA-RAG paper.

DecisionJournal. Every decision records The situation, the choice, the expected outcome, which packet_ids informed it, and the actual outcome (filled later). The doctrine effectiveness method computes average ratings per packet, creating a closed loop from decisions back to knowledge quality.

Real-World Eval. Doctrine-grounded responses vs. baseline Graded by LLM. Losing packets get demoted. Winning candidates get promoted.

7. FINAL CONCLUSION & THE PROMPT #

When your AI agent knows your entire project history, your preferences, your past mistakes, and your architectural vision, every single conversation starts at a higher level of intelligence than the last. That means better results with fewer tokens and more productivity. That's just the tip of the iceberg, though. Once you have a working memory, you can do a lot more dope stuff with it. From research to gain better intel to gain edges in the market, to self-improving agents, memory is the most important layer you need to get right. And I spent a lot of time and went through many trials and errors to share this methodology that simply works, unlike most other memory systems that actually doesn't. Finally, this prompt should help you get started. This is directly generated from my own Obsidian vault connected repo, so that you can actually get a prompt that actually works:

# Agent Memory Protocol — Obsidian Vault                                                           
  
  You maintain a structured memory system in an Obsidian vault. Your memory is not a
  log. It is a curated knowledge base that only stores what a human would find useful
  next time.

  ## The Quality Gate

  Before storing ANYTHING, ask: "Would a human find this useful next time?"

  STORE: domain insights, decisions + rationale, mistakes + what fixed them, boundaries
  where knowledge breaks down, user preferences confirmed across 2+ sessions.

  NEVER STORE: tool sequences (Bash→Edit), timing metrics, file modification facts,
  success rates, navigation patterns. This is telemetry, not memory.

  ## Vault Structure

  Maintain these directories in the Obsidian vault:

  vault/
    packets/           -- atomic knowledge units (one insight per file)
    doctrines/         -- synthesized truths from packet clusters
    state/
      dashboard.md     -- live stats: packet counts, coverage, gaps
      decisions.md     -- decision journal with doctrine grounding + outcomes
      lessons.md       -- one-line gotchas (most valuable file you own)
    ceremonies/        -- recurring review templates (weekly, monthly)

  ## Writing Packets

  Every piece of knowledge is a packet. Not a note. Not a paragraph. A structured
  unit with four mandatory fields.

  Template:
  Packet: [Title]

  - packet_id: [domain]-[source]-[slug]
  - source: [author, title, year]
  - confidence: high | medium | low
  - status: draft | candidate | promoted | validated
  - tags: [comma-separated]

  Claim

  [One sentence. The core insight. What is true.]

  Mechanism

  [One sentence. Why it works. The causal chain.]

  Boundary

  [One sentence. When it breaks. The conditions where this stops being true.]

  Contradiction

  [Where this clashes with other packets or common belief.]

  NEVER skip Boundary or Contradiction. Knowledge without limits is dangerous.

  ## Promotion Ladder

  Packets earn trust through evidence, not gut feel:

    draft       → wrote it, unvalidated
    candidate   → referenced 2+ times OR corrected a past mistake
    promoted    → used in a decision that produced a good outcome
    validated   → survived adversarial pressure or real-world testing

  Demote packets that lead to bad outcomes. Update the status: field immediately.
  When selecting knowledge to use, prefer promoted and validated packets. Ignore
  drafts unless nothing else is relevant.

  ## Doctrine Synthesis

  When 3+ packets share a tag, synthesize a doctrine — a higher-order insight that
  emerges from the cluster:

  Doctrine: [Theme]

  - cluster_tag: [shared tag]
  - packets: [[Packet A]], [[Packet B]], [[Packet C]]

  Meta-Claim

  [The insight that emerges from reading all packets together.]

  Tensions

  [Where the packets disagree with each other.]

  Gap

  [What you would need to learn to strengthen this doctrine.]

  Doctrines are your consolidated memory. Packets are episodes. Doctrines are
  understanding.

  ## Lessons Learned (lessons.md)

  Every time something goes wrong and you figure out why, add ONE LINE:

  Format: \`- [What happened] -- [what fixed it or what to do instead]\`

  Examples:
  - Running two extraction jobs in parallel corrupts the output -- run sequentially
  - Penalizing low scores kills performance -- use boosts only, never penalties
  - Expanding the pool without improving selection makes results worse
  - IP bans last 1-24 hours not minutes -- safe rate is 8s delay + 60s pause

  No essays. No context. Just the lesson and the fix. This file will become the
  single most valuable document in your vault.

  ## Decision Journal (decisions.md)

  When making a significant decision, record:

  [Date] [Decision Title]

  - Situation: [what prompted this]
  - Decision: [what you chose]
  - Grounded in: [[Packet X]], [[Packet Y]]
  - Expected outcome: [what you think will happen]
  - Actual outcome: [fill in later — good/bad/mixed]

  Review outcomes monthly. Packets that grounded good decisions get promoted.
  Packets that grounded bad decisions get demoted.

  ## Dashboard (dashboard.md)

  Update weekly with:
  - Total packets and count per status level
  - Coverage by tag (which areas are deep vs thin)
  - Top 3 gaps (fewest packets or lowest confidence)
  - Recent decisions and their outcomes

  ## Workflow — Every Session

  START of session:
  1. Read lessons.md for relevant gotchas
  2. Read dashboard.md for current state
  3. Search packets by tag for the current topic

  DURING session:
  4. When you learn something new → write a draft packet
  5. When something breaks → add a line to lessons.md
  6. When you make a decision → log it in decisions.md

  END of session:
  7. Promote any packets that proved useful today
  8. Demote any packets that led you astray
  9. Update dashboard.md counts
  10. If 3+ packets share a tag without a doctrine → synthesize one

  ## Three Rules

  1. STRUCTURE OVER VOLUME — one packet with Claim/Mechanism/Boundary/Contradiction
     beats 50 unstructured notes.

  2. PROMOTE THROUGH EVIDENCE — nothing enters the active pool without surviving
     pressure. Draft → Candidate → Promoted → Validated.

  3. LESSONS OVER KNOWLEDGE — your gotchas file saves more time than all your
     packets combined. Guard it. Grow it. Read it first.

Start using the above while creating a research agent, and don't forget to have your LLM customize the approach for your exact use case. Then you can evolve it for many more stuff. Good luck!