Turning Obsidian into AI's Own Memory — Local Cognitive OS with Hindsight and Hermes
It started with a single offhand remark from a developer: "I don't want Obsidian to be just a note-taking tool. I want it to become the long-term memory system for AI itself."
Ollama + Hindsight + PostgreSQL + Obsidian. Every process completes locally; the entire history of thought stays closed inside one's own machine. An experiment in how deeply an AI can understand context when constrained to "never leak externally."
This is not merely a combination of tools. It may be the emergence of a new layer where the infrastructure audits itself and humans and AI mutually extend each other's cognition.
This article assumes the following background. If any of these are unfamiliar, read this first to avoid getting lost.
- Hermes Agent — An OSS autonomous AI agent published by Nous Research. Runs resident as CLI / gateway and handles dialogue with the user. See "Hermes Agent — Execution engine of the second brain" for details.
- daily-chats/ — A folder inside the Obsidian Vault. Every time a Hermes session ends, that day's raw conversation log (questions, responses, tangents, hesitations, code snippets — all unedited text) is automatically exported as markdown here. This article treats it as the entry point of AI's long-term memory.
- knowledge/ and MOC — A separate folder in the Obsidian Vault. Summaries and insights extracted from daily-chats/ are organized here as themed MOCs (Map of Content — index notes that bundle related notes) and INDEX entries. The structured layer — the destination of AI's long-term memory.
- Hindsight — The memory mechanism in the Hermes ecosystem. It periodically scans daily-chats/, summarizes and vectorizes them, persists into PostgreSQL, and further accumulates summaries of summaries in a self-referential engine. The protagonist of this article.
- Ollama / Gemma3 — Ollama is a runtime for running LLMs locally; Gemma3 is Google's open-weight model. Together they let the summarization pipeline run with zero external API calls.
- Real-world benchmark with Gemma3 on Ollama: 23.4 tokens/sec — not "tolerable," but "daily usable."
- Hindsight repeatedly summarizes past summaries, forming a self-referential feedback loop. Infrastructure self-auditing begins to run entirely locally.
- Obsidian's daily-chats/ is redefined as AI's primary persistent memory store.
- The loop that gradually converts "hesitation, hypotheses, discomfort" into explicit knowledge is beginning to function as a cognitive OS.
What 23.4 tokens/sec Really Means for 'Being Local'
When running Gemma3 (or equivalent Gemma4-class model) on Ollama, the benchmark shows 23.4 tokens/sec. In an era of cloud dependency, this is not merely "tolerable speed" but "daily usable speed."
What matters is not the speed itself. It is the fact that every process completes locally. Raw conversation logs accumulated in daily-chats/ are instantly summarized by the LLM, structured by Hindsight, and persisted into PostgreSQL. No external APIs participate in this flow at all.
In other words, the entire history of thought remains closed inside one's own machine. This is more than privacy protection. It is an experiment in how deeply an AI can understand user context when constrained by the rule "never leak externally."
Related: For Hermes itself — the self-improving loop and resident execution — see "Hermes Agent — Execution engine of the second brain". This article wires that into Hindsight × local LLM.
Hindsight's Self-Referential Infrastructure
What stands out most is Hindsight's role.
Conventional RAG stops at "retrieve and answer." Hindsight is different. It repeatedly summarizes past summaries, accumulating meta-metadata from summaries of summaries. The vector and metadata layers etched into PostgreSQL gradually self-organize over time.
This is precisely "infrastructure self-auditing."
- Does today's daily-chat contradict last week's summary?
- How have the themes the user repeatedly touches evolved over the long term?
- To what extent do the summaries generated by the AI (or myself) distort the original context?
The foundation is now in place for the AI itself to periodically verify these questions. There is no need to wait for external human reviewers. The system relativizes itself and proposes corrections — such a loop may already be beginning to turn, entirely locally.
The Ultimate Human-AI Co-Creation: 'Making Obsidian the Memory'
What excites me most is not the tech stack, but the underlying idea.
The developer did not stop at treating Obsidian as a "second brain." They redefined it as "the primary memory device for AI." daily-chats/ is no longer a graveyard of miscellaneous logs. Every piece of text accumulated there is structured through Hindsight and functions as AI's long-term memory.
This is not a relationship where humans ask AI to "remember this for me."
It is a relationship where humans design the environment itself: "I'll write it here, so you go ahead and structure it on your own."
AI only becomes intelligent within the context it is given. The core of this pipeline may lie in the reversed idea that humans prepare the field to transform context into "persistent, searchable, and self-referential memory."
Related: A pipeline that stops treating Obsidian as just a "second brain" and lets AI grow a wiki and publish it externally is covered in "Obsidian → LLM Wiki → HTML → AI Deploy". Same Obsidian origin, but pointed outward instead of inward into memory.
The Emerging Loop of 'Tacit Knowledge to Explicit Knowledge'
The true destructive power of this architecture has not yet surfaced.
What gets written in daily-chats/ is not polished ideas, but "hesitation, hypotheses, discomfort, wavering judgment." Hindsight gradually converts that ambiguity into explicit knowledge. Eventually, Obsidian's knowledge/ layer may become not just a collection of MOCs, but a "knowledge graph that AI itself is cultivating."
Related: The "hesitation, hypothesis, discomfort" territory is continuous with the tacit-knowledge × tacit-thought layer that code can't capture. That side is mapped in "AI agents descending into what code can't write — long-tail × tacit knowledge × tacit thought". This article handles the infrastructure side: where to store that tacit thought.
When that happens, what will the developer witness?
- The moment a discomfort felt in the past is rediscovered by AI
- The moment a rough note suddenly gains meaning in a different context
- The moment the infrastructure quietly points out, "This theme contradicts what it was three months ago"
Everything local, everything persistent, everything self-referential — such a state may already be beginning to move at hand.
Each iteration: doubt left in daily-chats/ passes through summary → summary-of-summary and grows into the knowledge graph.
This Is Not About Tools — It Is About a Cognitive OS
Ollama + Hindsight + PostgreSQL + Obsidian.
What makes this combination special is not that any one piece is superior. It is because the circuit that converts the fluidity of human thought into a form AI can handle has finally closed.
The moment the developer decided to "make Obsidian the memory itself," technology transcended mere means.
It means that AI and humans have begun, without interference from anyone, to build with their own hands an OS for becoming wise together while compensating for each other's weaknesses.
This loop has only just begun.
I plan to deepen this architecture further and implement it as a practical Hermes skill. First, I want to close the loop locally: a periodic Hindsight scan script and automatic link-appending flow into Obsidian's knowledge/ layer.
From asking AI to "remember this," to designing the field in which AI grows
Making AI smarter by prompt-craft alone may not be enough anymore. We have reached the stage of building the field in which intelligence grows, with our own hands.
Making Obsidian the AI's memory itself is the first step. Don't hand over a polished answer set — leave thoughts laden with contradiction and doubt, and Hindsight will organize them over time.
- Humans write into daily-chats/, contradictions and all (leave the tacit in)
- Hindsight compounds summaries of summaries (structure via self-reference)
- knowledge/ becomes a graph that AI itself cultivates (the loop of becoming explicit)
The moment these three line up, the experience shifts from "being used by AI" or "using AI" to growing wise with AI. Everything local, everything persistent, everything self-referential — with no one in the way.