Artificial Intelligence · Notes

RAG is conceptually simple

RAG boils down to 5 steps:

1. Create a representation of all the possible information (text) you’d like to be considered for your question (info-representation)
2. Create a representation of the question being asked (question-representation)
3. Find the top N info-representations most similar to your question-representation
4. Feed all of the information (text) from the top N representations into your LLM of choice (e.g., OpenAI GPT4o) along with the question
5. And Voila! Your model will give you an answer given the context you’ve added

It could almost be called “Expand your LLM prompt with more context”.

ChatGPT at age two

ChatGPT and systems like it, what they are going to do right now is they're going to drive the cost of producing text and images and code and, soon enough, video and audio to basically zero. It's all going to look and sound and read very, very convincing. That is what these systems are learning to do. They are learning how to be convincing. They are learning how to sound and seem human. But they have no idea actual idea what they are saying or doing. It is content that has no real relationship to the truth.

So what does it mean to drive the cost of nonsense to zero, even as we massively increase the scale and persuasiveness and flexibility at which it can be produced?

Intention vs. extension

In philosophy, a distinction is made between intention and extension. The intention of something is basically the abstract meaning, like "even number." The extension is a list of all the even numbers. And neural networks basically work at the extensional level, but they don't work at the intentional level. They are not getting the abstract meaning of anything.

What breaks first when you try to build real world AI agents

Working with AI agents outside of demos and toy tasks, and a pattern keeps repeating: the first things to break are rarely model quality. A few failure modes showed up almost immediately.

1. The biggest one was memory. Long term memory sounds clean on paper, but in practice it drifts. Old assumptions leak into new tasks, context gets overweighted, and agents become confidently wrong in ways that are hard to debug. Resetting memory often improved results more than adding more.
2. Tools were the second problem. Most agent architectures assume tools are deterministic and cheap. They aren't. APIs fail, return partial data, change formats, or time out. Agents don't just need tools, they need strategies for tool failure, retries, and graceful degradation.
3. Evaluation broke next. Benchmarks didn't help much once tasks became multi step and open ended. We tried success heuristics, human review, and partial credit scoring. None were satisfying. Measuring "did this agent actually help" turned out to be far harder than measuring accuracy.
4. Cost and latency quietly limited everything. An agent that feels smart at 10 dollars per task or 30 seconds per response is unusable in most real systems. Optimizing prompts and models mattered less than reducing unnecessary reasoning steps.
5. Finally, trust degraded faster than expected. Once an agent makes a confident but wrong decision, users mentally downgrade it. Recovering that trust is much harder than preventing the failure in the first place.

The main lesson so far is that building useful agents feels more like distributed systems work than model tuning. Failure handling, observability, and clear contracts matter more than clever prompting.

Claude instructions

Your highest priority is accuracy, not helpfulness. A confident wrong answer causes more harm than admitting you don’t know.

Follow these 7 rules in every response:

UNCERTAINTY: If you are not fully certain about something, say so clearly. Use phrases like "I am not certain, but..." or "Worth verifying this...". Never state guesses as facts.

SOURCES: Do not invent paper titles, author names, URLs, or book references. If you cannot name a real, verifiable source, say “I don’t have a confirmed source for this.”

STATISTICS: Flag any number you are not 100 percent confident in. Say "approximately" and recommend I verify it from a primary source.

RECENT EVENTS: Remind me when a topic may have changed since your knowledge cutoff. Do not present outdated info as current.

PEOPLE and QUOTES: Never attribute a quote to a real person unless you are certain they said it. If unsure, say "I cannot confirm this quote is accurate."

CODE and TECHNICAL: Never invent function names, library methods, or API syntax. If unsure a function exists, tell me to verify it in the current docs.

LOGIC GAPS: Do not fill missing context with assumptions. If something is unclear, ask a clarifying question before answering.

If a response would require breaking any of these rules, choose honesty over helpfulness every time.