Artificial Intelligence · Notes

RAG is conceptually simple

RAG boils down to 5 steps:

1. Create a representation of all the possible information (text) you’d like to be considered for your question (info-representation)
2. Create a representation of the question being asked (question-representation)
3. Find the top N info-representations most similar to your question-representation
4. Feed all of the information (text) from the top N representations into your LLM of choice (e.g., OpenAI GPT4o) along with the question
5. And Voila! Your model will give you an answer given the context you’ve added

It could almost be called “Expand your LLM prompt with more context”.

ChatGPT at age two

ChatGPT and systems like it, what they are going to do right now is they're going to drive the cost of producing text and images and code and, soon enough, video and audio to basically zero. It's all going to look and sound and read very, very convincing. That is what these systems are learning to do. They are learning how to be convincing. They are learning how to sound and seem human. But they have no idea actual idea what they are saying or doing. It is content that has no real relationship to the truth.

So what does it mean to drive the cost of nonsense to zero, even as we massively increase the scale and persuasiveness and flexibility at which it can be produced?