RAG Over Your Own Notes

The RAG board draws a question going out to a library before it reaches the model. This lesson builds it over your own notes and measures whether it works.

Before you start. A language model, the AI that writes the answer, only knows what is in the text you hand it. So you cut your notes into chunks of a few paragraphs each. You find the chunks that match the question. You paste them in with “answer only from these”. That recipe is retrieval-augmented generation, RAG, from RAG (Retrieval-Augmented Generation). That lesson found matches by embeddings, lists of numbers where texts about the same thing sit close together; this one uses a simpler word-matching score instead. The first search engines matched words too, as Context Engineering (what goes in the window, and who decides) told. An eval set is a list of questions with the right answer written beside each. A second model call can grade each reply against a written rule, as Evals for Apps (the test suite for a program that is a string) showed.

The idea in 30 seconds

A notes assistant is five small pieces: cut the notes into chunks of a few paragraphs at their headings, score each chunk against the question with BM25 (a word-matching score), label each chunk with its note, have a real model answer from the best few, and cite the note after every sentence.
BM25 counts a shared word for more the rarer it is across your notes, so an unusual word in the question decides the match; it cannot match two different words with the same meaning.
A golden set (questions whose answering note you already know) lets you score retrieval on its own with hit@k (did the right note come back in the top k), apart from the answers, because the right note can still hand over the wrong chunk.
A second model call acting as judge, measuring faithfulness (the answer says only what the chunks say) passes an honest “Not in your notes” when retrieval failed, so only a check against the note that should have answered shows the retrieval failure.
Making the model quote its sources before it answers stopped both invented claims and false refusals on a 15-question test; query rewriting (asking the model to reword the question before searching) fixed a question that shared no words with its answer.

Read the story · Skip to the numbers