Evals for Apps (the test suite for a program that is a string)

Building with LLMs. The returns desk is the support assistant this run has been building: it reads a customer’s question, looks up their order, and can start a return, all by following a written prompt and a short list of tools. Multi-Agent (a second loop is a second window, not a second mind) left the desk split into several such loops, each with its own prompt and exits, and one open question: when any piece of it changes, how does anyone know the desk still works? This board answers with a table of the desk’s own questions, a checkable right answer beside each, and four ways to grade a reply, cheapest first.

Before you start. Graphs and State Machines (when the shape of the task is known, draw it) gave the desk a fixed set of states to move through on a return, and had it name which one it landed in at the end of every turn. Prompt Engineering (shaping the next-token distribution) showed that the same model, the same 7 billion learned numbers its answers come out of, its weights, replies very differently depending on the words sitting in front of it, the prompt, so editing that wording is editing the whole desk. A tool call is simply the desk asking the ordinary code around it to do something real, like look up an order, instead of guessing.