Red-Teaming (breaking your own model before a customer does)

Building with LLMs. The board draws three rings of checks around the returns desk. This lesson is about the people and programs paid to push on those rings before a stranger does.

Before you start. The returns desk is this run’s support assistant: a language model that looks up orders with lookup_order and opens returns with start_return. A guardrail is a check in ordinary code around the model; the one that matters most is a fixed series of checks every tool call must pass before it runs, each check called a rung, from Guardrails, Cost and Latency (the three budgets the harness enforces). An eval set is the desk’s own list of questions with a checkable right answer beside each, run on every change, from Evals for Apps (the test suite for a program that is a string). Safety training is the late round of training where people rank replies and the model learns to prefer the ones they picked, including polite refusals, from Fine-tuning & Alignment. Prompt injection is an instruction hidden inside text the model was only meant to read, covered from the security side in LLM & Agent Security.