AI/ML Ops - the deep-dive set

Nine boards covering three separate areas of the job, which the boards call planes: a model gets trained, it gets served, and what happens while it is live decides whether it keeps working. This lesson tells that loop once as a story, then walks each board’s own fight on its own.

Boards

Before you start. Inference at Scale (why an LLM is fast, and cheap) and Evaluating Models (how do you *know* it's good?) already showed you how a trained model turns a prompt into an answer; this note is about everything around that answer - where the file lives, what serves it, and who finds out when it drifts. Nothing here is new, only long: nine boards, not one.

The idea in 30 seconds

A trained model is one small file - Sculley’s small box. Everything else here is the ring around it: what serves that file, and what rebuilds it when it stops being right.
A registry is a librarian for that file: which version is live, and a label you move instead of a server you touch.
Serving a big model is limited by memory before it is limited by speed - the weights are bought once, the KV cache per conversation is not.
Reserving a fixed slot for every possible conversation wastes most of it; paging that memory out in small pieces, the way an operating system pages RAM, is where the real throughput gain came from.
Training at real scale fails constantly, and is expected to; checkpointing bounds how much any one failure costs.
A model can be perfectly correct on day one and quietly wrong by day ninety. Drift produces no crash, only a slow decline caught by traces, evals and detectors, or missed.

Read the story · Skip to the numbers