The cascade threshold
Every request goes to the small model first. How sure must it be before its answer goes out, and what does the frontier cost for the rest?
A thousand requests reach a 7B first, and it answers every one. If its reply sounds sure enough it goes out; if not, it is thrown away and the frontier model answers instead. The panels show how many the 7B answered alone, the wrong answers that got out, and the bill and the wait against always-frontier.
Board: Routing and Cascades →On a narrow screen, the three panels stack below the dial and the table scrolls sideways.
try this Drag How sure the 7B must be from 0.00 to 1.00 and watch Wrong answers that got out.
Included with a plan
This lab is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
see also ReAct: Synergizing Reasoning and Acting in Language Models in Papers.