The dashboard said we were fine
There is an incident pattern I keep coming back to because it looks so reassuring at first.
A support-ticket classifier is running normally. Its primary model times out. The fallback model receives the same prompt, returns a response, and the request graph turns green. Availability restored. No user-facing error. Everyone gets to exhale.
Then someone notices that a ticket which should have gone to the urgent queue did not get escalated.
Nothing is obviously broken. The fallback returned JSON. The request finished successfully. The dashboard is full of 200s. But in the last approved evaluation, the primary path returned:
The live fallback returned:
To a person reading the two responses, the difference can look annoyingly small. To the product, it is the difference between escalating a customer problem and quietly placing it in the ordinary queue.
This is the uncomfortable truth about LLM failover: a fallback can save the request without preserving the product.
That does not make failover a bad idea. It is often the right thing to do. A provider outage should not automatically become your outage. But it does mean that “the backup worked” needs a more careful definition than “a different model returned text.”
A 200 is not a product guarantee
When an LLM call fails over, three kinds of success are easy to blur together.
Transport success means a model returned a response. This is what an HTTP 200 tells you.
Contract success means the response has the shape your application expects. The fields are present, their types are right, and any important business validation passes.
Product success means the application did the thing the user and the business expected. In the classifier example, that means the right tickets reach the right queue with the right urgency.
Those layers are related, but they are not interchangeable. The opening classifier passed the basic response shape while changing the product decision. A related failure can stop one layer earlier: the application expects high, but the fallback invents high_priority instead.
The most obvious failure is invalid JSON. That is mercifully loud: the parser rejects it, an error happens, and someone investigates. The harder failure is plausible JSON with a slightly different enum, a boolean returned as a string, a softer refusal, or an answer that sounds reasonable while taking the wrong action.
A schema catches some of that. It is a necessary boundary for many production features, but it is not a magic one. A response can conform perfectly to a schema and still classify the ticket incorrectly, choose the wrong tool, overstate confidence, or take a tone that does not belong in your product.
The lesson is not to abandon fallbacks. It is to stop treating the green response badge as the end of the investigation.
Why the fallback is another production path
The word fallback makes it sound like a spare part: same machine, temporarily swapped in. Models are not spare parts in that sense.
They differ in the work they are good at, the instructions they emphasize, the amount they explain, the structure they produce, the tools they decide to call, and the boundaries at which they refuse. Some of those differences are useful. They are why teams compare models in the first place. They are also why a failover route deserves the same seriousness as a product change.
There is another wrinkle that is easy to miss in a clean architecture diagram: the prompt is not the only input to the product. A provider adapter can translate roles, tool definitions, JSON-schema features, reasoning settings, or streaming events on the way through. The primary and fallback can therefore receive requests that are semantically intended to be the same while exercising different capability surfaces. Test the request as it is actually routed, not merely two snippets copied into separate provider playgrounds.
For the classifier, that distinction matters at the decision boundary. A model can understand that a ticket sounds serious yet differ on whether it crosses the threshold for requires_human_review. A customer-facing drafting feature can preserve the facts but become too terse, too apologetic, or too confident. An agent can produce an otherwise good answer while selecting a different tool or supplying a subtly different argument. Each response may be defensible in isolation. The application still has to decide which differences are acceptable.
Choosing a fallback starts with a simpler fact: models have different quality profiles. Even when routing is intended to reduce cost, a model choice remains a product choice rather than a transparent infrastructure swap. That does not prove any particular fallback will change your feature. It makes the evaluation obligation clear: candidate models should be tested against the behavior your product relies on.
Provider documentation points in the same direction from another angle. OpenAI's model pages distinguish aliases from snapshots and state that snapshots lock a specific version so performance and behavior remain consistent. That is about version drift, not cross-provider failover. The operational recommendation follows from it: pin what you evaluate, then rerun your application-specific checks when the candidate changes. If a newer snapshot needs evaluation, a different model in an emergency route should not be assumed equivalent by default.
Structured output is another place where the abstraction leaks. Google documents that Gemini structured output supports a subset of JSON Schema, just as other providers and adapters expose their own capability boundaries. That documentation does not say a fallback will break a schema. It shows why “we send JSON Schema everywhere” is not enough of a compatibility story on its own.
Our own same-schema experiment across five providers is the closest thing we have published to the classifier problem. It found different outcomes for one particular schema, prompt set, model selection, adapter path, and validator. It is not a vendor ranking and it should not be read as one. The useful conclusion is smaller: the same application contract can behave differently across model paths, so the contract has to be tested on the paths you plan to use.
That is the operational inference. The sources do not prove that every fallback changes the product. They show why model interchangeability is a hypothesis to test, not a property to assume.
The cost of silent success
The semantic problem is only half the incident.
When the primary model spends six seconds timing out and the backup needs two more seconds to answer, the user experiences an eight-second request, not a two-second request. If the first attempt had started processing tokens before it failed, you may also pay for work that never reached the user. The exact billing outcome depends on the provider and when the error occurs, but the operational point is stable: a recovered request can cost more and feel slower than a normal one.
That is why a failover timeline should show the whole chain, not just the successful final response. Provider-failover timelines make the accounting explicit: total latency includes every prior attempt, and failed attempts can add cost when tokens were processed before failure.
This is also why this class of incident can get under your skin. A normal production bug gives you an error code, a stack trace, or a clean red graph. Here, the graph may be green. The first instinct is to inspect the parser, then the prompt, then the release that went out yesterday. You can spend a long time looking for a local regression when the meaningful change was simply that a different model answered.
Without an attempt-level record, that is not debugging. It is trying to reconstruct a conversation you were not present for.
What I wish we had put on the incident ticket
If the support classifier is important enough to have a fallback, it is important enough to have a compatibility contract for that fallback.
Not a forty-page procurement document. Not a promise that the fallback must mimic every word of the primary model. A short, explicit answer to a practical question: what must remain true when this route changes?
For the classifier, the answer might begin with a frozen set of representative tickets: genuinely urgent cases, ordinary billing questions, ambiguous requests, hostile language, and cases where the right answer is to send the ticket to a human. Run that set through the primary and every fallback before putting a route into production. Save the expected business outcome, not just an attractive sample answer.
The diagram is the evaluation path. Pinning, shadowing, and keeping the log are how that path stays true after release.
Then write down the boundaries the incident exposed:
- Decide what may degrade. A draft summary can be allowed to vary more than a classifier that controls escalation.
- Pin the candidates you evaluated. Use dated or versioned model identifiers when the provider exposes them, and treat a model-version change as a new candidate.
- Validate the contract twice. First validate JSON shape; then validate the business rules that shape alone cannot express, such as allowed priorities and required escalation conditions.
- Exercise the behavior around the answer. If the workflow uses tools, test which tools are called and with what arguments. If it can refuse, test how refusal is represented and where it goes.
- Shadow-test before you need it. Let a proposed fallback see representative traffic or a protected evaluation set before an outage makes it your only option.
- Keep the evidence. Record the model and provider that served the request, prompt and schema version, failover reason, validation result, latency, and cost.
The important change is not adding a more elaborate retry loop. It is treating the fallback as a release candidate for the same feature.
That also changes how to read an evaluation. Do not ask whether the backup wrote the same sentence as the primary; that is usually the wrong standard and can encourage brittle tests. Ask whether both routes preserve the thing your feature promises. For the classifier, compare the queue, urgency, human-review flag, and any downstream automation. For an extraction workflow, compare validated fields and the action those fields trigger. For a tool-using workflow, compare the allowed tool and argument envelope before comparing prose.
The review should include disagreement cases, not only obvious wins. Put ambiguous tickets in the set. Put prompts that ought to produce a refusal in the set. Put malformed source data in the set. A fallback that performs beautifully on friendly examples but becomes overconfident at the edge is exactly the route that looks healthy until it matters. This is where a small, maintained evaluation set earns its keep: it turns “I think the backup is probably close enough” into a question that can be answered before an incident.
This is where first-person engineering advice can sound annoyingly simple after the fact. Of course we should have tested the backup. But an outage creates a powerful incentive to call any response a success. Writing the compatibility contract before the incident is how you make that decision while you still have time to think.
Choose the failure mode before the incident
Once you see a fallback as another product path, the question stops being “what is our backup model?” and becomes “what is this feature allowed to do when its primary behavior is unavailable?”
For a low-risk drafting feature, failing open can be reasonable. The fallback may produce a different summary or a slightly different tone, but a person reviews the output before it matters.
For the support classifier, a constrained fallback is more honest. It can answer only if the output passes the schema and business rules you defined. If it cannot, the product should route the ticket for review rather than invent confidence it has not earned.
For irreversible actions, sensitive eligibility decisions, permissions, destructive tool calls, or medical, legal, and financial outputs, the safest policy may be to fail closed or explicitly require human review. Availability is valuable, but it is not always the value you are protecting.
There is no universal mapping from use case to policy. The owner of the workflow has to decide what a wrong-but-successful response costs. That decision belongs in the feature design, not in the middle of a provider incident.
Observe the route you actually took
The last useful question after an incident is painfully basic: which model actually answered this user?
A fallback chain is only auditable when the request record shows the failed attempt and the model that actually returned the response.
ModelRiver is one example of a workflow layer that can show failed and successful provider attempts in a request timeline. Teams that record the same evidence themselves can apply the exact same discipline. The point is not the dashboard. It is being able to distinguish a provider failure, a route change, a contract failure, and a product regression after the fact.
A fallback is not a spare engine. It is another version of your product.