41 Hours of AI Provider Outages, and the 1 Hour That Mattered

Between August 9 and November 7, 2026, Anthropic's API spent 41.0 hours under a published incident. OpenAI's text-generation API spent 55.6 hours. The two overlapped for 59 minutes [1][2].

That gap is the argument for failover in one line: AI providers fail often, but rarely at the same time. Making use of it is the hard part. You have to move a request to another provider mid-flight, get the answer to a browser that may have moved on, and let your own backend step in between AI calls. Most teams don't plan for that infrastructure when they make their first model call.

We build that infrastructure. ModelRiver is AI backend infrastructure in two layers:

• Gateway: routes each request across providers, fails over, caches, caps spend and logs cost.
• Orchestration: queues requests, calls your backend between AI steps, and delivers results to the browser as they happen.

So we measured it: what it takes to build, what it costs to keep running, what it does for each request, and what it does to your reliability. Where a number is an estimate, we show the formula and its inputs.

The headline numbers. Each is explained below, with its method and its limits.

Summary

• Failover across two providers cut exposure to provider incidents by 96%. With Anthropic as primary and OpenAI as backup, the time your requests had no healthy provider drops from 41.0 hours to 1.58 hours over 90 days. That already counts ModelRiver's own downtime [1][2][3].
• Every request whose primary model failed was still answered. In our benchmark that was 450 of 450, and each failover added a median 18.4 ms [4].
• Building the core yourself takes months to years of engineering. Routing with failover, async delivery, signed webhooks and orchestration took us 84 engineer-days. A standard, published estimation model (COCOMO II) puts the same code at 1,259 engineer-days. At the US median developer salary, that is $43,932 to $658,457 before you keep it running [4][5][6].
• Your code stays small. In Desk, a production app built on ModelRiver, the integration is 1.0% of the code, or 15.5% if you count every file that touches it. Without ModelRiver, the infrastructure Desk relies on could make up as much as 44% of the combined codebase [4][7].
• The hop is short. ModelRiver added a median 33.2 ms to a request, with production settings on one laptop [4].
• The price is small next to the engineering. ModelRiver's Pro plan is $49 a month for up to 500,000 requests. Renting the separate pieces you'd otherwise wire together costs $26.50–75 a month before any engineering [8]–[13].

What happens after you call the model

A model call in a tutorial is one HTTP request. A model call inside a product usually turns into a small pipeline:

1. Your app sends the request. If someone is waiting on a page, it should return at once and deliver the answer later.
2. The gateway picks a provider and model from the workflow's settings. It checks spend caps and input rules, then calls the provider.
3. If the provider fails, the gateway retries a dropped connection and then moves to a backup model, possibly at another provider, within the same request.
4. The orchestration layer takes over.
   • For an async request, it queues the job and gives the browser a short-lived, single-use token to listen with.
   • If the workflow needs your data or your rules, it sends a signed webhook to your backend, waits for your callback within a time limit, and then either continues with the next AI step or finishes.
5. The result is delivered to the browser as it happens, to your backend as a webhook, or both. That includes browsers that connect late, and jobs interrupted by a server restart.

We sent one request of each kind through a local ModelRiver server and counted what each one set in motion:

To you, an orchestrated request is one API call and one webhook handler. Behind it, ModelRiver ran 5 background jobs and wrote 3 records. It signed and delivered a webhook, waited for your backend to answer, and pushed 4 live updates to the browser. Without ModelRiver, you write, deploy and watch each of those pieces yourself.

41 hours, 1 hour: what failover buys you

Figure 1. API incidents on the providers' own status pages, August 9 to November 7, 2026.

We took every incident each provider published in those 90 days and merged overlapping ones, so no hour is counted twice [1][2]:

The gateway is also a dependency, so it has to be counted. ModelRiver's public status page checks the API and its database every minute. Over the same 90 days it recorded 99.97% uptime [3]: 35 failed checks, about 35 minutes. Adding that downtime as a worst case gives:

What this does and doesn't say about your error rate.

• It is not a per-request error rate. Providers don't publish per-request error rates, and an "incident" can mean anything from a few failed requests to a full outage. Hours under incident are an upper bound on the time requests were affected.
• Failover only rescues errors that should move to another model: network errors, rate limits (429), server errors (5xx), timeouts and missing models. A provider that is merely slow is only rescued once your timeout fires.
• A backup model isn't an identical model. Its answers can differ, which we covered in LLM failover in production.
• Google is left out of the overlap. Google AI Studio's status page listed 3 Gemini API incidents in the window, including an outage of about 2.5 hours on October 30 when "all requests using API keys" failed. It doesn't publish machine-readable windows [14].

The other half is whether failover actually works when it's needed. We configured workflows whose primary model always fails, and interleaved them with normal requests [4]:

All 450 of 450 requests whose primary failed were answered by a backup, after 600 failed provider calls. The caller saw HTTP 200, plus a list of the attempts so the failure stays visible in your logs.

What it takes to build

Below are the eleven request-handling features in ModelRiver, with the code and tests each needed [4]. Line counts depend on the programming language, so read them as relative size. Our implementation is written in a concise functional language, and other stacks would differ.

Figure 2. Lines of code per feature in ModelRiver, at one git commit (November 2026). Teal is production code; orange is tests.

Lines of code are the visible part. The harder part is the 74 decisions baked into them. We listed each one, and a script checks that each value really is on the line we cite. A sample:

None of these choices is hard on its own. The work is knowing they need making, and many were learned from a failure. The git history of the same files shows it [4]:

• 160 commits in 12 months, of which 48 (30%) are labelled as fixes.
• 57 commits came in the last 90 days.

Some of the fixes, quoted from commit messages: "close retry races", "Fix callback race condition", "resume stuck callback_ai", "Fix websocket socket ID collision", "normalize JSON schemas for structured output". Each is a production lesson a team building this itself would have to learn too.

What it costs: build, keep, rent

Your team's cost depends on your team, so we give a floor, a ceiling and the formula, and you can plug in your own rate.

Building the core. The core is routing with failover, async delivery, signed webhooks and orchestration: 21,374 lines of production code (16,991 lines excluding comments and blank lines) plus 15,837 lines of tests.

• The floor counts the days on which our engineers committed changes to these files. They used AI coding assistants and knew exactly what they were building. Days spent designing, debugging or operating without a commit aren't counted, so this is a minimum.
• The ceiling is COCOMO II, a widely used, published software cost model, with every setting at "nominal". It was calibrated on projects from before AI coding assistants, so treat it as an upper bound [5].
• A team building this from scratch will land somewhere between the two.
• The day rate is the US Bureau of Labor Statistics' May 2025 median annual wage for software developers, $135,980, divided by 260 working days. That is salary only: employer taxes, benefits and recruiting come on top [6].

Keeping it running. Upkeep doesn't stop after launch:

Renting the pieces. Hosted services exist for parts of this. For 100,000 AI requests a month, each with one live update and one webhook, and 500 browsers connected at peak:

That's $26.50–75 a month for the parts. Routing, failover, orchestration and the code that connects them are still yours to build, and so are the servers they run on.

Putting it together. In the first year, at the floor:

That comparison uses our floor. On the industry model's estimate, the build line alone is $658,457. Either way, the decisive cost isn't the rented parts. It's the engineering: about 4 to 60 engineer-months that go to infrastructure instead of your product.

What stays in your codebase

Desk is an open-source support app built on ModelRiver [7]. It has 27,894 lines of application code. One file, 274 lines long, wraps ModelRiver's API for async requests, reconnects, conversation sessions, usage and signals. That makes the integration 1.0% of the app. Counting every file that imports it, whole, it's 15.5%.

Desk uses routing with failover, async delivery, streaming, conversation memory and usage tracking. ModelRiver's implementation of those features is 22,270 lines, 80% the size of all of Desk. A single app wouldn't need all of it, but even a fraction would change who owns most of the code.

The smallest working integration is two files: a backend route that starts the request, and a frontend that receives the result. We ran them end to end:

That's 22 lines of code across the two files. Trusting ModelRiver's webhooks on your backend takes a 21-line signature check.

What ModelRiver costs you in speed

A service between your app and the provider adds time. To measure only ModelRiver's share, we used a mock provider that always answers in exactly 200 ms [4]:

Figure 3. Median time by request type, with production-like settings. Teal is the mock provider's fixed 200 ms; orange is everything else. Async rows start once the request has been accepted, which itself took a median 24.8 ms.

• A synchronous request took a median 238 ms. That leaves 33.2 ms for ModelRiver's own work, such as authentication, rate limiting, workflow lookup and logging, plus the local network. A second run on the same day gave 32.4 ms.
• Async results reached the browser 229 ms after the request was accepted, about 28.8 ms more than the provider itself took. A full orchestrated round trip through your own backend took 314 ms.
• Development mode is a trap for benchmarks. With the server's development defaults (code reloading, verbose logging), the same request added 143 ms. Measure any gateway with production settings.
• Throughput (indicative only): at 50 concurrent clients the laptop served 168.9 requests a second with no errors. It was also running the database, the mock and the client, so this is not a capacity claim.

What we found that doesn't flatter us

1. The SDK's automatic reconnect doesn't work for async requests. Listening tokens are single-use by design: the first connection worked (join ok) and a second with the same token was refused (socket rejected). The SDK's reconnect() reuses the stored token, so it will be refused too. A fresh token from your backend through /ai/reconnect works (join ok). Recovering after a page reload therefore needs a small route on your backend, and our docs' "automatic reconnection" wording overstates it.

   Update (November 9, 2026): fixed in client SDK 1.8.0. ModelRiver now gives the browser its next single-use token over the connection it already has, so the SDK reconnects on its own after a network drop, the 1-hour connection limit, or a page reload within about 5 minutes. No backend route is needed for those. For longer gaps, pass a getToken callback that asks your backend for a fresh token. The docs now describe exactly this; see Reconnecting.

2. The SDK is bigger than its README says. The README says "~15KB minified". The v1.7.0 bundle is 37,285 bytes minified and 10,957 bytes gzipped.

3. Webhook retries aren't individually visible in the delivery log. Three automatic attempts left one record showing "attempt 1". Retry timing itself was as documented [15]: 3 attempts, 17.2 s and 20.1 s apart.

How we measured, and the limits

• Code and history. Scripts read every file from one git commit and count lines per feature. A 5,650-line file that mixes several features is excluded everywhere, so the counts are low. Each of the 74 decisions is checked against the line that sets it. Commits count as fixes when the message says fix, bug, race, crash, leak or regression; that is a proxy.
• Live runs.
  • Server: a frozen copy of the same commit on an 8-core Apple M1 laptop, with its own database.
  • Mock provider: in OpenAI's format; successful calls take exactly 200 ms and failures are instant. The response cache was off and input guardrails were disabled.
  • Settings: a production build refuses to call a local http:// provider by design, so we ran the development build with production's logging level and without code reloading.
  • Machine load: each scenario waited until the machine's load was below 4, and the load was recorded.
• Public data.
  • Incidents come from the providers' own status-page feeds. OpenAI's are reported per component; we used Chat Completions and Responses for the overlap.
  • ModelRiver's uptime comes from its public status page.
  • Prices are list prices, read on November 7, 2026.
• Estimates. The cost ceiling is COCOMO II.2000 with nominal settings (A = 2.94, B = 0.91, scale factors summing to 18.97). The salary is the US BLS May 2025 median for software developers. Both are published, and the formulas are in the data file.
• What this can't tell you. ModelRiver's source is private, so the code and history figures come from us; the outside data and the raw results are published. Laptop latency isn't production latency. Incident hours aren't request error rates.

What this means if you're deciding

• You may not need any of this if your app makes one synchronous call to one provider and can show an error when that provider is down.
• You need most of it, whether you build it or buy it, if answers must reach a browser after the request returns, survive a provider incident, or pass through your own backend between AI steps.
• Use the tables to decide which parts to build, rent or skip, and how much of the 74-decision list you'd rather not learn the hard way.

FAQ

How often do AI APIs go down? In the 90 days to November 7, 2026, Anthropic's API spent 41.0 hours under a published incident (1.9% of the time). OpenAI's text-generation API spent 55.6 hours (2.6%). Incidents range from elevated error rates to outages, and OpenAI labelled none of its 18 API incidents a full outage, so the share of requests that actually failed was lower than these hours alone suggest [1][2].

Does multi-provider failover actually help? Yes, because providers rarely fail together. Anthropic's and OpenAI's incidents overlapped for 0.99 hours in 90 days. Routed through ModelRiver with failover, exposure to provider incidents fell by 96–97%, even after counting ModelRiver's own downtime [3][4].

How much latency does an AI gateway add? A median of 33.2 ms (p99 46.8 ms) in our production-like local test. Each failover added a median 18.4 ms. Network distance to the gateway comes on top, so measure from your own region [4].

How much does it cost to build AI infrastructure yourself? For routing with failover, async delivery, signed webhooks and orchestration: between 84 and 1,259 engineer-days by our measurements. At the US median developer salary that is $43,932 to $658,457, plus about 24 engineer-days a year on fixes [4][5][6].

What is AI orchestration, as opposed to an AI gateway? A gateway decides where one model call goes and what happens when it fails. Orchestration manages everything around the calls: queueing, waiting for your backend between AI steps, timeouts, and delivering results to the browser and your servers. ModelRiver does both.

Can I verify these numbers? Partly. The incident counts, prices, salary data and cost model all come from the public sources listed below. You can also measure ModelRiver's added latency from your own region: put a workflow in test mode, which returns sample data after a delay you set without calling a provider or using your quota, time a batch of requests, and subtract the delay. The code and history counts come from our private repository, so we publish the results, not the source.

Sources

All web sources were accessed on November 7, 2026. Numbers in brackets above point here.

1. Anthropic status page, incident feed (status.claude.com, incidents.json). Incidents tagged "Claude API (api.anthropic.com)", August 9 to November 7, 2026.
2. OpenAI status page (status.openai.com). Component-level incident impacts for the API components; Chat Completions and Responses for the overlap.
3. ModelRiver status page (status.modelriver.com). Per-minute health checks of the API and database, 90-day uptime.
4. ModelRiver, raw results for this post: results-2026-11-07.json (per-feature counts, benchmark results with machine load, incident overlap, uptime, cost-model inputs) and sources-2026-11-07.md (filtering rules and the full price tables).
5. Center for Software Engineering, University of Southern California. COCOMO II Model Definition Manual, version 2.1, 2000 (PDF). Constants A = 2.94, B = 0.91, nominal scale factors summing to 18.97.
6. U.S. Bureau of Labor Statistics, Occupational Employment and Wage Statistics, May 2025: Software Developers, SOC 15-1252, national median annual wage $135,980 (occupation page; data series OEUN000000000000015125213).
7. Desk, an open-source support app built on ModelRiver (github.com/modelriver/desk).
8. ModelRiver pricing (modelriver.com/pricing).
9. Supabase pricing (supabase.com/pricing).
10. Pusher Channels pricing (pusher.com/channels/pricing).
11. Upstash QStash pricing (upstash.com/pricing/qstash).
12. Hookdeck Outpost pricing (hookdeck.com/outpost/pricing).
13. Svix pricing (svix.com/pricing).
14. Google AI Studio status page (aistudio.google.com/status).
15. ModelRiver docs, Webhook delivery and retries.

Related reading: The backend complexity behind a simple AI request, Why AI needs to call your backend more than once, LLM failover in production: same prompt, different model, different product, and the docs on backend pipelines and webhook delivery and retries.