When GitHub or Microsoft 365 goes down, your agent pipeline goes with it
GitHub published its explanation of the August 17 outage a few days later, and it is the kind of postmortem worth reading twice. The incident lasted about eight hours. Peak traffic overloaded the Central US infrastructure after a scaling failure in a critical component; the error rate hit roughly 20% on web and API and roughly 50% on raw file and archive downloads; retry storms around Copilot made recovery slower; and there was no code change involved. Two weeks later, on September 1 and 2, Microsoft spent two days restoring Microsoft 365 after Exchange Online, Teams, SharePoint, OneDrive and M365 Copilot all went down together, traced to a shared authentication configuration across products. Two vendors, two root causes, the same lesson for anyone running agents.
What the two incidents share
Neither outage was a bad deploy. GitHub's was capacity: a component that should have scaled did not, under load that the AI coding boom has been steadily raising. Microsoft's was identity: a configuration shared by many products, so when it broke, everything behind it broke at once, and the AI assistant went down with the mail.
The detail I keep coming back to is the retry storm. Copilot clients and the agents built on them retried failed calls aggressively, which added load to the exact component that was already failing to scale. Your agent pipeline probably does the same thing. Every coding agent I have looked at treats a failed git fetch or API call as transient and retries in a tight loop, because on a normal day that is correct behaviour.
What your pipeline inherits
An agent pipeline is a stack of dependencies you did not build. A typical one: identity provider, GitHub for code and CI, a model API, a package registry, and a chat surface. Any one of them going down stops the loop. Two of them just did, for hours, and one of them took an identity layer down with it, which means that even the services that were healthy could not be reached by anyone who needed to log in first.
The M365 case is the sharper one. An enterprise Copilot deployment does not have its own login. It borrows the tenant's. When the tenant's auth breaks, the assistant is not degraded, it is gone, along with the documents and chat it was supposed to help with.
An AI assistant is exactly as available as the identity layer it authenticates through, and no amount of model uptime changes that.
Degraded mode, prepared in advance
- Mirror the repositories you cannot work without. A bare mirror on a box you control, synced hourly. During the GitHub outage, raw and archive downloads failed at 50%; a local mirror fails at 0%.
- Cache packages and containers. A pull-through registry cache means a build that already ran once can run again with the upstream down.
- Retry with backoff and a ceiling. Exponential backoff, jitter, and a hard stop after which the agent reports "upstream unavailable" instead of hammering the endpoint. You are not going to fix GitHub by retrying, and you are making it worse for everyone including yourself.
- Define what runs without CI. Which checks can run locally, which need the hosted runner, and what the merge policy is when the runner is gone. Decide this on a calm day.
- Keep a local model path. For code assistance specifically, a local model on a workstation keeps completions working when the cloud model or the identity in front of it is down. It is slower and dumber and it is there.
- Separate the assistant's identity from the tenant's where the vendor allows it. Not always possible; worth asking.
- Write the runbook for "vendor down, no ETA". Who decides to switch to degraded mode, what gets paused, what gets merged manually.
The honest gap
Both explanations are vendor-written. GitHub's says what failed and roughly how much; it does not say why the component failed to scale or what changed to prevent it. Microsoft's root cause is one sentence about a shared configuration. Neither gives error rates by region or by customer tier, so I cannot tell you whether your pipeline would have been in the 20% or the 50%. The mitigations above do not depend on those details, which is the point of preparing them before the next postmortem.