Microsoft Blames Months-Old Bug for Nearly 24-Hour Global 365 Outage
A dormant July update, a Teams traffic shift and lost server capacity combined to knock Outlook and Exchange offline worldwide, Microsoft said.
Microsoft has traced last week’s nearly 24-hour global outage of Outlook, Exchange Online and other Microsoft 365 services to a software defect that had been sitting unnoticed on its servers since July. The company’s preliminary post-incident report says the defect caused the failure only when two other problems hit at the same moment: a surge of rerouted traffic and a shortfall in server capacity. The outage left businesses, hospitals, courts and government offices around the world without email.
The report, dated November 15 and posted in the Microsoft 365 Admin Center, was described in detail by BleepingComputer, The Next Web and Cybersecurity News. Microsoft has said it will publish a fuller account on November 20.
What happened
According to BleepingComputer, the disruption began at 11:20 a.m. Eastern time on Thursday, November 12, and Microsoft confirmed that customer impact had ended at 11:10 a.m. on Friday, November 13. Cybersecurity News gave the same window in Coordinated Universal Time, from 4:20 p.m. on November 12 to 4:10 p.m. the next day. Exchange Online, Microsoft’s cloud email service, was hit hardest. Most users could not reach their mailboxes, and many messages were badly delayed. Teams and the Copilot AI assistant were also affected.
The Next Web, citing outage tracker Downdetector, reported that problems were logged in 74 countries and territories.
Why the systems failed
Microsoft’s report points to three factors that combined:
- A latent bug. A July update to Microsoft 365 networking gateway servers, the machines that route user connections into the company’s cloud, contained a defect that ran for more than three months without visible problems.
- Extra traffic. About 15 hours before the outage, Teams traffic automatically failed over to the gateway infrastructure that also carries Exchange Online, adding load to those servers.
- Reduced capacity. Nearly 14% of gateway server capacity was already out of service because of an unrelated issue.
Cybersecurity News reported that Microsoft said the remaining capacity would have been enough on its own, but the combination set off the defect. The outlets describe the bug in slightly different ways. BleepingComputer calls it a flaw in a gateway server update. The Next Web and Cybersecurity News say it sat in an internal Windows process on those servers. Cybersecurity News said the problem affected only Microsoft’s gateway servers, not ordinary Windows computers, and that customers did not need to change any settings.
The defect caused what engineers call lock contention, in which competing tasks wait on the same shared resources. Under certain traffic patterns, that drained the servers’ processors and memory until they stopped responding. Microsoft said its load testing missed the issue because lab machines were less powerful than production hardware. The company wrote that the problem “is worsened on machines with more cores and newer hardware,” BleepingComputer reported.
How a local fault went global
As each server failed, automated systems shifted its connections to the next healthy machine, which then hit the same bug. Microsoft said the failure spread across its fleet in under two hours, and at the worst point more than 80% of gateway servers were unresponsive. Safeguards meant to keep problems within one region also failed, so the outage went worldwide.
Recovery was slow for several reasons. The Next Web reported that Microsoft’s usual rollback tools and feature switches were “unable to quickly revert the change,” in the words of Charles Lamanna, Microsoft’s executive vice president of Copilot, Agents and Platform. Servers that came back online were swamped again by backlogged traffic. Teams, OneDrive and SharePoint could move to other infrastructure and recovered sooner, but Exchange Online could not switch paths at its scale, so Outlook users waited longest.
Engineers eventually wrote an update that restored the Windows connection-handling code to its earlier state. They took large groups of servers out of service, applied the fix, restarted them and brought traffic back step by step. Microsoft also extended the email delivery window so that queued messages would not bounce back as undeliverable, according to Cybersecurity News.
The Next Web reported that more than 1,000 engineers joined outage calls led by Lamanna and Scott Guthrie, who runs Microsoft’s cloud business. Lamanna apologized publicly on X, writing, “We are deeply sorry for the impact to all our customers,” BleepingComputer reported. On November 13, Microsoft said the incident was a technical problem “not related to cyber or AI / agents,” according to The Next Web.
What comes next
Microsoft says it has added server capacity, improved monitoring and added safeguards to stop traffic redirection from setting off similar chain reactions. The November 20 report is expected to give more detail on those repairs.
The outage is also drawing attention in Washington. The Next Web reported that Rep. Andrew Garbarino, the New York Republican who chairs the House Homeland Security Committee, is seeking information from Microsoft and from the Cybersecurity and Infrastructure Security Agency, the federal office responsible for protecting critical computer networks. The episode is a reminder of how much daily work for companies and public agencies now depends on a small number of cloud providers.


