With every webpage loaded, email sent, or video streamed, network traffic takes a complex journey…
What Happened to Our Services?
Microsoft has had its own share of outages recently and during the evening of December 15th Azure AD was the cloud culprit. As a result, the Exoprise sensors detected this Microsoft 365 outage more than an hour before Microsoft informed customers of the issue. Here’s some of the errors that users were experiencing if they attempted to sign into Microsoft services:
- Navigation HTTP failure (503 – Instance is overloaded)
- Loading the starting URL resulted in an unexpected unrecognized screen on https://login.microsoftonline.com/
- Error interacting with Teams (Failed to lookup the tenant id of the logged in user)
- Username submission failed: There was an issue looking up your account. Tap Next to try again
Most of our worldwide customers knew well in advance of the problem before users or business suffered. Depending on their timezone and location they were able to avoid potential business disruption. Here’s a composite of dashboards and alarm messages that they received:
Proactive Notifications
- Starting at approximately 8:35 PM EDT, synthetic sensors detected issues throughout the Azure AD login system
- Some of the errors that begin to appear for our customers included:
- Navigation http failure (503 – Instance is overloaded) for https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize
- Username submission failed: There was an issue looking up your account. Tap Next to try again.
- These were received in proactive notifications including Alarm Emails, WebHooks, Integrations with ServiceNow and more
- Some of the errors that begin to appear for our customers included:
You can read more about our Outage Notifications and Monitoring of Microsoft 365 Outages here:
- Proactive Microsoft Outage Detection
- Microsoft 365 Monitoring
- Real-user Monitoring for Microsoft Services
Microsoft 365 Outage Notification Approximately 40 Minutes Later
FeatureDisplayName: Portal ImpactDescription: Users may experience intermittent access issues to multiple Microsoft 365 services. LastUpdatedTime: Thu, 16 Dec 2021 02:13:32 +0000 Messages:
| PublishedTime: | Thu, 16 Dec 2021 02:03:37 +0000 |
| MessageText: | Title: Users may experience intermittent access issues to multiple Microsoft 365 services User Impact: Users may experience intermittent access issues to multiple Microsoft 365 services. More info: Impacted services include but are not limited to: -Exchange Online -Microsoft 365 admin center -Partner portal Additionally, active sessions may not be affected by this problem. Current status: We’ve halted all deployments to the service as a preliminary troubleshooting step. In parallel, we’re reviewing network diagnostic data to determine the cause of the problem. Scope of impact: Based on our initial investigation, this issue appears to be impacting any user. Next update by: Thursday, December 16, 2021, at 2:30 AM UTC |
Final Resolution
https://admin.microsoft.com/#/servicehealth/history/:/alerts/MO305726
MO305726, Microsoft 365 suite, Last updated: December 15, 2021 10:43 PM
Start time: December 15, 2021 7:11 PM, End time: December 15, 2021 9:25 PM
Issue type: Incident
Status: Service restored
User impact: Users may have experienced intermittent issues accessing multiple Microsoft 365 services.
December 15, 2021 10:43 PM
Title: Users may have experienced intermittent issues accessing multiple Microsoft 365 services
User Impact: Users may have experienced intermittent issues accessing multiple Microsoft 365 services.
More info: Impacted services included but were not limited to:
– Exchange Online
– Microsoft 365 admin center
– Partner portal
– Services leveraging GraphAPI
– Power Platform
Users may have received an error when attempting to sign into the affected service.
Final status: We’ve determined that components between Microsoft account (MSA) and Azure AD sign-in services were not functioning as expected. We have rerouted traffic to alternate infrastructure to resolve the issue. We have confirmed via customer reports and extended monitoring that the issue is mitigated.Scope of impact: This issue may have impacted any user attempting to log into multiple Microsoft 365 services.Start time: Wednesday, December 15, 2021, 7:11 PM (12/16/2021, 12:11 AM UTC)
End time: Wednesday, December 15, 2021, 9:25 PM (12/16/2021, 2:25 AM UTC)
Preliminary root cause: A shared component of the Microsoft account (MSA) and Azure AD sign-in services stopped responding when a combination of a configuration error and a routine update caused multiple redundant endpoints to become unreachable. This caused sign-in failures in Microsoft services for both Microsoft personal accounts and Azure AD accounts.
Next steps:
– We are reviewing diagnostic data from the impacted infrastructure to determine methods to prevent similar events in the future.
We’ll publish a post-incident report within five business days.
Related Posts
