Title: Users having multiple issues related to email flow, links within email messages and the Microsoft Defender portal
User Impact: Users may experience multiple issues related to email flow, links within emails and the Microsoft Defender portal
More info: Impacted scenarios include, but are not limited to: - Some customers might be experiencing delays with Automated Investigations in Defender for Office 365. Additionally, Deeplink integration between these investigations and actions may not load as expected. We are working on cancelling investigations that resulted from the erroneous alerts and working on reducing latency of rest of the automated investigations and system. - Users were previously unable to send or receive email from multiple legitimate domains such as Google, or DropBox. Messages from these domains would have been quarantined. This issue is resolved now. - Users were unable to access links within emails as they were identified as risky. This is no longer occurring for legitimate email. - Users may have been blocked from sending emails, if their messages were incorrectly flagged as spam or phishing attempts. We've unblocked all known impacted users. - Users may have previously experienced latency within the Microsoft Defender portal due to large numbers of erroneous alerts. We have mitigated the performance issues related to Defender portal.
Newly sent emails and links within emails are working correctly as of Monday, May 10, 2021 2:37 PM UTC.
This is continuation of multiple communications, and users may have seen aspects of this event reported previously under EX255432 and EX255435 before the full impact of this incident was understood.
Current status: We are working on cancelling investigations that resulted from the erroneous alerts and working on reducing latency of rest of the automated investigations and system. We expect our testing and deployment process to take an extended period of time.
Scope of impact: This could potentially impact any user.
Start time: Monday, May 10, 2021, at 12:00 AM UTC
Next update by: Tuesday, May 11, 2021, at 7:00 PM UTC
Posted May 11, 2021 - 08:22 CDT
Update
Title: Users having multiple issues related to email flow, links within email messages and the Microsoft Defender portal
User Impact: Users may experience multiple issues related to email flow, links within emails and the Microsoft Defender portal
More info: Impacted scenarios include, but are not limited to: - Users were previously unable to send or receive email from multiple legitimate domains such as Google, or DropBox. Messages from these domains would have been quarantined. This issue is resolved now. - Users were unable to access links within emails as they were identified as risky. This is no longer occurring for legitimate email. - Users may have been blocked from sending emails, if their messages were incorrectly flagged as spam or phishing attempts. We're working to unblock these users. - Users may have previously experienced latency within the Microsoft Defender portal due to large numbers of erroneous alerts. We have mitigated the performance issues related to Defender portal. - These same alerts are also causing delays in getting the latest alert and email information to admins in Threat Explorer. We're working to identify the most expedient way of restoring Threat ingestion.
Newly sent emails and links within emails are working correctly as of Monday, May 10, 2021 2:37 PM UTC.
This is continuation of multiple communications, and users may have seen aspects of this event reported previously under EX255432 and EX255435 before the full impact of this incident was understood.
Current status: We've completed reprocessing emails that were incorrectly quarantined during the impact window. We've identified a number of users that remain unable to send new messages and are working to unblock them; though, we've confirmed that no new users will be incorrectly blocked. Furthermore, we're investigating the most expedient means of optimizing Threat Explorer ingestion and resolving the ingestion delays.
Scope of impact: This could potentially impact any user.
Start time: Monday, May 10, 2021, at 12:00 AM UTC
Next update by: Monday, May 10, 2021, at 10:30 PM UTC
Posted May 10, 2021 - 16:35 CDT
Update
Title: Users having multiple issues related to email flow, links within email messages and the Microsoft Defender portal
User Impact: Users may experience multiple issues related to email flow, links within emails and the Microsoft Defender portal
More info: Impacted scenarios include, but are not limited to: - Users may be unable to send or receive email from various domains. Some of the domains include Google. - Users may notice legitimate messages are getting quarantined. - Users are unable to access links within emails as they're identified as risky. - Users may get blocked from sending emails, if their messages were incorrectly detected as spam or phish. - Admins might see delays in getting latest alert information and email information in Threat Explorer. - Microsoft Defender may be receiving a large amount of erroneous alerts, which could result in overall latency navigating within the Microsoft Defender portal.
Newly sent emails and links within emails are working correctly as of Monday, May 10, 2021 2:37 PM UTC.
This is an amalgamation of incidents and users may have previously seen these impacts reported under EX255432 and EX255435.
Current status: We've confirmed that newly sent emails and links within emails are working as expected. We've reprocessed emails for the most heavily impacted domains and we're monitoring email queues to ensure they deliver as expected. Once those emails email queues have drained, we'll continue to reprocess the emails on the remaining domains. Additionally, we've restored the latency issues with the Microsoft Defender portal.
Scope of impact: This could potentially impact any user.
Next update by: Monday, May 10, 2021, 3:30 PM (8:30 PM UTC)
Posted May 10, 2021 - 15:13 CDT
Monitoring
Title: Some users see legitimate email quarantined/marked as malicious in Exchange Online Protection & Defender for Office 365 ID: EX255432
Details Title: Some users see legitimate email quarantined/marked as malicious in Exchange Online Protection & Defender for Office 365
User Impact: Users are seeing legitimate email quarantined or marked as malicious within EOP and Microsoft Defender for Office 365.
More info: Users of Microsoft Defender for Office 365 and Microsoft 365 Defender may see following additional impact: - An increase in the number of URL related alerts for non-malicious URLs. - An increase in the number of Zapped Phish AIR investigations within Microsoft Defender for Office - Legitimate Emails being marked as malicious within Threat Explorer. - Delays in getting the latest email information within Threat Explorer.
Current status: We've identified a recent change in the infrastructure that serves these scenarios that inadvertently caused impact to the service. We're in the process of deploying a fix to restore the service and then reprocess any impacted URLs.
Scope of impact: Impact is specific to users who are served through the affected infrastructure.
Start time: Monday, May 10, 2021, at 12:00 AM UTC
Root cause: Legitimate URLs are incorrectly listed within our detection rules, resulting in impact.
Next update by: Monday, May 10, 2021, at 5:30 PM UTC
Posted May 10, 2021 - 10:35 CDT
Update
Title: Some users are seeing that legitimate email is being quarantined within the Exchange Online service
User Impact: Users are seeing that legitimate email is being quarantined within the Exchange Online service.
Current status: We're continuing with delisting the legitimate URLs from our anti-spam detection.
Scope of impact: Impact is specific to users who are served through the affected infrastructure.
Start time: Monday, May 10, 2021, at 6:26 AM UTC
Root cause: Legitimate URLs were incorrectly listed within our Anti-Spam detection rules, resulting in impact.
Next update by: Monday, May 10, 2021, at 4:00 PM UTC
Posted May 10, 2021 - 08:40 CDT
Identified
Title: Legitimate email is being sent to quarantine ID: EX255432
Service Degradation
Title: Some users are seeing that legitimate email is being quarantined within the Exchange Online service
User Impact: Users are seeing that legitimate email is being quarantined within the Exchange Online service.
Current status: We've identified that this issue is affecting a wider region and are working to delist the legitimate URLs from our anti-spam detection, to resolve the issue.
Scope of impact: Impact is specific to users who are served through the affected infrastructure.
Next update by: Monday, May 10, 2021, at 1:00 PM UTC