Title: Some users were unable to use the search functionality in multiple Microsoft 365 services
User impact: Users were unable to access search functionality in multiple Microsoft 365 services.
More info: Users were previously unable to access search functions within Microsoft 365 services, including but not limited to: - Microsoft Teams - SharePoint Online - Outlook desktop and web
Final status: We have identified and fixed the underlying issue which resulted in this severe impact to users and confirmed that it is no longer occurring within the originally impacted Microsoft 365 services. We've confirmed a rare secondary impact behavior in which users may receive incomplete returns from search queries. This issue is a result of functional issues within the search index encountered during search query requests and impacts approximately one percent of user queries.
We're continuing our previously described course of action to finalize a valid fix for this issue; however, any fix for this issue will require a detailed and time-intensive process to ensure proper service performance is maintained at current levels for the duration of the repair. We will provide ongoing SHD updates during this period, which will be provided through a separate notification.
Scope of impact: The original impact from this event would have affected any user in our worldwide production environment when their search query request was processed through the affected infrastructure. We’ve confirmed that this issue did not impact customers within our Government Cloud environments.
Start time: Monday, April 24, 2023, 2:00 AM (7:00 AM UTC)
Root cause: A recent service change inadvertently caused an encoding mismatch between the crawler and query return service, causing search query failures for users within Microsoft Teams, SharePoint Online, and Outlook desktop/web clients.
End time: Monday, April 24, 2023, 8:00 PM (4/25/2023, 1:00 AM UTC)
We will publish a preliminary Post-Incident Report within two business days, and a final version within five business days following resolution of the additional impact.
Posted Apr 27, 2023 - 18:08 CDT
Update
Title: Some users may be unable to use the search functionality in multiple Microsoft 365 services
User impact: Users may be unable to use the search functionality in multiple Microsoft 365 services.
More info: Users are no longer prevented from accessing search functions within Microsoft 365 services. Previously impacted service platforms included but were not limited to: - Microsoft Teams - SharePoint Online - Outlook desktop client - Outlook on the Web
Users may encounter a secondary impact in which search queries return partial results; though, this is a rare occurrence affecting approximately one percent of search requests and does not prevent user access to Microsoft 365 services. Users who encounter the secondary impact may be able to successfully return more complete results by altering their search query.
Current status: We’re validating the potential fix for the secondary issue on a limited set of internal systems. Once the fix is validated it will be implemented across all affected infrastructure. An estimated time for resolution will be provided after the initial validation is completed. We understand the impact this can have on your organization, and we appreciate your partnership and patience as we work to safely resolve the issue.
Scope of impact: This issue can affect any user in our worldwide production environment but will only occur for a given user when their request traffic is served through the affected infrastructure. We’ve confirmed that this issue does not impact customers within our Government Cloud environments.
Start time: Monday, April 24, 2023, 2:30 AM (7:30 AM UTC) .
Preliminary root cause: A recent service change inadvertently caused an encoding mismatch between the crawler and query return service, causing search query failures for users within Microsoft Teams, SharePoint Online, and Outlook desktop/web clients.
Next update by: Thursday, April 27, 2023, 12:30 PM (5:30 PM UTC)
Posted Apr 27, 2023 - 08:02 CDT
Update
Title: Some users may be unable to use the search functionality in multiple Microsoft 365 services
User impact: Users may be unable to use the search functionality in multiple Microsoft 365 services.
More info: Users are no longer prevented from accessing search functions within Microsoft 365 services. Users may encounter a secondary impact in which search queries return partial results; though, this is a rare occurrence affecting approximately one percent of search requests and does not prevent user access to Microsoft 365 services.
Previously impacted service platforms include but are not limited to: - Microsoft Teams - SharePoint Online - Outlook desktop client - Outlook on the Web
Current status: Our focus remains on understanding and addressing the residual impact from this event. We’re currently testing our proposed short-term changes within our testing internal environments to validate its efficacy. If successful, we will push our change to the affected Production infrastructure. Additionally, we are still observing a small percentage (approximately one) of active search traffic is still impacted. Due to our focus on retaining high levels of availability and caution regarding this incident, we expect our testing to take an extended period. We greatly appreciate the patience we’ve received from your organization and users.
Scope of impact: This issue can affect any user in our worldwide production environment but will only occur for a given user when their request traffic is served through the affected infrastructure. We’ve confirmed that this issue does not impact customers within our Government Cloud environments.
Start time: Monday, April 24, 2023, 2:30 AM (7:30 AM UTC)
Preliminary root cause: A recent service change inadvertently caused an encoding mismatch between the crawler and query return service, causing search query failures for users within Microsoft Teams, SharePoint Online, and Outlook desktop/web clients.
Next update by: Wednesday, April 26, 2023, 11:30 AM (4:30 PM UTC)
Posted Apr 26, 2023 - 07:05 CDT
Update
Title: Some users may be unable to use the search functionality in multiple Microsoft 365 services
User impact: Users may be unable to use the search functionality in multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to: - Microsoft Teams - SharePoint Online - Outlook desktop client - Outlook on the Web
While we were applying mitigations for the underlying issue, some users may have also experienced latency while attempting to perform a search and issues accessing the Outlook client and Outlook on the web.
Current status: The long-term primary fix has been deployed to the affected infrastructure. We’re monitoring the environment to ensure there is no residual impact.
Scope of impact: Impact is specific to some users who are served through the affected infrastructure. Customers hosted in any worldwide region may experience impact related to this event; though, the degree of impact can vary. Additionally, we’ve confirmed that there is no impact to any Government Cloud environments.
Start time: Monday, April 24, 2023, 2:30 AM (7:30 AM UTC) .
Next update by: Tuesday, April 25, 2023, 12:30 PM (5:30 PM UTC)
Posted Apr 25, 2023 - 10:32 CDT
Update
Title: Some users may be unable to use the search functionality in multiple Microsoft 365 services
User impact: Users may be unable to use the search functionality in multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to: - Microsoft Teams - SharePoint Online - Outlook desktop client - Outlook on the Web
Telemetry shows that impact associated to access issues with the Outlook client and Outlook on the web have been remediated as of Tuesday, April 25, 2023, 4:30 AM (9:30 AM UTC) .
Current status: The short-term mitigations have been applied across the affected infrastructure and we’re seeing positive feedback from customers and telemetry. Our validation workstreams for the long-term primary fix is ongoing and are in the final stages before deployment.
Scope of impact: Impact is specific to some users who are served through the affected infrastructure. Customers hosted in any worldwide region may experience impact related to this event; though, the degree of impact can vary. Additionally, we’ve confirmed that there is no impact to any Government Cloud environments.
Start time: Monday, April 24, 2023, 2:30 AM (7:30 AM UTC) .
Next update by: Monday, April 24, 2023, 7:30 PM (4/25/2023, 12:30 AM UTC)
Posted Apr 25, 2023 - 06:59 CDT
Update
Title: Some users may be unable to use the search functionality in multiple Microsoft 365 services
User impact: Users may be unable to use the search functionality in multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to: - Microsoft Teams - SharePoint Online - Outlook desktop client - Outlook on the Web
Current status: We're seeing positive telemetry signals from our interim change after rolling it out to a limited portion of affected infrastructure. We’re proceeding with deploying the interim change across all affected infrastructure and anticipate this will be completed around Monday, April 24, 2023, 8:00 PM (4/25/2023, 1:00 AM UTC) . This interim fix is intended to improve the user experience; however, it may not fully return the expected search behavior. In parallel, we're continuing to work through the validation process for our primary fix.
Scope of impact: Impact is specific to some users who are served through the affected infrastructure. Customers hosted in any worldwide region may experience impact related to this event; though, the degree of impact can vary.
Start time: Monday, April 24, 2023, 2:30 AM (7:30 AM UTC) .
Next update by: Monday, April 24, 2023, 7:30 PM (4/25/2023, 12:30 AM UTC)
Posted Apr 24, 2023 - 19:00 CDT
Update
Title: Some users may be unable to use the search functionality in multiple Microsoft 365 services
User impact: Users may be unable to use the search functionality in multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to: - Microsoft Teams - SharePoint Online - Outlook desktop client - Outlook on the Web
Current status: Our validations are progressing successfully, and we anticipate will complete within the next 30-60 minutes. Additionally, we're testing another change, which has shown some positive results and we're working to apply this change to a broader scope of impacted users to determine its efficacy.
Scope of impact: Impact is specific to some users who are served through the affected infrastructure. Customers hosted in any worldwide region may experience impact related to this event; though, the degree of impact can vary.
Start time: Monday, April 24, 2023, 2:30 AM (7:30 AM UTC)
Next update by: Monday, April 24, 2023, 5:30 PM (10:30 PM UTC)
Posted Apr 24, 2023 - 17:30 CDT
Update
Title: Some users may be unable to use the search functionality in multiple Microsoft 365 services
User impact: Users may be unable to use the search functionality in multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to: - Microsoft Teams - SharePoint Online - Outlook desktop client - Outlook on the Web
Current status: Our testing of the fix is progressing, and we'll provide an estimated completion timeline as soon as possible. While we did see some positive results from our ECS change, we determined further changes could result in issues with our primary fix and are therefore exploring other avenues for more immediate relief.
Scope of impact: Impact is specific to some users who are served through the affected infrastructure. Customers hosted in any worldwide region may experience impact related to this event; though, the degree of impact can vary.
Start time: Monday, April 24, 2023, 2:30 AM (7:30 AM UTC)
Next update by: Monday, April 24, 2023, 3:30 PM (8:30 PM UTC)
Posted Apr 24, 2023 - 14:55 CDT
Monitoring
Title: Some users may be unable to use the search functionality in multiple Microsoft 365 services
User impact: Users may be unable to use the search functionality in multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to: - Microsoft Teams - SharePoint Online - Outlook desktop client - Outlook on the Web
Current status: We've encountered some delays in our final testing of the fix, though we expect to be completed shortly. In the interim, we've deployed a change intended to improve search result behavior and we're monitoring to determine its impact to search results.
Scope of impact: Impact is specific to some users who are served through the affected infrastructure.
Start time: Monday, April 24, 2023, 2:30 AM (7:30 AM UTC)
Next update by: Monday, April 24, 2023, 1:30 PM (6:30 PM UTC)