We are continuing to monitor for any further issues.
Posted Dec 03, 2024 - 17:02 CST
Monitoring
Microsoft Update:
The fix has been deployed, and we've confirmed the databases have returned to a healthy state and are functioning within acceptable thresholds. After monitoring to ensure service health was stable, we've verified impact has been remediated.
- We're continuing to monitor the affected databases for any signs of unhealthy service. - We're developing wider, preventative measures to ensure this issue doesn't return.
Posted Dec 03, 2024 - 17:02 CST
Update
Microsoft Update:
We're preparing to deploy our fix and anticipate that this process should be underway by our next scheduled communications update. We'll provide a precise remediation timeline once available.
Posted Dec 03, 2024 - 08:16 CST
Update
Microsoft Update:
The aforementioned fix which was developed to manually update the metadata is undergoing the final stages of testing, and we expect to have a deployment timeline by our next update.
Posted Dec 02, 2024 - 15:51 CST
Update
Microsoft Update:
We've determined the error impacting certain portions of infrastructure is resulting from a table responsible for updating indexes on calls to the database being deleted and metadata not updating to reflect this table being We've determined the error impacting certain portions of infrastructure is resulting from a table responsible for updating indexes on calls to the database being deleted and metadata not updating to reflect this table being deleted. We suspect this table may have been deleted twice in two separate but concurrent transactions, leaving the metadata in a dysfunctional state. We’ve developed a fix to manually update the metadata so the replication mechanism can correct itself and are testing it to determine its effectiveness.
Posted Dec 02, 2024 - 12:55 CST
Identified
Microsoft Update:
We've identified an error which is occurring in the affected portions of infrastructure, and are developing a fix to address the issue while we investigate the cause of this error.
Posted Dec 02, 2024 - 10:38 CST
Investigating
Microsoft ID: EX946693
User impact Users may be unable to send or receive email messages in Exchange Online using Outlook on the web or desktop client.
Scope of impact Impact is specific to some users who are served through the affected infrastructure.
Root cause A table responsible for updating indexes on calls to the database was deleted, as intended, by an internal process. However, the corresponding metadata to reflect that deletion did not update correctly due to an unexpected service error that occurred on a subset of infrastructure, which resulted in the impact.