Previously you were notified about a major incident we experienced on November 16, 2023, that impacted several of the Zix Services. All impacted services have been restored. The root cause for the issues impacting the Zix Services was a firmware bug affecting the SAN storage application utilized by Zix. Working with our SAN provider, we were able to identify the issue and ultimately replaced the faulty systems. Below is a summary of the services that were impacted and the resolution. Impact
· Zix Email Encryption customers may have experienced issues sending and receiving emails through the service or delivering emails through the Secure Email Message Portal.
· Zix Email Threat Protection customers may have experienced intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction and Attachment Quarantine features.
· Zix Archiving customers using our legacy email archiving solution (console.zixcentral.com) may have experienced a disruption in the archiving of their email communication and may have experienced intermittent issues accessing their archive.
Resolution
· Zix Email Encryption services have been fully restored.
· Zix Email Threat Protection services have been fully restored.
· Zix Archiving services have been fully restored.
More information In the following weeks we will create a detailed Root Cause Analysis (RCA) and make the available to customers. If you would like to receive a copy of the RCA, you can request it by sending an email to Zix Support.
Posted Nov 30, 2023 - 09:19 CST
Monitoring
Previously you were notified about a major incident we experienced on November 16, 2023, that impacted several of the Zix Services. All impacted services have been restored. The root cause for the issues impacting the Zix Services was a firmware bug affecting the SAN storage application utilized by Zix. Working with our SAN provider, we were able to identify the issue and ultimately replaced the faulty systems. Below is a summary of the services that were impacted and the resolution. Impact
· Zix Email Encryption customers may have experienced issues sending and receiving emails through the service or delivering emails through the Secure Email Message Portal.
· Zix Email Threat Protection customers may have experienced intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction and Attachment Quarantine features.
· Zix Archiving customers using our legacy email archiving solution (console.zixcentral.com) may have experienced a disruption in the archiving of their email communication and may have experienced intermittent issues accessing their archive.
Resolution
· Zix Email Encryption services have been fully restored.
· Zix Email Threat Protection services have been fully restored.
· Zix Archiving services have been fully restored.
More information In the following weeks we will create a detailed Root Cause Analysis (RCA) and make the available to customers. If you would like to receive a copy of the RCA, you can request it by sending an email to Zix Support.
Posted Nov 28, 2023 - 19:40 CST
Update
Our backend engineers continue in the recovery efforts of systems affected by the data center.
Threat Service - ZixProtect advanced feature (message retraction) is still being worked.
Additional status page updates will be provided in 24 hours or when new information becomes
Posted Nov 28, 2023 - 09:33 CST
Update
Our backend engineers continue in the recovery efforts of systems affected by the data center.
Threat Service - ZixProtect advanced features are still being recovered.
Additional status page updates will be provided in 24 hours or when new information becomes available.
Posted Nov 27, 2023 - 07:24 CST
Update
Service Incident Affecting Zix Services:
The teams continue to work to resolve outstanding issues.
Threat Protection service - ZixProtect advanced features are still being worked through.
The data center has made progress on restoring affected hardware and is in recovery efforts.
We apologize for any inconvenience this is currently causing.
Additional status page updates will be provided when any new information is available.
Posted Nov 25, 2023 - 09:09 CST
Update
Service incident affecting Zix Services. While the team continues to work to resolve the outstanding issues, progress has been made: ZixArchive service has been restored. ZixProtect Link Protection restored. SMB ZixDLP systems restored.
We apologize for any inconvenience this may currently be causing. Additional updates to the status page will be provided when any new information is available.
Posted Nov 21, 2023 - 18:04 CST
Update
Service incident affecting Zix Services.
Team continues to work to resolve: Legacy email threat protection in ZixCentral (console.zixcentral.com) and the intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction, and Attachment Quarantine features.
Archiving service in ZixCentral (console.zixcentral.com) experiencing a disruption in the archiving of email communication and intermittent issues accessing their archived emails.
We apologize for any inconvenience this may currently be causing.
Additional updates to the status page will be provided when any new information is available.
Posted Nov 20, 2023 - 16:49 CST
Update
Customer Advisory
Service incident affecting Zix Services.
On November 16, 2023, the major incident process was initiated to resolve issues affecting several of the Zix Services.
Impact
· Zix Email Encryption customers may have experienced issues sending and receiving emails through the service or delivering emails through the Secure Email Message Portal.
· Zix Email Threat Protection customers may have experienced intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction and Attachment Quarantine features. · Zix Archiving customers using our legacy email archiving solution (console.zixcentral.com) may experience a disruption in the archiving of their email communication and may have intermittent issues accessing their archive.
Resolution
· Zix Email Encryption services have been fully restored.
· Zix Email Threat Protection services have been fully restored for customers using AppRiver/Secure Cloud email threat protection. Customers using our legacy email threat protection in ZixCentral (console.zixcentral.com) may still be experiencing intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction, and Attachment Quarantine features.
· Zix Archiving services in AppRiver/Secure Cloud are fully functional. Customers using the archiving service in ZixCentral (console.zixcentral.com) are experiencing a disruption in the archiving of their email communication and may have intermittent issues accessing their archived emails.
More information
We are actively working to resolve all outstanding issues.
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services - Starting to implement the workaround to restore mail flow. Legacy ETP (ZixProtect) - not working for outbound scans Legacy ZixArchive - still working on impact ZixProtect ECS - still working on impact; LinkProtection (ZixProtect) - still being worked to restore - Inbound email after 11/15 will not have links wrapped - Inbound email prior to 11/15 will have links wrapped but will come up with an error on the UI but still can proceed to the link (use at your discretion) Any SMB/Hosted Enterprise encryption service using outbound ETP scanning--(workaround is to point smart host directly to encryption servers)
Updates will be posted with new information.
Posted Nov 19, 2023 - 14:39 CST
Update
Hello All,
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services - Working on a workaround to restore mail flow Legacy ETP (ZixProtect) - not working for outbound scans Legacy ZixArchive - still working on impact ZixProtect ECS - still working on impact; LinkProtection (ZixProtect) - still being worked to restore - Inbound email after 11/15 will not have links wrapped - Inbound email prior to 11/15 will have links wrapped but will come up with an error on the UI but still can proceed to the link (use at your discretion) Any SMB/Hosted Enterprise encryption service using outbound ETP scanning--(workaround is to point smart host directly to encryption servers)
Posted Nov 19, 2023 - 14:16 CST
Update
Hello All,
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services - Working on a workaround to restore mail flow Legacy ETP (ZixProtect) - not working for outbound scans Legacy ZixArchive - still working on impact ZixProtect ECS - still working on impact; LinkProtection (ZixProtect) - still being worked to restore - Inbound email after 11/15 will not have links wrapped - Inbound email prior to 11/15 will have links wrapped but will come up with an error on the UI but still can proceed to the link (use at your discretion) Any SMB/Hosted Enterprise encryption service using outbound ETP scanning--(workaround is to point smart host directly to encryption servers)
Posted Nov 19, 2023 - 13:49 CST
Update
Hello All,
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services Legacy ETP (ZixProtect) Legacy ZixArchive ZixProtect ECS LinkProtection (ZixProtect) Any SMB/Hosted Enterprise encryption service using outbound ETP scanning--(workaround is to point smart host directly to encryption servers)
We will update you as soon as information becomes available.
Posted Nov 19, 2023 - 10:17 CST
Update
Hello All,
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services Any SMB/Hosted Enterprise encryption service using outbound ETP scanning Legacy ETP (ZixProtect) Legacy ZixArchive ZixProtect ECS LinkProtection (ZixProtect)
We will update you as soon as information becomes available.
Posted Nov 19, 2023 - 08:58 CST
Update
RSYNC of data continues to process as our engineers are working to restore the issues surrounding mail flow.
Our QA/DEV teams continue to test other services that have been impacted and are working to restore functionality.
The ZixPortal service (secure message centers) seems to have stabilized and appears to be operational at this point.
An update to the status page will be made when any new information is available.
Posted Nov 19, 2023 - 07:51 CST
Update
RSYNCs of data continue to process.
Our QA/DEV teams continue to test other services that have been impacted by this disruption of services and are working to restore functionality.
The ZixPortal service seems to have stabilized and appears to be operational at this point.
Posted Nov 18, 2023 - 19:11 CST
Update
RSYNCs of data to a new pool continue to process.
The ZixPortal service appears to be in a stable position and after testing all typical functions of users passed inspection.
Our QA/DEV teams continue to test other services that have been impacted by this disruption of services and are working to restore functionality to those services.
Posted Nov 18, 2023 - 17:43 CST
Update
Rsyncs are still progressing. The engineering teams are still attempting to bring up certain VMs to verify data integrity. Other teams continue to work with Oracle and VMware resources.
The next update will be at 5:00 PM Central or when we receive a major update from our engineers.
Thank you
Posted Nov 18, 2023 - 12:58 CST
Update
Our Engineers are still working with the vendor engineers on the rsyncs to move data off the bad drive pool.
Next update will be at 1:00 PM Central.
Posted Nov 18, 2023 - 11:13 CST
Update
Rsyncs are still progressing and taking longer than expected, the engineering team is attempting to bring up certain VMs to verify data integrity. The team continues to work with Oracle and VMware resources.
The next update will be at 11:00AM Central
Posted Nov 18, 2023 - 09:11 CST
Update
Rsync process is still underway and progressing. Some engineer teams remain engaged on a call with Oracle as extended teams perform integrity checks and other required VM functions.
The next update will be at 9:00AM Central; sooner if there is a major update to the progress.
Posted Nov 18, 2023 - 07:03 CST
Update
Our engineers have reported that all remaining RSYNC processes are underway. These are expected to run for approximately the next 4 hours.
The next update will be at 7:30 AM Central.
Posted Nov 18, 2023 - 02:05 CST
Update
Our engineers continue to run RSYNCs of data to the new drive pool. There is no ETA on when the RSYNCs are expected to be completed.
The next update will be at 2:00 AM Central.
Posted Nov 18, 2023 - 00:04 CST
Update
We do apologize for the delays in the release of information as we progress through this issue. Our engineers continue to validate data moved in preparation for the new drive install on the SAN.
We will continue to update the status as we get information from the engineering team.
Posted Nov 17, 2023 - 22:05 CST
Update
Hello,
Our engineers continue to work on the issue with vendor engineers. They currently are in a crucial part of the data copy to a new pool from the old pool of drives. Once this has been completed, the engineers will swap out damaged drives and finally start the copy back once the drives are tested.
There is no ETA for this action.
Posted Nov 17, 2023 - 14:28 CST
Update
Hello Team,
We are aware of the issue of sending emails outbound from O365 to the encryption services. This issue is the result of the SAN service issue we have been experiencing. To restore service admins can access their O365 Exchange Admin Center to temporarily disable the outbound transport rule and send directly from O365. The rule can be enabled when the SAN service issue has been resolved.
This is at the sender's discretion and must be aware that emails with sensitive data will receive no encryption.
Our engineers are still working with the vendor engineers. We have had some success and now are able to move the data to a new pool to replace the hard drives.
Thank you
Posted Nov 17, 2023 - 10:13 CST
Update
Our engineers are still working with our vendor engineers on the issue.
We will update the status page in 3 hours or as soon as information becomes available.
Posted Nov 17, 2023 - 07:32 CST
Update
Hello Team,
The initial work to remediate the issue was not successful. With the assistance of Oracle engineers, our teams have started the process of making adjustments to work through finding stability for the affected drives within the SAN.
This process will take a few hours to complete while the Oracle engineers look further into the system's logs.
The next update will be at 7 am Central.
Posted Nov 17, 2023 - 00:17 CST
Update
Our teams have started the process to remediate the affected disks of the SAN.
We expect these actions to take a few hours to complete.
We will update the status as we receive new information.
Posted Nov 16, 2023 - 17:29 CST
Update
Our Engineers and Oracle appear to have isolated the issue to a pool of drives within a SAN that require a reseating to reset them.
Our engineers are in a meeting with Opentext leadership to decide the best course of action to minimize the impact on the customer even further.
Posted Nov 16, 2023 - 16:27 CST
Update
Mail flow for our encryption services is being redirected to our second data center to assist with the slowdown of mail in our primary.
ZixPortal access is still being affected and we believe is related to the SAN issue that is currently ongoing.
Our engineers are working with the 3rd party vendor of the SAN to rectify the situation.
Information will be updated as it becomes available.
Thank you
Posted Nov 16, 2023 - 13:50 CST
Update
Our engineers are working with external vendor engineers to try and isolate the issue that is causing the disruption to the listed services.
We will update this page when new information has been released about the disruption of service.
Thank you
Posted Nov 16, 2023 - 12:29 CST
Update
No change to the statuses. Our engineers continue to work on the issue.
Posted Nov 16, 2023 - 10:54 CST
Update
We are continuing to work on a fix for this issue.
Posted Nov 16, 2023 - 08:40 CST
Update
Adding Threat Protection services for OUTBOUND scanning.
We are getting reports from customers that we are seeing "Relay Access Denied" messages from O365. This is being investigated.
Posted Nov 16, 2023 - 08:29 CST
Update
Adding Secure Cloud Encryption Services (AEE/ZEE) to the affected systems. The configuring of new domains added to the domains is impacted.
Posted Nov 16, 2023 - 07:50 CST
Update
We are continuing to work on a fix for this issue.
Posted Nov 16, 2023 - 07:18 CST
Identified
Hello,
Our engineers have identified an issue with ZixPortal and are currently working to remedy the situation. We do apologize for this inconvenience.