During a planned capacity expansion at one of our data centers, one of our devices was determined to be defective. Upon receiving the replacement, we attempted to sync it with the existing device but the pairing failed. At 7:00 AM CST on 08 February 2019, we rebooted the working device per the request of the vendor in an attempt to establish sync between both for the purpose of redundancy. Upon reboot, the working device failed.
At that point, we began restoring the working configuration. By 11:00 AM CST on 08 February 2019, the configuration was updated and loaded on the additional device. We’re working with the device manufacturer to determine why the device failed in the first place and to ensure redundancy is not jeopardized during any future upgrade plan.