On Tuesday January 13th, at approximately 1:45pm MT, Vitelity identified a critical issue impacting our core network and voice services. Upon investigation, it was determined that the incident was caused by Verizon disabling access to our core subnet 64.2.142.0/24 with no prior warning or notice. This caused a massive disruption in services to Vitelity and our customers.
What we did: Our network engineering team engaged Verizon with high priority to reactivate this subnet. Concurrently, we began rerouting our infrastructure to new IPs in spaces that we own, to prevent any future interruptions. We communicated via tickets and our status page the new information and worked to assist customers with migrating to the new host IPs as quickly as possible. The next day the old subnet did become available to us again, so customers who were unable to update could use the legacy IPs which allowed more time to migrate their services.
What customers need to do: If you were affected, and have not yet migrated to our new hosts for sending/receiving traffic, you will want to work to migrate as soon as you are able. While we do have the legacy subnet active again currently, Verizon could disable these again at any time. If this happens, we may not be able to reactivate them again, so we strongly encourage anyone who has not migrated yet to update your system(s). If you need assistance with this, please open a support ticket and our support team will provide you necessary information in order to update, which can also be found in the history of this event.
We thank you all for your extreme patience while we worked through this. We value your business, and sincerely apologize for any inconvenience. Emergency tickets opened during this event have been refunded. If you have any further questions or concerns, please open a ticket at portal.vitelity.net