Vitelity inbound SMS, Outbound calling issues and faxing delays

Incident Report for Vitelity

Postmortem

Vitelity inbound SMS, Outbound calling issues and faxing delays - Master Case 00788346

‌Event Start Time: 08/14/2020, 4:02 PM MT
Event Stop Time: 08/14/2020, 6:43 PM MT
Services Impacted: Vitelity inbound SMS, delayed outbound sms traffic
Master Case: 00788346

Event Summary:
Vitelity SMS customers reported 100% inbound sms failure as well as delays for outbound sms traffic. During this time, engineers were able to recover and resend inbound sms messages from 17:45pm MT and later. Inbound sms messages between 16:02 MT and 17:45pm MT were non-recoverable.

Reason For Outage:
The public facing side of the firewall had a software error that created a loop condition which caused the switch-port it was connected to go into error-disabled mode. The secondary firewall, which should have taken over for the primary, had a loose cable causing a down-port condition on the switch. The primary port was bounced to restore service and the secondary interface cable was replaced and re-terminated to restore redundant connectivity.

Recommendations for preventing a reoccurrence:
The DevOps team is migrating subnets from a software firewall to a hardware firewall.

Posted Aug 18, 2020 - 11:21 CDT

Resolved

Timeframe: 8/14/2020 - Down 4:02 PM MDT/UP 6:43 PM MDT

Service has been confirmed restored. Public Facing side of the firewall had a software error that created a loop condition which caused the switchport it was connected to to go into error-disabled mode. The secondary firewall which should have taken over for the primary had a loose cable causing a down-port condition on the switch. The primary port was bounced to restore service and the Secondary interface cable was replaced and re-terminated to restore redundant connectivity.

Posted Aug 14, 2020 - 21:02 CDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Aug 14, 2020 - 20:31 CDT

Identified

The issue has been identified and a fix is being implemented.

Posted Aug 14, 2020 - 19:58 CDT

Update

We are continuing to investigate this issue.

Posted Aug 14, 2020 - 19:49 CDT

Update

We are continuing to investigate this issue.

Posted Aug 14, 2020 - 19:15 CDT

Update

We are continuing to investigate this issue.

Posted Aug 14, 2020 - 19:13 CDT

Investigating

Some Vitelity customers may be experiencing intermittent or down services issues (inbound sms, outbound calling and faxing delays). We are working to restore service as soon as possible and will keep updates posted here.

Posted Aug 14, 2020 - 17:00 CDT

This incident affected: Messaging Services (Inbound Messaging (SMS)) and SIP Trunking (Outbound Calling).