Missing Network Change Event Alerts
Incident Report for AppNeta Services
Postmortem

On December 8th, 2022, at around 11:00am EST, AppNeta Engineering became aware of an issue characterized as missing Network Change Events. The issue continued in small pockets and for brief times, being addressed on a case by case basis until the final remediation was applied through the 15.24 release of AppNeta.

As a result of this issue, the App-01, App-05, App-07, App-10, App-13, App-13, App-14, App-19, App-20 application nodes have experienced sporadic, but brief windows of missing traceroute data, and the associated network change events.

AppNeta Engineering has conducted an investigation into the issue and identified the root cause as a fault in the messaging bus which caused our application to deadlock. This fault has been addressed in a software update to the messaging bus, which was applied on the 15.24 release of AppNeta, released on December 21, 2022.

In Addition to remediating the underlying software issue, AppNeta Engineering has addressed the gap in monitoring coverage which allowed this issue to go undetected. Should any similar fault be experienced in the future, our engineers will be proactively notified to remediate the issue promptly.

Posted Dec 23, 2022 - 13:16 EST

Resolved
This incident has been resolved.
Posted Dec 22, 2022 - 21:08 EST
Update
This issue has been addressed in the latest AppNeta release, deployed December 21st.
Posted Dec 22, 2022 - 21:08 EST
Update
We are continuing to monitor for any further issues.
Posted Dec 08, 2022 - 18:37 EST
Update
We are continuing to monitor for any further issues.
Posted Dec 08, 2022 - 16:10 EST
Monitoring
We have identified the issue and implemented a fix on all affected nodes. We are continuing to monitor the situation
Posted Dec 08, 2022 - 16:07 EST
Investigating
We are experiencing issues with Network Change Events processing and alerting on the App-01, App-05, App-07, App-10, App-11, App-13, App-14, App-19, and App-20 nodes. Customers may see stale routes and missing Network Change Event alerts from these nodes.

We are investigating the issue and impact and will provide regular updates
Posted Dec 08, 2022 - 14:44 EST
This incident affected: Delivery Monitoring.