On December 8th, 2022, at around 11:00am EST, AppNeta Engineering became aware of an issue characterized as missing Network Change Events. The issue continued in small pockets and for brief times, being addressed on a case by case basis until the final remediation was applied through the 15.24 release of AppNeta.
As a result of this issue, the App-01, App-05, App-07, App-10, App-13, App-13, App-14, App-19, App-20 application nodes have experienced sporadic, but brief windows of missing traceroute data, and the associated network change events.
AppNeta Engineering has conducted an investigation into the issue and identified the root cause as a fault in the messaging bus which caused our application to deadlock. This fault has been addressed in a software update to the messaging bus, which was applied on the 15.24 release of AppNeta, released on December 21, 2022.
In Addition to remediating the underlying software issue, AppNeta Engineering has addressed the gap in monitoring coverage which allowed this issue to go undetected. Should any similar fault be experienced in the future, our engineers will be proactively notified to remediate the issue promptly.