On 12-03-2024, two brief network outages were detected, affecting customers in the AMS and HAA data centres. During these outages, several customers have experienced packet loss and duplicate traffic.
In preparation of a future network migration, the issue was caused by a configuration change aimed at implementing a new network setup on hypervisors. This change inadvertently introduced a Layer 2 switching loop that propagated from our staging to our production environments.
We sincerely apologise for these outages, and the negative impact they had on our customers.
The outages occurred between 11:15:22 and 11:16:54 (CET), and between 13:51:22 and 13:57:08 (CET).
The first outage was detected by our engineers implementing the change. The second outage was detected by our monitoring solution after a configured grace period, after which our engineers reverted the change immediately.
We plan to improve our monitoring solutions to include information that leads to faster alerts in case of network issues, and also to improve our testing environment to ensure proper isolation from the production environment.