TLDR: Ubuntu unattended-upgrades and the systemd-networkd `ManageForeignRoutes=true` [1] default broke all Kubernetes networking globally, across multiple regions and cloud providers simultaneously, at 06:xx UTC. Recovery was difficult and slow due to the scale, particularly with cloud auto-scaling behavior kinda making things worse when all nodes failed simultaneously.

Please test how `systemctl restart systemd-networkd` affects your custom networking setup. It is quite likely to break.

DataDog also published their own post-mortem: [2]

[1] https://www.freedesktop.org/software/systemd/man/networkd.co...

[2] https://www.datadoghq.com/blog/2023-03-08-multiregion-infras...