nine_ch

We're a company providing hosting for SME in Switzerland and recently moved our managed services' ingress and our PaaS Deploio from ingress-nginx to haproxy-ingress (because ingress-nginx retired).

Problems came up with the defaults, not with the annotation mapping: 50 instead of 60 second timeouts (almost 10 times longer on exports that finished faster than a minute before), 2000 per pod for maxconn and unchanged HTTP/1.0 passing through to a rejecting S3 backend. And on musl, there was bcrypt basic auth behind a process-wide spinlock, pushing one pod past 2 CPU cores with 155 concurrent connections.

In the linked blogpost, you'll be able to read about the 8 differences and values we ended up with.

Yes. The blog version was drafted with AI assistance and then edited and reviewed by us. The facts, numbers and timeline are ours and match the technical postmortem PDF linked in the post, which is the drier, more complete version. Happy to go into any of the technical details here.

Yes, that's bunny.net, it's named in the post. Their Zurich PoP appears to sit on CDN77's network, which is on their side of the setup, not something we configured.

We're a Swiss managed hosting provider and run our own network. In August one of our customers was hit by a UDP amplification attack that we estimate peaked at 500 to 600 Gbit/s across all our links combined, several times what our uplinks can carry. Because we provide that customer's internet connection, the impact hit our network directly, and roughly three hours later the attack was broadened to our own services as well.

The write-up is mostly about what we got wrong: our automated detection only covered our own prefixes, not customer prefixes we announce on their behalf. Blackholing at the IXs we connect to only took effect via route servers, not on direct peerings. And we had no way to withhold a prefix from one specific upstream, so we had to build that while under full load. On top of that, our own website runs on the same shared platform as customer apps, so when it was targeted, unrelated applications were affected too.

What eventually ended it was moving exposed applications behind a CDN with DDoS protection. The full timeline is in the PDF postmortem linked from the post. Happy to answer questions.

We're a Swiss managed hosting provider. On April 30 a Linux kernel 0-day (local root via a few lines of Python) dropped with no patched kernel available yet for our Ubuntu fleet. This is a write-up of how we handled it, what worked, and what we'd do differently. Happy to answer questions.