Hacker News Reader

Comment by capableweb | Hacker News Reader

> However, due to the fast, atomic operations involved at many places, HAProxy was previously limited to 64 threads, and therefore 64 CPU cores, on 64-bit machines. This limit is now raised to 4096 threads by the introduction of thread groups.

This is such a welcome change! I've been in more than one situation where I rediscovered this limitations when trying to scale things while trying to remain monolith (due to costs and performance reasons) but noticing HAProxy not being able to utilize the newly upgraded machine. This will make a huge difference!

View on HN

Less than you think. You only need few cores to saturate tens of gigabits of traffic, HAProxy already is very performant.

It's future-proofing basically.

The page is ancient https://www.haproxy.org/10g.html but they were doing 10Gbits on Core2Duo in '09 (sans SSL but back then AES acceleration wasn't very common either)

Indeed, we've done 100Gbps about a year ago on a 8-core machine.

Usually the only reason to use many cores is SSL, but since OpenSSL 3.0 that totally collapses under load, even then you're forced to significantly lower the number of threads (or to downgrade to 1.1.1 that about any high traffic site does).

Do you know what's behind the performance degradation of OpenSSL 3.0? Has the problem been documented anywhere?

Horrible locking. 95% CPU spent in spinlocks. We're still doing measurements that we'll report with all data shortly. Anyway many of them were already collected by the project; there are so many that they created a meta-issue to link to them: https://github.com/openssl/openssl/issues/17627#issuecomment...

3.1-dev is slightly less worse but still far behind 1.1.1. They made it too dynamic, and certain symbols that were constants or macroes have become functions running over lists under a lock. We noticed the worst degradation in client mode where the performance was divided by 200 for 48 threads, making it literally unusable.

Maybe I'm biased by what I was doing with it and when, but HAProxy seemed better suited to processes than threads; there's not much state that's useful to share, and sharing the single process list of FDs is a scalability challenge (and presents opportunities for bugs around reuse; I fixed one of those).

Also, I found it best to work with everything avoiding cross CPU communication; making sure everything happens on the CPU core that handles the RX queue for the incoming connection. My NICs only did 16 queues, so more cores than that wasn't useful for me. Do they make nics with 128 queues now? (Selecting outgoing ports so traffic to/from origin stays on the same CPU is tricky, but also solves the problem of making sure there's no port contention between the processes)

>Also, I found it best to work with everything avoiding cross CPU communication; making sure everything happens on the CPU core that handles the RX queue for the incoming connection.

Yes, this is absolutely critical. Atomic ops (including taking a lock) when you have a large number of cores just completely kills performance.

>Do they make nics with 128 queues now?

Yep, the Intel E810 100gbit controller supports 256 queue pairs for just one example.

Hope you can process that traffic pretty fast, the amount of memory dedicated to packet buffers with that many queues is making my head spin a bit.

Oh nice, in that case, time for 256 core haproxy box ;)

My use case was just tcp proxy, inspecting the first few bytes to determine destination (http vs https vs proprietary) and add proxy protocol headers to the origin. Setup is somewhat expensive (tcp table manipulation isn't cheap), but data on established connections is simple and easy.

For pure TCP you rarely need threads, indeed! You may benefit from TCP splicing however if you're dealing with high bandwidth.

Yeah, TCP splicing would have been neat. The use case wasn't high bandwidth, and it was transitional: client software has IP addresses to try when DNS fails, but we moved to new hosting related to an aquisition, HAProxy at the old host until all clients in the field have updated.

The bottleneck was connection establishment in the kernel, and as traffic decreased it became harder to run meaningful tests without great risk (send all the traffic to one server to see if it can handle it is fun, but unsafe), and we had to keep a fair number of servers to keep the IPs from the hosting provider, so we had way more capacity than needed and optimizing more wasn't a good use of time :(

Indeed, that's fun at the beginning but past one point when you'd want to stop the servers taking 3 connections a day but you can't because these are the 3 most business critical connections, it starts to become much less fun.

Connection establishment in the kernel was an issue before kernel 3.8 I think though not sure. The problem is finding a spare port when you use a lot of concurrent connections and at some point it costs a lot. The "source" directive in haproxy was made to overcome this (you can define a per-server range so that the kernel doesn't have to scan its lists). But nowadays more efficient mechanisms were implemented in the kernel, that we support in haproxy (using IP_BIND_ADDRESS_NO_PORT) and we haven't witnessed these painful moments for a while now.

I was running on FreeBSD, and using port selection inside HAProxy to avoid the kernel selecting it, but still had contention on locks when inserting connections into the table. I can't remember if that was FreeBSD 11 or 12, but the next version made improvements which looked nice, but I wasn't able to test.

Somewhere my (not very great, but working) patch set is floating around, where I added rss aware port selection to HAProxy, so client and origin connections would both hash to the same NIC RX queue, and HAProxy never had to deal with cross-cpu communication for reading sockets.

This interests me. I've long been wondering how we could do that without knowing the NIC's hash and/or without having to brute-force the ports using the NIC's hash to reverse-map them. But maybe there are kernel facilities I'm not aware of to do that. If you happen to stumble upon it and share it to the mailing list, that would be great!

https://www.mail-archive.com/haproxy@formilux.org/msg34548.h...

I haven't looked at this code in a long time! Haven't gotten to a computer and it's hard to read this on a phone, so... And I left where I wrote it, so I can't confirm if there were any substantial changes after these. My fuzzy memory is this may be all the HAProxy changes, then I focused on kernel level changes.

Many thanks! Too bad this thread was not noticed and left unresponded. I'll have a look. Thanks again!

IMO for super high packet rate cases, DPDK would make more sense. For proxy protocol injection, this would be fine since you don't really care about a lot of the TCP stack, just keeping track of new connections, and doing the sequence number rewriting to account for the injected bytes. All of the congestion control etc can be handled by the endpoints.

Of course if HAProxy performs well enough, then that's still a million times easier.

Yeah, I had lots of fun ideas, but had a time crunch, HAProxy basically just worked to get started, and took well to optimizing, and pretty soon I had more capacity that would ever be needed, and never had a chance to rebuild it with exotic technology ;)

Actually that what I've been saying for years until users complained that processes had different vision of server states in fast changing environments, that it was not possible to perform dynamic updates at runtime because all of them had to be synchronized, that stick-tables were only on one process, that they had to manage their stats and sum them externally, that the LB algorithms were uneven with many processes because roundrobin starts from the same server on all processes, and leastconn didn't know other processes' load, that the maxconn size was unmanageable as it couldn't consider connections from other processes, that rate counters weren't shared, that maps and ACLs had to be loaded and managed in every process, eating a lot of memory or not being updatable, that memory usage was going through the roof because unused memory from one process was not usable by another one, etc... The list was so long that it was clear at some point that we had to move to threads to resolve all of this at once. And we're doing really good, I still hate sharing data so we're extremely careful to share very little. Look at the pools, the thread_info struct and the thread_group struct to see how much is local to the thread or to the group. Even the idle server connections are thread-local but can be stolen if needed (i.e. almost never any sharing unless really needed). So we've kept the old practices of processes with the facilities offered by threads, and that's what brought us this great scalability.

Yeah, that's fair. That's a big list of features I didn't need ;) and I was working on it when threads were new.

And admittedly threads didn't work great 5 years ago when we put them into 1.8. Fortunately 1.8 is dead soon, one less thing to worry about :-)

Reply on news.ycombinator.com