Hacker News Reader

Comment by BubRoss | Hacker News Reader

This actually someone asking and not an investigation and explanation. There isn't even a lot of due diligence to figuring it out - no profiling or resource usage other than CPUs. Also it is musl combined with docker causing a 30x slowdown.

If something is running 30x slower from linking in a different libc, I'm guessing it should not be that difficult to narrow down the cause at least a little bit.

View on HN

Yeah, upvoted. 30x slower on 48 core (?) system sounds suspiciously like excessive lock contention (or some other shared resource). Non-NUMA aware allocator (or other code) might also contribute to the issue.

Should be fairly easy to investigate.

He has so many layers in there it's going to be tough to find the problem.

"Ballista is an experimental distributed compute platform, powered by Apache Arrow, with support for Rust and JVM (Java, Kotlin, and Scala)."

Plus he's got Docker, the Rust library, musl, and jemalloc sometimes. There's no application. All this is just infrastructure.

Musl doesn't do much on its own. But it does do stdio buffering. Could it be that the buffering system is making too many I/O calls, like flushing on every write?

Excessive I/O is indeed another common issue. The number of times I've seen a slow system and discovered excessive number of flushes, often in something like logging system to be the root cause... Or gazillion of unbuffered 1 byte writes or reads.

Exactly, not sure why you're downvoted. His blog-post doesn't explain why it is slow but instead 'switches the allocator' and says he has "fixed" the issue. This is just a workaround, not a fix.

I'm quite surprised that there's no mention of profiling the actual allocator causing this regression to properly narrow the fault down to the source. Instead this blog-post encourages cargo-culting development to "fix slow code".

Yeah, you should be upvoted.

The author put zero effort into figuring things out.

Benchmarking in Docker in general is a mistake I believe.

You should probably be measuring your app's performance in a production-like environment though.

You should load test in a realistic environment. Comparative benchmarks should be done in a consistent environment.

I think it makes sense for software that is intended to run in Docker and frameworks like Kubernetes that use Docker.

Why? The only overhead you have in Docker is on syscalls (due to permission checks, namespaces, ...), everything else runs at 100% native speed - unlike assisted virtualization (at least IOMMU overhead plus overhead for anything involving the filesystem) or emulated virtualization (obvious overheads here).

If both sides of the comparison run inside Docker that is still a valid comparison. With Docker the benchmark should be a bit more reproducible by anyone.

There are serious and undocumented problems in Docker. For instance, I spent multiple days investigating an issue in my tests that eventually I realized was due to accessing memory returned by an `mmap` syscall on a Docker mounted FS causing a SIGBUS, but only some of the time.

That is super dangerous and shook my confidence in Docker.

Reply on news.ycombinator.com