HNHacker News
TopNewBestAskShowJobs

charleslmunger

1,327 karma · joined March 3, 2013

submissionscomments
charleslmunger··on HTTP/3 is everywhere but nowhere
Deploying QUIC led to substantial p95 and p99 latency improvements when I did it (admittedly a long time ago) in some widely used mobile apps. At first we had to correct our analysis for connection success rate because so many previously failing connections now succeeded slowly.

It's a material benefit over networks with packet loss and/or high latency. An individual human trying to accomplish something in an elevator, parking garage, or crowded venue will care about a connection being faster with a greater likelihood of success.

charleslmunger··on Microbenchmarks: Java Locks vs. Atomic
Did you use fair scheduling with ReentrantLock? That could account for a difference of intrinsic locks (synchronized) was a fair lock.

This type of benchmark is a pathological because the optimal scheduling policy is for one thread to starve all the others until it finishes, which is basically never what you'd want in a real project.

charleslmunger··on Microbenchmarks: Java Locks vs. Atomic
Why does reentrantlock exist? Well, one reason is that it supports APIs like tryLock and multiple condition variables.

Why is it faster than synchronized? This is probably because (depending on your jvm) ReentrantLock is built on LockSupport.park, which is virtual thread aware, and synchronized is operating directly on the platform threads.

charleslmunger··on The Big TDD Misunderstanding (2022)
>Anecdotally I've always found that tests which cover real life bugs are in the class of test with the highest chance of catching future regressions. So even if it does exist, I'm still mildly skeptical of the idea that tests that catch bugs that compilers have been provoked into also catching are strictly negative.

Maybe this is a static typing thing - if the test won't build because you've made the bug inexpressible in the type system, what test can you even write?

>If I write even a single line of production code I have specified what that line of code is going to do.

>combination of code investigation, spiking and dialog with stakeholders

Does "spiking" in this context mean writing code without TDD?

>I think this is conflating "TDD is a panacea" with "if it is valuable to write a test, it's always better to write it before".

That's a weaker formulation of TDD than I've seen espoused and practiced, which is usually something more like "before you make any behavioral changes to the code under test, you must write a test that fails; then make your edit so the test passes, and repeat". The problem with your approach at least for me is that until I've messed around a bit in the code seeing what's the best approach is to solving a problem, I don't know what the best structure for a final test is, or whether I can make the change in a way that leverages existing tests to detect the bug.

> I'm not really sure why fault injection should be treated as special.

Suppose you write a test reproducing a bug that reacts to allocation failing. One common way to do that is to inject an allocator that fails on the specific allocation call you had a bug with. But how do you target that specific call? One way is to make the Nth allocation in a test fail - but even slight changes to the production code will make this test start testing something completely different. The solution is to have a fuzz test that injects failed allocations randomly per run - now you can be reasonably confident that even if the prod code changes over time, your fuzz test will still all the allocation sites, preventing regression. I don't see why it's beneficial to write this first or last. Under your model the instrumented allocator setup is a replacement for the single test case that reproduces the original bug.

>If I'm reading it correctly, this looks like a class of bugs we discussed that is fixed by tightening up the typing. In which case, no test is strictly necessary, although I'd argue that it probably would not hurt either.

No, the original "bug" is that if you have two calls to SpaceAllocated and the second one races with a Fuse call on a other thread, you can see a smaller value result than before. This behavior wasn't guaranteed by the public API (so not a bug) but it's desirable to add. The fix is replacing a singly linked list with a doubly linked list, using tagged pointers to avoid adding storage cost for the second set of links. A test could be written for this; I could inspect the allocated addresses of three arenas and fuse them in a specific order, but I could not actually reproduce the ordering required without a test harness that allows full control of thread execution interleaving - which would be a huge amount of work and in my opinion not actually prevent future bugs, since that interleaving only meaningfully exists in the previous implementation. The full set of possible interleavings is way too large to productively explore. If I had started by writing the test, I would have spent a bunch of time messing around before giving up; because I did the implementation first, I had a much more informed idea of what would be required to test it, and changed my plan for preventing future bugs to documentation.

>I can't tell if this is refactoring or you're fixing a bug. Is there a scenario which would reproduce a bug?

In this case the "bug" is not really a bug - if we are allocating an arena, we put the overhead at the start rather than end of the first memory block. The reason we're doing this is that while it's totally valid to use the provided memory however we want, we can avoid faulting an extra page if we store the overhead near where we're about to allocate from. So the test would have to test "on a virtual memory platform, do we write to memory in a way that maximizes spatial locality". This is possible to do but it's a total mess of a test, and I have no idea how to do it on Windows. More importantly, anyone who changes this code is going to be doing it on purpose, and we don't guarantee to callers where we'll be placing any pointers we return.

>Again, I see no examples where writing a test after would have been better.

If TDD produces better results for you, that's great. I think I made a case with real examples where using TDD would have been at best neutral, but in practice would have made me spend more time to get the same results.

charleslmunger··on The Big TDD Misunderstanding (2022)
>This is actually a good (albeit somewhat niche) reason to not write a test scenario at all, but it's still not a great reason to write a test after instead of before.

But the before-test is strictly negative - it's a waste of time (deleted code, never submitted) and it possibly slowed down development (had to update the test as I messed with APIs).

>Yup. A test scenario which is of no interest to at least some stakeholders probably shouldnt be written at all.

And yet I see TDD practitioners as the primary source of such tests - if you are dogmatically writing a test for every intermediate change, you will end up with lots of extra tests that assert things in order to satisfy the TDD dogma rather than the specific needs of the problem. Obviously this can be avoided with judgement - but if you have sound independent judgement you don't need to adhere to specific philosophies about the order you make changes in.

>In fact it's probably a bit easier to link intentional behavior to a test while you have the spec in front of you and before the code is written.

When implementing to a spec you are absolutely right, but a very small amount of software is completely or even mostly specified in advance.

>I find people who write test after tend to (not always, but strong tendency) fit the test to the code rather than the requirement. This is really bad.

I agree this can lead to brittle tests and lack of spec adherence, but if you are iterating on intermediate state and writing tests as you go, the structure of the code you wrote 30 seconds ago is very much influencing the test you're writing now.

Another issue is that fault injection tests basically require coupling to the implementation - "make the Nth allocation fail" etc. The way I prefer to write these is to write the implementation first, then write the fuzz test - add a few bugs in the implementation, and fix/enhance the fuzz test until it catches them. Fuzz testing is one of the best bang-for-buck testing methodologies there is, and in my experience it's very hard to write a really good fuzz test unless you already have most of your implementation, so you can ensure your fuzz tester is actually exercising the stuff you want it to.

>Assuming Im understanding you correctly (you're building something like terraform?),

I write library code for mobile phones, mostly in Java/Kotlin. I recently did some open source work (warning: I am not actually very proficient with C, any good results are from enormous time spent and my code reviewers, constructive criticism very much welcome). Here's a few somewhat small, contained changes of mine, so we can talk about something concrete:

https://github.com/protocolbuffers/protobuf/pull/19893/files

This change alters a lock-free data structure to add a monotonicity invariant, when the space allocated is queried on an already-fused arena while racing with another fuse. I didn't add tests for this - I spent a fair bit of time thinking about how to do it, and decided that the type of test I would have to write to reliably reproduce this was not going to be net better at preventing a future bug, given its cost, than a comment in the implementation code and markdown documentation of the data structure. I don't know how I would really have made this change with a TDD methodology.

https://github.com/protocolbuffers/protobuf/pull/19933/files

This change moves a memory layout - again, I don't know how I would have written a test for this, besides something wild like querying smaps (not portable) to see if the final page of the arena allocation had faulted in.

https://github.com/protocolbuffers/protobuf/pull/19885/files

This change was written more in the way you recommend - but the whole change is basically a test. I debugged this by reading the code and thinking about it, then wrote up a pretty complicated fuzz test to help find any future races. I'm guessing that you would not consider adding debug asserts to be a violation of "write the test first"? So in this case, I followed TDD's order - not because I was following TDD, but because the code change was trivial and all the hard work was thinking about the data structures and memory model.

https://github.com/protocolbuffers/protobuf/pull/19688/files

All the tests were submitted before the implementation change here, but not because of TDD - in this case, I was trying to optimize performance, and wrote the whole implementation before any new tests - because changing the implementation required changing the API to no longer expose contiguous memory. But I did not want to churn all the users of the public API unless I knew my implementation was actually going to deliver a performance improvement - so I didn't write any tests for the API until I had the implementation pretty well in hand. Good thing too, because I actually had to alter the new API's behavior a few times to enable the performance I wanted, and if I had written all the tests as I went along, I'd have to go and rewrite them over and over. So in this case I wrote the implementation, got it how I wanted it, wrote and submitted the new API (implemented at first on the old implementation) and added tests, updated all callers to the new API, and then submitted the new implementation.

I don't think TDD would have led to better results in these cases, but you sound like a TDD believer and I'm always interested to hear anything that would make my engineering better.

charleslmunger··on The Big TDD Misunderstanding (2022)
I think having tests for all your diffs at the level of published commits/change lists/etc is totally reasonable for software you really care about. What's counterproductive is practicing TDD at the level of individual editor operations.

If I'm fixing a bug, I start by writing a test that reproduces the bug. If I can't do that, I fix the test harness until I can. Then I implement the change, making mental notes of each intermediate bug I think about along the way - things like "I should be careful to name this distinctly so that it's not confused with this other value in scope that has the same type". After that, I cull down that list until it's reasonable and not totally paranoid, and write tests covering those cases. Same thing for any bugs in in-progress code caught by manual testing, fuzzers, etc.

If you have discipline and use version control, you don't need to write tests before you write the actual code to get the same level of coverage as TDD and you waste a lot less time. I've often figured out late in the game how to make something a compile time failure rather than a runtime one - time to delete all those tests written along the way? Encode them all as negative compilation tests? Fundamentally the goal of testing is to describe what behaviors of the software are intentional rather than incidental, and to detect bugs that might be introduced by future changes to the software - TDD mixes both concerns and doesn't put any emphasis on preventing future bugs specifically.

Maybe other people work on different types of things and TDD is great for them, but I write primarily infrastructure code where correctness is critical and I have the luxury of time, and TDD doesn't produce better results for me. This is a case TDD feels like it should work well for, but in my experience it doesn't improve correctness, maintainability, or speed of delivery - at least compared to the alternative I described. I'm sure there's a universe of teams with sloppy practices out there that TDD would be an improvement for, but it's not helpful for me.

charleslmunger··on The Big TDD Misunderstanding (2022)
This is hilarious because it's a perfect interview (detects precisely the thing they're testing for) but also provides total adverse selection because the thing they're testing for is ridiculous.
charleslmunger··on I Wrote a WebAssembly VM in C
Metering also doesn't require a branch if you implement it with page faults. See "Implicit suspend checks" in https://android-developers.googleblog.com/2023/11/the-secret...
charleslmunger··on Scrolling like it's 2008: SwiftUI and Jetpack Compose
>The fact that these vendor-supported toolkits – SwiftUI and Compose – are built atop the platform-native scrolling mechanics

I can't speak for iOS, but neither RecyclerView nor Compose's scrolling is built on top of the Android platform's scrolling - they both implemented new and different ways to do scrolling versus the ScrollView or ListView the platform shipped with. Both were built using the same public APIs any developer could use.

The reason they offer the performance they do is implementation choices by the developers who wrote them, around what work to perform when. Lots of Android developers implemented scrolling using the platform APIs and got jank, others implemented scrolling before RecyclerView was even available and had smooth performance. There is no magic performance guarantee because you transpile to kotlin.

charleslmunger··on Case Study: ByteDance Uses eBPF to Enhance Networking Performance
>When you do that, the kernel network stack, at like the broad logic level, is sort of oblivious to the fact that the devices aren't real ethernet devices and don't have to go through the ethernet motions to transact.

Is that true even for virtio-net? I guess I just assumed all these virtual devices worked like virtiofs and had low overhead fast paths for host and guest communication.

charleslmunger··on C is not suited to SIMD (2019)
>Then you have to malloc the struct to be the sizeof the struct + whatever size you want the array to be.

Ultra pedantic, but you actually do not need to alloc struct+size; you want something like:

    // A flexible array member may have lower alignment requirements than the struct
    // overall - in that case, it can overlap with the trailing padding of the rest
    // of the struct, and a naive sizeof(base) + sizeof(flex) * count calculation
    // will not take into account that overlap, and allocate more than is required.
    #define SIZEOF_FLEX(type, member, count) MAX(sizeof(type), offsetof(type, member[count]))
I'm sure there's a more elegant way to calculate this, but I am no macro master.
charleslmunger··on Ask HN: How do you guard against ChatGPT use in technical interviews?
All the questions I come up with that are both narrow enough in scope to reasonably do in an interview and get unhelpful answers from an LLM end up being trick questions for humans too, which is not really fair or productive.

What's particularly frustrating is that I've got some actual unsolved algorithms problems I want a solution to. One might think that since the LLMs are smart enough to re-solve problems from their training data in interviews, they could help - alas, they have been of no help so far.

charleslmunger··on Branchless UTF-8 Encoding
>Beyond that, Intel recently updated their manual to retroactively define the behavior of BSR/BSF on zero inputs: it leaves the destination register unmodified.

This is very nice of them to do, but I found while optimizing a routine in protobuf that BSR is dramatically slower on AMD CPUs than LZCNT, and so I never want to use it again - it's pretty rare to have a function using BSR that can't use CLZ instead, and CLZ is faster on arm, AMD, and equivalent on Intel since haswell.

I believe there is also some errata where on some processors Intel LZCNT had a false dependency for the output register as an input, probably because of this BSR behavior, but compilers will insert a self-xor in loops where that carried dependency would matter.

charleslmunger··on Five years of React Native at Shopify
If your warm and cold starts are that close I'take a closer look at your measurement methodology. Could be your cold starts are actually warm, or your timing is starting late.
charleslmunger··on Five years of React Native at Shopify
>RAM access on modern hardware has a latency of something like 10 nanoseconds

What modern hardware are you using that this is true? That's faster than L3 cache on many processors.

charleslmunger··on Ask HN: How do you guard against ChatGPT use in technical interviews?
Sieve of Erastothenes is often introduced alongside prime numbers in elementary school math class and it has a funny/memorable name - it's not that weird for someone to know it. It is weird to lie about having a practical use case for it since running it to the point of a cryptographically useful prime length is infeasible and it requires O(n) memory.

It seems vanishingly unlikely that this type of question can provide any signal any more outside an in person interview. The incentives are just too strong for candidates and the tools are too good.

charleslmunger··on Uncut Currency
https://www.washingtonpost.com/wp-dyn/content/article/2008/0...
charleslmunger··on The Evolution of SRE at Google
> 1) tests take too long to run 2) difficult to root cause breakages.

I have gotten a ton of value out of targetting these problems specifically. Some low-cost-high-value test infra changes:

1. Passing tests are not allowed to emit error level logs without expecting them; if the expected error does not occur, the test fails. This is good in general, but also makes root causing test failures easier as if you encounter an unexpected error log you can attach it to the failure message, and often that pinpoints the root cause.

2. Have lots of off-by-default instrumentation. For example, if a test fails auto-rerun it with sanitizers on.

3. In tests, gather backtraces when moving tasks between threads. This makes identifying what triggered an error straightforward.

4. Collect data during test runs, like line coverage, in a sequential order that can be diffed between passing and failing runs.

As for making tests fast, the solution is aggressive optimization just like prod code - profile the test, and fix what makes it slow. Cut corners where it makes sense - should you ever fsync in a test? VM snapshotting can be very useful in this area for things that are slow to start up.

charleslmunger··on Go Protobuf: The New Opaque API
You'll be thrilled to hear about upb then, which was designed to be embeddable to power other languages without a from-scratch implementation - and now powers python protos.

https://github.com/protocolbuffers/protobuf/tree/main/upb

charleslmunger··on Home Assistant can not be secured for internet access
Home Assistant's Android app does work with client certificates.

https://companion.home-assistant.io/docs/getting_started/#tl...

charleslmunger··on Ask HN: What are the best programmable holiday lights?
If you're interested in DIY:

LED lights as a string or strip - sk8612 are RGBW, which you need if you want a nice looking white. Lots of strips of varying density and waterproofing are available everywhere; strings ("pixels") can be harder to find but are available on AliExpress.

Controller is https://quinled.info/

Runs wled which has a simple web interface, mobile apps, and works with home assistant.

charleslmunger··on The two factions of C++
>As an old-timer, I think you have some serious misconception about how RAII works, and what it does for you.

I appreciate the education :-)

>There's nothing that stops you from using arena allocators in C++.

This is true, but arenas have two wonderful properties - if your only disposable resource is memory, you don't need to implement disposal at all; and you can free large numbers of individual allocations in constant time for immediate reuse. RAII doesn't help for either of these cases, right?

>Use of destructors leads to use of destructors

I guess what I mean is... It's totally possible and common to have a zillion copies of std::vector in your binary, even though the basic functionality for trivially copyable trivially destructible types is identical and could be serviced from the same implementation, parameterized only on size and alignment. Destruction could be handled with a function pointer. But part of the reason templates are used so heavily seems to be that there's an expectation that libraries should handle types with destructors as the common case.

>lifetime management is a huge problem in C. Not at all trivial to clean things up when you're done.

Absolutely true if you're linking pairs of many malloc and free calls. But if you have a model where a per-frame or per-request or per-operation arena is used for all allocations with the same lifetime, you don't have this problem.

>And C's behavior would also be "UB" if you read after freeing a pointer.

The specific issue I ran into was the destructor of one thread_local reading the value of another thread_local. In C++17 the way to do this was to make one of them trivially destructible, as storage for thread locals is released after all code on that thread has finished, and the lifetime for trivially destructible types ends when storage is freed. In C++20 this was changed, such that the lifetime of a thread local ends when destroyed (rather than when storage is freed) if it's trivially destructible. C thread local lifetimes are tied to storage only and don't have this problem.

charleslmunger··on The two factions of C++
As a relative newcomer to C++, I have found RAII to be fine for writing in object-oriented style. But I would only subject myself to the suffering and complexity of C++ if I really wanted excellent performance, and while RAII does not generally have a runtime cost by itself, engineering for full performance tends to exclude the things that RAII makes easy. If you manage memory via arenas, you want to make your types trivially destructible. If you don't use exceptions, then RAII is not necessary to ensure cleanup. In addition, use of destructors tends towards template APIs that generate lots of duplicate code, when an API that used a C-style function pointer for generic disposal would have produced much smaller code.

And C++'s object model can add additional complexity and sources of UB. In C++20 previously valid code that reads a trivially destructible thread_local after it has been destroyed became UB, even though nothing has actually happened to the backing storage yet.

charleslmunger··on Using gRPC for (local) inter-process communication (2021)
You can run grpc over binder:

https://github.com/grpc/grpc-java/blob/master/binder/src/mai...

The overhead is low, and you get best practices like oneway calls and avoiding the transaction limit for free. It also comes with built in security policies for servers and clients.

charleslmunger··on Learn Rust the Dangerous Way
Clang emits a warning if you ask it to vectorize a loop and it can't; you can then promote that to an error. But even if the loop vectorizes, that doesn't mean that it vectorizes in the optimal way - my understanding is that pros write their loops with intrinsics specifying the operations they want for this reason.

https://godbolt.org/#z:OYLghAFBqd5QCxAYwPYBMCmBRdBLAF1QCcAaP...

charleslmunger··on There Is No Technology Fix for Car Bloat
From the article:

>A recent analysis by the Economist found that among the heaviest one percent of American cars, 12 people die inside smaller models for each person saved by the enormity of their vehicle.

What benefit exactly are hundreds of millions of people getting from having tall-grille cars? Many of these tall fronts are fake, they're taller than what is needed to fit the engine - purely an aesthetic choice, to give the look of a large and powerful machine.

charleslmunger··on The profit-obsessed monster destroying American emergency rooms
>Yet people still start businesses, make money, grumble endlessly about those regulations, and society marches on.

One such regulation is "certificate of need" laws, where you effectively need to ask the incumbent for permission to open your would-be competitor.

https://spn.org/articles/certificate-of-need-laws/

charleslmunger··on The sorry state of Java deserialization
Well, if you aren't constrained in the format, write it out as fixed width integers as small as you can, mmap the file, madvise as MADV_SEQUENTIAL, and scan with a branch free loop the compiler will vectorize. You'll saturate your read bandwidth for almost any speed.

Another trick I've seen is to set up a ring buffer by mapping the same physical page on either side of another, which allows you to always read a contiguous region even if it wraps around the end of your buffer. This would let you keep a traditional read loop without having to shift bytes.

In Java, these tools aren't all available. Java protobuf uses an object oriented style that comes with performance costs. Making the message object immutable avoids bugs and makes threading easier. If you like the object oriented style, you can have it, but you'll pay in allocation and cache misses if your objects are all tiny.

The protobuf wire format is optimized for flexibility - if you're storing only two ints per message, only 1/3 of the parsing you do is your actual content. You pay for the tag number, the message length delimiter, and then the tag numbers for each of your fields - four varints of framing for two varints of content. This lets you add and remove fields of any type in the message safely, but if you're optimizing for pure speed of many tiny snippets of data, you are paying for flexibility you may not need.

But if you like protobuf you should be able to get respectable performance in Java by making a single CodedInputStream with a large buffer (16kb at least) and using push limit/poplimit yourself to do parseDelimitedFrom repeatedly without making new stream objects or buffer wrappers every time. At that point I'd expect your bottleneck to be allocating and eventually GCing the message objects, but maybe escape analysis has gotten good enough for those to be stack allocated nowadays.

charleslmunger··on The sorry state of Java deserialization
I noticed:

1. Small read buffers. No reason to sequentially read and parse gigabytes only 4kb at a time.

2. parseDelimitedFrom created a new CodedInputStream on every message, which has its own internal buffer; that's why you don't see a buffered stream wrapper in the examples. Every iteration of the loop is allocating fresh 4kb byte[]s.

3. The nio protobuf code creates wrappers for the allocated ByteBuffer on every iteration of the loop.

But the real sin with the protobuf code is serializing the same city names over and over, reading parsing and hashing. Making a header with the city string mapped to an integer would dramatically shrink the file and speed up parsing. If that was done, your cost would essentially be the cost of decoding varints.

charleslmunger··on Valkey achieved one million RPS 6 months after forking from Redis
Linux perf would do this if you profile for "backend stalls".
← PreviousPage 4 of 16Next →