HNHacker News
TopNewBestAskShowJobs

twotwotwo

1,841 karma · joined June 24, 2012

submissionscomments
twotwotwo··on Databases and why their complexity is now unnecessary
A very simple thing about this (and many systems!) is that if your whole thing is "log writes and do the real work later", you lose read-your-writes and with it the idea that your app has a big persistent memory to play in.

This doesn't only matter if you're doing balance transfers or such; "user does a thing and sees the effects in a response" is a common wish. (Of course, if you're saving data for analytics or such and really don't care, that's fine too.)

When people use eventually-consistent systems in domains where they have to layer on hacks to hide some of the inconsistency, it's often because that's the best path they had out of a scaling pickle, not because that's the easiest way to build an app more generally.

I guess the other big thing is, if you're going to add asynchrony, it's not obvious this is where you want to add it. If you think of ETLs, event buses, and queues as tools, there are a lot more ways to deploy them--different units of work than just rows, different backends, different amounts of asynchrony for different things (including none), etc. Why lock yourself down when you might be able to assemble something better knowing the specifics of your situation?

This company's thing is riding the attention they get by making goofy claims, so I'm a bit sorry to add to that. I do wonder what happens once they're talking to actual or potential customers, where you can't bluff indefinitely.

twotwotwo··on Sigil II, a Doom WAD from J.Romero, has been released
Not inconsistent with this, what seems warped to me is having these huge expectations for individuals after their first success in the first place. (Writing this thinking about someone like Romero working on something like a game; lot of what follows doesn't carry over to, say, music or writing.)

They needed talent to pull it off, but also lots of stars aligned for a bold experiment to work out, and true experiments don't always work out. Conversely, for each success there were many talented folks who tried good ideas and didn't see them work out. The more surprising the first success was, the more foolish you'd be to expect a repeat, it seems like.

If that's the basic problem, you can't "fix" it in the sense that you can't make someone keep churning out revolutionary new things. But you might get better outcomes and fewer embarrassing failures the more you think of individual talent as one factor. Folks still have to iterate, get feedback, work with good people, sometimes put a lot of work into an idea then see it they have to rework it, or even release some quick experiments that don't do that well. You don't maximize your odds of great results by sending someone off to work in isolation as long as they want.

One thing I see hints of here and there, but don't really have a great picture of, is the inner workings of organizations that produce new/weird but successful things more than just once. It interacts with business, luck, and changes in the world outside of course, but internally--how do they balance single vision vs. input from many folks? Do they build a lot they throw away? Do they go outside the team a lot for feedback (in Romero's world "playtesting")? Of course there's no one Right Way to do things, especially across domains, but it could be fascinating to see a bit more how various places work.

twotwotwo··on Gobs of data (2011)
Yep, I very much agree with this. It's probably inevitable that languages with reflection grow some kind of language-specific serialization format because a) they can and b) there's often some use case where it looks handy to have a format adapted to the quirks of the language. Plus, when some of these bespoke formats were created, the world hadn't converged as much as it has now around a few common text and binary formats.

But now the interoperable serialization formats have a lot more energy spent on tools and such, and the formats are better-specified, to the point that you probably want to use them even where interoperability with other stuff doesn't force it.

twotwotwo··on Amazon Aurora Limitless Database
I'm glad someone from AWS is here! Some general info on the sharding approach would be interesting, e.g. whether secondary indices on sharded tables are scatter-gather, or separately sharded on the index columns, or you configure it per index, or what.

From what I hear from folks using things like Vitess, if you're used to a monolithic SQL database there's often some things to learn to mentally model your query cost well after you move to a sharded world, and understanding more up front can save heartburn later. Writing up those details well is a good thing that AWS could do.

twotwotwo··on Need a PRNG? Use a CSPRNG
I am pretty sympathetic to a CSPRNG being the default in standard libraries and so on. The relative performance has vastly changed since the first stdlib random()s; modern hardware- or SIMD-accelerated functions run at GBs/second per core, and CSPRNGs are flawless PRNGs--no fussing about whether PCG's subtle patterns matter--but the reverse is definitely not true.

Making the default secure would be a pretty significant mitigation against something that commonly makes "common security bugs" lists like https://developer.android.com/privacy-and-security/risks/wea...

If you consume many GBs/s/core of randomness, or you're writing a homebrew NES game, sure, you have a special case, code up a non-CSPRNG. But that isn't a reason for the default for everyone to remain a function that needs a warning label.

twotwotwo··on There's Math.random(), and then there's Math.random() (2015)
Sorry, I know you should be using a CSPRNG API for cryptography, and probably my glib phrasing there obscured that.

I'm saying the cost of running actual cryptographic primitives has dropped a ton over time: on computers from decades ago a cheaper flavor of pseudorandomness was clearly necessary, now hardware AES is very cheap. And webpages aren't typically massive doing HPC simulations or other things that will be bound by the PRNG taking a few cycles per byte.

So the memory/CPU benefit of keeping the bad PRNG around is not obviously still worth it to me. In your words, I think I disagree with their priorities, particularly because the cost savings are not what they used to be.

twotwotwo··on There's Math.random(), and then there's Math.random() (2015)
Sorry, to clarify, I know CSPRNG APIs are exposed and what you use for safe randomness.

What I'm saying is that outside rare, extreme situations--maybe some HPC Monte Carlo simulation where you need gobs of randomness--you might as well use strong primitives for all your randomness, because they work well and are now plenty cheap for the use case with hardware support, etc.

The games to find a fast non-cryptographic function that passes enough statistical tests, etc., don't make as much sense to me when cryptographic randomness costs a few cycles a byte. There is not much upside to every standard library exposing a bad random generator that we have to warn people not to use. The handful of people that really need xorshift128+ can figure something out.

Maybe still a position you disagree with, but at least clearer without the sarcastic gloss.

twotwotwo··on There's Math.random(), and then there's Math.random() (2015)
It's fun it's exactly 1:1 and you don't need any more output than the state!

It's sort of sad it isn't all AES-CTR or ChaCha12. Fast enough not to be the bottleneck in your webpage, and if you find any hint of a pattern in the output, good news, you get to publish a paper on it!

twotwotwo··on Bedazzled by Energy Efficiency (2018)
So a nasty form of this, referenced in the post but worth calling out, is that efficiency in one part of the economy can be effectively turned into higher usage elsewhere.

If your energy production is cheap enough in terms of variable cost, the producer might decide it's worth producing about the same amount at a lower price. If there are other energy users with enough price elasticity, the economy finds a new price where they pick up the slack.

So that idea isn't that the individual consumer drives more because they get better MPG, it's that individuals' demand for gas overall decreases, so industry, guided by price signals, uses more gas for cheaper air tickets or faster package delivery or something.

You've made the producer a bit less profitable, truly cut consumption in one sector, and provided some non-enviro benefits, but you haven't reduced overall consumption or environmental impact.

There are a few things that keep this from leading to total hopelessness:

- Elasticity isn't infinite and it doesn't have to always fully eat your efficiencies. People wouldn't take infinite trips if gas were free, and I suspect a huge change like a big country moving to electric cars really does have a net-beneficial effect on carbon emissions. You're just not guaranteed 1-for-1 efficiency-to-carbon-reduction.

- Lower demand could reduce production in the long term, as capital allocations get made, more than it does the short term. That is, even if utilities decide to keep existing coal plants going full blast when prices are down because variable costs are low, maybe they'll have reduced appetite to build new ones where they have to factor in fixed+variable costs.

- Not everything we're doing is energy efficiency. Gradually replacing our higher-carbon energy generation with lower-carbon methods reduces impact a different way from reducing consumption.

But it does make me think a carbon tax, ideally slowly phased in starting a long time ago, was maybe a better approach than it might look like: directly put economic decision-makers to work finding ways to cut the thing you don't want, instead of poking a complex system and hoping for the desired effect. Besides, charging for negative externalities is a better way to raise funds than charging for working or value added anyhow. It's just so hard politically.

There is also a place for non-economic policy here, e.g. if you want more everyday travel to use efficient mass transport rather than individuals' vehicles, governments just have to make the first move and put money into making the alternative systems complete, functional, and desirable to use.

twotwotwo··on A student’s desire to get out of a exam led to a compression algorithm
It looks like the ones I were PC Magazine, https://books.google.com/books?id=gCfzPMoPJWgC&lpg=PP1&pg=PA... and https://books.google.com/books?id=eX8w8B-OhIIC&lpg=PA371&pg=... .

Much appreciation to all the folks who put in the work to make these things interesting and accessible.

twotwotwo··on A student’s desire to get out of a exam led to a compression algorithm
It looks like it was Barry Simon in "Lab Notes" in PC Magazine of June and July 1993:

https://books.google.com/books?id=gCfzPMoPJWgC&lpg=PP1&pg=PA...

https://books.google.com/books?id=eX8w8B-OhIIC&lpg=PA371&pg=...

twotwotwo··on A student’s desire to get out of a exam led to a compression algorithm
There was a series of articles on lossless and lossy and compression techniques in, I think, PC Magazine that I read as a kid and made a big impression on me. I didn't, like, end up ready to write code to build Huffman trees or anything, but it did change it from a total black box into a bunch of smaller pieces each of which a mere mortal can understand.

The compression rabbit hole can be a rewarding one to go down if you haven't. Besides the tech itself making a lot of things work better (like a Web built on human-readable formats, or more specialized stuff like Parquet or other columnar formats, or video on the lossy side), it can give you some tools or perspective that apply elsewhere, e.g. to other probablistic stuff like caching and predictions for lossless compression, or other signal-processing stuff for lossy compression.

twotwotwo··on macOS 14 will support JPEG XL
Those words were about the JPEG recompressor (Brunsli) specifically, which saves about 22% but has a smoother transition from today's .jpg world than any of the new lossy formats; you can bit-for-bit reconstruct the original JPEG from the Brunsli-compressed one, and it's certainly much faster than AVIF to encode with CPUs. You can even insert Brunsli as a Content-Encoding that's transparent to the user (so right click still saves .jpg), and Chrome had an issue open to consider doing that before they backed out JXL support.
twotwotwo··on macOS 14 will support JPEG XL
Yeah, that is all odd to me; Google cited lack of "interest from the entire ecosystem" (https://bugs.chromium.org/p/chromium/issues/detail?id=117805...). Mozilla also had an experimental JXL implemenation out there. I'd hope, like you said, that the browser vendors had enough of a backchannel to keep a good thing from fizzling because each is uncertain if the others will implement it.

But if the Chrome folks did know they had all three major browser engines were ready to go, along with the positive noises from Facebook, Adobe, other parts of Google, and a few other notable folks, that seems like a solid base of support, unless you're really expecting the world to go all-in on a format before a browser supports it without a flag. So I guess I wonder if it's that the Chrome team had set the bar high, that they hadn't expected Apple to support JXL at the time they made the call, or something else.

twotwotwo··on macOS 14 will support JPEG XL
I hope it shows up in Safari and that this sign of interest from others gets it a second shot in Chrome.

Just the JPEG recompression is arguably worth the price of entry; it's less savings than AVIF, of course, but it's easier to adopt now, where re-encoding everything as AVIF is a much larger hump to get over. Its other modes provide some real benefits for users for very-high-quality and lossless images as well as any situation where progressive display helps. For folks producing images, is much easier to fit in than AVIF for anyone who needs to encode a lot on the CPU.

twotwotwo··on Pigz: Parallel gzip for modern multi-processor, multi-core machines
A part of this was that zstd and Brotli were able to use compression history windows of MBs not KBs, while DEFLATE maxes out at 32KB. RAM was thousands of times more expensive in the early 90s, so a smaller history window made sense.

There are also optimizations that only work on today's larger cores, and you have to actively update old code to get the advantages (happily some work is going into that): https://news.ycombinator.com/item?id=32533061 / https://news.ycombinator.com/item?id=32537545

That's not to minimize the clever ideas and amazing implementation work in new stuff. It's more that people were making smart decisions both then and now, more so than you might guess just from comparisons on today's hardware.

twotwotwo··on Man beats machine at Go in human victory over AI
Could play against adversarial policies become part of the training alongside self-play?

It's possible that would be a very slow, essentially "serial" process--adversaries don't really find a new trick until the AI can avoid the old, like with the pass attack in this paper. I'd also be curious whether training against adversaries makes the networks better or worse at general-purpose play or causes no change, but there's an argument that more-robust-but-slightly-worse-on-average is an overall improvement.

It also seems relevant to looking at weird behaviors of other models, like what we're seeing as people poke at LLMs more. Some of what we're seeing might simply be that they respond weirdly to some fraction of all prompts because they're not quite baked; also, we haven't defined their goals all that well yet. But automatic searches for problematic classes of input seem like a possible tool for keeping things more on the rails.

twotwotwo··on Improving Rust compile times to enable adoption of memory safety
There are hardware reasons even if you leave any software scaling inefficiency to the side. For tasks that can use lots of threads, modern hardware trades off per-thread performance for getting more overall throughput from a given amount of silicon.

When you max out parallelism, you're using 1) hardware threads which "split" a physical core and (ideally) each run at a bit more than half the CPU's single-thread speed, and 2) the small "efficiency" cores on newer Intel and Apple chips. Also, single-threaded runs can feed a ton of watts to the one active core since it doesn't have to share much power/cooling budget with the others, letting it run at a higher clock rate.

All these tricks improve the throughput, or you wouldn't see that wall-time reduction and chipmakers wouldn't want to ship them, but they do increase how long it takes each thread to get a unit of work done in a very multithreaded context, which contributes to the total CPU time being higher than it is in a single-threaded run.

twotwotwo··on Improving Rust compile times to enable adoption of memory safety
This also relates to something not directly about rustc: many-core CPUs are much easier to get than five years ago, so a CPU-hungry compiler needn't be such a drag if its big jobs can use all your cores.
twotwotwo··on Show HN: ClickHouse-local – a small tool for serverless data analytics
This inspired me to dig around a little more, and the details at https://clickhouse.com/docs/en/operations/settings/settings#... leave me more confident you can do large joins if your flow requires them. It still looks like CH's default setting is extra work if you're used to writing queries and letting the query planner worry about join strategies.
twotwotwo··on Show HN: ClickHouse-local – a small tool for serverless data analytics
Huh! I recently used this to play with CH, and had no idea it was so new! Also, I've been impressed with the range of formats/protocols/types of storage CH can use, and it's a good strategy by the developers because it makes it much more appealing to consider working with.

For what it's worth, my main uncertainty about CH at this stage is about just how we can/can't use joins while keeping memory use reasonable. (What I've gathered from playing around is CH may fail a query because it wants to create a hashtable that's too big for RAM, when other tools would spill temporary data to disk.) We can minimize joins by denormalizing, but can't completely get rid of them, and notably joins are really handy for the early steps of building "wide" denormalized tables from normalized source data.

I know CH has run the Star Schema Benchmark so it must support some join strategies; still, a lot of the examples focus on the single-table queries CH is best at, and so far I have less of an idea what join strategies CH can use and what the limitations are so I can work within them.

twotwotwo··on Chrome Responds "No" to JPEG XL
A JXL contributor says full JXL (not just the recompression) can fit in 174kB, and thinks a smaller version could fit in 50: https://news.ycombinator.com/item?id=33414678

If you have large JPEGs it's not hard to earn the bandwidth cost back. I am definitely curious about other aspects of WASM performance including speed on first run.

twotwotwo··on Chrome Responds "No" to JPEG XL
They have not closed a narrower bug about the lossless JPEG compression: https://bugs.chromium.org/p/chromium/issues/detail?id=110969...

I think it's a legitimately different tradeoff because the recompression is not like a new lossy format--less code, plus you don't need the whole ecosystem to move, big sites and CDNs can transparently start using it--but still substantial (22% on JPEG content out there). And by being very CPU-light it fills a fairly large gap where people are not ready to use AVIF, which needs either dedicated hardware or tons of CPU to encode.

One thing this is right about is a WASM polyfill is interesting. With that and a ServiceWorker, you can make the recompression pretty transparent (right-click saves a .jpg) but still save your 22%. The performance penalty of doing it in WASM is a question, but it's definitely possible.

twotwotwo··on It's not Ruby that's slow, it's your database
This is kind of relevant now: Mastodon is seeing a lot of growth all the sudden, the popular server for it is in Rails, and scaling up quickly is a challenge for server operators used to the levels of traffic from two months ago.

Twitter did rewrite their frontend to JVM languages because it was enough of a bottleneck to be worth it. If you can get Mastodon substantially further by serving some views using something besides Rails, or even going around ActiveRecord in places, that is a pretty different short-term to-do list than if the immediate problems are in the data layer. (Long term you have to scale both of course!)

twotwotwo··on Motto: Unsafe languages allow mostly-safe implementations
Title change? The linked content is fine but the submitted title ("Motto: Unsafe languages allow mostly-safe implementations") doesn't appear anywhere in it.
twotwotwo··on Removing the JPEG XL code and flag from Chromium
I think the decision on JPEG XL as a whole is made; there were some statements in favor of adopting it on Chrome's and Mozilla's trackers before the decision from folks at Adobe, Facebook, Intel, and a newspaper, and some developers of it and natural advocates of it were inside Google; general-public comments seem less impactful than all that. If I had to guess, the trigger was Apple supporting AVIF and Chrome folks concluding AVIF "won." They might also have decided they think Apple is unlikely to ever do JXL, but they probably wouldn't comment publicly on what they think someone else will do.

The case for a Brunsli filter in the meantime is legacy JPEG content is never going away, and there will also be a long "meantime" where we know AVIF is the endpoint but it's still impractical for many because of SW encoding cost. Brunsli saves ~20%, so it's not nothing, and it's CPU-cheap.

A WASM polyfill would kind of be cool. Someone did it for Bellard's "BPG" (HEIC in a different container) in a ServiceWorker, and one just doing JPEG1 transcoding would be even lighter (and, unlike this PNG-making polyfill, would save reasonably small JPG files if the user right-clicks): https://sequentialread.com/better-portable-graphics-bpg-on-t...

twotwotwo··on Removing the JPEG XL code and flag from Chromium
Someone implemented a BPG decoder as a WASM ServiceWorker returning PNG files, an approach that could work even better for Brunsli: https://sequentialread.com/better-portable-graphics-bpg-on-t...
twotwotwo··on Removing the JPEG XL code and flag from Chromium
Apart from numerical or visual comparisons, it does things other formats don't: https://news.ycombinator.com/item?id=27581195

The lossless recompression of existing .jpg content and faster software encoding for its native VarDCT format seem to provide immediate benefits in contexts where AVIF doesn't fit well now, either because AVIF encoding is slow or because AVIF would require a lossy reencoding of a lossily compressed source.

twotwotwo··on Removing the JPEG XL code and flag from Chromium
It's technically possible that, without all of JXL, they could implement the lossless JPEG1 recompressor part (Brunsli) as a Content-Encoding like gzip, which could save about 20% on all the JPEG content out there with no additional loss: https://bugs.chromium.org/p/chromium/issues/detail?id=110969...

Compared to all of JPEG XL, Brunsli is less code, and the adoption story is more compelling: you need CDNs/infra to support recompression, but don't need the entire ecosystem to change formats. Content-Encodings can be dropped if there isn't uptake--there's precedent with SDCH--so you could roll it out without a flag once it's well tested and give sites a reason to add support, avoiding a chicken and egg problem.

I don't think AVIF makes it irrelevant because 1) JPEG-to-Brunsli is a far smaller step than JPEG-to-AVIF, since Brunsli is far faster to encode than AVIF (especially in software) and adds no loss beyond the initial JPEG encode, and 2) even if AVIF and HEIC are widely adopted, there's a mind-boggling quantity of JPEGs (exabytes?) that won't go away. Dropping JXL as an image format makes the Content-Encoding more relevant, since the lossless recompression it provides is no longer going to be available by other means.

A star expressing interest in 1109698 couldn't hurt the chances (though I suspect they're slim). They haven't closed the bug, so I don't think they've officially said they'll never do it, though the current language does assume third_party/jpegxl exists.

twotwotwo··on U.S. workers have gotten less productive – no one is sure why
So, the economic value measured for a piece of code drops if fewer people buy the product that it's in or its price has to be cut, entirely separate from how fast anyone is coding. Similar logic applies to a lot of sectors, not just software. Macroeconomic changes are going to work their way back to these stats eventually, independent of any changes in how workers spend their day or the concrete stuff they produce. (Kind of like stormbrew and nostrademons said!)
← PreviousPage 4 of 19Next →