zigzag312

> The reason is that exploits in the image/font/docs parsing/rendering libraries can happen and are exploited in the wild.

Aren't risks similar when opening any untrusted web page in a browser?

The only difference is that browser sandbox and exploit mitigations are probably better than that of a PDF viewer.

This seems to make sense, but I need to read more about error correction to fully understand it. I was considering possibility that data could also contain patterns where error detection performs poorly due to bias, and I haven't seen how to include these estimates in probability calculations.

I can't say I agree with your logic here. We are not talking about any specific backup or anything like that. We are talking about the design of a file format that is going to be used globally.

Business running a lottery has to calculate the odds of anyone winning, not just the odds of a single person winning. Same, a designer of a file format has to consider chances for all users. What percent of users will be affected by any design decision.

For example, what if you would offer a guarantee that 32 bit hash will protect you from corruption, and compensate generously anyone who would get this type of corruption; how would you calculate probability then?

Then you just need a bigger hash.

> Uniformity isn’t directly important for error detection.

Is there any proof of this? I'm interested in reading more about it.

> detect all burst errors up to 32 bits in size

What if errors are not consecutive bits?

That's a good example. Thanks! It would be kind of an indirect tampering method.

It depends on how you calculate statistics. If you are designing a file format that over the lifetime of the format hundreds of millions of user will use (storing billions of files), what are the chances that 32 bits checksum won't be able to catch at least one corruption? During transfer over unstable wireless internet connection, storage on cheap flash drive, poor HDD with a higher error rate, unstable RAM etc. We want to avoid data corruption if we can even in less then ideal conditions. Cost of going from 32 bit to 64 bit hashes is very small.

> which can be done fast enough to appear instant on the CPU

Big scanned PDFs can be problfrom more efficient processing (if it had HW support for such technique)

> Your link shows CRC32 at 7963.20 MiB/s (~7.77 GiB/s) which indicates it's either very old or isn't measuring pure CRC32 throughput

It may not be fastest implementation of CRC32, but it's also done on old Ryzen 5 3350G 3.6GHz. Below the table are results done on different HW. On Intel i7-6820HQ CRC32 achieves 27.6 GB/s.

> measures 85 GB/s (GB, GiB, eh close enough) on the Apple M1. That's fast enough that I'm comfortable calling it limited by memory bandwidth on real-world systems.

That looks incredibly suspicious since Apple M1 has maximum memory bandwidth of 68.25 GB/s [1].

> I have personally seen bitrot and network transmission errors that were not caught by xxhash-type hash functions, but were caught by higher-level checksums. The performance properties of hash functions used for hash table keys make those same functions less appropriate for archival.

Your argument is meaningless without more details. xxhash supports 128 bits, which I doubt wouldn't be able to catch an error in you case.

SHA256 is an order of magnitude or more slower than non-cryptographic hashes. In my experience archival process usually has big enough effect on performance to care about it.

I'm beginning to suspect your primary reason for disliking xxhash is because it's not de facto standard like CRC or SHA. I agree that this is a big one, but you constantly imply like there's more to why xxhash is bad. Maybe my knowledge is lacking, care to explain? Why wouldn't 128 bit xxhash be more than enough for checksums of files. AFAIK the only thing it doesn't do is protect you against tampering.

> I don't know what kopia is, but according to your link it looks like their wire protocol involves each client downloading a complete index of the repository content, including a CAS identifier for every file. The semantics would be something like Git? Their list of supported algorithms looks reasonable (blake, sha2, sha3) so I wouldn't have the same concerns as I would if they were using xxhash or cityhash.

Kopia uses hashes for block level deduplication. What would be an issue, if they used 128 bit xxhash instead of 128 bit cryptographic hash like they do now (if we assume we don't need to protection from tampering)?

[1] https://en.wikipedia.org/wiki/Apple_M1

From SMHasher test results quality of xxhash seems higher. It has less bias / higher uniformity that CRC.

What bothers me with probability calculations, is that they always assume perfect uniformity. I've never seen any estimates how bias affects collision probability and how to modify the probability formula to account for non-perfect uniformity of a hash function.

I've seen few arguments that with the amount of data we have today the 2^(32-1) chance can happen, but I can't vouch their calculations were done correctly.

Readme in SMHasher test suite also seems to indicate that 32 bits might be too few for file checksums:

"Hash functions for symbol tables or hash tables typically use 32 bit hashes, for databases, file systems and file checksums typically 64 or 128bit, for crypto now starting with 256 bit."

It seems that at least JPEG can be decoded on the GPU [1] [2]

[1] https://docs.nvidia.com/cuda/nvjpeg/index.html

[2] https://github.com/CESNET/GPUJPEG

> the pages within a CBZ are going to be encoded (JPEG/PNG/etc) rather than just being bitmaps. They need to be decoded first, the GPU isn't going to let you create a texture directly from JPEG data.

It seems that JPEG can be decoded on the GPU [1] [2]

> CRC32 is limited by memory bandwidth if you're using a normal (i.e. SIMD) implementation.

According to smhasher tests [3] CRC32 is not limited by memory bandwidth. Even if we multiply CRC32 scores x4 (to estimate 512 bit wide SIMD from 128 bit wide results), we still don't get close to memory bandwidth.

The 32 bit hash of CRC32 is too low for file checksums. xxhash is definitely an improvement over CRC32.

> to actually check the integrity of archived files you want to use something like sha256, not CRC32 or xxhash

Why would you need to use a cryptographic hash function to check integrity of archived files? Quality a non-cryptographic hash function will detect corruptions due to things like bit-rot, bad RAM, etc. just the same.

And why is 256 bits needed here? Kopia developers, for example, think 128 bit hashes are big enough for backup archives [4].

[1] https://docs.nvidia.com/cuda/nvjpeg/index.html

[2] https://github.com/CESNET/GPUJPEG

[3] https://github.com/rurban/smhasher

[4] https://github.com/kopia/kopia/issues/692

Why would it need to be cryptographically secure for this use case?

If we want to have any meaningful debate on what effect something has, we need to try to separate what are the reactions/side effects caused by it and to try to avoid additional confounding changes that were not caused by it. Trying to undo the negative side effect you don't like by, for example, introducing new technological innovations, or in your case a new optimization even though there is no increase in anything that would help do develop this new optimization, is making a confounding change.

While speculating what might happen in some particular cases can be fun, the real questions that need to be answered are regarding the macroeconomic effects of UBI. Starting with what % of GDP will be needed for UBI that will cover all the basic needs (food, housing, healthcare, etc.)? And then focus, not just on what few good Samaritans might do, but on how households, firms, and governments would respond at scale: labor supply, wages and prices, inflationary pressure, taxes, effects on productivity, growth, etc.

But I think our discussion has run its course. While we clearly have different opinions, it was still fun and I hope we both learned something. Be well!

> Or facilities optimize to produce less trash so they can handle the newer trash load with less staff instead of paying extra.

You are breaking the principle of keep all other conditions constant. If it's possible to optimize, why didn't they do it before? They were already motivated to maximize profits. Optimization is also an additional work which you are conveniently ignoring.

> Food is extremely cheap to the point that the USG (effectively) sets price floors which prevents it from falling further. People do live on $3/day.

Food prices sometimes fall below costs because agriculture is volatile. It's not proof that food is inherently "extremely cheap" to produce. Also, absolute prices cannot be easily compared between countries and low quality food has negative health effects.

https://ourworldindata.org/grapher/food-expenditure-share-gd...

> Activation Energy - Many people have no choice to work a dead-end or low-paying job because they cannot afford to take a break to find higher paying work more suited to their skills.

If basic needs are so cheap that UBI can cover it, then doing low-paying job part time should also cover these basic needs. Something in your logic is not adding up.

> Opportunity Cost - The geologist example from above where you need to hide information from others so you don't suffer.

Opportunity cost is the loss of other alternatives when one alternative is chosen. Geologists not needing to hide information, because they don't need that money to survive is opportunity cost how? And even with UBI they would still benefit, if they get that additional money.

> Societal unrest - Literally right now there's a president in the US whose base is upset about how the technological progress was not shared with them to the point they want to throw away any current advantages to go backwards in time.

UBI fixes that issue how exactly?

Of course money is not the only thing that motivates people. But there's a lot of empirical evidence that it matters. A lot, unfortunately. And I say unfortunately as I would rather have it matter less. But me whishing it doesn't change the data.

UBI is a high concept pitch, that is memorable and catchy, but AFAIK it's not well supported either by psychological models or by empirical economics data. It gives some social safety net. Problem is that it gives a rather weak safety net. We can actually do better.

Can I ask you why exactly does it need to be UBI? If another system (more complex, with less sexy pitch) could provide a bigger safety net and have a more positive economic impact, wouldn't you rather choose that?

But knowledge and academic research and industry R&D are key for progress though. All of which require hard work.

You also don't balance equations in your examples. Your janitor goes to help a friend with rocketry, which seems like a net gain, but someone else now needs to stop helping a friend to replace that janitor's position. Otherwise researches at that facility where janitor had worked will have to do janitor's work instead of doing their own. You call work cooks and janitors are doing worthless for progress, but researches (or children in school) need to eat, need to have functional workplace/classroom, etc. While they might not make progress directly, they enable other people to make progress.

> We're long past the point where we barely need anybody to work to actually feed/house everybody

Why would we need UBI then? The price of food and of housing everybody would be dirt cheap, if that were really true. Value of anything is completely relative (which I find that many people have trouble grasping). If something requires very little work, then it will be very very cheap in an ideal free market.

Seems like you are making same binary assumption that people either work or they don't. The important question is probably how well/hard do people work. Lower productivity means people that work produce less so prices rise. Many make mistake thinking only about having money, but forget the supply part of the equation. If productivity is lower, there are literally less things available to everyone. And these equations are not linear. Look at the current RAM situation for example.

But the major issue is that the progress slows down. Effects of slower progress accumulate with time. At first you are only a few years behind, then you are a few decades behind etc. Imagine inventions, cures being available decades or hundreds of years later (depending on what timescale we look at).

I think UBI sounds nice, but is far from an optimal solution. Wouldn't be better, if we could solve same issues UBI promises to solve in a more efficient way (with less negative side effects)? UBI is just throwing money at the problem, hoping it will solve itself.

> Basic means it's not much, just enough to cover basic needs

But what are these basic needs that are not much? Housing costs, medical expenses...?

I agree, .NET Standard limitation unnecessarily complicates development experience. I think it's because some tools (Visual Studio) is still use legacy .NET Framework. I don't understand why they didn't integrate them via out of process architecture into these tools, since source generators didn't exist in the legacy framework anyway.

I sometimes generate code from plain CLI projects (avoiding source generators altogether), as whole debugging and DX is so much better.

NativeAOT compiled C# library can create C compatible exports which can be used with any language that supports C libraries.

Wrong? You are comparing number of possible values that can be stored not the amount of bits. Drive sizes are specified in bytes not possible combinations it can store.

Start menu in I Windows 11 is quite bad. One things that really annoys me is how little space the start menu has for pinned shortcuts.

Encapsulation

In a way, this is similar to tags vs folders.

Folders are hierarchical way of organizing, akin to inheritance and tags are compositional way of organizing.

I'm kind of waiting for any language to invent some sort of #hashtag interfaces to define contracts :)

DataFusion is another option I would be interested to see in a comparison like this.

Much better UX and integrated ReSharper.

> But no one who opts for canvas turns around and remembers to do their landmarks.

Not completely true. Flutter has been adding some accessibility for web canvas target. [1]

I think Avalonia is in in the make it work phase. Accessibility will probably be added in the make it right phase.

[1] https://docs.flutter.dev/ui/accessibility/web-accessibility

I would much rather see web apps become canvas rendered WASM versions of desktop apps than desktop apps become webview apps. Latter is what we have been seeing in the recent years unfortunately.

Canvas rendered cross-platform UI frameworks like Flutter & Avalonia targeting browsers (WASM), might shift the balance back in favor of desktop UI.

Originally, C# was more like Dart, but modern C# is like both Dart and Go combined and that's (IMO) the beauty of it.

Newish low level features of C#: https://em-tg.github.io/csborrow/