6,131 karma · joined February 7, 2013
edit: not Europe, just UK and Japan apparently: https://www.zdnet.com/article/ericsson-expired-certificate-c...
> paragraph, n.: a distinct section of a piece of writing, usually dealing with a single theme and indicated by a new line, indentation, or numbering
In the original comment you will find two of these, the former correcting an error in the parent comment, the latter making an observation based on the obvious brainwrong riddled throughout this thread
I'm done replying, you're of course free to continue checking in hazardous and suspect file IO code, as the rest of us are free to giggle at such things before ripping them out
(But that would be a sweeping generalization)
The "powerful (and consequently hazardous) OS feature" here is using mmap for general file IO, it:
- introduces resources leaks many developers can't profile
- introduces VM bottlenecks 99% of developers can't profile
- introduces random segfaults delivered to arbitrary threads in the running process, leading to crashes many developers can't diagnose
- the mmap() interface itself is fundamentally unsafe in that it allows partially overwriting random bits of VM (MAP_FIXED) with file views, and worse still, allows those mappings to be read-write
Once again, nobody has ever suggested avoiding virtual memory except you -- once again, that is impossible in a modern environment, but it is more than possible, and ultimately incredibly sensible, not to mention entirely on topic with regards to this thread and the article it is attached to, to suggest avoiding use of mmap for general file IO
My first comment was in reply to one claiming anonymous memory did not have the same problems as file-backed memory, indicating the parent did not understand they are the same thing. The subsequent reply was to another comment continuing to claim anonymous memory was somehow safer, both instances supporting the notion that most people in this thread don't seem to understand mmap at all.
What we're examining is a powerful (and consequently hazardous) OS feature that often provides only marginal performance improvement, yet introduces many exotic error paths into a program that have their own exotic problems (memory access in thread A can raise SEGV in thread B, async-signal safety), that 7 hours' commenting has not been sufficient to fully capture. This thread is full of upvoted miscomprehension, bad advice (spawn a child to deal with SEGV!?), obviously incorrect solutions (signalfd), and yet still manages to completely omit some critical characteristics of mmap, for example, that faults take a VM-global semaphore -- mmap can easily destroy multithreaded app performance in a way read() is immune to, because nobody expects file IO in one thread to cause malloc() latency in another.
If this isn't evidence for "avoid this feature wherever possible", I really don't know what is.
There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid mmap whenever possible -- it's clear almost nobody fully understands it
Its not clear why this needs to exist, as opposed to working with upstream to produce a single tree that can be built with a slimline profile enabled
It allows trading lower CPU for gzip-or-worse compression, and you can mix the settings within a single file. This means you can e.g. use the lowest setting (or no compression at all - it supports that too) to append to a file, while occasionally recompressing recent appends into a single block using the highest setting -- so the cost of compression can be amortized
The only petty annoyance with it is ecosystem support - e.g. GNU tar has no option for it, so it's slightly more painful to work with
Teehee. The method is not new at all, for example compressors like xzip do this out of the box, and almost the exact thing they're doing is basically how ZIP files work
The trouble with discarding state on every file is that it really hurts performance with small files, or when using anything like a modern codec, which gzip/deflate is not. Gzip maintains a 32kb dictionary which is quite easy to exceed with contemporary data, but with a modern compressor (like lzma2) losing that window will absolutely devastate ratios
The usual solution is so-called 'solid' compression, where the uncompressed input is partitioned into blocks spanning file boundaries. It configurably trades seek efficiency for reliably preserving compressor context -- including allowing seeking within files. Their format could be modified to support this while retaining backwards compatibility as good as the current method. What they have is already pretty much solid compression, except it only chunks large files. This is basically a weird special case of a simpler and more general design everyone else uses.
Finally on the compatibility angle, the end of stream is visible at an API level, so this isn't going to be 100% perfect. I'd expect one or more obscure implementations (maybe Windows apps? Java?) to potentially break
If memory serves it isn't a theoretical attack either, I read about it used against (Startcom maybe?) not so many years ago
People don't buy individual pieces of technology, they want packaged solutions that come with support. Tech almost matters less than good support
So you're left with a choice in this wonderful alternative non-optional world: some expensive temperature controlled oscillator with accompanying nuclear power plant integrated into every mouse, with a dual core processor just in case the host wishes to speak mouse-over-Thunderbolt, with a 240V connector on the mouse just in case the host wants to charge from it, or mice living on a separate low throughput bus where such requirements don't exist. We had that already, it was called the 1970s-1990s, it was an even bigger mess than what we have now, and is exactly what USB's mandate is to avoid
It's not like this is a new problem to USB, it's been a tri-modal specification from the outset to cope with completely different requirements of the peripherals that were to be unified, these horrible recent complexity outcomes are just a natural extension from the early days.
It wasn't so long ago that every budget peripheral manufacturer outside of e.g. printers and mice were forced to bundle expansion cards to implement custom busses just so you could talk to their e.g. scanner. This was still the reality of things as late as 1995 or so. Here we didn't just have custom connectors on the back of the machine, but custom cards that had to be installed to implement those connectors. In the 24 years since 1995, outside of display interfaces I count only 2 major new busses to date -- Firewire and USB. That sounds like a success to me
edit: yikes, looks like USB C doubled the number of those separate pairs, for a total of 6 data pins (2x RX, 2x TX, 2x USB 2.0 RX/TX)
It is the company's duty of care for the same reason that
- it is the city's duty of care if the child goes outside and falls down an unmaintained manhole
- it is the school's duty of care if a fire extinguisher malfunctions and kills the child
- it is the driver's duty of care if the child crosses the road and gets struck on a green crossing
In all these cases, the entities assume a certain privilege to operate due to the trust placed in them that enables the freedom for the child to go outside whatsoever. If that duty does not exist, then there is no trust the child can safely leave home unattended. In effect what you're arguing for is total control from the parent -- a much worse outcome for a child than self-moderation on behalf of the trusted entities they would otherwise have had the freedom to interact with.
It is the company's duty of care for the same reason that
- it is the city's duty of care if the child goes outside and falls down an unmaintained manhole
- it is the school's duty of care if a fire extinguisher malfunctions and kills the child
- it is the driver's duty of care if the child crosses the road and gets struck on a green crossing
In all these cases, the entities assume a certain privilege to operate due to the trust placed in them that enables the freedom for the child to go outside whatsoever. If that duty does not exist, then there is no trust the child can safely leave home unattended. In effect what you're arguing for is total control from the parent -- a much worse outcome for a child than self-moderation on behalf of the trusted entities they would otherwise have had the freedom to interact with.
Would you categorically state that if the service can't do it, the service should continue in its present form?
Walls are some of the oldest inventions of civilization, and YouTube currently lacks /any/ walls. I'm arguing for some walls in the right places, you're arguing for no walls whatsoever. We have a difference of opinion, it's fine.
Unfortunately this is inaccurate. There are billions of daily adolescent YouTube users, some of whom entering puberty that were exposed and continue to be exposed to the recommendation algorithm that created this mess in the first place, and for periods running into hours every evening.
At such an early stage in development, it is absolutely the company's duty of care to ensure that a 5 year old is not being sent up a gradient of a recommendation system that is encouraging them (with the help of the comments just removed) to view people of their own age in a sexual manner. Not only were children impacted by this, but the mechanism that enables it remains active to this day.
I agree with the idea of allowing children outside, but as per my reply to your previous comment, not if that means spending all day in the back yard of the village creepy old man. Balance is required in every situation, and denial of the kind your comment is riddled with accomplishes nothing.
> Would you ban beaches as well, because creeps might watch others on a beach?
No, but nor would I pretend people can shower and change without cubicles, or permit the nearby village to drown during the first storm because I deluded myself in the belief there was no need for a seawall.
There is no rationality to be found in an absolute "save the children at all costs" position, much as there is none in "freedom of speech at all costs". This issue like so many more is nuanced, and I'm surprised people here are so easily willing to deny its possibility, but prefer to get worked up by its very existence or the plain reality nuance is required to solve the problem. Progress is never due to people like that.