HNHacker News
TopNewBestAskShowJobs

wyoung2

168 karma · joined December 15, 2015

submissionscomments
wyoung2··on The First Rule of Microsoft Excel: Don’t Tell Anyone You’re Good at It
If the task is both undesirable, difficult to find people to fill, and yet widely sought after and greatly needed, the word you're looking for is "lucrative."

Example: plumbers.

Excel wizards: quit your whining and go make bank.

wyoung2··on The First Rule of Microsoft Excel: Don’t Tell Anyone You’re Good at It
What's the alternative?

Microsoft leaves Access out of many of its Office license plans, so it's an extra cost add-on to most people, hence uninteresting.

Filemaker is very nice, but even more costly in most cases.

I've never used Base in LibreOffice, but I also don't see people sharing their Base files, from which I infer that it isn't very popular. (Could be my myopia, though.)

Even if LibreOffice Base is gaining users much more rapidly than I think it is, they're missing the cloud component for offline and mobile users, which acts as a brake on adoption, given Office365 and Google Sheets pushing people to stick with spreadsheets when they want to share tabular data across sites and to mobile users.

Collabora Online is moving LibreOffice into the cloud, but they haven't done anything with Base, as far as I can see.

Apple's iWork web apps are very nice, and they have native clients on many of the platforms I care about, but there's no GUI database component at all.

Without a free, cloud-capable, normal-person focused GUI/web database system, I don't see Excel-as-database going away.

wyoung2··on The First Rule of Microsoft Excel: Don’t Tell Anyone You’re Good at It
The problem with debugging formula code is that it's hidden in the default view, and you have to know which cells to click on to find each bit of code.

In simple spreadsheets, the proper cell to click on is obvious, because the formula only updates the one cell where the formula is. The problem comes with multi-cell functions like Excel's VLOOKUP or Google Sheets' QUERY, where a cell that looks like it contains literal text might contain the complex formula call, and then all of those around it might look like plain text, but they're also the call's results. Editing any of this wrecks the sheet.

Despite these dangers, I recently perpetrated such a thing in Google Sheets, because that's the only platform I could get to easily work across many sites, some of which have users that are often offline. The next step up from that would be to stand up a custom web app, and the project simply did not demand that amount of effort. The next-best idea was to keep passing around .xls files via email; barf.

That's the seductive nature of these one-off spreadsheet "apps": they're quick and easy to stand up, yet difficult to extend and debug by the time that they grow to the point where they would justify development time for a proper application. By then, they've also gained a big enough user base and feature set that rebuilding the app properly also means a big effort in all axes: development, testing, deployment, and retraining.

wyoung2··on The original sources of MS-DOS 1.25 and 2.0
> So many painful memories or is that nostalgia.

The Greek roots of “nostalgia” translate as “the pain of returning home,” so it’s a false dichotomy. :)

https://en.m.wiktionary.org/wiki/nostalgia

wyoung2··on Times Newer Roman, a sneaky font designed to make essays look longer
> I don't have a "rational" argument

I do: Times New Roman was designed for a newspaper — Brittain’s “The Times” — which means it was designed in large part to keep the cost of paper and ink down. Compared to book fonts of the time, Times New Roman has uncommonly narrow letterforms and short descenders to allow for more text on a line and more lines in a column.

If you care more about aesthetics than about ink and paper cost, you should never use Times New Roman.

I second the recommendations for Garamond and Palatino.

wyoung2··on The “Developer Experience” Bait-And-Switch
Here are some relevant test results which are about 4.5 years old now:

https://modernweb.com/is-jquery-too-big-for-mobile/

At the time, your 3 round trips would have cost about a full second, which is roughly the same as the worst case JITting time for jQuery in those same tests. That worst-case result was on a slow Android 2 device.

Cellular network speeds have gone up, but certainly not as much as mobile processor speeds, so if you re-did those tests today with modern networks and devices, it’s probably a net benefit to pull the jQuery once, then JIT it on each page load, as compared to making even a single extra round-trip per page load.

> Processors aren’t getting any faster.

That’s true on the server side as well.

The article talks about externalized costs, but if you just shift the computing burden to the server, how do you pay for that?

You could load the page with more ads, which eats up the network bandwidth and JIT savings you just bought by moving the processing to the server.

Or, you could charge the users more than you currently do, which is economically little different than shifting the computing burden to the users, implicitly requiring them to buy faster mobile devices and better mobile data plans.

Consider also that the number of bugs created per line of code is roughly constant for each pairing of developer and programming language, so who pays for the costs of the extra bugs you’d expect to find in 2-3x more LOC?

TANSTAAFL.

> JS-heavy pages I've seen tend to have load times measured in seconds.

I doubt you’re comparing apples to apples.

There certainly are many very fat JS-heavy pages on the Internet, but what’s your comparison? If the JS-light alternatives aren’t accomplishing the same ends, then it’s not a fair comparison. You can’t compare, for instance, a web IRC gateway app to Facebook, even though you can use both to transmit plain text to another person.

Not that I’m defending Facebook. I’m just pointing out that they’ve got a wholly different thing going on there than the IRC gateway.

wyoung2··on The “Developer Experience” Bait-And-Switch
> server-side rendering

That costs an HTTP round trip, which can easily pay for the cost of tens of kB of JavaScript code on each hit.

Since a properly-optimized web app will have all or most of its JS code set to be cached indefinitely, the cost of the JS code is near zero; basically, just the cache lookup time. Contrast the HTTP round-trip, which you pay on each such interaction.

If you can render something on the client side without reaching back to the server, you should.

> plain-vanilla javascript

XHR and the browser DOM are horrid, inefficient APIs, in terms of lines of code per amount of functionality delivered to the end user.

I recently wrote some jQuery code to prototype a user-requested feature which ballooned to about 3x the size when the person managing the project demanded that it be done without any external dependencies. (It was about 2.7x the lines of code and 3.5x the bytes of code, the difference being that the XHR + DOM lines tended to be longer.)

In the same project, a separate rewrite from plain-old-JS to jQuery cut the code size nearly in half.

Since this plain-old-JS code was inline on the page, the user now pays for it on each page load, whereas an aggressively cached static library pull will be paid for only once as long as the user doesn’t toss the browser cache and re-visits the app often enough to keep the cached JS code in the cache.

Developer efficiency and user efficiency don’t have to be at odds. There’s a wide gray area between the article’s cherry-picked examples and a web app engineered for efficiency.

wyoung2··on Try OCaml
> What type should this function have:

If you don't call it, F# simply guesses "int -> int -> int". If you pass a float, you get back a curried function taking float and returning float. Now that the "add" function's type is known, you now can't pass an int: that's now known to be an error.

Seems sensible to me. No problem.

wyoung2··on Try OCaml
Yes, you're right, I shouldn't use the development tree for my comparison here. The runtime-only .NET Core 2.1.3 installer for my platform is only (cough) 26.3 MB.

If you then try to get that lower by using self-contained builds, you can only get "Hello, world!" down to about 20 MB:

    $ dotnet new console --language f# -n hello
    $ cd hello
    ...add <RuntimeIdentifier>osx.10.13-x64</RuntimeIdentifier> to fsproj
    $ dotnet publish --self-contained -c release
    $ cd bin/release/netcoreapp2.1/osx.10.13-x64/publish
    $ rm -r ?? ??-?? zh-*                 # nuke i18n
    $ tar -cvJf ~/Desktop/hello.tar.xz .
You'll notice that I'm giving you the benefit of high compression here. The unpacked tree is 70 MB!

The equivalent in C++ is 3.3 kB.

I'm not expecting miracles. As I said, I'd be happy with "only" a 3x ballooning here, roughly commensurate with the coding efficiency benefit. That's fair.

wyoung2··on Try OCaml
> promotion from int to float is not free!

Nor is it safe to allow implicit conversion in general, but F# shows that you don’t need separate operators for floating point. You just need a compiler that refuses to implicitly convert data types, as F# does, so you must use explicit conversions and type suffixes if you want to mix data types. If you manually convert, then on your head be the consequences.

This is not legal F#:

    let x = 1.0 + 1
But both of these are legal:

    let x = 1.0 + 1.0
    let y = 1.0 + float 1
F# doesn’t even let you mix single- and double-precision FP:

    let x = 1.0f + 1.0         // ILLEGAL
    let x = 1.0f + float32 1   // legal
wyoung2··on Try OCaml
> It’s impossible to divorce the two

It would surely be a lot of work, but “impossible?”

I think you’re missing a couple of major tricks by not doing a native code compiler for F#:

1. Now that Moore’s Law^H^H^HBusiness Rule has run dry, we can’t continue to just pile up the abstraction layers. Some sage once said that every problem in CS can be solved by adding another layer of abstraction, but there’s one that can’t: make it faster without buying more hardware. Given how popular FP is among the quants, a field often going after microsecond latencies, I’d think this would be a constant refrain from potential customers.

2. I don’t want to ship a dotnet runtime to my customers. Just this morning, I had to update a customer accessible only via a highly restrictive VPN, where they haven’t given my server DNS access, much less unfiiltered Internet access. Everything I ship to them has to be scp’d to the server over the VPN. I really really don’t want to send each such customer hundreds of megs of dotnet runtime stuff, then be forced to work around the failure of “dotnet restore”, so I’m going right back to C++ so I can get native binaries.

Hundreds of megs might be charitable. I just did a “du” of /usr/local/share/dotnet on a workstation here and got 1.4 GB. Presumably that compresses down to mere hundreds of megs. Contrast the size of a “Hello world” binary, which I’d expect to be under 10 kB.

I want to like F# on dotnet, but you’re making it hard for me to use it in practice.

wyoung2··on Try OCaml
For me, the attractions of F# over OCaml are:

1. Cleaner syntax. F# shows that we can do away with a lot of the visual noise in OCaml. I’m no particular fan of white space scoping, but I also don’t mind it.

2. Much better thought out standard library. The OCaml standard library was clearly accreted over long periods, with no strong designer making hard choices about naming and such. The example that keeps coming back to me is “big_int” vs all the other integer data types.

I believe #2 is what the OP meant by “minimal runtime.” A minimal F# will still need a good set of data types, a string library, etc.

wyoung2··on Americans can legally download 3-D printed guns starting next month
A drone capable of carrying the weight of a useful firearm and dealing with the recoil is a potential weapon in its own right.

If you were thinking of a lightweight one-shot deal that sends the drone flying in pieces away from the point of firing according to Newton's third law, then I question the operator's ability to aim it properly. Every minute of angle you're off in your aim at the start is 1 inch of error at 100 yards, and that's if all else is perfect.

With a lightweight armed drone lash-up, I think you'd end up with a system with an inherent accuracy measured in whole degrees, not minutes.

wyoung2··on Ask HN: What is one Vim trick most people don't know?
> Insert the contents of a file: :r <filename>

The file name can be a pipe instead.

Let’s say you have a Makefile based project and the dependencies aren’t as complete as they ought to be. With your cursor on this line in the Makefile:

    SOURCES = \
You can say

    :r !ls src/*.c
That will put the complete list of extant C source files into the file at that position, complete with the “src/“ path prefix. Then to format the list properly for “make” syntax, use Shift-V to select the list, hit the right angle bracket (“>”) once to indent it a level, then say:

     :s/$/ \/
This trick of replacing the start (“^”) or end (“$”) of a line is very often helpful. Here I’ve used it to ask Vim to put a space and backslash after every line to allow the SOURCES variable definition to continue from one line to the next.

(This will look like “:’<,’>s...” in your editor, with the bit between the colon and s being inserted by Vim; it means “front the start of the selection to the end.”)

Remove the final backslash and it’s done.

The advantage of working this way is that you aren’t manually transcribing things the computer already knows and can report on command.

The possibilities here are vast. For instance, if your C source files are buried under subdirectories of `src`, say this instead:

    :r !find src -name \*.c
wyoung2··on Ask HN: What is one Vim trick most people don't know?
The nice thing about using `=` instead of `gq` is that you can “set equalprg” to something smarter than the internal Vim formatter, such as `fmt` or `par`:

    https://www.gnu.org/software/coreutils/manual/coreutils.html#fmt-invocation

    http://www.nicemice.net/par/
These formatters will do nice things like choose line breaks intelligently to reduce the raggedness of the right edge. I just tested `gqip` on a paragraph in a Markdown document here that was formatted with `fmt`: the `fmt` output put breaks in such that all but the last line was within one character of equal, whereas `gqip` had line lengths varying as much as 5 characters on a 72 character line length.

A quicker alternative to Shift-V, arrows, then `=` to reformat a paragraph with `fmt` is this:

    :map <F2> !}fmt -w72<CR>
That works anywhere inside the paragraph like with your `gqip`, but it uses the smarter `fmt` paragraph formatter.

Select your keystroke to taste. I chose F2 so long ago that I forget why that was a good idea at the time.

wyoung2··on 'We're doomed': Mayer Hillman on the climate reality no one else dare mention
> Land use is actually staying constant or going down.

That's consistent with my claim that all the good farmland is already taken. The shrinkage in use of farmland could be that marginal farmland is falling away due to soil exhaustion and economic effects.

> humans are becoming more urban.

That sounds like the path towards arcologies.

You can see it now: former single-family dwellings in the large cities are being rented out to groups of yuppies, each of whom, despite being high earners, would struggle to pay the rent alone. The low earners are being forced out, but to where? Where does this road lead, if not to arcologies?

> It has nothing to do with McMansions, humans are becoming more urban.

Only if you define "urban" as identical to "metropolitan area."

Take any large US city, then include its complete metropolitan area, which I'll define here as the area of continuous land surrounded by either farmland or wasteland. I'll stretch the definition of farmland to include single-family-owned parcels of land which are large enough to support small alfalfa crops, horse pasturage, etc.

How many "cities" are in that metropolitan area? Why are they called "cities," even though there is no farmland between them? I'm fundamentally a rural sort, all my life, but I've been in such areas many times, and the only way I can tell that I've traveled from one city to another is that there's a sign situated between a laundromat and a strip mall, with the city boundary going down the alley between them. Once upon a time, they were in fact separate cities.

Take Los Angeles, California as an example. In what meaningful way is Anaheim not "L.A."? I get that its streets don't look much like those of Beverly Hills or Monrovia or Pasadena, but the whole area is one urbanized mess. You used to pass through farmland to get from each of these cities to the next. As an occasional visitor to the area, "L.A" to me extends from Ventura to the north down to San Clemente in the south, and inland almost to the Salton Sea. All that former farmland is paved over now.

> There is also tons of lower quality land that could be farmed if you really needed to as well.

Less productively, by definition.

> Around 45% of corn production is used to make transport fuel rather then food.

Sure, but that's just part of the same problem: as population and wealth rises, the wish to have personal rapid transit also rises, which leads to either:

1. Greater consumption pressure on our oil reserves.

2. A switch to biofuels of some sort, which in turn puts pressure on the rest of the bioeconomy.

3. A switch to electric vehicles, which requires massive solar cell arrays and massive distributed mines for rare earths, which stresses the environment still further.

...or some combination of the three, of course.

It's all an interdependent system. You can't move one part without affecting many other parts.

> food is not becoming more expensive other then being more expensive because of oil and transport cost.

The increasing transport costs are in part because of my original premise: cities used to have nearby farms to draw from. Now that we've pushed the boundaries of so many of our conurbanizations out to the wastelands, we've got to truck in food from much farther away than before.

In the US, some of the areas of most rapid population growth are out in the desert, places like Las Vegas and Phoenix. Why do you suppose that is?

> Most move to urban centers voluntary.

Yeah, voluntary in the same way that having children work in the coal mines was "voluntary" to our forebears. They had a choice: starve or go farming. Back then, we still had much of the West available for expansion. That's largely gone now.

At the grandparent level, my family were farmers. The land they did all of that on is now sold off for houses. My forebears made the desert bloom. The thing is, though, they did it around rivers which now run dry because the remaining farms are sucking up all of the water that isn't going into the houses or being dumped back on the desert to keep all those middle-America lawns green.

You speak of increasing productivity for our existing farmland, but all increasing curves flatten out eventually. There are no exponential curves in ecologies, only S-curves that haven't started to flatten out yet. (Or, collapsing curves!)

wyoung2··on 'We're doomed': Mayer Hillman on the climate reality no one else dare mention
> We have lots and lots of land that could be farmed if we needed it to.

Where?

As I understand it, pretty much all the good farmland is already in use for farming.

The only available farmland I'm aware of that is not being used for that is being squatted upon by McMansions in the suburbs. The land they're sitting on was formerly farmland, but it was found to be more economically valuable as land for upscale homes.

Think what it would mean for those economics to be flipped around, for farmland to become more valuable than McMansion lots. It means farming has become a very high-dollar economic activity, which means food has become really expensive, which means many fewer people would be buying McMansions.

So what then is your plan? That we put almost all the human population into arcologies like the Stephenson cyberpunk novels, so that we can free up the land to be farmland again?

wyoung2··on 500M of “Have I been Pwned” hashes broken
The apps become read-only, with export functions.

Source: https://support.1password.com/membership-billing-policy/

wyoung2··on Apple’s 2019 Mac Pro will be shaped by workflows
> The only really glaring issue is ConEmu and related Windows terminal ecosystem is still hot garbage compared to any Linux or OSX terminal.

...and that bites you in a surprising number of ways.

I recently tried porting some software that runs just fine under Ubuntu-on-x86 to Ubuntu-on-WSL, and I found three different failure modes in the console/pty mechanism before I gave up. I then tried running it under Cygwin and it ran correctly out of the box.

Perhaps the easiest way to see this is to try to run a program like GNU screen under WSL, but the core problem isn’t any specific application. Any program that does anything even moderately tricky with ptys is likely to fail under WSL.

wyoung2··on Apple’s 2019 Mac Pro will be shaped by workflows
> Lightroom just work[s] a ton better on Windows

That’s not my experience, at least with Classic.

Mine is a Mac house. My parents, though, are long-time Windows users; I only managed to get my parents to run a single iMac, which they gave away as soon as they could financially justify it.

I tell you that because that is where my cross-platform Lightroom experience comes from: almost all Mac, with occasional mentoring/support sessions under Windows at the parents’ house. With Lightroom on Windows, I’ve repeatedly observed:

1. Windows’ brain-dead default-mandatory file locking prevents perfectly reasonable operations, simply because Lightroom is busy working with one or more files in the background. Advisory locking as is default on POSIX type systems allows many things that Windows refuses to allow by default. (E.g. Rename a parent folder while a file in that folder is open for writing. Who cares, the FD is still valid!)

2. Lightroom can take a long time to shut down when it gets busy, as it too-frequently does. On the Mac, the app icon remains marked “running” while Lightroom grinds away, trying to figure out how to shut down, but on Windows, the app icon disappears from the task bar almost immediately, but Lightroom.exe remains running in the background, so that it is not obvious why reopening the program fails for minutes at a time.

(And why reopen? Because relaunching Lightroom often solves slowdown problems, and has for years upon years, which is a separate rant.)

3. Some plugins simply won’t run on Windows, at least not without dragging along a bunch of compatibility junk. Anything that depends on ExifTool, for example, requires dragging over a whole Perl environment just to run the plugin. If you have multiple plugins dependent on ExifTool, as I do, each one usually comes with its own Perl environment. Compare macOS, where at worst you have the CPAN module alone, and at best, it might simply require that you install ExifTool separately, that being a reasonable user requirement on an OS like macOS.

If your comment was referring to GPU acceleration and such, Adobe’s recent focus on that isn’t helping me anyway. Almost all of my problems with Lightroom’s speed are in the Library module, not the Develop module, where GPU acceleration doesn’t help much anyway.

wyoung2··on ZFS 128 bit storage: Are you high? (2004)
Because "Z" matches the goal of being "the last word in filesystems" than "Y".
wyoung2··on ZFS 128 bit storage: Are you high? (2004)
The article isn't talking about the size of raw disk blocks, better called the sector size. It's talking about ZFS blocks, which are variable in size, defaulting to 128 kB: https://en.wikipedia.org/wiki/ZFS#Variable_block_sizes

ZFS aside, the 512 byte sector isn't even a hard limit at the storage medium level. Disks with 528 byte sectors — and other odd sizes — have been available in the enterprise world for a long time now: https://en.wikipedia.org/wiki/Hard_disk_drive#Market_segment...

HDDs and SSDs with 4 kB sectors dominate the consumer market now: https://en.wikipedia.org/wiki/Advanced_Format

wyoung2··on Using Git Wrong
That pretty much describes Fossil.

As with Subversion, when you're not working on a project alone, you normally run Fossil on some central server somewhere. Then because it's a DVCS, you clone a copy of the repository to your working machine.

So far, it's vaguely like Git, but it diverges quickly because the default behavior of Fossil from that point on is basically identical to Subversion, only with a much saner branching model and more features. (Built-in wiki, ticket tracker, web UI...) A checkin on the local machine first checks the change into the local repo clone, then it attempts to sync the local repo with the remote repo it was cloned from. (That's a two-way sync, not just a push.) Thus, like Subversion, checkins normally immediately appear on the central server as soon as they're checked in.

The sync can fail, of course, but what happens then depends on why it failed.

If the sync failed because the tip of the branch you're working on has moved forward on the remote repo, because something else has been committed to it in the meantime, it aborts the checkin because to proceed would create an unintentional fork in the branch. So, it makes you do a pull first to merge the remote changes into your branch. You then re-build, re-test, and then check your changes in as the new branch tip. (You can force Fossil to accept it anyway, but then you have to deal with the forked branch.)

If the sync fails simply because the remote server couldn't be reached, your changes are checked into the local repo clone only. Once connectivity to the remote repo is re-established and you cause a sync, it pushes your local checkins to the remote server. Unlike the online sync failure case, Fossil won't refuse to sync your changes up to the remote even if it creates an unintentional fork, because otherwise there would be no way to sync. You must then resolve the the fork manually, which typically is done by selecting one of the fork tips as the new branch head and merging the other in. Alternately, you could call one of the fork tips a new branch, then merge later.

Fossil doesn't care which way you do it, because branch names are for the human only anyway. It wants you to resolve the fork only so you don't confuse the other humans, not to solve any confusion on Fossil's part. (Underneath, Fossil only cares about the relationships between the artifact hashes, not about human-friendly file and branch names.)

You can avoid the forked branch problem by checking changes in on a new branch when you know the remote repo is unreachable, such as when you have no Internet access and the remote repo is on a public server somewhere, or when you're working away from the office and the remote repo is inside a firewall and you're on the outside. Pushing a new branch never creates a fork.

If you don't like the autosync feature, you can turn it off, so it doesn't even try to sync changes between the local clone and the remote it cloned from on checkin. In this mode, it works more like Git, only with the day-to-day working sensibility of Subversion. You can then do manual pushes once your local repo is in a shape you feel like sharing.

If by "sharing with colleagues" you mean that you and your colleagues do not have mutual access to the same central repository, Fossil can cope with that, too. Fossil has a feature called "bundles" that pack up a subset of a repository, which can then be merged into a different repository. You can only do this if there is some history between the repositories, so that they at least share the same root — you can't just graft pieces of one repo onto a totally foreign repo — but it does let people who otherwise wouldn't have checkin rights on your repo send you changes. Because the Fossil bundle format preserves 100% of the Fossil feature set, it's basically an uber-patch(1) format, preserving things like checkin comments, branch points, file renames, etc., which unified diffs do not.

Git has some advantages, but you have to really need them bad to put up with the UX problems you mention. Examples:

1. Popularity. If you need tool integration — really need, not just want — then Git is more likely to be supported by your tools than Fossil.

2. GitHub. Some people's concept of "Git" is actually "GitHub". Fossil's web UI has a large subset of the features GitHub adds to Git, but if you need something Fossil lacks, such as the network effects stemming from GitHub's prime position in the software development community, it's fair to regretfully say you cannot use Fossil.

3. Rebase. Some people really really need this. Many more people think they need this. Fossil users prefer that a DVCS work more like an accounting system, which records what you actually did, not what you should have done, edited after the fact.

4. Working in private by default. Personally, I think this causes more problems than it solves. It encourages people to go off on wild tangents rather than collaborate with their peers. This mode of operation is essential in a development community the size and distributedness of the Linux kernel, but very few projects have this same pressure to decentralize. If you know all of your collaborators by name, you probably don't want everyone going off on their own wild tangents. If you can't safely commit to the trunk, check your stuff in on a branch so it doesn't break everyone's build, then merge when you're ready. Meanwhile, your coworkers can see what you've been up to, so no one's surprised when the branch finally gets merged in.

wyoung2··on ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot
No.

The first anecdote is about a TrueOS box — which previews what will become FreeBSD 12 — and the second is about a macOS Sierra box running OpenZFS on OS X.

Since TrueOS, O3X and ZoL are all based on OpenZFS, I expect that you will have the opportunity to replicate my experiences should you have disks that die. Now I don't know whether to wish you good luck or not. :)

wyoung2··on ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot
> might as well put in a full FS read in a cron

ZFS scrub is not the same thing.

If you do a full-filesystem read in a RAID system at the OS level, the redundant blocks won't be read: the RAID system will simply choose one of the copies to read based on which disk(s) is least heavily loaded at the moment. This is why reading on a 2-disk mirror is twice as fast as reading from a single one of the disks comprising the mirror.

During a ZFS scrub, all copies of every block are checked, and because the data is heavily checksummed, ZFS knows which copy is right if one of the 2+ redundant copies doesn't match its checksum.

Additionally, ZFS is structured as a Merkle tree (https://en.wikipedia.org/wiki/Merkle_tree) which avoids whole classes of ways traditional filesystems can become deranged at a structural level. ZFS always stores 3+ copies of certain types of filesystem metadata, even on a 1-disk ZFS pool, so that if one gets corrupted, it has 2+ others to choose from. When this same type of corruption happens on a traditional filesystem, well, let's just say that's why `/lost+found` exists.

> Most kinds of damage cannot be repaired on a live filesystem anyway.

See my post above, giving two anecdotes of ZFS actively repairing data on live filesystems. Both systems were in continuous use while these repairs proceeded, and no data were lost in either.

wyoung2··on ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot
Scrub/verify/patrols, whatever you want to call it, with RAID all it can do is say, "Well shit, these two copies don't match. What do you want me to do about it, boss?"

ZFS doesn't have to guess which copy is wrong. It knows, and it will automatically replace it.

More, ZFS will even do this on a ZFS mirror when reading half the data blocks from one disk and half from the other, because it reads the cryptographically-strong checksums in with each data block and checks them before delivering the data to the application. If the checksum doesn't match, it rewrites that block from the redundant copy on the other disk(s).

RAID can't do that. If one of a mirror's data blocks is corrupted on disk but with a correct ECC, so that the two blocks don't match but both read cleanly, RAID can't tell which one is correct, so it'll typically just force the system administrator to choose one disk to overwrite the other with. That exchanges astronomical odds against incorrect data for coin flip odds against.

wyoung2··on ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot
I've got a ZFS server here that regularly detects some small number of megs of incorrect data on each week's scrub. This week, it happens to be 4.28M. Every week, ZFS finds the correct copy and fixes it.

I have no idea what the problem is with this server. There are no SMART failures or kernel messages indicating hardware failure, and the system doesn't hard-crash. The thing is, I don't actually have to care, because ZFS is actively taking care of the problem. Until one of the disks goes so bad that SMART or the kernel's SATA layer or ZFS can point me at it, I can just passively let ZFS continue protecting me.

If this were a RAID, the first risk is that the RAID system wouldn't have a scrub command at all. Some do, but not all. Without such a command, those on-disk ECCs the author heaps so much praise on won't help him. I've got the same ECCs backing my ZFS, and clearly the data is getting corrupted anyway, somehow.

Let's keep the author's context in mind, which is apparently that we're going to use motherboard or software RAID, since he's budgeted $0 for a hardware RAID card, so the chances are higher that there is no scrub or verify command.

If our RAID implementation does happen to have a scrub or verify command, it might be forced to just kick one of the disks out or mark the whole array as degraded, depending on where in the chain the corruption happened. If it does that, it'll take a whole lot longer to rewrite one of the author's cheap 3 TB disks than it took ZFS on my file server to fix the few megs of corrupted blocks.

And that's not all. I have a second anecdote, the plural of which is "data," right? :)

Another ZFS-based system I manage had a disk die outright in it. SMART errors, I/O timeouts, the whole bit. Very easy to diagnose. So, I attached a third disk in an external hard disk enclosure to the pained ZFS mirror, which caused ZFS to start resilvering it.

Before I go on, I want to point out that this shows another case where ZFS has a clear advantage. In a typical hardware RAID setup, a 2-disk mirror is more likely to be done with a 2-port RAID card, because they're cheaper than 4-port and 8-port cards. That means there is a very real chance that you couldn't set up a 3-disk mirror at all, which means you're temporarily reduced to no redundancy during the resilver operation. Even if you've got a spare RAID port on the RAID card or motherboard, you might not have another internal disk slot to put the disk in. With ZFS, I don't need either: ZFS doesn't care if two of a pool's disks are in a high-end RAID enclosure configured for JBOD and the third is in a cheap USB enclosure.

The point of having a temporary 3-disk mirror is that the dying disk wasn't quite dead yet. That means it was still useful for maintaining redundancy during the resilvering operation. With the RAID setup, you might be forced to replace the dying disk with the new disk, which means you lose all your redundancy during the resilver.

Now as it happens, sometime during the resilver operation, `zfs status` began showing corruptions. ZFS was actively fixing them like a trooper, but this was still very bad. It turned out that the cheap USB external disk enclosure I was using for the third disk was flaky, so that when resilvering the new disk, it wasn't always able to write reliably. I unmounted the ZFS pool, moved the new disk to a different external USB disk enclosure, re-mounted the pool, and watched it pick the resilvering process right back up. Once that was done, I detached the dying disk from the mirror and did a scrub pass to clear the data errors, and I was back in business having lost no data, despite the hardware actively trying to kill my data twice over.

There are still cases where I'll use RAID over ZFS, but I'm under no illusions that ZFS has no real advantages over RAID. I've seen plenty of evidence to the contrary.

wyoung2··on ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot
The trivial case is that the data is corrupted in RAM prior to being written. If we take the simple case of a 2-disk mirror, the same wrong data is going to be written to both disks, the checksums will match, and the filesystems and underlying disks will be oblivious to the problem. ZFS can't help here, but neither can RAID-5.

The far more risky situations involve reading back data.

A properly-optimized RAID or RAID-like system will read half the blocks from one disk and half from the other when dealing with a 2-disk mirror.

With RAID-1, if the data blocks read cleanly from one disk — that is, the hard disk's ECC does its thing, as the author expects — but the data bytes are then corrupted in RAM during the DMA transfer, RAID won't detect the problem. Your application will simply have errors in those blocks, and it'll be oblivious to the problem unless there is some corruption detection ability in the data format.

With a ZFS mirror, things are different. If the blocks are cleanly read from the disk (again according to those in-drive ECC checks) but the bytes are corrupted during the DMA transfer to RAM, ZFS will detect it, because it always double-checks the hashes — cryptographycally-strong hashes, mind, not CRCs, as the author misstates — after reading the data in from disk. This will cause ZFS to attempt a second read from the corresponding block in the other side of the mirror. Assuming you don't get a second RAM corruption, the checksum will match this time, so ZFS will re-write the clean block to the first disk. ZFS is incorrectly assuming it was the drive that corrupted the block, but it doesn't matter because all that happens is a correct block is overwritten with the same correct block.

Now let's take a trickier case. What if your RAM is so flaky that it re-corrupts the clean block on its way back out to the first disk during this unnecessary re-write? ZFS will write the correct checksum along with that block's data, so that when it comes time to re-read that block, the checksum won't match the data. It doesn't matter whether the RAM corrupts the checksummed data or the checksum itself, because the odds are astronomically against both being corrupted in a way that causes the two to match. When ZFS is told to re-read that corrupted block, either by the application or by a background scrub, it will again decide it needs to overwrite the first disk's copy of the block with the copy from the second disk, which this time is in fact corrupted on-disk. Unless your RAM corrupts the data a third time, this time it will write the correct data to disk.

RAID can't do any of that. All RAID can do is say, "These two blocks don't match each other, but both have good on-disk ECC, so PANIC." Different RAID implementations do different things here. Some will just mark the array as degraded and force the operator to choose one disk to mirror back onto the other. If the operator guesses wrong, you've got two copies of the bad data now.

ZFS doesn't have to guess: it knows which copy is wrong with astronomical odds in favor of being correct.

wyoung2··on Red Hat Enterprise Linux 7.4 Released
About a month:

  7.0: 4 weeks
  7.1: 3 weeks
  7.2: 4 weeks
  7.3: 5 weeks
Source: https://en.m.wikipedia.org/wiki/CentOS#CentOS_releases
wyoung2··on Red Hat Enterprise Linux 7.4 Released
Another is the 3ware 8000 series. We got a dead machine in recently for repair and were faced with either moving the RAID drives and card over to a new server chassis while being restricted to the EOL'd CentOS 5 or upgrading to CentOS 7 but being forced to buy a 9000 series card and rebuild the array.

Lose-lose.

← PreviousPage 4 of 6Next →