pronoiac

Page turning? So, non-destructive, with cameras? How's the quality?

I got a copy of the 30-year old book from EBay or Amazon for $20, chopped the spine off, and fed it through a scanner. Doing that to a century-old book feels wrong!

ScanTailor was tricky to start with; dunno if there's a manual. I remember belatedly realizing that there's automation at each step, that one can then quickly skim and manually adjust.

For collaborative editing, git via GitHub worked for us. Tracking who did what, and when, is easy. It allowed for sweeping edits covering multiple chapters. Building some porcelain on top of that, for less technical folks, could be good.

Oh wow! I've worked on turning PAIP (Paradigms of Artificial Intelligence Programming) from a book into a bunch of Markdown files, but that's "only" about a thousand pages long, compared to the roughly 27000 pages long of all those volumes. I have advice, possibly helpful, possibly not.

Getting higher quality scans could save you some headaches. Check the Internet Archive. Or, get library copies, and the right camera setup.

Scantailor might help; it lets you semi-automate a chunk of things, with interactive adjustments. I don't know how its deskewing would compare to ImageMagick. The signature marks might be filtered out here.

I wrote out some of my process for handling scans here - https://github.com/norvig/paip-lisp/releases/tag/v1.2 . I maybe should blog about it.

If you get to the point of collaborative proofreading, I highly recommend Semantic Linefeeds - each sentence gets its own line. https://rhodesmill.org/brandon/2012/one-sentence-per-line/ I got there by:

* giving each paragraph its own line

* then, linefeed at punctuation, maybe with quotation marks and parentheses? It's been a while

As the credit's left to the end: this was written by Bruce Sterling.

I've dealt with something similar. We were able to spin up zombie reapers, looking for the cores / CPUs that were pegged at 100%, and prioritize the instances that were worst hit.

I guess this works right up until it doesn't? It's been a while, but I've seen AWS hit capacity for a specific instance size in a specific availability zone. I remember spot pricing being above the on-demand pricing, which might have been part of the issue.

Oh, writing my own Pinboard archive exporter is somewhere on my too-long to-do list. I should find out what would be good for importing into Archivebox. (WARC?)

I watched the 1997 Baz Luhrmann version of Romeo + Juliet online with some friends relatively recently, and I'm sort of proud that we managed to not spoil the ending for those who had never read or watched it.

Note, this blog post has spoilers for:

* Alien (1979)

* Close Encounters of the Third Kind (1977)

> Can a Flipper Zero be programmed for this? It connects to Micro SD cards and USB ports.

I wrote something barebones for this on my Synology, using just a shell script and (hardware accelerated) openssl, if memory serves; that acceleration was crucial for handling, say, an 8TB hard drive.

I'm midway through, but a nitpick:

> You're on your own figuring out the Commit IDs and punching them into the URL bar if you want something more granular.

There's a "commits" tab at the top, like: Conversation | [Commits] | Checks | Files changed

Example: https://github.com/raspberrypi/linux/pull/6330/commits

I emailed him.

The "Conan O'Brien Needs a Friend" podcast posted video for this segment: https://www.youtube.com/watch?v=eSjdYkqY5WI

Here's the article linked as https; I'll email the mods.

https://cantrip.org/sortfast.html

> I was unable to find a way to reach Ed Krol or Bruce Sterling ...

Bruce Sterling was on Twitter when this was written, and I follow him on Mastodon now. I'll contact him there.

You need at least two leading spaces to get that to render as code and preserve the layout, like:

     Out of office: 
            [John Doe]
                         [Stacy]
     Release:
            [code freeze]
                           [Staging date]
                                            [Release!]

Are you talking about lining up the pipeline symbols for columns? If it is, have you tried the AlignTab package?

I recently was running a variety of compressors over the course of days, and using Bash was great because, it doesn't parse the whole file before it starts running - I could edit and reprioritize next steps while the script was running. "Oh, that one's not working, skip the slower version" or "oh, space is an issue, work on freeing that up." It's a weird hack I wouldn't suggest in production, but it worked well for me.

Someone reached out to the FOIA liaison asking how an individual or organization might help, and got the reply that it's outside their scope, and to contact the NSA directly. Writing a paper letter to the NSA and cc'ing congresspeople might be good.

There were two sets of bugs:

* the new sort handled ties differently. They adjusted that, and they could have stopped there.

* the other was that the correct drivers were sensitive to loading order. They handled this and another driver bug, amdtemp. This was masked by the old sort.

If they'd found this years later, even investigating the first set would be a slower process - it wouldn't be fresh in minds, and it likely would then have other code relying on it, so adjusting it would be trickier.

I've used https://crt.sh to check my own logged SSL certs before, though it looks like they're seeing issues right now.

Oops! I edited my comment - moving the paragraph break - to make more clear that I got them used.

I recommend a NAS; you'll be in much better shape if you have a single drive fail. It's much easier to organize than some random external drives.

Am I parsing it right, that you want 24TB of storage? You could do 3x12TB or 4x8TB. Aim for CMR drives, not SMR; SMR is great at intermittent usage, but not certain parts of NAS, like filling a new drive. Try to get different brands, models, or at least batches; there have been firmware bugs hitting at a certain uptime.

You can get used drives; test them and verify their capacity. I used https://serverpartdeals.com , and they replaced one that didn't work well; they have 12TB drives for $97 right now. I didn't verify they're CMR but I'd be surprised.

Amazon and Newegg have a Synology 4-bay NAS for $400 or less. Mine has been easy to administrate locally. You might work from Synology's hard drive compatibility lists.

A rough total for those is about $700.

I expected general-purpose compressors to do better, but I was wrong. For the whole document:

  174355 pg11.txt
   60907 pg11.txt.gz-9
   58590 pg11.txt.zstd-9
   54164 pg11.txt.xz-9
   25360 [from blog post]

Buying refurbished also makes it much easier to avoid having the same brand/model/batch/uptime, for firmware and hardware issues. I do carefully test for bad sectors and verify capacity, just in case.

Looks like the Internet Archive got a copy, mirror: https://web.archive.org/web/20240430041911/https://only9fans...

Archive copy: https://web.archive.org/web/20240429061925/https://csvbase.c...

Mine all feel like "we do these not because they're easy, but because we thought they would be easy".

Turning a thousand-page book - PAIP - into a stack of Markdown files in a git repo, readable online. The print book received more editing and revisions than the ebook. I converted an ebook's ... odd formatting ... into Markdown, remade diagrams, generated new ePub and pdf files, and had the spine cut off a print copy to make a fresh scan. Working on that scan, I made Scantailor, an X program, easier to access from a Mac, via Docker. I tried different OCR engines, and pored over the diffs, incorporating dozens (hundreds?) of improvements. I got to find so many differences between Markdown engines. I have ideas on how to make Pandoc links between chapters. There's still a lot to do!

My current WIP: Lars Wirzenius posted about file systems with a billion (empty) files. I started exploring because I was curious, if I was remembering correctly, how well a mostly empty image file would recompress - like, drive_image.gz.gz. Lars offered a Rust program; I was curious about how other methods compared. Like, how about nested shell loops, tar, and touch? And, hey, how well can we archive and compress them? I've gotten to see some issues, bottlenecks, and outright failure modes with SMR hard drives, Samba re: sparse files, and parallel gzip compression. I've accumulated some shell script boilerplate to make it easier to go back and verify my processes, and harder to accidentally wipe out past work if I rerun it.

@'s don't do anything here. I've emailed the mods.

I have a rough draft, I'll see if I can make it presentable.