2,118 karma · joined November 29, 2016
jonah AT archive DOT org
https://bsky.app/profile/jonahedwards.com
-
4 data centers, 745 nodes, 28,000 spinning disks
Wayback Machine: 57 PetaBytes
Books/Music/Video Collections: 42 PetaBytes
Unique data: 99 PetaBytes
Total used storage: 212 PetaBytesAleator Press (specializing in computer-generated books) also recently published a letterpress book of pseudo-poetry from it: https://www.aleator.press/releases/wendit-tnce-inf
Re slowness, we're able to do a lot with a little, but there's always room for improvement. If you're interested in some of the specific infrastructural challenges, I did a presentation in February:
https://archive.org/details/jonah-edwards-presentation
and my colleague did a fantastic presentation detailing some of the internal workings of the Wayback Machine just last week:
Others like the Enidan J1 were specifically targeted toward people building clusters out of commodity hardware: https://web.archive.org/web/20010511215150/http://www.enidan...
(forecast is for the UTC day, so the prior overnight for NA)
E-mail's in my profile -
(STRIP was actually in the linux kernel, though I assume long gone or defunct -- the only things I still have talking via Ricochet are 2.4-era)
I have a couple of the lightpole radios (they listen for the modems on 900MHz and speak amongst themselves on 2.4GHz) and scored a single tower radio (talks to the lightpole radios and has an ethernet downlink) and have gotten them to speak with each other, but haven't had time to figure out the route mapping steps -- IIRC the deal is that the packet routing path information is stored upstream and delivered to the downstream radios. No idea if anyone ever decoded that format.
I messed around with this a lot in the immediate post-Ricochet era in between flashing silver WaveLAN cards to gold... fun times. It seems like it would be entirely possible now to run a simulated node using SDR.
We try to keep a regular upgrade cycle (to hold to our tight budget), typically with a tick-tock of adding new hardware (expanding within our existing footprint) and cycling out old hardware and drives as they reach the far end of the works/doesn't work spectrum. We have a local partner who takes care of some disposal for us, but we also have no shortage of physical storage space, so we will also accumulate (sometime intentionally -- nearly our entire "red box" deployment -- see "previous version": https://archive.org/web/petabox.php -- is packed into a shipping container. We don't like to throw things away!).
For a hypervisor, we use Ganeti (running over KVM). Because our fleet is so heterogenous we need to be able to control a lot of VM parameters in order to efficiently pack our computational resources, and Ganeti is kind of in a sweet spot for us in terms of providing a lot more tooling than a bunch of virsh scripts, and being much smaller than systems like OpenStack geared towards large, homogenous deployments).
Additionally, as I alluded to in the video, we regularly end up going down into the details and when this happens being able to closely examine and understand a disk's contents as written directly at the LBA using fibmaps and similar tooling is invaluable. I have personally been involved in the discovery of multiple possible-data-loss hard drive firmware bugs, and our catalog system is very paranoid about integrity checking. Modern hard drives are computers unto themselves and layering additional complexity on top of that (particularly complexity which might obscure or silently correct errors in the underlying datastream) is something I approach very carefully.