OK, so your format doesn't solve the trilemma; you have 'static' and 'single-file', but not 'efficient'. Seems like you might as well just go with MHT or SingleFile then...?
35,358 karma · joined April 8, 2009
About me: https://gwern.net/me
What I've written recently: https://gwern.net/changelog
Now, maybe you mean something like, 'a web server could additionally run some special CGI software or a plugin or do some fancy Lua scripting in order to munge a ZIP and split it up on the fly so as to do something like serve it to clients as a regular efficient multi-file HTML page'. Sure. I already cover that in the writeup, as we seriously considered this and got as far as writing a Lua nginx script to support special range requests. But then... it's not single-file. It's multi-file - whatever the additional special config file, script, plugin, or executable is.
Well, yes. That's why we created Gwtar and I didn't just use SingleFileZ. We would have preferred to not go to all this trouble and use someone else's maintained tool, but if it's not implemented, then I can't use it.
(Also, if it had been obvious to you how to do this window.stop+range-request trick beforehand, and you just hadn't gotten around to implementing it, it would have been nice if you had written it up somewhere more prominent; I was unable to find any prior art or discussion.)
And if you choose to require separate files and break single-file, then you have many options.
> surely you could host "warcviewer.html" and "warcviewer.js" next to "mycoolwarc.warc" "mycoolwrc.cdx"
I'm not familiar with warcviewer.js and Googling isn't showing it. Are you thinking of https://github.com/webrecorder/wabac.js ?
Yes. A web browser can't just read a .zip file as a web page. (Even if a web browser decided to try to download, and decompress, and open a GUI file browser, you still just get a list of files to click.) Therefore, far from satisfying the trilemma, it just doesn't work.
And if you fix that, you still generally have a choice between either no longer being single-file or efficiency. (You can just serve a split-up HTML from a single ZIP file with some server-side software, which gets you efficiency, but now it's no longer single-file; and vice-versa. Because if it's a ZIP, how does it stop downloading and only download the parts you need?)
As far as I know, we do not have any hash verification beyond that built into TCP/IP or HTTPS etc. I included SHA hashes just to be safe and forward compatible, but they are not checked.
There's something of a question here of what hashes are buying you here and what the threat model is. In terms of archiving, we're often dealing with half-broken web pages (any of whose contents may themselves be broken) which may have gone through a chain of a dozen owners, where we have no possible web of trust to the original creator, assuming there is even one in any meaningful sense, and where our major failure modes tend to be total file loss or partial corruption somewhere during storage. A random JPG flipping a bit during the HTTPS range request download from the most recent server is in many ways the least of our problems in terms of availability and integrity.
This is why I spent a lot more time thinking about how to build FEC in, like with appending PAR2. I'm vastly more concerned about files being corrupted during storage or the chain of transmission or damaged by a server rewriting stuff, and how to recover from that instead of simply saying 'at least one bit changed somewhere along the way; good luck!'. If your connection is flaky and a JPEG doesn't look right, refresh the page. If the only Gwtar of a page that disappeared 20 years ago is missing half a file because a disk sector went bad in a hobbyist's PC 3 mirrors ago, you're SOL without FEC. (And even if you can find another good mirror... Where's your hash for that?)
> Would W3C Web Bundles and HTTP SXG Signed Exchanges solve for this use case?
No idea. It sounds like you know more about them than I do. What threat do they protect against, exactly?
Beyond that, depending on how badly the server is tampering with stuff, of course it could break the Gwtar, but then, that is true of any web page whatsoever (never mind archiving), and why they should be very careful when doing so, and generally shouldn't.
Now you might wonder about 're-archiving': if the IA serves a Gwtar (perhaps archived from Gwern.net), and it injects its header with the metadata and timeline snapshot etc, is this IA Gwtar now broken? If you use a SingleFile-like approach to load it, properly force all references to be static and loaded, and serialize out the final quiescent DOM, then it should not be broken and it should look like you simply archived a normal IA-archived web page. (And then you might turn it back into a Gwtar, just now with a bunch of little additional IA-related snippets.) Also, note that the IA, specifically, does provide endpoints which do not include the wrapper, like APIs or, IIRC, the 'if_/' fragment. (Besides getting a clean copy to mirror, it's useful if you'd like to pop up an IA snapshot in an iframe without the header taking up a lot of space.)
How does it bypass the security restrictions which break SingleFileZ/Gwtar in local viewing mode? It's complex enough I'm not following where the trick is and you only mention single-origin with regard to a minor detail (forms).
(Not so fun fact: if you punch "filetype:djvu" into Google right now, you can easily page through what supposedly is every DjVu file on the Internet as far as Google knows, which is not many: "In order to show you the most relevant results, we have omitted some entries very similar to the 300 already displayed." I learned this the hard way when I began wondering why a bunch of DjVu fulltexts I hosted never seemed to show up in Google or Google Scholar...)
(On Gwern.net, we call this 'demo mode', and we use it to hide some UI elements after a certain number of interactions or page-views, under the theory that if you've, say, uncollapsed a collapse region a few times, you've gotten the idea, and you no longer need a big obtrusive text label saying 'click or hover to uncollapse'.)
But you could feed that back in! Just write it down. It's all tokens. As you read over descriptions and note down key pieces of family history or per-sibling details, that provides information about better annotating the next video for possible points of interest. And you can chat with the LLM and write down more general principles. It's not like a LLM like Gemini doesn't know an enormous amount about family life and things of sentimental value, and can't make good initial guesses. And when you do this, you still haven't used up more than a small fraction of the context window with these image references and text profiles and principles...
I wasn't suggesting anything about your siblings, but you, who are a developer. I was just talking about the actual download step, not what you did after that. (Obviously you were going to host them somewhere else in some other form. Probably not DVDs but a little quickie website or maybe just a Flash drive with a HTML file index, say, I don't know, lots of options here to make it user-friendly for your siblings on Christmas Day. The hard drive or Flash drive idea has the benefit of LOCKSS, especially if you use up the spare space providing PAR2 FEC.)
> I doubt any model could effectively label locations and people over 20 years of video.
Actually, Gemini is highly promptable with a large context window and a single still image only takes up ~300 tokens IIRC, so I think that you could probably do so! Just include, say, 3 photos of each person over time with a natural language description, and 1 photo of each location, and that might be enough to get back useful labels. Gemini can even do bounding boxes. (Google is quite proud of its vision and video analysis capabilities.) And you can run multiple passes or split up videos etc.
I'm not following here. Even if it was several terabytes of video (digitized at high resolution and minimal lossiness for archival purposes), that's plenty of time to download. Especially if you're a developer who can casually spin up a cloud or dedicated server to proxy through if need be? (And $2k sounds reasonable once you start going through "hundreds of hours" at a bare minimum, and again especially if you're a developer with real opportunity cost.)
Also, as far as the video analysis goes, Gemini might've been a better idea?
If you are serious about this, I think there are so many ways you could clean up, simplify, and calm the Claude Code terminal experience already.
I am not a CC user, but an enthusiastic CC user generously spent an hour or two last week or so showing me how it worked and walking through an non-publicly-implemented Gwern.net frontend feature (some CSS/JS styling of poetry for mobile devices).
It was highly educational and interesting, and Claude got most of the way to something usable.
Yet I was shocked and appalled by the CC UI/UX itself: it felt like the fetal alcohol syndrome lovechild of a Las Vegas slot machine and Tiktok. I did not realize that all those jokes about how using CC was like 'crack' or 'ADHD' or 'gambling' were so on point, I thought they were more, well, metaphorical about the process as a whole. I have not used such a gross and distracting UI in... a long time. Everything was dancing and bouncing around and distracting me while telling me nothing. I wasted time staring at the update monitor trying to understand if "Prognosticating..." was different from "Fleeblegurbigating..." from "Reticulating splines...", while the asterisk bounces up and down, or the colored text fades in and out, all simultaneously, and most of the screen was wasted, and the whole thing took pains to put in as much fancy TUI nonsense as it could. An absolute waste, not whimsy, of pixels. (And I was a little concerned how much time we spent zoned out waiting on the whole shabang. I could feel the productivity leaving my body, minute by minute. How could I possibly focus on anything else while my little friendly bouncing asterisk might finish at any instant...?!) Some description of what files are being accessed seems like you could spare the pixels for them.
So I was impressed enough with the functionality to move it up my list, but also much of it made me think I should look into GPT Codex instead. It sounds like the interfaces there respect my time and attention more, rather than treating me like a Zoomer.
> Pariyatti’s nonprofit mission, it should be noted, specifically incorporates a strict code of ethics, or sīla: not to kill, not to steal, not to engage in sexual misconduct, not to lie, and not to take intoxicants.
Not a whole lot of Pali in most LLM editorials.
I set my nginx to return the Markdown source (which is just $URL.md) for my website; any LLM which wants up-to-date docs from my website can do so as easily as `curl --header 'Accept: text/markdown' 'https://gwern.net/archiving'`. One simple flag. Boom, done.
You want people to keep contributing things like a unit-test that the current codebase fails on, or a design rationale for a feature, or an unenforced invariant. Then, after the expert human signs off on that, verifying that the new unit-test is in fact correct, then your trusted LLMs can solve any coding from there.
So, the ad hoc short-term Github solution is probably something like leaving pull requests enabled, but figuring out how to auto-reject ones which touch any implementation code. Depending on how inconvenient that is (maybe code/doc/tests are too intermingled, like typing declarations), you might try to channel everything through an issue.