nikisweeting

It's going to get worse before it gets better:

- https://docs.sweeting.me/s/ai-scraping-will-force-us-towards...

- https://docs.sweeting.me/s/so-you-want-to-build-a-social-net...

I've worked at multiple AI browser companies at this point and I think the best way forward is to start paying for content again honestly. Brave and BAT were too early but we need more things like that with lower friction and more primitives baked into the browser. The AI bots that create value for people and have money will happily pay to get access to the content, keeping it alive in the process and paying for the maintenance cost of allowing good bots.

DBSC, DPoP, and Google's latest "Google Cloud Fraud Defense" reCAPTCHA using phone are another approach where you use hardware secure enclaves to prevent cookies from leaving devices and attest to 1 visitor = 1 physical device. I don't like these because I want my personal agents to be able to access the things I can on my behalf. If I misbehave then block me & bots, not all bots, this is not as impossible as companies make it sound.

The worst possible approach is to require biometric or government ID, that's the one we all need to push hard against, as it's bad for everyone involved.

Screenshot is better for web content because OCR is not difficult anymore.

https://github.com/archiveteam/grab-site works well too, but neither are great with JS

When you get to that point just upload them to archive.org before you delete them

The ArchiveBox Browser Extension https://extension.archivebox.io/ has this built in. It captures both full MHTML and screenshots and uploads it to your server automatically, it was added a few months ago.

Also works from iOS and Android so you can save logged-in content directly without having to set up cookie sync (which it also supports now too directly from the extension).

Have you tried ArchiveBox since v0.9 was released? v0.9 was basically a complete rewrite with a new archiving engine, and it ships with ArchiveWebPage built-in as one of its plugins now.

As of the latest v0.9.77 it's now tested to be able to handle 2 million snapshots / 2TB on a 4CPU server, archiving @ 1 URL/min consistently while serving all the web UIs @ ~100ms. Also has automatic importing from social media, google docs, notion, canva, figma, and a growing list of sites with self-healing AI scripts & cookie sync from your browser so it doesn't need as much babysitting.

Amazing engineering work! Bought a license just to support it as well.

Can't wait for webmail next!

All the major browser agents already mask secrets in screenshots and use placeholders + redact the dom summaries, I guess the extra scrubbing of faces and other PII could be useful though.

Why not build it as a more general redaction library that can be plugged into any agent loop instead of as a whole agent of its own?

Yay glad to hear it! Feel free to open Github issues with any other feedback in the future!

Ok new compact UI is out in https://github.com/ArchiveBox/ArchiveBox/releases/tag/v0.9.6...

(but make sure to install the newer release after that added some more bugfixes)

let me know your thoughts

Fixed on dev and rolling out now. I didnt remove the preview images completely but I made it all significantly more compact. Down to 1 line with colorized icons for the most important outputs.

Ok thanks, super helpful.

I will push a fix today for the broken links and I'll try and make the icons more compact / more similar to the old style.

Do you want the preview screenshot/thumbnail on the left also smaller/hidable or is it just the icons part that you want closer to the old format?

Yes, it was (still is) an annoying Chromium bug that affects tons of browser automation projects, not just ArchiveBox.

But v0.9.x now has built-in chrome zombie killing at the start of every crawl, so it shouldnt happen anymore even if Chromium misbehaves! Definitely open a github issue if you see any performance issues at all!

Absolutely can definitely fix this relatively quickly, a few questions:

- did you do the full filesystem migration yet with `archivebox update --migrate-only`?

- what OS and screen size are you viewing on? and are you looking at the admin snapshot list or the public snapshot list?

- is it just the compactness of the old emoji icons on one line you preferred? or something more than that? the reason we changed it is because we have 20+ more icons now, so it no longer fits on one line easily and we cant easily represent every plugin with only common emoji

If you're able to share one of your old snapshot folders in a new Github issue that would be super helpful as then I can reproduce what you're seeing quickly.

It’s been a long time since our last major release back in 2024!

Thanks for hanging in there, but I promise this release was worth the wait!

In our last big announcement in 2024 we promised a plugin marketplace, P2P archiving by connecting servers together, and better UI / peformance. https://news.ycombinator.com/item?id=41860909

I'm pleased to share that all those promises have now been delivered! Please let me know your feedback and open issues, I'm excited to hear how the community uses this release (especially because we have 0 analytics, comments are the only feedback we get).

there are tricks to avoid needing distro support / kernel modules for ZFS: https://github.com/pirate/zfsbox

also tricks to make it easy to convert a root FS to ZFS now that Ubuntu Server 24.04 added native root-on-zfs support: https://github.com/pirate/zfsify

For those into deriving the laws of physics from pure math and game-of-life style iteration of simple rules, checkout the Wolfram Physics Project: https://www.wolframphysics.org/

https://github.com/pirate/wolfram-gauge-physics

which is rapidly becoming a game of steganography cat and mouse

You can choose what you pin when you run a node, no one can force you to host stuff you don't want to host.

I built this because I've struggled with agents making too many implicit assumptions during development.

They assume they need to write tests to ensure a feature is removed!

They assume they need to implement compat whenever an API changes!

They assume they must implement something from scratch when it may already exist!

Instead of interrupting their work and issuing corrections via the chat, I figured we could pick out these implicit assumptions and show them as an autogenerated form-style UI in the lower right. That way as I tab around to different apps I can keep one eye on the stream of (potentially incorrect) assumptions my agents are making, and correct them with one click instead of having to type in correction messages.

It works with Codex in the ChatGPT desktop app or CLI in iTerm.app for now, in the future I might add Claude support or support for Ghostty / other terminals.

and the US famously never subsidizes any of its industries...

If you want an $18 DIY / open hardware version of this using an LED cube off amazon & WLED, here you go: https://github.com/pirate/led-cube-agent-monitor

Strong disagree, I use tools like DeepWiki, Mintlify, Cubic daily to analyze complex codebases. Their diagrams on PRs and generated docs have made it much easier to quickly drop into a complex codebase and understand how it works / make meaningful changes. Especially DeepWiki's diagrams + interactive "Deep Research" / Codemap feature are amazing at building flow diagrams to answer specific questions.

I recently did a complete rewrite of our browser driver codebase by deeply analyzing playwright, puppeteer, selenium, and the chromium source code, relying heavily on auto generated diagrams to decide which parts to read myself / dive deeper on. Yes they occasionally leave things out if the underlying codebase hides the complexity or has no docs, but they're great for thoroughly documented mature codebases.

I have never seen anyone pull off seccomp nested sandboxing of Chrome in Docker before, if you manage to figure it out please let me know!

Singlefile supports scoped recursive crawls too: https://github.com/gildas-lormeau/single-file-cli#:~:text=an...

I highly recommend reading the singlefile source or https://archiveweb.page/ to see how they handle closed shadow DOMs, cross-origin iframes, websockets, media urls, deduping large assets, etc.

Docker is designed to be undetectable by default, the best way I have found is to set env IN_DOCKER=True manually in your Dockerfile + check that there is no $DISPLAY configured + that you're on linux. Usually if all/most of those are true you can safely add --no-sandbox --disable-setuid-sandbox --disable-dev-shm-usage etc. all the docker-specific flags. Thats what we do in https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile...

https://github.com/archiveteam/grab-site or browsertrix may be easier to use for some, it's what was used to save a lot of the data.gov stuff before it got taken down.

--no-sandbox is needed in docker, maybe they assume it will mostly run in docker?

I just added this to Lima recently, hopefully the macOS container machines support it too: https://github.com/lima-vm/lima/pull/4866

Very curious if they're implementing browser driving themselves or using an off-the-shelf library like stagehand, browser-use, etc. to drive the DOM. Hopefully they open source it if it's in Swift.

A11y-tree alone is not enough for many sites because lots of auth stuff happens in OOPIF frames that need special handling/stitching/interactive element filtering.

There's also the issues of many captchas around auth stuff being implemented using canvas elements (that are hard to instrument for browser agents without relying on CUA). Can their on-device 3B model really handle accurate CUA driving? I guess we'll see...