The adapter model is designed so adding a new site is a single YAML file (~30 min of work, or ~3 min with a coding agent). No Python code needed for most sites. PRs welcome if there's a site you'd want to see!
241 karma · joined August 28, 2015
The adapter model is designed so adding a new site is a single YAML file (~30 min of work, or ~3 min with a coding agent). No Python code needed for most sites. PRs welcome if there's a site you'd want to see!
web2cli makes direct HTTP requests using your browser cookies. No Chromium, no Selenium, no headless anything. The tricky part was TLS fingerprinting - Cloudflare blocks Python's default TLS stack (JA3 fingerprint mismatch), so web2cli uses curl_cffi with BoringSSL to impersonate Chrome at the TLS level. X.com was even harder - their search requires a cryptographic nonce generated by obfuscated browser JS, which the community reverse-engineered.
Six adapters today: HN, X, Discord, Slack, Stack Overflow, Reddit. Each adapter is a YAML file - writing a new one takes ~30 minutes (or ~3 minutes for your coding agent) and doesn't require Python code for most sites (although it's possible to add a custom python provider, like I did with X).
I'm working on web2cli Cloud - think "OAuth for sites that don't have OAuth." Your users log in via a sandboxed browser, your agent gets an opaque session token, cookies never touch your server.
Happy to go deep on the adapter architecture, anti-bot bypasses, or the economics of browser automation vs direct HTTP.
How to access one’s compute? Overriding GPU drivers?
- Database credentials inside the notebook
- Can't create final dashboards which are refreshed every X hours (which I'm doing in Redash)
I was thinking about joining their commission, so they could work together and help each other.
Is there any other path for an immigrant with a company incorporated in the US - paying taxes, books are right, etc.
Right now I'm working on a tool to achieve such kind of automation of a website, but without code. Mostly without the glitch part from the post. Working on any kind of hosting, no matter what underlying technology - WordPress, GH pages, whatever. With interface similar to IFTTT where you connect 3rd party apps to a website to populate content - like in the example from the post - data from Instagram on the website. Or data from Slack bot on website (D&D example). UX for this project is not an easy stuff. Also, we are thinking what services are most important to be on the platform at the very beginning. We would love to hear any kind of feedback. If you are interested find my email in my bio.
Any other suggestions about similar books where author is wondering what might happen next? I'm mostly interested in the tech part of it. Recently, I started reading Superinteligennce by Nick Bostrom. So far it's very good.
It was a solo attempt at the beginning. Right now there are 5 of us working on it and building a business around it.
Folks from Code for America were using it for very simple CMS to show dates of some events.
Guys from GeneralUI are using this for a quick solution for prototyping and have forms up and ready without setting up middlewares, servers, etc.
A customer from imgur is using this for some user research process (getting feedback).
Basically, most of them are using this in places where marketing or UX meets IT. Or there's no budget for IT to be involved, and they need to setup something on their own. We are trying to get all of those awesome use cases and make stories and blog posts from there. We are really amazed by some of them, what people can build if they need to.
From the other hand, it's hard to map JSON to a spreadsheet, because - what to do with the nested keys. Any ideas?