3,027 karma · joined January 2, 2012
https://namu.org
If you run the flow in the cloud, you can log into a site by adding your cookies to the cloud flow.
The two pieces that make it harder for someone to simply copy the extension code and run it themselves is (1) Javascript minification and (2) parts that talk to a server.
Someone can try to reverse engineer the calls to a server and run their own, but that's often more work than most people are willing to go through and it's made harder by the client code being minified.
I wish it were possible to open source extensions without running into this problem because I like sharing things I've built (https://github.com/dkthehuman/), but doing so effectively kills the possibility of building a business on top of it unless you maintain two code bases (one free and one paid) at which point it's more trouble than it's worth IMO.
As a solo developer, I don't have the resources to support two versions of the product (one for users with accounts, one for users without) because it'd require maintaining multiple ways to save, load, and share flows.
Thanks!
Of course, that doesn't stop websites from rate limiting you if you try to do too much too quickly. In general, I'd recommend being conservative and automating what you think a normal person would be capable of doing manually.
Link to the Discord is here: https://docs.browserflow.app/
Wrote more about it in this comment here: https://news.ycombinator.com/item?id=29256076
Here's a demo showing it in action: https://docs.browserflow.app/tutorials/tutorial-scrape-a-lis...
(The demo shows scraping each URL but you can perform arbitrary actions for each row of the sheet.)
As for Firefox, it's unfortunately not currently possible: https://news.ycombinator.com/item?id=29256938
There's also support for a few unofficial pseudo-selectors (:contains and :has — see https://docs.browserflow.app/guides/selectors#writing-your-o...) to make selecting elements more reliable.
Hope that helps! Agreed that creating reliable automations for the Web is challenging and hopefully Browserflow will make it easier for many folks.
If you run the flow in the cloud, it'll need some place to persist it so I've chosen Google Drive to start. If there's enough demand, I'd be open to storing it on Browserflow's servers and providing an API to access the files but I'd want to make sure that's something enough customers want before sinking my time into it.
Take a look at the "Take Screenshot" command and feel free to message me on the Browserflow Discord if you need help.
Maintaining flows as sites change is definitely a drawback for any scraping solution, so I built features like generating selectors by pointing and clicking to make it as easy as possible.
Browserflow Cloud has built-in support for rotating proxies and solving CAPTCHAs to get around the issues you mentioned. (They're currently in private beta.)
I added support for residential IPs to handle sites that employ aggressive bot detection, but it's in private beta and so far there hasn't been much of a demand for it. If it turns out that it's not really needed for the use cases people have, I might just remove the feature!
I'm using React for the Javascript!
It was a lot of work translating the extension code to work with headless browsers, but it means you can deploy your flows to the cloud and have it run automatically!
- Ease of use: Most existing tools have pretty clunky UX and are hard to get started on
- Reliability: I've had issues with many tools simply not working, especially when it comes to more complex sites
- Cloud: Browserflow lets you deploy your flows to the cloud to run them on a schedule whereas many tools are local-only
Hope that helps!
The cookies are used for as long as the user keeps them in the cloud flow. (Browserflow doesn't try to be smart and automatically refresh the cookies on your behalf because there are scenarios like using multiple accounts in the same browser, etc.) Most major sites use quite long expiration dates for cookies (a year is fairly common) so there usually aren't issues with cookies becoming invalid for a while.
I've thought about adding support for generating a self-contained bundle of Javascript code from the flow that anyone can self-host and run, but that hasn't been a priority for the current use cases. Will keep it in mind — thanks for the suggestion.
About 14 years ago, I fell in love with programming because it made me feel like a magician. I'd type in some incantations, click "Run", and voila! My lightning-powered thinking rock would do exactly as I commanded — in my case, make a virtual robot named Karel move around the screen in my computer science lab.
Nowadays, casting spells requires a bit more work. Most of our work happens inside complex web apps that each have their own custom spell books (APIs) — assuming they even provide one at all.
Let's take a task like managing your social media accounts. Suppose you want to reset your Twitter account and start from scratch. First step: Unfollow everyone. The problem is that you have hundreds of accounts to unfollow, and you don't exactly want to sit there all day clicking buttons.
If you're not a programmer, your options are limited to what others have built. You can hand over your credentials to all kinds of third-party apps and extensions in the hopes of finding one that works. Good luck.
If you're a programmer, you have more options. You have the power to cast spells. What if we used the official API?
You can sign up for a developer account, get an API key, download a client library, read about how OAuth works for the hundredth time, and then start digging through the API to find out how to complete your task.
That sounds tedious and creating a developer account for a one-off task feels like overkill. What if we simulated user actions in the browser instead?
You can install Selenium/Puppeteer/Playwright, read through its documentation to learn how to navigate and click, open the web inspector to figure out the right CSS selectors, run into some race condition where the elements aren't loading in time, sprinkle in some waits, and puzzle over how to handle elements being added from the infinite scrolling list.
That doesn't sounds too great either. Maybe it'd be faster to manually click those buttons after all...
I built Browserflow to automate tasks like this where people end up wasting time doing what computers do better. I wanted a tool that makes it easy for anyone, regardless of their technical background, to automate any task on any website. It includes the convenience of features like automatically generating automation steps by recording your actions while having the flexibility of letting you tweak anything manually or write custom Javascript. (If you want to see how simple it is to automate the Twitter task above using Browserflow, here's the demo: https://www.youtube.com/watch?v=UnsGTpcA-98)
Building a browser automation tool from scratch and making it robust enough to handle all kinds of websites took much, much longer than I expected. After working on it as a solo developer for 1.5 years, I'm super excited to finally share this with you all. Whether you're a professional developer or a non-technical marketer, I hope Browserflow will make you feel like a magician on the Web.
Would love to get your feedback!
An approach that worked for me was to provide an escape hatch (in my case, giving the user a way to run arbitrary Javascript on the page being automated) so that the built-in commands could be designed for the most common scenarios and users would still have a way to handle gnarly edge cases.