Testing, rendering, scraping, streaming, botting, and many more.

Think "what could I accomplish with a browser, with the slow human replaced by a fast program" and let your imagination run wild.

The space is interesting enough that people have jumped through a lot of hoops to make it work in the past; this makes it one less hoop.

Oh, and if you ever wondered why web captchas are a thing, one of the reasons is headless browsers.

A great example would be PDF generation for things like invoices. Rather than generating a pdf with something like PHP or Java. Render a regular html page with all the css you want (super easy compared to drawing a PDF using PHP) and then proceed to use a pdf printer on that page.

You could run such a thing as a microservice using a headless browser or PhantomJS. There are probably better ways to do this but that's one of the first things that popped into my head!

A few little personal things I've done with PhantomJS:

• A script that would go to Comcast's TV schedule for my area and make a list of all movies upcoming in the next two weeks on all channels that are included in my subscription. I could then grep that for a list of movies I've been looking for.

I couldn't just grab the page with curl and parse it, because JavaScript does most of the work. JavaScript fetches the listings, and when you advance the listings it it fetches the new listings and replaces the old ones on the page.

• A script that goes to the FCCs license information site and gets a list of all ham radio callsigns issued recently [1].

• A script that given a URL to a tactics problem on lichess gets the FEN for the position. I'd use this if I was doing tactics training there on my iPad and did not understand why my answer was wrong or why their answer was right.

I'd mail myself a link to the problem, and then later on my desktop I'd give that URL to this script and it would go to lichess to the problem page, and then from there to the board editor page for that position, and grab and give me the FEN, which I could then use to set up the position in Stockfish to analyze.

(This is no longer useful. They have made some changes at lichess and now they have a browser-based version of Stockfish on the problem pages, so I can answer my questions right there).

• A script that goes to everquest.com and gets the server population levels from the server population display on that page.

I don't think that there was anything in this one that actually needed a headless browser. As far as I recall it could have all been done with getting the page with curl and parsing it. It was just easier to do it in JavaScript using the DOM. (The lichess one may also have been that way).

[1] https://github.com/tzs/todays_hams

Web scraping for sites that discourage simple robots by checking for JS or serving content via JS.

Also, the darker stuff like click fraud and all the other kinds of fraud where you pretend there are humans doing something when in fact there's just a bot.

I wrote a PhantomJS script to download data from my bank accounts. They offer no API and disfunctional text-based exports, their websites are ridden with "good" (=terrible) web & security practices, like 3-characters-of-the-whole-password authentication, single-tab sessions, frames, etc. that makes it pretty much impossible to scrape with Python but relatively easy with a fully-fledged browser (although that still requires a lot of bank-specific boilerplate code).

I use https://github.com/bfirsh/needle/blob/master/README.md for automated UI regression testing. Using a headless browser means your test suite can run faster and with fewer dependencies.

Downloading any kind of web page where a simple wget or curl turns out empty, for instance anything made with React or other advanced JS frameworks.

Server side automation of multiple kinds (not just tests): screenshots, advanced crawlers, etc.

At work I was given permission by a vendor to screen scrape their site while they worked on building a real API. This site was extremely dependent on javascript. Including doing some really complex token passing between multiple domains that the company owned. Not to mention all of their js was minified and uglified so I had a very hard time understanding what it was doing.

It was the first time I wasn't able to successfully reverse engineer a site enough to scrape what I needed with just requests/beautifulsoup. I was however able to get it working just fine using phantomjs via selenium via splinter. It was a fun exercise, but part of me still feels like it was cheating.

While we're on the topic, does anyone know where one might find scripts to scrape bank statements so you don't have to download them manually every month? (This is one thing I would find headless browsers useful for...)

I used PhantomJS as part of a report generation pipeline which served no HTTP requests and contacted no outside servers. We made PDFs and ready-to-email, single-file HTML reports with some minimal interactive features. (Ready-to-email, single file == all images turned into data URIs, styles inlined, for HTML files sent as attachments)

PhantomJS loaded up an HTML file written earlier in the pipeline. The HTML consisted of a big slug of JSON containing all the relevant data (which would vary from one run to another) and a bunch of scripts and templates (which were fixed for any given report type). The scripts built into the HTML file would chew up the JSON slug and build up the DOM required for the report. Then the PhantomJS script would identify all the images in the DOM and replace all of them with data URIs, strip out the JSON slug to prevent giving away more data than contained in the DOM, and strip out all of the templating JavaScript, leaving behind only the JavaScript needed for the interactive features, which was inlined.

We went with PrinceXML for PDF generation. I was briefly nervous because I saw people praising PhantomJS' pdf generation capabilities... but then I saw the people saying, "we used PhantomJS for pdf generation, we used wkhtmltopdf, then we just paid some money to get something that wouldn't produce weird output some of the time." CSS Paged Media Module FTW, y'all.

We used it a lot for full automation tests for the UI. It's nice being able to interface with a full-featured browser that can run javascript, etc. And take screenshots when things go wrong.

We have a complex matrix of layouts and styles an user can chose from and we need to test them across all browser to make sure any improvement doesn't mess with others.

It's way cheaper to launch a browser headless at all the resolution we need and grab screenshot to visually compare them at glance, instead of goin one by one at hand

>Are there people who earn money by getting it to automatically fill out forms, enter competitions etc?

Yes. Tiny ex. https://news.ycombinator.com/item?id=1165680

My favorite use is automated browser testing. Examples:

- in Ruby, poltergeist (https://github.com/teampoltergeist/poltergeist)

- in Elixir, hound (https://github.com/HashNuke/hound)

We automate buying things on sites like Amazon.

I use it to automate turning webpages in pdf.

Automated tests.

I have used them in the past to convert graphs and reports to JPGs and PDFs so that I can automatically email them to people in the company who are unable (or unwilling) to use a web page.

Server side rendering, testing, automation just to name a few.

Automated browser testing on a CI server that doesn't have a gui.

automated testing and web-scraping mostly.