matsche

paperoni is awesome!

I once implemented the crossref api in rust in order to search for publications from the command line https://github.com/mattsse/crossref-rs

Yes, you can either limit how many request can be send concurrently or enforce a fixed or random delay in between consecutive requests.

Basically this a just a futures_timer::Delay [0] that is reset after each request, which is non blocking.

[0] https://docs.rs/futures-timer/3.0.2/futures_timer/

Thanks, you're right. I fixed it just now by also requiring the tokio feature for all the tests in the Cargo.toml.

The guys from corridor crew did a great break down of that amazing scene [0]

[0] https://youtu.be/pTl42COCNaM?t=255

I haven't used scraper too much. I personally find the predicate approach of select.rs [0] easier to use. However in this case the selector approach just made more sense. Standalone html5ever can be a bit cumbersome to work with directly, scraper is basically an implementation of the html5ever's `TreeSink` trait, where as `select.rs` uses the hmtl5ever `RcDom` to parse the document but stores it in a more convenient way. If you look for a minimal approach you should at select.rs which basicially only depends on html5ever

[0] https://github.com/utkarshkukreti/select.rs

tokio support just landed https://github.com/mattsse/chromiumoxide

Unfortunately this crate has been abandoned by its author for quite some time now. The main differences are that are that chromiumoxide uses async rust by design and that all the devtools protocol types are generated.

rust-headless-chrome was very helpful for some parts and bootstrapped some parts of chromiumoxide, such as the key and mouse clicking. However I looked at chromedp and puppeteer itself for the most part.

> The pdl files are the canonical protocol definitions and are maintained manually by the devtools team [0].

The JSON protocol definition files are then in fact being generated from those pdl files. You find them mirrored in the devtools-protocol-repo [1]

[0] https://chromedevtools.github.io/devtools-protocol/ [1] https://github.com/ChromeDevTools/devtools-protocol/

I started that way, but that was more work than writing a generator. Luckily I found https://github.com/chromedp/pdlgen

It uses async-tungstenite [0] for the websocket part. At the moment configured only with async-std-runtime but since async-tungstenite also supports tokio, the option to use tokio instead of async-std will be added as well.

[0] https://github.com/sdroege/async-tungstenite

It only supports the chrome devtools protocol[0]. chromiumoxide is therefor more like a (limited) puppeteer rust port. Support for javascript page injection is currently being worked on

[0] https://chromedevtools.github.io/devtools-protocol/

it baffles me that it had to come this far. There were warning signs all over it.

The German BaFin messed up so badly that one must either fundamentally doubt theircompetence or question their integrity.