marcell

Yes but I was one of only a handful of people with this access. It was heavily audited. AFAIK no one ever used it inappropriately.

I used to work at 23andMe, AMA

It uses GPT to extract the data: https://github.com/fetchfox/fetchfox/blob/public/src/lib/scr...

I haven't put together a good test framework yet, but qualitatively, the results are surprisingly good, and hallucinations are fairly low. The prompt tells GPT to say (not available) if needed, which helps.

I'm going to try the "generate selectors" approach as well. If you'd like to learn more or discuss just reach out via email (marcell.ortutay@gmail.com) or discord (https://discord.gg/mM54bwdu59 @ortutay)

Right now, not much. The extension is fairly basic in that is just looks at the raw text + HTML and sends it to the LLM.

The benefit of this approach is it's very simple and easy, but the downside is it sends a lot of unnecessary tokens to the LLM. That drives up the cost, slows things down, and hurts accuracy.

I'm working on a few improvements now to improve this.

FireFox versión coming soon!

I can’t predict the future but right now you can use ChatGPT to scrape maybe one or two pages at a time, but it’s harder to do a dozen or a few hundred. So that’s the niche I’m going for.

Try it out and see if you like it. Curious if you think it’s better or worse than ChatGPT for scraping dozens of pages.

If you don’t really care about the “morals” of the proxy services, just sign up for a few and see which one is reliable and has good cost. I used Luminati before, they have rebranded to Bright Data. Another is Oxy Labs.

I do want to offer a cloud version. If it’s something you’d be interested in, please email and maybe you’d be a good early user for it. You’d get 1:1 attention as one of our early users. Email is on the site and in my profile.

Sorry about that! I’ll add this in the next version, expect it to be fixed in a few days.

It’s actually a Chrome extension, and it runs in the user’s browser. You can configure the scraping rate and the app warns against higher rates (more than 5 tabs at a time).

I don’t think any sites are going to get hammered though, even at the fastest rates. The limiting factor is often LLM token rates.

I know! I used ChatGPT to get ideas for the name + logo, and didn't realize the issue until it was too late

One thing to note about FetchFox: it runs as a Chrome extension. This means it has a different interaction with anti-scraping measures than cloud based tools.

For one thing, many (most? all?) large sites ban Amazon IP's from accessing their websites. This is not a problem for FetchFox.

Also, with FetchFox, you can scrape a logged in session without exposing any sensitive information. Your login tokens/passwords are never exposed to any 3rd party proxy like they would be with cloud scraping. And if you use your own OpenAI API key, the extension developer (me) never sees any of the activity in your scraping. OpenAI does see it, however.

> And is there any reliable scraping services that can actually do scraping of those large companies' sites at a reasonable cost?

FetchFox :).

But besides that, the gold standard for scraping is proxied mobile IP requests. There are services that let you make requests which appear to come from a mobile IP address. These are very hard for big sites to block, because mobile providers aggregate many customer requests together.

The downside is mainly cost. Also, the providers in this space can be semi-sketchy, depending on how they get the proxy bandwidth. Some employ spyware, or embed proxies into mobile games without user knowledge/consent. Beware what you're getting into.

Good question, I actually haven't tried it with the image capture approach. I'll give that a shot and see how it performs. I'm planning to try many different AI extractors, and see which performs best.

So far, I've done some un-scientific testing to compare text vs. HTML. Text is a lot more effective on a per-token basis, and therefore lower cost. However, some data is only available in HTML.

I had another request for the exact same thing, actually. I'm planning to separate out the scraping library from the Chrome extension, and this project would be a good use case for that library.

Thanks for the kind words! I have a spend limit in place :)

Try it out and let me know if you like it :). If there's a bug I'll fix it!

More specifically, FetchFox is targeting a specific niche of scraping. It focuses on small scale scraping, like dozens or a few hundred pages. This is partly because, as a Chrome extension, it can only scrape what the user's internet connection can support. You can't scrape thousands or millions of pages on a residential connection.

But a separate reason is, I think that LLM's open up a new market and use case for scraping. FetchFox lets anyone scrape without coding knowledge. Imagine you're doing a research project, and want data from 100 websites. FetchFox makes that easy, whereas with traditional scraping you would have needed coding knowledge to scrape those sites.

As an example, I used FetchFox to research political bias in the media. I was able to get data from hundreds of articles without writing a line of code: https://ortutay.substack.com/p/analyzing-media-bias-with-ai . I think this tool could be used by many non-technical people in the same way.

Scraping is semi-controversial, but in this case it's just a user with a Chrome extension visiting the site. LinkedIn has lots and lots of shady patterns around showing different results to Google Bot vs. regular users to encourage logged in sessions. Many other sites like Pinterest and Twitter/X employ similar annoying patterns.

Imo, users should be allowed to use automation tools to access websites and collect data. Most of these sites thrive off of user generated content anyways, for example Reddit is built on UGC. Why shouldn't people be able to scrape it?

Yea! Check back in a week or so

Surprisingly yes, most of the time. I’ve put in a few optimizations:

1. Remove all <style> and <svg > tags. These rarely add value, and can dramatically increase token counts.

2. For the “crawl” step, I exclusively pull out <a> tags and only look at those. The “extract” step looks at full HTML

3. For now, it only looks at the first 50k text characters, and the first 120k HTML characters. This is to stay within token limits.

The last part will be what I focus on improving in the next version.

The ChatGPT part is pretty easy actually. You can just dump text and HTML and ask it a question, and it usually answers correctly.

The trickier part is “everything else” to make the extension work.

Yes! I actually just had someone else ask about Instagram. Try it out :)

I got these results just now: https://fetchfoxai.com/s/UOqL5HtuNe

If you want to do the same scrape, here is the prompt I used: https://imgur.com/XhguCk4

I'm working on a Chrome extension to do web scraping using OpenAI, and I've been impressed by what ChatGPT can do. It can scrape complicated text/html, and usually returns the correct results.

It's very early still but check it out at https://FetchFoxAI.com

One of the cool things is that you can scrape non-uniform pages easily. For example I helped someone scrape auto dealer leads from different websites: https://youtu.be/QlWX83uHgHs . This would be a lot harder with a "traditional" scraper.

The study found associations at r=0.17 for both BMI and educational attainment. Not a lot but not zero.

The marshmallow test deals with kids so it’s noisy by nature, that there are two mild associations is interesting. It has mild predictive value.

I think there’s a strong desire to have this test shown to be faulty. Perhaps because the test is so easy to do, parents do it on their own kids and don’t like the outcome.

There's very little information here. Is this using some existing medical process? Is it more like a Kickstarter / pre-order model?

Generally the way to hold a president accountable is to impeach him and remove him from office.

Under this ruling, the court system is generally not a way to keep him accountable.

So it’s not that there is no accountability or way to “punish”, a rogue president, it’s just a different method of accountability than what applies to you and me.

Yea but that can be explained by birth control. Before, just sex drive was enough to reproduce. Now you need a different disposition.

Fertility drop is a self correcting problem. The people who are not having kids will be less represented in the future gene pool. Those who are having 3 or 4+ kids, and whatever the associated genes are, will make up the majority within a few generations.

And also, humans probably overshot on reproduction in the 20th century. It was 5-10x growth. Cutting back and focusing on quality for a century or two seems like a good idea.

Trying to proselytize or raise a crisis about fertility is pointless.

> don’t ask why you need to remember four words instead of just horizontal/vertical, it’s still better than before

The reason is "display: flex" can be a column or row. The "align-items" property describes alignment along the axis, so vertical for column and horizontal for row. The "justify-content" property is the other direction.

Is there data that suggests testosterone would work as a male birth control?

Hormonal birth control was introduced in 1970’s, and were hearing about this decades later.

If Wegovy/etc are on a similar scale, how long until we find out those unintended side effects?