Jake232

Well, we essentially just sell the data to affiliates, who then get value out of it.

They either copy what their competition is doing, or expand / work on it and use it as a starting point.

We scrape hundreds of thousands of publishers who we know run the popup/redirect ads, with IP's from 100 + countries, spoofing hundreds of different mobile devices / orientations / etc. There's a little more magic to it than just those things, but that's the principle.

We see what popups appear, and then record this info and show it in the tool.

It's actually super valuable to the affiliates/advertisers purchasing popups. They can look at the competition and get a head start on what works.

It's pulling in low 5 figures/month with very little advertising, so there's definitely a demand for it.

I run AdsXposed (https://adsxposed.com) on the side. It's a project that scrapes online ads (specifically popups and redirects) and provides intelligence on them. Which publishers they popped on, which countries, to which devices, etc. It brings in pretty reasonable $$$.

You're correct. However for Rank Tracking these types of queries aren't needed.

Having built a ranktracker before, and still being good friends with people who run one of the biggest rank trackers on the market, using the google API's simply isn't an option.

Simply put, the API never returned the results in the exact same order as the actual search results did.

The best / most reliable rank trackers did (and still do) simply use proxies to get around the google captchas. I've scraped millions of pages from google over the years, and with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily.

This is especially true with the dropping cost of proxies. I'm obviously an exception considering I run several scraping SaaS and have generally specialized in it for years, but I can he hitting x.com with 30,000 different IP's within the hour.

Looks like somebody is using their own proxies for upvoting, 37 points in 7 minutes huh?

Lyft does much more aggressive deals though. When I moved to San Diego ~ 3 months back, they gave me 1month of half price rides. Considering I spend > $1000 per month on rides, that was over $500 of savings.

Can confirm. Launched a project recently with over 500 concurrent PhantomJS workers. Let's just say my hosting bill is significantly more expensive than it was.

It often attracts talented engineers; tech companies are always after talented engineers.

A comment dataset was actually already released a while back.

This caught me out too.

No. The form wasn't allowing xxx.yyy@gmail.com

You should allows period (.) in your signup form email field. I know you can just remove the period and the email will still get delivered to my gmail, but plenty of people likely don't know this.

I have scrapers built in Python that do well over a million pages per day, but that's not really a benchmark you can use. It all depends on the amount of computation required to extract the page data among other things.

You should be able to achieve > 120k per day for sure though. That's less than two per second.

I'm not sure that per-request would work; because the resources that a request takes can vary wildly in resources used / time taken. PiCloud (somewhat similar idea) used to charge based on processing time essentially (down to the millisecond I believe).

I personally think that is the correct kind of pricing for something like this; but monthly plans including X time/requests would likely be a good idea.

Your monthly expenses are likely 3x lower than the location he lives in though.

I'd recommend adding some kind previous work examples (Github maybe?).

You also don't state which city/location you're in; that could be useful for potential employers.

It may also be worth checking out if there's anything in last weeks HN Who's Hiring post to see if anything suits you: https://news.ycombinator.com/item?id=10152809

I think your pricing scheme[1] could put a lot of people off. I fall into the category where I'd be fine on the free tier (< 10 private services), and yet I don't want a free service.

I know if it's free, then it's going to be under some kind of fair usage policy, and you're going to rate limit me or have some kind of restrictions eventually. There's no way it can be sustainably free if I start to push it really hard. I'd prefer to just know the limits upfront, or have some kind of usage based pricing.

[1]. http://stackhut.com/#/pricing

Any kind of API / easy programattic access the results?

Ideal Scenario:

Can iFrame the form on own webapp (pass an ID iFrame in url), use it to collect data through custom form. Then can access result of this data via an API.

Coming at this from the other perspective, we currently work with a US based designer and 3 Russian python programmers that we found on TopTal, and they have been great. I'm not aware of the financials (I don't deal with that), but there's certainly some talented people on TopTal.

"I don't have faith that the company will succeed" - I would personally say this answers the question. Don't work somewhere you don't believe in.

One of my scrapers just alerted me to my link being posted here :). Glad to hear you're still finding it useful, I'll probably update it in the near future too; got a list of things I want to add to it.

I am on Sky in the UK, looks like it is an ISP issue then. I'll keep an eye on you guys though, I'm a big fan of being able to just run functions on-demand like this. I used to use PiCloud extensively before it was bought out.

Tried your public demo's, they don't seem to work.

Cool idea. This could easily be extended to support something like a proxy pool; that way you can rate limit / rotate proxies for X domain globally at this server level. That way it's across all your projects, rather than having to do it on a per project basis.

Adding xPath support as well as CSS selectors would be a good addition.

I feel like that is a fairly unusual usecase; plus you could just set the max distance parameter to the diameter of the Earth.

I can't seem to find the H1B statistics page, maybe I'm just looking in the wrong place though. Got a link?

That is a risk they take. That's what I was saying about transparency. If the Hola website made had made the whole situation more obvious, then users either take this risk or pay the $X a month for an actual VPN service.

I think they could have made it clearer, and changing up their FAQ didn't exactly make a great impression on a lot of users. That said; I think providing they tell users about this upfront, and don't waste their bandwidth on mobile connections etc (as they say they don't) - then I see no problem with this.

It's a way for users to get a VPN for "free". For most end-users they won't even notice/care about the odd HTTP request they're proxying. I think transparency may have been the issue here.