jon_r

If there's one thing I've learned in my years of software development it's that software is the great equaliser.

Everyone is capable of writing crappy spaghetti code.

Give the fuss you kicked up about meanpath earlier I'm not sure you can really claim "one-of-a-kind" here...

That said it looks good and the demo seems to work now, good job!

Just out of curiosity what's the UA of your crawler? Curious to see if it's hit any of my sites.

Well I guess that's the risk you take by using an off-the-shelf theme... your copy differs significantly if that's any consolation...

edit: (this is the theme https://wrapbootstrap.com/theme/beaker-responsive-working-la...)

Cool idea, I still wonder how you'll cope keeping that entire index in memory :)

The layouts don't really look anything alike and I think they may be using an off-the-shelf theme for their landing page. Also their site works... have you launched yet?

I was disappointed to find it doesn't work on my 3G, shame because I quite like the other tapbots

You'd think they could rate limit requests coming from EC2 ip addresses rather than blanket banning, but I guess value wise its cheaper to block all of them given the low likelihood of it being a real person.