HNHacker News
TopNewBestAskShowJobs

white_viel

12 karma · joined September 26, 2016

submissionscomments
white_viel··on GPT Bot Ignoring Robots.txt on my cloudflare worker
but i am impressed that gpt was able to decipher the worke.dev url on its own and access it.
white_viel··on GPT Bot Ignoring Robots.txt on my cloudflare worker
>> interesting. workers.dev domains can be a liability sometimes -- if you've mapped the worker to a real zone, then you probably don't want the workers.dev zone anymore.

maybe that is a good idea, prompt a user to disable the worker.dev domain. however, having the worker.dev domain open for me is more of a backup way to access the worker as the mapped domain is more for a hobby project.

> Interesting. workers.dev domains can be a liability sometimes

what about allowing the specific worker.dev to serve a robots.txt on its own that can be used to disable the AI bots?

i.e https://XXXXXXXXXXX.username.workers.dev/robots.txt should be allowed to be configure on a worker level. not sure how that affects the design of the infra, but it would be a good idea

white_viel··on GPT Bot Ignoring Robots.txt on my cloudflare worker
Finally managed to kill the traffic: 1. renamed the worker - the bot was bypassing the route on the domain and using the worker.dev domain 2. add WAF rules to block gptbot in user agent 3. serve zip bomb on request from GPT.

interesting that GPT was accessing the worker directly to bypass the WAF rule son Cloudflare

white_viel··on GPT Bot Ignoring Robots.txt on my cloudflare worker
update: the bot is back now with a vengeance, sending request at about 1 request per second. ignoring robots.txt and the status code 403
white_viel··on GPT Bot Ignoring Robots.txt on my cloudflare worker
serving a zip bomb and after 10 minutes, the traffic from the gpt bot disappeared..
white_viel··on GPT Bot Ignoring Robots.txt on my cloudflare worker
will be serving a zip bomb to the bot to see if they stay away from my proxy
white_viel··on Plex Update: Notice of a potential security incident
Jellyfin - an almost direct replacement of plex Emby - IMOO the best replacement if Plex is not viable
white_viel··on Why's that company so big? I could do that in a weekend
what about the bus stop broadcasting a signal with the id, name etc? the system is activated by the proximity..so you can have smthing like approaching station A, leaving station A.

it is scalable because the bus-stop transmitters are added , moved, removed as needed.

updating the data can also be done remotely