Tik Tok crawler fucked us up by taking a product-name (e-commerce) and inserting it into the search bar recursively with the results page. Respect the game, but not respecting the robots.txt crawl delay is awful.
We do that with MercadoLibre (way smaller Amazon for latam) and the limit they pagination results to 40 pages/2000 products. Never thought of the faceted search approach.
Thanks for the idea!
Thanks! sounds hard asf. I believe we can handle the inconsistence problem because we want <all> the products (so it'll be really general), however i don't think Amazon can list all their products on the webpage because of the huge amount. It'll be harder to reach the product url's than to scrap them :/