I know it's a solved problem now, but:

> One that couldn't easily be changed, either, because everyone's clients relied on the exact structure of the markup for screen scraping.

This doesn't seem like that hard a problem. Presumably the robots all sent their own distinct UA strings (as scrapers only get written to imitate browser UA strings if you ban non-browser UA strings.) So why not just filter your logs to find all the known scrapers, serve anything with those exact UA strings "legacy" table-based responses, and serve everyone else the new layout?