It was mainly for SEO, and to some extent protecting against website scrapers and crawlers.