You only need to download tokens for people who tested positive in the recent past (14 days - assuming you're not infected, any contact before that couldn't have infected you.)
While that's likely still a lot of data in areas with large daily case counts, at least it won't keep growing indefinitely.
BTW it's one token per day per person, to improve privacy. When someone tests positive, their tokens for the last 14 days are published.