I ended up writing a query to delete duplicate history entries older than X days from my Firefox history to solve the slowness problem, I would imagine you could do something similar if you really wanted to remove all but visited during X.