If visiting the site itself is prima facie evidence of wrongdoing -- like child porn -- I disagree. In the case Silk Road, I agree: users could merely be motivated by curiosity and therefore browsing isn't on its face a crime.
What about misspelled domains? DNS Hijacking or Bit Errors? Hostile redirects from innocuous hacked domains? I'm not against using access as evidence, but there needs to be more context than a single attempt to access a domain.