They do work like that. My house was raided as part of an investigation. It was about two years after the search that the DOJ indicted me for conspiracy to commit copyright infringement..
"Privilege" derives from the words for "private law". The privileged operate under different rules from the rest of us.
That doesn't mean they can't ever be taken down, but it does mean that when they are, it looks different than when law enforcement sets its sights on one of us.
I was given an opportunity to be 'interviewed' while being detained. The agents presented a letter from the AUSA basically outlining what they were looking for and explaining how it would be in my best interest to cooperate with the investigation. lol..
There were 9 other people involved, it was pretty big national news at the time.
It was almost two years later that the DOJ indicted me.
The Speedy Trial concerns are completely different depending on if you're in government custody or not.