I agree phishing is the most scary vulnerability for this.
In practice, when exchanging emails with my mom, I'm not concerned about it.
A sophisticated attacker has many easier ways to get into my stuff than creating a fake PortableSecret.
(more detail in a different comment: https://news.ycombinator.com/item?id=34084887)