That is why the pass phrase is required to have a lot of entropy. Even if you know the algorithm used, you're going to have to guess the pass phrase to verify that. And the pass phrase is harder to guess than most people's passwords are.
Aside from the implementation details that you've raised, I'm not finding as many flaws as I expected in it.