I'm not ruling out that there will be attackers that try to harvest passwords from multiple sites, join those sites by username or email, then try some ML to derive people's password algorithms. But there is a whole class of attackers that just won't bother with that level of sophistication.
If your algorithm is any good, you're already not the slowest person running from the bear (cf. easy passwords or exact-password-reuse).