I think the discrepancy here is coming from the fact that you are assuming the password is using English words, where others are assuming completely randomly selected characters from the set /[a-zA-Z0-9]/, plus some special characters, which yields substantially more than 1 bit per character.
https://en.wikipedia.org/wiki/Entropy_(information_theory)#c...
> English text, treated as a string of characters, has fairly low entropy, i.e., is fairly predictable. We can be fairly certain that, for example, 'e' will be far more common than 'z', that the combination 'qu' will be much more common than any other combination with a 'q' in it, and that the combination 'th' will be more common than 'z', 'q', or 'qu'. After the first few letters one can often guess the rest of the word. English text has between 0.6 and 1.3 bits of entropy per character of the message.[6]: 234
Consider the difference in entropy between this totally randomly generated password, consisting of 4 words each from a pool of 100,000:
Correct Horse Battery Staple 4lg(100000) = 33
And this one of the same length where each character was chosen from a bank of 68:
VIW&jubiHZUgBrFA8PI9Vy1E(%G 28*lg(68) = 170
That being said I don't understand your calculation of 4lg(100000). That seems like it can't possibly be correct. For starters it is entirely independent of the entropy content of the words in the dictionary. I could have 100000 strong random passwords of 1000 characters each in my dictionary. Could you explain this a bit, please because there's obviously something I'm missing?
Edit: Aah I think I get it - the assumption is both me and the attacker know the dictionary so the entropy content of the words doesn't matter, the only thing that matters is the joint probability distribution of the combinations I'm choosing. Is that correct?
I think you mean 66. More realistically, 52 for Diceware's 7776-word dictionaries, or 44 for XKCD's 2000 common words.
https://towardsdatascience.com/perplexity-of-language-models...
(I have not checked this claim, it's just what I found from googling "best large-language model character-perplexity 2023".)
A token in a LLM is generally more than one character, so I would guess that the entropy is a bit lower than that. Shannon estimated it at 0.6-1.3 bits/character in 1950 (https://mattmahoney.net/dc/entropy1.html)