The cache contains ephemeral _PUBLIC_ keys, that would otherwise be transmitted to anyone who requests them by the message recipient (perhaps through some encrypted channel, but without meaningful authentication). In essence it's the same thing as PGP's encryption subkeys, which are completely published, but the cache contains more of them as the keys are preferably only used once (there is one difference in that the cache gives only one key at a time and will not give the same public key again as long as it contains enough of them).

So: making the whole thing completely public only enables adversary to match session initializations with receivers, which the server can do by definition as it has to route the messages to correct recipient. (In the case without such central server, anybody observing the traffic could do that, as another role of the central server is to mask sender addresses on the lower protocol layers)