If you use an address under someone elses domain, they can. If you use your own domain, you can do that in the fediverse too - with the caveat that with email both range of offerings to do this without selfhosting and technical portability are better. (although the fediverse side of things hopefully will improve)
> I remember there being an attempt to make implement chat using the IMAP protocol.
Deltachat