One thing I like about PDF is the annotations (notes & highlights) are embedded in the PDF itself. That is not the case for EPUB files, each EPUB reader stores annotations in its own proprietary format.
Although for cases like Kavita, storing in the file would be problematic if multiple users want their own annotations without concerns of data leaking.