It shouldn’t be. IMAP doesn’t have to download all of them.
Because I haven't, and having everything in one folder works just fine with the clients I use.
Most importantly is the use of a mail indexer (notmuch currently) for searching and getting virtual mailbox viewes.
Right now, on my main mail, I have ~400 mails in Trash, but only 10 in Inbox and Archive + another dozen in Sent.
Right now, I've been looking for the email address of the landlord of the house I lived in until 2009. I'm happy that I didn't remove his email saying that his son would come on Friday to fix the door handle.