I actually have the opposite experience, at least with the books I read. Nonfiction is typically highly repetitive and redundant so I find it suits audiobooks far better. Meanwhile fiction has a lot of subtle nuance and rewards carefully reading each sentence.
Of course this depends on what books you read in each category, if it’s a physics textbook vs Colleen Hoover then it’s the opposite.