You do realize you’re saying “It’s not worth including some people,” including those with hearing loss, as if their inclusion is worth less than everyone else.
Inclusive design recognizes that by simply recognizing the challenges the extreme ends of the spectrum of ability may be facing, you can design outcomes that create a better experience for everyone, not just those with permanent disabilities. It’s no harder or even necessarily more expensive to design for inclusion from the beginning—it’s a mindset that leads to better, more inclusive, and more human products.