Absolutely - I auto deny reflexively, and probably miss out on notifcations I want.
It's weird - cookie popups (I can browse incognito or clear cookies or use an extension to control my own cookies) and the notification popup I found similarly annoying.
Just let me browse and control my browsing on my own terms. In the website text it can ask that I go into a notifications whitelist panel and add a url that can send notifications, or click on an element that triggers the pop-up etc.
Similarly, we keep on hearing these headlines about someone not having the right cookie popups or enough of them etc. I wish those would go away as well. The place to control cookies is with the user. I'm not sure disabling cookies even hides my activities from a site (obviously it doesn't).