You’ve touched on what is probably the biggest reason I don't use Siri more: Apple does not limit it to what’s important to me as user.
I have thousands of contacts, lots of photos, videos, and emails, all in Apple’s first-party apps and yet Siri is more likely to respond with a popular song or listing of news articles that’s only tangentially connected to my request.