I started declining small meetings that don't meet this bar. First, I'll reply with, "what's your agenda." If I get a flippant answer, I decline. It pisses my boss off, but it's extremely effective.
Flippant answer: "I'd like to talk to you about [product / project / job title]"