I think it's a reasonable thing to think about. I list the client if that's what seems reasonable. If prompted, I'd say "Technically I worked for an agency that provided next to zero value and I don't consider them relevant for the purpose of my CV"
If I was consulting through a company on more of a short-term basis for multiple clients, and that consulting company played a more significant role in employing me, than ya they'd get recognition. As it stands, I list the client I worked for, and in my current case I use the name of the one client I work for rather than the proxy company I technically work for.
There's a number of valid situations to be in, I usually do list "contract" if it was truly a contract, but otherwise I just do what seems reasonable, and if someone thinks that's deceptive, then I guess that's reasonable too.