This idea is explored a bit in the Ender's Game universe [1], it definitely seems pretty compelling that non-human intelligence might be so utterly incompatible with our own thoughts that even beginning to communicate might be entirely impossible.

[1] http://enderverse.wikia.com/wiki/Hierarchy_of_Foreignness