Having studied Japanese for three years full time I know the obstacles. I've yet to read a full novel. It's too painful. I just started reading one with the help of my Japanese partner who can fill me in on the reading and meaning when I try to read.
It would take too long otherwise.
You can now search for words using pinjin, voice or drawing (thanks Pleko).
And if you encounter unknown printed(!) signs you can just point your camera at it (now or soon, what's currently the best app?).
As for calligraphic handwriting this will forever remain elusive. You may consider it as an utterly useless hobby.
As much as I despise Siri, one things she is surprisingly capable of is answering: "hey Siri what does sneeze mean in Chinese"