I thought that, when nesting tmux instances, you just press the prefix key a number of times depending on how deep you need you need your next keystroke to propagate?
For me personally, I use ctrl-a instead of ctrl-b so I have this line in my ~/.tmux.conf:
#send prefix to nested sessions with C-a
bind-key C-a send-prefix
Lastly, iterm2 uses it's own keystrokes for it's tmux integration. (i.e.: cmd-D to split vertically)