screen (and tmux) basically store their own scrollback. The advantage here is that you can then page through the line history on another terminal or even another computer. The downside is that that your terminal (xterm/iterm/whatever) has to decide what to show when you page up. It has no way to read screen's internal scrollback, so it can either show you old non-screen lines, or hide the scrollback entirely. Switching into/out of screen will just add to the confusion.
(I think some terminals will hide the scrollback history when they spot the TTY in curses-style mode, and only bring it back when that process quits?)
One other annoyance for me is the behaviour of scrollback with a command like 'less' or 'more'. If you are paging down through a file, then scroll back up, some terminals will show you earlier lines of the file you are viewing, while others show you 'pre-less' scrollback.