A quick search suggests that emulators do something slightly different called run-ahead. This is simpler because it only has to cope with a fixed latency. The graphics displayed are consistently a fixed number of frames ahead of the game state. For every frame it saves the state, runs the emulation forward a few frames, displays the frame and reverts the state.
But I agree; when I really want to feel nostalgic, I take a c64 with tape rec and load a game. Sit very quiet after the load command as any movement makes it fail and you can rewind and restart the 20min game load. Somehow it feels very relaxing to me.
I have been slowly rectifying this by obtaining old systems and playing the games on flash carts, on original hardware, on a CRT.
I generally prefer it to emulation. That true zero-lag feeling of real hardware is such a treat. But emulation is cool too. Sometimes, save states make the experience 100x more enjoyable.
It's like music. Generally I listen digitally, but records are really fun in their own way too. For me it's not really a war between the two.
2600 stuff in particular is cheap and easy to find.