The very idea of "racing the beam" is madness distilled. How do you draw without a framebuffer and keep track of everything's position using just 128 bytes of RAM? This comment alone uses more RAM than that.
tl;dw - The background is defined with 20 bits, which are repeated or mirrored to make the background (eg 00000000111111110000 would either make the background 0000000011111111000000000000111111110000 or 0000000011111111000000001111111100000000). If you write to WSYNC ($D40A), the processor pauses until the start of the next scanline. Then you have to count machine cycles to make sure you modify the background at the right time to make the two halves different. Repeat on each scanline to fully define the background. There is no "framebuffer" per se
Essentially, the 2600 gives the programmer 2 sprites only. One sprite is dedicated to drawing the Pac-Man main character, while the other takes the role of all four ghosts. That's right, simultaneously 1 sprite for 4 distinct ghosts, wherever they are on the screen.
This singular sprite and the practice of "racing the beam" leads to a lot of flickering. I believe that the flickering is so severe that it impacts the collision detection, so that Pac-Man can slide right through a ghost in the right place and time.
I owned a copy, of course, and it was just so sad to compare it with its daddy in the arcade cabinet. However, once Ms. Pac-Man rolled around, they had learned a lot of lessons, and made that one a rousing technical success.
The signature hack on the system is to reuse sprites mid-frame. That's the "racing the beam": after an object has been drawn, reposition the sprite to draw something else further down on the screen, before the scanline beam gets there.
Pac-Man didn't do that. Instead, it draws only one ghost sprite per frame, and flickers between the four of them every four frames. The flickering and racing-the-beam are opposites, not one causing the other.
Atari's programmers knew how to race the beam for reusing sprites. Air-Sea Battle did it years before Pac-Man. What's different about Pac-Man is that the objects aren't constrained in their movement. They can overlap on the same scanline which makes drawing multiples with a single sprite impossible. The right solution to this and what Ms. Pac did is smart sprite reuse - if they're on the same scanline, flicker temporarily, but then when they move to be separated vertically, race the beam to display both together. This didn't happen for Pac-Man because Atari rushed it to release in too short a timeframe to implement that, and also the code wouldn't have fit into a standard 4k cartridge and would have needed more expensive hardware with bankswitching logic chips. 2600 Pac-Man's terribleness was a business decision, not a technical limitation.
There's a lot of myths around 2600 Pac-Man. The game was made in around 5 months, which was a normal amount of dev time for a 2600 game. The ghosts flickering was due to insufficient RAM, not insufficient ROM. Tod Frye, the game's programmer, had a kernel that would reposition the sprites as the screen drew, but it needed additional RAM. Frye really wanted Pac-Man to have a 2-player mode, as the arcade version did and he felt that having a 2-player mode gave the consumer more value. This meant that there wasn't enough RAM to get rid of the flicker, since the game had to keep track of the game state for each player. The 2600 version of Ms. Pac-Man was 1-player only, which is how they avoided the flickering.
You still needed a counter (index) to determine which byte in the ROM to copy over. You probably also wanted to put the graphics bitmap data at the beginning of a page (e.g. $FD00) so you would only need to increment 1 byte in the pointer and not worry about carrying over to the next byte.
The horizontal resolution was 160, and generally you were generating less than 255 lines of graphics, so tracking the X and Y position only took 2 bytes, and maybe a few other bytes for metadata.
First, there is no frame buffer (there could be no such thing in that memory space), but there are effectively line buffers for the different sprite types and background. These are not in the 128 bytes of general purpose RAM but memory mapped to the display controller.
Second, it maintains the patterns in these buffers until you change them and will repeat them on the next line unless you do anything to change them. This makes it easy to perform partial update, e.g. only update the background every 8 lines and only update sprite graphics on odd lines or whatever you need to do to save time.
Third, the video controller can be instructed (via a memory mapped register) to halt the CPU until the next horizontal "blanking" interval. This makes synchronizing to the beam position between lines much easier.
Fourth, there are no interrupts (except NMI, only triggered manually with the brk instruction) so timing is easily predictable.
Fifth, there is a hardware timer which makes it easier to keep track of your budget while in the vertical blanking portion of the screen, where you probably want to handle most if not all of your game logic. That way the game logic doesn't need to be strictly timed in the same way the screen is.
I made a game for it (https://text.garden/jupitersumo.html) expecting it to be difficult, but found that the simplicity and predictable timing made it surprisingly easy. On the C64 for example you will often find yourself wanting to synchronize to the beginning of a line or even every line for one reason or another, but doing that while dealing with the complexities of its timing is a whole science unto itself. For the main game screen I ended up using some 16 bytes of state in RAM to represent the stuff I need to keep track of while "racing the beam". The code that handles the display is maybe 150 instructions long and lives in the game ROM. The source code is linked from the page if you're interested in how it works.
Of course, in terms of what it displays, my game is very simple, only showing a background and the player sprites. There are much more complex display routines ("kernels") in the VCS game library. An interesting example is Pitfall II, which is very impressive considering the platform, but uses a coprocessor on the cartridge itself (called the DPC) to help sequence line register updates. From memory, I think it lets you enqueue data in advance, and during the display routine the CPU can just read a single register to get the next byte from a queue.