Regarding the Atari VCS (2600) it should be mentioned that the functionality of these ports is actually defined by the TIA (Television Interface Adaptor) chip. This provides the following capabilities (to be configured by a port register): dumped ports (for 4 paddles on 2 ports), latched ports (what we know as joystick ports), but you may also disable these latches, by which "these ports become two general purpose input ports" (Stella Guide [1]). Moreover, the ports can be configured to read two keyboards of 12 buttons each (4 rows of 3 columns), which was actually used by a few games that came with special controllers. Finally, port A, which handles all these controller inputs, can be configured as an output port using an 8-bit data direction register. (Meaning, you could set this up for full serial communication.)
So the Atari game ports were much more general and versatile in concept than just providing a connection for digital joysticks or paddles.