Mode X was convoluted, and if you were serious about double buffering, whole screen pixel perfect scrolling, etc - you had to master it.
But if all you wanted to do was set pixels, VGA mode 13 (IIRC) really was just setting a value to (0xA0000+y * 320+x) ; Turbo Pascal even let you define an "array[0..199][0..319] of byte absolute $a000:0" and treat it as an array; If you manipulated a small enough part of the screen, it was reasonable to do "off screen rending" (poor man's double buffering) and use the CPU to copy onto the screen.
CGA/EGA/Hercules were slightly more complicated because of bit packing and other weird hardware constraints, but only slightly so. As were the flat bitmap modes of most machines at the time, true. But the built in BASICs more than made up for this, as far as beginners were concerned.
On the Apple ][, it was all of "HGR2 : PLOT 100, 100 : LINE 200,200" IIRC; On the spectrum, it was "PLOT 100, 100 : DRAW 100, 100"; on the BBC B (spiritual grandfather of the Raspberry Pi) it was "MODE 7 : LINE (100,100)-(200,200)"
C64 Basic was indeed spartan, but C64 did have a pixel addressable mode that was almost a walk in the park; CPU was limited, so 99.9% of games used the redefined-charset trick, which let them update only one 40x25 charmap instead of a much bigger pixel map (and ... it was actually perfectly matched to the game designs and other memory constraints of the time - your elements had to be repetitive due to lack of memory/cpu to do other things, and the charmap just constrained those repetition to start on an 8-pixel boundary).
I may remember a detail wrong in those examples - but basically, the C64 and VIC20 basics gave you little help; But the ZX Spectrum, Oric Atmos, IBM PC jr, Apple ][, Dragon 32/64, Elan, MSX and all other contemporaries made it extremely easy.