Right. Colossus wasn't a general purpose computer at all. It was a key-tester, like a Bitcoin miner.
The big problem in the early days was memory. ENIAC had nothing writable that could hold a stored program. IBM had electronic arithmetic in R&D before WWII, and that came out as a product after the war, the IBM 603 (1946) and 604 (1948). Those were plugboard programmed machines with electronic arithmetic. Workarounds for memory cost, basically. Everybody involved recognized that with enough memory, they could get rid of the plugboards, but the hardware wasn't ready.
Many memory devices were tried. Delay lines. (Slow, not random access) Williams tube electrostatic storage. (Big and expensive.) High speed drums. (Slow, not random access.) Finally core memory was developed, the first thing that went fast and had random access.