But they did build those systems. They were called minicomputers. They took up more room than a microcomputer and cost a bunch more money. Because building a system like this is expensive both in engineering and component costs.
For example, one thing about the Gigatron is that it uses static RAM. Static RAM is fast, but expensive, and back in the early 80s even dynamic RAM was dear, and static RAM was several times more expensive than that. All the early 8080 and 6502 computers were using dynamic RAM. With static RAM, and fetching instructions out of ROM, and using a 74 series that wasn't available at the time, the Gigatron benefits from modern times.
The Original Commodore PET, up until the model "N" that was introduced in 1979, had static RAM. I had the 8k model back in 1978.