You will want to have a general idea of how a computer is internally structured but at a deeper level than is usually presented in popular material. I would suggest at least skimming through Tannenbaum's "Structured Computer Organization" (far more amenable to skimming than Patterson & Hennessy) and gleaning the high level concepts. Some may suggest Petzold's "Code" but I feel it's more aimed at a general audience than a technical audience. Others may suggest Nisan & Schocken's "The Elements of Computing Systems" used in their Nand2Tetris course but that's probably too low-level for what you want.
To get started with physical hardware, Ben Eater has videos for building a breadboarded computer based on the 6502 processor, which was used in the Apple II and Commodore 64: https://www.youtube.com/playlist?list=PLowKtXNTBypFbtuVMUVXN... that you can use to build your first computer. He also has parts kits available to make things easier: https://eater.net/6502. Some may suggest the videos he has for building an 8-bit computer from scratch (https://www.youtube.com/playlist?list=PLowKtXNTBypGqImE405J2...) but that also might be too low level for what you want.
From there, you'll probably understand enough to build breadboarded machines using other processors. Switching to an Intel 8088 which was used in the IBM PC will give you a 16-bit machine and the Motorola MC68000/MC68008 which was used in the early Macs, both of which are available in breadboardable DIP packages on eBay and elsewhere, will even give you a 32-bit machine to play around with.