I got into this stuff by buying an RC2014[0] kit and putting it together, and learning a bit about how CP/M works on it.
If you're interested in that sort of thing, you could do much worse than buying an RC2014 and just writing low-level programs for it for fun.
CP/M is really incredibly simple. You have some "kernel" type stuff towards the top of memory, with routines for reading/writing files on disk, and interacting with the console and a paper tape "reader + punch". When you type a command, the named program is loaded from disk into memory starting at 0x100 and then execution starts at 0x100. To interact with hardware, the program calls the kernel's routines (analogous to system calls). And that's basically all there is to it.
The OS for my CPU is very similar in operation to CP/M, but more pleasant to use for someone coming from a Unix background.
Windows 95 was DOS-compatible.
and so on. and even though the base OS changed starting with Windows 2000, lots of compatibility things were kept because they were supported in previous versions of Windows or MS-DOS.
All the way down to files with special names such as "AUX" or "CON", and the backslash as directory path separator are all from CP/M, though the path thing happened to maintain CP/M compatibility, and not because CP/M had filesystem directories, because early on (maybe always?) it didn't have directories.
Microsoft apparently wanted to break compatibility and use the forward slash, and even coded for the possibility by making the switch character configurable in the CONFIG.SYS of DOS 2.0, but IBM was against it.
DOS 2 needed to be compatible with DOS 1, and DOS 1 needed to be able to run CP/M programs in order to capture the business it was gaining at that point, so DOS 2 kept what DOS 1 had, and so on. all the way up to Windows 11 today.
The backslash didn't come from CP/M, as I said. it came because they needed a directory separator for DOS 2, and being visually similar to the UNIX directory separator character (a simple diagonal line), the backslash was chosen. so, as I said earlier (very poorly, looking back), the backslash was chosen because of CP/M, but was not copied from it.
Moreover, usually when you’re writing a 6502 program, you’re writing a program targeted towards an individual machine, with no OS in the way - the program you’re writing is written against a particular machine; it effectively is the OS.
So the answer is: learn how to write programs for a 6502-based machine first. Then, make your program an OS. Your ability to make it anything like a modern OS is going to be limited, though.
I've build the computer and simple OS/Monitor.
The various "osdev" communities are good places to hang out, though I personally find IRC to be somewhat better than the forums and such.
If you can reason about the code you write in a memory-byte-accurate and handwavingly "what it actually uses in assembly" manner, you can start in. Most hobby OS projects don't ever get past toy stage, a few do.
Unix, however, was designed to be portable, so getting Unix-like OSes running on all sorts of random stuff is easier than writing from scratch.
* There are only three data registers (A, X, Y), and they aren't interchangeable. Values get spilled to the stack or memory constantly.
* Memory addresses are 16 bits, so they cannot be stored in registers at all, and performing even simple operations on them (like incrementing an address) requires carry handling.
* The stack is hard-limited to 256 bytes, so pushing too many temporary values onto the stack risks stack overflow.
* Addressing modes are limited. There is no "address + immediate offset" addressing mode, making pointers to dynamically allocated structures awkward to work with, and some indexed addressing modes will roll over if they're used to index past the end of a 256-byte page.
* No instructions to push or pop anything besides A or status flags.
The combination of all of this basically means there's no good way to do local variables on the 6502. IIRC the C compilers typically reserve a bunch of memory for a separate "C stack" to avoid these limitations, but this is rather inefficient in terms of both performance and memory use.
Idiomatic 6502 assembly tends to have a mindset of "all variables are global", with only a few bytes for temporary/intermediate values that you have to juggle around carefully and deliberately.
and http://calc6502.com/RobotGame/summary.html comparing ASM, C and Forth of the same program
LDA var
CLC
ADC #10
STA var
That's 4 lines and many keypresses for a single addition! You might want to invent slightly more high-level language that would compile into assembly, like this: var, var2 = variables(2)
var += 10
var2 << 5 # Assign 5 to var2
By the way, you can use Python and operator overloading so that the lines above written in Python will generate assembly code when executed. I have used << instead of = in assignment because Python doesn't allow to overload it.They really only had to do a few things: boot from the disk, map files somehow to the disk, read/write/append to said files on the disk, and load and execute a program. Look for how each of the different systems tried to accomplish those tasks.
CP/M is the classic to look at. But consider things such as the Apple ROM, and the Apple DOS. The C64 ROM and how rather than having a DOS per se, it relied on a very "smart" disk drive. Atari has CIO, "Central I/O", which is an extensible device system. There's also FLEX, which is akin to CP/M in that it was a portable OS for 6800 based machines. TRS-DOS for the TRS-80 is a fascinating study, it has some really interesting features you find nowhere else.
Don't forget to look at the old Forth implementations, "language, editor, assembler and OS in 8K of memory". It's very (very) simple, but many folks in the day used it. It needs little more than block disk I/O. But, at the time, thats easier said than done if you take in to account things like sector interleave and other exotic disk layouts. The computers were so slow that you didn't put sectors back to back on the floppy disk. This is because while trying to read two consecutive sectors, by the time the system got done handling the read of the first sector, the second sector (assuming it was adjacent) would have passed the head already and you'd have to wait until it all came around again, which slowed things down dramatically. So, you'd alternate sectors (1 every 2 or 3 sectors), "interleaving" the sectors along the track. Obviously not something you'd need to worry about with modern solid state storage, but back in the day mapping a simple "load sector 123" in to the appropriate track and sector on the disk drive was an entire little subsystem of the OS.
The DOS part is the hardest part. Organizing files on the disk, keep track of the used sectors, the whole kit. There's a lot to do. The rest are little more than monitor routines. Read a serial port, write a serial port, handle an interrupt.
All of the old OS's pretty much have source code published somewhere today that you can audit. They're all in assembly (Z80, 8080, 6502, 6800). But they're a good study.
Doing a survey of them, you see how they try to all solve the same problems, but how they approach them in different ways.