Writing a simple operating system from scratch (2010) [pdf]
cs.bham.ac.uk
cs.bham.ac.uk
For me an Operating System means processes in userland, sadly it comes short of that.
I'm now pushing my own x86 32bit OS in Rust using [0][1]:
There is definitely a shortage of information on this stuff (for example writing a USB driver is really hard and I couldn't find easily available tutorials - you have to resort to raw *HCI and USB specs to get things done).
I'm no OS expert, besides using multiple of them daily, but isn't there different categories of Operating Systems? The ones you say are "A Real OS" are specifically multitasking OSes, while there also exists a section of OSes that only runs single tasks. Aren't those also real Operating Systems?
Another one great resource (that actually inspired me to tinker with all this awesome stuff) is this book [1]
This is a lot weaker of a statement than
>processes in userland
Even if you take a very broad definition of userland.
I think of an Operating System as something that provides a Hardware Abstraction Layer (HAL) and system calls to take advantage of that hardware. Drawing on the screen without knowing the details of the video card, file io without caring about IDE/SCSI, etc.
If it runs other programs but those programs still need to take direct control of the hardware then the OS is just a glorified boot loader.
(The byte code for R83's VM, called Pick assembly, came from Pick's earlier non-PC hardware systems, in which it was actually the hardware machine language, implemented by microcode; I believe the parts of the system which ran on top of the VM were a pretty direct port from those earlier platforms. R83 was succeeded by Advanced Pick, which supported both running as a standalone OS – AP/Native – or as a process on top of a host OS – AP/Unix and AP/DOS. AP/DOS was later succeeded by AP/NT for Windows NT, and Advanced Pick eventually evolved into D3, which is still around – now sold by Rocket Software. D3's virtual machine environment, VME, is a direct descendant of the same byte code VM, although it now supports much deeper integration with the AIX/Linux/Windows host platform than the original hosted AP versions did. Early MUMPS systems were similar in being hosted directly on the hardware, but Pick as an operating system arguably lasted a decade or more longer than MUMPS as an operating system did.)
Although there's still "userland" in Phantom, it's just that it uses type safety for memory protection, instead of MMU.
- the spec itself (obviously)
- USB Complete by Jan Axelson (quickly go through this one before reading the spec)
- USB The Universal Serial Bus by Benjamin Lunt (https://www.fysnet.net/the_universal_serial_bus.htm)
If you're writing your operating system, checkout other books by Benjamin Lunt:
It is for the author's definition of "complete", which he gives explicitly early on in the paper.
> For me an Operating System means processes in userland
As the reference to the author's late professor in the introduction to the paper will show you, it hasn't always been so. Once upon a time, an "operating system" just meant "the code you have to write before you can do anything at all with this computer". That code could be anything from a special-purpose system for one particular task, to something like MS-DOS, which could run other programs but had no notion of "processes in userland" (because there wasn't any "userland", every byte of code ran on the bare metal), to the kind of multi-tasking system we now generally think of when we think "operating system", like Linux or Windows or Mac OS X.
"So what this means is that a 32-bit operating system must provide its own drivers for all hardware of the machine (e.g. the keybaord, screen, disk drives, mouse, etc). Actually, it is possible for a 32-bit protected mode operating system to switch temporarily back into 16-bit mode whereupon it may utilise BIOS, but this teachnique can be more trouble than it is worth, especially in terms of performance."
--
In the toy 32-bit OS I am currently writing, having easy disk access and text output was more important to me than performance, so I decided to implement this technique to access the disk and screen via the BIOS instead of writing an ATA disk driver.
Although I could not find any minimal yet complete working examples of dropping to 16 bit and later resuming 32 bit mode, I was able to piece it together and write assembly functions called enter_16bit_real and resume_32bit_mode. See https://github.com/ironmeld/builder-hex0/blob/main/builder-h.... Those routines are working well but beware the project is a volatile work in progress and is coded in hex for bootstrapping reasons.
[0] Yes, I know, Windows 3.1 applications were almost always 16-bit unless you installed Win32S. It's somewhat complicated.
https://www.udemy.com/course/writing-your-own-operating-syst...
Page 37 says that pipelining is the reason we need to jmp after setting 32bit mode in CR0, but that's not true. The execution mode is part of the CS descriptor, and setting CR0 merely means that "on next segment (CS:EIP) jump" we should treat the segment part not as a 16bit segment, but as a segment descriptor selector.
Otherwise, dude, that would be a race against the pipelining with undefined behaviour.
So no, pipelining is an implementation detail you STILL don't have to care about.
Like, do you know the instruction pipeline length on a 386SX? No, because you don't have to. There's no rush to jump. (though what else are you going to do?)
Actually, you could continue executing 16bit code, loading only DS/ES with 32bit selectors. Why, I don't know, but you could.
> a near jump[…] may not be sufficient to flush the pipeline
Nope, that's not it. You are literally jumping into a 32bit address space by loading the descriptor. The descriptor is not loaded until you load a new selector into CS by using a far jump.
Mind you, this is from memory when I made my OS in high school. Back when there was a lot of moving floppies back and forth to iterate. :-/
For more on this (and the resource I wish I had back then), see https://wiki.osdev.org/
Up to 3 instructions could be pre-decoded and stored in that queue, while the execution unit was still busy with some other instruction (keep in mind that back in the day it was multiple clock cycles per instruction, not multiple instructions per cycle!)
Each decoded instruction included a microcode entry point address, depending on the opcode and the state of CR0 at the time of decoding. On the 486 this was further optimized to actually store the first micro-op in that queue.
If you set the "Protect Enable" bit in CR0, it does not affect instructions that have already been through the decode stage, so they would not run with protected mode semantics. A far jump would actually not work in that case, because it would load the CS base with (selector << 4) instead of going through the descriptor table. So you should first do a near jump to flush the pipeline, and only then jump to the protected mode CS.
In practice, you could get away with not doing this step, depending on the exact timing of the code used to switch to protected mode. A lot of instructions are short enough that the processor never has enough time to pre-decode anything following.
The Intel manuals did (and still do?) document you have to do this near jump first, though not the exact details of why.
It looks like the author of this PDF confused the two jumps.
Looks like Linux nowadays does the far jump immediately after setting cr0, thus doing two in one:
https://github.com/torvalds/linux/blob/master/arch/x86/boot/...
But the near and far jump (if you use both) still have VERY different purposes. And the CPU is not really in 32bit mode until all the segment descriptors have been loaded manually.
Linux didn't used to combine them:
https://elixir.bootlin.com/linux/2.2.26/source/arch/i386/boo...
Here we see the near instruction-flushing jump, then a bunch of code (that is still in 16bit mode!), and then the long jump that switches the instruction set to 32bit.
The comments in that last link explains it well, too.
Whoever wrote the modern Linux code doesn't seem to understand any of this. There is even a jump before the switch to protected mode, with the comment "Short jump to serialize on 386/486". Cargo-cult programming at its finest :)
As I said before, it may have always worked in practice without the jump, and modern Linux doesn't even support the 386 anymore.
Actually, with all the crazy out-of-order speculative execution going on, one could expect newer CPUs to have similar requirements, but I guess they also had to get better at providing the illusion that such internal details don't matter. Making LMSW / MOV CR0 a serializing instruction seems to be an easy way to do it.
It definitely is in real mode when the far jump is decoded. Or is that not the right technical definition of "real mode"?
Do we leave real mode as soon as CR0 bit 1 is set, even though nothing has changed until the code puts a new value into a segment register? I've always thought of it as when CS becomes 32bit, meaning the long jump.
Anyway, that's just terminology.
> "Short jump to serialize on 386/486". Cargo-cult programming at its finest :)
That's weird. I tracked down the commit:
https://github.com/torvalds/linux/commit/2ee2394b682c0ee99b0...
> Actually, with all the crazy out-of-order speculative execution going on
I would expect those to detect that dependency and not take effect. But I could be wrong.
Edit:
Interesting. The new Intel docs explicitly say to do the far jump "Immediately" after setting the PE bit in CR0.
https://www.intel.co.uk/content/www/uk/en/architecture-and-t... section 9.9.1.
Whereas the example code for the original 386 programming manual has the short jump, many other instructions, and then the far jump.
Haha, but then the actual example code on page 9-20 (new manual) still has the "clear prefetch queue" near jump.
Yes, that's what the bit is defined as doing. Protected mode was introduced on the 16-bit 80286, and 16-bit code segment descriptors are still supported to this day.
Even in real mode, the segment registers have hidden fields containing base, limit and access rights. The difference is that when they get loaded in real mode, the base will be set to the segment shifted left by 4 bits, with the limit¹ and access rights² generally left unchanged. What the PE bit actually affects is how segment load instructions operate, how interrupts/exceptions are handled, and a few other details.
Most instructions run exactly the same microcode in either mode: any memory access will form the address and do protection checks based on whatever is currently in those hidden fields. But a segment load (or far jump) decoded while PE=0 will execute different microcode than one with PE=1.
>That's weird. I tracked down the commit
Shuffling the code around may have fixed some alignment bug? The jump could likely be replaced by two NOPs, in any case the comment is completely wrong.
2.6.22 seems to be the last version using LMSW followed by a near jump, and presumably worked on that CPU (at least there is a comment mentioning bugfixes for Elan), so it isn't likely to be the cause of the problem.
¹ the limit on power-on/reset is 64K, but it is possible to change it to 4G, allowing access to all memory ("unreal mode")
² CS will always be made a writable data segment, something not possible to set up in protected mode without the use of LOADALL
edit: >Interesting. The new Intel docs explicitly say to do the far jump "Immediately" after setting the PE bit in CR0.
Well, it's probably not required to do any kind of jump immediately on modern CPUs, but it wouldn't be the first time Intel got something completely wrong: https://www.os2museum.com/wp/sgdtsidt-fiction-and-reality/
That's almost how flat real mode (aka "unreal mode") works, except it's returning from protmode with 4GB selector limits.
If you’re interested, the OSDev Wiki [0] is a great place to start, as are Bran’s Kernel Development tutorials, which I can’t seem to find on mobile right now.
UEFI gives you a much nicer initial environment — it's already in 64-bit long mode, for example, unlike BIOS which drops you into 8-bit real mode, with a maximum of one sector in which to perform all the gymnastics you need to do to load the rest of your program and switch to _at least_ 32-bit protected mode (notice how this PDF doesn't even go to 64-bit long mode).
UEFI also gives you better, easier to use boot facilities than BIOS: you can access a memory map, read from the boot file system, switch video modes, and even access network drivers. And you can access all this from long mode, unlike BIOS which only works in real mode. Then once you've booted you can take over the system by exiting the boot services, and have full control over the system just like you want for making an operating system.
That's my point though: today, BIOS is retro technology.
Nim: https://github.com/khaledh/axiom
Zig: https://github.com/khaledh/axiom-zig (this one goes into depth in disassembling ACPI DSDT bytecode)
There's tutorial type articles for these around: for C there's https://wiki.osdev.org/UEFI_App_Bare_Bones, for Rust there's this template https://github.com/rust-osdev/uefi-rs/tree/main/template and the crate's documentation.
I don't think the role of firmware requires it to be minimal. It's to provide a consistent boot environment on different hardware, and to provide useful services to the program. Even BIOS-based machines have significant amounts of code that run before they jump to the boot sector, so while UEFI is definitely more sophisticated I don't really see this as a drawback to it.
https://news.ycombinator.com/item?id=21951241
So yes, if you didn't use the DOS code (I mean… you overwrote it), then DOS is not there. And if DOS is not the OS, that just leaves your code to be the OS. :-)
Are there any such resources for ARM (preferably ARM64) that would cover similar "Get a basic OS up and running" topics that you could follow on a device or emulator?
Reminds me of some 'bare metal OS' series for Raspberry Pi: https://www.rpi4os.com and https://s-matyukevich.github.io/raspberry-pi-os and https://github.com/rsta2/circle
Recent x64 AMD and Intel PC's use UEFI secure boot and require signed boot images, so this simple approach won't boot out-of-the box anymore I suspect. Same for Mac M1. So some update would be useful, without requiring to disable UEFI secure boot.
UEFI offers some basic IO functionality, it's like BIOS in that regard. You don't design your OS around it, you use it to have easy graphics, networking etc during boot time, then you switch to your own drivers and don't touch UEFI again.
quote:
Sortix is a small self-hosting operating-system aiming to be a clean and modern POSIX implementation.
It is a hobbyist operating system written from scratch with its own base system, including kernel and standard library, as well as ports of third party software. It has a straightforward installer and can be developed under itself. Releases come with the source code in /src, ready for tinkering
When reading the "entering protected mode" section I started to question my own understanding of some the details. My first question is:
When a CPU starts up in real mode it's running in a kind of privileged state since there are none of the 4 protection rings that protected mode give us. We then build a GDT and switch the CPU to protected mode. In practical terms where does this code live GDT building and long mode jump code live? Is it in GRUB or whichever stage 2 bootloader is in use?
My second question was about the "chicken and egg" situation about how the kernel code inherits that privileged address space defined marked with ring 0 bits in the GDT. Is magic that once the GDT has been defined, when we subsequently decompresses the kernel into those memory addresses the kernel memory will have ring 0 bits in its descriptors?
This is how I've understood it but I've often found the texts about this a little murky and this part still feels a bit like magic to me I guess.
For the second question, you just build your GDT entries with the highest privelege, then you switch to protected mode.
The OS (or loader) is responsible for setting up descriptors and managing memory. In 64-bit and most 32-bit kernels, segmentation is basically not used anymore, and the descriptors are just flat address spaces (all start at 0, no limit) with different privilege levels.
Example of 32 bit flat GDT:
0000 (null descriptor; reserved by hardware and can't be used)
0008 kernel code (ring 0)
0010 kernel data (ring 0)
0018 user code (ring 3)
0020 user data (ring 3)
0028 TSS (hardware defined structure)
But you can also have segments in addition to paging, which is helpful for transitioning from/to 16-bit mode.
Hope that helps at least a little bit!
What I meant was the code segment register gets reloaded to some offset in that GDT that has those protection bits set to 0. According to your flat address space map above that would be at offsets 0008 and 0010. That makes a lot of sense, thanks.
I’ve been trying to play with building an OS for riscv (and / or maybe pic32) using qemu, but I’d like to simulate my own display, keyboards and other components.
Basically a digitised bread board where you can send compiled code to emulated chips and run.
It was a lot of fun!
https://www.arm.com/resources/education/books/operating-syst...
There's far too many of these courses targeting x86-32 and the deeply flawed UNIX architecture.
The closest we've got today is MINIX 3's book, Operating Systems: Design and Implementation.