How can the Linux kernel compile itself? | Hacker News Reader

How can the Linux kernel compile itself?

stackoverflow.com

34 pointsdchester19531 comments

> How can the linux kernel compile itself?

Short answer: it can't. Each software system relies for its own existence on a predecessor, usually one of smaller scope. Sort of like biology and evolution.

When I wrote Apple Writer in 1978, there were no high-level development tools available, just a primitive assembler built into the Apple II BIOS. By making direct hand entries, I used the BIOS assembler to write a better, faster assembler that supported alphabetic labels. Then I used that assembler to write a better one, and eventually I had a development tool suitable for writing applications.

Linux development has relied on a similar bootstrap process, on a larger scale.

Just out of curiosity, have you ever written a book about the process of bootstrapping an Apple II development environment and the little discoveries you made? I would read that in a heartbeat.

> Just out of curiosity, have you ever written a book about the process of bootstrapping an Apple II development environment and the little discoveries you made? I would read that in a heartbeat.

Thanks for this suggestion! I write a lot of articles, but I haven't written much about that period, possibly because other people were writing it for me. But I think it might be interesting to write about that time, how we created software before we had access to high-level languages.

In the meantime, here's an old piece I wrote in those days, even though not quite what you have in mind:

http://www.atariarchives.org/deli/cottage_computer_programmi...

The Atari Archives piece was part of what motivated me to ask about this. Another part is the series of books Jordan Mechner has released (in both print and electronically) about the development of Karateka and Prince of Persia. I've found that I enjoy following along the process of discovery almost as much as I enjoy the finished product. There is also a practical benefit: I gain more intellectual tools for tackling obstacles by learning the process others use to tackle obstacles.

If he thinks that's weird, then he ought to read Ken Thompson's article about trusting compilers.

http://cm.bell-labs.com/who/ken/trust.html

While I love the story, it's really amusing to watch how it gets re-discovered on HN every few years, blows everyone's minds, and then gets quoted all over the place, even where it is only mildly relevant. This might be our "does glass flow" meme.

Seemed relevant to me. The question asked about compilers and the kernel is what compiled the compiler. Ken Thompsons' talk related to hiding code by adding code to a compiler then compile the compiler, remove the code then continue to compile the compiler with the modified version that doesn't have the source code - your modification remains.

Point being: both relate to compiling compilers.

> the kernel is what compiled the compiler

That's not what a kernel does...

Who are you quoting? Certainly wasn't me!

Uh, it was the parent comment - do a search if you can't find it on perusal. It's possibly taken out of context but I don't see a reading of the passage in question that doesn't include that sentiment. Could you clarify?

The parent comment? I was the top level commenter on this. Do you mean the article itself? It reads:

"The kernel is written in C, however how did the kernel get compiled without a compiler installed?"

And...

"If the C compiler is installed on my machine before the kernel is compiled, how can the compiler itself get compiled without a compiler installed?"

A somewhat related albeit primarily entertaining idea is that behind the Quine: http://en.wikipedia.org/wiki/Quine_(computing)

Obligatory DDC link: http://www.dwheeler.com/trusting-trust

This is why I think more people should expose themselves to hardware design, CPU design, and compiler writing, if only at an abstract/academic level. I've worked with a lot of entry-level programmers, and even some that were not so entry-level, that wouldn't have a clue how to answer this question, which is quite depressing.

I'm not a programmer at all, and I found this answer quite obvious. Where else does new binary software come from, if not from a computer running older software?

It shouldn't be such a mystery, because it follows a biological model -- small, simple organisms produce more complex descendants by way of natural selection, all fueled by proteins. The same process creates complex software systems out of simple predecessors, all fueled by coffee.

The biological model also explains why people can get confused about it.

The chicken or egg problem shows that at some point you lose the connection to the predecessor and the process goes around in endless circles.

Then you might wonder how it ever came to be when the first thing depends on the second and the second depends on the first.

> The chicken or egg problem shows that at some point you lose the connection to the predecessor and the process goes around in endless circles.

No, this isn't so. It's a myth about biology that's repeated by people who don't understand natural selection. An egg is a chicken's way to make an insurance salesman.

You don't have a chicken/egg problem with either biological evolution or computers. With biological evolution abiogenesis is at the bottom of the turtle stack, with computers it is likely some guy poking holes in some cards or rewiring a computer by hand.

Ultimately you don't need life to create life, and ultimately you don't need a computer program to build a computer program. There are other methods, even though we don't see those methods executed today.

If you think through, it is not _that_ obvious; it can't be turtles all the way down. So, where did the first binary software come from?

The answer is that somebody punched in a program from a computer's console, probably somewhere in the early 1960s, but possibly earlier. That program was used to make it easier to enter other programs, etc.

Given that copying working code is so simple, chances are that most of the currently running hardware can be traced back to a few persons entering a boot loader by hand.

That might even be one person. For example, Windows NT was initially developed for Intel's i860 (http://en.wikipedia.org/wiki/Windows_NT#Development), so chances are that it was bootstrapped from a Unix machine. Because of that, it may trace back to the same initial boot loader as, for example, Mac OS X.

Well, to be detailed, both the first kernel and the first compiler was probably entered from a stack of punchcards or punch tape made with mechanical "typewriters"; punching a program from console is rather unwieldy and generally you should do that only for the bootloader that loads the rest of the (manually compiled) code from some electromechanical device.

Link to Ken Thompson, link to Quines and that one dude who always goes 'When I wrote Apple Writer...' - all in the first few comments.. thread complete?

i'm still waiting for some way to tangentially connect this issue with NSA backdooring.

I guess you found one.

This does make me wonder about something though: is there a crashplan for a compiler? Can a very minimal gcc be written which would be capable of bootstrapping the entire gcc compiler?

It seems like it would be relevant if a new architecture was developed at least.

If a new architecture was developed you could bring up the whole stack by cross-{assembling,compiling} from an already supported target. By doing so you could test the output using a simulator (or actual hardware) and then when the assembler/compiler is stable enough you could then cross-assemble/compile the assembler/compiler and then be self-hosting on the new architecture.

For any practical purposes, it is best to simply use the full complex software stack on another computer to generate binary code meant for that new architecture.

If we'd really need to bootstrap a compiler from scratch, then a possibility would be to hand-code a Forth system, and then find/modify/make an assembler written in forth; then write a compiler for some small language (scheme?) in that assembler, and then from scheme you can easily get to some C compiler that can compile the full GCC.

>Can a very minimal gcc be written which would be capable of bootstrapping the entire gcc compiler?

Yes. In fact, this is how gcc is built - in stages. First stage: build a compiler that can build the compiler. Recompile the compiler with the newly built compiler. Do it again. Repeat until completion. ;)

Another interesting thing to remember is that an embedded operating system (Android phone OS and software, for example) is likely being built on a PC set up with a cross-compile environment and not directly on the embedded hardware.

The bootloader can compile linux before booting : http://bellard.org/tcc/tccboot.html

http://en.wikipedia.org/wiki/Self-hosting

Reply on news.ycombinator.com