Yes, it's a fun exercise to build a tiny Linux install that's fully on the initrd. Not that you'd want to have that kind of system in daily use outside of special applications.

> I think it's even possible to embed the initrd in the kernel binary itself, but I've never really investigated that.

Yes, the kernel config has an option to embed the initrd in the kernel image. I'm not sure if there are any advantages to this.