This _is_ possible with Linux, but not at all trivial and likely impossible with general-purpose distros.
Interrupt handling and (on RP2040) dedicated multicore code is also nice.
I assume initramfs-only with special purpose pid0 and only the modules needed statically compiled into the kernel?
What else would it take?
Bootloaders need to initialize most of the devices and load the kernel image. Then they hand the control over to Linux which proceeds to re-init these devices again.
The userspace matters, but on recent computers it doesn't matter that much. You can get to sub-40ms with https://katacontainers.io/ That's a project that uses full VMs to run Docker images boot instead of kernel namespacing.