Shell internals and Unix internals are pretty intertwined. Amount of ancient syscalls and ioctls that exist directly to support shells is quite large.

If you're after how the shell parses commands, the EBNF grammar in the POSIX specification is a good start:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

Unfortunately, there's not much to a shell other than it's parsing capabilities.

It more likes "How shell invokes a command", I attended a POSIX programming course in university, one homework is implementing a simple shell, you need to handle the FDs, Pipes, environment variables by POSIX API as the article explains.