Write a Shell in C | Hacker News Reader

Write a Shell in C

brennan.io

276 pointsmindcrime80 comments

For anybody who's wondering why I chose to post this: I found this while Googling around trying to answer the question "how hard is it to write a custom shell, and how would you get started?" The reason I wanted to know that is this: I've always been intrigued by the idea of making "things" ( hand-waving around the details of what that means) available via ssh. Imagine something like a MUD, except not a MUD exactly. Just the idea of "You login to a server and you're immediately dropped into some program that provides $EXPERIENCE."

For the longest time I thought the "way to do that" probably involved some crazy sshd config or something, but then the penny dropped and I realized "Well shit, you could just make it a shell and set the user's shell entry to that program. Duh."

If somebody knows a better approach to that sort of thing, I'd love to hear about it. But for now I'm content that I've found an approach that seems workable.

(and if you're wondering more: No, this isn't anything interesting, or a startup project, or anything you'd probably even see on Show HN. It's just an example of the kind of random crap I think about in between doing other things.)

I used to think about a similar ideas. Because it should be possible to ssh to programs other than a "full-featured" shell (despite the "sh" in ssh). One idea was something like OpenBSD's auth-pf. The program might have only one purpose and one capabaility, e.g., to ephemerally open a port in the firewall. Then the user would access whatever remote program she wanted to access through that port, maybe opened briefly just for her.

I have seen "things" served using telnet. In this example, it's a mod for a game that allows for virtual machines to run inside the game, and then has a telnet server that allows you to access them from outside the game. It has a login utility that runs at the start, listing the available virtual machines.

https://github.com/KSP-KOS/KOS/blob/develop/src/kOS/UserIO/T...

Unfortunately, it appears the solution was to write an entirely custom telnet server. Author also says in the included documentation that SSH was the original plan but implementing all of SSH is far too complicated.

Using "command=" in authorized_keys is one way. I've used this to restrict a certain ssh key pair to only using allowing a few simple commands via a wrapper-shell script.

A couple of succinct examples I found via a quick google:

https://www.virtono.com/community/tutorial-how-to/restrict-e...

https://ctrlnotes.com/restrict-a-user-to-ssh-forced-command/...

You could also use Match in sshd_config and ForceCommand too.

Awesome. That's good stuff. I had figured there was a way to do with sshd configuration, but my (admittedly limited) initial Googling had not turned up anything. The "command=" thing seems pretty cool, since you can have a separate command for each certificate. I can see how that would be really useful.

I wrote a curses scheduling application for warehouse managers and essentially just set it as their login shell to a machine that only did timesheet stuff. No complaints and no problems for all the years it ran. Of course... they were already doing a lot of their work in a 5250 terminal emulator - and probably just assumed they were talking to an AS/400 instead of a headless disused Dell workstation. Don't get hung up on the shell thing, you only need to think about env variables, stdin/stdout, and maybe termcap/terminfo (depending on how fancy you want to get).

I wrote a curses scheduling application for warehouse managers and essentially just set it as their login shell to a machine that only did timesheet stuff.

Funny you would mention that... using a curses or TurboVision based TUI app is kinda what I was thinking about that inspired this hole avenue of exploration.

When I was a grad student, the entire campus was able to log in to their email by telneting to a well known host. Upon log in, you were dumped in to Pine.

Innocent times.

Hello. great article! I remember making one back in Uni. We implemented an LL parser it was fun. IT really starts to get tricky when you implement Pipes/Redirections and the parenthesis () operators. have fun hacking

Is this like creating your own Bash shell?

Nice work. Do you think you might do a follow-up adding redirection and pipes? Then a third with job control? ;)

I'm pretty sure something similar is done on nethack@alt.org

Had a course on Unix, one of the assignments was to create shell with pipes. I was surprised how simple it was: just fork all the programs between the pipes and link the output of one to the input of the other.

> char *buffer = malloc(sizeof(char) * bufsize);

Please don't do this. sizeof(char) is always and everywhere 1.

More importantly, don't do:

<type> *buffer = malloc(n * sizeof(<type>));

which frequently leads to bugs of the form

<new_type> *buffer = malloc(n * sizeof(<old_type>));

Instead, use

<type> *buffer = malloc(n * sizeof(*buffer));

which will always keep both sides consistent.

That's something I like to use for structured types, especially when the name is long and I can't be bothered writing "struct something" again, but I don't do it for char or other simple types.

Of course, it doesn't work when you're doing things like OOP in C, where the type on the left is a "superclass", or using flexible array members and allocating more than just a header's worth.

Moreover, don’t use C at all if you have the choice. A std::string in C++ makes mincemeat of the boilerplate that C requires for the same code.

In C++, you can make a very simple REPL that is superior to the one in the OP in almost every way.

  #include <iostream>
  #include <string>

  int main() {
    std::string input;

    while (true) {
      std::cout << "> ";
      std::getline(std::cin, input);

      if(input == "exit") {
        return EXIT_SUCCESS;
    } else {
        std::cout << "Unrecognized command: " << input << '\n';
      }
    }
  }

Code quoted from “Setting Up A Better REPL” (https://mikeloomisgg.github.io/2019-04-10-set-up-repl/). It offers good insight into why C is bad in light of the alternatives like Rust and C++.

Nothing stops you from coding a semantically equivalent solution in C, just with structs and function calls that abstract over the details of io and calling realloc correctly.

Does C have destructors?

Only if you make it and call it.

C has free, which deallocates a block of memory pointed to by a pointer. If it's a struct, the struct is deallocated. If the struct contains pointers, that memory is still there. If you've lose all the pointers to that chunk of memory, you now have a memory leak.

It is not uncommon to make your own allocators and deallocaters for complex structs.

> In C++, you can make a very simple REPL that is superior to the one in the OP in almost every way.

And in Scheme, you can make a very simple REPL that is vastly superior to the one of yours in every way. But I suspect that neither fulfills the goal of learning how things are done in C, which seemed to be the goal of the author of the article.

> A std::string in C++ makes mincemeat

Using bit-shift operators for input/output is a heinous crime against human dignity, basic decency and social order. Every use of it should be punished with 10 lashings, and the promoters of this atrocity should be tarred and feathered.

alternatively, one could consider that bit-shifting isn't nearly as common of a thing as I/O stream redirection.

I'm agnostic on C++'s std stream stuff but I am pretty anti-"yes bitwise logical operators are so important that they get priority over all these symobls".

among other things it's just not a very friendly syntax for the common case of formatting strings and getting input. printf/scanf are much less verbose (if a bit less safe).

That's a good point, actually. Beyond the "default formatting" case, it seems pretty hard to imagine a good syntax that would not be very awkward.

but it removes three classes of errors entirely when compared to printf / scanf:

- wrong arguments types (mostly solved by libraries such as fmt tho): printf("%d", 0.5);

- too much / not enough arguments passed: printf("%d", 0, 1);

- passing unsanitized strings as template argument: printf(some_string_i_got_from_the_network_lol, arg_1, arg_2);

the << method makes these three errors impossible.

Also << does not allow you to have something like printf's %n, which writes to one of its arguments.

fmt is able to give compile-time validation of string literals, but at a hefty compile-time cost. In my opinion the best solution would be string interpolation, but we're definitely not there yet...

> - wrong arguments types (mostly solved by libraries such as fmt tho): printf("%d", 0.5);

> - too much / not enough arguments passed: printf("%d", 0, 1);

-Wformat

> - passing unsanitized strings as template argument: printf(some_string_i_got_from_the_network_lol, arg_1, arg_2);

-Wformat-nonliteral

Ridiculous dogmatism

I would never use C++ if i had a choice. And i do have a choice.

The second tip is an important one. This also applies when using memset, memcpy/memcpy_s, and related memory manipulation functions.

C gives you many ways to shoot yourself in the foot, so you need to make sure to use available language features to your advantage.

FWIW, instead of `*buffer = malloc(n * sizeof(*buffer))` you can omit the extra parens and write `*buffer = malloc(n * sizeof *buffer)`

also, calloc is a nice idea too

> can ... write *buffer = malloc(n * sizeof *buffer)

The extra parentheses are there for the us humans who might need help distinguishing between * as dereferencing operator versus * as multiplication all happening within the space of eight or so characters.

CHAR_BIT is a minimum of 1 byte (8 bits), but is not guaranteed to be 1.

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf (Annex E)

> CHAR_BIT is a minimum of 1 byte (8 bits), but is not guaranteed to be 1.

It seems like you are conflating things. One is, a char does not have to be an octet. That is correct.

The other is the unit for sizeof(char) is always and everywhere 1. That is, the size of everything else is measured relative to char.

Please do see C FAQ[1]. In addition, please see 6.5.3.4.2 in [2]:

> When applied to an operand that has type char, unsigned char, or signed char, (or a qualified version thereof) the result is 1.

It does not matter if char is 11 bits therefore not an octet. That means you are in a universe where a byte is 11 bits long, sizeof(char) is still 1 as far as your program and the compiler are concerned.

Note that the same is true in C++ as well:

> Depending on the computer architecture, a byte may consist of 8 or more bits, the exact number being recorded in CHAR_BIT. sizeof(char), sizeof(char8_t), sizeof(signed char),sizeof(unsigned char), and sizeof(std::byte) are always equal to 1.

HTH.

[1]: http://c-faq.com/charstring/wchar.html

[2]: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf

[3]: https://en.cppreference.com/w/cpp/language/sizeof

You are right. I didn't look into sizeof.

So.. by the C standard, a byte is a size of char.

No. Instead, use

    <type> *buffer = calloc(n, sizeof(*buffer))

as that will also protect against multiplication overflow.

I don't know what I'm talking about, but according to [1] (2008):

> So naturally every existing calloc out there performs this overflow check and doesn’t just multiply its arguments together? Oh no. Even I was surprised.

> On OS X (10.4.11) calloc(65537, 65537) returns a non-NULL pointer. This is a bug.

> PHK’s code in NetBSD is typical, it just multiples the two arguments and uses the result, reckless to overflow.

> Hurray for FreeBSD: they do almost exactly what I suggest, but with a fast test that is just the tiniest bit unportable.

Which suggests you shouldn't rely on there being a multiplication overflow check.

[1] https://drj11.wordpress.com/2008/06/04/calloc-when-multiply-...

I always use `sizeof(char) * ...` in my projects when I deem fit. I feel like it helps with readability and it shows why I'm allocating a buffer with that specific size: I want it to hold exactly N characters.

Also recommend CMU's "shell lab": http://csapp.cs.cmu.edu/3e/labs.html

When I was new to systems programming, that lab — along with all the other labs part of "Computing Systems from a programmer's perspective, especially the binary bomb — helped me better understand lower level concepts such as debugging assembly, fork-exec, etc.

Credit to author for editing and maintaining the article, fixing bugs others have found.

  bufsize += LSH_RL_BUFSIZE;

Shouldn't that be:

  bufsize += LSH_RL_BUFSIZE * sizeof(char);

Probably but a char is often 1 byte so it doesn't make a difference

Sure but he does it the first time

I think the multiplication should be here:

    buffer = realloc(buffer, bufsize * sizeof(char));

This makes the code symmetric wrt. the malloc() call. And better allows the code to be rewritten when moving to e.g. a wide-character type.

> often 1 byte

char is always 1-byte according to the C standard. (Actually, I think it's more like the smallest addressable unit is 1-char, but it's been I while since I've read the spec)

It is now. There were more esoteric platforms for which it was defined as 2 bytes (eg csr8675 and the Kalimba DSP coprocessor was even 24-bit char, short and ints). Until recently C only defined that char was the smallest unit and had size 1. I think as of C11 that may have changed. It definitely did in C++ because basically any platform that mattered defined char as 1 byte and any platforms that didn’t also didn’t really care about portability of existing software that much as the porting is generally more involved anyway (eg I ported a crypto library to Kalimba and it was challenging to get the bit math correct but eventually I did).

> and the Kalimba DSP coprocessor was even 24-bit char, short and ints

Yes, but sizeof char would still be 1, unless I'm mistaken.

I think I might have made a major pedagogical mistake when I declared char == 1 byte. What I meant was that when you're working in C, a char is the smallest addressable unit. Now, that smallest addressable unit is conventionally known as a byte, but it might not be 8-bits wide. These days it is almost always is thankfully.

Thank you for clarifying that, because I don't think my original comment was clear enough. I've been meaning for a while to write a blog post on the origins of 8-bit bytes as well as other conventions in CS.

EDIT: I was wrong, a char is always sizeof 1, See https://news.ycombinator.com/item?id=26131228

// old reply:

I already replied to nanis above, so i+l just c/p it here.

CHAR_BIT is a minimum of 1 byte (8 bits), but is not guaranteed to be 1.

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf (Annex E)

sizeof(char) is 1, always. By definition. Sizeof()’s return answer is in units of “number of chars”.

sizeof(char) is a _minimum_ of 1, always, by definition. But it is not always 1, though most modern platforms are.

> Their implementation-defined values shall be equal or greater in magnitude... - ISO/IEC 9899:201x, Annex E, 5.2.4.2.1

Wrong.

The quoted part of the standard doesn’t talk about sizeof(char). Sizeof returns the size of its operand in char-sized units, therefore sizeof(char) is tautologically 1.

The question I had is if it shouldn't be using some kind of *= instead of +=, so it's amortized constant time.

> Shouldn't that be:

> bufsize += LSH_RL_BUFSIZE * sizeof(char);

sizeof(char) is always and everywhere 1.

I just finished building a shell last week for my University operating systems course. It's interesting to see the difference in quality when a person is looking to make a project for themselves that they are writing about vs fulfilling requirements from a document.

Although this seems to be missing a few things like & and pipe, I find it very interesting how so many people can Implement a shell differently with the same C syscalls.

Today I learned that getline is technically part of POSIX now. Does anyone know which libcs implement it, in addition to glibc?

getline was originally a GNU extension since at least 1991 (for inclusion in glibc 0.4, at least). It became part of POSIX-2008. It appeared in FreeBSD 8.0 (2009). From FreeBSD it made its way in to macOS 10.7 ("Lion", 2011). It appeared in NetBSD 6.0 (2012). It made its way from NetBSD to OpenBSD 5.2 (2012).

Did you have that memorized or is there somewhere you looked that up?

I checked the man pages and git history!

The Linux man-pages project[1] tends to do a pretty good job of telling you when different libcs got a given function, or compatibility concerns between different libcs. BSD man pages tend to just tell you where they copied their implementation from. But unfortunately the Linux man-pages getline(3) page only told me that "Both getline() and getdelim() were originally GNU extensions. They were standardized in POSIX.1-2008.". (I guess that saved me from searching my POSIX PDFs to determine the POSIX version. While I was a student and could legally aquire such things for free I hunted down every version of POSIX or SUS prior to 2016).

I checked the GNU libc git[2] to determine when GNU first got the function, but didn't quite get a clear answer, becuase much of the history prior to 1994 was mangled in the conversion from whatever they were using before to RCS (which they used before they converted to CVS, which they used before they converted to git). I didn't bother checking the GNU manual; it doesn't tend to do a good job of saying when things were introduced.

Next I checked macOS; the macOS 10.15 (Catalina) getline(3) man page confirmed that it appeared in POSIX.1-2008, and also told me that "These routines first appeared in FreeBSD 8.0" ("these routines" referring to their specific implementation of those routines.). I checked through opensource.apple.com archives[3] (actually, I used an import of those archives in to git[4]) to determine which version of macOS they first appeared in.

I checked the FreeBSD man page archive[5] to confirm that they first appeared in 8.0 (a getline(3) man page exists in 8.0 but not 7.4).

I checked the OpenBSD man page archive[6] next. In recent versions the "HISTORY" section says "These functions were ported from NetBSD to OpenBSD 5.2.", though that section doesn't exist in the 5.2 version of the man page. And I confirmed that no such man page exists in 5.1.

Finally, I checked the NetBSD man page archive[7], and bisected through the versions to determine when it was introduced.

I used Wikipedia to determine the release year for each of those versions.

[1]: https://www.kernel.org/doc/man-pages/ [2]: git://sourceware.org/git/glibc.git [3]: https://opensource.apple.com/ [4]: https://github.com/apple-open-source/macos [5]: https://www.freebsd.org/cgi/man.cgi [6]: https://man.openbsd.org/ [7]: https://man.netbsd.org/

Wow. Thanks!

That is some next-level research just for an HN commnet, wow. This is one time when I wish HN had something like Reddit gold or something. Great work!

Beautiful!

A nice, simple, well-documented shell in C!

Not too little, and not too much.

Just the right balance between functionality and complexity for education.

Favorited!

It's already there: https://github.com/danistefanovic/build-your-own-x#build-you...

Fun to see, and awarded full-on nostalgia. This was an actual assignment during my CS curriculum in the late 90s.

And it still is a fairly standard exercise for first or second semester undergrad students.

Just did it last week as the first assignment in my university level OS course (albeit a bit more complexity added)

Writing a basic shell was one of the MPs in Angrave’s Systems Programming course at U of I

https://github.com/angrave/SystemProgramming/wiki#2-processe...

I know its not C but this guy did it in Go among other small projects:

https://carltheperson.com/posts/10-things-linux

Stupid question but, why doesn’t the do while tight loop result in 100% cpu usage?

I often wonder how modern run loops work. They don’t burn cycles?

It spends most of the time waiting for IO.

If you would start it and run for ten minutes and not interact with it, you would have the program just sitting there, very efficiently doing nothing, waiting in getchar().

> why doesn’t the do while tight loop result in 100% cpu usage?

The main loop tries to read characters using the getchar() function, which in turn pokes them out of a buffer inside the "stdin" FILE struct.

If the buffer is empty, somewhere inside the libc, it calls the read() system call on the standard input file descriptor to get more.

If the kernel doesn't have any more data to return, the system call blocks. I.e. the kernel marks the thread as waiting for I/O and no longer schedules it. Once the kernel has data, the thread is scheduled again and the read() system call returns.

And before the obvious follow up question comes: If all programs are blocking on I/O and the kernel doesn't have any threads to schedule, it schedules an "idle thread" that spends its time slice throttling down the CPU. The CPU usage that process monitor programs are displaying is basically the amount of time a CPU core is running something that isn't the idle thread.

> it schedules an "idle thread" that spends its time slice throttling down the CPU

Specifically (AFAIK, at least back then) on x86 the kernel schedules an interrupt and sends a single HLT[0] instruction, and only wakes up when the interrupt triggers. A smartly set up interrupt would have the CPU wake up only as needed for what is required to unblock the pending blocked operations.

Smarter schedulers would go as far as coalescing active events together to rush to idle yet wake up less often, and spend as much time down as possible in increasingly deeper C-states[1].

Back in the day some kernels would loop over NOP[1] instructions, so the CPU would effectively never sleep!

[0]: https://en.wikipedia.org/wiki/HLT_(x86_instruction)

[1]: https://gist.github.com/wmealing/2dd2b543c4d3cff6cab7

[2]: https://en.wikipedia.org/wiki/NOP_(code)

Here's a one liner:

printf("a Shell");

Prefer puts, as the newline is included

One of the odd inconsistencies that I've never found a good answer to is why puts writes a string and a newline, but fputs doesn't write a newline. Likewise, and more problematic, is gets vs fgets --- the latter has a buffer length parameter, while the former doesn't and thus always leave the possibility of overflow. On the other hand, fprintf and printf, fscanf and scanf are consistent.

No one is perfect

Except for Bruce Schneier.

https://www.schneierfacts.com/

And Chuck Norris.

Why write it in C rather than in modern C++?

Toolchain is pretty similar, often more performant, you protect yourself against memory leaks, dangling file descriptors as such (yes, without needing a garbage collector [1]), and some decent libraries to help you out with what you're not writing yourself.

(Potential answer: Bigger executable if you're not careful. In some cases that could be a consideration. [2])

[1] : https://stackoverflow.com/a/48046118/1593077

[2] : https://softwareengineering.stackexchange.com/q/246167/63497

You can write a C compiler relatively easily. In fact there have been numerous articles submitted to HN about someone who did. I don't think anyone has done the same with C++, because of its immense complexity.

and some decent libraries to help you out with what you're not writing yourself.

If you're writing a shell you are probably already somewhere on the "bootstrap pilgrimage". Using something someone else has written is the exact opposite of what you want to do in that case. (Why not just use a shell that someone else has written then...?)

Some might say that as many people started writing c++ compilers as those of C but that the C++ crowd just haven't finished them. Perhaps another decade?

I don't envy anyone writing such a compiler. Its possible but I didn't start one. Seemed like less fun and more like red tape.

The pilgrimage thing is real. Done it myself. The journey from even boot to bash is real and the benefits of awareness it brings definitely appear in strange places. I went even further before the "boot" stage.

Reply on news.ycombinator.com