CGI was the late 90s equivalent of "function as a service". It was especially useful on multi-tenant web hosts, in a way that hasn't really been replicated since. The main advantage is absolute simplicity: you upload an executable, the webserver runs it under your uid/gid, it handles your request, and then it exits and doesn't consume any more resources.

The downside was the cost of starting an executable from scratch, which isn't that bad on Linux, plus the cost of whatever interpreter was handling the request if you didn't write it in C. Such as PHP or Perl ('use CGI' was very popular).

(what better solution would you choose in 2024 for a web server where you have 1,000 customers each of which has one or two CGI scripts that get on average one request every few minutes?)

You don’t get, when web servers were a new thing, why anyone continued using the same kind of executables they always had to handle workloads that required executable code.

Probably because there was not yet any data AT ALL about demand from the network, the patterns that random visitors could cause, etc.

“I don’t get why the Romans didn’t build suspension bridges with concrete, steel, and cables.”

It's actually pretty fast if written in a lean language like C/rust. See "cgit" for a good example. And if you want to distribute a program which talks HTTP, it's kind of the obvious choice - the alternative is each program packing it's own HTTP server, which needs a reverse proxy/SSL in front of it anyway.

Because back when CGI programs where the norm application servers weren't really a thing outside of maybe Java and similar ecosystem (by which I mean dotnet).

When you wrote a web application before the ~2010s you wouldn't start of with a library with webserver building blocks to which to attach routing + behaviour. It sounded crazy at that time, especially when you where building websites with a scripting language like PHP/Perl/Python/Ruby. It sorta became the norm with the rise in popularity of event driven IO, through projects like NodeJS and the rising popularity of Go.

Even today, at least in the PHP world, we still mostly use the CGI based approach to running code (though it's through a more modern interface such as "fast cgi"), with an inbetween service that handles process (fpm - fastcgi process manager). I'm sure other dynamic languages have something similar, though there is a large wave of people pushing for more application server style deployments in those ecosystems as well (which is more feasible today).

I have no idea why someone would use one now. In the mid-90’s it was pretty much the only practical cross-language way to create dynamic web content. Someone certainly could have written their CGI program as a separate web server, but:

1. proxy support was limited or non-existent in the main webservers that existed at the time, so routing the request over would have been a problem.

2. there were no frameworks to speak of to make any of this easier. CGI.pm would help parse and decode GET and POST data. That was just about all there was. There was no PHP (as it exists now,) no Rails, no http frameworks like actix-web. Nothing.

“Inefficient and prone to break” is an excellent description of CGI. Most practical people moved away from as quickly as possible.

CGI is a simple interface. It's simple to implement and simple to use. As a result, it's much less prone to break than most modern web architectures IMHO.

It does leave some performance on the table, which was addressed by things like mod_perl or FastCGI (incl. PHP-FPM). There's also the middle ground solution of mod_cgid, which has a specialized daemon fork the CGI process instead (presumably the specialized daemon has a much smaller memory image than the full server).

What should they have been using instead?