Fun fact, if you ran gws without its config files you would see the real front end for Google Search, News, etc.
Web configuration interface was in Java, writing some XML templates if I remember well.
So taking all of that, besides a very boring OS there was "nothing" or very little amount of open-source they were using.
It was more all homemade (except the OS).
Fun fact: There was a secret hardcoded password in clear (but only for physical access).
EDIT: Password was different for each instance, not the same as I thought.
Each GSA had a set of unique BIOS/root password generated during bootstrap though.
It was great to see how it was engineered, some parts were truly remarkable, my main interest was to learn about the ranking algorithm (not for SEO purposes, but because I thought it was fun and interesting).
We would have been in love 15 years ago when there was the GSA, sadly, our paths have separated :D