The C64 firmware was mapped out in detail when the C64 was in its heyday - you had books published with full commented disassembly. And because the C64 had the full 64K RAM, but used bank switching to overlay the ROMs, there were tons of "OS upgrades" that simply mirrored the ROM into the underlying RAM, made various modifications, and bank switched the ROMs out.
So yes there was, but most of hit happened 20-30 years ago already.
There were tons of commercial expansions based on this method, such as Simons' BASIC, and the far more ambitious Lightning/White Lightning, which I believe started as a Forth system, and was ported to other platforms too, but also came in a BASIC version. On the C64 - I don't know how different e.g. the Spectrum version was - it allowed multitasking up to 3 or 4 BASIC programs + sprite animation, and offered a lot of extensions for graphics and sound.
Many of these extensions led to equivalent ones written by enthusiasts or published with source in various books. E.g. a number of the books available here [1] have such extensions in them.
Later you got things like GEOS that provided a fully windowed user interface with word processor, spreadsheet, ide's etc. GEOS only really worked decently if you got a hard drive or RAM expansion, though, both of which were rare.
The most common "OS rework" on the C64, though, were freezer cartridges. So named because they had a "freeze" button that would jump into the code on the cartridge without touching anything in main memory, and let you mess around with it, and then restore the original state or dump everything to disk. Which made things like copying games fairly trivial most of the time, but also allowed for things like semi-automated "trainers" (you'd freeze, enable an option, go back into a game, lose a life, freeze again, and the cartridge would scan memory looking for memory locations that were likely locations of life counters, and scan for code that modified them).
There were a bunch of them, with Action Replay and The Final Cartridge being the most popular brands. Action Replay VI being the last one or the C64, but there are Action Replay cartridges and even PC expansion cards for a bunch of other platforms. The Final Cartridge ironically was also released in multiple version, with The Final Cartridge III also including a windowed user interface (which did not end up getting used much, as only very rudimentary software was included and hardly any third party software supported it).
In terms of contemporary stuff, there's the C64 version of Contiki [2] at least, and I'm sure there's more.
[1] http://www.bombjack.org/commodore/books.htm
[2] https://www.c64-wiki.com/index.php/Contiki