The F# compiler codebase is smaller than that of the C# compiler, but only in line count. F# code is massively more compact than C#, which is another way of saying that the concepts are dense too. There are some unique aspects of F# - custom metadata format for things not representable in C# or VB but must be carried over across F# projects, type inference and the massive number of rules for typechecking, a core library acting as a de facto replacement for a subset of the .NET libraries, custom IL reader/writer (this predates System.Reflection.Metadata), and many more - that lead to an enormous concept count that is anything but simple. Luckily, the codebase is quite maintainable and despite countless qualms I personally have with how things are organized, it's proven to be an excellent foundation for continually adding new features over the years.
While I don't get to use it nearly as often as I would like, every time I have it's been really fun to play with/use.
One of the nicest things is that I find is that every time I get to play with it, the language is intuitive enough and IDE support is rich enough that I'm never feeling like I've forgotten how to work with it.
I agree that the F# compiler's codebase is much smaller than C# (wc suggests 417k lines for dotnet/roslyn:src/Compilers/CSharp/ vs 190k lines for dotnet/fsharp:src/, which is hopefully all the right files).
But as far as simpler goes, the compiler is quite a bit complicated by having to implement F# in terms of .Net. Enum variants are actually classes inheriting from the class representing the enum, modules are actually classes, core library members being called by different names in F# vs not, the desugaring of computation expressions into state machines, ...
Then there's F#-specific stuff, like the core library, having two different kinds of parsers (#light on and off), two different kinds of generics, quotations, ...