> I mean more the type systems in languages like Scala and Haskell, think features like existential and dependent types. Type systems like Java’s and Typescript’s are great! They add value! But you can be an order of magnitude or two more safe with more advanced type systems and it’s profoundly beneficial when you learn how to make good use of them.
> Unfortunately, type-safety didn’t help me very much when developing my big feature....
I feel like the author is unintentionally reinforcing certain stereotypes around type systems. Typescript actually has an insanely powerful type system, in many ways much more advanced than Scala's type system and certainly more advanced than "vanilla" Haskell (i.e. something like Haskell2010). What often gives people the impression that a type system like Haskell is "strong" (or in the even more extreme case a type system like Elm, which has a reputation also for being "advanced" but is actually one of the simplest possible type systems with polymorphism) is that other features (mainly stuff around immutability) combine with the type system to enhance the guarantees that the type system can give. Very roughly speaking it is very hard to "go behind the type system's back" in these languages. However, the actual type system is not terribly advanced.
It's also weird to mention dependent types here because dependent types would have precisely solved the example the author brings up (this is leaving aside other issues that current dependently typed languages have).
For example in Idris you could write something like
firstCharOfString : String -> Maybe Char
firstCharOfString = ...
processData :
(firstNameLetter : Maybe Char) ->
(name : String) ->
(firstCharOfString name = firstNameLetter) ->
Result
processData = ...
Now you can't do this in Scala (although you can achieve something like dependently typed smart constructors with path-dependent types in Scala, but it's very fragile and I wouldn't recommend it) so I agree that this may not be the way you want to do things in Scala.To go back up a level, the way that non-dependent types of the sort you'd find in Scala help you when writing a compiler is when you split your compiler into different passes. You can write a statically typed representation at the end of each pass to act as an "anchor" to cut down on certain classes of bugs both between two successive passes, but also to ensure that certain bugs can't span multiple passes.
I suspect the reason that the author wasn't getting a lot of mileage out of the Scala type system is due to a monolithic "compiler" design. If everything is a single giant pass then you get a lot less lift from the Scala type system (but this is true even if you substitute human understanding for a type system).
> This is provenance. It doesn’t sound that hard but in practice it was surprisingly challenging to implement.
Provenance is indeed very difficult, especially in the presence of things that change your representation, but not your semantics. It can really muck up your code to have to carry around a lot of baggage that's unrelated to your final result, but also has to be updated through the various stages of data transformation your result goes through as well.
> I recently saw a PR to Scala that’s a great demonstration of exactly this. The private and sealed keywords have been in Scala forever and someone only just recently came up with the realisation that private classes are effectively sealed. Scala’s been in existence for 16 years!
I feel like this kind of misrepresents the situation. The original bug was filed 9 years ago and only just now fixed because it was very low priority (which makes me think a lot of people have come to the same realization and just not really bothered about it). Regardless the original sentiment that different options intersect in a combinatorial explosion is something I can get behind.
> Would the code just do the exact same thing but with more boilerplate using IORefs or huge StateT stacks? Would Haskell be so efficient in its optimisation that a more straight-forward FP approach would be sufficiently fast?
When comparing Haskell and Scala the answer is generally yes, functional idioms tend to run a lot faster in GHC than with scalac. Not necessarily IORef (which can be actually surprisingly slow due to boxing issues), but StateT (and by extension State) is way faster in GHC than in scalac. Similarly IO in Haskell is faster than the attempts to emulate it with cats-effect or ZIO. You pay an overhead for FP in Scala that you don't always have to in Haskell.
That being said I still prefer to write performance sensitive FP code, that is still performance-tolerant enough to be conceivably written in something like Java or Go, in Scala if the only choices were Scala or Haskell. As the author mentions, mutability contained entirely within functions isn't really all that icky to me and my mental model for low-level unboxed array hacking in Scala or Java is more robust than my mental model for intricately weaving Haskell code to make sure all the requisite GHC optimizations fire in terms of being fairly confident what the overall optimization process would look like.