Programming Language Scalability
blog.sulami.xyz
blog.sulami.xyz
> Scalability is often at odds with peak effectiveness, the maximum effectiveness of an engineer who is intimately familiar with the codebase...
I would think we would want to use the term "scalability" in this kind of context to say something more like "a programming language is more scalable if it makes it easier to have large numbers of engineers all working on features of a single app concurrently without stepping on each other's work (even if they have to go to great effort to learn how to do so)"?
(edit addition 25 minutes later:) As a maybe overly-trivial example, it can be really irritating to work with other developers in an object-oriented programming language that requires all the fields or methods of a class to be declared in one place, as you can end up with constant merge conflicts. If you can create a new file and declare fields and methods that extend an existing class -- preferably with the fields in that new file being scoped to only being accessible to methods in that new file -- that can be a massive help (even if it makes other specific aspects of understanding the functionality of the class a bit more difficult).
But I do like your usage too. We do need new words here.
This is optimizing for a wrong (minor) problem.
As the application grows, both accidental and essential complexity grows. More and more of engineers' time is spent analyzing existing code, and they typically spend less time writing new code. The limiting factor in large/old codebases (which is where the word "scalability" comes into play IMHO) is the inability of even experienced engineers to understand aspects of the codebase.
This is why I think e.g. Java has been so successful in the "big & complex" space. Java is annoying to write, but easy to read. I mean, some more conciseness would make it easier to read, but it's still quite OK. Very importantly, it's OK to read even if the dev writing it wasn't good / didn't observe best practices etc. I often have to deal with code written 20 years ago, and you can't change the past ...
Regarding the merge conflicts - they happen, most of the time causing only minor problems, sometimes it's a bigger deal. Some programming styles can help you avoid some minor merge conflicts, but I doubt they will be effective in avoiding the real troublesome merge conflicts (which go far beyond introducing a new field).
But the author's rankings look extremely reasonable. All companies have management problems and some languages cope better with that than others. Another way of interpreting this is "What costs does the language impose on programmers doing something novel" - usually novel is a bad sign for long term success.
Is that a virtue of the language (Clojure/lisp/any other) or a virtue of the 5 experienced programmers?
I ask because I’ve never delved deep into any language of the lisp family but I have seen that small experienced teams tend to produce good work regardless of the stack they choose. My guess is the recipe for good outcomes is: experience with the chosen stack + experience working together + small-ish team.
But Clojure is a language that supports [0] that sort of team better than Java or Python (I won't say Rust or Go because it isn't really a first choice for systems programming). The Clojure community also has a cultural expectation that teams will be small and highly skilled, for obvious reasons.
[0] I don't see how this sort of statement could ever be anything other than opinion, so I won't try to justify the assertion.
Bu that definition Java is probably the best just due to it's presence in teaching and while I prefer python, better threading support ...
but I do look forward to the rise of LOGO!!!
? FORWARD 20
? RIGHT 90
? FORWARD 20
? HIDETURTLE
?
(Really items like the available talent pool and what languages align best with business needs are better metrics, I'll put on my flame proof pants and say language choice is an architecturally insignificant implementation detail outside of concerns like the above)
Keeping things simple keeps them scalable. Cleverness and magic are antithetical to that goal.
This is the astute point of this article. It transcends the domain at hand: interchangeable software developers. It basically hits any professional talent pool. It even applies to various tools, machines, products we use in every day life.
The question one has to ask is what price one is willing to pay for scalability. And whether you’re paying for something you really need or not.
I agree with the framing that scaling a language is about scaling to more engineers easily.
Having said that, you can write scalably in any language, and write unscalably elsewhere. This is a bigger factor on scalability overall by far, than the language itself.
For example, I can make unmaintainable macro magic in rust. I can avoid dynamic pitfalls by using pydantic effectively in python.
Anyway, my 2c is speed to delivery of features matters more than scalability as defined here. Again, that depends more on technical components that have been invested in, rather than the selected language.
Rust is mainstream and boring?
No magic. No configuration. No cleverness. No DRY. Will it be beautiful? Not at all. Look at LLMs. We can make things that talk like people using ugly fucking matrix math and recursion so long as we keep it simple. We can scale it to infinity if we keep it simple.
It doesn’t matter what it is. QBASIC, JavaScript, Rust, jet engines, pneumatic circuits, CPU cores, manufacturing facilities… the list goes on.
Keep it so that it can be reasoned through by a single person and you can scale it forever.
The problem is that a lot of simple processes combined are no longer simple. So you need patterns which are general enough to write less processes, without being so general that developers write processes which are unintuitive and therefore complex by themselves.
My opinion is that tooling (perhaps with or without LLMs) can achieve the best of both worlds, by creating a simple query/projection (something in between) of complex code. Then the developer can get and mutate whatever properties they want from the code, without seeing unrelated information, and seeing seemingly-not-but-actually-related information presented in a concrete (simple) way.