The OP mentions Jackson. Jackson 1.x -> 2.x changed the package (so they are wholly separate libraries), Jackson 2.x -> 3.x will do it again. I've never had backwards compatibility issues with Jackson. I don't know what someone could be doing in jackson-module-scala that breaks with a minor version bump.
But Scala somehow seems to take all of that and intensify it. I have heard that Scala 3 is supposed to improve this, but I don't anticipate being able to adopt Scala 3 before the heat death of the universe.
Scala just adds more drugs to the cocktail.
When I first came to Java from .NET, I was initially shocked at how heavily it was used in the Java space. (.NET arguably has even more powerful reflection facilities than Java does, but they are typically avoided at all costs.) But it didn't take me very long to realize that many of the C# language features I would have chosen to use instead of reflection simply don't have equivalents in Java. So, I don't think that it's necessarily that Java developers are addicted to this stuff, so much as that they just haven't been given any other choice.
Ground zero for most of them was Guava, which I unfortunately could not avoid due to it being a transitive dependency. And, even more unfortunately, Guava types were being exposed on the direct dependency's public API, so package relocation was not an easy option.
What the article describes with Scala rings true, and I stopped using it (due to a job change) more than 2 years ago! So Scala was already going down this route 2 years ago.
It is still not a typical experience. In our team, we try _very_ hard to avoid reflection and these errors are very very rare (never in Production).
[1] - https://www.jbang.dev/documentation/guide/latest/dependencie...
A configurable middleground is needed, but it's not an easy task to get that done.
Doesn't solve the problem completely because it is normal that applications need libraries to interoperate; consider case where application is passing objects between two libraries that both use types from a third library.
The Java 9 Modules architecture would have been an opportunity to introduce something like that, but alas they didn’t.
I think that if you truly want to solve the problem, you need to extend the definition of the fully qualified classname to include the notion of a breaking version reference. Kind of like what folks now sometimes do the hard way by encoding it into the package name.
This would make it possible to directly address the two different versions from the same calling source file.
Best bet it to be picky about dependencies that respect backward compatiblity or do name changes (like commons collections4)
The issue is one of granularity of dependency: it can't be calculated from the code and dependency graph easily. Each class or function invocation has an associated metadata version, and something that tracks all the sub-ones, with compatibility matrices across the major versions of the library?
As in, a dependency / linked library has a new version. How do you know if that can be safely updated in your code? All you have fundamentally is a coarse version number for the entire library, it's types/classes, it's functions/procedures/methods/signatures. Does anything indicate which signatures changed aside from compilers? Does anything mark which functions have new logic in them?
Yikes. That can't be maintained by humans. No language has dared to do something that would resolve to that level of compatibility analysis. Can you imagine the time involved?
Maybe there's simpler indicators. Git repo as a service might be able to use the repo history to track what parts of a library actually changed with some metadata file at a sufficiently granular level.
Currently, convention dictates that you use a new class name or namespace or something similar for "breaking" changes, or a major version number boost with some assumed "service life" for the previous library major version. You know, if they do that. But there's instances (one of the java bytcode emission libraries IIRC) where the api remainined the same but was a breaking change.
Nothing exists to help a library maintainer be aware of what will be a breaking change. At granular levels that is inherent to the language design. At best, you have unit tests as the actually deep introspection of the interface and expected responses. But that is per-project.
As mentioned, you can also shade and dynamically renamespace a library. That almost is the solution, but shared libraries are a thing, and a if you have 10 libraries each shaded-using a common util library (ahem, slf4j, first item in the blog post), then that's precious ram wasted, even in modern RAM sizes. With three-deep versions like 4.0.7 or something similar, how would a build system know that it could safely reuse a shaded version? It basically can't. There's no language constructs for saying that it can.
To some degree I believe this a failure, in the case of Java, of some sort of overarching organization that tracks commons libraries, and while maybe not placing them in the language standard library, recommending best practices around those libraries across known usage patterns so library import/compiling can be gradually optimized, or common conflicts alleviated. It's almost like security patching to some degree. Maven and its repos aren't up to the task.
You have a new language? Likely nothing in it will killer app it, instead what makes modern new languages succeed isn't just novel language features, it's community and herding cats. It's what Zig just learned from one tweet, what Lisp's community never really learned for what otherwise should be the "ultimate" language. It's how Rust has made progress, and maybe Go too.
I wonder if in 200 years if there will actually be mature, stable languages with mature, stable libraries that haven't needed to change for a couple decades.
Anyway, "compatibility" is a hard hard hard problem.
Kotlin is unique in the sense that the language originated from a company whose primary business it is to provide developer tooling and IDEs (intellij). They know a thing or two about what developers like and appreciate in their products.
It shows. This language is very nice to use for programmers. Lot's of clever syntax that minimizes boiler plate. Lot's of clever IDE support. Lot's of convenient features. Good build tooling. And so on. In contrast, Scala was always a bit all over the place. Lot's of different takes on how to do things; not all of them very successful in retrospect, tooling was a bit iffy, and so on.
It's the reason Kotlin got popular with Android developers early on because it provided a massive improvement over plain Java and sort of worked as a drop in replacement and was easy to switch to. As soon as Kotlin became available (pre 1.0), the Android community was all over it because at the time they were stuck with a pretty outdated Java language version. It did not take long for Google to make it the language of choice and start providing lot's of Kotlin focused updates to their SDK.
For the same reason, it got popular with Java backend developers early on. It doesn't really matter what Java backend framework you use; Kotlin is a drop in replacement for Java and the chances are pretty good that your framework of choice already has extensive Kotlin support in the form of Kotlin DSLs, extension functions, etc. That's certainly true for Spring/Spring Boot, Quarkus, and most other mainstream server side Java frameworks you can name.
The comparison with Scala is fair because a lot of Kotlin libraries are inspired by work in the Scala community. Additionally, it seems a lot of former Scala developers are now doing a lot of Kotlin. For better or for worse, it seems to compete pretty effectively with Scala. Either language of course is a pretty big improvement over Java; even with the improvements that Oracle has been trickling in over the last few years.
Hah, I wish that was true. Looking at the last releases and the new UI which is about to come I highly doubt that is true. Although adding flashy new features instead of fixing bugs in a "visual" IDE is different than developing a programming language.
On the technical side
* I don't like the idiomatic Kotlin approach to concurrency. Its coroutines are more difficult and unusual than typical Future-based libraries in other JVM languages.
* I'm not convinced null safety is a big enough question to justify multiple cryptic operators.
Personally I dread the demise of Scala. The Haskell lovers and the Akka license changers are certainly squeezing regular developers out big time. Databricks built a C++ engine, Rust is increasingly popular in other new OSS query engines. Those trends don't bode well at at all. Java is more expressive than golang but I'd rather use something more modern than either of them.
I am. I've been working on a large c++ project recently and almost all the crashes in it would be solved by compiler- enforced null safety. Kotlin's null-safe operators don't look cryptic to me, they're more or the less the same as those from C#, Typescript etc. OTOH Java seems to go out of its way to make it hard to write null-safe code.
You really feel like you can do anything with it lately.