Hacker News Reader

Comment by badpun | Hacker News Reader

Java has the same dependency issues? I vividly remember spending days on jars shading in Maven. Basically, a single approach that would solve this, irrespective of the language, would be isolating dependency trees of every library from each other, essentially duplicating the code of many common libraries in RAM (like Docker, but for libraries inside the same process). There's an option to do that in Maven, but it's not the default. And it still would not work for dependencies where there should be only a single instance of it in RAM, like for example loggers...

View on HN

I only very rarely see this in Java, even in huge monoliths. Not that there's a technical reason for this; I think there's a cultural expectation that you don't break backwards compatibility with Java libraries. Perhaps the Scala world is more cutting-edge and fluid so there are different cultural expectations.

The OP mentions Jackson. Jackson 1.x -> 2.x changed the package (so they are wholly separate libraries), Jackson 2.x -> 3.x will do it again. I've never had backwards compatibility issues with Jackson. I don't know what someone could be doing in jackson-module-scala that breaks with a minor version bump.

Jackson doesn't have a stable API for building modules. You have to get elbow deep in their internals, which is already scary enough given that it's an old and highly performant library, but the bad part is that there's no formal commitment to maintaining compatibility. You just have to hope that the modules you use are popular enough that they will test them and make an effort not to break them. I've had an application break on a patch version bump of Jackson.

I have personally spent so much time fussing with runtime ClassDefNotFound and MethodNotFound errors in Java that I am no longer able to respond to assertions that Java is type-safe with anything but an incoherent stream of cursing and rage tears.

But Scala somehow seems to take all of that and intensify it. I have heard that Scala 3 is supposed to improve this, but I don't anticipate being able to adopt Scala 3 before the heat death of the universe.

Not sure what you're doing with Java but that's not a common problem at all. Are you just aggressively stripping code and dealing with the fallout?

Working on projects that involve big frameworks with deep and ever-shifting dependency trees. So upgrading an existing dependency or taking a new one could often be a risky move.

Ground zero for most of them was Guava, which I unfortunately could not avoid due to it being a transitive dependency. And, even more unfortunately, Guava types were being exposed on the direct dependency's public API, so package relocation was not an easy option.

Guava is an outlier. While they do honor semver, they’re already on version 31.1, meaning they have felt they needed to break backward compatibility dozens of times, and it’s likely your dependencies require conflicting versions just by accident of when they were written. I guess it’s because the maintainers live in a monorepo and just force any changes they need?

The other interesting corollary to this is: if your package relies on one like Guava, and it is not bumping its major version number every time it updates the version of Guava it depends on, then your package does not itself honor semver.

Much like libraries that run on Scala 2.11 or 2.12, I can imagine a system where I maintain several compatible minor versions (or named forks) of a library, each patched to use a different major version of Guava, and letting a caller solve for exactly which version of everything is compatible with everything else. But the more API churn we face, the harder this is.

Incompatible versions of transitive dependencies, incompatible libs that your web server added, this is the default in Java unless you pay close attention to your dependencies.

That wasn't my experience with Java back in the day.

What the article describes with Scala rings true, and I stopped using it (due to a job change) more than 2 years ago! So Scala was already going down this route 2 years ago.

> That wasn't my experience with Java back in the day

It is still not a typical experience. In our team, we try _very_ hard to avoid reflection and these errors are very very rare (never in Production).

Those sorts of java runtime errors largely depend on programmers' insistence on doing things via reflection, and it will be the same in any programming language: Like most drugs, at first reflection seems you've opened a new door into enlightenment, until eventually you find yourself sleeping in a gutter with half your teeth missing and wondering why it didn't work out as expected. A lot of Java open source is addicted to this stuff.

Scala just adds more drugs to the cocktail.

I don't think that this is entirely fair to Java programmers. Java's designers decided early on to lean fairly hard on reflection. Not just by giving Java strong reflective capabilities, but by sometimes deliberately choosing not to give Java great facilities for solving problems without using reflection.

When I first came to Java from .NET, I was initially shocked at how heavily it was used in the Java space. (.NET arguably has even more powerful reflection facilities than Java does, but they are typically avoided at all costs.) But it didn't take me very long to realize that many of the C# language features I would have chosen to use instead of reflection simply don't have equivalents in Java. So, I don't think that it's necessarily that Java developers are addicted to this stuff, so much as that they just haven't been given any other choice.

Java's type system supports code gen. See projects like the Checker framework or NullAway

The errors are created by failures on finding dependencies.

JBang [1] can help a lot here, especially with the new "BOM POM" feature. The experience feels closer to Go than Maven.

[1] - https://www.jbang.dev/documentation/guide/latest/dependencie...

Jep, this is really an issue of how the JVM and Classloaders are architected. Rust essentially chooses the other side of the sea, which is better in some sense, but worse in others (which we will see once the ecosystem grows).

A configurable middleground is needed, but it's not an easy task to get that done.

Java has the same dependency issues. But, from the article, it seems like there might be something cultural within the Scala community that contributes to the problem. Most Java libraries, in my experience, are relatively flat. The article gives an example of HikariCP, which depends on a random JDBC library from Akka. Obviously, that situation is going to end terribly given the popularity of Akka. As soon as Akka updates their library version, HikariCP has to as well or the two dependencies will get hopelessly out of sync. I don't know what `akka-persistence-jdbc ` does, but it can't be so important that it justifies the crazy dependency structure.

HikariCP is a Java library I thought, at least I’ve used it in many Java projects, so I’m not sure what that has to do with Scala.

Scala doesn't care about backwards compatibility. They break stuff all the time, And so major libraries are tenuously hanging on to whatever version they were targeted at. Akka prints out messages if it finds jars with even different minor versions.

Dependency management is painful almost everywhere. I see a glimmer of hope for the future in Unison, a language in which code exists as content-addressed ASTs. Dependency version conflicts cannot exist; there is not even a need to resolve a set of dependency versions.

https://www.unison-lang.org/learn/the-big-idea/

> Basically, a single approach that would solve this, irrespective of the language, would be isolating dependency trees of every library from each other

Doesn't solve the problem completely because it is normal that applications need libraries to interoperate; consider case where application is passing objects between two libraries that both use types from a third library.

Then your app would explicitly be doing the impendance mismatch resolution (converting objects from one version of library to the other) when doing the library calls.

which someone would have to code in as that's not automatically possible all the time.

You're pretty much screwed in that case but that's kind of by design and the entire point of versioning, isn't it?

A major problem is that Maven (and friends) doesn’t distinguish between API dependencies (the API of a library using types from the API of another library) and implementation dependencies (a library depending on another library as an implementation detail). If that distinction were made consistently, implementation dependencies could indeed be separated into a dedicated classloader for each library, and you’d only have to worry about API dependencies, which would go a long way.

The Java 9 Modules architecture would have been an opportunity to introduce something like that, but alas they didn’t.

OSGi allowed you to specify in excruciating detail what dependencies you had, whether API (i.e you wanted to re-export types from another library) or not, and much more. Unfortunately, it became far, far more complex than just implementing a simple module system, so as most things from the era (early 2000's) it became a huge, complex beast no one wanted to touch anymore.

I believe that separating interface from implementation doesn't really solve the problem, it just shrinks it. E.g. what if you depend on two libraries that re-export classes from different versions of a third library?

I think that if you truly want to solve the problem, you need to extend the definition of the fully qualified classname to include the notion of a breaking version reference. Kind of like what folks now sometimes do the hard way by encoding it into the package name.

This would make it possible to directly address the two different versions from the same calling source file.

I agree, but I believe the “shrinkage” would already reduce the problem substantially, and maybe more importantly, would make developers more careful about what they include as interface dependencies instead of as just implementation dependencies.

One thing is Scala plays fast and loose with backwards compatibility. So much so that akka actually prints out messages if they find jars with different versions. It's a pain in the arse.

Essentially OSGI is that. It is still a ballache tho, and slow to boot, so I would not recommend it.

Best bet it to be picky about dependencies that respect backward compatiblity or do name changes (like commons collections4)

Dependency conflicts is a fundamental issue of mature ecosystems, linux especially included.

The issue is one of granularity of dependency: it can't be calculated from the code and dependency graph easily. Each class or function invocation has an associated metadata version, and something that tracks all the sub-ones, with compatibility matrices across the major versions of the library?

As in, a dependency / linked library has a new version. How do you know if that can be safely updated in your code? All you have fundamentally is a coarse version number for the entire library, it's types/classes, it's functions/procedures/methods/signatures. Does anything indicate which signatures changed aside from compilers? Does anything mark which functions have new logic in them?

Yikes. That can't be maintained by humans. No language has dared to do something that would resolve to that level of compatibility analysis. Can you imagine the time involved?

Maybe there's simpler indicators. Git repo as a service might be able to use the repo history to track what parts of a library actually changed with some metadata file at a sufficiently granular level.

Currently, convention dictates that you use a new class name or namespace or something similar for "breaking" changes, or a major version number boost with some assumed "service life" for the previous library major version. You know, if they do that. But there's instances (one of the java bytcode emission libraries IIRC) where the api remainined the same but was a breaking change.

Nothing exists to help a library maintainer be aware of what will be a breaking change. At granular levels that is inherent to the language design. At best, you have unit tests as the actually deep introspection of the interface and expected responses. But that is per-project.

As mentioned, you can also shade and dynamically renamespace a library. That almost is the solution, but shared libraries are a thing, and a if you have 10 libraries each shaded-using a common util library (ahem, slf4j, first item in the blog post), then that's precious ram wasted, even in modern RAM sizes. With three-deep versions like 4.0.7 or something similar, how would a build system know that it could safely reuse a shaded version? It basically can't. There's no language constructs for saying that it can.

To some degree I believe this a failure, in the case of Java, of some sort of overarching organization that tracks commons libraries, and while maybe not placing them in the language standard library, recommending best practices around those libraries across known usage patterns so library import/compiling can be gradually optimized, or common conflicts alleviated. It's almost like security patching to some degree. Maven and its repos aren't up to the task.

You have a new language? Likely nothing in it will killer app it, instead what makes modern new languages succeed isn't just novel language features, it's community and herding cats. It's what Zig just learned from one tweet, what Lisp's community never really learned for what otherwise should be the "ultimate" language. It's how Rust has made progress, and maybe Go too.

I wonder if in 200 years if there will actually be mature, stable languages with mature, stable libraries that haven't needed to change for a couple decades.

Anyway, "compatibility" is a hard hard hard problem.

Reply on news.ycombinator.com