Dependency hell
I bet there is no one among you who is unfamiliar with the concept “Dependency hell”.
This problem exists in all technologies and languages. People feel distracted and helpless in the face of this problem. I personally hate dependency conflicts in JavaScript (npm), Ruby, Python. I just can do nothing to solve them.
But fortunately, I learnt how to solve dependency conflicts in Java. At least in Maven and Gradle – two most popular build tools in Java world. Let me share my knowledge with you.
Background
As an author of popular open-source library Selenide, I often have to investigates issues caused by Selenide dependencies, not Selenide itself. Selenium, Guava, WebDriverManager, Netty to name a few.
Let me show some examples.
Sample project
Let’s assume that you have a project using Selenide for automated UI tests. You can refer to todo mvc tests if you wish.
While you have only a dependency on Selenide, everything works fine.
build.gradle:
dependencies { testImplementation("com.codeborne:selenide:5.14.0") }
or pom.xml:
<dependencies> <dependency> <groupId>com.codeborne</groupId> <artifactId>selenide</artifactId> <version>5.14.0</version> <scope>test</scope> </dependency> </dependencies>
First problem
Now let’s assume you want to use TestNG in your project. Then you add a dependency:
build.gradle:
dependencies { testImplementation("org.testng:testng:7.3.0") testImplementation("com.codeborne:selenide:5.14.0") }
or pom.xml:
<dependency> <groupId>org.testng</groupId> <artifactId>testng</artifactId> <version>7.3.0</version> <scope>test</scope> </dependency>
This is the road to dependency hell. 🙂
Let’s run the tests
If you run mvn test
or gradle test
, your tests fail. Just because you added testng*.jar
– even if you don’t use it yet.
Let’s look at the test report
Your tests have failed with the following error:
java.lang.NoSuchMethodError: 'java.util.stream.Collector com.google.common.collect.ImmutableList.toImmutableList()' at org.openqa.selenium.chrome.ChromeOptions.asMap() at org.openqa.selenium.MutableCapabilities.merge() at com.codeborne.selenide.webdriver.MergeableCapabilities
Errors like “NoSuchMethod” clearly indicate that you have some mismatching versions of dependencies.
In this case, dependency selenium-chrome*.jar
(which contains ChromeOptions
) was built with one version of guava-*jar
(which contains class ImmutableList
), but is being run with another. We are stuck. Too many people don’t know what to do with this problem.
The cure
Luckily, both Gradle and Maven have a simple command that prints out a dependency tree of your project. Let’s try it.
Maven
run command mvn dependency:tree
in terminal.
org.selenide.examples:todomvc:jar:1.0-SNAPSHOT +- org.testng:testng:jar:7.3.0:test | +- com.google.inject:guice:jar:no_aop:4.2.2:test | | \- com.google.guava:guava:jar:25.1-android:test +- com.codeborne:selenide:jar:5.14.0:test | +- org.seleniumhq.selenium:selenium-java:jar:3.141.59:test | | +- org.seleniumhq.selenium:selenium-chrome-driver:jar:3.141.59:test
Gradle
run command gradle dependencies
in terminal.
testRuntimeClasspath - Runtime classpath of source set 'test'. +--- org.testng:testng:7.3.0 | +--- com.google.inject:guice:4.2.2 | | +--- javax.inject:javax.inject:1 | | \--- com.google.guava:guava:25.1-android +--- com.codeborne:selenide:5.14.0 | +--- org.seleniumhq.selenium:selenium-java:3.141.59 | | +--- org.seleniumhq.selenium:selenium-api:3.141.59 | | +--- org.seleniumhq.selenium:selenium-chrome-driver:3.141.59 | | | +--- org.seleniumhq.selenium:selenium-remote-driver:3.141.59 | | | | +--- com.google.guava:guava:25.0-jre -> 25.1-android
You see a similar output: both Maven and Gradle decided to use version guava:25.1-android
instead of guava:25.0-jre
– and it’s the wrong version.
Actually, the latest version of Guava is 30.0-jre
nowadays (you can check using “package search” site). But none of Selenide dependencies use the latest version. Almost all of them depend on guava:25.0-jre
, and only one depends on guava:25.1-android
. It’s testng:7.3.0
-> guice:4.2.2
-> guava:25.1-android
.
It’s a bug of guice:4.2.2
, if you ask me. By the way, it was fixed in guice:4.2.3
which depends on guava:27.1-jre
.
How to fix it?
Now we know that the problem is with guava version. But how to fix this problem?
There is no single correct answer. It depends on your needs.
Most probably you don’t actually need Guice to run your tests with TestNG. I would probably suggest to exclude all transitive dependencies of TestNG in this case.
build.gradle:
dependencies { testImplementation("org.testng:testng:7.3.0") { transitive = false } testImplementation("com.codeborne:selenide:5.14.0") }
or pom.xml:
<dependency> <groupId>org.testng</groupId> <artifactId>testng</artifactId> <version>7.3.0</version> <scope>test</scope> <exclusions> <exclusion> <artifactId>com.google.inject</artifactId> <groupId>guice</groupId> </exclusion> </exclusions> </dependency>
It’s not ideal, but it works.
See detailed case description here.
Now let’s look at another example.
SECOND PROBLEM
Selenide has method $("input").download()
for downloading files. It may use different ways to fetch the file, one of which is using embedded proxy server.
If you want this way, you are going to add a proxy dependency:
dependencies { testImplementation("com.codeborne:selenide:5.14.0") testRuntimeOnly("com.browserup:browserup-proxy-core:2.1.1") }
and write a test, something like that:
open("https://the-internet.herokuapp.com/download"); File file = $(byText("some-file.txt")).download(); assertThat(file.getName()).isEqualTo("some-file.txt");
But your test fails with a strange error:
org.openqa.selenium.WebDriverException: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED ... at com.codeborne.selenide.Selenide.open(Selenide.java:49) at org.selenide.selenoid.FileDownloadTest.download(FileDownloadTest.java:45)
Too many people have been complaining about that… 🙁
Most people start a panic here. But all you need to do is just read the log carefully. You will see the error:
[LittleProxy-0-ProxyToServerWorker-1] ERROR org.littleshoot.proxy.impl.ProxyToServerConnection - (HANDSHAKING) [id: 0xc05a41d5, L:/10.10.10.145:56103 - R:the-internet.herokuapp.com/52.1.16.137:443] : Caught an exception on ProxyToServerConnection java.lang.NoSuchMethodError: 'int io.netty.buffer.ByteBuf.maxFastWritableBytes()' at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:86)
Gotcha! We see NoSuchMethodError
which usually signals a versions conflict between your dependencies.
This time dependency browserup-proxy
was compiled with one version of Netty, but is being run with another.
Let’s run our familiar command gradle dependencies
or mvn dependency:tree
:
\--- com.browserup:browserup-proxy-core:2.1.1 +--- io.netty:netty-codec:4.1.44.Final +--- xyz.rogfam:littleproxy:2.0.0-beta-5 | +--- io.netty:netty-all:4.1.34.Final
We have two jars with different versions: netty-codec:4.1.44.Final
and netty-all:4.1.34.Final
.
It seems to be weird: library browserup-proxy-core
uses netty-codec
(and excludes netty-all
, by the way). But its dependency littleproxy
does use netty-all
(which presumably contains netty-codec
and some other netty sub-jars, hugh?). Obviously, they do conflict with each other.
How to fix it?
Again, there is no single correct answer.
There are many ways to fix the problem. Probably the easiest one is to declare Netty versions explicitly in build.gradle
or pom.xml
:
testRuntimeOnly("io.netty:netty-all:4.1.54.Final") testRuntimeOnly("io.netty:netty-codec:4.1.54.Final")
Now command gradle dependencies shows that Netty versions are matching:
\--- com.browserup:browserup-proxy-core:2.1.1 +--- io.netty:netty-codec:4.1.44.Final -> 4.1.54.Final +--- xyz.rogfam:littleproxy:2.0.0-beta-5 | +--- io.netty:netty-all:4.1.34.Final -> 4.1.54.Final
The test runs, proxy works, file is being downloaded. Everyone is happy. See detailed case description here.
Summary
I showed two examples of real-life problems caused by dependency conflicts. We saw how to investigate them using Maven or Gradle built-in tools. Now you know that it’s not impossible. You can do it. 🙂
Moral
Know your tools. Read your logs. Don’t be afraid.
Exclude as many dependencies as you can.
And feel like in dependency heaven.
Johannes Bühler December 2, 2020
Hi Andrei,
happy to read that you choose an interesting topic for the 2nd day of the advent of code. You might consider solving the 2 problem differently and more the gradle way.
The First Problem can be solved with constraints:
dependencies {
....
constraints {
implementation("com.google.guava:guava:27.1-jre") {
because("latest compatible version for all transitive dependencies")
}
}
}
see: https://docs.gradle.org/current/userguide/dependency_constraints.html
Instead of excluding all transitive dependencies you could also just exclude a certain dependency (https://docs.gradle.org/current/userguide/dependency_downgrade_and_exclude.html#sec:excluding-transitive-deps).
The Second Problem is a great showcase to let the usage of a deployed platform bom shine:
dependencies {
components.all()
....
}
open class NettyBomAlignmentRule: ComponentMetadataRule {
override fun execute(ctx: ComponentMetadataContext) {
ctx.details.run {
if (id.group.startsWith("io.netty")) {
// declare that Netty modules belong to the platform defined by the Netty BOM
belongsTo("io.netty:netty-bom:${id.version}", false)
}
}
}
}
see: https://docs.gradle.org/current/userguide/platforms.html
Another alternative to the platform approach would be to define a resolution strategy (https://docs.gradle.org/current/userguide/resolution_rules.html)
Please find the complete improved example here: https://github.com/jonnybbb/todomvc
A nice way to interactively query the dependencies can be experienced with build scans: https://scans.gradle.com/s/oo7ovsmyyn2jq/dependencies?dependencies=netty&expandAll
Andrei Solntsev December 3, 2020 — Post Author
@JOHANNES BÜHLER
Wow, thank you for the knowledge sharing!
These are really powerful features of Gradle. Though, they seem to be too complex for most of users. I am afraid very rarely people do use them.
Anyway, thank you again. I have merged your improved example to https://github.com/selenide-examples/todomvc
motasem July 28, 2021
happy to read this amazing article, but are you sure that dependency hell , is an npm problem , i see in their documentation that the node module loader was written exactly to solve this problem
Andrei Solntsev July 28, 2021 — Post Author
Exactly, that’s the difference between theory and practice!
In theory, npm module loader was written to solve this problem.
In practice, people still complain on npm…
Motasem July 28, 2021
And who have the right, npm or the people
Who’s didn’t understand node module system
Andrei Solntsev August 1, 2021 — Post Author
Look, the post was not about npm.
The post was about resolving dependency conflicts in Java (Maven/Gradle).
If there are good tools for resolving conflicts in npm – feel free to share it. Be constructive!
Motasem July 28, 2021
I believe that we misunderstood that node modules doesn’t have to be global, once we get this, we will never face dependency hell in nodeJs applications