JVM Advent

The JVM Programming Advent Calendar

How to debug dependency conflicts in Maven and Gradle

Dependency hell

I bet there is no one among you who is unfamiliar with the concept “Dependency hell”.

This problem exists in all technologies and languages. People feel distracted and helpless in the face of this problem. I personally hate dependency conflicts in JavaScript (npm), Ruby, Python. I just can do nothing to solve them.

But fortunately, I learnt how to solve dependency conflicts in Java. At least in Maven and Gradle – two most popular build tools in Java world. Let me share my knowledge with you.

 

Background

As an author of popular open-source library Selenide, I often have to investigates issues caused by Selenide dependencies, not Selenide itself. Selenium, Guava, WebDriverManager, Netty to name a few.

Let me shown some examples.

Sample project

Let’s assume that you have a project using Selenide for automated UI tests. You can refer to todo mvc tests if you wish.

While you have only a dependency on Selenide, everything works fine.

build.gradle:

dependencies {
  testImplementation("com.codeborne:selenide:5.14.0")
}

or pom.xml:

 <dependencies>
  <dependency>
    <groupId>com.codeborne</groupId>
    <artifactId>selenide</artifactId>
    <version>5.14.0</version>
    <scope>test</scope>
  </dependency>
</dependencies>

First problem

Now let’s assume you want to use TestNG in your project. Then you add a dependency:

build.gradle:
dependencies {
  testImplementation("org.testng:testng:7.3.0")
  testImplementation("com.codeborne:selenide:5.14.0")
}
or pom.xml:
<dependency>
  <groupId>org.testng</groupId>
  <artifactId>testng</artifactId>
  <version>7.3.0</version>
  <scope>test</scope>
</dependency>

This is the road to dependency hell. 🙂

Let’s run the tests

If you run mvn test or gradle test, your tests fail. Just because you added testng*.jar – even if you don’t use it yet.

Let’s look at the test report

Your tests have failed with the following error:

java.lang.NoSuchMethodError: 'java.util.stream.Collector
      com.google.common.collect.ImmutableList.toImmutableList()'
  at org.openqa.selenium.chrome.ChromeOptions.asMap()
  at org.openqa.selenium.MutableCapabilities.merge()
  at com.codeborne.selenide.webdriver.MergeableCapabilities

Errors like “NoSuchMethod” clearly indicate that you have some mismatching versions of dependencies.

In this case, dependency selenium-chrome*.jar (which contains ChromeOptions) was built with one version of guava-*jar (which contains class ImmutableList), but is being run with another. We are stuck. Too many people don’t know what to do with this problem.

The cure

Luckily, both Gradle and Maven have a simple command that prints out a dependency tree of your project. Let’s try it.

Maven

run command mvn dependency:tree in terminal.

org.selenide.examples:todomvc:jar:1.0-SNAPSHOT
+- org.testng:testng:jar:7.3.0:test
| +- com.google.inject:guice:jar:no_aop:4.2.2:test
| | \- com.google.guava:guava:jar:25.1-android:test
+- com.codeborne:selenide:jar:5.14.0:test
| +- org.seleniumhq.selenium:selenium-java:jar:3.141.59:test
| | +- org.seleniumhq.selenium:selenium-chrome-driver:jar:3.141.59:test
Gradle

run command gradle dependencies in terminal.

testRuntimeClasspath - Runtime classpath of source set 'test'.
+--- org.testng:testng:7.3.0
| +--- com.google.inject:guice:4.2.2
| | +--- javax.inject:javax.inject:1
| | \--- com.google.guava:guava:25.1-android
+--- com.codeborne:selenide:5.14.0
| +--- org.seleniumhq.selenium:selenium-java:3.141.59
| | +--- org.seleniumhq.selenium:selenium-api:3.141.59
| | +--- org.seleniumhq.selenium:selenium-chrome-driver:3.141.59
| | | +--- org.seleniumhq.selenium:selenium-remote-driver:3.141.59
| | | | +--- com.google.guava:guava:25.0-jre -> 25.1-android

You see a similar output: both Maven and Gradle decided to use version guava:25.1-android instead of guava:25.0-jre – and it’s the wrong version.

Actually, the latest version of Guava is 30.0-jre nowadays (you can check using “package search” site). But none of Selenide dependencies use the latest version. Almost all of them depend on guava:25.0-jre, and only one depends on guava:25.1-android. It’s testng:7.3.0 -> guice:4.2.2 -> guava:25.1-android.

It’s a bug of guice:4.2.2, if you ask me. By the way, it was fixed in guice:4.2.3 which depends on guava:27.1-jre.

How to fix it?

Now we know that the problem is with guava version. But how to fix this problem?

There is no single correct answer. It depends on your needs.

Most probably you don’t actually need Guice to run your tests with TestNG. I would probably suggest to exclude all transitive dependencies of TestNG in this case.

build.gradle:
dependencies {
  testImplementation("org.testng:testng:7.3.0") { 
    transitive = false 
  }
  testImplementation("com.codeborne:selenide:5.14.0")
}
or pom.xml:
<dependency>
  <groupId>org.testng</groupId>
  <artifactId>testng</artifactId>
  <version>7.3.0</version>
  <scope>test</scope>
  <exclusions>
    <exclusion>
      <artifactId>com.google.inject</artifactId>
      <groupId>guice</groupId>
    </exclusion>
  </exclusions>
</dependency>

It’s not ideal, but it works.

See detailed case description here.

 

Now let’s look at another example.

SECOND PROBLEM

Selenide has method $("input").download() for  downloading files. It may use different ways to fetch the file, one of which is using embedded proxy server.

If you want this way, you are going to add a proxy dependency:

dependencies {
  testImplementation("com.codeborne:selenide:5.14.0")
  testRuntimeOnly("com.browserup:browserup-proxy-core:2.1.1")
}

and write a test, something like that:

open("https://the-internet.herokuapp.com/download");
File file = $(byText("some-file.txt")).download();
assertThat(file.getName()).isEqualTo("some-file.txt");

But your test fails with a strange error:

org.openqa.selenium.WebDriverException: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED
   ...
   at com.codeborne.selenide.Selenide.open(Selenide.java:49)
   at org.selenide.selenoid.FileDownloadTest.download(FileDownloadTest.java:45)

Too many people have been complaining about that… 🙁

Most people start a panic here. But all you have to do is just read the log carefully. You will see the error:

[LittleProxy-0-ProxyToServerWorker-1] ERROR org.littleshoot.proxy.impl.ProxyToServerConnection
- (HANDSHAKING) [id: 0xc05a41d5, L:/10.10.10.145:56103
- R:the-internet.herokuapp.com/52.1.16.137:443]
: Caught an exception on ProxyToServerConnection
java.lang.NoSuchMethodError: 'int io.netty.buffer.ByteBuf.maxFastWritableBytes()'
at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:86)

Gotcha! We see NoSuchMethodError which usually signals a versions conflict between your dependencies.

This time dependency browserup-proxy was compiled with one version of Netty, but is being run with another.

Let’s run our familiar command gradle dependencies or mvn dependency:tree:

\--- com.browserup:browserup-proxy-core:2.1.1
+--- io.netty:netty-codec:4.1.44.Final
+--- xyz.rogfam:littleproxy:2.0.0-beta-5
| +--- io.netty:netty-all:4.1.34.Final

We have two jars with different versions: netty-codec:4.1.44.Final and netty-all:4.1.34.Final.

It seems to be weird: library browserup-proxy-core uses netty-codec (and excludes netty-all, by the way). But its dependency littleproxy does use netty-all (which presumably contains netty-codec and some other netty sub-jars, hugh?). Obviously, they do conflict with each other.

How to fix it?

Again, there is no single correct answer.

There are many ways to fix the problem. Probably the easiest one is to declare Netty versions explicitly in build.gradle or pom.xml:

testRuntimeOnly("io.netty:netty-all:4.1.54.Final")
testRuntimeOnly("io.netty:netty-codec:4.1.54.Final")

Now command gradle dependencies shows that Netty versions are matching:

\--- com.browserup:browserup-proxy-core:2.1.1
+--- io.netty:netty-codec:4.1.44.Final -> 4.1.54.Final
+--- xyz.rogfam:littleproxy:2.0.0-beta-5
| +--- io.netty:netty-all:4.1.34.Final -> 4.1.54.Final

The test runs, proxy works, file is being downloaded. Everyone is happy. See detailed case description here.

Summary

I showed two examples of real-life problems caused by dependency conflicts. We saw how to investigate them using Maven or Gradle built-in tools. Now you know that it’s not impossible. You can do it. 🙂

Moral

Know your tools. Read your logs. Don’t be afraid.

Exclude as many dependencies as you can.

And feel like in dependency heaven.

 

Author: Andrei Solntsev

Software developer at Codeborne (Estonia).

Creator of selenide.org

Next Post

Previous Post

7 Comments

  1. Johannes Bühler December 2, 2020

    Hi Andrei,
    happy to read that you choose an interesting topic for the 2nd day of the advent of code. You might consider solving the 2 problem differently and more the gradle way.

    The First Problem can be solved with constraints:


    dependencies {
    ....
    constraints {
    implementation("com.google.guava:guava:27.1-jre") {
    because("latest compatible version for all transitive dependencies")
    }
    }
    }

    see: https://docs.gradle.org/current/userguide/dependency_constraints.html

    Instead of excluding all transitive dependencies you could also just exclude a certain dependency (https://docs.gradle.org/current/userguide/dependency_downgrade_and_exclude.html#sec:excluding-transitive-deps).

    The Second Problem is a great showcase to let the usage of a deployed platform bom shine:

    dependencies {
    components.all()
    ....
    }

    open class NettyBomAlignmentRule: ComponentMetadataRule {
    override fun execute(ctx: ComponentMetadataContext) {
    ctx.details.run {
    if (id.group.startsWith("io.netty")) {
    // declare that Netty modules belong to the platform defined by the Netty BOM
    belongsTo("io.netty:netty-bom:${id.version}", false)
    }
    }
    }
    }

    see: https://docs.gradle.org/current/userguide/platforms.html
    Another alternative to the platform approach would be to define a resolution strategy (https://docs.gradle.org/current/userguide/resolution_rules.html)

    Please find the complete improved example here: https://github.com/jonnybbb/todomvc

    A nice way to interactively query the dependencies can be experienced with build scans: https://scans.gradle.com/s/oo7ovsmyyn2jq/dependencies?dependencies=netty&expandAll

  2. motasem July 28, 2021

    happy to read this amazing article, but are you sure that dependency hell , is an npm problem , i see in their documentation that the node module loader was written exactly to solve this problem

    • Andrei Solntsev July 28, 2021 — Post Author

      Exactly, that’s the difference between theory and practice!
      In theory, npm module loader was written to solve this problem.
      In practice, people still complain on npm…

      • Motasem July 28, 2021

        And who have the right, npm or the people
        Who’s didn’t understand node module system

        • Andrei Solntsev August 1, 2021 — Post Author

          Look, the post was not about npm.
          The post was about resolving dependency conflicts in Java (Maven/Gradle).
          If there are good tools for resolving conflicts in npm – feel free to share it. Be constructive!

  3. Motasem July 28, 2021

    I believe that we misunderstood that node modules doesn’t have to be global, once we get this, we will never face dependency hell in nodeJs applications

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© 2021 JVM Advent | Powered by Jetbrains LogoJetBrains & steinhauer.software Logosteinhauer.software

Theme by Anders Norén

%d bloggers like this: