JVM Crash
JVM crash is one of the toughest problem professional Java developers face. In case of a JVM crash, the operating system creates a core dump file which is a memory snapshot of a running process. A core dump is created by the operating system when a fatal or unhandled error like signal or system exception occurs. For professional Java developers, it is more common to report such errors to a JVM vendor because for a Java developer it is not common to troubleshoot native OS processes. That adds a significant wait period for troubleshooting and finding a fix for the crash. OpenJDK provides necessary tools for Java developers to diagnose such problems. With these tools we don’t need to wait for the JDK vendor, we can troubleshoot and find the root cause of the crash ourselves.
Common causes of JVM Crash
JVM crash can occur due to a bug
- in the Java HotSpot VM
- in a system library
- in a Java SE library or an API
- in application native code
- in the operating system (OS)
The majority of the above bugs falls under 2 categories:
- Running out of native memory. This is caused when native memory is exhausted, read more about native memory allocation here
- Segmentation faults. These are caused by a program trying to read or write an illegal memory location.
Creating core dump
On JVM crash a core dump is created by the Operating system. In some cases when JVM is hanged we may want to create a memory snapshot of running JVM to analyze this in detail, for this we can create a core dump of the JVM process manually. Let’s take a look at the command we need to run to create a core dump, and we will try to analyze that in the next section.
Java program to analyze in this post
This is a sample bad Java program Test.java that we will be using to create a core dump. For all our examples we will be using Java 18, as of writing this post it is built using JDK master branch. This post can help you to build JDK from the source.
public class Test
{
public static void main(String[] args)
{
Object object;
while(true)
{
object = new Object();
}
}
}
Operating system changes to create a core dump
Following are OS-level changes required on my Linux machine to enable creating a core dump file.
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope ulimit -c unlimited
In case you find difficulty generating core dump, refer detailed documentation from here.
Creating core dump
Commands to run a Java program for which we will create a core dump file.
/home/vipin/githubprojects/jdk/build/linux-x86_64-server-release/images/jdk/bin/javac Test.java /home/vipin/githubprojects/jdk/build/linux-x86_64-server-release/images/jdk/bin/java Test
Here is command to fetch PID of running Java process, to read more about jcmd see this post
$ jcmd 78480 Test 78501 jdk.jcmd/sun.tools.jcmd.JCmd
Following is a list of commands required to create a core dump file using GNU debugger gdb
gdb attach 78480 gcore jvm_dump.core detach quit
Analyzing core dump
A core dump file created by OS is more friendly for C and C++ developers, they use tools like gdb to get information out of the core dump files. OpenJDK provides CLI tool jhsdb which can be used to troubleshoot JVM core dump. Tools like gdb have no knowledge about Java data structures, while jhsdb is a Java developer’s friendly tool to analyze core dump files. jhsdb has a lot of features available, some important ones are
- we can use
jmapmode for memory information jstackmode for the stack tracejinfomode for command-line flags and the system properties.
For more details on all available modes, you can read about jhsdb here.
Let’s try to connect jhsdb with the core dump file jvm_dump.core created in the previous section. In practice, it could be either a core dump file generated from a crash or manually by the command line.
We will try to analyze the core dump using jhsdb in 2 modes
- clhsdb (Command line debugger)
- hsdb (GUI debugger)
jhsdb command line debugger clhsdb
Following command opens core dump in command line mode for further analysis
jhsdb clhsdb --core jvm_dump.core --exe /home/vipin/githubprojects/jdk/build/linux-x86_64-server-release/images/jdk/bin/java Opening core file, please wait... hsdb>
Now that we have the hsdb prompt we can run the threads command to get a list of threads along with thread ids.
hsdb> threads 78481 main State: IN_JAVA Stack in use by Java: 0x00007f040bb67a10 .. 0x00007f040bb67a30 Base of Stack: 0x00007f040bb69000 Last_Java_SP: null Last_Java_FP: null Last_Java_PC: null Thread id: 78481 ... 78488 Reference Handler State: BLOCKED Stack in use by Java: 0x00007f03c8bfe940 .. 0x00007f03c8bfea88 Base of Stack: 0x00007f03c8c00000 Last_Java_SP: 0x00007f03c8bfe940 Last_Java_FP: 0x00007f03c8bfe990 Last_Java_PC: 0x00007f03ed47b50b Thread id: 78488 ... ... ...
Now let’s try to get a stack trace for thread id 78481
hsdb> where 78481 Thread 78481 Address: 0x00007f0404023b50 Java Stack Trace for main Thread state = IN_JAVA - public static void main(java.lang.String[]) @0x00007f03c90003a0 @bci = 60, line = 19, pc = 0x00007f03f5015e4d (Compiled; information may be imprecise)
In the above output, we can see only one method call in the stack trace, because Test.java has only the main method. A thread dump is useful when we have a deadlock situation, or we want to find out what all threads are doing in the hanged java application.
In the following command, we are trying to print a histogram from the core dump, which shows an unusually high number of instances for Object. Here is an example of using jhisto
hsdb> jhisto Iterating over heap. This may take a while... Object Histogram: num #instances #bytes Class description -------------------------------------------------------------------------- 1: 933 7386016 int[] 2: 160637 2570192 java.lang.Object 3: 7546 366360 byte[] 4: 1432 176464 java.lang.Class 5: 7251 174024 java.lang.String 6: 1134 97736 java.lang.Object[] 7: 7 33032 char[] 8: 1032 33024 java.util.concurrent.ConcurrentHashMap$Node ... ...
We can use help to get a list of commands available clhsdb
hsdb> help
Available commands:
assert true | false
attach pid | exec core | remote_server
buildreplayjars [ all | app | boot ] | [ prefix ]
class name
classes
detach
...
...
...
thread { -a | id }
threads
tokenize ...
type [ type [ name super isOop isInteger isUnsigned size ] ]
universe
verbose true | false
versioncheck [ true | false ]
vmstructsdump
where { -a | id }
jhsdb gui debugger hsdb
The following command opens a GUI window that provides an interactive option to see details of the core dump. By default, it opens a list of threads as well.
jhsdb hsdb --core jvm_dump.core --exe /home/vipin/githubprojects/jdk/build/linux-x86_64-server-release/images/jdk/bin/java
In the tools menu there are various options available to see details of the core dump, below is the result when selected “Show VM Version”
When we select Object Histogram option from tools menu below is output:
Conclusion
JVM crash is not a common problem but when it happens we may need to raise it with the JVM vendor and wait for a significant duration for the RCA/resolution. jhsdb is a very useful command for Java developers we can analyze JVM crash ourselves and take corrective action as well.
If you want to get amazing Java jobs, I wrote an ebook 5 steps to Best Java Jobs. You can download this step-by-step guide for free!
Resources
- Java tools reference
- Detailed post on jhsdb
- Reasons for not Getting a Core File
- Build JDK from source
Author: Vipin Sharma
Vipin helps professional Java developers to learn language features so they can work on the best Java projects. Vipin is a senior Java developer and OpenJDK contributor.
Vipin is a senior developer at MSCI, and for more than 13 years he has worked on large Java projects in the Financial Sector.
On his blog, https://jfeatures.com Vipin shares his insights on Java Language Features and the evolution of the JVM.
steinhauer.software