How and Why is Unsafe used in Java?

Overview

sun.misc.Unsafe has been in Java from at least as far back as Java 1.4 (2004).  In Java 9, Unsafe will be hidden along with many other, for-internal-use classes. to improve the maintainability of the JVM.  While it is still unclear exactly what will replace Unsafe, and I suspect it will be more than one thing which replaces it, it raises the question, why is it used at all?

Doing things which the Java language doesn’t allow but are still useful.

Java doesn’t allow many of the tricks which are available to lower level languages.  For most developers this is very good thing, and it not only saves you from yourself, it also saves you from your co-workers.  It also makes it easier to import open source code because you know there is limits to how much damage they can do.  Or at least there is limits to how much you can do accidentally. If you try hard enough you can still do damage.

But why would you even try, you might wonder?  When building libraries many (but not all) of the methods in Unsafe are useful and in some cases, there is no other way to do the same thing without using JNI, which is even more dangerous and you lose the “compile once, run anywhere”

Deserialization of objects

When deserializing or building an object using a framework, you make the assumption you want to reconstitute an object which existed before.  You expect that you will use reflection to either call the setters of the class, or more likely set the internal fields directly, even the final fields.  The problem is you want to create an instance of an object, but you don’t really need a constructor as this is likely to only make things more difficult and have side effects.
public class A implements Serializable {
private final int num;
public A(int num) {
System.out.println("Hello Mum");
this.num = num;
}

public int getNum() {
return num;
}
}

In this class, you should be able to rebuild and set the final field, but if you have to call a constructor and it might do things which don’t have anything to do with deserialization.  For these reasons many libraries use Unsafe to create instances without calling a constructor

Unsafe unsafe = getUnsafe();
Class aClass = A.class;
A a = (A) unsafe.allocateInstance(aClass);

Calling allocateInstance avoids the need to call the appropriate constructor, when we don’t need one.

Thread safe access to direct memory

Another use for Unsafe is thread safe access to off heap memory.  ByteBuffer gives you safe access to off heap or direct memory, however it doesn’t have any thread safe operations.  This is particularly useful if you want to share data between processes.

import sun.misc.Unsafe;
import sun.nio.ch.DirectBuffer;

import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.lang.reflect.Field;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;

public class PingPongMapMain {
public static void main(String... args) throws IOException {
boolean odd;
switch (args.length < 1 ? "usage" : args[0].toLowerCase()) {
case "odd":
odd = true;
break;
case "even":
odd = false;
break;
default:
System.err.println("Usage: java PingPongMain [odd|even]");
return; }
int runs = 10000000;
long start = 0;
System.out.println("Waiting for the other odd/even");
File counters = new File(System.getProperty("java.io.tmpdir"), "counters.deleteme"); counters.deleteOnExit();

try (FileChannel fc = new RandomAccessFile(counters, "rw").getChannel()) {
MappedByteBuffer mbb = fc.map(FileChannel.MapMode.READ_WRITE, 0, 1024);
long address = ((DirectBuffer) mbb).address();
for (int i = -1; i < runs; i++) {
for (; ; ) {
long value = UNSAFE.getLongVolatile(null, address);
boolean isOdd = (value & 1) != 0;
if (isOdd != odd)
// wait for the other side.
continue;
// make the change atomic, just in case there is more than one odd/even process
if (UNSAFE.compareAndSwapLong(null, address, value, value + 1))
break;
}
if (i == 0) {
System.out.println("Started");
start = System.nanoTime();
}
}
}
System.out.printf("... Finished, average ping/pong took %,d ns%n",
(System.nanoTime() - start) / runs);
}

static final Unsafe UNSAFE;

static {
try {
Field theUnsafe = Unsafe.class.getDeclaredField("theUnsafe");
theUnsafe.setAccessible(true);
UNSAFE = (Unsafe) theUnsafe.get(null);
} catch (Exception e) {
throw new AssertionError(e);
}
}
}

When you run this in two programs, one with odd and the other with even. You can see that each process is changing data via  persisted shared memory.

In each program it maps the same are of the disks cache into the process.  There is actually only one copy of the file in memory.  This means the memory can be shared, provided you use thread safe operations such as the volatile and CAS operations.

The output on an i7-3970X is

Waiting for the other odd/even
Started
… Finished, average ping/pong took 83 ns

That is 83 ns round trip time between two processes. When you consider System V IPC takes around 2,500 ns and IPC volatile instead of persisted, that is pretty quick.

Is using Unsafe suitable for work?

I wouldn’t recommend you use Unsafe directly.  It requires far more testing than natural Java development.  For this reason I suggest you use a library where it’s usage has been tested already.  If you wan to use Unsafe yourself, I suggest you thoughly test it’s usage in a stand alone library.  This limits how Unsafe is used in your application and give syou a safer, Unsafe.

Conclusion

It is interesting that Unsafe exists in Java, and you might to play with it at home.  It has some work applications especially in writing low level libraries, but in general it is better to use a library which uses Unsafe which has been tested than use it directly yourself.

About the Author.

Peter Lawrey has the most Java answers on StackOverflow. He is the founder of the Performance Java User’s Group, and lead developer of Chronicle Queue and Chronicle Map, two libraries which use Unsafe to share persisted data between processes.

013 Things that Will go Horribly Wrong on This Friday, Dec 13, 2013

I’m a defensive programmer. A wise man once taught me:

Whatever can go wrong will go wrong

Another wise man taught me to…

expect the unexpected

Today, this will finally all make sense, as today, on this Friday the 13th, the following 10 programming-related things will go terribly wrong. In other words, your software is about to explode…

1. … because you have never really thought about names

Today, you’ll terribly regret never having thought about names, because that user will sign up to your application. That user’s name is null. I mean, literally, Null. Like Jack Null, Heather Null, Null Smith. Whatever. You hadn’t thought of the implications of such a user, and now you’ll have to fix it. And while you’re at it, please consider all the other mistakes and misbeliefs that you might have been making over the past years. Like:

  • Thinking that people’s names are case sensitive.
  • Thinking that people’s names are case insensitive.
  • Thinking that people’s names have any case at all.

Read this scary article about Falsehoods Programmers Believe About Names, to learn more about why your software will explode today because of people’s names.

2. … because you get SQL wrong

Yes. SQL. Some hate it, others love it. SQL has its own ways. But today, you’ll regret never having thought about SQL NULL. Today you’ll regret never having considered the difference between UNION and UNION ALL. Today, you’ll run out of memory, because you have thought you could perform that JOIN in Java (or C#, PHP, you name it), rather than in SQL.

Today, you’ll commit any of these 10 Common Mistakes Java Developers Make When Writing SQL

3. … because you still get SQL NULL wrong

As a matter of fact, you still didn’t understand SQL NULL. You’re not to blame, no one really understands SQL NULL. This must have been some elaborate Halloween joke by E. F. Codd, a sophisticated academic gag he lay upon us and upon the next 15 generations of programmers. But all that cynicism won’t help you, because today, NULL is going to explode right in your application.

If you don’t believe it, learn about NULL in SQL: Explaining its Behavior. And then, go back and check your SQL while you still can!

4. … because your Java code is overdesigned. Or underdesigned

It doesn’t help that you now think you know SQL. Because 80% of your application (which will blow up today, on this Friday, Dec 13th) is written in Java. And if it isn’t, then consider your Android phone, which is. And if that isn’t, either, then chances are, that any other device of yours is running Java.

And those who wrote that Java code, they didn’t write object-oriented Java code, but Perl-esque or C-esque Java code leading to a horrible amount of spaghetti. And then, when your architect (or you?) attempted to clean up the spaghetti, they just went mad on all the design patterns that we have had in Java for the last 10 years. Now, no one can maintain this application, which leads to it blowing up right now.

While this happens, consider reading Watch Out for These 10 Common Pitfalls of Experienced Java Developers & Architects. At least, you’ll know whom to blame. You should blame the humble architect.

5. … because your fellow developers keep finding new excuses

There’s no way around that. You and your fellow developers didn’t do your work. And you just keep finding new excuses for things you did (or didn’t do). Like:

  • Oh, you said you DIDN’T want that to happen?
  • The project manager said no one would want that feature

And that’s just the beginning. Today, all those Programming Excuses will unravel and show you the true nature of your application, explaining why it is about to explode.

6. … because you really really get SQL wrong

Really. It actually isn’t because of those developers or because of Java. It’s really SQL’s fault that your application explodes today. Who can ever memorise those caveats anyway? Because your application suffers from SQL injection and syntax errors due to the lack of bind variable usage. Because you didn’t use enough constraints. Because you thought that 50ms at development time was fast query execution. Yes, that “fast” query has now been running for 30 minutes in production and within a bit, it will make your application explode, because you haven’t read:

10 More Common Mistakes Java Developers Make When Writing SQL

7. … because you got dates and times wrong

Oh yes. That business-critical account statement. It goes from October 1 to October 31, 2010. Yes, and oh, your customer is using the CET / CEST time zone. Which means that chances are high that on this given end date, your customer’s country was switching from daylight savings time to winter time. As you’re an U.S. based developer, you hadn’t thought of that and did some date time arithmetic based on 24 hour dates, when in fact that particular day had 25 hours.

And don’t get me started on leap seconds. They jump right at you before you have updated your OS and/or JVM to get the latest in calendar technology. Too bad, you’re writing high-frequency-trading software instead of some geology software, where you don’t care about the odd leap second. Or leap year. Or leap millenium.

Sigh. When will we introduce the metric system also for our calendars and count in terms of Star Trek space time? But before that happens, let us learn about Falsehoods Programmers Believe About Time. And on a more serious note, Some Notes About Time. Clock’s ticking!

8. … because you get XML namespaces wrong

Now, this will go so terribly wrong, I don’t even want to go into this topic. Just a hint. This easy-to-read document will help you understand where you went wrong.

9. … because you’ll get CSS wrong

Finally, Microsoft has freed us from IE6 and previous horrible implementations of HTML, CSS et al. Unsupporting IE6 was a great step forward, helping us collectively forget what happened in the “dark ages of HTML” when we needed to double all HTML, CSS, JavaScript, to make things work on both Internet Explorer and all the other browser. Or did they?

Wait, mobile browsers appeared, HTML5 wasn’t standardised (or implemented?) quickly enough, Webkit set new defacto standards, aaagh. Like the mythological Hydra, the two-headed browser era has now transformed into an era where browsers have 360 heads. All different, utterly unmaintainable, according to Stefan Baumgartner whose hilarious talk about Mobile Browsers at Topconf 2013 has at least kept me from losing faith in mankind. Because at least we haven’t lost our humour.

Yet, today is the day your application will explode, and it is most likely (or just as likely) because of a CSS bug. Don’t believe it? Read The IE CSS Bug That Cost Me a Month’s Salary. And then go learn from Nicole Sullivan who has taught us about the 5 Mistakes of Massive CSS.

10. … because you get your encÕÐing wron错误的编码

Sigh. So you’re thinking that I am a Java and SQL person, and since you’re using some fancy language like Ruby or Python, you’re safe? Nope, you’re not. Not this time. Even your application will explode today, on this Friday, 13th because you got your encoding wrong. No one ever gets this right. Ever. You think that UTF-8 will save you? Did you hear of UTF-16 and UTF-32? Things can and will go wrong with UTF-8. Specifically because your application exports to CSV, which is consumed by Excel, which expects data to be encoded in Cp1251. Or maybe even an encoding without the € (Euro) sign. Sigh.

You know why we have ISO / IEC 8859-1 AND ISO / IEC 8859-15? Obviously, today, your application will explode precisely because you’ve chosen the wrong one. Yet, I know the truth. Don’t believe ISO / IEC’s or Wikipedia’s “official” story. As Thomas Müller (the H2 developer) once explained to me, it happened like this:

Once upon a time, an ISO engineer thought, we need a new encoding standard. So he went and created ISO 8859-1. Obviously, he didn’t name it ISO 8859, because he knew he was getting it wrong and his smart alec co worker will fix it just to prove he got it wrong. So he added the “dash one”. Then, a couple of years later, his co worker actually did show up and did complain that the euro sign was missing. And a couple of other very useful characters for 99% of us developers: ŠšŽžŒœŸ.

So the first guy said, “Oh well, who cares. Let’s leave things as they are”. And the other guy said: “No I’ll fix this”. And the first guy said, “No, don’t do it, it’ll cause so much pain”, and the other guy said: “No this is just wrong”. Much like in “Someone is Wrong in the ISO”. And after lots more of arguing, we now have even more permutations of encoding pairs to get wrong.

The above is how most things happened in the past, by the way. Which is only cold comfort for you as you are about to notice that your application will explode, today. Since you’re very unlikely to actually learn how this works correctly (most importantly not today, on this Friday, 13th), you might at least use this Python script to fix all wrong characters back from UTF-8 to ISO-8859-1 (or vice versa? Try both…). Read Fixing Common Unicode Mistakes with Python — After They’ve Been Made. Another good read is UTF-8: The Secret of Character Encoding. And finally: Geek and Poke’s point of view on the matter:

When it all began by Geek and Poke, Licensed CC-BY 3.0

11. (or 013) … because you get radixes wrong

If you’re unfortunate enough to code in JavaScript (and that is unfortunate, as we all know), you’ll run the risk of forgetting that parsing an int automatically “detects” the implicit radix of your number string. So, today, you’ll think that 013 has anything to do with the decimal 13, the hexadecimal 0x0D. You couldn’t get this any more wrong.

Because today is yet another day where you fell for JavaScript’s accepting leading zeroes as octal numbers. Read more about this behaviour here.

As you may have noticed, this list contains 013 things that will go wrong on this Dec 13. I.e. octal 13 on this decimal 13th day. Even this article got it wrong! It got it wrong twice, because the URL can no longer be changed from 10 things to 013 things. Today you won’t only get radixes wrong, but you’ll get lots of broken links or outdated websites, too!

TL;DR Today, everything will go wrong

Today is Friday the 13th. More specifically, 13.12.13 in my country’s date notation. See #7 to learn about what will go wrong with dates and times today, when you try to parse that date in other countries than mine. I wouldn’t be surprised if my clock showed 13.13.13 because of yet another piece of Code That Made Me Cry that went wrong today, and it would prove that today is a really bad day.

And don’t think about grabbing a coffee to cheer up after this article, because today, on this Friday the 13th, your coffee machine’s clock isn’t working because someone forgot a leap second. And even if it did work, there’s only DECAF left. Time to call it a weekend! Have a nice one, and don’t write any code, today!

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on!