Thank you all for the Java flavored advent of 2012!

Thank you everybody!

This concludes the series of posts for this years Java Advent Calendar. Thank you everybody for writing, reading and commenting on the articles. Thanks to Transylvania JUG for proving the inspiration for this project and a big thank you to all the contributors:

Below you can find a calendar of all our articles. Until next December!

Sunday Monday Tuesday Wednesday Thursday Friday Saturday
1 (Re)Start me up!
2 Java Management Extensions 3 Multi-threading in Java Swing with SwingWorker 4 Java 7 – The Language Grows Up 5 JAX-RS Bean Validation Error Message Internationalization 6 Quercus on Google App Engine – 2.0 7 Hand and Finger Detection using JavaCV 8 Of Hacking Enums and Modifying “final static” Fields
9 Changes to String.substring in Java 7 10 Java EE 6 Web Profile. On the cloud. Easy. 11 Using Builder Pattern in JUnit tests 12 Escaping from the JVM heap for memory intensive applications 13 Java Sound – Make A Noise 14 Functional Java collections 15 AChartEngine – A Charting Library for Android Applications
16 Cleaning Up After Yourself – Java Style 17 Java – far sight look at JDK 8 18
19 Java Runtime options 20 Ensuring the order of execution for tasks 21
JVM Classloaders
Pleading for Java
22 Weak, Weaker, Weakest
23 Waiting for the right moment 24 Java – The 2012 Review and Future Predictions

Update: Here is a nice advent calendar about numbers. See you in a year!

Java – The 2012 Review and Future Predictions

Hi all,
It’s a real privilege to close off the Java Advent calendar for 2012. It’s a wonderful initiative and I (like many of you) eagerly checked my feed every morning for the next addition. A big Thank You! has to go to +Attila-Mihaly Balazs and all of the other authors for rounding off 2012 in some style.
This post will focus on the events big and small that occurred in 2012 and also take a look at some future predictions for 2013. Some of the predictions will be honest guesses, others…. well lets just say that my Diabolical side will have taken over :-).
So without further adieu lets look at the year that was 2012 for Java…..

2012 – A Year in Review

2012 was a rocking year for Java, the JVM and the community. James Governer (RedMonk analyst) stated that “2012 was the dawning of a 2nd age for Java”.

Java enters the cloud (for real this time)

Java/JVM based cloud offerings became a serious reality in 2012 with a host of new PAAS and IAAS offerings. Cloudbees, JElastic, Heroku, Joyent, Oracle are just five of the large number of offerings out there now.

What does that mean for you as a developer? Well, it means lots of choice and the ability to try out this space very cheaply. I highly recommend that you try some of these providers out over the holidays (it takes minutes to set-up a free account) and see what all of the fuss is about.

Counter to this however is a lack of standardisation in this space and although JEE8 promises to change this (assuming the vendors get on board) – for the next few years you’ll need to be careful about being locked into a particular platform. If you’re a bit more serious about having agnostic services/code running on the various offerings then I can recommend looking at the jClouds API to assist you.

It’s fair to say that many of the offerings are still feeling their way in terms of getting the most out of the JVM. In particular multi-tenancy is an issue, as is Garbage Collection and performance on a virtualised environment.  Companies such as Waratek and jClarity (Disclaimer: I’m their CTO) now offer solutions  to alleviate those gaps.

The Java community thrives

The community continues to thrive despite many main stream tech media reports of “developers leaving the Java platform” or “Java is dead”. There are more Java User Groups (JUGs) than ever before, consisting of ~400,000 developers world wide. Notably, one of them, the London Java Community won several awards including the Duke’s Choice award and JCP Member of the Year (along with SouJava – the major Brazilian JUG).

The conference circuit is bursting at the seams with large, sold out in advance, world-class Java conferences such as JFokus, Devoxx and of course JavaOne. In addition to this the host of regional conferences that often pack in an audience of over 1000 people all continued to do well.

Oracle’s Java Magazine was launched and has grown to over 100,000 subscribers. Stalwarts like JaxEnter, Coderanch and the Javaposse continue to grow in audience sizes.


Further OpenJDK reforms happened over 2012 and a new scorecard is now in place for the wider community to give feedback on governance, openness and transparency. 2012 also saw a record number of individuals and organisations joining OpenJDK. In particular, the port to the ARM processor and support for running Java on graphic cards (Project Sumatra) were highlights this year.

Java Community Process (JCP)

The Java Community Process (JCP), Java’s standards body also continued its revival with record numbers of new sign-ups and a hotly contested election. As well as dealing with the important business of trademarks, IP and licensing for Java, a re-focus on the technical aspects for Java Specification Requests (JSRs) occurred. In particular the new Adopt a JSR programme is being strongly supported by the JCP.

Java and the JVM

The JVM continues to improve rapidly through OpenJDK – the number of Java Enhancement Proposals (JEPs) going into Java 8 is enormous. Jigsaw dropping out was a disappointing but given the lack of broader vendor support and the vast amount of technical work required, it was the correct decision.

JEE / Spring

JEE7 is moving along nicely (and will be out soon), bringing Java developers a standard way to deal with the modern web (JSON, Web Sockets, etc). Of course many developers are already using the SpringSource suite of APIs but it’s good to see advancement in the underlying specs.

Rapid Web Development

Java/JVM based rapid web development frameworks are finally gaining the recognition they deserve. Frameworks like JBoss’s SEAM, Spring Roo, Grails, Play etc all give Java developers parity with the Rails and Django crowd.

Mechanical Sympathy

A major focus of 2012 was on Mechanical Sympathy (as coined by Martin Thompson in his blog). The tide has turned, and we now have to contend with having multi-core machines and virtualised O/S’s. Java developers have had to start thinking about how Java and the JVM interacts with the underlying platform and hardware.

Performance companies like jClarity are building tooling to help developers understand this complex space, but it certainly doesn’t hurt to get those hardware manuals off the shelf again!

2013 – Future predictions

It’s always fun to gaze into the crystal ball and here are my predictions for 2013!

Java 8 will get delivered on time

Java 8 with Nashorn, Lambda, plus a port to the ARM processor will open up loads of new opportunities for developers working on the leading edge of web and mobile tech. I anticipate rapid adoption of Java 8 (much faster than 7).

However, the lack of JVMs present on iOS and Android devices will continue to curtail adoption there.

Commercial Java in the cloud

2013 will be the year of commercial Java/JVM in the cloud – many of the kinks will get ironed out with regards to mutli-tenancy and memory management and a rich SAAS ecosystem will start to form.

The organisations that enable enterprises to get their in house Java apps out onto the cloud will be the big commercial winners.

We’ll also see some consolidation in this space as the larger vendors snap up smaller ones that have proven technology.


OpenJDK will continue to truly open up with a public issue tracker based on JIRA, a distributed build farm available to developers and a far superior code review and patch system put in place.

Oracle, IBM and other major vendors have also backed initiatives to bring their in house test suites out into the open, donating them to the project for the good of all. 

JVM languages and polyglot

There will be a resurgence in Groovy thanks to its new static compilation capability and improved IDE tooling. Grails in particular will look like an even more attractive rapid development framework as it will offer decent performance for midrange web apps. 

Scala will continue to be hyped but will only be used successfully by small focused teams.  Clojure will continue to be popular for small niche areas.  Java will still outgrow them all in terms of real numbers and percentage growth.

A random prediction is that JRuby may well entice over Rails developers that are looking to take advantage of the JVM’s performance and scalability.

So that’s it from me, it was an amazing 2012 and I look forward to spending another year working with many of you and watching the rest making dreams into reality!

Martijn (@karianna – CTO of jClarity – aka “The Diabolical Developer”)

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Waiting for the right moment – in integration testing

When you have to test multi-threaded programs, there is always the need to wait until the system arrives at a particular state, at which point the test can verify that the proper state has been reached.

The usual way to do it is to insert a “probe” in the system which will signal a synchronization primitive (like a Semaphore) and the test waits until the semaphore gets signaled or a timeout passes. (Two things which you should never do – but are a frequent mistake – is to insert sleeps into your code – because they slow you down and are fragile – or to use the Object.wait method without looping around it – because you might get spurious wakeups which will result in spurious, hard to diagnose and very frustrating test failures).

This is all nice and good (although a little verbose – at least until the Java 8 lambdas arrive), but what if the second thread calls a third thread and doesn’t wait for it to finish, but in the test we want to wait for it? A concrete example would be: an integration test which verifies that a system composed out of a client which communicates trough a messaging middleware with a datagrid properly writes the data to the datagrid? Of course we will use a mock middleware and a mock datagrid, thus the startup/shutdown and processing will be very fast, but they would be still asynchronous (suppose that we can’t make it synchronous because the production one isn’t and the code is written such that it relies on this fact).

The situation is described visually in the sequence graph below: we have the test running on T0 and we would like for it to wait until the task on T3 has finished before it checks the state the system arrived to.

We can achieve this using a small modification to our execution framework (which probably is some kind of Executor). Given the following interface:

public interface ActivityCollector {
void before();
void after();

We would call before() at the moment a task is enqueued for execution and after() after it has executed (these will usually occur on different threads). If we now consider that before increments a counter and after decrements it, we can just wait for the counter to become zero (with proper synchronization) at which point we know that all the tasks were processed by our system. You can find an Executor which implements this here. In production you can of course use an implementation of the interface which does nothing, thus removing any performance overhead.

Now lets look at the interface which defines how we wait for the “processed” condition:

interface ActivityWatcher {
void await(long time, TimeUnit timeUnit);

Two personal design choices used here were: only provide a way to wait for a specific time and no longer (if the test takes too long that’s probably a performance regression one needs to take a look at) and to use unchecked exceptions to make testing code shorter.

A final feature would be to collect exceptions during the execution of the tasks and abort immediately if there is an exception somewhere rather than timing out. This means that we modify our interface as follows:

public interface ActivityCollector {
void before();
void after();
void collectException(Throwable t);

And the code wrapping the execution would be something like the following:

try {;
} catch (Throwable t) {
throw t;
} finally {

You can find an implementation of ActivityWatcher/ActivityCollector here (they are quite linked, thus the one class implementing them both). Happy testing!

A couple of caveats:

  • This requires some modification to your production code, so it might not be the best solution (for example you can try creating synchronous mocks of your subsystems and do testing that way).
  • This solution is not well suited for cases where Timers are involved because there will be times when “no tasks are waiting”, but in fact a task is waiting in a timer. You can work around this by using a custom timer which calls “before” when scheduling and “after” at the finish of the task.
  • The same issue can come up if you are using network communication for more authenticity (even if it is inside of the same process): there will be a moment when no tasks are scheduled because they are serialized in the OSs network buffer.
  • The ActivityCollector is a single point of synchronization. As such it might decrease performance and it might hide concurrency bugs. There are more complicated ways to implement it which avoids some of the synchronization overhead (like using a ConcurrentLinkedQueue), but you can’t eliminate it completely.

PS. This example is based on an IBM article I can’t seem to find (dear lazyweb: if somebody finds it, please leave a comment – before/after were called tick/tock in it) as well as work by my colleagues. My only role was to write it up and synthesize it.

Weak, Weaker, Weakest, Harnessing The Garbage Collector With Specialist References

When and when not to use specialist references in Java

Weak, Soft and Phantom references are dangerous and powerful. If they are used the wrong way they can destroy JVM performance; however, when used the correct way they can substantially enhance performance and program clarity.

Weak and Soft references are the more obvious of the three. They are pretty much the same thing actually! The idea is simply that they be used to access an object but will not prevent that object being reclaimed by the garbage collector:

Object y=new Object();
// y is a hard reference to the object
// and so that object cannot be reclaimed.

Obejct x=WeakReference<Object>(y);
// now x is a weak reference to the object
// (not to y - as y is just a variable).
// The object still cannot be reclaimed
// because y is still a hard reference to it.

// now there is only a weak reference to
//the object, it is eligible for garbage collection.

System.out.println("The object has gone away");
System.out.println("The object is " + x.get().toString());

Have you spotted the deliberate mistake? It is an easy one to miss and it will probably not show up in unit testing. It is exactly the sort of issue which makes me say:

Only Use Weak/Soft References If You Absolutely Have To
And Probably Not Even Then.
When the JVM is under memory pressure it might reclaim the object between the first and second invocations of the get method in the weak reference. This will result in the program throwing a null pointer exception when the toString method is invoked on null. The correct form for the code is:

Object x=x.get();
// Now we have null xor a hard reference to
// the object
System.out.println("The object has gone away");
System.out.println("The object is " + z.toString());

So they are mad, bad and dangerous to be with; why do we want them?

We have not fully touched on why they are really, really dangerous yet. To do that we need to see why we might want them and why we might need them. There are two common situations in which weak and soft references might seem like a good idea (we will look at the difference between soft and weak in a little bit). The first of these is in some form of RAM cache.

It works like this: We have some data, for example customer details, which is stored in a database. We keep looking it up and that is slow. What we can do is cache that data in RAM. However, eventually the RAM will fill up with names and addresses and the JVM throw an OutOfMemoryError. The solution is to store the names and addresses in objects which are only weakly reachable. Something like this:

ConcurrentHasMap>String,WeakReference>CustomerInfo<< cache=new ConcurrentHashMap><();
CustomerInfo currentCustomer=cache.get(customerName);

This innocent little pattern is quite capable of bringing a monster sized JVM to its knees. The pattern is using the JVM’s garbage collector to manage an in-memory cache. The garbage collector was never designed to do that. The pattern abuses the garbage collector by filling up the memory with weakly reachable objects which run the JVM out of heap space. When the JVM gets low in memory, it has to traverse all the reference, weak, soft and otherwise, in its heap and reclaim RAM. This is expensive and shows up as a processing cost. It is even worse on very big JVM instances with a lot of processor cores because the garbage collector may well end up having to perform a ‘stop the world’ full cycle and hence reduce performance down to single core levels!

I am not saying in memory cache technology is a bad idea. Oh no – it is a great idea. However, just throwing it against the garbage collector and not expecting trouble is a very poor design choice.

Weak vs Soft what is the difference? Well, there is much really. On some JVMs (the client hostspot JVM for example – but that might change at any time) weak reference are marked for preferential garbage collection. In other words, the garbage collector should make more effort to reclaim memory from the object graph to which they refer (and no soft or hard references refer) than for other memory. Soft references do not have this idea to them. However, this is just an optional idea on some JVMs and cannot be relied upon, and it is a bad idea anyway. I would suggest using either soft or weak references all the time and stick with it. Pick which ever you like the sound of. I prefer the name WeakReference, so tend to use that.

There is one other difference; an object which is referenced to by a soft reference and a weak reference, but not a hard reference, can have the situation where it can still be acquired from the .get() method of the weak reference but not that of the soft reference. The reverse is not possible not the other way around. Code that relies on this behaviour is probably wrong headed.

Good uses for weak references do exist. What weak references are great for it keeping track of objects which are being used else where. An example is from Sonic Field (an audio processing package). In this example, ‘slots’ in files contain audio data and are associated with objects in memory. This model does not use the weak references to refer to in-memory copies of the data. In memory objects use the slots. Weak references are used to allow the file management system to reuse slots.

The code using slots does not need (and should not need to) be concerned with the management of disk space. It is the concern of the file manager to do that. The file manager has weak references to the objects using the slots. When a new slot is requested, the file manager checks for any existing slots referred to via weak references which have been reclaimed (and hence return null from the get method). If it finds such a reference, it can reuse the slot.

Automatic notification of reclamation

Sometimes we might want to be told when a weak or soft (or the other sort – phantom) reference has been reclaimed. This can be done via the en-queuing system. We can do this using a reference queue:

WeakReference(T referent, ReferenceQueue<? super T> q)

We do something like this:

ReferenceQueue junkQ = new ReferenceQueue<>();
WeakReference<FileSlot> mySlot=new WeakReference<>(aSlot);
// In a different thread - make sure it is daemon!
WeakReference<FileSlot> isDead;
isDead = junkQ.remove();
// Take some action based on the fact it is dead
// But - it might not be dead - see end of post :(

But, remember, by the time weak reference ends on the junkQ calling .get() on it will return null. If you will have to store information to allow what ever action you are interesting it to happen somewhere else (like a ConcurrentHashMap where the reference is the key),

So What Is A Phantom Reference?

Phantom references are the one sort which, when you need them, you really need them. But on the face of it, they seem utterly useless. You see, whenever you invoke .get() on a phantom reference, you always get null back. It is not possible to use a phantom reference to get to the object to which it refers – ever. Well – that is not quite true. We can achieve this via JNI sometimes but we should never do so.

Consider the situation where you allocate native memory in JNI associated with a Java object. This is the sort of model which the DirectBuffers in the noi package of the JDK use. It is something I have used repeatedly in large commercial projects.

So, how do we reclaim that native memory? In the case of file like systems, it is possible to say that the memory is not reclaimed until the file is closed. This places the responsibility of resource management on the shoulders of the programmer; which is exactly what the programmer expects for things like files. However, for lighter weight objects, we programmers do not like to have to think about resource management – the garbage collector is there to do it for us.

We could place code in a finalizer which calls into the JNI code to reclaim the memory. This is bad (as in lethal) because JVMs make almost guarantee that they will call finalizers. So, don’t do that! But, phantom references come to the rescue! First we need to understand ‘phantom reachable’: A phantom reference will only become enqueued if the thing to which it refers cannot be reach via any other sort of reference (hard, weak or soft). At this point the phantom reference can be enqueued. If the object had a finalizer, then it will either have been ignored or run; but it will not have ‘brought the object back to life’. Phantom reachable objects are ‘safe’ for JNI native code (or any other code) to reclaim resources against.

So our code with phantom references can look like this:

ReferenceQueue<FileSlot> junkQ = new ReferenceQueue<>();
Phantom<FileSlot> mySlot=new Phantom<>(aSlot);
// In a different thread - make sure it is daemon!
Phantom<FileSlot> isDead;
long handle=lookUpHandle(isDead);

In this pattern we keep a handle which the native code can use to find and reclaim resources in a structure (another hashmap probably) in Java. When we are absolutely sure that Java object cannot be brought back to life – it is phantom reachable (i.e. a ghost – we can then safely reclaim the native resource. If your code does other ‘naughty’ things using sun.misc.unsafe (for example) this trick might be of use as well. For a full example which uses this technique – check out this post.

One final thought about phantom references. It is technically possible to implement the same pattern as above using weak references. However, that is not the purpose of weak references and such a pattern would be abusing them. Phantom references makes an absolute guarantee that an object really is dead and so resource can be reclaimed. For just one example, it is theoretically possible for a weak reference to be enqueued and then the object be brought back to life by its finalizer because the finalization queue is running slower than the weak reference queue. This sort of edge case horror story cannot happen with phantom references.

There is one little problem, which is a weakness of the JVM design. That is that the JNI global weak reference type has an undefined relationship with phantom references. Some people suggest that you can use a global weak reference even to get to am object even when it is enqueued as a phantom reference. This is a quirk of one particular implementation of the JVM and should never be used.

By: Dr Alexander J Turner: I would like to thank Attila for contacting me and organising this interesting project. Feel free to pop over to my blog at Nerds-Central at any time 🙂

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Under the JVM hood – Classloaders

By Simon Maple, @sjmapleZeroTurnaround Technical Evangelist

Classloaders are a low level and often ignored aspect of the Java language among many developers. At ZeroTurnaround, our developers have had to live, breathe, eat, drink and almost get intimate with classloaders to produce the JRebel technology which interacts at a classloader level to provide live runtime class reloading, avoiding lengthy rebuilds/repackaging/redeploying cycles. Here are some of the things we’ve learnt around classloaders including some debugging tips which will hopefully save you time and potential headdesking in the future.

A classloader is just a plain java object

Yes, it’s nothing clever, well other than the system classloader in the JVM, a classloader is just a java object! It’s an abstract class, ClassLoader, which can be implemented by a class you create. Here is the API:

public abstract class ClassLoader {

public Class loadClass(String name);

protected Class defineClass(byte[] b);

public URL getResource(String name);

public Enumeration getResources(String name);

public ClassLoader getParent()


Looks pretty straightforward, right? Let’s take a look method by method. The central method is loadClass which just takes a String class name and returns you the actual Class object. This is the method which if you’ve used classloaders before is probably the most familiar as it’s the most used in day to day coding. defineClass is a final method in the JVM that takes a byte array from a file or a location on the network and produces the same outcome, a Class object.

A classloader can also find resources from a classpath. It works in a similar way to the loadClass method. There are a couple of methods, getResource and getResources, which return a URL or an Enumeration of URLs which point to the resource which represents the name passed as input to the method.

Every classloader has a parent; getParent returns the classloaders parent, which is not Java inheritance related, rather a linked list style connection. We will look into this in a little more depth later on.

Classloaders are lazy, so classes are only ever loaded when they are requested at runtime. Classes are loaded by the resource which invokes the class, so a class, at runtime, could be loaded by multiple classloaders depending on where they are referenced from and which classloader loaded the classes which referen… oops, I’ve gone cross-eyed! Let’s look at some code.

public class A {

public void doSmth() {

B b = new B();




Here we have class A calling the constructor of class B within the doSmth of it’s methods.  Under the covers this is what is happening


The classloader which originally loaded class A is invoked to load the class B.

Classloaders are hierarchical, but like children, they don’t always ask their parents

Every classloader has a parent classloader. When a classloader is asked for a class, it will typically go straight to the parent classloader first calling loadClass which may in turn ask it’s parent and so on. If two classloaders with the same parent are asked to load the same class, it would only be done once, by the parent. It gets very troublesome when two classloaders load the same class separately, as this can cause problems which we’ll look at later.

When the JEE spec was designed, the web classloader was designed to work the opposite way – great. Let’s take a look at the figure below as our example.  

Module WAR1 has its own classloader and prefers to load classes itself rather than delegate to it’s parent, the classloader scoped by App1.ear. This means different WAR modules, like WAR1 and WAR2 cannot see each others classes. The App1.ear module has its own classloader and is parent to the WAR1 and WAR2 classloaders.  The App1.ear classloader is used by the WAR1 and WAR2 classloaders when they needs to delegate a request up the hierarchy i.e. a class is required outside of the WAR classloader scope. Effectively the WAR classes override the EAR classes where both exist. Finally the EAR classloader’s parent is the container classloader.  The EAR classloader will delegate requests to the container classloader, but it does not do it in the same way as the WAR classloader, as the EAR classloader will actually prefer to delegate up rather than prefer local classes. As you can see this is getting quite hairy and is different to the plain JSE class loading behaviour.

The flat classpath

We talked about how the system classloader looks to the classpath to find classes that have been requested. This classpath could include directories or JAR files and the order which they are looked through is actually dependant on the JVM you are using. There may be multiple copies or versions of the class you require on the classpath, but you will always get the first instance of the class found on the classpath.  It’s essentially just a list of resources, which is why it’s referred to as flat. As a result the classpath list can often be relatively slow to iterate through when looking for a resource.

Problems can occur when applications who are using the same classpath want to use different versions of a class, lets use Hibernate as an example. When two versions of Hibernate JARs exist on the classpath, one version cannot be higher up the classpath for one application than it is for the other, which means both will have to use the same version. One way around this is to bloat the application (WAR) with all the libraries necessary, so that they use their local resources, but this then leads to big applications which are hard to maintain. Welcome to JAR hell! OSGi provides a solution here as it allows versioning of JAR files, or bundles, which results in a mechanism to allow wiring to particular versions of JAR files avoiding the flat classpath problems.

How do I debug my class loading errors?



So, you’ve got an error/exception like the ones above. Well, does the class actually exist? Don’t bother looking in your IDE, as that’s where you compiled your class, it must be there otherwise you’ll get a compile time exception. This is a runtime exception so it’s in the runtime we want to look for the class which it says we’re missing… but where do you start? Consider the following piece of code…

Arrays.toString((((URLClassLoader) Test.class.getClassLoader())

This code returns an array list of all jars and directories on the classpath of the classloader the class Test is using. So now we can see if the JAR or location our mystery class should exist in is actually on the classpath. If it does not exist, add it! If it does exist, check the JAR/directory to make sure your class actually exists in that location and add it if it’s missing. These are the two typical problems which result in this error case.



Now it’s getting interesting! These are all subclasses of the IncompatibleClassChangeError. We know the classloader has found the class we want (by name), but clearly it hasn’t found the right version. Here we have a class called Test which is making an invocation to another class, Util, but BANG – We get an exception! Lets look at the next snippet of code to debug:

.replace('.', '/') + ".class");

We’re calling getResource on the classloader of class Test. This returns us the URL of the Util resource. Notice we’ve replaced the ‘.’ with a ‘/’ and added a ‘.class’ at the end of the String. This changes the package and classname of the class we’re looking for (from the perspective of the classloader) into a directory structure and filename on the filesystem – neat. This will show us the exact class we have loaded and we can make sure it’s the correct version. We can use javap -private on the class at a command prompt to see the byte code and check which methods and fields actually exist. You can easily see the structure of the class and validate whether it’s you or the Java runtime which is going crazy! Believe me, at one stage or another you’ll question both, and nearly every time it will be you! :o)



These can occur if two different classloaders load the same class and they try to interact… ouch! Yes, it’s now getting a bit hairy. This can cause problems as we do not know if they will load the classes from the same place. How can this happen? Lets look at the following code, still in the Test class:


The code looks pretty clean and safe, and it’s not clear how an error could emerge from this line. We’re calling a static factory method to get us an instance of the Test class and are invoking a method on it. Lets look at this supporting image to show the reason why an exception is being thrown.

Here we can see a web classloader (which loaded the Test class) will prefer local classes, so when it makes reference to a class, it will be loaded by the web classloader, if possible. Fairly straightforward so far.  The Test class uses the Factory class to get hold of an instance of the Util class which is fairly typical practice in Java, but the Factory class doesn’t exist in the WAR as it is an external library.  This is no problem as the web classloader can delegate to the shared classloader, which can see the Factory class. Note that the shared classloader is now loading it’s own version of the Util class as when the Factory instantiates the class, it uses the shared classloader (as shown in the first example earlier). The Factory class returns the Util object (created by the shared classloader) back to the WAR, which then tries to use the class, and effectively cast the class to a potentially different version of the same class (the Util class visible to the web classloader). BOOM!

We can run the same code as before from within both places (The Factory.instance() method and the Test class) to see where each of our Util classes are being loaded from.

.replace('.', '/') + ".class"));

Hopefully this has given you an insight into the world of classloading, and instead of not understanding the classloader, you can now appreciate it with a hint of fear and uncertainty! Thanks for reading and making it to the end. We’d all like to wish you a Merry Christmas and a happy new year from ZeroTurnaround!  Happy coding!
Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Pleading for Java

This post is based on an article I wrote almost 10 years ago in the GInfo Computer Science Magazine. I am surprised things did not change much since then. So, here I go again…
In Romanian high schools the programming languages used or teaching Computer Science are Pascal and C++. In college, the students learn other programming languages they might need in order to become software engineers. Unfortunately, this is true only from a theoretical perspective. Most Romanian software engineers claim they had to learn by themselves…
Imagine a junior high school student who’s dream is to become a software engineer. He starts by learning how to implement more and more complicated algorithms (using the above mentioned languages) and then also learns something about databases.
When she graduates from high school, she realizes show know nothing useful. She knows how tosolve many problems, but she has no idea how to create a simple application with a friendly user interface. She decides to go to college so that she can learn more…
Therefore, the student must go to college because she does not learn enough in high school. In my opinion, a high school graduate should be able to a software engineer without a college diploma. She should not be one of the best, but she should be able to join the crowd. Why? Because it is possible. There are many college students that work as programmers while still studying and there are also enough experienced programmers that do not have a University degree. Of course, they learn by themselves, but they learn. Therefore, it is possible!
About 20 years ago, high school students started by learning QBasic. Someone decided this cannot go on and, at some point, the first programming language was Pascal. A few years later the C language was introduced as an alternative to Pascal.
Things changed, so change is possible. Was it hard? Yes! Why? Because it will be harder for the kids to learn Pascal then it was to learn Basic. This was the official answer. Unfortunately the real answer was something like: it will be harder for the teachers to teach Pascal then it was to teach Basic. But, this was changed. The Basic language was forgotten; Pascal and C were introduced. Then, someone came with the idea to start with C. To war between Basic and Pascal was replaced by a war between Pascal and C. Why? Because it will be harder for the kids to learn C then it was to learn Pascal.
I remember that a computer science teacher said that a math teacher graduates from college and is able to teach math for the next 50 years without having tolearn anything new. The two teachers have the same salary. Therefore, why should the computer science teacher have to learn new things. Because she teaches computer science, not math (I would say)!
Anyway, another small (half)step was made. The programming language was no longer specified in the curricula. The teacher may choose Pascal or C or something else (Algol, Fortran, Java, C# or anything else comes to mind). Again, the theory is nice. In practice, for the Maturity exams at the end of high school, the solutions must be written in Pascal or C.
These were useful steps (or half-steps), but it is not enough. The current curricula creates several stars (who win medals in international olympiads) and many unknowledgeable student who cannot say what’s the difference between sorting an array and finding the last digit of a number. And… so what? Someone will teach them to draw some nice user interfaces using Delphi and they will become software engineers.
I think the effort to teach algorithms to all computer science students is useless. Algorithms might be necessary, but it is more important to know a modern programming language.
Let’s see now why it is more difficult for a student to learn Java rather than C or Pascal. The for is still a for, the while is still a while, the if is still an if. Therefore, the control structures that must be explained are the same. The subroutines are also the same. Some claim the Java syntax is more difficult. Might be, but it is practically identical to the C++syntax. But, we have objects in Java and the teacher must explain difficult concepts like inheritance, polymorphism etc. Does she really have to? Of course,but not from the beginning.
The teacher can start without adding anything complicated without adding anything to the current style. Then, object oriented paradigms can be added. I think people someone is missing an important point: at the beginning, the student know nothing. Why should we believe it is easier for the student to learn structured programming rather than object oriented programming? The only reason is: we think structured programming is easier. I am confident that the engineers that know both techniques have no problem using any of them. They do not think one of them is easier than the other. They both have some basic things that must be understood.
I claim that it is not more difficult to understand the object oriented paradigms and they are more useful. Why aren’t they more difficult? Because the student knows nothing. She is not accustomed to work based on some principles; the teachers are… Any new things can be assimilated; it all depends on the talent of the teacher.
I will now try to show that the Java language is a viable alternative. First of all, Java is a language that was accepted by the software engineer community. It is not just a trendy language that will disappear in a few years. There are many Java technologies that are developed based on the core language. Even if one cannot expect JSP or hibernate to be thought in high school, the student will be able to easily learn all these technologies if she knows the Java basics. Java is an object oriented language. Most languages actually used for developing software are object oriented. If one knows such a language, it will be easy to learn the others (when needed).
Please allow me to present some examples that will show how good the “coffee” actually is.
Operations with big integers are a pain for all students that take part in programming competitions. In most contests there is at least a problem that needs the implementation of such operations. If dividing is necessary, everything turns into a nightmare.
By using Java we do not have this problem. Someone already implemented these operations and we only have to use the already written code. The following example shows how easy it is to use big integers:
import java.math.BigInteger;
public class BigIntegers {
  public static void main(String[] args) {
    BigInteger a = new BigInteger(“2458”);
    BigInteger b = new BigInteger(“13”);
    System.out.println(a + ” + ” + b + ” = ” + a.add(b));
    System.out.println(a + ” – ” + b + ” = ” + a.subtract(b));
    System.out.println(a + ” * ” + b + ” = ” + a.multiply(b));
    System.out.println(a + ” / ” + b + ” = ” + a.divide(b));
    System.out.println(a + ” % ” + b + ” = ” + a.remainder(b));
But it does not stop here. We have operations for decimal numbers too:
import java.math.BigDecimal;
public class BigDecimals {
  public static void main(String[] args) {
    BigDecimal a = new BigDecimal(“2458”);
    BigDecimal b = new BigDecimal(“13”);
    System.out.println(a + ” + ” + b + ” = ” + a.add(b));
    System.out.println(a + ” – ” + b + ” = ” + a.subtract(b));
    System.out.println(a + ” * ” + b + ” = ” + a.multiply(b));
    System.out.println(a + ” / ” + b + ” = ” + a.divide(b, 5, BigDecimal.ROUND_DOWN));
One may say that, given a few hours, an engineer would have implemented all that. That might be true, but it would be a couple of wasted hours. But, Java allows us to save days or even months in some cases. How long do you think it would take you to implement the compressing of a regular file into a zip file? You would probably need a few days just to study the format of the zip file.
I think the next sequence of code needs no further comments:
public class ZIP {
  public static void main(String args[]) throws Exception {
    ZipOutputStream zo = new ZipOutputStream (new FileOutputStream(“”));
    ZipEntry e = new ZipEntry(“myfile.txt”);
    FileInputStream in = new FileInputStream(“myfile.txt”);
    byte b[] = new byte[16384];
    int i;
    while ((i = > 0)
      zo.write(b, 0, i);
Like I stated 10 years ago, I will not draw any conclusions. I am waiting for other opinions…
I apologize for my English. It is not what it should be… 🙂

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Java Runtime options

The Java runtime is a complex beast – and it has to be since it runs officially on seven platforms and unofficially on many more. Give this, it is normal that there are many knobs and dials to control how things function. The more well known ones are:

  • -Xmx for the maximum heap size
  • -client and -server for selecting the default set of parameters from classes of defaults
  • -XX:MaxPermGen for controlling the permanent generation size

Other than these, it is (very) rarely the case that you need to change the defaults. However, thanks to Java being open source you can see the list of options, their default values and a short explanation directly from the source code. Currently there are almost 800 options in there!

An other way to see the options (but one which doesn’t display the explanations unfortunately) is the following command:

java -XX:+UnlockDiagnosticVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version

These options are well worth studying. Not for tweaking them (since there is a wealth of testing behind the defaults the extent of which would be very hard to replicate), but rather to understand the different functionalities offered by the JVM (for example why you might not see stacktraces in exceptions).

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Ensuring the order of execution for tasks

Sometimes it is necessary to impose certain order on the tasks in a threadpool. Issue 206 of the JavaSpecialists newsletter presents one such case: we have multiple connections from which we read using NIO. We need to ensure that events from a given connection are executed in-order but events between different connections can be freely mixed.

I would like to present a similar but slightly different situation: we have N clients. We would like to execute events from a given client in the order they were submitted, but events from different clients can be mixed freely. Also, from time to time, there are “rollup” tasks which involve more than one client. Such tasks should block the tasks for all involved clients (but not more!). Let’s see a diagram of the situation:

As you can see tasks from client A and client B are happily processed in parallel until a “rollup” task comes along. At that point no more tasks of type A or B can be processed but an unrelated task C can be executed (provided that there are enough threads). The skeleton of such an executor is available in my repository. The centerpiece is the following interface:

public interface OrderedTask extends Runnable {
boolean isCompatible(OrderedTask that);

Using this interface the threadpool decides if two tasks may be run in parallel or not (A and B can be run in parallel if A.isCompatible(B) && B.isComaptible(A)). These methods should be implemented in a fast, non locking and time-invariant manner.

The algorithm behind this threadpool is as follows:

  • If the task to be added doesn’t conflict with any existing tasks, add it to the thread with the fewest elements.
  • If it conflicts with elements from exactly one thread, schedule it to be executed on that thread (and implicitly after the conflicting elements which ensures that the order of submission is maintained)
  • If it conflicts with multiple threads, add tasks (shown with red below) on all but the first one of them on which a task on the first thread will wait, after which it will execute the original task.

More information about the implementation:

  • The code is only a proof-of-concept, some more would would be needed to make it production quality (it needs code for exception handling in tasks, proper shutdown, etc)
  • For maximum performance it uses lock-free* structures where available: each worker thread has an associated ConcurrentLinkedQueue. To achieve the sleep-until-work-is-available semantics, an additional Semaphore is used**
  • To be able to compare a new OrderedTask with currently executing ones, a copy of their reference is kept. This list of copies is updated whenever new elements are enqueued (this is has the potential of memory leaks and if tasks are infrequent enough alternatives – like an additional timer for weak references – should be investigated)
  • Compared to the solution in the JavaSpecialists newsletter, this is more similar to a fixed thread pool executor, while the solution from the newsletter is similar to a cached thread pool executor.
  • This implementation is ideal if (a) the tasks are (mostly) short and (mostly) uniform and (b) there are few (one or two) threads submitting new tasks, since multiple submissions are mutually exclusive (but submission and execution isn’t)
  • If immediately after a “rollup” is submitted (and before it can be executed) tasks of the same kind are submitted, they will unnecessarily be forced on one thread. We could add code rearrange tasks after the rollup task finished if this becomes an issue.

Have fun with the source code! (maybe some day I’ll find the time to remove all the rough edges).

* somewhat of a misnomer, since there are still locks, only at a lower – CPU not OS – level, but this is the accepted terminology

** – benchmarking indicated this to be the most performant solution. This was inspired from the implementation of the ThreadPoolExecutor.

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Kawa – oldest functional language on JVM still going strong

by Per Bothner

So-called scripting languages are now very visible, but they have a long history, even on the Java platform. Of languages still in use and active development today, Kawa and Netrexx are (to my knowledge) the oldest, both created in 1996. (Java was publicly launched in 1995.) Kawa appears to have been the first scripting language to be compiled directly to Java bytescodes, and is the oldest still-active functional language on the JVM. Kawa was based on a simple Scheme interpreter written by Alex Milowski, but I re-wrote it as a compiler while I worked for Cygnus Solutions, probably the first company to commercialize Free Software. Kawa is now GNU software, available under an MIT license.

Kawa is still used for a number of projects. As an example MIT App Inventor(previously Google App Inventor) for Android is built on top of Kawa.

Scripting vs programming languages

Kawa tries to combine the advantages of scripting languages with those of Java-like programming languages.

Like other scripting languages, Kawa avoids boilerplate such as having to define a class and main method. Since Kawa is an expression language, the minimal Hello, worldprogram is just:

"Hello, world!"

Macros allow syntactic extension, and can be defined in libraries just like functions. It is trivial to implement a domain-specific languageon top of Scheme, at least as long as you use the core S-expression datum syntax. Kawa has convenient syntactic short-hands like colon notation.

Kawa is quite dynamic: It has eval, and a read-eval-print-loop. If you type in an expression or load a file to be evaluated, it is automatically(and quickly) compiled down to bytecode.

You don't need to specify the types of variables – but you can: Doing so helps document your program, may allow the compiler to generate faster code, and may allow the compiler to catch errors and inconsistencies.

Kawa also has the advantages a more conventional programming language, like Java or Scala. You can compile to a class file, which can be an application (with an automatically-generated main), an applet, a servlet, or other specified class. It has an optimizing compiler which does decent data-flow analysis and type interface. This combined with optional type declarations make it straightforward to write code that is as efficient as Java or Scala – and many times faster than languages like JRuby or Clojure. The compiler is good at detecting and reporting errors, which avoids many painful debugging sessions. Kawa's primary visibility boundary is a sourcefile modules, which is natural and easy for both human readers and compilers.

In the rest of this article I will touch on some cool features of Kawa; maybe one of them will inspire you to try it out.

Object construction

Kawa has a convenient terse syntax for object construction. For example, this creates a Swing JButton that calls do-itwhen pressed:

text: "Do it!"
(lambda (e) (do-it)))

Groovy has a SwingBuilder that allows similar compact object construction. However, Kawa's is built in: It works for any bean-like class with standard set or add methods. Kawa is also more efficient, since it doesn't create a helper object. Instead the compiler translates the constructor call to code like what you'd write by hand in Java.


Of course sometimes just using the beans naming convention isn't quite enough to yield a pleasant constructor API. For example, in the Android API you construct a tree of View objects. Each View constructor requires a parameter which is the current Activity, which looks like this:

(TextView (this) text: "Hello world!")

The reference (this) to the current Activityis tedious boiler-plate. That is why Kawa defines some special handling so you can leave out the (this):

(TextView text: "Hello world!")

This special handling for View classes is not hard-wired but is defined in an Android-specific library.

Read how to build and run Android applicationsand more about Android View expressions.


Kawa also has some special handling to ease JavaFX programming. This page has a simple example and up-to-date instructions. Also check out two slightly older blog articles: an introduction, and about animation.

Self-configuring web server

It is easy to write web services using Kawa. No setup or additional software is needed if using the server built in to JDK (version 6 or later). Just start Kawa like this:

kawa --http-auto-handler / appdir --http-start 8888

Here is a web page scriptthat replies back with the name of the requesting (client) host:

;; Hello world page script written in -*- scheme -*-
#<p>Hello, <b>&(request-remote-host)</b>!</p>

The script uses an XML literal with an embedded expression. Put this file in appdir/path, and point your browser to http://localhost:8888/path. Kawa will find the file, recognize the string -*- scheme -*-in the first line and realize it is a Scheme script, then compile (if necessary) and evaluate it, and return the result to the browser.

You can use the same page scripts on a servet container such as Tomcat or Glassfish, instead of using JDK's builtin server.

Lazy evaluation

A pure functional language uses lazy evaluation – i.e. an expression is only evaluated when the value is needed. Impure functional languages like Scheme use strict evaluation, familiar from Java and most commonly-used languages. Kawa has a hybrid functionality: The expression (delay E) evaluates to a promiseP that will evaluate the expression E when requested. You can call (force P) to explicitly force the resulting promise: It returns the result of evaluating E. The expression E is only evaluated once, and the result remembered in case P is forced later.

The forms delay and delay are standard. Kawa also supports implicit forcing, as you'd find in a pure functional language: If the promise is used in a context that requires a real non-promise value, such as (* P 2)the P then the result is as you wrote: (* (force P) 2). It makes it more convenient to use lazy values.

Kawa also has futures; these are also automatically forced when needed.

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!


Which was the first dynamic language on the JVM? Think you know? Read on!

/* This is a fully documented program */
say 'Hello World!'

In 1995 the NetRexx language was conceived by Mike Cowlishaw of IBM. We have an exact birthday – it was the 10th of December, 1995. Consequently, NetRexx is 17 years old, and this fact makes it the oldest alternative language for the JVM. Mike was involved in porting the Java VM to IBM’s platforms – and the first platform to receive a JVM was OS/2. No blog has enough space to lament OS/2 and the fate that is has met by IBM’s shabby treatment of its own intellectual property and Microsoft’s insincere stance – OS/2, the most important program for the future, does one remember?

Experience shows that it does not help crying over spilled milk, but a straight headed analysis shows that NetRexx and its ancestor Rexx were heavily impacted by the demise of OS/2 by IBM’s own hand. In 1995, by all measures, Rexx was at the top of the world. It was a heavy contender to BASIC and if only for this reason, Microsoft wanted it dead. It is a testament to the intrinsic strength of the Rexx family of languages that it survived the catastrophic series of events that the cancellation of OS/2 and the even lesser known Workplace OS formed. I remember these days vividly: I was scheduled to go to a RedBook writing session in Poughkeepsie when word came that it was all over for Workplace OS. a.k.a. Pink and Blue. Today, we are left with some parts of it, which have been packaged into the International Components for Unicode, and in a bizarre twist of fate, NetRexx has been open sourced using the ICU license.

Because Open Source it is. In the beginning of the nineties, a call for an Object Oriented Rexx became noticeable, and within IBM a project was started to provide one. Called Oryx (for Object Oriented Rexx) in those days, Simon Nash was charged with it, and it produced a Java avant-la-lettre – a collection class library married to a Rexx with an OO-syntax that somehow managed to be compatible to (what, with retroactive continuity, became) Classic Rexx. The Java reference is because the earliest implementations had a bytecode interpreter that predated Java itself a number of years.

In 1995, Mike Cowlishaw did some experimentation to see how a Rexx-like language would behave on the JVM. Some compromises were made to adhere to the JVM’s object model, and for example, the stem notation with dots, an important part of how Classic Rexx defines multidimensional, content-addressed arrays, was dropped, because the dot-notation clashed too much with Java’s method invocation syntax. In its place the indexed string, a notation with square brackets (these were a problem for a long time on EBCDIC producing keyboards) was introduced. Also, all string comparison in NetRexx was to be case insensitive – a feature that Mike always has regretted omitting from Classic Rexx (on the well-meant advice of others).

For the rest, NetRexx is a better Rexx than Rexx itself. In its current form, it is a translator that can compile to .class files, as well as interpret the program in a single shot. Orthogonal to this, there is full-blown application mode, in which the programmer declares all the classes and methods, or scripting mode, in which there is just a number of commands and method invocations specified, and the translator adds all the syntactic ceremony that the Java language requires before compiling or interpreting.

At the essence of NetRexx is the fact that one can write Java classes without all the syntax that is annoying to the programmer that is reared in the non-C tradition. C style syntax has a very paradoxical property, in which terseness has led to ceremonial syntax elements. These are avoided in NetRexx. It has been established that the same program contains up to 40% less lexical elements in NetRexx as compared to Java syntax.

In other respects NetRexx keeps close to Java. As all compiled programs are translated to Java source first, performance has kept up with improvements in the Java hotspot VM architecture. In contrast to Jython and JRuby, there is no performance penalty for unbounded dynamism in the language: one of the early slogans for NetRexx “strong typing without more typing” still is true today.

If you like clean syntax and great performance, you should try NetRexx. Integration with Java class libraries is excellent and transparent. NetRexx is the first alternative language for the JVM, and still the only alternative JVM language that was actually used to implement a part of the JVM runtime in: the bigdecimal library was first written in NetRexx. For the ones that are aware of Mike Cowlishaws’s effort on the part of better handling of decimals by computer(hardware|languages) this is no surprise. The IEEE decimal definition is in large parts equal to the Rexx Language ANSI Standard. It does not get more serious when you care for floating point precision.

With its ancestor Rexx, NetRexx shares the unbounded decimal arithmetic, the TRACE and PARSE statements (study them, and you will be sold), and a set of string functions that was and is still the best on the planet. When this has gotten you interested, know that NetRexx is free and open source, and downloadable from Release 3.02 will be here any day, and it is accompanied by documentation befitting any former IBM product. Currently, there is IDE support for Eclipse, Jedit and Emacs, and since version 3.01 there is no JDK required any more – a JRE will do.

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!