As they say, better late than never, here is the calendar with all of the 2014 posts:
As they say, better late than never, here is the calendar with all of the 2014 posts:
JDK 9 – a letter to Santa?!
Recompiling java runtime library with debug symbols
Mutation testing in java with PIT and gradle
Creating REST API with Spring Boot and MongoDB
Own your heap: Iterate class instances with JVMTI
Self healing applications are they real?
Developers want to be heard!
How and Why is Unsage used in Java?
Lightweight Integration with Java EE and Camel
A serpentine path to music
CMS pipelines for NetRexx on JVM
Thread local storage in Java
Managing Package Dependencies with Degraph
How jOOQ Leverages Generic Type Safety in its DSL
How jOOQ Allows for Fluent Functional-Relational Interactions in Java 8
How jOOQ Helps Pretend that Your Stored Procedures are a Part of Java
How is Java/JVM built? Adopt OpenJDK is your answer!
Doing microservices with micro-infra
|22 A persistent KeyValue Server in 40 lines and a sad fact||23
The Java Ecosystem – My top 5 highlights of 2014
A Musical Finale
We are all human and irrespective of our beliefs, it seems we all enjoy music of some form. For me some of the most beautiful music of all was written by Johan Sebastian Bach. Between 1708 and 1717 he wrote a set of pieces which are collectively called Orgelbüchlein (Little Organ Book). For this post and to celebrate the Java Advent Calendar I tasked Sonic Field to play this piece of music modelling the sounds of a 18th century pipe organ. If you did not know, yes some German organs of about that time really were able to produce huge sounds with reed pipes (for example, Passacaglia And Fugue the Trost Organ). The piece here is a ‘Choral Prelude’ which is based on what we would in English commonly call a Carol to be sung by an ensemble.
One note overlapping the next or:
One note entirely contained in a longer note?
Fortunately, tricks can be used to try and figure this out based on note length etc. The Java decoder always treats notes as fully contained. The Python method looks for very short notes which are contained in long ones and guesses the real intention was two long notes which ended up overlapped slightly. Here is the python (the Java is here on github).
print "Interpretation Pass"
print "Demerge pass:",endAt
midi=sorted(midi, key=lambda tup: tup)
midi=sorted(midi, key=lambda tup: tup)
# Merge interpretation
if dif<blip and tkey==nkey and ttickOff>=ntickOn and ttickOff<=ntickOff:
print "Separating: ",this,next," Diff: ",(ttickOff-ntickOn)
midiOut.append([ttickOn ,ntickOn ,tnote,tkey,tvelocity])
print "Removing blip: ",(ttickOff-ttickOn)
# iterate the loop
if not mute:
Then comes some real fun. If you know the original piece, you might have noticed that the introduction is not original. I added that in the midi editing software Aria Maestosa. It does not need to be done this way; we do not even need to use midi files. A lot of the music I have created in Sonic Field is just coded directly in Python. However, from midi is how it was done here.
Once we have a clean set of notes they need to be converted into sounds. That is done with ‘voicing’. I will talk a little about that to set the scene then we can get back into more Java code oriented discussion. After all, this is the Java advent calendar!
Voicing is exactly the sort of activity which brings Python to the fore. Java is a wordy language which has a large degree of strictness. It favours well constructed, stable structures. Python relies on its clean syntax rules and layout and the principle of least astonishment. For me, this Pythonic approach really helps with the very human process of making a sound:
print "##### Channel A #####"
Above is a ‘voice’. Contrary to what one might think, a synthesised sound does not often consist of just one sound source. It consists of many. A piece of music might have many ‘voices’ and each voice will be a composite of several sounds. To create just the one voice above I have split the notes into long notes and short notes. Then the actual notes are created by a call to doMidi. This takes advantage of Python’s ‘named arguments with default values’ feature. Here is the signature for doMidi:
|The most complex (unsurprisingly) voice to create is that
of a human singing. I have been working on this for
a long time and there is a long way to go; however, its
is a spectrogram of a piece of music which does
a passable job of sounding like someone singing.
The first argument is actually a reference to a function which will create the basic tone. The rest of the arguments describe how that tone will be manipulated in the note formation. Whilst an approach like this can be mimicked using a builder pattern in Java; this latter language does not lend it self to the ‘playing around’ nature of Python (at least for me).
For example, I could just run the script and add flatEvn=True to the arguments and run it again and compare the two sounds. It is an intuitive way of working.
Anyhow, once each voice has been composited from many tones and tweaked into the shape and texture we want, they turn up as a huge list of lists of notes which are all mixed together and written out to disk as a flat file format which is basically just a dump of the underlying double data. At this point it sounds terrible! Making the notes is often only half the story.
You see, real sounds happen in a space. Our Choral is expected to be performed in a church. Notes played without a space around them sound completely artificial and lack any interest. To solve this we use impulse response reverberation. The mathematics behind this is rather complex and so I will not go into it in detail. However in the next section I will start to look at this as a perfect example of why Java is not only necessary but ideal as the back end to Python/Jython.
public class CacheableFFT
private final int n, m;
// Lookup tables. Only need to recompute when size of FFT changes.
private final double cos;
private final double sin;
private final boolean forward;
public boolean isForward()
public int size()
public CacheableFFT(int n1, boolean isForward)
this.forward = isForward;
this.n = n1;
this.m = (int) (Math.log(n1) / Math.log(2));
// Make sure n is a power of 2
if (n1 != (1 << m)) throw new RuntimeException(Messages.getString("CacheableFFT.0")); //$NON-NLS-1$
cos = new double[n1 / 2];
sin = new double[n1 / 2];
double dir = isForward ? -2 * Math.PI : 2 * Math.PI;
for (int i = 0; i < n1 / 2; i++)
cos[i] = Math.cos(dir * i / n1);
sin[i] = Math.sin(dir * i / n1);
public void fft(double x, double y)
int i, j, k, n1, n2, a;
double c, s, t1, t2;
j = 0;
n2 = n / 2;
for (i = 1; i < n - 1; i++)
n1 = n2;
while (j >= n1)
j = j - n1;
n1 = n1 / 2;
j = j + n1;
if (i < j)
t1 = x[i];
x[i] = x[j];
x[j] = t1;
t1 = y[i];
y[i] = y[j];
y[j] = t1;
n1 = 0;
n2 = 1;
for (i = 0; i < m; i++)
n1 = n2;
n2 = n2 + n2;
a = 0;
for (j = 0; j < n1; j++)
c = cos[a];
s = sin[a];
a += 1 << (m - i - 1);
for (k = j; k < n; k = k + n2)
t1 = c * x[k + n1] - s * y[k + n1];
t2 = s * x[k + n1] + c * y[k + n1];
x[k + n1] = x[k] - t1;
y[k + n1] = y[k] - t2;
x[k] = x[k] + t1;
y[k] = y[k] + t2;
It would be complete lunacy to implement this methematics in JPython (dynamic late binding would give unusably bad performance). Java does a great job of running it quickly and efficiently. In Java this runs just about as fast as it could in any language plus the clean, simple object structure of Java means that using the ‘caching’ system as straight forward. The caching comes from the fact that the cos and sin multipliers of the FFT can be re-used when the transform is the same length. Now, in the creation of reverberation effects (those effects which put sound into a space) FFT lengths are the same over and over again due to windowing. So the speed and object oriented power of Java have both fed into creating a clean, high performance implementation.
But we can go further and make the FFT parallelised:
# tail out clicks due to amplitude at end of signal
grainLength = sf.Length(+convol)
for grain in sf.Granulate(signal_,grainLength):
Here we have the Python which performs the FFT to produce impulse response reverberation (convolution reverb is another name for this approach). The second function breaks the sound into grains. Each grain is then processes individually and they all have the same length. This performs that windowing effect I talked about earlier (I use a triangular window which is not ideal but works well enough due to the long window size). If the grains are long enough, the impact of lots of little FFT calculation basically the same as the effect of one huge one. However, FFT is a nLog(n) process, so lots of little calculations is a lot faster than one big one. In effect, windowing make FFT become a linear scaling calculation.
Note that the granulation process is performed in a future. We define a closure called revi and pass it to sf_do() which is executed it at some point in the future base on demand and the number of threads available. Next we can look at the code which performs the FFT on each grain – rii. That again is performed in a future. In other words, the individual windowed FFT calculations are all performed in futures. The expression of a parallel windowed FFT engine in C or FORTRAN ends up very complex and rather intractable. I have not personally come across one which is integrated into the generalised, thread pooled, future based schedular. Nevertheless, the combination of Jython and Java makes such a thing very easy to create.
Now that I hope I have put a good argument for hybrid programming between a great dynamic language (in this case Python) and a powerful mid level static language (in this case Java) it is time to look at how the two are fused together. There are many ways of doing this but Sonic Field picks a very distinct approach. It does not offer a general interface between the two where lots of intermediate code is generated and each method in Java is exposed separately into Python; rather it uses a uniform single interface with virtual dispatch.
Sonic Field defines a very (aggressively) simple calling convention from Python into Java which initially might look like a major pain in the behind but works out to create a very flexible and powerful approach.
Sonic Field defines ‘operators’ which all implement the following interface:
/* For Copyright and License see LICENSE.txt and COPYING.txt in the root directory */
* @author AlexTu
public interface SFPL_Operator extends Serializable
* <b>Gives the key word which the parser will use for this operator</b>
* @return the key word
public String Word();
* <b>Operate</b> What ever this operator does when SFPL is running is done by this method. The execution loop all this
* method with the current execution context and the passed forward operand.
* @param input
* the operand passed into this operator
* @param context
* the current execution context
* @return the operand passed forward from this operator
* @throws SFPL_RuntimeException
public Object Interpret(Object input, SFPL_Context context) throws SFPL_RuntimeException;
Here we can see how the arguments are processed into a list (which is Jython is implemented as an ArrayList) if there are more than one but are passed as a single object is there is only one. We can also see how the Python stack trace is passed into a thread local in the Java SFSignal object. Should an SFSignal not be freed or be double collected, this Python stack is displayed to help debug the program.
Definitely not! It works here because of the nature of the Sonic Field audio progressing architecture. We have processors which can be chained. Each processor has a simple input and output. The semantic content passed between Python and Java is quite limited. In more general purpose programming, this simple architecture, rather than being flexible and powerful, would be highly restrictive. In this case, the normal Jython interface with Java would be much more effective. Again, we can see a great example of this simplicity in the previous post when talking about threading (where Python access Java Future objects). Another example is the direct interaction of Python with SFData objects in this post on modelling oscillators in Python.
from com.nerdscentral.audio import SFData
for x in range(0,length):
Which violated the programming model of Sonic Field by creating audio samples directly from Jython, but at the same time illustrates the power of Jython! It also created one of the most unusual soundscapes I have so far achieved with the technology:
Well, that is all folks. I could ramble on for ever, but I think I have answered most if not all of the questions I set out in the first post. The key ones that really interest me are about creativity and hybrid programming. Naturally, I am obsessed with performance as I am by profession an optimisation consultant, but moving away from my day job, can Jython and Java be a creating environment and do they offer more creativity than pure Java?
|Transition State Analysis using
Too many years ago I worked on a similar hybrid approach in scientific computing. The GRACE software which I helped develop as part of the team at Bath was able to break new ground because it was easier to explore ideas in the hybrid approach than writing raw FORTRAN constantly. I cannot present in deterministic, reductionist language a consistent argument for why this applied then to science or now to music; nevertheless, experience from myself and others has show this to be a strong argument.
Whether you agree or disagree; irrespective of if you like the music or detest it; I wish you a very merry Christmas indeed.
The Java Ecosystem started off with a hiss and a roar in 2014 with the annual meeting of the Free Java room at FOSDEM. As well as the many fine deep technical talks on OpenJDK and related topics there was also a surprise presentation on the industry from Steve O’Grady (RedMonk Analyst). Steve gave a data lead insight into where Java ranked in terms of both popularity and scope at the start of 2014. The analysis on where Java is as a language is repeated on RedMonk’s Blog. The fact it remains a top two language didn’t surprise anyone, but it was the other angle that really surprised even those of us heavily involved in the ecosystem. Steve’s talk clearly showed that Java is aggressively diverse, appearing in industries such as social media, messaging, gaming, mobile, virtualisation, build systems and many more, not just Enterprise apps that people most commonly think about. Steve also showed that Java is being used heavily in new projects (across all of those industry sectors) which certainly killed the myth of Java being a legacy enterprise platform.
The arrival of Java 8 ushered in a new Functional/OO hybrid direction for the language giving it a new lease of life. The adoption rates have been incredible (See Typesafe’s full report on this) it was clearly the release that Java developers were waiting for.
Some extra thoughts around the highlights of this release:
RedHat have lead the effort to get the ARMv8 64-bit architecture supported in Java. This is clearly an important step to keep Java truly “Run anywhere” and alongside SAP’s PowerPC/AIX port represents two major ports that are primarily maintained by non-Oracle participants in OpenJDK. If you’d like to help get involved see the project page for more details.
Java still has a way to go before becoming a major player in the embedded space, but the signs in 2014 were encouraging with Java SE Embedded featuring regularly on the Raspberry Pi and Java ME Embedded getting a much needed feature parity boost with Java SE APIs.
The Java Community Process (JCP) is the standards body that defines what goes into Java SE, Java EE and the Java ME. It re-invented itself as a much more open community based organisation in 2013 and continued that good work in 2014, reversing the dropping membership trend. Most importantly – it now truly represents the incredible diversity that the Java ecosystem has. You can see the make up of the existing Executive Committee – you can see that institutions like Java User Groups sitting alongside industry and end user heavyweights such as IBM, Twitter and Goldman Sachs.
The number of new joiners to OpenJDK (see Mani’s excellent post on this) was higher than ever. OpenJDK now represents a huge melting pot of major tech companies such as Red Hat, IBM, Oracle, Twitter and of course the shock entry this year of Microsoft.
The Adopt a JSR and Adopt OpenJDK programmes continue to bring more day to day developers involved in guiding the future of various APIs with regular workshops now being organised globally around the world to test new APis and ideas out early and feed that back in OpenJDK and the Java EE specifications in particular.
Community conferences & the number of Java User Groups continue rise in numbers, JavaOne in particular having it’s strongest year in recent memory. It was also heartening to see a large number of community efforts helping kids learn to code with after school and weekend programmes such as Devoxx for Kids.
I’ll expect 2015 to be a little bit quieter in terms of changes for the core language or exciting new features to Java EE or Java ME as their next major releases aren’t due to 2016. On the community etc front I expect to see Java developers having to firmly embrace web/UI technologies such as AngularJS, More systems/Devops toolchains such as Docker, AWS, Puppet etc and of course migrate to Java 8 and all of the functional goodness it now brings! The community I’m sure will continue to thrive and the looming spectre of IoT will start to come into the mainstream as well. Java developers will likely have to wait until Java 9 to get a truly first class platform for embedded, but early adopters will want to begin taking a look at early builds throughout 2015. Java/JVM applications now tend to be complex, with many moving parts and distributed deployments. It can often take poor frustrated developers weeks to fix issues in production. To combat this there are a new wave of interesting analysis tools dealing with Java/JVM based applications and deployments. Oracle’s Mission Control is a powerful tool that can give lots of interesting insights into the JVM and other tools like Xrebel from ZeroTurnaround, jClarity’s Censum and Illuminate take the next step of applying machine learned analysis to the raw numbers. One final important note. Project Jigsaw is the modularisation story for Java 9 that will massively impact tool vendors and day to day developers alike. The community at large needs your help to help test out early builds of Java 9 and to help OpenJDK developers and tool vendors ensure that IDEs, build tools and applications are ready for this important change. You can join us in the Adoption Group at OpenJDK: http://adoptopenjdk.java.net I hope everyone has a great holiday break – I look forward to seeing the Twitter feeds and the GitHub commits flying around in 2015 :-).
Martijn (CEO – jClarity, Java Champion & Diabolical Developer)
OpenJDK history (2006 till date)
repo composition, language breakdown (metrics are estimated)
Fredrik Öhrström visited the LJC  in November 2011 where he showed us how to build OpenJDK on the three major platforms, and also distributed a four page leaflet with the benchmarks of the various components and how long they took to build. The new build system and the new makefiles are a result of the build system being re-written (build-infra).
Build Benchmarks page 2 
How has Java the language and platform built over the years ?
Please share your comments here or tweet at @theNeomatrix369.