A Serpentine Path To Music

A snapshot of the ffmpeg visualisation for the
Goldberg Variations as synthesised by Sonic Field.

Art And Programming Collide: An Introduction To The Post

Whilst there are a number of ways of making music with Java, as far as I can tell, Sonic Field is the only fully fledged, end to end synthesis system written in pure Java (no JNI and no call out to native servers etc.) which does not use the Javax Synthesiser extensions (so does not rely on the operating system to supply the actual synthesis engine).

Also, in many other ways, its heavy weight, uncompromisingly mathematical approach to music is unique. This project has consumed far too much of my time for the better part of three years; what can be said about this ab-initio synthesis system in just one post and what can we learn about Java programming from the project?

The answer is why?

  • Why write a mathematical engine in Java?
  • Why control it from a dynamic language (Python)?
  • Why make music using programs at all?
  • Why use ab-initio synthesis (and what is it)?

Maybe,  most interesting of all: 
  • Why is large scale musical mathematics so computationally demanding?
I will attempt to answer all these questions and shed some light into very ill frequented corners of Java/JVM programming. My approach will not be to answer each in turn, but these questions will form the armature upon which what follows will be constructed.

Sonic Field and how to achieve complex audio synthesis on the JVM proved to be too large a subject for one post. I have am extremely lucky to have been given chance to make two. This post is the hard work; we will cover the difficult subjects like threading, memory management and some digital audio concepts. The next post is a lot more fun; it will look at why I chose Python and some thoughts on creativity.

Now let us look at how digital sounds is represented and manipulated. 

Believe it or not, there are two ways right from the start. However, everyone (including myself with Sonic Field) picks the same one. Sound is made of waves. Waves can be represented mathematically using trigonometrical functions like sin and cos. We can then apply non linear differential equations to those mathematical forms to produce more complex equations. Keep adding more and more transformations and eventually we end up with one equation which represents an entire piece of music. This never happens because it is far, far too hard! Such an approach is referred to as ‘analytic’ and is practical only for the simplest of sounds.

The spectrogram of the piece of music
I rendered with Sonic Field for the
Java Advent Calendar
More tractable are numerical approaches. Here we consider sounds as a series of samples. Just like the samples that make up a video, one frame at a time, sound is generated and manipulated one sample at a time. We must understand this is an compromise and it brings with it a whole series of computational challenges of its own.

A quick note on licensing: Sonic Field is free as freedom:


Sonic Field Audio Processing And Synthesis Program
Copyright (C) 2012-2014 Dr Alexander J Turner

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
All the source code, including any Sonic Field patch scripts in this post are covered by the above licence. It only seems fair that anyone reading this should respect this as I am giving it away for free on Github!

Why samples are a challenge:

Consider that we want audio to be modelled up to 20kHz, i.e. we want to generate sounds which have frequencies up to 20 000 cycles per second. To actually generate that all using a sampling, digital system we need 40 000 samples per second. This is intuitively easy to understand as a sound goes ‘up and down’ in a wave so we need an up sample and a down sample for each cycle of the wave.
Now let us consider making a sound. We can start with a sine wave. This is a pure sound, and it is very boring indeed. How about making a more interesting sound? Well, real sounds consist of many different sine waves mixed together. A note at concert pitch A is 440Hz, it this were played on a trumpet we might get a sequence of frequencies like this:
440,880,1320,1760,2200,2640,3080,3520,3960,4400 and so on.
At the beginning of our piece we can see
that even a complex note is made from
bands of frequencies. 

Each frequency will be slightly less intense then the one beneath (usually) but they are all there. Now let us just consider 4400 or 8800; what happens when we pass that through a distortion filter to make it sound cooler? Well, we will end up adding frequencies which are multiples of the frequencies which made up the original trumpet sound. All this multiples are call harmonics.

Bored? we need some code! Sonic Field is controlled using Jython, the Python interpreter which runs on the JVM. I hope the reason for this will become obvious, but I will explicitly discuss this int he second post. Here is a piece of code which makes the basic sound like a trumpet (well, at least the trumpet rank from an organ):


def trumpetBase(length,freq,z=1.0):
voxA=[]
hc=1.0
freq=float(freq)
while hc*freq<20000:
hf=hc*freq
voxA.append(sf.NumericVolume(sf.PhasedSineWave(length,hf,random.random()),(1.0/hc)**z))
hc+=1

vox=sf.Mix(voxA)
vox=sf.Clean(vox)
vox=polish(sf.FixSize(vox),freq)
return sf.FixSize(vox)

OK, so you might not know Python (this being a Java Advent Calendar and all) so I will explain a little of what is happening here. Simply out their is a while loop which creates samples of different frequencies of sine waves to create the harmonics of the trumpet sound. The phase of the sine waves is randomised. It really does not matter if you do not know what that means. Anyhow, the loop which accumulates sine waves, then we mix them together and finally fix the volume (FixSize) to deviate +-1 from zero.
Not long now and we will be deep diving into Java, but a few more sentences to set the scene. But just back to that distortion. Consider I want the trumpet to sound more fragile and bright?

sig=sf.Power(trumpetBase(440,10000),1.25)
Now I have created a new ‘signal’ (i.e. a sequence of samples) which is a distortion of the original. In so doing multiples of the original harmonics will have been introduced. 5×4400 is over 20000. Our 40000 samples per second will no longer be enough. What we need to do is sample at a higher rate. But eventually we are defeated by this approach as we add distortion on distortion. The solution is to sample at a higher rate and then filter out the high frequencies periodically. We perform synthesis by repeating two basic steps, process then filter. The process adds frequencies and the filter takes out everything above when we need.
OK, so what rate? It seems that for most of my work 96 000 samples per second is about right. More makes near no difference to the sound quality but as I go below this number a horrid harsh effect called ‘aliasing’ starts to be unavoidable. This is caused by trying to model a frequency higher than twice the sample rate.

Here is a video I made talking about, explaining and illustrating aliasing.
Please note that the Sonic Field patch shown is pre the use of Python
and so is in a bespoke language I wrote called SFPL which is 
no longer used.
Finally we have started to see why making music with Java is so hard. 96 000 samples per second is a lot! In my research anything other than double precision floating point mathematics adds noise and reduces the over all quality of the sound. So, 96 000 * 8 bytes for 1 second of music. Basically, a single note can take many megabytes of memory to store.

How Sonic Field Represents Sounds:

The simplest way would be to just use a Java array of doubles. Sadly this is not anything like enough. I use a 16Gig machine and it would run out of memory very quickly if I used simple arrays. The reason being that sounds are made of sounds (as we saw before). Each of those sine waves in the trumpet sound might be 10 seconds long (about 5 megabytes) and there could be 40 harmonics, so that is 400 mega bytes. Then we want to generate notes in multiple threads, for 8 threads that might be 8 notes at once which is 1.6 Gigabytes. That is just to great a 10 second note which is relatively simple. More complex and longer notes (especially in ambient music where notes can last a very long time) quickly exhaust the capacity of my machine. I could use a bigger machine and address say 128 Gig; that would do for most music, but again, such a machine is likely to have more cores and the parallel memory use issue appears again. Basically, sound processing is memory lop sided compared to most other forms of processing.
Sonic Field address this issue with two techniques: Memory mapped files and generators. Of these, generators are probably the most interesting programmatically because they are surprisingly hard to implement and do not work as well as we might hope.


package com.nerdscentral.audio;

import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;

import com.nerdscentral.audio.pitch.CubicInterpolator;
import com.nerdscentral.sython.SFMaths;
import com.nerdscentral.sython.SFPL_RuntimeException;

public abstract class SFSignal implements AutoCloseable
{

private final static AtomicLong uniqueId = new AtomicLong(0);
private final long myId;
protected final AtomicInteger referenceCount = new AtomicInteger(1);

public abstract boolean isKilled();

public abstract SFSignal replicate();

private static ThreadLocal<String> pythonStack = new ThreadLocal<>();

public final SFData replicateEmpty()
{
return SFData.build(this.getLength());
}

public abstract double getSample(int index);

protected SFSignal()
{
myId = uniqueId.incrementAndGet();
}

/**
* Returns a linearly interpolated sample based on the samples either side of the the passed index. This is used for super
* sampling or pitch effects.
*
* @param index
* @return
*/
public final double getSampleLinear(double index)
{
double s = SFMaths.floor(index);
double e = SFMaths.ceil(index);
if (s < 0 || e >= getLength())
{
return 0;
}
if (s == e) return getSample((int) s);
double a = getSample((int) s);
double b = getSample((int) e);
return ((index - s) * b + (e - index) * a);
}

/**
* Returns a cubic interpolated sample based on the samples either side of the the passed index. This is used for super
* sampling or pitch effects. Cubic interpolation uses two samples either side of the required point and so at the ends of
* the sample this will fall back to linear interpolation.
*
* @param index
* @return
*/
public final double getSampleCubic(double index)
{
int s = (int) SFMaths.floor(index);
int e = (int) SFMaths.ceil(index);
if (s < 0 || e >= getLength())
{
return 0;
}
if (s > getLength() - 3 || index < 1)
{
if (s == e) return getSample(s);
double a = getSample(s);
double b = getSample(e);
return ((index - s) * b + (e - index) * a);
}
return CubicInterpolator.getValue(getSample(s - 1), getSample(s), getSample(s + 1), getSample(s + 2), index - s);
}

public abstract double setSample(int index, double value);

public abstract int getLength();

public abstract void setAt(int pos, SFSignal data) throws SFPL_RuntimeException;

public abstract void setFrom(int pos, SFSignal dataIn) throws SFPL_RuntimeException;

public abstract double[] getDataInternalOnly();

/**
* Get a globally unique identifier for this SFSingal
*
* @return
*/
public final long getUniqueId()
{
return myId;
}

public abstract void release();

@Override
public void close() throws RuntimeException
{
int c = referenceCount.decrementAndGet();
if (c == 0) release();
else if (c < 0) throw new RuntimeException(Messages.getString("SFSignal.1")); //$NON-NLS-1$
}

public void incrReference()
{
referenceCount.incrementAndGet();

}

public SFSignal __pos__()
{
incrReference();
return this;
}

public SFSignal __neg__()
{
close();
return this;
}

public int getReference()
{
return referenceCount.get();
}

public void decrReference()
{
referenceCount.decrementAndGet();
}

public static String getPythonStack()
{
return pythonStack.get();
}

public static void setPythonStack(String ps)
{
SFSignal.pythonStack.set(ps);
}

public void clear()
{
// NOOP on root class
}
}

[This AGPL code is on Github]

This is the abstract root class for all signal data in Sonic Field. It is a bit of a mess to be honest, the perfect example of a one person generated class which has ended up a little unfocused. However, it works and concrete sub-classes of this make up generators and memory mapped file handling. We can also see a bunch of code for integrating with Python in a nice way and for explicit resource management; normal Java garbage collection is no use for memory mapped file based storage!

So what is a generator?

The basic architecture of synthesis ends up being (you can argue theoretically about this, but in reality this is what happens):

  1. Make a signal
  2. Manipulate that signal making a new signal
  3. Repeat until the wanted sound appears

The obvious architecture is then to:

  1. Signals are samples.
  2. Read samples.
  3. Apply and algorithm.
  4. Write samples.

We represent samples as indexed sequences of double . Therefore, for the above model, each sound manipulator algorithm reads from a sequence and writes out to a sequence. However, this involves a huge amount of memory access and memory access is very slow. When we run out of memory the operating system starts flushing data to the memory mapped files to help cope; reading and writing then becomes even slower. Consider a manipulation like this:

  1. Distort a signal using the ‘Saturate’ algorithm
  2. Halve the resulting volume
  3. Mix with another signal

We do not need to create intermediate data storage for these steps, they can be chained together. This is what generators do; they chain with other generators overriding the getSample method. Here is the saturate implementation:

 
public static class Translator extends SFSingleTranslator
{

protected Translator(SFSignal input)
{
super(input);
}

@Override
public double getSample(int index)
{
double x = getInputSample(index);
double y = x >= 0 ? x / (x + 1) : x / (1 - x);
return y;
}
}
Here the base class SFSingleTranslator implements the ability to contain and access the signal source. Mix is a little more complex as it has multiple source signals, but again, it is hardly expensive:

static class Translator extends SFMultipleTranslator
{

protected Translator(List<SFSignal> input)
{
super(input);
}

@Override
public double getSample(int index)
{
double d = 0;
for (int mindex = 0; mindex < getNMembers(); ++mindex)
{
d += getInputSample(mindex, index);
}
return d;
}
}
I hope it is clear how each of these can use the previous as a source and so no storage is required.

Why generators are not the solution to everything:

Generators seem so cool that we might think signal processing is a solved problem; just make all manipulations a generator and the reading/writing issue only comes at the end of signal generation when we write our finished music to disk. Sadly, the real world is more complex. Whilst this ideal concept might work for a restricted notion of real time signal manipulation, for the complex world of non realtime manipulation in Sonic Field we have non sequential access and multi-threading to contend with.
First, let us consider the simplest form of non sequential access; this is multiple access to the same signal. Whilst the previous algorithms were simple, some are not. If we consider a 6 pole infinite impulse response filter, then we are looking at something which will take hundreds or thousands of cycles to compute one signal sample. We do not want to accidentally do this twice! However, generators can cause that to happen. Consider that we split a signal into two (for left and right).

# Read the signal
signal=sf.ReadFile("mono.wav")
# Filter it
signal=sf.ButterworthLowPass(signal,1000,6)
# Split it
left=+signal
right=signal
# Add Haas effect
left  = sf.Concatenate(sf.Silence(30),left)
# Write it out
sf.WriteFile32((left,right,"stereo,wav")
If the Butterworth Low Pass filter is a generator then it will get run twice, once for the left and once for  the right. This would be insanely expensive. In the case of expensive algorithms we want to write and the result not keep recomputing it. So, for low cost transformations Sonic Field uses generators and for high cost it does indeed translate to a SFData; i.e. a memory mapped file.
Left get even more complex than that; we also have some processing which can take place in many, many non sequential subsections. A processor might cut a signal into pieces, for granular synthesis for example. In granular synthesis we take sound and chop it into tiny pieces and then manipulate each individually. Here is a video I did describing the technique:

In cases like this we also do not want the signal being non sequentially processes to be a generator. To cope with such situations we have the ‘realise’ method which converts generators to ‘real’ signals but has no effect on those which are already storage backed.

Similarly, consider white noise. This is ‘random’ so one might think it could easily be created from a generator. However, we want the same point in a signal (say sample 112121) to always have the same value each time we read it. So, in effect, white noise must be ‘recorded’ into a real data store. Finally we have the filters which have an internal state. Things like low pass filters are implemented as a state machine. That state is based on the order in which samples are passed through the filter. As such, they cannot be implemented as a generator as samples in Sonic Field are random access exactly because we might want to do things like granulate!

How Java Is Terrible At Memory Management (and approaching a solution):

Now we have some background to why managing audio data is such a challenge we are ready to see why Java is a bit of a catastrophe for managing memory. Java has garbage collection which means we do not know how much memory is actually being used. There are some forms of instrumentation which are supposed to tell us the amount of memory being used, but they are not very ‘real time’. In a C++ based system (for example) we could intercept malloc and know, to the byte, the amount of memory being used from the heap. This luxury is not given to us in Java. I tried, oh I tried so hard, to get a reliable way of estimating when the JVM was about to throw an out of memory exception; but as far as I can tell, there is no fail-safe way of doing so.

However, all is not lost as there is a usable work around. Indeed, the work around is always evolving and I would not be shocked it it became almost as effective as a hand crafted memory management system in C++ but with the platform independence of Java. It is not there yet, but it is getting there.

Memory Mapped Signal Pages

Memory mapped files are somewhat supported in Java. It is frustrating as anything that we can only memory map byte buffers and not double arrays; nevertheless, we can use them with the appropriate accessor methods. So, what Sonic Field does is have 2 megabyte ‘chunks’ which are ByteBuffers created by memory mapping a file. Using explicit resource management (thanks to Java 7) we know when a chunk is being used and when it is free. When one is freed it is added to a ConcurrentLinkedDeque of free chunks. This means our chunks are recyclable. How about we take a look at some of the code which does this:


static
{
tempDir = new File(System.getProperty(SONIC_FIELD_TEMP));
String swapLimitraw = System.getProperty(SONIC_FIELD_SWAP_LIMIT);
if (swapLimitraw == null)
{
swapLimit = MAX_IN_RAM; // 64 megabytes or 87 seconds at 96000
}
else
{
swapLimit = Long.parseLong(swapLimitraw);
}
chunkLen = swapLimit / 2;

String pid = ManagementFactory.getRuntimeMXBean().getName();
try
{
if (tempDir != null)
{
coreFile = File.createTempFile("SonicFieldSwap" + pid, ".mem", tempDir); //$NON-NLS-1$ //$NON-NLS-2$
}
else
{
coreFile = File.createTempFile("SonicFieldSwap" + pid, ".mem"); //$NON-NLS-1$//$NON-NLS-2$
}
coreFile.deleteOnExit();
// Now create the actual file
coreFileAccessor = new RandomAccessFile(coreFile, "rw"); //$NON-NLS-1$
}
catch (IOException e)
{
throw new RuntimeException(e);
}
channelMapper = coreFileAccessor.getChannel();
}

}

[This AGPL code is on Github]

Here we can see the creation of the swap file (the name I use for the memory mapped file). The most important thing is that we get the channel for the swap file. This is the object we can then use for memory mapping. Now we can see the code which creates the mapping for a new SFData object:


private void makeMap(long size) throws IOException
{
long countDown = size;
int chunkCount = 0;
while (countDown > 0)
{
++chunkCount;
countDown -= chunkLen;
}

countDown = size;
chunks = new ByteBuffer[chunkCount];
chunkCount = 0;
while (countDown > 0)
{
ByteBuffer chunk = freeChunks.poll();
if (chunk == null) break;
chunks[chunkCount] = chunk;
++chunkCount;
countDown -= chunkLen;
}
if (countDown > 0)
{
synchronized (coreFile)
{
long from = coreFile.length();
while (countDown > 0)
{
ByteBuffer chunk = channelMapper.map(MapMode.READ_WRITE, from, chunkLen);
chunk.order(ByteOrder.nativeOrder());
chunks[chunkCount] = chunk;
++chunkCount;
from += chunkLen;
countDown -= chunkLen;
}
}
}
}

[This AGPL code is on Github]

It is lovely and simple! We just use up chunks on the free ‘stack’ and if there are not enough to represent all the signal we create a few more until we have enough.  I use a deque as a stack because the most recently added chunk is also most likely not to be ‘swapped out’ by the operating system; optimum performance is achieve by reusing the same chunks over and over. There is no concurrent stack in the standard JDK, so the deque made a good choice.

It is worth noting that recycling the last used chunk rather and using a first in first out approach gave a huge performance improvement on my machine. When I tried a first in first out approach the system would run fast until swapping started and then slow right down as it constantly read back in swapped out chunks. This was a complete waste of time because the swapped out chunks contained data that was not going to be used again. By recycling last in first out, this performance issue disappeared and Sonic Field achieved very efficient use of the operating system file cache and virtual memory systems.

Using explicit resource management, we can ensure that chunks which are no longer required get reused:


@Override
public void close() throws RuntimeException
{
int c = referenceCount.decrementAndGet();
if (c == 0) release();
else if (c < 0) throw new RuntimeException(Messages.getString("SFSignal.1")); //$NON-NLS-1$
}

Yes, Sonic Field uses reference counting. This is tide into the explicit resource management in that SFSingal implements AutoCloseable. I will discuss the mechanism behind this a little more when we discuss the way the Java interacts with Python. However, what matters for now is that when the reference count hits zero ‘release()’ is called:


@Override
public void release()
{
for (ByteBuffer chunk : chunks)
{
freeChunks.push(chunk);
}
chunks = null;
resourceTracker.remove(this);
}

Release is responsible for putting the now unused chunks back onto the free deque. As the free deque is concurrent, we do not have to use synchronization blocks anywhere other than when creating new chunks.

Heavy disk IO as the OS starts serialising
blocks of the memory mapped file to disk

Now I can hear you screaming at me. My ears are ringing and it has nothing todo with this noisy train I am on. “But you should use a direct double buffer” you say. Yes, nio has these double buffers so we would use put() and get() rather than putDouble() and getDouble(). Surely these are ‘blessed’ in some magical way to make them perform. Nope, that approach is very slightly slower than using ByteBuffers directly. In other words, getDouble and putDouble being used ‘under the covers’ or something very similar. Here is a simple test which creates a memory mapped signal and reads from it then creates another etc:

from java.lang import System
sig=sf.WhiteNoise(10000)
t0=System.nanoTime()
for x in range(1,10):
    sig=sf.Swap(sig)
    for z in range(1,5):
        sig1=sf.Realise(sf.Pcnt10(+sig))
    sig=sig1
        
t1=System.nanoTime()
for y in range(1,5):
    t0=System.nanoTime()
    for x in range(1,10):
        sig=sf.Swap(sig)
        for z in range(1,5):
            sig1=sf.Realise(sf.Pcnt10(+sig))
        sig=sig1
    t1=System.nanoTime()
    print t1-t0

Times using a double buffer:
1497275000
1474564000
1410076000
1474493000
Sonic Field using 48 Gig of swap file
on a machine with 16 Gig of RAM

Times using a byte buffer only:

1375515000
1376532000
1478389000
1438900000
On my machine the memory mapped files would not have had to actually resort to disk (SSD actually) as there was plenty of RAM available. Making sure the byte order is correct for the current platform (in my case LITTLE_ENDIAN, which one can get from ByteOrder.nativeOrder()) does help a little, which is interesting as it shows some form of optimisation going on.

A Quick Note On Caching

To anyone who has worked in software performance tuning, seeing a system with serialisation and deserialisation and writing to memory mapped files will immediately cause the word ‘cache’ to appear in bright neon in their mind’s eye. This is certainly the case for me, and until recently Sonic Field did have a caching scheme which simultaneously helped a bit, caused immense complexity and produced constant out of memory errors. As of now I have given up on that approach. Basically, the serialisation costs are painful but acceptable, especially considering that they can be reduced by ever more  aggressive use of generators. Caching systems add an overhead in themselves and so they offer less benefit than one might expect. One day I might dream up a better solution, but for now, this seems to be reliable if not exactly stella performance wise.

To finish this section on memory management, here is a piece of music which required a 48G swap file. This entire 32.5 minute sound scape, including all the reverberation, harmonic excitation and delay effects were all done in Sonic Field using double precision floating point mathematics at 96 000 samples per second. When I say it like that, I guess it is not such a shock that it took a lot of resources!

What Is Arbitrary Complexity Ab-initio Synthesis?

A quick lesson in synthesis. The pioneers of electronic music made notes and sounds. Later came the work of the legendary Dr Robert Moog. He created the concept of what we now call a synthesiser. As always, we could include many others in this creation, some working with him and some in parallel work. But let us just think about his work. He realised that very flexible sound creation could be achieved by oscillators, which create tones, and filters, which remove unwanted tones. These were ‘analogue’ electronic circuits which produced a limited number of tones and could filter them in a limited number of ways to produce sounds in ‘real time’.  The complexity of the sound was controlled by the complexity of the synthesiser. However, limitations in analogue technology set an upper limit to complexity; every circuit added some distortion and some noise, eventually these artefacts would overwhelm the desired sound and so the upper complexity limit was set.

These early pioneers created what we now think of as the ‘principles of synthesis’ and by so doing formed a wonderful working model for people to learn but also restricted creativity. Computational and digital synthesis tends to continue to follow the same path as the original analogue work. It is real time (press a key and out comes sound) and restricted (there are a fixed number of digital sound sources and filters etc). “Arbitrary complexity” simply means that there is no upper limit to the number of filters, oscillators and such that can be used. One key concept behind Sonic Field (which I have not yet quite gotten around to implementing) is that it can function in a cloud environment an coordinate an arbitrarily large number of nodes to create an arbitrarily complex piece of music or soundscape. Whilst the implementation is not quite there yet, the current architecture is very much suited to this approach.

There exists a completely different sort of digital sound creation which some people call synthesis (but I do not). This is sample looped based. Here sounds are recorded or created digital and stored as a ‘sample’ of that sound. These samples are then played back in a loop to create the notes heard. This is not ‘from first principles’ because the sounds already existed. Creating sounds by the direct manipulation of mathematical equations or the direct application of electronic circuits creates the sounds from first principles. This is where ‘ab-initio’ comes from.

So Why Java?

I get asked this quite a bit when I discuss Sonic Field with people. The first question often is why not do this in C++? There are many answers and as this post is in a Java advent calendar I do not intend to defend Java. I think the question comes from an incorrect view that Java is somehow non performant. I often hear people saying Java is interpreted, even now! My view is that Java has some huge specific benefits for this work. They centre around multi-threading, clustering and portability:
  1. I originally intended to run Sonic Field on top of hadoop so processing could be distributed in a cloud. I might well still do something like this. Java’s serialisation system and the maturity of the tooling around cloud computing in Java makes it an ideal candidate.
  2. Heavy multi-threading is more mature in Java than in C++. At the time is started the project C++11 was only just coming on line, so the contrast was even more stark.
  3. Being able to run Sonic Field anywhere and not having to recompile it for different platforms has been a huge benefit. I have run it on Windows, Linux and (almost exclusively for a couple of years now) on the Mac. I have never had to change a line of code to do these things.
  4. I like doing strange things which people do not expect; proving Java can handle large scale mathematical calculations has a perverse fun streak running through it!
On personal reflection, I do not think the project would have survived this long if it had been in C++. The effort of maintaining the build and updating from one version of gcc to the next and across to Visual Studio would have caused me to give up. Seriously, the memory management issues might one day start me porting off the JVM but I still hold hope I will keep coming up with a fixes for them rather than give up on this amazing platform; current evidence is that the memory mapped files approach is ‘good enough’ and I live in hope of coming up with either more tweaks to that or an even more effective approach.

An Aside On Threading

I said Java was good for threading. If you know anything about Python in C you will know it is single threaded (well, the interpreter is through a thing called the global interpreter lock). Jython is very multi-threaded. One of the driving concerns in Sonic Field has always been parallel execution be that via clustering or local threading. So I think it is time we took a look at the current threading model in Sonic Field (which changes almost as often as the memory management).

Now, this is a bit complex for a single post like this, so I will skim over a lot of the details. Nevertheless, I think this mechanism is a good demonstration of how Jython and Java can work well together.

import threading
import time
from java.util.concurrent import Executors, TimeUnit
from java.util.concurrent import Callable, Future
from java.lang import System
from java.lang import ThreadLocal
from java.lang import Thread
from java.util.concurrent import TimeUnit
from java.util.concurrent.locks import ReentrantLock

SF_MAX_CONCURRENT = int(System.getProperty("synthon.threads"))
SF_MAX_CONQUEUE = SF_MAX_CONCURRENT

print "Concurrent Threads: " + SF_MAX_CONCURRENT.__str__()
SF_POOL = Executors.newCachedThreadPool()

class sf_callable(Callable):
def __init__(self,toDo):
self.toDo=toDo

def call(self):
return self.toDo()

class sf_futureWrapper(Future):
def __init__(self,toDo):
self.toDo=toDo
self.gotten=False

def __iter__(self):
return iter(self.get())

def get(self):
if self.gotten:
return self.result
else:
self.result=self.toDo.get()
self.gotten=True
return self.result

def __pos__(self):
obj=self.get()
return +obj

def __neg__(self):
obj=self.get()
return -obj

class sf_getter(Future):
def __init__(self,toDo):
self.toDo=toDo
self.result=self.toDo()

def get(self):
return self.result

def __iter__(self):
return iter(self.get())

def __pos__(self):
obj=self.get()
return +obj

def __neg__(self):
obj=self.get()
return -obj

class sf_taskQueue(ThreadLocal):
def initialValue(self):
return []

SF_TASK_QUEUE=sf_taskQueue()

class sf_superFuture(Future):

def __init__(self,toDo):
self.toDo=toDo
queue=SF_TASK_QUEUE.get()
queue.append(self)
if len(queue)>SF_MAX_CONQUEUE:
self.submitAll()

def submit(self):
count=SF_POOL.getActiveCount()
if count<SF_MAX_CONCURRENT:
task=sf_callable(self.toDo)
self.future=sf_futureWrapper(SF_POOL.submit(task))
else:
self.future=sf_getter(self.toDo)

def submitAll(self):
queue=SF_TASK_QUEUE.get()
while(len(queue)):
queue.pop().submit()

def get(self):
self.submitAll()
while not hasattr(self,'future'):
Thread.yield()
r = self.future.get()
return r

def __iter__(self):
return iter(self.get())

def __pos__(self):
obj=self.get()
return +obj


def __neg__(self):
obj=self.get()
return -obj

def sf_do(toDo):
return sf_superFuture(toDo)

def shutdown_and_await_termination(pool, timeout):
pool.shutdown()
try:
if not pool.awaitTermination(timeout, TimeUnit.SECONDS):
pool.shutdownNow()
if not pool.awaitTermination(timeout, TimeUnit.SECONDS):
print >> sys.stderr, "Pool did not terminate"
except InterruptedException, ex:
pool.shutdownNow()
Thread.currentThread().interrupt()

def shutdownConcurrnt():
shutdown_and_await_termination(SF_POOL, 5)
This mechanism launches futures to process threads. It does this using closures which are really easy to do in Python. It gets much more complex than one might think though because of the issue of recursive launching (which I will touch on here in a bit but is covered in more detail in my post ‘The Embrace Of Meh‘). Sonic Field aims to have a simple approach to threading. The very idea that such a thing exists is a little naive! However, it does get some way towards this goal. Here is an example. Consider that we have function sf.FrequencyDomain() which is very expensive and a function xmulti() which cross multiplies the output of this expensive processor.


# Form A
def mixer(siga,sigb):
    return sf.CrossMultiply(
        sf.FrequencyDomain(siga),
        sf.FrequencyDomain(sigb)
    )

We can convert that into running in parallel simply by making the bulk of the work happen in a closure:


# Form B
def mixer(siga,sigb):
    def mixerInner():
        return sf.CrossMultiply(
            sf.FrequencyDomain(siga),
            sf.FrequencyDomain(sigb)
       )
    return sf_do(mixerInner)

I love this because it requires so little syntax and just works. sf_do creates a future for the execution of the closure. The snag comes when we want to use that future. Because in the Java world we need the result (a signal) not a future. How can we  fix this type safety issue? Despite the parallel submission code being written in Python it uses java.util.concurrent.Future from the JDK. This means we can dynamically dispatch of the instance type of arguments. All Sonic Field processors convert their argument dynamically from Object to what ever they require. They do this via the static methods on the com.nerdscentral.synthon.Caster class. The key one of those methods being this:


public static Object checkAutoTranslation(Object o) throws SFPL_RuntimeException
{

if (o instanceof Future)
{
Future doer = (Future) o;
try
{
Object d = doer.get();
return checkAutoTranslation(d);
}
catch (Throwable t)
{
throw new SFPL_RuntimeException(t);
}
}

if (o == null)
{
throw new SFPL_RuntimeException(Messages.getString("Caster.12")); //$NON-NLS-1$
}

return o;
}

I have highlighted in bold where the conversion occurs.

Naturally, tasks can submit tasks, so the system must be recursive. It will keep calling get() on Futures until it find a non Future result. However, you might have noticed that the code for creating futures is not as simple as it might be. What is a ‘SuperFuture’? This is a huge challenge. There is a finite number of threads available on any computer; when we have tasks submitting new tasks and each is in a new thread we can exhaust the maximum number easily. The classic solution is to use a thread pool. Sadly, if say we have 16 threads in a pool then one task submits 2 and then those submit 2 each and then those submit 2 each we have 1+2+4+8=15 threads. If any of the new leaf tasks then submit tasks we will see the pool exhausted. This means they the new tasks queue up for the pool to empty. But the pool never empties because it is full of blocked threads which are waiting for the completion of tasks which are queued up.

SuperFutures work around this because they can either result in a true Future which is asynchronously executed or they can, if the pool is getting dangerously full, be executed serially. i.e. a SuperFuture converts into either a Future (parallel) or Getter (serial) when it is executed. Its execution is delayed until an input queue hits a particular size of the result of the SuperFuture is requested from another Future or Getter. This delay effect helps fine grained tasks to end up as Getters and their parents to be the Futures.

I have not implemented a full work stealing system because the complexity of working out the dependencies in the Python puts me off. I want the system to be easy to use and I have not worked out an easy to use work stealing system yet; so for now the super future appraoch appears to work OK.

Wrapping It Up?

There is so much I wanted to discuss and did not! For example, I wanted to discuss explicit resource management in much more detail. I would love to have discussed caching in FFT. How about sub-normal floating point performance? Immutable data structures and threading for large scale mathematics?

What I hope I have shown is how some of the ‘heavy engineering’ in Sonic Field has allowed something one would often not associated with Java – huge scale mathematics – to work rather cleanly in this language. Sonic Field has taught me so much about practical threading issues, real memory management challenges and effective flexible processing architecture. I hope that at least some of these concepts will be of use to you the reader.

This post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on!

Published by

Alexander Turner

Life long technologist and creative.

4 thoughts on “A Serpentine Path To Music”

Leave a Reply