Filipe De Sousa

Playing Photographer

Photo, Hipster-ish

As a late Christmas present to myself, I decided to buy myself a camera. I’ve always played about with my dad’s cameras, and since having bought him a D3100 a few years ago, I’ve nicked it on occasion, or tried to, to get some interesting shots. It just didn’t cut it though, so I’ve gone the other way. I’m packing a Canon. To be precise, the EOS 100D, because although I’m massive, the idea of a DSLR that can be almost completely covered with one hand is nice for city shots.
It’s a bit of a shame we can’t just “borrow” each other’s lenses, but it does avoid the confusion if we were to buy similar ones (he’s a little careless though, so it’s whichever lens has most scratches, I’d assume is his).

Speaking of lenses, I find wide shots far more interesting, giving a sense of depth, taking in more of a scene, so I’ve ordered a Tokina 11-16mm (after having read so many reviews about it, the Sigma 8-16mm, and the Canon 10-22mm, I just gave up trying to choose and ordered the cheapest of the three, since it should be easy enough to sell on if I want to switch). Until that arrives, I’m pottering about during some of my lunch breaks with my Canon 40mm f/2.8 STM, and a Jupiter-37A 135mm f/3.5.
The Jupiter lens is a gem, a real treat to use, and makes me wish there were more with manual focus and aperture controls that felt that good to use. I haven’t really been able to judge distances correctly with it just yet (I think the distance guide is off by a big amount), so some of my shots are coming out unfocused, though when you set it to f/8, at a good long focus, you could just point-and-shoot. Originally, I bought a simple adaptor, in case it didn’t work out, there’s little point in buying anything but the cheapest adaptor. Now with an AF-confirm adaptor, it’s far easier to use.
Canon made a tasty little pancake with their 40mm lens. Stuck trying to choose between buying a 35mm or a 50mm, I went for the middle-ground, and I’m not disappointed. The 50mm/”nifty-fifty” is highly regarded, cheap and cheerful, with a nice big aperture of f/1.8, but the 40mm is sharper at f/2.8, not much more expensive, and a lot smaller, a little lighter and better built. It’s really paired up nicely with the EOS 100D, making a great compact combo.
So far, only my cats are unimpressed by the camera and lenses.

There have been some bright and lovely days already this month, and I’ve tried to take advantage of them during lunch. Lucky for me, I’m right next to Regent’s Park, which makes for some great shots.

As is normal for me, my geeky side is constantly looking at upgrading. Better lenses (some of those “L” lenses are rather more affordable than I realised), better bodies (hmmm, a 5D Mk III isn’t THAT much more, and I bet it could do loads more), better accessories (need an ND filter or three, maybe another CPL, a tripod, maybe a few cleaning apparatus, etc.). Once the Tokina arrives, hopefully that will subside, since I can get some of the wider shots I’ve wanted to take; the camera itself is a brilliant bit of kit, lightweight and tiny, even being smaller than some mirrorless cameras. With the 40mm on, it evens fits in a coat pocket.


Portable Object Format Serialisation With Inheritance

Coherence, Serialisation, Inheritance, Programming

The basics of POF serialisation are covered enough across the internet, but this article tries to show what you can do about serialising objects while accounting for changeable inheritance structures.
As part of the previous sprint at work we began implementing versioned and evolvable serialised objects for our Oracle Coherence cache in Portable Object Format (POF), which is currently not used in an easily evolvable way.
This post is a bit of a brain dump on how we’ve implemented versioning, which somebody may find useful, since the documentation isn’t always perfectly clear about some points meaning it’s been a trial-and-error implementation.

Implementation of POF serialisation and making objects evolvable is all done in code and requires hard-coding of the index numbers to keep it compatible between versions and languages, which makes for a rather tedious few days of work. When your colleagues haven’t written unit tests for some objects you’re making evolvable, you don’t have cache nodes to test on, and have multiple objects with deep inheritance models to adapt across multiple languages, it becomes an awful lot of fun to work on.

Before setting off I’m going to point out an excellent post by Shan He, which helped more than any other blog post, implementation example, or even the official documentation in understanding how to implement a solution. That post is occasionally quoted and paraphrased below, since it really illuminates some areas well alone.

Simple Serialisation

There are two basic ways to serialise objects into POF, which involve either making the object self-serialisable or using a separate serialiser class. Each have some advantages and disadvantages as is to be expected. With either approach, implementing the Evolvable interface or extending AbstractEvolvable in the serialised object is practically essential, and provides the base for evolvable object serialisation.

If you’re going for the self-serialisation route you implement the PortableObject interface (or the useful EvolvablePortableObject interface that simply combines Evolvable and PortableObject interfaces in one), using the method readExternal(PofReader) : void to deserialise an object from the PofReader, and using the method writeExternal(PofWriter) : void to serialise the object into the PofWriter.

For the separate serialiser, you write another class that implements PofSerializer, defining the method serialize(PofWriter, Object) : void to serialise to the PofWriter, and the method deserialize(PofReader) : Object to deserialise from the PofReader.

Modelling a Simplistic Bond

Let’s have a look at a really basic example of both approaches, serialising a simple representation of a Bond. Keeping the object simple:

FieldData Type
Contract NumberLong
CurrencyString
Counter PartyString
IssuerString
Maturity DateDate
PrincipalDouble
CouponDouble
PriceDouble

Simple Evolvable Portable Object Example

public class Bond extends AbstractEvolvable implements EvolvablePortableObject {
    /* Define object implementation version */
    public static final int IMPL_VERSION = 1;

    /* Define the indices of the serialised fields */
    public static final int CONTRACT_NUMBER_INDEX = 1;
    public static final int CURRENCY_INDEX = 2;
    public static final int COUNTERPARTY_INDEX = 3;
    public static final int INSTRUMENT_INDEX = 4;
    public static final int ISSUER_INDEX = 5;
    public static final int MATURITY_DATE_INDEX = 6;
    public static final int PRINCIPAL_INDEX = 7;
    public static final int COUPON_INDEX = 8;
    public static final int PRICE_INDEX = 9;

    /* Define the fields */
    private long contractNumber;
    private String currency;
    private String counterparty;
    private String instrument;
    private String issuer;
    private Date maturityDate;
    private double principal;
    private double coupon;
    private double price;

    /* Define no-args constructor and other constructors */
    public Bond() {
        super();
    }

    public Bond(long contractNumber, String currency, String counterparty,
        String instrument, String issuer, Date maturityDate,
        double principal, double coupon, double price) {

        super();
        this.contractNumber = contractNumber;
        this.currency = currency;
        this.counterparty = counterparty;
        this.instrument = instrument;
        this.issuer = issuer;
        this.maturityDate = maturityDate;
        this.principal = principal;
        this.coupon = coupon;
        this.price = price;
    }

    /* Implementation, getters/setters, toString, hashCode, equals */

    /* Serialisation method */
    @Override
    public void writeExternal(PofWriter writer) throws IOException {
        writer.setVersionId(Math.max(getImplVersion(), getDataVersion()));

        /* Serialise ourselves in ascending index order */
        writer.writeLong(CONTRACT_NUMBER_INDEX, getContractNumber());
        /* Skipping rest of writes */

        writer.writeRemainder(getFutureData());
    }

    /* Deserialisation method */
    @Override
    public void readExternal(PofReader reader) throws IOException {
        setDataVersion(reader.getVersionId());

        /* Deserialise ourselves in ascending index order */
        setContractNumber(reader.readLong(CONTRACT_NUMBER_INDEX));
        /* Skipping rest of reads */

        setFutureData(reader.readRemainder());
    }

    /* Remaining to implement as part of Evolvable - getImplVersion() */
    @Override
    public int getImplVersion() {
        return IMPL_VERSION;
    }
}

If you prefer not to break the single responsibility principle, and consider a class taking care of its own serialisation a breach of that principle, this is not the method for you. The one below will likely make more sense.

Simple Evolvable Object Serialiser Example

Object

For this example, we keep just the object’s fields and methods, implement just Evolvable by extending AbstractEvolvable, and remove the serialisation code.

public class Bond extends AbstractEvolvable {
    /* Define object implementation version */
    public static final int IMPL_VERSION = 1;

    /* Define the fields */
    private long contractNumber;
    private String currency;
    private String counterparty;
    private String instrument;
    private String issuer;
    private Date maturityDate;
    private double principal;
    private double coupon;
    private double price;

    /* Define no-args constructor and other constructors */
    public Bond() {
        super();
    }

    public Bond(long contractNumber, String currency, String counterparty,
        String instrument, String issuer, Date maturityDate,
        double principal, double coupon, double price) {

        super();
        this.contractNumber = contractNumber;
        this.currency = currency;
        this.counterparty = counterparty;
        this.instrument = instrument;
        this.issuer = issuer;
        this.maturityDate = maturityDate;
        this.principal = principal;
        this.coupon = coupon;
        this.price = price;
    }

    /* Implementation, getters/setters, toString, hashCode, equals */

    @Override
    public int getImplVersion() {
        return IMPL_VERSION;
    }
}

Serialiser

The serialisation code and field indices are now in the serialiser class, which does the heavy lifting on serialising and deserialising the class.

public class BondSerializer implements PofSerializer {
    /* Define indices for the serialised fields */
    public static final int CONTRACT_NUMBER_INDEX = 1;
    public static final int CURRENCY_INDEX = 2;
    public static final int COUNTERPARTY_INDEX = 3;
    public static final int INSTRUMENT_INDEX = 4;
    public static final int ISSUER_INDEX = 5;
    public static final int MATURITY_DATE_INDEX = 6;
    public static final int PRINCIPAL_INDEX = 7;
    public static final int COUPON_INDEX = 8;
    public static final int PRICE_INDEX = 9;

    /* Serialise the object into the writer */
    public void serialize(PofWriter writer, Object o) throws IOException {
        /* Check Object is of correct type and is not null before casting */
        if (!(o instanceof Bond)) {
            // Handle the case of an invalid/null object
        }
        Bond bond = (Bond) o;

        writer.setVersionId(Math.max(bond.getImplVersion(), bond.getDataVersion()));

        /* Serialise ourselves in ascending index order */
        writer.writeLong(CONTRACT_NUMBER_INDEX, bond.getContractNumber());
        /* Skipping rest of writes */

        writer.writeRemainder(bond.getFutureData());
    }

    /* Deserialise from the reader into a new object */
    public Object deserialize(PofReader reader) throws IOException {
        /* Instantiate a new Bond to deserialise into */
        Bond bond = new Bond();

        bond.setDataVersion(reader.getVersionId());

        /* Deserialise ourselves in ascending index order */
        bond.setContractNumber(reader.readLong(CONTRACT_NUMBER_INDEX));
        /* Skipping rest of reads */

        bond.setFutureData(reader.readRemainder());
    }
}

It is just more of the same, a case of defining the serialisation outside of the class instead of part of it, with both forms of serialisation displayed above being suitable for our current object hierarchy of one.

For the above examples a simple Bond contract was implemented, but what if we want to model a Future? Do we really need to re-implement all of the useful fields that apply to both? Of course not, we abstract things.

Serialisation and Inheritance

In POF serialisation there are three important attributes for evolvable objects, implementation version, data version and future data. Future data is a record of any data we may not be able to process because it’s from a newer implementation, and the version numbers define what version data being serialised or deserialised is, and also what it should contain.

By extending AbstractEvolvable, the storage of data version and future data is taken care of. The object must define its own getImplVersion() method returning the implementation version. Whether implementing PortableObject, or defining a separate PofSerializer, the base has been established.

What happens when you inherit from an evolvable object?

As stated above, to make objects backwards- and forwards-compatible and allow evolution, POF serialisation uses implementation version, data version and future data, which are stored in the object. When you are serialising a class with super-classes that require serialisation, where do you get the version number from, and what happens when adding new fields? The documentation does not clarify the situation, so below is my interpretation of what should happen.

Implementation version defines the version of the implementation of the object. Every time a serialisable attribute is changed in the object this version number should increase, since this number will also define the version number of data constructed by this implementation. The Evolvable interface defines the function getImplVersion() which is expected to return the version number of the object’s implementation. Its use is related to the data version, explained below.

Data version defines the version of the data being serialised or deserialised. If the object has just been instantiated the data version will be the same as the implementation version, but if it’s been previously deserialised the data could have come from a future implementation. This is why you see the call to Math.max(getImplVersion(), getDataVersion()) in the examples when calling PofWriter.setVersionId(), so we always define the latest version suitable for the data.

Future data is held in a Binary object, and is read via PofReader.readRemainder(), which does what it says on the tin, reading any remaining data we haven’t read. If deserialising from a newer implementation version, we might have some remaining data that should be kept in case we must serialise the same object again. As mentioned above, this is why we keep a copy of the data version around, and call setVersionId(Math.max(getImplVersion(), getDataVersion())) to set the version to the latest one.

As each layer should have its own implementation version, it should also hold onto its own data version and future data. This means AbstractEvolvable should not be extended, instead (re)implementing Evolvable at each level with private versions and data members. PortableObject also requires (re)implementation at each level, with sub-classes calling their super-class methods with a nested POF reader when deserialising, or writer when serialising.
There’s a fair portion of code being copied around, but it’s a necessary evil to have the implementation we want.

Solutions that account for inheritance using the afore-mentioned nested readers and writers have great flexibility. Due to the nesting implementation, each class can have index numbers without clashes. Sub-classes can be added to the inheritance structure without changing the serialisation or implementation of a super-class. Super-classes are none-the-wiser that they’re not the top-level serialised objects. The greatest benefit of all, is that a new serialised attribute of a super-class can be shared to all sub-classes, with no changes to their own serialisation.
Most articles and posts don’t detail this implementation, avoiding the subject due to not knowing its intent or use, or because they were written before it was available in the API.

POF serialisation becomes a bit tedious once you’ve a large enough class, as you write much the same code over and over. After declaring a new field, write the getter and setter methods, add a new index for the field to be serialised and deserialised at, increment the implementation version, and add the lines to serialise and deserialise the field. That’s the very basics of it, and if you have to add thirty fields, you must follow those steps thirty times. Imagine the extreme tedium of serialising classes five levels deep.

Modelling Bonds and Futures Contracts

We’re extending the modelled objects further to give us possibility for some new examples, adding Futures contracts to the system, which share a number of base attributes with Bonds contracts. First, defining what goes into each layer.

Contract Base

FieldData Type
Contract NumberLong
CurrencyString
Counter PartyString
BrokerString
PriceDouble

Contract number, currency, counterparty and price would have been found in both a Bond and a Future contract, but note the addition of the Broker field to the contract base so a Bond can have a Broker assigned to it as well as a Future.

Bond Contract

FieldData Type
IssuerString
Maturity DateDate
PrincipalDouble
CouponDouble

We’re keeping the above fields specifically only for Bonds, since they’re not relevant for all of the products we want to implement. The fields in the base were previously held exclusively in Bond, but they’ve now been abstracted out, as seen in the definition of Contract above.

Future Contract

FieldData Type
ContractString
Tenor CodeString
LotsLong

To make a basic Future contract we only need a few attributes, so this looks rather minimal and with good reason. Everything else we currently absolutely require is held in the base.

Evolvable Portable Objects with Inheritance Example

Contract

We’re making the Contract evolvable, and self-serialisable so that it may hold its own data version, implementation version and future data without clashes with sub-classes replacing them. As it’s able to be abstract, we’ll take advantage of that to ignore implementations of part of the Evolvable interface. Otherwise, this is very similar to the original Bond serialisation example.

public abstract class Contract implements EvolvablePortableObject {
    /* Define object implementation version */
    public static final int IMPL_VERSION = 1;

    /* Define the indices of the serialised fields */
    public static final int SUPERCLASS_INDEX = 0; // Should be the same in all sub-classes
    public static final int CONTRACT_NUMBER_INDEX = 1;
    public static final int CURRENCY_INDEX = 2;
    public static final int COUNTERPARTY_INDEX = 3;
    public static final int BROKER_INDEX = 4;
    public static final int PRICE_INDEX = 5;

    /* Define serialisation-specific fields */
    private int dataVersion;
    private Binary futureData;

    /* Define the fields */
    private long contractNumber;
    private String currency;
    private String counterparty;
    private String broker;
    private double price;

    /* Define no-args constructor and any others */
    protected Contract() {
        // Nothing to do here, no super(), nothing...
    }

    protected Contract(long contractNumber, String currency,
        String counterparty, String broker, double price) {

        this.contractNumber = contractNumber;
        this.currency = currency;
        this.counterparty = counterparty;
        this.broker = broker;
        this.price = price;
    }

    /* Implementation, getters/setters, toString, hashCode, equals */

    /* Serialisation method */
    @Override
    public void writeExternal(PofWriter writer) throws IOException {
        writer.setVersionId(Math.max(IMPL_VERSION, dataVersion));

        /* Serialise ourselves in ascending index order */
        writer.writeLong(CONTRACT_NUMBER_INDEX, getContractNumber());
        /* Skipping rest of writes */

        writer.writeRemainder(futureData);
    }

    /* Deserialisation method */
    @Override
    public void readExternal(PofReader reader) throws IOException {
        dataVersion = reader.getVersionId();

        /* Deserialise ourselves in ascending index order */
        setContractNumber(reader.readLong(CONTRACT_NUMBER_INDEX));
        /* Skipping rest of reads */

        futureData = reader.readRemainder();
    }
}

Bond

The first, and most important, thing to note are the calls to PofWriter.createNestedPofWriter() and PofReader.createNestedPofReader(), which will allow serialising and deserialising to and from a numbered field, without the object being handed the reader/writer instance knowing there’s a difference.
This is how we handle serialisation of our superclasses, by giving it a nested reader/writer, always using field number zero.

public class Bond extends Contract {
    /* Define object implementation version - can differ from Contract Base */
    public static final int IMPL_VERSION = 1;

    /* Define the indices of the serialised fields - can use same indices as super-class */
    public static final int INSTRUMENT_INDEX = 1;
    public static final int ISSUER_INDEX = 2;
    public static final int MATURITY_DATE_INDEX = 3;
    public static final int PRINCIPAL_INDEX = 4;
    public static final int COUPON_INDEX = 5;

    /* Define serialisation-specific fields */
    private int dataVersion;
    private Binary futureData;

    /* Define the fields */
    private String instrument;
    private String issuer;
    private Date maturityDate;
    private double principal;
    private double coupon;

    /* Define no-args constructor and any others */
    public Bond() {
        super();
    }

    public Bond(long contractNumber, String currency, String counterparty,
        String broker, double price, String instrument, String issuer,
        Date maturityDate, double principal, double coupon) {

        super(contractNumber, currency, counterparty, broker, price);
        this.instrument = instrument;
        this.issuer = issuer;
        this.maturityDate = maturityDate;
        this.principal = principal;
        this.coupon = coupon;
    }

    /* Implementation, getters/setters, toString, hashCode, equals */

    /* Serialisation method */
    @Override
    public void writeExternal(PofWriter writer) throws IOException {
        writer.setVersionId(Math.max(IMPL_VERSION, dataVersion));

        /* Serialise superclass first using nested writer */
        PofWriter subWriter = writer.createNestedPofWriter(SUPERCLASS_INDEX);
        super.writeExternal(subWriter);

        /* Serialise ourselves in ascending index order */
        writer.writeString(INSTRUMENT_INDEX, getInstrument());
        /* Skipping rest of writes */

        writer.writeRemainder(futureData);
    }

    /* Deserialisation method */
    @Override
    public void readExternal(PofReader reader) throws IOException {
        dataVersion = reader.getVersionId();

        /* Deserialise superclass first using nested reader */
        PofReader subReader = reader.createNestedPofReader(SUPERCLASS_INDEX);
        super.readExternal(subReader);

        /* Deserialise ourselves in ascending index order */
        setInstrument(reader.readString(INSTRUMENT_INDEX));
        /* Skipping rest of reads */

        futureData = reader.readRemainder();
    }

    /* Evolvable implementation */
    @Override public int getImplVersion() { return IMPL_VERSION; }
    @Override public int getDataVersion() { return dataVersion; }
    @Override public void setDataVersion(int dataVersion) { this.dataVersion = dataVersion; }
    @Override public Binary getFutureData() { return futureData; }
    @Override public void setFutureData(Binary futureData) { this.futureData = futureData; }
}

Future

This has the same structure as the Bond class, with differences in the fields used, so it’s nothing exciting.

public class Future extends Contract {
    /* Define object implementation version - can differ from Contract Base */
    public static final int IMPL_VERSION = 1;

    /* Define the indices of the serialised fields - can use same indices as super-class */
    public static final int CONTRACT_INDEX = 1;
    public static final int TENOR_CODE_INDEX = 2;
    public static final int LOTS_INDEX = 3;

    /* Define serialisation-specific fields */
    private int dataVersion;
    private Binary futureData;

    /* Define the fields */
    private String contract;
    private String tenorCode;
    private double lots;

    /* Define no-args constructor and any others */
    public Future() {
        super();
    }

    public Future(long contractNumber, String currency, String counterparty,
        String broker, double price, String contract, String tenorCode,
        double lots) {

        super(contractNumber, currency, counterparty, broker, price);
        this.contract = contract;
        this.tenorCode = tenorCode;
        this.lots = lots;
    }

    /* Implementation, getters/setters, toString, hashCode, equals */

    /* Serialisation method */
    @Override
    public void writeExternal(PofWriter writer) throws IOException {
        writer.setVersionId(Math.max(IMPL_VERSION, dataVersion));

        /* Serialise superclass first using nested writer */
        PofWriter subWriter = writer.createNestedPofWriter(SUPERCLASS_INDEX);
        super.writeExternal(subWriter);

        /* Serialise ourselves in ascending index order */
        writer.writeString(CONTRACT_INDEX, getContract());
        /* Skipping rest of writes */

        writer.writeRemainder(futureData);
    }

    /* Deserialisation method */
    @Override
    public void readExternal(PofReader reader) throws IOException {
        dataVersion = reader.getVersionId();

        /* Deserialise superclass first using nested reader */
        PofReader subReader = reader.createNestedPofReader(SUPERCLASS_INDEX);
        super.readExternal(subReader);

        /* Deserialise ourselves in ascending index order */
        setContract(reader.readString(CONTRACT_INDEX));
        /* Skipping rest of reads */

        futureData = reader.readRemainder();
    }

    /* Evolvable implementation */
    @Override public int getImplVersion() { return IMPL_VERSION; }
    @Override public int getDataVersion() { return dataVersion; }
    @Override public void setDataVersion(int dataVersion) { this.dataVersion = dataVersion; }
    @Override public Binary getFutureData() { return futureData; }
    @Override public void setFutureData(Binary futureData) { this.futureData = futureData; }
}

Alternatives

The above is inelegant, messy and pretty ugly. Here are some alternatives you might want to check out.

“How about if I’m trying not to break the single-responsibility principle?” I hear you ask.
Easy, reflection can help you, and to be more specific Alexey Ragozin’s open-source ReflectionPofSerializer can do this for you. Read about the implementation right here on Grid Dynamics’ blog.

If you’re not overly bothered about single-responsibility, want a solution at compile-time rather than reflection-based, or really need those extra few thousands operations per second then you’ll be interested in the open-source POF serializer generator written by Robert Varga.

Is inheritance really all that important? Considering some people consider extends to be evil, should we really be pushing to use it when it makes the job more difficult?
Google’s Protocol Buffers prefers nesting of messages rather than using an inheritance model.
Having learned ever since sixth form programming lectures that I should use inheritance all the time (they were a little extreme), it’s a big change to my normal thought processes. Overall this might be considered the better approach when you are given free-reign to refactor, or are writing from the ground up.
With nested or separately defined objects you’re free to either implement a PofSerializer for each object, or make them PortableObjects as you wish.

Changing serialised objects

The most useful parts of Shan He’s post are regarding changing serialised objects. Paraphrasing, He states there could be multiple implementation and data versions in a cluster at any point in time. (Most of this section paraphrases He, as his explanation is very good, and it would be a pretty thread-bare section if I just provided a link and said, “Here, read this.”)

For backwards compatibility a newer implementation version should provide suitable defaults for attributes missing from older data, and the wholeness of data should be kept for the benefit of older versions.
For forwards compatibility an older implementation version should keep the wholeness of the serialised data even for parts not recognised by the implementation.
Using setFutureData(reader.readRemainder()) in deserialisers and writer.writeRemainder(getFutureData()) in serialisers achieves the above by simply holding onto any future data, and handing it over where necessary.

To keep compatibility across versions no serialised data should ever be removed, instead only added to. With data access control via getters and setters. This brings with it implications as to version numbers and field indices for serialised items.

Implementation version should increase with each change to the implementation. Data versions of serialised objects should always be the higher of the current implementation version or the previous data version, whichever is the greater. The line writer.setDataVersion(Math.max(getImplVersion(), getDataVersion()) is used in serialiser methods for this very purpose.

As POF serialisation expects to write data in increasing index numbers, and read that data back in the same order, we should add fields in increasing index number order.
Since we will only ever add to serialised objects, we should never go back on index numbers or change the usage of an index number. If the type of a serialised field changes, so should its associated index number.

Testing serialisation

The majority of serialisation unit tests will revolve around making certain the right calls are made to the right methods, and they’re done the right number of times. This is where a good mocking framework should be employed to mock up PofReaders and PofWriters to your heart’s content. Add a few calls to Assert.assertEquals(expected, actual) to be absolutely certain nothing odd has occurred, and you’re golden.

When testing that your objects serialise and deserialise correctly under normal circumstances, you need to use an ExternalizableHelper and a PofContext. You register your user types to the PofContext, and use the ExternalizableHelper to serialise and deserialise your object type.

It’s much easier to show an example than to explain how to use ExternalizableHelper and PofContext in conjunction.

@Test
public void testBondSerialisationDeserialisation() {
    // Register serialisable user types required to serialise your object
    SimplePofContext pofContext = new SimplePofContext();
    pofContext.registerUserType(1001, Bond.class, new PortableObjectSerializer(1001));

    // Make sure you have whatever object you want to test serialisation on
    Bond original = new Bond(123, "EUR", "ABC", "DEF", 100.0, "GHI", "JKL", new Date(456789), 800.0, 9.0);

    // Use ExternalizableHelper.toBinary to serialise your object to Binary
    Binary bin = ExternalizableHelper.toBinary(original, pofContext);
    // Use ExternalizableHelper.fromBinary to deserialise object from Binary
    Object obj = ExternalizableHelper.fromBinary(bin, pofContext);

    // Now do your tests - some simple examples
    assertTrue(obj instanceof Bond);
    Bond copy = (Bond) obj;
    assertEquals(original, copy);
    // Good idea to do your own checks of individual members
}

The test is rather simple, and easy to add to. We don’t want to test that the PofSerializer, PofContext or ExternalizableHelper work, or that the Binary format is valid. Rather, we’re only interested in making sure our object is correctly serialised and deserialised, and the resulting object is equivalent to the original.

One thing you should probably not test is the byte array values of serialised objects. You can test using byte arrays via the ExternalizableHelper.toByteArray() and ExternalizableHelper.fromByteArray() functions, but you probably shouldn’t. It wastes your time whenever there is a change to a serialised object, having to manually edit the byte arrays to change the expected data. Only if you rely upon the byte array values should you be testing against them.

Testing serialisation code’s use of the PofReader and PofWriter is rather easy using a good mocking framework, since they’re both interfaces. You could even test the nesting, making sure your code uses the correct reader and writer for each serialised object, and the sequence in which fields are serialised or deserialised, as in this example.

@RunWith(JMock.class)
public class FutureTest {
    Mockery context = new JUnit4Mockery();
    // Setup mocks for PofWriter tests
    @Mock PofWriter writer;
    @Mock PofWriter subWriter;
    // Setup mocks for PofReader tests
    @Mock PofReader reader;
    @Mock PofReader subReader;
    // Setup sequence - only one needed
    @Auto Sequence sequence;

    // Define some test data e.g.:
    // Contract
    private final long contractNumber = 1234567890;

    // Future
    private final String contract = "ABCD";

    @Test
    public void testFutureSerialisation() {
        context.checking(new Expectations() {{
            // Invocations in Future
            oneOf (writer).setVersionId(Future.IMPL_VERSION); inSequence(sequence);
            oneOf (writer).createNestedPofWriter(Future.SUPERCLASS_INDEX); inSequence(sequence); will (returnValue(subWriter));

                // Invocations in Contract
                oneOf (subWriter).setVersionId(Contract.IMPL_VERSION); inSequence(sequence);
                oneOf (subWriter).writeLong(Contract.CONTRACT_NUMBER_INDEX, contractNumber); inSequence(sequence);
                // Write rest of serialised fields
                oneOf (subWriter).writeRemainder(aNull(Binary.class)); inSequence(sequence);

            // Continue invocations in Future
            oneOf (writer).writeString(Future.CONTRACT_INDEX, contract); inSequence(sequence);
            // Write rest of serialised fields
            oneOf (writer).writeRemainder(aNull(Binary.class)); inSequence(sequence);
        }});

        // Instantiate object to serialise using test data
        Future future = new Future(contractNumber, currency, counterparty,
                                   broker, price, contract, tenorCode, lots);
        // Serialise to check the expectations are correct
        future.writeExternal(writer);
    }

    @Test
    public void testFutureDeserialisation() {
        context.checking(new Exceptions() {{
            // Invocations in Future
            oneOf (reader).getVersionId(); inSequence(sequence); will (returnValue(Future.IMPL_VERSION));
            oneOf (reader).createNestedPofReader(Future.SUPERCLASS_INDEX); inSequence(sequence); will (returnValue(subReader));

                // Invocations in Contract
                oneOf (subReader).getVersionId(); inSequence(sequence); will (returnValue(Contract.IMPL_VERSION));
                oneOf (subReader).readLong(Contract.CONTRACT_NUMBER_INDEX); inSequence(sequence); will (returnValue(contractNumber));
                // Read rest of serialised fields
                oneOf (subReader).readRemainder(); inSequence(sequence); will (returnValue(null));

            // Continue invocations in Future
            oneOf (reader).readString(Future.CONTRACT_INDEX); inSequence(sequence); will (returnValue(contract));
            // Write rest of serialised fields
            oneOf (reader).readRemainder(); inSequence(sequence); will (returnValue(null));
        }});

        // Instantiate Future object with default constructor to deserialise
        Future future = new Future();
        // Deserialise to check the expectations are correct
        future.readExternal(reader);

        // Check data is the same as expected with asserts
        assertEquals(contractNumber, future.getContractNumber());
        assertEquals(contract, future.getContract());
        // Skipping rest of asserts
    }
}

Shan He touches on testing serialisation in his post, and suggests creating two projects to test with. One project would contain the older implementation, the other project containing the newer implementation. The two then transfer serialised objects checking they serialise and deserialise correctly between the versions in test.
When upgrading a Coherence cluster, nodes can be brought down one at a time, upgraded and brought back up.
To test the case of an old version node passing an object to a new version, the old implementation constructs an object, serialises and passes it to the new implementation. The new implementation would deserialise, re-serialise and return the object. The old implementation deserialises and checks its equality.
To test the case of a new version node passing an object to an old version, just swap the roles and have the new implementation construct the object at the start, and check the object at the end.

Conclusion

The process of making a class serialisable in Portable Object Format is not complicated, but can be tedious. There can be a large amount of duplicate code, making potential for errors higher, but if both the super- and sub-classes being serialised can evolve, that duplication can be the only way to implement a solution. Hopefully this post has helped in figuring out what’s needed for a working implementation in other systems.

Useful References


Blast from the past

Computers

Packard Bell 890391 Legend 105CD

  • Intel Pentium 90MHz
  • 8MB EDO RAM
  • 1.44MB Floppy Drive (basically compulsory)
  • 1.6GB Hard Drive (upgraded from puny 540MB)
  • Dual-speed CD-ROM
  • 15” SVGA CRT monitor
  • Microsoft Windows 95

That was the first computer my family ever bought, some time in ‘95. Aren’t the specs just amazing?
It provided me with my first taste of computing, mostly using it for games (hell, what else was there to do with them? All my school work had to be hand written) where I probably wasted hundreds of hours of my life playing Sim City, Duke Nukem and C&C: Red Alert, never once stopping to consider how any of the programs were created.

That came far later, when I went to sixth-form where the choice between Computing and IT made me realise computers could be fun, but not just because of the games you could play on them.
I was already somewhat a computer geek, having built my own computers, repaired other peoples’ and gotten into the really basic stages of learning programming.
While the main point behind IT was to learn to use word processing and spreadsheet applications, Computing taught how those applications could be put together, and what went on behind the scenes. Something I found far more interesting.

I guess that lead me to where I am today. Writing code for a living.
Funny to think how exposure to a computer may have leaned me towards it, and how a foggy memory of an ancient bit of kit could remind me of that.


Messing around with PyGame

Programming, Python, Personal, Games, PyGame

I recently came to the realisation that I had never written a graphical game. Text-driven, or simple question-and-answer types, I have written, mostly just for practice and understanding of certain ideas. When it comes to something with graphics drawn onto the screen, whether 2D sprites or 3D models, I’ve never ventured into that.
Having become so accustomed to Python, and using it so often as-of late, this made a logical choice as the language to use. It helped that I had read up on PyGame, and thought it made for an ideal starting point. I was right, too. Within an hour, I had all the dependencies checked off the list and PyGame was running with just a one-colour screen up to confirm it was working.

So far, I’ve managed to write up a few bits for an RPG, using some basic sprites. It has some animations, map-building and character movement. It has a long way to go before it’s an enjoyable game, but it’s a base to start something bigger on.
Currently working on some form of viewport that follows the player’s sprite, allowing for large maps that go off-screen. This is much less fun than the rest of the project thus far, not to mention requiring more work too, but once I’m done, I can start adding more interesting features, like transition points between maps, which should begin to make it more of a real game.

So far, PyGame has just made it really easy to put things together. It’s a great framework that just gets out of your way when you don’t need it, and provides easy access to everything when you do. This just makes it very easy to learn, both the framework and about making games.
I’m more excited about writing code for this game than for most of my previous projects, mostly because I can enjoy it afterwards, and who doesn’t like that?


Learning with the Raspberry-Pi

Programming, Operating Systems, Raspberry-Pi

Today I learned that Cambridge University has a free course, Baking Pi: Operating Systems Development. My Pi is just sitting around, doing little or nothing, and this is probably the best way to get around that. I’ve always wondered about Operating Systems and developing them, even having taken the module at University that turned out to not have much detail, and this is going to be a brilliant way to learn. Now I just need to get a working setup for my Pi, or it’ll be the same as not having one, really.

Anyone struggling for ideas, with a Pi lying about, and with an avid interest in programming and operating systems, should really check it out. It won’t be easy, but nothing interesting in computer science ever is, or should be.


Another Day, Another Idea

Programming, Facebook, Social, App Idea

I suck as a friend, because I forget birthdays all the time. I barely remember my own, let alone remembering everybody else’s, so how about a social app that uses information from Facebook (I was going to also state “and other social sites,” but there are few popular ones that have personal information) about your friends — think name, birthday and gender — to personalise one of a number of random messages, to make things seem more personal. Though it sounds like something that only horrible friends would use, it could really be a nice idea overall, depending on the options the app has.

Feature ideas:

  • Name fill-in
  • Message categories to choose from
  • Random choice of message
  • Discernment of gender for choosing suitable messages
    (no woman wants to read, “How’s it hanging?” Actually, nobody wants to read that…)
  • Ability to choose which friends to send messages to
  • Personalisation of messages (category, specific messages, nicknames, etc.) for each friend

Simple set of ideas, and an app that barely gives functionality to people. That seems to me to be exactly what people want, much of the time.


Idea for a Translation Site

Idea, Programming, Translation

Before I get started, let me state outright that this is just an idea, one that might not work as expected. My understanding of how Android handles translations is quite simple, and where I base some of my, possibly wrong, assumptions are made. Again, this is just an idea and nothing more.

So let me get started. A website that allows voting on translations to various languages for a given word or phrase. This is not the last part of the suggestion, as this would just allow choosing the best translations. The real magic would be in putting together a list of translations, for various chosen languages, into the correct format for the application. For Android, this would be multiple strings.xml files with the translations labelled with IDs (suitable IDs for each translation being used), split into various folders, labelled with the various regions and dialects that the translations are in. This would take most of the hassle out of translating an application, as long as the developer is willing to put in a little work to read strings in, rather than using literals everywhere. A little extra magic would be taking in a suitable file, for the system to figure out and automate the translation.

This idea might work, it might revolutionise the translation of applications, and it may well help lazy developers (guilty, right here) in getting their app seen and used by more people around the world. If you think this idea is useful, in any way, shape or form, run with it. I’m not sure I could do much with this idea, but my opinion is that something similar to this would be particularly useful, it just needs someone to work on it and implement it.
I hate being seen as conceited, especially for such a stupidly simplistic idea, so I leave it to someone wiser and better at developing ideas into products than I am, as I still think it’s a cool idea, with a lot of uses.

Word of the day: Petrichor
Scent of rain on dry earth.


What is wrong with the 40-hour work week?

Work

Everything I have heard about working at American companies seems to boils down to long hours, and working while at home. I can’t stand the idea of not having a work-life separation (even if I have no life to speak of, I have a right to protect it). I do over 40 hours a week, every week, sitting in a relatively uncomfortable chair with monitors too bright, air conditioning beating down on my eyes, and lights that do more burning than brightening. I’ve done longer days of 11 hours, finding that as my upper limit, where I haven’t found much of a productivity gain past the 8 hour mark. In fact, it just puts me into a bit of a mood by the time I get home, and leaves me tired. Considering I rarely sleep that well that can carry over to the next day, and I just end up losing so much time and productivity. So I just keep thinking, what is the point? What is the American obsession with working themselves silly?

For me, there is no real advantage to working stupidly long hours. I don’t get paid past the 40-hour mark as it stands, so I can’t claim that the compensation is good enough. In terms of learning, I do it more effectively when I’m happy, calm and well-rested. There are plenty of successful people in all industries that work a more reasonable amount of hours than the seemingly expected minimum of 60. After all it should come down to the quality of the work done, not quantity of the hours worked. If working long hours lowers the quality of the work, where are the advantages?

This post has mostly been a rant, but maybe it will stop me from responding to people on Twitter so questioningly. I have recently seen plenty of tweets along the lines of “If you don’t want to work more than 60 hours, work at McDonald’s,” which is an unfairly judging remark, as there are people I know working at fast-food places, hotels and bars that work 60 hours a week because they need the money, but cannot find better. Are they passionate about the job? Definitely not, but the money is a necessity.
The insinuation that working more than a certain number of hours implies passion for the work, and only “passionate” people need apply for the computing industry does not pan out. If everybody is different, then it stands to reason that everybody learns and works in different ways. This industry is fraught with personal preferences from monitor setups, programming language preferences, keyboard and mouse choices, even to where the curly braces should be placed on a line. Does it not stand to reason that everybody should make up their own mind over how many hours they work?

Again I must state, this post is ranting on. I tend to have strong views over people that issue statements in a one-size-fits-all style, especially one such as this. At times, I partially agree, but just keep remembering that people will never all fit into one box. Hell, try to fit all humans into a box, and suddenly you have the questions of gender, race and religion cropping up, because even if we are all the same species each and every one of us is different, and these differences should always be celebrated and never mocked.


Blog Test On MDEdit

Programming, Personal, Android

I’ve been working on MDEdit on-and-off, just as an excuse to keep up with some Android development and to play with Java. It has become a pretty nice, usable app, though I haven’t really given it a proper test just yet, so I’m writing this post on my Touchpad running CM9, and with the latest MDEdit installed on it. It’s not missing a beat, as I’d expect from such a simplistic app, and that makes me pretty happy!

There’s not much to add on this point, so instead, I’m going to try a little bit of markdown, and there is some more below.

Missing Features

There are many features still missing, mostly because of my experimenting rather than implementing. Let’s add a list of them below.

  • Settings activity
  • Changing editor colours
  • Changing viewer HTML
  • Portrait orientation
    Has landscape based on sensor rotation, but any other mode destroys the fragments and causes endless null values.
  • Spacing around edges of editor
    The padding hasn’t helped, and writing edge-to-edge is an annoyance
  • Changing the markdown parser
    Obviously having some issues with lists
  • Changing text size
    The text is far too small on my tablet

Possible Release

The real answer is eventually but in some part of my mind, I actually think never.
As far as markdown editors go, this is very barebones, and limited in appeal. The only real “feature” is the fact it converts markdown-formatted plain text to HTML, big whoop! Wanna fight about it?!

Wrap it up

This post was mainly an excuse to play with the app. The app itself just gets out of the way and lets me write, with the swiping to go between editor and viewer being the only break in concentration, though being better than split-screen view of editor and viewer in one.

The app has potential to at least be my favourite little editor on my tablet. It’s capable, and that’s the most important thing in reality.