The answer to the question at the end of the previous post

What is the answer to the question at the end of the previous post:

Is using an enum for analyzer family a good way to customize the processing in the client application (CT)

Well the answer in the case of our projet was no. The stable part of the application ecosystem was the CT and the analysis cycle step in it. Adding an enum that we would then use to skip step in the processing would have made it impossible for us to add new Analyzer class without opening the code of the CT to support the new class.(remember the OCP or Open Close Principle)
What is an API designer to do in such a case ? Well, what we did was define an interface for the steps in the analysis cycle of the CT and implement a kind of Decorator for that in all SM(I) instances. Of course, concrete instances of this interface are provided by the IInstrumentDefinition factory. The main interface for this is something like:

interface IAnalysisSteps
{
void doStep (String stepName);
}

The CT supplies a default implementation of this that is in fact a supertype that looks like this:

interface IAnalysisStepHandler extends IAnalysisStep
{
void skipStep (String stepName);
}

The IInstrumentDefinition interface has a factory method that looks like:

IAnalysisStep getAnalysisStep (IAnalysisStepHandler analysisStepHandler);

The SM(I) implements a version of IAnalysisSteps that will do one of two thing when it gets called:

- Simply call the default implementation of the IAnalysisStepHandler
- Call the skipStep(String) version of the method in the IAnalysisStepHandler

Now SM(I) instances can customize the steps in the CT without actually having to open the code of the CT. What if new analysis steps are added to the CT ? Well, in that case you don't have any choice but to open the CT and to support that you simply have to implement the SM(I) IAnalysisSteps decorator such that if an analysis step is unknown it is skipped. Since this step was originally not present in the CT skipping it should be a good default behavior for old SM(I) instances. I left out some details above like having different parameters for different analysis steps. That however is easily handled using a single additional parameter of type Map<INamedParameter>. If you don't remember what a INamedParameter is simply look at the previous blog entry.

Good software design principles (part 5)

AS you have seen the two main interfaces of the SMF are:
- IInstrumentDefinition: an Abstract Factory that defines methods to get different classes to work with analyzers.
- IInstrument: a Strategy that implements all the steps needed to acquire data from an analyzer. Concrete instances of this are returned by concrete implementation of IInstrumentDefinition dynamically loaded from a .jar file.
The only class that needs to be public (exported) in the SM(I) .jar is the implementation of the IInstrumentDefinition
Now, how do you customize the behaviour of the client application (CA) to work properly with a given SM(I) ? After all, the steps in a analysis sequence of the CA might not all be appropriate for the data returned by a given SM(I).
Would having a method that defines an analyser family be a good idea ? Something like:
Family getFamily()
Where Family would be an enum. Something like:

public enum Family
{
FTIR,
PARTICLE_COUNTER,
MASS_SPECTROMETER
}

Is this a good idea ? Think about this in light of the OCP.

Good software design principle part 4

Another thing that was a big success in the SMF was the use of List and Map(s) (Dictionary in .net). If a class looks like a List, smells like a List and sounds like a list then it probably is a List. This might sound trivial but I have seen project where a class with getters and setters was defined to handle a concept that obviously was a Map. People had to constantly update the interface when new properties where added and the whole thing really was a nightmare. Using data structures such as Lists and Maps makes programming in Java feel much more like dynamic programming. So the IInstrumentDefinition includes a number of methods that actually return unmodifiable Maps. The Maps are generic Map<String, INamedParameter> where the INamedParameter interface is a little one element heterogeneous container. The INamedParameter defines the name, class and value of a parameter or variable. A Map of those is a nice little package that can easily give access to the List<String> of INamedParameter names. Once you have the name you can get the actual INamedParameter from the Map.

Good software design principle part 3

In the previous post I gave a summary of the problem to be solved using the Successful Module. I explained that in fact the problem is solved using a Successful Module Framework (SMF) and Successful Module Instances SM(i). The SMF is a library used by the client application (CA) and the SM(i) to interact with each other without building cumbersome circular references and dependencies. In Java all modules are .jar files. At compile time both the CA and the SM(i) know the SMF but they don't know each other. Of course at runtime the CA needs to know how to load the SM(i) but that information is limited to the file location. Once it is loaded the CA interacts with the SM(i) as if it consisted only of classes (more precisely interfaces) defined in the SMF. The CA loads the SM(i) .jar and looks for a class that implements an interface called IInstrumentDefinition that marks it as the point of entry into the module. In fact the CA knows only interfaces (pure abstract classes in C++) defined in the SMF. This is one of the key to the success of the whole project. No concrete classes in the API. To understand what the IInstrumentDefinition is we need to enumerate some of the interaction between the CA and an analyzer. The CA needs to:
1- Define the value of parameters used by the analyzer (resolution, number of scan, etc...)
2- Find the list of values that a particular analyzer can supply (the data it can return, etc...)
3- Start an acquisition
4- Read the status of the analyzer
5- Read the data at the end of an acquisition cycle
So. Does the IInstrumentDefinition need to supply operations (methods) related to all these interactions ? No, not at all. In fact it turns out that many of the operations listed above are defined in a separate class called an IInstrument. Things like operation 3, 4 and 5 are defined in the IInstrument. We found that in practice it was very useful to have a specific interface - the IInstrumentDefinition - to define the functions that would provide the more "static" information about an analyzer and that it made sense that this interface be the entry point into the module. We also identified another very important function of the IInstrumentDefinition: Abstract Factory.
The instrument definition is a factory for a host of classes used when interacting with a specific analyzer. Of course the most important of the factory method is the getInstrument() method that returns a IInstrument.

Good software design principles part 2

For the benefit of this blog we will call the module mentioned in the previous post the SM for successful module. Remember that the objective for this module was to make our software capable of acquiring and analysing data from any third party instrument. When thinking about a problem like this many of you might immediately think about the Open Close Principle. If you did not think about this and/or if you do not know what the OCP is, here is a definition:

software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification

Here is a link to the Wikipedia article: OCP
Trying to satisfy the OCP has a number of implication. One of them is that you should not have to modify the application code each time you want to support a new instrument. So of course the first thing that was decided was that the module should be loaded dynamically at runtime based on information in the application configuration. This is not earth shattering or revolutionary but it is a very useful concept. What we will be doing with the SM is to build a plugin infrastructure for our application instrument interface. Plugin are used in Eclipse, Netbeans and a whole menagerie of applications. In Java this is easy to do you just pack your SM in a .jar. Now the tricky par is to define the API that this jar file should implement. Another more immediate task is to define how the whole thing will be structured in terms of module. Now from the discussion we know that the starting point is this:

Now with a project like this you want to get your dependencies right from the beginning. In the case of this project it is easy to understand that has illustrated on the diagram communication between the two module goes both way. The Application supplies the SM with parameters and possibly other information and the SM returns statuses and results to the Application. We will probably need to allocate objects and possibly implement interfaces on the application side as well as in the SM. The question then is: where do we define those classes and interfaces ? The answer is: in a third module. Now the high level view of the project looks like this:

Now except for the weird looking arrow the elements on that diagram are packages with their dependencies. You have:

1) Application.
2) SM(I). This is a specific SM Instance (I added the I between parentheses to highlight that).
3) SMF the Successful Module Framework

As you can see there is no circular dependency in the standard UML elements. At compile time the APplication depends only on the SMF and the SM(I) also knows only the SMF. Now I added a non UML element (the weird broken arrow not quite connected) to represent the runtime dependencies in the system. I think having this extra arrow makes everything obvious and clean. In my next blog entry we will continue on our analysis of the SM and SMF. In fact for a while the emphasis will turn on the SMF and the key patterns used in that module the most important being:

- Abstract Factory
- Strategy

Now a closing comment. It goes without saying that before you start on a project like this a good analysis and requirements definition phase is in order. This is beyond the scope of the current thread but we might come back to this or insert a few blog entries about this phase later.

Good software design principles and weekends at the beach

I work on software that is used to read and analyse data from instruments made by the company I work for: Fourier Transform Infrared Spectrometer. The software I work on is a client/server type continuous acquisition and analysis software. I work on the server component of that software.
One key component used by the server is the data acquisition component - sometimes called the data acquisition driver. These days data acquisition module is a more appropriate name for it since this component is written in Java and does not really corresponds to what we would call a driver. In the old days, when our software was written in C/C++ the thing really was a driver. However, in recent years the link between the PC and our instrument was changed from a proprietary protocol to good old Ethernet TCP/IP link. Also, in the meantime, software development switched to Java. Of course, all of this is very nice since Java has very good Network communication libraries.
Now, our software is very flexible and has very good data processing capabilities so why not use it to analyse data from other instruments ? Well that's exactly what I was asked to do a few years ago: make our software capable of acquiring and analysing data from any third party instrument. Now, while this may sound like a simple task, doing it right really is not that simple.
In my next few blog entries I'm going to talk about how this problem was solved. Now of course, I'm not going to discuss this at a level of details that could get me in trouble with my employer but this is not a problem since the key to the success of the project are not in the kind of implementation details that could be considered trade secret. No actually the key to the success of this project was using well known and documented good software design principles. I think you will enjoy this.

The Builder and Null Object pattern meet the Java Proxy class

I love the Null Object pattern. It is a simple and useful pattern in many situations. The archetypal example is in the context of JUnit tests (or whatever is your favourite testing framework). In unit tests, Mock objects are often little more than Null Objects. Obviously, there are very good libraries for mocking objects in unit tests and I guess I could use this for Null objects in regular code but that would bring a lot of dead code into my application since those libraries provide some pretty fancy features. I need something simpler.
I have used the Java dynamic java.lang.reflect.Proxy class for things like debugging proxy and other applications so I though using this class for Null Object would be great (if you need a refresher on Java Dynamic Proxy check Explore the Dynamic Proxy API). In fact since a Null object has only one immutable state you only have to worry about the return value of its method. I want to keep this simple so what I will do is I will use two Map(s) in my proxy handler: one with return types for keys (Class) and one with method names for keys (String). This will allow me to define return value for different return types and return values for different method names. This implementation does not support having different return value for methods with the same name and different parameters (overloaded methods). This should not be a big limitation since a Null object is likely to return the same value in this case. Here are the declarations for my Map(s):

The returnTypeLookup must include values for primitive types since methods that return those cannot default to null. As a bonus the Map includes a consistent set of values for primitive wrapper classes.
The code below shows how the Map(s) are used in the invoke method of my handler:

As you can see we first check if there is a custom return value for the invoked method name and then we check if there is a custom value for the return type. This makes it possible to handle most common cases where you want to override the default.
Now how can we customize our Proxy ? We could use different constructors for our invocation handler or some extra methods to override values. I have considered several options and finally I think I have found a very elegant solution. Here is what creating a Null object using my API looks like:

ISomeInterface r = 
	NullObjectProxy.newInstance (ISomeInterface.class)
		.whenReturnType (boolean.class, true)
		.whenMethodName ("someMethodName", new SomeUserDefinedType())
		.build ();

To achieve this I use a generic Factory method and a Builder class.In my next post I will show you how to define a class that can be used with such a flexible and programmer friendly syntax.

Unit tests and the Layers Pattern (part 4)

Last time we looked at the layers in OPC UA module into a little bit more details. Now lets look at how the tests were structured. I was not please at how the diagram represented the organisation of the tests so here is a modified version:

========================================================
 Interface (Java) :             (Unit tests)(Unit tests)
======================================|===========|=====
 Logical (Java)   :    Unit tests     |           |
======================================|===========|=====
 Low-level access :                   |           |
......................................|...........|.....
            Java  :                   |           |
......................................|...........|.....
   JNI (ANSI C++) :                   V           |
..................................................|.....
          C++/CLI :                               |  
..................................................V.....
             C#   :    Unit tests  
========================================================

The unit tests are divided into the following categories:

Horizontal (single layer)

Test of classes in the logical layer

Most are classical Junit tests. Mostly, they test class methods in an isolated manner. I do have “behavior driven” tests here that use a mock implementation of the lower layers. Because of the layered approach, the Mock implementation is quite simple. It uses a Map in the background with backdoor methods to setup parameter values. The methods that fake UA method calls don't do anything except changed predefined parameter values.

Test of classes in the low-level layer

Same thing here except that I use Nunit since this is written in C#. The difference is that I don't have lower levels in my code. The next layer is external and that is the OPC UA framework. I was able to expand my tests here using generics and conditional compilation. I had to do this because the OPC UA framework does not use a lot of interface or abstraction. I ended up having to work very hard to test some part of the code. A lot of the tests here are for the special Queue used for subscriptions.

Vertical (multiple layers)

Tests of the JNI interface

Here I use a separate DLL that does not use the C++/CLI layer. This allows me to test the JNI part of the code in isolation so that if I have a bug I know that the problem is in the pure native C++ layer. I could have used horizontal tests here but they would have been very limited since most of the code is mostly JNI mechanics.

Tests of all upper layers

These tests go from the logical down to the low-level layer. The low-level layer however is Mocked so this group of tests is mainly a test of the C++/CLI mechanic. Of course there is a small amount of redundant tests of the JNI code here. This is unavoidable. However, because the JNI code is tested in isolation elsewhere this is not a problem. I know that if I have a bug here there is a high probability that the bug is in the C++/CLI mechanic.

Conclusion

Structuring the code in layers allows to more easily test more code. Having different group of tests allows to quickly find the source of a bug. You avoid much debugging using this modular approach. You also can tests more stuff as part of the build because you can use Mock implementations of key components and avoid having to use an actual OPC UA server on the build machine.
The code is tested with an actual OPC UA server as part of manual tests. These are JUnit tests that I run manually on my development machine and that use all real layers. Finally, system and integration tests close the loop.

Unit tests and the Layers Pattern (part 3)

Last time we looked at the layers for my OPC UA client project without going into too much details. For convenience I have repeated the diagram below.

============================================
 Interface (Java) :             (Unit tests)
======================================|=====
 Logical (Java)   :              Unit tests
======================================|=====
 Low-level access :                   |
......................................|.....
            Java  :                   |
......................................|.....
   JNI (ANSI C++) :                   V
......................................|.....
          C++/CLI :                   |
......................................V.....
             C#   :              Unit tests     
============================================

Lets describe the layers into a little bit more details:

Interface

As described in part 1 this is were you define the public API for the module. In my Java code this is made up mostly of Java interfaces. In C++ I would use pure abstract classes. The interface also defines things like Enum and constants that are part of the interface. In my project this is in a separate group of package (namespace) and one could go as far as putting this in a totally separate projects. Putting the interface in a separate project helps make he separation between the interface and the rest of the code even more explicit and this helps to avoid some type of errors were the interface is contaminated with implementation elements from other layers. In my case I kept all the Java code in the same project and it went fairly well.

Logical

The logical layer is the part that uses the low-level access layer to implement actual business logic. Things like:

if (parameterX.value == aSpecificValue)
 {
 // Do something 
 }
else
 {
 // Do something else call a  UA  MethodY()
 }

In this layer I actually have a state machine that switches state and takes different action based on parameter values. The logical layer uses other sublayers (configuration persistence, ...) but we won't go into those details here because it would make things too complicated. This layer contains a good number of unit tests (horizontal).

Low-level access

This layer defines an interface of its own. In my case this interface is not visible from outside the module. It defines the following method:

• read one or more parameters


• write one or more parameters


• call UA methods


• subscribe for update notification for one or more parameters


• fetch data updated through the subscription mechanism

The only code in this layer is the code necessary to use the UA framework to perform the tasks listed above. And in fact the only part that contains more complicated logic is the part that manages the subscription and this is mainly a kind of smart queue mechanism. This code is the part responsible for most of the unit tests located directly in the layer (horizontal).

Parting comments

I want to emphasize that except for the sublayers in the low-level access layer the layers have nothing to do with the use of different languages. The same layers would have been present with an all Java module. In other words if a Java OPC UA framework had been available in a sufficiently advanced state for my project the layers would have been the same.
Next time we will keep exploring the layers and how the unit tests were structured.

Unit tests and the Layers Pattern (part 1)

In previous postings I talked about techniques that I use when adding unit tests to legacy code. The two techniques were Introduce middle man and Extract static method. I admit that those techniques may not sound very impressive at first. However, they did help me to add unit tests to legacy code on a number of occasions where it would have been difficult otherwise. Adding tests to code late in the development (or in a later version of the software) is quite a challenge. Ideally you want to add unit tests early in the development. In fact you want to design your code for testing.
I should point out that on my blog when I talk about unit tests I mean automated unit tests that can run as part of the build process on any machine. When I talk about the other types of unit tests I will use an expression like manual unit tests on some other more precise expression depending on the exact type of tests..
Today I will talk about one of the most powerful design technique I have found for writing code to be unit tested. The technique is organized around an architectural Pattern calledLayers.
Anyone familiar with networking theory knows about the OSI layers. The OSI model is one good example of this pattern even though sometimes the actual layers implemented in the real world do not exactly match the theory. If you are not familiar with the OSI model it's not a problem because the actual example that I am going to present is not very complicated and should allow you to understand the pattern. The Layers pattern is presented in the excellant book A system of pattern by Bushmann, Meunier, Rohnert, Sommerlad and Stal:

The Layers pattern helps to structure applications that can be decomposed into groups of subtasks in which each group of subtasks is at a particular level of abstraction.

The Pattern is particularly useful when building modules that access hardware (device drivers) or that communicate with server or other external entity. In the case of a device driver for example you could split the module into three layers (I show the abbreviations that I will use later for those layers in parentheses):


========================
Interface (IL)
========================
Logical (LL)
========================
Low-level access (LLAL)
========================


The responsibilities are divided as:

1. Interface Layer (IL): defines the methods and classes used to interact with the module from a client application.


2. Logical Layer (LL): implements the functions defined in the interface layer using the data supplied by the low-level layer. Also uses the low-level layer to request additional information when needed. It is the layer where most of the complicated business logic is located.


3. Low-level access Layer (LLAL): handles the low-level details. This can be communicating with the hardware, communication with the server or other low-level operations.

There are other benefits to using the layers Pattern but here we will focus on how it can benefit unit tests.

• Often the problem with unit testing client/server or code that interfaces with hardware is that it is difficult to have the setup on all machine that needs to run the automated unit tests. Layers can help mitigate this by separating the part that interfaces with the hardware or server from the rest of the code.


• Also sometimes it is difficult to test specific scenario when using the actual hardware or when interacting with a server.

In both cases, the Layers pattern can help by making the building and use of mock implementation easier. Having a simple well defined low-level layer is a key ingredient here.
Before we dive into an example lets consider some of the pitfalls that you might encounter when using Layers:


Communication overhead between layers

You have to watch out for this and if needed make compromise in order to get acceptable communication costs. You may need to transfer some responsibilities between layers if things degenerate.


Coupling between layers

You want to avoid static circular dependencies. In a clean design the dependencies will look something like:

|IL| /__ |LL| __\ |LLAL|

The dependencies are from LL to the other layers. Again, you may need to compromise and/or add packages (namespaces) to break circular dependencies. Sometimes simply adding an interface to a layer will do the trick (invert the dependency). In practice you will often end-up with something like:

|package-1||package-2|
  /      \  /     \ 
|IL| /__ |LL| __\ |LLAL|

The extra dependencies here are all towards package-1 or package-2. Of course you can have additional packages (or namespaces) inside each layers.

Next time we will look at a nice example.