In one of my previous articles I have described the idea of asynchronous operations. Apart from many advantages asynchronous operations do have some  disadvantages as well – they are complicated. It’s relatively complicated to implement a big, multiplatform application. Of course, it is possible to wrap native functions from your operating system with classes that will highly simplify the use of this complicated approach and they also can hide the inter-platform differences. Correct encapsulation and abstraction can result in a much more user-friendly interface.

Basic object oriented framework

Let’s start by thinking what would the core of such wrappers look like. We definitely need to wrap select() or WaitForMultipleObjects() (depending on the platform). This class should contain a collection of IDs of the previously created IO streams. On incoming data it should automatically perform a callback and then return to waiting for other events. Let’s call this class Reactor because it reacts to events reported by the operating system.

Certainly the Reactor class doesn’t “know” what to do with the data received. The decision belongs to business logic that should be placed outside of the wrapper. As I have already mentioned, we need a callback. As there are multiple signals that can be received from a stream (data, error, notification about closed socket etc.), it’s best to create a generic interface, or rather (because C++ does not support the interface concept directly) an abstract class with several purely virtual methods. Implementations of this abstract class can be then registered as callbacks for handling IO events. Let’s call this abstract class EventHandler because it handles the events.

Let’s try to write a simple piece of code that shows what we are aiming at:

class EventHandler
{
friend class Reactor;
private:
virtual void handleInput()=0;
virtual void handleOutput()=0;

virtual void handleClose() =0;
virtual void handleError(int err) =0;
};

The Reactor class cooperates with the Event Handler class. It needs to contain a collection of registered pairs: an ID of a stream and a callback for handling events. It also needs the methods for adding, removing and manipulating callbacks and handlers. And – of course – for starting and stopping the asynchronous loop. An exemplary Reactor interface is shown below. Handle represents a OS-specific ID of a stream – int on POSIX platforms and HANDLE on Windows (such types are usually defined as typedefs in a platform specific C++ headers):

//brief overview of the Reactor class
class Reactor
{
void registerHandler (Handle handle, EventHandler * eventHandler); //adds new pair
void removeHandler(Handle handle); // removes pair.
void startEventLoop(); //blocks the tread and waits for asynchronous events
void stopEventLoop(); //ends the loop and unblocks the thread
};

Now we can create, for example, a class that would handle data coming from a socket. It should inherit the EventHandler class and provide implementation of its purely virtual methods. Once we have this class, we can create several instances and open several sockets. Then, pairs consisting of a socket and a handler can be registered in Reactor to handle several connections in one thread.

Usually you create only one Reactor instance per thread. There can be several threads, each running its internal event loop using Reactor. A common pattern is, therefore, to use thread specific storage. Instead of passing a Reactor reference  to each object using it, it is much simpler to create a function called, let’s say, Reactor & giveMeThisThreadReactor().

More advanced functionality

We already have everything we need to handle IO operations correctly. But it turns out that there’s more that can be achieved with Reactor. Let’s look at cross thread notifications, timers and OS signals.

Cross-thread notification system

Let’s imagine that you have several threads in your application. Each of those threads has their Reactor and each thread maintains several sockets, files and other asynchronous IO sources. Let’s say that one of your threads needs to send a message to another thread. How to do that? Of course the solution is to use synchronized message queues. It’s a very elegant solution commonly used in the majority of mature systems. The problem though is that the thread that handles IO operations gets blocked and won’t check the queue until some IO data arrives. To avoid that we not only need to have a queue but also a way to activate the receiving thread.

Timers

Another feature commonly used by applications are timers. For example when you perform HTTP get request you always want to make sure that your application won’t hang forever if the other party stops responding. To achieve that you have to set a time limit and simply end the request after the given time. Since the receiving thread is blocked on IO operations, it can’t check the time. It’s a very similar situation to the one with a synchronized queue – your thread is blocked, not responding and waiting for some IO data. Fortunately both select() and WaitForMultipleObjects() functions allow you to pass a timeout value. If you wrap this functionality correctly you can have a solid timer framework with a user capable of adding and canceling asynchronous timers.

Signals

Signals are very short messages that are passed from your operating system to the application that you are running. For example, if you press Ctrl + C your operating system sends SIGINT to your application. If the application does not provide any handler for this signal, operating system will kill it. If it does, it handles the signal in the way implemented by the programmer. Of course, signals are asynchronous in their nature, so they look like something that we would like to have in our Reactor-EventHandler framework.

Complexity problem

After this analysis, you surely have noticed that our framework becomes a little complex. It gets even more complex if you want to create a multiplatform solution because then you need to hide all the platform specific details inside the Reactor, write duplicated, platform-specific code, use a lot of C++ #ifdef-s and test your code on all platforms. This framework grows even further when you notice that signals, asynchronous notifications and timers are implemented in a different way in each operating system and your code will significantly differ for each platform. It would probably take months to implement it on your own.Fortunately there are ready-to-use libraries where all that work has already been done.

Libraries for handling asynchronous IO

Asio from Boost

There are actually two main libraries that are commonly used in C++ world to handle asynchronous IO operations. The first one is asio that comes from boost (http://bost.org). Boost is the most solid standard of system programming. Unfortunately this library does not support signals nor cross-thread notifications. You could probably implement it on your own but what’s the point of using a library then?

Adaptive Communication Environment

A very solid solution Adaptive Communication Environment(http://www.cs.wustl.edu/~schmidt/ACE.html). Contrary to boost, it provides all mentioned functionality. Its API consists of ACE_Reactor and ACE_Event_Handler classes. As you can guess they almost precisely match what I have described earlier as Reactor and EventHandler. Additionally, ACE allows you to use handles of your operating system directly, it has its own wrappers around sockets, files servers etc. that make use of IO much simpler and it is suitable for real time solutions.

A Real life example

While working for Wind Mobile company I had an opportunity to maintain the NaviVoice application. NaviVoice is a layer of a phone card management stack and it is purely asynchronous because of the speed requirements. Since it was supposed to be migrated from Windows to Linux (but it was using direct operating system calls), ACE and the Reactor framework was a simply perfect solution to do that. Not only did this framework prove to be useful but, basically, it saved me months of slow implementation and writing multiplatform code. Huge advantage of using an external library is that you don’t have to write more code. In fact you need to delete much of your own code and therefore make your product much easier to maintain. You can also limit the number of tests because such libraries tend to be heavily covered with tests and already used by thousands of companies around the world. You can skip part of the documentation too – ACE is, of course, very well documented.

The final result was an application that had less code, could work on several platforms and was much easier to maintain. Simply – an almost perfect success story.