IoT (Internet of Things) has recently been one of the most popular buzzwords. Despite being over-hyped, we’re indeed heading towards a foreseeable world in which all sorts of things are inter-connected. Before IoT became a hot acronym, I was heavily involved in building a Home-Area-Network SaaS platform over the course of 5 years in a previous startup I cofounded, so it’s no stranger to me.

At the low-level device network layer, there used to be platform service companies providing gateway hardware along with proprietary APIs for IoT devices running on sensor network protocols (such as ZigBee, Z-Wave). The landscape has been evolving over the past couple of years. As more and more companies begin to throw their weight behind building products in the IoT ecosystem, open standards for device connectivity emerge. One of them is MQTT (Message Queue Telemetry Transport).

Message Queue Telemetry Transport

MQTT had been relatively little-known until it was standardized at OASIS a couple of years ago. The lightweight publish-subscribe messaging protocol, MQTT, has since been increasingly adopted by major players, including Amazon, as the underlying connectivity protocols for IoT devices. It’s TCP/IP based but its variant, MQTT-SN (MQTT for Sensor Networks), covers sensor network communication protocols such as ZigBee. There are also quite a few MQTT message brokers, including HiveMQ, Mosquitto and RabbitMQ.

IoT makes a great use case for Akka actor systems which come with lightweight loosely-coupled actors in decentralized clusters with robust routing, sharding and pub-sub features, as mentioned in a previous blog post. The actor model can be rather easily structured to emulate the operations of a typical IoT network that scales in device volume. In addition, availability of MQTT clients for Akka such as Paho-Akka makes it easy to communicate with MQTT brokers.

A Scala-based IoT application

UPDATE: An expanded version of this application with individual actors representing each of the IoT devices, each of which maintains its own internal state and setting, is now available. Please see the Akka Actors IoT v.2 blog post for details.

In this blog post, I’m going to illustrate how to build a scalable distributed worker system using Akka actors to service requests from a MQTT-based IoT system. A good portion of the Akka clustering setup is derived from Lightbend’s Akka distributed workers template. Below is a diagram of the application:

As shown in the diagram, the application consists of the following components:

1. IoT

• A DeviceRequest actor which:
  • simulates work requests from IoT devices
  • publishes requests to a MQTT pub-sub topic
  • re-publishes requests upon receiving failure messages from a topic subscriber
• An IotAgent actor which:
  • subscribes to the mqtt-topic for the work requests
  • sends received work requests via ClusterClient to the master cluster
  • sends DeviceRequest actor a failure message upon receiving failure messages from Master actor
• A MQTT pub-sub client, MqttPubSub, for communicating with a MQTT broker
• A configuration helper object, MqttConfig, consisting of:
  • MQTT pub-sub topic
  • URL for the MQTT broker
  • Serialization methods to convert objects to byte arrays, and vice versa

2. Master Cluster

• A fault-tolerant decentralized cluster which:
  • manages a singleton actor instance among the cluster nodes (with a specified role)
  • delegates ClusterClientReceptionist on every node to answer external connection requests
  • provides fail-over of the singleton actor to the next-oldest node in the cluster
• A Master singleton actor which:
  • registers Workers and distributes work to available Workers
  • acknowledges work request reception with IotAgent
  • publishes work results to a work-results topic via Akka distributed pub-sub
  • maintains work states using persistence journal
• A PostProcessor actor in the master cluster which:
  • simulates post-processing of the work results
  • subscribes to the work-results topic

3. Workers

• An actor system of Workers each of which:
  • communicates via ClusterClient with the master cluster
  • registers with, pulls work from the Master actor
  • reports work status with the Master actor
  • instantiates a WorkProcessor actor to perform the actual work
• WorkProcessor actors which process the work requests

Source code is available at GitHub.

A few notes:

1. Neither IotAgent nor Worker actor system is a part of the master cluster, hence both need to communicate with the Master via ClusterClient.
2. Rather than having the Master actor spawn child Workers and push work over, the Workers are set up to register with the Master and pull work from it – a model similar to what Derek Wyatt advocated in his post.
3. Paho-Akka is used as the MQTT pub-sub client with configuration information held within the helper object, MqttConfig.
4. The helper object MqttConfig consists of MQTT pub-sub topic/broker information and methods to serialize/deserialize the Work objects which, in turn, contains Device objects. The explicit serializations are necessary since multiple JVMs will be at play if one launches the master cluster, IoT and worker actor systems on separate JVMs.
5. The test Mosquitto broker at tcp://test.mosquitto.org:1883 serves as the MQTT broker. An alternative is to install a MQTT broker (Mosquitto, HiveMQ, etc) local to the IoT network.
6. The IotAgent uses Actor’s ask method (?), instead of the fire-and-forget tell method (!), to confirm message receipt by the Master via a Future return. If the receipt confirmation is not so important, using the tell method will be a much preferred choice for performance.
7. This is primarily a proof-of-concept application of IoT using Akka actors, hence code performance optimization isn’t a priority. In addition, for production systems, a production-grade persistence journal (e.g. Redis, Cassandra) should be used and multiple-Master via sharding could be considered.

Test-running

Similar to how you would test-run Lightbend’s distributed workers template, you may open up separate command line terminals and run the different components on separate JVMs, adding and killing the launched components to observe how the systems scale out, fail over, persist work states, etc. Here’s an example of test-run sequence:

Below are some sample console output.

Console Output: Master seed node with persistence journal:

Console Output: IotAgent-DeviceRequest node:

Console Output: Worker node: