Tag Archives: akka actors

An Akka Actor-based Blockchain

As proposed at the beginning of this blockchain mini blog series, we’ll have an Actor-based blockchain application at the end of the series. The fully functional application is written in Scala along with the Akka toolkit.

While this is part of a blog series, this post could still be viewed as an independent one that illustrates the functional flow of a blockchain application implemented in Scala/Akka.

What is a blockchain?

Summarizing a few key characteristics of a blockchain (primarily from the angle of a cryptocurrency system):

  • At the core of a cryptocurrency system is a distributed ledger with a collection of transactions stored in individual “blocks” each of which is successively chained to another, thus the term “blockchain”.
  • There is no centralized database storing the ledger as the authoritative data source. Instead, each of the decentralized “nodes” maintains its own copy of the blockchain that gets updated in a consensual fashion.
  • At the heart of the so-called “mining” process lies a “consensus” algorithm that determines how participants can earn the mining reward as an incentive for them to collaboratively grow the blockchain.
  • One of the most popular consensus algorithms is Proof of Work (PoW), which is a computationally demanding task for the “miners” to compete for a reward (i.e. a certain amount of digital coins) offered by the system upon successfully adding a new block to the existing blockchain.
  • In a cryptocurrency system like Bitcoin, the blockchain that has the highest PoW value (generally measured by the `length`, or technically referred to as `height`) of the blockchain overrides the rest.

Beyond cryptocurrency

While blockchain is commonly associated with cryptocurrency, the term has been generalized to become a computing class (namely blockchain computing) covering a wide range of use cases, such as supply chain management, asset tokenization. For instance, Ethereum, a prominent cryptocurrency, is also a increasingly popular computing platform for building blockchain-based decentralized applications. Its codebase is primarily in Golang and C++.

Within the Ethereum ecosystem, Truffle (a development environment for decentralized applications) and Solidity (a JavaScript alike scripting language for developing “smart contracts”), among others, have prospered and attracted many programmers from different industry sectors to develop decentralized applications on the platform.

In the Scala world, there is a blockchain framework, Scorex 2.0 that allows one to build blockchain applications not limited to cryptocurrency systems. Supporting multiple kinds of consensus algorithms, it offers a versatile framework for developing custom blockchain applications. Its predecessor, Scorex, is what powers the Waves blockchain. As of this post, the framework is still largely in experimental stage though.

How Akka Actors fit into running a blockchain system

A predominant implementation of the Actor model, Akka Actors offer a comprehensive API for building scalable distributed systems such as Internet-of-Things (IoT) systems. It comes as no surprise the toolset also works great for what a blockchain application requires.

Lightweight and loosely-coupled by design, actors can serve as an efficient construct to model the behaviors of the blockchain mining activities that involve autonomous block creations in parallel using a consensus algorithm on multiple nodes. In addition, the non-blocking interactions among actors via message passing (i.e. the fire-and-forget `tell` or query-alike `ask` methods) allow individual modules to effectively interact with custom logic flow or share states with each other. The versatile interaction functionality makes actors useful for building various kinds of modules from highly interactive routines such as simulation of transaction bookkeeping to request/response queries like blockchain validation.

On distributed cluster functionality, Akka provides a suite of cluster features — cluster-wide routing, distributed data replication, cluster sharding, distributed publish/subscribe, etc. There are different approaches to maintaining the decentralized blockchains on individual cluster nodes. For multiple independent mining processes to consensually share with each others their latest blockchains in a decentralized fashion, Akka’s distributed pub/sub proves to be a superb tool.

A blockchain application that mimics a simplified cryptocurrency system

UPDATE: A new version of this application that uses Akka Typed actors (as opposed to the Akka classic actors) is available. An overview of the new application is at this blog post. Also available is a mini blog series that describes the basic how-to’s for migrating from Akka classic to Akka Typed.

It should be noted that Akka Actors has started moving towards Typed Actors since release 2.5, although both classic and typed actors are being supported in the current 2.6 release. While the Akka Typed API which enforces type-safe code is now a stable release, it’s still relatively new and the API change is rather drastic, requiring experimental effort to ensure everything does what it advertises. Partly because of that, Akka classic actors are used in this blockchain application. Nonetheless, the code should run fine on both Akka 2.5 and 2.6.

Build tool for the application is the good old sbt, with the library dependencies specified in `built.sbt` and all the configurative values of the Akka cluster and blockchain specifics such as the proof-of-work difficulty level, mining reward, time-out settings in `application.conf`:

akka {

  actor {
    provider = "cluster"

    default-dispatcher {
      throughput = 10
    }

    allow-java-serialization = on
    warn-about-java-serializer-usage = off
  }

  remote.artery {
    enabled = on
    transport = tcp
    canonical {
      hostname = "127.0.0.1"
      port = 2551
    }
  }

  cluster {
    seed-nodes = [
      "akka://blockchain@127.0.0.1:2551",
      "akka://blockchain@127.0.0.1:2552"
    ]
  }
}

blockchain {
  proof-of-work {
    difficulty = 3
    nonce = 0
    reward = 99
    network-name = "akka-blockchain"
    network-account-key = "MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEApKyCqvNjx+YfVYFippxviV0UgEgpYJbV0luEJTkcbnvIIolqXR5JJACbk8TDW72F9pnwKqQhM0vzpsbat+kubl5FeO0CPk/Gk2/aAcoP11DNqYAgqvCKEyZlJFmwnC/6JecCBdBb+G/+v894LESD8y4upjZIRdrrC4UQowlm4+6mPJTbB0U+2q7rwv8FXmR9KHmRxtGoMh885bKTXwqOO0pqh0/MMSaW5pzS6s5bUiX7ekl0EPwTyiXbuyUZ3nZcpm/v7lCvXWlyRju8g2pg9e0BShyfI7w1qqw1xEeRVwU1qIlN4Q1bz82t1O+7l308VtiXH2lM9Vhn+Sfz68cjtmJ4uSwZKeV91kaOJdr1OafM9ryfPv2MGDVvSGUk+TEMORysFcRS59VsON8styFJ6hGx1/GKyfUilCS2G2dS/9NvTJjV64nPTQ1mufT4Kd8tBa3/DTmjrqsFjfzYsc/kVz0QsGN7PxgKPf5+aSzrG2y9GQy69jAb6wpMoLcux2zlZvobYR+qjoBIsn+5Y5J0Fs9Jvwh0yd0KGdaDkMZnPGYTQo1vpC4gqowt7q4lba0FRV3jt/bm1B8Pu+YWk4MasNk1wfBMLNOr1CGaiSXFIl0NyATViOOyq6FEnHdF8L1MmLM9v7Yj3l4q5ykF0u/JnDXErC7CwtECRVr6gX3LWfcCAwEAAQ=="
  }

  timeout {
    mining = 20000
    block-validation = 1000
  }

  simulator {
    transaction-feed-interval = 15000
    mining-average-interval = 20000
  }
}

Note that Artery TCP remoting, as opposed to the classic Netty-base remoting, is used.

With the default configuration, the application will launch an Akka cluster on a single host with two seed nodes at port 2551 and 2552 for additional nodes to join the cluster. Each user can participate the network with their cryptographic public key (for collecting mining reward) provided as an argument for the main program on one of the cluster nodes to perform simulated mining tasks.

For illustration purpose, the main program will either by default enter a periodic mining loop with configurable timeout, or run a ~1 minute quick test by adding “test” to the program’s argument list.

Functional flow of the blockchain application

Rather than stepping through the application logic in text, the following diagram illustrates the functional flow of the Akka actor-based blockchain application:

Akka Blockchain - functional flow

Below is a summary of the key roles played by the various actors in the application:

Blockchainer – A top-level actor that maintains a distributed copy of the blockchain and transaction queue for a given network participant (e.g. a miner) identified by their cryptographic public key. It collects submitted transactions in the queue and updates the blockchain according to the consensual rules via the cluster-wide distributed pub/sub. Playing a managerial role, the actor delegates mining work to actor Miner and validation of mined blocks to actor BlockInspector.

Miner – A child actor of Blockchainer responsible for processing mining tasks, carrying out computationally demanding Proof of Work using a non-blocking routine and returning the proofs back to the parent actor via the Akka `ask` pattern.

BlockInspector – Another child actor of Blockchainer for validating content of a given block, typically a newly mined block. The validation verifies that generated `proof` and goes “vertically” down to the nested data structure (transactions/transactionItems, merkleRoot, etc) within a block as well as “horizontally” across all the preceding blocks. The result is then returned to the parent actor via Akka `ask`.

Simulator – A top-level actor that simulates mining requests and transaction submissions sent to actor Blockchainer. It spawns periodic mining requests by successively calling Akka scheduler function `scheduleOnce` with randomized variants of configurable time intervals. Transaction submissions are delegated to actor TransactionFeeder.

TransactionFeeder – A child actor of actor Simulator responsible for periodically submitting transactions to actor Blockchainer via an Akka scheduler. Transactions are created with random user accounts and transaction amounts. Since accounts are represented by their cryptographic public keys, a number of PKCS#8 PEM keypair files under “{project-root}/src/main/resources/keys/” were created in advance to save initial setup time.

As for the underlying data structures including `Account`, `Transactions`, `MerkleTree`, `Block` and `ProofOfWork`, it’s rather trivial to sort out their inter-relationship by skimming through the relevant classes/companion objects in the source code. For details at the code level of 1) how they constitute the “backbone” of the blockchain, and 2) how Proof of Work is carried out in the mining process, please refer to the previous couple of posts of this mini series.

Complete source code of the blockchain application is available at GitHub.

Test running the blockchain application

Below is sample console output with edited annotations from an Akka cluster of two nodes, each running the blockchain application with the default configuration on its own JVM.

###
### Annotated Console Log: NODE #1
###

sbt:akka-blockchain> runMain akkablockchain.Main 2551 src/main/resources/keys/account0_public.pem
runMain akkablockchain.Main 2551 src/main/resources/keys/account0_public.pem
[info] Running akkablockchain.Main 2551 src/main/resources/keys/account0_public.pem
<<<
<<<--- Running on port# 2551 with minerKeyFile `account0_public.pem`
<<<
[INFO] [04/16/2020 18:08:40.349] [run-main-0] [ArteryTcpTransport(akka://blockchain)] Remoting started with transport [Artery tcp]; listening on address [akka://blockchain@127.0.0.1:2551] with UID [7294460728338242243]
[INFO] [04/16/2020 18:08:40.378] [run-main-0] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Starting up, Akka version [2.6.4] ...
[INFO] [04/16/2020 18:08:40.466] [run-main-0] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Registered cluster JMX MBean [akka:type=Cluster]
[INFO] [04/16/2020 18:08:40.467] [run-main-0] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Started up successfully
[INFO] [04/16/2020 18:08:40.570] [blockchain-akka.actor.internal-dispatcher-3] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing
[INFO] [04/16/2020 18:08:41.056] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Ack] akkablockchain.actor.Blockchainer@e1b6066: Subscribing to 'new-transactions' ...
[INFO] [04/16/2020 18:08:41.056] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Ack] akkablockchain.actor.Blockchainer@e1b6066: Subscribing to 'new-block' ...
[WARN] [04/16/2020 18:08:41.192] [blockchain-akka.remote.default-remote-dispatcher-8] [akka.stream.Log(akka://blockchain/system/Materializers/StreamSupervisor-1)] [outbound connection to [akka://blockchain@127.0.0.1:2552], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1:2552,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
[WARN] [04/16/2020 18:08:41.192] [blockchain-akka.remote.default-remote-dispatcher-9] [akka.stream.Log(akka://blockchain/system/Materializers/StreamSupervisor-1)] [outbound connection to [akka://blockchain@127.0.0.1:2552], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1:2552,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
[INFO] [04/16/2020 18:08:42.023] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(1cab, 3000/2, 2020-04-17 01:08:42) is published.
[INFO] [04/16/2020 18:08:42.027] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(1cab, 3000/2, 2020-04-17 01:08:42) to transaction queue.
[INFO] [akkaMemberChanged][04/16/2020 18:08:45.722] [blockchain-akka.actor.internal-dispatcher-3] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Node [akka://blockchain@127.0.0.1:2551] is JOINING itself (with roles [dc-default]) and forming new cluster
[INFO] [04/16/2020 18:08:45.725] [blockchain-akka.actor.internal-dispatcher-3] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - is the new leader among reachable nodes (more leaders may exist)
[INFO] [akkaMemberChanged][04/16/2020 18:08:45.732] [blockchain-akka.actor.internal-dispatcher-3] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Leader is moving node [akka://blockchain@127.0.0.1:2551] to [Up]
<<<
<<<--- Cluster seed node #1, bound to port# 2551, is up
<<<
[info] new client connected: network-1
[INFO] [04/16/2020 18:08:57.007] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(4700, 4000/2, 2020-04-17 01:08:57) is published.
[INFO] [04/16/2020 18:08:57.009] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(4700, 4000/2, 2020-04-17 01:08:57) to transaction queue.
[WARN] [04/16/2020 18:09:00.800] [blockchain-akka.remote.default-remote-dispatcher-9] [Association(akka://blockchain)] Outbound control stream to [akka://blockchain@127.0.0.1:2552] failed. Restarting it. akka.remote.artery.OutboundHandshake$HandshakeTimeoutException: Handshake with [akka://blockchain@127.0.0.1:2552] did not complete within 20000 ms
[INFO] [04/16/2020 18:09:10.980] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 22000 millis
[INFO] [04/16/2020 18:09:10.980] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:09:10.980] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(1cab, 3000/2, 2020-04-17 01:08:42), T(4700, 4000/2, 2020-04-17 01:08:57))
[INFO] [04/16/2020 18:09:10.989] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:09:11.998] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(dea2, 1500/1, 2020-04-17 01:09:11) is published.
[INFO] [04/16/2020 18:09:11.999] [blockchain-akka.actor.default-dispatcher-14] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(dea2, 1500/1, 2020-04-17 01:09:11) to transaction queue.
[INFO] [04/16/2020 18:09:24.812] [blockchain-akka.actor.internal-dispatcher-2] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Received InitJoin message from [Actor[akka://blockchain@127.0.0.1:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#-281590563]] to [akka://blockchain@127.0.0.1:2551]
[INFO] [04/16/2020 18:09:24.813] [blockchain-akka.actor.internal-dispatcher-2] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Sending InitJoinAck message from node [akka://blockchain@127.0.0.1:2551] to [Actor[akka://blockchain@127.0.0.1:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#-281590563]] (version [2.6.4])
[INFO] [akkaMemberChanged][04/16/2020 18:09:24.912] [blockchain-akka.actor.internal-dispatcher-6] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Node [akka://blockchain@127.0.0.1:2552] is JOINING, roles [dc-default]
[INFO] [akkaMemberChanged][04/16/2020 18:09:25.430] [blockchain-akka.actor.internal-dispatcher-6] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Leader is moving node [akka://blockchain@127.0.0.1:2552] to [Up]
<<<
<<<--- Allowed node #2, bound to port# 2552, to join the cluster
<<<
[INFO] [04/16/2020 18:09:26.615] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(20b3, 5000/2, 2020-04-17 01:09:26) to transaction queue.
[INFO] [04/16/2020 18:09:27.003] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(1270, 6500/3, 2020-04-17 01:09:27) is published.
[INFO] [04/16/2020 18:09:27.004] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(1270, 6500/3, 2020-04-17 01:09:27) to transaction queue.
[INFO] [04/16/2020 18:09:32.996] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 17000 millis
[INFO] [04/16/2020 18:09:32.996] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:09:33.004] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(4700, 4000/2, 2020-04-17 01:08:57), T(dea2, 1500/1, 2020-04-17 01:09:11), T(20b3, 5000/2, 2020-04-17 01:09:26), T(1270, 6500/3, 2020-04-17 01:09:27))
[INFO] [04/16/2020 18:09:33.005] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:09:38.763] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:09:38.767] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771) is valid. Updating blockchain.
<<<
<<<--- Adding a locally mined block to local `blockchain`
<<<
[INFO] [04/16/2020 18:09:38.768] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:09:41.579] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(dd7d, 7000/3, 2020-04-17 01:09:41) to transaction queue.
[INFO] [04/16/2020 18:09:42.004] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(af31, 5000/2, 2020-04-17 01:09:42) is published.
[INFO] [04/16/2020 18:09:42.004] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(af31, 5000/2, 2020-04-17 01:09:42) to transaction queue.
[INFO] [04/16/2020 18:09:50.018] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 20000 millis
[INFO] [04/16/2020 18:09:50.018] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:09:50.020] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(dea2, 1500/1, 2020-04-17 01:09:11), T(20b3, 5000/2, 2020-04-17 01:09:26), T(1270, 6500/3, 2020-04-17 01:09:27), T(dd7d, 7000/3, 2020-04-17 01:09:41), T(af31, 5000/2, 2020-04-17 01:09:42))
[INFO] [04/16/2020 18:09:50.020] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:09:56.582] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(c28e, 6500/3, 2020-04-17 01:09:56) to transaction queue.
[INFO] [04/16/2020 18:09:56.996] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(6a16, 2000/1, 2020-04-17 01:09:56) is published.
[INFO] [04/16/2020 18:09:56.996] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(6a16, 2000/1, 2020-04-17 01:09:56) to transaction queue.
[ERROR] [04/16/2020 18:10:08.044] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.Mining] akkablockchain.actor.Blockchainer@e1b6066: ERROR: java.util.concurrent.TimeoutException: BLK(Hg/S, T(4700, 4099/3), 2020-04-17 01:09:50, 3, 0): 18000 milliseconds
<<<
<<<--- Mining of a block timed out; the assembled block will be discarded
<<<
[INFO] [04/16/2020 18:10:08.044] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:10:10.036] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 22000 millis
[INFO] [04/16/2020 18:10:10.036] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:10:10.036] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(20b3, 5000/2, 2020-04-17 01:09:26), T(1270, 6500/3, 2020-04-17 01:09:27), T(dd7d, 7000/3, 2020-04-17 01:09:41), T(af31, 5000/2, 2020-04-17 01:09:42), T(c28e, 6500/3, 2020-04-17 01:09:56), T(6a16, 2000/1, 2020-04-17 01:09:56))
[INFO] [04/16/2020 18:10:10.036] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:10:11.573] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(a784, 2500/2, 2020-04-17 01:10:11) to transaction queue.
[INFO] [04/16/2020 18:10:11.998] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(64bb, 3000/2, 2020-04-17 01:10:11) is published.
[INFO] [04/16/2020 18:10:11.998] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(64bb, 3000/2, 2020-04-17 01:10:11) to transaction queue.
[INFO] [04/16/2020 18:10:17.526] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662) is valid. Updating blockchain.
<<<
<<<--- Adding a mined block thru Akka distributed pub/sub to local `blockchain`
<<<
[INFO] [04/16/2020 18:10:17.526] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:10:26.572] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(1c91, 4500/2, 2020-04-17 01:10:26) to transaction queue.
[INFO] [04/16/2020 18:10:26.997] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(ea85, 5000/3, 2020-04-17 01:10:26) is published.
[INFO] [04/16/2020 18:10:26.997] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(ea85, 5000/3, 2020-04-17 01:10:26) to transaction queue.
[ERROR] [04/16/2020 18:10:28.055] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.Mining] akkablockchain.actor.Blockchainer@e1b6066: ERROR: java.util.concurrent.TimeoutException: BLK(lxoY, T(dea2, 1599/2), 2020-04-17 01:10:10, 3, 0): 18000 milliseconds
<<<
<<<--- Mining of a block timed out; the assembled block will be discarded
<<<
[INFO] [04/16/2020 18:10:28.056] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:10:32.053] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 15000 millis
[INFO] [04/16/2020 18:10:32.053] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:10:32.054] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(1270, 6500/3, 2020-04-17 01:09:27), T(dd7d, 7000/3, 2020-04-17 01:09:41), T(af31, 5000/2, 2020-04-17 01:09:42), T(c28e, 6500/3, 2020-04-17 01:09:56), T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26))
[INFO] [04/16/2020 18:10:32.054] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:10:35.550] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[ERROR] [04/16/2020 18:10:35.551] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: ERROR: BLK(pryD, T(20b3, 5099/3), 2020-04-17 01:10:32, 3, 4014397) is invalid!
<<<
<<<--- A mined block failed validation due to associated transactions existing in local `blockchain`
<<<
[INFO] [04/16/2020 18:10:35.551] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:10:41.571] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(e73e, 4000/2, 2020-04-17 01:10:41) to transaction queue.
[INFO] [04/16/2020 18:10:41.995] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(05c4, 2000/2, 2020-04-17 01:10:41) is published.
[INFO] [04/16/2020 18:10:41.996] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(05c4, 2000/2, 2020-04-17 01:10:41) to transaction queue.
[INFO] [04/16/2020 18:10:42.173] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136) is valid. Updating blockchain.
[INFO] [04/16/2020 18:10:42.173] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:10:47.074] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 26000 millis
[INFO] [04/16/2020 18:10:47.074] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:10:47.075] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(dd7d, 7000/3, 2020-04-17 01:09:41), T(af31, 5000/2, 2020-04-17 01:09:42), T(c28e, 6500/3, 2020-04-17 01:09:56), T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41))
[INFO] [04/16/2020 18:10:47.075] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:10:56.570] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(dd31, 2000/1, 2020-04-17 01:10:56) to transaction queue.
[INFO] [04/16/2020 18:10:57.000] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(ab28, 3500/2, 2020-04-17 01:10:57) is published.
[INFO] [04/16/2020 18:10:57.001] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(ab28, 3500/2, 2020-04-17 01:10:57) to transaction queue.
[ERROR] [04/16/2020 18:11:05.093] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.Mining] akkablockchain.actor.Blockchainer@e1b6066: ERROR: java.util.concurrent.TimeoutException: BLK(VGhP, T(1270, 6599/4), 2020-04-17 01:10:47, 3, 0): 18000 milliseconds
<<<
<<<--- Mining of a block timed out; the assembled block will be discarded
<<<
[INFO] [04/16/2020 18:11:05.093] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:11:09.680] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597) is valid. Updating blockchain.
<<<
<<<--- Adding a mined block thru Akka distributed pub/sub to local `blockchain`
<<<
[INFO] [04/16/2020 18:11:09.680] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:11:11.573] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(3be4, 4500/3, 2020-04-17 01:11:11) to transaction queue.
[INFO] [04/16/2020 18:11:12.003] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(424e, 6000/3, 2020-04-17 01:11:12) is published.
[INFO] [04/16/2020 18:11:12.003] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(424e, 6000/3, 2020-04-17 01:11:12) to transaction queue.
[INFO] [04/16/2020 18:11:13.093] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 25000 millis
[INFO] [04/16/2020 18:11:13.093] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:11:13.094] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(af31, 5000/2, 2020-04-17 01:09:42), T(c28e, 6500/3, 2020-04-17 01:09:56), T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57), T(3be4, 4500/3, 2020-04-17 01:11:11), T(424e, 6000/3, 2020-04-17 01:11:12))
[INFO] [04/16/2020 18:11:13.094] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597), BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:11:18.026] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433) is valid. Updating blockchain.
<<<
<<<--- Adding a locally mined block to local `blockchain`
<<<
[INFO] [04/16/2020 18:11:18.026] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:11:21.022] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[ERROR] [04/16/2020 18:11:21.023] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: ERROR: BLK(3FyP, T(dd7d, 7099/4), 2020-04-17 01:11:13, 3, 8943763) is invalid!
<<<
<<<--- A mined block failed validation due to associated transactions existing in local `blockchain`
<<<
[INFO] [04/16/2020 18:11:21.023] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:11:26.563] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(99be, 2500/1, 2020-04-17 01:11:26) to transaction queue.
[INFO] [04/16/2020 18:11:27.002] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(0e91, 4000/2, 2020-04-17 01:11:27) is published.
[INFO] [04/16/2020 18:11:27.003] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(0e91, 4000/2, 2020-04-17 01:11:27) to transaction queue.
[INFO] [04/16/2020 18:11:38.113] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 16000 millis
[INFO] [04/16/2020 18:11:38.113] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:11:38.114] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(c28e, 6500/3, 2020-04-17 01:09:56), T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57), T(3be4, 4500/3, 2020-04-17 01:11:11), T(424e, 6000/3, 2020-04-17 01:11:12), T(99be, 2500/1, 2020-04-17 01:11:26), T(0e91, 4000/2, 2020-04-17 01:11:27))
[INFO] [04/16/2020 18:11:38.115] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433), BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597), BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:11:41.568] [blockchain-akka.actor.default-dispatcher-29] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(2e89, 6000/3, 2020-04-17 01:11:41) to transaction queue.
[INFO] [04/16/2020 18:11:41.993] [blockchain-akka.actor.default-dispatcher-29] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(1925, 1000/1, 2020-04-17 01:11:41) is published.
[INFO] [04/16/2020 18:11:41.996] [blockchain-akka.actor.default-dispatcher-29] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(1925, 1000/1, 2020-04-17 01:11:41) to transaction queue.
[INFO] [04/16/2020 18:11:44.242] [blockchain-akka.actor.default-dispatcher-29] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232) is valid. Updating blockchain.
<<<
<<<--- Adding a mined block thru Akka distributed pub/sub to local `blockchain`
<<<
[INFO] [04/16/2020 18:11:44.242] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:11:54.132] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 20000 millis
[INFO] [04/16/2020 18:11:54.132] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[ERROR] [04/16/2020 18:11:54.132] [blockchain-akka.actor.default-dispatcher-29] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.Mine(BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232), T(c28e, 6500/3, 2020-04-17 01:09:56)) received but akkablockchain.actor.Miner@320de6d4 is busy!
[INFO] [04/16/2020 18:11:54.132] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57), T(3be4, 4500/3, 2020-04-17 01:11:11), T(424e, 6000/3, 2020-04-17 01:11:12), T(99be, 2500/1, 2020-04-17 01:11:26), T(0e91, 4000/2, 2020-04-17 01:11:27), T(2e89, 6000/3, 2020-04-17 01:11:41), T(1925, 1000/1, 2020-04-17 01:11:41))
[INFO] [04/16/2020 18:11:54.133] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232), BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433), BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597), BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[ERROR] [04/16/2020 18:11:54.133] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.Mining] akkablockchain.actor.Blockchainer@e1b6066: ERROR: akkablockchain.actor.Blockchainer$BusyException: akkablockchain.actor.Miner@320de6d4 is busy!
<<<
<<<--- A mining request rejected by the busy miner; associated transactions to be put back in local `transaction queue`
<<<
[INFO] [04/16/2020 18:11:54.134] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Prepended T(c28e, 6500/3, 2020-04-17 01:09:56) to transaction queue.
[INFO] [04/16/2020 18:11:54.136] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[ERROR] [04/16/2020 18:11:54.137] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: ERROR: BLK(yVLA, T(af31, 5099/3), 2020-04-17 01:11:38, 3, 16916400) is invalid!
<<<
<<<--- A mined block failed validation due to associated transactions existing in local `blockchain`
<<<
[INFO] [04/16/2020 18:11:54.137] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:11:56.565] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(7d6c, 7500/3, 2020-04-17 01:11:56) to transaction queue.
[INFO] [04/16/2020 18:11:57.000] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(3002, 8000/3, 2020-04-17 01:11:57) is published.
[INFO] [04/16/2020 18:11:57.001] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(3002, 8000/3, 2020-04-17 01:11:57) to transaction queue.
[INFO] [04/16/2020 18:11:57.305] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: BLK(l++u, T(6a16, 2099/2), 2020-04-17 01:11:46, 3, 11875208) is valid. Updating blockchain.
<<<
<<<--- Adding a locally mined block to local `blockchain`
<<<
[INFO] [04/16/2020 18:11:57.305] [blockchain-akka.actor.default-dispatcher-12] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:12:11.570] [blockchain-akka.actor.default-dispatcher-29] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(bd60, 1000/1, 2020-04-17 01:12:11) to transaction queue.
[INFO] [04/16/2020 18:12:11.993] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(26a0, 3000/1, 2020-04-17 01:12:11) is published.
[INFO] [04/16/2020 18:12:11.994] [blockchain-akka.actor.default-dispatcher-29] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(26a0, 3000/1, 2020-04-17 01:12:11) to transaction queue.
[INFO] [04/16/2020 18:12:14.150] [blockchain-akka.actor.default-dispatcher-29] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 16000 millis
[INFO] [04/16/2020 18:12:14.150] [blockchain-akka.actor.default-dispatcher-29] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:12:14.151] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57), T(3be4, 4500/3, 2020-04-17 01:11:11), T(424e, 6000/3, 2020-04-17 01:11:12), T(99be, 2500/1, 2020-04-17 01:11:26), T(0e91, 4000/2, 2020-04-17 01:11:27), T(2e89, 6000/3, 2020-04-17 01:11:41), T(1925, 1000/1, 2020-04-17 01:11:41), T(7d6c, 7500/3, 2020-04-17 01:11:56), T(3002, 8000/3, 2020-04-17 01:11:57), T(bd60, 1000/1, 2020-04-17 01:12:11), T(26a0, 3000/1, 2020-04-17 01:12:11))
[INFO] [04/16/2020 18:12:14.151] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(l++u, T(6a16, 2099/2), 2020-04-17 01:11:46, 3, 11875208), BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232), BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433), BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597), BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:12:14.791] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@e1b6066: BLK(g+dc, T(a784, 2599/3), 2020-04-17 01:12:10, 3, 4362943) is valid. Updating blockchain.
<<<
<<<--- Adding a mined block thru Akka distributed pub/sub to local `blockchain`
<<<
[INFO] [04/16/2020 18:12:14.791] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[WARN] [akkaUnreachable][04/16/2020 18:12:23.924] [blockchain-akka.actor.internal-dispatcher-11] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Marking node as UNREACHABLE [Member(address = akka://blockchain@127.0.0.1:2552, status = Up)].
[INFO] [04/16/2020 18:12:27.000] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(4632, 4000/3, 2020-04-17 01:12:27) is published.
[INFO] [04/16/2020 18:12:27.001] [blockchain-akka.actor.default-dispatcher-30] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(4632, 4000/3, 2020-04-17 01:12:27) to transaction queue.
[INFO] [akkaDeadLetter][04/16/2020 18:12:27.102] [blockchain-akka.actor.default-dispatcher-30] [akka://blockchain/deadLetters] Message [akka.cluster.GossipStatus] from Actor[akka://blockchain/system/cluster/core/daemon#1177403680] to Actor[akka://blockchain/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://blockchain/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][04/16/2020 18:12:27.102] [blockchain-akka.actor.default-dispatcher-30] [akka://blockchain/deadLetters] Message [akka.cluster.GossipStatus] from Actor[akka://blockchain/system/cluster/core/daemon#1177403680] to Actor[akka://blockchain/deadLetters] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://blockchain/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][04/16/2020 18:12:27.102] [blockchain-akka.actor.default-dispatcher-30] [akka://blockchain@127.0.0.1:2552/system/distributedPubSubMediator/new-transactions] Message [akkablockchain.actor.Blockchainer$AddTransactions] from Actor[akka://blockchain/user/blockchainer#559536709] to Actor[akka://blockchain@127.0.0.1:2552/system/distributedPubSubMediator/new-transactions#1378150251] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://blockchain@127.0.0.1:2552/system/distributedPubSubMediator/new-transactions#1378150251] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[WARN] [04/16/2020 18:12:27.451] [blockchain-akka.remote.default-remote-dispatcher-5] [akka.stream.Log(akka://blockchain/system/Materializers/StreamSupervisor-1)] [outbound connection to [akka://blockchain@127.0.0.1:2552], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1:2552,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
<<<
<<<--- Detected cluster node #2 no longer reachable
<<<
[INFO] [04/16/2020 18:12:30.171] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Start mining in 21000 millis
[INFO] [04/16/2020 18:12:30.172] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[ERROR] [04/16/2020 18:12:30.172] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.Mine(BLK(g+dc, T(a784, 2599/3), 2020-04-17 01:12:10, 3, 4362943), T(6a16, 2000/1, 2020-04-17 01:09:56)) received but akkablockchain.actor.Miner@320de6d4 is busy!
<<<
<<<--- A mining request rejected by the busy miner; associated transactions to be put back in local `transaction queue`
<<<
[ERROR] [04/16/2020 18:12:30.172] [blockchain-akka.actor.default-dispatcher-30] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.Mining] akkablockchain.actor.Blockchainer@e1b6066: ERROR: akkablockchain.actor.Blockchainer$BusyException: akkablockchain.actor.Miner@320de6d4 is busy!
[INFO] [04/16/2020 18:12:30.172] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Transaction queue: Queue(T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57), T(3be4, 4500/3, 2020-04-17 01:11:11), T(424e, 6000/3, 2020-04-17 01:11:12), T(99be, 2500/1, 2020-04-17 01:11:26), T(0e91, 4000/2, 2020-04-17 01:11:27), T(2e89, 6000/3, 2020-04-17 01:11:41), T(1925, 1000/1, 2020-04-17 01:11:41), T(7d6c, 7500/3, 2020-04-17 01:11:56), T(3002, 8000/3, 2020-04-17 01:11:57), T(bd60, 1000/1, 2020-04-17 01:12:11), T(26a0, 3000/1, 2020-04-17 01:12:11), T(4632, 4000/3, 2020-04-17 01:12:27))
[INFO] [04/16/2020 18:12:30.172] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Prepended T(6a16, 2000/1, 2020-04-17 01:09:56) to transaction queue.
[INFO] [04/16/2020 18:12:30.173] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/simulator] [MiningLoop] Blockchain: List(BLK(g+dc, T(a784, 2599/3), 2020-04-17 01:12:10, 3, 4362943), BLK(l++u, T(6a16, 2099/2), 2020-04-17 01:11:46, 3, 11875208), BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232), BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433), BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597), BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[ERROR] [04/16/2020 18:12:32.172] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.Mining] akkablockchain.actor.Blockchainer@e1b6066: ERROR: java.util.concurrent.TimeoutException: BLK(d13g, T(c28e, 6599/4), 2020-04-17 01:12:14, 3, 0): 18000 milliseconds
[INFO] [04/16/2020 18:12:32.172] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[WARN] [04/16/2020 18:12:35.612] [blockchain-akka.remote.default-remote-dispatcher-9] [akka.stream.Log(akka://blockchain/system/Materializers/StreamSupervisor-1)] [outbound connection to [akka://blockchain@127.0.0.1:2552], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1:2552,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
[INFO] [04/16/2020 18:12:41.997] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@e1b6066: T(552a, 4000/2, 2020-04-17 01:12:41) is published.
[INFO] [04/16/2020 18:12:41.997] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2551/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@e1b6066: Appended T(552a, 4000/2, 2020-04-17 01:12:41) to transaction queue.
[INFO] [akkaDeadLetter][04/16/2020 18:12:43.022] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2552/system/distributedPubSubMediator/new-transactions] Message [akkablockchain.actor.Blockchainer$AddTransactions] from Actor[akka://blockchain/user/blockchainer#559536709] to Actor[akka://blockchain@127.0.0.1:2552/system/distributedPubSubMediator/new-transactions#1378150251] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://blockchain@127.0.0.1:2552/system/distributedPubSubMediator/new-transactions#1378150251] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[WARN] [04/16/2020 18:12:43.767] [blockchain-akka.remote.default-remote-dispatcher-9] [akka.stream.Log(akka://blockchain/system/Materializers/StreamSupervisor-1)] [outbound connection to [akka://blockchain@127.0.0.1:2552], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1:2552,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
[INFO] [akkaClusterLeaderIncapacitated][04/16/2020 18:12:44.327] [blockchain-akka.actor.internal-dispatcher-3] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2551] - Leader can currently not perform its duties, reachability status: [akka://blockchain@127.0.0.1:2551 -> akka://blockchain@127.0.0.1:2552: Unreachable [Unreachable] (1)], member status: [akka://blockchain@127.0.0.1:2551 Up seen=true, akka://blockchain@127.0.0.1:2552 Up seen=false]

Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
<<<
<<<--- User terminated program on the node
<<<
###
### Annotated Console Log: NODE #2
###

sbt:akka-blockchain> runMain akkablockchain.Main 2552 src/main/resources/keys/account1_public.pem
runMain akkablockchain.Main 2552 src/main/resources/keys/account1_public.pem
<<<
<<<--- Running on port# 2552, also a seed node, with minerKeyFile `account1_public.pem`
<<<
[info] Running akkablockchain.Main 2552 src/main/resources/keys/account1_public.pem
[INFO] [04/16/2020 18:09:24.088] [run-main-0] [ArteryTcpTransport(akka://blockchain)] Remoting started with transport [Artery tcp]; listening on address [akka://blockchain@127.0.0.1:2552] with UID [7768051775546777328]
[INFO] [04/16/2020 18:09:24.112] [run-main-0] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2552] - Starting up, Akka version [2.6.4] ...
[INFO] [04/16/2020 18:09:24.217] [run-main-0] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2552] - Registered cluster JMX MBean [akka:type=Cluster]
[INFO] [04/16/2020 18:09:24.217] [run-main-0] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2552] - Started up successfully
[INFO] [04/16/2020 18:09:24.271] [blockchain-akka.actor.internal-dispatcher-6] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2552] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing
[INFO] [04/16/2020 18:09:24.594] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Ack] akkablockchain.actor.Blockchainer@608e81c9: Subscribing to 'new-transactions' ...
[INFO] [04/16/2020 18:09:24.594] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Ack] akkablockchain.actor.Blockchainer@608e81c9: Subscribing to 'new-block' ...
[INFO] [04/16/2020 18:09:24.884] [blockchain-akka.actor.internal-dispatcher-3] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2552] - Received InitJoinAck message from [Actor[akka://blockchain@127.0.0.1:2551/system/cluster/core/daemon#1177403680]] to [akka://blockchain@127.0.0.1:2552]
[INFO] [04/16/2020 18:09:24.975] [blockchain-akka.actor.internal-dispatcher-2] [Cluster(akka://blockchain)] Cluster Node [akka://blockchain@127.0.0.1:2552] - Welcome from [akka://blockchain@127.0.0.1:2551]
<<<
<<<--- Seed node #2, bound to port# 2552, joined the cluster
<<<
[INFO] [04/16/2020 18:09:26.591] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(20b3, 5000/2, 2020-04-17 01:09:26) is published.
[INFO] [04/16/2020 18:09:26.594] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(20b3, 5000/2, 2020-04-17 01:09:26) to transaction queue.
[INFO] [04/16/2020 18:09:27.015] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(1270, 6500/3, 2020-04-17 01:09:27) to transaction queue.
[INFO] [04/16/2020 18:09:38.795] [blockchain-akka.actor.default-dispatcher-14] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771) is valid. Updating blockchain.
<<<
<<<--- Adding a mined block thru Akka distributed pub/sub to local `blockchain`
<<<
[INFO] [04/16/2020 18:09:38.796] [blockchain-akka.actor.default-dispatcher-14] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:09:41.575] [blockchain-akka.actor.default-dispatcher-14] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(dd7d, 7000/3, 2020-04-17 01:09:41) is published.
[INFO] [04/16/2020 18:09:41.576] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(dd7d, 7000/3, 2020-04-17 01:09:41) to transaction queue.
[INFO] [04/16/2020 18:09:42.008] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(af31, 5000/2, 2020-04-17 01:09:42) to transaction queue.
[INFO] [04/16/2020 18:09:54.548] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Start mining in 16000 millis
[INFO] [04/16/2020 18:09:54.548] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:09:54.549] [blockchain-akka.actor.default-dispatcher-14] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Transaction queue: Queue(T(20b3, 5000/2, 2020-04-17 01:09:26), T(1270, 6500/3, 2020-04-17 01:09:27), T(dd7d, 7000/3, 2020-04-17 01:09:41), T(af31, 5000/2, 2020-04-17 01:09:42))
[INFO] [04/16/2020 18:09:54.549] [blockchain-akka.actor.default-dispatcher-14] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Blockchain: List(BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:09:56.576] [blockchain-akka.actor.default-dispatcher-14] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(c28e, 6500/3, 2020-04-17 01:09:56) is published.
[INFO] [04/16/2020 18:09:56.578] [blockchain-akka.actor.default-dispatcher-14] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(c28e, 6500/3, 2020-04-17 01:09:56) to transaction queue.
[INFO] [04/16/2020 18:09:56.998] [blockchain-akka.actor.default-dispatcher-14] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(6a16, 2000/1, 2020-04-17 01:09:56) to transaction queue.
[INFO] [04/16/2020 18:10:10.563] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Start mining in 25000 millis
[INFO] [04/16/2020 18:10:10.563] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:10:10.565] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Transaction queue: Queue(T(1270, 6500/3, 2020-04-17 01:09:27), T(dd7d, 7000/3, 2020-04-17 01:09:41), T(af31, 5000/2, 2020-04-17 01:09:42), T(c28e, 6500/3, 2020-04-17 01:09:56), T(6a16, 2000/1, 2020-04-17 01:09:56))
[INFO] [04/16/2020 18:10:10.565] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Blockchain: List(BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:10:11.570] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(a784, 2500/2, 2020-04-17 01:10:11) is published.
[INFO] [04/16/2020 18:10:11.570] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(a784, 2500/2, 2020-04-17 01:10:11) to transaction queue.
[INFO] [04/16/2020 18:10:12.001] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(64bb, 3000/2, 2020-04-17 01:10:11) to transaction queue.
[INFO] [04/16/2020 18:10:17.518] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:10:17.520] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662) is valid. Updating blockchain.
<<<
<<<--- Adding a locally mined block to local `blockchain`
<<<
[INFO] [04/16/2020 18:10:17.521] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:10:26.569] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(1c91, 4500/2, 2020-04-17 01:10:26) is published.
[INFO] [04/16/2020 18:10:26.569] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(1c91, 4500/2, 2020-04-17 01:10:26) to transaction queue.
[INFO] [04/16/2020 18:10:26.999] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(ea85, 5000/3, 2020-04-17 01:10:26) to transaction queue.
[ERROR] [04/16/2020 18:10:35.565] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: ERROR: BLK(pryD, T(20b3, 5099/3), 2020-04-17 01:10:32, 3, 4014397) is invalid!
<<<
<<<--- A mined block failed validation due to associated transactions existing in local `blockchain`
<<<
[INFO] [04/16/2020 18:10:35.566] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:10:35.581] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Start mining in 23000 millis
[INFO] [04/16/2020 18:10:35.581] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:10:35.582] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Transaction queue: Queue(T(dd7d, 7000/3, 2020-04-17 01:09:41), T(af31, 5000/2, 2020-04-17 01:09:42), T(c28e, 6500/3, 2020-04-17 01:09:56), T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26))
[INFO] [04/16/2020 18:10:35.582] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Blockchain: List(BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:10:41.568] [blockchain-akka.actor.default-dispatcher-25] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(e73e, 4000/2, 2020-04-17 01:10:41) is published.
[INFO] [04/16/2020 18:10:41.569] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(e73e, 4000/2, 2020-04-17 01:10:41) to transaction queue.
[INFO] [04/16/2020 18:10:41.997] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(05c4, 2000/2, 2020-04-17 01:10:41) to transaction queue.
[INFO] [04/16/2020 18:10:42.165] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:10:42.167] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136) is valid. Updating blockchain.
<<<
<<<--- Adding a mined block thru Akka distributed pub/sub to local `blockchain`
<<<
[INFO] [04/16/2020 18:10:42.167] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:10:56.567] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(dd31, 2000/1, 2020-04-17 01:10:56) is published.
[INFO] [04/16/2020 18:10:56.568] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(dd31, 2000/1, 2020-04-17 01:10:56) to transaction queue.
[INFO] [04/16/2020 18:10:57.003] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(ab28, 3500/2, 2020-04-17 01:10:57) to transaction queue.
[INFO] [04/16/2020 18:10:58.601] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Start mining in 16000 millis
[INFO] [04/16/2020 18:10:58.601] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:10:58.603] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Transaction queue: Queue(T(af31, 5000/2, 2020-04-17 01:09:42), T(c28e, 6500/3, 2020-04-17 01:09:56), T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57))
[INFO] [04/16/2020 18:10:58.604] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Blockchain: List(BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:11:09.668] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:11:09.672] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597) is valid. Updating blockchain.
<<<
<<<--- Adding a locally mined block to local `blockchain`
<<<
[INFO] [04/16/2020 18:11:09.672] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:11:11.570] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(3be4, 4500/3, 2020-04-17 01:11:11) is published.
[INFO] [04/16/2020 18:11:11.571] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(3be4, 4500/3, 2020-04-17 01:11:11) to transaction queue.
[INFO] [04/16/2020 18:11:12.005] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(424e, 6000/3, 2020-04-17 01:11:12) to transaction queue.
[INFO] [04/16/2020 18:11:14.619] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Start mining in 17000 millis
[INFO] [04/16/2020 18:11:14.619] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:11:14.620] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Transaction queue: Queue(T(c28e, 6500/3, 2020-04-17 01:09:56), T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57), T(3be4, 4500/3, 2020-04-17 01:11:11), T(424e, 6000/3, 2020-04-17 01:11:12))
[INFO] [04/16/2020 18:11:14.621] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Blockchain: List(BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597), BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:11:18.014] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:11:18.019] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433) is valid. Updating blockchain.
<<<
<<<--- Adding a mined block thru Akka distributed pub/sub to local `blockchain`
<<<
[INFO] [04/16/2020 18:11:18.019] [blockchain-akka.actor.default-dispatcher-4] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[ERROR] [04/16/2020 18:11:21.031] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: ERROR: BLK(3FyP, T(dd7d, 7099/4), 2020-04-17 01:11:13, 3, 8943763) is invalid!
<<<
<<<--- A mined block failed validation due to associated transactions existing in local `blockchain`
<<<
[INFO] [04/16/2020 18:11:21.031] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:11:26.561] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(99be, 2500/1, 2020-04-17 01:11:26) is published.
[INFO] [04/16/2020 18:11:26.562] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(99be, 2500/1, 2020-04-17 01:11:26) to transaction queue.
[INFO] [04/16/2020 18:11:27.005] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(0e91, 4000/2, 2020-04-17 01:11:27) to transaction queue.
[INFO] [04/16/2020 18:11:31.639] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Start mining in 15000 millis
[INFO] [04/16/2020 18:11:31.639] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:11:31.640] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Transaction queue: Queue(T(6a16, 2000/1, 2020-04-17 01:09:56), T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57), T(3be4, 4500/3, 2020-04-17 01:11:11), T(424e, 6000/3, 2020-04-17 01:11:12), T(99be, 2500/1, 2020-04-17 01:11:26), T(0e91, 4000/2, 2020-04-17 01:11:27))
[INFO] [04/16/2020 18:11:31.640] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Blockchain: List(BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433), BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597), BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:11:41.565] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(2e89, 6000/3, 2020-04-17 01:11:41) is published.
[INFO] [04/16/2020 18:11:41.566] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(2e89, 6000/3, 2020-04-17 01:11:41) to transaction queue.
[INFO] [04/16/2020 18:11:41.998] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(1925, 1000/1, 2020-04-17 01:11:41) to transaction queue.
[INFO] [04/16/2020 18:11:44.227] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:11:44.231] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232) is valid. Updating blockchain.
<<<
<<<--- Adding a locally mined block to local `blockchain`
<<<
[INFO] [04/16/2020 18:11:44.231] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:11:46.659] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Start mining in 24000 millis
[INFO] [04/16/2020 18:11:46.659] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:11:46.660] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Transaction queue: Queue(T(a784, 2500/2, 2020-04-17 01:10:11), T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57), T(3be4, 4500/3, 2020-04-17 01:11:11), T(424e, 6000/3, 2020-04-17 01:11:12), T(99be, 2500/1, 2020-04-17 01:11:26), T(0e91, 4000/2, 2020-04-17 01:11:27), T(2e89, 6000/3, 2020-04-17 01:11:41), T(1925, 1000/1, 2020-04-17 01:11:41))
[INFO] [04/16/2020 18:11:46.660] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Blockchain: List(BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232), BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433), BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597), BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[ERROR] [04/16/2020 18:11:54.143] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: ERROR: BLK(yVLA, T(af31, 5099/3), 2020-04-17 01:11:38, 3, 16916400) is invalid!
<<<
<<<--- A mined block failed validation due to associated transactions existing in local `blockchain`
<<<
[INFO] [04/16/2020 18:11:54.143] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:11:56.562] [blockchain-akka.actor.default-dispatcher-13] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(7d6c, 7500/3, 2020-04-17 01:11:56) is published.
[INFO] [04/16/2020 18:11:56.563] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(7d6c, 7500/3, 2020-04-17 01:11:56) to transaction queue.
[INFO] [04/16/2020 18:11:57.003] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(3002, 8000/3, 2020-04-17 01:11:57) to transaction queue.
[INFO] [04/16/2020 18:11:57.295] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:11:57.301] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: BLK(l++u, T(6a16, 2099/2), 2020-04-17 01:11:46, 3, 11875208) is valid. Updating blockchain.
<<<
<<<--- Adding a mined block thru Akka distributed pub/sub to local `blockchain`
<<<
[INFO] [04/16/2020 18:11:57.301] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.
[INFO] [04/16/2020 18:12:10.678] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Start mining in 16000 millis
[INFO] [04/16/2020 18:12:10.678] [blockchain-akka.actor.default-dispatcher-27] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Getting transaction queue and blockchain ...
[INFO] [04/16/2020 18:12:10.679] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Transaction queue: Queue(T(64bb, 3000/2, 2020-04-17 01:10:11), T(1c91, 4500/2, 2020-04-17 01:10:26), T(ea85, 5000/3, 2020-04-17 01:10:26), T(e73e, 4000/2, 2020-04-17 01:10:41), T(05c4, 2000/2, 2020-04-17 01:10:41), T(dd31, 2000/1, 2020-04-17 01:10:56), T(ab28, 3500/2, 2020-04-17 01:10:57), T(3be4, 4500/3, 2020-04-17 01:11:11), T(424e, 6000/3, 2020-04-17 01:11:12), T(99be, 2500/1, 2020-04-17 01:11:26), T(0e91, 4000/2, 2020-04-17 01:11:27), T(2e89, 6000/3, 2020-04-17 01:11:41), T(1925, 1000/1, 2020-04-17 01:11:41), T(7d6c, 7500/3, 2020-04-17 01:11:56), T(3002, 8000/3, 2020-04-17 01:11:57))
[INFO] [04/16/2020 18:12:10.680] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/simulator] [MiningLoop] Blockchain: List(BLK(l++u, T(6a16, 2099/2), 2020-04-17 01:11:46, 3, 11875208), BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232), BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433), BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597), BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136), BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662), BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771), BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0))
[INFO] [04/16/2020 18:12:11.567] [blockchain-akka.actor.default-dispatcher-28] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Req.SubmitTransactions] akkablockchain.actor.Blockchainer@608e81c9: T(bd60, 1000/1, 2020-04-17 01:12:11) is published.
[INFO] [04/16/2020 18:12:11.568] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(bd60, 1000/1, 2020-04-17 01:12:11) to transaction queue.
[INFO] [04/16/2020 18:12:11.996] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.AddTransactions] akkablockchain.actor.Blockchainer@608e81c9: Appended T(26a0, 3000/1, 2020-04-17 01:12:11) to transaction queue.
[INFO] [04/16/2020 18:12:14.774] [blockchain-akka.actor.default-dispatcher-15] [akka://blockchain@127.0.0.1:2552/user/blockchainer/miner] [Mining] Miner.DoneMining received.
[INFO] [04/16/2020 18:12:14.786] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer] [Cmd.UpdateBlockchain] akkablockchain.actor.Blockchainer@608e81c9: BLK(g+dc, T(a784, 2599/3), 2020-04-17 01:12:10, 3, 4362943) is valid. Updating blockchain.
<<<
<<<--- Adding a locally mined block to local `blockchain`
<<<
[INFO] [04/16/2020 18:12:14.786] [blockchain-akka.actor.default-dispatcher-26] [akka://blockchain@127.0.0.1:2552/user/blockchainer/blockInspector] [Validation] BlockInspector.DoneValidation received.

Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
<<<
<<<--- User terminated program on the node
<<<

Note that, for illustration purpose, each block as defined in trait `Block`’s `toString` method:

sealed trait Block {
  def hash: Array[Byte]
  def hashPrev: Array[Byte]
  def merkleRoot: MerkleTree
  def transactions: Transactions
  def timestamp: Long
  def difficulty: Int
  def nonce: Long
  def length: Int

  override def toString: String = {
    val datetime = timestampToDateTime(timestamp)
    val transInfo = s"T(${transactions.id.substring(0, 4)}, ${transactions.items.map(_.amount).sum}/${transactions.items.size})"
    s"BLK(${bytesToBase64(hash).substring(0, 4)}, ${transInfo}, ${datetime}, ${difficulty}, ${nonce})"
  }
}

is represented in an abbreviated format as:

BLK( hash, Trans(id, total-amount/num-of-trans), date-time, difficulty, proof )

where `proof` is the first incremented `nonce` value in PoW that satisfies the requirement at the specified `difficulty` level.

As can be seen in the latest copies of `blockchain` maintained on the individual cluster nodes, they get updated via distributed pub/sub in accordance with the consensual rule, but still may differ from each other (typically by one or more most recently added blocks) when examined at any given point of time.

// Blockchain on Node #1 [04/16/2020 18:12:30 US/Pacific]:
List(
  BLK(g+dc, T(a784, 2599/3), 2020-04-17 01:12:10, 3, 4362943),
  BLK(l++u, T(6a16, 2099/2), 2020-04-17 01:11:46, 3, 11875208),
  BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232),
  BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433),
  BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597),
  BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136),
  BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662),
  BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771),
  BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0)
)

// Blockchain on Node #2 [04/16/2020 18:12:10 US/Pacific]:
List(
  BLK(l++u, T(6a16, 2099/2), 2020-04-17 01:11:46, 3, 11875208),
  BLK(yIsc, T(c28e, 6599/4), 2020-04-17 01:11:31, 3, 13515232),
  BLK(9+Ca, T(af31, 5099/3), 2020-04-17 01:11:14, 3, 3918433),
  BLK(vuLL, T(dd7d, 7099/4), 2020-04-17 01:10:58, 3, 11900597),
  BLK(zycW, T(1270, 6599/4), 2020-04-17 01:10:35, 3, 8360136),
  BLK(639L, T(20b3, 5099/3), 2020-04-17 01:10:10, 3, 7614662),
  BLK(n6T2, T(1cab, 3099/3), 2020-04-17 01:09:33, 3, 7028771),
  BLK(XF1C, T(----, 0/0), 1970-01-01 00:00:00, 0, 0)
)

Reliability and efficiency

The application is primarily for proof of concept, hence the abundant side-effecting console logging for illustration purpose. From a reliability and efficiency perspective, it would benefit from the following enhancements to be slightly more robust:

  • Fault tolerance: Akka Persistence via journals and snapshots over Redis, Cassandra, etc, woud help recover an actor’s state in case of a system crash. In particular, the distributed `blockchain` copy (and maybe `transactionQueue` as well) maintained within actor Blockchainer could be crash-proofed with persistence. One approach would be to refactor actor Blockchainer to delegate maintenance of `blockchain` to a dedicated child PersistentActor.
  • Serialization: Akka’s default Java serializer is known for not being very efficient. Other serializers such as Protocol Buffers, Kryo should be considered.

Feature enhancement

Feature-wise, the following enhancements would help make the application one step closer to a real-world cryptocurrency system:

  • Data privacy: Currently the transactions stored in the blockchain isn’t encrypted, despite PKCS public keys are being used within individual transactions. The individual transaction items could be encrypted, each of which to be stored with the associated cryptographic public key/signature, requiring miners to validate the signature while allowing only those who have the private key for certain transactions to see the content.
  • Self regulation: A self-regulatory mechanism that adjusts the difficulty level of the Proof of Work in accordance with network load would help stabilize the currency. As an example, in a recent drastic plunge of the Bitcoin market value in mid March, there was reportedly a significant self-regulatory reduction in the PoW difficulty to temporarily make mining more rewarding that helped dampen the fall.
  • Currency supply: In a cryptocurrency like Bitcoin, issuance of the mining reward by the network is essentially the “minting” of the digital coins. To keep inflation rate under control as the currency supply grows, the rate of coin minting must be proportionately regulated over time. For instance, Bitcoin has a periodic “halfing” mechanism that reduces the mining reward by half for every 210,000 blocks added to the blockchain and will cease producing new coins once the total supply reaches 21 million coins.
  • Blockchain versioning: Versioning of the blockchain would make it possible for future algorithmic changes by means of a `fork`, akin to Bitcoin’s soft/hard forks, without having to discard the old system.
  • User Interface: The existing application focuses mainly on how to operate a blockchain network, thus supplementing it with, say, a Web-based user interface (e.g. using Play framework) would certainly make it a more complete system.

Scala Promises – Futures In Your Hands

In the previous blog post, we saw how Scala Futures serve as a handy wrapper for running asynchronous tasks and allow non-blocking functional transformations via composable functions. Despite all the goodies, a plain Future, once started, is read-only.

A “manipulable” Future

To make things a little more interesting, let’s take a glimpse into an interesting “container” that holds an “uncertain” Future. Scala provides another abstraction called Promise that allows programmers to have some control in the “when” and “what” in completing a Future. A Promise is like a container holding a Future which can be completed by assigning a value (with success or failure) at any point of time. The catch is that it can only be completed once.

The Promise companion object has the following `apply` method that creates a `DefaultPromise`:

object Promise {
  def apply[T](): Promise[T] = new impl.Promise.DefaultPromise[T]()
  // ...
}

As shown below, the DefaultPromise class extends AtomicReference to ensure that a Promise instance will be completed in an atomic fashion.

class DefaultPromise[T] extends AtomicReference[AnyRef](Nil) with Promise[T]

A trivial producer and consumer

A common use case of Promise is like this:

  1. a Promise which holds an “open” Future is created
  2. run some business logic to come up with some value
  3. complete the Promise by assigning its Future the value via methods like `success()`, `failure()`, `tryComplete()`, etc
  4. return the “closed” Future

Here’s a hello-world example of Scala Promise used in a trivialized producer and consumer:

import scala.concurrent.{Future, Promise}
import scala.util.{Success, Failure}
import scala.concurrent.ExecutionContext.Implicits.global
import java.util.concurrent.ThreadLocalRandom

val p = Promise[Int]()
val f = p.future

val producer = Future {
  val r = ThreadLocalRandom.current.nextInt(0, 2000)
  println("Inside producer: r = " + r)
  if (r < 1000)
    p success r
  else
    p failure (new Exception("r > 999!"))
}

val consumer = Future {
  println("Inside consumer!")
  f onComplete {
    case Success(r) => println("Success: r = " + r)
    case Failure(e) => println("Error: " + e)
  }
}

The above code snippet is rather self-explanatory. The producer running in one thread completes the Promise’s future based on the result of a randomly generated integer and the consumer in another thread checks and reports the value of the completed future.

// * Test running #1:

// Inside producer: r = 1278
// producer: scala.concurrent.Future[scala.concurrent.Promise[Int]] = Future()
// consumer: scala.concurrent.Future[Unit] = Future()
// Error: java.lang.Exception: r > 999!

// * Test running #2:

// Inside producer: r = 547
// producer: scala.concurrent.Future[scala.concurrent.Promise[Int]] = Future()
// consumer: scala.concurrent.Future[Unit] = Future()
// Success: r = 547

Simulating a CPU-bound task, again

For up-coming illustrations, let’s borrow the CPU-bound task simulation `doWork` method used in the coding examples from the previous blog post:

case class Result(val id: Int, val millis: Long)

def doWork(id: Int): Result = {
  import java.util.concurrent.ThreadLocalRandom
  val millis = ThreadLocalRandom.current.nextLong(1000, 2000)
  val millisTMO = 1500
  val start = System.currentTimeMillis()

  while ((System.currentTimeMillis() - start) < millis) {}

  if (millis < millisTMO) {
    println("Work Result: " + Result(id, millis))
    Result(id, millis)
  } else {
    println(s"TimeOut: work id $id would take ${millis - millisTMO}ms above limit to complete!")
    throw new Throwable("TimeOut!")
  }
}

Revisiting first completed Future

Recall that method `Future.firstCompletedOf` from the previous blog post can be used to capture the first completed Future out of a list of Futures running in parallel:

val firstCompleted = Future.firstCompletedOf( (1 to 4).map( id =>
  Future{ doWork(id) }.map( Right(_) ).recover{ case e => Left(e) }
) )

Now, let’s see how `firstCompletedOf` is actually implemented in Scala Future using Promise:

object Future {
  // ...

  def firstCompletedOf[T](futures: TraversableOnce[Future[T]])(implicit executor: ExecutionContext): Future[T] = {
    val p = Promise[T]()
    val firstCompleteHandler = new AtomicReference[Promise[T]](p) with (Try[T] => Unit) {
      override def apply(v1: Try[T]): Unit = getAndSet(null) match {
        case null => ()
        case some => some tryComplete v1
      }
    }
    futures foreach { _ onComplete firstCompleteHandler }
    p.future
  }

  // ...
}

In the `firstCompletedOf` method implementation, the helper callback function `firstCompleteHandler` for each of the Futures in the input list ensures by means of an `AtomicReference` that the first completed Future will be the Promise’s future.

First completed Future with a condition

What if we want to get the first completed Future from a number of Futures whose values meet a certain condition? One approach would be to derive from the `firstCompletedOf` method implementation.

We pick the default ExecutionContext like how we did in some coding examples from the previous blog. Besides the list of Futures, the derived method `firstConditionallyCompletedOf[T]` would also take a `T => Boolean` filtering condition as a parameter. Piggybacking on the core logic from method `firstCompletedOf`, we simply apply the input filter to each of the Futures in the input list before the callback.

import scala.concurrent.{ExecutionContext, Future, Promise, Await}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}

import java.util.concurrent.atomic.AtomicReference

def firstConditionallyCompletedOf[T](futures: List[Future[T]], condition: T => Boolean)(
  implicit ec: ExecutionContext): Future[T] = {

  val p = Promise[T]()
  val firstCompleteHandler = new AtomicReference[Promise[T]](p) with (Try[T] => Unit) {
    override def apply(v1: Try[T]): Unit = getAndSet(null) match {
      case null => ()
      case some => some tryComplete v1
    }
  }
  futures.foreach{ _.filter(condition).onComplete(firstCompleteHandler) }
  p.future
}

val firstCompleted = firstConditionallyCompletedOf(
  (1 to 4).map( id =>
    Future{ doWork(id) }.map( Right(_) ).recover{ case e => Left(e) }
  ),
  (e: Either[Throwable, Result]) => e match {
    case Left(_) => false
    case Right(res) => res.millis < 1250
  }
)

Await.ready(firstCompleted, 10.seconds)

// * Test running #1:

// Work Result: Result(4, 1037)
// Work Result: Result(2, 1268)
// Work Result: Result(3, 1386)
// TimeOut: work id 1 would take 139ms above limit to complete!
// res1: firstCompleted.type = Future(Success(Right(Result(4, 1037))))

// * Test running #2:

// Work Result: Result(4, 1320)
// TimeOut: work id 1 would take 22ms above limit to complete!
// TimeOut: work id 3 would take 253ms above limit to complete!
// TimeOut: work id 2 would take 438ms above limit to complete!
// res2: firstCompleted.type = Future(Failure(
//   java.util.NoSuchElementException: Future.filter predicate is not satisfied
// ))

First N completed Futures

While at it, rather than just the first completed Future, what if we want to capture the first few completed Futures? Deriving from the `firstCompletedOf` implementation wouldn’t quite work – the way the helper callback function `firstCompleteHandler` is structured wouldn’t be useful now that we have a list of Futures to be captured.

We’ll take a straight forward approach of using a `var` list for capturing the first N (or the size of input Futures, whichever smaller) Future results and update the list inside a `synchronized` block. Since we want to capture the first few completed Futures (success or failure), we make the return Future consisting of a `List[Either[Throwable, T]]`, rather than just `List[T]`.

import scala.concurrent.{ExecutionContext, Future, Promise, Await}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}

def firstNCompletedOf[T](n: Int, futures: List[Future[T]])(implicit ec: ExecutionContext):
  Future[List[Either[Throwable, T]]] = {

  val nLimit = n min futures.size
  val p = Promise[List[Either[Throwable, T]]]()
  var list = List.empty[Either[Throwable, T]]
  futures foreach { _.onComplete{ tryT =>
    synchronized {
      if (list.size < nLimit)
        list ::= tryT.toEither
      else
        p trySuccess list.reverse
    }
  } }
  p.future
}

val firstNCompleted = firstNCompletedOf( 3, (1 to 5).toList.map( id => Future{ doWork(id) } ) )

Await.ready(firstNCompleted, 15.seconds)

// Work Result: Result(1,1168)
// Work Result: Result(5,1180)
// TimeOut: work id 3 would take 200ms above limit to complete!
// TimeOut: work id 2 would take 399ms above limit to complete!
// TimeOut: work id 4 would take 448ms above limit to complete!

// res1: firstNCompleted.type = Future(Success(List(
//   Right(Result(1,1168)), Right(Result(5,1180)), Left(java.lang.Throwable: java.lang.Throwable: TimeOut!)
// )))

Simulating a non-CPU-bound task

Rather than keeping the CPU busy (thus CPU-bound), a non-CPU-bound asynchronous task does not demand extensive processing resource. The following snippet defines a method that mimics a non-CPU-bound asynchronous task which could be, say, a non-blocking call to a remote database. This time, we’re going to run on an Akka Actor system, using the ExecutionContext that comes with its default dispatcher. Besides the Fork/Join Executor provided by the dispatcher, we pick the Akka runtime library also to leverage its high-throughput scheduler.

In this example, a Promise which contains a Future is created and after a random duration, the scheduler triggers the completion of the Future with success or failure depending on the random time.

import scala.concurrent.{ExecutionContext, Future, Promise, Await}
import scala.concurrent.duration._
import akka.actor.ActorSystem
implicit val system = ActorSystem("system")
implicit val ec = system.dispatcher

def nonCPUbound[T](value: T): Future[T] = {  
  import java.util.concurrent.ThreadLocalRandom
  val millis = ThreadLocalRandom.current.nextLong(1000, 2000)
  val millisTMO = 1500
  val promise = Promise[T]

  system.scheduler.scheduleOnce(FiniteDuration(millis, MILLISECONDS)) {
    if (millis < millisTMO) {
      println(s"OK: Got value [${value}] in ${millis}ms")
      promise.success(value)
    } else {
      println(s"TimeOut: It would take ${millis - millisTMO}ms above limit to complete!")
      promise.failure(new Throwable("TimeOut!"))
    }
  }

  promise.future
}

Launching method `nonCPUbound()` with some value a few times would yield results similar to the following:

Await.result(nonCPUbound("something"), 2.seconds)

// * Test running #1:

// OK: Got value [something] in 1259ms
// res1: String = something

// * Test running #2:

// TimeOut: It would take 384ms above limit to complete!
// [ERROR] ... java.lang.Throwable: TimeOut!

CPU-bound versus non-CPU-bound tasks

By wrapping a CPU-bound task like `doWork()` with a Future, the task becomes non-blocking but it still consumes processing power. The default ExecutionContext via the `scala.concurrent.ExecutionContext.Implicits.global` import will optimally set `scala.concurrent.context.maxThreads` to the number of CPU cores of the machine the application resides on. One can raise the `maxThreads` and handcraft a custom `ExecutionContext.fromExecutor(Executors.newFixedThreadPool(numOfThreads))` to allow more threads to be run. To set the value of `maxThreads` to, say 16, simply add the following `javaOptions` to `build.sbt`.

javaOptions += "-Dscala.concurrent.context.maxThreads=16"

However, that wouldn’t necessarily make more instances of `Future{ doWork() }` than the number of CPU cores execute in parallel since each of them consumes CPU resource while executing.

On the other hand, a non-CPU-bound task like `nonCPUbound()` takes little CPU resource. In this case, configuring an ExecutionContext with more threads than the CPU cores of the local machine can increase performance, since none of the individual tasks would consume anywhere near the full capacity of a CPU core. It’s not uncommon to configure a pool of hundreds of threads to handle a large amount of such tasks on a machine with just a handful of CPU cores.

Futures or Promises?

While the Scala Future API extensively utilizes Promises in its function implementations, we don’t need to explicitly use Promises very often as the Futures API already delivers a suite of common concurrent features for writing asynchronous code. If the business logic doesn’t need Promises, just stick to the Futures API. But for cases in which you need to provide a “contract to be fulfilled at most once in the future”, say, between two modules like the `producer/consumer` example above, Promises do come in handy.

HTTPS Redirection With Akka HTTP

Akka HTTP is a HTTP-based toolkit built on top of Akka Stream. Rather than a framework for rapid web server development, it’s principally designed as a suite of tools for building custom integration layers to wire potentially complex business logic with a REST/HTTP interface. Perhaps for that reason, one might be surprised that there isn’t any example code for something as common as running a HTTPS-by-default web server.

Almost every major website operates using the HTTPS protocol by default for security purpose these days. Under the protocol, the required SSL certificate and the bidirectional encryption of the communications between the web server and client does ensure the authenticity of the website as well as avoid man-in-the-middle attack. It might be an over-kill for, say, an information-only website, but the ‘lock’ icon indicating a valid SSL certificate on the web browser address bar certainly makes site visitors feel more secure.

In this blog post, I’ll assemble a snippet using Akka HTTP to illustrate how to set up a skeletal web server which redirects all plain-HTTP requests to the HTTPS listener. For testing purpose in a development environment, I’ll also include steps of creating a self-signed SSL certificate. Note that such self-signed certificate should only be used for internal testing purpose.

HTTP and HTTPS cannot serve on the same port

Intuitively, one would consider binding both HTTP and HTTPS services to the same port on which all requests are processed by a HTTPS handler. Unfortunately, HTTPS uses SSL/TLS protocol which for security reason can’t be simply downgraded to HTTP upon detecting unencrypted requests. A straight-forward solution would be to bind the HTTP and HTTPS services to separate ports and redirect all requests coming into the HTTP port to the HTTPS port.

First let’s create ‘build.sbt’ with necessary library dependencies under the project root subdirectory:

name := "akka-http-secureserver"

version := "1.0"

scalaVersion := "2.11.12"

libraryDependencies ++= Seq(
  "com.typesafe.akka" %% "akka-actor" % "2.4.20",
  "com.typesafe.akka" %% "akka-stream" % "2.4.20",
  "com.typesafe.akka" %% "akka-http" % "10.0.11"
)

Next, create the main application in, say, ${project-root}/src/main/SecureServer.scala:

import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import akka.http.scaladsl.model.{HttpEntity, ContentTypes, StatusCodes}
import akka.http.scaladsl.{Http, ConnectionContext, HttpsConnectionContext}
import akka.http.scaladsl.server.Directives._
import com.typesafe.sslconfig.akka.AkkaSSLConfig
import java.security.{SecureRandom, KeyStore}
import javax.net.ssl.{SSLContext, KeyManagerFactory, TrustManagerFactory}
import java.io.InputStream
import scala.io.StdIn

object SecureServer {
  def main(args: Array[String]) {

    implicit val system = ActorSystem("my-system")
    implicit val materializer = ActorMaterializer()
    implicit val ec = system.dispatcher

    val password: Array[Char] = "mypassword".toCharArray  // Unsafe to provide password here!

    val ks: KeyStore = KeyStore.getInstance("PKCS12")
    val keystore: InputStream = getClass.getClassLoader.getResourceAsStream("server.p12")

    require(keystore != null, "Keystore required!")
    ks.load(keystore, password)

    val keyManagerFactory: KeyManagerFactory = KeyManagerFactory.getInstance("SunX509")
    keyManagerFactory.init(ks, password)

    val tmf: TrustManagerFactory = TrustManagerFactory.getInstance("SunX509")
    tmf.init(ks)

    val sslContext: SSLContext = SSLContext.getInstance("TLS")
    sslContext.init(keyManagerFactory.getKeyManagers, tmf.getTrustManagers, new SecureRandom)
    val httpsContext: HttpsConnectionContext = ConnectionContext.https(sslContext)

    val hostName = "dev.genuine.com"
    val portHttp = 8080
    val portHttps = 8443

    val route =
      scheme("http") {
        extract(_.request.uri) { uri =>
          redirect( uri.withScheme("https").withAuthority(hostName, portHttps),
            StatusCodes.MovedPermanently
          )
        }
      } ~
      pathSingleSlash {
        get {
          complete( HttpEntity( ContentTypes.`text/html(UTF-8)`,
            "Welcome to Akka-HTTP!"
          ) )
        }
      } ~
      path("hello") {
        get {
          complete( HttpEntity( ContentTypes.`text/html(UTF-8)`,
            "Hello from Akka-HTTP!"
          ) )
        }
      }

    Http().bindAndHandle(route, hostName, portHttp)
    Http().bindAndHandle(route, hostName, portHttps, connectionContext = httpsContext)

    println(s"Server online at https://${hostName}:${portHttps}/\nPress RETURN to stop...")
    StdIn.readLine()

    system.terminate()
  }
}

The top half of the main code are initialization routines for the Akka actor system, stream materializer (which are what Akka HTTP is built on top of) and creating HTTPS connection context. The rest of the code is a standard Akka HTTP snippet with URL routing and server port binding. A good portion of the code is borrowed from this Akka server-side HTTPS support link.

Within the ‘scheme(“http”)’ routing code block is the core logic for HTTPS redirection:

    // HTTPS redirection
    uri =>
      redirect( uri.withScheme("https").withAuthority(hostName, portHttps),
        StatusCodes.MovedPermanently
      )

Note that there is no need for applying ‘withAuthority()’ if you’re using standard HTTPS port (i.e. 443).

Next step would be to put in place the PKCS #12 formatted file, ‘server.p12’, which consists of the PKCS private key and X.509 SSL certificate. It should be placed under ${project-root}/src/main/resources/. At the bottom of this blog post are steps for creating the server key/certificate using open-source library, OpenSSL.

Once the private key/certificate is in place, to run the server application from a Linux command prompt, simply use ‘sbt’ as below:

# Run SecureServer
cd ${project-root}
sbt "runMain SecureServer"

To test it out from a web browser, visit http://dev.genuine.com:8080/hello and you should see the URL get redirected to https://dev.genuine.com:8443/hello. The web browser will warn about security of the site and that’s just because the SSL certificate is a self-signed one.

Generating server key and self-signed SSL certificate in PKCS #12 format

#
# Steps to generate private key and self-signed X.509 certificate in PKCS #12 format
#

## Generate private key
openssl genrsa -des3 -out server.key 2048

# --
Generating RSA private key, 2048 bit long modulus
..................................+++
.......................................................+++
e is 65537 (0x10001)
Enter pass phrase for server.key: genuine
Verifying - Enter pass phrase for server.key:
# --

## Generate CSR
openssl req -new -key server.key -out server.csr

# --
Enter pass phrase for server.key:
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:California
Locality Name (eg, city) []:Sunnyvale
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Genuine
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:dev.genuine.com
Email Address []:postmaster@genuine.com
A challenge password []:
# --

## Remove pass phrase
cp server.key server.key.orig
openssl rsa -in server.key.orig -out server.key

# --
Enter pass phrase for server.key.orig:
writing RSA key
# --

## Generate certificate
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

# --
Signature ok
subject=/C=US/ST=California/L=Sunnyvale/O=Genuine/CN=dev.genuine.com/emailAddress=postmaster@genuine.com
Getting Private key
# --

## Convert to PKCS #12 or PFX format
openssl pkcs12 -export -out server.p12 -inkey server.key -in server.crt 

# --
Enter Export Password:
Verifying - Enter Export Password:
# --

## Move the PKCS #12 file to the server application resources subdirectory
mv server.p12 ${project-root}/src/main/resources/