The Hyperledger consortium has published a whitepaper that defines performance metrics for blockchains. The metrics should apply to most enterprise blockchains, not just Hyperledger solutions. Broad participation is partly the result of the recent cooperation with the Enterprise Ethereum Alliance and also Hyperledger’s large membership that includes R3 and Digital Asset.
Blockchains and distributed ledger technologies have numerous benefits but are less performant than centralized databases. That’s because work has to be duplicated on multiple servers or nodes. Additionally, each blockchain technology has unique characteristics. Nonetheless, people want to try to compare like-for-like.
However, the white paper points out that before creating comparable benchmarks, it’s necessary to agree on standard definitions about what to measure. Hence the paper focuses on the metrics as a precursor towards benchmarks.
Blockchain metrics
A transaction involves changing data on a blockchain. For any database, blockchain or otherwise, performance metrics relate to reading and writing data. Hence the paper defines two metrics for reading and two for writing.
The most significant bottleneck with blockchains relates to writing or blockchain transactions.
The big difference with a blockchain compared to a database is that it’s not just a matter of copying data to each server. Additionally, the nodes have to arrive at a consensus that the transaction is acceptable. The combined delay of these two steps is called Transaction Latency. Or more formally “a network-wide view of the amount of time taken for a transaction’s effect to be usable across the network.”
Transaction Throughput is generally known as transactions per second (TPS). It’s the rate at which valid transactions are committed by the blockchain network in a defined period, usually seconds. The commit rate is for all nodes, not just a single node.
If you compare the two metrics to the performance of a vehicle, Latency might be how long it takes to reach 60 mph, and Throughput is the top speed.
The document also specifies other factors to evaluate performance and the need for test results to be reproducible. For example, if the nodes are all in the same room rather than spread across the globe, then clearly performance will be better. The hardware used on the nodes will have a significant impact, and so on.
Enterprise blockchain standards
In reality, the paper isn’t big news because these measures are already widely used. However, there are some nuances for enterprise blockchains because they use different consensus mechanisms. For example, one might measure transaction latency on public blockchains as when a transaction is available on 90% of nodes. But for widely used enterprise consensus mechanisms, the threshold will often be 100%.
While this may seem like a baby step, nonetheless the document is a crucial step toward creating standards.
