On Thursday the Deutsche Börse and central bank Deutsche Bundesbank announced the successful completion of tests for their blockchain prototypes. However, while the press release gave no hint of any issues, the report accompanying the announcement showed a significant technical problem during one of the experiments.
Tests were performed with Hyperledger Fabric and Digital Asset’s technology. The Deutsche Börse is a member of Hyperledger, part of the Linux Foundation, and an investor in Digital Asset (DA). “The tests showed that both prototypes are suitable for the productive operation of a realistic financial market infrastructure,” the announcement states.
Ledger Insights disagrees and spoke with the Deutsche Börse team, which stood by its findings.
The tests with Digital Asset’s technology went well, though these were performed by Digital Asset themselves, not by the Deutsche Börse. In the past two weeks, two sets of positive third-party results have also been published for DA’s technology. Ledger Insights has no issue here.
However, a closer look reveals that in the case of the Hyperledger Fabric tests performed by Deutsche Börse, the technology demonstrated the ability to process high transaction rates but results showed that many transactions failed.
There is a fix being worked on which should address the issue, but it wasn’t tested and isn’t yet publicly available. Hence objectively, it’s arguable the tests only proved throughput and neither accuracy nor suitability. The Deutsche Börse might reasonably be of the opinion that Fabric is suitable or will be soon. But that’s not the statement made in the press release.
Is Ledger Insights splitting hairs? As an analogy, imagine if a car company published results saying its auto is capable of 180 mph. Now there might be vibration at that speed which the company neglects to mention. But without that knowledge, you assume that the driver still has control and can safely stop the car. We contend that if the auto is unsafe at 180mph, a company shouldn’t say the vehicle is capable of that speed. Similarly here, if this went into production it has the potential to materially affect a trader’s finances.
Furthermore, this isn’t a startup putting a positive spin on a new feature. It’s a press release by the stock exchange operator and regulator relating to tests. Test results need to be factual, just as figures in financial announcements need to be accurate.
One important caveat. We aren’t making an assertion that Deutsche Börse is trying to hide anything. On the contrary, it has been transparent in the report and Deutsche Börse was forthright in conversation. Its report was also far more detailed compared to the DTCC‘s test results a couple of weeks ago, which is to be lauded. Many institutions such as Project Jasper and the Monetary Authority of Singapore are sharing research results, which helps to accelerate learning across the blockchain industry.
This is merely a debate about conclusions, particularly in press releases. And in this case, we remain cautious about the wording of the stated results.
The tests
Financial market infrastructures are more technically demanding than your average blockchain.
Jens Hachmeister, Deutsche Börse’s Managing Director for DLT, Crypto Assets and New Market Structures outlined the aim of the research to Ledger Insights: “The main reason for the test and the main goal was, first of all, especially from a German federal bank point of view, to understand whether this kind of technology would, in principle, be sufficient and adequate to run delivery versus payment processes in a certain magnitude.”
Delivery versus payment in this context involves having both an asset and tokenized cash on the distributed ledger so that a trade can settle instantly rather than waiting for a payment to clear the bank.
The magnitude refers to both latency and throughput in transactions per second. Latency is similar to a busy website where sometimes there’s a delay in a page loading.
Hachmeister continued: “And what we did, more or less, was we used Hyperledger Fabric as it stands, off the shelf. There was no tuning, no nothing.” That compares to most financial market software which is heavily customized.
While Hyperledger Fabric 1.0 (not the latest version) met these load demands, it also experienced transaction conflicts which resulted in many transactions failing. Given these blockchain systems are intended for market trading, it’s not acceptable for a trade to fail and be retried a little later. Prices move too quickly.
A known problem
There is a known Hyperledger Fabric issue regarding concurrency or dealing with transactions simultaneously.
“The fact is that those issues that we have seen were known to IBM and at this stage, they were already working to resolve these issues with the next version of Hyperledger Fabric,” Hachmeister explained. (For clarity, these tests were run by the Deutsche Börse team, not IBM.)
In 2016, IBM and Digital Asset contributed Hyperledger Fabric to the open-source Hyperledger Project.
Hyperledger’s public task manager shows the task as work in progress but unresolved. Ledger Insights contacted the task owner, Sheehan Anderson, who works for State Street. He previously worked for IBM and was one of the co-founders of Hyperledger Fabric in 2015-16. The solution proposed is his, though several IBMers have since contributed.
A solution was originally targeted for Hyperledger Fabric version 1.3 which was made available earlier this month but has been pushed to version 2, due for release in the first quarter of 2019.
Asked whether the task would fix the Deutsche Börse issue, Anderson’s email response stated: “I can’t say with certainty that it will resolve the problem without more technical detail, though it’s likely related.”
Hachmeister explained that it was decided at the start of the research that tests wouldn’t be re-run with later versions. The sector is fast moving, and any results will always be out of date. Moreover, according to Fouad Aberkane, Lead Architect and Technology Lead DLT/Blockchain at Deutsche Börse, milestones would need to be rescheduled repeatedly to accommodate new releases.
The report stated: “Therefore, we conclude that Hyperledger Fabric seems to be sufficiently performant in order to consider it for a productive implementation for the use case under consideration, conditional on the fact that IBM’s (at this point in time) proprietary solution for the architectural issue works sufficiently well.”
Was it a success?
Taking account of these points, Ledger Insights asked Hachmeister whether the tests were a success. He stood by the conclusions because the objectives for the tests were latency and transactions per second. “To what extent does this scenario meet certain criteria in terms of throughput and latency? And not the question how many errors would the software produce? You have to take a step back to go back to the goal of this exercise,” he said.
“One can say that despite the errors, which occurred within Hyperledger Fabric 1.0, still the test requirements were met under the test conditions,” continued Hachmeister. “It would be unfair to Hyperledger to state that it failed and didn’t meet our defined test criteria. The test criteria we had defined were met.” And he stated the error rate would have been lower in version 1.1.
However, in discussing different technologies, Hachmeister recognizes the importance of both scalability and reliability in the context of regulated market infrastructure. “We will always choose systems which not only survive things in terms of scalability but as well have the reliability to be the underlying technology for regulated market infrastructure even in the new context of distributed ledger,” said Hachmeister.
To reiterate, Deutsche Börse has been entirely transparent. But it’s Ledger Insights’ position that the press release’s conclusion that tests demonstrated suitability for financial markets was not completely accurate. We hold the two venerable names on that announcement to a higher standard.
Disclosure: Ledger Insights is a media partner for the Hyperledger Global Forum.
