credo-ts: Performance degradation on networks with large ammount of nodes
TL;DR
Although it seems that ledger network performance doesn’t directly impact a wallet’s ability to complete a transaction (such as accepting a credential), the number of network connections does matter. Wallets with fewer connections to the ledger network tend to complete transactions noticeably faster. This highlights the importance of optimizing the number of network connections for efficiency.
Problem
When using Aries Bifold, BC Wallet, or AFJ to “Accept” a credential, the process can become frustratingly slow on ledgers with many nodes.
Analysis
The table below provides information about test results. These tests were conducted on the sovrin:staging environment using the LSBC Test credential.
In testing process, the same tests were run three times for each platform. However, in the table, we’ve displayed only the two best results for each platform, with the exception of Orbi Edge.
For Orbi Edge, the first test result is shown, even though it was slower. This initial test showed a notably higher number of network connections. Subsequent tests for Orbi Edge were more optimized.
In the table, you’ll find two key columns:
- “Network”: This column indicates the number of network connections that were established during the test.
- “Duration”: This column provides information about the time it took for the wallet to register either “success” or “failure” when accepting the credential.
| No. | Platform | Network | Duration | Comment |
|---|---|---|---|---|
| 1 | Bifold iOS | 28 | 28 sec | AFJ/IndyVDR |
| 2 | Bifold iOS | 36 | 23 sec | AFJ/IndyVDR |
| 3 | Trinsic iOS | 206 | 60 sec | Fail |
| 4 | Trinsic iOS | 209 | 60 sec | Fail |
| 5 | Lissi iOS | 230 | 60 sec | Fail |
| 6 | Lissi iOS | 242 | 60 sec | Fail |
| 7 | Node.js Linux | AFJ/IndyVDR | ||
| 8 | Node.js Linux | AFJ/IndyVDR | ||
| 9 | Orbi Edge iOS | 7 | 5 sec | |
| 10 | Orbi Edge iOS | 2 | 3 sec |
During testing, both Trinsic and Lissi encountered issues when attempting to accept the credential. After waiting for ~60 seconds, an error message appeared.
It was noted via logging that when accepting a credential, AFJ makes a series of network calls, which are logged as follows:
- Get credential definition
- Get transaction
- Get credential definition
- Get transaction
- Get schema
- Get revocation registry definition
To accurately assess network performance and duration, these calls were replicated using the IndyVDR Proxy and cURL. The results of these tests are documented in the table below.
| No. | Platform | Network | Duration | Comment |
|---|---|---|---|---|
| 1 | IndyVDR Linux | 8 | 4 sec | |
| 2 | IndyVDR Linux | 10 | 4 sec |
It is not know what framworks Orbi Edge, Lissi, or Trinsic use.
In the case of Lissi and Trinsic, our observations indicate that they scan all configured ledgers while in the process of accepting a credential. This thorough scanning approach likely played a role in reaching a timeout at approximately 60 seconds, resulting in a test failure.
NOTE Preliminary testing with Trinsic showd it getting similar results to Orbi Edge, however, after a re-install this was not the case and the above results were collected.
Conclusion
This slowness seems to stem from AFJ or IndyVDR making numerous network calls. While each call is quick on its own, the cumulative effect can lead to significant delays when considering response processing.
Our test results reveal some key insights:
-
Ledger Network Performance: It’s worth noting that ledger network performance doesn’t seem to significantly impact the results, as evidenced by the efficient performance of IndyVRD Linux and Orbi Edge. Orbi Edge impressively completes transactions in just 3 seconds. Even when we eliminate duplicate network calls from IndyVDR Linux, it achieves similar results.
-
Number of Network Connections: On the other hand, the number of network connections appears to be a notable factor in test results. Tests that establish fewer connections, possibly the minimum required, tend to complete significantly faster compared to those that create multiple network connections.
In light of these findings, we recommend that AFJ and IndyVDR consider optimizing their ledger network interactions by:
-
Removing Duplicate Network Calls: Identify and eliminate any duplicate network calls that are part of the same transaction to reduce redundancy.
-
Batching Queries: Implement a strategy to batch queries, sending them to the same two nodes over the same connection, rather than establishing multiple network connections for each query.
-
Network Reconciliation: Consider periodic network reconciliation or scheduling intervals for this process, separating it from critical transactions such as accepting a credential or processing proof requests.
-
Caching Transactions: Explore the possibility of caching ledger transactions, given their immutability, to improve efficiency.
These optimizations could enhance the overall performance of ledger network interactions. Thank you for your attention to these recommendations, which aim to streamline the process for a smoother user experience.
How To Reproduce
Use a demo on Sovrin Test/Staging. If you don’t have one, use this email verification service. Watch your firewall logs, time the result. If you have a pfSense based router with pfTop you can run this filter: tcp dst port 9700||9702||9744||9777||9778||9799 and out.
About this issue
- Original URL
- State: open
- Created 8 months ago
- Reactions: 3
- Comments: 18 (16 by maintainers)
@Patrik-Stas When I rand the test I recorded the IP addresses of the nodes IndyVDR (AFJ) was connecting to and they matched the currently active nodes on Sovrin Staging/Test. Or another way, when you “Accept” a credential, IndyVDR only only connects to Active nodes. Also, reconciling the ledger (figuring out what nodes are active and what ones are not) takes place during initialization, not during credential acceptance.
For example, in test 1 above connected to these nodes comprising the 28 connections (I removed duplicates):
All these nodes are active ledger nodes, except these two, which are removed in subsequent blocks. The genesis block I was using was two transactions each of which removed one of these nodes below.
They way I understand how the ledgers work is IndyVDR will use the provided genesis block as a starting point to establish network connections. The genesis block may be behind so it will fetch any new transactions and rebuild the current network state (what nodes are alive and what IPs they are on). Then it uses this list to make queries. Even if I remove a few IPs from the genesis block its still going to get them from other nodes as the ledger is always the source of truth. Also, from what we know of IndyVDR it does not automatically rebuild the network once it does it’s initial startup and reconciliation.