redb: Significant Slowdown when Upgrading from 0.13.0 to 0.20.0
While working on the upgrade from redb 0.13.0 to redb 0.20.0 for ord in https://github.com/ordinals/ord/pull/2141, we have encountered a very significant slowdown on our workload. Indexing signet with --index-sats takes about 19 minutes on my Mac with 0.13.0 and takes 6 hours and 45 minutes with 0.20.0. Process sys time goes up by a factor of 400. Details of what we changed for the upgrade are in the PR. We also tried with redb::Durability::Eventual - still running, but does not seem significantly better.
Before (0.13.0):
time target/release/ord --signet --index-sats index
1095.31s user 35.47s system 99% cpu 18:53.03 total
After (0.20.0):
time target/release/ord --signet --index-sats index
5543.28s user 14072.40s system 80% cpu 6:45:53.23 total
These are just our initial findings, will do more investigation. We suspect this is due to the change in https://github.com/cberner/redb/blob/master/CHANGELOG.md#0140---2023-03-26 removing mmap. Any ideas for how to get back to old speed would be much appreciated. As it stands, this would be prohibitively slow to index mainnet - takes good 12 hours currently, and would go up to more than a week if we see the same slowdown.
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 1
- Comments: 19 (11 by maintainers)
I fixed the added overhead from removing
WriteStrategyin https://github.com/cberner/redb/pull/609This brings it down to:
Next I’m going to work on the caching heuristic, which will hopefully remove the need to use a 100GB cache size
@veryordinally , so I noticed one thing that ord may be doing inefficiently. This
value_cacheis a HashMap so the insertion order will be arbitrary. If you can insert them in the same order that they are sorted in redb you’ll get better cache hits when updating the file, because values are stored sorted on diskOk, redb 0.22.0 should be much faster. I was able to index mainnet up to height 790k using an 8GB cache in 31hrs 42min
I created a branch which uses a BTreeMap for both the value cache and the range cache. I didn’t benchmark because I’m on a different machine, so it wouldn’t be apples to apples, so try it and see if it makes a difference: https://github.com/casey/ord/tree/ordered-insertion
One downside is that unlike a HashMap, BTreeMap storage can’t be drained and reused, so this will reallocate the caches when they’re committed, which might degrade performance.
Some findings: it looks like this is mainly a perf regression switching from mmap to the pure userspace backend. Which is rather unfortunate, because that was required to make redb safe. Hopefully, I can optimize this to get that performance back.
Findings below: Indexed up to block 100k on signet:
edit added the last item in the list. Seems like increasing the cache size helps a lot.
I’m able to reproduce it on my Linux desktop using signet. Will dig in shortly!