torrent: Hashing is very slow at 16-32 MB/s
Hello,
I want to use the torrent library in a project that will distribute files across a number of heterogenous storage nodes. I wrote a very simple client, the source code is here: https://gist.github.com/ambientsound/bbc30b258bdd253d5fbaa6ce339e2ff1
Now, after adding a torrent, the client starts hashing. The hash speed is very slow, however, stuck between 16-32 MB/s. The filesystem can handle around 250 MB/s when doing sha1sum from the command line.
I expect 10x the performance I can see from the torrent library, and thus assume that I’m doing something wrong. Can you provide some help?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 22 (10 by maintainers)
I’ll take another look at your pprof, I think the best way forward will be to modify the storage Piece API slightly to allow persistent file descriptor use. That should be enough, but if it isn’t, some custom user-space caching of chunks will be necessary, that’s a larger change.