storm: slow inserts/saves
I am calling db.Save(&model)
to save (or rather insert) two million objects.
This is incredibly slow. Is there a faster/better way of doing this with storm
?
About this issue
- Original URL
- State: open
- Created 6 years ago
- Comments: 28 (9 by maintainers)
Commits related to this issue
- replace Storm with home-built solution because perf issues https://github.com/asdine/storm/issues/222#issuecomment-472791001 — committed to function61/varasto by joonas-fi 5 years ago
@mvdan @AndersonBargas This library is not maintained these days because I’m putting all or my energy in another database which solves lots of Storm design issues (no reflection, SQL as primary API, works with Bolt and Badger), if you like Storm I definitely recommend taking a look at Genji.
Regarding Storm, even if I’m not actively working on it, I definitely accept PRs, especially regarding performance and stability
It was just an example because Batch is usually used for concurrent writes accross multiple goroutines (ex: webservers, etc.)
You can use the
Batch()
option Then all the Storm write methods (Save, Update, DeleteStruct, etc.) will use the bolt Batch method. If you only want that for one particular bucket, you can enable the option just for it using thisExample:
Actually this is more related to how BoltDB synchronizes data on disk, the time spent on Storm functions is an order of magnitude lower than the disk sync time. This bbolt thread explains it well https://github.com/coreos/bbolt/issues/94
The same line of thinking let me try the serialization alone. A million serializations took around 14s. Writing with storm I canceled after much much longer. This lets me to believe that the serialization isn’t really the bottle neck.