annoy: Can't pickle annoy.Annoy objects
Hello,
I have an Issue when I try to pickle an annoy object. Here Is the code generating the exception:
def save_checkpoint(self, checkpoint_prefix): super().save_checkpoint(checkpoint_prefix) with open(os.path.join(self.ap.task_parameters.checkpoint_save_dir, str(checkpoint_prefix) + '.dnd'), 'wb') as f: pickle.dump(self.networks['main'].online_network.output_heads[0].DND, f, pickle.HIGHEST_PROTOCOL)
This code is comming from the nec_agent of Intel / NervanaSystems Coach Deep reinforcement learning librarie.
I’m running that code on:
- windows 10 64bits
- Python 3.6.5
- annoy 1.15.0
Thank for the help,
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (2 by maintainers)
sorry, the main point of using annoy is that it uses mmap to load a file from disk into memory very quickly. that requires the file to be on disk initially. not sure why you can’t just download the file to a temporary file and load it that way?
Sorry to resurect this old thread. My usecase is that I’d like to run multiple annoy retrievals in parallel using Python joblib, and that requires the annoy index to be available to each process worker. joblib then tries to pickle the object and send it to each process.
Any ideas on how to deal with this, given that annoy objects cannot be pickled?
@erikbern I am trying to read my saved Annoy model from Google Cloud Storage, but the
loadmethod seems to require me to have a path as opposed to being able to just read the file from Storage (from a pickled object, for example). Is there a way to load my model without needing a direct path?