spikeinterface: `BlockingIOError` when attempting to use `n_jobs>1` when writing a binary file.

I get the following error whenever running export_to_phy() using n_jobs > 1 and chunk_duration = "100s" on a 8,000 second dataset:

BlockingIOError: [Errno 11] Resource temporarily unavailable

I’m on Linux and have the nofile limit set to 1,000,000. I first encountered this error during preprocessing steps, but found a combination of n_jobs and chunk_duration parameters that kept the number of open files below the limit without filling up RAM. However, the exporter function throws the same error when run on more than one core, which means the PC extraction step can’t be parallelized and is extremely slow. Any thoughts as to what’s happening?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (10 by maintainers)

Most upvoted comments

Cool. So this is an issue with the overall parallelization and not just the PC step. That’s really good to know. Are you changing any other job_kwargs (could you just list all the job_kwargs you put in? )