lenstronomy: MCMC sampler does not work with MPI=False and threadCount > 1

The schwimmbad.choose_pool method, when mpi=False and threadCount > 1 returns a MultiPool object that has no is_master attribute, hence the following error :


  File "/Users/aymericg/Documents/EPFL/PhD_LASTRO/Code/Forked/lenstronomy_forked/lenstronomy/Workflow/fitting_sequence.py", line 94, in fit_sequence
    mcmc_output = self.mcmc(**kwargs)
  File "/Users/aymericg/Documents/EPFL/PhD_LASTRO/Code/Forked/lenstronomy_forked/lenstronomy/Workflow/fitting_sequence.py", line 230, in mcmc
    threadCount=threadCount, progress=progress)
  File "/Users/aymericg/Documents/EPFL/PhD_LASTRO/Code/Forked/lenstronomy_forked/lenstronomy/Sampling/sampler.py", line 149, in mcmc_emcee
    if pool.is_master():
AttributeError: 'MultiPool' object has no attribute 'is_master'

In addition, I see that the new schwimmbad way of dealing with pools calls the multiprocessing module instead of the multiprocess, as it was done before in lenstronomy, and which is more robust because of pickling issues in multiprocessing. @ajshajib , could you please look into that since you introduced this change ?

I did not test for the PSO, but it’s likely to give the same result.

In the meantime, I will temporarily make my own fork of schwimmbad to fix that, but it’s not viable to link lenstronomy to this fork.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20

Commits related to this issue

Most upvoted comments

Ok thanks @sibirrer , I’ll prepare the changes

Ok no problem, but what’s your preference @sibirrer ? I propose a change to schwimmbad master branch, or I write a “hack” in lenstronomy ?

up to you @aymgal. It would be great if we can minimize outside dependence on ‘unstable’ development. If you implement it in lenstronomy, I would not call it a ‘hack’ if you well document the in- and outputs, its current purpose and reason for doing it there.

@ajshajib @aymgal please use the branch multithreading_mpi to suggest changes in lenstronomy.