pyvista: v=0.30.1 Multiblock creation/appending is dramatically faster than v=0.32.1 when adding meshes.

A minimally reproducible example can be used below for testing.

import pyvista as pv
import numpy as np
from time import perf_counter as pf

def create_spline():
    line = np.random.randint(0, 100, (20, 3))
    spline = pv.Spline(line)
    tube = spline.tube(radius=1)
    return tube
    

splines = [create_spline() for _ in range(1000)]

t = pf()
block = pv.MultiBlock(splines)

mesh = block.combine()
print (pf() - t)
mesh.plot()

The choice of a tube mesh can be replaced here with any other mesh and the problem still persists.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 15 (15 by maintainers)

Most upvoted comments

First take: https://github.com/pyvista/pyvista/pull/1805. We should get this right so I don’t expect it to be merged too soon. We’ll have to make sure we’re not breaking something or missing important corner cases.

adeak on Nov 15, 2021

So quick to identify this.

To be fair I haven’t proven it yet, but it’s a very very good guess 😃

Is there anything I can do to help?

Probably not, but the fact that you’ve flagged this is already huge help for us 😃 Thank you.

Looking at the implementation of MultiBlock, there are multiple things that jump out at me. When you initialise a MultiBlock from a sequence of meshes, .append is called with each mesh. In turn this does self[index] = block, which calls __setitem__.

However __setitem__ ends with https://github.com/pyvista/pyvista/blob/828fa349134e036efac9f47852a1bed4bf51c929/pyvista/core/composite.py#L471-L472

Note that self.refs is a list. So first we do an O(n) sweep over the list to see if the new data is in there. Then if it’s not in there, we call append. Then at the end of MultiBlock.append we do that again… https://github.com/pyvista/pyvista/blob/828fa349134e036efac9f47852a1bed4bf51c929/pyvista/core/composite.py#L336-L337

And there has to be one more leak here:

>>> import pyvista as pv
>>> mb = pv.MultiBlock([pv.Sphere()])
>>> len(mb.refs)
3

The first two items are the mesh itself, and the third item is the mesh in a one-element tuple…

So there’s a lot of cruft in the implementation, and now that membership comparison triggers the new heavy __eq__ instead of object.__eq__ which was blasing-fast based on self is other, this is becoming obvious. We’ll have to refactor MultiBlock to fix this.

adeak on Nov 15, 2021

Aaaaah I know what this is!

After https://github.com/pyvista/pyvista/pull/1593:

>>> import pyvista as pv
>>> pv.Sphere() == pv.Sphere()
True

Preceding commit:

>>> import pyvista as pv
>>> pv.Sphere() == pv.Sphere()
False

In between __eq__ was implemented for datasets. This recurses into the attributes of meshes, which ends up being slow if you do it a lot of times. I bet MultiBlock creation triggers this for each submesh for whatever reason.

adeak on Nov 15, 2021

This was broken by https://github.com/pyvista/pyvista/pull/1593/. I’m not very happy about this, because that was a major PR with 2800 lines lines touched across 57 files 😰

I’ll try to figure out what’s off here. The fact that scalars are missing is a good sign I think. And in any case MultiBlocks are more of less localised.

adeak on Nov 15, 2021

Hold on, something’s weird here. I recreated the environment and the timing script from scratch to answer your question… and now I get 0.1 second runtime with the old setup! And I see scalars, which weren’t there before. Weird. I’m starting to think I messed up something in my environment during my first attempt… sorry about that.

For the sake of exactness:

Old setup

--------------------------------------------------------------------------------
  Date: Mon Nov 15 19:39:20 2021 CET

                OS : Linux
            CPU(s) : 8
           Machine : x86_64
      Architecture : 64bit
       Environment : Python
        GPU Vendor : Intel
      GPU Renderer : Mesa Intel(R) UHD Graphics 620 (WHL GT2)
       GPU Version : 4.6 (Core Profile) Mesa 20.3.5

  Python 3.8.7 (default, Jan 27 2021, 23:12:13)  [GCC 10.2.1 20210110]

           pyvista : 0.30.1
               vtk : 9.0.1
             numpy : 1.21.4
           imageio : 2.10.4
           appdirs : 1.4.4
            scooby : 0.5.7
            meshio : 4.4.6
--------------------------------------------------------------------------------

Runtime 0.1 second.

Newer setup, only updating pyvista:

--------------------------------------------------------------------------------
  Date: Mon Nov 15 19:44:49 2021 CET

                OS : Linux
            CPU(s) : 8
           Machine : x86_64
      Architecture : 64bit
       Environment : Python
        GPU Vendor : Intel
      GPU Renderer : Mesa Intel(R) UHD Graphics 620 (WHL GT2)
       GPU Version : 4.6 (Core Profile) Mesa 20.3.5

  Python 3.8.7 (default, Jan 27 2021, 23:12:13)  [GCC 10.2.1 20210110]

           pyvista : 0.32.1
               vtk : 9.0.1
             numpy : 1.21.4
           imageio : 2.10.4
           appdirs : 1.4.4
            scooby : 0.5.7
            meshio : 4.4.6
--------------------------------------------------------------------------------

Runtime 240 seconds.

I checked pyvista 0.31.1 as a first step of bisecting, and it’s also fast (even slightly faster than the oldest version). So this indeed on our side, thanks for finding it. I’ll track down the change that broke this.

As an aside I wouldn’t be too surprised at different runtimes, because my laptop isn’t very strong either, and it doesn’t have a discrete GPU.

adeak on Nov 15, 2021