distributed: Bug introduced in distributed 2022.5.1 -- unpack_frames

What happened:

Seems like potentially two bugs related to each other or one bug that is causing two different flakey behaviors.

First bug is that the compute call is returning an array asking for PiB of memory: https://github.com/AllenCellModeling/aicsimageio/runs/6594900687?check_suite_focus=true#step:8:137

Second bug is that the unpack_frames is requiring a buffer of a certain length but isn’t receiving it: https://github.com/AllenCellModeling/aicsimageio/runs/6594900687?check_suite_focus=true#step:8:709

Ultimately both bugs use the unpack_frames function in the traceback though.

What you expected to happen:

This exact same code ran fine on 2022.5.0: https://github.com/AllenCellModeling/aicsimageio/runs/6563614927?check_suite_focus=true

(I also checked the underlying file reading library version tifffile and it stays the same 2022.5.4 for both the PR CI and the post merge CI.)

Minimal Complete Verifiable Example:

Unlike the last bug I caught via CI (#6255) this one is seems deep in the weeds of worker magic but I do think I have a cuplrit from the changelog.

• https://github.com/dask/distributed/pull/6333 – the actual code for host_array in utils was changed in 2022.5.0 but not in 2022.5.1 however there was a change made in that PR ensures a memory view for each frame to unpack. Memory view -> incorrect allocation size / buffer size seems like a likely culprit?

Anything else we need to know?:

Environment:

• Dask version: 2022.5.1
• Python version: CI tests on 3.8, 3.9, and 3.10, all are failing
• Operating System: CI tests on Windows, MacOS, and Ubuntu – MacOS tests fail fast, the others tests are taking 3 hours so I am assuming something is going wrong with them too.
• Install method (conda, pip, source): pip

Cluster Dump State: