ray: [Bug] Random 'pid=None' console output

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Clusters

What happened + What you expected to happen

A script that connects to a local cluster randomly prints (fun pid=None) during computations. Sometime also ip=some_ip is attached. Can’t find anything in the docs. Any hints what’s going on or how to debug/get more info? In the script I set os.environ["RAY_SCHEDULER_EVENTS"] = "0" to disable some random ray (autoscaler) output. Not sure if this has something to do with this issue, but is the only ray env var I’m changing.

Versions / Dependencies

name: puma-lab channels:

• pyviz
• conda-forge
• defaults dependencies:
• _libgcc_mutex=0.1=conda_forge
• _openmp_mutex=4.5=1_gnu
• abseil-cpp=20210324.2=h9c3ff4c_0
• anyio=3.5.0=py38h578d9bd_0
• aplus=0.11.0=py_1
• argon2-cffi=21.3.0=pyhd8ed1ab_0
• argon2-cffi-bindings=21.2.0=py38h497a2fe_1
• arrow-cpp=6.0.1=py38h44d4a2f_7_cpu
• asgiref=3.4.1=pyhd8ed1ab_0
• astropy=5.0=py38h6c62de6_0
• async_generator=1.10=py_0
• attrs=21.4.0=pyhd8ed1ab_0
• aws-c-cal=0.5.11=h95a6274_0
• aws-c-common=0.6.2=h7f98852_0
• aws-c-event-stream=0.2.7=h3541f99_13
• aws-c-io=0.10.5=hfb6a706_0
• aws-checksums=0.1.11=ha31a3da_7
• aws-sdk-cpp=1.8.186=hb4091e7_3
• babel=2.9.1=pyh44b312d_0
• backcall=0.2.0=pyh9f0ad1d_0
• backports=1.0=py_2
• backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
• backports.zoneinfo=0.2.1=py38h497a2fe_4
• blake3=0.2.1=py38ha8db356_0
• bleach=4.1.0=pyhd8ed1ab_0
• blosc=1.21.0=h9c3ff4c_0
• bokeh=2.4.2=py38h578d9bd_0
• bqplot=0.12.32=pyhd8ed1ab_0
• branca=0.4.2=pyhd8ed1ab_0
• brotli=1.0.9=h7f98852_6
• brotli-bin=1.0.9=h7f98852_6
• brotlipy=0.7.0=py38h497a2fe_1003
• brunsli=0.1=h9c3ff4c_0
• bzip2=1.0.8=h7f98852_4
• c-ares=1.18.1=h7f98852_0
• c-blosc2=2.0.4=h5f21a17_1
• ca-certificates=2021.10.8=ha878542_0
• cached-property=1.5.2=hd8ed1ab_1
• cached_property=1.5.2=pyha770c72_1
• certifi=2021.10.8=py38h578d9bd_1
• cffi=1.15.0=py38h3931269_0
• cfitsio=4.0.0=h9a35b8e_0
• charls=2.2.0=h9c3ff4c_0
• charset-normalizer=2.0.10=pyhd8ed1ab_0
• click=8.0.3=py38h578d9bd_1
• clickhouse-cityhash=1.0.2.3=py38h709712a_3
• clickhouse-driver=0.2.2=py38h497a2fe_1
• cloudpickle=2.0.0=pyhd8ed1ab_0
• colorama=0.4.4=pyh9f0ad1d_0
• colorcet=3.0.0=pyhd8ed1ab_0
• commonmark=0.9.1=py_0
• cramjam=2.5.0=py38ha8db356_0
• cryptography=36.0.0=py38h9ce1e76_0
• cycler=0.11.0=pyhd8ed1ab_0
• cytoolz=0.11.2=py38h497a2fe_1
• dask=2021.12.0=pyhd8ed1ab_0
• dask-core=2021.12.0=pyhd8ed1ab_0
• dataclasses=0.8=pyhc8e2a94_3
• datashader=0.13.0=pyh6c4a22f_0
• datashape=0.5.4=py_1
• debugpy=1.5.1=py38h709712a_0
• decorator=5.1.1=pyhd8ed1ab_0
• defusedxml=0.7.1=pyhd8ed1ab_0
• distributed=2021.12.0=py38h578d9bd_0
• entrypoints=0.3=pyhd8ed1ab_1003
• fastapi=0.71.0=pyhd8ed1ab_0
• fastparquet=0.7.2=py38h6c62de6_1
• filelock=3.4.2=pyhd8ed1ab_1
• flit-core=3.6.0=pyhd8ed1ab_0
• fonttools=4.28.5=py38h497a2fe_0
• freetype=2.10.4=h0708190_1
• frozendict=2.0.6=py38h497a2fe_1
• fsspec=2022.1.0=pyhd8ed1ab_0
• future=0.18.2=py38h578d9bd_4
• geos=3.10.1=h9c3ff4c_1
• gflags=2.2.2=he1b5a44_1004
• giflib=5.2.1=h36c2ea0_2
• gitdb=4.0.9=pyhd8ed1ab_0
• gitpython=3.1.26=pyhd8ed1ab_0
• glog=0.5.0=h48cff8f_0
• grpc-cpp=1.42.0=ha1441d3_1
• h11=0.12.0=pyhd8ed1ab_0
• h5py=3.6.0=nompi_py38hfbb2109_100
• hdf5=1.12.1=nompi_h2750804_103
• heapdict=1.0.1=py_0
• holoviews=1.14.7=py_0
• hvplot=0.7.3=py_0
• icu=69.1=h9c3ff4c_0
• idna=3.1=pyhd3deb0d_0
• imagecodecs=2021.11.20=py38h678ac2f_1
• imageio=2.13.5=pyh239f2a4_0
• importlib-metadata=4.10.0=py38h578d9bd_0
• importlib_metadata=4.10.0=hd8ed1ab_0
• importlib_resources=5.4.0=pyhd8ed1ab_0
• ipydatawidgets=4.2.0=pyhd3deb0d_0
• ipykernel=6.6.1=py38he5a9106_0
• ipyleaflet=0.15.0=pyhd8ed1ab_3
• ipympl=0.8.5=pyhd8ed1ab_0
• ipython=7.31.0=py38h578d9bd_0
• ipython_genutils=0.2.0=py_1
• ipyvolume=0.6.0a8=pyhd8ed1ab_0
• ipyvue=1.7.0=pyhd8ed1ab_0
• ipyvuetify=1.8.1=pyhd8ed1ab_0
• ipywebrtc=0.6.0=pyhd8ed1ab_0
• ipywidgets=7.6.5=pyhd8ed1ab_0
• jbig=2.1=h7f98852_2003
• jedi=0.18.1=py38h578d9bd_0
• jinja2=3.0.3=pyhd8ed1ab_0
• jpeg=9d=h36c2ea0_0
• json5=0.9.5=pyh9f0ad1d_0
• jsonschema=4.3.3=pyhd8ed1ab_0
• jupyter-server-mathjax=0.2.3=pyhd8ed1ab_0
• jupyter_client=7.1.0=pyhd8ed1ab_0
• jupyter_contrib_core=0.3.3=py_2
• jupyter_contrib_nbextensions=0.5.1=pyhd8ed1ab_2
• jupyter_core=4.9.1=py38h578d9bd_1
• jupyter_highlight_selected_word=0.2.0=py38h578d9bd_1005
• jupyter_latex_envs=1.4.6=pyhd8ed1ab_1002
• jupyter_nbextensions_configurator=0.4.1=pyhd8ed1ab_2
• jupyter_server=1.13.1=pyhd8ed1ab_0
• jupyterlab=3.2.6=pyhd8ed1ab_0
• jupyterlab-git=0.34.1=pyhd8ed1ab_0
• jupyterlab_pygments=0.1.2=pyh9f0ad1d_0
• jupyterlab_server=2.10.3=pyhd8ed1ab_0
• jupyterlab_widgets=1.0.2=pyhd8ed1ab_0
• jxrlib=1.1=h7f98852_2
• kiwisolver=1.3.2=py38h1fd1430_1
• krb5=1.19.2=hcc1bbae_3
• lcms2=2.12=hddcbb42_0
• ld_impl_linux-64=2.36.1=hea4e1c9_2
• lerc=3.0=h9c3ff4c_0
• libaec=1.0.6=h9c3ff4c_0
• libblas=3.9.0=12_linux64_openblas
• libbrotlicommon=1.0.9=h7f98852_6
• libbrotlidec=1.0.9=h7f98852_6
• libbrotlienc=1.0.9=h7f98852_6
• libcblas=3.9.0=12_linux64_openblas
• libcurl=7.81.0=h2574ce0_0
• libdeflate=1.8=h7f98852_0
• libedit=3.1.20191231=he28a2e2_2
• libev=4.33=h516909a_1
• libevent=2.1.10=h9b69904_4
• libffi=3.4.2=h7f98852_5
• libgcc-ng=11.2.0=h1d223b6_11
• libgfortran-ng=11.2.0=h69a702a_11
• libgfortran5=11.2.0=h5c6108e_11
• libgomp=11.2.0=h1d223b6_11
• libiconv=1.16=h516909a_0
• liblapack=3.9.0=12_linux64_openblas
• libllvm10=10.0.1=he513fc3_3
• libnghttp2=1.43.0=h812cca2_1
• libnsl=2.0.0=h7f98852_0
• libopenblas=0.3.18=pthreads_h8fe5266_0
• libpng=1.6.37=h21135ba_2
• libprotobuf=3.19.3=h780b84a_0
• libsodium=1.0.18=h36c2ea0_1
• libssh2=1.10.0=ha56f1ee_2
• libstdcxx-ng=11.2.0=he4da1e4_11
• libta-lib=0.4.0=h516909a_0
• libthrift=0.15.0=he6d91bd_1
• libtiff=4.3.0=h6f004c6_2
• libutf8proc=2.7.0=h7f98852_0
• libwebp-base=1.2.1=h7f98852_0
• libxml2=2.9.12=h885dcf4_1
• libxslt=1.1.33=h0ef7038_3
• libzlib=1.2.11=h36c2ea0_1013
• libzopfli=1.0.3=h9c3ff4c_0
• llvmlite=0.36.0=py38h4630a5e_0
• locket=0.2.0=py_2
• lxml=4.7.1=py38hf1fe3a4_0
• lz4-c=1.9.3=h9c3ff4c_1
• markdown=3.3.6=pyhd8ed1ab_0
• markupsafe=2.0.1=py38h497a2fe_1
• matplotlib=3.3.2=0
• matplotlib-base=3.3.2=py38h5c7f4ab_1
• matplotlib-inline=0.1.3=pyhd8ed1ab_0
• mistune=0.8.4=py38h497a2fe_1005
• msgpack-python=1.0.3=py38h1fd1430_0
• multipledispatch=0.6.0=py_0
• munkres=1.1.4=pyh9f0ad1d_0
• nb_conda_kernels=2.3.1=py38h578d9bd_1
• nbclassic=0.3.5=pyhd8ed1ab_0
• nbclient=0.5.9=pyhd8ed1ab_0
• nbconvert=6.4.0=py38h578d9bd_0
• nbdime=3.1.1=pyhd8ed1ab_0
• nbformat=5.1.3=pyhd8ed1ab_0
• ncurses=6.2=h58526e2_4
• nest-asyncio=1.5.4=pyhd8ed1ab_0
• networkx=2.6.3=pyhd8ed1ab_1
• notebook=6.4.6=pyha770c72_0
• numba=0.53.1=py38h8b71fd7_1
• numpy=1.22.0=py38h6ae9a64_0
• olefile=0.46=pyh9f0ad1d_1
• openjpeg=2.4.0=hb52868f_1
• openssl=1.1.1l=h7f98852_0
• orc=1.7.2=h1be678f_0
• packaging=21.3=pyhd8ed1ab_0
• pandas=1.3.5=py38h43a58ef_0
• pandoc=2.16.2=h7f98852_0
• pandocfilters=1.5.0=pyhd8ed1ab_0
• panel=0.12.6=py_0
• param=1.12.0=pyh6c4a22f_0
• parquet-cpp=1.5.1=2
• parso=0.8.3=pyhd8ed1ab_0
• partd=1.2.0=pyhd8ed1ab_0
• patsy=0.5.2=pyhd8ed1ab_0
• pcre=8.45=h9c3ff4c_0
• pexpect=4.8.0=pyh9f0ad1d_2
• pickleshare=0.7.5=py_1003
• pillow=8.4.0=py38h8e6f84c_0
• pip=21.3.1=pyhd8ed1ab_0
• progressbar2=3.53.1=pyh9f0ad1d_0
• prometheus_client=0.12.0=pyhd8ed1ab_0
• prompt-toolkit=3.0.24=pyha770c72_0
• psutil=5.9.0=py38h497a2fe_0
• ptyprocess=0.7.0=pyhd3deb0d_0
• pyarrow=6.0.1=py38he7e5f7d_7_cpu
• pycparser=2.21=pyhd8ed1ab_0
• pyct=0.4.6=py_0
• pyct-core=0.4.6=py_0
• pydantic=1.9.0=py38h497a2fe_0
• pyerfa=2.0.0.1=py38h6c62de6_1
• pygments=2.11.2=pyhd8ed1ab_0
• pykalman=0.9.5=py_1
• pyopenssl=21.0.0=pyhd8ed1ab_0
• pyparsing=3.0.6=pyhd8ed1ab_0
• pyrsistent=0.18.0=py38h497a2fe_0
• pysocks=1.7.1=py38h578d9bd_4
• python=3.8.12=hb7a2778_2_cpython
• python-dateutil=2.8.2=pyhd8ed1ab_0
• python-tzdata=2021.5=pyhd8ed1ab_0
• python-utils=3.0.0=pyhd8ed1ab_0
• python_abi=3.8=2_cp38
• pythreejs=2.3.0=pyhd8ed1ab_0
• pytz=2021.3=pyhd8ed1ab_0
• pytz-deprecation-shim=0.1.0.post0=py38h578d9bd_1
• pyviz_comms=2.1.0=py_0
• pywavelets=1.2.0=py38h6c62de6_1
• pyyaml=6.0=py38h497a2fe_3
• pyzmq=22.3.0=py38h2035c66_1
• re2=2021.11.01=h9c3ff4c_0
• readline=8.1=h46c0cb4_0
• requests=2.27.1=pyhd8ed1ab_0
• rich=11.0.0=pyhd8ed1ab_0
• s2n=1.0.10=h9b69904_0
• scikit-image=0.19.1=py38h43a58ef_0
• scipy=1.7.3=py38h56a6a73_0
• send2trash=1.8.0=pyhd8ed1ab_0
• setuptools=60.5.0=py38h578d9bd_0
• shapely=1.8.0=py38h800f7b8_4
• six=1.16.0=pyh6c4a22f_0
• smmap=3.0.5=pyh44b312d_0
• snappy=1.1.8=he1b5a44_3
• sniffio=1.2.0=py38h578d9bd_2
• sortedcontainers=2.4.0=pyhd8ed1ab_0
• sqlite=3.37.0=h9cd32fc_0
• starlette=0.17.1=pyhd8ed1ab_0
• statsmodels=0.13.1=py38h6c62de6_0
• ta-lib=0.4.19=py38h6c62de6_3
• tabulate=0.8.9=pyhd8ed1ab_0
• tblib=1.7.0=pyhd8ed1ab_0
• terminado=0.12.1=py38h578d9bd_1
• testpath=0.5.0=pyhd8ed1ab_0
• thrift=0.15.0=py38h709712a_1
• tifffile=2021.11.2=pyhd8ed1ab_0
• tk=8.6.11=h27826a3_1
• toolz=0.11.2=pyhd8ed1ab_0
• tornado=6.1=py38h497a2fe_2
• tqdm=4.62.3=pyhd8ed1ab_0
• traitlets=5.1.1=pyhd8ed1ab_0
• traittypes=0.2.1=pyh9f0ad1d_2
• typing-extensions=4.0.1=hd8ed1ab_0
• typing_extensions=4.0.1=pyha770c72_0
• tzdata=2021e=he74cb21_0
• tzlocal=4.1=py38h578d9bd_1
• unicodedata2=14.0.0=py38h497a2fe_0
• urllib3=1.26.8=pyhd8ed1ab_1
• uvicorn=0.16.0=py38h578d9bd_0
• vaex=4.7.0=pyhd8ed1ab_0
• vaex-astro=0.9.0=pyhd8ed1ab_0
• vaex-core=4.7.0.post1=py38hdaa53af_0
• vaex-hdf5=0.11.1=pyhd8ed1ab_0
• vaex-jupyter=0.7.0=pyhd8ed1ab_0
• vaex-ml=0.16.0=pyhd8ed1ab_0
• vaex-server=0.8.0=pyhd8ed1ab_0
• vaex-viz=0.5.1=pyhd8ed1ab_0
• wcwidth=0.2.5=pyh9f0ad1d_2
• webencodings=0.5.1=py_1
• websocket-client=1.2.3=pyhd8ed1ab_0
• wheel=0.37.1=pyhd8ed1ab_0
• widgetsnbextension=3.5.2=py38h578d9bd_1
• xarray=0.20.2=pyhd8ed1ab_0
• xyzservices=2021.11.0=pyhd8ed1ab_1
• xz=5.2.5=h516909a_1
• yaml=0.2.5=h7f98852_2
• zeromq=4.3.4=h9c3ff4c_1
• zfp=0.5.5=h9c3ff4c_8
• zict=2.0.0=py_0
• zipp=3.7.0=pyhd8ed1ab_0
• zlib=1.2.11=h36c2ea0_1013
• zstandard=0.16.0=py38h497a2fe_2
• zstd=1.5.1=ha95c52a_0
• pip:
  • aiohttp==3.8.1
  • aiohttp-cors==0.7.0
  • aioredis==1.3.1
  • aiosignal==1.2.0
  • async-timeout==4.0.2
  • blessed==1.19.0
  • cachetools==4.2.4
  • colorful==0.5.4
  • deprecated==1.2.13
  • frozenlist==1.2.0
  • google-api-core==2.4.0
  • google-auth==2.3.3
  • googleapis-common-protos==1.54.0
  • gpustat==1.0.0b1
  • grpcio==1.43.0
  • hiredis==2.0.0
  • multidict==5.2.0
  • nvidia-ml-py3==7.352.0
  • opencensus==0.8.0
  • opencensus-context==0.1.2
  • protobuf==3.19.3
  • py-spy==0.3.11
  • pyasn1==0.4.8
  • pyasn1-modules==0.2.8
  • ray==1.9.2
  • redis==4.1.0
  • rsa==4.8
  • smart-open==5.2.1
  • wrapt==1.13.3
  • yarl==1.7.2

Reproduction script

Sorry, can’t share original source. Happy to provide any logs though. Also any hints on how to refactor to a minimal example so I could share a script is very welcome. I really don’t know where to start debugging this problem (looks like an issue somewhere deep in ray).

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!