neptune-client: BUG: TypeError: stat: path should be string, bytes, os.PathLike or integer, not _io.BytesIO

Describe the bug

When the run starts (catboost + optuna), after the first round of optimization, the below error is shown. No further sync happens after this error.

This error also occurs when I try to sync the run afterwards (neptune sync --object ...).

Unexpected error occurred in Neptune background thread: Killing Neptune asynchronous thread. All data is safe on disk and can be later synced manually using `neptune sync` command.
Exception in thread NeptuneAsyncOpProcessor:
Traceback (most recent call last):
  File "/home/username/miniconda3/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/threading/daemon.py", line 53, in run
    self.work()
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/operation_processors/async_operation_processor.py", line 214, in work
    self.process_batch(batch, version)
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/threading/daemon.py", line 77, in wrapper
    result = func(self_, *args, **kwargs)
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/operation_processors/async_operation_processor.py", line 227, in process_batch
    processed_count, errors = self._processor._backend.execute_operations(
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/backends/hosted_neptune_backend.py", line 481, in execute_operations
    self._execute_upload_operations_with_400_retry(
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/backends/hosted_neptune_backend.py", line 580, in _execute_upload_operations_with_400_retry
    return self._execute_upload_operations(
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/backends/hosted_neptune_backend.py", line 546, in _execute_upload_operations
    upload_errors = upload_file_attribute(
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/backends/hosted_file_operations.py", line 108, in upload_file_attribute
    _multichunk_upload_with_retry(
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/backends/hosted_file_operations.py", line 319, in _multichunk_upload_with_retry
    return _multichunk_upload(
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/new/internal/backends/hosted_file_operations.py", line 369, in _multichunk_upload
    for idx, chunk in enumerate(chunker.generate()):
  File "/home/username/miniconda3/lib/python3.8/site-packages/neptune/internal/storage/datastream.py", line 66, in generate
    last_change = os.stat(self._filename).st_mtime
TypeError: stat: path should be string, bytes, os.PathLike or integer, not _io.BytesIO

Reproduction

It happens for all my recent runs.

Expected behavior

• No error
• syncing to the app.neptune.ai

Environment

The output of pip list:

Package                           Version
--------------------------------- -----------
aiohttp                           3.8.1
aiohttp-cors                      0.7.0
aioredis                          1.3.1
aiosignal                         1.2.0
alembic                           1.7.7
altair                            4.2.0
altgraph                          0.17.2
argon2-cffi                       21.3.0
argon2-cffi-bindings              21.2.0
astor                             0.8.1
asttokens                         2.0.5
async-generator                   1.10
async-timeout                     4.0.2
attrs                             21.4.0
autoflake                         1.4
autopage                          0.5.0
autopep8                          1.6.0
backcall                          0.2.0
backports.functools-lru-cache     1.6.4
backports.lzma                    0.0.14
beautifulsoup4                    4.11.1
biopython                         1.79
bleach                            5.0.0
blessings                         1.7
blinker                           1.4
bokeh                             2.4.2
boto3                             1.21.43
botocore                          1.24.43
bravado                           11.0.3
bravado-core                      5.17.0
brotlipy                          0.7.0
cachetools                        5.0.0
catboost                          1.0.5
category-encoders                 2.4.0
certifi                           2021.10.8
cffi                              1.15.0
chardet                           4.0.0
charset-normalizer                2.0.12
ciso8601                          2.2.0
click                             8.1.2
cliff                             3.10.1
cloudpickle                       2.0.0
cmaes                             0.8.2
cmd2                              2.4.0
colorama                          0.4.4
colorful                          0.5.4
colorlog                          6.6.0
colour                            0.1.5
comet-ml                          3.28.2
conda                             4.12.0
conda-pack                        0.7.0
conda-package-handling            1.8.1
configobj                         5.0.6
contextlib2                       21.6.0
cryptography                      36.0.2
csb                               1.2.5
cycler                            0.11.0
cytoolz                           0.11.2
dask                              2022.4.1
dataclasses                       0.8
debugpy                           1.6.0
decorator                         5.1.1
defusedxml                        0.7.1
Deprecated                        1.2.13
distributed                       2022.4.1
dtreeviz                          1.3.5
dulwich                           0.20.35
emcee                             3.1.1
entrypoints                       0.4
everett                           3.0.0
executing                         0.8.3
fastjsonschema                    2.15.3
filelock                          3.6.0
fitparse                          1.2.0
Flask                             2.0.3
flit_core                         3.7.1
fonttools                         4.32.0
frozenlist                        1.3.0
fsspec                            2022.3.0
future                            0.18.2
garmin-ical-export                1.0.4
gitdb                             4.0.9
GitPython                         3.1.27
google-api-core                   2.5.0
google-auth                       2.6.5
googleapis-common-protos          1.56.0
gplearn                           0.4.1
gpustat                           0.6.0
graphviz                          0.20
greenlet                          1.1.2
GridDataFormats                   0.7.0
grpcio                            1.45.0
h5py                              3.6.0
HeapDict                          1.0.1
hiredis                           2.0.0
hyperopt                          0.2.7
idna                              3.3
imbalanced-learn                  0.9.0
importlib-metadata                4.11.3
importlib-resources               5.7.1
iniconfig                         1.1.1
ipykernel                         6.13.0
ipython                           8.2.0
ipython-genutils                  0.2.0
ipywidgets                        7.7.0
iso3166                           2.0.2
itsdangerous                      2.1.0
jedi                              0.18.1
Jinja2                            3.1.1
jmespath                          1.0.0
joblib                            1.1.0
json-tricks                       3.15.5
jsonref                           0.2
jsonschema                        4.4.0
jupyter                           1.0.0
jupyter-client                    7.2.2
jupyter-console                   6.4.3
jupyter-contrib-core              0.3.3
jupyter-contrib-nbextensions      0.5.1
jupyter-core                      4.9.2
jupyter-highlight-selected-word   0.2.0
jupyter-latex-envs                1.4.6
jupyter-nbextensions-configurator 0.4.1
jupyterlab-pygments               0.2.2
kaleido                           0.2.1
keras                             2.8.0
kiwisolver                        1.4.2
libmambapy                        0.22.1
lightgbm                          3.3.2
llvmlite                          0.36.0
locket                            0.2.0
lux-api                           0.5.1
lux-widget                        0.1.11
lxml                              4.8.0
lz4                               4.0.0
macholib                          1.15.2
Mako                              1.2.0
mamba                             0.22.1
Markdown                          3.3.6
MarkupSafe                        2.1.1
matplotlib                        3.5.1
matplotlib-inline                 0.1.3
mistune                           0.8.4
mljar-supervised                  0.11.2
modin                             0.13.2
MolVS                             0.1.1
monotonic                         1.5
mpi4py                            3.1.3
mrcfile                           1.3.0
msgpack                           1.0.3
multidict                         6.0.2
munkres                           1.1.4
mypy-extensions                   0.4.3
nb-conda                          2.2.1
nb-conda-kernels                  2.3.1
nbclient                          0.6.0
nbconvert                         6.5.0
nbformat                          5.3.0
neptune-client                    0.16.1
neptune-optuna                    0.9.13
nest-asyncio                      1.5.5
networkx                          2.7
nni                               2.6.1
notebook                          6.4.11
numba                             0.53.1
numpy                             1.22.3
nvidia-ml-py3                     7.352.0
oauthlib                          3.2.0
olefile                           0.46
opencensus                        0.8.0
opencensus-context                0.1.2
optuna                            2.10.0
packaging                         21.3
paho-mqtt                         1.6.1
pandas                            1.4.0
pandocfilters                     1.5.0
parso                             0.8.3
partd                             1.2.0
pathlib2                          2.3.7.post1
patsy                             0.5.2
pbr                               5.8.1
pexpect                           4.8.0
pickleshare                       0.7.5
Pillow                            9.1.0
pip                               22.0.4
plotly                            5.7.0
pluggy                            1.0.0
Pmw                               2.0.1
prettytable                       3.2.0
prometheus-client                 0.14.1
prompt-toolkit                    3.0.29
protobuf                          3.19.4
psutil                            5.9.0
ptyprocess                        0.7.0
pure-eval                         0.2.2
py                                1.11.0
py4j                              0.10.9.3
pyaml                             21.10.1
pyarrow                           6.0.1
pyasn1                            0.4.8
pyasn1-modules                    0.2.7
pycairo                           1.21.0
pycodestyle                       2.8.0
pycosat                           0.6.3
pycparser                         2.21
pycryptodome                      3.14.1
pyflakes                          2.4.0
Pygments                          2.11.2
pyinstaller                       4.9
pyinstaller-hooks-contrib         2022.2
PyJWT                             2.3.0
pymongo                           4.0.1
pyOpenSSL                         22.0.0
pyparsing                         3.0.8
pyperclip                         1.8.2
PyPrind                           2.11.3
Pypubsub                          4.0.3
PyQt5                             5.15.6
PyQt5-Qt5                         5.15.2
PyQt5-sip                         12.9.1
PyQtChart                         5.15.5
PyQtChart-Qt5                     5.15.2
PyQtWebEngine                     5.15.5
PyQtWebEngine-Qt5                 5.15.2
pyrsistent                        0.18.1
pyserial                          3.5
PySocks                           1.7.1
pytest                            7.1.1
python-dateutil                   2.8.2
PythonWebHDFS                     0.2.3
pytz                              2022.1
pyu2f                             0.1.5
PyYAML                            6.0
pyzmq                             22.3.0
qed                               1.0.1
qtconsole                         5.3.0
QtPy                              2.0.1
ray                               1.10.0
redis                             4.2.2
reportlab                         3.6.7
requests                          2.27.1
requests-oauthlib                 1.3.1
requests-toolbelt                 0.9.1
responses                         0.18.0
rsa                               4.8
ruamel-yaml-conda                 0.15.80
s3transfer                        0.5.2
schema                            0.7.5
schwimmbad                        0.3.2
scikit-learn                      1.0.2
scikit-optimize                   0.9.0
scikit-plot                       0.3.7
scipy                             1.8.0
seaborn                           0.11.2
semantic-version                  2.9.0
Send2Trash                        1.8.0
service-identity                  21.1.0
setproctitle                      1.2.2
setuptools                        62.1.0
sh                                1.13.1
shap                              0.40.0
simplejson                        3.17.6
six                               1.16.0
slicer                            0.0.7
smart-open                        5.2.1
smilite                           2.3.0
smmap                             3.0.5
smogn                             0.1.2
sortedcontainers                  2.4.0
soupsieve                         2.3.1
SQLAlchemy                        1.4.35
stack-data                        0.2.0
statsmodels                       0.13.2
stevedore                         3.5.0
swagger-spec-validator            2.7.4
tabulate                          0.8.9
tblib                             1.7.0
tenacity                          8.0.1
terminado                         0.13.3
testpath                          0.6.0
threadpoolctl                     3.1.0
tinycss2                          1.1.1
toml                              0.10.2
tomli                             2.0.1
toolz                             0.11.2
torch                             1.10.2
torchvision                       0.11.3
tornado                           6.1
tqdm                              4.64.0
traitlets                         5.1.1
typeguard                         2.13.3
typing_extensions                 4.2.0
unicodedata2                      14.0.0
UpSetPlot                         0.6.0
urllib3                           1.26.9
vobject                           0.9.6.1
wcwidth                           0.2.5
webencodings                      0.5.1
websocket-client                  1.3.2
websockets                        10.2
Werkzeug                          2.1.1
wheel                             0.37.1
widgetsnbextension                3.6.0
wordcloud                         1.8.1
wrapt                             1.14.0
wurlitzer                         3.0.2
wxPython                          4.1.1
xgboost                           1.5.2
yapf                              0.32.0
yarl                              1.7.2
zict                              2.1.0
zipp                              3.8.0

The operating system you’re using: Ubuntu 20.04.4 LTS The output of python --version: Python 3.8.13

Further info

This error doesn’t occur with the same script and data, the same neptune-client and neptune-optuna versions but run on another system (ICM topola, Linux hpc 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux and Python 3.9.12).

pip list (here everything works fine):

Package                       Version
----------------------------- ---------
aiohttp                       3.8.1
aiohttp-cors                  0.7.0
aioredis                      1.3.1
aiosignal                     1.2.0
alembic                       1.7.7
async-timeout                 4.0.2
attrs                         21.4.0
autopage                      0.5.0
backports.functools-lru-cache 1.6.4
blessings                     1.7
blinker                       1.4
bokeh                         2.4.2
boto3                         1.21.44
botocore                      1.24.44
bravado                       11.0.3
bravado-core                  5.17.0
brotlipy                      0.7.0
cachetools                    5.0.0
catboost                      1.0.5
category-encoders             2.4.0
certifi                       2021.10.8
cffi                          1.15.0
charset-normalizer            2.0.12
click                         8.1.2
cliff                         3.10.1
cloudpickle                   2.0.0
cmaes                         0.8.2
cmd2                          2.4.0
colorama                      0.4.4
colorful                      0.5.4
colorlog                      6.6.0
colour                        0.1.5
comet-ml                      3.28.2
conda                         4.12.0
conda-package-handling        1.8.1
configobj                     5.0.6
cryptography                  36.0.2
cycler                        0.11.0
cytoolz                       0.11.2
dask                          2022.4.1
dataclasses                   0.8
Deprecated                    1.2.13
dill                          0.3.4
distributed                   2022.4.1
dtreeviz                      1.3.5
dulwich                       0.20.35
everett                       3.0.0
filelock                      3.6.0
fonttools                     4.32.0
frozenlist                    1.3.0
fsspec                        2022.3.0
future                        0.18.2
gitdb                         4.0.9
GitPython                     3.1.27
google-api-core               2.5.0
google-auth                   2.6.5
googleapis-common-protos      1.56.0
gpustat                       0.6.0
graphviz                      0.20
greenlet                      1.1.2
grpcio                        1.45.0
HeapDict                      1.0.1
hiredis                       2.0.0
idna                          3.3
importlib-metadata            4.11.3
importlib-resources           5.7.1
iniconfig                     1.1.1
Jinja2                        3.1.1
jmespath                      1.0.0
joblib                        1.1.0
jsonref                       0.2
jsonschema                    4.4.0
kaleido                       0.2.1
kiwisolver                    1.4.2
libmambapy                    0.22.1
lightgbm                      3.3.2
llvmlite                      0.38.0
locket                        0.2.0
lz4                           4.0.0
Mako                          1.2.0
mamba                         0.22.1
Markdown                      3.3.6
MarkupSafe                    2.1.1
matplotlib                    3.5.1
mljar-supervised              0.11.2
modin                         0.13.2
monotonic                     1.5
msgpack                       1.0.3
multidict                     6.0.2
munkres                       1.1.4
neptune-client                0.16.1
neptune-optuna                0.9.13
numba                         0.53.1
numpy                         1.22.3
nvidia-ml-py3                 7.352.0
oauthlib                      3.2.0
opencensus                    0.9.0
opencensus-context            0.1.2
optuna                        2.10.0
packaging                     21.3
pandarallel                   1.5.5
pandas                        1.4.0
partd                         1.2.0
patsy                         0.5.2
pbr                           5.8.1
Pillow                        9.1.0
pip                           22.0.4
plotly                        5.7.0
pluggy                        1.0.0
prettytable                   3.2.0
prometheus-client             0.14.1
protobuf                      3.19.4
psutil                        5.9.0
py                            1.11.0
pyarrow                       7.0.0
pyasn1                        0.4.8
pyasn1-modules                0.2.7
pycosat                       0.6.3
pycparser                     2.21
PyJWT                         2.3.0
pyOpenSSL                     22.0.0
pyparsing                     3.0.8
pyperclip                     1.8.2
pyrsistent                    0.18.1
PySocks                       1.7.1
pytest                        7.1.1
python-dateutil               2.8.2
pytz                          2022.1
pyu2f                         0.1.5
PyYAML                        6.0
ray                           1.10.0
redis                         4.2.2
requests                      2.27.1
requests-oauthlib             1.3.1
requests-toolbelt             0.9.1
rsa                           4.8
ruamel-yaml-conda             0.15.80
s3transfer                    0.5.2
scikit-learn                  1.0.2
scikit-plot                   0.3.7
scipy                         1.8.0
seaborn                       0.11.2
semantic-version              2.9.0
setproctitle                  1.2.2
setuptools                    62.1.0
shap                          0.40.0
simplejson                    3.17.6
six                           1.16.0
slicer                        0.0.7
smart-open                    5.2.1
smmap                         3.0.5
sortedcontainers              2.4.0
SQLAlchemy                    1.4.35
statsmodels                   0.13.2
stevedore                     3.5.0
swagger-spec-validator        2.7.4
tabulate                      0.8.9
tblib                         1.7.0
tenacity                      8.0.1
threadpoolctl                 3.1.0
tomli                         2.0.1
toolz                         0.11.2
tornado                       6.1
tqdm                          4.64.0
typing_extensions             4.2.0
unicodedata2                  14.0.0
urllib3                       1.26.9
wcwidth                       0.2.5
websocket-client              1.3.2
wheel                         0.37.1
wordcloud                     1.8.1
wrapt                         1.14.0
wurlitzer                     3.0.2
xgboost                       1.5.2
yarl                          1.7.2
zict                          2.1.0
zipp                          3.8.0