wandb: [CLI]: TypeError: _WandbInit._pause_backend() takes 1 positional argument but 2 were given

@drscotthawley @jason-brian-anderson @YooSungHyun sorry to hear you’re running into this issue, could you please try to downgrade your ipython package to 8.16.1 version, and then restart your Jupyter kernel? You can downgrade as follows from the terminal: pip install ipython==8.16.1

A quick update on the issue @ here: our SDK team has implemented a fix, which has been shipped in the latest release of wandb (v0.16.0). You can now upgrade IPython if you are using wandb version 0.16.0. For more details, you can check out the pull request here and our release notes. Please let us know if you’re still encountering any issues.

Hey folks, we’re aware of the issue. The root cause is coming from the most recent version of IPython. We’re working on a fix.

For posterity as noted above, in the meantime time if you downgrade IPython to 8.16.1 the issue should go away.

Thanks. @thanos-wandb. Still, I want to echo @jason-brian-anderson’s question: For what reason is wandb taking control of every jupyter cell execution?

Great, we’re glad this unblocks you for now, @drscotthawley, we will raise to our SDK team and follow-up with you here. It seems something broke with latest ipython: https://pypi.org/project/ipython/8.17.2/#history

@thanos-wandb That worked! Hooray! Thank you.

fwiw, i’m running numpy ‘1.26.1’

the most interesting thing to me is that every invocation of any subsequent cell execution in jupyterlab thereafter. what is wandb doing such that every subsequent, non-related cell execution is somehow wrapped by this resume/pause method? i don’t think i’ve ever seen anything like this before

here is how i reproduced it:

import numpy as np
import wandb
import transformers
print(np.__version__)
print(wandb.__version__)
print(transformers.__version__)
import wandb
run = wandb.init()
assert run is wandb.run

gives:

1.26.1
0.15.12
4.34.1
wandb: Currently logged in as: jason-anderson-professional. Use `wandb login --relogin` to force relogin
Tracking run with wandb version 0.15.12
Run data is saved locally in /app/jupyter/lora_gptq/wandb/run-20231107_140745-86c160ks
Syncing run [morning-eon-3](https://wandb.ai/jason-anderson-professional/lora_gptq-jupyter_lora_gptq/runs/86c160ks) to [Weights & Biases](https://wandb.ai/jason-anderson-professional/lora_gptq-jupyter_lora_gptq) ([docs](https://wandb.me/run))
View project at https://wandb.ai/jason-anderson-professional/lora_gptq-jupyter_lora_gptq
View run at https://wandb.ai/jason-anderson-professional/lora_gptq-jupyter_lora_gptq/runs/86c160ks
Error in callback <bound method _WandbInit._pause_backend of <wandb.sdk.wandb_init._WandbInit object at 0x7ff247a80100>> (for post_run_cell), with arguments args (<ExecutionResult object at 7ff247a82ad0, execution_count=1 error_before_exec=None error_in_exec=None info=<ExecutionInfo object at 7ff247a80190, raw_cell="import numpy as np
import wandb
import transformer.." store_history=True silent=False shell_futures=True cell_id=04e878bb-37eb-4914-949e-1992fae4c8ff> result=None>,),kwargs {}:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: _WandbInit._pause_backend() takes 1 positional argument but 2 were given

and running the next cell with: print('hello world')

gives:

Error in callback <bound method _WandbInit._resume_backend of <wandb.sdk.wandb_init._WandbInit object at 0x7ff247a80100>> (for pre_run_cell), with arguments args (<ExecutionInfo object at 7ff1df76aa10, raw_cell="print('hello world')" store_history=True silent=False shell_futures=True cell_id=4b869ce9-75c6-4b2f-879b-d35b15cf1d74>,),kwargs {}:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: _WandbInit._resume_backend() takes 1 positional argument but 2 were given
hello world
Error in callback <bound method _WandbInit._pause_backend of <wandb.sdk.wandb_init._WandbInit object at 0x7ff247a80100>> (for post_run_cell), with arguments args (<ExecutionResult object at 7ff1df76a680, execution_count=2 error_before_exec=None error_in_exec=None info=<ExecutionInfo object at 7ff1df76aa10, raw_cell="print('hello world')" store_history=True silent=False shell_futures=True cell_id=4b869ce9-75c6-4b2f-879b-d35b15cf1d74> result=None>,),kwargs {}:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: _WandbInit._pause_backend() takes 1 positional argument but 2 were given

why is wandb even running at all in this cell? it’s like it’s @decorating every cell ?!

python isn’t wrong to assert that the method only takes 1 arg:

    def _resume_backend(self) -> None:
        if self.backend is not None and self.backend.interface is not None:
            logger.info("resuming backend")  # type: ignore
            self.backend.interface.publish_resume()

i’m not even sure how to start trouble shooting given the weird control over jupyter’s cell execution.

Hey all, thanks a lot for reporting this! I’m currently investigating this so will post an update as soon as I have something

Hi folks. I’m getting this a lot (e.g. for wandb.init() and wandb.finish() and wandb.log() ) on my brand-new, fresh environment with recent update to Ubuntu 22.04 LTS, using Python 3.10. I get the error with wandb versions 0.15.12 and 0.15.10. (I started with 0.15.12 and then tried downgrading, but it didn’t help.)

On Discord, wandbot suggested setting settings=wandb.Settings(start_method="fork") when I call wandb.init(), but adding that had no effect.

This messaging is burdensome, in that the error appears twice even when performing non-wandb operations, e.g. even “2+2” produces the error:

I suspect the error happens when wandb.finish() isn’t called before the user does “something” (not exactly sure what), such as restarts the kernel or runs wandb.init() again.

Looking at processes on my machine, I see there are currently 3 instances of wandb-service running. 2 processes might make sense as I’m doing two simultaneous runs (on two different GPUs), but not sure where the 3rd is coming from. I’ll try killing all the wandb-service processes, re-running wandb.login(), and see if that fixes things… 🤞

UPDATE: Nope. That had no effect. Error persists even after killing processes, restarting kernel, re-running wandb.login(), etc. Also tried uninstalling & re-isntalling (the latest) wandb. Nothing helped.