wandb: [CLI]: TypeError: _WandbInit._pause_backend() takes 1 positional argument but 2 were given

Describe the bug

if i used run = wandb.init(entity="bart_tadev", project='GPT-4 in Python', name="test") in jupyter notebook error raised TypeError: _WandbInit._pause_backend() takes 1 positional argument but 2 were given

run = wandb.init(entity="bart_tadev", project='GPT-4 in Python', name="test")
wandb: Currently logged in as: bart_tadev. Use `wandb login --relogin` to force relogin
Tracking run with wandb version 0.15.12
Run data is saved locally in /data01/bart/private/kaggle/AI-gen-detector/wandb/run-20231103_163805-aqkwtf24
Syncing run [test](https://wandb.ai/bart_tadev/GPT-4%20in%20Python/runs/aqkwtf24) to [Weights & Biases](https://wandb.ai/bart_tadev/GPT-4%20in%20Python) ([docs](https://wandb.me/run))
View project at https://wandb.ai/bart_tadev/GPT-4%20in%20Python
View run at https://wandb.ai/bart_tadev/GPT-4%20in%20Python/runs/aqkwtf24
Error in callback <bound method _WandbInit._pause_backend of <wandb.sdk.wandb_init._WandbInit object at 0x7f252ade0610>> (for post_run_cell), with arguments args (<ExecutionResult object at 7f252add6e10, execution_count=4 error_before_exec=None error_in_exec=None info=<ExecutionInfo object at 7f252add79d0, raw_cell="run = wandb.init(entity="bart_tadev", project='GPT.." store_history=True silent=False shell_futures=True cell_id=[vscode-notebook-cell://ssh-remote%2B7b22686f73744e616d65223a22284750553037295f3139322e3136382e302e323335227d/data01/bart/private/kaggle/AI-gen-detector/wandb_gpt_api.ipynb#W3sdnNjb2RlLXJlbW90ZQ%3D%3D>](vscode-notebook-cell://ssh-remote%2B7b22686f73744e616d65223a22284750553037295f3139322e3136382e302e323335227d/data01/bart/private/kaggle/AI-gen-detector/wandb_gpt_api.ipynb#W3sdnNjb2RlLXJlbW90ZQ%3D%3D%3E) result=None>,),kwargs {}:

Additional Files

No response

Environment

WandB version: wandb==0.15.12

OS: ubuntu 18.04

Python version: 3.11

Versions of relevant libraries:

Additional Context

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 9
  • Comments: 27 (10 by maintainers)

Commits related to this issue

Most upvoted comments

@drscotthawley @jason-brian-anderson @YooSungHyun sorry to hear you’re running into this issue, could you please try to downgrade your ipython package to 8.16.1 version, and then restart your Jupyter kernel? You can downgrade as follows from the terminal: pip install ipython==8.16.1

A quick update on the issue @ here: our SDK team has implemented a fix, which has been shipped in the latest release of wandb (v0.16.0). You can now upgrade IPython if you are using wandb version 0.16.0. For more details, you can check out the pull request here and our release notes. Please let us know if you’re still encountering any issues.

Thanks. @thanos-wandb. Still, I want to echo @jason-brian-anderson’s question: For what reason is wandb taking control of every jupyter cell execution?

Great, we’re glad this unblocks you for now, @drscotthawley, we will raise to our SDK team and follow-up with you here. It seems something broke with latest ipython: https://pypi.org/project/ipython/8.17.2/#history

@thanos-wandb That worked! Hooray! Thank you.

Here’s a clue: this seems to be some kind of interaction between wandb and jupyter-notebook. If i run those same lines of code in CLI Python for that environment, i see no errors!

fwiw, i’m running numpy ‘1.26.1’

the most interesting thing to me is that every invocation of any subsequent cell execution in jupyterlab thereafter. what is wandb doing such that every subsequent, non-related cell execution is somehow wrapped by this resume/pause method? i don’t think i’ve ever seen anything like this before

here is how i reproduced it:

import numpy as np
import wandb
import transformers
print(np.__version__)
print(wandb.__version__)
print(transformers.__version__)
import wandb
run = wandb.init()
assert run is wandb.run

gives:

1.26.1
0.15.12
4.34.1
wandb: Currently logged in as: jason-anderson-professional. Use `wandb login --relogin` to force relogin
Tracking run with wandb version 0.15.12
Run data is saved locally in /app/jupyter/lora_gptq/wandb/run-20231107_140745-86c160ks
Syncing run [morning-eon-3](https://wandb.ai/jason-anderson-professional/lora_gptq-jupyter_lora_gptq/runs/86c160ks) to [Weights & Biases](https://wandb.ai/jason-anderson-professional/lora_gptq-jupyter_lora_gptq) ([docs](https://wandb.me/run))
View project at https://wandb.ai/jason-anderson-professional/lora_gptq-jupyter_lora_gptq
View run at https://wandb.ai/jason-anderson-professional/lora_gptq-jupyter_lora_gptq/runs/86c160ks
Error in callback <bound method _WandbInit._pause_backend of <wandb.sdk.wandb_init._WandbInit object at 0x7ff247a80100>> (for post_run_cell), with arguments args (<ExecutionResult object at 7ff247a82ad0, execution_count=1 error_before_exec=None error_in_exec=None info=<ExecutionInfo object at 7ff247a80190, raw_cell="import numpy as np
import wandb
import transformer.." store_history=True silent=False shell_futures=True cell_id=04e878bb-37eb-4914-949e-1992fae4c8ff> result=None>,),kwargs {}:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: _WandbInit._pause_backend() takes 1 positional argument but 2 were given


and running the next cell with: print('hello world')

gives:

Error in callback <bound method _WandbInit._resume_backend of <wandb.sdk.wandb_init._WandbInit object at 0x7ff247a80100>> (for pre_run_cell), with arguments args (<ExecutionInfo object at 7ff1df76aa10, raw_cell="print('hello world')" store_history=True silent=False shell_futures=True cell_id=4b869ce9-75c6-4b2f-879b-d35b15cf1d74>,),kwargs {}:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: _WandbInit._resume_backend() takes 1 positional argument but 2 were given
hello world
Error in callback <bound method _WandbInit._pause_backend of <wandb.sdk.wandb_init._WandbInit object at 0x7ff247a80100>> (for post_run_cell), with arguments args (<ExecutionResult object at 7ff1df76a680, execution_count=2 error_before_exec=None error_in_exec=None info=<ExecutionInfo object at 7ff1df76aa10, raw_cell="print('hello world')" store_history=True silent=False shell_futures=True cell_id=4b869ce9-75c6-4b2f-879b-d35b15cf1d74> result=None>,),kwargs {}:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: _WandbInit._pause_backend() takes 1 positional argument but 2 were given


why is wandb even running at all in this cell? it’s like it’s @decorating every cell ?!

python isn’t wrong to assert that the method only takes 1 arg:

    def _resume_backend(self) -> None:
        if self.backend is not None and self.backend.interface is not None:
            logger.info("resuming backend")  # type: ignore
            self.backend.interface.publish_resume()

i’m not even sure how to start trouble shooting given the weird control over jupyter’s cell execution.

I’m seeing this as well. Started after i rebuilt my docker container that pulled the most recent version of wandb yesterday (0.15.12) it has a weird interaction with jupyterlab(running Version 4.0.8). I’m not even importing wandb, and just by running trl.sfttrainer i get the error:

from trl import SFTTrainer
trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=dataset,
    peft_config=config,
    dataset_text_field="text",
    tokenizer=tokenizer,
    packing=False,
    max_seq_length=512)

But not only then, but all subsequent cell executions thereafter.

I literally get the error while running the single command

print('hello world')

talk about imposter syndrome when you can’t even run hello world 😉