wandb: [CLI] WandbCallback AttributeError: 'NoneType' object has no attribute 'item'

Description

I’m getting a temperamental (i.e. doesn’t consistently occur) AttrributeError from the keras WandbCallbackwith wandb 0.12.1 after calling model.fit in a train script:

Traceback (most recent call last):
  File "/data/hpcdata/users/tomand/code/icenet2/icenet2/train_icenet2.py", line 350, in <module>
    WandbCallback(
  File "/data/hpcdata/users/tomand/anaconda/envs/icenet2_tf24/lib/python3.9/site-packages/wandb/integration/keras/keras.py", line 430, in __init__
    previous_best = wandb.run.summary.get(
  File "/data/hpcdata/users/tomand/anaconda/envs/icenet2_tf24/lib/python3.9/site-packages/wandb/sdk/wandb_summary.py", line 34, in get
    return self._as_dict().get(key, default)
  File "/data/hpcdata/users/tomand/anaconda/envs/icenet2_tf24/lib/python3.9/site-packages/wandb/sdk/wandb_summary.py", line 127, in _as_dict
    return self._get_current_summary_callback()
  File "/data/hpcdata/users/tomand/anaconda/envs/icenet2_tf24/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 859, in _summary_get_current_summary_callback
    return proto_util.dict_from_proto_list(ret.item)
AttributeError: 'NoneType' object has no attribute 'item'

wandb: Waiting for W&B process to finish, PID 36822
wandb: Program failed with code 1.  Press ctrl-c to abort syncing.

The strangest thing is the inconsistency. With no changes to the script, this error will pop up on a couple attempts and then it will run fine just five minutes later.

I’m not really sure where to start in terms of providing further information or a reproducible example. Is this already on your radar? Let me know if you need more info.

Environment

OS: CentOS Linux 7
Environment: SLURM job on remote HPC with conda environment
Python Version: 3.9.7

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16

Most upvoted comments

Hi, upgrading to latest version 12.7 the problem seems resolved. Thank you for your support.

In any case I found that when using a notebook, is possibile to overcame the issue running two times the same cell (I don’t know why this work).

Thank you again, Matteo

Inviato da Postahttps://go.microsoft.com/fwlink/?LinkId=550986 per Windows

Da: Anmol @.> Inviato: venerdì 3 dicembre 2021 01:05 A: @.> Cc: @.>; @.> Oggetto: Re: [wandb/client] [CLI] WandbCallback AttributeError: ‘NoneType’ object has no attribute ‘item’ (Issue #2839)

Hi @matteoferrante https://github.com/matteoferrante, @tom-andersson https://github.com/tom-andersson , I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/wandb/client/issues/2839#issuecomment-985100605, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AILKTAP7JVHKXMCT3NNR5NLUPACTJANCNFSM5G244B6Q. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

matteoferrante on Dec 3, 2021

@anmolmann resolved the error by upgrading to latest version. The error does not persist in the latest version(in my case).

bp-high on Nov 30, 2021

Thank you @anmolmann, much appreciated

tom-andersson on Nov 12, 2021