tensorboard: Callback failing: AttributeError: 'TrackableWeightHandler' object has no attribute 'name'

Diagnostics

Diagnostics output

--- check: autoidentify
INFO: diagnose_tensorboard.py source unavailable
INFO: diagnose_tensorboard.py source unavailable

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=8, releaselevel='final', serial=0)
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=8, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Darwin', nodename='MacBook-Pro-3.lan', release='19.6.0', version='Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64', machine='x86_64')
INFO: os.uname(): posix.uname_result(sysname='Darwin', nodename='MacBook-Pro-3.lan', release='19.6.0', version='Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64', machine='x86_64')
INFO: sys.getwindowsversion(): N/A
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: None
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==2.4.0
INFO: installed: tensorboard==2.4.0
INFO: installed: tensorflow==2.4.0
INFO: installed: tensorflow==2.4.0
INFO: installed: tensorflow-estimator==2.4.0
INFO: installed: tensorflow-estimator==2.4.0

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.4.0'
INFO: tensorboard.version.VERSION: '2.4.0'

--- check: tensorflow_python_version
INFO: tensorflow.__version__: '2.4.0'
INFO: tensorflow.__version__: '2.4.0'
INFO: tensorflow.__git_version__: 'v2.4.0-rc4-71-g582c8d236cb'
INFO: tensorflow.__git_version__: 'v2.4.0-rc4-71-g582c8d236cb'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/Users/dennisyurkevich/.pyenv/versions/3.7.8/bin/tensorboard\n'
INFO: which tensorboard: b'/Users/dennisyurkevich/.pyenv/versions/3.7.8/bin/tensorboard\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 1024>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 1024>
Loopback infos: [(<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): 'macbook-pro-3.lan'
INFO: socket.getfqdn(): 'macbook-pro-3.lan'

--- check: stat_tensorboardinfo
INFO: directory: /var/folders/gn/91n6_2ts4r5drxqt__hnn36w0000gn/T/.tensorboard-info
INFO: directory: /var/folders/gn/91n6_2ts4r5drxqt__hnn36w0000gn/T/.tensorboard-info
INFO: .tensorboard-info directory does not exist
INFO: .tensorboard-info directory does not exist

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/Users/dennisyurkevich/.pyenv/versions/3.7.8/lib/python3.7/site-packages']; bad_roots (0): []
INFO: tensorboard_roots (1): ['/Users/dennisyurkevich/.pyenv/versions/3.7.8/lib/python3.7/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.11.0
aiohttp==3.6.2
alabaster==0.7.12
alembic==1.4.2
anyconfig==0.9.11
AppNexus-client==0.8.1
appnope==0.1.0
argcomplete==1.12.0
arrow==0.17.0
astroid==2.4.2
astunparse==1.6.3
async-timeout==3.0.1
attrs==19.3.0
Babel==2.9.0
backcall==0.2.0
backoff==1.10.0
beautifulsoup4==4.9.1
binaryornot==0.4.4
bleach==3.1.5
blis==0.4.1
bokeh==2.1.1
boto==2.49.0
boto3==1.14.30
botocore==1.17.30
CacheControl==0.12.6
cachetools==4.1.1
catalogue==1.0.0
cdx-toolkit==0.9.25
Cerberus==1.3.2
certifi==2020.6.20
cffi==1.14.1
chardet==3.0.4
click==7.1.2
cloudevents==0.3.0
cloudpickle==1.6.0
clusteval==1.0.2
colorcet==2.0.2
comcrawl==1.0.2
configparser==5.0.0
cookiecutter==1.6.0
courlan==0.2.3
coverage==5.3
crcmod==1.7
cryptography==3.0
cssselect==1.1.0
cycler==0.10.0
cymem==2.0.3
Cython==0.29.21
dask==2.25.0
datashape==0.5.2
decorator==4.4.2
defusedxml==0.6.0
dill==0.3.3
distributed==2.21.0
docker==4.3.1
docutils==0.15.2
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.1/en_core_web_sm-2.3.1.tar.gz
enstop==0.2.5
entrypoints==0.3
fastcore==1.3.10
filelock==3.0.12
firebase-admin==4.4.0
Flask==1.1.2
flatbuffers==1.12
fsspec==0.6.3
functions-framework==2.0.0
funcy==1.14
furl==2.1.0
future==0.18.2
gast==0.3.3
gensim==3.8.3
google-api-core==1.23.0
google-api-python-client==1.12.3
google-auth==1.23.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.1
google-cloud-bigquery==2.6.0
google-cloud-bigquery-storage==2.1.0
google-cloud-core==1.4.1
google-cloud-firestore==1.9.0
google-cloud-logging==1.15.1
google-cloud-secret-manager==2.0.0
google-cloud-storage==1.30.0
google-crc32c==0.1.0
google-pasta==0.2.0
google-resumable-media==0.7.0
googleapis-common-protos==1.52.0
graphviz==0.8.4
grpc-google-iam-v1==0.12.3
grpcio==1.32.0
gunicorn==20.0.4
h5py==2.10.0
hdbscan==0.8.26
HeapDict==1.0.1
holoviews==1.13.3
html2text==2020.1.16
htmldate==0.7.2
httplib2==0.18.1
humanfriendly==8.2
ibm-cloud-sdk-core==1.5.1
ibm-watson==4.5.0
idna==2.10
imagesize==1.2.0
importlib-metadata==1.7.0
importlib-resources==3.3.0
ipykernel==5.3.4
ipython==7.16.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
isort==4.3.21
itsdangerous==1.1.0
jedi==0.17.2
jellyfish==0.8.2
Jinja2==2.11.2
jinja2-time==0.2.0
jmespath==0.10.0
joblib==0.16.0
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.6
jupyter-console==6.1.0
jupyter-core==4.6.3
jusText==2.2.0
kedro==0.16.6
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
libcst==0.3.13
lightfm==1.15
llvmlite==0.34.0
locket==0.2.0
lxml==4.5.2
Mako==1.1.3
Markdown==3.2.2
MarkupSafe==1.1.1
matplotlib==3.3.0
mccabe==0.6.1
mistune==0.8.4
more-itertools==8.4.0
msgpack==1.0.0
multidict==4.7.6
multipledispatch==0.6.0
murmurhash==1.0.2
mxnet==1.7.0.post1
mypy-extensions==0.4.3
nbconvert==5.6.1
nbdev==1.1.5
nbformat==5.0.7
networkx==2.4
nltk==3.5
notebook==6.0.3
numba==0.51.2
numexpr==2.7.1
numpy==1.19.4
oauthlib==3.1.0
opt-einsum==3.3.0
orderedmultidict==1.0.1
packaging==20.4
pandas==1.0.5
pandocfilters==1.4.2
panel==0.9.7
param==1.9.3
parsel==1.6.0
parso==0.7.1
partd==1.1.0
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
pip==20.3.3
pip-autoremove==0.9.1
pip-tools==5.3.0
pipx==0.15.4.0
plac==1.1.3
plotly==4.12.0
pluggy==0.13.1
ply==3.11
poyo==0.5.0
preshed==3.0.2
prometheus-client==0.8.0
promise==2.3
prompt-toolkit==3.0.5
proto-plus==1.11.0
protobuf==3.12.2
psutil==5.7.2
ptyprocess==0.6.0
py==1.9.0
-e git+https://github.com/AirGrid/py-utils.git@b89ba5a0ff82fe15510cf82613a1d1a1808baa76#egg=py_utils
py-zipkin==0.20.0
pyarrow==2.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.5.0
pycld3==0.20
pycparser==2.20
pyct==0.4.6
pydash==4.9.0
Pygments==2.6.1
PyJWT==1.7.1
pyLDAvis==2.1.2
pylint==2.5.3
pynndescent==0.4.8
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyrsistent==0.16.0
pytest==5.4.3
pytextrank==2.0.3
python-dateutil==2.8.1
python-editor==1.0.4
python-json-logger==0.1.11
pytz==2020.1
pyviz-comms==0.7.6
PyYAML==5.3.1
pyzmq==19.0.1
qtconsole==4.7.5
QtPy==1.9.0
readability-lxml==0.8.1
regex==2020.7.14
requests==2.24.0
requests-file==1.5.1
requests-oauthlib==1.3.0
retrying==1.3.3
rsa==4.6
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
s3transfer==0.3.3
sacremoses==0.0.43
scikit-learn==0.23.2
scipy==1.5.2
seaborn==0.10.1
segtok==1.5.10
Send2Trash==1.5.0
sentencepiece==0.1.94
setuptools==50.0.0
six==1.15.0
sklearn==0.0
smart-open==2.1.0
snowballstemmer==2.0.0
sortedcontainers==2.2.2
soupsieve==2.0.1
spacy==2.3.2
Sphinx==3.3.1
sphinx-autodoc-typehints==1.11.1
sphinx-markdown-builder==0.5.4
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
SQLAlchemy==1.3.19
SQLAlchemy-Utils==0.36.8
srsly==1.0.2
stanza==1.1.1
tabulate==0.8.7
tbb==2019.0
tblib==1.7.0
tenacity==6.2.0
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow==2.4.0
tensorflow-addons==0.12.0
tensorflow-datasets==4.1.0
tensorflow-estimator==2.4.0
tensorflow-metadata==0.26.0
tensorflow-recommenders==0.3.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
textmap @ git+https://github.com/TutteInstitute/TextMAP.git@150d939b4fb5b6dbcda057ef782ef442c042e7a5
thinc==7.4.1
Thingy==0.8.5
threadpoolctl==2.1.0
thriftpy2==0.4.11
tldextract==2.2.2
tokenizers==0.9.4
toml==0.10.1
toolz==0.10.0
toposort==1.5
torch==1.6.0
torchvision==0.7.0
tornado==6.0.4
tqdm==4.55.1
trafilatura==0.5.2
traitlets==4.3.3
transformers==4.1.1
typed-ast==1.4.1
typeguard==2.10.0
typing-extensions==3.7.4.2
typing-inspect==0.6.0
umap-learn==0.4.6
unify==0.5
untokenize==0.1.1
uritemplate==3.0.1
urllib3==1.25.11
userpath==1.4.1
vectorizers @ git+https://github.com/TutteInstitute/vectorizers.git@eff54167d0e68bbf35ec946a95ffd59444454c77
w3lib==1.22.0
warcio==1.7.4
wasabi==0.8.0
watchdog==0.10.3
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.48.0
Werkzeug==1.0.1
wheel==0.36.2
whichcraft==0.6.1
widgetsnbextension==3.5.1
wikipedia==1.4.0
Wikipedia-API==0.5.4
wordcloud==1.8.0
wrapt==1.12.1
xarray==0.16.0
yake @ git+https://github.com/LIAAD/yake@2c9fedbe0e3195a273a3c3daef3d41d4d7c5b3b3
yapf==0.30.0
yarl==1.5.1
zenserp==0.2
zict==2.0.0
zipp==3.1.0

INFO: pip freeze --all:
absl-py==0.11.0
aiohttp==3.6.2
airgrid @ git+https://github.com/AirGrid/mind.git@0a778be6e462881799b8c7b74e5d14ba05067332#subdirectory=serverless_crawler/client
alabaster==0.7.12
alembic==1.4.2
anyconfig==0.9.11
AppNexus-client==0.8.1
appnope==0.1.0
argcomplete==1.12.0
arrow==0.17.0
astroid==2.4.2
astunparse==1.6.3
async-timeout==3.0.1
attrs==19.3.0
Babel==2.9.0
backcall==0.2.0
backoff==1.10.0
beautifulsoup4==4.9.1
binaryornot==0.4.4
bleach==3.1.5
blis==0.4.1
bokeh==2.1.1
boto==2.49.0
boto3==1.14.30
botocore==1.17.30
CacheControl==0.12.6
cachetools==4.1.1
catalogue==1.0.0
cdx-toolkit==0.9.25
Cerberus==1.3.2
certifi==2020.6.20
cffi==1.14.1
chardet==3.0.4
click==7.1.2
client @ git+https://github.com/AirGrid/mind.git@482982a9dbac1275c85cd7e441feec525b041caa#subdirectory=serverless_crawler/client
cloudevents==0.3.0
cloudpickle==1.6.0
clusteval==1.0.2
colorcet==2.0.2
comcrawl==1.0.2
configparser==5.0.0
cookiecutter==1.6.0
courlan==0.2.3
coverage==5.3
crcmod==1.7
cryptography==3.0
cssselect==1.1.0
cycler==0.10.0
cymem==2.0.3
Cython==0.29.21
dask==2.25.0
datashape==0.5.2
decorator==4.4.2
defusedxml==0.6.0
dill==0.3.3
distributed==2.21.0
docker==4.3.1
docutils==0.15.2
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.1/en_core_web_sm-2.3.1.tar.gz
enstop==0.2.5
entrypoints==0.3
fastcore==1.3.10
filelock==3.0.12
firebase-admin==4.4.0
Flask==1.1.2
flatbuffers==1.12
fsspec==0.6.3
functions-framework==2.0.0
funcy==1.14
furl==2.1.0
future==0.18.2
gast==0.3.3
gensim==3.8.3
google-api-core==1.23.0
google-api-python-client==1.12.3
google-auth==1.23.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.1
google-cloud-bigquery==2.6.0
google-cloud-bigquery-storage==2.1.0
google-cloud-core==1.4.1
google-cloud-firestore==1.9.0
google-cloud-logging==1.15.1
google-cloud-secret-manager==2.0.0
google-cloud-storage==1.30.0
google-crc32c==0.1.0
google-pasta==0.2.0
google-resumable-media==0.7.0
googleapis-common-protos==1.52.0
graphviz==0.8.4
grpc-google-iam-v1==0.12.3
grpcio==1.32.0
gunicorn==20.0.4
h5py==2.10.0
hdbscan==0.8.26
HeapDict==1.0.1
holoviews==1.13.3
html2text==2020.1.16
htmldate==0.7.2
httplib2==0.18.1
humanfriendly==8.2
ibm-cloud-sdk-core==1.5.1
ibm-watson==4.5.0
idna==2.10
imagesize==1.2.0
importlib-metadata==1.7.0
importlib-resources==3.3.0
ipykernel==5.3.4
ipython==7.16.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
isort==4.3.21
itsdangerous==1.1.0
jedi==0.17.2
jellyfish==0.8.2
Jinja2==2.11.2
jinja2-time==0.2.0
jmespath==0.10.0
joblib==0.16.0
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.6
jupyter-console==6.1.0
jupyter-core==4.6.3
jusText==2.2.0
kedro==0.16.6
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
libcst==0.3.13
lightfm==1.15
llvmlite==0.34.0
locket==0.2.0
lxml==4.5.2
Mako==1.1.3
Markdown==3.2.2
MarkupSafe==1.1.1
matplotlib==3.3.0
mccabe==0.6.1
mistune==0.8.4
more-itertools==8.4.0
msgpack==1.0.0
multidict==4.7.6
multipledispatch==0.6.0
murmurhash==1.0.2
mxnet==1.7.0.post1
mypy-extensions==0.4.3
nbconvert==5.6.1
nbdev==1.1.5
nbformat==5.0.7
networkx==2.4
nltk==3.5
notebook==6.0.3
numba==0.51.2
numexpr==2.7.1
numpy==1.19.4
oauthlib==3.1.0
opt-einsum==3.3.0
orderedmultidict==1.0.1
packaging==20.4
pandas==1.0.5
pandocfilters==1.4.2
panel==0.9.7
param==1.9.3
parsel==1.6.0
parso==0.7.1
partd==1.1.0
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
pip==20.3.3
pip-autoremove==0.9.1
pip-tools==5.3.0
pipx==0.15.4.0
plac==1.1.3
plotly==4.12.0
pluggy==0.13.1
ply==3.11
poyo==0.5.0
preshed==3.0.2
prometheus-client==0.8.0
promise==2.3
prompt-toolkit==3.0.5
proto-plus==1.11.0
protobuf==3.12.2
psutil==5.7.2
ptyprocess==0.6.0
py==1.9.0
-e git+https://github.com/AirGrid/py-utils.git@b89ba5a0ff82fe15510cf82613a1d1a1808baa76#egg=py_utils
py-zipkin==0.20.0
pyarrow==2.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.5.0
pycld3==0.20
pycparser==2.20
pyct==0.4.6
pydash==4.9.0
Pygments==2.6.1
PyJWT==1.7.1
pyLDAvis==2.1.2
pylint==2.5.3
pynndescent==0.4.8
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyrsistent==0.16.0
pytest==5.4.3
pytextrank==2.0.3
python-dateutil==2.8.1
python-editor==1.0.4
python-json-logger==0.1.11
pytz==2020.1
pyviz-comms==0.7.6
PyYAML==5.3.1
pyzmq==19.0.1
qtconsole==4.7.5
QtPy==1.9.0
readability-lxml==0.8.1
regex==2020.7.14
requests==2.24.0
requests-file==1.5.1
requests-oauthlib==1.3.0
retrying==1.3.3
rsa==4.6
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
s3transfer==0.3.3
sacremoses==0.0.43
scikit-learn==0.23.2
scipy==1.5.2
seaborn==0.10.1
segtok==1.5.10
Send2Trash==1.5.0
sentencepiece==0.1.94
setuptools==50.0.0
six==1.15.0
sklearn==0.0
smart-open==2.1.0
snowballstemmer==2.0.0
sortedcontainers==2.2.2
soupsieve==2.0.1
spacy==2.3.2
Sphinx==3.3.1
sphinx-autodoc-typehints==1.11.1
sphinx-markdown-builder==0.5.4
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
SQLAlchemy==1.3.19
SQLAlchemy-Utils==0.36.8
srsly==1.0.2
stanza==1.1.1
tabulate==0.8.7
tbb==2019.0
tblib==1.7.0
tenacity==6.2.0
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow==2.4.0
tensorflow-addons==0.12.0
tensorflow-datasets==4.1.0
tensorflow-estimator==2.4.0
tensorflow-metadata==0.26.0
tensorflow-recommenders==0.3.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
textmap @ git+https://github.com/TutteInstitute/TextMAP.git@150d939b4fb5b6dbcda057ef782ef442c042e7a5
thinc==7.4.1
Thingy==0.8.5
threadpoolctl==2.1.0
thriftpy2==0.4.11
tldextract==2.2.2
tokenizers==0.9.4
toml==0.10.1
toolz==0.10.0
toposort==1.5
torch==1.6.0
torchvision==0.7.0
tornado==6.0.4
tqdm==4.55.1
trafilatura==0.5.2
traitlets==4.3.3
transformers==4.1.1
typed-ast==1.4.1
typeguard==2.10.0
typing-extensions==3.7.4.2
typing-inspect==0.6.0
umap-learn==0.4.6
unify==0.5
untokenize==0.1.1
uritemplate==3.0.1
urllib3==1.25.11
userpath==1.4.1
vectorizers @ git+https://github.com/TutteInstitute/vectorizers.git@eff54167d0e68bbf35ec946a95ffd59444454c77
w3lib==1.22.0
warcio==1.7.4
wasabi==0.8.0
watchdog==0.10.3
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.48.0
Werkzeug==1.0.1
wheel==0.36.2
whichcraft==0.6.1
widgetsnbextension==3.5.1
wikipedia==1.4.0
Wikipedia-API==0.5.4
wordcloud==1.8.0
wrapt==1.12.1
xarray==0.16.0
yake @ git+https://github.com/LIAAD/yake@2c9fedbe0e3195a273a3c3daef3d41d4d7c5b3b3
yapf==0.30.0
yarl==1.5.1
zenserp==0.2
zict==2.0.0
zipp==3.1.0

Issue description

The keras callback is failing to write files to the folder:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-81-3f8d7320ec70> in <module>
     18           verbose=0,
     19           validation_data=cached_test,
---> 20           callbacks=[tensorboard_callback]
     21          )
     22 

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1143           epoch_logs.update(val_logs)
   1144 
-> 1145         callbacks.on_epoch_end(epoch, epoch_logs)
   1146         training_logs = epoch_logs
   1147         if self.stop_training:

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_epoch_end(self, epoch, logs)
    426     for callback in self.callbacks:
    427       if getattr(callback, '_supports_tf_logs', False):
--> 428         callback.on_epoch_end(epoch, logs)
    429       else:
    430         if numpy_logs is None:  # Only convert once.

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_epoch_end(self, epoch, logs)
   2337 
   2338     if self.histogram_freq and epoch % self.histogram_freq == 0:
-> 2339       self._log_weights(epoch)
   2340 
   2341     if self.embeddings_freq and epoch % self.embeddings_freq == 0:

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _log_weights(self, epoch)
   2395         for layer in self.model.layers:
   2396           for weight in layer.weights:
-> 2397             weight_name = weight.name.replace(':', '_')
   2398             summary_ops_v2.histogram(weight_name, weight, step=epoch)
   2399             if self.write_images:

AttributeError: 'TrackableWeightHandler' object has no attribute 'name'

Code:

model = RecModel()

log_dir = 'logs/fit/' + datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=LEARNING_RATE))
history = model.fit(cached_train, 
          epochs=EPOCHS, 
          verbose=0, 
          validation_data=cached_test, 
          callbacks=[tensorboard_callback]
         )

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 4
Comments: 18

Most upvoted comments

@ydennisy a bit of a crude workaround to get this working is to subclass the TensorBoard Callback and explicitly check if there is a name attribute. This assumes that you do not want a histogram of those experimental layers (which is probably a fair assumption as not much learning happening there).

I fully expect there will be a more principled way to handle these experimental layers in the future but hopefully this should get you around this current issue.

class TBCallback(keras.callbacks.TensorBoard):
    def _log_weights(self, epoch):
        with self._train_writer.as_default():
            with summary_ops_v2.always_record_summaries():
                for layer in self.model.layers:
                    for weight in layer.weights:
                        if hasattr(weight, "name"):
                            weight_name = weight.name.replace(':', '_')
                            summary_ops_v2.histogram(weight_name, weight, step=epoch)
                            if self.write_images:
                                self._log_weight_as_image(weight, weight_name, epoch)
                self._train_writer.flush()

mikedavy on Feb 22, 2021

Unfortunately https://github.com/tensorflow/tensorboard/issues/4530#issuecomment-774331534 is still the best suggestion I have (avoid problematic layers if using the keras callback). I’ve bumped the internal issue as well.

nfelt on Oct 20, 2022

This issue arises in particular when using categorical string layers (e.g. tensorflow.keras.layers.experimental.keras.preprocessing.StringLookup). We’re looking into how this can be fixed, but it’s more of an issue with the way the layer is structured than with the TensorBoard callback per se so we don’t have much control over how quickly it will be resolved. (Googlers, see b/169618190.)

For now my best suggestion is to avoid using that layer; it is marked as experimental still.

nfelt on Feb 5, 2021

This looks like the same issue as: https://github.com/tensorflow/tensorflow/issues/41244

EDIT: Setting histogram_freq=0 stops this issue.

Is there a way to manually assign names to all layers?

ydennisy on Jan 22, 2021