nbformat: Notebook Validation Error using nbformat 5.x with notebooks that have widgets

I’m getting this error when using nbconvert to run notebooks that have widgets in them. This only occurs after upgrading from nbformat 4.x to 5.0.3. Here’s a full traceback, which you can also peruse at https://travis-ci.org/ketch/riemann_book/jobs/637262913. The tests there have been running just fine for two years, but started throwing this error after switching to a newer version of nbformat.

Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
    nbformat.validate(nbc, relax_add_props=True)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbformat/validator.py", line 277, in validate
    raise error
nbformat.validator.NotebookValidationError: {'version_major': 2, 'version_minor': 0, 'model_id': '1cfed2a3899e49a3af35520ca75e8220'} is not valid under any of the given schemas
Failed validating 'oneOf' in display_data['properties']['data']['patternProperties']['^(?!application/json$)[a-zA-Z0-9]+/[a-zA-Z0-9\\-\\+\\.]+$']:
On instance['cells'][47]['outputs'][1]['data']['application/vnd.jupyter.widget-view+json']:
{'model_id': '1cfed2a3899e49a3af35520ca75e8220',
 'version_major': 2,
 'version_minor': 0}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/python/3.6.7/lib/python3.6/logging/__init__.py", line 994, in emit
    msg = self.format(record)
  File "/opt/python/3.6.7/lib/python3.6/logging/__init__.py", line 840, in format
    return fmt.format(record)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/traitlets/config/application.py", line 117, in format
    return super(LevelFormatter, self).format(record)
  File "/opt/python/3.6.7/lib/python3.6/logging/__init__.py", line 577, in format
    record.message = record.getMessage()
  File "/opt/python/3.6.7/lib/python3.6/logging/__init__.py", line 338, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/home/travis/virtualenv/python3.6.7/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/jupyter_core/application.py", line 268, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 340, in start
    self.convert_notebooks()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 510, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 481, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 410, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 179, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 197, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super(NotebookExporter, self).from_notebook_node(nb, resources, **kw)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 139, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 321, in _preprocess
    preprocessor)
Message: 'Notebook is invalid after preprocessor {}'
Arguments: (<nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x7f6f383ca748>,)
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.7/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/jupyter_core/application.py", line 268, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 340, in start
    self.convert_notebooks()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 510, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 481, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 410, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 179, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 197, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super(NotebookExporter, self).from_notebook_node(nb, resources, **kw)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 139, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
    nbformat.validate(nbc, relax_add_props=True)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/nbformat/validator.py", line 277, in validate
    raise error
nbformat.validator.NotebookValidationError: {'version_major': 2, 'version_minor': 0, 'model_id': '1cfed2a3899e49a3af35520ca75e8220'} is not valid under any of the given schemas
Failed validating 'oneOf' in display_data['properties']['data']['patternProperties']['^(?!application/json$)[a-zA-Z0-9]+/[a-zA-Z0-9\\-\\+\\.]+$']:
On instance['cells'][47]['outputs'][1]['data']['application/vnd.jupyter.widget-view+json']:
{'model_id': '1cfed2a3899e49a3af35520ca75e8220',
 'version_major': 2,
 'version_minor': 0}
Traceback (most recent call last):
  File "test.py", line 50, in <module>
    run_tests()
  File "test.py", line 44, in run_tests
    _, errors = _notebook_run(filename)
  File "test.py", line 23, in _notebook_run
    subprocess.check_call(args)
  File "/opt/python/3.6.7/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['jupyter', 'nbconvert', '--to', 'notebook', '--execute', '--ExecutePreprocessor.timeout=60', '--ExecutePreprocessor.kernel_name=python3', '--output', '/tmp/tmp6zp9xx8t.ipynb', 'Euler.ipynb']' returned non-zero exit status 1.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 20 (7 by maintainers)

Commits related to this issue

Save notebook with jupyter lab to fix issue with new notebook format in the "classic" notebook. https://github.com/jupyter/nbformat/issues/161#issuecomment-577412867 — committed to ericpre/hyperspy-demos by ericpre 4 years ago
Save notebook with jupyter lab to fix issue with new notebook format in the "classic" notebook. https://github.com/jupyter/nbformat/issues/161#issuecomment-577412867 — committed to ericpre/hyperspy-demos by ericpre 4 years ago
Fix issue with newer version of nbformat. The newer version of nbformat (5.x) was failing to process the old notebook format. The fix was to set nbformat_minor: 4. This issue and solution are discuss... — committed to zivy/SimpleITK-Notebooks by zivy 4 years ago
Save notebook with jupyter lab to fix issue with new notebook format in the "classic" notebook. See https://github.com/jupyter/nbformat/issues/161#issuecomment-577412867 — committed to ericpre/hyperspy_tutorial by ericpre 4 years ago
Save notebook with jupyter lab to fix issue with new notebook format in the "classic" notebook. See https://github.com/jupyter/nbformat/issues/161#issuecomment-577412867 — committed to ericpre/hyperspy_tutorial by ericpre 4 years ago
Bump notebook versions to 4.4 This should hopefully allow nbformat to work again, see https://github.com/jupyter/nbformat/issues/161 — committed to ska-telescope/sdp-par-model by scpmw 4 years ago
Bump notebook versions to 4.4 This should hopefully allow nbformat to work again, see https://github.com/jupyter/nbformat/issues/161 — committed to ska-telescope/sdp-par-model by scpmw 4 years ago
Allow notebook format version 4.1 mimebundle keys ending in `+json` to have arbitrary JSON There are a number of people posting issues with nbformat 5 being stricter about validating notebook format ... — committed to jasongrout/nbformat by jasongrout 4 years ago

Most upvoted comments

Yes that is Notebook Classic. To repair the issue launch jupyter lab, load the notebook and save it back. This will automatically update the notebook format so classic will be able to load and manipulate the document again. Usually this process is just updating the minor version number as suggested above, but it could include other small changes to fix the notebook in other places as the format changed over the years.

The longer explanation is below:

Essentially nbformat 5 release added validation for the older format versions instead of using the 4.4 format checks regardless of document version. This was causing invalid format issues with nbformat < 5 when evaluating notebook documents, so the newer package is more strict to avoid errors in notebook processing. This caused manipulations of older formats to fail in places where they were succeeding before because those manipulations were likely not actually spec compliant and your notebook was acting like a 4.4 notebook as best it can. The classic notebook server has been in a frozen / undeveloped state for many years, with lab, nteract, colab, and sagemaker making new interfaces.

+12

MSeal on Jan 22, 2020

It looks like the notebook in question is stored in the 4.1 schema, which wouldn’t tolerate the “fancy” widget mime type. It’s valid on disk, but once it gets your widgets added, it becomes invalid. Older nbformat would probably tolerate it, but a major version bump can cause some trouble, for sure!

Quick fixes would be:

open and save all the notebooks once
see if replacing "nbformat_minor": 1 with "nbformat_minor": 4 fixes it, or uncovers entirely new problems (unlikely)
go back to 4.x if it was working for you!

The stuff built into nbconvert --nbformat won’t let you migrate between minor versions, and dropping down to v3 to go back up to v4 seems more brittle than changing one number. Perhaps we could offer that.

+12

bollwyvl on Jan 15, 2020

I don’t remember the error message now, and I can’t seem to reproduce it…

I guess it might have looked this way:

slayoo on Feb 13, 2020

Yes that is Notebook Classic. To repair the issue launch jupyter lab, load the notebook and save it back. This will automatically update the notebook format so classic will be able to load and manipulate the document again. Usually this process is just updating the minor version number as suggested above, but it could include other small changes to fix the notebook in other places as the format changed over the years.

The longer explanation is below:

Essentially nbformat 5 release added validation for the older format versions instead of using the 4.4 format checks regardless of document version. This was causing invalid format issues with nbformat < 5 when evaluating notebook documents, so the newer package is more strict to avoid errors in notebook processing. This caused manipulations of older formats to fail in places where they were succeeding before because those manipulations were likely not actually spec compliant and your notebook was acting like a 4.4 notebook as best it can. The classic notebook server has been in a frozen / undeveloped state for many years, with lab, nteract, colab, and sagemaker making new interfaces.

REOPENING:

I am reopening this issue, because although a great reply was given that explains things well, we need some type of more automated solution for it.

I am looking for a bulk conversion approach to convert a batch of “old v4 classic format notebooks” into the newer JupyterLab format.

What utility exists to do this, and what are some command line examples to do the bulk conversion?

richlysakowski on Mar 1, 2020

@MSeal Sure I can. But should I?

I was just saying it’s possible and may not be worth writing into nbformat.

Options shouldn’t get more strict on old files as the time goes.

At some point old versions need to fall off supported paths. Within the same major version should probably be supported, so it doesn’t really apply here.

If this was not, then, when changing your library behaviour, add a ticket for the notebook lib to fix that issue, and provide a way to communicate this from upper level library to nbformat library.

Sure, it just wasn’t clear the validation added would cause old notebooks to complain in classic when this was released. This change was contributed from different people at different times. I only ask to be a little understanding that most of us are maintaining this in our free time and there’s a lot of projects interconnected we have to track. It would help us if you could open an issue in the other projects even as not all of us work on all the projects. It’d help even more if you had the time to try to make the change in that repo. If you’d be kind enough to open that I can comment if there’s clarifying questions needed. Otherwise I can get to doing that later, just time limited on my end.

Why users should become the ones who get the error message on their faces when they did nothing wrong?

I agree, but it’s not like this was malciously done. Matter of fact it was added because there were bugs with loading 4.x versions that were actually 4.x spec instead of 4.4 and causing weird parse errors, rather than a fairly clear “Hey your version is out of date” message where it’s not always the right thing to bump the minor version.

Today nbformat doesn’t do notebook version updates on client’s behalf. Lab auto-upgrades this already, and most things are tested against lab or the headless execution modes like papermill. So there is a clear work-around, and clear guidance for how to remedy the issue as well for both actual old versions and incorrectly version indicators in files.

MSeal on Mar 5, 2020