jupyter-book: Exit with non-zero status if notebooks encounter an unexpected exception.

Summary of the problem

Our project is switching to jupyter-book for documentation. Our documentation is built as part of our CI process using github actions. One challenge we’re encountering is ensuring that changes to our project code don’t break examples in our documentation.

Since the notebooks used for documentation are executed as part of our CI process now, it seems natural that any unexpected errors in our notebook should trigger a failure on our CI system. However, this is not the case.

When the notebooks are executed jupyter-book will report a successful build of documentation even if unexpected exceptions were raised. (in sphinx.py, the build process returns app.statuscode which doesn’t take into account whether any exceptions were raised in the notebook).

The solution we’d like

During the build process, myst-nb keeps track of which notebooks are successfully built, and which ones failed, as described here: https://myst-nb.readthedocs.io/en/latest/use/execute.html#execution-statistics

We propose allowing a new flag (perhaps -f or --fail-on-exception) that would cause the build process to return a non-zero value (perhaps the number of failed notebooks) in the event of an unexpected exception.

Notebooks meant to demonstrate exceptions could still allow them to be raised without issue by tagging their cells with raises-exception. The machinery for this is already in place and works.

This system would allow our documentation to “test itself” during the build process. At least on github actions, the log files which further describe the exceptions encountered can be saved as build artifacts to provide more information about what’s going wrong in the notebooks.

Alternatives we’ve considered.

We had a test system in place that tests notebooks and looks for errors. This works but is really inefficient. Our notebooks are evaluated twice during the build process and it basically doubles the job time on the CI system.

We’d considered other third-party notebook testing packages like testbook, but we didn’t have a great experience with them.

Having the jupyter-book build process serve as its own testing process under CI seems like a reasonable solution.

Additional context

I have a branch that demonstrates the functionality here: https://github.com/robfalck/jupyter-book/tree/fail_on_exception

As a demonstration, I created a new JupyterBook and made four copies of notebooks.ipynb. All of these notebooks included a new cell with

assert 1 == 2

Two of these notebooks tag that cell with raises-exception, and two do not.

Executing with the command

jupyter-book build -f .

Results in the following warnings during the build:

WARNING: Execution Failed with traceback saved in /Users/rfalck/Projects/test_error_book/_build/html/reports/notebooks2.log
WARNING: Execution Failed with traceback saved in /Users/rfalck/Projects/test_error_book/_build/html/reports/notebooks2.log

and the build ends with

The following notebooks encountered unexpected exceptions.
Add the tag 'raises-exception' to the offending cells to ignore these exceptions.

notebooks2
subfolder/notebooks2

===============================================================================

Building your book, returns a non-zero exit code (2). Look above for the cause.

===============================================================================

This demonstrates that the change allows exceptions when the raises-exception tag is present. The docs will also completely build when the -f switch is not present, so it doesn’t break current behavior.

I’m not sure what the correct process is here for submitting pull requests. I’d be happy to submit one with my branch, or make any changes that are wanted before doing so.