Cirq: Notebook test slow tests
Probably worth trying to speed up that notebook test as it is long poll in the CI. Maybe there is a pattern to add something that can “fix” the notebooks to smaller parameters to speed them up?
63.27s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/cirq/contrib/quimb/Cirq-to-Tensor-Networks.ipynb]
42.70s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/quantum_walks.ipynb]
33.26s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/educators/qaoa_ising.ipynb]
28.50s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/hidden_linear_function.ipynb]
27.31s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/educators/chemistry.ipynb]
24.82s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/qaoa.ipynb]
21.05s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/variational_algorithm.ipynb]
14.91s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/cirq/contrib/quimb/Contract-a-Grid-Circuit.ipynb]
11.20s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/rabi_oscillations.ipynb]
10.77s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/interop.ipynb]
9.48s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/educators/intro.ipynb]
9.44s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/educators/textbook_algorithms.ipynb]
7.76s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/circuits.ipynb]
7.73s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/educators/ion_device.ipynb]
7.45s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/simulation.ipynb]
7.29s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/basics.ipynb]
6.34s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/qubits.ipynb]
6.30s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/transform.ipynb]
6.26s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/educators/neutral_atom.ipynb]
6.21s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/qudits.ipynb]
6.19s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/protocols.ipynb]
6.13s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/custom_gates.ipynb]
6.04s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/noise.ipynb]
6.03s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/tutorials/shor.ipynb]
5.93s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/cirq/contrib/svg/example.ipynb]
5.84s call dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/docs/gates.ipynb]
0.11s setup dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/cirq/contrib/quimb/Contract-a-Grid-Circuit.ipynb]
0.06s setup dev_tools/notebook_test.py::test_notebooks[/Users/bacon/code/github/Cirq/cirq/contrib/svg/example.ipynb]
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 25 (12 by maintainers)
Commits related to this issue
- only test notebooks that are checked into git (#3713) This helps with local running of the notebook tests - it will only pick up notebooks that are checked into git. Related to https://github.com/qu... — committed to quantumlib/Cirq by balopat 3 years ago
- Notebook tests against both released Cirq and PRs (#3751) This PR adds notebook testing based on the PR branch in a non-isolated environment (hence it's faster). The isolated notebook tests against t... — committed to quantumlib/Cirq by balopat 3 years ago
- Speed up notebook tests by adding extra replacements file (#4077) Provides the ability to speed up notebook execution tests by substituting smaller values for parameters that impact the runtime of th... — committed to quantumlib/Cirq by dabacon 3 years ago
- Speed up notebook tests by adding extra replacements file (#4077) Provides the ability to speed up notebook execution tests by substituting smaller values for parameters that impact the runtime of th... — committed to rht/Cirq by dabacon 3 years ago
So I have a PR that can speed up the notebook tests significantly by parameterizing the notebooks and supplying reasonable numbers during test. However it is incompatible with the tensorflow nb document formatter which removes all sort of things including the tag you need to use for the parameterized cell. Grrr.
Thanks for investigating and those are some good ideas @dabacon. My 2 cents is we should give high priority to readability to notebooks in their intended form. Prioritize making our users happy rather than our github actions VM happy. It sounds like you’re on the same page!
if we could parametrize the notebooks in a way that they could be run in a “fast” mode for tests, each of them < 10s, it would save a lot of time, and create a faster feedback loop. Anything above 30 seconds is a bit of a suspect for me that could be probably easily sped up.