three.js: Automatic regression testing with CI
Description of the problem
Right now, it seems threejs is not testing the actual shaders code in any automated way and just relies on testing the examples and hoping that works out.
With the growing number of examples, that is quickly becoming too much.
Is there a plan to solidify this part of the testing pipeline, such as is done by https://github.com/GoogleWebComponents/model-viewer (who use threejs) or https://github.com/Kitware/vtk-js-datasets/tree/master/data/vtkjs ?
Thanks for the great library!
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 18 (6 by maintainers)
The framework I was working on take cares of that. It basically hooks all the APIs possible, for example
math.random, performance.now(), date.now()returning always the same sequence based on a seed number that can be define by example. The same goes for the canvas size, as most examples set it to full window size, so depending on how big your browser window or monitor is, the example will be rendering at different sizes. I hook the canvas and webgl context for that and make sure each test run always at the same size, no matter your device. Also I added VR support, so you can set a static pose for the camera, or even replay some recorder movement. The same goes for the key and mouse input, you can play the example in record mode, na it will record every input and then replay on the exact frame you introduced it.Basically I tried to ensure that every run is deterministic, otherwise it would be very hard to compare the numbers. I’m planning to do a video/post about the framework so you could better understand how this work before integrating it here. But basically when I was designing it I had three.js in mind too for the type of problems we have been having in the past in regressions. Apart from stats on cpu time/idle, fps, and so, it also records the webgl drawcalls, textures, programs, so problems as when the frustum culling was broken, will be quite easy to spot with it.
I have pretty much all the pieces to start working on this using a framework I was working on in the last months for exactly this type of tasks: https://github.com/MozillaReality/webgfx-tests This will help both with the regression testing on rendering (comparing with reference images) but also on performance regression. Adding it to three.js is on my to-do list hopefully I could get unblocked soon and start working on it soon 😃
Yep I was working on an implementation based on something that the Mozilla Games team was doing on their testing basically what you said, you hook the fetch/xhr request and wait until they are done, you can configure time threshold on them to start or to wait for these requests. Also the test could generate config files based on the execution so it could know how many request the example will be doing and wait for them beforehand without need to “guess” them. The RAF loop is basically waiting until all the requests are ready, and it will start as soon as they are done
This would be fantastic – I was hoping that #13017 would get merged but it looks like it got held up.
I’ve been looking into writing something for my own work to support CI and using Mapbox’s pixelmatch package to perform the image diffs. If there could be any sort of plan or vision around this I’d be happy to help move it forward.