webpack: Lazy compilation can fail when switching "routes" quickly
Bug report
What is the current behavior?
When using the experimental lazy compilation feature (in Storybook), when you change stories quickly (thus triggering many different lazy compilation jobs), sometimes the HMR for the lazy update can fail in various ways, which can lead to react-refresh errors or simply the whole app having to reload itself.
If the current behavior is a bug, please provide the steps to reproduce.
It is not entirely predictable, and I have not managed to reproduce outside of Storybook (yet), but this is a very very simple Storybook project and I think it is quite obvious the bug is in Webpack, but I am ready to be corrected đ
https://github.com/tmeasday/storybook-lazy-compilation-webpack â see the instructions from the repo.
The key point in this reproduction is that when a bunch of lazy updates are created on the server, the browser, which is otherwise doing nothing, will sometimes 404 on a hot-update.json request:
Note we have seen other (less reproducible) issues around HMR and lazy compilation, that you might see if you play with this repository, including:
- Sometimes the storybook fails to load initially with a HMR error. This definitely seems to be a timing issue with when the auto-launched browser loads relative to the initial entries compilation (it happens maybe 10% of the time or less when I simply run
rm -rf node_modules/.cache/ && yarn storybookand let the browser launch).
- You can also trigger a bug by changing stories in the sidebar directly quickly while lazy compilation is going on (i.e. start SB, open up, change stories as fast as you can (you can use
option-downandoption-upto do this). However I cannot make this happen reliably enough to create a reproduction that would be useful. If youâd like to see what I mean I can likely get a screen recording of it happening at least.
What is the expected behavior?
Lazy compilation should be fairly transparent to the consumer, beyond import() requests taking a little longer than they otherwise might. If updates come over the wire from compilations triggered by other browsers, they should not fail.
Other relevant information: webpack version: 5.70.0 Node.js version: 14.18.1 Operating System: MacOS Additional tools: Storybook 6.5.0-alpha.48
NOTE
The tool we are using here to trigger the bug is the Storybook test runner, which opens the Storybook up in 4 separate, concurrent playwright instances, each of which will trigger a different dynamic (and thus lazy-compiled) import() statement.
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 1
- Comments: 21 (16 by maintainers)
@vankop I dug into this a bit and I think I have some more context on what is going wrong here, in case it helps. Apologies in advance for my lack of understanding of how HMR stuff works in detail.
So, just to reiterate, whatâs happening in the failing repro above is that we are
import()-ing three files with a very small delay between each import.Actually the issue can happen even when even importing two files with a delay, but it doesnât happen every time. Itâs interesting to see what happen when it does work vs when it does.
When the lazy compilation works, on the terminal (with all logging turned on), I see:
Notice in particular the two messages
.... is now in use and will be compiled.that come in around the same time. This message is logged right before thelazyCompilationBackendcallscompiler.watching.invalidate().The two modules do not actually get âusedâ at the same time, there is a 2ms gap between them, but for whatever reason the new compilation triggered by the invalidate hasnât actually started yet.
On the client, on the HMR socket we see a really straightforward set of messages for a single HMR update and it all works fine:
When it doesnât work, it is because the compilation starts before the second invalidation:
In this case, notice the âCompilation startingâŚâ message is logged before the ââŚb.js is now in use and will be compiled.â message. Ultimately this leads to the following messages over the websocket:
(Hereâs where I get a bit more hazy) â what this means is that there are actually two âchangesâ in compilation â we go from an initial hash, to a second hash with
a.js, to a third hash witha.jsandb.js. However, the HMR mechanism tries to fetch thehot-update.jsonbased on the initial hash, after the compilation of the third hash, and it no longer exists in the dev server.I am not sure exactly what the root problem is. I suspect that maybe the lazy compilation backend shouldnât (or shouldnât be able to) invalidate the compilation while it is compiling. Perhaps somewhere in the pipeline there should be a queue of invalidations that blocks on previous compilations?
Yeah, sorry đ
Thank you I will look soon
@emiliegervais let me reopen that issue and we can discuss there.
@tmeasday Something which I am still facing too in my repo. Issue: https://github.com/storybookjs/storybook/issues/22706 This is how it looks. If you see in the video, whenever i click on any component it is downloading duplicate assets and resources and the page continuosly refreshes. I am using
https://github.com/webpack/webpack/assets/8790510/e0a3647b-9bae-4857-a14e-bceec54bc241
As a note we are still working around this problem in Storybook in a way which probably isnât ideal for performance: https://github.com/storybookjs/storybook/blob/15dd0ad6891e964f807a22e3e3827c0e7f599a1b/code/lib/core-webpack/src/to-importFn.ts#L50-L55