flow: The `vaadinPrepareFrontendTask` gradle task is not cached
Description of the bug
We have upgrade to Vaadin 24.2.0 with the hopes to benefit from the newly introduced incremental Gradle task caching for vaadinPrepareFrontendTask (related PR is https://github.com/vaadin/flow/pull/17150).
Unfortunately our local and CI builds seem to be unable to benefit from this feature. Upon closer investigation we noticed Gradle scans show the following notice:
Overlapping outputs: Gradle does not know how file 'package.json' was created (output property 'taskOutputProperties.packageJson').
Task output caching requires exclusive access to output paths to guarantee correctness
Expected behavior
The vaadinPrepareFrontendTask should show UP-TO-DATE or FROM CACHE in the gradle build scans and the task should not be computing again.
Minimal reproducible example
Not sure how to reproduce yet, but we don’t have any special setup.
We are using the gradle vaadin plugin and run gradle build.
Versions
- Vaadin / Flow version: 24.2.0
- Java version: 17
- OS version: ubuntu-22.04 (GH CI)
- Browser version (if applicable):
- Application Server (if applicable):
- IDE (if applicable):
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 6
- Comments: 51 (31 by maintainers)
Commits related to this issue
- fix: Do not touch package.json files unless content has changed (#18045) In some cases, at least the package.json file is overwritten with the same content it already had. This causes the timestamp t... — committed to vaadin/flow by tepi 3 years ago
- fix: Do not touch package.json files unless content has changed (#18045) In some cases, at least the package.json file is overwritten with the same content it already had. This causes the timestamp t... — committed to vaadin/flow by tepi 3 years ago
- fix: Do not touch package.json files unless content has changed (#18045) (#18055) In some cases, at least the package.json file is overwritten with the same content it already had. This causes the ti... — committed to vaadin/flow by vaadin-bot 3 years ago
Hey guys,
Apologies for the late reply.
What you’ve described is exactly what’s happening. The original problem is that the
package.jsonis overwritten during the CI build and then for some reason it’s never able to be cached. Even if thepackage.jsonandpackage-lock.jsonare synced and committed to VCS thevaadinPrepareFrontendtask is still not cached. We also tried clearing the GH Actions cache and running a few runs on themainbut the result remains the same.I am not entirely sure where this is coming from exactly, but the closest I could get to reproducing the problem was by introducing out-of-sync in the
package.jsonas shown in https://github.com/steve-todorov/base-starter-gradle/tree/issue-17941. This leads me to believe something additional is happening in thevaadinBuildFrontendtask that might be changing thepackage.jsonorpackage-lock.jsoncausing the problem.I’ll record the
package.jsonandpackage-lock.jsonduring the CI and diff them in hopes to get more clarity and report back.It would be super amazing to get this fixed. Shaving 3+ mins off each Github Actions build would be a game changer for my project.
I did some tests both locally and on our gradle starter repository.
Locally, the cache seems to work fine. If I build the project and then do a clean (removing also
frontend/generatedstuff) and rebuild, the log says:vaadinPrepareFrontend FROM-CACHE.If I do not remove the
frontend/generatedfolder, I get a non-cachable warning, but not for package.jsonOn GitHub actions I always get the non-cachable warning for
package.jsoneven if the file is not changed in any way; in the job summary the checksum, creation and modification times are the same before and after the gradle build.Excerpt of GH build</summary
Hey @tepi, @mshabarov ,
We’ve just upgraded to 24.2.5, but it looks like this does not fix the problem for us. We are still getting the same error in GH Actions CI (after complete cache nuking):
Locally it seems to be working as expected. Is there any
project paththat you might be caching? For example thenode_modules/.vaadin/vaadin.jsonseems to contain an absoluteprojectFolder. If you are caching that on the CI, then that path might be changing depending on which agent the job lands on.I have
rebasedthe fork of thebase-starterand you can see the build here: https://github.com/steve-todorov/base-starter-gradle/actions/runs/7037473808Scan is here: https://scans.gradle.com/s/e6d5xqndde4qe/performance/execution
Should we re-open this ticket or open a new one?
With the provided example run locally,
vaadinBuildFrontendupdatespackage.jsononce (the outdated frontend dependency) and after that no more updates.@steve-todorov thanks for the update, we will investigate further if the proposed workaround works well.
@steve-todorov Yup, that sounds like a good way forward. I’ve created a feature request at https://github.com/gradle/gradle/issues/27696 and I agree to continue exactly as you suggested. Let’s discuss the ticket with the Gradle guys a bit first; depending on the outcome we can decide appropriate course of action.
@mvysny thanks for your continuous efforts into debugging and solving this problem!
Although I see your point of view as valid, I believe waiting for this feature request to be resolved might take a considerable time. Let’s open a ticket and see how it goes and if the Gradle team would be open to looking into this.
In the mean time or if Gradle decide to decline fixing this as it is “working as intended” – it might be more useful to create the necessary additional tasks and properly document why they’re needed and how they work. That should make things easier to understand for other team members and potentially easier to maintain.
I think I finally understand what the problem is.
Vaadin plugin is modifying
package.jsonin-place - it reads the json file, and if there is a need for modification, it modifies thepackage.jsonfile. Therefore, the property should be annotated both with@InputFileand@OutputFilebut that doesn’t work. I’ve asked a question on SO, hopefully we’ll learn more: https://stackoverflow.com/questions/77807234/gradle-cache-task-which-processes-a-file-in-placeTherefore, currently the
package.jsonis annotated only with@OutputFile. However, that creates a huge build-cache-related problem which is not immediately visible. If thepackage.jsonfile doesn’t exist then everything works: the outcome ofvaadinPrepareFrontendis cached in the build cache and the build cache is reused during the next run ofvaadinPrepareFrontend. However, if thepackage.jsonfile exists before the build cache is populated, then the following happens:package.jsonexists and Gradle doesn’t have it in its build cache, it will print the “Gradle does not know how file” blah blah overlapping error message. It will run thevaadinPrepareFrontendtask fully.Caching disabled for task ':compileJava' because:. That means that even thoughvaadinPrepareFrontendran fully, its outcome is not recorded in Gradle’s build cache!package.jsonexists and Gradle doesn’t have it in its build cache. Goto 1.This behavior is reproducible also with the Java Compiler Gradle task as documented at https://mvysny.github.io/gradle-overlapping-outputs/ .
I’m not sure whether this is a bug in Gradle or whether this is desired behavior; it could be a Gradle protection from some kind of cache corruption as documented in the forums: https://discuss.gradle.org/t/gradle-7-0-seems-to-take-an-overzealous-approach-to-inter-task-dependencies/39656 . Therefore, instead of trying to find a workaround to cache output of a task which ran into “overlapping outputs”, I’d propose to mark the package.json property properly.
I’ve added
frontend/generatedinto github’s action cache to see if that would solve it. Unfortunately it does not. This problem might be related to NPM doing something behind the scenes to thepackage.jsonor perhaps there is an environment difference that causes the problem to appear in GH Actions.If anybody has time to dig into this you could use a custom self-hosted runner from https://github.com/myoung34/docker-github-actions-runner and try to debug it locally (i.e. log into the container and check what’s really going on)
Reopened for further investigation.
Fix has been applied to Vaadin Flow 24.3 (pre) and 24.2, will be released soon for both branches and available for testing.
I’m looking into the issue currently, no need to create a new issue, I will create one when needed. Right now I can’t reliably reproduce the timestamp update of
package.jsonfile. With identical setup it sometimes updates and sometimes does not update for my local build. Will keep you posted when I figure out what is going on.Can confirm point 1 from @mvysny above; the timestamp does update even though the contents do not change. This might be the underlying reason for this behavior. Unfortunately could not find any way to disable this update - rather I consider it a bug that needs to be fixed, since it makes no sense to write the file if the contents are identical.
Thanks @mvysny @oliveryasuna @knoobie for you analysis! I like the idea to open a new one, but before I would like to understand what changes in
package.jsonare made byvaadinBuildFrontendfor @steve-todorov .Thanks for the issue. We will try to reproduce with our Gradle-based starters and get back to you soon.