flow: The `vaadinPrepareFrontendTask` gradle task is not cached

Description of the bug

We have upgrade to Vaadin 24.2.0 with the hopes to benefit from the newly introduced incremental Gradle task caching for vaadinPrepareFrontendTask (related PR is https://github.com/vaadin/flow/pull/17150).

Unfortunately our local and CI builds seem to be unable to benefit from this feature. Upon closer investigation we noticed Gradle scans show the following notice:

Overlapping outputs: Gradle does not know how file 'package.json' was created (output property 'taskOutputProperties.packageJson'). 
Task output caching requires exclusive access to output paths to guarantee correctness

image

Expected behavior

The vaadinPrepareFrontendTask should show UP-TO-DATE or FROM CACHE in the gradle build scans and the task should not be computing again.

Minimal reproducible example

Not sure how to reproduce yet, but we don’t have any special setup. We are using the gradle vaadin plugin and run gradle build.

Versions

  • Vaadin / Flow version: 24.2.0
  • Java version: 17
  • OS version: ubuntu-22.04 (GH CI)
  • Browser version (if applicable):
  • Application Server (if applicable):
  • IDE (if applicable):

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 6
  • Comments: 51 (31 by maintainers)

Commits related to this issue

Most upvoted comments

Hey guys,

Apologies for the late reply.

@mvysny If vaadinBuildFrontend constantly keeps overwriting/modifying the package.json, then that’s definitely a bug which needs to be reported as a separate issue and resolved by Vaadin. You can try to reproduce such a bug by running full build vaadinBuildFrontend in production mode once on your dev machine, then commit package.json to git, then run the same build on other machines - the package.json should not be modified by consequent builds, and ultimately the vaadinPrepareFrontend task should be cached properly.

What you’ve described is exactly what’s happening. The original problem is that the package.json is overwritten during the CI build and then for some reason it’s never able to be cached. Even if the package.json and package-lock.json are synced and committed to VCS the vaadinPrepareFrontend task is still not cached. We also tried clearing the GH Actions cache and running a few runs on the main but the result remains the same.

I am not entirely sure where this is coming from exactly, but the closest I could get to reproducing the problem was by introducing out-of-sync in the package.json as shown in https://github.com/steve-todorov/base-starter-gradle/tree/issue-17941. This leads me to believe something additional is happening in the vaadinBuildFrontend task that might be changing the package.json or package-lock.json causing the problem.

I’ll record the package.json and package-lock.json during the CI and diff them in hopes to get more clarity and report back.

It would be super amazing to get this fixed. Shaving 3+ mins off each Github Actions build would be a game changer for my project.

I did some tests both locally and on our gradle starter repository.

Locally, the cache seems to work fine. If I build the project and then do a clean (removing also frontend/generated stuff) and rebuild, the log says :vaadinPrepareFrontend FROM-CACHE.

If I do not remove the frontend/generated folder, I get a non-cachable warning, but not for package.json

Non-cacheable because Gradle does not know how file 'frontend/generated/vaadin.ts' was created (output property 'outputProperties$vaadin_gradle_plugin.generatedTsFolder'). Task output caching requires exclusive access to output paths to guarantee correctness (i.e. multiple tasks are not allowed to produce output in the same location). [OVERLAPPING_OUTPUTS]

On GitHub actions I always get the non-cachable warning for package.json even if the file is not changed in any way; in the job summary the checksum, creation and modification times are the same before and after the gradle build.

Excerpt of GH build</summary
Pre build details
Checksums:

a550b63a7820f031dc93c27124801dbd  package.json
da23cc6f131fce28dfdb79955f59ed53  package-lock.json
File stats

  File: package.json
  Size: 10593     	Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d	Inode: 283258      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/  runner)   Gid: (  127/  docker)
Access: 2023-12-05 10:31:10.656979086 +0000
Modify: 2023-12-05 10:31:06.344961327 +0000
Change: 2023-12-05 10:31:06.344961327 +0000
 Birth: 2023-12-05 10:31:06.344961327 +0000


  File: package-lock.json
  Size: 576628    	Blocks: 1128       IO Block: 4096   regular file
Device: 801h/2049d	Inode: 283257      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/  runner)   Gid: (  127/  docker)
Access: 2023-12-05 10:31:10.708979337 +0000
Modify: 2023-12-05 10:31:06.344961327 +0000
Change: 2023-12-05 10:31:06.344961327 +0000
 Birth: 2023-12-05 10:31:06.340961311 +0000

Post build details
Checksums:

a550b63a7820f031dc93c27124801dbd  package.json
da23cc6f131fce28dfdb79955f59ed53  package-lock.json
File stats

  File: package.json
  Size: 10593     	Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d	Inode: 283258      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/  runner)   Gid: (  127/  docker)
Access: 2023-12-05 10:31:10.656979086 +0000
Modify: 2023-12-05 10:31:06.344961327 +0000
Change: 2023-12-05 10:31:06.344961327 +0000
 Birth: 2023-12-05 10:31:06.344961327 +0000


  File: package-lock.json
  Size: 576628    	Blocks: 1128       IO Block: 4096   regular file
Device: 801h/2049d	Inode: 283257      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/  runner)   Gid: (  127/  docker)
Access: 2023-12-05 10:32:20.589286067 +0000
Modify: 2023-12-05 10:32:10.629243784 +0000
Change: 2023-12-05 10:32:10.629243784 +0000
 Birth: 2023-12-05 10:31:06.340961311 +0000

Hey @tepi, @mshabarov ,

We’ve just upgraded to 24.2.5, but it looks like this does not fix the problem for us. We are still getting the same error in GH Actions CI (after complete cache nuking):

image

Locally it seems to be working as expected. Is there any project path that you might be caching? For example the node_modules/.vaadin/vaadin.json seems to contain an absolute projectFolder. If you are caching that on the CI, then that path might be changing depending on which agent the job lands on.

I have rebased the fork of the base-starter and you can see the build here: https://github.com/steve-todorov/base-starter-gradle/actions/runs/7037473808

Scan is here: https://scans.gradle.com/s/e6d5xqndde4qe/performance/execution

Should we re-open this ticket or open a new one?

With the provided example run locally, vaadinBuildFrontend updates package.json once (the outdated frontend dependency) and after that no more updates.

@steve-todorov thanks for the update, we will investigate further if the proposed workaround works well.

@steve-todorov Yup, that sounds like a good way forward. I’ve created a feature request at https://github.com/gradle/gradle/issues/27696 and I agree to continue exactly as you suggested. Let’s discuss the ticket with the Gradle guys a bit first; depending on the outcome we can decide appropriate course of action.

@mvysny thanks for your continuous efforts into debugging and solving this problem!

Although I see your point of view as valid, I believe waiting for this feature request to be resolved might take a considerable time. Let’s open a ticket and see how it goes and if the Gradle team would be open to looking into this.

In the mean time or if Gradle decide to decline fixing this as it is “working as intended” – it might be more useful to create the necessary additional tasks and properly document why they’re needed and how they work. That should make things easier to understand for other team members and potentially easier to maintain.

I think I finally understand what the problem is.

Vaadin plugin is modifying package.json in-place - it reads the json file, and if there is a need for modification, it modifies the package.json file. Therefore, the property should be annotated both with @InputFile and @OutputFile but that doesn’t work. I’ve asked a question on SO, hopefully we’ll learn more: https://stackoverflow.com/questions/77807234/gradle-cache-task-which-processes-a-file-in-place

Therefore, currently the package.json is annotated only with @OutputFile. However, that creates a huge build-cache-related problem which is not immediately visible. If the package.json file doesn’t exist then everything works: the outcome of vaadinPrepareFrontend is cached in the build cache and the build cache is reused during the next run of vaadinPrepareFrontend. However, if the package.json file exists before the build cache is populated, then the following happens:

  1. Since package.json exists and Gradle doesn’t have it in its build cache, it will print the “Gradle does not know how file” blah blah overlapping error message. It will run the vaadinPrepareFrontend task fully.
  2. However, note the following message which is present too: Caching disabled for task ':compileJava' because:. That means that even though vaadinPrepareFrontend ran fully, its outcome is not recorded in Gradle’s build cache!
  3. Next time the build starts, package.json exists and Gradle doesn’t have it in its build cache. Goto 1.

This behavior is reproducible also with the Java Compiler Gradle task as documented at https://mvysny.github.io/gradle-overlapping-outputs/ .

I’m not sure whether this is a bug in Gradle or whether this is desired behavior; it could be a Gradle protection from some kind of cache corruption as documented in the forums: https://discuss.gradle.org/t/gradle-7-0-seems-to-take-an-overzealous-approach-to-inter-task-dependencies/39656 . Therefore, instead of trying to find a workaround to cache output of a task which ran into “overlapping outputs”, I’d propose to mark the package.json property properly.

I’ve added frontend/generated into github’s action cache to see if that would solve it. Unfortunately it does not. This problem might be related to NPM doing something behind the scenes to the package.json or perhaps there is an environment difference that causes the problem to appear in GH Actions.

If anybody has time to dig into this you could use a custom self-hosted runner from https://github.com/myoung34/docker-github-actions-runner and try to debug it locally (i.e. log into the container and check what’s really going on)

Reopened for further investigation.

Fix has been applied to Vaadin Flow 24.3 (pre) and 24.2, will be released soon for both branches and available for testing.

I’m looking into the issue currently, no need to create a new issue, I will create one when needed. Right now I can’t reliably reproduce the timestamp update of package.json file. With identical setup it sometimes updates and sometimes does not update for my local build. Will keep you posted when I figure out what is going on.

Can confirm point 1 from @mvysny above; the timestamp does update even though the contents do not change. This might be the underlying reason for this behavior. Unfortunately could not find any way to disable this update - rather I consider it a bug that needs to be fixed, since it makes no sense to write the file if the contents are identical.

Thanks @mvysny @oliveryasuna @knoobie for you analysis! I like the idea to open a new one, but before I would like to understand what changes in package.json are made by vaadinBuildFrontend for @steve-todorov .

Thanks for the issue. We will try to reproduce with our Gradle-based starters and get back to you soon.