flow: The `vaadinPrepareFrontendTask` gradle task is not cached

Description of the bug

We have upgrade to Vaadin 24.2.0 with the hopes to benefit from the newly introduced incremental Gradle task caching for vaadinPrepareFrontendTask (related PR is https://github.com/vaadin/flow/pull/17150).

Unfortunately our local and CI builds seem to be unable to benefit from this feature. Upon closer investigation we noticed Gradle scans show the following notice:

Overlapping outputs: Gradle does not know how file 'package.json' was created (output property 'taskOutputProperties.packageJson'). 
Task output caching requires exclusive access to output paths to guarantee correctness

Expected behavior

The vaadinPrepareFrontendTask should show UP-TO-DATE or FROM CACHE in the gradle build scans and the task should not be computing again.

Minimal reproducible example

Not sure how to reproduce yet, but we don’t have any special setup. We are using the gradle vaadin plugin and run gradle build.

Versions

Vaadin / Flow version: 24.2.0
Java version: 17
OS version: ubuntu-22.04 (GH CI)
Browser version (if applicable):
Application Server (if applicable):
IDE (if applicable):

About this issue

Original URL
State: open
Created 8 months ago
Comments: 41 (26 by maintainers)

Commits related to this issue

fix: Do not touch package.json files unless content has changed (#18045) In some cases, at least the package.json file is overwritten with the same content it already had. This causes the timestamp t... — committed to vaadin/flow by tepi 7 months ago
fix: Do not touch package.json files unless content has changed (#18045) In some cases, at least the package.json file is overwritten with the same content it already had. This causes the timestamp t... — committed to vaadin/flow by tepi 7 months ago
fix: Do not touch package.json files unless content has changed (#18045) (#18055) In some cases, at least the package.json file is overwritten with the same content it already had. This causes the ti... — committed to vaadin/flow by vaadin-bot 7 months ago

Most upvoted comments

Hey guys,

Apologies for the late reply.

@mvysny If vaadinBuildFrontend constantly keeps overwriting/modifying the package.json, then that’s definitely a bug which needs to be reported as a separate issue and resolved by Vaadin. You can try to reproduce such a bug by running full build vaadinBuildFrontend in production mode once on your dev machine, then commit package.json to git, then run the same build on other machines - the package.json should not be modified by consequent builds, and ultimately the vaadinPrepareFrontend task should be cached properly.

What you’ve described is exactly what’s happening. The original problem is that the package.json is overwritten during the CI build and then for some reason it’s never able to be cached. Even if the package.json and package-lock.json are synced and committed to VCS the vaadinPrepareFrontend task is still not cached. We also tried clearing the GH Actions cache and running a few runs on the main but the result remains the same.

I am not entirely sure where this is coming from exactly, but the closest I could get to reproducing the problem was by introducing out-of-sync in the package.json as shown in https://github.com/steve-todorov/base-starter-gradle/tree/issue-17941. This leads me to believe something additional is happening in the vaadinBuildFrontend task that might be changing the package.json or package-lock.json causing the problem.

I’ll record the package.json and package-lock.json during the CI and diff them in hopes to get more clarity and report back.

steve-todorov on Nov 8, 2023

I did some tests both locally and on our gradle starter repository.

Locally, the cache seems to work fine. If I build the project and then do a clean (removing also frontend/generated stuff) and rebuild, the log says :vaadinPrepareFrontend FROM-CACHE.

If I do not remove the frontend/generated folder, I get a non-cachable warning, but not for package.json

Non-cacheable because Gradle does not know how file 'frontend/generated/vaadin.ts' was created (output property 'outputProperties$vaadin_gradle_plugin.generatedTsFolder'). Task output caching requires exclusive access to output paths to guarantee correctness (i.e. multiple tasks are not allowed to produce output in the same location). [OVERLAPPING_OUTPUTS]

On GitHub actions I always get the non-cachable warning for package.json even if the file is not changed in any way; in the job summary the checksum, creation and modification times are the same before and after the gradle build.

Excerpt of GH build</summary

Pre build details
Checksums:

a550b63a7820f031dc93c27124801dbd  package.json
da23cc6f131fce28dfdb79955f59ed53  package-lock.json
File stats

  File: package.json
  Size: 10593     	Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d	Inode: 283258      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/  runner)   Gid: (  127/  docker)
Access: 2023-12-05 10:31:10.656979086 +0000
Modify: 2023-12-05 10:31:06.344961327 +0000
Change: 2023-12-05 10:31:06.344961327 +0000
 Birth: 2023-12-05 10:31:06.344961327 +0000


  File: package-lock.json
  Size: 576628    	Blocks: 1128       IO Block: 4096   regular file
Device: 801h/2049d	Inode: 283257      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/  runner)   Gid: (  127/  docker)
Access: 2023-12-05 10:31:10.708979337 +0000
Modify: 2023-12-05 10:31:06.344961327 +0000
Change: 2023-12-05 10:31:06.344961327 +0000
 Birth: 2023-12-05 10:31:06.340961311 +0000

Post build details
Checksums:

a550b63a7820f031dc93c27124801dbd  package.json
da23cc6f131fce28dfdb79955f59ed53  package-lock.json
File stats

  File: package.json
  Size: 10593     	Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d	Inode: 283258      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/  runner)   Gid: (  127/  docker)
Access: 2023-12-05 10:31:10.656979086 +0000
Modify: 2023-12-05 10:31:06.344961327 +0000
Change: 2023-12-05 10:31:06.344961327 +0000
 Birth: 2023-12-05 10:31:06.344961327 +0000


  File: package-lock.json
  Size: 576628    	Blocks: 1128       IO Block: 4096   regular file
Device: 801h/2049d	Inode: 283257      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/  runner)   Gid: (  127/  docker)
Access: 2023-12-05 10:32:20.589286067 +0000
Modify: 2023-12-05 10:32:10.629243784 +0000
Change: 2023-12-05 10:32:10.629243784 +0000
 Birth: 2023-12-05 10:31:06.340961311 +0000

mcollovati on Dec 5, 2023

With the provided example run locally, vaadinBuildFrontend updates package.json once (the outdated frontend dependency) and after that no more updates.

tepi on Nov 6, 2023

It would be super amazing to get this fixed. Shaving 3+ mins off each Github Actions build would be a game changer for my project.

benstpierre on Dec 21, 2023

I’ve added frontend/generated into github’s action cache to see if that would solve it. Unfortunately it does not. This problem might be related to NPM doing something behind the scenes to the package.json or perhaps there is an environment difference that causes the problem to appear in GH Actions.

If anybody has time to dig into this you could use a custom self-hosted runner from https://github.com/myoung34/docker-github-actions-runner and try to debug it locally (i.e. log into the container and check what’s really going on)

steve-todorov on Dec 12, 2023

Reopened for further investigation.

mshabarov on Nov 30, 2023

Hey @tepi, @mshabarov ,

We’ve just upgraded to 24.2.5, but it looks like this does not fix the problem for us. We are still getting the same error in GH Actions CI (after complete cache nuking):

Locally it seems to be working as expected. Is there any project path that you might be caching? For example the node_modules/.vaadin/vaadin.json seems to contain an absolute projectFolder. If you are caching that on the CI, then that path might be changing depending on which agent the job lands on.

I have rebased the fork of the base-starter and you can see the build here: https://github.com/steve-todorov/base-starter-gradle/actions/runs/7037473808

Scan is here: https://scans.gradle.com/s/e6d5xqndde4qe/performance/execution

Should we re-open this ticket or open a new one?

steve-todorov on Nov 30, 2023

Fix has been applied to Vaadin Flow 24.3 (pre) and 24.2, will be released soon for both branches and available for testing.

mshabarov on Nov 17, 2023

I’m looking into the issue currently, no need to create a new issue, I will create one when needed. Right now I can’t reliably reproduce the timestamp update of package.json file. With identical setup it sometimes updates and sometimes does not update for my local build. Will keep you posted when I figure out what is going on.

tepi on Nov 15, 2023

Can confirm point 1 from @mvysny above; the timestamp does update even though the contents do not change. This might be the underlying reason for this behavior. Unfortunately could not find any way to disable this update - rather I consider it a bug that needs to be fixed, since it makes no sense to write the file if the contents are identical.

tepi on Nov 13, 2023

Thanks @mvysny @oliveryasuna @knoobie for you analysis! I like the idea to open a new one, but before I would like to understand what changes in package.json are made by vaadinBuildFrontend for @steve-todorov .

mshabarov on Nov 8, 2023

Thanks for the issue. We will try to reproduce with our Gradle-based starters and get back to you soon.

mshabarov on Oct 31, 2023