gatsby: Build stuck at running jobs (image transformation)

@LekoArts I have come to the conclusion together with other teammates working on the same project that we cannot even make a proper assessment why this problem occurs and we are not able to detect if it’s a resource problem, a graphQL problem or if it has something to do with gatsby internals.

I see that many people are struggling with this error. Hence, there may be a common drawback of the new version. Therefore, it should be handled with high priority.

For me, this problem happened for a long time, tried directly with Gatsby Cloud to work on a solution, but they basically shrugged to the issue. I could get to the bottom of the problem: it happens on gatsby-plugin-sharp, when there are too many images to process. I tried all these variables, I tried to change the underlying code to throttle the image processing etc, and couldn’t get to any point I was happy with.

Then I decided to just stick with using Shopify’s CDN instead of processing the images and never looked back. But I know that this is a bummer and not everyone can “disable” image processing.

We’re trying to upgrade to 4.4 from 3 as well and are running into this exact issue - both in Gatsby Cloud https://www.gatsbyjs.com/dashboard/e156da66-cda0-4df5-b3c0-a7fdca6bf65e/sites/43774e74-f15a-4923-b6f7-d215d0ba104b/builds/e82328f4-29ff-44bc-945e-81b886afd8f8/details#rawLogs and locally:

success Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 211.318s - 2175/2175 10.29/s ⠋ Merge worker state

ERROR

Assertion failed: all worker queries are not dirty (worker #3)`

Following because I get this problem a lot.

Thanks for providing the URLs. We’ve looked at the builds from @NickBarreto @DennisKraaijeveld and in summary these are the findings:

• It’s not stuck at “run queries in workers” as that step finishes
• Sometimes (not always) it hangs on running image transformation (jobs). If you add up the times of the activities it doesn’t go up to e.g. 20min, it only takes that long because in between it waits for communication from the outside which it doesn’t get. It retries that a bunch of times and in some cases it then fails the build, in some cases it continues
• The PR https://github.com/gatsbyjs/gatsby/pull/34061 won’t fix it but it’ll give us and the users a way of seeing this being stuck and make debugging things easier

Adding a comment here that I’m just running a query in GraphiQL to test a few things and when I execute it with

  allFile {
    edges {
      node {
        childImageSharp {
          gatsbyImageData(formats: AUTO)
        }
      }
    }

it hangs on

success Writing page-data.json files to public directory - 0.302s - 3/5 16.54/s
[                            ]   0.024 s 0/2 0% Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs

@wardpeet Hello, but what if the site is not even deployed on gatsby cloud? For production the site is deployed on AWS but the builds also fail locally. :<

Any people on Gatsby Cloud who are having this issue that can send me an email at ward@gatsbyjs.com with your user email and site name so we can investigate 🙏

Keeping this issue open for others and I would also like to follow the development of it.

@bkaldor and others finding this issue: I guess there might be different reasons the build stops/stalls and this issue can get messy. I summarized below:

• You might be upgrading to v4 and have some old component with a dirty query. Then adding the debug info using the the comment above by @pieh should give you enough info to trace and fix it.
• You might be using Gatsby cloud and its an image processing thing, which appears (not confirmed, see comment below) to be fixed in comment above.
• If it’s an image processing issue on local gatsby build, try playing with the NODE_OPTIONS=--max_old_space_size and GATSBY_CPU_COUNT environment variables.
• Also make sure you upgraded gatsby-cli to v4.

Also seeing this 👀

Probably workaround is to add env var: GATSBY_DISABLE_CACHE_PERSISTENCE=true

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

@pieh Thanks! Yeah I was going that route but your snippet really helped:

So apparantly I still had an old Image (image.js) component laying around from an earlier version/iteration which was used in one place and that debug info showed me:

    trackedQueries: Map(6) {
      'sq--src-components-header-header-js' => { dirty: 0, running: 0 },
      'sq--src-components-meta-js' => { dirty: 0, running: 0 },
      'sq--src-components-blocks-latest-posts-js' => { dirty: 0, running: 0 },
      'sq--src-components-media-image-js' => { dirty: 4, running: 0 },
      'sq--src-components-forms-ebook-form-js' => { dirty: 0, running: 0 },
      'sq--src-components-node-body-js' => { dirty: 0, running: 0 }
    },

Adding that debug info by default might help a lot of people migrating and running into an issue like this.

I have read the thread but I’m not using Gatsby Cloud.

Currently migrating from v3 to v4 and testing everything on local and now this quite often happens on gatsby build.

success run queries in workers - 14.927s - 96/96 6.43/s
success Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 50.515s - 206/206 4.08/s
not finished Merge worker state - 0.276s

Never had issues before, it’s a moderate site, definitely nothing large. Is there a way to get more debug info here what’s going on?

Edit: played around with NODE_OPTIONS=--max_old_space_size and GATSBY_CPU_COUNT on my local 32Gb laptop, but without success.

@engineergit I ran into the same issue as @stephzero1 did 👆🏻 up there using Gatsby: 4.19.2, Node: 16.14.2, npm: 8.5.0, macOS: 12.4. Turns out it’s related to sharp (this one and this one).

What resolved this for me was to simply run $ brew install vips (thanks to @lovell 's suggestion).

I just ran into this issue with my blog, hosted on Vercel. I found a workaround for Vercel and spotted a couple of things that might be of interest to anyone working on this issue.

Background: I’m writing a plugin that sources my Instagram posts so they can be included in my blog alongside regular markdown posts. A markdown node is created for each Instagram post; the images are downloaded by gatsby-remark-images-remote for processing with gatsby-plugin-sharp. Basically: my site has a ton of images.

I develop on macOS (M1) and didn’t encounter this issue until I tried to deploy on Vercel. The error message is similar to those reported above:

success Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 34.278s - 256/256 7.47/s

ERROR

Assertion failed: all worker queries are not dirty (worker #3)

Error: Assertion failed: all worker queries are not dirty (worker #3)

...
...
...

Scouring the comments here, I was able to reproduce locally by setting the environment variable GATSBY_CPU_COUNT=2. I soon realised that by increasing the value of this environment variable, my site would build just fine again.

Surprisingly, the same trick works on Vercel: simply override the default build command (find the Project Settings page then jump down to Build & Development Settings) then my site deploys just fine:

GATSBY_CPU_COUNT=8 gatsby build

I can’t find much documentation on Vercel’s build environment but I presume it’s a tiny VM running somewhere where GATSBY_CPU_COUNT gets set by default to 1 and this is why I don’t typically see this issue on my M1 (because my site actually deploys pretty quickly on Vercel, I suspect it’s not in fact such a tiny VM at all).

A couple of observations that may help debug this issue:

• The value of GATSBY_CPU_COUNT needed to get my site to build is proportional to the number of images. Even without overriding the default I was able to deploy my site on Vercel if I simply limited the number of Instagram posts to ~10. I currently set it to 8.
• I configure gatsby-transformer-remark to use both gatsby-remark-images and gatsby-remark-images-remote. If I disable one or the other, my site deploys just fine on Vercel regardless of how many images I include.

@NickBarreto we fixed the issue, you were seeing on the cloud side.

@LekoArts I run multiple builds through and through for the entire week, only having the same results as @tyhopp. I will add the logging output below. I could not find absolutely no pattern to understand which image or why some images fail the rendering engines validation, as every build I ran, the images were random.

gatsby:gatsby-plugin-sharp Start processing /Users/my-user/Desktop/projects/my-project/public/static/186be79251497704906aefc1d719d805/0ceda/GettyImages-1225946537.jpg +0ms
  gatsby:gatsby-plugin-sharp Start processing /Users/my-user/Desktop/projects/my-project/public/static/186be79251497704906aefc1d719d805/92368/GettyImages-1225946537.jpg +0ms
  gatsby:gatsby-plugin-sharp Start processing /Users/my-user/Desktop/projects/my-project/public/static/186be79251497704906aefc1d719d805/fbd6b/GettyImages-1225946537.webp +65ms
  gatsby:gatsby-plugin-sharp Start processing /Users/my-user/Desktop/projects/my-project/public/static/186be79251497704906aefc1d719d805/dd2c8/failed Validating Rendering Engines - 2.853s
 ERROR #98001  WEBPACK

Built Rendering Engines failed validation failed validation.

Please open an issue with a reproduction at https://github.com/gatsbyjs/gatsby/issues/new for more help

I looked into ways to expose more verbose logging for image processing, and one way that’s available now (since gatsby-plugin-sharp@2.0.25) is to run a build with this debug env var:

DEBUG=gatsby:gatsby-plugin-sharp npm run build

This should allow you to see logs for which image is being processed for sharp and how long each takes, which should reveal if the issue is related to stalling on a particular image. Example log output:

gatsby:gatsby-plugin-sharp Start processing my-gatsby-site/public/static/2ce198ad053443a8411db71e1288ac2c/fc149/2909411689_fa73cef69f_o.avif +211ms
gatsby:gatsby-plugin-sharp Start processing my-gatsby-site/public/static/2ce198ad053443a8411db71e1288ac2c/0e4d3/2909411689_fa73cef69f_o.avif +1ms
gatsby:gatsby-plugin-sharp Start processing my-gatsby-site/public/static/2ce198ad053443a8411db71e1288ac2c/447c6/2909411689_fa73cef69f_o.webp +0ms

Still investigating how to expose the extra log output from stalled jobs locally, will share in this thread if I can get that working since it should also be helpful.

@buzinas Until this problem is solved I will need to research how this problem occurs and try to either bypass it internally or make a workaround for the configuration that is used.

I am running into the same issue. I also played with the .env variables NODE_OPTIONS / GATSBY_CPU_COUNT but had no luck to get my build running.

I am hosting my application on a ubuntu 20.04 server with dokku/docker, 8G of memory and 4 cores. Also Gatsby Cloud is giving my the same output. Just my M1 Mac builds the page locally without any problems.

Since downgrading to GATSBY V3 it works again.

Getting the same issue as well, sometimes I have to re-run the build 4-5 times until one succeeds (running with GATSBY_CPU_COUNT=1 on a single-CPU instance).

This is still a massive blocker for my team in order for us to move from Gatsby 3 to Gatsby 4. How does one go about getting traction towards this issue, which appears to be a larger impacting issue than just our site?

I am having This same issue. Here is a link to the build failurebuild fail

Thank you very much with the assistance on my query @pieh and great spot on the relative paths 😅 You are probably right on that, though I am curious how the change from V3 to V4 had cause that to become an issue. Let me try the workaround and revert back here.

In the mean time, would the gatsby-source-directus imageFile field as pointed out here potentially cause issues on this as well? I noticed in the migration guide, specifically here that there’s a new way to call createRemoteFileNode and the current source plugin seems to be deprecated.

@askibinski if you are hitting this issue locally - could you manually edit a file in your node_modules - node_modules/gatsby/dist/redux/reducers/queries.js - if you are on 4.4 line should be line 439

And instead of just

throw new Error(`Assertion failed: all worker queries are not dirty (worker #${workerId})`);

Let’s add information on our state that assertion fails on:

throw new Error(`Assertion failed: all worker queries are not dirty (worker #${workerId})\n${require(`util`).inspect(queryStateChunk, { depth: Infinity })}`);

This should additionally print information like one below alongside assertion error:

{
    byNode: Map(2) {
      'Site' => Set(1) { '/using-typescript/' },
      'SiteBuildMetadata' => Set(1) { '/using-typescript/' }
    },
    byConnection: Map(0) {},
    queryNodes: Map(0) {},
    trackedQueries: Map(1) { '/using-typescript/' => { dirty: 0, running: 0 } },
    trackedComponents: Map(0) {},
    deletedQueries: Set(0) {},
    dirtyQueriesListToEmitViaWebsocket: []
  }

I currently have no idea how we end up in situation like that. Possibly something fails earlier and we swallow/ignore error? Or maybe we have some stale state?

@NickBarreto (and other folks following this issue)

What would you advise as a next step? Watch this PR until it is merged into a release, then upgrade to that release and do a few further builds to gather more diagnostic details?

I did publish “canary” (gatsby@alpha-job-progress) from the PR branch and we are running that (+ some additionaly debugging code on top) internally with test site that we are able to reproduce the issue (eventually, as it does need multiple runs to eventually reproduce the problem). You can use that canary release yourself, but it really won’t help much for unstucking builds (it will just show additional progress bar in logs tab)

So in short, we don’t need more information from you folks (at least about being stuck on image generation in Gatsby Cloud), we already can reproduce and are in process of tracking down the problem and we will post update here once we find the problem, implement a fix and have reasonably high level of confidence that the fix is correct ( we can never be 100% sure due to intermittent nature of the problem )