gatsby: Build is not incremental

Description

My Gatsby 3.0 site is being built on every push with a Github action. I’m seeing that many pages are being re-built even though I only made a change in a single page.

Steps to reproduce

This is a bit tricky for now because I can reproduce in a private repo. But the steps are simple, my site has a variety of markdown based pages as well as plain old JSX pages.

  1. I build.
  2. I make a change in a single JSX page, and I see the build is slow and all of the pages are updated.
  3. I build again without any changes and I see verbose Found 102 changed pages. Still slow.

Here are the logs to show it happening:

ran@mbp2:~/dev/staking/topstakingrewards $ gatsby build --verbose --log-pages
verbose set gatsby_log_level: "verbose"
verbose set gatsby_executing_command: "build"
verbose loading local command from: /Users/ran/dev/staking/topstakingrewards/node_modules/gatsby/dist/commands/build.js
verbose running command: build

success open and validate gatsby-configs - 0.097s
success load plugins - 1.553s
success onPreInit - 0.040s
success initialize cache - 0.005s
success copy gatsby files - 0.072s
success onPreBootstrap - 0.015s
success createSchemaCustomization - 0.021s
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 0 changed pages
success Checking for changed pages - 0.004s
success source and transform nodes - 0.188s
success building schema - 0.424s
info Total nodes: 319, SitePage nodes: 103 (use --verbose for breakdown)
verbose Number of node types: 10. Nodes per type: SitePage: 103, SitePlugin: 51, Site: 1, SiteBuildMetadata: 1, Directory: 10, File: 77, ImageSharp: 28, MarkdownRemark: 35, DatabaseXlsx__Prod: 12,
DatabaseXlsx: 1
success createPages - 0.108s
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 102 changed pages
success Checking for changed pages - 0.002s
success createPagesStatefully - 0.068s
success Cleaning up stale page-data - 0.006s
success update schema - 0.052s
success onPreExtractQueries - 0.002s
success extract queries from components - 3.998s
success write out redirect data - 0.001s
success Build manifest and related icons - 0.100s
success onPostBootstrap - 0.212s
info bootstrap finished - 11.961s
success write out requires - 0.005s
success Building production JavaScript and CSS bundles - 39.526s
success Building HTML renderer - 4.133s
info There are no new or changed html files to build.
info Generated public/sw.js, which will precache 9 files, totaling 584266 bytes.
The following pages will be precached:
/offline-plugin-app-shell-fallback/index.html
success onPostBuild - 0.120s
info Done building in 56.248277421 sec

ran@mbp2:~/dev/staking/topstakingrewards $ echo "// adding a comment" >> src/pages/faq.jsx
ran@mbp2:~/dev/staking/topstakingrewards $ st
On branch dev
Your branch is up to date with 'origin/dev'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   src/pages/faq.jsx

no changes added to commit (use "git add" and/or "git commit -a")
ran@mbp2:~/dev/staking/topstakingrewards $ gatsby build --log-pages --verbose
verbose set gatsby_log_level: "verbose"
verbose set gatsby_executing_command: "build"
verbose loading local command from: /Users/ran/dev/staking/topstakingrewards/node_modules/gatsby/dist/commands/build.js
verbose running command: build

success open and validate gatsby-configs - 0.110s
success load plugins - 1.576s
success onPreInit - 0.045s
success initialize cache - 0.006s
success copy gatsby files - 0.102s
success onPreBootstrap - 0.019s
success createSchemaCustomization - 0.023s
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 0 changed pages
success Checking for changed pages - 0.004s
success source and transform nodes - 0.197s
success building schema - 0.427s
info Total nodes: 319, SitePage nodes: 103 (use --verbose for breakdown)
verbose Number of node types: 10. Nodes per type: SitePage: 103, SitePlugin: 51, Site: 1, SiteBuildMetadata: 1, Directory: 10, File: 77, ImageSharp: 28, MarkdownRemark: 35, DatabaseXlsx__Prod: 12,
DatabaseXlsx: 1
success createPages - 0.104s
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 102 changed pages
success Checking for changed pages - 0.002s
success createPagesStatefully - 0.070s
success Cleaning up stale page-data - 0.005s
success update schema - 0.053s
success onPreExtractQueries - 0.002s
success extract queries from components - 3.983s
success write out redirect data - 0.002s
success Build manifest and related icons - 0.097s
success onPostBootstrap - 0.206s
info bootstrap finished - 11.850s
success write out requires - 0.004s
success Building production JavaScript and CSS bundles - 38.290s
success Rewriting compilation hashes - 0.003s
success Building HTML renderer - 4.297s
success Building static HTML for pages - 0.429s - 109/109 254.32/s
info Generated public/sw.js, which will precache 9 files, totaling 584266 bytes.
The following pages will be precached:
/offline-plugin-app-shell-fallback/index.html
success onPostBuild - 0.139s
info Done building in 56.103523451 sec
info Built pages:
Updated page: /articles/pos-vs-pow/
Updated page: /articles/staking-pools/
Updated page: /articles/what-is-staking/
Updated page: /articles/staking-risk/
Updated page: /providers/binanceus/
Updated page: /providers/kraken/
Updated page: /about-us/
Updated page: /cookie-policy/
Updated page: /disclaimer/
Updated page: /privacy-notice/
Updated page: /terms-of-use/
Updated page: /providers/binance-vs-rocketpool
Updated page: /providers/binance-vs-kraken
Updated page: /providers/binance-vs-staked
Updated page: /providers/binance-vs-guarda
Updated page: /providers/binance-vs-stakefish
Updated page: /providers/binance-vs-bloxstaking
Updated page: /providers/binance-vs-ankr
Updated page: /providers/binance-vs-allnodes
Updated page: /providers/binance-vs-binanceus
Updated page: /providers/rocketpool-vs-binance
Updated page: /providers/rocketpool-vs-kraken
Updated page: /providers/rocketpool-vs-staked
Updated page: /providers/rocketpool-vs-guarda
Updated page: /providers/rocketpool-vs-stakefish
Updated page: /providers/rocketpool-vs-bloxstaking
Updated page: /providers/rocketpool-vs-ankr
Updated page: /providers/rocketpool-vs-allnodes
Updated page: /providers/rocketpool-vs-binanceus
Updated page: /providers/kraken-vs-binance
Updated page: /providers/kraken-vs-rocketpool
Updated page: /providers/kraken-vs-staked
Updated page: /providers/kraken-vs-guarda
Updated page: /providers/kraken-vs-stakefish
Updated page: /providers/kraken-vs-bloxstaking
Updated page: /providers/kraken-vs-ankr
Updated page: /providers/kraken-vs-allnodes
Updated page: /providers/kraken-vs-binanceus
Updated page: /providers/staked-vs-binance
Updated page: /providers/staked-vs-rocketpool
Updated page: /providers/staked-vs-kraken
Updated page: /providers/staked-vs-guarda
Updated page: /providers/staked-vs-stakefish
Updated page: /providers/staked-vs-bloxstaking
Updated page: /providers/staked-vs-ankr
Updated page: /providers/staked-vs-allnodes
Updated page: /providers/staked-vs-binanceus
Updated page: /providers/guarda-vs-binance
Updated page: /providers/guarda-vs-rocketpool
Updated page: /providers/guarda-vs-kraken
Updated page: /providers/guarda-vs-staked
Updated page: /providers/guarda-vs-stakefish
Updated page: /providers/guarda-vs-bloxstaking
Updated page: /providers/guarda-vs-ankr
Updated page: /providers/guarda-vs-allnodes
Updated page: /providers/guarda-vs-binanceus
Updated page: /providers/stakefish-vs-binance
Updated page: /providers/stakefish-vs-rocketpool
Updated page: /providers/stakefish-vs-kraken
Updated page: /providers/stakefish-vs-staked
Updated page: /providers/stakefish-vs-guarda
Updated page: /providers/stakefish-vs-bloxstaking
Updated page: /providers/stakefish-vs-ankr
Updated page: /providers/stakefish-vs-allnodes
Updated page: /providers/stakefish-vs-binanceus
Updated page: /providers/bloxstaking-vs-binance
Updated page: /providers/bloxstaking-vs-rocketpool
Updated page: /providers/bloxstaking-vs-kraken
Updated page: /providers/bloxstaking-vs-staked
Updated page: /providers/bloxstaking-vs-guarda
Updated page: /providers/bloxstaking-vs-stakefish
Updated page: /providers/bloxstaking-vs-ankr
Updated page: /providers/bloxstaking-vs-allnodes
Updated page: /providers/bloxstaking-vs-binanceus
Updated page: /providers/ankr-vs-binance
Updated page: /providers/ankr-vs-rocketpool
Updated page: /providers/ankr-vs-kraken
Updated page: /providers/ankr-vs-staked
Updated page: /providers/ankr-vs-guarda
Updated page: /providers/ankr-vs-stakefish
Updated page: /providers/ankr-vs-bloxstaking
Updated page: /providers/ankr-vs-allnodes
Updated page: /providers/ankr-vs-binanceus
Updated page: /providers/allnodes-vs-binance
Updated page: /providers/allnodes-vs-rocketpool
Updated page: /providers/allnodes-vs-kraken
Updated page: /providers/allnodes-vs-staked
Updated page: /providers/allnodes-vs-guarda
Updated page: /providers/allnodes-vs-stakefish
Updated page: /providers/allnodes-vs-bloxstaking
Updated page: /providers/allnodes-vs-ankr
Updated page: /providers/allnodes-vs-binanceus
Updated page: /providers/binanceus-vs-binance
Updated page: /providers/binanceus-vs-rocketpool
Updated page: /providers/binanceus-vs-kraken
Updated page: /providers/binanceus-vs-staked
Updated page: /providers/binanceus-vs-guarda
Updated page: /providers/binanceus-vs-stakefish
Updated page: /providers/binanceus-vs-bloxstaking
Updated page: /providers/binanceus-vs-ankr
Updated page: /providers/binanceus-vs-allnodes
Updated page: /404/
Updated page: /articles/
Updated page: /calculator/
Updated page: /faq/
Updated page: /
Updated page: /providers/
Updated page: /404.html
Updated page: /offline-plugin-app-shell-fallback/

ran@mbp2:~/dev/staking/topstakingrewards $ gatsby build --log-pages --verbose
verbose set gatsby_log_level: "verbose"
verbose set gatsby_executing_command: "build"
verbose loading local command from: /Users/ran/dev/staking/topstakingrewards/node_modules/gatsby/dist/commands/build.js
verbose running command: build

success open and validate gatsby-configs - 0.108s
success load plugins - 1.483s
success onPreInit - 0.041s
success initialize cache - 0.005s
success copy gatsby files - 0.099s
success onPreBootstrap - 0.017s
success createSchemaCustomization - 0.025s
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 0 changed pages
success Checking for changed pages - 0.003s
success source and transform nodes - 0.184s
success building schema - 0.432s
info Total nodes: 319, SitePage nodes: 103 (use --verbose for breakdown)
verbose Number of node types: 10. Nodes per type: SitePage: 103, SitePlugin: 51, Site: 1, SiteBuildMetadata: 1, Directory: 10, File: 77, ImageSharp: 28, MarkdownRemark: 35, DatabaseXlsx__Prod: 12,
DatabaseXlsx: 1
success createPages - 0.112s
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 102 changed pages
success Checking for changed pages - 0.002s
success createPagesStatefully - 0.069s
success Cleaning up stale page-data - 0.007s
success update schema - 0.052s
success onPreExtractQueries - 0.002s
success extract queries from components - 4.138s
success write out redirect data - 0.001s
success Build manifest and related icons - 0.107s
success onPostBootstrap - 0.219s
info bootstrap finished - 11.878s
success write out requires - 0.005s
success Building production JavaScript and CSS bundles - 38.493s
success Building HTML renderer - 4.039s
info There are no new or changed html files to build.
info Generated public/sw.js, which will precache 9 files, totaling 584266 bytes.
The following pages will be precached:
/offline-plugin-app-shell-fallback/index.html
success onPostBuild - 0.119s
info Done building in 55.015417453 sec

Expected result

Fast build to update only /faq

Actual result

Slow build and everything is updated.

Environment

  System:
    OS: macOS 10.15.7
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
    Shell: 5.1.0 - /usr/local/bin/bash
  Binaries:
    Node: 15.4.0 - /usr/local/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 7.5.3 - /usr/local/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 89.0.4389.82
    Safari: 14.0.1
  npmPackages:
    gatsby: ^3.0.4 => 3.0.4 
    gatsby-cli: ^3.0.0 => 3.0.0 
    gatsby-codemods: ^2.0.0 => 2.0.0 
    gatsby-plugin-google-gtag: ^3.0.0 => 3.0.0 
    gatsby-plugin-image: ^1.0.1 => 1.0.1 
    gatsby-plugin-manifest: ^3.0.0 => 3.0.0 
    gatsby-plugin-netlify-cms: ^5.0.0 => 5.0.0 
    gatsby-plugin-offline: ^4.0.0 => 4.0.0 
    gatsby-plugin-postcss: ^4.0.0 => 4.0.0 
    gatsby-plugin-react-helmet: ^4.0.0 => 4.0.0 
    gatsby-plugin-remark-collection: ^0.1.0 => 0.1.0 
    gatsby-plugin-sharp: ^3.0.1 => 3.0.1 
    gatsby-plugin-sitemap: ^3.0.0 => 3.0.0 
    gatsby-remark-autolink-headers: ^3.0.0 => 3.0.0 
    gatsby-remark-images: ^4.0.0 => 4.0.0 
    gatsby-remark-relative-images: ^2.0.2 => 2.0.2 
    gatsby-source-filesystem: ^3.0.0 => 3.0.0 
    gatsby-transformer-excel: ^3.0.0 => 3.0.0 
    gatsby-transformer-remark: ^3.0.0 => 3.0.0 
    gatsby-transformer-sharp: ^3.0.0 => 3.0.0 
  npmGlobalPackages:
    gatsby-cli: 3.0.0

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 20 (14 by maintainers)

Most upvoted comments

I did a bunch of incremental builds with gatsby@alpha-inc-build-diag and the results were always the same:

  • I didn’t do any data change, which was reported correctly by Gatsby:
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 0 changed pages
  • Every very page gets rebuild for the same reason:
"{path-to-page}": [
  "FLAG_DIRTY_STATIC_QUERY_RESULT_CHANGED",
  "FLAG_DIRTY_BROWSER_COMPILATION_HASH",
  "FLAG_DIRTY_SSR_COMPILATION_HASH"
],

Edit 1: I could resolve FLAG_DIRTY_STATIC_QUERY_RESULT_CHANGED. Via gatsby-plugin-build-date we are using a static query to get the current build time. This query is used in our shared Layout component that is used in every template. As the build time changes on an incremental build, the static query result was different and all pages need to be rebuild. We removed the static query from our Layout component and just put it into very unique page that is only build once.

Edit 2: I finally fixed it! The fix is quite special for our project, but maybe it helps others to spot it in their project. It was a “trial and error” approach: deleting all code in the gatsby-ssr, changing some config in the gatsby-config and removing some createPages variables until the incremental build was working. In the end, we had 2 problems:

tl;dr: Check for all usages of Date in your create-pages files and gatsby-config. It changes between incremental builds and could therefore be the reason why all pages need to rebuild.

Like in the static query, we also used the date in a shared Gatsby context object we put into every createPage. It looks kinda like this:

actions.createPage({
  path: urlPath,
  context: {
    meta: {
      buildTime: new Date().toUTCString(),
    }
    channelId: context.channelId,
  },
});

I just removed it, as it was not necessary anymore in this project. The information is also redundant as we already have it with the build date plugin.

The second problem was even more unique: Due to some infrastructure constrains we need to copy all assets into a separate subfolder. We are using gatsby-plugin-asset-path for this. The gatsby-config looks like that:

module.exports = {
  assetPrefix: process.env.GATSBY_ASSET_PREFIX
    ? '/' + process.env.GATSBY_ASSET_PREFIX + `/${Date.now().toString()}`
    : undefined,
  plugins: [
    ...(process.env.GATSBY_ASSET_PREFIX
      ? [
          {
            resolve: 'gatsby-plugin-asset-path',
            options: {
              fileTypes: ['js', 'map', 'css', 'json', 'png', 'webmanifest'],
            },
          },
        ]
      : [])
  ]

The problem there was the Date.now().toString() method. With an incremental build, the date changed which resulted in the flags FLAG_DIRTY_BROWSER_COMPILATION_HASH and FLAG_DIRTY_SSR_COMPILATION_HASH. The fix was to just remove the date in the path. We added it right at the begging of the project to make sure caching was always working between deploys. This is not necessary as a proper Nginx config already handles that.

Conclusion Incremental build are now working for us. Only the nodes we change get rebuild, all other pages stay the same 🎉

Gatsby was working correctly all the time. The problems we had were valid reasons to rebuild all pages. The issue is that it’s very complicated to track down the problem. The log messages was super helpful to spot the issues, I would propose that the flag-log-message makes it into the normal Gatsby package! It would also be helpful to make the error messages more detailed - not sure if this is technically working (like “there was a change in your gatsby-config”) - and to have a documentation for every flag that helps to track down the error.

+4
feedm3 on Mar 17, 2021

so i have tried your steps , and now i have the build-diff.txt file for my 10500 page 😃 and i understand nothing from it , its mixed up and super hard to know anything inside , any hint what should i look for or how can i read it ?

So generating diff is first step - just to talk concretely - here’s example diff I created replicating some of what feedm3 described in their comments - https://gist.github.com/pieh/f0e10a2b8a4eb33177c85cf0a80fb0e3 - I used diff -u -r --exclude="*.html" public-first-build public > inc.diff to remove some of the noise that .html changing would add to the diff.

DISCLAIMER: this is just example - it will be unlikely that it can apply to every situation - this is more showing general steps of tracking down root cause

The few things you can get from it are:

Genareted JS bundle did change

You can tell by seeing that both chunk-map.json and webpack.stats.json are different and also content of app-data.json - on top of that you get lines like Only in public: component---src-pages-index-js-2b24b822e1fe82516116.js - this tell you that something in that component changed (there is no diff because hash in file changed so filenames for them changed) - from here you can compare in next step is exact diff between assets generated for that page template - it would be something like https://www.diffchecker.com/HbqV7mlf - ok now we see that Date is inlined in js files and that date will change between builds - then it’s matter of finding this piece in source code and going from there to understand what can be done about it. In my case it was quite straight forward because I did that on purpose, but source of it was <div>Define plugin: {process.env.DEFINED_DATE}</div> and I defined process.env.DEFINED_DATE in gatsby-node:

exports.onCreateWebpackConfig = ({ actions, plugins }) => {
  const dateString = new Date().toString()
  actions.setWebpackConfig({
    plugins: [
      plugins.define({
        "process.env.DEFINED_DATE": JSON.stringify(dateString),  
      }),
    ],
  })
}

Used data has changed

Other things in the diff is we see changes in: a) page-data/sq/d/2907560070.json b) page-data/using-typescript/page-data.json files. Those are results of a) static query (sq/d/<number>.json gives it away ) and b) page query

Page query is quite straight forward to figure out because it points to actual page and you can see what data changed.

Static query is a bit more messy - you can see what content changed but not quite easily to tell which static query it is - we don’t expose much information about it, so probably fastest way is just to check static queries you have in your site and just match the result from .json file to static query in your code by comparing what data is there and what you query in those. So result in public is something like:

{"data":{"site":{"buildTime":"2021-03-18 10:47 am UTC"}}}

And I track it down to static query in one of my components that is used in all pages:

  const data = useStaticQuery(graphql`
    {
      site {
        buildTime(formatString: "YYYY-MM-DD hh:mm a z")
      }
    }
  `)

Because it’s used on all pages, when result of it changes, it will “invalidate” all html files (of course there can be situation when static query is used on subset of all pages too, but point is that changing it can invalidate 1 or more pages)

Note - you can see additional changes in the diff like _headers (that’s from gatsby-plugin-gatsby-cloud / gatsby-plugin-netlify type plugins and rss.xmlthose changes are result of changes listed above and would suggest to not put attention (those should not be source of problems - they just manifest them for similar reason inc builds doesn’t work). Focus on changes in .js files and in page-data directory

+1
pieh on Mar 18, 2021

@feedm3 thanks for super detailed description of what was causing to not hit “happy path” for inc builds. I do wonder now what kind of diagnostic information we could actually display to users to help with tracking root causes down because gatsby@alpha-inc-build-diag doesn’t really have a lot of details and is just starting point to know what to look for

It rebuilds only 1 changed page (the one that was changed, so this is correct) but it says that 1823 pages were changed. So the log message is not correct here.

Agree, it’s confusing (especially because that part of logs actually doesn’t relate at all which html files get rebuild 😦 )

+1
pieh on Mar 17, 2021
Read more comments on GitHub
← gatsby: Build HTML error: Cannot resolve 'file' or 'directory' ../public/chunk-manifest.json
gatsby: Build script returned non-zero exit code: 1 netlify →