gatsby: Gatsby build on Netlify fails(Image generation): Command did not finish within the time limit
Description
The build on Netlify fails with error: Command did not finish within the time limit.
When I run a local build the build is much faster.
Steps to reproduce
Running gatsby build on Netlify
Expected result
The build should succeed in a reasonable amount of time, just like when running the build locally:
Build log:
success open and validate gatsby-config — 0.010 s
success load plugins — 0.172 s
success onPreInit — 0.469 s
success delete html and css files from previous builds — 0.005 s
success initialize cache — 0.007 s
success copy gatsby files — 0.039 s
success onPreBootstrap — 0.004 s
⠁ Starting to fetch data from Prismic
⠈ source and transform nodesFetch Prismic data: 933.771ms
success source and transform nodes — 12.536 s
success building schema — 0.997 s
success createPages — 0.170 s
success createPagesStatefully — 0.149 s
success onPreExtractQueries — 0.004 s
success update schema — 0.830 s
success extract queries from components — 0.294 s
success run graphql queries — 19.262 s — 199/199 10.33 queries/second
success write out page data — 0.007 s
success write out redirect data — 0.001 s
Generating image thumbnails [==============================] 1690/1690 367.4 secs 100%
info bootstrap finished - 389.838 s
success onPostBootstrap — 0.011 s
success Building production JavaScript and CSS bundles — 7.609 s
success Building static HTML for pages — 1.887 s — 198/198 265.21 pages/second
info Done building in 399.357 sec
Actual result
The build fails: Command did not finish within the time limit.
Build log:
4:15:29 PM: Build ready to start
4:15:31 PM: build-image version: 42bca793ccd33055023c56c4ca8510463a56d317
4:15:31 PM: buildbot version: 6bab8b64bbd90091082af19fedf16bf73d502e5e
4:15:31 PM: Fetching cached dependencies
4:15:31 PM: Starting to download cache of 254.7KB
4:15:31 PM: Finished downloading cache in 119.957538ms
4:15:31 PM: Starting to extract cache
4:15:31 PM: Failed to fetch cache, continuing with build
4:15:31 PM: Starting to prepare the repo for build
4:15:32 PM: No cached dependencies found. Cloning fresh repo
4:15:32 PM: git clone git@bitbucket.org:oneshoewebsite/website2018
4:15:38 PM: Preparing Git Reference refs/heads/develop
4:15:39 PM: Starting build script
4:15:39 PM: Installing dependencies
4:15:40 PM: Downloading and installing node v8.12.0...
4:15:40 PM: Downloading https://nodejs.org/dist/v8.12.0/node-v8.12.0-linux-x64.tar.xz...
4:15:40 PM:
#
4:15:40 PM: 1.9%
4:15:41 PM:
#################################
4:15:41 PM: ###### 54.9%
4:15:41 PM:
####################################
4:15:41 PM: #################################### 100.0%
4:15:41 PM: Computing checksum with sha256sum
4:15:41 PM: Checksums matched!
4:15:43 PM: Now using node v8.12.0 (npm v6.4.1)
4:15:43 PM: Attempting ruby version 2.3.6, read from environment
4:15:44 PM: Using ruby version 2.3.6
4:15:44 PM: Using PHP version 5.6
4:15:44 PM: Started restoring cached node modules
4:15:44 PM: Finished restoring cached node modules
4:15:45 PM: Started restoring cached yarn cache
4:15:45 PM: Finished restoring cached yarn cache
4:15:45 PM: Installing yarn at version 1.3.2
4:15:45 PM: Installing Yarn!
4:15:45 PM: > Downloading tarball...
4:15:45 PM: [1/2]: https://yarnpkg.com/downloads/1.3.2/yarn-v1.3.2.tar.gz --> /tmp/yarn.tar.gz.fCkWccGqjP
4:15:45 PM: % Total % Received % Xferd Average Speed Time Time Time Current
4:15:45 PM: Dload Upload Total Spent Left Speed
4:15:45 PM:
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
4:15:45 PM:
100 91 100 91 0 0 985 0 --:--:-- --:--:-- --:--:-- 989
4:15:45 PM: 0
4:15:45 PM: 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 608 0 0 2085 0 --:--:-- --:--:-- --:--:-- 593k
4:15:45 PM: 1
4:15:45 PM: 00 865k 100 865k 0 0 991k 0 --:--:-- --:--:-- --:--:-- 991k
4:15:45 PM: [2/2]: https://yarnpkg.com/downloads/1.3.2/yarn-v1.3.2.tar.gz.asc --> /tmp/yarn.tar.gz.fCkWccGqjP.asc
4:15:45 PM:
100 95 100 95 0 0 2529 0 --:--:-- --:--:-- --:--:-- 2529
4:15:45 PM:
0 0 0 612 0 0 4564
4:15:45 PM: 0 --:--:-- --:--:-- --:--:-- 4564
4:15:46 PM: 1
4:15:46 PM: 00 1027 100 1027 0 0 4551 0 --:--:-- --:--:-- --:--:-- 4551
4:15:46 PM: > Verifying integrity...
4:15:46 PM: gpg: Signature made Thu 02 Nov 2017 04:44:10 PM UTC using RSA key ID FD2497F5
4:15:46 PM: gpg: Good signature from "Yarn Packaging <yarn@dan.cx>"
4:15:46 PM: gpg: WARNING: This key is not certified with a trusted signature!
4:15:46 PM: gpg: There is no indication that the signature belongs to the owner.
4:15:46 PM: Primary key fingerprint: 72EC F46A 56B4 AD39 C907 BBB7 1646 B01B 86E5 0310
4:15:46 PM: Subkey fingerprint: 6A01 0C51 6600 6599 AA17 F081 46C2 130D FD24 97F5
4:15:46 PM: > GPG signature looks good
4:15:46 PM: > Extracting to ~/.yarn...
4:15:46 PM: > Adding to $PATH...
4:15:46 PM: > We've added the following to your /opt/buildhome/.profile
4:15:46 PM: > If this isn't the profile of your current shell then please add the following to your correct profile:
4:15:46 PM: export PATH="$HOME/.yarn/bin:$HOME/.config/yarn/global/node_modules/.bin:$PATH"
4:15:46 PM:
4:15:46 PM: > Successfully installed Yarn 1.3.2! Please open another terminal where the `yarn` command will now be available.
4:15:46 PM: Installing NPM modules using Yarn version 1.3.2
4:15:47 PM: yarn install v1.3.2
4:15:47 PM: warning package.json: No license field
4:15:47 PM: warning oneshoe@1.0.0: No license field
4:15:47 PM: [1/4] Resolving packages...
4:15:48 PM: [2/4] Fetching packages...
4:15:48 PM: warning Pattern ["gatsby-link@next"] is trying to unpack in the same destination "/opt/build/.yarn_cache/v1/npm-gatsby-link-2.0.0-rc.2-e6f54bc9ae8f825136fbdb1ad789c2dbbf025d8a" as pattern ["gatsby-link@^2.0.0-rc.2"]. This could result in a non deterministic behavior, skipping.
4:16:07 PM: info fsevents@1.2.4: The platform "linux" is incompatible with this module.
4:16:07 PM: info "fsevents@1.2.4" is an optional dependency and failed compatibility check. Excluding it from installation.
4:16:07 PM: [3/4] Linking dependencies...
4:16:07 PM: warning " > @babel/plugin-proposal-class-properties@7.0.0" has unmet peer dependency "@babel/core@^7.0.0-0".
4:16:07 PM: warning "gatsby > mini-css-extract-plugin > schema-utils > ajv-errors@1.0.0" has unmet peer dependency "ajv@>=5.0.0".
4:16:07 PM: warning " > slick-carousel@1.8.1" has unmet peer dependency "jquery@>=1.8.0".
4:16:07 PM: warning " > eslint-config-airbnb@16.1.0" has unmet peer dependency "eslint-plugin-jsx-a11y@^6.0.2".
4:16:14 PM: [4/4] Building fresh packages...
4:16:29 PM: success Saved lockfile.
4:16:29 PM: Done in 42.18s.
4:16:29 PM: NPM modules installed using Yarn
4:16:30 PM: warning package.json: No license field
4:16:30 PM: Started restoring cached go cache
4:16:30 PM: Finished restoring cached go cache
4:16:30 PM: unset GOOS;
4:16:30 PM: unset GOARCH;
4:16:30 PM: export GOROOT='/opt/buildhome/.gimme/versions/go1.10.linux.amd64';
4:16:30 PM: export PATH="/opt/buildhome/.gimme/versions/go1.10.linux.amd64/bin:${PATH}";
4:16:30 PM: go version >&2;
4:16:30 PM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.10.linux.amd64.env';
4:16:30 PM: go version go1.10 linux/amd64
4:16:30 PM: Installing missing commands
4:16:30 PM: Verify run directory
4:16:30 PM: Executing user command: npm run build:nl
4:16:30 PM: > oneshoe@1.0.0 build:nl /opt/build/repo
4:16:30 PM: > GATSBY_LANGUAGE=nl-nl gatsby build
4:16:32 PM: success open and validate gatsby-config — 0.007 s
4:16:33 PM: success load plugins — 0.163 s
4:16:33 PM: success onPreInit — 0.764 s
4:16:33 PM: success delete html and css files from previous builds — 0.089 s
4:16:33 PM: success initialize cache — 0.008 s
4:16:34 PM: success copy gatsby files — 0.012 s
4:16:34 PM: success onPreBootstrap — 0.006 s
4:16:34 PM: Starting to fetch data from Prismic
4:16:35 PM: Fetch Prismic data: 1085.642ms
4:16:42 PM: success source and transform nodes — 8.514 s
4:16:44 PM: success building schema — 1.562 s
4:16:44 PM: success createPages — 0.214 s
4:16:44 PM: success createPagesStatefully — 0.328 s
4:16:44 PM: success onPreExtractQueries — 0.004 s
4:16:45 PM: success update schema — 1.102 s
4:16:46 PM: success extract queries from components — 0.352 s
4:17:30 PM: success run graphql queries — 44.577 s — 199/199 4.46 queries/second
4:17:30 PM: success write out page data — 0.006 s
4:17:30 PM: success write out redirect data — 0.001 s
4:30:04 PM: info bootstrap finished - 813.661 s
4:30:04 PM: success onPostBootstrap — 0.010 s
4:30:34 PM: success Building production JavaScript and CSS bundles — 30.560 s
4:30:40 PM: success Building static HTML for pages — 5.619 s — 198/198 108.75 pages/second
4:30:40 PM: info Done building in 849.863 sec
4:30:40 PM: Execution timed out after 15m0s
4:30:40 PM: Error running command: Command did not finish within the time limit
4:30:40 PM: Failing build: Failed to build site
4:30:40 PM: failed during stage 'building site': Command did not finish within the time limit
4:30:41 PM: Finished processing build request in 15m9.995337154s
Environment
System: OS: macOS High Sierra 10.13.6 CPU: x64 Intel® Core™ i7-4770HQ CPU @ 2.20GHz Shell: 3.2.57 - /bin/bash Binaries: Node: 10.7.0 - /usr/local/bin/node Yarn: yarn install v0.24.6 [1/4] Resolving packages… [2/4] Fetching packages… [3/4] Linking dependencies… [4/4] Building fresh packages… success Saved lockfile. Done in 74.20s. - ~/.node/bin/yarn npm: 6.1.0 - /usr/local/bin/npm Browsers: Chrome: 69.0.3497.81 Firefox: 61.0.1 Safari: 11.1.2 npmPackages: gatsby: next => 2.0.0-rc.15 gatsby-image: next => 2.0.0-rc.1 gatsby-link: next => 2.0.0-rc.2 gatsby-plugin-mailchimp: ^2.2.3 => 2.2.3 gatsby-plugin-react-helmet: next => 3.0.0-rc.1 gatsby-plugin-sharp: next => 2.0.0-rc.3 gatsby-plugin-styled-components: next => 3.0.0-rc.1 gatsby-source-prismic-lang: ^2.0.0-alpha.5 => 2.0.0-alpha.5 gatsby-transformer-remark: next => 2.1.1-rc.1 gatsby-transformer-sharp: next => 2.1.1-rc.3 npmGlobalPackages: gatsby-cli: 1.1.58
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 13
- Comments: 59 (32 by maintainers)
@DSchau
#8379 Looks super promising ! That’d be a great step forward imho.
–
I think I misphrased this. Let me try again:
I don’t get a cache invalidation when I change an image. I do when, as you mentioned, I edit
package.jsonorgastby-*.jsfiles. Sorry for that.–
I see your point and it’s something I can’t say I had considered. I had my use case in mind which is rather basic: I have a server serving a Gatsby site. When I push code to a production branch, I like to grab the new code, run a build and then serve that new build.
What happens right now is: I edit the
package.jsonfile (I update a dependency, say React for example), I push my code and then get a 30 minutes (super CPU intensive, which I guess could have side effects if you’re using this same box for other purposes) build time before the new version of the site is ready to be served. I find it a bit of a waste of time / power to recompute all images (and everything else, but images are the most expensive by far) because I bumped a dependency.Same thing happens on my local machine too. With better hardware, it’s faster of course (still takes a good 10/15 minutes), but every time I update a
gastby-*.jsfile orpackage.json, I’m down for 15 minutes of Node hogging my CPU and me waiting to test the changes. Each time. So my React bump example above is actually 15 minutes (local machine) + 30 minutes (droplet) of just building before the new version is live.Of course, I could only build locally and deploy compiled code directly, but I prefer to do it this way personally (for a few reasons which are beyond this - already too long - reply).
Not a major issue though, Gatsby is still one of the - if not THE - best webdev experience I’ve had. I absolutely love it.
My take on this is that Gatsby needs to be smarter when clearing cache, especially images. The problem here is, as far as I can tell, Gatsby will rebuild everything whenever you change a « core » file, including images.
This is what takes forever. I don’t use netlify but I have a similar build set up on a $5 DO droplet and my rebuild when cache is cleared takes about 30 minutes (most of which is just image transformation).
I think Gatsby should try to preserve transformed images cache as long as it can, this would most likely solve this issue as well as greatly reduce build times when you just update dependencies or a « core » file used by Gatsby.
Okay, So I fixed mine by doing this
Clear cache and deploy site!
We already have credit card in and paying for build minutes, just had 4 builds fail due to this 15mins limit (1 hour of build time wasted which we’ll have to pay for…) please get our limit up to 30 mins too, we have too many routes to be limited to 15m. Site: bitcompare https://bitcompare.net Thanks
We did some changes in our image processing plugins but unsure if it actually relates to this issue. It seems like clearing your cache might fix the issue. Would love some feedback of some people struggling.
@Coriou No, I’m not using that plugin.
@azamatsmith it did - thanks. Not ideal, but it worked. Will still want to actually fix this, but will do for now.
@lol-russo This particular error might be due to the fact that data is not present for the featuredImage field on all nodes. I found that I have to default nodes for data with holes in it for Gatsby to build completely.
I would verify that every record has that field populated first. If that does end up being your problem, I’d either fix the data, or create a plugin to manage the data as it is being consumed.
@azamatsmith Currently working on this issue for my project. I figured out that it was the image transformations that were taking so long (makes sense). Also, Gatsby invalidates cache quite often (can be triggered by different things, like editing your gatsby-node file). I guess I could tackle the cache clearing issue (make sure transformed images are kept for example) but I took a different approach.
What I’m trying to do is to write my own transformer that would be compatible with gatsby-image but instead of manipulating images locally, it’d rely on a remote service (looking at Pixboost right now). This would require no image manipulation at build, thus drastically improving build times regardless of cache.