gatsby: [fsevents bug] Stuck at "source and transform nodes" / "createPagesStatefully" on MacOS

For me, cleaning the cache with gatsby clean solved the issue.

Hey everybody, the patched version of fsevents has been published. You should be able to simply delete your yarn.lock file, and run yarn again. Each of your dependencies should automatically pick up fsevents@2.1.1, which should resolve the issue.

from https://github.com/gatsbyjs/gatsby/issues/6385#issuecomment-531949380 - I am also seeing this issue but gatsby clean did not solve it. It seems like it might just be the CLI printout is hanging, which is why resizing the terminal fixes it?? 🤷‍♂ 😕 ❓❗️

@JaKXz - Thank-you! This was driving me nuts. The fix does seem to be resizing the terminal window in VS Code. Just drag it up or down a bit and the develop task chugs along happily. I tested in a couple of different cases both yarn and npm, workspaces and not. Seemed to be working in all those cases for me. It also seems to freeze or hang at createPagesStatefully more often than source and transform nodes now for me. I am going to leave the issue open for now, this may need to be fixed in a less “hacky” way by someone with more know how than me.

For me, limiting concurrent downloads as described here seems to help.

I added the following line to my .env file. The default is 200. GATSBY_CONCURRENT_DOWNLOAD=50

Not sure if it solves your problem but maybe it helps someone 😃

This kind of seems like a kernel+shell related issue. Probably some way node interacts with your kernel and/or your shell. I only say this because I can’t seem to replicate the issues using Linux or Windows. I can’t seem to find any known issues, so either a) it’s some combination of code patterns unique to Gatsby + the interaction with the system, or b) I just don’t know the right question to ask yet.

If resizing the terminal fixes the issue, then it might be some glitch with between node and kqueue causing a block in the event loop. Resizing the terminal sends the process a SIGWINCH signal, which interrupts the event loop, which frees it up and allows it to continue.

Can you try running kill -SIGWINCH ${pid} on the running process when it freezes? Should act the same as resizing the terminal.

I’m also interested in whether or not this happens in all shells. Based on the information so far, this has been failing in bash and zsh, and this is probably one of the common factors between all of the terminal emulators. Can you guys try one or more of sh, csh, ksh, tcsh, etc… ?

If the problem occurs in all of the shells, then I would guess that it’s the way the kernel is handling node’s event loop. This could also be why it’s an intermittent problem. If some function gets less processor time (maybe because of variable system load), it might block for too long, and maybe the kernel tries to reuse an event node somewhere, resulting in a deadlock? I’m not super familiar with the internals of the api, but I bet that source and transform nodes is fairly filesystem IO intensive. That means there’s probably a lot of offloading happening, so who knows what’s really happening on lower levels.

I think it would be a good idea to try to narrow down the surface area of this bug. It occurs most frequently in source and transform nodes, it looks like, so maybe try narrowing it down to what plugin is blocking. Try adding the following lines to node_modules/gatsby/dist/utils/api-runner-node.js:

+ 347       if (api === `sourceNodes`) {
+ 348         debugger;
+ 349       }
  350       resolve(runAPI(plugin, api, Object.assign({}, args, {
  351         parentSpan: apiSpan
  352       })));

Then run node inspect node_modules/.bin/gatsby develop. It will break as soon as it starts, so you have to hit c when you get to the debug> prompt, and wait for it to continue. When it breaks on that debugger line, write exec console.log(plugin), and note what it says in resolve. Then hit c to continue. Just do this until it hangs… if it hangs.

Because of the nature of the event loop, I bet that it won’t even hang while debugging. Those interrupts are probably enough to keep it on a track. Async bugs can be a real pain to track down. If it doesn’t hang while using the debugger, replace that debugger line with:

reporter.log(plugin.resolve);

Hopefully that will turn something up. It would be nice to see what plugin is causing the block. If we can figure that out, we might be able to figure out what needs to be optimized/refactored in order to sort this out.

Be careful - deleting your whole yarn.lock file can also cause other packages to be upgraded.

Another, more precise option if you’re using yarn is to delete just the lines relating to fsevents in the yarn.lock and re-run yarn. This will cause only the affected package to be upgraded. So for example, I removed the following lines:

- 
- fsevents@^2.0.6:
-   version "2.0.7"
-   resolved "https://registry.yarnpkg.com/fsevents/-/fsevents-2.0.7.tgz#382c9b443c6cbac4c57187cdda23aa3bf1ccfc2a"
-   integrity sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==

Another way of doing this is to use the resolutions feature of Yarn (https://yarnpkg.com/lang/en/docs/selective-version-resolutions/). By adding the following to your package.json and then running yarn, it will also upgrade transitive dependencies and update your yarn.lock file:

  "resolutions": {
    "fsevents": "^2.1.1"
  }

After you have done this and your lockfile is updated, you can remove the resolutions property from package.json again.

Edit: removed previous, incorrect version of instructions.

I have this issue on iTerm 2, so it’s not VS Code specific.

Hello,

I can do little but confirm that I’m seeing this issue on a number of gatsby starters. And, as you point out, it sometimes just decides to build or to stop building, hanging at either “source and transform nodes” or createPagesStatefully.

Quite frustrating and leading to trying the most outrageous attempts to fix it.

Weirdly resizing the terminal window worked for me

Hey guys, @ehowey, please check out PR #17938. If anybody is willing to test, please do, and drop a line on the PR.

You can snag my repo, and use it as your gatsby source in your site by using gatsby-dev-cli. First, you need gatsby-dev-cli

# Use your package manager of choice, and install globally
npm install -g gatsby-dev-cli

Then, just clone the repo, and bootstrap it.

git clone --single-branch -b macos-fsevents-fix https://github.com/Js-Brecht/gatsby
cd gatsby
yarn bootstrap
# Set the target repo path for gatsby-dev
gatsby-dev -p "${PWD}"

Then go to the site you wish to use the repo in, and run gatsby-dev

# Disable file watching, unless you intend on making changes to the gatsby repo
gatsby-dev --scan-once

@hbthen3rd

Update: I manage to fix things by restoring my old “yarn.lock” file.

My experience is that the problem can go away for no clear reason only to return later. You may find the problem returning despite restoring yarn.lock. Keep us posted.

This is got to me one of the most odd bugs I have ever encountered because the behaviour seems to change without there being discernible change in the code. In some cases the build hangs at source and transform nodes(60% of time), in some cases it hangs at Create Pages Statefully(20%), in some cases it builds successfully (20%). I have had it display all three of these behaviours without changing a line of code.

I have replicated this behaviour in gatsby-starter-blog as well, which is really odd. Again it makes me think this is an issue on my end locally. Interestingly it only hangs at createPagesStatefully.

I am now starting to think that it has something to do with a library that got updated by homebrew automatically - which one I have no idea but it is the only thing I can think of which I have not tested.

Here is a list of the things I have tried so far:

• Changing node versions, tried 12, 10 and 8

• Reverting to a previous point in my commit history that was working - it still fails now.

• Commenting out plugins and areas of the gatsby-config

• Commenting out the contents of my gatsby-node.js

• Test it again on Netlify, still built successfully, which is what makes me think it must have something to do with my computer.

• Deleting the 4 pages in my src/pages directory and putting in a very basic index.mdx file

• Deleting all the node_module and lock files, reinstalling

• Restarting computer

• Creating a fresh yarn workspace with new clones of the theme/starter from Github.

• Testing gatsby-starter-blog, similar behaviour but it hangs on createPagesStatefully

Got the same issue and could not track the reason. This happen to me both on mac and pc. Could try to relate this to internet speed, however this happened to me also when I was connected to high speed network. What seems to be working for me right now is to add this in my env

GATSBY_CONCURRENT_DOWNLOAD=5

and running using --v8-pool-size=128

I’m using npm. Deleting node_modules, running npm i --save fsevents@2.1.1 and then npm i did the trick for me. Took 19s to startup and I’m using the gatsby-lumen-starter as a base.

– Edit:

I actually finished what I was working on, pushed to Netlify and it was completely unable to build because of fsevents (error fsevents@2.1.1: The platform 'linux' is incompatible with this module).

I switched to yarn and had the same issue so I yarn removed fsevents entirely, and now local and netlify are both working…

@RomanHotsiy that indeed sorts it! Thanks!

@canvaspixels does resizing the terminal window unstuck your build? If so, please try this and let us know if that helped https://github.com/gatsbyjs/gatsby/pull/17938#issuecomment-540661991

This has been solved thanks to the diligent work of @Js-Brecht and @RomanHotsiy. It was an upstream issue in fsevents, way beyond what I could have figured out on my own, and should be fixed in future updates once these changes are implemented and migrate down from fsevents into gatsby itself. For now a reliable workaround is to resize the terminal window, there is a way to fix this in your repo itself which is discussed here: https://github.com/gatsbyjs/gatsby/pull/17938 however you would have to redo this fix after any changes to your node_modules folder and might not be worth the work depending on how often you update your package versions.

I will leave the issue open until the fix migrates downstream into gatsby itself.

For anybody else finding this issue, try

CHOKIDAR_USEPOLLING=1 gatsby develop

That should disable fsevents on MacOS, which seems to be a reliable workaround.

+1 Doesn’t work in iTerm2, but work in Terminal 🤷‍♂️

@ehowey Just so I know I’m understanding you correctly, it did hang while you were in the debugger? In that case, it looks to me like gatsby-source-filesystem is the culprit, which makes sense to me… especially because of existing, related, gatsby issues.

I’d like to see if the plugin can handle a test run. If it hangs while running tests, then I think that will be an easy way to see where it is failing. If not, well… will just have to do some dissecting/debugging, which will be difficult for me, since I don’t have MacOS.

Can you download the main gatsby repo, and run the gatsby-source-filesystem tests?

> git clone git@github.com:gatsbyjs/gatsby.git gatsby-js
> cd gatsby-js
> yarn bootstrap
> yarn jest -- -i './packages/gatsby-source-filesystem/src'

Also, are you doing all of this with your minimal reproduction repository? I see that gatsby-plugin-mdx was run twice, so that tells me that it is not a barebones repository. In a perfect world, it shouldn’t matter, but I think it would be better to run as simple a setup as possible. If you can’t get it to fail reliably with the minimal repo, then use whichever one fails most consistently (in the same place, every time (or close to))

😉

kill -SIGWINCH ${pid} must be run from another shell instance. When the build hangs, open another terminal window/tab, and run the command from there. This little snippet should work:

pid=$(ps -ef |grep -E 'node.*index\.js develop' |awk '{ print $2 }').
kill -SIGWINCH $pid

# Can also be run like this
kill -SIGWINCH $(ps -ef |grep -E 'node.*index\.js develop' |awk '{ print $2 }')

If there are errors, try running the commands seperately:

# Run first
ps -ef grep -E 'node.*index\.js develop'
# Example output
     UID     PID    PPID  TTY        STIME COMMAND
********   39928       1 cons0    06:47:38 /path/to/node /path/to/gatsby-cli/lib/index.js develop
# Second column is the pid you want
kill -SIGWINCH 39928

If you come up empty-handed after running the tests, I think it will be useful to have a core dump, so we can get a stack trace. But, one step at a time.

If resizing works, I suspect this bug is related buffer, somewhere needs stdout flush.