turborepo: discovery failed: bad grpc status code: The operation was cancelled

Verify canary release

  • I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue

https://github.com/dBianchii/kodix-turbo

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Linux

Which canary version will you have in your reproduction?

turbo 2.0.3

Describe the Bug

When trying to run turbo watch dev -F @kdx/kdx... for my project, I get this error:

gabriel@gabriel-ubuntu:~/Documents/Github/kodix-turbo$ turbo watch dev -F @kdx/kdx...
• Packages in scope: @kdx/api, @kdx/auth, @kdx/date-fns, @kdx/dayjs, @kdx/db, @kdx/eslint-config, @kdx/kdx, @kdx/locales, @kdx/prettier-config, @kdx/react-email, @kdx/shared, @kdx/tailwind-config, @kdx/tsconfig, @kdx/ui, @kdx/upstash-dev, @kdx/validators
• Running dev in 16 packages
• Remote caching disabled
  × failed to connect to daemon
  ╰─▶ server is unavailable: channel closed

Expected Behavior

Should run the project in watch mode without any problems

To Reproduce

Clone https://github.com/dBianchii/kodix-turbo and run pnpm dev:kdx after running pnpm i

Additional context

I tested it on 2.0.4-canary.4

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 29
  • Comments: 50 (9 by maintainers)

Most upvoted comments

I also get this error sometimes:

× failed to connect to daemon
  ╰─▶ server is unavailable: channel closed

But after runing turbo daemon clean the error goes back to “bad grpc status code”

Big thanks to @krlvi for getting this closed in #8800! If anyone is still runs into this, please open up a new issue, so we don’t miss a comment hiding in a closed issue. Thanks all!

@NicholasLYang I’ve seen this problem on both mac and linux (Ubuntu variant) with this repo: https://github.com/t3-oss/create-t3-turbo

On mac the problem only occurs on a case-sensitive volume so using a case insensitive volume is a temporary fix for mac users.

We are seeing this issue 100% of the time we run turbo watch dev on a Mac case-sensitive volume.

This only happens for us on a Mac case-sensitive volume.

If the repo is copied to a non case-sensitive volume, this error does not occur.

Looking forward to a patch for this.

run:

turbo daemon status

then you can see 3 files in the terminal, and delete all of them. then:

turbo daemon stop
turbo daemon start

It’s not fixed on my Linux machine. It’s related to cache in my case. If all dependencies of watched packages are cached, and “FULL TURBO” is shown, then it’s always thrown.

obraz

Workaround:

  1. Remove .turbo/
  2. npx turbo daemon clean, it’ll throw an error complaining about missing .turbo/ in main directory, but it’ll complain without doing this
  3. now watching works

imho, something is wrong with caching, and daemon might incorrectly handle “corrupted” files in .turbo/

Fully reproducible here: https://github.com/DashHub-ai/DashHub

man this is just boring

Gotcha. I’ll try to reproduce and fix these issues. I will note that turbo watch is not compatible with --no-daemon, since we use the daemon to watch your files and produce change events.

If I have it correct, there’s a few issues at hand here:

Is that an accurate summary?

The fix mentioned in PR #9406 does not seem to fix this issue for MacOs case-sensitive volumes. I created a new issue, as requested, targeting this MacOS issue specifically #9536

It didn’t work for me on macOS in case-insensitive volumes either.

I’ve posted a temporary solution that worked for me here: https://github.com/vercel/turborepo/issues/9536#issuecomment-2564209789

The fix mentioned in PR https://github.com/vercel/turborepo/pull/9406 does not seem to fix this issue for MacOs case-sensitive volumes. I created a new issue, as requested, targeting this MacOS issue specifically https://github.com/vercel/turborepo/issues/9536

I ran into this problem too and thought to give it a try to fix it. I have a PR here https://github.com/vercel/turborepo/pull/9406 that I have manually verified to fix the issue, at least under Linux.

It would be awesome if someone else can try this patch as well for some additional validation.

+1 on Mac on case-sensitive volume; works fine when copied to the system drive.

It’s so freaking annoying, let’s do sth with this, pls.

+1 Having same result on Windows with WSL, and on Linux (Linux Mint 22). Using the same project in both OS.


Edit: I managed to fix it. I’m using a docker compose file to spin up a db locally. I was saving the data on ./data in the same folder as the monorepo. After running docker compose up, everything broke. Running docker compose down -v to delete everything, included volumes, removing the entire folder (as root), cleaning the project (deleting all .turbo/ and node_modules), and cleaning the turbo daemon, seems to solve it for me.

+1 from us also. Sadly unable to find a way to reliably reproduce, but it is causing significant disruptions in our development workflow.

Similar constraints that @dobrac mentioned above. If there’s anything else we find out I’ll add here, but for now we aren’t sure what to do other than just deal with it.

All good! Since I haven’t been able to repro consistently, nothing to provide unfortunately. Sorry 😕 If I can lock it down then yes, would be happy to provide a repro

[…] npx turbo daemon clean to clean the daemon manully. After that ,the watch mode works~ 🚀

Just experienced this; running npx turbo daemon clean allowed me to start a turbo watch job again 🤝

turbo daemon clean

@Cypher1 You’re right, execute npx turbo daemon logs , I also find the similar logs: image

And execute npx turbo daemon clean to clean the daemon manully. After that ,the watch mode works~ 🚀

I’ve run in to this fairly reliably (not consistent enough for repro though) when i leave the turbo watch [cmds] running in my terminal and then my computer goes to sleep. If I wake up my computer and try to make a change to my code I receive

unable to connect to daemon after 3 retries

My command is turbo watch dev pkg-dev

Which does the following in my monorepo:

  • Two persistent Fastify servers with tsx watch --clear-screen=false src/server.ts (maybe i should use turbo daemon to watch instead of using tsx watch?)
  • One persistent NextJS app with next dev
  • 6 packages watched with the turbo daemon tsup

If I try to run my turbo watch [cmd] again after this fails, all my processes in the terminal just hang. I have to go to my activity monitor where i normally see 2 node processes still active that I have to force quit. I then can run my command again and it works

The turbo daemon logs don’t seem too helpful at the moment but here you go

2024-06-25T08:23:41.397334Z  WARN turborepo_lib::package_changes_watcher: hashes are the same, no need to rerun
2024-06-25T08:23:41.551849Z  WARN turborepo_lib::package_changes_watcher: changed_files: {AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/20.pack.gz_")}
2024-06-25T08:23:41.551857Z  WARN turborepo_lib::package_changes_watcher: changed_packages: Ok(Some({WorkspacePackage { name: Other("foundry-client"), path: AnchoredSystemPathBuf("apps/web") }}))
2024-06-25T08:23:41.551987Z  WARN turborepo_lib::package_changes_watcher: hashes are the same, no need to rerun
2024-06-25T08:23:42.552668Z  WARN turborepo_lib::package_changes_watcher: changed_files: {AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/20.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/8.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/12.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/8.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/index.pack.gz.old"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/index.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/20.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/5.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/0.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/0.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/12.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/index.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/5.pack.gz")}
2024-06-25T08:23:42.552677Z  WARN turborepo_lib::package_changes_watcher: changed_packages: Ok(Some({WorkspacePackage { name: Other("foundry-client"), path: AnchoredSystemPathBuf("apps/web") }}))
2024-06-25T08:23:42.565069Z  WARN turborepo_lib::package_changes_watcher: hashes are the same, no need to rerun
2024-06-25T17:14:53.505513Z  WARN turborepo_lib::package_changes_watcher: changed_files: {AnchoredSystemPathBuf("packages/trpc/src/scripts/migrate.ts")}
2024-06-25T17:14:53.505556Z  WARN turborepo_lib::package_changes_watcher: changed_packages: Ok(Some({WorkspacePackage { name: Other("@foundrydev/trpc"), path: AnchoredSystemPathBuf("packages/trpc") }}))
2024-06-25T17:14:53.954880Z ERROR turborepo_lib::daemon::server: package changes stream closed: channel closed

Hopefully this helps a little bit

I have the very same issue with watch. I’m running turbo inside a Docker container. Logs I can see:

2024-06-21T10:53:34.000+0000 [DEBUG] turborepo_lib::daemon::connector: looking for pid in lockfile: AbsoluteSystemPathBuf("/tmp/turbod/f53b52ad6d21cceb/turbod.pid")
2024-06-21T10:53:34.000+0000 [DEBUG] turborepo_lib::daemon::connector: found pid: 55
2024-06-21T10:53:34.000+0000 [DEBUG] turborepo_lib::daemon::connector: got daemon with pid: 55
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: creating AbsoluteSystemPath("/tmp/turbod/f53b52ad6d21cceb")
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: watching AbsoluteSystemPath("/tmp/turbod/f53b52ad6d21cceb")
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: creating AbsoluteSystemPath("/tmp/turbod/f53b52ad6d21cceb")
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: watching AbsoluteSystemPath("/tmp/turbod/f53b52ad6d21cceb")
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: connecting to socket: /tmp/turbod/f53b52ad6d21cceb/turbod.sock
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: socket error: transport error

I see the same thing in my repo. Some notes:

  • turbo daemon status was fine.
  • The file was readable. The PID in it was still running.
  • turbo daemon status reported no issues.
  • I had a hunch after seeing that turbo daemon status -vv was fine and cloned my repo onto the same drive as /var and it works. Might be a red herring. Also possibly useful: the dev drive is case sensitive. MacOS.

Hope this helps. Thanks for your hard work