turborepo: Persistent tasks in watch mode don't wait for first run of their dependencies to finish

Verify canary release

  • I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue

https://github.com/OliverJAsh/turbo-watch-deps-of-persistent

What package manager are you using / does the bug impact?

Yarn v2/v3/v4 (node_modules linker only)

What operating system are you using?

Mac

Which canary version will you have in your reproduction?

2.0.6-canary.0

Describe the Bug

Persistent tasks in watch mode don’t wait for first run of their dependencies to finish.

  • Package b depends on package a.
  • Both packages have a build task.

In non-watch / run mode, build a correctly runs before build b:

$ TURBO_UI=false yarn run turbo run build --filter b
• Packages in scope: b
• Running build in 1 packages
• Remote caching disabled
a:build: cache bypass, force executing 7c09bd93554e1d1c
a:build: A
b:build: cache bypass, force executing 7f551e27d3b8aa66
b:build: B

 Tasks:    2 successful, 2 total
Cached:    0 cached, 2 total
  Time:    1.516s

However, in watch mode, Turbo runs both builds in parallel. Build b does not wait for build a to finish even though it’s a dependency:

$ TURBO_UI=false yarn run turbo watch build --filter b
• Packages in scope: b
• Running build in 1 packages
• Remote caching disabled
• Packages in scope: b
• Running build in 1 packages
• Remote caching disabled
a:build: cache bypass, force executing 7c09bd93554e1d1c
b:build: cache bypass, force executing 25c5580a26314aac
b:build: B
a:build: A

Expected Behavior

Persistent tasks in watch mode should wait for the first run of their dependencies to finish, otherwise they may fail.

To Reproduce

  1. Clone https://github.com/OliverJAsh/turbo-watch-deps-of-persistent.
  2. Run yarn.
  3. Run TURBO_UI=false yarn run turbo watch build --filter b.

Additional context

I believe the idea is that persistent tasks should self-heal: https://github.com/vercel/turbo/issues/8164#issuecomment-2122878984

However, this is inefficient and can lead to confusing error messages.

In our case we’re using webpack watch and rebuilding is CPU intensive, so it would be better if webpack waited for its dependencies to finish building before starting.

Furthermore, our webpack watch task can only self-heal after the build has started—it can’t self-heal if the configuration itself is missing dependencies.

We only expect the order to be respected for the first run, not subsequent to that.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 39
  • Comments: 18 (2 by maintainers)

Most upvoted comments

@NicholasLYang IMHO this feels like a huge blocker of turborepo adoption. Given it isn’t a priority I feel like I am missing something. As someone trying to use turbo watch since it showed up in canary I don’t quite understand what it is currently solving without this. Is there an example repository somewhere that would show how someone can leverage turbo watch?

If i am understanding correctly, the usual set up is:

  • apps with persistent dev tasks (next, vite, etc.)
  • packages with a build step

Whenever a package changes, the dependent apps need to know about it. Without it, one would have to restart turbo dev on every change.

I tried various work arounds:

  • make dev dependent on dev and add a bundler to every package that supports some sort of watch mode, but you can’t make persistent tasks dependent on other persistent tasks
  • remove persistent, and remove all hot reloads, hasn’t been very succesful

The best solution for now has been using packages without a build step and just export the ts files directly as much as possible and let the host apps bundler bundle the package instead.

Appreciate any guidance / insight!

Is there a way to configure it so that only the non-persistent tasks are executed after the first run?

I have a configuration like so:

    "build": {
      "dependsOn": [
        "^build"
      ],
      "outputs": [
        "dist/**"
      ]
    },
    "dev": {
      "dependsOn": ["^build"],
      "cache": false,
      "persistent": true,
      "interruptible": true,
    },

and while this does wait for the first run of dependencies to finish, subsequent changes to packages also runs dev again. dev for me is a Nest dev server that doesn’t need to be rerun.

@dylanjmcdonald In the context of the kitchen sink example, if you change the code inside of @repo/logger the api tsup --watch won’t detect the change as its not aware when one of its dependencies change.

This led me to then make the following changes:

  1. Change the dev script in the root package.json to turbo watch dev
  2. Delete the dev script inside of the packages/logger and packages/ui because we are using the turbo watch command now
  3. Make the dev task inside of api interruptible so that the api server is restarted when its dependencies change

This leaves us with the following setup:

  1. Apps inside the apps/ directory have dev scripts that are both persistent but only apps/api is interruptible
  2. Packages inside the packages/ directory have a build script and rely on turbo watch to be rebuilt

However, with such setup (which i believe is an extremely common one in monorepos) there is a bug currently (turbo@2.3.3).

Repro steps:

  1. Update the code inside packages/ui package
  2. packages/ui gets rebuilt (as it should)
  3. The interruptible api dev task gets terminated and isn’t restarted.

I believe this bug is documented here => https://github.com/vercel/turborepo/issues/9421

I haven’t been able to find a good workaround for this issue.

UPDATE: A neat workaround that I found was to mark all persistent tasks as non-interruptible and use the watch command from tsx to monitor and restart those tasks. tsx watch (v4.19.2) also detects changes inside internal package dependencies of the watched package.

@KajSzy I experienced something similar to this and it turned out it was a bug in the latest v2.2.3 release where persistent tasks were accidently stopped in watch mode: https://github.com/vercel/turborepo/pull/9330. Fixed in v2.2.4-canary.2+