nx: Changes to package-lock.json trigger an Affected on all packages

Another problem with this: so let’s say you have 100 projects in your monorepo (could be much lower). You want to update a single dependency to a new version. Now you want to know which projects would be updated by this change so you can do proper regression testing… currently all 100 projects would be shown as affected, but wouldn’t it be nice if you could run nx:affected and determine which projects are actually using that dependency and run your regression testing on only those projects?

the pnpm workspace workflow definitely seems to not be ideal here.

Adding a package to an app or package in the workspace ends up marking the entire repo as affected. Would love to find a way to resolve this without ignoring the changes

We have a yarn.lock and it seems to work adding yarn.lock to the .nxignore file. Now we only get the affected services/packages as expected. Perhaps there will be some side affects that i’m not considering?

Our pipeline looks roughly like this:

• nx affected --target=version
  • https://github.com/jscutlery/semver
  • on PRs, only bumps versions in the package.json (e.g. 1.0.0 to 1.0.1)
  • on merge to main, bumps versions, creates a commit and tag (e.g. 1.0.0 to 1.0.1 again, so that it gets cache from the PR run)
  • we found it was safer to invalidate cache on PRs instead of on main
• nx affected --targets=lint,test,build,docker
  • because of the version bump, cache is invalidated any lib has to re-run lint, test and build
  • apps have to run docker (https://github.com/gperdomor/nx-tools/tree/main/packages/nx-container)
• nx affected --target=deploy
  • https://github.com/edbzn/ngx-deploy-npm for libs
  • on PRs, it’s just a --dryRun
  • on merge to main, publish to our internal registry
  • it makes more sense for us to do like this because we don’t have a monorepo, but a shared repo where libs live
  • other apps in other repos have to install them from the registry, and once a fix that got merged, we immediately want it available to install in the other repo
• nx affected --target=upgrade
  • proprietary upgrade script for some (not all) applications
  • on merge to main, upgrades the applications in production

So when one lib or app adds one dependency, say via pnpm install --filter {my-lib} {some-dependency}, the lockfile makes everything affected, triggering a release cycle for all projects in the repo, and a full rebuild.

I understand your pain and can inform you that solution is on the way.

We are implementing a logic to restrict hash scope for custom targets.

This would for example allow you to specify that your docker push target is not affected by any changes in the installed packages and solely depends on the project’s built artifact.

But even still, let’s say I use docker cache. I still have to build and push all new containers as my development environment builds and pushes containers tagged with their git hash. That means not only to I have to build and push, but I also have all of my kubernetes pods restart and reload a new container even though nothing in the apps have changed.

I am having the same issue, this is a big problem:(

do you have update on that docs

Not yet

what would be a way to hook into that calculation

Currently, there is no way to extract that information as far as I know. The calculation happens in realtime as you execute the target. But maybe @FrozenPandaz would know more.

Btw. the print-affected is deprecated. You should use nx show

It’d be great to have an article/tutorial with examples of a docker (nx-container) and/or npm (semver & ngx-deploy-npm) target that uses the externalDependencies option.

It’s usually docker push or npm publishing that will suffer when the lockfile affects every project.

Ok I do see that the build task is cached but what I do not understand how building and pushing a docker container stamped with the version number (git hash) and updating kubernetes can be a cachable operation. If changing a single dependency of one project, updates the lock file then all of my deploy targets will run, which due to be docker image builds, push and kubernetes updates will have to run as I do not see how caching can work for that. Unless the image is actually built and pushed, updating kubernetes with the new version will fail as it won’t be able to find the image.

This also causes very high resource usage on our kubernetes cluster as all of the pods need to roll over.

@meeroslav @meeroslav

Creating a .nxignore with content

pnpm-lock.yaml

doesn’t work for me. The cache is invalided if I change any integrity hash value. Example:

  /remeda@1.9.0:
    resolution:
      {
        integrity: sha512-dQolqsREsfF8hV6f67wiS4q/DkdsGr714NM1MwYqdMJEStDwkug6UD7ALPbEX7O4vWR3dFW6qJomKxHYhkX5lbAb==,
      }
    dev: false

The cache is invalided for all commands. Also, for packages that don’t depend on this library.

nx version: 16.2.2

I am using custom commands from package.json, not executors. Is there any workaround for it? This is a huge problem for my team. We are migrating to nx, and we will plan to keep adding more packages and split the existing code. The CI times will grow instead of shrink.

I’m still keeping this closed as the affected works as expected, but caching is currently a bit limiting (which is why we are making changes to it)

I am happy to tell you we are actually working on a solution for this. It’s currently in the design phase but you should expect it in a few weeks.

In the meantime, you can use .nxignore as @smailc suggested.

We have an existing monorepo with ~60 services and ~60 packages that we added nx to. Some operations do not use caching:

• typecheck - we separated typechecking to another job using npm script tsc --noEmit and use esbuild during build step. AFAIK there is no official executor to perform typechecking only, so even though we have caching enabled for this step, it won’t work
• deploy jobs (for each environment) - this is not cachable, it deploys different projects using serverless framework / ecs / eks. Updating a single dependency now causes 60 services to deploy
• each ecs deploy job creates an artefact that is later used to confirm successful deployment in ecs in the following wait_for_deployment step that queries the status of deployments for each service until it’s in a stable state
• we have e2e tests that we want to run only if a subset of services is affected, for example, any of 6 services that are tested by that workflow

@ianldgs, changing project’s package.json should mark that project as affected, since package.json is a file belonging to that project.

Hey all,

The logic behind this is not as trivial as you would have expected.

Explanation

The reason why print-affected and the affected itself mark all the projects is because every change in the lock file can affect all of them. You run nx commands via executors (e.g. nx:run-commands or @nrwl/js:tsc). Without knowing what target (and what executor) you are running it’s impossible to tell whether the project is affected or not so it’s safer to mark it as (potentially) affected.

Imagine you have a react app, and you update rxjs. Although your project might not depend on it, you might have an executor that looks like this:

import { range } from 'rxjs';

export default async function buildExecutor(
  options: Schema,
  context: ExecutorContext
) {
  // ...your executor code which uses the range from rxjs
}

Now each build of your app is actually affected by changes in the rxjs.

Caching to the rescue

Despite every project is marked as affected, once you run the actual command (or task) we will have more information and can at that point decide if the project had actually changed or not. The task consists of the following parameters:

• sources of the project (should be intact)
• command (unchanged)
• target executor’s dependencies (e.g. @nrwl/js and everything it depends on)

If your target hasn’t been affected by the change, you will get instant replay from the cache, otherwise as long as any of those is different we will run the full command.

Known executors only

This unfortunately only works with known executors (e.g. @nrwl/*) where we can guarantee the list of dependencies. For all the other executors, including the nx:run-commands we will fall back to all affected mode since we have no idea what that executor might be running and which packages it depends on.

Workaround

There is a workaround that will land in v16 where you can skip checking package.json and lock files. In your nx.json you need to add the following:

{
  "pluginsConfig": {
    "@nrwl/js": { "analyzePackageJson": false }
  }
}

This will disable all the checks so your graph will no longer have any of the external packages included as a node.