nx: nx affected:e2e is hanging on last app running

I’m having the same issue when running tests in our NX app on Bitbucket Cloud. It seems to be connected to running tests in parallel. When max-parallel is 3, all the tests run, but the actual command never seems to finish on Bitbucket (works fine when I run it locally). When I set max-parallel to 1, it finishes fine on Bitbucket Cloud. This is the line in question:

npx nx affected --target=test --base=origin/master --parallel --max-parallel=3 --ci --code-coverage

I’ve also asked a question on the Bitbucket Forum here: https://community.atlassian.com/t5/Bitbucket-questions/CI-Pipeline-hangs-when-using-nx-with-max-parallel-higher-than-1/qaq-p/2161467#M86672

I am seeing the same problems after migrating to nx 14.5 and Angular 14. It seems that certain of my libs cause this e2e run to hang on our CI (Bitbucket cloud). I have only just started investigating why this happens, but since the log does not print any errors - and adding --verbose does not output anything extra, its hard to toubleshoot.

Can this be related to some dependsOn settings? I noticed that the migration to nx 14.5 adds a dependsOn block in nx.json.

I am using a run-many script to split the affected libs/apps into seperate jobs, so that I can run them seperately in my CI pipeline. The weird part is, that if I run the same run-many script locally, it completes as expected. And because we use the cloud cache, running the CI pipeline afterwards completes it.

I’m also seeing this having just attempted to update my repo from nx@14.8.6 to nx@15.8.7. Everything works perfectly locally, but when CI (CircleCI) runs I’m getting no cache hits after the update, and the final unit test (using Jest) hangs indefinitely. I can’t share my repo as it is proprietary, but if I can find more details or a resolution I’ll share.

still having the same issue with the latest version of NX, and the only way to make it work on the CI is using --runInBand command

I’m also seeing this having just attempted to update my repo from nx@14.8.6 to nx@15.8.7. Everything works perfectly locally, but when CI (CircleCI) runs I’m getting no cache hits after the update, and the final unit test (using Jest) hangs indefinitely. I can’t share my repo as it is proprietary, but if I can find more details or a resolution I’ll share.

I’ve run into the similar issue reported by @jfahrenkrug.

When running jest tests on CircleCI with

$ nx affected --target=test --base=$NX_BASE --head=$NX_HEAD

Some tests run successfully but the remaining tests hanged without any output logs. And the 10mins timeout reached with the following error on CI pipeline

Too long with no output (exceeded 10m0s): context deadline exceeded

After delving into the documents/issues, I found the following articles which recommend to use --runInBand for Jest and letting Nx handle the degrees of parallelism

• nx.dev/packages/jest#performance-in-ci
• Running test in angular is faster for sequential runs than parallel runs #8964 (comment)

So I tried to modify the command as

$ nx affected --target=test --base=$NX_BASE --head=$NX_HEAD --runInBand

Everything works fine and the performance of the test has indeed improved. Instead of reaching the 10mins timeout, the tests are finished in 1min.

But I still don’t know why the difference in performance is so large. Actually, I don’t think it’s the root cause 😭

I’m seeing the same issue with the build step of my (large, >150 project) monorepo - sometimes after the last project has been built then the command hangs until it is killed by CircleCI.

I’ve also observed when run locally that it sometimes takes more than 30 seconds to upload to Nx Cloud after a project has been built, and also that I have not been getting cache hits from Nx Cloud when running.

I’ve run into the similar issue reported by @jfahrenkrug.

When running jest tests on CircleCI with

$ nx affected --target=test --base=$NX_BASE --head=$NX_HEAD

Some tests run successfully but the remaining tests hanged without any output logs. And the 10mins timeout reached with the following error on CI pipeline

Too long with no output (exceeded 10m0s): context deadline exceeded

After delving into the documents/issues, I found the following articles which recommend to use --runInBand for Jest and letting Nx handle the degrees of parallelism

• https://nx.dev/packages/jest#performance-in-ci
• https://github.com/nrwl/nx/issues/8964#issuecomment-1043461553

So I tried to modify the command as

$ nx affected --target=test --base=$NX_BASE --head=$NX_HEAD --runInBand

Everything works fine and the performance of the test has indeed improved. Instead of reaching the 10mins timeout, the tests are finished in 1min.

But I still don’t know why the difference in performance is so large. Actually, I don’t think it’s the root cause 😭

Here is my command:

node --max_old_space_size=8192 node_modules/@nrwl/cli/bin/nx.js affected --base=origin/master --head=HEAD --target=e2e --exclude='$(EXCLUDE-E2E)' --parallel=1 --spec=**/integration/**/ci/**/*

and it still hangs on E2E. I see all tests are run but nothing happens after the last one is run.

The same command is working without DTE feature enabled.

Here is the whole pipeline:

    - script: yarn nx-cloud start-ci-run
      displayName: "Start-CI-Run"
    - script: node --max_old_space_size=8192 node_modules/@nrwl/cli/bin/nx.js affected --base=origin/master --head=HEAD --target=build --parallel --max-parallel=4
      displayName: Build
    - script: node --max_old_space_size=8192 node_modules/@nrwl/cli/bin/nx.js affected --base=origin/master --head=HEAD --target=test --parallel --max-parallel=4
      displayName: Test
    - script: node --max_old_space_size=8192 node_modules/@nrwl/cli/bin/nx.js affected --base=origin/master --head=HEAD --target=lint --parallel --max-parallel=4
      displayName: Lint
    - script: node --max_old_space_size=8192 node_modules/@nrwl/cli/bin/nx.js affected --base=origin/master --head=HEAD --target=e2e --exclude='$(EXCLUDE-E2E)' --parallel=1 --spec=**/integration/**/ci/**/*
      displayName: E2E
    - script: yarn nx-cloud stop-all-agents

I was finally able to reproduce this by telling nx to run in parallel

nx run-many --target=e2e --all --skip-nx-cache --parallel --max-parallel 3

before all tests looked to be executing one at a time preventing the port collision.

So as mentioned by @alexw10 using a different port for each project should prevent this issue.

also setting max-parallel to 1 also works with the caveat that only 1 project will run at a time if you do not want to change the ports

nx run-many --target=e2e --all --skip-nx-cache --max-parallel 1

Side note: using the --verbose flag will print the error about the port being used and preventing hanging 🤔

Still looking into why Nx isn’t exiting with an error when this port collision happens.

So one of our devs seems to have resolved this issue. He ran both e2e tests separately at same time and noticed port already in use error. In angular.json he updated the ports to be different for e2e’s running and this seems to have resolved the problem…

Not sure if this is an action item for you guys or not but I guess at a minimum we have a work around

Hi @alexw10, thanks for the reproduction steps. Seems like this issue slipped through the cracks, but we are looking into it now.

We are experiencing something similar where cypress component tests complete and the result is published to nx cloud, but then it hangs until our build pipeline times out.

We are running in to what seems to be the same issue with NX hanging after running some of our E2E tests and never finishing after getting NX SUCCESS confirmation. In our case its (luckily not on all runs) happens after running our shorter/faster component tests. Furthermore, It have been observed that the caching stage for our jest test at times can take up to 8 min. before our pipeline moves on.

I’ve seen talk about NX potentially losing track of the info it needs to send (This is only theorized), particularly for the jest test, which could be solved by using --runInBand, though the author have later removed the issue from stackoverflow.

Not sure if it’s exactly the same issue we are facing, but wanted to increase the attention around it.

Is no one else hitting this? 😦 really causing us not to go to Angular 13/Latest nx