nx: Nx is very slow in a large monorepo (>500k files) due to globbing of `**/project.json`

I’m working on setting up a JS monorepo within a very large Ruby monolith, and running into this same issue. The **/project.json causes all of the directories to be crawled, while we only care about a small subset. I’ve tried leveraging .nxignore to speed it up, but it doesn’t seem to help. I think the problem lies in this section of code: https://github.com/nrwl/nx/blob/73bc2e1c915fac40e29db930297121646362733b/packages/nx/src/config/workspaces.ts#L630-L646

Note that the globSync call is made without taking .nxignore into consideration. The ig value is passed into deduplicateProjectFiles, but not until after we have already globed the file system without considering .nxignore. Would it be possible to parse the .nxignore and add the values to ALWAYS_IGNORE before calling globSync? Something like this:

  try {
    const nxIgnoreContent = readFileSync(`${root}/.nxignore`, 'utf-8')
    nxIgnoreContent.split('\n').forEach((rawLine) => {
      const line = rawLine.trim()
      if (line) {
        ALWAYS_IGNORE.push(line)
      }
    })
    ig.add(nxIgnoreContent);
  } catch {}

@vdumitraskovic I created this patch (yarn patch or patch-package):

diff --git a/src/config/workspaces.js b/src/config/workspaces.js
index 5dad79d352ed3ffa6dc4a3f6cda0656fad099262..2473944ed2c7a3c1c9405f60dad8821af735cfb9 100644
--- a/src/config/workspaces.js
+++ b/src/config/workspaces.js
@@ -464,9 +464,14 @@ function getGlobPatternsFromPackageManagerWorkspaces(root) {
 }
 exports.getGlobPatternsFromPackageManagerWorkspaces = getGlobPatternsFromPackageManagerWorkspaces;
 function normalizePatterns(patterns) {
-    return patterns.map((pattern) => removeRelativePath(pattern.endsWith('/package.json')
+    // Return both package.json and project.json per pattern.
+    // See https://github.com/nrwl/nx/issues/13637
+    const paths = patterns.map((pattern) => removeRelativePath(pattern.endsWith('/package.json')
         ? pattern
         : (0, path_2.joinPathFragments)(pattern, 'package.json')));
+
+
+    return paths.flatMap(pattern => [pattern, pattern.replace(/package\.json$/, 'project.json')]);
 }
 function removeRelativePath(pattern) {
     return pattern.startsWith('./') ? pattern.substring(2) : pattern;
@@ -489,7 +494,9 @@ function globForProjectFiles(root, nxJson, ignorePluginInference = false) {
         .map((glob) => (glob.startsWith('/') ? glob.substring(1) : glob));
     const projectGlobPatterns = [
         'project.json',
-        '**/project.json',
+        // '**' glob causes a major slowdown. The normalizePatterns function is patched to bring back project.json
+        // See https://github.com/nrwl/nx/issues/13637
+        // '**/project.json',
         ...globsToInclude,
     ];
     if (!ignorePluginInference) {

This is still an issue.

Hey, sorry for the lack of answers on this one. Its certainly not stale, and the globbing performance is something that we are aware of. Generally tuning .nxignore can help out quite a bit, but there are some similar issues that would probably shake out to work for this one too.

For instance, #13843 would be solved by a similar solution.

We don’t want to adopt the package.json workspaces field in particular because that ties Nx down to JS and requiring the root package.json. Additionally, the current setup allows you to have some projects managed by npm workspaces and some only by Nx. I could see a world where we add a similar field into nx.json, but I’m not sure exactly what that would look like. I’ll try to get with @FrozenPandaz and see what his thinking is, probably need some input from @vsavkin on this one too.