nx: "Invalid Cache Directory" error when running tasks locally despite clean local cache

Current Behavior

After upgrading to Nx 16.6.0, we are seeing the new “invalid cache directory” error when running tasks locally. After running nx reset as the error suggests and ensuring that the local Nx cache in node_modules has been removed, we still get the error.

If we use --skip-nx-cache when running tasks then we don’t get the error, which makes sense, but does that mean it’s throwing an error for the artifact retrieved from the remote Nx cloud cache? Again, local cache is empty.

Am I misunderstanding how this should work or could it be a bug?

Expected Behavior

If the local Nx cache is removed, running a task locally should not result in the Invalid Cache Directory error.

GitHub Repo

No response

Steps to Reproduce

1. Make a change to an app locally that would result in a cache miss for the build task, but do not run the task locally (or make sure connection to Nx Cloud is disabled temporarily)
2. Push change to git remote and CI runs build task (Nx Cloud cache miss)
3. Build task artifact is now stored in remote cache. Locally, run nx reset to clear local cache, then run build task.
4. Get “Invalid Cache Directory for Task” error.

Nx Report

>  NX   Report complete - copy this into the issue template

   Node   : 16.14.2
   OS     : darwin-x64
   yarn   : 1.22.19
   
   nx (global)        : 16.4.0
   nx                 : 16.6.0
   @nx/js             : 16.6.0
   @nx/jest           : 16.6.0
   @nx/linter         : 16.6.0
   @nx/workspace      : 16.6.0
   @nx/cypress        : 16.6.0
   @nx/devkit         : 16.6.0
   @nx/eslint-plugin  : 16.6.0
   @nx/react          : 16.6.0
   @nrwl/tao          : 16.6.0
   @nx/web            : 16.6.0
   @nx/webpack        : 16.6.0
   nx-cloud           : 16.0.5
   typescript         : 5.1.6
   ---------------------------------------
   Community plugins:
   @jscutlery/semver : 2.30.1
   @nxkit/playwright : 2.2.0
   ---------------------------------------

Failure Logs

nx run-many -t build
 >  NX   Ran target build for 10 projects (183ms)
 
    ✔    0/0 succeeded [0 read from cache]
 
    ✖    0/0 targets failed, including the following:
 
 
   View structured, searchable error logs at https://nx.app/runs/<redacted>


 >  NX   Invalid Cache Directory for Task "api:build"

   The local cache artifact in "/<redacted>/node_modules/.cache/nx/<redacted>" was not been generated on this machine.
   As a result, the cache's content integrity cannot be confirmed, which may make cache restoration potentially unsafe.
   If your machine ID has changed since the artifact was cached, run "nx reset" to fix this issue.
   Read about the error and how to address it here: https://nx.dev/recipes/troubleshooting/unknown-local-cache
   
   Pass --verbose to see the stacktrace.

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

No response