azure-pipelines-tasks: Error in post-job step of CacheV2 task when caching yarn packages

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: CacheV2 (https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/CacheV2)

Environment

Issue Description

Using the CacheV2 task to cache Yarn packages, we’re getting an error in the post-job step of the cache task.

The task is defined exactly as in the docs.

variables:
  YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn

steps:
- task: Cache@2
  inputs:
    key: 'yarn | "$(Agent.OS)" | yarn.lock'
    restoreKeys: |
       yarn | "$(Agent.OS)"
    path: $(YARN_CACHE_FOLDER)
  displayName: Cache Yarn packages

We’re using the vs2017-win2016 image.

Error logs

Starting: Cache Yarn packages
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.0.0
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - yarn                         [string]
 - "Windows_NT"                 [string]
 - common/config/rush/yarn.lock [file] --> 0551FA989D7CDCB2101F1397E8E7C5468396D4C72935548B6A74BEB126F0296F
Resolved to: yarn|"Windows_NT"|57F7K6kT3NtH/ZuSuWsxueTn4wCxuYls973WEnqCJgE=
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 06101a50-81b0-41b6-ab13-b130ee5c98ae
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `yarn|"Windows_NT"|57F7K6kT3NtH/ZuSuWsxueTn4wCxuYls973WEnqCJgE=`
Information, There is a cache miss.
tar: D\:\\a\\1\\.yarn: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session 06101a50-81b0-41b6-ab13-b130ee5c98ae
##[error]Process returned non-zero exit code: 2
Finishing: Cache Yarn packages

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 19
  • Comments: 30 (1 by maintainers)

Most upvoted comments

@jfheins thanks for the hint.

Current configuration works for me in cloud-hosting:

pool:
  vmImage: 'ubuntu-latest'
variables:
 - name: YARN_CONFIG_CACHE
    value: /home/vsts/.cache/yarn/v6
...
- task: Cache@2
  displayName: Cache node modules
  inputs:
    key: 'yarn | "$(Agent.OS)" | yarn.lock'
    restoreKeys: |
      yarn | "$(Agent.OS)"
    path: $(YARN_CONFIG_CACHE)

Same issue here, but this time with npm

Starting: Cache npm Task : Cache Description : Cache files between runs Version : 2.0.1 Author : Microsoft Corporation Help : https://aka.ms/pipeline-caching-docs Resolving key:

  • npm [string]
  • “Windows_NT” [string]
  • ./product/package.json [file] --> 66FF0092057590D58252E7A1365E9520D41F83083CE0C8C1DA0E038DC4C2D9D1 Resolved to: npm|“Windows_NT”|2DZ3WcfStD108j8qipq/0v97TF5eyy6fpvtPYfpDrZM= Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session xxxxxxxxxx-1282-485c-a6f4-b92a59ed4955 Information, Getting a pipeline cache artifact with one of the following fingerprints: Information, Fingerprint: npm|"Windows_NT"|2DZ3WcfStD108j8qipq/0v97TF5eyy6fpvtPYfpDrZM= Information, There is a cache miss. tar: Cannot change to directory E:\Agents\3\283\s\node_modules: No such file or directory tar: Error is not recoverable: exiting now Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session XXXXXXXX282-485c-a6f4-b92a59ed4955 ##[error]Process returned non-zero exit code: 2 Finishing: Cache npm

Same issue here on a hosted linux agent

2020-06-09T21:30:32.6150811Z ##[section]Starting: Cache yarn packages
2020-06-09T21:30:32.6163777Z ==============================================================================
2020-06-09T21:30:32.6164193Z Task         : Cache
2020-06-09T21:30:32.6164523Z Description  : Cache files between runs
2020-06-09T21:30:32.6164828Z Version      : 2.0.1
2020-06-09T21:30:32.6165145Z Author       : Microsoft Corporation
2020-06-09T21:30:32.6165519Z Help         : https://aka.ms/pipeline-caching-docs
2020-06-09T21:30:32.6165941Z ==============================================================================
2020-06-09T21:30:32.9975064Z Resolving key:
2020-06-09T21:30:33.0154988Z  - yarn      [string]
2020-06-09T21:30:33.0162685Z  - "Linux"   [string]
2020-06-09T21:30:33.0743802Z  - yarn.lock [file] --> EA85735E476B7CE9FF55138A52CE89B7299F92856B621075773F035440C87D93
2020-06-09T21:30:33.0785039Z Resolved to: yarn|"Linux"|QlvZU5gmusF+bBuImFOlW68Qf4aNmdJ1pmhH2LChb5o=
2020-06-09T21:30:33.6622757Z Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session c6edd63d-1238-44dd-a471-3439ac2f402e
2020-06-09T21:30:33.6861181Z Information, Getting a pipeline cache artifact with one of the following fingerprints:
2020-06-09T21:30:33.6862267Z Information, Fingerprint: `yarn|"Linux"|QlvZU5gmusF+bBuImFOlW68Qf4aNmdJ1pmhH2LChb5o=`
2020-06-09T21:30:34.2316763Z Information, There is a cache miss.
2020-06-09T21:30:34.2834153Z tar: /home/vsts/work/1/.yarn: Cannot open: No such file or directory
2020-06-09T21:30:34.2834821Z tar: Error is not recoverable: exiting now
2020-06-09T21:30:34.5635719Z Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session c6edd63d-1238-44dd-a471-3439ac2f402e
2020-06-09T21:30:34.5689658Z ##[error]Process returned non-zero exit code: 2
2020-06-09T21:30:34.5844564Z ##[section]Finishing: Cache yarn packages```

Try adding an s between $(Pipeline.Workspace) and .yarn, so variable goes from:

YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn

To:

YARN_CACHE_FOLDER: $(Pipeline.Workspace)/s/.yarn

This is because when using checkout: self, it checks out the repository to $(Pipeline.Workspace)/s, as is explained here. Unless your using magic, your .yarn folder should be inside the repository.

I think caching will only work on a self hosted agent?

No, this task is especially good for the Azure hosted agents. In a self hosted scenario, most of the caching works without this task. (For example: yarn has a machine wide global cache that persists in a self hosted agent)

The tricky thing is to get the cache directory right. For example, an Azure agent mighthave the yarn cache in “/home/vsts/.cache/yarn” while a self hosted might need “/home/AzDevOps/.cache/yarn”

I’d recommend to include a step that logs out the cache directories for easier troubleshooting:

      - script: |
          node -v
          npm -v
          yarn -v
          yarn cache dir
          npm config get cache
        displayName: "Log versions & paths"

I was running into this exact issue and fixed it by adding YARN_CACHE_FOLDER to the pipeline variables.

I had the Cache task as part of a task group and in the pipeline that uses the task group, there is a required field that asks for the value of YARN_CACHE_FOLDER which is used as ‘path’ for the Cache task. Filled in the field with $(Pipeline.Workspace)/.yarn but the post-job kept failing with same error as above.

Once I also defined YARN_CACHE_FOLDER=$(Pipeline.Workspace)/.yarn in the pipeline variables, the post-job stopped failing.

@johnterickson you are correct. The variable wasn’t being set.

After changing this part:

  - script: |
      corepack enable
      corepack prepare pnpm@latest-8 --activate
      pnpm config set store-dir $(pnpm_config_cache)
    displayName: "Setup pnpm"

to this:

  - script: |
      corepack enable
      corepack prepare pnpm@latest-8 --activate
    displayName: "setup pnpm"

  - script: pnpm config set store-dir $(pnpm_config_cache)
    displayName: "setup pnpm config"

I stopped experiencing this issue.

I’m not sure why the pipeline I created started throwing the /home/vsts/work/1/.yarn: Cannot open: No such file or directory error as I use the exact same setup other places, with $(Pipeline.Workspace)/.yarn. I tried using different vmImages with no luck. Here’s how I solved it for the ones interested.

          - bash: echo "##vso[task.setvariable variable=yarnCacheDir;]$(yarn cache dir)"
            displayName: Set yarn cache dir variable
          - task: Cache@2
            inputs:
              key: '"yarn" | "$(Agent.OS)" | yarn.lock'
              restoreKeys: |
                yarn | "$(Agent.OS)"
              path: $(yarnCacheDir)
            displayName: Cache Yarn packages