turbo: Unauthorized cache warning
This is a bug. We would appreciate reproductions if you have one to provide. 🙏
Discussed in https://github.com/vercel/turbo/discussions/6740
<div type='discussions-op-text'>Originally posted by dcantu96 December 7, 2023
Hello I just updated from 1.10.16 to 1.11.0 and I started to see the warning below when I ran a turbo command. Some other helpful notes are that our company hosts the remote cache servers and that the TURBO_TOKEN is set in the root .env file. It’s been working without warnings until now
WARNING artifact verification failed: Error making HTTP request: HTTP status client error (401 Unauthorized) for url (https://turborepo.company.app/v8/artifacts/123ID?teamId=MY_TEAM_ID)
```</div>About this issue
- Original URL
- State: open
- Created 7 months ago
- Reactions: 10
- Comments: 35 (15 by maintainers)
I’ve encountered this issue only when using ducktors / turborepo-remote-cache with S3. We actually unhooked S3 just to get around the issue temporarily.
@attila There is a config file at the XDG_CONFIG_DIR (on macos, it’s under
${HOME}/Library/Application Support/turborepo/, the daemon stuff is mostly in a/tmp/directory and shouldn’t affect the upload to the cache. Other than that, there’s the local cache itself atnode_modules/.cache/turbo, and task logs in each package under.turbo.Glad to hear the clean checkout is working though. I wouldn’t spend too much time debugging beyond that. This issue has collected a few different problems, and I don’t think anyone else has reported this specific variant.
@gsoltis
Here's the sanitised log output from the command above
I observed that I am no longer getting errors and there is a remote cache hit on subsequent attempts (I cleaned up
**/.turboandnode_modules/.cache/turbobefore subsequent runs)I am still trying to understand the differences between our own and the scaffolded monorepos, so far no luck.
In the meantime, I took the liberty to run a task with verbose logging on our monorepo that shows the issue.
The command I ran was a simple typescript “linting” command that only has the “^topo” task as a dependency.
Here's the sanitised log output from the monorepo where it does not work
I’ll continue the investigation as to why this Bad file descriptor error occurs, but any pointers you see from the above logs is much appreciated, thank you.
Having bumped into this issue, I can also confirm this using ducktors / turborepo-remote-cache.
Moreover, it seems that this warning may not be remote caching issue, but a generic “artifact verification” related problem as I could just confirm the same bug with local filesystem caches:
The task I tested all the above with does not have any extra outputs, only stdout
Hope that helps.
Is there any alternatives to ducktors / turborepo-remote-cache that don’t have this issue?
@dhoulker I think you have a different scenario, I’ve opened a new issue for you at https://github.com/vercel/turbo/issues/7359
Yes we use the docker container for ours
Continuing to look into this. However, we still have been unable to reproduce it, and it looks like the
412 Precondition Failedis something that ducktors / turborepo-remote-cache returns, so digging in to that as well.I do believe that the
artifact verificationerror has been addressed. If anyone is still running into this, please open a new issue.Still prevalent in 1.12 (and no --go-fallback option)
WARNING failed to contact remote cache: Error making HTTP request: HTTP status client error (412 Precondition Failed) for url (https://subdomain.azurewebsites.net/v8/artifacts/0c30e2a7c696e865?slug=team_uol)@attila Interesting. My suspicion is that the credentials are fine, but something significant to your setup is different in the TLS stack between the Rust and Go implementations.
If you’re comfortable with it, can you email me (greg.soltis@vercel.com) the domain name (please no credentials or private keys) that is hosting your remote cache? I’d like to try to confirm if we can make a TLS connection to it.
Also, any details you’re aware of about the TLS setup would be helpful. For instance: is the root cert for the domain signed by one of the default Certificate Authorities? Or do you have a custom root CA on your machine?
Hi all, we’re continuing to look into this. If people who ran into issues could try again with
turbo@1.11.4-canary.2and see if that version has issues as well that would be helpful. If the issue persists, please include the full error message along with how you’re providing the various pieces of authentication:TURBO_TOKEN/--token/ stored in config file?TURBO_TEAM/--team/ stored team ID in.turbo/config.jsonHi @admmasters, could you open a new issue? And if possible, include the following details: Are you using a proxy? Can you validate that you are getting cache hits using the
--go-fallbackflag from the local remote cache (using--remote-onlyto skip the filesystem cache)?Yes it works perfectly with the
--go-fallback, so looks like an incompatibility with the rust client.