k3s: Broken `sha256sum-amd64.txt` files in recent releases

temp work around while is fixed is to disable hashes verification on install script. Lines 546 and 554

546     #download_hash
547 
548     if installed_hash_matches; then
549         info 'Skipping binary downloaded, installed k3s matches hash'
550         return
551     fi
552 
553     download_binary
554     #verify_binary
555     setup_binary

temp work around while is fixed is to disable hashes verification on install script. Lines 546 and 554

546     #download_hash
547 
548     if installed_hash_matches; then
549         info 'Skipping binary downloaded, installed k3s matches hash'
550         return
551     fi
552 
553     download_binary
554     #verify_binary
555     setup_binary

Thank you. Temporary workaround in case you use curl -sfL https://get.k3s.io | sh - in CI:

curl -sfL https://get.k3s.io | sed "s/download_hash$/#download_hash/" | sed "s/verify_binary$/#verify_binary/" | sh -

Test to verify it’s working correctly and only replacing these two lines:

curl -sfL https://get.k3s.io | sed "s/download_hash$/#download_hash/" | sed "s/verify_binary$/#verify_binary/" > disabled_ver.sh
curl -sfL https://get.k3s.io > original.sh
diff original.sh disabled_ver.sh
546c546
<     download_hash
---
>     #download_hash
554c554
<     verify_binary
---
>     #verify_binary

This is only a workaround and should be removed as soon as CDN is back up!

All the sha256sum-amd64.txt of 1.21 (from 1.21.0 to 1.21.5) seem to be working again, for example:

https://github.com/k3s-io/k3s/releases/download/v1.21.5%2Bk3s1/sha256sum-amd64.txt

I haven’t tried other versions, though, but this is encouraging.

I have an issue open too - https://support.github.com/ticket/personal/0/1319404 and we’ve reached out to CNCF to see if they can shake anyone loose.

GitHub resolved my ticket just before midnight last night:

Thanks for your patience while we worked on this!

I just wanted to let you know that we’ve deployed a fix that should have resolved the issue.

Sorry again for the trouble, and please let us know if there’s anything else we can do.

We’re seeing the same thing with the sha256sum files across all of our releases and repos. There appears to be some sort of outage with the GitHub artifacts CDN, which is hosted by fastly in front of S3. I don’t believe there’s anything we can do about it.

brandond@dev01:~$ curl -svL https://github.com/k3s-io/k3s/releases/download/v1.21.5%2Bk3s1/sha256sum-amd64.txt 2>&1 | grep -E '> GET|< location:|<Error'
> GET /k3s-io/k3s/releases/download/v1.21.5%2Bk3s1/sha256sum-amd64.txt HTTP/2
< location: https://github-releases.githubusercontent.com/135516270/d392d2d3-6890-4573-8818-bdbf25ef59b3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210920T213039Z&X-Amz-Expires=300&X-Amz-Signature=fd8a5395a78840a14207050a44dc916913723ba1b8b65af9a1e5c2f3cd7f1e07&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream
> GET /135516270/d392d2d3-6890-4573-8818-bdbf25ef59b3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210920T213039Z&X-Amz-Expires=300&X-Amz-Signature=fd8a5395a78840a14207050a44dc916913723ba1b8b65af9a1e5c2f3cd7f1e07&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream HTTP/2
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAIWNJYAX4CSVEH53A</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256

I can also recreate this with any other repo, for example a personal repo here: https://github.com/brandond/releases-testing/releases/tag/v0.1.0

brandond@dev01:~/repos/releases-testing$ gh release upload v0.1.0 sha256sum-amd64.txt
HTTP 502: Error uploading to https://objects.githubusercontent.com/github-production-release-asset-2e65be: 401 (https://uploads.github.com/repos/brandond/releases-testing/releases/49944902/assets?label=&name=sha256sum-amd64.txt)
brandond@dev01:~/repos/releases-testing$ gh release upload v0.1.0 sha256sum-amd64.txt
HTTP 422: Validation Failed (https://uploads.github.com/repos/brandond/releases-testing/releases/49944902/assets?label=&name=sha256sum-amd64.txt)
ReleaseAsset.name already exists
brandond@dev01:~/repos/releases-testing$ gh release upload v0.1.0 sha256sum-amd64.txt --clobber
HTTP 422: Validation Failed (https://uploads.github.com/repos/brandond/releases-testing/releases/49944902/assets?label=&name=sha256sum-amd64.txt)
ReleaseAsset.name already exists

If I attempt to edit this release, the file shows an error next to it in the artifact list. Deleting it and saving the release gives me a 500 error from GH.

I opened a ticked with Github Support about 2 hours ago (~1730EDT). Seems to be effecting the same file on all releases, so no ability to use an older release. I started experiencing problems around 1400EDT and relayed that information to them in the ticket.

We’re seeing the same thing with the sha256sum files across all of our repos. There appears to be some sort of outage with the GitHub artifacts CDN, which is hosted by fastly in front of S3. I don’t believe there’s anything we can do about it.

That’s a darn shame. Do people sometimes get into the habit of hosting these files locally? I’m still experimenting with installing k3s to host my Rancher instance, so was relying heavily on this for development today.