sharp: Status 403 from https://github.com/lovell/sharp-libvips/releases/download/...

I’ve contacted GitHub support about this.

A reply from GitHub:

“I have escalated this to our data center team. I do not have an ETA currently but we’ll follow up as soon as we have some news. I’m sorry for the trouble this is causing in the meantime!”

Given three people have just reported this problem within a few minutes of each other, I’d guess there are currently (or were recently) temporary networking/permission problems somewhere between GitHub and AWS, which hosts GitHub releases.

I’m aware of some intermittent problems earlier today in the AWS us-west-2 region that might be related.

The closing response from GitHub:

“This wasn’t the result of any rate limits. This was a configuration change that was rolled back - we’re sorry for the trouble this caused!”

Whilst I have everyone’s attention, it’s always great to learn more about what people are using sharp for in the survey at #35 if you’re able to share details publicly.

This is happening on circleci as well.

> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz
/home/circleci/repo/node_modules/sharp/install/libvips.js:81
          throw new Error(`Status ${response.statusCode}`);
          ^

Error: Status 403
    at /home/circleci/repo/node_modules/sharp/install/libvips.js:81:17
    at f (/home/circleci/repo/node_modules/once/once.js:25:25)
    at ClientRequest.protocol.request.res (/home/circleci/repo/node_modules/simple-get/index.js:63:5)
    at Object.onceWrapper (events.js:273:13)
    at ClientRequest.emit (events.js:182:13)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:555:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:109:17)
    at TLSSocket.socketOnData (_http_client.js:441:20)
    at TLSSocket.emit (events.js:182:13)
    at addChunk (_stream_readable.js:283:12)
make: Entering directory '/home/circleci/repo/node_modules/sharp/build'
  TOUCH Release/obj.target/libvips-cpp.stamp
  CXX(target) Release/obj.target/sharp/src/common.o
../src/common.cc:25:22: fatal error: vips/vips8: No such file or directory
 #include <vips/vips8>
                      ^
compilation terminated.
sharp.target.mk:128: recipe for target 'Release/obj.target/sharp/src/common.o' failed
make: *** [Release/obj.target/sharp/src/common.o] Error 1
make: Leaving directory '/home/circleci/repo/node_modules/sharp/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Linux 4.4.0-141-generic
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/circleci/repo/node_modules/sharp
gyp ERR! node -v v10.11.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN ajv-keywords@3.4.0 requires a peer of ajv@^6.9.1 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.7 (node_modules/nodemon/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.7: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! sharp@0.21.3 install: `(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the sharp@0.21.3 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2019-03-28T16_41_50_493Z-debug.log```

We’ve made our fix public (all versions in Github Releases are available) Add this as an environment variable in CircleCI: SHARP_DIST_BASE_URL=https://s3-us-west-2.amazonaws.com/sharp-distro/

Temporary until real fix comes in from Github Release

Thanks for the reports of this situation returning to normal. I’m going to leave this open until I hear back from GitHub as I suspect the lack of errors may be because the shared S3 bucket used for releases is no longer experiencing peak traffic and is therefore no longer being rate-limited (if that is the reason).

The prebuilt libvips binaries provided by sharp were previously hosted on Bintray but they imposed a limit, originally 1TB/month then increased to 4TB/month in 2017. See https://github.com/lovell/sharp/issues/964 for the kind of problem that led to the use of GitHub Releases instead.

It’s a good example of the tragedy of the commons, albeit Microsoft-provided, so if you work at a for-profit organisation then please consider self-hosting these files in your CI environments.

FWIW we are no longer receiving 403s on our end and Sharp install is back to normal.

That works.

1. Pull down one of the tarballs that matches your sharp version and destination platform, e.g.: https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz

2. Throw them on s3 / dropbox / whatever. Change your build script to either set the SHARP_DIST_BASE_URL environment variable for the container, or directly prepend the npm install command with the override: SHARP_DIST_BASE_URL= https://s3.amazonaws.com/<YOUR BUCKET> npm install

3. Build is working again.

Happening in Circle for me as well:

info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz
/home/circleci/project/node_modules/sharp/install/libvips.js:80
          throw new Error(`Status ${response.statusCode}`);
          ^
Error: Status 403
    at /home/circleci/project/node_modules/sharp/install/libvips.js:80:17

@TimDurward I am getting a 403 with that URL.

Oops, we only uploaded one version there. All versions on the releases page should be there now.

I don’t believe that it actually has to do with simple-get. Here is a test script that demonstrates the issue:

const https = require('https');
  
const opts = {
  agent: null,
  hostname: 'github.com',
  port: null,
  protocol: 'https:',
  auth: null,
  path:
   '/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz',
  headers: { 'accept-encoding': 'gzip, deflate' }
}

const req =  https.request(opts, (res) => {
 console.log(res.statusCode);
});

req.end();

This script seems to work EXCEPT for running on AWS (regardless of the base image) with Node 10 (other versions of node work fine). We are able to run that script on other environments (local, Digital Ocean, etc.) with any version of Node 8+.

I don’t know if anyone else can confirm or if others are experiencing different results.

Edit:

By work I mean returns 302 rather than 403.

There has been no response from GitHub support so I have sent another message both providing and requesting more details.

Additional evidence of the same problem affecting Electron can be seen at https://twitter.com/mipearson/status/1111533780706062336

My best guess is that (new?) rate limiting by source IP address is being applied to the shared AWS S3 bucket used by GitHub for releases. Perhaps Microsoft want to rate limit what they pay Amazon.

If anyone affected by this is currently paying GitHub or CircleCI for support then I suggest you contact them.

It looks like you’ve all found the SHARP_DIST_BASE_URL environment variable, which was added to workaround exactly this kind of scenario.

And I’m wondering if the install script is correct, it seems that some header files are missing to finish the fallback compiling.