Ghost: 502 on image upload, but works fine with `ghost run`

1. Download the library https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz

2. Place the tarball in the /home/ghost-mgr/.npm/_libvips/ folder (you can download it in here through wget)

3. Edit ghostFolder/current/node_modules/sharp/package.json file, change the current version to 8.7.4

4. Restart blog, I don’t know if it’s necessary but I did it

This problem still happens in Ghost 3.0.0

This problem still happens in Ghost 3.0.0

I was really hoping this would be fixed in such a major version update. I am facing the same problem

Ghost CLI Version is 1.12.0

Hosted on a Digital Ocean droplet

Hi all! Just to let you know that Ghost 3.9.0 has shipped with sharp 0.24.1. You might need to update with --force if you run into trouble with the binary dependencies.

@daniellockyer Thank you! I had this 502 issue and avoided upgrading Ghost in case it breaks temporary solution that’s described here. I upgraded Ghost (it didn’t require --force) a few minutes ago, and it works now. Seems like image upload problem is no longer with us.

Upgraded from 2.31.0 to 3.2.0 using “ghost upgrade” and having the same issue as everyone else, npm version 6.12.1 and sharp@0.23.4 “ghost doctor” shows no sign of any issue.

I am able to upload SVG files but not PNGs or JPGs.

Everything works if using “ghost run” in lieu of “ghost start”

I’m using a tmux instance as a workaround for the time being. To add on to what @BlueHatbRit said, you can also write tmux new -s name then to detach press ctrl + b and then d, and to re-attach, tmux a -t name

I’ll try to post a backtrace later.

@ErisDS I’ve tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you.

Then I thought maybe its a problem with the libvips library. So I re-downloaded v8.7.4 manually with wget. Then I modified ghostFolder/current/node_modules/sharp/package.json to use library v8.7.4 instead of v8.8.1 and now it works again.

I’m guessing version 8.8.1 has a problem with CentOS 7.x since its working for you.

What OS is @everyoneElse using?

I fixed this problem as described by @andreborud and it works now with ghost 3.0.2 on DIgital Ocean. Droplet runs on the ubuntu Ghost package from their market.

ok to summarize, it seems the problem is being cause by the newer libvips in the sharp package not working correctly on some OS mainly shown in the thread to be CentOS and DigitalOcean’s Ubuntu image

So I had the same issue running ghost from alpine docker image https://hub.docker.com/_/ghost/, when I tried to upload a picture my nginx showed 502 and the container just died with no log or anything.

So I tried to do the gdb debug as @lovell proposed just with the change to attach to already running process. When I tried to do that it showed ptrace: Operation not permitted quick google search reveals this https://stackoverflow.com/a/46676868 that the docker restrictions prevent GDB to work properly and that I have to supply --cap-add=SYS_PTRACE --security-opt seccomp=unconfined in order to do debug. And weirdly the issue disappeard. After some more testing it was the seccomp=undefined option which fixed the issue.

Not sure if it helps the debug on non-dockerized environment.

I tried to do gdb from the host machine but that didn’t yield much info because of docker. But strace did produce some usable output https://gist.github.com/BrnoPCmaniak/66068fbbcacff1a748b98fb59dcad826 the last thing it did was that it loaded sharp’s libharfbuzz.so.0 and then it crashed.

TL;DR

I run ghost in docker:alpine after 3.0 version this issue arises and --security-opt seccomp=unconfined fixed it.

@jagracey can you confirm if sudo -u ghost NODE_ENV=production ghost run works?

I have a Digital Ocean droplet that I recently upgraded to 3.0.3 with this same issue, and I can confirm that the workaround posted by @johnpoelman worked for me. I can now upload images again.

Hi, I’m the sharp maintainer, is this problem only occurring with Digital Ocean VMs / “droplets”? Please can someone provide full details of which type/size/location this occurs with, including the output of cat /proc/cpuinfo.

If anyone is able to use gdb node to get a backtrace of the crash that would be amazing. Here’s an example of what to type (run this from the directory that contains node_modules):

$ gdb node
GNU gdb (Ubuntu 8.2.91.20190405-0ubuntu3) 8.2.91.20190405-git
Copyright (C) 2019 Free Software Foundation, Inc.
...

(gdb) run -e "require('sharp')"

Starting program: /usr/bin/node -e "require('sharp')"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
...
[Detaching after fork from child process 15193]
[Inferior 1 (process 15183) exited normally]

In this example there is no crash, but if there was you can type backtrace to get full details.

Rolling back to 4.3.3 Alpine container fixed it for me. Must be something to do with the latest version

Edit: It looks like it did work. However, I had to specify sharp v0.24, e.g. npm install sharp@0.24. The default npm install sharp installed v0.23.4

Having the same issue. Image uploads work fine under “ghost run”, but fail when ghost is started normally.

I tried removing sharp from my current version (3.7.0), and then reinstalling via npm install sharp per the above instructions, but no luck. I actually had a few hours of screwing things up, force re-installing the current version, having dependency issues, fixing things, breaking them again. I can’t begin to say exactly what I did, but none of it worked anyway.

Running Ubuntu 18.04.4 LTS in AWS. I’ll be happy to provide any other information that might be useful, or try anything else that someone wants me to for troubleshooting. Just let me know what commands to run.

@darrenmhill regardless of the importance of this issue, it’s quite difficult to fix as we’re not able to create a RCA with the updated version of libvps. In the meantime, many people have mentioned that migrating to a new droplet or downgrading libvps and upgrading sharp in your installation is a valid workaround.

I’ve tried every fix here, including downgrading libvips version used by sharp and installing a brand new ghost instance and copying over content and although it solves the 502 error on upload, responsive images are no longer generated. Respective folders and images are not created inside content/images/size/. Has anyone managed to fix this part of the issue?

Running ghost 3.4.0 on AWS EC2. I had the problem since ghost ver 2. Today I checked my nodeJs version. It was version 8. After updating to the latest version 10 everything seems to work again.

The problem with running NODE_ENV=production ghost run as ghost-mgr is that it’ll only work so long as you have a terminal open. The moment you close your terminal ghost shuts down.

Surely there must be a better image uploading package that doesn’t have all of these issues that can be used especially because the main way suggested for implementing Ghost on an outside server (via the pre-built Digital Ocean droplet) is the one that is broken.

This is a huge issue for a lot of us and desperately needs to be fixed.

Can confirm: - migrating v2 to v3 fails, NODE_ENV=production ghost run* is a temp fix (blogs=3) - fresh v3 install works fine (blogs=2)

We’re running on small Digital Ocean droplets, using the marketplace Ghost app.

sudo -i -u ghost-mgr
cd /var/www/ghost
NODE_ENV=production ghost run

Nope it does not:

$ node -e "console.log(require('sharp').versions)"
{
  cairo: '1.17.2',
  croco: '0.6.13',
  exif: '0.6.21',
  expat: '2.2.7',
  ffi: '3.2.1',
  fontconfig: '2.13.91',
  freetype: '2.10.1',
  fribidi: '1.0.5',
  gdkpixbuf: '2.36.12',
  gettext: '0.20.1',
  gif: '5.1.4',
  glib: '2.56.4',
  gsf: '1.14.46',
  harfbuzz: '2.5.3',
  jpeg: '2.0.2',
  lcms: '2.9',
  orc: '0.4.28',
  pango: '1.42.4',
  pixman: '0.38.4',
  png: '1.6.37',
  svg: '2.45.5',
  tiff: '4.0.10',
  vips: '8.8.1',
  webp: '1.0.3',
  xml: '2.9.9',
  zlib: '1.2.11'
}

Don’t mean to just add another +1 here, but I’ve got two blogs running affected by this. Both are on a (somewhat dated, but kept up to date via regular apt-get upgrades) ghost DO droplet.

Ghost-CLI version: 1.12.0
Ghost version: 3.0.0 (at /var/www/ghost)
Linux personal-blog 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux`

I’ve run through these instructions from above and oddly enough, the 502s persist despite getting nearly identical output. Initially, it had been using the cached version from ghost-mgr’s npm cache, then after deleting it, it downloaded a new one from https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz.

Happy to try and share the results of any other ideas anyone has. 😃

**Edit: ** Downgrading libvips to 8.7.4 in sharp’s package.json seemed to do the trick, as here. Guess I’ll just keep doing that with every release for a while. 🤷‍♂

FWIW I keep trying to reproduce this and I can’t! I do however have a potentially useful debug process - it’s tricky, but here we go:

Step 1. Modify ghost-cli to be super verbose:

which ghost will tell you roughly where ghost-cli is installed

E.g. on a DO droplet it should be /usr/local/bin/ghost.

We want to edit the file lib/tasks/yarn-install.js. The path to the actual files will be up 2 levels from which ghost, in lib/node_modules/ghost-cli.

So, if your which ghost gives you /usr/local/bin/ghost then you need to edit the file /usr/local/lib/node_modules/ghost-cli/lib/tasks/yarn-install.js.

Change line 65 from:

const observable = yarn(['install', '--no-emoji', '--no-progress'], {

to:

const observable = yarn(['install', '--no-emoji', '--no-progress', '--verbose'], {

Step 2. Rerun update

First figure out what version of Ghost you are currently on with ghost version. Then update to that exact version again, with the force and verbose flags:

e.g. ghost version -> 2.37.0

Then run ghost update v2.37.0 --force --verbose.

This is going to give TONNES of output from yarn, but thankfully the interesting bit is at the end.

You should see some output like:

node-pre-gyp info install unpacking node-v64-linux-x64/node_sqlite3.node
node-pre-gyp info tarball done parsing tarball
[sqlite3] Success: "/var/www/ghost/versions/2.37.0/node_modules/sqlite3/lib/binding/node-v64-linux-x64/node_sqlite3.node" is installed via remote
node-pre-gyp info ok
[16:06:22] → verbose 27.014 info sharp Using cached /home/ghost-mgr/.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz
[16:06:23] → Done in 26.92s.
[16:06:23] Installing dependencies [completed]

See that line in the middle with verbose 27.014 info sharp? That’s the bit we’re looking for - some output from sharp.

Please follow these steps if you can and share any sharp-related output at the end of the yarn output.

---

Additional info: The output from sharp comes from this file, in my test case, this line which says it’s reusing cached sharp.

I did these steps, then I deleted the path that it mentioned:

rm /home/ghost-mgr/.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz

Re-ran the force upgrade, and this time I got a different line:

verbose 28.056 info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz

Image uploads continue to work fine.

Oh P.S. @ekerstein can you also share what OS and version are you running on & what version of node and npm you are on? Is it still node 10.16?

@ekerstein Your debug info is very helpful but also baffling!

Is there anything in your journal or syslog at all?

Given that the second log doesn’t even get output, that means the require is dying and causing the process to crash without an exception and before the log even gets sent to stdout - there’s a try-catch block there on purpose, it’s meant to catch any problems with requiring sharp, but clearly it isn’t able to.

This points at a problem with the install of sharp’s c-libraries.

Could you possibly change directory into /versions/[latest] and try 1) deleting the sharp directory, and then running npm install sharp just to see if there’s any output?

I’m sorry to make you jump, but I’ve not been able to reproduce this issue. It’s happening to many people so it’s definitely an issue, but without reproduction we don’t have a hope in hell at fixing it.

It’s also almost certainly a problem with the sharp install process, but we don’t even yet understand it enough to report it.

---

To anyone else - please don’t just comment here saying “me too”. We know it’s a problem.

If you want to help, share your full system information (os, node version, npm version, node install method, ghost-cli version or ghost install method) and if possible ghost & system logs.

@andreborud Honestly, I haven’t tried against since rolling back to 2.30.2. So I just tested it again now:

• Checked my nginx configuration and it was already at 50M. Also, the file I uploaded when I got a 502 error was only 1-2mb. So unless the config file is in another folder, I’m not sure that’s the issue.
• The nginx config also doesn’t explain why rolling back to a previous version would make it work again.

I came across this issue related to the config server.port being 2369 while nginx was on 2368. I realized I had the same issue! So I did ghost config set server.port 2368, restarted, and updated to the latest version. My site loaded fine but when I went to try and access my posts it gave me a 500 error. I did ghost run and it gave me this:

InternalServerError: Cannot read property '0' of undefined
    at new GhostError (/var/www/ghost/versions/2.32.0/core/server/lib/common/errors.js:10:26)
    at _private.prepareError (/var/www/ghost/versions/2.32.0/core/server/web/shared/middlewares/error-handler.js:51:19)
    at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:71:5)
    at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
    at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:67:12)
    at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:635:15
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:260:14)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/route.js:127:14)
    at apiImpl.then.then.catch (/var/www/ghost/versions/2.32.0/core/server/api/shared/http.js:97:17)
    at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:93:12)

TypeError: Cannot read property '0' of undefined
    at forEach (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relative-to-absolute.js:22:43)
    at Array.forEach (<anonymous>)
    at Object.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relativ                        e-to-absolute.js:21:39)
    at UrlUtils.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/index.js:387:22)
    at Object.forPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/url.js:34:36)
    at Object.mapPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/mapper.js:35:9)
    at frame.response.posts.models.data.map.model (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/post                        s.js:15:56)
    at Array.map (<anonymous>)
    at Object.all (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/posts.js:15:36)
    at serializeOptionsShared (/var/www/ghost/versions/2.32.0/core/server/api/shared/serializers/handle.js:107:58)
    at /var/www/ghost/versions/2.32.0/core/server/lib/promise/sequence.js:10:31
    at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Object.gotValue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:168:18)
    at Object.gotAccum (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:155:25)
    at Object.tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromiseCtx (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:641:10)
    at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:97:12)
    at _drainQueue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:15:14)

I rolled back to 2.30.2 again and everything is working. So now I’m not sure if the issue is related to the port 2368 that I changed or something else.

As you got it working again by downgrading I’m not sure if nginx is the problem. On my droplet running CentOS I had to change the max size of any file being uploaded in the base config of nginx located in /etc/nginx/nginx.conf. I don’t remember the default size 2 or 5mb.

http {
    ...
    client_max_body_size 50M; <- change this line
    ...
}