deployer: Phar problems: crc32 mismatch on file

I’m having some issues with deployer connections. When I try to php ./vendor/bin/dep ssh test it fails 50% of the time. This is both on my local machine running Arch as well on my Gitlab server in CI/CD. Php version is 8.0.7 on Arch, 8.0.5 on the alpine based docker image used for CI/CD in Gitlab.

Deployer version: 7.0.0-beta.23
Deployment OS: ubuntu (docker container) / archlinux

In testing I have reduced my deploy.yaml to a minimal setup like this:

config:
  application: APPNAME
  user: USER
  host: HOST
  base_deploy_path: "~/domains"
  repository: "REPONAME"

hosts:
  test:
    remote_user: "{{user}}"
    hostname: "{{host}}"
    deploy_path: "{{base_deploy_path}}/APPLICATION_URL"

This is what I get when the CD fails with this issue:

$ php ./vendor/bin/dep deploy test
In ClassLoader.php line 444:
                                                                               
  include(phar:///builds/APPLICATION_NAME/vendor/deployer/dist/dep/vendor/com  
  poser/../../src/Selector/Selector.php): Failed to open stream: phar error:   
  internal corruption of phar "/builds/APPLICATION_NAME/vendor/deployer/dist/  
  dep" (crc32 mismatch on file "src/Selector/Selector.php")                    
                                                                               
deploy [--tag TAG] [--revision REVISION] [--branch BRANCH] [-o|--option OPTION] [-l|--limit LIMIT] [--no-hooks] [--plan] [--start-from START-FROM] [--log LOG] [--profile PROFILE] [--] [<selector>...]

The application server is running OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017. I could try to have that updated to a newer build as it seems a bit old to me.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 38 (14 by maintainers)

Most upvoted comments

🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀

Schrank on Sep 18, 2021

Released 7.0.0-beta.25. Please test it out! Should be fixed now!

antonmedv on Sep 18, 2021

Yes, thanks a lot. I already know how to fix it. Will write pr today and release it.

antonmedv on Sep 16, 2021

Just chiming in here because I’ve encountered and addressed similar problems for some client project with https://github.com/clue/phar-composer in the past and it looks like this could be related to inherited file descriptors in forked PHP processes. See https://github.com/reactphp/child-process/issues/51 and in particular the linked issues for all the gory details. (Yep, messing with these subtle, low-level details in PHP is what I happen to do for a living and actually spend a lot of time on.)

The gist: Forking is easy – except it’s not.

Whenever you spawn a child process or fork a process on Unix-based platforms, all child processes will inherit all open file descriptors by default. On Unix-based platforms, this can be controlled through the FD_CLOEXEC and O_CLOEXEC flags (neither of which are exposed to PHP).

In itself, this isn’t a big deal for many applications as child processes would usually disregard any inherited file descriptors and simply open new ones themselves. This is however a major problem when using pcntl_fork() and Phar archives in PHP: The forked process will have a reference to the open Phar archive and the same underlying file descriptor. This means that if either the parent or the child process reads from the Phar archive, it will advance the FD position without the other process being aware. This is why one of the processes may read garbled/invalid data and would report a CRC error in this case.

This can be avoided by making sure each child process has a unique reference to the Phar archive, i.e. by using exec() (and family) instead of pcntl_fork() or not using Phar archives at all in combination with pcntl_fork(). Additionally, you may be able to work around this issue by making sure all classes are loaded into memory before forking, as the child process would no longer have to seek in the Phar archive.

I think https://stackoverflow.com/questions/29413013/phar-internal-corruption-crc32-mismatch-during-process-fork does a pretty good job at explaining what’s going on here.

I hope this helps 👍

clue on Sep 16, 2021

Tested using PHP 8.0 & PHP 7.4 on Github Actions (ubuntu-latest) and MacOS Big Sur (latest build): everything works perfectly. Tests on Windows build agents will follow next week, but I will keep silence unless any issue found.

Thank you @antonmedv for quick resolution of this nasty bug! Great tool!

zahhar on Sep 19, 2021

The first attempt was successful. If someone can, check with yourself.

zorca on Sep 1, 2021

https://twitter.com/shyim97/status/1418271370421719041?s=21 I read something in phpunit that they require all classes before executing any code to avoid a bug. Maybe that’s help here too?

Schrank on Jul 22, 2021

Managed to reproduce the same issue down to at least r10, until the version differences got big enough that my deploy.php no longer worked.

I also attempted building the phar using build script from 6.x branch and code from r24, but still got the same errors (I don’t really know much about building phar archives, so perhaps I did something wrong / there’s some internal cache somewhere / etc - so please take this with a grain of salt).

indrek-k on Jul 16, 2021

@Schrank A colleague suggested it could be a race condition in phar loading. Where the crc is checked while it is still loading or multiple processes are loading the phar at the same time and corrupting the unpacked files.

An important piece of information missing on this issue: I’m using the deployer/dist package installed with composer.

I’ve tried with the source build ‘deployer/deployer’ and lo and behold, everything works fine. So this issue seems to be only with the deployer/dist package. And it seems to be related to composer class loading. I’ve found something kind of related maybe: https://stackoverflow.com/questions/29413013/phar-internal-corruption-crc32-mismatch-during-process-fork

coffeeneer on Jul 2, 2021