argo-cd: Application sync hang with "unable to fork off sideband demultiplexer"

After having the same problem, it would come from the version of docker/containerd which is deployed.

The new GLIBC library in 22.04 uses CLONE3 for syscal which is not supported by SECCOMP profile older versions of Docker.

Workaround:

• Use base image 20.04

Recommendation :

• Update docker/containerd

With this minimal Dockerfile I’ve created a somehow reproducible environment:

ARG VERSION=21.04
FROM ubuntu:$VERSION
COPY ./old-ubuntu-fix.sh /usr/bin/old-ubuntu-fix.sh
RUN chmod 755 /usr/bin/old-ubuntu-fix.sh && old-ubuntu-fix.sh
RUN apt update && apt install -y git && rm -rf /var/cache/apt/
RUN useradd user
USER 1000

old-ubuntu-fix.sh:

#!/bin/bash
set -x
UBUNTU_VERSION=$(cat /etc/lsb-release  | grep DISTRIB_RELEASE | cut -d = -f 2)

if [ "$UBUNTU_VERSION" == "21.04" ]; then
  sed -i -E 's/http:\/\/releases.ubuntu.com\/ubuntu/http:\/\/old-releases.ubuntu.com\/ubuntu/g' /etc/apt/sources.list
  sed -i -E 's/http:\/\/archive.ubuntu.com\/ubuntu/http:\/\/old-releases.ubuntu.com\/ubuntu/g' /etc/apt/sources.list
  sed -i -E 's/http:\/\/security.ubuntu.com\/ubuntu/http:\/\/old-releases.ubuntu.com\/ubuntu/g' /etc/apt/sources.list
fi

Using as VERSIONs:

• 21.04
• 22.04

21.04

$ git clone ssh://...
...
Resolving deltas: 100% (348/348), done.
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=21.04
DISTRIB_CODENAME=hirsute
DISTRIB_DESCRIPTION="Ubuntu 21.04"
$ git --version
git version 2.30.2
$ ldd $(which git)
	linux-vdso.so.1 (0x00007ffe6ecf8000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007fd41fcde000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd41fcc2000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd41fca0000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd41fab4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd4200f3000)

22.04

$ git --version
git version 2.34.1
$ git clone ssh://...
Cloning into 'gitops'...
error: cannot create async thread: Operation not permitted
fatal: fetch-pack: unable to fork off sideband demultiplexer

$ ldd $(which git)
	linux-vdso.so.1 (0x00007fffd3bdf000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f9866c42000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9866c26000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f98669fe000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f986707f000)

$ dpkg-query -l git
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version             Architecture Description
+++-==============-===================-============-===========================>
ii  git            1:2.34.1-1ubuntu1.4 amd64        fast, scalable, distributed

This effectively confirms that:

• The issue happens with a Docker Image based on Ubuntu 22.04 (when running on a Ubuntu 16.04.7 LTS / VMware Tanzu host?)
• The git binary without libpthread causes the issue
• The git version doesn’t really matter (bitnami’s git worked well there, but libpthread was present on the image I’ve tried)
• It seems to be an issue caused (or visible only) by missing linked library.

Now the question is:

• Is it really an issue with VMware Tanzu as an host? Can someone test this somewhere else?
• Where should we file this bug?
• Is this something that’s already known to the Ubuntu / VMware community?

@nice-pink: I did not get the meaning of this part of your post:

This error is only shown for apps running in the same cluster and with code from the same git repo.

I use the same Argo CD instance to manage multiple clusters. There is one repository containing the manifests for each cluster. Only the apps which are running in the same cluster than the Argo CD instance with their manifests saved in the same repository as the ones from Argo CD are showing the error “unable to fork off sideband demultiplexer”. All other apps are synced properly.