podman: `podman run --detach` doesn't start the process, not known to `podman inspect`

/kind bug

Description

During OKD bootstrapping on Fedora CoreOS 31 etcd-signer container is started in a detached mode:

podman run --quiet --net=host \
	--name etcd-signer \
	--detach \
	--volume /opt/openshift/tls:/opt/openshift/tls:ro,z \
	"<etcd-signer pullspec>" \
	serve \
	--cacrt=/opt/openshift/tls/etcd-signer.crt \
...

Occasionally (~60% of all runs) this container is reported as started, but podman logs -f <id> immediately exists and podman inspect <id> returns

Error: error inspecting container 62a09bea7faf0fd723d3018edd6715ebee517f90a22517f7a97078f8b5c8d6c7: error getting container from store "62a09bea7faf0fd723d3018edd6715ebee517f90a22517f7a97078f8b5c8d6c7": container not known

Other podman runs are not using detached mode and work as expected.

Output of podman version:

Version:            1.7.0
RemoteAPI Version:  1
Go Version:         go1.13.5
OS/Arch:            linux/amd64

Output of podman info --debug:

Details

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.5
  podman version: 1.7.0
host:
  BuildahVersion: 1.12.0
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.10-2.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
  Distribution:
    distribution: fedora
    version: "31"
  MemFree: 6243008512
  MemTotal: 8340058112
  OCIRuntime:
    name: crun
    package: crun-0.10.6-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.10.6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: ip-10-0-13-190
  kernel: 5.4.15-200.fc31.x86_64
  os: linux
  rootless: false
  uptime: 1m 35.57s
registries:
  registry.svc.ci.openshift.org/origin/4.4-2020-02-06-060436:
    Blocked: false
    Insecure: false
    Location: registry.svc.ci.openshift.org/origin/4.4-2020-02-06-060436
    MirrorByDigestOnly: true
    Mirrors:
    - Insecure: false
      Location: quay.io/vrutkovs/okd-release
    Prefix: registry.svc.ci.openshift.org/origin/4.4-2020-02-06-060436
  registry.svc.ci.openshift.org/origin/release:
    Blocked: false
    Insecure: false
    Location: registry.svc.ci.openshift.org/origin/release
    MirrorByDigestOnly: true
    Mirrors:
    - Insecure: false
      Location: quay.io/vrutkovs/okd-release
    Prefix: registry.svc.ci.openshift.org/origin/release
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 15
  GraphDriverName: overlay
  GraphOptions:
    overlay.mountopt: nodev,metacopy=on
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 14
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman 1.7.0-2.fc31.x86_64

List of other installed packages: https://github.com/coreos/fedora-coreos-config/blob/90b01c127c5cdba89543cdd62612bb5e4a249ab2/manifest-lock.x86_64.json#L865

Additional environment details (AWS, VirtualBox, physical, etc.):

Happens on AWS / vSphere / baremetal machines occasionally

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (8 by maintainers)

Most upvoted comments

I’ll look at this either tomorrow or next week 😄

haircommander on Feb 6, 2020