cri-o: Fedora 32 / cri-o 1.18 ... service fails to start

I have a fresh install of Fedora 32 and cri-o 1.18. crio fails to start at systemctl enable --now crio

Steps to reproduce the issue:

  1. Install Fedora 32
  2. Install crio per https://cri-o.io/
  3. systemctl enable --now crio

Describe the results you received:

systemctl status crio --no-pager -l

● crio.service - Container Runtime Interface for OCI (CRI-O) Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2020-04-30 16:31:17 PDT; 12min ago Docs: https://github.com/cri-o/cri-o Process: 922 ExecStart=/usr/bin/crio $CRIO_CONFIG_OPTIONS $CRIO_RUNTIME_OPTIONS $CRIO_STORAGE_OPTIONS $CRIO_NETWORK_OPTIONS $CRIO_METRICS_OPTIONS (code=exited, status=1/FAILURE) Main PID: 922 (code=exited, status=1/FAILURE) CPU: 208ms

Apr 30 16:31:17 cube0 crio[922]: time=“2020-04-30 16:31:17.508525817-07:00” level=info msg=“AppArmor is disabled by the system or at CRI-O build-time” Apr 30 16:31:17 cube0 crio[922]: time=“2020-04-30 16:31:17.522845028-07:00” level=info msg=“Found CNI network crio (type=bridge) at /etc/cni/net.d/100-crio-bridge.conf” Apr 30 16:31:17 cube0 crio[922]: time=“2020-04-30 16:31:17.533307816-07:00” level=info msg=“Found CNI network 200-loopback.conf (type=loopback) at /etc/cni/net.d/200-loopback.conf” Apr 30 16:31:17 cube0 crio[922]: time=“2020-04-30 16:31:17.576317230-07:00” level=info msg=“Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist” Apr 30 16:31:17 cube0 crio[922]: time=“2020-04-30 16:31:17.576341624-07:00” level=info msg=“Update default CNI network name to crio” Apr 30 16:31:17 cube0 crio[922]: time=“2020-04-30 16:31:17.793353021-07:00” level=warning msg=“Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled” Apr 30 16:31:17 cube0 crio[922]: time=“2020-04-30 16:31:17.796965936-07:00” level=fatal msg=“Version string empty” Apr 30 16:31:17 cube0 systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE Apr 30 16:31:17 cube0 systemd[1]: crio.service: Failed with result ‘exit-code’. Apr 30 16:31:17 cube0 systemd[1]: Failed to start Container Runtime Interface for OCI (CRI-O).

Describe the results you expected: ● crio.service - Container Runtime Interface for OCI (CRI-O) Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled) Active: active (running) since fooor evvv aaaah

Additional information you deem important (e.g. issue happens only occasionally): I tried setting selinux to permissive. systemctl restart yields same failure.

Output of crio --version:

[root@cube0 ~]# crio --version
crio version
Version:
GitCommit:
GitTreeState:
BuildDate:
GoVersion:     go1.14
Compiler:      gc
Platform:      linux/amd64
Linkmode:      unknown: `ldd crio` failed: ldd: ./crio: No such file or directory
  (exit status 1)
[root@cube0 ~]# which crio
/usr/bin/crio
[root@cube0 ~]# ldd /usr/bin/crio
	linux-vdso.so.1 (0x00007fffbc79d000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1149e9c000)
	libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007f1149e4e000)
	libgpgme.so.11 => /lib64/libgpgme.so.11 (0x00007f1149dfc000)
	libassuan.so.0 => /lib64/libassuan.so.0 (0x00007f1149de6000)
	libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f1149dc2000)
	libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x00007f1149d64000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f1149d57000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f1149b8d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f114d9f6000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f1149b60000)
	libudev.so.1 => /lib64/libudev.so.1 (0x00007f1149b36000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f11499f0000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f1149957000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f114994e000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1149933000)

Additional environment details (AWS, VirtualBox, physical, etc.): physical … HP EliteDesk Mini G2 (Intel Core i5 / 16 gig mem)

Bugzilla is here

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 19 (11 by maintainers)

Most upvoted comments

Hi,

Do we know why we were failing here? --log-level debug and strace didn’t help pinpoint the problem.

Yes, the version string was empty, from the logs:

Apr 30 16:31:17 cube0 crio[922]: time="2020-04-30
16:31:17.796965936-07:00" level=fatal 
msg="Version string empty"

During the build CRI-O uses -X in the Makefile to set some values that were not shipping inside the RPM. Here the full patch inside Fedora packaging:

$ git show 6ee7c76731cbb141d9929ac0a0952dd85b7f8bb2
commit 6ee7c76731cbb141d9929ac0a0952dd85b7f8bb2
Author: Douglas Schilling Landgraf <dougsland@redhat.com>
Date:   Wed Apr 29 10:25:22 2020 -0400

    build: include version data during the build
    
    Resolves: github.com/cri-o/cri-o/issues/3684

diff --git a/cri-o.spec b/cri-o.spec
index 62d5156..eb3372e 100644
--- a/cri-o.spec
+++ b/cri-o.spec
@@ -16,16 +16,26 @@
 %define gobuild(o:) GO111MODULE=off go build -buildmode pie -compiler gc -tags="rpm_crashtraceback ${BUILDTAGS:-}" -ldflags "${LDFLAGS:-} -B 0x$(head -c20 /dev/urandom|od -An -tx1|tr -d ' \\n') -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '" -a -v -x %{?**}; 
 %endif
 
+# Global vars
 %global provider github
 %global provider_tld com
 %global project cri-o
 %global repo cri-o
+
+# Related: github.com/cri-o/cri-o/issues/3684
+%global build_timestamp %(date -u +'%Y-%m-%dT%H:%M:%SZ')
+%global git_tree_state clean
+%global criocli_path ""
+
 # https://github.com/cri-o/cri-o
 %global import_path %{provider}.%{provider_tld}/%{project}/%{repo}
+
+# Commit for the builds
 %global commit0 7d79f42b28ad00cf2e7d86604a5a4007303ac328
 %global shortcommit0 %(c=%{commit0}; echo ${c:0:7})
 %global git0 https://%{import_path}
 
+# Services
 %global service_name crio
 
 # Used for comparing with latest upstream tag
@@ -100,6 +110,15 @@ ln -s vendor src
 export GOPATH=$(pwd)/_output:$(pwd)
 export BUILDTAGS="$(hack/btrfs_installed_tag.sh) $(hack/btrfs_tag.sh) $(hack/libdm_installed.sh) $(hack/libdm_no_deferred_remove_tag.sh) $(hack/seccomp_tag.sh) $(hack/selinux_tag.sh)"
 export GO111MODULE=off
+
+# Related: github.com/cri-o/cri-o/issues/3684
+export LDFLAGS="-X %{import_path}/internal/pkg/criocli.DefaultsPath=%{criocli_path}
+-X  %{import_path}/internal/version.buildDate=%{build_timestamp}
+-X  %{import_path}/internal/version.gitCommit=%{commit0}
+-X  %{import_path}/internal/version.version=%{version}
+-X  %{import_path}/internal/version.gitTreeState=%{git_tree_state}"
+
 %gobuild -o bin/%{service_name} %{import_path}/cmd/%{service_name}
 %gobuild -o bin/%{service_name}-status %{import_path}/cmd/%{service_name}-status
 

Boring is good. I always say CRI-O should be like a file system, it just does it’s job, and I should seldom need to think about it.

@mazzystr @harche @haircommander the latest bits are in the stable channel. Please consider closing this ticket. Thanks!

# cat /etc/fedora-release 
Fedora release 32 (Thirty Two)

# dnf module info cri-o:1.18
Last metadata expiration check: 0:34:42 ago on Fri 08 May 2020 08:44:59 AM EDT.
Name             : cri-o
Stream           : 1.18 [e] [a]
Version          : 3220200429145402
Context          : 43bbeeef
Architecture     : x86_64
Profiles         : default
Default profiles : 
Repo             : updates-modular
Summary          : Kubernetes Container Runtime Interface for OCI-based containers
Description      : CRI-O is the Kubernetes Container Runtime Interface for OCI-based containers.
Requires         : platform:[f32]
Artifacts        : cri-o-2:1.18.0-2.module_f32+8778+fdcfac52.src
                 : cri-o-2:1.18.0-2.module_f32+8778+fdcfac52.x86_64
                 : cri-o-debuginfo-2:1.18.0-2.module_f32+8778+fdcfac52.x86_64
                 : cri-o-debugsource-2:1.18.0-2.module_f32+8778+fdcfac52.x86_64