buildah: `buildah images` very slow

Description

My containers store has a lot of large images, totaling ~40-50 tagged images & ~1TB size. For some reason, this causes buildah images (as well as podman image ls) to run very slowly:

> time buildah images
... output ...
real	1m33.037s
user	2m23.364s
sys	0m4.053s

I’m not sure what the exact cause is, but I think it is abnormal/unusual for performance of merely listing images to deteriorate at sizes as small as ~50 images and ~1TB. Perhaps I’m doing something wrong?

Steps to reproduce the issue:

  1. Have a lot of images, maybe?
  2. time buildah images or time podman image ls

Describe the results you received: Listing images takes ~2minutes.

> time buildah images
... output ...
real	1m33.037s
user	2m23.364s
sys	0m4.053s

(Perhaps relatedly, podman system df is also kind of slow, but much less so–only takes about 15s.)

Describe the results you expected: Listing images should happen approximately instantly.

Output of ~rpm -q buildah or apt list buildah~ xbps-query buildah:

architecture: x86_64
changelog: https://github.com/containers/buildah/blob/master/CHANGELOG.md
filename-sha256: 25806183e98deae0653cf8c2faa61e61426343e5cb50fb3ca6f3088e61d2d6a3
filename-size: 13MB
homepage: https://github.com/containers/buildah
installed_size: 29MB
license: Apache-2.0
maintainer: Cameron Nemo <cnemo@tutanota.com>
pkgname: buildah
pkgver: buildah-1.20.1_1
repository: https://alpha.us.repo.voidlinux.org/current
run_depends:
	runc>=0
	containers.image>=0
	glibc>=2.32_1
	libgpgme>=1.12.0_2
	libassuan>=2.0.1_1
	libgpg-error>=1.6_1
	device-mapper>=2.02.110_1
shlib-requires:
	libpthread.so.0
	libgpgme.so.11
	libassuan.so.0
	libgpg-error.so.0
	libdl.so.2
	libdevmapper.so.1.02
	libc.so.6
short_desc: Dockerfile compatible OCI image building tool
source-revisions: buildah:b177108fa1

Output of buildah version:

Version:         1.20.1
Go Version:      go1.16.3
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:
image Version:   5.10.5
Git Commit:
Built:           Wed Dec 31 16:00:00 1969
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Version:      3.2.1
API Version:  3.2.1
Go Version:   go1.16.5
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of cat /etc/*release:

NAME="void"
ID="void"
DISTRIB_ID="void"
PRETTY_NAME="void"

Output of uname -a:

Linux vic 5.12.14_1 #1 SMP 1625102948 x86_64 GNU/Linux

Output of ~cat /etc/containers/storage.conf~ no conf, using defaults–here’s buildah info:

{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "\"void\"",
            "version": "unknown"
        },
        "MemFree": 58960666624,
        "MemTotal": 67415769088,
        "OCIRuntime": "runc",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 12,
        "hostname": "vic",
        "kernel": "5.12.14_1",
        "os": "linux",
        "rootless": true,
        "uptime": "27m 38.37s"
    },
    "store": {
        "ContainerStore": {
            "number": 1
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.mount_program=/usr/bin/fuse-overlayfs"
        ],
        "GraphRoot": "/home/kshi/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 176
        },
        "RunRoot": "/var/tmp/containers-user-1000/containers"
    }
}

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 24 (20 by maintainers)

Most upvoted comments

I found the cause (Podman issue https://github.com/containers/podman/issues/11997) and am working on a fix.