containerized-data-importer: CDI import from URL is significantly slower than a manual wget + virtctl image-upload

@cloudymax Just released 1.57.0 which should not require the ‘cert’ hack shown above. It should be similar speed.

nbdkit changes are upstream in 1.35.8, mainly this commit: https://gitlab.com/nbdkit/nbdkit/-/commit/a74b289ee15a7c75dceb8a96403f1fa8ebd72e88

I did some tests with the URLs under discussion and we are better than wget for the Debian and Ubuntu cases. The Windows URL is weird, more below.

https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img (HTTP/1.1)

nbdkit before change: 180s nbdkit after change, connections=16: 42.4s nbdkit after change, connections=32: 21.6s wget: 20.2s nbdkit after change, connections=64: 17.2s

https://gemmei.ftp.acc.umu.se/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 (HTTP/1.1)

nbdkit before change: 100.6s nbdkit after change, connections=16: 23.7s wget: 20.9s nbdkit after change, connections=32: 14.0s nbdkit after change, connections=64: 9.1s

https://www.itechtics.com/?dl_id=173 This URL redirects twice, one to windowstan.com which is proxied by cloudflare, and a second time to software.download.prss.microsoft.com. Unfortunately the second redirect is not an actual HTTP redirect, but a message in the body of the 502 error message that cloudflare frequently sends if you attempt any automated download. So I did these tests using the microsoft.com URL directly. The microsoft site uses HTTP/2.

nbdkit before change if you use the windowstan.com URL: over 80 minutes, and very unreliable nbdkit before change if you use the microsoft.com URL: 3m03 nbdkit after change, connections=1: 3m24 nbdkit after change, connections=2: 3m05 nbdkit after change, connections=16: 3m04 wget: 1m04

Some tips:

• “Pre-resolving” the URLs helps greatly.
• If IPv6 is broken, use ipresolve=v4 to force IPv4 which will save a bit of time.
• connections parameter controls the number of HTTP connections to the remote server. Reducing will put less load on / more friendly to the server, increasing will make things faster. Default is 16.
• HTTP/2 is way more efficient than HTTP/1.1. We will now use HTTP/2 multiplexing, usually up to 100 streams per HTTP connection. curl ought to default to HTTP/2 if available automatically.

Oh and we found and fixed an actual bug in curl: https://github.com/curl/curl/pull/11557

I reimplemented the plugin using curl “multi” handle support, and the results are way better: https://listman.redhat.com/archives/libguestfs/2023-July/032167.html (see patch 2 for some test timings)

Quick question: Does the container environment have broken IPv6? The Ubuntu URL resolves to IPv6 then IPv4, and curl tries the IPv6 addresses first. In the Beaker test environment IPv6 is broken and this causes a delay. I added a feature upstream to force IPv4 which improves performance.

Anyway, I’m on holiday today, will take another look at this tomorrow.

I eliminated nbdkit from the equation and I am running the go http client to directly get the data from inside the pod, and it was still slow until I increased the amount of memory available to the pod, so I used this yaml to create the DataVolume

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: jammy-pvc
  annotations:
    cdi.kubevirt.io/storage.bind.immediate.requested: "true"
spec:
  source:
    http:
      # certConfigMap with an empty cert is to work around this issue:
      # https://github.com/kubevirt/containerized-data-importer/issues/2358
      certConfigMap: empty-cert
      url: https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 5Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: empty-cert
data:
  ca.pem: ""

And I increased the amount of memory allocated to the CDI importer pod by modifying the CDI CR to look like this

apiVersion: cdi.kubevirt.io/v1beta1
kind: CDI
metadata:
  name: cdi
spec:
  config:
    featureGates:
    - HonorWaitForFirstConsumer
    podResourceRequirements:
      limits:
        cpu: "4"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 250Mi
  imagePullPolicy: IfNotPresent
  infra:
    nodeSelector:
      kubernetes.io/os: linux
    tolerations:
    - key: CriticalAddonsOnly
      operator: Exists
  workload:
    nodeSelector:
      kubernetes.io/os: linux

In particular note the podResourceRequirements which increased the amount of memory and cpu available. I went a little overboard, it doesn’t require that much, but apparently the defaults are just not enough. This allowed me to import using the http go client into the scratch space, and convert from there. It took maybe a minute if that on my local machine which a 1G fiber connection. The emptyCert is to trick CDI into thinking it cannot use nbdkit and fall back to the scratch space method which is a straight http download and then convert.

CDI runs inside a container on the host. So I just double checked, it looks like 1.57 is running 1.34 nbdkit, and 1.56 is running 1.30. And I think we need to add a plugin to 1.34 that is not needed in 1.30.

@cloudymax could you try 1.56 instead of 1.57-rc1 to see if it works better?

nbdkit is even faster this morning …

$ time ./nbdkit -r -U - curl https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img --run 'nbdcopy -p $uri null:'
 █ 100% [****************************************]
real	1m10.973s
user	0m1.613s
sys	0m1.976s

I should note that I’m running a slightly modified nbdkit. The curl plugin uses a parallel thread model which seems to make it a bit faster. And I gave in and wrote a qcow2 filter.

The two issues are overlapping but slightly distinct. In BZ 2013000 / #1981, the problem was with Fedora redirects being flaky, sometimes resolving to a 404 mirror. We decided in the end to solve that by using the nbdkit-retry-request-filter which basically retries single requests until we get a good mirror. In that case pre-resolving the URL was thought to be bad (because we might pre-resolve it to a 404, and then it wouldn’t work at all) and so #1981 was not merged. In this case the Debian mirrors all appear to be good (ie they don’t resolve to 404s that I’ve seen), but it’s the issue that we’re doing the resolves on every block (curl internally doesn’t cache them) and they happen to be especially slow on the central Debian server, so pre-resolving would be a good thing.

I’ve finally worked out what’s going on here. About 3/4 of the time is spent doing 302 redirects from the orignal URL located on cloud.debian.org to the mirror (which for me happens to be saimei.ftp.acc.umu.se, but I guess that depends where you are located in the world).

The way curl works is it doesn’t cache 302 redirects, so basically on every single block that we request we’re doing a whole new redirect.

If you “manually” resolve the redirect first then for me it is ~ 4 times faster, and near the speed of plain wget.

It turns out we’ve actually hit this bug before in a slightly different context:

https://bugzilla.redhat.com/show_bug.cgi?id=2013000

I proposed a patch for that bug, adding an effective-url=true flag which would do the pre-resolving once inside the curl plugin. There were various problems with that patch which you can read about here:

https://listman.redhat.com/archives/libguestfs/2021-October/026882.html

I wonder if CDI can do a simple HEAD request to the site that is submitted in order to resolve redirects, and then use the resolved URI when opening nbdkit? It should be fairly trivial if CDI has access to the curl library. I can suggest some code for this if you like. It would also be useful to give better errors for things like 404s.

The fundamental problem is still qemu-img convert lacking support for NBD multi-conn which is not something we can fix in nbdkit.

But anyway a while back I had this patch to nbdkit to enable parallel curl plugin. It had some unfortunate effects because of fairness which I describe in the thread there. I looked at it again and I think we should give it another go, so I pushed this commit:

https://gitlab.com/nbdkit/nbdkit/-/commit/f2163754860d041c4cb12dace90591c280eccae8

With this patch and upping the default number of connections (connections=8) it’s goes from 20 mins to about 4 mins, which while still not great (see above about qemu) is probably the best we can do until qemu gets fixed.

Hey! Thanks for reporting this

This sounds similar to previous issues we had like https://github.com/kubevirt/containerized-data-importer/issues/2358, but the changes that tackle those did not get backported to your CDI version (v1.54.3). The author of the PR that fixes this (https://github.com/kubevirt/containerized-data-importer/pull/2701) mentions:

On my machine this reduces a Ubuntu cloud image import from 90 minutes to 1 minute.

If you want to try that change, you can install CDI 1.57 RC