kamaji: Race condition in container image CI when the commit has multiple tags

Hi,

we created an additional datastore of type etcd using your helm chart clastix/kamaji-etcd. We then tried to migrate a cluster from the old to the new datastore (both of type etcd), using this tutorial: https://kamaji.clastix.io/guides/datastore-migration/

This did not work as expected, the migration job gets an image pull backoff:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  6m4s                  default-scheduler  Successfully assigned kamaji-system/migrate-cust1-capv-foo-5jhqw to capv-kamaji-mgmt-md-0-788d8649ffxfd49c-9sjt2
  Normal   Pulling    4m39s (x4 over 6m3s)  kubelet            Pulling image "clastix/kamaji:vhelm-v0.12.4"
  Warning  Failed     4m38s (x4 over 6m3s)  kubelet            Failed to pull image "clastix/kamaji:vhelm-v0.12.4": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/clastix/kamaji:vhelm-v0.12.4": failed to resolve reference "docker.io/clastix/kamaji:vhelm-v0.12.4": docker.io/clastix/kamaji:vhelm-v0.12.4: not found
  Warning  Failed     4m38s (x4 over 6m3s)  kubelet            Error: ErrImagePull
  Warning  Failed     4m12s (x6 over 6m2s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    56s (x20 over 6m2s)   kubelet            Back-off pulling image "clastix/kamaji:vhelm-v0.12.4"

It seems the calculation of the version of the image has an error. kamaji:vhelm-v0.12.4 looks like it tries to find the version (here v0.3.3) by looking into the helm chart (here v0.12.4), but the result doesnt create the correct string.

We use kamaji v0.3.3 together with cluster-api v1.5.0 with infrastructure-vsphere v1.7.0 and your kamaji plugin control-plane-kamaji v0.3.0.

About this issue

  • Original URL
  • State: closed
  • Created 10 months ago
  • Comments: 20 (10 by maintainers)

Most upvoted comments

Ah thanks!!! Now it worked, by editing the KamajiControlPlane. (We still have --migrate-image ion place.

https://github.com/clastix/kamaji/blob/973392bd85549b83d6ce96edad1a2f0e429f8808/Makefile#L159

We should return with no v prefix.

I’ll address this for the v0.3.4 release. In the meanwhile you can solve your issue without inflecting the image manually, just specify the --migrate-image CLI flag by specifying the fully qualified container image.

Thanks for giving this a try!