image-automation-controller: Lack of context for failing reconciliation

I received this message from Bitbucket:

Syahrul commented:

G’day, Paul

A quick update on this issue.

Our development team noticed a pattern with the FluxCD issue with Bitbucket cloud. After thorough analysis, it has been determined that the issue is most likely caused by the go-git library being used by FluxCD. This library prematurely closes the connection before a push operation is completed.

To address this matter, we will release a fix tomorrow to mitigate the problem. Once the mitigation process is complete, we will provide you with an update.

We appreciate your patience and encourage you to reach out if you have any additional questions.

– Mohammad Syahrul Support Engineer APAC, Bitbucket cloud

Bitbucket rolled their fix, and for us everything has been working perfectly again.

We use both flux and bitbucket and have been absolutely pulling our hair out over this issue. For what it’s worth we found that moving from https:// to ssh:// git URLs seemed to make the behavior go away. That isn’t always practical to do however so here’s hoping that bitbucket’s fix works out.

I specifically spoke to them about “object not found” and gave some technical details … i’m fairly sure it will fix this.

Wondering if “object not found” issue will be fixed, or its related to something else 😃

have raised a support ticket with Bitbucket… still no solution though

Same here, since it was blocking us - we switched manifest repository location to another provider… and now thinking to switch everything 😃

@mantasaudickas, We’re also using bitbucket and have the scenario of multiple clusters using the same repo. Are you having good results so far?

I have switched one project to gitlab and another to github (2 independent clients). So far error message “object not found” is gone in both of them. I did not tried your mentioned options.

The reason for switch actually was bitbucket issue - that once FluxCD makes a push - its not possible to get that last push anymore (while it is visible using UI, but not fetchable to local copies and not visible in git command line history)… I don’t know if its a flux or bitbucket issue, but it was solved by migrating to other providers.

There are a couple of reports of this type of failure (or potentially unrelated failures eg. git error code 128) that are showing up in the Slack channel, I haven’t seen them filter down to reports for IAC as of yet, but something to be aware of.

I will load up some Image Update Automation controls today or tomorrow and try to reproduce this issue, one or the other issue, there is not much context to go on for what is causing the failure. I understand this report is not about one specific failure, but the general case of failure not being reported very clearly with a good obvious link to a really specific root cause.

{"level":"error","ts":"2023-04-03T19:27:38.561Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"a43e903f-de19-4eaa-a7cd-e64a804d77fa","error":"malformed unpack status: \u00010069\u0001001dunpack index-pack failed\n0043ng refs/heads/main error processing packfiles: exit status 128\n0000"}

This is another example of that. This is the error returned from Git, and I’m not sure how much helpful parsing we can do, but to refocus, the subject of this report is about making it clearer what has gone wrong when IAC fails. Maybe we can come up with some common failure scenarios and start classifying errors to raise those as conditions, based on a pattern matching.