kustomize-controller: Azure Keyvault access stops working correctly with Kustomize-controller >=0.22.0
When using Azure KV with sops-stored secrets in kustomize controller >= 0.22.0 a strange behaviour appears: decryption is rejected due to a bad parameter (HTTP Status code 400) error, with no further indication about what’s wrong.
In fact, using Azure Monitor Key Vault functionality and enabling detailed logging (even for failures) I was able to pinpoint the error.
REST API address generated by internal SOPS in kustomize controller is:
https://my-keyvault-name.vault.azure.net/keys/sops-key/1d....my-key-version.../decrypt?api-version=7.3-preview
Instead of
https://my-keyvault-name.vault.azure.net/keys/sops-key/1d....my-key-version.../decrypt?api-version=2016-10-01
Which is generated when I use sops in macOS CLI.
api-version is surely strange, because it does not resembles any API version scheme used in Azure (right?), but it is close enough to the latest version number in Azure SDK for Go, so may be it is a bug on a recent version of that library.
So I started searching history of go.mod and testing recent versions of kustomize-controller that were in use before bumping Flux.
0.22.0 and 0.22.1 exhibit the very same failure, some maybe what bricked Azure KV is this version (0.4.0) of azkeys library, introduced in 0.22.0
kustomize-controller 0.21.1 works just fine, instead.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 22 (12 by maintainers)
That means the cause of the issue was the broken fallback, which was restored in
v0.22.2(and fixed at the time of https://github.com/fluxcd/kustomize-controller/issues/595#issuecomment-1079297839).I will be making some structural changes to the code so this can not happen again, while adding some more tests for other KMS solutions to ensure regression bugs (or compatibility issues with upstream) will be detected earlier, as already done for Azure Key Vault in #604.
Given all this, I think this issue can be closed, but do not hesitate to comment (or open a new issue) if new problems and/or questions arise. Thanks all 🙇
Now available as an official release in https://github.com/fluxcd/flux2/releases/tag/v0.28.3
Manual patches to overwrite the source-controller version can be removed when updating to https://github.com/fluxcd/flux2/releases/tag/v0.28.5. This should solve all reported problems in this issue (and the general
v0.22.xrange of this controller).Confirmed, @hiddeco !
In a staging cluster I’ve used Flux 0.28.4 + patch to use kustomize-controller 0.22.3 (since in Flux 0.28.4 the default kustomize-controller is 0.22.2), with the following outcome: I can use secretRef.name in spec.decryptor and the decryption process works fine.
Thank you!
@log2 for the issues you described in https://github.com/fluxcd/kustomize-controller/issues/595#issuecomment-1079215862, I expect https://github.com/fluxcd/kustomize-controller/releases/tag/v0.22.3 to now work without issues.
@BertelBB this might solve your issues as well. If not, please provide me with the output of
kubectl get kustomization <name> -oyaml --show-managed-fields, plus some details about your specific SOPS setup.Can you please give the following (AMD64) image a try?
hiddeco/kustomize-controller:fix-azkv-fallback-2495ea9Think I found the issue, will prepare a patch.
The other change between
v0.21.xandv0.22.xis an update of SOPS itself (fromv3.7.1tov3.7.2). Can you confirm the v3.7.2 SOPS binary works correctly on your machine?