argo-cd: namespace not found vs. CreateNamespace=true

Describe the bug I’m getting a namespace not found error when deploying an app. It worked yesterday but stopped working today.

Since yesterday all nodes have only been scaled down and up again. No configuration has been changed apart from the fact I deleted the app along with the namespace and was trying to deploy it again. I also tried with a different namespace but it didn’t work.

App definition:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  labels:
    argocd.argoproj.io/instance: test-hello-world-app
  name: test-hello-world
  namespace: test-hello-world
spec:
  destination:
    namespace: test-hello-world
    server: 'https://kubernetes.default.svc'
  project: sebolabs-test
  source:
    helm:
      valueFiles:
        - values.yaml
        - values-test.yaml
    path: apps/hello-world
    repoURL: 'https://...'
    targetRevision: trying-out-things
  syncPolicy:
    automated:
      prune: true
    retry:
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 1m
      limit: 1
    syncOptions:
      - CreateNamespace=true

The sebolabs-test project accepts test-* namespaces + ArgoCD is configured with application.namespaces: "test-*". Just like it was explained yesterday here.

As I was running out of ideas on how to get it working again I updated ArgoCD to v2.5.4 but that didn’t change anything.

To Reproduce

See above.

Expected behavior

The test-hello-world namespace should be automatically created and the app should be successfully deployed.

Screenshots

N/A

Version

v2.5.4+86b2dde

Logs

time="2022-12-08T21:29:29Z" level=info msg=Syncing application=argocd/test-hello-world-app skipHooks=false started=false syncId=00008-QWOTG
time="2022-12-08T21:29:29Z" level=info msg="Tasks (dry-run)" application=argocd/test-hello-world-app syncId=00008-QWOTG tasks="[Sync/0 resource argoproj.io/Application:test-hello-world/test-hello-world nil->obj (,,)]"
time="2022-12-08T21:29:29Z" level=info msg="Applying resource Application/test-hello-world in cluster: https://172.20.0.1:443, namespace: test-hello-world"
time="2022-12-08T21:29:29Z" level=info msg="Updating operation state. phase: Running -> Running, message: 'one or more objects failed to apply, reason: namespaces \"test-hello-world\" not found. Retrying attempt #4 at 9:28PM.' -> 'one or more tasks are running'" application=argocd/test-hello-world-app syncId=00008-QWOTG
time="2022-12-08T21:29:29Z" level=info msg="Applying resource Application/test-hello-world in cluster: https://172.20.0.1:443, namespace: test-hello-world"
time="2022-12-08T21:29:29Z" level=info msg="Apply failed" application=argocd/test-hello-world-app dryRun=false message="namespaces \"test-hello-world\" not found" syncId=00008-QWOTG task="Sync/0 resource argoproj.io/Application:test-hello-world/test-hello-world nil->obj (,,)"
time="2022-12-08T21:29:29Z" level=info msg="Adding resource result, status: 'SyncFailed', phase: 'Failed', message: 'namespaces \"test-hello-world\" not found'" application=argocd/test-hello-world-app kind=Application name=test-hello-world namespace=test-hello-world phase=Sync syncId=00008-QWOTG
time="2022-12-08T21:29:29Z" level=info msg="Updating operation state. phase: Running -> Failed, message: 'one or more tasks are running' -> 'one or more objects failed to apply, reason: namespaces \"test-hello-world\" not found'" application=argocd/test-hello-world-app syncId=00008-QWOTG

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 4
Comments: 30 (2 by maintainers)

Most upvoted comments

We are still seeing the same error with

syncPolicy:
   automated:
     prune: true
     selfHeal: true
   syncOptions:
     - CreateNamespace=true

nicl-dev on Jul 24, 2023

The ApplicationSet example of the git-generator-directory is a good example to reproduce this bug Adding spec.destination.namespace does not change anything.

https://github.com/argoproj/argo-cd/tree/master/applicationset/examples/git-generator-directory

You just have to add the namespace restrictions to the project name used in the example. I could replicate it deploying this ApplicationSet from another Application. Not sure if it’s the App of App that introduce this issue. I remember using this mechanism without any issue several month ago.

nicolas-vivot on Dec 16, 2022

Okee-doke, here’s a minimal reproduction of the issue:

app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-hello-world-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/crenshaw-dev/argo-cd-11625
    path: resource
  destination:
    server: 'https://kubernetes.default.svc'
    # uncomment this, and the sync will work
    #namespace: test-hello-world
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

contents of resource to be deployed:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test
  namespace: test-hello-world

This conflicts with my understanding of how the CreateNamespace is supposed to work.

My belief was that it would create a namespace each time it encountered a resource destined for a namespace which does not exist.

The reality is that it just creates the namespace specified in spec.destination.namespace, regardless of whether any resource actually uses that namespace.

@ashutosh16, @leoluz was I just incorrect about how this feature works? Or is this a regression?

crenshaw-dev on Dec 13, 2022

Hi @sebolabs, given your original app definition above it can be that you try to deploy your argocd application into the wrong namespace (test-hello-world), because your metadata.namespace and your spec.destination.namespace are identical. Is your ArgoCD application really deployed into “test-hello-world” namespace? Otherwise you need to change the metadata.namespace to the namespace where you deployed ArgoCD itself.

hahawu42 on Nov 6, 2023

Just had this issue when trying to recreate an app. I had to remove the namespace to purge some annotations we set with managedNamespaceMetadata. We are running v2.6.14, though.

It’s an app-of-apps structure where I removed the whole app from the mother-app but it was still unable to recreate the namespace. It could only recreate the cluster-scoped resources.

EDIT: Nevermind. And on a totally unrelated note: never use “Prettier - Code Formatter” for VSCode without double-checking the changes it has made…

Honken77 on Nov 2, 2023

No worries. For testing, and as expected, I created the namespace with kubectl and everything got nicely synced. Then I went and deleted both the app and the namespace, redeployed the parent app, and ended up with the same error.

sebolabs on Dec 13, 2022

How are you deploying the test-hello-world app? From an app-of-apps, applicationset, kubectl or other?

The error message appears to be nonsensical. For Argo CD to throw an error when trying to deploy an Application which is installed in test-hello-world, that Application must first exist. For the Application to exist in that namespace, the namespace must exist.

My theory is that you’re deploying from an app-of-apps and that the parent app needs CreateNamespace=true. But that could be a bad guess.

crenshaw-dev on Dec 12, 2022