kubernetes: Error syncing deployment, replica set already exists

Env: Kubernetes 1.3.3 on AWS

I ran into the same problem described into #26673 but now it’s happening on every deploy. I would have reopened the issue, but I was not the author so I could not.

The bug presents itself because I have a build that does kubectl rollout status deployment/application-unstable just after a kubectl apply -f <new deployment manifest> and it gets stuck forever because the rollout is never reported as finished (it does finish tho).

Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.615754   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.557964   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.509632   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.456868   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.409374   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.357072   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.306676   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.257030   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.206504   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.157741   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.112898   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.059652   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.007476   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.956218   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.910275   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.857240   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.806429   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.756885   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.706639   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.656417   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.607126   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.557562   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.506025   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.456588   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.422213   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.356907   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.313353   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.257953   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.209894   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.160954   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.107205   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.056387   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.006022   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:25 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:25.956830   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:25 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:25.906450   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 42 (36 by maintainers)

Commits related to this issue

Merge pull request #36584 from kargakis/add-more-logging-in-the-deployment-controller Automatic merge from submit-queue Do not handle AlreadyExists errors yet Until we fix https://github.com/kubern... — committed to kubernetes/kubernetes by deleted user 8 years ago
Merge pull request #44774 from kargakis/uniquifier Automatic merge from submit-queue Switch Deployments to new hashing algo w/ collision avoidance mechanism Implements https://github.com/kubernetes... — committed to kubernetes/kubernetes by deleted user 7 years ago
Merge pull request #44774 from kargakis/uniquifier Automatic merge from submit-queue Switch Deployments to new hashing algo w/ collision avoidance mechanism Implements https://github.com/kubernetes... — committed to kubernetes-retired/cluster-registry by deleted user 7 years ago

Most upvoted comments

Hash collisions are definitely a problem. I wrote up a test and can get a collision within a couple hundred versions (only version changes). Here’s pod_test.go:

package pod

import (
    "testing"
    "strings"
    "strconv"
    "encoding/json"
    "fmt"
    "k8s.io/kubernetes/pkg/api"
)

var podSpec string = `
{
    "metadata": {
        "creationTimestamp": null,
        "labels": {
            "app": "cats"
        }
    },
    "spec": {
        "containers": [
            {
                "name": "cats",
                "image": "registry/test/cats:v0.@@VERSION@@.0",
                "ports": [
                    {
                        "name": "http",
                        "containerPort": 9077,
                        "protocol": "TCP"
                    }
                ],
                "env": [
                    {
                        "name": "DEPLOYMENT_ENVIRONMENT",
                        "value": "cats-stubbed-functional"
                    },
                    {
                        "name": "APP_NAME",
                        "value": "cats"
                    }
                ],
                "resources": {
                    "limits": {
                        "cpu": "1",
                        "memory": "1Gi"
                    },
                    "requests": {
                        "cpu": "1",
                        "memory": "1Gi"
                    }
                },
                "livenessProbe": {
                    "httpGet": {
                        "path": "/private/status",
                        "port": 9077,
                        "scheme": "HTTP"
                    },
                    "initialDelaySeconds": 30,
                    "timeoutSeconds": 1,
                    "periodSeconds": 10,
                    "successThreshold": 1,
                    "failureThreshold": 3
                },
                "readinessProbe": {
                    "httpGet": {
                        "path": "/private/status",
                        "port": 9077,
                        "scheme": "HTTP"
                    },
                    "initialDelaySeconds": 1,
                    "timeoutSeconds": 1,
                    "periodSeconds": 10,
                    "successThreshold": 1,
                    "failureThreshold": 3
                },
                "terminationMessagePath": "/dev/termination-log",
                "imagePullPolicy": "IfNotPresent"
            }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {}
    }
}
`

func TestPodTemplateSpecHash(t *testing.T) {
    seenHashes := make(map[uint32]int)

    for i := 0; i < 1000; i++ {
        specJson := strings.Replace(podSpec, "@@VERSION@@", strconv.Itoa(i), 1)
        spec := api.PodTemplateSpec{}
        json.Unmarshal([]byte(specJson), &spec)
        hash := GetPodTemplateSpecHash(spec)
        fmt.Printf("version: %d hash: %d\n", i, hash)
        if v, ok := seenHashes[hash]; ok {
            t.Fatalf("Hash collision, old: %d new: %d", v, i)
        }
        seenHashes[hash] = i
    }
}

/usr/local/go/bin/go test -v k8s.io/kubernetes/pkg/util/pod -run ^TestPodTemplateSpecHash$
<snip>
version: 119 hash: 2663716803
version: 120 hash: 1102452667
version: 121 hash: 1297618876
version: 122 hash: 1492785085
version: 123 hash: 1687951294
version: 124 hash: 1883117503
version: 125 hash: 2078283712
version: 126 hash: 2273449921
version: 127 hash: 2468616130
version: 128 hash: 2663782339
version: 129 hash: 2858948548
version: 130 hash: 1297684412
version: 131 hash: 1492850621
version: 132 hash: 1688016830
version: 133 hash: 1883183039
version: 134 hash: 2078349248
version: 135 hash: 2273515457
version: 136 hash: 2468681666
version: 137 hash: 2663847875
version: 138 hash: 2859014084
version: 139 hash: 3054180293
version: 140 hash: 1492916157
version: 141 hash: 1688082366
version: 142 hash: 1883248575
version: 143 hash: 2078414784
version: 144 hash: 2273580993
version: 145 hash: 2468747202
version: 146 hash: 2663913411
version: 147 hash: 2859079620
version: 148 hash: 3054245829
version: 149 hash: 3249412038
version: 150 hash: 1688147902
version: 151 hash: 1883314111
version: 152 hash: 2078480320
version: 153 hash: 2273646529
version: 154 hash: 2468812738
version: 155 hash: 2663978947
version: 156 hash: 2859145156
version: 157 hash: 3054311365
version: 158 hash: 3249477574
version: 159 hash: 3444643783
version: 160 hash: 1883379647
version: 161 hash: 2078545856
version: 162 hash: 2273712065
version: 163 hash: 2468878274
version: 164 hash: 2664044483
version: 165 hash: 2859210692
version: 166 hash: 3054376901
version: 167 hash: 3249543110
version: 168 hash: 3444709319
version: 169 hash: 3639875528
version: 170 hash: 2078611392
version: 171 hash: 2273777601
version: 172 hash: 2468943810
version: 173 hash: 2664110019
version: 174 hash: 2859276228
version: 175 hash: 3054442437
version: 176 hash: 3249608646
version: 177 hash: 3444774855
version: 178 hash: 3639941064
version: 179 hash: 3835107273
version: 180 hash: 2273843137
version: 181 hash: 2469009346
version: 182 hash: 2664175555
version: 183 hash: 2859341764
version: 184 hash: 3054507973
version: 185 hash: 3249674182
version: 186 hash: 3444840391
version: 187 hash: 3640006600
version: 188 hash: 3835172809
version: 189 hash: 4030339018
version: 190 hash: 2469074882
version: 191 hash: 2664241091
version: 192 hash: 2859407300
version: 193 hash: 3054573509
version: 194 hash: 3249739718
version: 195 hash: 3444905927
version: 196 hash: 3640072136
version: 197 hash: 3835238345
version: 198 hash: 4030404554
version: 199 hash: 4225570763
version: 200 hash: 907286458
version: 201 hash: 1102452667
    pod_test.go:98: Hash collision, old: 120 new: 201
exit status 1
FAIL    k8s.io/kubernetes/pkg/util/pod  0.043s

I’m guessing we can’t improve the hash function at this point, to maintain backwards compatibility? We should at least handle the collision somehow - what do you suggest?

jsravn on Oct 18, 2016