rancher: 422 Body: [code=NotUnique, fieldName=name] when multiple containers use the same data volume

Rancher Version: v1.0.0

Docker Version: 1.10.3

OS: Ubuntu 15.04

Steps to Reproduce:

docker-compose --verbose up with this docker-compose.yml:

app:
  image: mageinferno/magento2-nginx:1.9.9-2
  ports:
    - "80:80"
  links:
    - phpfpm
    - db
  volumes_from:
    - appdata
  environment:
    - APP_MAGE_MODE=production
    - VIRTUAL_HOST=shop.liquid-trading.de
  labels:
    io.rancher.sidekicks: appdata, phpfpm, setup

appdata:
  image: tianon/true
  volumes:
    - "/var/magento2/src:/src"

phpfpm:
  image: mageinferno/magento2-php:7.0.2-fpm-1
  links:
    - db
  volumes_from:
    - appdata
  environment:
    - APP_MAGE_MODE=default
    - PHP_MEMORY_LIMIT=2048M

db:
  image: mariadb:10.1.10
  ports:
    - "3306:3306"
  volumes:
    - "/var/magento2/mysql:/var/lib/mysql"

setup:
  image: mageinferno/magento2-php:7.0.2-fpm-1
  command: /usr/local/bin/mage-setup
  links:
    - db
  volumes_from:
    - appdata

Results:

INFO[0000] Creating service app
DEBU[0000] Creating service app with hash: digest.ServiceHash{Service:"04eebe96f990ab64972bce5193990b3e6b8da6a2", LaunchConfig:"792bc6859a5da13c8bc9ce0c1e502a38418e3cb8", SecondaryLaunchConfigs:map[string]string{"appdata":"a4b64bdd56ddf28671e2787b7fc583bb55b804e9", "phpfpm":"5b7975000c982dd602f04b98f9e55f63d3b9817c", "setup":"367a2ef15c2a8d522e9f05f9ed23229eb02b05bd"}}
ERRO[0000] Failed Creating app : Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name] from [http://rancher.storyx.company:8080/v1/services]

Expected:

Spinning up all containers.

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 20 (4 by maintainers)

Most upvoted comments

I get same issue with v1.6.10…

Tested it with Elasticsearch Cluster 5.5.1 package, getting the same error with different services inside same package.

It happens once you deploy the package from catalog, you remove it and immediately redeploy it with same name.

Redeploy 1 failing es-data service…

time=“2017-10-04T14:00:38Z” level=info msg=“Stack Create Event Received” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:38Z” level=info msg=“Creating volume template es-storage-volume”
time=“2017-10-04T14:00:38Z” level=info msg=“[es-cluster:]: Creating project ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:38Z” level=info msg=“[es-cluster:es-data]: Creating ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:38Z” level=info msg=“[es-cluster:es-client]: Creating ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:38Z” level=info msg=“[es-cluster:es-master]: Creating ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:38Z” level=info msg=“Creating service es-client”
time=“2017-10-04T14:00:38Z” level=info msg=“Creating service es-master”
time=“2017-10-04T14:00:38Z” level=info msg=“Creating service es-data”
time=“2017-10-04T14:00:39Z” level=error msg=“Failed Creating es-data : Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]”
time=“2017-10-04T14:00:39Z” level=error msg=“Failed to start: es-data : Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]”
time=“2017-10-04T14:00:39Z” level=info msg=“[es-cluster:es-client]: Created ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:39Z” level=info msg=“[es-cluster:es-master]: Created ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:39Z” level=info msg=“[es-cluster:es-sysctl]: Creating ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:39Z” level=info msg=“[es-cluster:es-sysctl]: Created ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:39Z” level=info msg=“[es-cluster:es-storage]: Creating ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:39Z” level=info msg=“[es-cluster:es-storage]: Created ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:39Z” level=error msg=“Stack Create Event Failed: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
time=“2017-10-04T14:00:39Z” level=info msg=“[es-cluster:]: Project created ” eventId=05c0a828-4b59-4d11-a2e3-5c17124cf11b resourceId=1st358
2017-10-04 14:00:39,871 ERROR [4cdf2363-83f3-40e8-9d05-bcc6f92d2078:6461488] [stack:358] [stack.create] [] [cutorService-67] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [stack.create.reply;handler=rancher-compose-executor]: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]
2017-10-04 14:00:40,177 ERROR [:] [] [] [] [cutorService-67] [.e.s.i.ProcessInstanceDispatcherImpl] Agent error for [stack.create.reply;handler=rancher-compose-executor]: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]

Redeploy 2 failing es-client service…

time=“2017-10-04T14:04:57Z” level=info msg=“Stack Create Event Received” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:57Z” level=info msg=“Creating volume template es-storage-volume”
time=“2017-10-04T14:04:57Z” level=info msg=“[es-cluster:]: Creating project ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:57Z” level=info msg=“[es-cluster:es-data]: Creating ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:57Z” level=info msg=“[es-cluster:es-master]: Creating ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:57Z” level=info msg=“[es-cluster:es-client]: Creating ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:57Z” level=info msg=“Creating service es-master”
time=“2017-10-04T14:04:57Z” level=info msg=“Creating service es-data”
time=“2017-10-04T14:04:57Z” level=info msg=“Creating service es-client”
time=“2017-10-04T14:04:57Z” level=error msg=“Failed Creating es-client : Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]”
time=“2017-10-04T14:04:58Z” level=info msg=“[es-cluster:es-master]: Created ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:58Z” level=info msg=“[es-cluster:es-data]: Created ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:58Z” level=info msg=“[es-cluster:es-sysctl]: Creating ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:58Z” level=info msg=“[es-cluster:es-storage]: Creating ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:58Z” level=info msg=“[es-cluster:es-storage]: Created ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:58Z” level=info msg=“[es-cluster:es-sysctl]: Created ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:58Z” level=error msg=“Failed to start: es-client : Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]”
time=“2017-10-04T14:04:58Z” level=error msg=“Stack Create Event Failed: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
time=“2017-10-04T14:04:58Z” level=info msg=“[es-cluster:]: Project created ” eventId=e3a06db5-b774-4b52-9314-6d4af25377bd resourceId=1st359
2017-10-04 14:04:58,967 ERROR [d2c9b4f1-8149-4212-b4f8-5b228a8a240b:6462029] [stack:359] [stack.create] [] [cutorService-63] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [stack.create.reply;handler=rancher-compose-executor]: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]
2017-10-04 14:04:59,142 ERROR [:] [] [] [] [cutorService-63] [.e.s.i.ProcessInstanceDispatcherImpl] Agent error for [stack.create.reply;handler=rancher-compose-executor]: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=name, baseType=error] from [http://localhost:8080/v2-beta/projects/1a14/services]

Waiting a couple of minutes between remove and redeploy the package, issue doesn’t occur…

It seems a race condition, Services are removed async and some of them could still exist transitioning to removed once the stack is already removed. Redeploying so quickly from the catalog could eventually generates status code 422, notUnique service name error.

rawmind0 on Oct 4, 2017