spinnaker: Failed to trigger the bake and deploy pipeline

Issue Summary:

It failed to trigger the bake and deploy pipeline (createServerGroup) and it failed to trigger the disable and destroy pipeline. After investigated, it seems that the failure is caused by the commit https://github.com/spinnaker/clouddriver/pull/5791. If reverted this commit, it works fine. @ovidiupopa07, so could you take a look this issue and fix it asap since we’re blocked now? Thanks.

Error Message while triggering the bake and deploy pipeline:

{
    "phase": "ORCHESTRATION",
    "status": "Orchestration failed: CreateAzureServerGroupWithAzureLoadBalancerAtomicOperation | ManagementException: [Status code 400, "{  "error": {    "code": "LoadBalancingRuleBackendAdressPoolAndBackendAddressPoolsCannotBeSetAtTheSameTimeWithDifferentValue",    "message": "BackendAddressPool and BackendAddressPools[] in LoadBalancingRule xxxx-st-frontend-rule0 cannot be set at the same time with different value.",    "details": []  }}"]"
}

Error Message while triggering the disable and destroy pipeline:

"type": "determineTargetServerGroup",
      "name": "determineTargetServerGroup",
      "status": "TERMINAL",
      "context": {
        "exception": {
          "exceptionType": "NotFoundException",
          "shouldRetry": false,
          "details": {
            "stackTrace": "com.netflix.spinnaker.orca.clouddriver.pipeline.servergroup.support.TargetServerGroup$NotFoundException: Unable to locate current_asg_dynamic in my-azure-account/westus2/xxxx-st-frontend\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat,
            "error": "Unexpected Task Failure",
            "errors": [
              "Unable to locate current_asg_dynamic in my-azure-account/westus2/xxxx-st-frontend"
            ]
          },
          "operation": "determineTargetServerGroup"
        },

Cloud Provider(s):

Azure

Environment:

ubuntu 18.04

Feature Area:

Clouddriver

Steps to Reproduce:

  1. hal config version edit --version branch:upstream/master
  2. hal deploy apply
  3. Create a new Spinnaker Application
  4. Create a new load balancer
  5. Create the bake and deploy pipeline
  6. Create the disable and destroy pipeline
  7. Trigger the bake and deploy pipeline. This creates a new server group below the given load balancer but it failed
  8. Trigger the disable and destroy pipeline but it failed

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17

Commits related to this issue

Most upvoted comments

Hi @neil-yechenwei. I fixed the issue: image

The disable and destroy pipeline works too image image image

I will open a PR later today. Thanks for pointing this out 👍

+1
ovidiupopa07 on Oct 28, 2022

hi @neil-yechenwei. I managed to reproduce it. If the load balancer has no rule defined everything works fine. If a rule is defined it fails when adding a new backend pool.

Anyway, it is strange why is not failing with the old azure SDK but is failing with the new one. I’ve gone through the SDKs and I don’t see any big difference. I will further investigate the reason why this is happening and I will come up with a solution

Actually, I just saw something. In the old SDK, there is only one property backendAddressPool defined in LoadBalancingRuleInner.

In the latest SDK there are 2 properties image

Anyway, I will check how can we fix this.

+1
ovidiupopa07 on Oct 27, 2022

Hi @neil-yechenwei. Unfortunately, I was out last week and had to take care of some personal matters. I started to work on this issue yesterday. Currently, I am trying to setup everything on Azure -> to reproduce the problem. I will let you know when I have something to share.

+1
ovidiupopa07 on Oct 25, 2022

Thanks @ovidiupopa07 for the update. We have to ensure there is no breaking change or blocking issue on master branch in case the bug/issue is brought into stable release. So we can wait until it’s fixed. Once the issue is fixed, please let us know. Thanks in advance.

Yes, I agree, I will fix it for sure. I was mentioning this as a workaround until I fix the issue in the master branch, just to unblock you. I will let you know when I push the fix.

+1
ovidiupopa07 on Oct 14, 2022

@ovidiupopa07 , Sure. Hopes it would be helpful.

The json content for Bake and Deploy pipeline:

[
  {
    "application": "azurep0lb031344",
    "id": "xx-xx-xx-xx",
    "index": 0,
    "keepWaitingPipelines": false,
    "lastModifiedBy": "anonymous",
    "limitConcurrent": true,
    "name": "BakeAndDeploy",
    "schema": "1",
    "stageCounter": 2,
    "stages": [
      {
        "baseLabel": "release",
        "baseOs": "ubuntu-1604",
        "cloudProviderType": "azure",
        "extendedAttributes": {},
        "name": "Bake",
        "osType": "linux",
        "package": "",
        "refId": "BAKE",
        "regions": [
          "westus2"
        ],
        "requisiteStageRefIds": [],
        "type": "bake",
        "user": "[anonymous]"
      },
      {
        "clusters": [
          {
            "account": "my-azure-account",
            "application": "azurep0lb031344",
            "capacity": {
              "max": 1,
              "min": 1
            },
            "cloudProvider": "azure",
            "customScriptsSettings": {
              "commandToExecute": ""
            },
            "detail": "frontend",
            "enableInboundNAT": false,
            "freeFormDetails": "frontend",
            "image": {
              "imageName": "",
              "isCustom": "true",
              "offer": "",
              "ostype": "",
              "publisher": "",
              "region": "westus2",
              "sku": "",
              "uri": "",
              "version": ""
            },
            "instanceTags": {},
            "instanceType": "Standard_B1ms",
            "interestingHealthProviderNames": [],
            "loadBalancerName": "azurep0lb031344-st-frontend",
            "loadBalancerType": "Azure Load Balancer",
            "name": "azurep0lb031344-st-frontend",
            "osConfig": {},
            "region": "westus2",
            "rollback": {},
            "selectedProvider": "azure",
            "sku": {
              "capacity": 1,
              "name": "Standard_B1ms",
              "tier": "Standard"
            },
            "stack": "st",
            "strategy": "",
            "subnet": "default",
            "type": "createServerGroup",
            "upgradePolicy": "Manual",
            "useSourceCapacity": false,
            "user": "[anonymous]",
            "viewState": {
              "disableImageSelection": true,
              "disableStrategySelection": true,
              "expectedArtifacts": [],
              "hideClusterNamePreview": false,
              "instanceProfile": "custom",
              "loadBalancersConfigured": true,
              "mode": "createPipeline",
              "networkSettingsConfigured": true,
              "readOnlyFields": {},
              "securityGroupsConfigured": true,
              "showImageSourceSelector": true,
              "submitButtonLabel": "Add",
              "templatingEnabled": true,
              "useAllImageSelection": false,
              "usePreferredZones": true,
              "useSimpleCapacity": true
            },
            "vnet": "spin-xxx-vnet",
            "vnetResourceGroup": "spin-xxx-rg",
            "zones": [],
            "zonesEnabled": false
          }
        ],
        "name": "Deploy",
        "refId": "DEPLOY",
        "requisiteStageRefIds": [
          "BAKE"
        ],
        "type": "deploy"
      }
    ],
    "triggers": []
  }
]

The json content for Disable and Destroy pipeline:

[
  {
    "id": "xx-xx-xx-xx",
    "name": "BakeAndDeploy",
    "application": "azurep0lb031344",
    "schema": "1",
    "triggers": [],
    "index": 0,
    "lastModifiedBy": "anonymous",
    "stageCounter": 2,
    "stages": [
      {
        "baseLabel": "release",
        "baseOs": "ubuntu-1604",
        "cloudProviderType": "azure",
        "extendedAttributes": {},
        "name": "Bake",
        "osType": "linux",
        "package": "",
        "refId": "BAKE",
        "regions": [
          "westus2"
        ],
        "requisiteStageRefIds": [],
        "type": "bake",
        "user": "[anonymous]"
      },
      {
        "clusters": [
          {
            "account": "my-azure-account",
            "application": "azurep0lb031344",
            "capacity": {
              "max": 1,
              "min": 1
            },
            "cloudProvider": "azure",
            "customScriptsSettings": {
              "commandToExecute": ""
            },
            "detail": "frontend",
            "enableInboundNAT": false,
            "freeFormDetails": "frontend",
            "image": {
              "imageName": "",
              "isCustom": "true",
              "offer": "",
              "ostype": "",
              "publisher": "",
              "region": "westus2",
              "sku": "",
              "uri": "",
              "version": ""
            },
            "instanceTags": {},
            "instanceType": "Standard_B1ms",
            "interestingHealthProviderNames": [],
            "loadBalancerName": "azurep0lb031344-st-frontend",
            "loadBalancerType": "Azure Load Balancer",
            "name": "azurep0lb031344-st-frontend",
            "osConfig": {},
            "region": "westus2",
            "rollback": {},
            "selectedProvider": "azure",
            "sku": {
              "capacity": 1,
              "name": "Standard_B1ms",
              "tier": "Standard"
            },
            "stack": "st",
            "strategy": "",
            "subnet": "default",
            "type": "createServerGroup",
            "upgradePolicy": "Manual",
            "useSourceCapacity": false,
            "user": "[anonymous]",
            "viewState": {
              "disableImageSelection": true,
              "disableStrategySelection": true,
              "expectedArtifacts": [],
              "hideClusterNamePreview": false,
              "instanceProfile": "custom",
              "loadBalancersConfigured": true,
              "mode": "createPipeline",
              "networkSettingsConfigured": true,
              "readOnlyFields": {},
              "securityGroupsConfigured": true,
              "showImageSourceSelector": true,
              "submitButtonLabel": "Add",
              "templatingEnabled": true,
              "useAllImageSelection": false,
              "usePreferredZones": true,
              "useSimpleCapacity": true
            },
            "vnet": "spin-xxx-vnet",
            "vnetResourceGroup": "spin-xxx-rg",
            "zones": [],
            "zonesEnabled": false
          }
        ],
        "name": "Deploy",
        "refId": "DEPLOY",
        "requisiteStageRefIds": [
          "BAKE"
        ],
        "type": "deploy"
      }
    ],
    "keepWaitingPipelines": false,
    "limitConcurrent": true
  },
  {
    "id": "xx-xx-xx-xx",
    "name": "DisableAndDestroy",
    "application": "azurep0lb031344",
    "schema": "1",
    "triggers": [],
    "index": 1,
    "lastModifiedBy": "anonymous",
    "stageCounter": 2,
    "stages": [
      {
        "cloudProvider": "azure",
        "cloudProviderType": "azure",
        "cluster": "azurep0lb031344-st-frontend",
        "credentials": "my-azure-account",
        "moniker": {
          "app": "azurep0lb031344",
          "cluster": "azurep0lb031344-st-frontend",
          "detail": "frontend",
          "stack": "st"
        },
        "name": "Disable Server Group",
        "refId": "DISABLE",
        "regions": [
          "westus2"
        ],
        "requisiteStageRefIds": [],
        "target": "current_asg_dynamic",
        "type": "disableServerGroup"
      },
      {
        "cloudProvider": "azure",
        "cloudProviderType": "azure",
        "cluster": "azurep0lb031344-st-frontend",
        "credentials": "my-azure-account",
        "interestingHealthProviderNames": [],
        "moniker": {
          "app": "azurep0lb031344",
          "cluster": "azurep0lb031344-st-frontend",
          "detail": "frontend",
          "stack": "st"
        },
        "name": "Destroy Server Group",
        "refId": "DESTROY",
        "regions": [
          "westus2"
        ],
        "requisiteStageRefIds": [
          "DISABLE"
        ],
        "target": "current_asg_dynamic",
        "type": "destroyServerGroup"
      }
    ],
    "keepWaitingPipelines": false,
    "limitConcurrent": true
  }
]
+1
neil-yechenwei on Oct 27, 2022
Read more comments on GitHub
← spinnaker: Error from server (AlreadyExists): error when creating "STDIN": namespaces "spinnaker" already exists
spinnaker: Fiat Service Account causes NPE if the Service Account is not a user in LDAP →