spinnaker: Orca: Spinnaker upgrade from 1.33 to 1.34 or 1.34 fails due to liquibase.exception.MigrationFailedException

I am trying to upgrade spinnaker(using haylard) from 1.33 to 1.34 or 1.35 and I am getting failures in orca pods.

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. 2025-03-19 12:16:46.304 ERROR 1 --- [ main] o.s.boot.SpringApplication : [] Application run failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'redisOrcaQueueConfiguration': Unsatisfied dependency expressed through method 'redisQueueObjectMapper' parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'taskResolver' defined in com.netflix.spinnaker.orca.config.OrcaConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.netflix.spinnaker.orca.TaskResolver]: Factory method 'taskResolver' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dependsOnExecutionTask' defined in URL [jar:file:/opt/orca/lib/orca-core-8.51.1.jar!/com/netflix/spinnaker/orca/pipeline/tasks/DependsOnExecutionTask.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sqlExecutionRepository' defined in class path resource [com/netflix/spinnaker/config/SqlConfiguration.class]: Unsatisfied dependency expressed through method 'sqlExecutionRepository' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [com/netflix/spinnaker/config/SqlConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.CommandExecutionException: liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset db/changelog/20201214-create-compressed-executions-table.yaml::create-orchestrations-stages-compressed-executions-table::apattan: Reason: liquibase.exception.DatabaseException: Table 'orchestration_stages_compressed_executions' already exists [Failed SQL: (1050) CREATE TABLE orca_sandbox.orchestration_stages_compressed_executions (id CHAR(26) NOT NULL, compressed_body LONGBLOB NOT NULL, CONSTRAINT PK_ORCHESTRATION_STAGES_COMPRESSED_EXECUTIONS PRIMARY KEY (id))] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.resolveMethodArguments(AutowiredAnnotationBeanPostProcessor.java:768) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:720) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:920) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:745) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:423) at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164) at org.springframework.boot.builder.SpringApplicationBuilder$run$0.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139) at com.netflix.spinnaker.orca.Main.main(Main.groovy:80) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'taskResolver' defined in com.netflix.spinnaker.orca.config.OrcaConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.netflix.spinnaker.orca.TaskResolver]: Factory method 'taskResolver' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dependsOnExecutionTask' defined in URL [jar:file:/opt/orca/lib/orca-core-8.51.1.jar!/com/netflix/spinnaker/orca/pipeline/tasks/DependsOnExecutionTask.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sqlExecutionRepository' defined in class path resource [com/netflix/spinnaker/config/SqlConfiguration.class]: Unsatisfied dependency expressed through method 'sqlExecutionRepository' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [com/netflix/spinnaker/config/SqlConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.CommandExecutionException: liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset db/changelog/20201214-create-compressed-executions-table.yaml::create-orchestrations-stages-compressed-executions-table::apattan: Reason: liquibase.exception.DatabaseException: Table 'orchestration_stages_compressed_executions' already exists [Failed SQL: (1050) CREATE TABLE orca_sandbox.orchestration_stages_compressed_executions (id CHAR(26) NOT NULL, compressed_body LONGBLOB NOT NULL, CONSTRAINT PK_ORCHESTRATION_STAGES_COMPRESSED_EXECUTIONS PRIMARY KEY (id))] It seems to be related to these issues https://github.com/spinnaker/orca/pull/4620 https://github.com/spinnaker/orca/pull/4727 https://github.com/spinnaker/spinnaker/issues/6941

The issue seems to be due to the checksums associated with the new compression tables added in 1.34.

create-pipelines-compressed-executions-table apattan db/changelog/20201214-create-compressed-executions-table.yaml 2025-03-19 11:58:28 51 EXECUTED 9:762181febfdfd0a1245de8e200adcdcb create-pipeline-stages-compressed-executions-table apattan db/changelog/20201214-create-compressed-executions-table.yaml 2025-03-19 12:00:13 52 EXECUTED 9:dd9555942f5320c5d28ae0798718637f create-orchestrations-compressed-executions-table apattan db/changelog/20201214-create-compressed-executions-table.yaml 2025-03-19 12:02:09 53 EXECUTED 9:04ca4b6e7ad9bf098421ce9582c7ba29 create-orchestration-stages-compressed-executions-table apattan db/changelog/20201214-create-compressed-executions-table.yaml 2025-03-19 12:03:30 54 EXECUTED

A workaround i tried was the do the following for example:


use orca_dev;


SET @max_orderexecuted = (SELECT MAX(ORDEREXECUTED) FROM DATABASECHANGELOG);

INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, EXECTYPE)
VALUES (
    'create-pipelines-compressed-executions-table',
    'apattan',
    'db/changelog/20201214-create-compressed-executions-table.yaml',
    NOW(),
    @max_orderexecuted + 1,
    'EXECUTED'
);

This seemed to work for 3 of the tables but when we get to the last table it does not work. the table names that have the issues are as follows

  1. pipelines_compressed_executions
  2. pipeline_stages_compressed_executions
  3. orchestrations_compressed_executions
  4. orchestration_stages_compressed_executions

it seems to be during liquibase migration and Im not sure why it is saying table already exists. https://github.com/spinnaker/orca/blob/release-1.37.x/orca-sql/src/main/resources/db/changelog/20201214-create-compressed-executions-table.yaml.

I have downgraded and upgraded many ties and facing same issue. Easily reproducible for me.

Thanks Gary

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

Yes Kiran i believe that we are safe to upgrade to the later versions now with that change so I’m happy with that thanks!!

I have tested scenario of 1.33.3 -> 1.34.3 using Postgresql using this integration test - PostgresMigrationContainerTest which succeeded.

And proceeded with a new integration test with MySql db for the same scenario of migrating from 1.33.3 -> 1.34.3 and it also succeeded.

In both the scenarios, 1.33.3 didn’t execute 20201214-create-compressed-executions-table.yaml. Only the second container executed the changlog and succeeded.

Other cases like 1.33.0 / 1.33.1/ 1.33.2 -> 1.34.x were failing due to the first container not being healthy even after several retries. Test may have limitations to show the exact reason.

I am looking into it.

using mysql