spinnaker: ECS create server group is throwing exception because of problem with Orca running with Java 17

Issue Summary:

We wanted to upgrade to 1.33.0 and ECS deployments started failing with the an exception:

Exception ( Create Server Group )
Exception evaluating property 'value' for java.util.ArrayList, Reason: groovy.lang.MissingPropertyException: No such property: value for class: java.lang.String

Downgrading to 1.32.3 resolved the problem.

Cloud Provider(s):

AWS ECS

Environment:

Tested in two environments:

  • production env running on GKE
  • local dev environment running from localgit

Feature Area:

Pipeline stage “Deploy” in combination with a “Find Image from Tags” stage

Description:

This is the complete stacktrace from the execution:

groovy.lang.MissingPropertyException: Exception evaluating property 'value' for java.util.ArrayList, Reason: groovy.lang.MissingPropertyException: No such property: value for class: java.lang.String
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.getAtIterable(DefaultGroovyMethods.java:8760)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.getAt(DefaultGroovyMethods.java:8748)
	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:2055)
	at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:329)
	at com.netflix.spinnaker.orca.clouddriver.tasks.providers.ecs.EcsServerGroupCreator.getOperations(EcsServerGroupCreator.groovy:125)
	at com.netflix.spinnaker.orca.clouddriver.tasks.servergroup.ServerGroupCreator.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
	at com.netflix.spinnaker.orca.clouddriver.tasks.servergroup.CreateServerGroupTask.execute(CreateServerGroupTask.groovy:55)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:166)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:122)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withLoggingContext(RunTaskHandler.kt:473)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.access(RunTaskHandler.kt:89)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:122)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:121)
	at com.netflix.spinnaker.orca.q.handler.AuthenticationAware.withAuth-0(AuthenticationAware.kt:51)
	at com.netflix.spinnaker.security.AuthenticatedRequest.lambda-zsh(AuthenticatedRequest.java:272)
	at com.netflix.spinnaker.orca.q.handler.AuthenticationAware.withAuth(AuthenticationAware.kt:51)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withAuth(RunTaskHandler.kt:89)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:121)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:119)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:292)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:281)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:69)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:61)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:86)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:75)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.withExecution(OrcaMessageHandler.kt:96)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withExecution(RunTaskHandler.kt:89)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.withStage(OrcaMessageHandler.kt:75)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withStage(RunTaskHandler.kt:89)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.withTask(OrcaMessageHandler.kt:61)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:89)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:281)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle-0(RunTaskHandler.kt:119)
	at com.netflix.spinnaker.orca.lock.NoOpRunOnLockAcquired.execute(ExternalLock.kt:191)
	at com.netflix.spinnaker.orca.lock.RetriableLock.get(RetriableLock.java:128)
	at com.netflix.spinnaker.orca.lock.RetriableLock.get(RetriableLock.java:103)
	at com.netflix.spinnaker.kork.core.RetrySupport.retry(RetrySupport.java:34)
	at com.netflix.spinnaker.orca.lock.RetriableLock.lock(RetriableLock.java:57)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withLocking(RunTaskHandler.kt:298)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:118)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:89)
	at com.netflix.spinnaker.q.MessageHandler.invoke(MessageHandler.kt:36)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:46)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:89)
	at com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:72)
	at com.netflix.spinnaker.q.QueueProcessor.invoke-0(QueueProcessor.kt:90)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

So, something is happening in orca in EcsServerGroupCreator.groovy line 125 I thought. That’s where a Deploy stage with image referenced from a previous Find Image from Tags stage tries to get the imageId. I searched if the code changed, but there weren’t any changes in the last 5 years.

My next thought was maybe it’s related to Java 17 upgrade of orca.

I tested multiple scenarios locally:

  • version 1.32.3, all services running Java 11: all fine, no exception
  • version 1.33.0, all services running Java 17: exception in orca for ECS create server group
  • version 1.33.0, orca running Java 11, rest running Java 11: all fine, no exception
  • version 1.33.0, all services running Java 11: all fine, no exception

Steps to Reproduce:

  • Start Spinnaker 1.33.0 with all services running Java 17
  • create a pipeline with the following stages
    • Find Image from Tags
    • Deploy to ECS server group
      • container mapping should reference the previous stage output
  • Run the pipeline, it will fail the Deploy stage after 1 second with an exception

Additional Details:

It only fails in combination with the Find Image from Tags stage reference. Single Deploy stages with fixed container mappings are working.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 20 (19 by maintainers)

Most upvoted comments

The fix has been merged and backported in 1.36.x, 1.35.x and 1.34.x

Just to note, I too am experiencing this same issue, If I have a stage with find image tag for a docker image, I get the same error. But if I disable the stage and use hardcoded values, I can deploy as usual.

spinnaker version 1.34.3 image in aws ecr