spinnaker: ECS create server group is throwing exception because of problem with Orca running with Java 17
Issue Summary:
We wanted to upgrade to 1.33.0 and ECS deployments started failing with the an exception:
Exception ( Create Server Group )
Exception evaluating property 'value' for java.util.ArrayList, Reason: groovy.lang.MissingPropertyException: No such property: value for class: java.lang.String
Downgrading to 1.32.3 resolved the problem.
Cloud Provider(s):
AWS ECS
Environment:
Tested in two environments:
- production env running on GKE
- local dev environment running from localgit
Feature Area:
Pipeline stage “Deploy” in combination with a “Find Image from Tags” stage
Description:
This is the complete stacktrace from the execution:
groovy.lang.MissingPropertyException: Exception evaluating property 'value' for java.util.ArrayList, Reason: groovy.lang.MissingPropertyException: No such property: value for class: java.lang.String
at org.codehaus.groovy.runtime.DefaultGroovyMethods.getAtIterable(DefaultGroovyMethods.java:8760)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.getAt(DefaultGroovyMethods.java:8748)
at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:2055)
at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:329)
at com.netflix.spinnaker.orca.clouddriver.tasks.providers.ecs.EcsServerGroupCreator.getOperations(EcsServerGroupCreator.groovy:125)
at com.netflix.spinnaker.orca.clouddriver.tasks.servergroup.ServerGroupCreator.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
at com.netflix.spinnaker.orca.clouddriver.tasks.servergroup.CreateServerGroupTask.execute(CreateServerGroupTask.groovy:55)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:166)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:122)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withLoggingContext(RunTaskHandler.kt:473)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.access(RunTaskHandler.kt:89)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:122)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:121)
at com.netflix.spinnaker.orca.q.handler.AuthenticationAware.withAuth-0(AuthenticationAware.kt:51)
at com.netflix.spinnaker.security.AuthenticatedRequest.lambda-zsh(AuthenticatedRequest.java:272)
at com.netflix.spinnaker.orca.q.handler.AuthenticationAware.withAuth(AuthenticationAware.kt:51)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withAuth(RunTaskHandler.kt:89)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:121)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:119)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:292)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:281)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:69)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:61)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:86)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:75)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.withExecution(OrcaMessageHandler.kt:96)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withExecution(RunTaskHandler.kt:89)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.withStage(OrcaMessageHandler.kt:75)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withStage(RunTaskHandler.kt:89)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.withTask(OrcaMessageHandler.kt:61)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:89)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:281)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle-0(RunTaskHandler.kt:119)
at com.netflix.spinnaker.orca.lock.NoOpRunOnLockAcquired.execute(ExternalLock.kt:191)
at com.netflix.spinnaker.orca.lock.RetriableLock.get(RetriableLock.java:128)
at com.netflix.spinnaker.orca.lock.RetriableLock.get(RetriableLock.java:103)
at com.netflix.spinnaker.kork.core.RetrySupport.retry(RetrySupport.java:34)
at com.netflix.spinnaker.orca.lock.RetriableLock.lock(RetriableLock.java:57)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withLocking(RunTaskHandler.kt:298)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:118)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:89)
at com.netflix.spinnaker.q.MessageHandler.invoke(MessageHandler.kt:36)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler.invoke(OrcaMessageHandler.kt:46)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:89)
at com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:72)
at com.netflix.spinnaker.q.QueueProcessor.invoke-0(QueueProcessor.kt:90)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
So, something is happening in orca in EcsServerGroupCreator.groovy line 125 I thought. That’s where a Deploy stage with image referenced from a previous Find Image from Tags stage tries to get the imageId.
I searched if the code changed, but there weren’t any changes in the last 5 years.
My next thought was maybe it’s related to Java 17 upgrade of orca.
I tested multiple scenarios locally:
- version 1.32.3, all services running Java 11: all fine, no exception
- version 1.33.0, all services running Java 17: exception in orca for ECS create server group
- version 1.33.0, orca running Java 11, rest running Java 11: all fine, no exception
- version 1.33.0, all services running Java 11: all fine, no exception
Steps to Reproduce:
- Start Spinnaker 1.33.0 with all services running Java 17
- create a pipeline with the following stages
- Find Image from Tags
- Deploy to ECS server group
- container mapping should reference the previous stage output
- Run the pipeline, it will fail the Deploy stage after 1 second with an exception
Additional Details:
It only fails in combination with the Find Image from Tags stage reference.
Single Deploy stages with fixed container mappings are working.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 20 (19 by maintainers)
The fix has been merged and backported in 1.36.x, 1.35.x and 1.34.x
Just to note, I too am experiencing this same issue, If I have a stage with find image tag for a docker image, I get the same error. But if I disable the stage and use hardcoded values, I can deploy as usual.
spinnaker version 1.34.3 image in aws ecr