charts: artifactory-ha cannot find join key during migration from 6.x to 7.x

Is this a request for help?: Yes

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes: helm: v2.11.0 Kubernetes: 1.14 (eks)

Which chart: starting from 2.0.0 till 2.4.4

What happened: I am trying to migrate artifactory-ha from version 6.16 (chart 1.3.6) to 7.x.x. (chart 2.x.x) It block on the initConainer : migration-artifactory-ha, on the logs I see :

[main] - Bootstrap joinKey found in [/opt/jfrog/artifactory/var/bootstrap/access/etc/security/join.key:]. Deleting original

then

[INFO ] [7751b16fcf75824 ] [security_keys.go:128          ] [main                ] - Cluster join: Join key is missing. Pending for 15 seconds with 2m0s timeout
.
.
.
 [ERROR] [b8eb8c79386c4388] [tifactoryHomeConfigListener:59] [ocalhost-startStop-2] - Failed initializing Home. Failed to init db properties:
[ERROR] [b8eb8c79386c4388] [actoryContextConfigListener:82] [ocalhost-startStop-2] - Failed initializing Artifactory context: Artifactory home not initialized.
[SEVERE] [                ] [org.apache.catalina.core.StandardContext] [org.apache.catalina.core.StandardContext startInternal] - One or more listeners failed to start. Full details will be found in the appropriate container log file
[SEVERE] [                ] [org.apache.catalina.core.StandardContext] [org.apache.catalina.core.StandardContext startInternal] - Context [/artifactory] startup failed due to previous errors
[INFO ] [7751b16fcf75824 ] [security_keys.go:128          ] [main                ] - Cluster join: Join key is missing. Pending for 55 seconds with 2m0s timeout

[ERROR] [606175f05af5d63d] [o.s.b.SpringApplication:822   ] [ocalhost-startStop-1] - Application run failed
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.boot.autoconfigure.jersey.JerseyAutoConfiguration': Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jerseyConfig' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-application-5.3.9.jar!/org/jfrog/access/rest/config/JerseyConfig.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'httpServiceImpl' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/service/http/HttpServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'internalConfigurationServiceImpl' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/service/configuration/InternalConfigurationServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessConfigurationStorageServiceImpl' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/service/db/AccessConfigurationStorageServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessConfigsDao' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/db/dao/AccessConfigsDao.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessJdbcHelperImpl' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/db/util/AccessJdbcHelperImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbProperties' defined in class path resource [org/jfrog/access/server/db/spring/DbConfigFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.jfrog.storage.DbProperties]: Factory method 'dbProperties' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'internalAccessHome' defined in class path resource [org/jfrog/access/server/home/InternalAccessHomeConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.jfrog.access.server.home.InternalAccessHome]: Factory method 'internalAccessHome' threw exception; nested exception is java.lang.IllegalStateException: Failed to init db properties:
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:156)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:743)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:390)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
        at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:151)
        at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:131)
        at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:91)
        at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:172)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:614)
        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1823)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.boot.autoconfigure.jersey.JerseyAutoConfiguration': Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jerseyConfig' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-application-5.3.9.jar!/org/jfrog/access/rest/config/JerseyConfig.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'httpServiceImpl' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/service/http/HttpServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'internalConfigurationServiceImpl' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/service/configuration/InternalConfigurationServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessConfigurationStorageServiceImpl' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/service/db/AccessConfigurationStorageServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessConfigsDao' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/db/dao/AccessConfigsDao.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessJdbcHelperImpl' defined in URL [jar:file:/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access/WEB-INF/lib/access-server-core-5.3.9.jar!/org/jfrog/access/server/db/util/AccessJdbcHelperImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbProperties' defined in class path resource [org/jfrog/access/server/db/spring/DbConfigFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.jfrog.storage.DbProperties]: Factory method 'dbProperties' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'internalAccessHome' defined in class path resource [org/jfrog/access/server/home/InternalAccessHomeConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.jfrog.access.server.home.InternalAccessHome]: Factory method 'internalAccessHome' threw exception; nested exception is java.lang.IllegalStateException: Failed to init db properties:
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:769)

.
.
.
[FATAL] [2f98cd3358259ccf] [bootstrap.go:92               ] [main                ] - Cluster join: Failed resolving join key: failed resolving 'shared.security.joinKey' key; file does not exist: /opt/jfrog/artifactory/var/etc/security/join.key
Waiting for Artifactory to start --- sleeping for 10 seconds
Waiting for Artifactory to start --- sleeping for 10 seconds
Waiting for Artifactory to start --- sleeping for 10 seconds

and to be clear, in my values files, i’ve commented the masterKey. then I added a join key.

What you expected to happen: migration to be OK.

How to reproduce it (as minimally and precisely as possible):

1. install chart 1.3.6 with a masterKey
2. edit values file, remove masterKey declaration and add a joinKey
3. install chart 2.4.4

Anything else we need to know: why artifactory found the join key in /opt/jfrog/artifactory/var/bootstrap/access/etc/security/join.key then deletes it ? Why join key and master key are written to different paths : https://github.com/jfrog/charts/blob/master/stable/artifactory-ha/templates/artifactory-primary-statefulset.yaml#L167 ?

I’ve seen #674 but it does not help