opensearch-k8s-operator: Masters won't start if using security config

Hi,

We need to migrate one cluster, and while testing we are having issues with setting security config. Every time we try to set up security config on operator, the master nodes won’t start.

Error on master is:

022-08-03T21:14:21,966][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:33126
[2022-08-03T21:14:45,194][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:40054
[2022-08-03T21:14:52,015][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:53602
[2022-08-03T21:15:22,037][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:40324
[2022-08-03T21:15:49,810][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:42414
[2022-08-03T21:15:52,007][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:42418
[2022-08-03T21:16:22,027][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:48282

The environment variables OPENSEARCH_USER and OPENSEARCH_PASSWORD are being set up correctly with the created password, which is the same as the one setup on the security config secret.

But the security config directory inside master is using default config, not the one on the secret.

"security": {
      "config": {
          "securityConfigSecret": {
              "name": "opensearch-cluster-securityconfig-secret",
          },
          "adminCredentialsSecret": {
              "name": "opensearch-cluster-admin-secret",
          }
      }
  }

{"apiVersion":"opensearch.opster.io/v1","kind":"OpenSearchCluster","metadata":{"labels":{"app.kubernetes.io/managed-by":"pulumi"},"name":"opensearch-cluster","namespace":"opensearch-system-f92c6ffc"},"spec":{"confMgmt":{"smartScaler":true},"dashboards":{"enable":true,"replicas":1,"resources":{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"500m","memory":"1Gi"}},"version":"2.1.0"},"general":{"serviceName":"opensearch-cluster","version":"2.1.0"},"nodePools":[{"component":"masters","diskSize":"20Gi","persistence":{"pvc":{"accessModes":["ReadWriteOnce"],"storageClass":"gp2-retain"}},"replicas":3,"resources":{"limits":{"cpu":"1","memory":"3Gi"},"requests":{"cpu":"500m","memory":"2Gi"}},"roles":["master","data"]}],"security":{"config":{"adminCredentialsSecret":{"name":"opensearch-cluster-admin-secret"},"securityConfigSecret":{"name":"opensearch-cluster-securityconfig-secret"}}}}}

Operator is running on its namespace, and cluster and secrets isolated on same namespace.

No pod with name opensearch-cluster-securityconfig-update was found.

On controller we could find this:

1.659562337162273e+09 INFO controller.opensearchcluster No security specified. Not doing anything {"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster", "name": "opensearch-cluster", "namespace": "opensearch-system-f92c6ffc"}

This is not the true, security is specified, the new admin and user was injected correctly.

We are using Operator 2.0, v2.0-ARM image (maybe docs should be more explicit about this image tag, it was hard to found about it 😄 ).

Would you mind helping us with this? We are just trying to set up securityconfig to finish this cluster migration.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 19

Most upvoted comments

Hi @danielbichuetti

Apologies on the delay here, but I wanted to let you know we got it working.

For anyone who might be coming across this with a similar issue, the b64 encoding was automatically being applied on the secret by pulumi, so it wasn’t needed. There was also a misunderstanding in that I thought Daniel had been saying the credentials for the opensearch admin and opensearch dashboard user had to be identical due to operator expectations. This is not the case.

Thanks again for all the help and have a great day.

d-maki on Aug 16, 2022

I would like to say that, in my humble perspective, the project should focus a bit more into better docs. They are a bit poor and some parts outdated. For example, setting dashboard user secret could be found only reading the code itself.

This would help project growth. Usually, who looks for an Operator wants an easier life. Not being tortured by lack of documentation. Which the OpenSearch helm charts are much better.

After struggling for some days with the Operator, we are now making usage of it. Soon, we will publish a post about it.

Have a great day!

danielbichuetti on Aug 6, 2022