security: [BUG] Securityadmin error: exits with node reported failures
What is the bug?
Executing /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh throws error as FAIL: Expected 2 nodes to return response, but got 0
Full error
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755 **
**************************************************************************
Security Admin v7
Will connect to my-first-cluster.default.svc.cluster.local:9200 ... done
Connected as "CN=admin,OU=my-first-cluster"
OpenSearch Version: 2.0.1
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: my-first-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Populate config from /usr/share/opensearch/config/opensearch-security/
Will update '/config' with /usr/share/opensearch/config/opensearch-security/config.yml (legacy mode)
SUCC: Configuration for 'config' created or updated
Will update '/roles' with /usr/share/opensearch/config/opensearch-security/roles.yml (legacy mode)
SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml (legacy mode)
SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml (legacy mode)
SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml (legacy mode)
SUCC: Configuration for 'actiongroups' created or updated
Will update '/nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml (legacy mode)
SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml (legacy mode)
SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /usr/share/opensearch/config/opensearch-security/audit.yml (legacy mode)
SUCC: Configuration for 'audit' created or updated
FAIL: 2 nodes reported failures. Failure is /{"_nodes":{"total":2,"successful":0,"failed":2,"failures":[{"type":"failed_node_exception","reason":"Failed node [E_Dyk7VUR_ee4wykVYJSoA]","node_id":"E_Dyk7VUR_ee4wykVYJSoA","caused_by":{"type":"static_resource_exception","reason":"static_resource_exception: Unable to load static tenants"}},{"type":"failed_node_exception","reason":"Failed node [G4U098vuRCGF8RTI3KPRPA]","node_id":"G4U098vuRCGF8RTI3KPRPA","caused_by":{"type":"static_resource_exception","reason":"Unable to load static tenants"}}]},"cluster_name":"my-first-cluster","configupdate_response":{"nodes":{},"node_size":0,"has_failures":true,"failures_size":2}}
FAIL: Expected 2 nodes to return response, but got 0
Done with failures
How can one reproduce the bug?
Start the docker container with some persistence storage and when executed /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh throws this error.
What is the expected behavior? Executing Securityadmin script should create an security index as expected, when it works logs successful message as
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755 **
**************************************************************************
Security Admin v7
Will connect to my-first-cluster.default.svc.cluster.local:9200 ... done
Connected as "CN=admin,OU=my-first-cluster"
OpenSearch Version: 2.0.1
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: my-first-cluster
Clusterstate: YELLOW
Number of nodes: 2
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Populate config from /usr/share/opensearch/config/opensearch-security/
Will update '/config' with /usr/share/opensearch/config/opensearch-security/config.yml (legacy mode)
SUCC: Configuration for 'config' created or updated
Will update '/roles' with /usr/share/opensearch/config/opensearch-security/roles.yml (legacy mode)
SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml (legacy mode)
SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml (legacy mode)
SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml (legacy mode)
SUCC: Configuration for 'actiongroups' created or updated
Will update '/nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml (legacy mode)
SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml (legacy mode)
SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /usr/share/opensearch/config/opensearch-security/audit.yml (legacy mode)
SUCC: Configuration for 'audit' created or updated
SUCC: Expected 7 config types for node {"updated_config_types":["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"],"updated_config_size":7,"message":null} is 7 (["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"]) due to: null
SUCC: Expected 7 config types for node {"updated_config_types":["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"],"updated_config_size":7,"message":null} is 7 (["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"]) due to: null
Done with success
What is your host/environment?
- OS: 2.0.1
- Version [e.g. 22]
- Plugins: Docker container
docker.io/opensearchproject/opensearch:2.0.1
Do you have any additional context?
Following the issue in past https://github.com/opensearch-project/helm-charts/issues/158, this was not resolved with config_version: 2 in action_groups.yml, is there a co-relation with config_version: 2?
This issue is raised to help OpenSearch Kubernetes Operator compatible with 2.0.0 series of OpenSearch. https://github.com/Opster/opensearch-k8s-operator/issues/176
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 33 (22 by maintainers)
@prudhvigodithi Thanks for reviewing the issue, to me the key part that needs to be followed up on is:
It looks like it might be useful to see this to a larger timeout for environments when the cluster is slower to start up, whereas today its fixed at the default of 30 seconds.
[TRIAGE] @peternied can you follow up with this issue to make sure the issue remains. Thank you.
Actually I was about to update this. It’s possible that this specific node was installed with 2.1.0, and right after upgraded to 2.2.0. I did follow the logs, but the only exception I noticed was “Unable to load static tenants”. Having said that, I removed its datadir and afterwards everything ran just fine. So in this case, that was one way of solving the problem…