OpenSearch: [BUG] Cannot update cluster settings after upgrading to 3.0.0
Describe the bug
After upgrading our clusters from 2.19.2 to version 3.0.0, we cannot do any PUT requests to _cluster/settings as it results with an error message:
{
"error": {
"root_cause": [
{
"type": "settings_exception",
"reason": "unknown setting [archived.plugins.index_state_management.metadata_migration.status] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"suppressed": [
{
"type": "settings_exception",
"reason": "unknown setting [archived.plugins.index_state_management.template_migration.control] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
}
]
}
],
"type": "settings_exception",
"reason": "unknown setting [archived.plugins.index_state_management.metadata_migration.status] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"suppressed": [
{
"type": "settings_exception",
"reason": "unknown setting [archived.plugins.index_state_management.template_migration.control] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
}
]
},
"status": 400
}
When using GET _cluster/settings, I can see these archived settings but I cannot remove these settings in any way:
"archived": {
"plugins": {
"index_state_management": {
"metadata_migration": {
"status": "1"
},
"template_migration": {
"control": "-1"
}
}
}
}
Trying to set them to null:
PUT _cluster/settings
{
"persistent":{
"archived.plugins.index_state_management.metadata_migration.status": null,
"archived.plugins.index_state_management.template_migration.status": null
}
}
{
"error": {
"root_cause": [
{
"type": "settings_exception",
"reason": "unknown setting [archived.plugins.index_state_management.template_migration.control] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
}
],
"type": "settings_exception",
"reason": "unknown setting [archived.plugins.index_state_management.template_migration.control] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
},
"status": 400
}
And now we are stuck as we cannot reinstall opensearch-index-management temporarily to get rid of them. How can we resolve this issue? Would it be possible to make the archived settings validation more lax or offer a way to get rid of these settings?
Related component
Plugins
To Reproduce
Not sure how to reproduce exactly - I just know that we had indexes with ISM enabled in 2.19.2, then we upgraded to 3.0.0 (also, migrated from indexes to data streams during the upgrade) and now we cannot change cluster settings due to this bug.
Expected behavior
Any cluster setting updates/changes are successful.
Additional Details
Plugins [opensearch@opensearch-cluster-master-0 ~]$ opensearch-plugin list opensearch-alerting opensearch-anomaly-detection opensearch-asynchronous-search opensearch-cross-cluster-replication opensearch-custom-codecs opensearch-flow-framework opensearch-geospatial opensearch-job-scheduler opensearch-knn opensearch-ltr opensearch-ml opensearch-neural-search opensearch-notifications opensearch-notifications-core opensearch-observability opensearch-performance-analyzer opensearch-reports-scheduler opensearch-security opensearch-security-analytics opensearch-skills opensearch-sql opensearch-system-templates prometheus-exporter query-insights repository-azure
Screenshots
Host/Environment (please complete the following information):
- OS: Azure AKS cluster with Ubuntu 22.04.5 LTS nodes
- Version: 3.0.0
Additional context Can provide additional information if needed.
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 1
- Comments: 16 (9 by maintainers)
@taaniel Apologies, I missed the detail that you attempted to delete them per the workaround.
It looks like your delete command had a small mistake. It should be “control” on the second setting instead of “status”. Can you try this command?
I’m curious about how exactly you got to this situation. Did you remove the index management plugin on the 3.0 version when you did the upgrade?