security: [BUG] Security plugin transport layer(multi-node cluster) does not work due to missing required reflective access (JDK17)

Describe the bug Security plugin 1.3.0.0 does not work on OpenSearch 1.3.0 multi-node clusters without required reflective access. This issue was originated from: https://github.com/opensearch-project/security/issues/1619

To Reproduce Steps to reproduce the behavior:

1. Create a OpenSearch 1.3.0 cluster with at least 2 nodes.
2. Install security plugin 1.3.0.0(security plugin repo main branch) on each of the nodes.
3. Send a basic request(such as _cat/nodes or _cat/indices) to the OpenSearch cluster.
4. See error:

% curl -k -XGET -u admin:admin https://localhost:9200/_cat/indices
{"error":{"root_cause":[{"type":"security_exception","reason":"Unexpected exception indices:monitor/stats"}],"type":"security_exception","reason":"Unexpected exception indices:monitor/stats"},"status":500}% 

Stacktrace in logs:

org.opensearch.OpenSearchSecurityException: Unexpected exception cluster:monitor/nodes/info
         at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:376) [opensearch-security-1.3.0.0.jar:1.3.0.0]
         at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.3.0.0.jar:1.3.0.0]
         at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.3.0.0.jar:1.3.0.0]
         at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.client.support.AbstractClient$ClusterAdmin.nodesInfo(AbstractClient.java:813) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.rest.action.cat.RestNodesAction$1.processResponse(RestNodesAction.java:127) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.rest.action.cat.RestNodesAction$1.processResponse(RestNodesAction.java:115) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.rest.action.RestActionListener.onResponse(RestActionListener.java:60) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:103) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:97) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.action.ActionListener$2.onResponse(ActionListener.java:104) [opensearch-1.3.0.jar:1.3.0]
         at org.opensearch.action.ActionListener.completeWith(ActionListener.java:351) [opensearch-1.3.0.jar:1.3.0]
 

Digging into the reason of the error: 1.

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "Unexpected exception cluster:monitor/nodes/info"
      }
    ],
    "type": "security_exception",
    "reason": "Unexpected exception cluster:monitor/nodes/info",
    "caused_by": {
      "type": "exception_in_initializer_error",
      "reason": null,
      "caused_by": {
        "type": "inaccessible_object_exception",
        "reason": "Unable to make field private java.lang.String java.io.ObjectStreamClass.name accessible: module java.base does not \"opens java.io\" to unnamed module @5b0575d0"
      }
    }
  },
  "status": 500
}

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "Unexpected exception cluster:monitor/nodes/info"
      }
    ],
    "type": "security_exception",
    "reason": "Unexpected exception cluster:monitor/nodes/info",
    "caused_by": {
      "type": "no_class_def_found_error",
      "reason": "Could not initialize class org.opensearch.security.support.Base64Helper$DescriptorNameSetter"
    }
  },
  "status": 500
}

Expected behavior Requests gets expected response like the following:

% curl -k -XGET -u admin:admin https://localhost:9200/_cat/indices                                                                                    
green open security-auditlog-2022.02.19 l9rrGwzOR9C4ZWqMyMYHwg 1 1  1 0  23.2kb  11.6kb
green open security-auditlog-2022.02.18 qvL83MJ1RbG1YgOW4MixAQ 1 1 22 0 244.9kb 154.3kb
green open .kibana_1                    peXycy4TSoWTYm8Uh6TemQ 1 1  1 0    10kb     5kb
green open .opendistro_security         3RaQH4lCQuayi6CKLXptWg 1 1  9 0 106.1kb    53kb

Plugins Please list all plugins currently enabled. Security plugin

Host/Environment (please complete the following information):

• OS: [Linux]

Additional context PR#1278 in security plugin requires these reflective access since the changes use reflection. But reverting the PR is not an option since the PR itself supports OpenSearch backward compatibility for ODFE. Without this PR(#1278), security plugin would not work in mixed cluster(ODFE nodes + OpenSearch nodes), which would fail rolling upgrade from ODFE to OpenSearch. In order to fix the issue, JVM Options where OpenSearch cluster runs need to be appended with something like this to grant the reflective access. Security plugin had exactly same error message in Integration Tests, which was resolved by such changes.