OpenSearch: Plugin security permission exception "org.opensearch.secure_sm.ThreadPermission" "modifyArbitraryThreadGroup"

Is your feature request related to a problem? Please describe.

SQL plugin depend on hadoop-fs to access s3. when init S3AFileSystem. SQL plugin get exception "org.opensearch.secure_sm.ThreadPermission" "modifyArbitraryThreadGroup"
Try to add permission in plugin-security.policy, it does not work. I think the reason is additional permissions will only be granted to the jars in SQL plugin
I found the similar issue. but I do not think we can do similar thing.

Describe the solution you’d like Alt1. Plugin could config additional security permissions.

Describe alternatives you’ve considered Alt2. Disable modifyArbitraryThreadGroup check in SecureSM. Alt3. Plugin could override security manager. System.setSecurityManager(new PluginSecureSM()); Plugin can do it, but I am not sure is it the best practice.

Additional context

SQL Plugin: https://github.com/penghuo/os-sql/blob/hp/test/maximus-m1/

=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Mac OS X 11.3 (x86_64)
  JDK Version           : 11 (AdoptOpenJDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
  Random Testing Seed   : A3FC7A76E87A50BA
  In FIPS 140 mode      : false
=======================================

[2022-11-23T00:13:13,051][INFO ][o.o.s.l.p.RestSqlAction  ] [debugTest-0] [a15e8a52-bc72-4cc3-96fb-2d357d57d993] Incoming request /_plugins/_sql: ( SELECT * FROM table LIMIT number )
[2022-11-23T00:13:13,110][WARN ][stderr                   ] [debugTest-0] log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
[2022-11-23T00:13:13,111][WARN ][stderr                   ] [debugTest-0] log4j:WARN Please initialize the log4j system properly.
[2022-11-23T00:13:13,112][WARN ][stderr                   ] [debugTest-0] log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2022-11-23T00:13:14,685][WARN ][stderr                   ] [debugTest-0] access: caller thread=Thread[opensearch[debugTest-0][sql-worker][T#2],5,main]
[2022-11-23T00:13:14,686][WARN ][stderr                   ] [debugTest-0] access: caller group=java.lang.ThreadGroup[name=main,maxpri=10]
[2022-11-23T00:13:14,687][WARN ][stderr                   ] [debugTest-0] access: target group=java.lang.ThreadGroup[name=InnocuousForkJoinWorkerThreadGroup,maxpri=10]
[2022-11-23T00:13:14,741][WARN ][stderr                   ] [debugTest-0] access: caller thread=Thread[opensearch[debugTest-0][sql-worker][T#2],5,main]
[2022-11-23T00:13:14,742][WARN ][stderr                   ] [debugTest-0] access: caller group=java.lang.ThreadGroup[name=main,maxpri=10]
[2022-11-23T00:13:14,742][WARN ][stderr                   ] [debugTest-0] access: target group=java.lang.ThreadGroup[name=InnocuousForkJoinWorkerThreadGroup,maxpri=10]
[2022-11-23T00:13:14,744][ERROR][o.o.s.l.p.RestSqlAction  ] [debugTest-0] a15e8a52-bc72-4cc3-96fb-2d357d57d993 Server side error during query execution
java.security.AccessControlException: access denied ("org.opensearch.secure_sm.ThreadPermission" "modifyArbitraryThreadGroup")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
        at java.security.AccessController.checkPermission(AccessController.java:897) ~[?:?]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:322) ~[?:?]
        at org.opensearch.secure_sm.SecureSM.checkThreadGroupAccess(SecureSM.java:248) ~[opensearch-secure-sm-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at org.opensearch.secure_sm.SecureSM.checkAccess(SecureSM.java:194) ~[opensearch-secure-sm-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:313) ~[?:?]
        at java.lang.Thread.<init>(Thread.java:418) ~[?:?]
        at java.lang.Thread.<init>(Thread.java:704) ~[?:?]
        at java.lang.Thread.<init>(Thread.java:625) ~[?:?]
        at java.util.concurrent.ForkJoinWorkerThread.<init>(ForkJoinWorkerThread.java:119) ~[?:?]
        at java.util.concurrent.ForkJoinWorkerThread$InnocuousForkJoinWorkerThread.<init>(ForkJoinWorkerThread.java:224) ~[?:?]
        at java.util.concurrent.ForkJoinPool$InnocuousForkJoinWorkerThreadFactory$1.run(ForkJoinPool.java:3227) ~[?:?]
        at java.util.concurrent.ForkJoinPool$InnocuousForkJoinWorkerThreadFactory$1.run(ForkJoinPool.java:3225) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.util.concurrent.ForkJoinPool$InnocuousForkJoinWorkerThreadFactory.newThread(ForkJoinPool.java:3224) ~[?:?]
        at java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1328) ~[?:?]
        at java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1352) ~[?:?]
        at java.util.concurrent.ForkJoinPool.signalWork(ForkJoinPool.java:1476) ~[?:?]
        at java.util.concurrent.ForkJoinPool.externalPush(ForkJoinPool.java:1903) ~[?:?]
        at java.util.concurrent.ForkJoinTask.fork(ForkJoinTask.java:704) ~[?:?]
        at java.util.stream.AbstractTask.compute(AbstractTask.java:324) ~[?:?]
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) ~[?:?]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[?:?]
        at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408) ~[?:?]
        at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:919) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.apache.hadoop.fs.statistics.impl.EvaluatingStatisticsMap.entrySet(EvaluatingStatisticsMap.java:166) ~[hadoop-common-3.3.4.jar:?]
        at java.util.Collections$UnmodifiableMap.entrySet(Collections.java:1481) ~[?:?]
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.copyMap(IOStatisticsBinding.java:171) ~[hadoop-common-3.3.4.jar:?]
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.snapshotMap(IOStatisticsBinding.java:215) ~[hadoop-common-3.3.4.jar:?]
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.snapshotMap(IOStatisticsBinding.java:198) ~[hadoop-common-3.3.4.jar:?]
        at org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.snapshot(IOStatisticsSnapshot.java:161) ~[hadoop-common-3.3.4.jar:?]
        at org.apache.hadoop.fs.statistics.IOStatisticsSnapshot.<init>(IOStatisticsSnapshot.java:124) ~[hadoop-common-3.3.4.jar:?]
        at org.apache.hadoop.fs.statistics.IOStatisticsSupport.snapshotIOStatistics(IOStatisticsSupport.java:49) ~[hadoop-common-3.3.4.jar:?]
        at org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.<init>(S3AInstrumentation.java:901) ~[hadoop-aws-3.3.4.jar:?]
        at org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.<init>(S3AInstrumentation.java:786) ~[hadoop-aws-3.3.4.jar:?]
        at org.apache.hadoop.fs.s3a.S3AInstrumentation.newInputStreamStatistics(S3AInstrumentation.java:677) ~[hadoop-aws-3.3.4.jar:?]
        at org.apache.hadoop.fs.s3a.statistics.impl.BondedS3AStatisticsContext.newInputStreamStatistics(BondedS3AStatisticsContext.java:115) ~[hadoop-aws-3.3.4.jar:?]
        at org.apache.hadoop.fs.s3a.S3AInputStream.<init>(S3AInputStream.java:163) ~[hadoop-aws-3.3.4.jar:?]
        at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1498) ~[hadoop-aws-3.3.4.jar:?]
        at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1441) ~[hadoop-aws-3.3.4.jar:?]
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976) ~[hadoop-common-3.3.4.jar:?]
        at org.opensearch.sql.filesystem.storage.fs.FSScanOperator$FSFileReader.lambda$new$0(FSScanOperator.java:160) ~[filesystem-2.4.0.0-SNAPSHOT.jar:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at org.opensearch.sql.common.utils.AccessController.doPrivileged(AccessController.java:14) ~[common-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.filesystem.storage.fs.FSScanOperator$FSFileReader.<init>(FSScanOperator.java:160) ~[filesystem-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.filesystem.storage.fs.FSScanOperator.lambda$open$3(FSScanOperator.java:92) ~[filesystem-2.4.0.0-SNAPSHOT.jar:?]
        at com.google.common.collect.Iterators$6.transform(Iterators.java:826) ~[guava-31.0.1-jre.jar:?]
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:52) ~[guava-31.0.1-jre.jar:?]
        at org.opensearch.sql.filesystem.storage.fs.FSScanOperator.hasNext(FSScanOperator.java:101) ~[filesystem-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.opensearch.executor.protector.ResourceMonitorPlan.hasNext(ResourceMonitorPlan.java:74) ~[opensearch-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.planner.physical.LimitOperator.open(LimitOperator.java:43) ~[core-2.4.0.0-SNAPSHOT.jar:?]
        at java.util.Collections$SingletonList.forEach(Collections.java:4856) ~[?:?]
        at org.opensearch.sql.planner.physical.PhysicalPlan.open(PhysicalPlan.java:33) ~[core-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$0(OpenSearchExecutionEngine.java:46) [opensearch-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.schedule(OpenSearchNodeClient.java:182) [opensearch-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.execute(OpenSearchExecutionEngine.java:40) [opensearch-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.executor.QueryService.lambda$executePlan$1(QueryService.java:66) [core-2.4.0.0-SNAPSHOT.jar:?]
        at java.util.Optional.ifPresentOrElse(Optional.java:203) [?:?]
        at org.opensearch.sql.executor.QueryService.executePlan(QueryService.java:64) [core-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.executor.QueryService.execute(QueryService.java:43) [core-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:50) [core-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:33) [opensearch-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:47) [opensearch-2.4.0.0-SNAPSHOT.jar:?]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]

About this issue

Original URL
State: open
Created 2 years ago
Comments: 15 (4 by maintainers)

Most upvoted comments

Oh I see, I think the way HTTPAuthenticator are instantiated in security plugin, you do not have options to inject additional context. You could try with:

        final ScheduledThreadPoolExecutor scheduler = new ScheduledThreadPoolExecutor(
                1,
                OpenSearchExecutors.daemonThreadFactory(settings, "expiration-scheduler"),
                new OpenSearchAbortPolicy()
            );

reta on Apr 13, 2023

Thanks @reta. I add suggested policy, but still having same issue.

grant {
  // For Spring IOC
  permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
  permission java.lang.RuntimePermission "accessDeclaredMembers";
  permission java.lang.RuntimePermission "defineClass";
  permission java.lang.RuntimePermission "getClassLoader";
  permission java.lang.RuntimePermission "accessUserInformation";
  permission java.net.NetPermission "getProxySelector";
  permission java.net.SocketPermission "*", "accept,connect,resolve";

  // ml-commons client
  permission java.lang.RuntimePermission "setContextClassLoader";

  // filesystem
  permission java.util.PropertyPermission "*", "read,write";
  permission javax.security.auth.AuthPermission "getSubject";
  permission java.security.AllPermission;

  permission org.opensearch.secure_sm.ThreadPermission "modifyArbitraryThreadGroup";
};

grant codeBase "${codebase.hadoop-common}" {
  // rest client uses system properties which gets the default proxy
  permission org.opensearch.secure_sm.ThreadPermission "modifyArbitraryThreadGroup";
  permission java.security.AllPermission;
};

penghuo on Dec 14, 2022