digdag: Parallel loop failing with java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;

Hi, trying to run a parallel loop using Digdag latest version, and running into a cryptical Java error. Here’s the error trace:

io.digdag.core.agent.OperatorManager: Task failed with unexpected error: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
	at io.digdag.core.database.AESGCMSecretCrypto.decryptSecret(AESGCMSecretCrypto.java:173)
	at io.digdag.core.database.DatabaseSecretStore.getSecret(DatabaseSecretStore.java:49)
	at io.digdag.core.agent.GrantedPrivilegedVariables.lambda$privilegedSecretProvider$0(GrantedPrivilegedVariables.java:25)
	at io.digdag.spi.SecretProvider.getSecret(SecretProvider.java:20)
	at io.digdag.util.UserSecretTemplate.lambda$format$0(UserSecretTemplate.java:55)
	at io.digdag.util.UserSecretTemplate.replaceAll(UserSecretTemplate.java:65)
	at io.digdag.util.UserSecretTemplate.format(UserSecretTemplate.java:55)
	at io.digdag.core.agent.GrantedPrivilegedVariables.lambda$buildAccessor$1(GrantedPrivilegedVariables.java:62)
	at io.digdag.core.agent.GrantedPrivilegedVariables.get(GrantedPrivilegedVariables.java:81)
	at io.digdag.util.CommandOperators.collectEnvironmentVariables(CommandOperators.java:18)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runCommand(PyOperatorFactory.java:296)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runCode(PyOperatorFactory.java:159)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runTask(PyOperatorFactory.java:115)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:365)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:298)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:151)
	at io.digdag.core.agent.ExtractArchiveWorkspaceManager.withExtractedArchive(ExtractArchiveWorkspaceManager.java:77)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:149)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:132)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.MultiThreadAgent.lambda$run$0(MultiThreadAgent.java:132)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Any help would be very appreciated. Thank you!

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 21 (17 by maintainers)

Most upvoted comments

Hi, @FredericoCoelhoNunes Thank you for your comment. @szyn I confirmed this issue has been fixed in Digdag 0.10.2. Thanks!

hiroyuki-sato on Aug 12, 2021

Maybe we’d better explicitly add --release 8 option to javac like this

Thanks, that’s good to know, and I agree with you.

But, of course, the easiest fix is to build with Java 8 and release as 0.10.2 😉

I think so too. Let me build with Java 8 and release it as 0.10.2 later.

szyn on Jul 21, 2021

Maybe we’d better explicitly add --release 8 option to javac like this

diff --git a/build.gradle b/build.gradle
index 240042c7a8..6167dc0c04 100644
--- a/build.gradle
+++ b/build.gradle
@@ -55,7 +55,10 @@ subprojects {
     sourceCompatibility = 1.8
     targetCompatibility = 1.8

-    compileJava.options.encoding = 'UTF-8'
+    compileJava {
+        options.encoding = 'UTF-8'
+        options.compilerArgs.addAll(['--release', '8'])
+    }
     compileTestJava.options.encoding = 'UTF-8'

     dependencies {

With this option, I confirmed this issue isn’t reproduced even with the combination of Java 11 for build and Java 8 for execution.

But, of course, the easiest fix is to build with Java 8 and release as 0.10.2 😉

FYI: latest Gradle has release option https://docs.gradle.org/current/dsl/org.gradle.api.tasks.compile.CompileOptions.html#org.gradle.api.tasks.compile.CompileOptions:release

komamitsu on Jul 21, 2021

@fernandomrtnz Thanks for reporting this issue! And @hiroyuki-sato Thanks for your support as always 😉

I also reproduced this issue with 0.10.1 while didn’t reproduce with 0.10.0. Let me look into it.

komamitsu on Jul 21, 2021