aws-sdk-java-v2: Java SDK does not support EKS IAM for service accounts

EKS IAM Service Account Role introduces a new environment variable “AWS_WEB_IDENTITY_TOKEN_FILE” and based on the documentation on these two pages, the Java SDK should use “AWS_WEB_IDENTITY_TOKEN_FILE” for credentials if exists.

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#pod-configuration

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

I have tried the latest Java SDK 2.9.19 and it doesn’t seem to work. There is a similar discussion on DotNet SDK at here:

https://github.com/aws/aws-sdk-net/issues/1413

I couldn’t find docs saying “AWS_WEB_IDENTITY_TOKEN_FILE” is in the list of credentials chain for SDK Java 2. I am wondering if this is implemented or not.

Expected Behavior

Based on the EKS doc link above, Java SDK should recognize the environment variable “AWS_WEB_IDENTITY_TOKEN_FILE” and use it to call AssumeRoleWithWebIdentity for access/secret/session tokens.

Current Behavior

I am getting Access Denied with Java SDK in a correctly setup EKS pod with service account. In the same pod, I was able to run aws s3 ls and it worked, which means the token is correct.

Exception in thread "main" software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: 467C66AAEAD354A9)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleErrorResponse(HandleResponseStage.java:115)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleResponse(HandleResponseStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:58)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:41)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:113)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:86)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:240)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:96)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:120)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.s3.DefaultS3Client.listBuckets(DefaultS3Client.java:2035)
        at com.example.s3.S3BucketOps.main(S3BucketOps.java:63)

Steps to Reproduce (for bugs)

The code I am using to test is from sample S3 code at here: https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3BucketOps.java

I set up a pod in EKS with service account and IAM role properly setup. In the pod, I used aws cli to test the token and it worked. However, the Java SDK didn’t work.

Your Environment

  • AWS Java SDK version used: 2.9.19
  • JDK version used: 1.8
  • Operating System and version: Linux running from a container (pod on EKS)

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 13
  • Comments: 20 (2 by maintainers)

Commits related to this issue

Most upvoted comments

@endre-synnes The issue is due to WebIdentityTokenFileCredentialsProvider is not in the default credentials provider chain

The workaround for now is to specify the provider in the client initialization:

import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider;
...
S3Client s3 = S3Client.builder()
                              .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create())
                              .region(region)
                              .build();

Looking forward that WebIdentityTokenFileCredentialsProvider to be added to the default credentials provider chain.

+19
starchx on Oct 18, 2019

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

+11
thereisnospoon on Mar 19, 2020

Hi, It still doesn’t work for me with 2.10.72. It seems that java sdk doesn’t get credentials from AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN envs. (FYI, I already added security context to pods to be able to access the token file)

+8
imcheck on Mar 19, 2020

Hi, We’re still on sdk v1, anyone who’s able to workaround this on v1? Using the following dependencies:

  • aws-java-sdk-core (v1.12.239)
  • aws-java-sdk-sns (v1.11.125)
+7
itowk on Aug 1, 2022

Hi all, this should be part of the default chain. We will prepare a change for this.

+7
dagnir on Oct 23, 2019

Hi Jiang,

Could you please let me the code snippet you have used with StsWebIdentityCredentialsProviderFactory.java

S3Client s3 = S3Client.builder()
                              .region(region)
                              .build();

and include sts with the latest version from maven repo, today it is 2.11.14

		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>sts</artifactId>
		</dependency>
+6
jiang-wei on Apr 20, 2020

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

Not really.

When I try to run s3 client, it says WebIdentityCredentialProvider needs sts been loaded into class path So I searched and get StsWebIdentityCredentialsProviderFactory.java

Then I included

		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>sts</artifactId>
		</dependency>

It just works 😃

+6
jiang-wei on Apr 20, 2020

still happens

SDK version: software.amazon.awssdk:ssm:2.17.19
JDK version used: Java 11
Operating System and version: openjdk:11.0.11-jdk-slim
EKS version: 1.21

only after I added software.amazon.awssdk:sts:2.17.19 fixed

+4
llnformer on Aug 17, 2021

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

Not really.

When I try to run s3 client, it says WebIdentityCredentialProvider needs sts been loaded into class path So I searched and get StsWebIdentityCredentialsProviderFactory.java

Then I included

		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>sts</artifactId>
		</dependency>

It just works 😃

Surprised that this works, but it does. Any idea why the separate declaration of sts is required?

+4
rod-dvla on Dec 9, 2020

Having the same issue using the AWS Java SDK version: 1.11.653.

  • JDK version used: 11
  • Operating System and version: Linux running from a container (pod on EKS)
  • Kubernetes version 1.14

According to the documentation this version of the SDK should also work fine. My application uses the role of the node instead of the one I have added through a service account.

+4
endre-synnes on Oct 17, 2019

This is an interesting discussion and somewhat related to a use case I am trying to figure out. My service needs to access resources in a different AWS account from EKS, so I want to use profiles and assume role using a config like this:

[profile eks-account-role]
role_arn = <eks-account-role-arn>
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token

[profile other-account-role]
role_arn = <other-account-role-arn>
source_profile = eks-account-role
role_session_name = x-account-session

Using the following code to access the STS client (for other purposes than assuming roles).

stsClient = StsClient.builder().build();

Then I set the environment variable AWS_PROFILE=other-account-role in the container.

But when I go to run it, the service is still running as the eks-account-role. I’m assuming the default credential provider chain will pick up the AWS profile defined in the config and use that method for authentication, but it doesn’t seem to be happening. However, when I try this with the AWS CLI it works as expected (I have properly setup the other role to be assumed by the eks role).

I have found a workaround where if I specify to use profile credentials explicitly it works as expected.

stsClient = StsClient.builder()
                .credentialsProvider(ProfileCredentialsProvider.create())
                .build();

Now this isn’t a big deal as it is a small amount of additional code, but I am curious what the SDK is doing under the hood and why it is different from the CLI. A couple things to consider, EKS seems to be injecting two environment variables into the container that could be effecting the behavior AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. However, I am running the CLI from within the container with the same variables present.

Thoughts?

+2
campbel on Jun 10, 2020

@endre-synnes I am also experiencing the same problem despite the documentation saying it should work. We should file an issue at https://github.com/aws/aws-sdk-java/issues

+1
thundergolfer on Oct 29, 2019
Read more comments on GitHub
← aws-sdk-java-v2: First DynamoDB request is very slow
aws-sdk-java-v2: LambdaConversionException: Invalid receiver type interface org.apache.http.Header; not a subtype of implementation type interface org.apache.http.NameValuePair →