amazon-kinesis-client-python: Problem trying to refresh assume-role credentials automatically with KCL on EC2

I’ve just tested it using the line below (with redactions), and was able to access a stream in another account:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account id>:role/pfifer-cross-account|pfifer-session

I’ve tested this with version 1.11.151, and 1.11.129 of the STS SDK.

To get more information I’ve updated the pom.xml to include some logging libraries. You can use this without a logging configuration file, but it will default to DEBUG. You can configure the logging using these steps:

1. In you application directory create a directory called logback (you can call it something else if you want to).
2. Download this logback.xml file, and save it to logback/logback.xml
3. Start the script again like: CLASSPATH_PREFIX=./logback ./bin/multilang-daemon config/kcl.local.properties

It defaults to logging output to the console, but if you switch the logger to MAIN it will output to a file. This also enable debug logging, which can help us understand what is going on.

The STSAssumeRoleSessionCredentialsProvider isn’t in the aws-java-sdk-core jar, so you would need to add the aws-java-sdk-sts jar to the class path.

If you don’t mind using Apache Maven this is a pom.xml that will create a launch script that will setup the class path for you.

To create the launcher script:

1. Create an empty directory
2. Create a new file in the directory called pom.xml
3. Install Apache Maven if you don’t already have it
4. In the directory created in 1, run the command mvn package
5. If everything works you should have a script in target/appassembler/bin/multilang-daemon, and all the jars are in target/appassembler/repo.
6. Copy the directory target/appassembler to wherever you need it.
7. To start your application run the mutlilang-daemon script with the location of your property file.

Found solution myself. For those experiencing the same problem and using python:

1. I did what @pfifer suggested (it was very helpful). Afterwards, I took aws-java-sdk-sts-1.11.129.jar from /target/appassembler/repo/com/amazonaws/aws-java-sdk-sts/1.11.129/ and put it into lib/python2.7/site-packages/amazon_kclpy (the folder with python KCL library.)

Of course, it would have been much better if the file was added to the library by default.

1. In the sample.parameters file I wrote:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account>:role/<role>|<session_name>
AWSCredentialsProviderDynamoDB = DefaultAWSCredentialsProviderChain
AWSCredentialsProviderCloudWatch = DefaultAWSCredentialsProviderChain

Previously, I didn’t specify the session_name, cause boto3 didn’t need it. That was a mistake. session_name can be anything, up to you.

In my case I need different credentials for DynamoDB and for Kinesis. In this commit I found how to do it.

Hi - I’m also trying to assume a role and so far have been unable to. I get the same message as you if I use the same format for AWSCredentialsProvider. I’m curious to know where you found the format in the first place? I haven’t been able to find any kind of documentation for it.

Edit: Found the source code: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/AWSCredentialsProviderPropertyValueDecoder.java#L72

If this doesnt work, add sts-sdk manually using this command wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-sts/1.12.428/aws-java-sdk-sts-1.12.428.jar -O "$APP_ROOT"/kcl/aws-sdk-sts.jar

and append java path of the location KCL_COMMAND="$(python "$APP_ROOT"/kcl/helper.py --print_command --java /usr/bin/java --properties "$APP_ROOT"/kcl/consumer.properties --classpath "$APP_ROOT"/kcl/*.jar)"

Works perfectly 💯