aws-sdk-ruby: Lightsail credentials fetching from IMDS is slow

Thanks for the offering on Lightsail reconfiguration; I really do appreciate it. Still, I prefer regular instances to special treatment, so people who maintain the servers after me would have an easier time figuring things out.

Some of our services are running in a regular VPC; we still need IPSec peering and stuff. However, we are in the direction of another way around; we think EC2 is overly complicated for our workload and start moving over to Lightsail. Lightsail is actually doing a better job than our self-managed VPC setup on some aspects. Weighing the cost-benefit, I don’t think that is the way to go for us.

Option 3 seems to be the way to go for most people. If someone comes to this thread from Google, that is probably what you are looking for. I also want to add that you probably need to config http_read_timeout instead of http_open_timeout because the packet that has a low hop limit is the one containing the secret, which excludes HTTP header and stuff.

Otherwise, TTL mangling is relatively safe too. The reason is that the condition -s 169.254.169.254 -d 172.26.0.0/16 ensures that at least one additional node is required to form a loop, and that node will be the one that decreases the TTL value. --ttl-inc 1 is unlikely to flood the network, but you may need to double-check your setup if you want more hops.

I understand every party, including the Lightsail team who decide against more ec2 permissions. Kudos to the Ruby SDK team for arranging things out.

I’ve talked to some members of the Lightsail team and also poked around a bit. Per my understanding as a background, Lightsail EC2 instances (like all EC2 instances) have linked IAM role (AmazonLightsailInstanceRole/instance-id). You can view it by calling aws sts get-caller-identity from an EC2 instance. The credentials returned by IMDS (instance metadata service) can be fetched with curl http://169.254.169.254/latest/meta-data/iam/security-credentials/AmazonLightsailInstanceRole, and the SDK tries to fetch and use these credentials. The issue is that, the SDK cannot fetch these using IMDSv2 in a container unless you modify your hop limit. I tried convincing the Lightsail team to blanket permissions to this role for EC2’s modify metadata operation but they declined. So that leaves us with a few options.

Option 1) Lightsail will modify the metadata on your behalf if you can provide a customer ID, region, instance ID, and the value of the hop limit (depends on your container routing). The downside of this option, is that it’s manual and new instances will not have your hop limit. If you want to do this, I can give you my email or some way to securely provide me that information.

Option 2, preferred IMO) Migrate your containers to EC2 instances without Lightsail. You can export snapshots but you will have to wire up storage and other expected features. The Lightsail engineer said that your use case may be more complex than Lightsail wants to offer. If you are running your own containers, you may as well just run them on vanilla EC2 instances, and then you have full control over them. You can then use the AWS CLI to modify your instance metadata.

Option 3) No architecture changes, and try to fail fast to use V1 credential fetching. It does seem like you expect your credentials from IMDS on the EC2 host. You can initialize an instance of Aws::InstanceProfileCredentials with 0 retries, and a smaller http_open_timeout, and pass it to your S3 client: Aws::S3::Client.new(credentials: Aws::InstanceProfileCredentials.new(..)). The major benefit is that this is “quick and dirty” and not necessarily a solution to the problem.