pulumi-cloud: Travis failure in master: EC2 instance failed to create

Not clear if this was one-off or likely to re-occur.

https://travis-ci.com/pulumi/pulumi-cloud/builds/58711465

It looks like one of the two VMs that was created never responded that it was ready (presumably never fired the cfn-signal in it’s userdata script?).

The VM that never responded doesn’t show anything too unusual happening - it was running for the full 15 minute period:

https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=us-east-2#Instances:sort=instanceState

Unfortunately we don’t have logging for the EC2 VM boot process so it’s hard to see what might have gone wrong on the VM.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 45 (45 by maintainers)

Most upvoted comments

Here’s one way we can get logs from these instances, if we catch this within an hour of the failure:

Go to the list of EC2 instances in the pulumi-testing account in us-east-2 (Ohio). Look for terminated instances. https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=us-east-2#Instances:instanceState=terminated;sort=instanceId
Right-click each terminated instance and select “Get System Log” under “Instance Settings”.

The CloudFormation event log will show us the instance that successfully came up, e.g.: so we can look at the other one.

If there are a lot of instances, we can filter them to the appropriate autoscaling group. Looking at the CloudFormation stack will give you the physical ID of the autoscaling group. From there you can filter the EC2 instances that have a matching aws:autoscaling:groupName tag.

mmdriley on Jan 16, 2018