AutoSpotting: Spot EC2 instances fail to launch with "Server.InternalError: Internal error on launch"

Issue type

• Bug Report

Build number

2017/10/26 22:50:27.070251 Starting autospotting agent, build nightly-461 expiring on 26-Dec-2017

Configuration

Using the cloudformation template; all default parameters except “Regions”:

ExecutionFrequency | rate(5 minutes)
LambdaHandlerFunction | handler.Handle
LambdaMemorySize | 256
LambdaS3Bucket | cloudprowess
LambdaZipPath | nightly/lambda.zip
LogRetentionPeriod | 7
MinOnDemandNumber | 0
MinOnDemandPercentage | 0.0
Regions | us-east-1

Environment

• AWS region: us-east-1
• Type of environment: VPC
• Anonymized launch configuration:

"MyLaunchConfig": {
      "Type": "AWS::AutoScaling::LaunchConfiguration",
      "Properties": {
        "ImageId" : "ami-80861296",
        "InstanceType": "m4.xlarge",
        "InstanceMonitoring": "false",
        "IamInstanceProfile" : { "Ref" : "MyInstanceProfile" },
        "KeyName": "keyname",
        "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
          "#!/bin/sh\n",
          "echo hello", "\n"
        ]]}},
        "SecurityGroups": [
          "sg1", "sg2", "sg3", "sg4"
        ],
        "BlockDeviceMappings" : [
          {
            "DeviceName" : "/dev/sda1",
            "Ebs" : {
              "VolumeSize" : "8",
              "VolumeType" : "gp2",
              "DeleteOnTermination" : "true"
            }
          },
          {
            "DeviceName" : "/dev/xvdb",
            "Ebs" : {
              "VolumeSize" : "20",
              "VolumeType" : "gp2",
              "DeleteOnTermination" : "true"
            }
          },
          {
            "DeviceName" : "/dev/xvdc",
            "Ebs" : {
              "VolumeSize" : "20",
              "VolumeType" : "gp2",
              "DeleteOnTermination" : "true"
            }
          }
        ]
      }
    }

Summary

All the instances launched via Autospotting result in “State transition reason message: Server.InternalError: Internal error on launch”.

Steps to reproduce

Unknown at this point.

Expected results

Working EC2 spot instances.

Actual results

Every spot instances launched by Autospotting are terminated due to “Server.InternalError: Internal error on launch”, as shown in the EC2 console. In the Spot console, this shows as “Status: instance-terminated-by-user”.

START RequestId: 74b8d322-baa1-11e7-b1e9-15ba8e5cc822 Version: $LATEST
2017/10/26 23:00:26.976017 Starting autospotting agent, build nightly-461 expiring on 26-Dec-2017
2017/10/26 23:00:27.009913 Parsed command line flags: regions='us-east-1' min_on_demand_number=0 min_on_demand_percentage=0.0 allowed_instance_types=
2017/10/26 23:00:27 main.go:97: Scanning for available AWS regions
2017/10/26 23:00:27 main.go:69: Enabled to run in us-east-1, processing region.
2017/10/26 23:00:27 region.go:66: Creating connections to the required AWS services in us-east-1
2017/10/26 23:00:27 connections.go:29: Creating Service connections in us-east-1
2017/10/26 23:00:27 connections.go:43: Created service connections in us-east-1
2017/10/26 23:00:27 region.go:70: Scanning for enabled AutoScaling groups in us-east-1
2017/10/26 23:00:27 region.go:280: Processing page 1 of DescribeTagsPages for us-east-1
2017/10/26 23:00:27 region.go:282: us-east-1 has enabled ASG: software-user-us-dev-MyAutoScale
2017/10/26 23:00:27 region.go:314: Processing page 1 of DescribeAutoScalingGroupsPages for us-east-1
2017/10/26 23:00:27 region.go:77: Scanning full instance information in us-east-1
2017/10/26 23:00:27 spot_price.go:24: us-east-1 Requesting spot prices
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.2xlarge in us-east-1e skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.2xlarge in us-east-1a skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.2xlarge in us-east-1b skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.8xlarge in us-east-1b skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.8xlarge in us-east-1a skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.8xlarge in us-east-1e skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.4xlarge in us-east-1a skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.xlarge in us-east-1e skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.4xlarge in us-east-1b skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.8xlarge in us-east-1c skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.4xlarge in us-east-1c skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.4xlarge in us-east-1e skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.large in us-east-1a skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.large in us-east-1c skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.2xlarge in us-east-1c skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.xlarge in us-east-1c skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.large in us-east-1d skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.4xlarge in us-east-1d skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.xlarge in us-east-1a skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.large in us-east-1b skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.xlarge in us-east-1d skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.large in us-east-1e skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.xlarge in us-east-1b skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.2xlarge in us-east-1d skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:231: us-east-1 Instance data missing for c3.8xlarge in us-east-1d skipping because this region is currently not supported
2017/10/26 23:00:27 region.go:82: Scanning instances in us-east-1
2017/10/26 23:00:28 region.go:116: Processing page 1 of DescribeInstancesPages for us-east-1
2017/10/26 23:00:28 region.go:88: Processing enabled AutoScaling groups in us-east-1
2017/10/26 23:00:28 autoscaling.go:173: Finding spot instance requests created for software-user-us-dev-MyAutoScale
2017/10/26 23:00:28 autoscaling.go:233: Spot instance requests were previously created for software-user-us-dev-MyAutoScale
2017/10/26 23:00:28 autoscaling.go:245: Adding instances to software-user-us-dev-MyAutoScale
2017/10/26 23:00:28 autoscaling.go:140: No default value for on-demand instances specified, skipping.
2017/10/26 23:00:28 autoscaling.go:762: software-user-us-dev-MyAutoScale Counting already running on demand instances 
2017/10/26 23:00:28 autoscaling.go:778: software-user-us-dev-MyAutoScale Found 3 on-demand instances running on a total of 3
2017/10/26 23:00:28 autoscaling.go:148: Currently more than enough OnDemand instances running
2017/10/26 23:00:28 autoscaling.go:404: spot bids were found, continuing
2017/10/26 23:00:28 autoscaling.go:465: software-user-us-dev-MyAutoScale No active unfulfilled bid was found
2017/10/26 23:00:28 autoscaling.go:211: us-east-1 software-user-us-dev-MyAutoScale Would launch a spot instance in us-east-1a
2017/10/26 23:00:28 autoscaling.go:543: Trying to launch spot instance in us-east-1a first finding an on-demand instance to use as a template
2017/10/26 23:00:28 autoscaling.go:552: Found on-demand instance 0xc4206e69e8
2017/10/26 23:00:28 launch_configuration.go:31: Launch configuration would attach 0 ephemeral volumes if available
2017/10/26 23:00:28 instance.go:259: Comparing m2.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m2.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m2.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing cr1.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type cr1.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 cr1.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing i2.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i2.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i2.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m4.16xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m4.16xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m4.16xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing r3.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r3.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r3.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing f1.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type f1.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 f1.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing cc2.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type cc2.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 cc2.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing t2.micro with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type t2.micro in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 t2.micro instances
2017/10/26 23:00:28 instance.go:259: Comparing t2.medium with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type t2.medium in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 t2.medium instances
2017/10/26 23:00:28 instance.go:259: Comparing m4.10xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m4.10xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m4.10xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m3.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m3.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m3.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing x1e.32xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type x1e.32xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 x1e.32xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing r3.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r3.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r3.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing c1.medium with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type c1.medium in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 c1.medium instances
2017/10/26 23:00:28 instance.go:259: Comparing r4.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r4.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r4.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing r4.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r4.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r4.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing i3.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i3.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i3.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing d2.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type d2.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 d2.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing d2.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type d2.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 d2.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m1.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m1.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m1.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing cg1.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type cg1.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 cg1.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m2.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m2.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m2.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing i2.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i2.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i2.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing g3.16xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type g3.16xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 g3.16xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing i3.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i3.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i3.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing g2.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type g2.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 g2.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing t2.nano with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type t2.nano in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 t2.nano instances
2017/10/26 23:00:28 instance.go:259: Comparing m1.medium with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m1.medium in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m1.medium instances
2017/10/26 23:00:28 instance.go:259: Comparing m3.large with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m3.large in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m3.large instances
2017/10/26 23:00:28 instance.go:259: Comparing g3.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type g3.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 g3.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing x1.16xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type x1.16xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 x1.16xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing r4.16xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r4.16xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r4.16xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing d2.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type d2.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 d2.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing i2.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i2.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i2.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m3.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m3.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m3.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing t2.small with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type t2.small in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 t2.small instances
2017/10/26 23:00:28 instance.go:259: Comparing t2.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type t2.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 t2.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing x1.32xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type x1.32xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 x1.32xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing p3.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type p3.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 p3.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m4.large with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m4.large in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m4.large instances
2017/10/26 23:00:28 instance.go:259: Comparing r3.large with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r3.large in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r3.large instances
2017/10/26 23:00:28 instance.go:259: Comparing p3.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type p3.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 p3.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing p3.16xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type p3.16xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 p3.16xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing p2.16xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type p2.16xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 p2.16xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing i3.large with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i3.large in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i3.large instances
2017/10/26 23:00:28 instance.go:259: Comparing i3.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i3.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i3.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing g2.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type g2.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 g2.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing i2.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i2.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i2.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing hs1.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type hs1.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 hs1.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m4.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m4.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m4.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m3.medium with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m3.medium in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m3.medium instances
2017/10/26 23:00:28 instance.go:259: Comparing g3.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type g3.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 g3.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing i3.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i3.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i3.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing t2.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type t2.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 t2.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m4.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m4.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m4.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing i3.16xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type i3.16xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 i3.16xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m1.large with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m1.large in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m1.large instances
2017/10/26 23:00:28 instance.go:259: Comparing hi1.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type hi1.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 hi1.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m4.2xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m4.2xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m4.2xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing f1.16xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type f1.16xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 f1.16xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m1.small with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m1.small in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m1.small instances
2017/10/26 23:00:28 instance.go:259: Comparing c1.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type c1.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 c1.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing t1.micro with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type t1.micro in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 t1.micro instances
2017/10/26 23:00:28 instance.go:259: Comparing p2.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type p2.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 p2.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing p2.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type p2.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 p2.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing d2.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type d2.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 d2.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing m2.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type m2.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 m2.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing t2.large with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type t2.large in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 t2.large instances
2017/10/26 23:00:28 instance.go:259: Comparing r4.large with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r4.large in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r4.large instances
2017/10/26 23:00:28 instance.go:259: Comparing r3.4xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r3.4xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r3.4xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing r4.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r4.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r4.xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing r4.8xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r4.8xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r4.8xlarge instances
2017/10/26 23:00:28 instance.go:259: Comparing r3.xlarge with m4.xlarge
2017/10/26 23:00:28 autoscaling.go:737: software-user-us-dev-MyAutoScale Counting already running spot instances of type r3.xlarge in AZ us-east-1a
2017/10/26 23:00:28 autoscaling.go:748: software-user-us-dev-MyAutoScale Found 0 r3.xlarge instances
2017/10/26 23:00:28 autoscaling.go:569: Finished searching for best spot instance in us-east-1a
2017/10/26 23:00:28 autoscaling.go:570: Replacing an on-demand m4.xlarge instance having the ondemand price 0.2
2017/10/26 23:00:28 autoscaling.go:572: Launching best compatible instance: r3.xlarge with current spot price: 0.0361
2017/10/26 23:00:28 autoscaling.go:582: Bidding for spot instance for software-user-us-dev-MyAutoScale
2017/10/26 23:00:29 autoscaling.go:620: software-user-us-dev-MyAutoScale Created spot instance request sir-c6ig7j5g
2017/10/26 23:00:29 spot_instance_request.go:107: software-user-us-dev-MyAutoScale Failed to create tags for the spot instance request sir-c6ig7j5g retrying in 5 seconds...
2017/10/26 23:00:34 spot_instance_request.go:116: software-user-us-dev-MyAutoScale successfully tagged spot instance request sir-c6ig7j5g
2017/10/26 23:00:34 spot_instance_request.go:24: software-user-us-dev-MyAutoScale Waiting for spot instance for spot instance request sir-c6ig7j5g
2017/10/26 23:00:49 spot_instance_request.go:39: software-user-us-dev-MyAutoScale Done waiting for an instance.
2017/10/26 23:00:49 spot_instance_request.go:51: software-user-us-dev-MyAutoScale Found new spot instance i-0c895bde4f9250c39
2017/10/26 23:00:49 spot_instance_request.go:52: Tagging it to match the other instances from the group
2017/10/26 23:00:50 region.go:116: Processing page 1 of DescribeInstancesPages for us-east-1
2017/10/26 23:00:50 instance.go:303: us-east-1 Tagging spot instance i-0c895bde4f9250c39
2017/10/26 23:00:51 instance.go:308: Instance i-0c895bde4f9250c39 was tagged with the following tags: [{
Key: "Name",
Value: "software-masters-user-us"
} {
Key: "spot-enabled",
Value: "true"
}]
2017/10/26 23:00:51.060402 Execution completed, nothing left to do
END RequestId: 74b8d322-baa1-11e7-b1e9-15ba8e5cc822
REPORT RequestId: 74b8d322-baa1-11e7-b1e9-15ba8e5cc822	Duration: 24085.22 ms	Billed Duration: 24100 ms Memory Size: 256 MB	Max Memory Used: 48 MB