kops: Spot request for spot instance group expires after 1 week
Kops version: 1.7.0 Kube version: 1.6.2 Cloud provider: AWS
I successfully created a spot instance group ~1 month ago. Today I noticed that the spot instance requests were in state request-canceled-and-instance-running.
After quick investigation it seems like kops sets date Valid Until to today + 7 days so spot instance request is cancelled. Is there any way I can change this during creation of the instance group? I did not find anything about it in docs here.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 17 (12 by maintainers)
Thanks for reply. I see, I think you are right about spot fleets. AWS does not seem to expose
valid untilin any SDK (by curiosity I checked Java and JS).Does anyone know if there is another way to achieve spot instances using kops even though it requires manual tweak (i.e. changing something in AWS that is not reflected in state store)? From what I know there is no way to modify an active spot instance request which otherwise would be the easiest way to simply prolong
Valid Until.UPDATE: My quick fix was to add a scheduled action on the ASG to every day scale down and then up with
OldestInstanceas termination policy in order to rotate the nodes and hence get new spot instance request. This works for us considering it is a test environment and just wanted a quick way of having it more cost efficient.@duro yes i would say its a non-issue, from my understanding of the docs and testing of the behavior you should not get charged extra or have any undesirable effects from the way kops uses spot instances
I don’t think this is an issue exactly if you read the documentation from amazon. I just tested this myself as well we had kops create a ig with spot instances they have now expired after 7 days but our instances are still running as spot instances.
The answer from the amazon employee explains it well and provides links to amazon documentation that explains it as well.
https://forums.aws.amazon.com/thread.jspa?messageID=827030
To add a little more, it since the instances stay running everything is fine until the price goes above your max bid at that point it will terminate the spot instance. However kops sets on the launch configuration that is attached to the autoscaling group that it should launch spot instances so when the asg drops below its node count it will try to recreate the spot requests and hence a new spot instance.