elastic-ci-stack-for-aws: SQS queues not being cleaned up by lifecycled

When starting a new buildkite agent instance, lifecycled automatically creates a SQS queue for the instance (https://github.com/lox/lifecycled/blob/master/main.go#L71)

The plan is for lifecycled to remove the queue upon receiving SIGTERM (when the instances shuts down) (https://github.com/lox/lifecycled/blob/master/main.go#L77-L90)

The problem is that lifecycled gets a SIGPIPE instead, thus not cleaning up after itself. Seems like the problem is that stderr is being redirected to stdout and piped to syslog.

Nov 27 15:21:34 ip-10-0-1-241 lifecycled: time="2017-11-27T15:21:34Z" level=info msg="Looking up instance id from metadata service"
Nov 27 15:21:35 ip-10-0-1-241 lifecycled: time="2017-11-27T15:21:35Z" level=info msg="Listening for lifecycle notifications"
Nov 27 15:21:49 ip-10-0-1-241 init: lifecycled main process (27036) killed by PIPE signal

We’ve been able to get it to work by just redirecing stdout, but since lifecycled logs to stderr by default we won’t get any logs. Unsure if the problem lies within lifecycled or the configuration for upstart.

start on startup
respawn
respawn limit unlimited
post-stop exec sleep 65
script
    set -a
    . /etc/lifecycled
    mkfifo /tmp/lifecycled-log-fifo
    ( logger -t lifecycled </tmp/lifecycled-log-fifo & )
    exec >/tmp/lifecycled-log-fifo
    #exec 2>&1
    rm /tmp/lifecycled-log-fifo
    exec /usr/bin/lifecycled
end script

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 19 (11 by maintainers)

Most upvoted comments

In the meantime, something like this would delete lifecycled SQS queues for instances that aren’t running any more:

#!/bin/bash
set -euo pipefail

instances=($(aws ec2 describe-instances \
  --filters "Name=tag:Role,Values=buildkite-agent" "Name=instance-state-name,Values=running" \
  --query 'Reservations[*].Instances[*].InstanceId' \
  --output text))

for queue in $(aws sqs list-queues \
  --queue-name-prefix "lifecycled" \
  --query 'QueueUrls[*]' \
  --output text) ; do

  running=0
  for i in "${instances[@]}" ; do
    queue_instance=$(echo "$queue" | cut -d/ -f5 | sed 's/^lifecycled-//' )
    if [[ "$queue_instance" == "$i" ]] ; then
      running=1
    fi
  done

  if [[ $running == 0 ]] ; then
    aws sqs delete-queue --queue-url "$queue"
  fi
done

lox on Apr 4, 2018

We’ve had a few reports of this and a replacement for lifecycled is a fair way off, so we’ll have a look at what we can do to fix this! Thanks for the patience folks.

lox on Feb 12, 2018

This was fixed in the 4.x series.

lox on Jul 27, 2019

We created a tool for cleaning up SQS queues: https://github.com/buildkite/lifecycled#cleaning-up-leftover-sqs-queues

@meganemura there shouldn’t be multiple SNS topics?

lox on Apr 19, 2018

Exceeding 85% of our free usage tier for SQS. I think we actually exceeded 1,000,000 messages so there’s $0.40c we’ll never get back 😉

ritchiey on Apr 4, 2018