containerpilot: Cannot start a service after backend's events

I’m working on moving an existing application into autopilot. The whole application is a Docker Compose app and I’m currently focusing on a part where an API (developed in js with the sailsjs framework) relies on an underlying mongodb database.

The containerpilot.json of the API is the following one:

$ cat containerpilot.json
{
  "consul": "localhost:8500",
  "preStart": "/app/manage.sh prestart",
  "services": [
    {
      "name": "api",
      "health": "/usr/bin/curl -o /dev/null --fail -s http://localhost/health",
      "poll": 3,
      "ttl": 10,
      "port": 80
    }
  ],
  "coprocesses": [
    {
      "command": ["/usr/local/bin/consul", "agent",
                  "-data-dir=/data",
                  "-config-dir=/config",
                  "-rejoin",
                  "-retry-join", "{{ if .CONSUL_HOST }}{{ .CONSUL_HOST }}{{ else }}consul{{ end }}",
                  "-retry-max", "10",
                  "-retry-interval", "10s"],
      "restarts": "unlimited"
    }
  ],
  "backends": [
    {
      "name": "mongodb-replicaset",
      "poll": 3,
      "onChange": "/app/manage.sh db-change"
    }
  ]
}

I use the following script to handle the prestart and db-change events.

$ cat manage.sh
#!/bin/sh

event=$1
echo "Received event:$event"

if [ "$event" = "prestart" ];then
  while [[ "$(curl -s http://${CONSUL_HOST}:8500/v1/health/service/mongodb-replicaset | grep passing)" = "" ]]
  do
    echo "db is not yet healthly..."
    sleep 5
  done
  echo "db is healthly, moving on..."
  exit 0
fi

# If db not accessible anymore, restart the api service
if [ "$event" = "db-change" ];then
  pkill -SIGHUP node
fi

Part of the docker-compose file that I use.

version: '2'
services:
  # Consul
  consul:
    image: consul:0.7.2
    restart: always
    ports:
      - 8500:8500
    dns:
      - 127.0.0.1
    command: agent -server -client=0.0.0.0 -bootstrap -ui
  db:
    image: autopilotpattern/mongodb
  api:
    build: api
    environment:
      - CONSUL_HOST=consul
    ports:
      - 1337:80
    restart: always

What I’m expecting to have:

  • run the whole stack
  • API and DB services up in the Consul UI
  • stop the mongodb service
  • db disappear from Consul UI
  • API restart and wait for the db to be up again (within manage.sh)
  • API unhealthy in Consul UI
  • start the mongodb service
  • API detects the db is available (and exits the loop in manage.sh)
  • API becomes healthy in the Consul UI

What I’m having:

  • run the whole stack
  • DB service is up
  • API service is never up in Consul UI

In fact, when using the pkill -SIGHUP node command (triggered when a db event is detected), the API never becomes healthy, it seems there are several DB events that make it restart. What are the backends events that are expected to reach the API’s onChange ?

Any idea what I’m missing ?

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

My fault, I had the wrong command provided to containerpilot entrypoint in the Dockerfile. 3 services are up and running now, this is great. I’ll add Piloted module now. Thanks a lot for your help.