Bug Report

• Concourse version: 3.9.1
• Deployment type (BOSH/Docker/binary): Docker
• Infrastructure/IaaS: VSphere-provided Ubuntu 16.04.3 kernel v4.4.0-62 running Docker v1.13.1
• Browser (if applicable): Chrome
• Did this used to work? Yes, in 3.9.0 and earlier

Starting with release 3.9.1, running concourse via http://concourse-ci.org/docker-repository.html, after loading a pipeline, we see all resources turn orange with the error base resource type not found.

Digging through the logs, we see: concourse-worker: (these are ‘git’ resource types)

{"timestamp":"1520631653.062031031","source":"atc","message":"atc.syncer.JavaHelloWorld:radar.start","log_level":1,"data":{"session":"42.3"}}
{"timestamp":"1520631653.063035488","source":"atc","message":"atc.syncer.JavaHelloWorld:scheduler.start","log_level":1,"data":{"interval":"10s","session":"42.4"}}
{"timestamp":"1520631653.085903168","source":"atc","message":"atc.syncer.JavaHelloWorld:radar.scan-resource.interval-runner.tick.failed-to-find-or-create-resource-config-check-session","log_level":2,"data":{"error":"base resource type not found","pipeline-scoped-name":"JavaHelloWorld:resource:HelloWorld","session":"42.3.1.1.1"}}

concourse-web:

timestamp":"1520631240.584894657","source":"guardian","message":"guardian.start.completed","log_level":1,"data":{"session":"6"}}
{"timestamp":"1520631240.585029602","source":"guardian","message":"guardian.list-containers.starting","log_level":1,"data":{"session":"31"}}
{"timestamp":"1520631240.585165977","source":"guardian","message":"guardian.list-containers.finished","log_level":1,"data":{"session":"31"}}
{"timestamp":"1520631240.585197687","source":"guardian","message":"guardian.started","log_level":1,"data":{"addr":"127.0.0.1:7777","network":"tcp"}}

concourse-db: (looks the same in 3.9.0)

PostgreSQL init process complete; ready for start up.

LOG:  database system was shut down at 2018-03-09 21:34:06 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
ERROR:  relation "migration_version" does not exist at character 21
STATEMENT:  SELECT version FROM migration_version
ERROR:  relation "migration_version" does not exist at character 21
STATEMENT:  SELECT version FROM migration_version

If we edit the docker-compose to point to v3.9.0 (or 3.8.0), these symptoms go away. This is confirmed on three diff deployed servers we have so far running concourse.

We have 300+ servers deployed running concourse that get rebuilt to the newest version of concourse every couple of months, so this will become a big issue for us soon and we need to know if we should update the server build template to peg to 3.9.0 to avoid widespread issues.

This is an example of how the resource definition looks for one of the git resources failing:

resources:
- name: HelloWorld
  type: git
  webhook_token: <redacted>
  source: 
    uri: git@<redacted>.git
    branch: master
    private_key: ((github-private-key-value))
  check_every: 24h

This is a sanitized example of the docker-compose.yml used for this. We observed the same behavior with a docker-compose v2 format as well (not that we expect this to have bearing on the issue we see):

version: '3'

services:
  concourse-db:
    image: postgres:9.6
    restart: unless-stopped
    environment:
      POSTGRES_DB: concourse
      POSTGRES_USER: concourse
      POSTGRES_PASSWORD: <redacted>
      PGDATA: /database
  concourse-web:
    image: concourse/concourse
    links: [concourse-db]
    command: web
    depends_on: [concourse-db]
    ports: ["80:8080"]
    volumes: ["./keys/web:/concourse-keys"]
    restart: unless-stopped # required so that it retries until concourse-db comes up
    environment:
      CONCOURSE_EXTERNAL_URL: "<redacted>"
      CONCOURSE_GITHUB_AUTH_CLIENT_ID: "<redacted>"
      CONCOURSE_GITHUB_AUTH_CLIENT_SECRET: "<redacted>"
      CONCOURSE_GITHUB_AUTH_ORGANIZATION: "<redacted>"
      CONCOURSE_GITHUB_AUTH_AUTH_URL:  "https://<redacted>/login/oauth/authorize"
      CONCOURSE_GITHUB_AUTH_TOKEN_URL: "https://<redacted>/login/oauth/access_token"
      CONCOURSE_GITHUB_AUTH_API_URL:   "https://<redacted>/api/v3/"
      CONCOURSE_PUBLICLY_VIEWABLE: "true"
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: concourse
      CONCOURSE_POSTGRES_PASSWORD: <redacted>
      CONCOURSE_POSTGRES_DATABASE: concourse
      CONCOURSE_ENCRYPTION_KEY: "<redacted>"
      CONCOURSE_VAULT_URL: "<redacted>"
      CONCOURSE_VAULT_AUTH_BACKEND: "approle"
      CONCOURSE_VAULT_AUTH_PARAM: "<redacted>"
  concourse-worker:
    image: concourse/concourse
    privileged: true
    links: [concourse-web]
    depends_on: [concourse-web]
    command: worker
    restart: unless-stopped
    volumes: ["./keys/worker:/concourse-keys", "/opt/concourse/:/opt/concourse/"]
    environment:
      CONCOURSE_TSA_HOST: concourse-web
      CONCOURSE_WORK_DIR: "/opt/concourse/"
      https_proxy: "<redacted>"
      http_proxy: "<redacted>"
      no_proxy: "<redacted>"