knex: Why can't I do knex migrate:latest using Docker-Composer?

I have the following docker-compose.yml where I’m starting two services:

  • postgres
  • nodejs app

After the Postgres DB is up I try to run knex migrate:latest to create the DB structure. But for some reason the migration keep telling me that is unable to connect. This is strange because:

  • I’m able to connect to the DB using Postico and work with the DB no problem.
  • The nodejs app is able to connect because I get an error message telling me that the table that I’m trying to select from dose not exists.
  • I’m able to ping the DB container from the Web container

Question

What am I missing?

File: docker-compose.yml

version: '2'

services:

  postgres:
    image: "postgres"
    ports:
     - "5433:5432"
    networks:
     - curted
    environment:
     POSTGRES_DB: dbname
     POSTGRES_USER: admin
     POSTGRES_PASSWORD: admin
    volumes:
     - database:/var/lib/postgresql/data

  example.com:
    build: .
    ports:
     - "5000:5000"
    volumes:
     - .:/code
    networks:
     - example
    working_dir: /code
    environment:
     DATABASE_URL: postgres://admin:admin@172.18.0.2:5433/dbname
     NODE_ENV: local
    command: bash -c "sleep 10 && npm run knex migrate:latest --file knexfile.js"
    depends_on:
     - postgres

networks:
  curted:
    driver: bridge

volumes:
  database:

The knexfile.js

module.exports = {
	local: {
		client: 'pg',
		connection: process.env.DATABASE_URL,
		ssl: false,
		migrations: {
			tableName: 'knex_migrations'
		}
	},
	staging: {
		client: 'pg',
		connection: process.env.DATABASE_URL,
		ssl: true,
		migrations: {
			tableName: 'knex_migrations'
		},
		pool: {
			min: 1,
			max: 2
		}
	},
	production: {
		client: 'pg',
		connection: process.env.DATABASE_URL,
		ssl: true,
		migrations: {
			tableName: 'knex_migrations'
		},
		pool: {
			min: 1,
			max: 2
		}
	}
};

Erro Message

curted.com_1  | > CuratedProducts@1.0.0 knex /home/app
curted.com_1  | > knex "migrate:latest"
curted.com_1  |
curted.com_1  | Using environment: local
curted.com_1  | Error: connect ECONNREFUSED 172.18.0.2:5433
curted.com_1  |     at Object.exports._errnoException (util.js:1022:11)
curted.com_1  |     at exports._exceptionWithHostPort (util.js:1045:20)
curted.com_1  |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1090:14)

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 28 (6 by maintainers)

Most upvoted comments

Hey guys, I found the reason and the solution. The solution is generally speaking the same as @Bram-Zijp . Here is my knexfile.js:

module.exports = {

  development: {
    client: 'mysql2',
      debug:true,
    connection: {
        host:'docker-mysql',
        user: 'root',
        password: '1qaz',
        port: '',
        database: 'docker_db'
    },
      migrations: {
          tableName: 'knex_migrations'
      }
  },
};

The key is to set host with the name of the db service in docker-compose. Setting port to empty string or 3306 both will okay ( I don’t know why but it works).

The ECONNREFUSED error means the web app cannot connect the db, and I’ve tried many hosts and ports but all the tries failed, like localhost:3306 and 0.0.0.0:13306 and so on. The reason is the localhost:3306 actually refers the container itself, not the docker host.

Plus, make sure the migration is executed after db fully initializing.

References:

  1. https://medium.com/@niratattri/building-a-node-js-application-and-deploying-through-docker-meet-docker-aa8ae677ea12
  2. https://stackoverflow.com/questions/45399941/node-js-connect-to-mysql-docker-container-econnrefused

OK. I’ll dig deeper, and will let you know.

A little late to the party, but I thought I’d share a Node script that I run when using docker-compose with a Knex migration service.

"use strict";

const config = require("./knexfile");
const Knex = require("knex");

const knex = Knex(config);

async function runMigrations(retries, delay) {
  for (let i = 1; i !== retries; i++) {
    try {
      return await knex.migrate.latest();
    } catch (err) {
      await setTimeoutAsync(delay * i);
    }
  }

  throw new Error(`Unable to run migrations after ${retries} attempts.`);
}

function setTimeoutAsync(ms) {
  return new Promise(resolve => {
    setTimeout(resolve, ms);
  });
}

runMigrations(10, 500)
  .catch(err => {
    console.log(err);
    process.exit(1);
  })
  .then(res => {
    console.log(`Batch ${res[0]} run: ${res[1].length} migrations`);
    console.log(res[1].join("\n"));
    process.exit();
  });

This is a file called index.js that is run by npm start, in a directory called db off of the root directory of my project. The docker-compose.yml file looks like this:

version: "3.7"

services:
  postgres:
    environment:
      - POSTGRES_DB=mydb
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 10
    ports:
      - "5432:5432"
    image: postgres:9.6
    restart: always
    volumes:
      - ".:/data"

  migrations:
    build: ./db
    depends_on:
      - postgres
    command: bash -c "npm start"
    environment:
      - DOCKER=true
      - NODE_ENV=development
      - PGHOST=postgres
      - PGPORT=5432
      - PGUSER=postgres
      - PGPASSWORD=
      - PGDATABASE=mydb
    volumes:
      - "./db:/db"

I was running the migrate commands from my terminal instead of from inside the docker container. When I ran the migration from within the container which is hosting the node.js app it migrated fine.

I use this in my knexfile.js as connection:

      connection: {
            host: 'db',
            database: 'cms',
            user: 'cms',
            password: 'test',
        },

this is my docker-compose.yml

version: '2'
services:
  web:
    build: .
    ports:
      - 3001:3001
    depends_on:
      - db
    links:
      - db
  db:
    image: postgres:10
    volumes:
      - ./data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: cms
      POSTGRES_USER: cms
      POSTGRES_PASSWORD: test

docker exec CONTAINER_NAME sh -c "npm run migrate" The migrate script in my package.json looks like this: "migrate": "knex migrate:rollback && knex migrate:latest && knex seed:run",

Hope this helps those who are struggling.

@davidgatti I have the same issue and resolve it by following code: (I run following code in web container in the startup script, and DB in other contaiter on which web container depends)

echo "Running migrations ..."
for i in $(seq 1 30); do
    knex migrate:latest
    [ $? = 0 ] && break
    echo "Reconnecting db ..." && sleep 1
done

Below are some logs:

Running migrations ...
Using environment: production
Error: connect ECONNREFUSED 172.19.0.3:5432
    at Object.exports._errnoException (util.js:953:11)
    at exports._exceptionWithHostPort (util.js:976:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1080:14)
Reconnecting db ...
Using environment: production
Error: connect ECONNREFUSED 172.19.0.3:5432
    at Object.exports._errnoException (util.js:953:11)
    at exports._exceptionWithHostPort (util.js:976:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1080:14)
Reconnecting db ...
Using environment: production
error: the database system is starting up
    at Connection.parseE (/opt/xxx/node_modules/pg/lib/connection.js:539:11)
    at Connection.parseMessage (/opt/xxx/node_modules/pg/lib/connection.js:366:17)
    at Socket.<anonymous> (/opt/xxx/node_modules/pg/lib/connection.js:105:22)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:172:18)
    at Socket.Readable.push (_stream_readable.js:130:10)
    at TCP.onread (net.js:542:20)
Reconnecting db ...
Using environment: production
Already up to date

The logs show that DB is not ready in the first three connecting tries. So if we run knex migration just one time, it will be failed in most cases. But I do set timeout parameters (acquireTimeout & bailAfter) in knexfile.js like this:

  production: {
    client: 'pg',
    connection: {
      host: 'dbserver',
      database: mydb,
      user:     myuser,
      password: mypwd
    },
    pool: {
      min: 2,
      max: 10,
      acquireTimeout: 30*1000,
      bailAfter: 30*1000
    },
    migrations: {
      tableName: 'knex_migrations'
    }
  },

I expect that knex will keep trying to connect DB in 30 seconds. But it’s not. Is this a knex bug? or Is there any other proper parameter can control the timeout/retrying?