acme-companion: certs are not generated, but no error in logs

I have many URL certified using acme, one of these doesn’t seem to be certified, but no error is present in log:

I’m using latest version of both nginx-proxy and acme-companion.

When doing docker-compose up the logs are:

$ docker-compose up --build
Building nginx
Step 1/3 : FROM nginxproxy/nginx-proxy:latest
 ---> e07e1525f3c2
Step 2/3 : COPY conf.d/* /etc/nginx/conf.d/
 ---> 89ef9964b8c1
Step 3/3 : RUN ls -l /etc/nginx/conf.d/
 ---> Running in 141c79cee4bf
total 12
-rw-r--r-- 1 root root 1093 Apr 13 15:40 default.conf
-rw-r--r-- 1 root root   93 Jun  8 09:14 timeout.conf
-rw-r--r-- 1 root root   87 Jun  8 09:37 uploadsize.conf
Removing intermediate container 141c79cee4bf
 ---> de01d71b3400

Successfully built de01d71b3400
Successfully tagged nginxproxy/nginx-proxy:latest
Recreating nginx-proxy ... done
Recreating nginx-proxy-letsencrypt ... done
Attaching to nginx-proxy, nginx-proxy-letsencrypt
nginx-proxy    | Custom dhparam.pem file found, generation skipped
nginx-proxy    | forego     | starting dockergen.1 on port 5000
nginx-proxy    | forego     | starting nginx.1 on port 5100
nginx-proxy-letsencrypt | Info: running acme-companion version v2.1.0-10-gcd3b51b
nginx-proxy    | dockergen.1 | 2021/06/10 16:13:21 Generated '/etc/nginx/conf.d/default.conf' from 28 containers
nginx-proxy    | dockergen.1 | 2021/06/10 16:13:21 Running 'nginx -s reload'
nginx-proxy    | dockergen.1 | 2021/06/10 16:13:21 Watching docker events
nginx-proxy    | dockergen.1 | 2021/06/10 16:13:21 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
nginx-proxy-letsencrypt | Info: Custom Diffie-Hellman group found, generation skipped.
nginx-proxy-letsencrypt | Reloading nginx proxy (5226814fb63282edf2f6fffc5abc85ba0618da1b813c05a7fdd388270b64d409)...
nginx-proxy-letsencrypt | 2021/06/10 16:13:22 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
nginx-proxy-letsencrypt | 2021/06/10 16:13:22 [notice] 60#60: signal process started
nginx-proxy-letsencrypt | 2021/06/10 16:13:23 Generated '/app/letsencrypt_service_data' from 28 containers
nginx-proxy-letsencrypt | 2021/06/10 16:13:23 Running '/app/signal_le_service'
nginx-proxy-letsencrypt | 2021/06/10 16:13:23 Watching docker events
nginx-proxy-letsencrypt | 2021/06/10 16:13:23 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/signal_le_service'
nginx-proxy-letsencrypt | Reloading nginx proxy (5226814fb63282edf2f6fffc5abc85ba0618da1b813c05a7fdd388270b64d409)...
nginx-proxy-letsencrypt | 2021/06/10 16:13:23 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
nginx-proxy-letsencrypt | 2021/06/10 16:13:23 [notice] 82#82: signal process started
nginx-proxy-letsencrypt | Creating/renewal devops.bancolini.com certificates... (devops.bancolini.com registry.bancolini.com)
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:25 UTC 2021] Domains not changed.
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:25 UTC 2021] Skip, Next renewal time is: Mon Aug  9 09:14:17 UTC 2021
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:25 UTC 2021] Add '--force' to force to renew.
nginx-proxy-letsencrypt | Creating/renewal gems.bancolini.com certificates... (gems.bancolini.com)
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:26 UTC 2021] Domains not changed.
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:26 UTC 2021] Skip, Next renewal time is: Mon Aug  2 09:44:01 UTC 2021
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:26 UTC 2021] Add '--force' to force to renew.
nginx-proxy-letsencrypt | Creating/renewal openproject.bancolini.com certificates... (openproject.bancolini.com)
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:27 UTC 2021] Domains not changed.
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:27 UTC 2021] Skip, Next renewal time is: Mon Aug  2 09:44:30 UTC 2021
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:27 UTC 2021] Add '--force' to force to renew.
nginx-proxy-letsencrypt | Creating/renewal pm.bancolini.com certificates... (pm.bancolini.com)
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:28 UTC 2021] Domains not changed.
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:28 UTC 2021] Skip, Next renewal time is: Mon Aug  2 09:44:47 UTC 2021
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:28 UTC 2021] Add '--force' to force to renew.
nginx-proxy-letsencrypt | Creating/renewal test.bancolini.com certificates... (test.bancolini.com)
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:29 UTC 2021] Domains not changed.
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:29 UTC 2021] Skip, Next renewal time is: Mon Aug  2 09:44:59 UTC 2021
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:29 UTC 2021] Add '--force' to force to renew.
nginx-proxy-letsencrypt | Creating/renewal time.bancolini.com certificates... (time.bancolini.com)
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:30 UTC 2021] Domains not changed.
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:30 UTC 2021] Skip, Next renewal time is: Mon Aug  2 09:45:22 UTC 2021
nginx-proxy-letsencrypt | [Thu Jun 10 16:13:30 UTC 2021] Add '--force' to force to renew.
nginx-proxy-letsencrypt | Sleep for 3600s

So, no apparent errors here: devops.bancolini.com which is the problematic one, does not seem to throw errors.

But the website, using chrome, gives this error:

NET::ERR_CERT_AUTHORITY_INVALID
Subject: letsencrypt-nginx-proxy-companion

Issuer: letsencrypt-nginx-proxy-companion

Expires on: Jun 3, 2022

Current date: Jun 10, 2021

And if I check the certs folder, I see:

$ ls -l certs/
total 36
-rw-r--r-- 1 root root 1870 giu  3 09:43 default.crt
-rw-r--r-- 1 root root 3268 giu  3 09:43 default.key
drwxr-xr-x 2 root root 4096 giu 10 15:45 devops.bancolini.com
-rw-r--r-- 1 root root  424 giu  3 09:44 dhparam.pem
drwxr-xr-x 2 root root 4096 giu  3 09:44 gems.bancolini.com
lrwxrwxrwx 1 root root   30 giu  3 09:44 gems.bancolini.com.chain.pem -> ./gems.bancolini.com/chain.pem
lrwxrwxrwx 1 root root   34 giu  3 09:44 gems.bancolini.com.crt -> ./gems.bancolini.com/fullchain.pem
lrwxrwxrwx 1 root root   13 giu  3 09:44 gems.bancolini.com.dhparam.pem -> ./dhparam.pem
lrwxrwxrwx 1 root root   28 giu  3 09:44 gems.bancolini.com.key -> ./gems.bancolini.com/key.pem
drwxr-xr-x 2 root root 4096 giu  3 09:44 openproject.bancolini.com
lrwxrwxrwx 1 root root   37 giu  3 09:44 openproject.bancolini.com.chain.pem -> ./openproject.bancolini.com/chain.pem
lrwxrwxrwx 1 root root   41 giu  3 09:44 openproject.bancolini.com.crt -> ./openproject.bancolini.com/fullchain.pem
lrwxrwxrwx 1 root root   13 giu  3 09:44 openproject.bancolini.com.dhparam.pem -> ./dhparam.pem
lrwxrwxrwx 1 root root   35 giu  3 09:44 openproject.bancolini.com.key -> ./openproject.bancolini.com/key.pem
drwxr-xr-x 2 root root 4096 giu  3 09:44 pm.bancolini.com
lrwxrwxrwx 1 root root   28 giu  4 07:36 pm.bancolini.com.chain.pem -> ./pm.bancolini.com/chain.pem
lrwxrwxrwx 1 root root   32 giu  4 07:36 pm.bancolini.com.crt -> ./pm.bancolini.com/fullchain.pem
lrwxrwxrwx 1 root root   13 giu  4 07:36 pm.bancolini.com.dhparam.pem -> ./dhparam.pem
lrwxrwxrwx 1 root root   26 giu  4 07:36 pm.bancolini.com.key -> ./pm.bancolini.com/key.pem
drwxr-xr-x 2 root root 4096 giu  3 09:45 test.bancolini.com
lrwxrwxrwx 1 root root   30 giu  3 09:45 test.bancolini.com.chain.pem -> ./test.bancolini.com/chain.pem
lrwxrwxrwx 1 root root   34 giu  3 09:45 test.bancolini.com.crt -> ./test.bancolini.com/fullchain.pem
lrwxrwxrwx 1 root root   13 giu  3 09:45 test.bancolini.com.dhparam.pem -> ./dhparam.pem
lrwxrwxrwx 1 root root   28 giu  3 09:45 test.bancolini.com.key -> ./test.bancolini.com/key.pem
drwxr-xr-x 2 root root 4096 giu  3 09:45 time.bancolini.com
lrwxrwxrwx 1 root root   30 giu  3 09:45 time.bancolini.com.chain.pem -> ./time.bancolini.com/chain.pem
lrwxrwxrwx 1 root root   34 giu  3 09:45 time.bancolini.com.crt -> ./time.bancolini.com/fullchain.pem
lrwxrwxrwx 1 root root   13 giu  3 09:45 time.bancolini.com.dhparam.pem -> ./dhparam.pem
lrwxrwxrwx 1 root root   28 giu  3 09:45 time.bancolini.com.key -> ./time.bancolini.com/key.pem

As you can see, there exists a devops.bancolini.com folder, but no links. checking inside devops.bancolini.com directory, it just have an hidden .companion file:

$ ls -al certs/devops.bancolini.com/
total 12
drwxr-xr-x 2 root root 4096 giu 10 15:45 .
drwxr-xr-x 8 root root 4096 giu 10 16:13 ..
-rw-r--r-- 1 root root   19 giu 10 16:13 .companion

But checking into the acme directory i get:

$ ls -al acme/ced@bancolini.com/devops.bancolini.com/
total 44
drwxr-xr-x 3 root root 4096 giu  3 09:43 .
drwxr-xr-x 9 root root 4096 giu  3 09:45 ..
drwxr-xr-x 2 root root 4096 giu 10 09:14 backup
-rw-r--r-- 1 root root 3751 giu 10 09:14 ca.cer
-rw-r--r-- 1 root root 2236 giu 10 09:14 devops.bancolini.com.cer
-rw-r--r-- 1 root root  972 giu 10 09:14 devops.bancolini.com.conf
-rw-r--r-- 1 root root 1716 giu 10 09:14 devops.bancolini.com.csr
-rw-r--r-- 1 root root  242 giu 10 09:14 devops.bancolini.com.csr.conf
-rw-r--r-- 1 root root 3243 giu 10 09:14 devops.bancolini.com.key
-rw-r--r-- 1 root root 5987 giu 10 09:14 fullchain.cer

Which is very similar to the other, working, domains.

The docker-compose.yml file is:

version: '3.7'

networks:
  default:
    external:
      name: webproxy

# /home/docker/persistence/nginx-data/ 
services:
  nginx:
    restart: always
    image: nginxproxy/nginx-proxy:latest
    build: .
    container_name: nginx-proxy
    hostname: nginx-proxy
    labels:
      - "com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /home/docker/persistence/nginx-data/certs:/etc/nginx/certs
      - /home/docker/persistence/nginx-data/vhost.d:/etc/nginx/vhost.d
      - /home/docker/persistence/nginx-data/html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro

  letsencrypt:
    restart: always
    image: nginxproxy/acme-companion:latest
    container_name: nginx-proxy-letsencrypt
    hostname: nginx-proxy-letsencrypt
    depends_on:
      - "nginx"
    volumes:    
      - /home/docker/persistence/nginx-data/certs:/etc/nginx/certs
      - /home/docker/persistence/nginx-data/vhost.d:/etc/nginx/vhost.d
      - /home/docker/persistence/nginx-data/html:/usr/share/nginx/html
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/docker/persistence/nginx-data/acme:/etc/acme.sh
    environment: 
      DEFAULT_EMAIL: ced@bancolini.com

And the one for the gitlab service is:

version: "3.7"

networks:
  default:
    external:
      name: webproxy

services:
  runner:
    image: "gitlab/gitlab-runner:latest"
    depends_on:
      - "gitlab"
    container_name: abs-gitlab-runner
    restart: always
    volumes:
      - "/home/docker/persistence/gitlab-runners-data:/etc/gitlab-runner"
      - "/var/run/docker.sock:/var/run/docker.sock"

  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    container_name: abs-gitlab
    restart: always
    hostname: devops.bancolini.com
    volumes:
      - '/home/docker/persistence/gitlab-data/config:/etc/gitlab'
      - '/home/docker/persistence/gitlab-data/data:/var/opt/gitlab'
      - '/home/docker/persistence/nginx-data/acme/ced@bancolini.com/devops.bancolini.com/fullchain.cer:/etc/gitlab/ssl/devops.bancolini.com.crt'
      - '/home/docker/persistence/nginx-data/acme/ced@bancolini.com/devops.bancolini.com/devops.bancolini.com.key:/etc/gitlab/ssl/devops.bancolini.com.key'
      - '/home/docker/persistence/nginx-data/acme/ced@bancolini.com/devops.bancolini.com/fullchain.cer:/etc/gitlab/ssl/registry.bancolini.com.crt'
      - '/home/docker/persistence/nginx-data/acme/ced@bancolini.com/devops.bancolini.com/devops.bancolini.com.key:/etc/gitlab/ssl/registry.bancolini.com.key'
    restart: always
    ports:
      - '80'
      - '443'
      - '5050:5050'
      - '22:22'
    environment:
      VIRTUAL_HOST: devops.bancolini.com,registry.bancolini.com
      VIRTUAL_PORT: 443
      VIRTUAL_PROTO: https
      LETSENCRYPT_HOST: devops.bancolini.com,registry.bancolini.com
      LETSENCRYPT_EMAIL: ced@bancolini.com

How can I debug this issue? I really need to have devops.bancolini.com up and running.

Thank you very much, Gabriele

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (7 by maintainers)

Most upvoted comments

In fact there are no certs for devops.bancolini.com, just a directory, the other domains, instead, got the directory and the certs and the links to those certs inside /etc/nginx/certs

Okay, I think I got lost between the different ls results.

First: /etc/acme.sh is meant to persist the internal state of acme.sh. It shouldn’t be used for any other purpose, should not be directly manipulated for anything else that issue fixing and files inside it shouldn’t be mounted inside other containers.

You have the wanted certificates inside /etc/acme.sh but not the copies inside /etc/nginx/certs. The container can’t detect and correct this situation by itself.

Could you try docker exec nginx-proxy-letsencrypt /app/force_renew ?

buchdag on Jun 10, 2021