acme-companion: jrcs/letsencrypt-nginx-proxy-companion - HTTPS is throwing 500 Internal Server Error

Hello,

I’m stuck with this prod issue for couple of days and it is hurting a customer now. So if anyone would be kind enough to help - I’d really appreciate here 😃

I’m using jrcs/letsencrypt-nginx-proxy-companion for generating SSL certs for a number of companion apps running on sidecar docker-compose. There are more than 100 dockers and each one is running with a subdomain.

I pulled the latest docker tag of jrcs/letsencrypt-nginx-proxy-companion couple of days ago and issue started from there.

Issue:

1. More than half of the apps are running fine with HTTPS. Valid Letsencrypt cert is showing there.
2. Few apps (docker-compose) are not running with HTTPS - showing “500 Internal Server Error | nginx/1.19.3” error
3. But they are running fine with HTTP
4. Lots of subdomains are getting SSL renewed when /app/force_renew is run. But some of them are not getting renewed

Create new order error. Le_OrderFinalize not found. {
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: friis1.marvinxr.com: see https://letsencrypt.org/docs/rate-limits/",
  "status": 429
}

While some of them are auto-renewed (all of them have same configuration)

Creating/renewal spiderman.marvinxr.com certificates... (spiderman.marvinxr.com)
[Thu Jun  9 13:53:29 UTC 2022] Domains not changed.
[Thu Jun  9 13:53:29 UTC 2022] Skip, Next renewal time is: Mon Aug  8 09:00:49 UTC 2022

I have tried restarting/force_renew certificates quite some time (I know it’s not the best practice but the client has a big launch and cannot wait). The issue is not getting solved at all.

jrcs/letsencrypt-nginx-proxy-companion docker-compose:

version: '2'

services:

  nginx-proxy:
    image: jwilder/nginx-proxy
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - conf:/etc/nginx/conf.d
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - dhparam:/etc/nginx/dhparam
      - certs:/etc/nginx/certs:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
    restart: always

  letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: nginx-proxy-le
    depends_on:
      - nginx-proxy
    volumes_from:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: always

volumes:
  conf:
  vhost:
  html:
  dhparam:
  certs:
  acme:

networks:
  default:
    external:
      name: letsencrypt

sidecar application docker-compose:

version: '3.4'

networks:
  application_network:
services:
    ar_app:
      restart: "always"
      container_name: friis1
      build:
        context: app
      environment:
         VIRTUAL_HOST: friis1.marvinxr.com
         PORT: 6000
         LETSENCRYPT_EMAIL: support@marvinai.live
         LETSENCRYPT_HOST: friis1.marvinxr.com
      expose:
        - "6000"


networks:
  default:
    external:
      name: letsencrypt

Dockerfile for application docker-compose (from above)

FROM python:3.11.0b3-alpine3.16

WORKDIR /app

COPY requirements.txt ./

RUN pip install -r requirements.txt

COPY . /app

CMD ["python","-u","server.py"]

I can see the full list of certificate chain and acme keys stored corrctly. friis1.marvinxr.com (directory)

• friis1.marvinxr.com.conf
• friis1.marvinxr.com.csr
• friis1.marvinxr.com.csr.conf
• friis1.marvinxr.com.key

friis1.marvinxr.com.chain.pem friis1.marvinxr.com.crt friis1.marvinxr.com.dhparam.pem friis1.marvinxr.com.key

The certificate was working perfectly for over a month. But then they got regenerated on June 7th. The problem starts from there.

Doing curl gives me this result:

curl https://friis1.marvinxr.com
curl: (60) SSL certificate problem: self signed certificate More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

Mozilla says Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT https://friis1.marvinxr.com/

The certificate is not trusted because it is self-signed.

HTTP Strict Transport Security: false HTTP Public Key Pinning: false

Certificate chain:

-----BEGIN CERTIFICATE----- MIIFOTCCAyGgAwIBAgIUJURatmDfv7J5KmupgfDJEbixz7cwDQYJKoZIhvcNAQEL BQAwLDEqMCgGA1UEAwwhbGV0c2VuY3J5cHQtbmdpbngtcHJveHktY29tcGFuaW9u MB4XDTIyMDYwODIxMzcwOFoXDTIzMDYwODIxMzcwOFowLDEqMCgGA1UEAwwhbGV0 c2VuY3J5cHQtbmdpbngtcHJveHktY29tcGFuaW9uMIICIjANBgkqhkiG9w0BAQEF AAOCAg8AMIICCgKCAgEAwvj257m7ha/E1VTS7wkOEqunZg/yAY/VWyDGHSE9LtvX y0xS08DL1dPpo4TJyX2dmdgQTtj3R/5CfVIPNdk9fy6VRmq7kPomKioSDN+04Eip I/0RaSYtRyhSWxzb7JLPTeb4J4eVGUlpyghaZV3jds2Eb8EESARR81muhtN432Mb tgu6SIf4CR541ArjlBaAnRe94tA0/dL0la4kKtzgFM7dh7qm7vQkDbB5bCMFGd9J BculjqWxbbDU9n3bXF3uKpXeF8dJV3SFzbv3xeerBB/m28isSzPpnBjXi/JMgs4X ZY2T91Zek9dqcGGlcPzkey9obvhazot9fwaTE89vwmuDzsReJuiR/q7vLpEgKGLb lCEAyJAGMczPj2qF/MPI6qOK+WaN7yjis4mVeQdVL0gLqtE1rPFAMSTKXmjvacTr Jpm/IKzPQFquqWuUR/UInOvSwp8AjiHlQTcpx6d8nLPNEvCw6w7nWSPVDAYD4qup VxcAXSGsqgm+p2lLo9vcD7ZZvdpsQN3s2l+P+6if3CFK0yHDjTf6rIUwmXbtOIww VH+fScVSrZLYyk9X3YpXGX/k3vg+LfkOpgHepcqW0A4WpTsVtg5vGEfUct0z6Cmu Ia0IZvecLtjkXeoRYxJiyzk790ZdDmmKfcJ2uS/tVW6B+iQgseinR/ydikPFwoUC AwEAAaNTMFEwHQYDVR0OBBYEFGs4WOV5xq4B6saFQKhYJDLJbu4wMB8GA1UdIwQY MBaAFGs4WOV5xq4B6saFQKhYJDLJbu4wMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI hvcNAQELBQADggIBAD0V/v3+s+RfSdBgDdpp95numDNwQi+y1RuN3VBB/JTAIfz/ pj0b5MU8dAL36fZdMHmV/cwUPxydZkUYBqZ6bpdjPIVVuT/gdr6Sk81Fba2PXHFq Tk6DtbaQpEV47t+aIiPuI4c28cTkU+Ww/jRX8fwaB53RywlbLmshOpSFLLBCRbPP GE85NaFettEnn6Hv3+oBmeVRDpI8z2pNhibcSiptScsPQQ2vpQlnTyAPoBgJv4Ro 65g1qSkPPMfzHx/Syk3yVgCOzZ7DfmGTLnn5sWOM09K+tk3zZUS0zZ6ObJ0rcCfs TtVZeMdHLplrmpjIJGtq64UCkMQKtS7Hn3NkXAwkulpCKptDktikJnX+yOvONftc 6bpwOSmAcSUUqsFJJOhVibPNc81MdC4peUE9L9PPwhCnC/yDBKAd3nUKtnNTeXtp GmPtLopSTZ/6kUPdsOBcPbKprck2SlnpO7pii2LD8cZvWVLppCLUAqGfyCHCdo4n 0oiOiArcZLiI5vOgyBg5SuE8nEevP8NcdV3F4zq9jCOZ9zjoAK5PDDaqhXZe8V/2 3McpsjL0EGtzAXHiPVEgOFaalsrpPGx73oa5Crz9HsrBwdFD6y3fJu7u90Iq6oOp qdRag9mVQ/+GrtGqN4XGGtsrugF9RHp4I/oTmOr6nV9mlLunHwVQqlbuN1Pn -----END CERTIFICATE-----

Then I check it from https://www.ssllabs.com --> it says the cert is invalid

My question is: how to recreate them and make it work asap?

P.S. - It’s an AWS server and the inbound security rules are as per the documentation - allowed I have gone through almost all issues here and I have checked the pre-requisites multiple times already (inluding DNS, wildcard etc.)

Please help. I’m quite clueless now. Maybe a small thing that I’m missing here.

Thanks in advance for your help.