harbor: Internal TLS not working when tutorial followed 2.0.0

Expected behavior and actual behavior: Expected is that Harbor started and is running properly Actual is that nginx-photon is constantly restarted and rest of pod reports errors.

Steps to reproduce the problem: Followed all the steps of tutorial to deploy Harbor with internal tls enabled. In steps there is written to run command docker run -v /:/hostfs goharbor/prepare:v2.0 gencert -p /path/to/internal/tls/cert but version should be v2.0.0. Installation does not show any errors.

Versions: Please specify the versions of following systems.

  • harbor version: 2.0.0
  • docker engine version: 19.03.1
  • docker-compose version: 1.25.5

Additional context:

  • Log files:
  • proxy.log
May 14 12:24:07 172.18.0.1 proxy[60892]: 2020/05/14 10:24:07 [emerg] 1#0: SSL_CTX_load_verify_locations("/harbor_cust_cert/harbor_internal_ca.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/harbor_cust_cert/harbor_internal_ca.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib)
May 14 12:24:07 172.18.0.1 proxy[60892]: nginx: [emerg] SSL_CTX_load_verify_locations("/harbor_cust_cert/harbor_internal_ca.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/harbor_cust_cert/harbor_internal_ca.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib)
May 14 12:24:07 172.18.0.1 proxy[60892]: 2020/05/14 10:24:07 [emerg] 1#0: SSL_CTX_load_verify_locations("/harbor_cust_cert/harbor_internal_ca.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/harbor_cust_cert/harbor_internal_ca.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib)
May 14 12:24:07 172.18.0.1 proxy[60892]: nginx: [emerg] SSL_CTX_load_verify_locations("/harbor_cust_cert/harbor_internal_ca.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/harbor_cust_cert/harbor_internal_ca.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib)`
  • core.log
May 14 12:24:04 172.18.0.1 core[60892]: Appending trust CA to ca-bundle ...
May 14 12:24:04 172.18.0.1 core[60892]:  /harbor_cust_cert/core.crt Appended ...
May 14 12:24:04 172.18.0.1 core[60892]: CA appending is Done.
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/controller/artifact/processor/processor.go:58]: the processor to process media type application/vnd.oci.image.index.v1+json registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/controller/artifact/processor/processor.go:58]: the processor to process media type application/vnd.docker.distribution.manifest.list.v2+json registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/controller/artifact/processor/processor.go:58]: the processor to process media type application/vnd.docker.distribution.manifest.v1+prettyjws registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/controller/artifact/processor/processor.go:58]: the processor to process media type application/vnd.oci.image.config.v1+json registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/controller/artifact/processor/processor.go:58]: the processor to process media type application/vnd.docker.container.image.v1+json registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/controller/artifact/processor/processor.go:58]: the processor to process media type application/vnd.cncf.helm.config.v1+json registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/controller/artifact/processor/processor.go:58]: the processor to process media type application/vnd.cnab.manifest.v1 registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/native/adapter.go:36]: the factory for adapter docker-registry registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/harbor/adaper.go:31]: the factory for adapter harbor registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/dockerhub/adapter.go:25]: Factory for adapter docker-hub registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/huawei/huawei_adapter.go:27]: the factory of Huawei adapter was registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/googlegcr/adapter.go:29]: the factory for adapter google-gcr registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/awsecr/adapter.go:47]: the factory for adapter aws-ecr registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/azurecr/adapter.go:15]: Factory for adapter azure-acr registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/aliacr/adapter.go:31]: the factory for adapter ali-acr registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/jfrog/adapter.go:30]: the factory of jfrog artifactory adapter was registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/quayio/adapter.go:38]: the factory of Quay.io adapter was registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/helmhub/adapter.go:30]: the factory for adapter helm-hub registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/replication/adapter/gitlab/adapter.go:17]: the factory for adapter gitlab registered
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [ERROR] [/common/config/manager.go:118]: loadSystemConfigFromEnv failed, config item, key: clair_db_port,  err: strconv.Atoi: parsing "": invalid syntax
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/core/controllers/base.go:299]: Config path: /etc/core/app.conf
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/core/main.go:111]: initializing configurations...
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/core/config/config.go:83]: key path: /etc/core/key
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [ERROR] [/common/config/manager.go:118]: loadSystemConfigFromEnv failed, config item, key: clair_db_port,  err: strconv.Atoi: parsing "": invalid syntax
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/core/config/config.go:60]: init secret store
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/core/config/config.go:63]: init project manager
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/core/config/config.go:95]: initializing the project manager based on local database...
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/core/main.go:113]: configurations initialization completed
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [INFO] [/common/dao/base.go:84]: Registering database: type-PostgreSQL host-postgresql port-5432 databse-registry sslmode-"disable"
May 14 12:24:05 172.18.0.1 core[60892]: 2020-05-14T10:24:05Z [ERROR] [/common/utils/utils.go:102]: failed to connect to tcp://postgresql:5432, retry after 2 seconds :dial tcp 172.18.0.3:5432: connect: connection refused
May 14 12:24:07 172.18.0.1 core[60892]: 2020-05-14T10:24:07Z [ERROR] [/common/utils/utils.go:102]: failed to connect to tcp://postgresql:5432, retry after 2 seconds :dial tcp 172.18.0.3:5432: connect: connection refused
May 14 12:24:09 172.18.0.1 core[60892]: 2020-05-14T10:24:09Z [INFO] [/common/dao/base.go:89]: Register database completed
May 14 12:24:09 172.18.0.1 core[60892]: 2020-05-14T10:24:09Z [INFO] [/common/dao/pgsql.go:118]: Upgrading schema for pgsql ...
May 14 12:24:10 172.18.0.1 core[60892]: 2020-05-14T10:24:10Z [INFO] [/core/main.go:78]: User id: 1 updated its encrypted password successfully.
May 14 12:24:10 172.18.0.1 core[60892]: 2020-05-14T10:24:10Z [INFO] [/core/main.go:198]: Removing Trivy scanner
May 14 12:24:10 172.18.0.1 core[60892]: 2020-05-14T10:24:10Z [INFO] [/core/main.go:220]: Removing Clair scanner
May 14 12:24:10 172.18.0.1 core[60892]: 2020-05-14T10:24:10Z [INFO] [/core/main.go:156]: initializing notification...
May 14 12:24:10 172.18.0.1 core[60892]: 2020-05-14T10:24:10Z [INFO] [/pkg/notification/notification.go:47]: notification initialization completed
May 14 12:24:10 172.18.0.1 core[60892]: 2020-05-14T10:24:10Z [INFO] [/core/main.go:162]: internal TLS enabled, Init TLS ...
May 14 12:24:10 172.18.0.1 core[60892]: 2020-05-14T10:24:10Z [INFO] [/core/main.go:166]: load client key: /etc/harbor/ssl/core.key client cert: /etc/harbor/ssl/core.crt
May 14 12:24:10 172.18.0.1 core[60892]: 2020-05-14T10:24:10Z [INFO] [/core/main.go:175]: Version: v2.0.0, Git commit: 87602132
May 14 12:24:10 172.18.0.1 core[60892]: 2020/05/14 10:24:10.402 #033[1;34m[I]#033[0m [asm_amd64.s:1357]  https server Running on https://:8443
May 14 12:24:16 172.18.0.1 core[60892]: 2020/05/14 10:24:16.445 [server.go:3054]  [HTTP] http: TLS handshake error from 172.18.0.9:36534: remote error: tls: bad certificate
May 14 12:24:29 172.18.0.1 core[60892]: 2020/05/14 10:24:29.470 [server.go:3054]  [HTTP] http: TLS handshake error from 172.18.0.9:36548: remote error: tls: bad certificate
May 14 12:24:34 172.18.0.1 core[60892]: 2020/05/14 10:24:34.778 #033[1;44m[D]#033[0m [transaction.go:62]  |      127.0.0.1|#033[97;42m 200 #033[0m|    304.516µs|   match|#033[97;44m GET     #033[0m /api/v2.0/ping   r:/api/v2.0/ping
May 14 12:24:48 172.18.0.1 core[60892]: 2020/05/14 10:24:48.497 [server.go:3054]  [HTTP] http: TLS handshake error from 172.18.0.9:36598: remote error: tls: bad certificate
May 14 12:25:04 172.18.0.1 core[60892]: 2020/05/14 10:25:04.935 #033[1;44m[D]#033[0m [transaction.go:62]  |      127.0.0.1|#033[97;42m 200 #033[0m|    319.823µs|   match|#033[97;44m GET     #033[0m /api/v2.0/ping   r:/api/v2.0/ping

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 19 (6 by maintainers)

Most upvoted comments

I believe this issue still exists in harbor v2.2.0. After enabling internal_tls, and running install.sh, the nginx containers enters a restart loop due to the missing CA at common/config/shared/trust-certificates. Copying the file over solves this problem.

Could solve it in my case. The container mounted directory /harbor_cust_cert/ only listed an empty core.crt file which seem to get generated on each docker-compose up. I’ve simply copied the harbor_internal_ca.crt file (generated via gencert) to the hard-coded directory make/common/config/shared/trust-certificates and redeployed containers.

Likely something on my end as I run Harbor on aarch64. As this does not work out-of-the-box I use my own script to build related Harbor images, build scanner binaries, alter some files and do some other stuff.

@kuburoman, as your proxy.log shows the same entries, could you check the content of make/common/config/shared/trust-certificates?