origin: Get https://docker-registry.default.svc:5000/v2/: net/http: TLS handshake timeout

Version

oc version
oc v3.11.0+62803d0-1
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://master.service.dc1.consul:443
openshift v3.11.0+5a84bad-168
kubernetes v1.11.0+d4cacc0

Steps To Reproduce

fresh install of Openshift 3.11
try to deploy the default Appache template to test the cluster

Current Result

Cloning "https://github.com/openshift/httpd-ex.git " ...
--
  | Commit:	0ac6da93a1f65fe9175cb1b7838cfca7b23d5fbe (Merge pull request #15 from adambkaplan/sclorg-rename)
  | Author:	Honza Horak <hhorak@redhat.com>
  | Date:	Fri Aug 3 13:08:12 2018 +0200
  | pulling image error : Get https://docker-registry.default.svc:5000/v2/:  net/http: TLS handshake timeout
  | error: build error: unable to get docker-registry.default.svc:5000/openshift/httpd@sha256:d1256b39182b0ac5290c946dc44fc11055524683113a1b5e3a55d83044a185cb

Expected Result

The deployment is ok

Additional Information

I haven’t overriden the certificate. As the doc says “you should not need to replace the certificate used by the registry service itself.”

The registry pod is running

oc get pod
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-94xbh    1/1       Running   0          1d
registry-console-1-mn8tb   1/1       Running   0          1d
router-1-jfdtp             1/1       Running   0          1d
router-1-sr44x             1/1       Running   0          1d

But when I try to connect to the service

oc get service
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                   AGE
docker-registry    ClusterIP   172.30.95.44    <none>        5000/TCP                  1d
kubernetes         ClusterIP   172.30.0.1      <none>        443/TCP,53/UDP,53/TCP     1d
openshift-master   ClusterIP   172.30.34.165   <none>        443/TCP                   1d
registry-console   ClusterIP   172.30.141.37   <none>        9000/TCP                  1d
router             ClusterIP   172.30.32.90    <none>        80/TCP,443/TCP,1936/TCP   1d

openssl s_client -connect 172.30.95.44:5000
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1555591482
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

I checked taht the registry certificate was ok with the ca

openssl verify -verbose -CAfile /etc/origin/master/ca.crt /etc/origin/master/registry.crt 
/etc/origin/master/registry.crt: OK

Info about the certificate:

sudo openssl x509 -in /etc/origin/master/registry.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 14 (0xe)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=openshift-signer@1555495169
        Validity
            Not Before: Apr 17 10:08:49 2019 GMT
            Not After : Apr 16 10:08:50 2021 GMT
        Subject: CN=172.30.95.44
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b4:ff:6c:a7:2d:2b:35:22:d6:21:b6:5a:45:e9:
                    f9:b5:42:f9:a8:38:60:90:48:71:10:28:bf:55:cf:
                    aa:d5:48:0a:70:62:fc:4f:97:52:de:aa:ad:d6:8d:
                    39:60:9a:64:d2:c2:20:98:91:65:01:b8:2a:e8:fb:
                    e5:6f:f8:96:c0:19:6d:62:c2:6f:74:72:43:eb:0d:
                    f8:bd:18:5e:e3:8b:83:00:f8:22:c1:96:f5:ad:74:
                    c0:18:38:99:c7:74:5c:3c:19:07:20:c5:9e:6c:fe:
                    61:36:07:1c:fa:6b:3f:da:eb:24:90:ea:19:53:34:
                    1c:4a:45:9c:b3:39:2f:f1:52:52:ed:4e:fe:35:cd:
                    b6:6d:81:4f:f5:2c:65:7a:c3:35:4a:da:03:a8:79:
                    41:fc:6a:62:63:1c:49:b4:c8:6e:90:2c:8e:ed:7e:
                    ee:81:41:ab:da:49:77:11:4a:8c:5e:c0:c1:20:89:
                    b7:9f:b3:37:56:0b:d9:2d:aa:c1:66:42:5c:3b:0a:
                    c1:da:db:79:fd:b1:d7:36:cb:a1:e7:f0:88:27:02:
                    2f:74:fd:26:81:8a:82:42:e9:73:00:02:cd:55:2d:
                    15:14:9f:d2:9c:60:fa:7f:0b:88:6b:24:79:ab:d1:
                    f6:f1:dd:a0:74:60:3f:f0:eb:e5:c0:79:d0:f7:dc:
                    b2:a5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:docker-registry-default.gitops.kmt.orange.com, DNS:docker-registry.default.svc, DNS:docker-registry.default.svc.cluster.local, DNS:172.30.95.44, IP Address:172.30.95.44
    Signature Algorithm: sha256WithRSAEncryption
         5c:ac:6d:d8:b1:df:f8:0f:d2:3f:76:26:1b:94:97:38:ad:10:
         92:c6:2e:f1:5e:e1:fc:d1:2c:ce:59:fd:a3:0e:57:58:12:b8:
         2c:b4:ee:bc:36:86:95:4b:46:f1:7e:ff:12:a1:53:dd:85:1c:
         bc:3c:27:8b:0b:e7:ff:cc:b0:d7:7e:b1:9f:9a:c0:fc:47:4f:
         4e:e9:f0:51:ba:1e:fb:c5:76:49:7a:fa:3d:ff:36:4a:79:79:
         59:0e:8f:54:90:08:7e:f1:7a:f4:9e:96:67:72:82:95:08:c6:
         93:80:f0:f2:d6:65:cf:59:82:94:f0:13:de:a1:fc:1e:0e:f4:
         dd:15:59:4a:12:99:20:dd:6c:25:ed:af:49:ab:a0:f2:cf:f3:
         a9:be:2d:7d:3f:6b:75:d5:d9:50:9d:a9:8a:62:79:82:64:9a:
         63:36:4d:86:79:12:e1:0b:e7:ca:80:af:84:41:be:20:b5:50:
         dc:6b:1d:ac:c8:38:58:c0:35:16:10:41:59:c4:20:a5:c5:bd:
         1e:79:9b:42:8f:da:52:06:38:3a:95:8a:58:5d:84:d9:fb:08:
         e9:e8:fa:66:d2:6c:2a:1e:6c:08:d9:84:ce:e4:cc:1c:fc:c2:
         2f:95:24:c7:46:97:5b:48:2b:da:c8:e7:9c:c0:bb:bd:66:03:
         38:17:50:48

Any ideas?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (4 by maintainers)

Most upvoted comments

We experienced the same issue in our environment and it was a MTU problem. We’ll probably be filing another issue for this, but basically our env uses 1450 (instead of 1500) for MTU and in that case OpenShift install should adjust it to 1400. This has worked in the past, but recently it didn’t - i.e.: it left everything at 1450 - and we experienced this issue. After manually adjusting our nodes to use 1400, it worked fine again.

oybed on May 10, 2019

As ROOT let resolv.conf editable. chattr -ai /etc/resolv.conf

Edit it vi /etc/resolv.conf

Add those on search entry search cluster.local default.svc.cluster.local svc.cluster.local cluster.local default.svc

Activate it systemctl restart NetworkManager

ameagol on Nov 23, 2019

@woland7 there are multiple ways you can do this: During install you can set the openshift_node_sdn_mtu inventory variable, or set the node configmap values for MTU. Or, post install you can change the configmap values for the MTU for your nodes (including any infra and/or master nodes) and do a node reboot.

> oc get cm -n openshift-node node-config-compute -o yaml | grep -B1 mtu
    networkConfig:
      mtu: 1400

oybed on Jun 4, 2019

We experienced the same issue in our environment and it was a MTU problem. We’ll probably be filing another issue for this, but basically our env uses 1450 (instead of 1500) for MTU and in that case OpenShift install should adjust it to 1400. This has worked in the past, but recently it didn’t - i.e.: it left everything at 1450 - and we experienced this issue. After manually adjusting our nodes to use 1400, it worked fine again.

I had a Openshift setup running over Openstack VM’s with this same problem and it worked perfectly. The networks settings of MTU was 1400 for eth0, 1500 for docker0 and 1450 for tun0. Changing docker0 to 1300 and tun0 to 1350 solved my issue.

Rikeard on May 14, 2019