sealed-secrets: Timeout on fetch cert

On initial installation on AWS I get the following timeout error:

$ kubeseal --fetch-cert -v 10000
I0302 16:37:04.066027   36889 loader.go:359] Config loaded from file:  /home/pal/.kube/config
I0302 16:37:04.066646   36889 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://REDACTED.sk1.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/http:sealed-secrets-controller:/proxy/v1/cert.pem'
I0302 16:37:34.951625   36889 round_trippers.go:438] GET https://REDACTED.sk1.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/http:sealed-secrets-controller:/proxy/v1/cert.pem 503 Service Unavailable in 30884 milliseconds
I0302 16:37:34.951681   36889 round_trippers.go:444] Response Headers:
I0302 16:37:34.951696   36889 round_trippers.go:447]     Audit-Id: 8f9e456d-7cd3-42e6-8871-bdd2e99608fa
I0302 16:37:34.951703   36889 round_trippers.go:447]     Date: Mon, 02 Mar 2020 15:37:34 GMT
I0302 16:37:34.951775   36889 request.go:947] Response Body: Error: 'dial tcp 10.167.172.10:8080: i/o timeout'
Trying to reach: 'http://10.167.172.10:8080/v1/cert.pem'
I0302 16:37:34.951834   36889 request.go:1150] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: invalid character 'E' looking for beginning of value
error: cannot fetch certificate: the server is currently unable to handle the request (get services http:sealed-secrets-controller:)

I applied the controller at https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.9.8/controller.yaml and installed a precompiled cli from https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.9.8/kubeseal-linux-amd64

Some additional debugging:

$ kubectl --namespace kube-system describe svc sealed-secrets-controller
Name:              sealed-secrets-controller
Namespace:         kube-system
Labels:            name=sealed-secrets-controller
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"name":"sealed-secrets-controller"},"name":"sealed-secrets-cont...
Selector:          name=sealed-secrets-controller
Type:              ClusterIP
IP:                172.20.209.255
Port:              <unset>  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.167.172.10:8080
Session Affinity:  None
Events:            <none>

I am able to do a port forward like so:

kubectl --namespace kube-system port-forward svc/sealed-secrets-controller 8081:8080

And then curl the cert:

$ curl localhost:8081/v1/cert.pem
-----BEGIN CERTIFICATE-----
....

About this issue

Original URL
State: open
Created 4 years ago
Comments: 23

Most upvoted comments

For anyone who referred to this issue, I have had the same problem with EKS. The reason is that the Cluster API server needs to call the controller. However, the request was blocked by the security group of the node where the pod is deployed.

So, you need to allow the port 8080 for the entry rule of the node’s security group.

I hope that could help someone.

renxunsaky on Aug 31, 2022

JFYI: I’m using provider Linode and getting same error:

$ kubeseal --fetch-cert -v 10 > kubeseal.pem

I0730 14:44:00.291228   23849 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer blabla/api/v1/namespaces/kube-system/services/http:sealed-secrets-controller:/proxy/v1/cert.pem'
I0730 14:44:30.974004   23849 round_trippers.go:443] GET https://blabla.net:443/api/v1/namespaces/kube-system/services/http:sealed-secrets-controller:/proxy/v1/cert.pem 503 Service Unavailable in 30682 milliseconds
I0730 14:44:30.974048   23849 round_trippers.go:449] Response Headers:
I0730 14:44:30.974069   23849 round_trippers.go:452]     Content-Length: 191
I0730 14:44:30.974079   23849 round_trippers.go:452]     Date: Fri, 30 Jul 2021 11:44:30 GMT
I0730 14:44:30.974089   23849 round_trippers.go:452]     Cache-Control: no-cache, private
I0730 14:44:30.974098   23849 round_trippers.go:452]     Content-Type: application/json
I0730 14:44:30.974161   23849 request.go:968] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"error trying to reach service: dial tcp 10.2.193.7:8080: i/o timeout","reason":"ServiceUnavailable","code":503}
error: cannot fetch certificate: error trying to reach service: dial tcp 10.2.193.7:8080: i/o timeout

No network rules, no custom controller configuration, +just fresh cluster.

Workaround: using kube proxy and offline sealing

SerhatTeker on Jul 30, 2021

Thanks @renxunsaky !

For people not familiar with manual customisation of security groups:

In AWS console, search for “Security Groups”
Find your Security Group dedicated for node (For me, it was “clustername-node”)
Check the box on the left, then “Actions” > “Edit inbound rules”.
Clic “Add Rule”. Type “Custom TCP”, Port range “8080”, Source “Custom”, select the security related to Cluster API. Add a description.
Save rule.

Here we go

cosad3s on Jul 10, 2023

Same timeout with DigitalOcean in October 2021. Kubeseal v0.16.0, K8s 1.21.3. Helm chart 1.16.1, using Flux v2.

kube proxy workaround worked (different namespace due to Flux)

flurdy on Oct 23, 2021

There is another approach detailed in #282 Both can and should coexist since they strike different tradeoffs.

I’ll close this issue when the PR that implements the fix lands.

mkmik on Mar 3, 2020