cert-manager: Getting "cluster doesn't provide client-ca-file"
I attempted at deploying cert-manager 0.5.2 with Helm but running into the following error and the pod for the webhook is in a crashloop.
I get the following error message,
ERROR: logging before flag.Parse: I0116 10:26:12.457905 1 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: webhook/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */*" -H "Authorization: Bearer <not given for security reasons>" https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
ERROR: logging before flag.Parse: I0116 10:26:12.672853 1 round_trippers.go:405] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 214 milliseconds
ERROR: logging before flag.Parse: I0116 10:26:12.672880 1 round_trippers.go:411] Response Headers:
ERROR: logging before flag.Parse: I0116 10:26:12.672885 1 round_trippers.go:414] Audit-Id: 90541976-528d-4d36-a88a-aa2d50630c9d
ERROR: logging before flag.Parse: I0116 10:26:12.672891 1 round_trippers.go:414] Content-Type: application/json
ERROR: logging before flag.Parse: I0116 10:26:12.672896 1 round_trippers.go:414] Content-Length: 1634
ERROR: logging before flag.Parse: I0116 10:26:12.672900 1 round_trippers.go:414] Date: Wed, 16 Jan 2019 10:26:12 GMT
ERROR: logging before flag.Parse: I0116 10:26:12.672937 1 request.go:874] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"ce2b6f64-17b8-11e9-a6dd-021a269d3ce8","resourceVersion":"39","creationTimestamp":"2019-01-14T04:56:51Z"},"data":{"requestheader-allowed-names":"[\"front-proxy-client\"]","requestheader-client-ca-file":"<not given for security reasons>","requestheader-extra-headers-prefix":"[\"X-Remote-Extra-\"]","requestheader-group-headers":"[\"X-Remote-Group\"]","requestheader-username-headers":"[\"X-Remote-User\"]"}}
Error: cluster doesn't provide client-ca-file
Usage:
[flags]
Flags:
--admission-control-config-file string File with admission control configuration.
--alsologtostderr log to standard error as well as files
--audit-log-batch-buffer-size int The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
--audit-log-batch-max-size int The maximum size of a batch. Only used in batch mode. (default 400)
--audit-log-batch-max-wait duration The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode. (default 30s)
--audit-log-batch-throttle-burst int Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode. (default 15)
--audit-log-batch-throttle-enable Whether batching throttling is enabled. Only used in batch mode.
--audit-log-batch-throttle-qps float32 Maximum average number of batches per second. Only used in batch mode. (default 10)
--audit-log-format string Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Requires the 'AdvancedAuditing' feature gate. Known formats are legacy,json. (default "json")
--audit-log-maxage int The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.
--audit-log-maxbackup int The maximum number of old audit log files to retain.
--audit-log-maxsize int The maximum size in megabytes of the audit log file before it gets rotated.
--audit-log-mode string Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking. (default "blocking")
--audit-log-path string If set, all requests coming to the apiserver will be logged to this file. '-' means standard out.
--audit-policy-file string Path to the file that defines the audit policy configuration. Requires the 'AdvancedAuditing' feature gate. With AdvancedAuditing, a profile is required to enable auditing.
--audit-webhook-batch-buffer-size int The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
--audit-webhook-batch-max-size int The maximum size of a batch. Only used in batch mode. (default 400)
--audit-webhook-batch-max-wait duration The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode. (default 30s)
--audit-webhook-batch-throttle-burst int Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode. (default 15)
--audit-webhook-batch-throttle-enable Whether batching throttling is enabled. Only used in batch mode. (default true)
--audit-webhook-batch-throttle-qps float32 Maximum average number of batches per second. Only used in batch mode. (default 10)
--audit-webhook-config-file string Path to a kubeconfig formatted file that defines the audit webhook configuration. Requires the 'AdvancedAuditing' feature gate.
--audit-webhook-initial-backoff duration The amount of time to wait before retrying the first failed request. (default 10s)
--audit-webhook-mode string Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking. (default "batch")
--authentication-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io.
--authentication-skip-lookup If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
--authentication-token-webhook-cache-ttl duration The duration to cache responses from the webhook token authenticator. (default 10s)
--authorization-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io.
--authorization-webhook-cache-authorized-ttl duration The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
--authorization-webhook-cache-unauthorized-ttl duration The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
--bind-address ip The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). (default 0.0.0.0)
--cert-dir string The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
--client-ca-file string If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
--contention-profiling Enable lock contention profiling, if profiling is enabled
--disable-admission-plugins stringSlice admission plugins that should be disabled although they are in the default enabled plugins list. Comma-delimited list of admission plugins: Initializers, MutatingAdmissionWebhook, NamespaceLifecycle, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
--enable-admission-plugins stringSlice admission plugins that should be enabled in addition to default enabled ones. Comma-delimited list of admission plugins: Initializers, MutatingAdmissionWebhook, NamespaceLifecycle, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
--enable-swagger-ui Enables swagger ui on the apiserver at /swagger-ui
-h, --help help for this command
--http2-max-streams-per-connection int The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default. (default 1000)
--kubeconfig string kubeconfig file pointing at the 'core' kubernetes server.
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--logtostderr log to standard error instead of files (default true)
--profiling Enable profiling via web interface host:port/debug/pprof/ (default true)
--requestheader-allowed-names stringSlice List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
--requestheader-client-ca-file string Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers
--requestheader-extra-headers-prefix stringSlice List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
--requestheader-group-headers stringSlice List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
--requestheader-username-headers stringSlice List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
--secure-port int The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 443)
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--tls-cert-file string File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
--tls-cipher-suites stringSlice Comma-separated list of cipher suites for the server. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). If omitted, the default Go cipher suites will be used
--tls-min-version string Minimum TLS version supported. Value must match version names from https://golang.org/pkg/crypto/tls/#pkg-constants.
--tls-private-key-file string File containing the default x509 private key matching --tls-cert-file.
--tls-sni-cert-key namedCertKey A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
-v, --v Level log level for V logs (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
ERROR: logging before flag.Parse: F0116 10:26:12.756308 1 cmd.go:42] cluster doesn't provide client-ca-file
Looking at extension-apiserver-authentication
I can see that client-ca-file
is missing.
kubectl get configmap extension-apiserver-authentication -n kube-system -o yaml
apiVersion: v1
data:
requestheader-allowed-names: '["front-proxy-client"]'
requestheader-client-ca-file: |
<not given for security reasons>
requestheader-extra-headers-prefix: '["X-Remote-Extra-"]'
requestheader-group-headers: '["X-Remote-Group"]'
requestheader-username-headers: '["X-Remote-User"]'
kind: ConfigMap
metadata:
creationTimestamp: 2019-01-14T04:56:51Z
name: extension-apiserver-authentication
namespace: kube-system
resourceVersion: "39"
selfLink: /api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
uid: ce2b6f64-17b8-11e9-a6dd-021a269d3ce8
Any idea what could be causing this ? I have not tried reaching out to AWS yet.
Environment details::
- Kubernetes version (e.g. v1.10.2): 1.11.5
- Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): AWS EKS
- cert-manager version (e.g. v0.4.0): 0.5.2
- Install method (e.g. helm or static manifests): Helm
/kind bug
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (5 by maintainers)
Commits related to this issue
- Upgrade to the latest version of `cert-manager` - We need to apply the `cert-manager` CRDs ourselves [1]. - We have to disable some `cert-manager`-specific validation in the `ingress-system` namespa... — committed to alphagov/gsp by samcrang 5 years ago
- Upgrade to the latest version of `cert-manager` - We need to apply the `cert-manager` CRDs ourselves [1]. - We have to disable some `cert-manager`-specific validation in the `ingress-system` namespa... — committed to alphagov/gsp by samcrang 5 years ago
@gordcorp nice find! Thanks for digging into this for us.
Hopefully the 1.13 apimachinery PR will land soon, but if that takes a while it may be worth us putting in a patch to bump within the 1.10 series and cherry picking this into v0.6 😄
/reopen /priority important-soon /help
I did some digging, and noticed there was a feature request in apiserver to address this issue: https://github.com/kubernetes/kubernetes/issues/65724 cert-manager is using apiserver 1.10.0, and the PR which added the feature was cherry-picked into 1.10.7, so I tried it by bumping cert- apiserver to 1.10.7 and it worked for me on EKS. I was going to put in a PR, but I saw there is already one to bump all the k8s dependencies to 1.13 which would include the fix for this issue. Since that’s already well in progress, I wont bother with my PR, but let me know if it’s still needed.
@MilanDasek In case you are wondering version
0.6.2
has already been released withv1.10.12
k8 deps so you no longer need to manually build it.I’ve opened https://github.com/jetstack/cert-manager/pull/1344 to bump the 1.10 dependencies 😄