security: Can't use Let’s Encrypt certificate

I run Open Distro for Elasticsearch in Docker container with Docker compose. When I mount self-signed certificates and CA, Elasticsearch works as expected. When I mount Let’s Encrypt certificates, I get errors:

[2019-05-22T15:33:17,757][ERROR][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [es01] Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.
...
Caused by: org.elasticsearch.ElasticsearchSecurityException: Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /usr/share/elasticsearch/config/node-key.pem
...
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /usr/share/elasticsearch/config/node-key.pem

Kibana as well as test Nginx and Apache containers work fine with the same pem files.

Here are some validations to show that certificates are fine.

Openssl verify returns OK:

openssl verify -verbose -CAfile lets_root_chain.pem fullchain.pem
fullchain.pem: OK

lets_root_chain.pem is a concatenation of isrgrootx1.pem.txt (ISRG Root X1 (self-signed)) and letsencryptauthorityx3.pem.txt (Let’s Encrypt Authority X3 (Signed by ISRG Root X1)).

fullchain.pem and privkey.pem are the pair:

openssl x509 -noout -modulus -in fullchain.pem | openssl md5
openssl rsa -noout -modulus -in privkey.pem | openssl md5
(stdin)= 8e3628a899bce3a92471e5c99dcbc62c
(stdin)= 8e3628a899bce3a92471e5c99dcbc62c

I’ve tried both wildcard and single host certificates. And of course the key has no password.

elasticsearch.yml

...
# Transport layer TLS
opendistro_security.ssl.transport.pemcert_filepath: node-cert.pem
opendistro_security.ssl.transport.pemkey_filepath: node-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false

# REST layer TLS
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: node-cert.pem
opendistro_security.ssl.http.pemkey_filepath: node-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
...

docker-compose.yml

...
    volumes:
      - es-data01:/usr/share/elasticsearch/data
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./internal_users.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml
      - ${SSL_CERT}:/usr/share/elasticsearch/config/node-cert.pem
      - ${SSL_KEY}:/usr/share/elasticsearch/config/node-key.pem
      - ${SSL_CA}:/usr/share/elasticsearch/config/root-ca.pem
...

.env:

SSL_CA=./lets_root_chain.pem
SSL_CERT=./fullchain.pem
SSL_KEY=./privkey.pem

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 9
Comments: 19 (2 by maintainers)

Most upvoted comments

installed standalone security plugin to ECK image, met the same problem with ECK generated self-signed certs

xial-thu on May 7, 2020

Something I ran into with TLS is that ODFE doesn’t support some format of certificates. I’m not sure if Lets’s Encrypt will support this format for the server key.

IMPORTANT: Convert these to PKCS#5 v1.5 to work correctly with the JDK. Output from this command will be used in all the config files.

https://aws.amazon.com/blogs/opensource/add-ssl-certificates-open-distro-for-elasticsearch/

jeff-cook on Jun 17, 2019

Just encase anyone else experiencing this basically 2 issues with LE certs and opendistro-es

First is private key type: Fix is:

  privateKey:
    encoding: PKCS8

Second is the lack of ca.crt in cert-manager LE secrets. This can be worked around by setting the …CASubPath: tls.crt technically at this point to my understand your validating the cert is the cert vs it signed by the ISRG Root X1 but it does allow es to start with LE certs and then those certs be externally valid in the browser.

Full Example

elastic-example-com.yaml

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: elastic-example-com
  namespace: elastic
spec:
  commonName: elastic.example.com
  dnsNames:
  - 'elastic.example.com'
  privateKey:
    encoding: PKCS8
  secretName: elastic-example-com-tls
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer

values.yml

---
elasticsearch:
  ssl:
    transport:
      enabled: true
      existingCertSecret: elastic-example-com-tls
      existingCertSecretCertSubPath: tls.crt
      existingCertSecretKeySubPath: tls.key
      existingCertSecretRootCASubPath: tls.crt
...
...
    opendistro_security.nodes_dn:
      - 'CN=elastic.example.com'
...

jamestutton on Feb 24, 2021

/cc @mikemaxey @maraino @dopey for an option on step ca certificate to choose the private key format?

@QuingKhaos I’ve added an issue for this some time ago https://github.com/smallstep/cli/issues/387, and now I think we should be providing PKCS#8 keys by default.

maraino on Feb 22, 2021

Same problem for cfssl on EC PRIVATE KEY

mhf-ir on Aug 11, 2019