pomerium: OCSP staple sometimes missing

What happened?

Note I cannot reproduce this issue but I’m pretty certain it happened.

It looks like in some cases the HTTPS response does not contain OCSP staple result, which conflicts with OCSP Must-Staple and cause HTTPS connection to fail.

This only happened to one of my routes. Chrome would fail to connect. When I test the server using ssllabs.com it returned A+ with warning that there’s no OCSP stapling in the response.

Restart the server seems to have fixed the issue. I do notice the route’s certificate was recently renewed on Feb 19 but not sure if this is related.

What did you expect to happen?

How’d it happen?

Ran x
Clicked y
Saw error z

What’s your environment like?

Pomerium version (retrieve with pomerium --version): 0.13.0
Server Operating System/Architecture/Cloud:

What’s your config.yaml?

# Paste your configs here
# Be sure to scrub any sensitive values

What did you see in the logs?

# Paste your logs here.
# Be sure to scrub any sensitive values

Additional context

Add any other context about the problem here.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (8 by maintainers)

Most upvoted comments

Thanks for opening this issue. Until now I’ve been unable to find the cause of the Chrome error: ERR_SSL_VERSION_OR_CIPHER_MISMATCH and Firefox: SSL_ERROR_NO_CYPHER_OVERLAP

Running the ssllabs.com test, I’m getting the same

However for me, the error is completely persistent and predictably happens within 1-2 weeks of running Pomerium via docker-compose. I cannot get the same logs as others here, the only thing I see in the debug output is:

{"level":"debug","service":"envoy","name":"filter","time":"2021-06-09T10:42:02Z","message":"tls inspector: new connection accepted"}
{"level":"debug","service":"envoy","name":"filter","time":"2021-06-09T10:42:02Z","message":"tls:onServerName(), requestedServerName: redacted"}
{"level":"debug","service":"envoy","name":"conn_handler","time":"2021-06-09T10:42:02Z","message":"[C345975] new connection"}
{"level":"debug","service":"envoy","name":"connection","time":"2021-06-09T10:42:02Z","message":"[C345975] TLS error: 268435589:SSL routines:OPENSSL_internal:CONNECTION_REJECTED"}
{"level":"debug","service":"envoy","name":"connection","time":"2021-06-09T10:42:02Z","message":"[C345975] closing socket: 0"}
{"level":"debug","service":"envoy","name":"conn_handler","time":"2021-06-09T10:42:02Z","message":"[C345975] adding to cleanup list"}

If needed, I can provide access to a server that has the issue, but will take 1-2 weeks to setup.

With the specific error, TLS error: 268435589 being zero-hits on google.

Restarting the affected container immediately allows connections to pass again in Chrome and Firefox.

Is there a workaround for this other than regular scheduling of proxy and authenticate containers to restart? Unfortunately my healthcheck with curl does not reject the missing OSCP stapling, so I’m also a bit at a loss of how exactly to detect this problem.

dsschneidermann on Jun 9, 2021

This also still happen to me. I decide to post my full configuration (with information redacted) in case it can help the investigation.

docker-compose config (in jsonnet format):

  pomerium: uses_adguard {
    image: 'docker.io/pomerium/pomerium:v0.13.6',
    hostname: 'foo.bar.com',
    sysctls: {
      'net.ipv6.conf.all.disable_ipv6': 0,
    },
    environment: super.environment {
      AUTHENTICATE_SERVICE_URL: 'https://foo.bar.com',
      AUTOCERT: '1',
      AUTOCERT_MUST_STAPLE: '1',
      COOKIE_SECRET: '${POMERIUM_COOKIE_SECRET}',
      GRPC_CLIENT_DNS_ROUNDROBIN: '0',
      HTTP_REDIRECT_ADDR: ':80',
      IDP_CLIENT_ID: '${POMERIUM_CLIENT_ID}',
      IDP_CLIENT_SECRET: '${POMERIUM_CLIENT_SECRET}',
      IDP_PROVIDER: 'google',
      IDP_PROVIDER_URL: 'https://accounts.google.com',
      JWT_CLAIMS_HEADERS: 'email',
      LOG_LEVEL: 'info',
      METRICS_ADDRESS: ':9090',
      POLICY: std.base64(importstr 'pomerium/pomerium.yaml'),
      POMERIUM_DEBUG: '1',
      SHARED_SECRET: '${POMERIUM_SHARED_SECRET}',
      SIGNING_KEY: std.base64(importstr 'pomerium/ed25519_private.pem'),
      SIGNING_KEY_ALGORITHM: 'EdDSA',
    },
    volumes: [
      '/opt/pomerium:/data/autocert',
    ],
    ports: [
      '443:443/tcp',
      '80:80/tcp',
      '8843:443/tcp',
    ],
  },

pomerium.yaml file:

- from: https://1.bar.com
  to: http://1
  pass_identity_headers: true
  allowed_users:
    - A@gmail.com
- from: https://2.bar.com
  to: http://2:8080
  allowed_users:
    - A@gmail.com
  pass_identity_headers: true
- from: https://3.bar.com
  to: http://3:9000
  allowed_users:
    - A@gmail.com
- from: https://4.bar.com
  to: http://4.local
  timeout: 10m
  allowed_users:
    - A@gmail.com
- from: https://5.bar.com
  to: http://5.local
  timeout: 10m
  allowed_users:
    - A@gmail.com
- from: https://6.bar.com
  to: http://6
  path: /foo-bar
  allow_public_unauthenticated_access: true
- from: https://6.bar.com
  to: http://6
  allowed_users:
    - A@gmail.com
    - B@gmail.com
- from: https://7.bar.com
  to: http://7.local
  allowed_users:
    - A@gmail.com
- from: https://8.bar.com
  to: http://8.local
  allowed_users:
    - A@gmail.com
- from: https://9.bar.com
  to: http://172.17.x.x:xxxx
  allow_websockets: true
  allow_public_unauthenticated_access: true
- from: https://10.bar.com
  to: http://10
  allow_public_unauthenticated_access: true
- from: https://11.bar.com
  to: http://11:9090
  allowed_users:
    - A@gmail.com
- from: https://12.bar.com
  to: http://12:3000
  allowed_users:
    - A@gmail.com
    - B@gmail.com
  pass_identity_headers: true
- from: https://13.bar.com
  to: http://13
  pass_identity_headers: true
  allowed_users:
    - A@gmail.com
- from: https://14.bar.com
  to: http://14.local
  allowed_users:
    - A@gmail.com
- from: https://15.bar.com
  to: https://15:xxxxx
  allow_public_unauthenticated_access: true
  tls_server_name: aaa.bbb.com
  allow_websockets: true
- from: https://15.bar.com:443
  to: https://15:xxxxx
  allow_public_unauthenticated_access: true
  tls_server_name: aaa.bbb.com
  allow_websockets: true
- from: https://16.bar.com
  to: https://16:8443
  set_request_headers:
    Referer: https://16:8443
  remove_request_headers:
    - Origin
  allow_websockets: true
  tls_skip_verify: true
  allow_public_unauthenticated_access: true
- from: https://16.bar.com:8843
  to: https://16:8843
  allow_public_unauthenticated_access: true
  set_request_headers:
    Referer: https://16:8443
  remove_request_headers:
    - Origin
  allow_websockets: true
  tls_skip_verify: true
- from: https://17.bar.com
  to: https://17:8843
  allow_public_unauthenticated_access: true
  set_request_headers:
    Referer: https://17:8443
  remove_request_headers:
    - Origin
  allow_websockets: true
  tls_skip_verify: true
- from: https://18.bar.com
  to: http://192.168.x.x:xxxx
  allowed_users:
    - A@gmail.com
- from: https://19.bar.com
  to: http://192.168.x.x:xxxx
  allowed_users:
    - A@gmail.com
- from: https://20.bar.com
  to: http://20.local:xxxx/
  allowed_users:
    - A@gmail.com
  timeout: 10m
  set_request_headers:
      Authorization: Basic xxxx
- from: https://21.bar.com
  allow_public_unauthenticated_access: true
  redirect:
      host_redirect: foo.com
      path_redirect: /bar/baz
- from: https://22.bar.com
  to: http://22:4000
  allowed_users:
    - A@gmail.com
  allow_websockets: true
- from: https://23.bar.com
  to: http://23.local:8080
  remove_request_headers:
    - Origin
    - Referer
  allow_public_unauthenticated_access: true

yegle on Jun 5, 2021

And it happened again. ERR_SSL_VERSION_OR_CIPHER_MISMATCH SSLlabs reports: This server certificate supports OCSP must staple but OCSP response is not stapled. And a lot of handshake failures:

Even with POMERIUM_DEBUG TRUE, I dont find anything in the pomerium logs regarding the failed requests. I will leave it in this state, if something needs to be tested.

fightforlife on Jun 5, 2021