grpc: Decryption error: TSI_DATA_CORRUPTED

What version of gRPC and what language are you using?

grpc-node 1.24.2

What operating system (Linux, Windows,…) and version?

  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 5.4.42-v8+
      Architecture: arm64

openssl:arm64/buster 1.1.1d-0+deb10u3 uptodate

What runtime / compiler are you using (e.g. python version or version of gcc)

node

What did you do?

As GRPC-Client, connecting to a Server using Server-Side SSL encryption. Using duplex streaming.

What did you expect to see?

A stable connection.

What did you see instead?

recurring disconnects:

ssl_transport_security.cc:483] Corruption detected.
ssl_transport_security.cc:459] error:14187180:SSL routines:ssl_do_config:bad value
secure_endpoint.cc:208]     Decryption error: TSI_DATA_CORRUPTED

Anything else we should know about your project / environment?

This seems to be related to the environment. I can reproduce this on a Raspberry Pi 3 having Debian 10 (arm64) installed. When using Ubuntu 18 (arm64) instead on the same device, the problem does not occur.

Therefore my question: What system libraries (or certs) are used by the C++ layer? And which version of this library would you recommend to use?

Additional info: Seems like unary calls are working, just (duplex) streaming is affected.

I have seen this decryption error was already mentioned in other tickets / environments, but seems like they never got fixed: #12642 #15506

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 15 (5 by maintainers)

Commits related to this issue

ci: re-enable generator on debian buster My goal is to reproduce the suspected gRPC + OpenSSL bug described in grpc/grpc#23144, which we also saw in https://github.com/googleapis/google-cloud-cpp/pul... — committed to devjgm/google-cloud-cpp by devjgm 3 years ago
Clear SSL error stacks before performing SSL I/O. OpenSSL's SSL_get_error returns an error if a previous unrelated error happened to be on the stack before the most recent SSL I/O operation. These er... — committed to bdhess/grpc by bdhess 3 years ago

Most upvoted comments

@ZhenLian Thanks for looking into the issue.

The error occurs on client side. Both, client and server use Node.js (common code project). At the beginning (clean OS install), the error occured conistently (every duplex streaming request failed). Later, I realized the default openssl config on Debian 10 is different from Ubuntu 19.10 (which worked fine for months). Debian 10 got an additional MinProtocol and CipherString entry. After having removed MinProtocol it first seemed to work, but I experienced the error again after about an hour. After removing both, MinProtocol and CipherString the connection is pretty stable.

To clarify: Debian 10 was used on client side when the problem occured. Server is Ubuntu 19.10.

Note: For me, it was not clear that the systems openssl config must be considered when using grpc on Node.js. Mentioning this dependency in the docs would be helpful.

sjkummer on Jun 17, 2020