grpc: Python/Core: Error parsing http status code - `401` converted to `UNKNOWN: Stream removed`

What version of gRPC and what language are you using?

gRPC v1.56.2 Python

What operating system (Linux, Windows,…) and version?

Mac (client) Linux (server)

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.10

What did you do?

Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.

#31657 (closed as fixed in v1.51.3) appears to have resurfaced for versions ==v1.53.1 and >=v1.54.2. We first noticed it while running v1.56.2.

Our particular issue is reproducible by running a gRPC service behind an NGINX reverse proxy with subrequest authentication enabled. When the auth endpoint returns a 401, the client errors with UNKNOWN: Stream removed instead of translating the http error to the gRPC code: UNAUTHENTICATED (16). A complete, minimal reproducible example is available here: https://github.com/jeevb/grpc-regression-mre

What did you expect to see?

From gRPC v1.51.3:

E0728 20:36:16.829609000 8093441536 hpack_parser.cc:1218]              Error parsing metadata: error=invalid value key=content-type value=text/html
Traceback (most recent call last):
  File "/Users/jeev/Workspace/repos/personal/grpc-regression-mre/echo/client.py", line 27, in <module>
    main()
  File "/Users/jeev/Workspace/repos/personal/grpc-regression-mre/echo/client.py", line 22, in main
    response = stub.Echo(request)
  File "/Users/jeev/Workspace/repos/personal/grpc-regression-mre/echo/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/jeev/Workspace/repos/personal/grpc-regression-mre/echo/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
  status = StatusCode.UNAUTHENTICATED
  details = "Received http2 header with status: 401"
  debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:8443 {created_time:"2023-07-28T20:36:16.829747-07:00", grpc_status:16, grpc_message:"Received http2 header with status: 401"}"

What did you see instead?

From gRPC v1.56.2:

E0728 19:54:05.103107000 8093441536 hpack_parser.cc:991]               Error parsing 'content-type' metadata: invalid value
Traceback (most recent call last):
  File "/Users/jeev/Workspace/repos/personal/grpc-regression-mre/echo/client.py", line 27, in <module>
    main()
  File "/Users/jeev/Workspace/repos/personal/grpc-regression-mre/echo/client.py", line 22, in main
    response = stub.Echo(request)
  File "/Users/jeev/Workspace/repos/personal/grpc-regression-mre/echo/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/jeev/Workspace/repos/personal/grpc-regression-mre/echo/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Stream removed"
    debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Stream removed", grpc_status:2, created_time:"2023-07-28T19:54:05.10316-07:00"}"

Complete log

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

Anything else we should know about your project / environment?

A complete, minimal reproducible example is available here: https://github.com/jeevb/grpc-regression-mre