traefik: gRPC-Web middleware sends incorrect content-length to backend servers

Welcome!

• Yes, I’ve searched similar issues on GitHub and didn’t find any.
• Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.

What did you do?

I am using grpc-web in my react frontend and grpc-java in my backend application. After seeing grpc-web support from Traefik I decided to give it a try and it worked flawlessly in the beginning. After upgrading grpc-java from 1.41.0 to 1.51.0 I realised that in version 1.42.0 there was a internal change that started validating the content-length header.

What did you see instead?

After upgrading the grpc-java version the proxied requests through the grpc-web middleware stopped working with the following error:

io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Received amount of data 22 does not match content-length header 32
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:153) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$ContentLength.increaseReceivedBytes(DefaultHttp2ConnectionDecoder.java:806) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.verifyContentLength(DefaultHttp2ConnectionDecoder.java:230) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.access$300(DefaultHttp2ConnectionDecoder.java:53) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:303) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onDataRead(DefaultHttp2ConnectionDecoder.java:691) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:250) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[grpc-netty-shaded-1.52.1.jar:1.52.1]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

Here are some related issues that I found: https://github.com/grpc/grpc-java/issues/9019 and https://github.com/grpc/grpc/issues/29216

I have also tried to remove the content-length header with a header middleware but it did not made any difference.

What version of Traefik are you using?

3.0.0-beta2

What is your environment & configuration?

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: grpcweb
spec:
  grpcWeb:
    allowOrigins:
      - "*"
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: remove-content-length-header
spec:
  headers:
    customRequestHeaders:
      Content-Length: "" # Removes
      content-length: "" # Removes
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    meta.helm.sh/release-name: backend
    meta.helm.sh/release-namespace: dev
  labels:
    app.kubernetes.io/managed-by: Helm
  name: grpc-ingress-tls
  namespace: dev
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`grpc-web.example.com`) && PathPrefix(`/Auth`)
    middlewares:
    - name: grpcweb
    - name: remove-content-length-header
    services:
    - name: auth
      port: 6565
      scheme: h2c

If applicable, please paste the log output in DEBUG level

No response