netty: IndexOutOfBoundsException when upgrading from 4.1.87 to 4.1.88

Hey there,

We’re experiencing test failures when our CI tries to automatically upgrade from 4.1.87 to 4.1.88. I can reproduce locally.

One of the broken test is https://github.com/gatling/gatling/blob/main/gatling-http-client/src/test/java/io/gatling/http/client/BasicHttpTest.java#L261-L289.

I can work on a pure netty reproducer, unless this immediately rings a bell for you guys 😃

Java version: 1.8.0_342 OS: irrelevant, happens both on Linux and MacOS.

Cheers

[error] Caused by: java.lang.IndexOutOfBoundsException: index: -2147483630, length: 2147483647 (expected: range(0, 2048))
[error]     at io.netty.buffer.AbstractByteBuf.checkRangeBounds(AbstractByteBuf.java:1390)
[error]     at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1397)
[error]     at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1384)
[error]     at io.netty.buffer.ByteBufUtil.lastIndexOf(ByteBufUtil.java:718)
[error]     at io.netty.buffer.AbstractByteBuf.indexOf(AbstractByteBuf.java:1254)
[error]     at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1022)
[error]     at io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:633)
[error]     at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:284)
[error]     at io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:239)
[error]     at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
[error]     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
[error]     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
[error]     at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
[error]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
[error]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[error]     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[error]     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[error]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
[error]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[error]     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[error]     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
[error]     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
[error]     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
[error]     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
[error]     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
[error]     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
[error]     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[error]     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

About this issue

Original URL
State: closed
Created a year ago
Comments: 40 (40 by maintainers)

Commits related to this issue

Don't fail on HttpObjectDecoder's maxHeaderSize greater then (Integer.MAX_VALUE - 2) (Fixes #13213) Motivation: Integer's overflow while computing indexOf indexes Modification: Fix the overflow by... — committed to franz1981/netty by franz1981 a year ago
Don't fail on HttpObjectDecoder's maxHeaderSize greater then (Integer.MAX_VALUE - 2) (Fixes #13213) Motivation: Integer's overflow while computing indexOf indexes Modification: Fix the overflow by... — committed to franz1981/netty by franz1981 a year ago
Don't fail on HttpObjectDecoder's maxHeaderSize greater then (Integer.MAX_VALUE - 2) (Fixes #13213) Motivation: Integer's overflow while computing indexOf indexes Modification: Fix the overflow by... — committed to franz1981/netty by franz1981 a year ago
Don't fail on HttpObjectDecoder's maxHeaderSize greater then (Integer.MAX_VALUE - 2) (Fixes #13213) Motivation: Integer's overflow while computing indexOf indexes Modification: Fix the overflow by... — committed to franz1981/netty by franz1981 a year ago
Don't fail on HttpObjectDecoder's maxHeaderSize greater then (Integer.MAX_VALUE - 2) (Fixes #13213) Motivation: Integer's overflow while computing indexOf indexes Modification: Fix the overflow by... — committed to netty/netty by franz1981 a year ago

Most upvoted comments

@franz1981 Connected. I see your messages but can’t reply yet 😃

Sorry, I enforced those limits on Discord to prevent guerrilla marketing spam. 😅

hyperxpro on Feb 13, 2023

@franz1981 Connected. I see your messages but can’t reply yet 😃

slandelle on Feb 13, 2023

Yes, our whole test suite passes.

slandelle on Feb 13, 2023

@franz1981 Just tested your fix. All good!

slandelle on Feb 13, 2023

I reckon I might be the only person disabling this limit, sorry for this 😂

slandelle on Feb 13, 2023

eheh yep: the reason is because in the maxHeaderSize’s set we allow scanning that amount of bytes, but that doesn’t include any CRLF found (as in the original impl), hence we allow scanning a total of (maxHeaderSize + 2) bytes, but sadly I’ve used an int to store it and making the subsequent

// maxBodySizeWithCRLF is < 0 because of integer overflow :(
final int toProcess = Math.min(maxBodySizeWithCRLF, readableBytes);
// toProcess is negative :(

Sending a branch to try now 😃

franz1981 on Feb 13, 2023

I can run Reactor Netty although I do not see the issue that @slandelle reported.

I bet it, because in theory using such larger header max size, will probably blow up the client/server much earlier, but still…is a good test. I’m both adding the fix and the test itself, sending a PR soon that anyone can run the test suite in case

franz1981 on Feb 13, 2023

I can run Reactor Netty although I do not see the issue that @slandelle reported.

violetagg on Feb 13, 2023

@franz1981 My current test suite uses Netty for the client but Jetty as the server so that I get more chances to uncover bugs with different stacks. Before trying to build a pure netty test, I think we should identify the kind of payload that causes this issue. I’m going to add logs on my side.

slandelle on Feb 13, 2023

@franz1981 Please let me know if you want me to try a patch.

slandelle on Feb 13, 2023

@franz1981 I confirm that reverting #12321 fixes the issue.

slandelle on Feb 13, 2023