netty: FullHttpRequest doesn't return proper content anymore

@fredericBregier Thanks for the detailed answer! 👍🏻

I want to use HttpObjectAgregator and FullHttpRequest to keep the code simple, even though it may not be the most efficient. Sometimes I think that Netty is too low level for my needs. 😄 But since everything works just fine, I have no reason to rewrite code to some other high level framework. 🤷‍♂️

I want to extract all the possible representations of the POST data. If it is a form, then I want to have key -> value pairs. If it is just a simple plain content, I want to have it as a String.

So I just changed the order. First I get the content from FullHttpRequest and then I use HttpPostRequestDecoder, and now I don’t care if it modifies the request or not. 😄

@bxqgit OK, I understand.

So you are indeed using a Multipart decoding. I have found several issues in the way you using it:

• In general, it is not recommended to use the HttpObjectAgregator in your pipeline when facing Multipart. The reason is that it will fully load everything in memory. But as you’re using a Multipart, it might be “too much” for the JVM. So you might change this as follow:

  • First, remove the HttpObjectAgregator from your pipeline

  • Then adapt your code as follow (inspired from https://github.com/netty/netty/blob/4.1/example/src/main/java/io/netty/example/http/upload/HttpUploadServerHandler.java )

    // The factory in general is initialized once
    private static final HttpDataFactory factory =
        new DefaultHttpDataFactory(DefaultHttpDataFactory.MINSIZE); // Disk if size exceed
    
    // Store the decoder between each and every "chunk" for the same request
    private HttpPostRequestDecoder decoder;
    
    @Override
    public void channelRead0(ChannelHandlerContext ctx, HttpObject msg) throws Exception {
        if (msg instanceof HttpRequest) {
            HttpRequest request = (HttpRequest) msg;
            try {
                decoder = new HttpPostRequestDecoder(factory, request);
            } catch (ErrorDataDecoderException e1) {
                // Your error handling
                return;
            }
        }
    
        // check if the decoder was constructed before (if not, could be an issue)
        if (decoder != null) {
            if (msg instanceof HttpContent) {
                // New chunk is received
                HttpContent chunk = (HttpContent) msg;
                try {
                    decoder.offer(chunk);
                } catch (ErrorDataDecoderException e1) {
                    // Your error handling
                    return;
                }
            }
            // check for end of multipart (which is also an HttpContent)
            if (chunk instanceof LastHttpContent) {
                // finaly use all data (may be use before the end,
                // but keep in mind that not all data are there, so up to your business logic)
                readHttpDataChunkByChunk()
                // Finalize the decoder
                reset();
            }
        }
    }
    
    private void reset() {
        // destroy the decoder to release all resources
        decoder.destroy();
        decoder = null;
    }
    
    /**
     * Example of reading request by chunk and getting values from chunk to chunk
     */
    private void readHttpDataChunkByChunk() {
        try {
            while (decoder.hasNext()) {
                InterfaceHttpData data = decoder.next();
                if (data != null) {
                    // new value to use in your way
                    useTheNewData(data);
                    data.release();
                }
            }
        } catch (EndOfDataDecoderException e1) {
            // end
        }
    }
    

• Now to your issue:

  • Whatever using onely one FullHttpRequest or using multiple HttpContent (when chunked, the first item is a HttpRequest without any content, followed by multiple HttpContent, ending with a last LastHttpContent that closes the chunk reception)
  • When you offer the chunks to the decoder, the decoder consumes each and every chunk or the only one FullHttpRequest if not chunked
  • This means that the request is now fully read, and therefore the ByteBuf has its readerIndex sets to writerInder.
  • So the reason you cannot read it anymore (except if you reset readerIndex to 0 as you did).
  • The change occurs since previously the decoder did not act as needed. It should read all buffers, and not keeping them unchanged (indeed, it was really depending on how the chunks arrived, so not consistent). All decoders reads the input ByteBufs and therefore the HttpRequest is not immutable regarding its content.

So it is not an issue but a normal way. You’re not using a decoder correctly.

I don’t know why you want to get the “full” content of a Multipart request into a String, but if so, you can build it through the HttpData retrieves from the decoder, as follow:

    /**
     * NEW: add a StringBuilder that you can define as attached to the session and initialize
     * and destroyed for each and every HttpRequest as the decoder
     */
    private StringBuilder builder = null;
    @Override
    public void channelRead0(ChannelHandlerContext ctx, HttpObject msg) throws Exception {
        if (msg instanceof HttpRequest) {
            HttpRequest request = (HttpRequest) msg;
            try {
                decoder = new HttpPostRequestDecoder(factory, request);
            } catch (ErrorDataDecoderException e1) {
                // Your error handling
                return;
            }
            // NEW: initialize the StringBuilder
            builder = new StringBuilder();
        }
        ...
            // check for end of multipart (which is also an HttpContent)
            if (chunk instanceof LastHttpContent) {
                // finaly use all data (may be use before the end,
                // but keep in mind that not all data are there, so up to your business logic)
                readHttpDataChunkByChunk()
                // NEW: Use your StringBuilder
                final String s = builder.toString();
                // Finalize the decoder
                reset();
            }
        }
    }

    /**
     * Example of reading request by chunk and getting values from chunk to chunk
     */
    private void readHttpDataChunkByChunk() {
        try {
            while (decoder.hasNext()) {
                InterfaceHttpData data = decoder.next();
                if (data != null) {
                    // new value to use in your way
                    useTheNewData(data);
                    // NEW: Change here: use the StringBuilder
                    builder.append(data.getString(UTF_8));
                    data.release();
                }
            }
        } catch (EndOfDataDecoderException e1) {
            // end
        }
    }
    
    private void reset() {
        // destroy the decoder to release all resources
        decoder.destroy();
        decoder = null;
        // NEW: StringBuilder clean
        builder = null;
    }

Hoping this is clear enough to help you.

@bxqgit Hi, could you elaborate a bit ? If I understand, your code is doing something like:

1. the server uses HttpServerCodec and HttpObjectAgregator
2. it receives one body data only, so not multipart, using standard codec only
3. it creates from the buffer given by HttpObjectAgregator a FullHttpRequest: does it done manually?
4. when it is done, the buffer seems empty but it is not (changing readerIndex to 0 gives back access to the data)

If this is this first scenario, where comes the Multipart things?

Maybe you’re using Multpart things, so it might be point 3 as: 3. Uses HttpPostMultipartRequestDecoder as:

• HttpDataFactory factory = new DefaultHttpDataFactory(true or false or limit size);
• HttpPostMultipartRequestDecoder decoder = new HttpPostMultipartRequestDecoder(factory, request);
• for each HttpContent, doing decoder.offer(content)
• once done, retrieving the HttpData(s), and creates a new HttpRequest from the HttpData(s)?
• then destroying decoder and factory
  • factory.cleanAllHttpData(); and decoder.destroy();

If this is the second case, why are you using HttpObjectAgregator ? There is no need for that, and of no use with HttpPostMultipartRequestDecoder.

Maybe you’re using Multpart things, but as encoder, so it might be point 3 as: 3. Uses HttpPostRequestEncoder as:

• HttpPostRequestEncoder encoder = new HttpPostRequestEncoder(true);
• add the received content from original request to the encoder: one or the other
  • encoder.addBodyAttribute(...);
  • encoder.addBodyFileUpload(...);
• add maybe other datas
• retrieve the final request:
  • HttpRequest newRequest = encoder.finalizeRequest();
  • factory.cleanAllHttpData(); and decoder.destroy();

Then, here the HttpObjectAgregator usage is probably correct. It is only encoder part that is concerned. As there are Junit tests doing like this, if you could gives more information, that could be helpful.

Which scenario is it?