open-liberty: Spurious IOException logged when attempting to send messages to closed SSE clients
Describe the bug
While working with Server-Sent Event (SSE), I see the exception below printed many, many times. The application seems to behave as expected, but the logs are really noisy. This is despite having configured server.xml with:
<httpOptions throwIOEForInboundConnections="false" />
And despite explicitly catching and ignoring exceptions when sending messages:
try {
sseEventSink
.send(sseEvent)
.exceptionally(ignored -> {
// Importantly, if I log a message here, it is printed *exactly* as many times as the exception below.
// This exception happens mostly when trying to send messages to clients who have closed the connection.
System.out.println("¯\_(ツ)_/¯");
return null;
});
} catch (Exception ignored) {
// Yes, sometimes this branch runs too...! I have no clue why; one would expect this to be handled
// at the CompletionStage level, but I suppose that's a problem for a different occasion.
}
It’s proven impossible for me to trace down the source of the exception or to find a way to prevent it from being printed. Here is the exception in question:
[INFO] [2022-07-13T08:37:37.919+0000] 00000292 srt E com.ibm.ws.webcontainer.srt.SRTServletResponse closeResponseOutput SRVE8056E: An unexpected exception occurred closing the output stream.
[INFO] java.io.IOException: Broken pipe
[INFO] at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
[INFO] at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62)
[INFO] at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
[INFO] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:76)
[INFO] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:53)
[INFO] at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:532)
[INFO] at com.ibm.ws.tcpchannel.internal.NioSocketIOChannel.write(NioSocketIOChannel.java:201)
[INFO] at com.ibm.ws.tcpchannel.internal.NioSocketIOChannel.attemptWriteToSocketUsingNIO(NioSocketIOChannel.java:139)
[INFO] at com.ibm.ws.tcpchannel.internal.SocketIOChannel.attemptWriteToSocket(SocketIOChannel.java:198)
[INFO] at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:605)
[INFO] at com.ibm.ws.tcpchannel.internal.WorkQueueManager.processWork(WorkQueueManager.java:309)
[INFO] at com.ibm.ws.tcpchannel.internal.NioTCPWriteRequestContextImpl.processAsyncWriteRequest(NioTCPWriteRequestContextImpl.java:102)
[INFO] at com.ibm.ws.tcpchannel.internal.TCPWriteRequestContextImpl.writeInternal(TCPWriteRequestContextImpl.java:320)
[INFO] at com.ibm.ws.tcpchannel.internal.NioTCPWriteRequestContextImpl.processSyncWriteRequest(NioTCPWriteRequestContextImpl.java:58)
[INFO] at com.ibm.ws.tcpchannel.internal.TCPWriteRequestContextImpl.write(TCPWriteRequestContextImpl.java:96)
[INFO] at com.ibm.ws.channel.ssl.internal.SSLWriteServiceContext.write(SSLWriteServiceContext.java:171)
[INFO] at com.ibm.ws.http.channel.internal.HttpServiceContextImpl.synchWrite(HttpServiceContextImpl.java:3141)
[INFO] at com.ibm.ws.http.channel.internal.HttpServiceContextImpl.sendFullOutgoing(HttpServiceContextImpl.java:2874)
[INFO] at com.ibm.ws.http.channel.internal.inbound.HttpInboundServiceContextImpl.finishResponseMessage(HttpInboundServiceContextImpl.java:937)
[INFO] at com.ibm.ws.http.channel.internal.outbound.HttpOutputStreamImpl.flushBuffers(HttpOutputStreamImpl.java:538)
[INFO] at com.ibm.ws.http.channel.internal.outbound.HttpOutputStreamImpl.close(HttpOutputStreamImpl.java:625)
[INFO] at com.ibm.ws.webcontainer.osgi.response.WCOutputStream.close(WCOutputStream.java:216)
[INFO] at com.ibm.ws.webcontainer31.osgi.response.WCOutputStream31.close(WCOutputStream31.java:89)
[INFO] at com.ibm.ws.webcontainer.srt.SRTServletResponse.closeResponseOutput(SRTServletResponse.java:521)
[INFO] at com.ibm.ws.webcontainer40.srt.SRTServletResponse40.closeResponseOutput(SRTServletResponse40.java:215)
[INFO] at com.ibm.ws.webcontainer.srt.SRTServletRequest.closeResponseOutput(SRTServletRequest.java:3474)
[INFO] at com.ibm.ws.webcontainer.async.CompleteRunnable.run(CompleteRunnable.java:63)
[INFO] at com.ibm.ws.webcontainer.async.ServiceWrapper.wrapAndRun(ServiceWrapper.java:236)
[INFO] at com.ibm.ws.webcontainer.async.ContextWrapper.run(ContextWrapper.java:28)
[INFO] at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:245)
[INFO] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[INFO] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[INFO] at java.base/java.lang.Thread.run(Thread.java:833)
Steps to Reproduce I haven’t isolated the problem, but I generally accept SSE messages from a controller, i.e.
@GET
@Produces(SERVER_SENT_EVENTS)
public void onSseClientConnected(@Context Sse sse, @Context SseEventSink sink) {
// ...
sink.send(...); // on a separate thread, after a varying delay
// ...
}
Then open a few SSE connections, i.e.
var client = ClientBuilder.newBuilder().build();
var target = client.target(sseUrl);
var source = SseEventSource.target(target).build();
source.register(onEvent);
source.open();
Then wait for some expected messages, and close the connection from the client side. I do this in several JUnit tests, run sequentially.
Expected behavior
No error should be logged, as no exception has gone uncaught.
Diagnostic information:
- OpenLiberty Version: whichever ships with liberty-maven-plugin 3.6.1 (how could I quickly find out the OpenLiberty version I’m using?)
- Affected feature(s): no idea
- Java Version: 17, though I don’t think this is the issue
- server.xml configuration: I have outlined above the only relevant section
Additional context
Not entirely related, but I have this other unresolved problem with SSE: https://stackoverflow.com/questions/72684257/detect-closed-server-sent-event-clients-in-jakarta-ee
I see them as related in the sense that if I were able to detect regular client closures, I would not attempt to send messages to closed clients so often, considerably reducing the amount of times the exception above is logged, which would be a good enough start.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 1
- Comments: 27 (9 by maintainers)
And if you can (while waiting for the official case open with IBM), add this to the server.xml
<logging traceSpecification="*=info=enabled:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:GenericBNF=all:HTTPDispatcher=all:com.ibm.ws.http*=all" maxFileSize="40" maxFiles="20" traceFormat="BASIC"/>recreate the issue and attach it here. Please keep the recreation to the main steps (to reduce the trace output since these are full webcontainer and channel trace).
Thanks,
Hi,
Can you please open a support SalesForce case for this with IBM support? It will help tracking the issue/solution timely as well as prioritizing our works.
Thanks,
We are also an enterprise using SSE in production. Our application log size has spiked up due to this exception being written, which in-turn has increased our bills too. (because we use external logging service).
Please can you not log this exception.
I’m not sure if I’m doing anything wrong, but the suggested trace specification not only did not work, but it actually broke my existing logging configuration, making SQL statements show up in the logs, whereas they normally wouldn’t, thanks to
eclipselink.sql=warning: