flow: Unexpected message id from the client (sync ids different)
Description of the bug
I receive sometimes the same error message on my production system. But I do not have any idea when this happen:
2021-12-27 07:27:43,174 INFO | https-jsse-nio-18443-exec-10 | com.vaadin.flow.server.communication.ServerRpcHandler | Ignoring old duplicate message from the client. Expected: 1748, got: 1747
2021-12-27 07:28:01,354 WARN | https-jsse-nio-18443-exec-5 | de.app.ApplicationVaadinServiceInitListener | Unexpected error
java.lang.UnsupportedOperationException: Unexpected message id from the client. Expected sync id: 1751, got 1752. more details logged on DEBUG level.
at com.vaadin.flow.server.communication.ServerRpcHandler.handleRpc(ServerRpcHandler.java:310)
at com.vaadin.flow.server.communication.UidlRequestHandler.synchronizedHandleRequest(UidlRequestHandler.java:115)
at com.vaadin.flow.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40)
at com.vaadin.flow.server.VaadinService.handleRequest(VaadinService.java:1565)
at com.vaadin.flow.server.VaadinServlet.service(VaadinServlet.java:302)
at com.vaadin.flow.spring.SpringServlet.service(SpringServlet.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
It seems it is the same bug as https://github.com/vaadin/flow/issues/9778
Minimal reproducible example
Unfortunately I’m not possible to reproduce the issue. I only find the messages in the logfile.
Versions
- Vaadin / Flow version: 21.0.9
- Java version: 17.0.1
- OS version: CentOS
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 6
- Comments: 57 (18 by maintainers)
Links to this issue
Commits related to this issue
- fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) Attempts to fix the synchronization issue related to the usage of the Login reported in #12640. The Login comp... — committed to vaadin/flow by mcollovati 2 years ago
- fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) Attempts to fix the synchronization issue related to the usage of the Login reported in #12640. The Login comp... — committed to vaadin/flow by mcollovati 2 years ago
- fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) Attempts to fix the synchronization issue related to the usage of the Login reported in #12640. The Login comp... — committed to vaadin/flow by mcollovati 2 years ago
- fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) (#20449) Attempts to fix the synchronization issue related to the usage of the Login reported in #12640. The L... — committed to vaadin/flow by vaadin-bot 2 years ago
- fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) (#20448) Attempts to fix the synchronization issue related to the usage of the Login reported in #12640. The L... — committed to vaadin/flow by vaadin-bot 2 years ago
- fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) Attempts to fix the synchronization issue related to the usage of the Login reported in #12640. The Login compo... — committed to vaadin/flow by mcollovati 2 years ago
- fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) Attempts to fix the synchronization issue related to the usage of the Login reported in #12640. The Login compo... — committed to vaadin/flow by mcollovati 2 years ago
- fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) (#20455) Attempts to fix the synchronization issue related to the usage of the Login reported in #12640. The L... — committed to vaadin/flow by mcollovati 2 years ago
- fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) (CP: 2.11) (#20456) * fix: delay session expiration handling to prevent canceling ongoing navigation (#19983) ... — committed to vaadin/flow by mcollovati 2 years ago
That’s basically what we suspect we would have to end up doing with two additions:
An update on the login issue.
The problem seems to be that when you submit the form, Spring Security changes the sessionId, but at the same time the client also sends a UIDL login event with the “old” session cookie. The UIDL requests then sends back a “Session expired” message, and this could be fine, since we will get a new UI after for submission because of the redirect issued by Spring Security. However, when the browser is going to be redirected, it also sends an UNLOAD UIDL message with the new session ID, and this is the request that fails because it has
clientId=3, but theclientId=2message was previously rejected because of session expiration.I see the error on my development machine from time to time. So I can exclude 1,2,4,5. We actually switched from Sophos to SentinelOne in the mean time so I guess I can also rule out 3. The only patterns that I see is that is happening when logging in and it is always request 2 that is missing. Everything after this seems fine. But the Logincode and most of the spring security code is located in a shared project and in the other application I wasn’t able to reproduce the error. I will be away for 2 weeks but after that I would love to schedule a meeting, where we could try to find the cause on my machine.
That error is actually reported multiple times and has been discussed on many tickets. The error itself is not a bug as is, but logged as part of the failsafe mechanism in Vaadin.
When packets are sent back and forth between server and browser, they include serial number so that both parties can verify that no packet is being lost.
There are natural reasons for packet loss
When communication goes un-sync Vaadin detects that and does resynchronize. That is done on purpose in order to restore the operation.
When you see this error, one should observe does it happen in correlation with some usage pattern of the application, frequency, when it is happening etc. All these can give some hints whether it is due some natural cause of the error.
As you see the list of the natural causes are all flaky, hence deterministic re-production may be difficult.
The cases where it has been a bug in Vaadin have been difficult to debug, but with rigorous attempt of following the statistics, close collaboration with the reporter, ruling out natural causes we have been able to find those and fixed them. Naturally we would like to hope that we have identified all of them, and only the natural causes remain.
I’m using Vaadin 24.2.3 with Java 21, SpringBoot 3.2 and Spring Framework 6.1 (on macOS Ventura 13.5) and still see this issue:
Any news on this?
Cheers, Fabian
Will do - but will take a while since it only occurred very infrequently in PROD and next PROD deployment will not be before next week.
We also see this in our production systems, and I want to share our findings.
It happens rarely, not reproducible, but since we have so many users, we see it several times per hour.
The bigger part comes during end of session, the RPC typically contains “UNLOAD”:true (as written above by ssindelar). This is probably nothing which affects users.
But we also have a significant number of events when this happens in the middle of users’ work. Server e.g. receives id=91, but expects id=90.
We have hacked code into ServerRpcHandler, when this situation is detected, we temporarily unlock the session, and let the thread wait (with some timeout) on a sync object. Then processing continues, often a few milliseconds later we can process message #90. When this is processed we notify the sync object and the other thread awakes. It restores the lock on the session and processing continues. On client side everything seems to be ok, users apparently see no error in this situation.
So it looks like client does not strictly enforce sequential processing of messages, it sends out request #91 before answer for #90 has been received. We use XHR as communication mechanism. If this is something that the client is allowed to do, due to the asynchronous nature of http requests it surely can happen that processing on server will be done in arbitrary sequence. Therefore code on server should handle this transparently (e.g. like we have tried)
To be precise, the buffer would not fully eliminate the need for resync occasionally, but would reduce likelihood of the occurrence. I would assume it would reduce with great amount as what I have seen, in most typical cases it would be enough to retain just the previous one in the memory. Naturally, the size of buffer could be configurable.
@jorgheymans ‘invalid json response from server’ is not usually related to synchronization as that error is more often related to unexpected closing of the session while client is still running. Then when there is next request, Vaadin does bootstraping and severs the index page, which is naturally a invalid json. This invalid json error can be avoided if there is Vaadin-Refresh token in the content served. Then client will notice this and also restart itself.
I think I tracked down the problem. Yesterday in the late afternoon the problem resurfaced with a nearly 100% reproduceability long enough so I could do some debugging before disappearing again a couple of minutes later.
I think the cause of the problem is a request with the “wrong” session id. The expection with the “Unexpected message id from the client” is a result of this but not the cause of the problem. Sadly it couldn’t test my findings because it error disappeared so suddenly it reoccured.
What I found:
The first thing I noticed was that the second request after clicking the Login Button was recieving a session expired response.
To understand whats happening we need to do a short deep dive into Spring Security and Tomcats session handling. During the login Spring Security rotates the session id to prevent a Session Fixation Attack. With a couple of tracepoints I was able to see this happening in the ManagerBase classs of Tomcat.
When the response of the first request arrives the cookie is updated with the new session id. The second request is send most of the time before the first one returns. So it get’s send with the “old” session id. This usually works because the login takes some time and the second request has got the session before the id is changed.
With a breakpoint in ManagerBase.findSession I was able to reproduce the exception by stopping both threads and letting the login update the session id before continueing the second request:
While it’s currently not happening without the breakpoints and manual continuation, I’m very confident I would see a request with the outdated session id that can’t find it’s session. When a VaadinRequest can’t find the session a reconnect is triggered. Now it depends on the timing of the reconnect if the user ends up at the Login or the page after. A reconnect on the page after the login usually goes unnoticed because it looks like a refresh or a little longer load time.
I activated the debug logging, and got the following (unfortunately no special information):
2022-01-02 20:25:10,673 DEBUG | https-jsse-nio-18443-exec-9 | com.vaadin.flow.server.communication.ServerRpcHandler | Unexpected message id from the client. Expected sync id: 2, got 1. Message start: