traefik: v2.11.1 lingeringTimeout can break some connections
Welcome!
- Yes, I’ve searched similar issues on GitHub and didn’t find any.
- Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.
What did you do?
2.11.1 shouln’t breaks connections.
What did you see instead?
2.11.1 breaks connections in some use cases.
There is currently a CVE which affects Traefik, it’s not yet published (work in progress).
The 2.11.1 release was packed with this PR, introducing respondingTimeouts.lingeringTimeout to mitigate the vulnerability. This implementation was carefully chosen after many discussions to avoid being breaking at all costs. Sadly, we haven’t anticipated all use cases, and many issues have been opened by the community since the release.
We made the decision to rollback the lingeringTimeout PR, and to provide a simpler fix by tomorrow. ⚠️ We haven’t been able to find a fix for this vulnerability that is not breaking at some point. However, the new fix we are going to provide is simpler to understand and configure to fit your needs.
We will update this issue while progress is made.
What version of Traefik are you using?
2.11.1 & 3.0.0-rc4
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 28
- Comments: 45 (10 by maintainers)
Links to this issue
Commits related to this issue
- Set ReadTimeout to 300s - The default value is 60s - https://github.com/traefik/traefik/issues/10598 — committed to b-data/docker-deployment-traefik by benz0li 3 years ago
I wasn’t going to feed the troll, but you keep posting - and it needs to be addressed.
How much do you pay for Traefik @Majestic7979? I assume you are on a paid support plan given how entitled you feel, so it’s probably best you raise that with the team directly - rather than in this issue.
If you aren’t paying for Traefik, and you are using
:latestor blindly updating the version / container without checking, then you have no right to be abusive of the team, maintainers and others working towards a resolution.You didn’t get thumbs down for speaking the truth, you got it for completely missing the mark - a sign I would have thought a reasonable person would see and understand.
Your comment suggesting that misunderstanding is linked to English skills is dismissive and prejudiced. It’s important to communicate respectfully and avoid making assumptions about others’ language abilities or cultural backgrounds. Let’s strive to keep our dialogue inclusive and supportive.
The fact that people are providing you with a service that you clearly rely on, for (I’m assuming) free, and you have the gall to attack them because English might be their second language really highlights your character.
I hope that this was born out of frustration, and given time you may come to realise how inappropriate your contribution is and will apologise to all involved.
Please be comprehensive: We are humans and humans make mistakes.
We made a mistake with
lingeringTimeoutand we are trying to fix it as quickly as we can.Then it’s not semver compliant but this is exceptional because there are not a lot of possibilities to manage the situation, and we are sorry about that.
Hello,
Traefik v2.11.2 & 3.0.0-rc5 have just been released and published on Docker Hub v2.11.2 image & 3.0.0-rc5 image. Those releases mitigates the
CVE-2024-28869https://github.com/traefik/traefik/security/advisories/GHSA-4vwx-54mw-vqfw.Both versions revert the
lingeringTimeoutand introduce a default value of60sforreadTimeout. ⚠️ Be aware that you should adapt this value to your own use case. The general guideline is to set it right above the maximum needed by your applications.The only use case which should be impacted is the large request payload (upload of a large file for example): this could require a higher timeout value. Otherwise, you shouldn’t be impacted. If this behaviour impacts your production setup, please open an issue to inform the maintainers.
We did everything we could to address this vulnerability quickly & with a minimal impact.
lingeringTimeoutwas an attempt to avoid being breaking as much as possible, but eventually this proved not to be the case. There will be a follow up on this, we would like to provide a more flexible solution in the future (as an example, we could inspire from https://github.com/traefik/traefik/issues/10598#issuecomment-2050051693). Don’t hesitate to open issue to propose your ideas. Thank you all for your contributions, feedback & kindness ❤️I am editing my response to remove the solution because I received several thumbs down for complaining. This is toxic harassment. I have blocked and reported all the members involved.
The following Additional Arguments resolved this issue for us with
v2.11.1& Helm Releasev27.0.1.Hopefully this helps someone 😊
v2.12.0 would be an option.
I’ve created an absolutely unofficial table here https://github.com/traefik/traefik/wiki/respondingTimeouts-for-applications
If you have input please update it with your findings. I’ve thrown what I’ve seen in there to start it up.
If you’re adhering to semver and don’t increment the major version for a breaking change are you actually doing semver, or are you doing a ‘traefiks-almost-semver-but-different’ thing?
This isn’t rhetorical. If you’re not incrementing the major version for a breaking change you aren’t following semver:
Sure. We are prioritizing v2.11 for today, and v3.0 porbably tomorrow.
Quick update: we will introduce a default value for readTimeout (which is the time needed by the server to read a request headers + body). We have chosen a persmissive default value of
60sto avoid being breaking in most cases. The only use case which can be impacted is the large request payload (upload of a large file for example): this could require a higher timeout value. We also based our decision on other reverse proxies behaviors.Yes unfortunately this is the second day in a row I’ve had to figure out what the heck is going on with my Traefik deployment breaking all of my selfhosting.
Yesterday, per suggestions from others, I had applied the
transport.respondingTimeouts.tcp.lingeringTimeout: 0workaround to mytraefik.ymlto avoid my websocket applications being broken.With today’s 2.11.2 release, Traefik now refuses to startup because those same configuration clauses are no longer valid:
2024/04/12 09:04:37 command traefik error: field not found, node: tcpThanks @Majestic7979 for providing the steps needed to fix.
The v2.11.2 and v3.0.0-rc5 have been created.
Now you have to wait for Docker to create the official images.
https://github.com/docker-library/official-images/pull/16591
https://doi-janky.infosiftr.net/job/multiarch/view/images/view/traefik/
You can use v2.11.0 or v3.0.0-rc3
v2.11.1 or v3.0.0-rc4 already fixed the CVE.
High
When would a stable version (without the CVE fix) be uploaded to dockerhub?
Also, any expectations on when a version that closes the CVE will be released?
What is the severity of this vulnerability?
Will a feature like “MinRate” from apache be made available that, if timeout=0 cannot be safely used, allows for good clients that upload at a certain rate to keep doing so while avoiding SlowLoris-like attacks?
Imagine the usual File Synchronization use case, especially from less than optimal connections.
Minor versions in semver are not supposed to be breaking either. So the problem would be the same with v2.12.0. When there is a vulnerability, and when there is no way to fix it without being breaking, the only way is to do a breaking bug fix. We aren’t happy about this situation, but that’s the best we can do (again, we tried to no be breaking at all costs introducing the lingering timeout, but in fact this was clearly breaking).
I am not a troll, so I am not going to read whatever you said past your name-calling. You should learn to respect others by not calling them names if you want them to read what you’ve got to say. I’m also blocking you and reporting the harassment. Just because you think differently or have a different opinion does not turn your oponent on a debate into a troll, especially when I have contributed with a fix to the issue, that was really disgusting by you. Enjoy your likes and hearts, looks like you have many fans, congratulations, I am proud to see people supporting bullying and harassment so openly, many of them are even German, looks like in Germany some people still have the mindset that it is acceptable to attack others. I am autistic by the way.
Anybody harassing me with either name calling or thumbs down with a clear view to cause me distress will be blocked and reported to GitHub. Bullying and harassment is against the site’s guidelines. Please understand that having different opinions and complaining about issues is not a reason to bully and/or harass anyone online.
Update: GitHub concluded that there was bullying and harassment. I politely request anyone thinking of harassing me over my comment to think again and desist immediately from going forward with bullying and to leave me alone. I have the right to make a complaint about an application without being targeted with either bad comments or bad reactions. I will report all of the abuse and GitHub may even terminate accounts that are found to be abusive. Please respect the Terms of Service and do not use this platform to target members you disagree with.
ℹ️ If one is using Traefik in front of a registry, the default
ReadTimeoutvalue of 60s may be too low.👉 Pushing of a single layer for more than 60s results in a failure.
@ldez
Of course I understand that it was a mistake and you’re doing your best to fix things. I’m a bit confused as to why you think that I was claiming otherwise. Bumping semver isn’t a punishment, it’s just a machine readable sort of changelog; numbers are free.
Maintaining semver is making a promise to your consumers that you will update the version in a particular way based on things like implementing a breaking change.
The current solution, introducing a default value for
readTimeout, still seems to be acknowledged as a breaking change - Is that the case? If so, please consider making 2.11.2 into 3.0.0 and making the current 3.0.0-rc into a 4.0.0-rc; it’s not too late to follow through on your commitment to sane versioning.Regardless, thanks for the prompt response on this issue; CVEs are not fun and acting proactively to ensure that users receive a fix is admirable, it’s just a shame that the issues we’ve seen weren’t caught by some sort of automation.
Here’s to many more years of not having to think about updates!
@emilevauge There’s no way to set respondingTimeouts at a service and router level right? Like @troykelly mentioned Portainer and a few other services I run aren’t going to like this. It would be great to override respondingTimeouts for specific paths as an example.
I think the solution that @waffshappen mentioned would cover off some of the remaining issues though so I look forward to that implementation. Not entirely sure what the solution looks like for websockets that are mostly in-active.
@emilevauge + team great work through the day to get this mitigated as quickly as possible.
I’m not sure that this only impacts large payloads, I’m seeing tools like Portainer not working properly because of timeouts disconnecting, setting as
0resolves the issue - but as I understand it, then the Possible DoS vulnerability with Content-length header isn’t mitigated - is that correct?I understand completely that this is needed to mitigate the issue, but I think that there might be a fairly prolific impact on long-held interactions.
All Working in rc-5
Ho hum… it’s hardly an attack or aggressive… but a casual observation of your profile suggests you are doing this to get haters and you complain if you get them. Does not compute? Just be nice.
good news - fixed for ASP.NET 8 Blazor and Dozzle - thanks
+1, if it’s possible to manage to reintegrate semantic versioning, that would be great - especially helpful given there have been some rapid-fire changes with some breaking configuration expectations.
Understand that given how quickly things have had to move, that opportunity might not exist any more - it might be something that can be identified in a PIR so it can be built into the process next time?
Does setting the readTimeout to 0 still protect you from the CVE? Edit: I came here because Home Assistant was constantly refreshing and eventually found my way to this issue. But some issues on Home Assistant (like home-assistant/frontend/issues/20493) have a user setting it to 0, so hence why I asked for clarification. On a side note, I updated to 2.11.2 a few minutes ago, and the pages on my Home Assistant instance does not constantly refreshes every few seconds anymore without me having to change the entrypoint configuration.
Anything related to
lingeringTimeouthas been reverted in v2.11.2 and v3.0.0-rc5:respondingTimeouts.xxxoptions has been removedlingeringTimeoutoptions have been removedrespondingTimeouts.http.xxxoptions have been removedrespondingTimeouts.tcp.xxxoptions have been removedSure, it will also be merged in v3.
An additional symptom is an inability to log in to services like GitLab, Nextcloud, Synology dashboard, etc. if hosted behind Traefik and they use LDAP for authentication.
So another breaking change (new default value for
readTimeout) in a patch release?