nats.go: Unexpected Request timeout
When developing application using Nats, for events and request managing, we have some issues with request timeout. Created a snippet code, that imitates our code behavior natsrequest
Problem
Program fails with message like Source.NNNN.N request: *Request.Publish: nats: timeout.
Number of iterations that succeeds may vary.
Program structure
- Events - imitates events every 100ms.
- Sources - imitates requests from up to 2000 sources, starts from 200 with step 200. Each source sends a request every 500ms.
- Destinations - every destination subscribes for all 100 events and one source request. One goroutine for events processing, other for requests.
There is two Nats connection instances for Events, Sources and one encoded connection for Destinations.
Server version 2.6.2
Code was tested first on Win10, used a release binary. Timeout was set to 5 seconds and was exceeded. After that it was decreased to 1 second, just for time economy.
After that it was tested also in WSL Ubuntu 20.04.2 LTS. Nats was installed via docker.
Some program log for WSL:
2021/10/17 15:50:54.784505 Source.919 answer `Source.919.8 response` time 948.380228ms
2021/10/17 15:50:55.987874 Source.606.9 request: *Request.Publish: nats: timeout
2021/10/17 15:50:55.987891 Source.69.9 request: *Request.Publish: nats: timeout
2021/10/17 15:50:55.987907 Source.1173.9 request: *Request.Publish: nats: timeout
2021/10/17 15:50:55.987943 Source.1098.9 request: *Request.Publish: nats: timeout
Question
Is it something wrong with code, or this is just hardware issue, or maybe we should use different connection for request processing?
Reasoning
We experienced such error in our development process - our program send request timeout error after some time under stress test. After inspecting some logs, we saw that the most time delay was between moments
- request answer publish
- getting answer in source
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (6 by maintainers)
Clear. @derekcollison, @wallyqs thank you for help and advices. Closing issue.