status-go: SpiffWorkflow node not sending messages
Problem
The SpiffWorkflow node appears to have issues sending messages. Spiff shows errors like:
{
"level": "ERROR",
"message": "Error executing Service Task. Received error code ReadTimeout from service waku/SendMessage ::: 500 (Internal Server Error) - The service you called is experiencing technical difficulties. ::: The original message: HTTPSConnectionPool(host=test.app.spiff.status.im, port=443): Read timed out. (read timeout=45). ",
"loggerName": "spiffworkflow_backend",
"processName": "MainProcess",
"processID": 36,
"threadName": "ThreadPoolExecutor-0_9",
"threadID": 139861463910080,
"timestamp": "2024-05-20T13:11:21.225Z"
}
But debugging this is difficult because as far as I can tell 98% of the node logs are errors:
jakubgs@node-01.sm-eu-zur1.spiff.app-test:~ % zgrep -v ERROR /var/log/docker/spiff-workflow-wakunode/docker.* | wc -l
267247
jakubgs@node-01.sm-eu-zur1.spiff.app-test:~ % zgrep ERROR /var/log/docker/spiff-workflow-wakunode/docker.* | wc -l
17513290
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 28 (27 by maintainers)
This is something I have noticed in the past and noticed today also. Normally I open the Status app only when I want to test the messages. That can be once in few days or once in a week sometimes. I have noticed that most of the time when I open the status app then the instances start to error for some time. Yesterday and today morning I created about 100+ instances and none of them errored. I didn’t have the Status app opened at that time. But later today I opened the Status app and then created those instances (which I shared above) and they started getting the error. This is something I have observed in the past also. Just shared what I have observed. Not sure whether it’s a valid point or not.
These I ran after starting the Status app and some of them errored.
I ran 50 instances of this in the morning before opening the Status app without any error. And another 100+ instances yesterday without any error.