status-go: SpiffWorkflow node not sending messages

Problem

The SpiffWorkflow node appears to have issues sending messages. Spiff shows errors like:

{
  "level": "ERROR",
  "message": "Error executing Service Task. Received error code ReadTimeout from service waku/SendMessage ::: 500 (Internal Server Error) - The service you called is experiencing technical difficulties. ::: The original message: HTTPSConnectionPool(host=test.app.spiff.status.im, port=443): Read timed out. (read timeout=45). ",
  "loggerName": "spiffworkflow_backend",
  "processName": "MainProcess",
  "processID": 36,
  "threadName": "ThreadPoolExecutor-0_9",
  "threadID": 139861463910080,
  "timestamp": "2024-05-20T13:11:21.225Z"
}

But debugging this is difficult because as far as I can tell 98% of the node logs are errors:

jakubgs@node-01.sm-eu-zur1.spiff.app-test:~ % zgrep -v ERROR /var/log/docker/spiff-workflow-wakunode/docker.* | wc -l
267247
jakubgs@node-01.sm-eu-zur1.spiff.app-test:~ % zgrep ERROR /var/log/docker/spiff-workflow-wakunode/docker.* | wc -l 
17513290

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 28 (27 by maintainers)

Most upvoted comments

This is something I have noticed in the past and noticed today also. Normally I open the Status app only when I want to test the messages. That can be once in few days or once in a week sometimes. I have noticed that most of the time when I open the status app then the instances start to error for some time. Yesterday and today morning I created about 100+ instances and none of them errored. I didn’t have the Status app opened at that time. But later today I opened the Status app and then created those instances (which I shared above) and they started getting the error. This is something I have observed in the past also. Just shared what I have observed. Not sure whether it’s a valid point or not.

These I ran after starting the Status app and some of them errored. image

I ran 50 instances of this in the morning before opening the Status app without any error. And another 100+ instances yesterday without any error.

image