goakt: Cluster does not rebalance

Bug description Cluster does not rebalance

How to reproduce it? Same example as #573 , start 3 nodes cluster with docker compose, after everything is up, issue docker compose down node1, the node will be shutdown but the running actors are not re-deployed.

Expected behavior Actors on dead node should be re-deployed on remaining nodes

Screenshots If applicable, add screenshots to help explain your problem.

Library Version:

  • Go-Akt version: 2.10.0
  • Go version: 1.23

Additional context

node3-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.2:50052/Source"}
node3-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.2:50052/Plugin-multiply"}
node3-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] MULTIPLY: 0.51119995"}
node3-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.2:50052/Plugin-demo"}
node3-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] DEMO: 1.0223999"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.3:50052/Source"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.3:50052/Plugin-multiply"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] MULTIPLY: 0.62782377"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.3:50052/Plugin-demo"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] DEMO: 1.2556475"}
node2-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.4:50052/Source"}
node2-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.4:50052/Plugin-multiply"}
node2-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] MULTIPLY: 0.564825"}
node2-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.4:50052/Plugin-demo"}
node2-1  | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] DEMO: 1.12965"}
node2-1 exited with code 2
node3-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.2:50052/Source"}
node3-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.2:50052/Plugin-multiply"}
node3-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] MULTIPLY: 0.34328568"}
node3-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.2:50052/Plugin-demo"}
node3-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] DEMO: 0.68657136"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.3:50052/Source"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.3:50052/Plugin-multiply"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] MULTIPLY: 0.66686666"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.3:50052/Plugin-demo"}
node1-1  | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] DEMO: 1.3337333"}

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25 (16 by maintainers)

Most upvoted comments

Great news. Although I agree on handling both SIGTERM and SIGINT I’m concerned about node crashing for whatever reason, like power failure, where the actor system cannot be shutdown properly and I expect the remaining nodes will take over and restart the failed actors.

@webfrank both issues have been fixed. The PR has been merged. You can use the latest commit hash on the main branch before I cut a new release.