goakt: Cluster does not rebalance
Bug description Cluster does not rebalance
How to reproduce it? Same example as #573 , start 3 nodes cluster with docker compose, after everything is up, issue docker compose down node1, the node will be shutdown but the running actors are not re-deployed.
Expected behavior Actors on dead node should be re-deployed on remaining nodes
Screenshots If applicable, add screenshots to help explain your problem.
Library Version:
- Go-Akt version: 2.10.0
- Go version: 1.23
Additional context
node3-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.2:50052/Source"}
node3-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.2:50052/Plugin-multiply"}
node3-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] MULTIPLY: 0.51119995"}
node3-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.2:50052/Plugin-demo"}
node3-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] DEMO: 1.0223999"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.3:50052/Source"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.3:50052/Plugin-multiply"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] MULTIPLY: 0.62782377"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.3:50052/Plugin-demo"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] DEMO: 1.2556475"}
node2-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.4:50052/Source"}
node2-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.4:50052/Plugin-multiply"}
node2-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] MULTIPLY: 0.564825"}
node2-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.4:50052/Plugin-demo"}
node2-1 | {"level":"info","ts":"2024-12-19T15:39:51Z","msg":"[PLUGIN] DEMO: 1.12965"}
node2-1 exited with code 2
node3-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.2:50052/Source"}
node3-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.2:50052/Plugin-multiply"}
node3-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] MULTIPLY: 0.34328568"}
node3-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.2:50052/Plugin-demo"}
node3-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] DEMO: 0.68657136"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[SOURCE] Tick: goakt://iotbuilder@172.18.0.3:50052/Source"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.3:50052/Plugin-multiply"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] MULTIPLY: 0.66686666"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] Tick: goakt://iotbuilder@172.18.0.3:50052/Plugin-demo"}
node1-1 | {"level":"info","ts":"2024-12-19T15:39:54Z","msg":"[PLUGIN] DEMO: 1.3337333"}
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 25 (16 by maintainers)
Great news. Although I agree on handling both SIGTERM and SIGINT I’m concerned about node crashing for whatever reason, like power failure, where the actor system cannot be shutdown properly and I expect the remaining nodes will take over and restart the failed actors.
@webfrank both issues have been fixed. The PR has been merged. You can use the latest commit hash on the main branch before I cut a new release.