goakt: Performance regression after updating to 2.5.0

Bug description

I’m building a demo web app using goakt. It spawns over 600 vehicle actors during its runtime.

With goakt version 2.4.0, I can get over 35k rps when running load test for the request to get the vehicle position. The memory usage is over 17GB.

> hey -c 100 "http://localhost:8080/vehicle?id=0012.02247" -z 30s
Summary:
  Success rate:	1.0000
  Total:	30.0036 secs
  Slowest:	0.4775 secs
  Fastest:	0.0000 secs
  Average:	0.0028 secs
  Requests/sec:	35629.9656

After updating to version 2.5.0, the number of rps dropped to only 3.5k. The memory usage is only <40MB (which is superb).

> hey -c 50 "http://localhost:8080/vehicle?id=0018.00222" -z 30s
Summary:
  Success rate:	1.0000
  Total:	30.0029 secs
  Slowest:	0.3449 secs
  Fastest:	0.0001 secs
  Average:	0.0139 secs
  Requests/sec:	3588.8494

How to reproduce it?

  1. Clone my repo https://github.com/sdil/goakt-realtimemap
  2. Run the server with go run . on first terminal
  3. Run load test on /vehicle?id=<actor name> endpoint. I’m using https://github.com/rakyll/hey so the full command looks like this: hey -c 50 "http://localhost:8080/vehicle?id=<actor name>" -z 30s. You can get the actor name from the log in the terminal
  4. Observe the throughput.

Change the goakt version to 2.4.0 and repeat the same process.

Expected behavior

The performance to remain around the same.

Screenshots

N/A

Library Version:

  • Go-Akt version: 2.5.0
  • Go version: go1.22.5 darwin/arm64

Additional context

When running with 1 actor, the performance is kinda similar. But when I spawn 10 actors, it started degrading significantly.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 55 (34 by maintainers)

Most upvoted comments

@sdil using the mpsc queue seems increase some throughput:

=== RUN   TestBenchmark_BenchTell
starting benchmark for (30s): num workers:(50)
total actors spawned: (2000)
total workers: (50), total messages sent: (105993035), total messages received: (105993035) - duration: (30s)
messages per second: (3533101)
--- PASS: TestBenchmark_BenchTell (52.73s)
PASS

@sdil closing the issue. Let us revisit it when things get worst.

That’s pretty good. I wonder how does that change improves the performance

The previous implementation of the receive is just looping even when there is no messages in the mailbox which creates unnecessary busy threads. The enhancement is to react only messages are dropped in the mailbox.

@sdil I will spend some time and see how best to tweak the code.

I can’t message you either.

@sdil I will have through your implementation and see what is going on.

Ahh i saw that this morning. I’ll give it a try later

I noticed you used generics for your mpsc implementation. I believe, I tried this way and moved back to the 'any" type due to performance issues. But, honestly I had no enough time to dig into it.

Interesting. Let me take your advice and see the outcome. Thanks for pointing out. @halturin I found this article https://planetscale.com/blog/generics-can-make-your-go-code-slower

Please let us leave the issue open.

Thank you for the release @Tochemey great work! I’m pretty sure we can find some other optimizations in the future 😀

@sdil I have released 2.5.1 to include the mspc queue and few fixes. At the moment, Go-Akt is not yet close to ergo. Let us leave the issue open while I dig more in some the implementation and see where we can gain some more performance.

Personally, I liked go-akt syntaxes because it feels closer to MS Orleans. I really liked that I can do scheduling, passivation, etc. I was abit struggled with proto messaging, but that’s because I never used it. Now I understand the type checking it provides.

Ergo looks nice too but I haven’t give it a try yet. Ergo looks closer to Erlang & OTP but I struggled with Elixir in the past because of its syntax and paradigm.

What are the current cluster provider engine integration?(k8, NATS, etc…)

its in the roadmap now.

didnt get the first two.

maybe I may abandon Go-Akt and redirect people to ergo as long as it does not require licensing 😃 because that is the goal of Go-Akt.

please, don’t. we might have different vision of “where to go/how to do”.

see v300 then 😃 its about to release. preparing the docs now. planned for coming September.

https://github.com/ergo-services/examples/tree/v300 - examples. see “demo” there (it has “observer” app embedded, open http://localhost:9911 to see the node/network/processes etc)

@Tochemey i believe you can reach the same numbers easily

I hope so 😃

Did you consider MPSC queue as suggested by @halturin? It seems like the same queue implemented in Akka

Ref: https://concurrencyfreaks.blogspot.com/2014/04/multi-producer-single-consumer-queue.html

Thanks. I am going to give it a try 😃

@sdil I have been able to reproduce it. I will take a look at what may be the issue in the weekend. Thanks