nng: nng creates too many threads

Describe the bug I’m running a 32-core/64-thread threadripper machine, and it looks like in total nng creates 151 threads.

While this could be the wanted case for some apps, for the application that I’m writing it’s all around way too much, and clutters up the profiling tools quite a bit.

Thread name	Thread count
nng:iocp	64
nng:ipc:dial	1
nng:resolver	4
nng:task	16
nng:reap2	1
nng:timer	1
nng:aio:expire	64

Expected behavior Ideally, thread creation is left to the application instead of being done by the nng framework, that way I can control scheduling of work and fit it into my async execution runtime, instead of having nng take up resource (and arguably more important) screen real estate in the profiler.

In our application we’re trying to severely limit our number of threads and schedule work on our own.

Actual Behavior See above

To Reproduce Simply run nng on a high core-count machine.

** Environment Details **

NNG version: we’re using the nng-rs crate, I think it’s a bit behind on latest nng
Operating system and version: Windows
Compiler and language used: rustc
Shared or static library: unknown (and probably irrelevant)

Additional context I found an additional issue on limiting the amount of threads being spun up in #769 this seems to require compile time toggles? Ideally this is something that can be set at runtime (we have a few different use-cases all sharing the same codebase - one case where run the service inproc and want a low thread count, one where we run the service as a daemon and we don’t really care about thread count).

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 23 (15 by maintainers)

Commits related to this issue

fixes #1572 nng creates too many threads This further limits some of the thread counts, but principally it offers a new runtime facility, nng_init_set_parameter(), which can be used to set certain ru... — committed to nanomsg/nng by gdamore 6 months ago
fixes #1572 nng creates too many threads This further limits some of the thread counts, but principally it offers a new runtime facility, nng_init_set_parameter(), which can be used to set certain ru... — committed to nanomsg/nng by gdamore 6 months ago
fixes #1572 nng creates too many threads This further limits some of the thread counts, but principally it offers a new runtime facility, nng_init_set_parameter(), which can be used to set certain ru... — committed to nanomsg/nng by gdamore 6 months ago
fixes #1572 nng creates too many threads This further limits some of the thread counts, but principally it offers a new runtime facility, nng_init_set_parameter(), which can be used to set certain ru... — committed to nanomsg/nng by gdamore 6 months ago
fixes #1572 nng creates too many threads This further limits some of the thread counts, but principally it offers a new runtime facility, nng_init_set_parameter(), which can be used to set certain ru... — committed to nanomsg/nng by gdamore 6 months ago

Most upvoted comments

I’m using 1619 as the EXPIRE_THREADS tunable, and a PR is out… (I’m waiting to merge that as I’ve uncovered some other worse bugs that I want to ensure are fixed first.)

I expect this will merge.

I’ll revisit the poller for Windows to do something similar for it as well. Longer term we need to look at ways to improve the parallelism in the pollers anyway – in nanomq they are running into scalability issues beyond 64 cores with the single epoll() thread.

gdamore on Apr 24, 2023

So initially Im going to add compile time defaults that are sensible. I may add runtime for these as well.

gdamore on Apr 20, 2023

I made a quick-and-dirty local patch to enable initialization-time tweaking of thread counts. It’s still a bit rough around the edges but works well enough for me to evaluate.

It adds the following API functions:

void nng_set_ncpu_max(int);  // limit the number of CPUs reported
void nng_set_pool_thread_limit_min(int); // configure minimum number of threads in thread pools
void nng_set_pool_thread_limit_max(int); // configure maximum number of threads in thread pools
void nng_set_resolve_thread_max(int); // configure number of resolve threads

patch is here

I wasn’t sure what would be a good naming scheme for these as I didn’t see any similar function to mimic, feedback is welcomed! (I already know nng_set_resolve_thread_max is a misnomer as it sets the actual count)

bionicbeagle on Nov 30, 2022

In head, we got read of the nni_timer thread. It’s only thread, but still an improvement, if only a small one.

gdamore on Dec 18, 2023

Btw, there are compile time fixes for these that * did* merge. Runtime is a different matter. Probably it will still need to be set up by the application before calling nng_init() or other functions, because the initial threads have to be configured at startup.

gdamore on Sep 14, 2023

did a fix ever merge?

gedalia on Jun 23, 2023

I’ve been evaluating nng for use in some tooling and this issue is a blocker for us as well. Much like @Jasper-Bekkers we often use machines with high core counts and the number of threads is a problem for debuggability and performance profiling since the threads cause lots of clutter as well as making the library needlessly resource intensive regardless of whether threads do actual work or not. In the case of IOCP especially there’s typically no real need to have a huge number of threads as long as you only do I/O related work in the completion handlers.

We are currently consuming nng via vcpkg and therefore using build-time defines to reconfigure nng is not really a great option. Ideally there would be an API to allow application-specific tuning. I’m currently not familiar enough with the project to propose what that would look like but might take a stab at it if there is no attractive alternative.

bionicbeagle on Nov 28, 2022