tilelive: `--concurrency` in `tilelive-copy` doesn't appear to have any affect
It appears (to me) that the --concurrency argument to tilelive-copy doesn’t have any affect. No matter what value I put in top shows that 4 node processes are running, regardless if I’m using a machine with 4 CPU cores (my desktop) or a server with 8. Regardless of the --concurrency value, I get approximately the same throughput when doing a copy. Changing --concurrency to ten times it’s old value doesn’t appear to increase (or decrease) the throughput.
Is there something I’m missing to get increased concurrency?
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 26 (5 by maintainers)
@mapsam thanks so much for going out of your way to look into this. I’ll give this a shot this AM (Korean time) and report back!
Thanks for the info- it’s helpful to know which tilelive plugins you’re using since this module only connects the interfaces. Upon quick glance it looks like this could be fixed in the tilelive-bridge & tilelive-mapnik modules, which both use mapnik-pool (which sets threadpool size based on number of cpus here). Each module sets the size differently:
UV_THREADPOOL_SIZEin the environment, otherwise defaults torequire('os').cpus().lengthcpus() * 2and is not configurable it seems. Would gladly accept a PR that first triesprocess.env.UV_THREADPOOL_SIZEand backs up to making a best guess.My suggestion would be to test changing that number in whichever way makes sense for you to see if that helps speed things up for you. @ianthetechie Glad to hear you’re working on something better, feel free to share your work or backport fixes in the tilelive modules!
@yhahn This issue have been open for quite a while. If altering concurrency is not supported, it may be better to remove the documentation and close this issue. If there is a way to configure it, it would be super if you would share it (I don’t mind writing an updated readme).