pueue: [BUG] excessive memory and CPU usage with large outputs

I’m using pueue to queue up some large file copies – and it’s pretty nice!

However I notice after a while, it’s taking several seconds of pueued CPU time to pueue add date a new task. I would have expected it’s just adding it to some kind of in-memory or on-disk queue, and the time should be proportional to the fairly short command line.

I also see pueued seems to be using a lot of memory for what it’s doing:

  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND 
16885 pts/11   SNl+   2:54     71  1930 1043077 779556 13.3 pueued

I don’t know how to reproduce it, but maybe it’s connected to some of these commands producing a lot of output? Ideally it would just stay on files on disk and not be in memory.

This is pueue 0.3.1 from Cargo, on Debian Buster.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 19 (19 by maintainers)

Commits related to this issue

Avoid repeatedly cloning large output strings Hold stdout and stderr in Arc refcounts so that the task can be cloned or sent between threads without cloning the potentially-large buffers. In particu... — committed to sourcefrog/pueue by sourcefrog 4 years ago

Most upvoted comments

Ok. This took a little more refactoring than expected. Anyhow, we now have a client that can read logs from disk 😃)

Daemon:

Client:

Nukesor on May 1, 2020

The heap profile has been taken with valgrind and the --tool=massif option. Visualization via massif-visualizer

Nukesor on Apr 30, 2020