cargo: support and eventually upgrade to an alternate jobserver protocol standard that does not leak threads, in collaboration with zig
Problem
The GNU make jobserver protocol leaks threads if a process dies while holding a token. For example:
- Parent process creates the pipe and puts 2 tokens into the pipe, then spawns children A and B.
- Child Process A reads a token and starts doing work, but then crashes before putting the token back.
- Child Process B has 100 tasks to do, but only proceeds to do them using 1 logical core, leaving an entire core idle because it has been “leaked” by Child Process A.
It could get even worse - if another leak occurred in this scenario, the entire process tree would deadlock.
The fundamental problem is that the protocol relies on a process manually releasing resources before it dies, rather than relying on kernel-managed resources that are automatically released when the process dies.
Proposed Solution
Let’s work together on a standard protocol that does not have this flaw, and then pressure GNU make, Ninja, and other third party projects to support it in future versions.
Here is an initial proposal (copied from https://github.com/ziglang/zig/issues/20274). This is something I thought of today and have not evaluated yet. I’m happy to workshop it with the Cargo team.
Initially proposed scheme based on advisory record locks
Root process opens a temporary file (perhaps with `shm_open`) and writes N bytes to the file, where N is thread pool size. This file descriptor is passed down the process tree. Each byte in the file represents an available logical core. Each thread in each process's thread pool holds an advisory record lock (`fnctl(F_SETLKW)`) on the corresponding byte while working. Advisory record locks are automatically released by the kernel when a process dies.In theory this is exactly the same overhead as reading and writing byte-sized tokens to a pipe (1 syscall each for lock and unlock), the difference being the resource is automatically released by the kernel on process death.
Alternate scheme based on the root process listening on a unix domain socket instead: https://github.com/rust-lang/cargo/issues/14102#issuecomment-2177815756
The idea would be to deprecate the legacy GNU make jobserver protocol and try to standardize various open source projects on the better protocol. In the beginning, it could be merely supported when the parent process implements this new standard, and have a flag for choosing it when Cargo is the root process in the tree. Later on, if the new standard achieves widespread adoption, then the default could change.
Notes
If there is interest, I will follow up here with performance data points and other observations as I evaluate this strategy in the Zig project.
Happy hacking!
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 21 (5 by maintainers)
Alright, never mind, there was a solution right in front of my face. Here’s an alternate scheme:
Root process listens on a unix domain socket. The address is passed down the process tree via environment variable. To get a thread token a child process connects to the socket. To return the token, disconnect. The root process only accepts N concurrent connections and stops accepting when it’s maxed out.
When a child process dies, the OS causes the connection to end, allowing the root process to accept a new connection.
@andrewrk A correction: it looks like signals might be viable after all. https://www.man7.org/linux/man-pages/man3/pthread_kill.3.html exists. You could install a signal handler, invoke
pthread_kill, the handler will run on that thread, the handler can set a flag, and after the handler returns,connectshould return withEINTRand the code that invokedconnectcan check the flag.It’d be mildly painful for something general-purpose like zig’s worker pool, you’d probably want to let the user pass in a signal number that it’s OK for you to use (which can have a reasonable default), but it should work.
Update: I’m still interested in this but have put the project on ice for the time being. If y’all end up looking into it and figuring out something that works well, Zig will follow Cargo’s lead. For now though, I was unable to come up with something satisfactory with the available primitives.
@andrewrk Based on the earlier experiment, I had thought
connectnever blocked in the first place, which is why we need theread.Investigating further, it looks like eventually
connectwill block. depending on the backlog size specified inlisten. (Looking at the zig PR thread, it looks like you found something similar.)For instance, if you have a backlog of
2, and your listener accepts only one connection and loops forever (never accepting other connections), the first connect will succeed (being accepted), the next 3 (?!) will successfullyconnectand block inread, and after that, additionalconnectattempts will block.Likewise, with a backlog of
1, the first connection succeeds (being accepted), the next 2 (?!) will successfullyconnectand block inread, and after that, additionalconnectattempts will block.This behavior seems weird enough that it probably can’t be counted on, and I’d be shocked if it doesn’t vary by platform, so I’m not sure if counting on it is a good idea. I’m not sure if it’s viable to count on setting the backlog to a large number (in any case you shouldn’t set it to more than SOMAXCONN), nor should you count on any particular backlog behavior from the jobserver (since many implementations are unlikely to change the default). I think you’re going to have to allow for either
connectorreadblocking.From what I can tell, the behavior of non-blocking
connectis also both finicky and non-standard here; I don’t think you’re likely to be able to usepollto wait for connect to succeed. On Linux at least, non-blocking UNIX sockets just failconnectwithEAGAINrather than returningEINPROGRESS, so you I don’t think you canpollfor completion ofconnect. This seems like a substantial deficiency. I haven’t found any reasonable way to do a non-blockingconnectto a UNIX socket on Linux; there are various unreasonable ways that require jumping through substantial hoops (e.g. requiringio_uring, or spawning a thread/process).You could use the sockopt
SO_SNDTIMEO, which is standardized by POSIX, but I’m not sure all platforms apply that timeout toconnectand not justsend/write. In any case, this would cause you to have to repeatedly wake up when the timeout expires and re-invokeconnect, which seems wasteful, and it’d incur a delay before you can kill worker threads. That might be acceptable, though, since you expect most blocks to occur onread(for whichpollworks) rather thanconnect. But in any case, you’d have to verify that it applies toconnecton all targets.I don’t think you can portably send a signal to a specific thread (rather than a process), leaving aside all the other reasons not to want to use signals.
pthread_cancelsimilarly seems like a bad idea.I expect the behavior of closing the fd (or race-free equivalent like
dup2ing/dev/nullover the fd) would be finicky and non-standard. From reading the Linux code, I think that ifconnecthas reached the point of waiting to connect, it’s just going to keep waiting and not care about its socket having been closed.You could move the blocking from the worker threads themselves to a separate thread that handles making the connections, which would allow you to terminate the worker threads (since they could wait on something like a channel), but that just moves the problem to the connection thread.
If you want something to be able to block in either
connectorread, but you also want to be able to bail out and end the worker pool by some means other than your process ending, I don’t have anything else obvious to suggest; non-blockingconnectdoesn’t work on UNIX sockets, and blocking connect can’t easily be terminated.@andrewrk wrote:
This seems like a great solution that’s reasonably portable; it’d work on any UNIX, and it’d work on modern Windows.
@andrewrk wrote:
It’d be an extra syscall, but one thing that should work portably on any UNIX system would be to connect and then read one byte, and have the jobserver always write a single byte to each socket after accepting it.
If that turns out to be the only way to get this to work, then another possible protocol, which might be faster for multi-job processes but marginally slower for single-job processes, would be for clients of this protocol to first write a byte to the connected socket for each job it wants to start, and then read a byte per job. This would allow connecting a socket once and then requesting multiple jobs from it, rather than connecting a socket per job. For a multi-job process, that reduces the overhead to a
writeandreadper job (both of which may be possible to batch), and it doesn’t require the overhead of updating the fd table repeatedly. But for a single-job process, this would mean one more syscall (awrite) on top of what’s now already asocket/connect/read.It’s unfortunate that this protocol already requires 3 syscalls to start each job, and another to close it (if not closing it via process exit). This isn’t substantially worse than the 2+1 of the current jobserver protocol (which requires
openandreadand a subsequentwrite) or the 1+1 of the old one (inherit fd, do areadand subsequentwrite), but it’s not ideal. However, I don’t see any obvious way to do substantially better, even without taking portability into account, while preserving the property of automatically releasing on close.