rustup: Rustup (including proxies) is not safe for concurrent use
(Edited to capture all the details that have emerged over time)
Recovering from this bug:
Usually just doing a rustup component remove NAME && rustup component add NAME will fix things. Sometimes removing the entire toolchain will be needed. In rare cases uninstalling rustup entirely will be needed.
User model of Rustup
Rustup may be run as three different commands:
- rustup-init to install rustup (& by default a toolchain)
- rustup to explicitly query or modify an installation (including rustup itself and one or more toolchains)
- as a proxy of rustc, cargo etc (& which can implicitly trigger installation, upgrade of modification of a toolchain e.g. through toolchain files)
Locking in Rustup
Rustup gets run concurrently in two very different contexts: within a single (machine, user), it may be run concurrently by the user, or an IDE, or both, to perform tasks ranging from toolchain installation, component addition, documentation opening. All of these require multiple reads to be made to the rustup data with a consistent view; some of them require writes to be made as well.
Rustup may also be run across machines, where a different machine but a shared rustup install is present - (machineA, userA) + (machineB, userA) - and in this case the same set of operations may take place, with the same requirements.
Whatever consistency solution we adopt would be best if it can deliver both use cases, and not require manual lock administration as lockdir style solutions do, nor additional running network daemons.
Proxies
Rustup has one set of proxies shared across all toolchains; the proxies are held open while any process from a toolchain is running - e.g. IDE’s hold rls open for extended periods.
We need a lock to prevent concurrent attempts to install new proxies, and we need a notification mechanism back to the running proxies to allow them to be notified to exit when an update is required (because of presumed limitations of in-use-file-replacement on Windows, though recent changes may mean we can avoid this)
Toolchains
We have many toolchains in one rustup installation; stable, beta, nightly, dated nightly versions, and custom versions. Adding a toolchain adds a directory and a hash file; we need a lock to prevent collisions attempting to move the directory into place. Deleting a toolchain does a recursive rm in-place, which also needs a lock to prevent other rustup invocations presuming that the toolchain is actually installed during the time the deletion takes place (or perhaps we need to rename-then-delete, though that can run into race conditions with virus scanners, especially if the toolchain was just installed). Further, permitting deletions at any point will require notifications to running rls process proxies from that toolchain to cause them to shutdown, or the .exe is likely not deletable on Windows.
Components
Components such as rls are added into a toolchain directory, and also involve writing to a metadata file within the toolchain tree itself. This needs to be locked to avoid corruption/dropped writes. As with toolchains, we need proxy notification for component removal, as well as a way to make sure that a component that is being removed does not have new instances of it spawned between the start of the removal and the completion of the removal.
Downloads
We download packages for rustup itself, toolchains and additional components for toolchains, and (a corner case) custom installer executables for toolchains. We also download digital signature metadata files.
The same file can be downloaded by two different rustup invocations for two different reasons. For instance, downloading nightly and a dated nightly for today, will download the same file(s).
We used to leak partially downloaded files, and recently started deleting all download dir contents when rustup finished running; this is causing errors now.
We need some mechanism to deal with leaks, but also to permit concurrent execution of rustup to be downloading files without interruption. Possibly a date based mechanism or locking based mechanism would be sufficient.
Network file systems & rustup
Linux handles SMB mounts and locking on that filesystem well, at least per my reading of the source - a rustup dir on an SMB mounted file system using regular posix locks should have those locks reflected as SMB locks.
NFS is well known for having poor lock behaviour when the network services are not running or are firewalled; the underlying syscalls are only non-blocking on the filedescriptor themselves, and EWOULDBLOCK is defined as the lock being already held, not the OS being unable to determine if the lock is already held…
[EWOULDBLOCK]
The file is locked and the LOCK_NB option was specified.
So it is likely that we will see bug reports of rustup hanging when ~/.rustup is on NFS and the NFS server’s lock RPC service is misconfigured or flaky.
I think this be mitigated by emitting an NFS specific log message when taking a lock out on NFS once per process; with a config option to disable that for users that are confident they don’t need it… and a bug reporting hint to tell us they have disabled it.
Locks and notifications
OS locks typically don’t have callback notifications built in; polling is one answer, or some form of lightweight message bus (e.g. zmq) with clear rules tied into the lock management layer. We have to be careful about race conditions though: in particular notifying before or after as appropriate.
About this issue
- Original URL
- State: open
- Created 10 years ago
- Reactions: 28
- Comments: 31 (25 by maintainers)
Commits related to this issue
- Attach `/cargo-lock` locks to all dune rules using rustup or cargo There is currently an issue with `rustup`[0] that can cause the toolchain to fall into an unusable state if multiple instances are r... — committed to MinaProtocol/mina by mrmr1993 5 years ago
- fix(gitpod): workaround rustup's concurrent issue Related: rust-lang/rustup#988 Related: rust-lang/rustup#2417 — committed to pan93412/ferrumfix by pan93412 4 years ago
After thinking about this issue for quite a while, I have formed the following early ideas around a hypothetical file-based global RwLock:
rustupcommand, called “Rustup” below): Acquires write lock on enter, releases on exit.cargoandrust-analyzer, called “Proxy” below): Acquires read lock on enter, releases on exit.execon Unix, unless we can get the cleanup code running when the current PID is repurposed (which seems possible by usingflockon dummy files).The annoying thing here is that we also have to consider the subprocesses we’re creating (since the critical sections could be nested), so:
… and it looks like if we’re making a re-entrant lock, the uniqueness of its ownership should be based on the root of the process’ subtree (up to a certain Rustup/proxy). This info might be conveniently passed on via a new env variable and/or via the lock file itself. Something like C#'s “named mutex” is close to what I’m looking for, which happens to be implemented using lock files on Unix.
All these points rely on the fact that a proxy cannot implicitly modify the current installation (so can be safely granted read-only access), which has fortunately been ensured by #3635.
Further down the line, more “Rustup mode” operations could be made to acquire read locks by default instead, and these locks might better be managed on a per-toolchain basis.
I’m actively trying to unblock this issue, although I’m not sure about the correctness of this reasoning… I’m trying this out in a local repo… Any input is welcome. 🤔
Update: @weihanglo suggests that such a RwLock already exists in Cargo (or more precisely, its variation, as it holds two reentrant file-based locks to form a custom 3-mode global RwLock), so that should be a solid starting point.
cc @rbtcollins what was the concern in https://github.com/rust-lang/rustup/issues/988#issuecomment-582285054 that made you think Cargo’s lock has semantics different enough from what we’d expect?
Just found this issue again and the solution seemed to be to close all Visual Studio Code instances before doing the
rustup updateand that worked, but unfortunately it does look like Visual Studio Code is locking these files despite the windows not being active or in use.Ok, so here’s a bit of a specification, I think this ties together all the various bits involved.
Locking in Rustup
Rustup gets run concurrently in two very different contexts: within a single (machine, user), it may be run concurrently by the user, or an IDE, or both, to perform tasks ranging from toolchain installation, component addition, documentation opening. All of these require multiple reads to be made to the rustup data with a consistent view; some of them require writes to be made as well.
Rustup may also be run across machines, where a different machine but a shared rustup install is present - (machineA, userA) + (machineB, userA) - and in this case the same set of operations may take place, with the same requirements.
Whatever consistency solution we adopt would be best if it can deliver both use cases, and not require manual lock administration as lockdir style solutions do, nor additional running network daemons.
Proxies
Rustup has one set of proxies shared across all toolchains; the proxies are held open while any process from a toolchain is running - e.g. IDE’s hold rls open for extended periods.
We need a lock to prevent concurrent attempts to install new proxies, and we need a notification mechanism back to the running proxies to allow them to be notified to exit when an update is required (because of presumed limitations of in-use-file-replacement on Windows, though recent changes may mean we can avoid this)
Toolchains
We have many toolchains in one rustup installation; stable, beta, nightly, dated nightly versions, and custom versions. Adding a toolchain adds a directory and a hash file; we need a lock to prevent collisions attempting to move the directory into place. Deleting a toolchain does a recursive rm in-place, which also needs a lock to prevent other rustup invocations presuming that the toolchain is actually installed during the time the deletion takes place (or perhaps we need to rename-then-delete, though that can run into race conditions with virus scanners, especially if the toolchain was just installed). Further, permitting deletions at any point will require notifications to running rls process proxies from that toolchain to cause them to shutdown, or the .exe is likely not deletable on Windows.
Components
Components such as
rlsare added into a toolchain directory, and also involve writing to a metadata file within the toolchain tree itself. This needs to be locked to avoid corruption/dropped writes. As with toolchains, we need proxy notification for component removal, as well as a way to make sure that a component that is being removed does not have new instances of it spawned between the start of the removal and the completion of the removal.Downloads
We download packages for rustup itself, toolchains and additional components for toolchains, and (a corner case) custom installer executables for toolchains. We also download digital signature metadata files.
The same file can be downloaded by two different rustup invocations for two different reasons. For instance, downloading nightly and a dated nightly for today, will download the same file(s).
We used to leak partially downloaded files, and recently started deleting all download dir contents when rustup finished running; this is causing errors now.
We need some mechanism to deal with leaks, but also to permit concurrent execution of rustup to be downloading files without interruption. Possibly a date based mechanism or locking based mechanism would be sufficient.
Network file systems & rustup
Linux handles SMB mounts and locking on that filesystem well, at least per my reading of the source - a rustup dir on an SMB mounted file system using regular posix locks should have those locks reflected as SMB locks.
NFS is well known for having poor lock behaviour when the network services are not running or are firewalled; the underlying syscalls are only non-blocking on the filedescriptor themselves, and EWOULDBLOCK is defined as the lock being already held, not the OS being unable to determine if the lock is already held…
So it is likely that we will see bug reports of rustup hanging when
~/.rustupis on NFS and the NFS server’s lock RPC service is misconfigured or flaky. I think this be mitigated by emitting an NFS specific log message when taking a lock out on NFS once per process; with a config option to disable that for users that are confident they don’t need it… and a bug reporting hint to tell us they have disabled it.Locks and notifications
OS locks typically don’t have callback notifications built in; polling is one answer, or some form of lightweight message bus (e.g. zmq) with clear rules tied into the lock management layer. We have to be careful about race conditions though: in particular notifying before or after as appropriate.
Regarding https://github.com/rust-lang/rustup/issues/988#issuecomment-2340369437, I’ve made a working on Unix PoC for a mocked
rustup/cargo/rust-analyzerscenario using a combination ofexecvp(),fcntl()andopen()syscalls on a marker file.The outcome has been positive overall:
rustupcommands, for example, when trying to acquire a write lock, would be blocked byrust-analyzer’s read lock.The two key points mentioned in the previous comment seem to have been implemented, namely the inheritance of locks from parent process to child, and the upgrade of a read lock to a write lock on
rustupcommands, thanks to the fact thatopen()works acrossexec*()ifCLOEXECis not set (note: this is unfortunately not the case withstd::fs). The currently-open FD is passed to the subprocess via an extra env variable, where its ownership is recovered byFile::from_raw_fd(). This seems alright in terms of handle ownership, but I’m not quite sure if this is acceptable security-wise.However, this comes with a notable limitation: since these properties (lock inheritance and lock upgrade) are limited to subprocesses launched using
exec*(), this implementation might deadlock any proxy trying to runrustupcommands (in incompatible ways) that require a write lock, e.g.cargo runrunning a program that usesCommand::spawn()to runrustup toolchain install. Fortunately, thanks to the shared nature of the read lock, other use cases (including proxies) seem to work without issues, regardless of the nested critical sections.The lock is currently not reentrant in-process, but when integrating the solution into
rustup, it should be relatively easy to adaptcargo’s approach to make it so.Next up, I’ll try to clean up the PoC and make it public for reference.
There are still many questions to be answered, to name a few:
rustupshould be executed withexec*()only)?rustupthroughCommand::spawn()and friends? How should we help handle the potential migration?cargosuffers from the same limitation: https://github.com/rust-lang/cargo/issues/6412@rbtcollins OK. Just to clarify my own understanding of the issue as @matklad described it, if the update crashes while it is happening, it could leave the system in an indeterminate state, which is enough of a problem that we want some method of avoiding it completely, correct? If that is true, below is a first stab at solving the two problems. Every process is required to follow the steps below, or everything fails.
/tmp. My current understanding is that this is not guaranteed to be atomic, but that doesn’t matter, the only requirement is that they try to create this file./tmp/rustup_lock_file. Bonus points if the contents of the file include a short readme so that anyone wondering what this file is can get an idea as to what it’s for. It may even contain a URL pointing to this issue, or something similar./tmp.~/.rustupnamedstate_machine. If this file does not exist, then the last process to do any work succeeded, and the system is in a clean state. Since the process owns the lock, it can now safely create thestate_machinefile, and update its contents with status messages, state information, etc., etc., etc. safely. Once all work is done, the process first deletes thestate_machinefile, and then deletes the/tmp/rustup_lock_file, confirming that the former is truly committed before executing the latter.There are a few advantages to the above algorithm:
rustup, rebooting might clear that issue out, letting the process continue to completion after reboot.state_machinefile gives a hint as to where the process was when it failed. This could be useful when debugging what’s going on, and for general awareness of the current state of the process (e.g.,tail -f ~/.rustup/state_machineis a quick and dirty way of watching the current progress of the process).The one big disadvantage is that this is not a truly transactional algorithm. I don’t yet know how to do that, but maybe some of the database literature would help with that?
rustup needs to have concurrency support bolted on in general. Need to extract cargo’s flocking code.
i got bit by this issue today. i was able to solve this by stopping running rust-analyzer processes, uninstalling my rustup toolchains, and trying again. the error message provided was not particularly helpful or clear though, by the rust project’s usual standards.
thank you for the remediation steps outlined here.
I think Windows is a bit simpler here. On Windows the rustup process stays alive for the duration of the proxied binary no?
Locking a file can be done either by setting the
share_modeor by usingLockFileEx.A third option would be to craft an rwlock from interprocess synchronization objects. However this may be trickier to get right.
Further thoughts: we need to mutually exclude operations on toolchains: removal and upgrade affect the entire toolchain as well as operations on components (add/remove components). I think locking the dir of the toolchain is probably the right control point.