go: proposal: runtime: allow termination of blocked syscalls
Many syscalls may block indefinitely, for example file ops trying paths on a network filesystem or peripheral device that’s unavailable. Such syscalls can often be terminated, but Go is unable to do so at present. So today, Go apps…
- can’t retry a dir, file, etc while it’s unavailable without leaking an OS thread on every attempt.
- can’t let you abandon a stalled task.
- may return immediately to a stalled state after abort and restart.
CIFS (#39237, #38836) and FUSE (#40846) support this by returning EINTR instead of restarting after a signal. NFS on Linux used to do the same, but dropped it a while ago. So not all hung syscalls can be terminated gracefully.
Rust used to retry syscalls on EINTR, but dropped the practice outside of some high-level APIs.
The runtime should provide a way to terminate blocked syscalls. On unix, this entails sending the blocked thread a signal. Windows has an analogous mechanism, CancelSynchronousIo(). See also
https://docs.microsoft.com/en-us/windows/win32/fileio/canceling-pending-i-o-operations.
The solution must not add context.Context variants of all stdlib APIs that make blocking syscalls. Mandating Context arguments for termination would force them into third party package APIs, and code importing those packages would then be broken. If a package author did not amend its API, the callers would have to manage stalled ops some other way, and likely leak resources on every op retry.
I think the simplest way to allow this (other ideas welcome) is to asynchronously post a threadId & metadata to the app (edit: or an internal table) immediately before and after trying a blocking syscall.
The following API could be introduced as experimental (edit: or internal). If we also want cancellable variants of stdlib APIs, Add/DropSyscallPost() could take an argument limiting its scope to the current goroutine, for use by the variants.
A file-oriented variation of this appears in https://github.com/golang/go/issues/41054#issuecomment-692267226. A runtime-internal variation is suggested in https://github.com/golang/go/issues/41054#issuecomment-683155474.
package runtime
// A callback typically implemented to maintain a table of ThreadIds and syscall metadata.
// The name is OS-specific; if "-", thread has completed its syscall.
type PostSyscall func(thread ThreadId, name string, args ...interface{})
// Notify the app of blocking syscalls, by invoking post asynchronously. Each function added is invoked.
// An error results if post is already known.
func AddSyscallPost(post PostSyscall) error
// Stop notifications via post.
// An error results if post is unknown.
func DropSyscallPost(post PostSyscall) error
// Try to terminate a syscall blocked in thread.
// An error results if thread is invalid.
func TerminateSyscall(thread ThreadId) error
// The error returned by syscall after termination.
// The caller should retry the syscall if TerminateSyscall() wasn't called for this thread.
type InterruptError struct { thread ThreadId }
Discussion of this began (more or less) with https://github.com/golang/go/issues/40846#issuecomment-676777624
Changelog
27-Aug: add InterruptError and improve PostSyscall docs
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 7
- Comments: 23 (14 by maintainers)
I am strongly opposed to any form of calling back to user code while deep in invoking a syscall. Even if we carefully write down rules as to what is permitted, people will inevitably write code that breaks those rules, and we will be tied to what people write. We went through years of pain with cgo vs. the garbage collector, and that was in an area where we had explicitly said there was no compatibility guarantee.
That said, perhaps this API could use a channel.
Similarly, the notion of marking the API is experimental is a non-starter. Once people start using an API, it is fixed. We can not break people once they have started using some feature. That’s not how Go works.
Nevertheless, I believe that
context.Contextis the right approach for canceling operations in Go. I don’t think it’s necessary to introduce another approach.There may be a problem here but it doesn’t seem like a new problem or an urgent one.
As discussed above, I don’t think that this specific proposal is a good fix for that problem.
I don’t think we can synchronously call a user function as we are calling a syscall. At the moment of calling a syscall we are in a fairly restricted execution envrionment. Calling user code at that point seems like a footgun.
(I thought that by “asynchronously” you meant that the user code would run in a separate goroutine. I think that could be done safely. I don’t think it would be safe to call a user function and wait for it to return.)
This is incredibly invasive and breaks essentially all the abstractions that the Go runtime works hard to establish. I have a hard time seeing why we would do that.
There may be a problem to solve here, but the answer can’t be tearing down all the abstractions we have. Are you going to let the user TerminateSyscall in the middle of the runtime asking the operating system for more memory? It’s just going to lead to an incredible number of subtle bugs.
I don’t really understand why this API would be necessary in the runtime proper. Isn’t this already possible to implement as a third-party library today, using
runtime.{Lock,Unlock}OSThread(),EINTR, and eithergolang.org/x/sys/unix.TgkillorC.pthread_kill?After catching up on https://github.com/golang/go/issues/40846, it’s not clear to me where this API comes from? It seems like that thread was slowly converging around having versions of os.Stat, etc which take a Context that interrupts the system call when cancelled. Lots of details to be worked out, but a pretty clear API.
On the other hand, I’m worried that this API is both very low-level, and difficult to use. In particular:
@ianlancetaylor I think calling user code could be generally safe so long as this is limited to standard library calls like os.Stat/Readdir, etc, since those aren’t used in any fragile states within the runtime itself. If this is intended to wrap every syscall, then we’re definitely going to have many cases where it is nearly impossible to call user code (e.g., mmap call trying to allocate more stack space in morestack).