wasmtime: Add `nix` support for missing host function calls.

At a quick glance, the following functions are not yet provided by nix, but would be useful to have in nix for use by wasi-common: mkdirat, unlinkat, renameat, linkat, clock_getres, clock_gettime, posix_fallocate, posix_fadvise.

Currently code in wasi-common and lucet-wasi is calling libc interfaces for these functions directly, which requires unsafe code. Having nix provide these will mean the unsafe code is shifted to nix instead. This may not seem like a win on a theoretical level, but the more we can push unsafe code into dedicated utilities that do simple unsafe things with safe, testable, interfaces, the easier it is to audit everything.

I’ve marked this as “help wanted” – this is not an urgent project, but it is a nicely scoped, straightforward, and subdividable project, and it’s a good chance to see some of the low-level parts of the software stack that wasi-common is built on. And mentoring is available 😃.

Additional resources:

Documentation for these functions in POSIX can be found here: http://pubs.opengroup.org/onlinepubs/9699919799/
Here’s an example of implementing a very simple function in nix, though the functions above are more complex and will need testcases 😃.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 23 (10 by maintainers)

Commits related to this issue

Merge #1084 1084: sys/stat: implement mkdirat r=asomers a=kevinwern See: https://github.com/CraneStation/wasi-common/issues/16 https://linux.die.net/man/2/mkdirat My first contribution to this r... — committed to nix-rust/nix by bors[bot] 5 years ago
Merge #1089 1089: implement posix_fadvise r=asomers a=kevinwern See: https://github.com/CraneStation/wasi-common/issues/16 http://man7.org/linux/man-pages/man2/posix_fadvise.2.html Conditional... — committed to nix-rust/nix by bors[bot] 5 years ago
Merge #1089 1089: implement posix_fadvise r=asomers a=kevinwern See: https://github.com/CraneStation/wasi-common/issues/16 http://man7.org/linux/man-pages/man2/posix_fadvise.2.html Conditional... — committed to nix-rust/nix by bors[bot] 5 years ago
Implement path_filestat_set_times using nix instead of libc. As advised in #16. — committed to marmistrz/wasi-common by marmistrz 5 years ago
Implement path_filestat_set_times using nix instead of libc. As advised in #16. — committed to CraneStation/wasi-common by marmistrz 5 years ago

Most upvoted comments

I think unlinkat is close to merging. I will give linkat and symlinkat a go if no one else has started these.

jlb6740 on Jul 15, 2019

(moved to https://github.com/CraneStation/wasi-common/issues/132)

nomeata on Oct 18, 2019

memcpy should be able to be replaced with std::ptr::copy_nonoverlapping

programmerjake on Jun 24, 2019

Continuing to chip away at this, also did a bit of grepping here for direct references to libc functions. I ended up with:

clock_getres (https://github.com/nix-rust/nix/pull/1100/)
clock_gettime (https://github.com/nix-rust/nix/pull/1100/)
fdopendir (@jlb6740 working on this)
futimens (this is actually done already)
linkat( @jlb6740 working on this )
memcpy (think this is more refactoring on our end, see dec_slice_of vs enc_slice_of–other call should disappear if we use nix for directory reading)
mkdirat (https://github.com/nix-rust/nix/pull/1084)
posix_fadvise, (https://github.com/nix-rust/nix/pull/1089)
posix_fallocate ( nix-rust/nix#1105 )
readdir_r ( assessing using nix::dir for this )
renameat (worked on indepently of this issue, https://github.com/nix-rust/nix/pull/1097)
sched_yield (https://github.com/nix-rust/nix/pull/1050, https://github.com/nix-rust/nix/pull/1090)
seekdir ( assessing using nix::dir for this )
symlinkat ( @jlb6740 working on this )
unlinkat, (https://github.com/nix-rust/nix/pull/1058)
telldir ( assessing using nix::dir for this )
utimensat (this is actually done already)

I need to look more into whether or not nix::dir as it is can replace the directory reading/iterating calls listed here. I think that’s the case.

Any input @sunfishcode @kubkon ? I’ll keep track of things here to the best of knowledge.

kevinwern on Sep 24, 2019

@jlb6740 noted that on the list. Go ahead!

kevinwern on Jul 16, 2019

@kevinwern That list looks good! As a minor update, sched_yield is now in upstream nix here so we just need to update wasi-common to use it.

That’s a good catch. One caveat here though. It is currently quite unfortunate that sched_yield landed in sched module in nix crate which is compiled only for linux-like hosts. This precludes its use on macos for example. I have now submitted a PR to remedy this nix-rust/nix#1090, and I guess when it lands, we can safely move away from using libc::sched_yield directly.

kubkon on Jun 24, 2019

Definitely up to help out however I can…I remember confirming that the list @sunfishcode had at the top of this issue covered everything but I will double-check that tomorrow. Regardless, I’ll try to pick off things bit by bit.

kevinwern on Jun 12, 2019

Hey, that’s awesome, thanks! It’d be great if we could get all missing host function calls added to nix and especially those that we rely on internally in wasi-common. Would you be willing to scan the repo, find any calls that rely directly on libc, and try adding them to nix? For example, we’re still missing posix_fadvise.

Feel free to ask any questions, and I’ll be happy to help out as much as I can, and @sunfishcode can mentor and straighten me out in case I’m saying something stupid 😉

kubkon on Jun 10, 2019