libs-team: ACP: Add nul-terminated version of `core::panic::Location::file`
Proposal
Problem statement
When using #[track_caller] in codebases that mix C and Rust, you often wish to pass the caller’s filename to a C api. However, this usually requires a nul-terminated string.
Motivating examples or use cases
I would like to utilize this in the Linux kernel to implement a Rust equivalent of the following utility:
/**
* might_sleep - annotation for functions that can sleep
*
* this macro will print a stack trace if it is executed in an atomic
* context (spinlock, irq-handler, ...). Additional sections where blocking is
* not allowed can be annotated with non_block_start() and non_block_end()
* pairs.
*
* This is a useful debugging help to be able to catch problems early and not
* be bitten later when the calling function happens to sleep when it is not
* supposed to.
*/
#define might_sleep() do { __might_sleep(__FILE__, __LINE__); might_resched(); } while (0)
It’s essentially an assertion that crashes the kernel if a function is used in the wrong context. The filename and line number is used in the error message when it fails. Unfortunately, the __might_sleep function requires the filename to be a nul-terminated string.
Note that unlike with things like the file!() macro, it’s impossible for us to do this ourselves statically. Copying the filename at runtime into another string to nul-terminate it is also not a great solution because we need to create the string even if the assertion doesn’t fail, as the assertion is checked on the C side.
Solution sketch
Add a new function core::panic::Location::file_with_nul that returns a &CStr instead of a &str.
This has the implication that the compiler must now always store a nul-byte in the filename when generating the string constants.
Alternatives
It could make sense to return *const c_char instead of &CStr to avoid having to compute the length when all you need is a pointer you can pass into C code. This could be important as possible future work involves reducing the size of Location by removing the length. In this case, the existing core::panic::Location::file function would be updated to compute the length using the nul-terminator. Right now, the &CStr return value forces us to compute the length even when we don’t need it.
Links and related work
An implementation can be found at https://github.com/rust-lang/rust/pull/131828.
For more context, please see zulip and the Linux kernel mailing list. This is one of RfL’s wanted features in core.
Adding a nul-terminator to the Location string has been tried before in https://github.com/rust-lang/rust/pull/117431. However, back then, it was motivated by reducing the size of Location, and the previous PR did not actually expose the c string in the API.
What happens now?
This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.
Possible responses
The libs team may respond in various different ways. First, the team will consider the problem (this doesn’t require any concrete solution or alternatives to have been proposed):
- We think this problem seems worth solving, and the standard library might be the right place to solve it.
- We think that this probably doesn’t belong in the standard library.
Second, if there’s a concrete solution:
- We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
- We’re not sure this is the right solution, and the alternatives or other materials don’t give us enough information to be sure about that. Here are some questions we have that aren’t answered, or rough ideas about alternatives we’d want to see discussed.
cc @ojeda @Noratrieb
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 6
- Comments: 54 (37 by maintainers)
Commits related to this issue
- sched/core: Add __might_sleep_precision() Add __might_sleep_precision(), Rust friendly version of __might_sleep(), which takes a pointer to a string with the length instead of a null-terminated strin... — committed to intel-lab-lkp/linux by fujita 2 years ago
- sched/core: Add __might_sleep_precision() Add __might_sleep_precision(), Rust friendly version of __might_sleep(), which takes a pointer to a string with the length instead of a null-terminated strin... — committed to intel-lab-lkp/linux by fujita 2 years ago
- sched/core: Add __might_sleep_precision() Add __might_sleep_precision(), Rust friendly version of __might_sleep(), which takes a pointer to a string with the length instead of a null-terminated strin... — committed to intel-lab-lkp/linux by fujita 2 years ago
- sched/core: Add __might_sleep_precision() Add __might_sleep_precision(), Rust friendly version of __might_sleep(), which takes a pointer to a string with the length instead of a null-terminated strin... — committed to intel-lab-lkp/linux by fujita 2 years ago
- sched/core: Add __might_sleep_precision() Add __might_sleep_precision(), Rust friendly version of __might_sleep(), which takes a pointer to a string with the length instead of a null-terminated strin... — committed to intel-lab-lkp/linux by fujita 2 years ago
- sched/core: Add __might_sleep_precision() Add __might_sleep_precision(), Rust friendly version of __might_sleep(), which takes a pointer to a string with the length instead of a null-terminated strin... — committed to JCKeep/linux by fujita 2 years ago
- Auto merge of #135054 - cramertj:file-cstr, r=m-ou-se Add Location::file_with_nul This is useful for C/C++ APIs which expect the const char* returned from __FILE__ or std::source_location::file_name... — committed to rust-lang/rust by bors a year ago
- Auto merge of #135054 - cramertj:file-cstr, r=m-ou-se Add Location::file_with_nul This is useful for C/C++ APIs which expect the const char* returned from __FILE__ or std::source_location::file_name... — committed to rust-lang/rust by bors a year ago
- Auto merge of #135054 - cramertj:file-cstr, r=m-ou-se Add Location::file_with_nul This is useful for C/C++ APIs which expect the const char* returned from __FILE__ or std::source_location::file_name... — committed to model-checking/verify-rust-std by bors a year ago
I did a rustc-perf run on the relevant PR to measure the concrete binary size impact: https://github.com/rust-lang/rust/pull/135054#issuecomment-2578809459
It confirms what I suspected. While the binaries do get bigger of course, rustc-perfs rounding to one decimal place rounds it to 0.0%, making it very clear that it’s completely negligible.
Now there are people who really really care about binary size. But this one byte will definitely not push then over the edge. No, they won’t want any of this location information, they will use the unstable (or potentially stable in the future)
-Zlocation-detail=noneflag to remove all the information, which is what will actually satisfy their needs. We do not need to care about them here.Therefore, I really recommend to accept this ACP, at least if the upsides are considered useful (which I think they are), as the downsides of binary size can be ignored.
bonus: with a small sacrifice of speed on accesses, we could even improve binary size with this with the scheme described in https://github.com/rust-lang/rust/pull/117431. we don’t have to of course, and I don’t think that should be part of the discussion
if combined with the optimization in
Location’s size, it’s probably smaller to use nul-terminated strings, since each string is only needed for a whole file and only needs one more byte whereas the size field is duplicated for every tracked location and is either 4 or 8 extra bytes in each one.Using null terminated strings may also unlock linker string merging size optimizations, which could further decrease binary size. It seems unlikely to me that anyone cares about the tiny size increase - those who really really care about location size are gonna use something like -Zlocation-detail=none anyways, which deletes all this info.
Here’s the thing. We don’t truly need this for the kernel; it would be annoying to work around it, but we can change the C code and have it accept a length instead of a nul-terminated string.
However, I think it’s an unfortunate direction because adding Rust to existing C/C++ codebases is the future, and if we want Rust to work well in such codebases, we have to admit that they are not just going to give up their nul-terminated strings. I’m certain that this feature (or something equivalent) is not limited to only being useful in the kernel, and that any other codebase that does C/C++ interop will require this feature in order to have good error messages when something goes wrong when Rust calls into C/C++.
After all, when Rust is in the minority in a codebase, we want the experience of using Rust there to be good. Having the Rust side give worse error messages than you get in C/C++ is not a good look. But that is the current state in the kernel right now, and fixing it would be a lot easier with this feature.
I am not sure that there is in fact a “cost” here.
Every file path already has a de-facto terminator: it is suffixed by “.rs”, and this causes it to be “prefix-free”: https://en.wikipedia.org/wiki/Prefix_code
This is the same property possessed by NUL-terminated CStr. Each has “\0” at the end, which means no CStr can be a prefix of another CStr. Thus the argument about the cost seems wildly speculative, unless we wish to introduce a very curious new state of affairs, like not providing the “.rs” suffix!
Meanwhile, these file paths also could benefit from linker-driven deduplication (which revolves around the fact that CStrs can share a suffix).
Wow that’s extraordinarily silly. I don’t think we should engineer anything long-term around such an accident of history, and instead work with linkers to improve their deduplication to not be so C-centric.
The maximum binary size overhead measured in our benchmark suite is 0.04%, and most benchmarks reported the size change is below their significance threshold for binary size changes. https://github.com/rust-lang/rust/pull/132081#issuecomment-2434978157
The average binary size increase is 0.01%.
I think if people are going to claim that this feature has a cost to all users of Rust, they should be able to demonstrate that cost measured by our benchmark suite. I have previously measured the overhead of the
track_callerfeature in https://github.com/rust-lang/rust/pull/129704, so I’ve posted https://github.com/rust-lang/rust/pull/132081 to collect some measurements of the overhead of adding this null terminator. I’d like to have that number just so that people know what it is.There’s another way out of this, depending on the vibes people are willing to tolerate. For interop with C generally, we could add a compiler flag that null-terminates
track_callerstrings. Users of the null-terminated API would have to build everything with the flag, and would just saylocation.file().as_ptr(), and thestrlenfor that pointer would be doing https://github.com/rust-lang/unsafe-code-guidelines/issues/256.Of course this can’t happen on Linux, but has anyone thought about what happens when a user of this null-terminated API encounters an interior null byte?
Well, considering these are always full paths, the only coalescing that I think is possible here is merging together a module root with its children, so these two files:
I’d expect this to be rather low-yield, because it doesn’t work at all in projects that use
mod.rs, and only finds one merge per module.It seems unfortunate if all the
track_callerdata in the binary needs to be bigger for everyone just because some people want to pass it to a random C API sometimes.Could we have
file_cstr!()instead, so only people dealing in C strings need to deal with it? Yes, that’s not as nice astrack_caller, but oh well?But it becomes a problem if we later go through with the size optimization from https://github.com/rust-lang/rust/pull/117431. Then, the length is no longer known, so it really does have to be computed by calling
strlenor similar.To feel out whether this has a chance of going anywhere, I started a discussion about giving linkers the ability to dedup our strings without
\0https://inbox.sourceware.org/binutils/CALNs47sfhiiCPi4o=otZ4k3nEt=byB=hv3yEowLO5rKU8CKt+Q@mail.gmail.com/T/#u.@Amanieu
This isn’t correct-- I’ve been asking to use it for interop with other C++ APIs (absl) which require a null-terminated file API.
This kind of deduplication should work even without a trailing nul, right? Suffixes can be shared either way.
There is already a perf run for that in the PR linked above, which I can link again here: https://github.com/rust-lang/rust/pull/117431
main.rs
foo.js
Alas, I regret not picking the file extension
.workingjubileenow…