firecracker: Firecracker panics cause a seccomp violation
Thanks @serban300 for pointing this out.
When Firecracker panic!s, it emits a blacklisted SYS_mremap which breaks the seccomp filter.
To repro, I added a dummy panic! in the api_server thread just after installing the seccomp filters:
diff --git a/api_server/src/lib.rs b/api_server/src/lib.rs
index afb8f42..5b43c7c 100644
--- a/api_server/src/lib.rs
+++ b/api_server/src/lib.rs
@@ -149,6 +149,8 @@ impl ApiServer {
);
}
+ panic!("Terminating");
+
// This runs forever, unless an error is returned somewhere within f (but nothing happens
target/x86_64-unknown-linux-musl/debug/firecracker --api-sock /tmp/a.sock
2019-05-10T11:13:12.182703303 [anonymous-instance:ERROR:src/main.rs:55] Firecracker panicked at 'Terminating', api_server/src/lib.rs:152:9
2019-05-10T11:13:12.307724144 [anonymous-instance:ERROR:vmm/src/sigsys_handler.rs:69] Shutting down VM after intercepting a bad syscall (25).
2019-05-10T11:13:12.307818289 [anonymous-instance:ERROR:vmm/src/sigsys_handler.rs:75] Failed to log metrics while stopping: Logger was not initialized.
The mremap originates in libbacktrace:
#0 __mremap (old_addr=0x7ffff7db9000, old_len=233472, new_len=new_len@entry=237568, flags=flags@entry=1) at src/mman/mremap.c:14
#1 0x0000000000a2784c in realloc (p=0x7ffff7db9020, n=234656) at src/malloc/malloc.c:397
#2 0x00000000009dca63 in __rbt_backtrace_vector_grow ()
#3 0x00000000009dddab in add_unit_addr ()
#4 0x00000000009de931 in add_unit_ranges ()
#5 0x00000000009decb6 in find_address_ranges ()
#6 0x00000000009df23d in build_address_map ()
#7 0x00000000009e2309 in build_dwarf_data ()
#8 0x00000000009e24da in __rbt_backtrace_dwarf_add ()
#9 0x00000000009dc3bf in elf_add ()
#10 0x00000000009dc7e5 in __rbt_backtrace_initialize ()
#11 0x00000000009d708a in fileline_initialize ()
#12 0x00000000009d7135 in __rbt_backtrace_pcinfo ()
#13 0x00000000009d56ce in backtrace::symbolize::libbacktrace::resolve::h746412056539717c (symaddr=0x9d23ac <backtrace::backtrace::trace_unsynchronized::h2c34a021aa0e95d7+44>, cb=...)
at $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.15/src/symbolize/libbacktrace.rs:171
#14 0x00000000009cd498 in backtrace::symbolize::resolve_unsynchronized::hb7eccc81fdd0039a (addr=0x9d23ac <backtrace::backtrace::trace_unsynchronized::h2c34a021aa0e95d7+44>, cb=...)
at $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.15/src/symbolize/mod.rs:62
#15 0x00000000009cd460 in backtrace::symbolize::resolve::h53981fa6789bc746 (addr=0x9d23ac <backtrace::backtrace::trace_unsynchronized::h2c34a021aa0e95d7+44>, cb=...)
at $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.15/src/symbolize/mod.rs:51
#16 0x00000000009c84de in backtrace::capture::Backtrace::resolve::h748b5356e9e44001 (self=0x7fffffff7db8)
at $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.15/src/capture.rs:146
#17 0x00000000009c8050 in backtrace::capture::Backtrace::new::h622dc69a5dec5128 ()
at $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.15/src/capture.rs:72
#18 0x0000000000400f84 in firecracker::main::_$u7b$$u7b$closure$u7d$$u7d$::hdc43da8746fa8ee1 (info=0x7fffffff80a8) at src/main.rs:59
#19 0x0000000000a135ea in std::panicking::rust_panic_with_hook::h28b9ce6fa7a5033b () at src/libstd/panicking.rs:495
#20 0x00000000006132b8 in std::panicking::begin_panic::hc75a354ae6cfa9ae (msg=..., file_line_col=0xd21b18) at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panicking.rs:425
#21 0x00000000004068d4 in api_server::ApiServer::bind_and_run::h4ab39ac0c86ac506 (self=0x7fffffffd8e0, path=..., start_time_us=..., start_time_cpu_us=..., seccomp_level=2)
at api_server/src/lib.rs:152
#22 0x0000000000403e5f in firecracker::main::h28360df3fa9d0378 () at src/main.rs:168
#23 0x0000000000402a10 in std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h3e2f7377df0eeb37 () at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/rt.rs:74
#24 0x0000000000a12f13 in std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::hbd394198e0f45efb () at src/libstd/rt.rs:59
#25 std::panicking::try::do_call::hf93a787b72e1d226 () at src/libstd/panicking.rs:310
#26 0x0000000000a1e059 in __rust_maybe_catch_panic () at src/libpanic_abort/lib.rs:39
#27 0x0000000000a138da in std::panicking::try::h9b83fe1076812e50 () at src/libstd/panicking.rs:289
#28 std::panic::catch_unwind::h8a94b67bdbd8163d () at src/libstd/panic.rs:398
#29 std::rt::lang_start_internal::h7b3bd8c78881c37d () at src/libstd/rt.rs:58
#30 0x00000000004029e9 in std::rt::lang_start::h0fdb6015f3270167 (main=0x4031e0 <firecracker::main::h28360df3fa9d0378>, argc=3, argv=0x7fffffffdd28)
at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/rt.rs:74
#31 0x00000000004042ca in main ()
The panic! terminates the process anyway, but the backtrace is lost because of the seccomp issue/
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (16 by maintainers)
Commits related to this issue
- syscalls: whitelist SYS_mremap Called by libbacktrace to make room for the stack trace. Related to, but does not fix #1088. Signed-off-by: Alexandra Iordache <aghecen@amazon.com> — committed to aghecenco/firecracker by aghecenco 5 years ago
- syscalls: whitelist SYS_mremap Called by libbacktrace to make room for the stack trace. Related to, but does not fix #1088. Signed-off-by: Alexandra Iordache <aghecen@amazon.com> — committed to aghecenco/firecracker by aghecenco 5 years ago
I like the idea, but it introduces a new artefact for users to collect. Firecracker already has 2 output channels in the logs and metrics destinations, and
libbacktraceallowed us to leverage one. While stack traces currently don’t work at all, getting them to work with this quirk of an extra requirement might not be desirable for all users - especially since they expect the stack trace to come through the logging channel.If we take a step back and look at the problem, our pain is with
backtrace-rswhich swallows a bunch of C sources and pulls in the extra dependency onccto compile them. Last time we had this problem, it was withlibseccompand we considered doing exactly that - gulp up the C sources, compile them and present them as a Rust crate. We ended up not doing that and instead rewriting in native Rust only what we needed fromlibseccomp. I think the best thing to do here is to repeat that - take what we need fromlibbacktrace, rewrite it, and profit 🍺Not as-is, but filtered via
/proc/self/coredump_filter. Actually, we could set that up to initially include only the stack (i.e. filter=0). Fromman core:How about dropping libbacktrace, together with the concept of logging a stack trace on panic? We could then just use a simple core dump, and rely on the user to configure
/proc/<PID>/coredump_filter.