libc: static binary crashing with NULL symbol

target: ppc64le-unknown-linux-gnu & stable-x86_64-unknown-linux-gnu

a statically compiled kata agent binary crashes in what looks to be a dynamic-library-related function. The binaries were built using the latest libc so that we could pick up the fix for statically compiling on non-x86 architectures (https://github.com/rust-lang/libc/pull/2046).

However, we are seeing this: dl-call-libc-early-init.c:37: _dl_call_libc_early_init: Assertion sym != NULL failed

The place @amulyam24 has narrowed it down to is possibly in this openpty call: https://github.com/kata-containers/kata-containers/blob/main/src/agent/rustjail/src/container.rs#L881

I was once stuck on an issue with go that involved ptys, and in that case TCSETS and TCGETS values were wrong for ppc64le. See https://github.com/golang/go/issues/19560. So I can’t help but be suspicious, but the dl_ part of this makes me think it’s probably not related.

Unfortunately, there’s not a lot of info in the logs, and we haven’t been able to find a small recreator. I’m opening this here in hopes that someone has some tips for getting more information.

kata-issue-1387-trace.out.txt

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 3
  • Comments: 21 (7 by maintainers)

Most upvoted comments

@clnperez I don’t know your exact requirements. If you need static binaries for isolation from the host environment, you need to stick to a certain subset if using glibc. It so happens that openpty is in that subset only starting with glibc 2.33. There is no good way to check whether an application sticks to the subset, as every static link currently pulls in the dynamic loader (so its presence in the static binary does not tell you anything about compatibility). We’re making slow progress towards improved static linking, but other things have higher priority for upstream work.

Using one of the smaller libcs instead might be a better alternative for you. The other alternative would be to inject the application along with its own dynamically-linked glibc.

Statically linked glibc binaries aren’t very portable. In general, you need to run them on a system with exactly the same glibc version. (Dynamically linked binaries can run on newer glibc versions, too.) In glibc 2.33, the behavior of openpty changed due to the commit, Linux: Require properly configured /dev/pts for PTYs, so openpty is safer to use in statically linked programs.

Thank you @Jakob-Naucke ! I reported this issue to glibc here: https://sourceware.org/bugzilla/show_bug.cgi?id=27790

I think it’s also noteworthy that when you build dynamically instead, you will get the error /lib64/libc.so.6: version `GLIBC_2.32’ not found

Running a program dynamically linked against a newer glibc on a system with an older glibc is unsupported because the older glibc can’t support all the features of the new one. IMHO, the real question is in the static case which does make usage of dynamic linking via dl_open(). I prefer to collect more details before giving you an answer.