bpftrace: Accessing pointers broken on LLVM <12

bpftrace --info

System
  OS: Linux 5.6.4-arch1-1 #1 SMP PREEMPT Mon, 13 Apr 2020 12:21:19 +0000
  Arch: x86_64

Build
  version: v0.10.0-63-g4b37
  LLVM: 10
  foreach_sym: yes
  unsafe uprobe: no
  btf: no
  bfd: yes
  bpf_attach_kfunc: yes

Kernel helpers
  probe_read: yes
  probe_read_str: yes
  probe_read_user: yes
  probe_read_user_str: yes
  probe_read_kernel: yes
  probe_read_kernel_str: yes
  get_current_cgroup_id: yes
  send_signal: yes
  override_return: yes

Kernel features
  Instruction limit: 1000000
  Loop support: yes

Map types
  hash: yes
  percpu hash: yes
  array: yes
  percpu array: yes
  stack_trace: yes
  perf_event_array: yes

Probe types
  kprobe: yes
  tracepoint: yes
  perf_event: yes
  kfunc: no

What reproduces the bug?

This works:

#include <linux/fs.h>
#include <linux/blk_types.h>
#include <linux/blkdev.h>
#include <linux/blk-cgroup.h>
#include <linux/cgroup-defs.h>
#include <linux/sched.h>

kprobe:blk_update_request
{
        $req = (struct request *)arg0;
        printf("cgroup name: %p\n", $req->bio->bi_blkg->blkcg->css.cgroup->kn->parent);
}

with output like this:

cgroup name: 0xffffa02fdd173480
cgroup name: 0xffffa02fdd173480
cgroup name: (nil)
cgroup name: (nil)

But this script does not work:

#include <linux/fs.h>
#include <linux/blk_types.h>
#include <linux/blkdev.h>
#include <linux/blk-cgroup.h>
#include <linux/cgroup-defs.h>
#include <linux/sched.h>

kprobe:blk_update_request
{
        $req = (struct request *)arg0;
        $kn = $req->bio->bi_blkg->blkcg->css.cgroup->kn;
        printf("cgroup name: %p\n", $kn->parent);
}

with output like:

cgroup name: (nil)
cgroup name: (nil)
cgroup name: (nil)
cgroup name: (nil)
cgroup name: (nil)
cgroup name: (nil)
cgroup name: (nil)
cgroup name: (nil)
cgroup name: (nil)
cgroup name: (nil)

There are no non-nils.

I think there might have been some change w/ handling pointers already on the stack. Maybe it’s eliding a bpf_probe_read()? Could make sense cuz then kernel will fail the dereference and give you 0x0.

cc @mmisono

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 19 (8 by maintainers)

Most upvoted comments

I filed a bug against llvm upstream: https://bugs.llvm.org/show_bug.cgi?id=47591

From https://llvm.org/docs/LangRef.html, llvm.lifetime.{start,end} encloses a region of lifetime of a memory object. From the above IR, I see

  %"struct cgroup.kn" = alloca i64, align 8
...
  %15 = bitcast i64* %"struct cgroup.kn" to i8*
  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %15)
  %probe_read4 = call i64 inttoptr (i64 4 to i64 (i64*, i32, i64)*)(i64* nonnull %"struct cgroup.kn", i32 8, i64 %14)
  %16 = load i64, i64* %"struct cgroup.kn", align 8
  call void @llvm.lifetime.end.p0i8(i64 -1, i8* nonnull %15)
  %17 = bitcast %printf_t* %printf_args to i8*
  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %17)
  %18 = add i64 %16, 8
  %19 = bitcast i64* %"struct kernfs_node.parent" to i8*
  %20 = getelementptr inbounds %printf_t, %printf_t* %printf_args, i64 0, i32 0
  store i64 0, i64* %20, align 8
  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %19)
  %probe_read5 = call i64 inttoptr (i64 4 to i64 (i64*, i32, i64)*)(i64* nonnull %"struct kernfs_node.parent", i32 8, i64 %18)

After call void @llvm.lifetime.end.p0i8(i64 -1, i8* nonnull %15), the compiler is free to reuse the same stack location, and based on the above, the compiler is supposed to do load first and then reuse the stack. This is exactly llvm7 asm code is doing.

Need to do a little bit study on why llvm did the above reordering. I guess this might be a llvm bug. One comment mentioned it is after BPF DAG->DAG Pattern Instruction Selection. Let me take a further check and then will comment back.

Cannot reproduce this one and #1558 with the llvm12 build from #1716 \o/

The bug is fixed in llvm12. This LLVM patch https://reviews.llvm.org/D91833 fixed the problem and I added a test case (derived from this issue) in BPF https://reviews.llvm.org/D92451.