bpftrace: Unable to perform more than 4096 instructions in BEGIN

CentOS 7.7, using the following repo to build/package/install Bpftrace:

NOTE: Forked from https://github.com/fbs/el7-bpf-specs

I am running into an issue with handling large BEGIN. To reproduce:

printf "BEGIN{\n%s\n}\n\nEND{clear(@a)}\n" "$(for n in {1..400};do printf "\t@a[$n] = 1;\n"; done)" | bpftrace /dev/stdin

Results in Bpftrace prematurely exiting with this error:

Error loading program: BEGIN (try -v)

Adding -v as suggested, lowering from {1..400} to an acceptable {1..372} so that it runs, and adding a grep for some registers does indeed point us at the issue:

$ printf "BEGIN{\n%s\n}\n\nEND{clear(@a)}\n" "$(for n in {1..372};do printf "\t@a[$n] = 1;\n"; done)" | bpftrace -v /dev/stdin | grep "(b7) r1 ="
Attaching END
1: (b7) r1 = 24896
3: (b7) r1 = 30002
5: (b7) r1 = 0
11: (b7) r1 = 2
22: (b7) r1 = 3
33: (b7) r1 = 4
44: (b7) r1 = 5
55: (b7) r1 = 6
66: (b7) r1 = 7
77: (b7) r1 = 8
88: (b7) r1 = 9
99: (b7) r1 = 10
110: (b7) r1 = 11
...
4026: (b7) r1 = 367
4037: (b7) r1 = 368
4048: (b7) r1 = 369
4059: (b7) r1 = 370
4070: (b7) r1 = 371
4081: (b7) r1 = 372

It appears to take 11 instructions for each @a[<N>] = 1 line and the 373rd attempt pushes us past some limit of 4096 instructions per function (BEGIN in this case).

Is there a way to increase the limit?

Ideally, for testing I would like to increase it (in a local compile of LLVM for example) to (65535 * 11 + 4096) for a total of 724,981 instructions allowed per-function. This would allow me to – at the very worse-case scenario – flag each/every PID in the system for tracing.

What I am doing is analyzing running jobs outside of Bpftrace for some condition and then stuffing @flag[pid] = integer; into the BEGIN block of Bpftrace to seed the running system state into an analyzer that continues with realtime events. I am usually just a few hundred instructions over the limit and even a small increase would be greatly helpful, but handling the worst-case scenario of every pid in the system matching the pre-trace condition would be nice.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 39 (15 by maintainers)

Most upvoted comments

nice hack 😃

I’m not entirely sure what you’re building here so it might not apply to your use case. But as a work around that doesn’t require patching the kernel you could consider using bpftool to write to the map, instead of using a BEGIN probe.

$ for i in {0..200}; do echo map update id 199 key $i 0 0 0 0 0 0 0  value 1 0 0 0 0 0 0 0; done | sudo bpftool batch file  /dev/stdin

$ sudo bpftrace  -e 'i:s:1 {@[pid]=1 }'
^C
@[39]: 1
@[173]: 1
@[90]: 1
@[82]: 1
@[177]: 1
@[174]: 1
@[195]: 1
@[83]: 1
@[99]: 1
@[101]: 1
@[1]: 1
@[167]: 1
@[12]: 1
@[96]: 1
@[48]: 1
@[7]: 1

fbs on Feb 7, 2020

Thanks so much! Here’s the patch that we applied to 3.10.0-1062.12.1 (latest kernel release on CentOS 7.7):

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index f628971988449..a445194b5fb6d 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -421,6 +421,7 @@ struct bpf_array {
 	};
 };
 
+#define BPF_COMPLEXITY_LIMIT_INSNS      1000000 /* yes. 1M insns */
 #define MAX_TAIL_CALL_CNT 32
 
 struct bpf_event_entry {
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index afca36f53c492..1d65e56594dbc 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1557,7 +1557,8 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr)
 	/* eBPF programs must be GPL compatible to use GPL-ed functions */
 	is_gpl = license_is_gpl_compatible(license);
 
-	if (attr->insn_cnt == 0 || attr->insn_cnt > BPF_MAXINSNS)
+	if (attr->insn_cnt == 0 ||
+	    attr->insn_cnt > (capable(CAP_SYS_ADMIN) ? BPF_COMPLEXITY_LIMIT_INSNS : BPF_MAXINSNS))
 		return -E2BIG;
 
 	if (type == BPF_PROG_TYPE_KPROBE &&
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 6dcfeb44bb8ef..b631e89e7a515 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -176,7 +176,6 @@ struct bpf_verifier_stack_elem {
 	struct bpf_verifier_stack_elem *next;
 };
 
-#define BPF_COMPLEXITY_LIMIT_INSNS	131072
 #define BPF_COMPLEXITY_LIMIT_STACK	1024
 
 #define BPF_MAP_PTR_UNPRIV	1UL

With this patch and only this patch we are able to get past our problem. The original test-program of:

printf "BEGIN{\n%s\n}\n\nEND{clear(@a)}\n" "$(for n in {1..400};do printf "\t@a[$n] = 1;\n"; done)" | bpftrace /dev/stdin

No longer has an issue starting. No recompile of bpftool, bcc, llvm, or bpftrace was required, just that kernel patch.

freebsdfrau on Feb 7, 2020