bpftrace: bpftrace not able to attack kprobe with nf_nat with kernel 6.1.66-1

With latest debian kernel 6.1. bpftrace is not able to attach kprobe for nf_nat functions. It gives error - cannot attach kprobe, probe entry may not exist

What reproduces the bug? Provide code if possible.

bpftrace -e 'kprobe:nf_nat_ipv4_manip_pkt { printf("func called\n"); }'

/ # bpftrace -e 'kprobe:nf_nat_ipv4_manip_pkt { printf("func called\n"); }'
Attaching 1 probe...
cannot attach kprobe, probe entry may not exist
ERROR: Error attaching probe: 'kprobe:nf_nat_ipv4_manip_pkt'

Mentioned below is working and non working output: Not working :

/ # bpftrace --info
System
  OS: Linux 6.1.0-15-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.66-1 (2023-12-09)
  Arch: x86_64

Build
  version: v0.17.1
  LLVM: 16.0.3
  unsafe uprobe: no
  bfd: yes
  libdw (DWARF support): yes

Kernel helpers
  probe_read: yes
  probe_read_str: yes
  probe_read_user: yes
  probe_read_user_str: yes
  probe_read_kernel: yes
  probe_read_kernel_str: yes
  get_current_cgroup_id: yes
  send_signal: yes
  override_return: no
  get_boot_ns: yes
  dpath: yes
  skboutput: no

Kernel features
  Instruction limit: 1000000
  Loop support: yes
  btf: yes
  map batch: yes
  uprobe refcount (depends on Build:bcc bpf_attach_uprobe refcount): yes

Map types
  hash: yes
  percpu hash: yes
  array: yes
  percpu array: yes
  stack_trace: yes
  perf_event_array: yes

Probe types
  kprobe: yes
  tracepoint: yes
  perf_event: yes
  kfunc: yes
  iter:task: yes
  iter:task_file: yes
  kprobe_multi: no
  raw_tp_special: yes

$ sudo bpftrace -l | grep "manip"
kfunc:nf_nat:l4proto_manip_pkt
kfunc:nf_nat:nf_nat_ipv4_manip_pkt
kfunc:nf_nat:nf_nat_ipv6_manip_pkt
kfunc:nf_nat:nf_nat_manip_pkt
kprobe:l4proto_manip_pkt
kprobe:nf_nat_ipv4_manip_pkt
kprobe:nf_nat_ipv6_manip_pkt
kprobe:nf_nat_manip_pkt

/ # bpftrace -e 'kprobe:nf_nat_ipv4_manip_pkt { printf("func called\n"); }'
Attaching 1 probe...
cannot attach kprobe, probe entry may not exist
ERROR: Error attaching probe: 'kprobe:nf_nat_ipv4_manip_pkt'

/ # bpftrace -e 'kfunc:nf_nat:nf_nat_ipv4_manip_pkt { printf("func called\n"); }'
Attaching 1 probe...

func called
func called
func called
^C

/ #

Working:

/ # bpftrace --info
System
  OS: Linux 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29)
  Arch: x86_64

Build
  version: v0.17.1
  LLVM: 16.0.3
  unsafe uprobe: no
  bfd: yes
  libdw (DWARF support): yes

Kernel helpers
  probe_read: yes
  probe_read_str: yes
  probe_read_user: yes
  probe_read_user_str: yes
  probe_read_kernel: yes
  probe_read_kernel_str: yes
  get_current_cgroup_id: yes
  send_signal: yes
  override_return: no
  get_boot_ns: yes
  dpath: yes
  skboutput: no

Kernel features
  Instruction limit: 1000000
  Loop support: yes
  btf: yes
  map batch: yes
  uprobe refcount (depends on Build:bcc bpf_attach_uprobe refcount): yes

Map types
  hash: yes
  percpu hash: yes
  array: yes
  percpu array: yes
  stack_trace: yes
  perf_event_array: yes

Probe types
  kprobe: yes
  tracepoint: yes
  perf_event: yes
  kfunc: yes
  iter:task: yes
  iter:task_file: yes
  kprobe_multi: no
  raw_tp_special: yes

/ # bpftrace -l | grep "manip"
kprobe:l4proto_manip_pkt
kprobe:nf_nat_ipv4_manip_pkt
kprobe:nf_nat_ipv6_manip_pkt
kprobe:nf_nat_manip_pkt

/ # bpftrace --version
bpftrace v0.17.1
/ # bpftrace -e 'kprobe:nf_nat_ipv4_manip_pkt { printf("func called\n"); }'
Attaching 1 probe...
func called
func called

^C

/ # bpftrace -e 'kfunc:nf_nat:nf_nat_ipv4_manip_pkt { printf("func called\n"); }'
Attaching 1 probe...
func called
func called
func called

^C
/ #

About this issue

Original URL
State: open
Created 6 months ago
Comments: 17 (9 by maintainers)

Most upvoted comments

I use modprobe to load kernel modules: https://opensource.com/article/18/5/how-load-or-unload-linux-kernel-module I believe it has to do with kconfig settings that determines which ones get loaded by default.

Can i use tracepoints with bpf and use bpf maps?

Sure, but there might not be 1 to 1 matching of kprobe to tracepoint. You’ll probably have to dig around in the kernel code to see if there is something you can use.

jordalgo on Jan 4, 2024

@jordalgo here is output which is same in both. Debian 6.1.55-1

debian:~$ sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep "nf_nat_ipv4_manip_pkt"
nf_nat_ipv4_manip_pkt [nf_nat]
debian:~$ uname -a
Linux connector1 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
debian~$

Debian - 6.1.66-1

~$ sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep "nf_nat_ipv4_manip_pkt"
nf_nat_ipv4_manip_pkt [nf_nat]
~$ uname -a
Linux  6.1.0-15-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.66-1 (2023-12-09) x86_64 GNU/Linux
~$

pkopensrc on Jan 2, 2024

As attached It was working fine till 6.1.55 and broken recently. It it was inline it shouldn’t show in bpftrace -l.

Here is output of llvm-dwarfdump: root@pk-ntest-12new:/boot# llvm-dwarfdump-14 vmlinuz-6.1.0-15-cloud-amd64 | grep -A 5 DW_TAG_inlined_subroutine | grep -C 1 nf_nat_ipv4_manip_pkt root@pk-ntest-12new:/boot#

No changes in function. static bool nf_nat_ipv4_manip_pkt(struct sk_buff *skb, unsigned int iphdroff, const struct nf_conntrack_tuple *target, enum nf_nat_manip_type maniptype)

pkopensrc on Dec 25, 2023