WSL: [WSL1] Cannot debug anything after upgrading to 22.04

Version

Microsoft Windows [Version 10.0.19044.1526]

WSL Version

  • WSL 2
  • WSL 1

Kernel Version

Linux version 4.4.0-19041-Microsoft (Microsoft@Microsoft.com) (gcc version 5.4.0 (GCC) ) #1237-Microsoft Sat Sep 11 14:32:00 PST 2021

Distro Version

Linux version 4.4.0-19041-Microsoft (Microsoft@Microsoft.com) (gcc version 5.4.0 (GCC) ) #1237-Microsoft Sat Sep 11 14:32:00 PST 2021

Other Software

GNU gdb (Ubuntu 11.1-0ubuntu2) 11.1 gcc version 11.2.0 (Ubuntu 11.2.0-7ubuntu2)

Repro Steps

Consider the following C file:

#include <stdio.h>

int main(void)
{
    printf("Hello World!\n");
    return 0;
}
  • build the program with proper flags
  • debug it with gdb, and try to insert breakpoints

Trying to debug it results in the following;

image

Trying to debug the same program makes LLDB hang.

Expected Behavior

GDB and LLDB should have worked

Actual Behavior

GDB doesn’t work (see strace), LLDB hangs.

Diagnostic Logs

strace_gdb_nofollowfork.txt

(did not follow fork - caused program startup failure)

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 12
  • Comments: 23 (2 by maintainers)

Most upvoted comments

Are you enabled jammy-proposed in sources.list ?

Is it cause by gdb version different?

Yes. Binary patch is really hack, so you have to find the correct offset for each different binary . for GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1, the new patch offset is 0x335C7D, and the one-line patch command is echo -en '\x90\x90' | sudo dd of=/usr/bin/gdb count=2 bs=1 conv=notrunc seek=$((0x335C7D))

If so how to construct the generic command for temp fix?

I suppose there is no generic command, but there is generic method, as shown below.

Download gdb_12.0.90-0ubuntu1_amd64.deb and its debug info gdb-dbgsym_12.1-0ubuntu1~22.04_amd64.ddeb, then do binary reversing engineering (with a decompiler).

Find linux_nat_target::xfer_partial function (its location is different from each binary), see decompiled pseudo code here:

unsigned int __fastcall linux_nat_target::xfer_partial(
        linux_nat_target *const this,
        target_object object,
        const char *annex,
        unsigned __int64 readbuf,
        unsigned __int64 writebuf,
        ULONGEST offset,
        ULONGEST len,
        ULONGEST *xfered_len)
{
...
  switch ( object )
  {
    case TARGET_OBJECT_SIGNAL_INFO:
...
    case TARGET_OBJECT_MEMORY:
// HERE: when gdb try to access memory, it will enter this case
      if ( inferior_ptid.m_pid == null_ptid.m_pid
        && inferior_ptid.m_lwp == null_ptid.m_lwp
        && inferior_ptid.m_tid == null_ptid.m_tid )
      {
        return 0;
      }
      v13 = target_gdbarch();
      v14 = gdbarch_addr_bit(v13);
      if ( (unsigned __int64)v14 <= 0x3F )
        v9 &= ~(-1LL << v14);
      v15 = proc_mem_file_map._M_h._M_buckets[inferior_ptid.m_pid % proc_mem_file_map._M_h._M_bucket_count];
      if ( v15 )
      {
...
// HRER: after pread64 or pwrite64 on /proc/<pid>/mem, if no error, will return 1
                return 1;
...
      }      
// HERE: but if /proc/<pid>/mem is not valid, it just return 0 and fails later
      return 0;
    case TARGET_OBJECT_AUXV:
...
  }
  if ( object != TARGET_OBJECT_OSDATA )
// HERE: however, ptrace is also available, so the hack patch lets code jump here even if it match case TARGET_OBJECT_MEMORY
    return inf_ptrace_target::xfer_partial(
             this,
             object,
             annex,
             (gdb_byte *)readbuf,
             (const gdb_byte *)writebuf,
             offset,
             len,
             xfered_len);
  v33 = linux_common_xfer_osdata(annex, (gdb_byte *)readbuf, offset, len);
  *xfered_len = v33;
  return v33 != 0;
}

Have a look at the disassembly code:

.text:0000000000335C71                 cmp     esi, 0Fh
.text:0000000000335C74                 jz      loc_335DD0
.text:0000000000335C7A                 cmp     esi, 1    ; case TARGET_OBJECT_MEMORY
.text:0000000000335C7D                 jz      short loc_335CD8      ; HERE: patch it to `nop`, so it will continue execute and reach 0x335CCC, where is our target
.text:0000000000335C7F                 mov     r10, rdx
.text:0000000000335C82                 cmp     esi, 6
.text:0000000000335C85                 jz      loc_335E30
.text:0000000000335C8B                 cmp     esi, 0Eh
.text:0000000000335C8E                 jz      loc_335FC8
.text:0000000000335C94                 mov     rax, [rsp+168h+var_40]
.text:0000000000335C9C                 sub     rax, fs:28h
.text:0000000000335CA5                 jnz     loc_33620B
.text:0000000000335CAB                 mov     [rsp+168h+xfered_len], rbx
.text:0000000000335CB3                 mov     [rsp+168h+len], r14
.text:0000000000335CBB                 add     rsp, 138h
.text:0000000000335CC2                 pop     rbx
.text:0000000000335CC3                 pop     rbp
.text:0000000000335CC4                 pop     r12
.text:0000000000335CC6                 pop     r13
.text:0000000000335CC8                 pop     r14
.text:0000000000335CCA                 pop     r15
.text:0000000000335CCC                 jmp     _ZN17inf_ptrace_target12xfer_partialE13target_objectPKcPhPKhmmPm ; inf_ptrace_target::xfer_partial(target_object,char const*,uchar *,uchar const*,ulong,ulong,ulong *)

The hack method is forcing code to jump to inf_ptrace_target::xfer_partial even if the object matchs case TARGET_OBJECT_MEMORY, and it just needs to patch the two-byte instruction jz short loc_335CD8 at 0x335C7D to nop ; nop (aka “\x90\x90”).

For gdb 12.0.90-0ubuntu1 from Ubuntu 22.04 (download link), a temporary solution is patching two bytes in /usr/bin/gdb which offset at 0x335bad from \x74\x59 to \x90\x90 (or simply use this command: echo -ne '\x90\x90' | sudo dd of=/usr/bin/gdb seek=$((0x335bad)) bs=1 count=2 conv=notrunc), this will re-enable the ptrace fallback and the patched binary should work well.

However, I hope this issue keeps open, until WSL1 add support of /proc/<PID>/mem feature. The file is really important in Linux /proc Filesystem, but it is still not solved after 6 years since year 2016 when #547 opens !

I installed GDB from Ubuntu Support Team PPA (add that repo then sudo apt update && sudo apt dist-upgrade) and the error doesn’t happen anymore.

The real reason for this issue is that WSL1 lacks /proc/*/mem files (as well as /proc/*/task/*/mem files).

A tracer can read memory of tracee by either reading /proc/<PID>/mem file or using ptrace(PTRACE_PEEKTEXT, ...) / ptrace(PTRACE_PEEKDATA, ...) syscall, and old version gdb will fallback to use ptrace when it fails to read /proc/<PID>/mem.

However, in this commit on 1 Jul 2021, palves says:

So the
next step is to keep the /proc/<pid>/mem file open across memory
accesses.  If we have this, then it doesn't make sense anymore to even
have the ptrace fallback, so the patch disables it.

Gdb disables such fallback and always assumes /proc/<PID>/mem is available, but WSL1 does not has this file, so gdb on Ubuntu 22.04 cannot access memory under WSL1.

Related issue: #547

For now this issue is on our backlog. The reason for this is that on the team we’re prioritizing work items for WSL 2, since the majority of our users and new features are on WSL 2 distros. We are still supporting WSL 1 for very high impact breaking bugs (security, etc.), but as there is a work around for this it doesn’t currently meet our bar for fixing.

If you’re hitting this issue I’d recommend trying to upgrade to WSL 2. I understand there are still some blockers for folks (Networking, cross-os file performance etc.), but we have made many improvements to it recently to address these (Like our recent networking and memory management improvements). And if you’re still unable to upgrade I’d love to better understand why to help us prioritize work in the future. Thank you!

@craigloewen-msft Notice this is also an issue in WSL 2. I have it, and a few others above pointed out it also happens to them.

Im using wsl2 right now with the same issue, this is really annoying and tricky, I tried to use older-version of gdb(12.0->11.2), but the same problem exists, I truly hope wsl will sooner or later fix this problem but Im not very positive about this, as far as I know this is not a new problem. Maybe using virtual machine will be end of journey.

Update : This problem is fundamentally due to the newer version of gdb only support /proc/PID/mem while the older version supports ptrace as well, so if you really need gdb, I highly recommend you roll back your system version, for example from Ubuntu 22.04LTS to Ubuntu 20.04LTS, Im using Arch and roll back my system version doesn’t work while for Ubuntu it is useful, so the problem I guess is somehow related to the system as well, Im just confused

Seeing the same problems as everyone else for GDB 12.1 on WSL1 Ubuntu 22.04 LTS.

I didn’t want to patch my GDB using dd so I ended up just downloading the GDB 13.2 tarball and running the good old

./configure && make && make install

Of course, the downside is that apt is not aware of this GDB, which lives in /usr/local/bin for me, but it works fine.

This worked like a charm for me.

Are you enabled jammy-proposed in sources.list ?

Is it cause by gdb version different?

Yes. Binary patch is really hack, so you have to find the correct offset for each different binary . for GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1, the new patch offset is 0x335C7D, and the one-line patch command is echo -en '\x90\x90' | sudo dd of=/usr/bin/gdb count=2 bs=1 conv=notrunc seek=$((0x335C7D))

If so how to construct the generic command for temp fix?

I suppose there is no generic command, but there is generic method, as shown below.

Download gdb_12.0.90-0ubuntu1_amd64.deb and its debug info gdb-dbgsym_12.1-0ubuntu1~22.04_amd64.ddeb, then do binary reversing engineering (with a decompiler).

Find linux_nat_target::xfer_partial function (its location is different from each binary), see decompiled pseudo code here:

unsigned int __fastcall linux_nat_target::xfer_partial(
        linux_nat_target *const this,
        target_object object,
        const char *annex,
        unsigned __int64 readbuf,
        unsigned __int64 writebuf,
        ULONGEST offset,
        ULONGEST len,
        ULONGEST *xfered_len)
{
...
  switch ( object )
  {
    case TARGET_OBJECT_SIGNAL_INFO:
...
    case TARGET_OBJECT_MEMORY:
// HERE: when gdb try to access memory, it will enter this case
      if ( inferior_ptid.m_pid == null_ptid.m_pid
        && inferior_ptid.m_lwp == null_ptid.m_lwp
        && inferior_ptid.m_tid == null_ptid.m_tid )
      {
        return 0;
      }
      v13 = target_gdbarch();
      v14 = gdbarch_addr_bit(v13);
      if ( (unsigned __int64)v14 <= 0x3F )
        v9 &= ~(-1LL << v14);
      v15 = proc_mem_file_map._M_h._M_buckets[inferior_ptid.m_pid % proc_mem_file_map._M_h._M_bucket_count];
      if ( v15 )
      {
...
// HRER: after pread64 or pwrite64 on /proc/<pid>/mem, if no error, will return 1
                return 1;
...
      }      
// HERE: but if /proc/<pid>/mem is not valid, it just return 0 and fails later
      return 0;
    case TARGET_OBJECT_AUXV:
...
  }
  if ( object != TARGET_OBJECT_OSDATA )
// HERE: however, ptrace is also available, so the hack patch lets code jump here even if it match case TARGET_OBJECT_MEMORY
    return inf_ptrace_target::xfer_partial(
             this,
             object,
             annex,
             (gdb_byte *)readbuf,
             (const gdb_byte *)writebuf,
             offset,
             len,
             xfered_len);
  v33 = linux_common_xfer_osdata(annex, (gdb_byte *)readbuf, offset, len);
  *xfered_len = v33;
  return v33 != 0;
}

Have a look at the disassembly code:

.text:0000000000335C71                 cmp     esi, 0Fh
.text:0000000000335C74                 jz      loc_335DD0
.text:0000000000335C7A                 cmp     esi, 1    ; case TARGET_OBJECT_MEMORY
.text:0000000000335C7D                 jz      short loc_335CD8      ; HERE: patch it to `nop`, so it will continue execute and reach 0x335CCC, where is our target
.text:0000000000335C7F                 mov     r10, rdx
.text:0000000000335C82                 cmp     esi, 6
.text:0000000000335C85                 jz      loc_335E30
.text:0000000000335C8B                 cmp     esi, 0Eh
.text:0000000000335C8E                 jz      loc_335FC8
.text:0000000000335C94                 mov     rax, [rsp+168h+var_40]
.text:0000000000335C9C                 sub     rax, fs:28h
.text:0000000000335CA5                 jnz     loc_33620B
.text:0000000000335CAB                 mov     [rsp+168h+xfered_len], rbx
.text:0000000000335CB3                 mov     [rsp+168h+len], r14
.text:0000000000335CBB                 add     rsp, 138h
.text:0000000000335CC2                 pop     rbx
.text:0000000000335CC3                 pop     rbp
.text:0000000000335CC4                 pop     r12
.text:0000000000335CC6                 pop     r13
.text:0000000000335CC8                 pop     r14
.text:0000000000335CCA                 pop     r15
.text:0000000000335CCC                 jmp     _ZN17inf_ptrace_target12xfer_partialE13target_objectPKcPhPKhmmPm ; inf_ptrace_target::xfer_partial(target_object,char const*,uchar *,uchar const*,ulong,ulong,ulong *)

The hack method is forcing code to jump to inf_ptrace_target::xfer_partial even if the object matchs case TARGET_OBJECT_MEMORY, and it just needs to patch the two-byte instruction jz short loc_335CD8 at 0x335C7D to nop ; nop (aka “\x90\x90”).

It works! Thank you so much.

For gdb 12.0.90-0ubuntu1 from Ubuntu 22.04 (download link), a temporary solution is patching two bytes in /usr/bin/gdb which offset at 0x335bad from \x74\x59 to \x90\x90 (or simply use this command: echo -ne '\x90\x90' | sudo dd of=/usr/bin/gdb seek=$((0 x335bad)) bs=1 count=2), this will re-enable the ptrace fallback and the patched binary should work well. However, I hope this issue keeps open, until WSL1 add support of /proc/<PID>/mem feature. The file is really important in Linux /proc Filesystem, but it is still not solved after 6 years since year 2016 when #547 opens !

after I run this command, the gdb binary file is damaged. -bash: /usr/bin/gdb: cannot execute binary file: Exec format error

in:

  • wsl1
  • ubuntu 22.04

so what is problem? anybody knows? thanks!

Sorry for that. I miss the conv=notrunc argument for dd command. The correct command should be echo -ne '\x90\x90' | sudo dd of=/usr/bin/gdb seek=$((0x335bad)) bs=1 count=2 conv=notrunc. (Before this, you should re-install gdb to recover the original /usr/bin/gdb file (e.g. by sudo apt-get install --reinstall gdb) )

Ran into this yesterday, agree this is pretty horrible, ended up using a different workaround that worked for my limited needs. Took a dump in an old debugger that couldn’t read the symbols, opened the dump in a new debugger.

Unfortunately we’re short on people. @craigloewen-msft for visibility

Seeing the same problems as everyone else for GDB 12.1 on WSL1 Ubuntu 22.04 LTS.

I didn’t want to patch my GDB using dd so I ended up just downloading the GDB 13.2 tarball and running the good old

./configure && make && make install

Of course, the downside is that apt is not aware of this GDB, which lives in /usr/local/bin for me, but it works fine.