go: runtime: netpoll port_getn() fails with impossible errno on illumos

Here is a script that will trace some of the port_getn() calls for a process:

#!/usr/sbin/dtrace -qs

pid$target::port_getn:entry
{
        self->x = timestamp;
        self->fd = args[0];
        self->ngetp = args[3];
        self->nget = *self->ngetp;

        printf("%u [%d/%d:%s] port_getn(%d, nget %p = %d)...\n",
            timestamp, pid, tid, execname,
            self->fd, self->ngetp, self->nget);
}

syscall::portfs:entry
/self->x/
{
        printf("%u [%d/%d:%s] SYSCALL portfs %d entry\n",
            timestamp, pid, tid, execname,
            (int)arg0);
}

syscall::portfs:return
/self->x/
{
        printf("%u [%d/%d:%s] SYSCALL portfs return %d (errno %d)\n",
            timestamp, pid, tid, execname,
            (int)arg0, errno);
}

pid$target::port_getn:return
/self->x/
{
        this->dur = (timestamp - self->x) / 1000;

        /*
         * Get the per-thread errno location for this thread:
         */
        this->fsbase = curthread->t_lwp->lwp_pcb.pcb_fsbase;
        this->ulwp = (userland struct pid`ulwp *)this->fsbase;
        this->errnop = this->ulwp->ul_errnop;

        printf("%u [%d/%d:%s] -> %d (errnop %p = %d, nget = %d) %dus\n",
            timestamp, pid, tid, execname,
            (int)arg1, this->errnop, *this->errnop, *self->ngetp, this->dur);

        if ((int)arg1 < 0 && *this->errnop == 0) {
                printf("WARNING: oddball return detected\n");
                ustack();
                printf("\n");
        }

        self->x = 0;
        self->fd = 0;
        self->ngetp = 0;
        self->nget = 0;
}

e.g.,

some output from an acme-dns process:

dns2 # /var/tmp/portgetn.d -p $(pgrep acme-dns)
13334996135948040 [597/6:acme-dns] port_getn(4, nget fffffc7feb3ff1c8 = 1)...
13334996135962747 [597/6:acme-dns] SYSCALL portfs 6 entry
13334996135968032 [597/6:acme-dns] SYSCALL portfs return 0 (errno 0)
13334996135973638 [597/6:acme-dns] -> 0 (errnop fffffc7feec8234c = 0, nget = 0) 25us

13334996135986941 [597/6:acme-dns] port_getn(4, nget fffffc7feb3ff1c8 = 1)...
13334996135991899 [597/6:acme-dns] SYSCALL portfs 6 entry
13335017554884999 [597/6:acme-dns] SYSCALL portfs return 1 (errno 0)
13335017554908883 [597/6:acme-dns] -> 0 (errnop fffffc7feec8234c = 0, nget = 1) 21418921us

13335017555435247 [597/6:acme-dns] port_getn(4, nget fffffc7feb3ff1c8 = 1)...
13335017555446540 [597/6:acme-dns] SYSCALL portfs 6 entry
13335017555451788 [597/6:acme-dns] SYSCALL portfs return 0 (errno 0)
13335017555456734 [597/6:acme-dns] -> 0 (errnop fffffc7feec8234c = 0, nget = 0) 21us

13335017555465954 [597/6:acme-dns] port_getn(4, nget fffffc7feb3ff1c8 = 1)...
13335017555470793 [597/6:acme-dns] SYSCALL portfs 6 entry

Critically this script looks at two things:

• the libc-level port_getn() call, including the per-thread errno value that Go should be collecting via errno()
• the underlying portfs system call made by port_getn() which is what truss is showing you as well, including the errno value that the kernel put in place for the system call return (this is separate but related to the libc errno value)

There is a conditional print if we detect a failure return (-1) where errno is 0, which should print something like:

WARNING: oddball return detected

              libc.so.1`port_getn+0x51
              acme-dns`runtime.netpoll+0x1ef
              acme-dns`runtime.startTheWorldWithSema+0x236
              acme-dns`runtime.gcMarkTermination.func3+0x26
              acme-dns`runtime.systemstack+0x73
              acme-dns`runtime.gcMarkDone+0x1ff
              acme-dns`runtime.gcBgMarkWorker+0x29d
              acme-dns`runtime.goexit+0x1

It would be good to capture this sort of trace output while the problem is occurring, so that we can determine whether the C library is mishandling the error numbers here, or if it is some part of the Go machinery for fetching thread local errno after a C library call, or something else.