go: syscall: memory corruption when forking on OpenBSD, NetBSD, AIX, and Solaris

I managed to reproduce this on NetBSD with a C program!

forkstress.c:

#include <pthread.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <sys/wait.h>
#include <unistd.h>

void __attribute((noinline)) spin(uint64_t loops) {
  for (volatile uint64_t i = 0; i < loops; i++) {
  }
}

struct thing {
  uint64_t b;
  uint32_t c;  // Making this (plus 'sink' below) uint64_t may make repro take longer?
};

volatile struct thing* page;

volatile uint32_t sink;

int ready;

void* thread(void* arg) {
  __atomic_store_n(&ready, 1, __ATOMIC_SEQ_CST);

  while (1) {
    // Spin not strictly required, but it speeds up repro in my case.
    spin(40*1000);

    // Atomic not required, this works too:
    // page->c = sink;
    __atomic_store_n(&page->c, sink, __ATOMIC_SEQ_CST);
    sink++;
  }
}

int main(void) {
  page = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
  if (page == MAP_FAILED) {
    perror("mmap");
    return 1;
  }

  pthread_t thread_id;
  int ret = pthread_create(&thread_id, NULL, &thread, NULL);
  if (ret != 0) {
    perror("pthread_create");
    return 1;
  }

  // Wait for child thread to start.
  //
  // This is not required to repro, but eliminates racing fork+thread create as
  // a possibility.
  while (!__atomic_load_n(&ready, __ATOMIC_SEQ_CST)) {
  }

  int64_t i = 0;
  while (1) {
    i++;
    if (i % 10000 == 0) {
      printf("Loop %d...\n", i);
    }

    page->b = 102;
    // Does not work with libc fork(). libc fork() is significantly slower,
    // which may be the problem.
    uint64_t pid = syscall(SYS_fork);
    if (pid == 0) {
      /* Child */
      _exit(0);
    }

    /* Parent */
    /* spin(40*1000); may speed up repro. */
    page->b = 2;
    uint64_t pb = page->b;
    if (pb != 2) {
      printf("Corruption! pb, page->b = %lu, %lu\n", pb, page->b);
      _exit(1);
    }

    int status;
    ret = waitpid(pid, &status, 0);
    if (ret < 0) {
      perror("waitpid");
      return 1;
    }
    if (WEXITSTATUS(status) != 0) {
      printf("Bad child status %#x\n", status);
      return 1;
    }
  }
}

Build and run:

$ cc -pthread forkstress.c
$ ./a.out & ./a.out & ./a.out  & ./a.out
Loop 10000...
Corruption! pb, page->b = 102, 2
Loop 10000...
Loop 10000...
Loop 10000...
Corruption! pb, page->b = 102, 2
Loop 20000...
Loop 20000...
Corruption! pb, page->b = 102, 2

Notes:

• Anecdotally, running several instances at the same time seems to speed up repro more than a linear speedup (system under more load?). That said, a single instance will eventually fail.
• Sometimes I get several failures immediately, sometimes it takes >60s to get a failure.
• The most immediately interesting part here is that I call the fork syscall directly rather than using fork() from libc. I think this is simply because libc fork() is significantly slower to return than syscall(SYS_fork), and we seem to have a small race window.
  • I’ve looked through _malloc_prefork/postfork and (I believe) all of the registered atfork callbacks (here, here, and here), and none of them seem important, as neither thread is interacting with pthread or malloc.

The summarized behavior we see is:

1. page->b = 102
2. fork()
3. page->b = 2
4. Read page->b, observe 102 instead of 2.
5. When logging the corruption, we load page->b again, which typically observes 2 again.

All while another thread is spinning writing to page->c (unrelated word in the same page).

While debugging oxidecomputer/omicron#1146 I saw that this bug mentions Solaris and wondered if it might affect illumos as well, since the failure modes look the same for my issue. For the record, I don’t think my issue was caused by this one. I ran the Go and C test programs for several days without issue, and I ultimately root-caused my issue to illumos#15254. I mention this in case anyone in the future is wondering if illumos is affected by this. I don’t know whether Solaris (or any other system) has the same issue with preserving the %ymm registers across signal handlers, but that can clearly cause the same failure modes shown here.

OpenBSD applied the following change:

https://github.com/openbsd/src/commit/43687ba57c7d88063c6fa2df2386adbd1a0cf241

Based on a cursory skim (without having thought much about the details of the change), I suspect that this change is designed to fix another copy-on-write bug just like https://reviews.freebsd.org/D14347 and like I described in https://github.com/golang/go/issues/34988#issuecomment-997000571.

However, I have no theory for how the FreeBSD fix or the NetBSD fix could affect the problem we detected here, because that copy-on-write bug – and the fix in FreeBSD and NetBSD – makes sense only if the TLB IPI handler runs between the store and load of the memory location at issue, whereas our measurements on NetBSD indicate that that’s physically implausible for the issue in this thread.

In NetBSD, we have not yet committed a fix for that COW bug because our draft fixes (like the one in https://github.com/golang/go/issues/34988#issuecomment-997000571) seem to have the side effect of suppressing the issue in this thread. I’m still hoping to hear from AMD with an idea about what could be going wrong in the 10ns window we observed but they haven’t gotten back to me yet. However, if we don’t hear anything before NetBSD 10.0 is ready we’ll probably just apply the unrelated-COW fix.

I tried the linked C program on aix with a couple modifications. I had to modify it to use fork, I am not sure if aix can fork without going through libc, so the test might not really indicate anything in this case. It did not reproduce anything after running for 5 or so minutes.

FWIW, the NetBSD issue I filed is at http://gnats.netbsd.org/56535.

@bokunodev the issue you are describing is not related to this bug. You perhaps accidentally replied to this bug, or need to file a new issue.

spoke too soon:

package main

import "os/exec"

func main() {
	loop := func() {
		for {
			err := exec.Command("/usr/bin/true").Run()
			if err != nil {
				panic(err)
			}
		}
	}
	for i := 0; i < 100; i++ {
		go loop()
	}
	select {}
}

https://gist.githubusercontent.com/jrick/a071767cde2d2d71b210135cf8282b04/raw/6fcd814e5a93a6a1d204c2d00b0a1f4195664d61/gistfile1.txt