cc-rs: make resource failure when cc parallel is enabled

Here is how to override default make with gmake on Github action runner on M1 which fixes build issue

- name: install make
        if: matrix.os == 'macos-14'
        run: |
          brew install make
          gmake --version
          mkdir -p $HOME/bin
          sudo ln -s /opt/homebrew/bin/gmake $HOME/bin/make
          echo "$HOME/bin/" >> $GITHUB_PATH

I have written a small rust-script, and confirmed that this is a macOS-only problem:

(The code below is a cargo-script, just paste it in a *.rs file, chmod +x it and then run it)

#!/usr/bin/env cargo +nightly -Zscript
```cargo
[dependencies]
os_pipe = "1.1.5"
nix = { version = "0.28.0", features = ["process"] }
libc = "0.2.153"
```

use std::{io::Error, thread::sleep, time::Duration};

fn get_flags(fd: &impl std::os::unix::io::AsRawFd) -> Result<i32, Error> {
    let flags = unsafe { libc::fcntl(fd.as_raw_fd(), libc::F_GETFL, 0) };
    if flags == -1 {
        Err(Error::last_os_error())
    } else {
        Ok(flags)
    }
}

fn set_flags(fd: std::os::unix::io::RawFd, flags: std::os::raw::c_int) -> Result<(), Error> {
    if unsafe { libc::fcntl(fd, libc::F_SETFL, flags) } == -1 {
        Err(Error::last_os_error())
    } else {
        Ok(())
    }
}

fn set_non_blocking(pipe: &impl std::os::unix::io::AsRawFd) -> Result<(), Error> {
    // On Unix, switch the pipe to non-blocking mode.
    // On Windows, we have a different way to be non-blocking.
    let fd = pipe.as_raw_fd();

    let flags = get_flags(pipe)?;
    set_flags(fd, flags | libc::O_NONBLOCK)
}

Fn set_blocking(pipe: &impl std::os::unix::io::AsRawFd) -> Result<(), Error> {
    // On Unix, switch the pipe to non-blocking mode.
    // On Windows, we have a different way to be non-blocking.
    let fd = pipe.as_raw_fd();

    let flags = get_flags(pipe)?;
    set_flags(fd, flags & (!libc::O_NONBLOCK))
}

fn is_blocking(pipe: &impl std::os::unix::io::AsRawFd) -> bool {
    (get_flags(pipe).unwrap() & libc::O_NONBLOCK) != 0
}

fn main() {
    let (read, write) = os_pipe::pipe().unwrap();

    println!(
        "read is blocking = {}, write is blocking = {}",
        is_blocking(&read),
        is_blocking(&write),
    );

    if unsafe { nix::unistd::fork().unwrap().is_child() } {
        sleep(Duration::from_secs(1));

        set_non_blocking(&read).unwrap();

        println!("set_non_blocking(read)");
        println!(
            "child read is blocking = {}, write is blocking = {}",
            is_blocking(&read),
            is_blocking(&write),
        );

        sleep(Duration::from_secs(2));

        set_blocking(&read).unwrap();

        println!("set_blocking(read)");
        println!(
            "child read is blocking = {}, write is blocking = {}",
            is_blocking(&read),
            is_blocking(&write),
        );
    }

    loop {
        println!(
            "read is blocking = {}, write is blocking = {}",
            is_blocking(&read),
            is_blocking(&write),
        );

        sleep(Duration::from_secs(1));
    }
}

The output of this program is:

read is blocking = false, write is blocking = false
read is blocking = false, write is blocking = false
set_non_blocking(read)
child read is blocking = true, write is blocking = false
read is blocking = true, write is blocking = false
read is blocking = true, write is blocking = false
set_blocking(read)
child read is blocking = false, write is blocking = false
read is blocking = false, write is blocking = false
read is blocking = false, write is blocking = false
read is blocking = false, write is blocking = false
read is blocking = false, write is blocking = false
read is blocking = false, write is blocking = false
read is blocking = false, write is blocking = false

which confirmed that setting blocking from children also affects its parent.

Users of my CLI have just also reported and I confirmed that I still see this error (the reports came from M1 users and I also have M1 Pro), even though I have cc v1.0.88 (only one) and openssl-sys v0.9.101. Here is the command to reproduce:

cargo install cargo-near

Thanks, confirmed that I can reproduce this on my M2

I’ve opened #985 to fix this, verified locally by building cargo-near that it indeed fixed this issue.

Can someone double-check it on their system/CI to see if it fixed the issue please?

BTW, your script has the same output on Linux as the one you pasted.

Working on a solution to not set O_NONBLOCK and instead use a thread on unix.

At that point, why keep the custom jobserver implementation? Wasn’t the main reason to avoid using a thread in the first place? Wouldn’t it make sense to just go back to using the jobserver crate?

Just piling up, this also happened on macOS when we upgraded Firefox to use cc 1.0.88.

Can you try #974 and see if there’s any line in the build.rs output like Failed to set read end of jobserver pipe back to blocking?

I patched my build using the commit from #974 and got the following output which is the same error as before and it doesn’t show the message you asked about:

[openssl-sys 0.9.101] running cd "/Users/erik/risc0/target/release/build/openssl-sys-ec116a4f5be84652/out/openssl-build/build/src" && MAKEFLAGS="-j --jobserver-fds=13,14 --jobserver-auth=13,14" "make" "build_libs"
[openssl-sys 0.9.101] perl "-I." "-Iutil/perl" "-Mconfigdata" "-MOpenSSL::paramnames" "util/dofile.pl" "-oMakefile" crypto/params_idx.c.in > crypto/params_idx.c
[openssl-sys 0.9.101] make: *** read jobs pipe: Resource temporarily unavailable.  Stop.
[openssl-sys 0.9.101] make: *** Waiting for unfinished jobs....
[openssl-sys 0.9.101] thread 'main' panicked at /Users/erik/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-src-300.2.3+3.2.1/src/lib.rs:611:9:
[openssl-sys 0.9.101] 
[openssl-sys 0.9.101] 
[openssl-sys 0.9.101] 
[openssl-sys 0.9.101] Error building OpenSSL:
[openssl-sys 0.9.101]     Command: cd "/Users/erik/risc0/target/release/build/openssl-sys-ec116a4f5be84652/out/openssl-build/build/src" && MAKEFLAGS="-j --jobserver-fds=13,14 --jobserver-auth=13,14" "make" "build_libs"
[openssl-sys 0.9.101]     Exit status: exit status: 2
[openssl-sys 0.9.101] 
[openssl-sys 0.9.101] 
[openssl-sys 0.9.101]     
[openssl-sys 0.9.101] note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

@n-eq which crate are you building?

I’d like to check its build.rs.

I’m working on a project I can’t share, but it’s using openssl and zstd. I’ll try to wrap up a minimal reproducible example asap

cc 1.0.87 has released, which shall fix this issue.

Note that since gnu make 4.3 they did switch to nonblocking pipes. Though due to an optimization in the linux kernel blocking pipe reads should actually perform better. jobserver-rs accepts both approaches.

Edit: Pipes, not sockets

At least on Linux you can open /dev/fd/$FD to open a separate FD pointing to the same underlying file.