restic: MacOS Mojave: error: Open: ... operation not permitted

I recently started using Restic and have been trying to get it to work as a cron job called from the root crontab sudo crontab -e so that I could feel a little more secure having my backup script in a file only accessible with sudo privileges. I was getting the exact same errors as @n8henrie but now I have a working solution and would like to know if this works for others here.

First a little background of my setup:

I have a backup script in /Users/myuser/bin named restic-backup.sh with permissions 700 (only root/sudo read/write/execute). I execute this file with my root crontab sudo crontab -e. I use iTerm as my default terminal. I have installed restic and zsh with Homebrew.

macOS version 10.14.5

restic-backup.sh:

My backup script file has the following in it.

#!/usr/local/bin/zsh
restic_path="/usr/local/bin"
logFile="/Users/myuser/Documents/Backups/configurations/Mac/backup_logs/$(date +%F_%H%M)_restic.log"
unset HISTFILE
export RESTIC_REPOSITORY="..."
export AWS_ACCESS_KEY_I'd="..."
export AWS_SECRET_ACCESS_KEY="..."
export RESTIC_PASSWORD="..."
$restic_path/restic --verbose backup /Users/myuser  &> $logFile

Then in my root crontab file: sudo crontab -e I have:

0 */2 * * * /Users/myuser/bin/restic-backup.sh

In Full Disk Access:

cron ==> /usr/sbin/cron iTerm.app ==> /Applications/iTerm.app

Like @n8henrie, I thought that the programs actually accessing the files like restic would be the ones that require FDA but instead it seems that the programs making the initial sudo request need the FDA: cron in the automated case, and iTerm.app in the manual case.

I noted above that adding Terminal.app to the System Preferences -> Security & Privacy -> Full Disk Access list was a workaround, since when I manually ran my script (sudo /usr/local/bin/bash mybackup.sh) it seemed to work without the permissions errors.

For some reason, when I checked my logs this morning from the automated overnight restic run (based on a launchd script at /Library/LaunchDaemons/com.n8henrie.restic.plist which runs nightly with root privileges), the permissions errors are still there.

And unfortunately, I am now unable to get the workaround to work – even with Terminal.app in FDA, I still get the permissions errors when I run sudo launchctl start com.n8henrie.restic or sudo /usr/local/bin/bash mybackup.sh.

So the workaround doesn’t seem to work, or something has changed.

EDIT: Gah, just added all 3 to FDA – Terminal.app, /usr/local/bin/bash, and /usr/local/Cellar/restic/0.9.2/bin/restic – and now it ran without errors. 🤷‍♂️

FWIW, here is my log output, which contains the list of files getting errors. As you can see, there are some big concerns, like my photos library.

Unbelievable, this seems to work and is far cleaner and more simple.

// Runrestic provides a binary to run my restic backup script in MacOS Mojave with Full Disk Access
package main

import (
    "log"
    "os"
    "os/exec"
    "path/filepath"
)

func main() {
    ex, err := os.Executable()
    if err != nil {
        log.Fatal(err)
    }
    dir := filepath.Dir(ex)
    script := filepath.Join(dir, "restic-backup.sh")
    cmd := exec.Command("/usr/local/bin/bash", script)
    if err := cmd.Run(); err != nil {
        log.Fatal(err)
    }
}

The resulting binary can be chown root and chmod 0700, then add to Full Disk Access, and it seems to work. It can then be added to a /Library/LaunchDaemons plist to run automatically.

First 2 runs are working so far, I’m hopeful that this doesn’t end up like several false starts above.

• https://apple.stackexchange.com/a/343260/27415

Using a binary similar to https://github.com/restic/restic/issues/2051#issuecomment-442872479 is working for me. I went with c as I don’t have go installed right now. For others to copy/paste:

1. backup.c

#include <stdlib.h>
int main(void) {
  int status = system("./backup.sh");
  int ret = WEXITSTATUS(status);
  return ret;
}

1. compile: gcc -Wall -o backup backup.c
2. whitelist the backup binary and use it as you like

For anyone who wants to bundle restic into a .app bundle, here’s a somewhat generic tutorial I wrote a few months ago about how to do it for any Go program, including the code if you just want to change a few settings and be on your way: https://medium.com/@mattholt/packaging-a-go-application-for-macos-f7084b00f6b5

I’m a little surprised to see the issue closed already – this seems like a major incompatibility with a major platform, and closing it at this point may put it “out of sight, out of mind.”

I understand that this is not primarily a restic issue, but it seems like there are some reasonable steps that could be taken for MacOS users. As is, the ability of a user to back up much of their most important personal data (e.g. photos) is compromised by default on the current version of MacOS. That seems like a big deal to me, for any backup software!

Some suggestions / possibilities (that I’m happy to help work on):

• Possibilities for fixing the problem
  • Is it possible to provide a proper MacOS wrapper application which can get added to the FDA list, which contains the restic binary
  • If able to figure out how to do the above, instead of providing the application, could we include instructions for how users can accomplish it themselves with some sort of Makefile or XCode?
• Workarounds
  • Provide info in the docs regarding the most secure / minimal binaries that need to be added to FDA
  • Provide links to info regarding (and risks of) disabling SIP
• Warnings
  • Include info in the docs somewhere that Mojave users will not have their data fully backed up by default

Overall, the situation seems not terribly dissimilar to full backup without root on Linux, with disabling SIP altogether being something like running as root, and determining and recommending a reasonably secure compromise with FDA being something like using setcap.

Unfortunately my Macbook is in the shop this week so I won’t be able to work on any of these right away, but I’d be interested to hear thoughts from others; perhaps I’m way off base, or possibly ignorant of specifics of the restic team workflow for managing issues.

Curiously, I am still seeing, including a reboot:

go binary in FDA go binary not in FDA go binary without sudo err err go binary with sudo err err launchd running go binary RUNS err

Thanks!

The solution for me was to create .plist file that calls restic directly and put all the parameters either inside or into separate files, using -p, --exclude-files, --files-from options. And of course give restic binary FDA permissions:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>my.backup_agent</string>

	<key>ProgramArguments</key>
	<array>
		<string>/usr/local/bin/restic</string>
		<string>backup</string>

		<string>-r</string>
		<string>s3:https://MY.STORAGE.SERVER/....</string>

		<string>-p</string>
		<string>.config/backup/restic.pwd</string>

		<string>--files-from</string>
		<string>.config/backup/backup.lst</string>

		<string>--exclude-file</string>
		<string>.config/backup/exclude.lst</string>
	</array>

	<key>EnvironmentVariables</key>
	<dict>
		<key>AWS_ACCESS_KEY_ID</key>
		<string>XXX</string>

		<key>AWS_SECRET_ACCESS_KEY</key>
		<string>YYY</string>
	</dict>

	<key>WorkingDirectory</key>
	<string>/Users/ME</string>

	<key>StandardErrorPath</key>
	<string>/Users/ME/log/backup.log</string>

	<key>StandardOutPath</key>
	<string>/Users/ME/log/backup.log</string>

	<key>StartCalendarInterval</key>
	<dict>
		<key>Hour</key>
		<integer>13</integer>

		<key>Weekday</key>
		<array>
		<integer>1</integer>
		<integer>2</integer>
		<integer>3</integer>
		<integer>4</integer>
		<integer>5</integer>
		</array>
	</dict>
</dict>
</plist>

Yes, I run from launchd. Some interesting reads here

https://eclecticlight.co/2018/10/02/how-privacy-protection-is-enforced-through-the-attribution-chain/

https://eclecticlight.co/2018/09/06/working-with-mojaves-privacy-protection/

I too settled on wrapping my restic backup script in a .app bundle to be able to give it FDA. It sucks having to do this, but it seems to be the only practical way forward.

At first I tried giving just the restic binary FDA, but that did not make it work. I also tried giving my backup script (a Bash script that calls restic) FDA, but that didn’t work either. I did not try @armhold’s wrapper.

I used Platypus as suggested above, and it works great. It produces an .app bundle that I can then give FDA in the system settings of macOS, and this resolves the issue.

The only downside I’ve noticed during use is that when the .app starts (for me it’s started by crontab using open -ga ~/Applications/Backup.app, the current window loses focus. This can be a serious annoyance for my users, but it is what it is. At least we have working backups again. I thought the -g switch would take care of that, but it doesn’t change anything unfortunately.

I very briefly tried to see what happens when you delete the .app bundle (move it to Trash, empty trash) and replace it with a new .app bundle having the same name. My observations are that when you remove the old one, the entry in FDA in system preferences disappears, but when you put the new one back in the same place, the entry appears again, indicating that the system recognizes and would consider the new .app to have FDA. However, when I ran the new app after that, I got the original errors back. Once I removed the FDA entry for the app in system preferences and added a new FDA entry for it, the errors were gone again. So for now, I am presuming that when replacing the .app bundle, I need to replace the FDA entry as well, for it to work. It might very well be that if one just replace parts of the .app bundle, it will keep working though. This needs further investigaton AFAICT.

Still working on this.

It continues to be pretty painful, but I finally have a workaround that I think will work for my purposes.

I’ve detailed the process here, but the bottom line is:

You can use Script Editor.app to make an AppleScript into an application bundle. That AppleScript can run your restic backup script (in my case /path/to/restic-backup.sh, where restic-backup.sh is a bash script that runs restic with my desired settings), and you can add the resulting application bundle to FDA in order to get access to the protected files.

However, this makes it difficult to automate running backups on a system level. The built-in open command works, but runs the application under your user – so while the FDA protected files above are backed up, you’ll get permissions errors on any files that are only root readable (e.g. root-owned 0600 stuff).

My workaround at this point is to have the AppleScript call the backup script with sudo and give my user privileges to run that script with root privileges without a password (sudo visudo, NOPASSWD:, etc.).

It’s not pretty, but it seems to be working.

I’d like to leave this open for any suggestions from MacOS users with more expertise. If nothing more satisfying comes up, I could add this info to the docs (although this solution honestly doesn’t seem very satisfactory).

Platypus created .app didn’t work (wasn’t able to find restic).

You must have done something wrong then - I’m using a Platypus generated app on several macOS clients of different versions and the app works. Try again - add restic to the resources bundled in the app, and make the app use that restic 😃

Another way to address this if you’re running things via launchd: LaunchControl now ships with a helper called fdautil that you can whitelist, and then run commands using fdautil exec. It will only allow commands you’ve whitelisted via LaunchControl or fdautil set.

There’s a bit of info about it at https://www.soma-zone.com/LaunchControl/FAQ.html, and more details in the app’s help window.

@atticusmatticus @russelldavis – I think one of the recent MacOS updates may have changed something again – I had definitely tried both of those strategies with no luck. But I had also tried the below, which had stopped working, but is now working again (kind of):

package main

import (
	"fmt"
	"io/ioutil"
)

func main() {
	fmt.Println("Starting...")
	matches, err := ioutil.ReadDir("/Users/me/Library/Mail")
	if err != nil {
		fmt.Println("Err:", err)
	} else {
		for _, match := range matches {
			fmt.Println(match.Name())
		}
	}
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Label</key>
        <string>com.me.gotest</string>
        <key>ProgramArguments</key>
        <array>
            <string>/Users/me/go/src/github.com/me/gotest/gotest</string>
        </array>
        <key>StartInterval</key>
        <integer>15</integer>
        <key>StandardErrorPath</key>
        <string>/Users/me/go/src/github.com/me/gotest/stderr.txt</string>
        <key>StandardOutPath</key>
        <string>/Users/me/go/src/github.com/me/gotest/stdout.txt</string>
    </dict>
</plist>

1. go build
2. Add the gotest binary (and nothing else) to Full Disk Access
3. copy the plist to ~/Library/LaunchAgents/
4. Load the launchd daemon: launchctl load -w ~/Library/LaunchAgents/com.me.gotest.plist

Now, curiously, with the gotest binary, but not launchd (or launchctl) added to FDA, I still can’t run directly:

$ ls -l gotest
-rwxr-xr-x 1 me staff 2142552 Jul  2 09:15 gotest
$ ./gotest
Starting...
Err: open /Users/me/Library/Mail: operation not permitted
$ sudo ./gotest
Password:
Starting...
Err: open /Users/me/Library/Mail: operation not permitted

But it is running without errors from the launchd daemon (every 15 seconds as configured), which is not running as root (~/Library/ not /Library/, and launchctl load not sudo launchctl load):

$ cat stdout.txt | head
Starting...
.DS_Store
PersistenceInfo.plist
V6
Starting...
.DS_Store
PersistenceInfo.plist
V6
Starting...
.DS_Store

Ok, thanks for the feedback! I’ll reopen this issue, maybe we can figure it out what’s going on and then add some documentation to the manual.