zrepl: Placeholders break encryption key inheritance of root_fs, unencrypted data is leaked

Hello, I have the following setup and use case:

  • Source: A server at a secure-ish location w/o encryption
  • Pull destination: A NAS at a shared location with native encryption enabled at pool/backup.
  • The source’s tank/a and tank/a/b/c are replicated, but tank/a/b is not. Zrepl will create a placeholder for it.

Expected behavior: All files under pool/backup are encrypted and the received filesystems inherit pool/backup’s keys.

Actual behavior: Zrepl explicitly disabled encryption for tank/a/b, causing unencrypted data to be written/leaked onto the destination’s hardware. Eek!

Indeed it is impossible to have tank/a/b/c encrypted because the inheritance chain is broken by the placeholder. Setting encrypting properties explicitly can work, but is clunky at best. Setting encryption=aes-256-gcm in properties.inherit at least makes the whole thing fail instead of leaking unencrypted data.

Analysis and Source-Level Fix

I found that this behavior was intentionally introduced in commit 8ff83f2f1ac6faf364627f14c62880f35af4835e to serve issue #342. IMO, that behavior is dangerous. Please never disable my encryption automatically 😦.

Removing lines 92-96 in zfs/placeholder.go resolves the issue: The placeholder is created with default properties, inherits its encryption, and the plain send stream is re-encrypted like expected.

Recommendation to fix:

Make placeholder properties configurable 😃. Afterwards, drop the hardcoded and dangerous encryption=off completely.

If the user expects an encrypted send stream, then the user is free to set e.g. placeholders.properties = {encryption: off} in their job. This info would nicely fit into existing docs regarding encrypted sending. Otherwise, placeholders simply inherit their properties and are thus free of surprises.

Bonus: canmount and similar properties become configurable with placeholders for free.

Configs

Source:

  - name: foo
    type: source
    serve:
      type: tcp
      listen: 10.42.1.1:8888
      clients:
        10.42.1.32: example
    filesystems:
      "tank/a<": true
      "tank/a/b": false
      "tank/a/b/c<": true
    snapshotting:
      type: periodic
      interval: 15m
      prefix: zrepl_

Pull:

jobs:
  - name: bar
    type: pull
    root_fs: pool/backup
    recv:
      properties:
        override:
          canmount: noauto
          mountpoint: none
        inherit:
          - keylocation
          - keyformat
          - encryption
    connect:
      type: tcp
      address: 10.42.1.1:8888
    interval: 15m
    pruning:
      keep_sender:
        - type: not_replicated
        - type: last_n
          count: 10
        - type: grid
          grid: 1x1h(keep=all) | 24x1h | 30x1d
          regex: "zrepl_.*"
        - type: regex
          regex: "^manual_.*"
      keep_receiver:
        - type: grid
          grid: 1x1h(keep=all) | 24x1h | 30x1d | 3x30d
          regex: "zrepl_.*"

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 21 (13 by maintainers)

Commits related to this issue

Most upvoted comments

I like explicit better than implicit, so if I had to pick I would go for an explicit recv.placeholders.encryption and explain the reason for requiring configuration here to the user in zrepl’s docs. Zrepl can also notify affected users by erroring out if it observes a placeholder with encryption=off despite recv.placeholders.encryption being undefined.

Ok, I’m convinced. @InsanePrawn thoughts?

@problame Ok I tested the commit with this config:

---
global:
  logging:
    - type: stdout
      level: debug
      format: human
jobs:
- name: test-sink
  root_fs: test-sink/zrepl-encrypted-sink
  serve:
    listener_name: test-sink
    type: local
  recv:
    placeholder:
      encryption: inherit # Without this I was getting an error
  type: sink
- connect:
    client_identity: nas
    dial_timeout: 2s
    listener_name: test-sink
    type: local
  filesystems:
    vault/backups/postgres: true
    vault/backups/gmail: true
  name: nas-test-backups
  pruning:
    keep_receiver:
    - grid: 6x30d
      regex: ^zrepl_test_
      type: grid
    keep_sender:
    - type: not_replicated
  snapshotting:
    interval: 168h # 7 days
    prefix: zrepl_test_
    type: periodic
  type: push

Vol test-sink/zrepl-encrypted-sink was created using zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt test-sink/zrepl-encrypted-sink. After zrepl has finished syncing, I could see that all created subvolumes reported encryption=on using zfs get -r encryption test-sink/zrepl-encrypted-sink