sops: sops/3.6.0 encrypted yaml file is broken (even to sops)

I have the following yaml file:

## Configuration for prometheus-node-exporter subchart
##
prometheus-node-exporter:
  podLabels:
    ## Add the 'node-exporter' label to be used by serviceMonitor to match standard common usage in rules and grafana dashboards
    ##

    jobLabel: node-exporter
  extraArgs:
    - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
    - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$

When it’s encrypted with sops, I get the output as below

prometheus-node-exporter:
## Add the 'node-exporter' label to be used by serviceMonitor to match standard common usage in rules and grafana dashboards
##  podLabels:
        jobLabel: node-exporter
    extraArgs:
    - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
    - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
sops:
    kms:
    -   arn: arn:aws:kms:eu-central-1:0000000:key/75f9aedd-ce9e-4b85-b112-00000000
        created_at: '2020-07-16T17:43:07Z'
        enc: AQICAHgquprv1l5xoZPFQCgP7aJyPm2SG/VQw5mKqJ1aWMOo1wGMTOvYqCZquWh0DzogTo7uAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMgkcK1GZYgX5WXfFJAgEQgDsRmTsCzr7vNGSLvP0sZ7zGQcXqg+5pEHNOp3H7jjVdyUAFY/LowjnEmWzWGdQCku4hljPHv/N3MXrjMw==
        aws_profile: ""
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2020-07-16T17:43:08Z'
    mac: ENC[AES256_GCM,data:N5EwksB9NZFd3Z1hLREp6lgOIc3K00XO6K54xQ7dDnYiywzozXvXFV1/5adZW4a3XaTfP3UkyzCFkWkfPesIJ1roLwjcYmwBzDke5wsl4MW2jNy8/M9B8u8sCmj6MpZY149gS6RV+8gETBKa9P9phGNAW4GLT+4DjXpBUIuXDiY=,iv:aCQQ7kjZBosxQipF68UiymJtvlsGWz6kCZX7Q/WiPEU=,tag:cMuYOyQajwV3kEMWtahzzA==,type:str]
    pgp: []
    encrypted_regex: (newrelic|certificate|private|pass|key|value)|(api_url)
    version: 3.6.0

This output is a broken yaml file. sops can’t even decrypt it:

$ sops -d foobar.yaml
Error unmarshaling input YAML: yaml: line 4: did not find expected key

$ ryaml < foobar.yaml
Traceback (most recent call last):
        3: from -e:2:in `<main>'
        2: from /usr/lib/ruby/2.7.0/psych.rb:565:in `load_stream'
        1: from /usr/lib/ruby/2.7.0/psych.rb:459:in `parse_stream'
/usr/lib/ruby/2.7.0/psych.rb:459:in `parse': (<unknown>): did not find expected key while parsing a block mapping at line 1 column 1 (Psych::SyntaxError)

The first comment in my original file is gone. I noticed these lines caused issue.

    ##

    jobLabel: node-exporter

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 43 (27 by maintainers)

Most upvoted comments

I’m unfortunately not able to create releases ever since I left Mozilla. @ajvb is currently the only person able to do it to my knowledge, and I imagine they’re quite busy so they don’t like to do releases super often.

On Mon, 16 Nov 2020 at 14:48, Felix Fontein notifications@github.com wrote:

@kim0 https://github.com/kim0 I think the worst bugs are fixed, at least the ones I know about / which annoyed me. What’s mainly missing is a new release.

@autrilla https://github.com/autrilla is there a chance of a new bugfix release anytime soon? Would be really appreciated.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/mozilla/sops/issues/695#issuecomment-728044958, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARH4V4IKZXEPIXBKTDBHILSQEURXANCNFSM4O4VESBQ .

autrilla on Nov 16, 2020

This is the smallest I could get the example and keep it broken:

a:
  b:
    #
    c: {}
  d:

The “Some comments are not included in the output” is known. The top-of-file comments are always going to get removed. This is documented and unfortunately hard to fix.

autrilla on Jul 17, 2020

I built from develop branch and indeed it looks much improved! Thanks! Alas, one tiny issue still remains. The yaml file ytt needs looks like

#@data/values
---
key: value

So yeah, that first yaml document is empty. When I try to encrypt that file, I get

Error unmarshalling file: Error unmarshaling input YAML: yaml: line 1: did not find expected node content

Can I hope to get a fix so that after encryption/decryption, the resulting file is exactly equal to the input? Thanks a lot for your help!

kim0 on Nov 16, 2020

@felixfontein actually, your issue seems unrelated to the original one in this thread. Really, we should fix these issues by migrating to the v3 YAML parser, which natively supports comments, and does it properly, instead of the way I hacked it into the v2 parser.

autrilla on Oct 4, 2020

I created a PR to bump the yaml library version and add tests for all the problems mentioned in this issue in #754.

felixfontein on Oct 7, 2020

Either works for me. This seems testable with a unit test, so I’d prefer that to keep things simple. So yeah, those look fine. The test should probably be put in https://github.com/mozilla/sops/blob/master/stores/yaml/store_test.go though.

autrilla on Oct 3, 2020

(I’m using sops 3.6.1)

felixfontein on Sep 25, 2020

I’m getting a lot of similar errors with some YAML files. I’ve built a following minimal example:

a:
 b:
 # foo

When cycled through sops -e / sops -d, I get warnings:

[SOPS]	 WARN[0000] Found possibly unencrypted comment in file. This is to be expected if the file being decrypted was created with an older version of SOPS.  comment="ENC[AES256_GCM,data:i0gx/Q==,iv:/FOmU45WS/1e6Zv80MIGsxSpXtQFl6DaIGGGhEJGTB8=,tag:c5Bp8X3M8eu2i1Ahny+V4Q==,type:comment]"

The resulting decrypted file:

#ENC[AES256_GCM,data:i0gx/Q==,iv:/FOmU45WS/1e6Zv80MIGsxSpXtQFl6DaIGGGhEJGTB8=,tag:c5Bp8X3M8eu2i1Ahny+V4Q==,type:comment]
a:
    b: null

I.e. not only did the comment end up encrypted, it also moved the comment around (and when cycling again, it will vanish because it is now at the top of the file).

Is this the same bug, or should I create a new issue for this?

felixfontein on Sep 25, 2020

That PR has landed, but we need to write a test on the SOPS side to make sure there’s no regressions, and also we need to upgrade the yaml dependency.

autrilla on Jul 20, 2020