omegaconf: Unexpected behavior for multiline-strings

Describe the bug Hey, I’m trying to have a config, where some values are in an escaped interpolation, but in a multiline-string. I think this is a really specific use case, but I think that this applies to all escaped characters.

To Reproduce

from omegaconf import OmegaConf as oc, Node

correct = {
    "config": {
        "subconfig": "MULTILINE \n    ${tag:ENV}",
    },
    "values": "${config}",
}

wrong = {
    "values": "${config}",
    "config": {
        "subconfig": "MULTILINE \n    ${tag:ENV}",
    },
}


def escape_tag(input: str, _node_: Node):
    """resolver function, that returns an escaped tag with the input"""
    if "\n" in str(_node_):  # multiline-strings
        escape = "\\\\\\"  # \\\
    else:
        escape = "\\"  # \

    return escape + "${" + input + "}"


oc.register_new_resolver("tag", escape_tag)


cfg_correct = oc.create(correct)
cfg_wrong = oc.create(wrong)

oc.resolve(cfg_correct)
oc.resolve(cfg_wrong)

print(cfg_correct["values"])
# >>> {'values': {'subconfig': 'MULTILINE \n    \\${ENV}'}}
print(cfg_wrong["values"])
# >>> {'values': {'subconfig': 'MULTILINE \n    \\\\\\${ENV}'}

### Another bug ?
print(cfg_correct["values"]["subconfig"])
# >>> [...] ${ENV}
print(cfg_wrong["values"]["subconfig"])
# >>> [...] \${ENV}

Explanation Here I have two configs correct and wrong and the only difference between them is the key order in the python file. Then I have a custom resolver that produces me an escaped interpolation. In multiline-strings \ has to be escaped and because of the double escapes (https://github.com/omry/omegaconf/pull/695#discussion_r621335546) we need 3 \. Now the diff is that in the wrong config, each \ is escaped again, and it produces 6 \.

Another bug (?) is, that when I print the actual strings, that the string gets dumped as multiline and not as quoted, but this could be intended.

Expected behavior I expect, that the order in the config doesn’t matter and both configs produce \${ENV}.

If there is any kind of workaround like sorting the keys in the config, I would be happy already.

Additional context

  • OmegaConf version: latest / master / 2.4.0.dev0
  • Python version: 3.10.12
  • Operating system: Linux Ubuntu
  • Please provide a minimal repro

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 16 (9 by maintainers)

Commits related to this issue

Most upvoted comments

but in the meantime I updated https://github.com/omry/omegaconf/pull/1113 so that resolve() now takes a flag to control whether or not to escape interpolation strings.

Sounds great. Thank you really much!

I tried it.

Alright. Then I just got lucky with my testscript…

I’m not sure I entirely follow what you’re trying to do here, but this breaks a bunch of tests.

You tried already, or an assumption?

I tried it.

I’ll need to spend a bit more time understanding your use case and be able to tell whether there’s a better way to achieve what you want, but in the meantime I updated #1113 so that resolve() now takes a flag to control whether or not to escape interpolation strings. I reverted the default behavior to the old one for now, and hopefully the problem you originally mentioned in this issue should be fixed.

I don’t know if I’m doing something wrong, but if I run your code, my example above is still incorrect. Even if I need to change the tag resolver, then the output for both configs should be equal. I will try again later today, if I have more time

Works now, I edited the wrong file in my env

I still wonder what causes the different behavior regarding the order of the keys in my example. Will it fix that too?

Yes I expect so. (just be careful that my first PR was bugged, I just pushed a fixed version)

Is there a general problem with producing (escaped) interpolations

Yes (IMO).

I have a tentative fix in #1113. I can’t guarantee if/when a new version will be released with this fix, so you might need to use a custom omegaconf version for now.

Note that this may break some of your existing code if you are relying on the current buggy behavior (for instance I think you’ll need to change your tag resolver to achieve what you want with this PR).

Interesting – I believe this is the same root issue as in https://github.com/omry/omegaconf/issues/1081#issuecomment-1570443922

I’ll have a quick look to check if option (2) mentioned in that post would be easy to implement.