omegaconf: Unexpected behavior for multiline-strings
Describe the bug Hey, I’m trying to have a config, where some values are in an escaped interpolation, but in a multiline-string. I think this is a really specific use case, but I think that this applies to all escaped characters.
To Reproduce
from omegaconf import OmegaConf as oc, Node
correct = {
"config": {
"subconfig": "MULTILINE \n ${tag:ENV}",
},
"values": "${config}",
}
wrong = {
"values": "${config}",
"config": {
"subconfig": "MULTILINE \n ${tag:ENV}",
},
}
def escape_tag(input: str, _node_: Node):
"""resolver function, that returns an escaped tag with the input"""
if "\n" in str(_node_): # multiline-strings
escape = "\\\\\\" # \\\
else:
escape = "\\" # \
return escape + "${" + input + "}"
oc.register_new_resolver("tag", escape_tag)
cfg_correct = oc.create(correct)
cfg_wrong = oc.create(wrong)
oc.resolve(cfg_correct)
oc.resolve(cfg_wrong)
print(cfg_correct["values"])
# >>> {'values': {'subconfig': 'MULTILINE \n \\${ENV}'}}
print(cfg_wrong["values"])
# >>> {'values': {'subconfig': 'MULTILINE \n \\\\\\${ENV}'}
### Another bug ?
print(cfg_correct["values"]["subconfig"])
# >>> [...] ${ENV}
print(cfg_wrong["values"]["subconfig"])
# >>> [...] \${ENV}
Explanation
Here I have two configs correct and wrong and the only difference between them is the key order in the python file.
Then I have a custom resolver that produces me an escaped interpolation. In multiline-strings \ has to be escaped and because of the double escapes (https://github.com/omry/omegaconf/pull/695#discussion_r621335546) we need 3 \. Now the diff is that in the wrong config, each \ is escaped again, and it produces 6 \.
Another bug (?) is, that when I print the actual strings, that the string gets dumped as multiline and not as quoted, but this could be intended.
Expected behavior
I expect, that the order in the config doesn’t matter and both configs produce \${ENV}.
If there is any kind of workaround like sorting the keys in the config, I would be happy already.
Additional context
- OmegaConf version: latest / master / 2.4.0.dev0
- Python version: 3.10.12
- Operating system: Linux Ubuntu
- Please provide a minimal repro
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 16 (9 by maintainers)
Commits related to this issue
- Properly escape interpolation-like strings in resolved configs Fixes #1112 Fixes #1081 — committed to odelalleau/omegaconf by odelalleau a year ago
- Properly escape interpolation-like strings in resolved configs Fixes #1112 Fixes #1081 — committed to odelalleau/omegaconf by odelalleau a year ago
Sounds great. Thank you really much!
Alright. Then I just got lucky with my testscript…
I tried it.
I’ll need to spend a bit more time understanding your use case and be able to tell whether there’s a better way to achieve what you want, but in the meantime I updated #1113 so that
resolve()now takes a flag to control whether or not to escape interpolation strings. I reverted the default behavior to the old one for now, and hopefully the problem you originally mentioned in this issue should be fixed.I don’t know if I’m doing something wrong, but if I run your code, my example above is still incorrect. Even if I need to change thetagresolver, then the output for both configs should be equal. I will try again later today, if I have more timeWorks now, I edited the wrong file in my env
Yes I expect so. (just be careful that my first PR was bugged, I just pushed a fixed version)
Yes (IMO).
I have a tentative fix in #1113. I can’t guarantee if/when a new version will be released with this fix, so you might need to use a custom omegaconf version for now.
Note that this may break some of your existing code if you are relying on the current buggy behavior (for instance I think you’ll need to change your
tagresolver to achieve what you want with this PR).Interesting – I believe this is the same root issue as in https://github.com/omry/omegaconf/issues/1081#issuecomment-1570443922
I’ll have a quick look to check if option (2) mentioned in that post would be easy to implement.