hydra: [Bug] Hydra resolver does not work in compose_api

🐛 Bug

Description

hydra:key substitution does not work in interactive Python session and Jupyter notebook. hydra.key does, but it is not recommended (according to https://github.com/facebookresearch/hydra/issues/1786#issuecomment-914734857).

Checklist

I checked on the latest version of Hydra
I created a minimal repro (See this for tips).

To reproduce

** Minimal Code/Config snippet to reproduce **

conf/config.yaml

defaults:
  - env: dev

conf/env/prod.yaml

spark:
    appName: ${hydra:job.name}

conf/env/dev.yaml

spark:
    appName: ${hydra.job.name}

run.py

from hydra import initialize, compose

with initialize(config_path="conf"):
    conf = compose(config_name="config.yaml", return_hydra_config=True)
    print(conf.env.spark.appName)
    # prints current file name, e.g. "run"

with initialize(config_path="conf"):
    conf = compose(config_name="config.yaml", return_hydra_config=True, overrides=["env=prod"])
    print(conf.env.spark.appName)
    # raises exception

** Stack trace/error message **

$ python run.py
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 357, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/base.py", line 196, in _format_and_raise
    type_override=type_override,
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/_utils.py", line 821, in format_and_raise
    _raise(ex, cause)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/_utils.py", line 719, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 446, in _get_impl
    key=key, value=node, default_value=default_value
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 69, in _resolve_with_default
    throw_on_resolution_failure=True,
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/base.py", line 622, in _maybe_resolve_interpolation
    memo=memo if memo is not None else set(),
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/base.py", line 483, in _resolve_interpolation_from_parse_tree
    parse_tree=parse_tree, node=value, key=key, parent=parent, memo=memo
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/base.py", line 669, in resolve_parse_tree
    ).with_traceback(sys.exc_info()[2])
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/base.py", line 662, in resolve_parse_tree
    return visitor.visit(parse_tree)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 205, in accept
    return visitor.visitConfigValue(self)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
    return self.visit(ctx.getChild(0))
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 339, in accept
    return visitor.visitText(self)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/grammar_visitor.py", line 298, in visitText
    return self.visitInterpolation(c)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
    return self.visit(ctx.getChild(0))
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 1030, in accept
    return visitor.visitInterpolationResolver(self)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/grammar_visitor.py", line 182, in visitInterpolationResolver
    args_str=tuple(args_str),
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/base.py", line 653, in resolver_interpolation_callback
    inter_args_str=args_str,
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/base.py", line 596, in _evaluate_custom_resolver
    inter_args_str,
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 435, in resolver_wrapper
    ret = resolver(*args, **kwargs)
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/hydra/core/utils.py", line 215, in <lambda>
    lambda path: OmegaConf.select(cast(DictConfig, HydraConfig.get()), path),
  File "/home/maxim/Repo/ds-demo-project/venv/lib/python3.7/site-packages/hydra/core/hydra_config.py", line 31, in get
    raise ValueError("HydraConfig was not set")
omegaconf.errors.InterpolationResolutionError: ValueError raised while resolving interpolation: HydraConfig was not set
    full_key: env.spark.appName
    object_type=dict

Expected Behavior

Substitutionshydra: are not causing an exception inside a context created by compose function with return_hydra_config=True. HydraConfig instance should be initialized.

System information

Hydra Version : both 1.1.0 from pypi and 2a9f181 commit from main branch
Python version : both 3.7.8 and 3.10.2
Virtual environment type and version : venv
Operating system : Linux x86_64 5.16.5-1

Additional context

Same behavior has been observed several times: #716 https://github.com/facebookresearch/hydra/issues/1786#issuecomment-913110496

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 6
Comments: 19 (2 by maintainers)

Most upvoted comments

Just a note that I also ran into this problem. FYI my context:

I want to use the Compose API for my unit tests
I want to use in my config an interpolation of the form ${hydra:runtime.choices.xyz} to access which config group was used (and this is failing with HydraConfig was not set)

I would expect HydraConfig to be available within my with initialize(...) context (possibly with not all fields that are found when using @hydra.main(...), but at least if we have the config we can manipulate it for the purpose of the test)

Edit: my current workaround is the following (sketch):

with initialize(....):
    cfg = compose(
        ...,
        return_hydra_config=True,
    )
    HydraConfig.instance().set_config(cfg)

odelalleau on Sep 21, 2022

Yes indeed, the ConfigStore is persistent global state.

For Hydra’s test suite, we have a hydra_restore_singletons fixture that is used to save and then later restore the global state so that the tests can be run independently.

Jasha10 on Feb 15, 2022

No, I was talking about hydra.main. Please disregard my previous comment.

The root of the problem is that the hydra: resolver relies on global state (i.e. the state of the HydraConfig singleton). When using the compose API, the hydra: resolver fails because the global state (HydraConfig) is not set up.

It would be nice to have an API that has the best of both worlds from @hydra.main and compose:

@hydra.main allows for use of the the hydra: resolver and access to HydraConfig (within the context of the function decorated by @hydra.main). However, hydra.main does not easily allow control over the overrides via python code.
compose allows passing overrides and its use is consistent with the functional programming paradigm, but access to HydraConfig is not available.

Again brainstorming about possible APIs, it would be nice if compose could function as a context manager:

# a possible API:
with initialize(...):
    with compose(..., context=True) as conf:
        # ${hydra:} resolver works and HydraConf is available in this context
    # ${hydra:} resolver and HydraConf no longer available

Thanks for the link to #716. The context is useful. For now I will put this issue on the wishlist.

Jasha10 on Feb 15, 2022

Any progress here?

Thanks for looking into this @odelalleau! I’ll try to put together an implementation for compose_context.

I’m thinking that a usage pattern involving OmegaConf.resolve could even make the cfg object behave nicely after the context has ended:
with initialize(...):
    with compose_context(...) as cfg:
        OmegaConf.resolve(cfg)  # makes cfg usable outside of the context
...
application_logic(cfg.foo)

jasonkhadka on Apr 6, 2023

Any update on this?

maxoppelt on Mar 24, 2023

@Jasha10, Utilizing a context is a nice idea, but I think you should not abuse compose(). How about adding a new context for it?

with compose_with_global_hydra_config(...):
   ...

Alternatively, you can just document in the Compose API page how to regain access to the Hydra config (like the example from @odelalleau).

omry on Feb 24, 2023

Any updates on this?

jbergq on Feb 22, 2023

Thanks for looking into this @odelalleau! I’ll try to put together an implementation for compose_context.

I’m thinking that a usage pattern involving OmegaConf.resolve could even make the cfg object behave nicely after the context has ended:

with initialize(...):
    with compose_context(...) as cfg:
        OmegaConf.resolve(cfg)  # makes cfg usable outside of the context
...
application_logic(cfg.foo)

Jasha10 on Sep 22, 2022

Thanks for the report, @dolfinus. I can reproduce the behavior from the example you provided.

The issue here is that the hydra: resolver (${hydra:...}) depends on global state, namely the state of HydraConfig. When using @hydra.main, using the hydra: resolver results in a query to this global state.

Meanwhile, the compose API is a functional API; after the call to compose has returned, the global HydraConfig state is not retained. This is why we are getting the “HydraConfig was not set” ValueError in the traceback above.

I’m not sure what the best way to get around this is… Is your usecase limited to looking up hydra.job.name? Or is there some other hydra setting that you wanted to query?

Jasha10 on Feb 8, 2022