meltano: Nested config values in setting arrays and objects don't have environment variables expanded

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3245

Originally created by @alxthm on 2022-02-15 09:06:57

Related to #2366

What is the current bug behavior?

I was having trouble using environment variables in the meltano.yml file, and it seems they are only supported for top-level configuration options.

If I define SOME_ENV_VAR=SOME_VALUE in the .env file and I write this configuration yaml:

plugins:
  extractors:
    - name: tap-spreadsheets-anywhere
      variant: ets
      pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
      config:
        test: $SOME_ENV_VAR
        tables:
          - path: $SOME_ENV_VAR
            name: my_table

Then this is the actual configuration used by meltano:

$ meltano config tap-spreadsheets-anywhere
{
  "tables": [
    {
      "path": "$SOME_ENV_VAR",
      "name": "my_table"
    }
  ],
  "test": "SOME_VALUE"
}

And the tables.path variable is not set.

What is the expected correct behavior?

The environment variable SOME_ENV_VAR should expend to SOME_VALUE in the tables array as well, and we should have:

$ meltano config tap-spreadsheets-anywhere
{
  "tables": [
    {
      "path": "SOME_VALUE",
      "name": "my_table"
    }
  ],
  "test": "SOME_VALUE"
}

Steps to reproduce

add the configuration above in meltano.yml
add the line SOME_ENV_VAR=SOME_VALUE to your .env file
see the result with meltano config tap-spreadsheets-anywhere

Relevant logs and/or screenshots

Please use code blocks (```) to format console output

Possible fixes

If you can, link to the line of code that might be responsible for the problem or suggest a fix

Further regression test

Ensure we automatically catch similar issues in the future

Write additional adequate test cases and submit test results
Test results should be reviewed by a person from the team

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 4
Comments: 27 (17 by maintainers)

Commits related to this issue

fix: Expand enviroment variables in array setting values Closes https://github.com/meltano/meltano/issues/3171 — committed to meltano/meltano by edgarrmondragon 8 months ago
fix: Expand enviroment variables in array setting values Closes https://github.com/meltano/meltano/issues/3171 — committed to meltano/meltano by edgarrmondragon 8 months ago
fix: Expand enviroment variables in array setting values Closes https://github.com/meltano/meltano/issues/3171 — committed to meltano/meltano by edgarrmondragon 8 months ago
feat: Expand environment variables in array setting values (#8268) * fix: Expand enviroment variables in array setting values Closes https://github.com/meltano/meltano/issues/3171 * Do not support ... — committed to meltano/meltano by edgarrmondragon 7 months ago

Most upvoted comments

@cjohnhanson - This came up again today in regards to stream map config (ref above) and would probably be worth picking up if you have capacity opening up.

Cc @sbalnojan

aaronsteers on Mar 31, 2023

I have the same problem, trying to use tables-setting in tap-s3-csv.

    - name: tap-s3-csv
      config:
        bucket: my_bucket
        start_date: '2023-01-01T00:00:00Z'
        tables:
          - table_name: my_table
            key_properties:
              - Date
              - App Name
            search_prefix: prefix_name/
            search_pattern: executionDate=${TAP_S3_CSV_EXECUTION_DATE}/part-(.*)\.csv

andreigavrilov-onefootball on Jul 12, 2023

Coming from here: https://meltano.slack.com/archives/C01TCRBBJD7/p1686739597595969

Another one running into this issue. I am a bit shocked that custom env vars can’t be injected into the YML. Instead, I am going to inject a whole object into a plugin’s defined env var instead.

When is this issue expected to be resolved? I am pretty sure more will run into this.

Thanks!

mgarciadelojo on Jun 14, 2023

@cjohnhanson - This came up again today in regards to stream map config (ref above) and would probably be worth picking up if you have capacity opening up.

@aaronsteers I’ll dive back in this week, but how far I can get will definitely depend on cloud onboarding workload.

cjohnhanson on Apr 4, 2023

@tayloramurphy bumping here related to another user needing this https://github.com/meltano/meltano/issues/3435#issuecomment-1333985077.

pnadolny13 on Dec 1, 2022

@aaronsteers object settings can be worked around by using the dot syntax, e.g. my.nested.setting and can expand environment variables that way. For example: https://github.com/meltano/sdk/issues/1073

edgarrmondragon on Nov 3, 2022

@pandemicsyn if I’ve learned anything from time spent in config/env var stuff, it’s that these types of changes are always more complex than you would think.

As per @cjohnhanson comment, this may be another example in favour of an EnvVarsService #6253 to handle resolution in one place.

kgpayne on Aug 24, 2022

Another user asked for this in Slack. Also in relation to https://hub.meltano.com/extractors/tap-spreadsheets-anywhere#tables.

https://meltano.slack.com/archives/C01TCRBBJD7/p1660854848113529?thread_ts=1660828425.701879&cid=C01TCRBBJD7

edgarrmondragon on Aug 18, 2022

@tayloramurphy - Yes, agreed. I think a simplified flow from the Miro board:

Evaluate environment variables from contexts.
Evaluate settings values, with values injected from step 1 if/when appropriate.
Pass fully resolved settings values into plugin context (either in json or mapped “output” env vars, or both.)

aaronsteers on Aug 5, 2022

@cjohnhanson in looking at the Miro board I think there’s an easy path here (at least easy to say!).

I think the precedence order we’ve defined in https://github.com/meltano/meltano/issues/6386#issuecomment-1195887590 would be evaluated and resolved for any env key first and then that final result set would be passed to plugin settings and config to be expanded.

That said, I’d be ok with solving this for deeper levels of expansion and documenting whatever the current actual behavior is. @aaronsteers thoughts?

tayloramurphy on Aug 5, 2022