meltano: bug: Projects without `environments` nor `default_environment` see error where `dev` environment is not found

Meltano Version

2.5.0

Python Version

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

Description

Users with projects created from scratch (not using meltano init) or started before Environments where introduced may not be using default_environment nor environments in their meltano.yml.

This causes an error because in 2.5.0 we made default_environment a project setting that defaults to dev. Any project without a dev environment, as those exemplified above, will see an error because Meltano will be looking for dev.

Code

$ meltano install
Environment 'dev' was not found

This is reproducible with the following meltano.yml:

plugins:
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

Workaround

From AJ:

There should be two ways to work around this issue.

Running meltano environment add dev as a one-time fix will resolve the issue, even if no other changes are made and no environment-specific settings are added.

Adding the --no-environment CLI arg will override the default dev environment name.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 16 (7 by maintainers)

Most upvoted comments

@tayloramurphy - I’ve added a “Workarounds” section above in the description. Since the workarounds seem pretty clear, I’m not seeing this as a high urgency item - or at least not high enough urgency that we’d want to introduce new tech debt or promote unsupported execution paths.

aaronsteers on Aug 30, 2022

Upgrade to 2.8.0 has helped with this issue! (caused another - but I think solvable) 👍

alanmcruickshank on Oct 19, 2022

@tayloramurphy - I’m on the fence a bit due to the importance of environments for many of our default behaviors.

We had some discussion on this over here:

#3200

If there was a reasonable way to warn or even hard fail with a specific instruction, I would feel a bit better about that path forward.

There’s another topic wrapped up here also, which is that some non-invocation commands shouldn’t need an environment or apply the default one.

I think these are the commands that can’t apply an environment and probably shouldn’t have a default environment applied:

These could have an environment applied or not, but probably should follow the meltano config behavior of the user specifying it explicitly:

These could be run without an environment and should auto-apply the default if available:

This should really not be run without an environment, because behaviors like auto-state tracking won’t work as expected when no environment is set:

Ideally if we change the “default default environment” to none, I’d like to pair this with a hard failure of meltano run if no environment is specified.

Orthogonal to those changes, and worthwhile on its own, I do think we should skip applying the default from all commands except elt, invoke, and run - which are the commands that perform workloads within the environments themselves. That would be an alternate solution to the noted failures during meltano install. This would match the recent changes on meltano config, which is that a user can still override the environment but we don’t default them into the environment if they didn’t specify it. (I think @tayloramurphy’s phrasing was that “default_environment” is more precisely explained as something like “default runtime environment”.)

aaronsteers on Aug 26, 2022

@edgarrmondragon what’s a reasonable resolution path here? Do we default to default_environment: none but put a prompt for users to consider adding environments? I don’t think suddenly forcing everything through dev is a great experience on an existing project.

@tayloramurphy Yeah, I think defaulting to none in settings.yml is the right call. New projects created with meltano init will still get default_environment: dev and the corresponding environment.

edgarrmondragon on Aug 26, 2022