dvc: params: untracked values in params files are not ignored in params/exp commands

Bug Report

exp show: Invalid or wrong values for column names and values

Description

I have an experiment trying different parameters, it is the last stage of the pipeline, and the only with checkpointed outputs. Following this procedure: run the experiment (dvc exp run), change a parameter in params.yaml and run it again, do it several times. To visualize the results I run dvc exp show, but I get wrong column names and values.

I have a doubt about the intended behavior of this feature, when showing experiments, the idea is to show the comparison of metrics along parameters. It is expected to show the difference in metrics for ALL stages that contain metrics and parameters, or only those being experimented with (the ones that are run when iterating experiments)?

Trying to hide the trash columns with --exclude-metrics allows me to remove some columns, though some others are not recognized. But, in any case, it doesn’t remove the strange field values. I include a photo to avoid breaking the format:

2021-02-11-171417_1895x84_scrot

Reproduce

Reproducing is a bit computationally expensive, it requires training a bert model and doing some operations on it’s embeddings, it can be accomplished in the following way (at least one GPU with enough RAM, 5-8hrs of computations).

mkdir ./venv
virtualenv ./venv/dvc-reproduce
source ./venv/dvc-reproduce/bin/activate
git clone https://github.com/geblanco/quail-experiments
cd quail-experiments
git checkout restrucutre-experiments
mkdir ~/dvc_cache
dvc config --local core.remote local_dvc
dvc remote add --local -d local_dvc "~/dvc_cache"
dvc repro merge_norm_with_lengths_oversample_05
dvc exp run

Expected

The command should show a table with the parameters tracked by the stage, it’s changes and results in metrics.

Environment information

Output of dvc version:

$ dvc version
2.0.0a0+02cfe3

Additional Information (if any):

I have the intuition that it has to do with the parsing of parameters from a file other than params.yaml. To be more clear, here is the output in json of the show command:

There are various entries for data/specs/bert-train-no-empty-answers_experiment.json stated as parameters, with the whole file contents as parameters, although the stage using that file only uses a part of it as parameters (see stage bert-train-no-empty-answers in dvc.yaml below)

dvc exp show --show-json

{                                                                     
    "workspace": {
        "baseline": {
            "timestamp": null,
            "params": {
                "data/specs/bert-train-no-empty-answers_experiment.json": {
                    "inputs": [
                        "/home/gb/Documents/Research/quail-experiments/data/quail_no_empty_answers/train.json",
                        "/home/gb/Documents/Research/quail-experiments/data/quail_no_empty_answers/dev.json",
                        "/home/gb/Documents/Research/quail-experiments/data/models/bert-base-uncased"
                    ],
                    "scripts": [
                        "/home/gb/Documents/Research/quail-experiments/src/processing/run.sh"
                    ],
                    "metrics": [
                        "train_metrics.json",
                        "eval_metrics.json"
                    ],
                    "outputs": [
                        "data/models/bert-train-no-empty-answers/config.json",
                        "data/models/bert-train-no-empty-answers/pytorch_model.bin",
                        "data/models/bert-train-no-empty-answers/special_tokens_map.json",
                        "data/models/bert-train-no-empty-answers/tokenizer_config.json",
                        "data/models/bert-train-no-empty-answers/training_args.bin",
                        "data/models/bert-train-no-empty-answers/vocab.txt"
                    ],
                    "results": [
                        "train_predictions.json",
                        "train_nbest_predictions.json",
                        "eval_predictions.json",
                        "eval_nbest_predictions.json"
                    ],
                    "command": [
                        "./src/processing/run.sh",
                        "data/specs/bert-train-no-empty-answers_experiment.json"
                    ],
                    "params": {
                        "meta": "bert-train-no-empty-answers",
                        "data_dir": "data/quail_no_empty_answers",
                        "cache_dir": "/tmp",
                        "model_name_or_path": "data/models/bert-base-uncased",
                        "output_dir": "data/models/bert-train-no-empty-answers",
                        "metrics_dir": "data/metrics/bert-train-no-empty-answers",
                        "results_dir": "data/results/bert-train-no-empty-answers",
                        "model_type": "bert",
                        "task_name": "generic",
                        "do_train": true,
                        "do_eval": true,
                        "fp16": true,
                        "fp16_opt_level": "O1",
                        "save_total_limit": 0,
                        "save_steps": 0,
                        "max_seq_length": 484,
                        "num_train_epochs": 2,
                        "per_device_eval_batch_size": 8,
                        "per_device_train_batch_size": 2,
                        "gradient_accumulation_steps": 8,
                        "learning_rate": 5e-05,
                        "warmup_steps": 500,
                        "save_logits": true
                    },
                    "qualified_name": "bert-train-no-empty-answers",
                    "experiment_dir": "bert-train-no-empty-answers",
                    "experiment_name": "bert-train-no-empty-answers"
                },
                "params.yaml": {
                    "classification": {
                        "iterations": 200,
                        "popsize": 50,
                        "selection": 10,
                        "early_stop": 6,
                        "balanced": false,
                        "memory": 64,
                        "autogoal": false,
                        "pipeline": "logreg",
                        "sweep_features": false,
                        "test_size": 0.2,
                        "metric": "weighted_f1",
                        "multi_layer": {
                            "lr": 0.001,
                            "epochs": 100,
                            "batch_size": 200
                        }
                    },
                    "features": {
                        "normalization": true,
                        "oversample": false,
                        "text_length": false,
                        "embeddings": false,
                        "logits": false,
                        "context": false,
                        "question": false,
                        "endings": false
                    }
                }
            },
            "queued": false,
            "metrics": {
                "data/metrics/bert-train-no-empty-answers/eval_metrics.json": {
                    "eval_loss": 1.0892217214358713,
                    "eval_acc": 0.5046210720887245
                },
                "data/metrics/bert-train-no-empty-answers/train_metrics.json": {
                    "eval_loss": 0.5457088775723894,
                    "eval_acc": 0.8294667399670148
                },
                "data/metrics/classification/scores.json": {
                    "0": {
                        "precision": 0.0,
                        "recall": 0.0,
                        "f1-score": 0.0,
                        "support": 315
                    },
                    "1": {
                        "precision": 0.8268279274326553,
                        "recall": 1.0,
                        "f1-score": 0.9052061390309961,
                        "support": 1504
                    },
                    "accuracy": 0.8268279274326553,
                    "macro avg": {
                        "precision": 0.41341396371632766,
                        "recall": 0.5,
                        "f1-score": 0.45260306951549806,
                        "support": 1819
                    },
                    "weighted avg": {
                        "precision": 0.6836444215825803,
                        "recall": 0.8268279274326553,
                        "f1-score": 0.7484497158343145,
                        "support": 1819
                    }
                }
            }
        }
    },
    "1a64cf41824def258db67785f8b249d7c95cd46c": {
        "baseline": {
            "timestamp": "2021-02-10T17:01:56",
            "params": {
                "data/specs/bert-train-no-empty-answers_experiment.json": {
                    "inputs": [
                        "/home/gb/Documents/Research/quail-experiments/data/quail_no_empty_answers/train.json",
                        "/home/gb/Documents/Research/quail-experiments/data/quail_no_empty_answers/dev.json",
                        "/home/gb/Documents/Research/quail-experiments/data/models/bert-base-uncased"
                    ],
                    "scripts": [
                        "/home/gb/Documents/Research/quail-experiments/src/processing/run.sh"
                    ],
                    "metrics": [
                        "train_metrics.json",
                        "eval_metrics.json"
                    ],
                    "outputs": [
                        "data/models/bert-train-no-empty-answers/config.json",
                        "data/models/bert-train-no-empty-answers/pytorch_model.bin",
                        "data/models/bert-train-no-empty-answers/special_tokens_map.json",
                        "data/models/bert-train-no-empty-answers/tokenizer_config.json",
                        "data/models/bert-train-no-empty-answers/training_args.bin",
                        "data/models/bert-train-no-empty-answers/vocab.txt"
                    ],
                    "results": [
                        "train_predictions.json",
                        "train_nbest_predictions.json",
                        "eval_predictions.json",
                        "eval_nbest_predictions.json"
                    ],
                    "command": [
                        "./src/processing/run.sh",
                        "data/specs/bert-train-no-empty-answers_experiment.json"
                    ],
                    "params": {
                        "meta": "bert-train-no-empty-answers",
                        "data_dir": "data/quail_no_empty_answers",
                        "cache_dir": "/tmp",
                        "model_name_or_path": "data/models/bert-base-uncased",
                        "output_dir": "data/models/bert-train-no-empty-answers",
                        "metrics_dir": "data/metrics/bert-train-no-empty-answers",
                        "results_dir": "data/results/bert-train-no-empty-answers",
                        "model_type": "bert",
                        "task_name": "generic",
                        "do_train": true,
                        "do_eval": true,
                        "fp16": true,
                        "fp16_opt_level": "O1",
                        "save_total_limit": 0,
                        "save_steps": 0,
                        "max_seq_length": 484,
                        "num_train_epochs": 2,
                        "per_device_eval_batch_size": 8,
                        "per_device_train_batch_size": 2,
                        "gradient_accumulation_steps": 8,
                        "learning_rate": 5e-05,
                        "warmup_steps": 500,
                        "save_logits": true
                    },
                    "qualified_name": "bert-train-no-empty-answers",
                    "experiment_dir": "bert-train-no-empty-answers",
                    "experiment_name": "bert-train-no-empty-answers"
                },
                "params.yaml": {
                    "classification": {
                        "iterations": 200,
                        "popsize": 50,
                        "selection": 10,
                        "early_stop": 6,
                        "balanced": false,
                        "memory": 64,
                        "autogoal": false,
                        "pipeline": "logreg",
                        "sweep_features": false,
                        "test_size": 0.2,
                        "metric": "weighted_f1",
                        "multi_layer": {
                            "lr": 0.001,
                            "epochs": 100,
                            "batch_size": 200
                        }
                    },
                    "features": {
                        "normalization": false,
                        "oversample": false,
                        "text_length": false,
                        "embeddings": false,
                        "logits": false,
                        "context": false,
                        "question": false,
                        "endings": false
                    }
                }
            },
            "queued": false,
            "metrics": {
                "data/metrics/bert-train-no-empty-answers/eval_metrics.json": {
                    "eval_loss": 1.0892217214358713,
                    "eval_acc": 0.5046210720887245
                },
                "data/metrics/bert-train-no-empty-answers/train_metrics.json": {
                    "eval_loss": 0.5457088775723894,
                    "eval_acc": 0.8294667399670148
                }
            },
            "name": "restrucutre-experiments"
        },
        "7936e1fd371ceca8599b312930a27e3acdbadbe5": {
            "checkpoint_tip": "7936e1fd371ceca8599b312930a27e3acdbadbe5",
            "timestamp": "2021-02-11T11:47:46",
            "params": {
                "data/specs/bert-train-no-empty-answers_experiment.json": {
                    "inputs": [
                        "/home/gb/Documents/Research/quail-experiments/data/quail_no_empty_answers/train.json",
                        "/home/gb/Documents/Research/quail-experiments/data/quail_no_empty_answers/dev.json",
                        "/home/gb/Documents/Research/quail-experiments/data/models/bert-base-uncased"
                    ],
                    "scripts": [
                        "/home/gb/Documents/Research/quail-experiments/src/processing/run.sh"
                    ],
                    "metrics": [
                        "train_metrics.json",
                        "eval_metrics.json"
                    ],
                    "outputs": [
                        "data/models/bert-train-no-empty-answers/config.json",
                        "data/models/bert-train-no-empty-answers/pytorch_model.bin",
                        "data/models/bert-train-no-empty-answers/special_tokens_map.json",
                        "data/models/bert-train-no-empty-answers/tokenizer_config.json",
                        "data/models/bert-train-no-empty-answers/training_args.bin",
                        "data/models/bert-train-no-empty-answers/vocab.txt"
                    ],
                    "results": [
                        "train_predictions.json",
                        "train_nbest_predictions.json",
                        "eval_predictions.json",
                        "eval_nbest_predictions.json"
                    ],
                    "command": [
                        "./src/processing/run.sh",
                        "data/specs/bert-train-no-empty-answers_experiment.json"
                    ],
                    "params": {
                        "meta": "bert-train-no-empty-answers",
                        "data_dir": "data/quail_no_empty_answers",
                        "cache_dir": "/tmp",
                        "model_name_or_path": "data/models/bert-base-uncased",
                        "output_dir": "data/models/bert-train-no-empty-answers",
                        "metrics_dir": "data/metrics/bert-train-no-empty-answers",
                        "results_dir": "data/results/bert-train-no-empty-answers",
                        "model_type": "bert",
                        "task_name": "generic",
                        "do_train": true,
                        "do_eval": true,
                        "fp16": true,
                        "fp16_opt_level": "O1",
                        "save_total_limit": 0,
                        "save_steps": 0,
                        "max_seq_length": 484,
                        "num_train_epochs": 2,
                        "per_device_eval_batch_size": 8,
                        "per_device_train_batch_size": 2,
                        "gradient_accumulation_steps": 8,
                        "learning_rate": 5e-05,
                        "warmup_steps": 500,
                        "save_logits": true
                    },
                    "qualified_name": "bert-train-no-empty-answers",
                    "experiment_dir": "bert-train-no-empty-answers",
                    "experiment_name": "bert-train-no-empty-answers"
                },
                "params.yaml": {
                    "classification": {
                        "iterations": 200,
                        "popsize": 50,
                        "selection": 10,
                        "early_stop": 6,
                        "balanced": false,
                        "memory": 64,
                        "autogoal": false,
                        "pipeline": "logreg",
                        "sweep_features": false,
                        "test_size": 0.2,
                        "metric": "weighted_f1",
                        "multi_layer": {
                            "lr": 0.001,
                            "epochs": 100,
                            "batch_size": 200
                        }
                    },
                    "features": {
                        "normalization": true,
                        "oversample": false,
                        "text_length": false,
                        "embeddings": false,
                        "logits": false,
                        "context": false,
                        "question": false,
                        "endings": false
                    }
                }
            },
            "queued": false,
            "metrics": {
                "data/metrics/bert-train-no-empty-answers/eval_metrics.json": {
                    "eval_loss": 1.0892217214358713,
                    "eval_acc": 0.5046210720887245
                },
                "data/metrics/bert-train-no-empty-answers/train_metrics.json": {
                    "eval_loss": 0.5457088775723894,
                    "eval_acc": 0.8294667399670148
                },
                "data/metrics/classification/scores.json": {
                    "0": {
                        "precision": 0.0,
                        "recall": 0.0,
                        "f1-score": 0.0,
                        "support": 315
                    },
                    "1": {
                        "precision": 0.8268279274326553,
                        "recall": 1.0,
                        "f1-score": 0.9052061390309961,
                        "support": 1504
                    },
                    "accuracy": 0.8268279274326553,
                    "macro avg": {
                        "precision": 0.41341396371632766,
                        "recall": 0.5,
                        "f1-score": 0.45260306951549806,
                        "support": 1819
                    },
                    "weighted avg": {
                        "precision": 0.6836444215825803,
                        "recall": 0.8268279274326553,
                        "f1-score": 0.7484497158343145,
                        "support": 1819
                    }
                }
            },
            "name": "exp-6b84d",
            "checkpoint_parent": "b5f88821b90096dd4ed14ca397e7c1fdd28fdd64"
        },
        "b5f88821b90096dd4ed14ca397e7c1fdd28fdd64": {
            "checkpoint_tip": "b5f88821b90096dd4ed14ca397e7c1fdd28fdd64",
            "checkpoint_parent": "335edd1f67f3908e70b872f307a2500cbd47c6ac"
        },
        "335edd1f67f3908e70b872f307a2500cbd47c6ac": {
            "checkpoint_tip": "335edd1f67f3908e70b872f307a2500cbd47c6ac",
            "checkpoint_parent": "e842154c2a8268bcba9da8bc5e9775b2e13d409e"
        },
        "e842154c2a8268bcba9da8bc5e9775b2e13d409e": {
            "checkpoint_tip": "e842154c2a8268bcba9da8bc5e9775b2e13d409e",
            "checkpoint_parent": "d1db1a587c83847812f47e9ba4e68dac95886350"
        },
        "d1db1a587c83847812f47e9ba4e68dac95886350": {
            "timestamp": "2021-02-10T18:07:58",
            "params": {
                "data/specs/bert-train-no-empty-answers_experiment.json": {
                    "inputs": [
                        "/home/gb/Documents/Research/quail-experiments/data/quail_no_empty_answers/train.json",
                        "/home/gb/Documents/Research/quail-experiments/data/quail_no_empty_answers/dev.json",
                        "/home/gb/Documents/Research/quail-experiments/data/models/bert-base-uncased"
                    ],
                    "scripts": [
                        "/home/gb/Documents/Research/quail-experiments/src/processing/run.sh"
                    ],
                    "metrics": [
                        "train_metrics.json",
                        "eval_metrics.json"
                    ],
                    "outputs": [
                        "data/models/bert-train-no-empty-answers/config.json",
                        "data/models/bert-train-no-empty-answers/pytorch_model.bin",
                        "data/models/bert-train-no-empty-answers/special_tokens_map.json",
                        "data/models/bert-train-no-empty-answers/tokenizer_config.json",
                        "data/models/bert-train-no-empty-answers/training_args.bin",
                        "data/models/bert-train-no-empty-answers/vocab.txt"
                    ],
                    "results": [
                        "train_predictions.json",
                        "train_nbest_predictions.json",
                        "eval_predictions.json",
                        "eval_nbest_predictions.json"
                    ],
                    "command": [
                        "./src/processing/run.sh",
                        "data/specs/bert-train-no-empty-answers_experiment.json"
                    ],
                    "params": {
                        "meta": "bert-train-no-empty-answers",
                        "data_dir": "data/quail_no_empty_answers",
                        "cache_dir": "/tmp",
                        "model_name_or_path": "data/models/bert-base-uncased",
                        "output_dir": "data/models/bert-train-no-empty-answers",
                        "metrics_dir": "data/metrics/bert-train-no-empty-answers",
                        "results_dir": "data/results/bert-train-no-empty-answers",
                        "model_type": "bert",
                        "task_name": "generic",
                        "do_train": true,
                        "do_eval": true,
                        "fp16": true,
                        "fp16_opt_level": "O1",
                        "save_total_limit": 0,
                        "save_steps": 0,
                        "max_seq_length": 484,
                        "num_train_epochs": 2,
                        "per_device_eval_batch_size": 8,
                        "per_device_train_batch_size": 2,
                        "gradient_accumulation_steps": 8,
                        "learning_rate": 5e-05,
                        "warmup_steps": 500,
                        "save_logits": true
                    },
                    "qualified_name": "bert-train-no-empty-answers",
                    "experiment_dir": "bert-train-no-empty-answers",
                    "experiment_name": "bert-train-no-empty-answers"
                },
                "params.yaml": {
                    "classification": {
                        "iterations": 200,
                        "popsize": 50,
                        "selection": 10,
                        "early_stop": 6,
                        "balanced": false,
                        "memory": 64,
                        "autogoal": false,
                        "pipeline": "logreg",
                        "sweep_features": false,
                        "test_size": 0.2,
                        "metric": "weighted_f1",
                        "multi_layer": {
                            "lr": 0.001,
                            "epochs": 100,
                            "batch_size": 200
                        }
                    },
                    "features": {
                        "normalization": true,
                        "oversample": false,
                        "text_length": false,
                        "embeddings": false,
                        "logits": false,
                        "context": false,
                        "question": false,
                        "endings": false
                    }
                }
            },
            "queued": false,
            "metrics": {
                "data/metrics/bert-train-no-empty-answers/eval_metrics.json": {
                    "eval_loss": 1.0892217214358713,
                    "eval_acc": 0.5046210720887245
                },
                "data/metrics/bert-train-no-empty-answers/train_metrics.json": {
                    "eval_loss": 0.5457088775723894,
                    "eval_acc": 0.8294667399670148
                },
                "data/metrics/classification/scores.json": {
                    "0": {
                        "precision": 0.0,
                        "recall": 0.0,
                        "f1-score": 0.0,
                        "support": 329
                    },
                    "1": {
                        "precision": 0.8191313908741067,
                        "recall": 1.0,
                        "f1-score": 0.9005741915986704,
                        "support": 1490
                    },
                    "accuracy": 0.8191313908741067,
                    "macro avg": {
                        "precision": 0.40956569543705335,
                        "recall": 0.5,
                        "f1-score": 0.4502870957993352,
                        "support": 1819
                    },
                    "weighted avg": {
                        "precision": 0.6709762355153485,
                        "recall": 0.8191313908741067,
                        "f1-score": 0.7376885901495431,
                        "support": 1819
                    }
                }
            },
            "name": "exp-a1b47"
        }
    }
}

stages:
  dl_quail:
    cmd: ./src/etl/dl_quail.sh
    deps:
    - src/etl/dl_quail.sh
    outs:
    - data/quail/dev.json
    - data/quail/dev_answers.json
    - data/quail/eval.py
    - data/quail/train.json
    - data/quail/train_answers.json
  dl_models:
    cmd: python ./src/etl/dl_models.py -m bert-base-uncased --overwrite
    deps:
    - src/etl/dl_models.py
    outs:
    - data/models/bert-base-uncased
  tr_quail_train_to_race:
    cmd: src/etl/preprocess_quail_to_race.py data/quail/train.json data/quail/train_answers.json
      -o data/quail_race_fmt/train.json
    deps:
    - data/quail/train.json
    - data/quail/train_answers.json
    - src/etl/preprocess_quail_to_race.py
    outs:
    - data/quail_race_fmt/train.json
  tr_quail_dev_to_race:
    cmd: src/etl/preprocess_quail_to_race.py data/quail/dev.json data/quail/dev_answers.json
      -o data/quail_race_fmt/dev.json
    deps:
    - data/quail/dev.json
    - data/quail/dev_answers.json
    - src/etl/preprocess_quail_to_race.py
    outs:
    - data/quail_race_fmt/dev.json
  tr_quail_train_no_empty_answers:
    cmd: python src/etl/choices/reformat_dataset.py -d data/quail_race_fmt -o data/quail_no_empty_answers
      -s train --no_answer_text "not enough information"
    deps:
    - data/quail_race_fmt
    - src/etl/choices/reformat_dataset.py
    outs:
    - data/quail_no_empty_answers/train.json
  tr_quail_dev_no_empty_answers:
    cmd: python src/etl/choices/reformat_dataset.py -d ./data/quail_race_fmt -o ./data/quail_no_empty_answers
      -s dev --no_answer_text "not enough information" --keep_matching_text --index_list_path
      ./data/quail_no_empty_answers/dev_index_list.json
    deps:
    - data/quail_race_fmt
    - src/etl/choices/reformat_dataset.py
    outs:
    - ./data/quail_no_empty_answers/dev_index_list.json
    - data/quail_no_empty_answers/dev.json
  bert-train-no-empty-answers:
    cmd: ./src/processing/run.sh data/specs/bert-train-no-empty-answers_experiment.json
    deps:
    - data/models/bert-base-uncased
    - data/quail_no_empty_answers/dev.json
    - data/quail_no_empty_answers/train.json
    - src/processing/run.sh
    params:
    - data/specs/bert-train-no-empty-answers_experiment.json:
      - params.do_eval
      - params.do_train
      - params.fp16
      - params.fp16_opt_level
      - params.gradient_accumulation_steps
      - params.learning_rate
      - params.max_seq_length
      - params.meta
      - params.model_type
      - params.num_train_epochs
      - params.per_device_eval_batch_size
      - params.per_device_train_batch_size
      - params.task_name
      - params.warmup_steps
    outs:
    - data/models/bert-train-no-empty-answers/config.json
    - data/models/bert-train-no-empty-answers/pytorch_model.bin
    - data/models/bert-train-no-empty-answers/special_tokens_map.json
    - data/models/bert-train-no-empty-answers/tokenizer_config.json
    - data/models/bert-train-no-empty-answers/training_args.bin
    - data/models/bert-train-no-empty-answers/vocab.txt
    - data/results/bert-train-no-empty-answers/eval_nbest_predictions.json
    - data/results/bert-train-no-empty-answers/eval_predictions.json
    - data/results/bert-train-no-empty-answers/train_nbest_predictions.json
    - data/results/bert-train-no-empty-answers/train_predictions.json
    metrics:
    - data/metrics/bert-train-no-empty-answers/eval_metrics.json
    - data/metrics/bert-train-no-empty-answers/train_metrics.json
  extract_no_empty_answers_embeddings:
    cmd: python src/etl/embeddings/extract/extract_embeddings.py -s train --args_file
      data/specs/bert-eval-no-empty-answers_experiment.json --extract params --scatter_dataset
      -o /data/gblanco/datasets/embeddings/quail_no_empty_answers_2 --oversample 0.5
    deps:
    - data/models/bert-train-no-empty-answers
    - data/quail_no_empty_answers/train.json
    - data/specs/bert-eval-no-empty-answers_experiment.json
    - src/etl/embeddings/extract/extract_embeddings.py
    outs:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/embeddings
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_embeddings
  extract_no_empty_answers_lengths:
    cmd: python src/etl/embeddings/extract/extract_lengths.py -d data/quail_no_empty_answers
      -s train -t generic -o /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/train_text_lengths
    deps:
    - data/quail_no_empty_answers/train.json
    - src/etl/embeddings/extract/extract_lengths.py
    outs:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/train_text_lengths_data.pkl
  extract_no_empty_answers_lengths_oversample_05:
    cmd: python src/etl/embeddings/extract/extract_lengths.py -d /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_embeddings/
      -s train -t generic -o /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/train_text_lengths_oversample
    deps:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_embeddings/train.json
    - src/etl/embeddings/extract/extract_lengths.py
    outs:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/train_text_lengths_oversample_data.pkl
  normalize_no_emtpy_answers_embeddings:
    cmd: python src/etl/embeddings/transform/normalize_embeddings.py -d /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/embeddings
      -n 50 --scatter_dataset -o /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/normalized_embeddings
    deps:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/embeddings
    - src/etl/embeddings/transform/normalize_embeddings.py
    outs:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/normalized_embeddings
  normalize_no_emtpy_answers_embeddings_oversample_05:
    cmd: python src/etl/embeddings/transform/normalize_embeddings.py -d /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_embeddings/embeddings
      -n 50 --scatter_dataset -o /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_normalized_embeddings
    deps:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_embeddings/embeddings
    - src/etl/embeddings/transform/normalize_embeddings.py
    outs:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_normalized_embeddings
  merge_norm_with_lengths:
    cmd: python src/etl/embeddings/transform/merge_embeddings_and_lengths.py -e /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/normalized_embeddings
      -d /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/train_text_lengths_data.pkl
      -o /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/normalized_embeddings_with_lengths
      --scatter_dataset
    deps:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/normalized_embeddings
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/train_text_lengths_data.pkl
    - src/etl/embeddings/transform/merge_embeddings_and_lengths.py
    outs:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/normalized_embeddings_with_lengths
  merge_norm_with_lengths_oversample_05:
    cmd: python src/etl/embeddings/transform/merge_embeddings_and_lengths.py -e /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_normalized_embeddings
      -d /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/train_text_lengths_oversample_data.pkl
      -o /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_normalized_embeddings_with_lengths
      --scatter_dataset
    deps:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_normalized_embeddings
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/train_text_lengths_oversample_data.pkl
    - src/etl/embeddings/transform/merge_embeddings_and_lengths.py
    outs:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2/oversample_normalized_embeddings_with_lengths
  classification:
    cmd: python src/etl/embeddings/classify/classification.py -d /data/gblanco/datasets/embeddings/quail_no_empty_answers_2
      -o /data/gblanco/results/quail_no_empty_answers_2 --metrics_dir data/metrics/classification
      --train --eval
    deps:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers_2
    - src/etl/embeddings/classify/classification.py
    params:
    - classification
    - features
    outs:
    - /data/gblanco/results/quail_no_empty_answers_2:
        checkpoint: true
    metrics:
    - data/metrics/classification/scores.json:
        cache: false

I hope this information is useful enough, but, if anything is needed just let me know!

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 32 (32 by maintainers)

Most upvoted comments

@geblanco params command has now --deps flag and exp show has --param-deps flag to show only used params

+3
pared on Apr 20, 2021

In order not to break the current version of DVC, i think we should go with the opposite of --include-untracked. Let’s provide --used or --only-used. That way we keep the previous behaviour and add functionality to fix the current issue. Refactoring the default behavior for params we should probably leave for the next major release.

Okay, let’s go ahead with this plan and add it to params, diff, and exp show. Thoughts on the --used name? I’m partial to --tracked. Thoughts @jorgeorpinel ?

+3
dberenbaum on Mar 9, 2021

Agree! Is that what iterative/enhancement-proposals/pull/1 is for?

This issue is hopefully going to be addressed by #5550. That’s going to be a pretty limited change to provide an option to exclude untracked parameters. iterative/enhancement-proposals/pull/1 is for related changes building off of this discussion but less focused on fixing the specific issue here. Basically, we’re trying to keep the issue narrowly focused and use the enhacement proposal for the broader changes we want to make.

+2
dberenbaum on Mar 20, 2021

@jorgeorpinel

But if this should also apply to metrics then “deps” isn’t that meaningful. Could also add --outs for metrics to keep control over each concept separate.

I don’t think we should be concerned here with any outs. The main problem in this issue is that we allow sourcing multiple params from a single file. This is not a case in outs because we prevent stages from sharing outputs. So we are considering metrics on file level, and here we want to consider params on dict/file entry level.

+2
pared on Mar 11, 2021

@dberenbaum the fix is in https://github.com/iterative/dvc/pull/5550 I am still working on that

+1
pared on Mar 19, 2021

Okay, looks like this keeps getting more complicated! Maybe we jumped too far ahead here.

Are there any options for 2.x that won’t need to be reverted in 3.0? Can --filter solve this for now since that will still be useful in 3.0? That could at least allow @geblanco to specify something like --filter=params.yaml:classification?

IMO, this still needs more thought about how to only see parameters relevant to a particular experiment/pipeline, but that can be addressed for 3.0.

+1
dberenbaum on Mar 8, 2021

So in order to introduce the change for --include-untracked, we would need to break 2 already existing behaviours. 1 - reading from all used params files 2 - always considering params.yaml a params file.

As @efiop mentioned in here params and metrics are created based on assumptions introduced as we were developing it, sometimes by user. Current state might be confusing.

In order not to break the current version of DVC, i think we should go with the opposite of --include-untracked. Let’s provide --used or --only-used. That way we keep the previous behaviour and add functionality to fix the current issue. Refactoring the default behavior for params we should probably leave for the next major release.

+1
pared on Mar 8, 2021

@pared @pmrowla @skshetry My thoughts based on this feedback (which mostly match what I would expect):

  1. Agree with everyone that only tracked parameters should be included by default. --include-untracked seems fine as an option, although I don’t see it being that useful (does it provide enough benefit over git diff at that point?).
  2. --targets seems similarly not that useful compared to just using git diff. It’s possible that it would be more useful if it took dvc stages instead of filenames, since that is more aligned with the dvc workflow.
  3. --filter seems like a good option since seeing a subset of params (and metrics) seems like a common scenario. Would this only be a filename filter? Could it include more, like --filter=params.yaml:classification?
  4. This probably needs another issue, but this raises questions about what to include in params, metrics, and exp show when there are multiple pipelines. It would be great to pass a stage name and auto-filter params diff/metrics diff/exp show.
+1
dberenbaum on Mar 5, 2021

Thanks to you all for this great software you’re crafting!

@dberenbaum I’ll try to answer inline

Yes effectively, it seems natural to me to only take into account parameters that have been explicitly tracked down (-p <param_file>:<param>), also I don’t see many use-cases for the --include-untracked feature, though always good to have the option. On the contrary, tracking the entire file would just be a matter of passing -p data/specs/bert-train-no-empty-answers_experiment.json? Albeit when using --targets, as it can be used to diff files not tracked by DVC/git, it makes a lot of sense to track all parameters and exclude based on --filter.

  • You want to ignore the non-params section of data/specs/bert-train-no-empty-answers_experiment.json, right? That seems like it can be addressed by the discussion above.

When thinking about diffing parameters, I think more in terms of tracked parameters and their values, rather than the source those came from. So I guess it would be great to get the diff from all tracked parameters, regardless of its origin (as is done right now). Also, as the path of each tracked parameter is showed, filtering the output based on file is as easy as greping the output. Though an option for this filtering could be added for non-grep-friend users (analogous to --target <tracked param file> and --filter <tracked params inside file> .

  • Do you want to see params from both data/specs/bert-train-no-empty-answers_experiment.json and params.yaml?

The last two questions are very interesting in terms of researching methodology, I’m currently trying to find my way through it. Right now, what I usually do regarding the classification step is:

  1. Add the classifier stage to DVC with certain parameters tracked.
  2. Test the classifier with those parameters and measure its performance. If satisfied -> end; else, go to step 1 and try another classifier.

When trying new classifiers, sometimes more parameters are required, in such scenario I just modify the current classification stage within DVC to track my new parameters, though it does not seem to be the best procedure. In some cases, the code for a new classifier is really different from the previous one (i.e.: going from a RegressionTree to a whole transformers-based model), in those cases, the entire set of parameters might be changed, and it makes a lot sense to separate both source code files, and probably, add a different DVC stage, splitting the pipeline from the previous stage (the one before classification), this yields a tree with two nodes that are intended for the same purpose: classify previously processed data. The problem here, is that then, you have to point the output results of both stages to separate files (to avoid output conflicts), if we later add a stage that processes those results (joining both classification nodes) then, it has to depend on both results files, and so on… That is why I always end up modifying the DVC stage.

  • Do you have plans to add to this pipeline or set of parameters? For example, trying other classification layers that would need a different set of hyperparameters? If so, how would you plan to organize this?

Yes, that makes a lot of sense too. When a pipeline is split into two different nodes, and each node tracks different parameters (and probably continue evolving separately) it would be great to track those separately too. Otherwise, parameters would get tangled and it becomes harder to see the progress of each pipeline. For this scenario, maybe it would be possible to get a clearer idea with some use cases (I add one at the end, does it makes sense?)

For the last question, I’m thinking that multiple pipelines might be needed at times with separate parameters in each. Should dvc include only the parameters tracked by the current pipeline?

                 +---------+
                 | prepare |
                 +---------+
                      *
                      *
                      *
                +-----------+
                | featurize |
                +-----------+
                **      **
            **              **
         **                    **
       **                        *
      *                           **
+-------+                       +--------------+
| train |                       | augment data |
+-------+                       +--------------+
    *                                   *
    *                                   *
    *                                   *
+----------+                       +----------+
| evaluate |                       |   train  |
+----------+                       +----------+
    *                                   *
    *                                   *
    *                                   *
+------------------------+          +----------+
| prepare for production |          | evaluate |
+------------------------+          +----------+
                                        *                     
                                        *                     
                                        *                     
                        +--------------------------------------+
                        | add to database for further analysis |
                        +--------------------------------------+

In this case, it would make sense to track parameters for the first pipeline separately from the second one, as both serve for different purposes and can evolve within different timelapses.

+1
geblanco on Mar 5, 2021

@pmrowla, --include-untracked makes sense. For --targets, I also think that the behaviour should be as-is, as otherwise, it won’t be consistent between DVC repo and Git repo, but don’t have a strong opinion here.

Maybe introduce --filter=params.yaml to show only the tracked dependencies from the targets? 🤷🏽

+1
skshetry on Mar 5, 2021

To get the current behavior (meaning, include untracked values in DVC-tracked params dependencies), what about just -u/--include-untracked like git?

With --targets, I get that the current behavior is documented to be an intentional params diff use case:

Note that unlike dvc diff, this command doesn’t always need DVC files to find params files (see --targets option). For that reason, it doesn’t require an existing DVC project to run in. It can work in any Git repo.

but do we know many DVC users are actually using it this way?

If it’s not actually a common use case, I think it’s reasonable to just require both -u and --targets

+1
pmrowla on Mar 5, 2021

@pmrowla collect is responsible for gathering outs/deps, we can apply some filtering only after we actually read those dependencies in _read_params.

+1
pared on Mar 4, 2021

I don’t know if it was a feature or a bug, but from my user perspective, it’s annoying to get parameters into params diff and metrics when those parameters are not explicitly required for tracking.

+1
geblanco on Mar 4, 2021

It seems like it should be changed to me, or at the very least our internal collect methods for params should have a flag to only collect tracked parameters

+1
pmrowla on Mar 3, 2021
Read more comments on GitHub
← dvc.org: yarn: lots of errors when running dev app; can't test locally
dvc: Permission denied with windows share accessed from a linux dvc client →