syne-tune: Promotion Logic Bug

There seems to be a problem with the Hyperband promotion logic.

How to reproduce: Add type="promotion" to https://github.com/awslabs/syne-tune/blob/main/benchmarking/nursery/benchmark_automl/baselines.py#L69

Run python benchmarking/nursery/benchmark_automl/benchmark_main.py --num_seeds 1 --method ASHA --benchmark lcbench-airlines

  File "/syne-tune/benchmarking/nursery/benchmark_automl/benchmark_main.py", line 209, in <module>
    tuner.run()
  File "/syne-tune/syne_tune/tuner.py", line 240, in run
    raise e
  File "/syne-tune/syne_tune/tuner.py", line 175, in run
    new_done_trial_statuses, new_results = self._process_new_results(
  File "/syne-tune/syne_tune/tuner.py", line 345, in _process_new_results
    done_trials_statuses = self._update_running_trials(
  File "/syne-tune/syne_tune/tuner.py", line 465, in _update_running_trials
    decision = self.scheduler.on_trial_result(trial=trial, result=result)
  File "/syne-tune/syne_tune/optimizer/schedulers/hyperband.py", line 779, in on_trial_result
    task_info = self.terminator.on_task_report(trial_id, result)
  File "/syne-tune/syne_tune/optimizer/schedulers/hyperband.py", line 1124, in on_task_report
    rung_sys.on_task_report(trial_id, result, skip_rungs=skip_rungs)
  File "/syne-tune/syne_tune/optimizer/schedulers/hyperband_promotion.py", line 221, in on_task_report
    assert resource == milestone, (
AssertionError: trial_id 1: resource = 4 > 3 milestone. Make sure to report time attributes covering all milestones```

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16

Commits related to this issue

Most upvoted comments

I can give this a try