MachineLearningNotebooks: AutoMLEnsembleException: Could not find any models for running ensembling.
I’m facing this exception with ensemble in AutoML:
AutoMLEnsembleException: Could not find any models for running ensembling.
This exception is showing up in the voting ensemble child run.
AutoML Settings:
automl_settings = {
"featurization": "auto",
"primary_metric": "AUC_weighted",
"max_concurrent_iterations": vm_max_nodes-1,
"max_cores_per_iteration": 4,
"model_explainability": False,
"debug_log": "automl.log",
"experiment_timeout_hours": 5,
"iteration_timeout_minutes": 120,
"iterations": 40,
"enable_early_stopping": True,
"enable_voting_ensemble": True,
"enable_stack_ensemble": True}
SDK Info:
- Python version: 3.7.6
- AzureML SDK version: 1.1.5.1
Traceback:
Type: System Class: AutoMLEnsembleException Message: AutoMLEnsembleException: Could not find any models for running ensembling. Traceback: File “fit_pipeline.py”, line 158, in fit_pipeline remote) File “pipeline_run_helper.py”, line 323, in run_pipeline raise status.with_traceback(status.traceback) File “limit_function_call_spawn.py”, line 129, in execute **kwargs) File “spawn_client.py”, line 120, in run_in_proc raise err
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 15
I’d separate in two sections, kwargs specific to StackEnsemble and kwargs applicable to both Stack & Voting ensemble. The ensemble_download_models_timeout parameter is common to both, while the rest that are already in the docs are only applicable to Stack Ensemble.
@jadhosn But I think you could still provide this parameter through kwargs as discussed with Razvan. So you still can try that configuration.
This parameter is comparable to the other ensemble related parameters explained here:
https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#ensemble
There are multiple default arguments that can be provided as kwargs in an AutoMLConfig object to alter the default stack ensemble behavior. Like:
stack_meta_learner_type,stack_meta_learner_train_percentage, etc.We need to document
ensemble_download_models_timeout_secin that section, too.Setting it like the following in kwargs, the timeout will be set to 10 minutes, for instance:
"ensemble_download_models_timeout_sec": 600@jadhosn - Looks like that parameter
ensemble_download_models_timeout_secis not public. We’ll document the behavior and analyze if we should make it public for custom configuration.During Ensemble generation we download the fitted models from the previous child runs. By default we allocate 5 minutes for downloading these models in parallel, which seems not enough for your models. The timeout is for all the models to be downloaded. There is no maximum AFAIK. If the timeout is hit and there are models downloaded then the ensembling proceeds with as many models it has downloaded (it’s not required that all of them to finish within that timeout).
It looks like we the download of models required for ensembling procedure didn’t finish within the default timeout of 5 minutes. Please increase that timeout by setting this parameter in the AutoMLConfig: ensemble_download_models_timeout_sec to a larger value.