LightGBM: [python-package] Can't retrieve best_iteration after Optuna optimization

Environment info

LightGBM version or commit hash: 4.1.0 Optuna version:3.6.0 Optuna_Integration version:3.6.0

Command(s) you used to install LightGBM

pip install lightgbm

Description

Hi there, I am new to LightGBM and currently I can’t find any useful solutions from google/stackoverflow/github issues, so I wonder if posting a new issue would be helpful, pardon me for the inappropriate behavior since I’m using ‘issue’ to ask a ‘question’.

Here’s my problem:

I was using Optuna to optimize my LightGBM model. At the same time I was using LightGBM callbacks early_stopping(50) to early stop the iterations. I have set the best model in loops of optimization, and retrieved the best model(best booster) from the user_attr. Since the early_stopping callbacks was set, the training output logs showed some content like this below:

Early stopping, best iteration is:
[30]	train_set's auc: 0.982083	valid_set's auc: 0.874471
Training until validation scores don't improve for 100 rounds

Assuming that the auc value above valid_set's auc: 0.874471 was indeed the best value from all iterations, the best_iteration should be [30] as showed above. However, I got -1 from invoking best_model.best_iteration like this below:

In: print(best_model.best_iteration)
Out: -1

My question is: How can I get the correct best_iteration value from the best model retrieved from study object?

Thanks to whom may solving my problem! Looking forward to your reply 😃

Reproducible example

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import lightgbm as lgb
import optuna
from lightgbm import early_stopping

dataset = load_breast_cancer()
x_train, x_test, y_train, y_test = train_test_split(dataset.data, dataset.target, test_size=0.2)

def objective(trial, train_set, valid_set, num_iterations): 
    params = {
        'objective':'binary', 
        'metric': ['auc'],
        'verbosity':-1, 
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.5)
    }

    pruning_callback = optuna.integration.LightGBMPruningCallback(trial, 'auc', valid_name='valid_set')
    
    model = lgb.train(
        params,
        num_boost_round=num_iterations,
        train_set=train_set, 
        valid_sets=[train_set, valid_set], 
        valid_names=['train_set', 'valid_set'],
        callbacks=[pruning_callback, early_stopping(50)]
    )
    
    trial.set_user_attr(key='best_booster', value=model)

    prob_pred = model.predict(x_test, num_iteration=model.best_iteration)
    return roc_auc_score(y_test, prob_pred, labels=[0,1])

train_set = lgb.Dataset(x_train, label=y_train)
valid_set = lgb.Dataset(x_test, label=y_test)
func = lambda trial: objective(trial=trial, train_set=train_set, valid_set=valid_set, num_iterations=num_iterations)

num_iterations = 100
study = optuna.create_study(
    pruner=optuna.pruners.HyperbandPruner(), 
    direction='maximize'
)

def save_best_booster(study, trial):
    if study.best_trial.number == trial.number:
        study.set_user_attr(key='best_booster', value=trial.user_attrs['best_booster'])

study.optimize(
    func, 
    n_trials=30,
    show_progress_bar=True,
    callbacks=[save_best_booster]
)

trial = study.best_trial
best_model=study.user_attrs['best_booster']

print(best_model.best_iteration)