LightGBM: Unexpected Behavior for Early Stopping with Custom Metric

Hello -

When passing a custom metric to feval argument in Booster.train(), I’m getting what I believe to be unexpected behavior. Or what I presume to be undesirable. The Booster is not returning the best iteration as determined by the validation set as can be seen with the Reproducible example and Output below.

Environment:

lightgbm: 2.2.3 os: Linux CPU ami: 4.14.138-114.102.amzn2.x86_64

Reproducible example

import numpy as np
import lightgbm as lgb

np.random.seed(90210)
N = 100
x = np.random.uniform(size = (N, 7))

lp = (x[:,0]*1.34 + x[:,1]*x[:,2]*0.89 + x[:,2]*0.05 + x[:,2]*x[:,3]*1.34 + x[:,3]*0.31
      + np.random.normal(loc = 2, scale = 0.5, size = 100))

y = np.exp(lp - lp.mean()) / (1 + np.exp(lp - lp.mean()))
y = (y > 0.5).astype(np.uint8)

in_trn = np.random.binomial(1, p = 0.6, size = 100).astype(np.bool)
x_trn = x[ in_trn]
x_val = x[~in_trn]
y_trn = y[ in_trn]
y_val = y[~in_trn]

l_trn = lgb.Dataset(x_trn, y_trn)
l_val = lgb.Dataset(x_val, y_val)

def feval(prd, dmat):
    act = dmat.get_label()
    cls = (prd > 0.5).astype(np.uint8)
    tp = (cls * act).sum()
    fp = ((cls == 1) & (act == 0)).sum()
    savings = (100 * tp -75 * fp) / len(act)
    return 'savings', savings, True

params = {
        'learning_rate':0.05,
        'reg_lambda': 1,
        'feature_fraction': 0.75,
        'bagging_fraction': 0.7,
        'bagging_freq': 1,
        'boosting_type': 'gbdt',
        'objective': 'binary',
        'reg_alpha': 2,
        'num_leaves':  31,
        'min_data_in_leaf': 1,
        'feature_fraction_seed': 201,
        'bagging_seed': 427,
        'metric': 'None',
        }

mod = lgb.train(params, l_trn, num_boost_round = 1000, valid_sets = [l_trn,l_val],
                        valid_names = ['trn','val'], early_stopping_rounds = 100,
                        verbose_eval = 1, feval = feval)

Output

[1] trn’s savings: 16.6667 val’s savings: 5.40541 Training until validation scores don’t improve for 100 rounds. [2] trn’s savings: 30.5556 val’s savings: 21.6216 [3] trn’s savings: 34.127 val’s savings: 14.8649 [4] trn’s savings: 36.9048 val’s savings: 14.8649 [5] trn’s savings: 36.1111 val’s savings: 16.8919 [6] trn’s savings: 34.5238 val’s savings: 18.2432 [7] trn’s savings: 35.7143 val’s savings: 18.2432 [8] trn’s savings: 34.5238 val’s savings: 20.2703 [9] trn’s savings: 32.1429 val’s savings: 18.2432 [10] trn’s savings: 36.9048 val’s savings: 20.2703 [11] trn’s savings: 40.0794 val’s savings: 20.2703 [12] trn’s savings: 44.8413 val’s savings: 22.2973 [13] trn’s savings: 44.8413 val’s savings: 19.5946 [14] trn’s savings: 43.254 val’s savings: 19.5946 [15] trn’s savings: 44.8413 val’s savings: 21.6216 [16] trn’s savings: 44.8413 val’s savings: 21.6216 [17] trn’s savings: 43.6508 val’s savings: 21.6216 [18] trn’s savings: 42.0635 val’s savings: 21.6216 [19] trn’s savings: 44.8413 val’s savings: 19.5946 [20] trn’s savings: 44.8413 val’s savings: 21.6216 [21] trn’s savings: 46.0317 val’s savings: 21.6216 [22] trn’s savings: 46.0317 val’s savings: 21.6216

… … [110] trn’s savings: 44.4444 val’s savings: 23.6486 [111] trn’s savings: 43.254 val’s savings: 23.6486 [112] trn’s savings: 43.254 val’s savings: 23.6486 [113] trn’s savings: 43.254 val’s savings: 23.6486 [114] trn’s savings: 43.254 val’s savings: 23.6486 [115] trn’s savings: 43.254 val’s savings: 23.6486 [116] trn’s savings: 43.254 val’s savings: 23.6486 [117] trn’s savings: 43.254 val’s savings: 23.6486 [118] trn’s savings: 43.254 val’s savings: 23.6486 [119] trn’s savings: 43.254 val’s savings: 23.6486 [120] trn’s savings: 43.254 val’s savings: 23.6486 [121] trn’s savings: 43.254 val’s savings: 23.6486 Early stopping, best iteration is: [21] trn’s savings: 46.0317 val’s savings: 21.6216

Other Notes When I remove the training set from valid_sets and update valid_names accordingly, I do indeed get the correct iteration returned.

mod = lgb.train(params, l_trn, num_boost_round = 1000, valid_sets = l_val,
                        valid_names = ['val'], early_stopping_rounds = 100,
                        verbose_eval = 1, feval = feval)

[1] val’s savings: 5.40541 Training until validation scores don’t improve for 100 rounds. [2] val’s savings: 21.6216 [3] val’s savings: 14.8649 [4] val’s savings: 14.8649 [5] val’s savings: 16.8919 [6] val’s savings: 18.2432 [7] val’s savings: 18.2432 [8] val’s savings: 20.2703 [9] val’s savings: 18.2432 [10] val’s savings: 20.2703 [11] val’s savings: 20.2703 … … [72] val’s savings: 21.6216 [73] val’s savings: 21.6216 [74] val’s savings: 21.6216 [75] val’s savings: 21.6216 [76] val’s savings: 21.6216 [77] val’s savings: 21.6216 [78] val’s savings: 21.6216 [79] val’s savings: 23.6486 [80] val’s savings: 23.6486 [81] val’s savings: 23.6486 [82] val’s savings: 23.6486 [83] val’s savings: 23.6486 [84] val’s savings: 23.6486 … … [172] val’s savings: 23.6486 [173] val’s savings: 23.6486 [174] val’s savings: 23.6486 [175] val’s savings: 23.6486 [176] val’s savings: 23.6486 [177] val’s savings: 23.6486 [178] val’s savings: 23.6486 [179] val’s savings: 23.6486 Early stopping, best iteration is: [79] val’s savings: 23.6486