LightGBM: [dask] Distributed Lightgbm randomly hangs when multiple train calls are submitted

Description: While submitting more than 1 Distributed Lightgbm train calls at the same time using dask, sometimes the execution is successful, sometimes, it just hangs forever. My machine has 8 workers and 2 threads per worker.

Reproducible Example:

from dask.distributed import Client, wait
from sklearn.datasets import make_classification
import dask.array as da
from dask.distributed import worker_client
import dask
import lightgbm as lgb

n_samples, n_features, num_test_samples = 10000, 10, 100
dask.config.set({'distributed.worker.daemon': False})


# Verbosity of printing messages. Valid values of 0 (silent), 1 (warning), 2 (info), and 3 (debug)
def invoke_dis_lgbm(X, y, number_of_estimators):
    with worker_client() as client_lgbm:
        lgbm_cls = lgb.DaskLGBMClassifier(client=client_lgbm,boosting_type='gbdt', objective='binary', num_leaves=50,
                                    learning_rate=0.1, n_estimators=number_of_estimators, max_depth=5,
                                    bagging_fraction=0.9, feature_fraction=0.9, reg_lambda=0.2,
        )
        lgbm_cls.fit(X, y)
    return 


def main(client):
    X_local, y_local = make_classification(n_samples=n_samples, n_features=n_features, random_state=12345)
    X = da.from_array(X_local, chunks=(n_samples//8, n_features), name='train_feature')
    y = da.from_array(y_local, chunks=(n_samples//8), name='train_label')

    X = client.persist(X)
    _ = wait(X)
    y = client.persist(y)
    _ = wait(y)

    futures = []
    results = []
    # here I am invoking 5 train calls
    for i in range(100, 105):
        f1 = client.submit(invoke_dis_lgbm, X, y, i)
        futures.append(f1)
    for i, f in enumerate(futures):
        results.append(f.result())
    return


if __name__ == '__main__':
    with Client('127.0.0.1:8786') as client:
        main(client)

The main problem seems to be this one: [LightGBM] [Fatal] Socket recv error, code: 104, I have checked the issue https://github.com/microsoft/LightGBM/issues/4625, However the proposed solution there was for Macos, mine is Linux.

Here is the complete log:

Finding random open ports for workers
Finding random open ports for workers
Finding random open ports for workers
Finding random open ports for workers
Finding random open ports for workers
[LightGBM] [Info] Trying to bind port 57189...
[LightGBM] [Info] Binding port 57189 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 37783...
[LightGBM] [Info] Binding port 37783 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 40885...
[LightGBM] [Info] Binding port 40885 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 56053...
[LightGBM] [Info] Binding port 56053 succeeded
[LightGBM] [Info] Trying to bind port 39767...
[LightGBM] [Info] Binding port 39767 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 34465...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Binding port 34465 succeeded
[LightGBM] [Info] Trying to bind port 56931...
[LightGBM] [Info] Binding port 56931 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Info] Trying to bind port 47941...
[LightGBM] [Info] Listening...
[LightGBM] [Info] Binding port 47941 succeeded
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Listening...
[LightGBM] [Info] Trying to bind port 41189...
[LightGBM] [Info] Binding port 41189 succeeded
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Listening...
[LightGBM] [Info] Trying to bind port 42587...
[LightGBM] [Info] Binding port 42587 succeeded
[LightGBM] [Info] Trying to bind port 57309...
[LightGBM] [Info] Binding port 57309 succeeded
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Listening...
[LightGBM] [Info] Trying to bind port 51349...
[LightGBM] [Info] Binding port 51349 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 48809...
[LightGBM] [Info] Trying to bind port 47085...
[LightGBM] [Info] Binding port 47085 succeeded
[LightGBM] [Info] Binding port 48809 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 200 milliseconds
[LightGBM] [Info] Listening...
machines source code ->  127.0.0.1:59689,127.0.0.1:53785,127.0.0.1:33347,127.0.0.1:55639,127.0.0.1:49899,127.0.0.1:37989,127.0.0.1:55235,127.0.0.1:38371
[LightGBM] [Info] Trying to bind port 43799...
[LightGBM] [Info] Binding port 43799 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 55275...
[LightGBM] [Info] Binding port 55275 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 17218 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 22383 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 6 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 29097 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Info] Connected to rank 1
[LightGBM] [Info] Connected to rank 2
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 260 milliseconds
[LightGBM] [Info] Local rank: 0, total number of machines: 8
[LightGBM] [Warning] num_threads is set=2, n_jobs=-1 will be ignored. Current value: num_threads=2
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9
[LightGBM] [Warning] feature_fraction is set=0.9, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9
[LightGBM] [Info] Connected to rank 0
[LightGBM] [Info] Connected to rank 2
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Info] Local rank: 1, total number of machines: 8
[LightGBM] [Warning] num_threads is set=2, n_jobs=-1 will be ignored. Current value: num_threads=2
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9
[LightGBM] [Warning] feature_fraction is set=0.9, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Info] Connected to rank 0
[LightGBM] [Info] Connected to rank 1
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Info] Local rank: 2, total number of machines: 8
[LightGBM] [Warning] num_threads is set=2, n_jobs=-1 will be ignored. Current value: num_threads=2
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9
[LightGBM] [Warning] feature_fraction is set=0.9, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 260 milliseconds
distributed.scheduler - INFO - Remove client Client-worker-93612a86-7253-11ec-b5b5-0200170338e3
distributed.nanny - INFO - Worker process 30133 was killed by signal 11
distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://127.0.0.1:39441', name: tcp://127.0.0.1:39441, memory: 6, processing: 5>
distributed.core - INFO - Removing comms to tcp://127.0.0.1:39441
distributed.scheduler - INFO - Close client connection: Client-worker-93612a86-7253-11ec-b5b5-0200170338e3
distributed.nanny - WARNING - Restarting worker
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 439 milliseconds
distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:36255
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:36255
distributed.worker - INFO -          dashboard at:            127.0.0.1:35889
distributed.worker - INFO - Waiting to connect to:       tcp://127.0.0.1:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          2
distributed.worker - INFO -       Local Directory: /tmp/olautoml.NTB74suIuO/dask-worker-space/worker-kn5k12ew
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <WorkerState 'tcp://127.0.0.1:36255', name: tcp://127.0.0.1:36255, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:36255
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to:       tcp://127.0.0.1:8786
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 4 failed, waiting for 4639 milliseconds
[LightGBM] [Fatal] Socket recv error, code: 104
distributed.scheduler - INFO - Remove worker <WorkerState 'tcp://127.0.0.1:43295', name: tcp://127.0.0.1:43295, memory: 6, processing: 6>
distributed.core - INFO - Removing comms to tcp://127.0.0.1:43295
distributed.nanny - INFO - Worker process 30119 was killed by signal//127.0.0.1:46173
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to:       tcp://127.0.0.1:8786
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.core - INFO - Starting established connection
distributed.core - INFO - Starting established connection
distributed.core - INFO - Starting established connection
distributed.core - INFO - Starting established connection
Finding random open ports for workers
machines source code ->  127.0.0.1:32963,127.0.0.1:55109,127.0.0.1:56591,127.0.0.1:57205,127.0.0.1:34665,127.0.0.1:54907,127.0.0.1:37213,127.0.0.1:53191
Finding random open ports for workers
Finding random open ports for workers
Finding random open ports for workers
Finding random open ports for workers
[LightGBM] [Info] Trying to bind port 32963...
[LightGBM] [Info] Binding port 32963 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 55109...
[LightGBM] [Info] Binding port 55109 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
machines source code ->  127.0.0.1:58799,127.0.0.1:36661,127.0.0.1:36481,127.0.0.1:40383,127.0.0.1:34539,127.0.0.1:53015,127.0.0.1:60303,127.0.0.1:47631
machines source code ->  127.0.0.1:50151,127.0.0.1:59425,127.0.0.1:53269,127.0.0.1:41435,127.0.0.1:34057,127.0.0.1:40317,127.0.0.1:59965,127.0.0.1:60019
[LightGBM] [Info] Trying to bind port 34665...
[LightGBM] [Info] Trying to bind port 54907...
[LightGBM] [Info] Binding port 34665 succeeded
[LightGBM] [Info] Binding port 54907 succeeded
[LightGBM] [Info] Trying to bind port 56591...
[LightGBM] [Info] Binding port 56591 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Info] Trying to bind port 57205...
[LightGBM] [Info] Binding port 57205 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 53191...
[LightGBM] [Info] Trying to bind port 37213...
[LightGBM] [Info] Binding port 53191 succeeded
[LightGBM] [Info] Binding port 37213 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Info] Trying to bind port 50151...
[LightGBM] [Info] Binding port 50151 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 53809...
[LightGBM] [Info] Binding port 53809 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 45151...
[LightGBM] [Info] Binding port 45151 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 49601...
[LightGBM] [Info] Binding port 49601 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 40769...
[LightGBM] [Info] Binding port 40769 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 34539...
[LightGBM] [Info] Binding port 34539 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Trying to bind port 53269...
[LightGBM] [Info] Binding port 53269 succeeded
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Listening...
[LightGBM] [Info] Trying to bind port 33649...
[LightGBM] [Info] Binding port 33649 succeeded
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 200 milliseconds
[LightGBM] [Info] Connected to rank 1
[LightGBM] [Info] Connected to rank 2
[LightGBM] [Info] Connected to rank 3
[LightGBM] [Info] Connected to rank 4
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Info] Local rank: 0, total number of machines: 8
[LightGBM] [Warning] num_threads is set=2, n_jobs=-1 will be ignored. Current value: num_threads=2
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9
[LightGBM] [Warning] feature_fraction is set=0.9, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9
[LightGBM] [Info] Connected to rank 0
[LightGBM] [Info] Connected to rank 2
[LightGBM] [Info] Connected to rank 3
[LightGBM] [Info] Connected to rank 4
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Info] Local rank: 1, total number of machines: 8
[LightGBM] [Warning] num_threads is set=2, n_jobs=-1 will be ignored. Current value: num_threads=2
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9
[LightGBM] [Warning] feature_fraction is set=0.9, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9
[LightGBM] [Info] Connected to rank 0
[LightGBM] [Info] Connected to rank 1
[LightGBM] [Info] Connected to rank 3
[LightGBM] [Info] Connected to rank 4
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Info] Local rank: 2, total number of machines: 8
[LightGBM] [Warning] num_threads is set=2, n_jobs=-1 will be ignored. Current value: num_threads=2
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9
[LightGBM] [Warning] feature_fraction is set=0.9, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9
[LightGBM] [Info] Connected to rank 0
[LightGBM] [Info] Connected to rank 1
[LightGBM] [Info] Connected to rank 2
[LightGBM] [Info] Connected to rank 3
[LightGBM] [Info] Connected to rank 4
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 0
[LightGBM] [Info] Connected to rank 1
[LightGBM] [Info] Connected to rank 2
[LightGBM] [Info] Connected to rank 3
[LightGBM] [Info] Connected to rank 0
[LightGBM] [Info] Connected to rank 4
[LightGBM] [Info] Connected to rank 1
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 2
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Info] Connected to rank 4
[LightGBM] [Info] Connected to rank 0
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 1
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 2
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Info] Connected to rank 3
[LightGBM] [Info] Connected to rank 5
[LightGBM] [Info] Connected to rank 0
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 1
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Info] Connected to rank 2
[LightGBM] [Info] Connected to rank 3
[LightGBM] [Info] Connected to rank 4
[LightGBM] [Info] Connected to rank 6
[LightGBM] [Info] Connected to rank 7
[LightGBM] [Info] Local rank: 7, total number of machines: 8
[LightGBM] [Info] Local rank: 6, total number of machines: 8
[LightGBM] [Info] Local rank: 4, total number of machines: 8
[LightGBM] [Info] Local rank: 5, total number of machines: 8
[LightGBM] [Info] Local rank: 3, total number of machines: 8
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 439 milliseconds
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Info] Listening...
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 200 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 260 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 338 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 439 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 570 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 741 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 963 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1251 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 1626 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2113 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 2746 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 3569 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 4639 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 6030 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 7838 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 10189 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 7 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 5 failed, waiting for 13245 milliseconds
[LightGBM] [Warning] Connecting to rank 3 failed, waiting for 13245 milliseconds

Environment: Linux Lightgbm: 3.2.1 dask: 2021.9.1 distributed: 2021.9.1

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 19

Most upvoted comments

Hi @vishalvvs, from https://github.com/microsoft/LightGBM/issues/4942#issuecomment-1014128096 it seems that the original problem in this issue wasn’t the right approach to achieving a speed up in training multiple models. Are you trying the same? If you’re facing a different problem please open a new issue with the description of your problem.

jmoralez on Aug 12, 2022

@rudra0713 may I ask what you’re trying to do? It seems like you want to train several models in parallel changing only the number of trees. If that’s the case you could just train a single model for the maximum number of iterations that you want and set num_iteration to the desired number of iterations when calling predict.

In case the number of trees is just to simulate setting a different hyperparameter and what you’re trying to do is just train several models in parallel you could modify your fit function to just call the regular lightgbm sklearn API with something like the following:

def train_local_model(X, y, number_of_estimators):
    lgbm_cls = lgb.LGBMClassifier(boosting_type='gbdt', objective='binary', num_leaves=50,
                                  learning_rate=0.1, n_estimators=number_of_estimators, max_depth=5,
                                  bagging_fraction=0.9, feature_fraction=0.9, reg_lambda=0.2,
      )
    lgbm_cls.fit(X, y)
    return lgbm_cls


def main(client):
    X, y = make_classification(n_samples=n_samples, n_features=n_features, random_state=12345)
    futures = []
    # here I am invoking 5 train calls
    for i in range(100, 105):
        f1 = client.submit(train_local_model, X, y, i)
        futures.append(f1)
    results = client.gather(futures)
    return results

The dask API for lightgbm is meant to train a single model using several workers. The main use case is when your data doesn’t fit in a single machine so you build a cluster of machines where each machine holds a piece of the data and train a model that way.

jmoralez on Jan 11, 2022