LightGBM: Dask tests randomly fail with socket error code 104
[LightGBM] [Fatal] Socket send error, code: 104
distributed.worker - WARNING - Compute Failed
Full logs:
2021-03-15T22:41:00.2549100Z ============================= test session starts ==============================
2021-03-15T22:41:00.2550462Z platform linux -- Python 3.8.3, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
2021-03-15T22:41:00.2550974Z rootdir: /LightGBM
2021-03-15T22:41:00.2551347Z collected 395 items
2021-03-15T22:41:00.2552034Z
2021-03-15T22:41:01.7305167Z ../tests/python_package_test/test_basic.py ......................... [ 6%]
2021-03-15T22:41:11.7101364Z ../tests/python_package_test/test_consistency.py ...... [ 7%]
2021-03-15T22:43:21.2950317Z ../tests/python_package_test/test_dask.py .............................. [ 15%]
2021-03-15T22:49:09.6962741Z ......................F............................s...............s.... [ 33%]
2021-03-15T22:52:32.9692687Z ...........s.............................s................. [ 48%]
2021-03-15T22:52:32.9703455Z ../tests/python_package_test/test_dual.py s [ 48%]
2021-03-15T22:52:37.5705591Z ../tests/python_package_test/test_engine.py ............................ [ 55%]
2021-03-15T22:53:14.1203842Z ......................................... [ 66%]
2021-03-15T22:53:14.8460675Z ../tests/python_package_test/test_plotting.py ..... [ 67%]
2021-03-15T22:53:17.2262198Z ../tests/python_package_test/test_sklearn.py ........................... [ 74%]
2021-03-15T22:53:21.2526973Z ......x.............................................x................... [ 92%]
2021-03-15T22:53:22.5839380Z .......................ss... [ 99%]
2021-03-15T22:53:22.6106534Z ../tests/python_package_test/test_utilities.py . [100%]
2021-03-15T22:53:22.6107017Z
2021-03-15T22:53:22.6107367Z =================================== FAILURES ===================================
2021-03-15T22:53:22.6108982Z _ test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-multiclass-classification-joblib] _
2021-03-15T22:53:22.6109743Z
2021-03-15T22:53:22.6110501Z serializer = 'joblib', task = 'multiclass-classification', set_client = True
2021-03-15T22:53:22.6111514Z tmp_path = PosixPath('/tmp/pytest-of-root/pytest-0/test_model_and_local_version_a4')
2021-03-15T22:53:22.6111976Z
2021-03-15T22:53:22.6112692Z @pytest.mark.parametrize('serializer', ['pickle', 'joblib', 'cloudpickle'])
2021-03-15T22:53:22.6113592Z @pytest.mark.parametrize('task', tasks)
2021-03-15T22:53:22.6114399Z @pytest.mark.parametrize('set_client', [True, False])
2021-03-15T22:53:22.6115265Z def test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly(serializer, task, set_client, tmp_path):
2021-03-15T22:53:22.6115881Z
2021-03-15T22:53:22.6116330Z with LocalCluster(n_workers=2, threads_per_worker=1) as cluster1:
2021-03-15T22:53:22.6116901Z with Client(cluster1) as client1:
2021-03-15T22:53:22.6117516Z
2021-03-15T22:53:22.6117868Z # data on cluster1
2021-03-15T22:53:22.6118419Z if task == 'ranking':
2021-03-15T22:53:22.6118855Z X_1, _, _, _, dX_1, dy_1, _, dg_1 = _create_ranking_data(
2021-03-15T22:53:22.6119408Z output='array',
2021-03-15T22:53:22.6119770Z group=None
2021-03-15T22:53:22.6120208Z )
2021-03-15T22:53:22.6120487Z else:
2021-03-15T22:53:22.6120847Z X_1, _, _, dX_1, dy_1, _ = _create_data(
2021-03-15T22:53:22.6121228Z objective=task,
2021-03-15T22:53:22.6121788Z output='array',
2021-03-15T22:53:22.6122097Z )
2021-03-15T22:53:22.6122398Z dg_1 = None
2021-03-15T22:53:22.6122670Z
2021-03-15T22:53:22.6123122Z with LocalCluster(n_workers=2, threads_per_worker=1) as cluster2:
2021-03-15T22:53:22.6123707Z with Client(cluster2) as client2:
2021-03-15T22:53:22.6124061Z
2021-03-15T22:53:22.6124441Z # create identical data on cluster2
2021-03-15T22:53:22.6125302Z if task == 'ranking':
2021-03-15T22:53:22.6125786Z X_2, _, _, _, dX_2, dy_2, _, dg_2 = _create_ranking_data(
2021-03-15T22:53:22.6126446Z output='array',
2021-03-15T22:53:22.6126827Z group=None
2021-03-15T22:53:22.6127132Z )
2021-03-15T22:53:22.6127432Z else:
2021-03-15T22:53:22.6127803Z X_2, _, _, dX_2, dy_2, _ = _create_data(
2021-03-15T22:53:22.6128192Z objective=task,
2021-03-15T22:53:22.6128741Z output='array',
2021-03-15T22:53:22.6129060Z )
2021-03-15T22:53:22.6129373Z dg_2 = None
2021-03-15T22:53:22.6129654Z
2021-03-15T22:53:22.6130027Z model_factory = task_to_dask_factory[task]
2021-03-15T22:53:22.6130410Z
2021-03-15T22:53:22.6130685Z params = {
2021-03-15T22:53:22.6131024Z "time_out": 5,
2021-03-15T22:53:22.6131370Z "n_estimators": 1,
2021-03-15T22:53:22.6131743Z "num_leaves": 2
2021-03-15T22:53:22.6132053Z }
2021-03-15T22:53:22.6132311Z
2021-03-15T22:53:22.6132823Z # at this point, the result of default_client() is client2 since it was the most recently
2021-03-15T22:53:22.6133837Z # created. So setting client to client1 here to test that you can select a non-default client
2021-03-15T22:53:22.6134494Z assert default_client() == client2
2021-03-15T22:53:22.6134898Z if set_client:
2021-03-15T22:53:22.6135351Z params.update({"client": client1})
2021-03-15T22:53:22.6135726Z
2021-03-15T22:53:22.6136198Z # unfitted model should survive pickling round trip, and pickling
2021-03-15T22:53:22.6136989Z # shouldn't have side effects on the model object
2021-03-15T22:53:22.6137496Z dask_model = model_factory(**params)
2021-03-15T22:53:22.6137972Z local_model = dask_model.to_local()
2021-03-15T22:53:22.6138374Z if set_client:
2021-03-15T22:53:22.6138816Z assert dask_model.client == client1
2021-03-15T22:53:22.6139217Z else:
2021-03-15T22:53:22.6139624Z assert dask_model.client is None
2021-03-15T22:53:22.6139987Z
2021-03-15T22:53:22.6140968Z with pytest.raises(lgb.compat.LGBMNotFittedError, match='Cannot access property client_ before calling fit'):
2021-03-15T22:53:22.6141929Z dask_model.client_
2021-03-15T22:53:22.6142282Z
2021-03-15T22:53:22.6142689Z assert "client" not in local_model.get_params()
2021-03-15T22:53:22.6143234Z assert getattr(local_model, "client", None) is None
2021-03-15T22:53:22.6143649Z
2021-03-15T22:53:22.6144225Z tmp_file = str(tmp_path / "model-1.pkl")
2021-03-15T22:53:22.6144646Z _pickle(
2021-03-15T22:53:22.6145096Z obj=dask_model,
2021-03-15T22:53:22.6145486Z filepath=tmp_file,
2021-03-15T22:53:22.6145901Z serializer=serializer
2021-03-15T22:53:22.6146253Z )
2021-03-15T22:53:22.6146831Z model_from_disk = _unpickle(
2021-03-15T22:53:22.6147245Z filepath=tmp_file,
2021-03-15T22:53:22.6147666Z serializer=serializer
2021-03-15T22:53:22.6148042Z )
2021-03-15T22:53:22.6148306Z
2021-03-15T22:53:22.6148984Z local_tmp_file = str(tmp_path / "local-model-1.pkl")
2021-03-15T22:53:22.6149450Z _pickle(
2021-03-15T22:53:22.6149789Z obj=local_model,
2021-03-15T22:53:22.6150198Z filepath=local_tmp_file,
2021-03-15T22:53:22.6150627Z serializer=serializer
2021-03-15T22:53:22.6151001Z )
2021-03-15T22:53:22.6151367Z local_model_from_disk = _unpickle(
2021-03-15T22:53:22.6151820Z filepath=local_tmp_file,
2021-03-15T22:53:22.6152247Z serializer=serializer
2021-03-15T22:53:22.6152594Z )
2021-03-15T22:53:22.6152866Z
2021-03-15T22:53:22.6153234Z assert model_from_disk.client is None
2021-03-15T22:53:22.6153620Z
2021-03-15T22:53:22.6153909Z if set_client:
2021-03-15T22:53:22.6154355Z assert dask_model.client == client1
2021-03-15T22:53:22.6154754Z else:
2021-03-15T22:53:22.6155149Z assert dask_model.client is None
2021-03-15T22:53:22.6155527Z
2021-03-15T22:53:22.6156494Z with pytest.raises(lgb.compat.LGBMNotFittedError, match='Cannot access property client_ before calling fit'):
2021-03-15T22:53:22.6157331Z dask_model.client_
2021-03-15T22:53:22.6157654Z
2021-03-15T22:53:22.6158055Z # client will always be None after unpickling
2021-03-15T22:53:22.6158488Z if set_client:
2021-03-15T22:53:22.6158948Z from_disk_params = model_from_disk.get_params()
2021-03-15T22:53:22.6159456Z from_disk_params.pop("client", None)
2021-03-15T22:53:22.6159946Z dask_params = dask_model.get_params()
2021-03-15T22:53:22.6160440Z dask_params.pop("client", None)
2021-03-15T22:53:22.6160910Z assert from_disk_params == dask_params
2021-03-15T22:53:22.6161313Z else:
2021-03-15T22:53:22.6161781Z assert model_from_disk.get_params() == dask_model.get_params()
2021-03-15T22:53:22.6162437Z assert local_model_from_disk.get_params() == local_model.get_params()
2021-03-15T22:53:22.6162890Z
2021-03-15T22:53:22.6163352Z # fitted model should survive pickling round trip, and pickling
2021-03-15T22:53:22.6164143Z # shouldn't have side effects on the model object
2021-03-15T22:53:22.6164603Z if set_client:
2021-03-15T22:53:22.6165003Z > dask_model.fit(dX_1, dy_1, group=dg_1)
2021-03-15T22:53:22.6165286Z
2021-03-15T22:53:22.6165781Z ../tests/python_package_test/test_dask.py:833:
2021-03-15T22:53:22.6166341Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-03-15T22:53:22.6167087Z /root/.local/lib/python3.8/site-packages/lightgbm/dask.py:683: in fit
2021-03-15T22:53:22.6167624Z return self._lgb_dask_fit(
2021-03-15T22:53:22.6168348Z /root/.local/lib/python3.8/site-packages/lightgbm/dask.py:570: in _lgb_dask_fit
2021-03-15T22:53:22.6168856Z model = _train(
2021-03-15T22:53:22.6169673Z /root/.local/lib/python3.8/site-packages/lightgbm/dask.py:399: in _train
2021-03-15T22:53:22.6170300Z results = client.gather(futures_classifiers)
2021-03-15T22:53:22.6171247Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/client.py:1963: in gather
2021-03-15T22:53:22.6171870Z return self.sync(
2021-03-15T22:53:22.6172683Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/client.py:837: in sync
2021-03-15T22:53:22.6173274Z return sync(
2021-03-15T22:53:22.6174064Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/utils.py:351: in sync
2021-03-15T22:53:22.6174710Z raise exc.with_traceback(tb)
2021-03-15T22:53:22.6175528Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/utils.py:334: in f
2021-03-15T22:53:22.6176148Z result[0] = yield future
2021-03-15T22:53:22.6176905Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/tornado/gen.py:762: in run
2021-03-15T22:53:22.6177518Z value = future.result()
2021-03-15T22:53:22.6178363Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/client.py:1828: in _gather
2021-03-15T22:53:22.6179115Z raise exception.with_traceback(traceback)
2021-03-15T22:53:22.6179931Z /root/.local/lib/python3.8/site-packages/lightgbm/dask.py:118: in _train_part
2021-03-15T22:53:22.6180648Z model.fit(data, label, sample_weight=weight, init_score=init_score, **kwargs)
2021-03-15T22:53:22.6181515Z /root/.local/lib/python3.8/site-packages/lightgbm/sklearn.py:890: in fit
2021-03-15T22:53:22.6182242Z super().fit(X, _y, sample_weight=sample_weight, init_score=init_score, eval_set=valid_sets,
2021-03-15T22:53:22.6183161Z /root/.local/lib/python3.8/site-packages/lightgbm/sklearn.py:683: in fit
2021-03-15T22:53:22.6183747Z self._Booster = train(params, train_set,
2021-03-15T22:53:22.6184520Z /root/.local/lib/python3.8/site-packages/lightgbm/engine.py:228: in train
2021-03-15T22:53:22.6185159Z booster = Booster(params=params, train_set=train_set)
2021-03-15T22:53:22.6185980Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:2229: in __init__
2021-03-15T22:53:22.6186643Z train_set.construct()
2021-03-15T22:53:22.6187415Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1468: in construct
2021-03-15T22:53:22.6188203Z self._lazy_init(self.data, label=self.label,
2021-03-15T22:53:22.6189025Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1270: in _lazy_init
2021-03-15T22:53:22.6189641Z self.__init_from_np2d(data, params_str, ref_dataset)
2021-03-15T22:53:22.6190480Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1312: in __init_from_np2d
2021-03-15T22:53:22.6191162Z _safe_call(_LIB.LGBM_DatasetCreateFromMat(
2021-03-15T22:53:22.6191639Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-03-15T22:53:22.6191868Z
2021-03-15T22:53:22.6192516Z > raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
2021-03-15T22:53:22.6193250Z E lightgbm.basic.LightGBMError: Socket send error, code: 104
2021-03-15T22:53:22.6193719Z
2021-03-15T22:53:22.6194402Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:110: LightGBMError
2021-03-15T22:53:22.6195296Z ----------------------------- Captured stdout call -----------------------------
2021-03-15T22:53:22.6195841Z Finding random open ports for workers
2021-03-15T22:53:22.6196338Z [LightGBM] [Warning] Set TCP_NODELAY failed
2021-03-15T22:53:22.6197015Z [LightGBM] [Info] Trying to bind port 53263...
2021-03-15T22:53:22.6197580Z [LightGBM] [Info] Binding port 53263 succeeded
2021-03-15T22:53:22.6198267Z [LightGBM] [Warning] Set TCP_NODELAY failed
2021-03-15T22:53:22.6198777Z [LightGBM] [Info] Listening...
2021-03-15T22:53:22.6199312Z [LightGBM] [Warning] Set TCP_NODELAY failed
2021-03-15T22:53:22.6199808Z [LightGBM] [Info] Connected to rank 1
2021-03-15T22:53:22.6200445Z [LightGBM] [Info] Local rank: 0, total number of machines: 2
2021-03-15T22:53:22.6201400Z [LightGBM] [Warning] num_threads is set=1, n_jobs=-1 will be ignored. Current value: num_threads=1
2021-03-15T22:53:22.6202249Z [LightGBM] [Warning] Set TCP_NODELAY failed
2021-03-15T22:53:22.6202747Z [LightGBM] [Info] Trying to bind port 53263...
2021-03-15T22:53:22.6203241Z [LightGBM] [Info] Binding port 53263 succeeded
2021-03-15T22:53:22.6203857Z [LightGBM] [Warning] Set TCP_NODELAY failed
2021-03-15T22:53:22.6204356Z [LightGBM] [Info] Listening...
2021-03-15T22:53:22.6204910Z [LightGBM] [Warning] Set TCP_NODELAY failed
2021-03-15T22:53:22.6205402Z [LightGBM] [Info] Connected to rank 1
2021-03-15T22:53:22.6205984Z [LightGBM] [Info] Local rank: 0, total number of machines: 2
2021-03-15T22:53:22.6206926Z [LightGBM] [Warning] num_threads is set=1, n_jobs=-1 will be ignored. Current value: num_threads=1
2021-03-15T22:53:22.6208005Z ----------------------------- Captured stderr call -----------------------------
2021-03-15T22:53:22.6208611Z [LightGBM] [Fatal] Socket send error, code: 104
2021-03-15T22:53:22.6209356Z distributed.worker - WARNING - Compute Failed
2021-03-15T22:53:22.6209936Z Function: _train_part
2021-03-15T22:53:22.6210283Z args: ()
2021-03-15T22:53:22.6212943Z kwargs: {'model_factory': <class 'lightgbm.sklearn.LGBMClassifier'>, 'params': {'boosting_type': 'gbdt', 'class_weight': None, 'colsample_bytree': 1.0, 'importance_type': 'split', 'learning_rate': 0.1, 'max_depth': -1, 'min_child_samples': 20, 'min_child_weight': 0.001, 'min_split_gain': 0.0, 'n_estimators': 1, 'num_leaves': 2, 'objective': None, 'random_state': None, 'reg_alpha': 0.0, 'reg_lambda': 0.0, 'silent': True, 'subsample': 1.0, 'subsample_for_bin': 200000, 'subsample_freq': 0, 'time_out': 5, 'tree_learner': 'data', 'num_threads': 1, 'machines': '127.0.0.1:53263,127.0.0.1:53263', 'local_listen_port': 53263, 'num_machines': 2}, 'list_of_parts': [{'data': array([[ 4.79103195, 3.09061255],
2021-03-15T22:53:22.6214882Z [ 4.91540212, 4.32875111],
2021-03-15T22:53:22.6215384Z [-2.13422549, 4.47383292],
2021-03-15T22:53:22.6215916Z [-3.78590626, 2.75426122],
2021-03-15T22:53:22.6216242Z [ 3.96417396, 5.56464366],
2021-03-15T22:53:22.6216820Z [-4.84679372, 2.48515278],
2021-03-15T22:53:22.6217147Z [ 5.8861859 , 4.17457781],
2021-03-15T22:53:22.6217627Z [-3.47805843, 4.29698467],
2021-03-15T22:53:22.6218152Z [-4.46341769, -4.46572975],
2021-03-15T22:53:22.6218606Z [-5.0708925 ,
2021-03-15T22:53:22.6219299Z Exception: LightGBMError('Socket send error, code: 104')
2021-03-15T22:53:22.6219754Z
2021-03-15T22:53:22.6220165Z [LightGBM] [Fatal] Socket send error, code: 104
2021-03-15T22:53:22.6220935Z distributed.worker - WARNING - Compute Failed
2021-03-15T22:53:22.6221506Z Function: _train_part
2021-03-15T22:53:22.6221856Z args: ()
2021-03-15T22:53:22.6224466Z kwargs: {'model_factory': <class 'lightgbm.sklearn.LGBMClassifier'>, 'params': {'boosting_type': 'gbdt', 'class_weight': None, 'colsample_bytree': 1.0, 'importance_type': 'split', 'learning_rate': 0.1, 'max_depth': -1, 'min_child_samples': 20, 'min_child_weight': 0.001, 'min_split_gain': 0.0, 'n_estimators': 1, 'num_leaves': 2, 'objective': None, 'random_state': None, 'reg_alpha': 0.0, 'reg_lambda': 0.0, 'silent': True, 'subsample': 1.0, 'subsample_for_bin': 200000, 'subsample_freq': 0, 'time_out': 5, 'tree_learner': 'data', 'num_threads': 1, 'machines': '127.0.0.1:53263,127.0.0.1:53263', 'local_listen_port': 53263, 'num_machines': 2}, 'list_of_parts': [{'data': array([[-3.52676238, 3.92717109],
2021-03-15T22:53:22.6226811Z [-4.83921752, -4.30921238],
2021-03-15T22:53:22.6227139Z [ 2.08122878, 3.97348612],
2021-03-15T22:53:22.6316466Z [-4.47917424, -4.18565898],
2021-03-15T22:53:22.6316996Z [-4.97468167, 4.7870846 ],
2021-03-15T22:53:22.6317441Z [-3.17745509, -5.22084365],
2021-03-15T22:53:22.6317718Z [ 4.25755039, 3.92555408],
2021-03-15T22:53:22.6318142Z [-4.68002472, 4.2322537 ],
2021-03-15T22:53:22.6318562Z [-2.53435123, -4.2257763 ],
2021-03-15T22:53:22.6318961Z [-4.90802408, -
2021-03-15T22:53:22.6319756Z Exception: LightGBMError('Socket send error, code: 104')
2021-03-15T22:53:22.6320133Z
2021-03-15T22:53:22.6320472Z =============================== warnings summary ===============================
2021-03-15T22:53:22.6321004Z tests/python_package_test/test_basic.py::test_basic
2021-03-15T22:53:22.6321612Z tests/python_package_test/test_engine.py::test_reference_chain
2021-03-15T22:53:22.6322265Z tests/python_package_test/test_engine.py::test_init_with_subset
2021-03-15T22:53:22.6322869Z tests/python_package_test/test_engine.py::test_fpreproc
2021-03-15T22:53:22.6323534Z tests/python_package_test/test_engine.py::test_dataset_params_with_reference
2021-03-15T22:53:22.6324697Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1433: UserWarning: Overriding the parameters from Reference Dataset.
2021-03-15T22:53:22.6325770Z _log_warning('Overriding the parameters from Reference Dataset.')
2021-03-15T22:53:22.6326177Z
2021-03-15T22:53:22.6326712Z tests/python_package_test/test_basic.py::test_add_features_equal_data_on_alternating_used_unused
2021-03-15T22:53:22.6327502Z tests/python_package_test/test_basic.py::test_add_features_same_booster_behaviour
2021-03-15T22:53:22.6328181Z tests/python_package_test/test_engine.py::test_sliced_data
2021-03-15T22:53:22.6328819Z tests/python_package_test/test_engine.py::test_monotone_penalty_max
2021-03-15T22:53:22.6329474Z tests/python_package_test/test_engine.py::test_forced_bins
2021-03-15T22:53:22.6330768Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:448: UserWarning: Usage of np.ndarray subset (sliced data) is not recommended due to it will double the peak memory cost in LightGBM.
2021-03-15T22:53:22.6331833Z _log_warning("Usage of np.ndarray subset (sliced data) is not recommended "
2021-03-15T22:53:22.6332231Z
2021-03-15T22:53:22.6332787Z tests/python_package_test/test_basic.py::test_add_features_equal_data_on_alternating_used_unused
2021-03-15T22:53:22.6333580Z tests/python_package_test/test_basic.py::test_add_features_same_booster_behaviour
2021-03-15T22:53:22.6334343Z tests/python_package_test/test_basic.py::test_add_features_from_different_sources
2021-03-15T22:53:22.6335603Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:2129: UserWarning: Cannot add features from NoneType type of raw data to NoneType type of raw data.
2021-03-15T22:53:22.6336520Z Set free_raw_data=False when construct Dataset to avoid this
2021-03-15T22:53:22.6336989Z _log_warning(err_msg)
2021-03-15T22:53:22.6337227Z
2021-03-15T22:53:22.6337786Z tests/python_package_test/test_basic.py::test_add_features_equal_data_on_alternating_used_unused
2021-03-15T22:53:22.6338582Z tests/python_package_test/test_basic.py::test_add_features_same_booster_behaviour
2021-03-15T22:53:22.6339334Z tests/python_package_test/test_basic.py::test_add_features_from_different_sources
2021-03-15T22:53:22.6340431Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:2131: UserWarning: Reseting categorical features.
2021-03-15T22:53:22.6341274Z You can set new categorical features via ``set_categorical_feature`` method
2021-03-15T22:53:22.6341930Z _log_warning("Reseting categorical features.\n"
2021-03-15T22:53:22.6342271Z
2021-03-15T22:53:22.6342767Z tests/python_package_test/test_basic.py::test_add_features_from_different_sources
2021-03-15T22:53:22.6343972Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:2129: UserWarning: Cannot add features from list type of raw data to ndarray type of raw data.
2021-03-15T22:53:22.6344709Z Freeing raw data
2021-03-15T22:53:22.6345054Z _log_warning(err_msg)
2021-03-15T22:53:22.6345293Z
2021-03-15T22:53:22.6345925Z tests/python_package_test/test_basic.py::test_add_features_from_different_sources
2021-03-15T22:53:22.6347390Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:2129: UserWarning: Cannot add features from list type of raw data to csr_matrix type of raw data.
2021-03-15T22:53:22.6348144Z Freeing raw data
2021-03-15T22:53:22.6348486Z _log_warning(err_msg)
2021-03-15T22:53:22.6348852Z
2021-03-15T22:53:22.6349350Z tests/python_package_test/test_basic.py::test_add_features_from_different_sources
2021-03-15T22:53:22.6350596Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:2129: UserWarning: Cannot add features from list type of raw data to DataFrame type of raw data.
2021-03-15T22:53:22.6351341Z Freeing raw data
2021-03-15T22:53:22.6351700Z _log_warning(err_msg)
2021-03-15T22:53:22.6351923Z
2021-03-15T22:53:22.6352373Z tests/python_package_test/test_consistency.py: 10 warnings
2021-03-15T22:53:22.6353478Z /root/.local/lib/python3.8/site-packages/lightgbm/engine.py:148: UserWarning: Found `num_trees` in params. Will use it instead of argument
2021-03-15T22:53:22.6354375Z _log_warning("Found `{}` in params. Will use it instead of argument".format(alias))
2021-03-15T22:53:22.6354760Z
2021-03-15T22:53:22.6355212Z tests/python_package_test/test_consistency.py: 10 warnings
2021-03-15T22:53:22.6356293Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1222: UserWarning: data keyword has been found in `params` and will be ignored.
2021-03-15T22:53:22.6357211Z Please use data argument of the Dataset constructor to pass this parameter.
2021-03-15T22:53:22.6358074Z _log_warning('{0} keyword has been found in `params` and will be ignored.\n'
2021-03-15T22:53:22.6358456Z
2021-03-15T22:53:22.6358864Z tests/python_package_test/test_dask.py: 188 warnings
2021-03-15T22:53:22.6359809Z /root/.local/lib/python3.8/site-packages/lightgbm/dask.py:285: UserWarning: Parameter n_jobs will be ignored.
2021-03-15T22:53:22.6360577Z _log_warning(f"Parameter {param_alias} will be ignored.")
2021-03-15T22:53:22.6360926Z
2021-03-15T22:53:22.6361432Z tests/python_package_test/test_dask.py::test_training_does_not_fail_on_port_conflicts
2021-03-15T22:53:22.6362169Z tests/python_package_test/test_dask.py::test_training_does_not_fail_on_port_conflicts
2021-03-15T22:53:22.6362907Z tests/python_package_test/test_dask.py::test_training_does_not_fail_on_port_conflicts
2021-03-15T22:53:22.6363635Z tests/python_package_test/test_dask.py::test_training_does_not_fail_on_port_conflicts
2021-03-15T22:53:22.6364730Z /root/.local/lib/python3.8/site-packages/lightgbm/dask.py:285: UserWarning: Parameter num_threads will be ignored.
2021-03-15T22:53:22.6365512Z _log_warning(f"Parameter {param_alias} will be ignored.")
2021-03-15T22:53:22.6365860Z
2021-03-15T22:53:22.6367162Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-binary-classification-pickle]
2021-03-15T22:53:22.6368672Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6369454Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6369972Z Hosting the HTTP server on port 46153 instead
2021-03-15T22:53:22.6370394Z warnings.warn(
2021-03-15T22:53:22.6370639Z
2021-03-15T22:53:22.6371892Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-binary-classification-joblib]
2021-03-15T22:53:22.6373796Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-binary-classification-joblib]
2021-03-15T22:53:22.6375296Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6376092Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6376586Z Hosting the HTTP server on port 40445 instead
2021-03-15T22:53:22.6377134Z warnings.warn(
2021-03-15T22:53:22.6377378Z
2021-03-15T22:53:22.6378746Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-binary-classification-cloudpickle]
2021-03-15T22:53:22.6380745Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-multiclass-classification-joblib]
2021-03-15T22:53:22.6382437Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6383227Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6383739Z Hosting the HTTP server on port 41447 instead
2021-03-15T22:53:22.6384157Z warnings.warn(
2021-03-15T22:53:22.6384402Z
2021-03-15T22:53:22.6385721Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-multiclass-classification-pickle]
2021-03-15T22:53:22.6387465Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6388253Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6388766Z Hosting the HTTP server on port 37785 instead
2021-03-15T22:53:22.6389186Z warnings.warn(
2021-03-15T22:53:22.6389429Z
2021-03-15T22:53:22.6390831Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-multiclass-classification-cloudpickle]
2021-03-15T22:53:22.6392479Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6393263Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6393776Z Hosting the HTTP server on port 44229 instead
2021-03-15T22:53:22.6394195Z warnings.warn(
2021-03-15T22:53:22.6394437Z
2021-03-15T22:53:22.6395574Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-regression-pickle]
2021-03-15T22:53:22.6396964Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6397770Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6398284Z Hosting the HTTP server on port 36047 instead
2021-03-15T22:53:22.6398709Z warnings.warn(
2021-03-15T22:53:22.6398952Z
2021-03-15T22:53:22.6400080Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-regression-joblib]
2021-03-15T22:53:22.6401460Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6402238Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6402753Z Hosting the HTTP server on port 41291 instead
2021-03-15T22:53:22.6403171Z warnings.warn(
2021-03-15T22:53:22.6403418Z
2021-03-15T22:53:22.6404616Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-regression-cloudpickle]
2021-03-15T22:53:22.6406035Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6406831Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6407334Z Hosting the HTTP server on port 44011 instead
2021-03-15T22:53:22.6407753Z warnings.warn(
2021-03-15T22:53:22.6407996Z
2021-03-15T22:53:22.6409106Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-ranking-pickle]
2021-03-15T22:53:22.6410444Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6411236Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6411853Z Hosting the HTTP server on port 34579 instead
2021-03-15T22:53:22.6412284Z warnings.warn(
2021-03-15T22:53:22.6412528Z
2021-03-15T22:53:22.6413658Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-ranking-joblib]
2021-03-15T22:53:22.6414991Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6415900Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6416395Z Hosting the HTTP server on port 41603 instead
2021-03-15T22:53:22.6416813Z warnings.warn(
2021-03-15T22:53:22.6417057Z
2021-03-15T22:53:22.6418236Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-ranking-cloudpickle]
2021-03-15T22:53:22.6419617Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6420412Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6420908Z Hosting the HTTP server on port 41269 instead
2021-03-15T22:53:22.6421327Z warnings.warn(
2021-03-15T22:53:22.6421570Z
2021-03-15T22:53:22.6422854Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-binary-classification-pickle]
2021-03-15T22:53:22.6424368Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6425159Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6425658Z Hosting the HTTP server on port 42025 instead
2021-03-15T22:53:22.6426094Z warnings.warn(
2021-03-15T22:53:22.6426448Z
2021-03-15T22:53:22.6427842Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-binary-classification-cloudpickle]
2021-03-15T22:53:22.6429421Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6430220Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6430716Z Hosting the HTTP server on port 40825 instead
2021-03-15T22:53:22.6431151Z warnings.warn(
2021-03-15T22:53:22.6431837Z
2021-03-15T22:53:22.6433351Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-multiclass-classification-pickle]
2021-03-15T22:53:22.6434929Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6435719Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6436219Z Hosting the HTTP server on port 39257 instead
2021-03-15T22:53:22.6436649Z warnings.warn(
2021-03-15T22:53:22.6436875Z
2021-03-15T22:53:22.6438223Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-multiclass-classification-joblib]
2021-03-15T22:53:22.6439779Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6440570Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6441071Z Hosting the HTTP server on port 33183 instead
2021-03-15T22:53:22.6441509Z warnings.warn(
2021-03-15T22:53:22.6441736Z
2021-03-15T22:53:22.6443165Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-multiclass-classification-cloudpickle]
2021-03-15T22:53:22.6444813Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6445615Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6446109Z Hosting the HTTP server on port 35323 instead
2021-03-15T22:53:22.6446546Z warnings.warn(
2021-03-15T22:53:22.6446934Z
2021-03-15T22:53:22.6448141Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-regression-pickle]
2021-03-15T22:53:22.6449521Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6450312Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6450918Z Hosting the HTTP server on port 39173 instead
2021-03-15T22:53:22.6451352Z warnings.warn(
2021-03-15T22:53:22.6451579Z
2021-03-15T22:53:22.6452752Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-regression-joblib]
2021-03-15T22:53:22.6454123Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6454922Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6455424Z Hosting the HTTP server on port 41287 instead
2021-03-15T22:53:22.6455857Z warnings.warn(
2021-03-15T22:53:22.6456100Z
2021-03-15T22:53:22.6457293Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-regression-cloudpickle]
2021-03-15T22:53:22.6458720Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6459522Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6460018Z Hosting the HTTP server on port 33235 instead
2021-03-15T22:53:22.6460452Z warnings.warn(
2021-03-15T22:53:22.6460697Z
2021-03-15T22:53:22.6461800Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-ranking-pickle]
2021-03-15T22:53:22.6463154Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6463938Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6464433Z Hosting the HTTP server on port 33147 instead
2021-03-15T22:53:22.6464867Z warnings.warn(
2021-03-15T22:53:22.6465110Z
2021-03-15T22:53:22.6466207Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-ranking-joblib]
2021-03-15T22:53:22.6467735Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6468518Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6469018Z Hosting the HTTP server on port 32887 instead
2021-03-15T22:53:22.6469451Z warnings.warn(
2021-03-15T22:53:22.6469694Z
2021-03-15T22:53:22.6470855Z tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[False-ranking-cloudpickle]
2021-03-15T22:53:22.6472264Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.
2021-03-15T22:53:22.6473044Z Perhaps you already have a cluster running?
2021-03-15T22:53:22.6473540Z Hosting the HTTP server on port 42065 instead
2021-03-15T22:53:22.6473974Z warnings.warn(
2021-03-15T22:53:22.6474218Z
2021-03-15T22:53:22.6474618Z tests/python_package_test/test_dask.py::test_errors
2021-03-15T22:53:22.6475629Z /root/.local/lib/python3.8/site-packages/lightgbm/dask.py:312: RuntimeWarning: coroutine '_wait' was never awaited
2021-03-15T22:53:22.6476278Z wait(parts)
2021-03-15T22:53:22.6476494Z
2021-03-15T22:53:22.6476889Z tests/python_package_test/test_dask.py::test_errors
2021-03-15T22:53:22.6478217Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/distributed/scheduler.py:1219: RuntimeWarning: coroutine 'PooledRPCCall.__getattr__.<locals>.send_recv_from_rpc' was never awaited
2021-03-15T22:53:22.6479176Z self._annotations = {}
2021-03-15T22:53:22.6479428Z
2021-03-15T22:53:22.6479961Z tests/python_package_test/test_engine.py::test_binary
2021-03-15T22:53:22.6481112Z /root/.local/lib/python3.8/site-packages/lightgbm/engine.py:148: UserWarning: Found `num_iteration` in params. Will use it instead of argument
2021-03-15T22:53:22.6482012Z _log_warning("Found `{}` in params. Will use it instead of argument".format(alias))
2021-03-15T22:53:22.6482412Z
2021-03-15T22:53:22.6482870Z tests/python_package_test/test_engine.py::test_pandas_categorical
2021-03-15T22:53:22.6483640Z tests/python_package_test/test_engine.py::test_linear_trees
2021-03-15T22:53:22.6484269Z tests/python_package_test/test_engine.py::test_save_and_load_linear
2021-03-15T22:53:22.6485394Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1705: UserWarning: categorical_feature in Dataset is overridden.
2021-03-15T22:53:22.6486145Z New categorical_feature is [0]
2021-03-15T22:53:22.6486869Z _log_warning('categorical_feature in Dataset is overridden.\n'
2021-03-15T22:53:22.6487258Z
2021-03-15T22:53:22.6487721Z tests/python_package_test/test_engine.py::test_pandas_categorical
2021-03-15T22:53:22.6488803Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1705: UserWarning: categorical_feature in Dataset is overridden.
2021-03-15T22:53:22.6489708Z New categorical_feature is ['A']
2021-03-15T22:53:22.6490414Z _log_warning('categorical_feature in Dataset is overridden.\n'
2021-03-15T22:53:22.6490805Z
2021-03-15T22:53:22.6491279Z tests/python_package_test/test_engine.py::test_pandas_categorical
2021-03-15T22:53:22.6492355Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1705: UserWarning: categorical_feature in Dataset is overridden.
2021-03-15T22:53:22.6493276Z New categorical_feature is ['A', 'B', 'C', 'D']
2021-03-15T22:53:22.6494014Z _log_warning('categorical_feature in Dataset is overridden.\n'
2021-03-15T22:53:22.6494400Z
2021-03-15T22:53:22.6494856Z tests/python_package_test/test_engine.py::test_pandas_categorical
2021-03-15T22:53:22.6495953Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1705: UserWarning: categorical_feature in Dataset is overridden.
2021-03-15T22:53:22.6496882Z New categorical_feature is ['A', 'B', 'C', 'D', 'E']
2021-03-15T22:53:22.6497606Z _log_warning('categorical_feature in Dataset is overridden.\n'
2021-03-15T22:53:22.6498025Z
2021-03-15T22:53:22.6498507Z tests/python_package_test/test_engine.py::test_pandas_categorical
2021-03-15T22:53:22.6499601Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1705: UserWarning: categorical_feature in Dataset is overridden.
2021-03-15T22:53:22.6500366Z New categorical_feature is []
2021-03-15T22:53:22.6501067Z _log_warning('categorical_feature in Dataset is overridden.\n'
2021-03-15T22:53:22.6501453Z
2021-03-15T22:53:22.6501889Z tests/python_package_test/test_engine.py::test_pandas_sparse
2021-03-15T22:53:22.6502538Z tests/python_package_test/test_sklearn.py::test_pandas_sparse
2021-03-15T22:53:22.6504015Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/pandas/core/generic.py:5673: PerformanceWarning: Concatenating sparse arrays with multiple fill values: '[0, nan, False]'. Picking the first and converting the rest.
2021-03-15T22:53:22.6505200Z return self._mgr.as_array(transpose=self._AXIS_REVERSED)
2021-03-15T22:53:22.6505560Z
2021-03-15T22:53:22.6506031Z tests/python_package_test/test_engine.py::test_int32_max_sparse_contribs
2021-03-15T22:53:22.6507687Z /root/miniconda/envs/test-env/lib/python3.8/site-packages/scipy/sparse/_index.py:82: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2021-03-15T22:53:22.6508752Z self._set_intXint(row, col, x.flat[0])
2021-03-15T22:53:22.6509033Z
2021-03-15T22:53:22.6509466Z tests/python_package_test/test_engine.py::test_init_with_subset
2021-03-15T22:53:22.6510475Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1959: UserWarning: Cannot subset str type of raw data.
2021-03-15T22:53:22.6511148Z Returning original raw data
2021-03-15T22:53:22.6511746Z _log_warning("Cannot subset {} type of raw data.\n"
2021-03-15T22:53:22.6512059Z
2021-03-15T22:53:22.6512542Z tests/python_package_test/test_engine.py::test_monotone_constraints
2021-03-15T22:53:22.6513215Z tests/python_package_test/test_engine.py::test_monotone_penalty
2021-03-15T22:53:22.6513892Z tests/python_package_test/test_engine.py::test_monotone_penalty_max
2021-03-15T22:53:22.6514663Z tests/python_package_test/test_engine.py::test_get_split_value_histogram
2021-03-15T22:53:22.6515369Z tests/python_package_test/test_sklearn.py::test_pandas_categorical
2021-03-15T22:53:22.6516041Z tests/python_package_test/test_sklearn.py::test_pandas_categorical
2021-03-15T22:53:22.6516731Z tests/python_package_test/test_sklearn.py::test_pandas_categorical
2021-03-15T22:53:22.6517403Z tests/python_package_test/test_sklearn.py::test_pandas_categorical
2021-03-15T22:53:22.6518089Z tests/python_package_test/test_sklearn.py::test_pandas_categorical
2021-03-15T22:53:22.6519185Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1702: UserWarning: Using categorical_feature in Dataset.
2021-03-15T22:53:22.6520136Z _log_warning('Using categorical_feature in Dataset.')
2021-03-15T22:53:22.6520487Z
2021-03-15T22:53:22.6520967Z tests/python_package_test/test_engine.py::test_dataset_update_params
2021-03-15T22:53:22.6522137Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:1222: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.
2021-03-15T22:53:22.6523181Z Please use categorical_feature argument of the Dataset constructor to pass this parameter.
2021-03-15T22:53:22.6524111Z _log_warning('{0} keyword has been found in `params` and will be ignored.\n'
2021-03-15T22:53:22.6524495Z
2021-03-15T22:53:22.6524936Z tests/python_package_test/test_plotting.py::test_plot_metrics
2021-03-15T22:53:22.6526034Z /root/.local/lib/python3.8/site-packages/lightgbm/plotting.py:328: UserWarning: More than one metric available, picking one to plot.
2021-03-15T22:53:22.6526908Z _log_warning("More than one metric available, picking one to plot.")
2021-03-15T22:53:22.6527287Z
2021-03-15T22:53:22.6527834Z tests/python_package_test/test_sklearn.py::test_binary_classification_with_custom_objective
2021-03-15T22:53:22.6529224Z /root/.local/lib/python3.8/site-packages/lightgbm/sklearn.py:922: UserWarning: Cannot compute class probabilities or labels due to the usage of customized objective function.
2021-03-15T22:53:22.6530136Z Returning raw scores instead.
2021-03-15T22:53:22.6530683Z _log_warning("Cannot compute class probabilities or labels "
2021-03-15T22:53:22.6531060Z
2021-03-15T22:53:22.6531466Z tests/python_package_test/test_sklearn.py: 12 warnings
2021-03-15T22:53:22.6532481Z /root/.local/lib/python3.8/site-packages/lightgbm/basic.py:739: UserWarning: Converting data to scipy sparse matrix.
2021-03-15T22:53:22.6533422Z _log_warning('Converting data to scipy sparse matrix.')
2021-03-15T22:53:22.6533772Z
2021-03-15T22:53:22.6534238Z tests/python_package_test/test_utilities.py::test_register_logger
2021-03-15T22:53:22.6535605Z /root/.local/lib/python3.8/site-packages/lightgbm/plotting.py:357: UserWarning: Attempting to set identical bottom == top == 1.0 results in singular transformations; automatically expanding.
2021-03-15T22:53:22.6536521Z ax.set_ylim(ylim)
2021-03-15T22:53:22.6536748Z
2021-03-15T22:53:22.6537418Z -- Docs: https://docs.pytest.org/en/stable/warnings.html
2021-03-15T22:53:22.6538045Z =========================== short test summary info ============================
2021-03-15T22:53:22.6539514Z FAILED ../tests/python_package_test/test_dask.py::test_model_and_local_version_are_picklable_whether_or_not_client_set_explicitly[True-multiclass-classification-joblib]
2021-03-15T22:53:22.6540661Z = 1 failed, 385 passed, 7 skipped, 2 xfailed, 294 warnings in 744.12s (0:12:24) =
2021-03-15T22:53:26.4930107Z ##[error]Process completed with exit code 255.
2021-03-15T22:53:26.4959628Z Cleaning up orphan processes
Link: https://github.com/microsoft/LightGBM/runs/2116868032
cc @jameslamb
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 15 (1 by maintainers)
Just seen this error at
master:Logs:
For anyone subscribed to this issue, I THINK I may have found the root cause, and it might be possible to fix this without needing to mark some Dask tests as flaky.
https://github.com/microsoft/LightGBM/pull/4132
Adding for context here, error code
104means “Connection reset by peer” (link, also mentioned in https://github.com/microsoft/LightGBM/pull/3952#pullrequestreview-610403805).I think there is a root underlying problem to the flaky tests and to some of the instability @ffineis saw in #3952.
@StrikerRUS I’d like to explore these
pytestplugins that allow you to mark certain tests as flaky and tellpytestto retry them up tontimes on failure: https://docs.pytest.org/en/stable/flaky.html#plugins. Would you support a PR that adds something like that?I do think doing that might cover up some underlying reliability problems, but it would prevent the Dask tests from blocking PRs across this project, and we could then work on slowly removing the “flaky” markers as underlying problems are fixed.
+2 failures with
104error code today. It is getting frequent and annoying…UPD: +3
UPD: +4
I’m a little confused by the double-negative nature of
rerun_filter, but hopefully it can be used to excludes re-tries of failures that are notSocket recv error code: 104errors? If it can be used to target this one known error type, then test retries should really only end up comprising a small minority of the total test attempts, because it’s uncommon relative to the 60+ other passing tests. I dunno I think this is a good idea.