catboost: CatboostError: catboost/cuda/cuda_lib/cuda_manager.cpp:199: Condition violated: `State == nullptr'

Problem: Trying to train model using GPU on google colab, installed cuda 9.1, while running fit() got this error

CatboostError                             Traceback (most recent call last)
<ipython-input-29-12dd739d7ba3> in <module>()
      1 
      2 print("Started Training")
----> 3 model.fit(train_pool)

/usr/local/lib/python3.6/dist-packages/catboost/core.py in fit(self, X, y, cat_features, sample_weight, baseline, use_best_model, eval_set, verbose, logging_level, plot, column_description, verbose_eval, metric_period, silent, early_stopping_rounds)
   2393         return self._fit(X, y, cat_features, None, sample_weight, None, None, None, None, baseline,
   2394                          use_best_model, eval_set, verbose, logging_level, plot, column_description,
-> 2395                          verbose_eval, metric_period, silent, early_stopping_rounds)
   2396 
   2397     def predict(self, data, ntree_start=0, ntree_end=0, thread_count=-1, verbose=None):

/usr/local/lib/python3.6/dist-packages/catboost/core.py in _fit(self, X, y, cat_features, pairs, sample_weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, use_best_model, eval_set, verbose, logging_level, plot, column_description, verbose_eval, metric_period, silent, early_stopping_rounds)
   1070 
   1071         with log_fixup():
-> 1072             self._train(train_pool, eval_sets, params, allow_clear_pool)
   1073 
   1074         if (not self._object._has_leaf_weights_in_model()) and allow_clear_pool:

/usr/local/lib/python3.6/dist-packages/catboost/core.py in _train(self, train_pool, test_pool, params, allow_clear_pool)
    816 
    817     def _train(self, train_pool, test_pool, params, allow_clear_pool):
--> 818         self._object._train(train_pool, test_pool, params, allow_clear_pool)
    819         setattr(self, '_is_fitted', True)
    820         setattr(self, '_random_seed', self._object._get_random_seed())

_catboost.pyx in _catboost._CatBoost._train()

_catboost.pyx in _catboost._CatBoost._train()

CatboostError: catboost/cuda/cuda_lib/cuda_manager.cpp:199: Condition violated: `State == nullptr'

catboost version: 0.9.1.1 Operating System: Linux (Google Colab)

GPU: K80

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 21 (8 by maintainers)

Most upvoted comments

Can you please try running the latest version 0.10? we have static linkage with CUDA now, and this should not happen