rasa: Bug in system
I use Python3.5 And make such training example
{
"rasa_nlu_data": {
"common_examples": [
{
"text": "What category of assets does the American Century One Choice 2025 C fund hold?",
"entities": [],
"intent": "getFundAssetType"
},
{
"text": "Asset alloaction of American Century One Choice 2025 C",
"entities": [],
"intent": "getFundAssetType"
},
....
and got
/home/vshebuniayeu/anaconda3/lib/python3.5/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
'precision', 'predicted', average, warn_for)
Because of these bug, intent classification is working wrong
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 19 (12 by maintainers)
@vladimircape of course we are happy to help, but in order to receive help I believe @amn41 would like everyone to abide by the Code of Conduct. Seeing as this discussion topic isn’t progressing anywhere I am inclined to lock it, but I would like to provide one more chance the opportunity for us to assist you.
Rasa is bringing multiple tools together in order to offer functionality similar to API.ai and other “chatbot” providers. One of the tools they include is sklearn. The message you are seeing is actually just being passed by Rasa from sklearn metrics. Some more reading information if you’d like to dive in further.
This is not a bug, but is in fact a feature of the underlying tools telling you that something is not right with the model that has been generated.
@tmbo and my response is to say that: in the past whenever we have seen this warning, it was a strong indicator that the training data was insufficient.
If you’d like to share your training data with us either here, in a gist, or privately via e-mail we’d be happy to take a look. You may also have better luck using the
mitiepipeline which qualitatively requires less training data, but tends to be slower to train.As for Rasa itself, the GoLastMile team has managed to get 1500+ stars, 400+ unique cloners, and 14000+ viewers. I think they’re doing wonderful work.
@wrathagom If you are not “is not member of the company building RASA” why you answer for question which belongs to RASA.
Instead of a thousand words, you could have done testing during this time, but apparently you have such a business model, a mass of crush
@vladimircape I would just remind you of our code of conduct - please refrain from statements like “It’s junior issues in Data Science”
@wrathagom You are not right "As Tom said above it appears that you are using too few training examples. In this case the model created from the training data will be of little to no use. Increase the amount of training data.
This isn’t a lookup, so using the same sentence doesn’t guarantee a perfect match. The sentence has to be processed by the model, and if the model wasn’t generated correctly (because of too few training examples) then it will not classify the intent correctly." First of all it’s depend on what kind of classifier you use, some of them required more training data ,some of them not. And if classifier is bad, increading training data will not work. It’s junior issues in Data Science. And you are talking only Statistical Models, if your system use Grammatical models, you don’t need a lot of data,
And you still don’t want to answer to my question, if you use statisticals models, where is your test of accuracy of prediction of intent, as usuall you should have training data, cv data, and test data, and based on test ,show in percentage what is the accuracy.Please provide it?
This just indicates that you don’t have enough training data. How do you experience the
"... intent classifcation is working wrong"?