sherpa-ncnn: [Help wanted] sherpa-ncnn for android crash when using LSTM model
如何在安卓平台的sherpa-ncnn Demo中使用LSTM模型?
描述:
当我只是修改对应的ModelConfig时,修改部分代码如下方附录代码,sherpa-ncnn App闪退,看起来是在jni中Java_com_k2fsa_sherpa_ncnn_SherpaNcnn_new中调用sherpa_ncnn::Model::Create创建sherpa_ncnn::LstmModel::LstmModel时出现的问题。
推测:
是我在使用LSTM模型时必须要如下示图2中LOGE所述,添加SherpaMetaData 到encoder_xxx.ncnn.param中吗?我应该如何添加SherpaMetaData?
崩溃日志:


以下是我修改的SherpaNcnn.kt部分代码:
fun getModelConfig(type: Int, useGPU: Boolean): ModelConfig? {
when (type) {
//crash
1 -> {
val modelDir = "sherpa-ncnn-streaming-zipformer-bilingual-zh-en-2023-02-13"
return ModelConfig(
encoderParam = "$modelDir/encoder_jit_trace-pnnx.ncnn.param",
encoderBin = "$modelDir/encoder_jit_trace-pnnx.ncnn.bin",
decoderParam = "$modelDir/decoder_jit_trace-pnnx.ncnn.param",
decoderBin = "$modelDir/decoder_jit_trace-pnnx.ncnn.bin",
joinerParam = "$modelDir/joiner_jit_trace-pnnx.ncnn.param",
joinerBin = "$modelDir/joiner_jit_trace-pnnx.ncnn.bin",
tokens = "$modelDir/tokens.txt",
numThreads = 4,
useGPU = useGPU,
)
}
2 -> {
val modelDir = "sherpa-ncnn-conv-emformer-transducer-2022-12-08/v2"
return ModelConfig(
encoderParam = "$modelDir/encoder_jit_trace-pnnx-epoch-15-avg-3.ncnn.param",
encoderBin = "$modelDir/encoder_jit_trace-pnnx-epoch-15-avg-3.ncnn.bin",
decoderParam = "$modelDir/decoder_jit_trace-pnnx-epoch-15-avg-3.ncnn.param",
decoderBin = "$modelDir/decoder_jit_trace-pnnx-epoch-15-avg-3.ncnn.bin",
joinerParam = "$modelDir/joiner_jit_trace-pnnx-epoch-15-avg-3.ncnn.param",
joinerBin = "$modelDir/joiner_jit_trace-pnnx-epoch-15-avg-3.ncnn.bin",
tokens = "$modelDir/tokens.txt",
numThreads = 4,
useGPU = useGPU,
)
}
//crash
3 -> {
val modelDir = "sherpa-ncnn-lstm-transducer-small-2023-02-13"
return ModelConfig(
encoderParam = "$modelDir/encoder_jit_trace-pnnx.ncnn.param",
encoderBin = "$modelDir/encoder_jit_trace-pnnx.ncnn.bin",
decoderParam = "$modelDir/decoder_jit_trace-pnnx.ncnn.param",
decoderBin = "$modelDir/decoder_jit_trace-pnnx.ncnn.bin",
joinerParam = "$modelDir/joiner_jit_trace-pnnx.ncnn.param",
joinerBin = "$modelDir/joiner_jit_trace-pnnx.ncnn.bin",
tokens = "$modelDir/tokens.txt",
numThreads = 4,
useGPU = useGPU,
)
}
}
return null
}
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 30 (20 by maintainers)
针对低端设备,我们特别提供了一个文档 https://k2-fsa.github.io/sherpa/ncnn/pretrained_models/small-models.html
这个文档里面的模型,都是小模型,适合在低端设备上跑。
设备的计算性能,会影响推理速度。如果推理速度慢了, RTF > 1 的话,就会丢音频数据,从而影响准确度。
所以,在低端设备上,第一件事,就是使用 executable
sherpa-ncnn去测一下 RTF.注:
sherpa-ncnn跑完后,会自动输出 RTF 。RTF < 1 , 才可以正常使用。你也可以测试不同解码方法和不同线程数量的 RTF.我陆续会把所有模型都试一遍
希望后续可以解决lstm模型和zipformer模型在sherpa-ncnn-android上的问题
我本地试 非 vulkan 的版本,是没问题的 (可以加载模型,并且打开界面, lstm 模型的识别,现在应该有点问题, 可能是更新 ncnn 导致的)
你试的 conv-emformer 或者 zipformer 会有问题么.