sherpa-ncnn: [Help wanted] sherpa-ncnn for android crash when using LSTM model

如何在安卓平台的sherpa-ncnn Demo中使用LSTM模型?

描述:

当我只是修改对应的ModelConfig时,修改部分代码如下方附录代码,sherpa-ncnn App闪退,看起来是在jniJava_com_k2fsa_sherpa_ncnn_SherpaNcnn_new中调用sherpa_ncnn::Model::Create创建sherpa_ncnn::LstmModel::LstmModel时出现的问题。

推测:

是我在使用LSTM模型时必须要如下示图2中LOGE所述,添加SherpaMetaDataencoder_xxx.ncnn.param中吗?我应该如何添加SherpaMetaData

崩溃日志:

image

image

以下是我修改的SherpaNcnn.kt部分代码:

fun getModelConfig(type: Int, useGPU: Boolean): ModelConfig? {
    when (type) {
        //crash
        1 -> {
            val modelDir = "sherpa-ncnn-streaming-zipformer-bilingual-zh-en-2023-02-13"
            return ModelConfig(
                encoderParam = "$modelDir/encoder_jit_trace-pnnx.ncnn.param",
                encoderBin = "$modelDir/encoder_jit_trace-pnnx.ncnn.bin",
                decoderParam = "$modelDir/decoder_jit_trace-pnnx.ncnn.param",
                decoderBin = "$modelDir/decoder_jit_trace-pnnx.ncnn.bin",
                joinerParam = "$modelDir/joiner_jit_trace-pnnx.ncnn.param",
                joinerBin = "$modelDir/joiner_jit_trace-pnnx.ncnn.bin",
                tokens = "$modelDir/tokens.txt",
                numThreads = 4,
                useGPU = useGPU,
            )
        }
        2 -> {
            val modelDir = "sherpa-ncnn-conv-emformer-transducer-2022-12-08/v2"
            return ModelConfig(
                encoderParam = "$modelDir/encoder_jit_trace-pnnx-epoch-15-avg-3.ncnn.param",
                encoderBin = "$modelDir/encoder_jit_trace-pnnx-epoch-15-avg-3.ncnn.bin",
                decoderParam = "$modelDir/decoder_jit_trace-pnnx-epoch-15-avg-3.ncnn.param",
                decoderBin = "$modelDir/decoder_jit_trace-pnnx-epoch-15-avg-3.ncnn.bin",
                joinerParam = "$modelDir/joiner_jit_trace-pnnx-epoch-15-avg-3.ncnn.param",
                joinerBin = "$modelDir/joiner_jit_trace-pnnx-epoch-15-avg-3.ncnn.bin",
                tokens = "$modelDir/tokens.txt",
                numThreads = 4,
                useGPU = useGPU,
            )
        }
        //crash
        3 -> {
            val modelDir = "sherpa-ncnn-lstm-transducer-small-2023-02-13"
            return ModelConfig(
                encoderParam = "$modelDir/encoder_jit_trace-pnnx.ncnn.param",
                encoderBin = "$modelDir/encoder_jit_trace-pnnx.ncnn.bin",
                decoderParam = "$modelDir/decoder_jit_trace-pnnx.ncnn.param",
                decoderBin = "$modelDir/decoder_jit_trace-pnnx.ncnn.bin",
                joinerParam = "$modelDir/joiner_jit_trace-pnnx.ncnn.param",
                joinerBin = "$modelDir/joiner_jit_trace-pnnx.ncnn.bin",
                tokens = "$modelDir/tokens.txt",
                numThreads = 4,
                useGPU = useGPU,
            )
        }
    }
    return null
}

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 30 (20 by maintainers)

Most upvoted comments

针对低端设备,我们特别提供了一个文档 https://k2-fsa.github.io/sherpa/ncnn/pretrained_models/small-models.html

这个文档里面的模型,都是小模型,适合在低端设备上跑。

设备的计算性能,会影响推理速度。如果推理速度慢了, RTF > 1 的话,就会丢音频数据,从而影响准确度。

所以,在低端设备上,第一件事,就是使用 executable sherpa-ncnn 去测一下 RTF.

注:sherpa-ncnn 跑完后,会自动输出 RTF 。RTF < 1 , 才可以正常使用。你也可以测试不同解码方法和不同线程数量的 RTF.

我陆续会把所有模型都试一遍

希望后续可以解决lstm模型和zipformer模型在sherpa-ncnn-android上的问题

我本地试 非 vulkan 的版本,是没问题的 (可以加载模型,并且打开界面, lstm 模型的识别,现在应该有点问题, 可能是更新 ncnn 导致的)

你试的 conv-emformer 或者 zipformer 会有问题么.