How to combine vad, asr, punc and nnlm models inside pipeline

[LauraGPT]

You could combine the VAD, ASR, NNLM and PUNC models freely in pipeline. The following is the demo of Paraformer-large.

The pipeline which only contains ASR model：

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
)

The pipeline which contains NNLM and ASR models：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    lm_model='damo/speech_transformer_lm_zh-cn-common-vocab8404-pytorch',
    lm_weight=0.15,
    beam_size=10,
)

The pipeline which contains VAD and ASR models：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
)

The pipeline which contains VAD, ASR and PUNC models：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
    punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',
)

The pipeline which contains VAD, ASR, PUNC and NNLM models：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
    punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',
    lm_model='damo/speech_transformer_lm_zh-cn-common-vocab8404-pytorch',
    lm_weight=0.15,
    beam_size=10,
)

[LauraGPT]

Notes：

For the Paraformer-large-long, the pipeline defaults to combine VAD, ASR and PUNC models. For example, the following pipeline decodes with VAD, ASR and PUNC models.

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
)

It equals to

    inference_pipeline = pipeline(
        task=Tasks.auto_speech_recognition,
        model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
        vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
        punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',
    )

If you want to disable the PUNC, just to set punc_model='':

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    punc_model='',
)

[mattzheng]

报错：TypeError: 'NoneType' object is not callable

[xiaomingnio]

对于读取本地模型，如
inference_pipeline = pipeline(
task=Tasks.auto_speech_recognition,
model='/root/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
vad_model='/root/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
punc_model='/root/damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',）
只有model会生效，vad_model和punc_model不生效，仍读取hub模型。

[LauraGPT]

Yeah, it is a bug. We would fix it.

[hildazzz]

长达7分钟的干净语音文件，使用：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
    punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',
)

没有输出time_stamp，这是为什么呢？sentences的输出也是空的。我希望得到时间戳和对应的文本内容（带有标点符号），谢谢。

[LauraGPT]

You could generate timestamp by:

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
)

[hildazzz]

RuntimeError: AutomaticSpeechRecognitionPipeline: cuDNN version incompatibility: PyTorch was compiled against (8, 5, 0) but found runtime version (8, 2, 1). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN.Looks like your LD_LIBRARY_PATH contains incompatible version of cudnnPlease either remove it from the path or install cudnn (8, 5, 0)

按照教程使用，报错如上，请问是哪里有问题？

[xiaomingnio]

You can use this command in you terminal-----> unset LD_LIBRARY_PATH