Speech-to-Text RNNT + KenLM#

Encoder model + RNNT loss + KenLM

This tutorial is available as an IPython notebook at malaya-speech/example/stt-transducer-model-lm.

This module is not language independent, so it not save to use on different languages. Pretrained models trained on hyperlocal languages.

[1]:

import malaya_speech
import numpy as np
from malaya_speech import Pipeline

List available RNNT model#

[2]:

malaya_speech.stt.available_transducer()

[2]:
Size (MB) Quantized Size (MB) WER CER WER-LM CER-LM Language
tiny-conformer 24.4 9.14 0.212811 0.081369 0.199683 0.077004 [malay]
small-conformer 49.2 18.1 0.198533 0.074495 0.185361 0.071143 [malay]
conformer 125 37.1 0.163602 0.058744 0.156182 0.05719 [malay]
large-conformer 404 107 0.156684 0.061971 0.148622 0.05901 [malay]
conformer-stack-2mixed 130 38.5 0.103608 0.050069 0.102911 0.050201 [malay, singlish]
conformer-stack-3mixed 130 38.5 0.234768 0.133944 0.229241 0.130702 [malay, singlish, mandarin]
small-conformer-singlish 49.2 18.1 0.087831 0.045686 0.087333 0.045317 [singlish]
conformer-singlish 125 37.1 0.077792 0.040362 0.077186 0.03987 [singlish]
large-conformer-singlish 404 107 0.070147 0.035872 0.069812 0.035723 [singlish]

Lower is better. Mixed models tested on different dataset.

Load RNNT model#

def deep_transducer(
    model: str = 'conformer', quantized: bool = False, **kwargs
):
    """
    Load Encoder-Transducer ASR model.

    Parameters
    ----------
    model : str, optional (default='conformer')
        Model architecture supported. Allowed values:

        * ``'tiny-conformer'`` - TINY size Google Conformer.
        * ``'small-conformer'`` - SMALL size Google Conformer.
        * ``'conformer'`` - BASE size Google Conformer.
        * ``'large-conformer'`` - LARGE size Google Conformer.
        * ``'conformer-stack-mixed'`` - BASE size Stacked Google Conformer for (Malay + Singlish) languages.
        * ``'conformer-stack-3mixed'`` - BASE size Stacked Google Conformer for (Malay + Singlish + Mandarin) languages.
        * ``'small-conformer-singlish'`` - SMALL size Google Conformer for singlish language.
        * ``'conformer-singlish'`` - BASE size Google Conformer for singlish language.
        * ``'large-conformer-singlish'`` - LARGE size Google Conformer for singlish language.

    quantized : bool, optional (default=False)
        if True, will load 8-bit quantized model.
        Quantized model not necessary faster, totally depends on the machine.

    Returns
    -------
    result : malaya_speech.model.tf.Transducer class
    """

[3]:

small_model = malaya_speech.stt.deep_transducer(model = 'small-conformer')
model = malaya_speech.stt.deep_transducer(model = 'conformer')

Load Quantized deep model#

To load 8-bit quantized model, simply pass quantized = True, default is False.

We can expect slightly accuracy drop from quantized model, and not necessary faster than normal 32-bit float model, totally depends on machine.

[4]:

quantized_small_model = malaya_speech.stt.deep_transducer(model = 'small-conformer', quantized = True)
quantized_model = malaya_speech.stt.deep_transducer(model = 'conformer', quantized = True)

WARNING:root:Load quantized model will cause accuracy drop.
WARNING:root:Load quantized model will cause accuracy drop.

Load sample#

[5]:

ceramah, sr = malaya_speech.load('speech/khutbah/wadi-annuar.wav')
record1, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-36-06_294832.wav')
record2, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-40-56_929661.wav')
shafiqah_idayu, sr = malaya_speech.load('speech/example-speaker/shafiqah-idayu.wav')
mas_aisyah, sr = malaya_speech.load('speech/example-speaker/mas-aisyah.wav')
khalil, sr = malaya_speech.load('speech/example-speaker/khalil-nooh.wav')

[6]:

import IPython.display as ipd

ipd.Audio(ceramah, rate = sr)

[6]:

As we can hear, the speaker speaks in kedahan dialects plus some arabic words, let see how good our model is.

[7]:

ipd.Audio(record1, rate = sr)

[7]:

[8]:

ipd.Audio(record2, rate = sr)

[8]:

[9]:

ipd.Audio(shafiqah_idayu, rate = sr)

[9]:

[10]:

ipd.Audio(mas_aisyah, rate = sr)

[10]:

[11]:

ipd.Audio(khalil, rate = sr)

[11]:

Install pyctcdecode#

From PYPI#

pip3 install pyctcdecode==0.1.0 pypi-kenlm==0.1.20210121

From source#

Check https://github.com/kensho-technologies/pyctcdecode how to build from source incase there is no available wheel for your operating system.

Building from source should only take a few minutes.

Load pyctcdecode#

I will use dump-combined for this example.

[12]:

import kenlm
from pyctcdecode.language_model import LanguageModel

[13]:

lm = malaya_speech.language_model.kenlm(model = 'dump-combined')

[14]:

kenlm_model = kenlm.Model(lm)
language_model = LanguageModel(kenlm_model, alpha = 0.01, beta = 0.5)

Predict using beam decoder language model#

def beam_decoder_lm(self, inputs, language_model,
                    beam_width: int = 5,
                    token_min_logp: float = -20.0,
                    beam_prune_logp: float = -5.0,
                    temperature: float = 0.0,
                    score_norm: bool = True):
    """
    Transcribe inputs using beam decoder + KenLM.

    Parameters
    ----------
    inputs: List[np.array]
        List[np.array] or List[malaya_speech.model.frame.Frame].
    language_model: pyctcdecode.language_model.LanguageModel
        pyctcdecode language model, load from `LanguageModel(kenlm_model, alpha = alpha, beta = beta)`.
    beam_width: int, optional (default=5)
        beam size for beam decoder.
    token_min_logp: float, optional (default=-20.0)
        minimum log probability to select a token.
    beam_prune_logp: float, optional (default=-5.0)
        filter candidates >= max score lm + `beam_prune_logp`.
    temperature: float, optional (default=0.0)
        apply temperature function for logits, can help for certain case,
        logits += -np.log(-np.log(uniform_noise_shape_logits)) * temperature
    score_norm: bool, optional (default=True)
        descending sort beam based on score / length of decoded.

    Returns
    -------
    result: List[str]
    """

[15]:

%%time

small_model.beam_decoder_lm([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil],
                          language_model)

CPU times: user 25.6 s, sys: 2.13 s, total: 27.7 s
Wall time: 21.7 s

[15]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allah maha ini',
 'helo nama saya pusing saya tak suka mandi ketat saya masak',
 'helo nama saya husin saya suka mandi saya mandi tetek hari',
 'nama saya syafiqah hidayah',
 'sebut perkataan uncle',
 'tolong sebut anti kata']

[16]:

%%time

model.beam_decoder_lm([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil],
                          language_model)

CPU times: user 33.5 s, sys: 3.34 s, total: 36.8 s
Wall time: 24.7 s

[16]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah maaf ini',
 'helo nama saya pusing saya tak suka mandi ke tak saya masam',
 'helo nama saya husin saya suka mandi saya mandi tiap tiap hari',
 'nama saya syafiqah idayu',
 'sebut perkataan angka',
 'tolong sebut antika']

RNNT model beam decoder language model not able to utilise batch processing, if feed a batch, it will process one by one.