Speech-to-Text CTC#

Encoder model + CTC loss

This tutorial is available as an IPython notebook at malaya-speech/example/stt-ctc-model.

This module is not language independent, so it not save to use on different languages. Pretrained models trained on hyperlocal languages.

This is an application of malaya-speech Pipeline, read more about malaya-speech Pipeline at malaya-speech/example/pipeline.

This interface deprecated, use HuggingFace interface instead.

[1]:

import malaya_speech
import numpy as np
from malaya_speech import Pipeline

`pyaudio` is not available, `malaya_speech.streaming.stream` is not able to use.

[2]:

import logging

logging.basicConfig(level=logging.INFO)

[3]:

import warnings
warnings.filterwarnings('default')

List available CTC model#

[4]:

malaya_speech.stt.ctc.available_transformer()

/home/husein/dev/malaya-speech/malaya_speech/stt/ctc.py:144: DeprecationWarning: `malaya.stt.ctc.available_transformer` is deprecated, use `malaya.stt.ctc.available_huggingface` instead
  warnings.warn(
INFO:malaya_speech.stt:for `malay-fleur102` language, tested on FLEURS102 `ms_my` test set, https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt
INFO:malaya_speech.stt:for `malay-malaya` language, tested on malaya-speech test set, https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt
INFO:malaya_speech.stt:for `singlish` language, tested on IMDA malaya-speech test set, https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt

[4]:
Size (MB) Quantized Size (MB) malay-malaya Language
hubert-conformer-tiny 36.6 10.3 {'WER': 0.238714008166, 'CER': 0.060899814, 'W... [malay]
hubert-conformer 115 31.1 {'WER': 0.2387140081, 'CER': 0.06089981404, 'W... [malay]
hubert-conformer-large 392 100 {'WER': 0.2203140421, 'CER': 0.0549270416, 'WE... [malay]

Google Speech-to-Text accuracy#

We tested on the same malay dataset to compare malaya-speech models and Google Speech-to-Text, check the notebook at benchmark-google-speech-malay-dataset.ipynb.

[5]:

malaya_speech.stt.google_accuracy

[5]:

{'malay-malaya': {'WER': 0.16477548774, 'CER': 0.05973209121},
 'malay-fleur102': {'WER': 0.109588779, 'CER': 0.047891527},
 'singlish': {'WER': 0.4941349, 'CER': 0.3026296}}

Load CTC model#

def transformer(
    model: str = 'hubert-conformer',
    quantized: bool = False,
    **kwargs,
):
    """
    Load Encoder-CTC ASR model.

    Parameters
    ----------
    model : str, optional (default='hubert-conformer')
        Check available models at `malaya_speech.stt.ctc.available_transformer()`.
    quantized : bool, optional (default=False)
        if True, will load 8-bit quantized model.
        Quantized model not necessary faster, totally depends on the machine.

    Returns
    -------
    result : malaya_speech.model.wav2vec.Wav2Vec2_CTC class
    """

[7]:

model = malaya_speech.stt.ctc.transformer(model = 'hubert-conformer-large')

Load Quantized deep model#

To load 8-bit quantized model, simply pass quantized = True, default is False.

We can expect slightly accuracy drop from quantized model, and not necessary faster than normal 32-bit float model, totally depends on machine.

[5]:

quantized_model = malaya_speech.stt.deep_ctc(model = 'hubert-conformer-large', quantized = True)

WARNING:root:Load quantized model will cause accuracy drop.

Load sample#

[6]:

ceramah, sr = malaya_speech.load('speech/khutbah/wadi-annuar.wav')
record1, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-36-06_294832.wav')
record2, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-40-56_929661.wav')

[8]:

import IPython.display as ipd

ipd.Audio(ceramah, rate = sr)

[8]:

As we can hear, the speaker speaks in kedahan dialects plus some arabic words, let see how good our model is.

[9]:

ipd.Audio(record1, rate = sr)

[9]:

[10]:

ipd.Audio(record2, rate = sr)

[10]:

Predict using greedy decoder#

def greedy_decoder(self, inputs):
    """
    Transcribe inputs using greedy decoder.

    Parameters
    ----------
    input: List[np.array]
        List[np.array] or List[malaya_speech.model.frame.Frame].

    Returns
    -------
    result: List[str]
    """

[9]:

%%time

model.greedy_decoder([ceramah, record1, record2])

CPU times: user 16.5 s, sys: 5.54 s, total: 22 s
Wall time: 4.2 s

[9]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah maini',
 'helo nama saya esin saya tak suka mandi ketak saya masak',
 'helo nama saya musin saya suka mandi saya mandi titiap hari']

[10]:

%%time

quantized_model.greedy_decoder([ceramah, record1, record2])

CPU times: user 16.7 s, sys: 5.5 s, total: 22.2 s
Wall time: 4.15 s

[10]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah maini',
 'helo nama saya esin saya tak suka mandi ketak saya masak',
 'helo nama saya musin saya suka mandi saya mandi titiap hari']

Predict using beam decoder#

def beam_decoder(self, inputs, beam_width: int = 100):
    """
    Transcribe inputs using beam decoder.

    Parameters
    ----------
    input: List[np.array]
        List[np.array] or List[malaya_speech.model.frame.Frame].
    beam_width: int, optional (default=100)
        beam size for beam decoder.

    Returns
    -------
    result: List[str]
    """

[11]:

%%time

model.beam_decoder([ceramah, record1, record2])

CPU times: user 26.9 s, sys: 11.8 s, total: 38.7 s
Wall time: 21.9 s

[11]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah ma ini',
 'helo nama saya esin saya tak suka mandi ketak saya masak',
 'helo nama saya musin saya suka mandi saya mandi titiap hari']

[12]:

%%time

quantized_model.beam_decoder([ceramah, record1, record2])

CPU times: user 26.5 s, sys: 11 s, total: 37.5 s
Wall time: 19.3 s

[12]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah ma ini',
 'helo nama saya esin saya tak suka mandi ketak saya masak',
 'helo nama saya musin saya suka mandi saya mandi titiap hari']

Predict logits#

def predict_logits(self, inputs, norm_func=softmax):
    """
    Predict logits from inputs.

    Parameters
    ----------
    input: List[np.array]
        List[np.array] or List[malaya_speech.model.frame.Frame].
    norm_func: Callable, optional (default=malaya.utils.activation.softmax)


    Returns
    -------
    result: List[np.array]
    """

[13]:

%%time

logits = model.predict_logits([ceramah, record1, record2])

CPU times: user 28.1 s, sys: 11.9 s, total: 40 s
Wall time: 22.6 s

[16]:

logits, logits[0].shape, logits[1].shape

[16]:

([array([[1.8236330e-14, 1.0061867e-09, 9.1841962e-10, ..., 1.0257798e-12,
          1.4030079e-09, 2.5916536e-04],
         [8.9639464e-15, 1.7033805e-09, 4.6274323e-10, ..., 6.6502495e-13,
          8.0938090e-10, 4.3023058e-04],
         [3.1887525e-15, 2.8573975e-08, 1.4154197e-10, ..., 3.1815506e-13,
          2.4540896e-09, 1.0617726e-03],
         ...,
         [8.8733091e-16, 1.2685842e-08, 6.6173206e-11, ..., 3.7230733e-13,
          4.0759787e-12, 1.5587657e-04],
         [6.6120927e-16, 6.5794703e-09, 5.7139283e-11, ..., 4.9776345e-13,
          4.2690807e-12, 8.7887602e-05],
         [5.4730716e-16, 2.1361308e-09, 9.3025414e-11, ..., 5.9886417e-13,
          1.5507430e-11, 1.0300719e-04]], dtype=float32),
  array([[2.5148340e-15, 3.8227799e-09, 1.0598683e-09, ..., 4.1977169e-13,
          7.1061929e-10, 3.9750684e-04],
         [2.9357024e-14, 4.2996367e-11, 8.0736609e-09, ..., 3.3508845e-12,
          3.6292098e-11, 3.5416318e-07],
         [1.9910680e-15, 7.9035152e-09, 3.4416633e-10, ..., 1.3513049e-12,
          1.3454201e-09, 1.0257948e-04],
         ...,
         [3.6507369e-16, 3.2854697e-09, 3.0207747e-10, ..., 8.0089668e-13,
          1.2993007e-10, 1.5945344e-03],
         [4.2754438e-16, 6.5015402e-09, 2.0224432e-10, ..., 8.0650563e-13,
          1.6817953e-10, 1.4560440e-03],
         [3.1852618e-16, 6.3009313e-09, 1.1704311e-10, ..., 5.8428116e-13,
          2.6164873e-10, 1.3219280e-03]], dtype=float32),
  array([[7.4554222e-15, 8.0248808e-10, 4.3944155e-09, ..., 7.1492242e-13,
          1.8890089e-09, 8.0676808e-05],
         [2.0308339e-15, 2.7493043e-09, 9.9622988e-10, ..., 3.6468658e-13,
          4.4663002e-09, 4.2535696e-04],
         [9.8700091e-16, 5.8224430e-09, 9.5636510e-10, ..., 2.5640658e-13,
          6.0229244e-10, 5.0613942e-04],
         ...,
         [6.6229983e-16, 2.5211668e-09, 1.0523804e-10, ..., 8.1805699e-13,
          2.0467164e-10, 1.5298247e-03],
         [7.8388564e-16, 3.5194940e-09, 8.4871700e-11, ..., 6.7016152e-13,
          1.1333524e-10, 1.0733902e-03],
         [6.9106448e-16, 3.5260113e-09, 6.5353133e-11, ..., 4.5797551e-13,
          1.2999080e-10, 9.0871076e-04]], dtype=float32)],
 (499, 39),
 (299, 39))

You can use output from predict_logits to feed into ctc-decoders or pyctcdecode with language model to get better results.

[ ]: