Speech-to-Text CTC + CTC Decoders#

Encoder model + CTC loss + CTC Decoders with KenLM

This tutorial is available as an IPython notebook at malaya-speech/example/stt-ctc-model-ctc-decoders.

This module is not language independent, so it not save to use on different languages. Pretrained models trained on hyperlocal languages.

This is an application of malaya-speech Pipeline, read more about malaya-speech Pipeline at malaya-speech/example/pipeline.

This interface deprecated, use HuggingFace interface instead.

[1]:

import os

os.environ['CUDA_VISIBLE_DEVICES'] = ''

[2]:

import malaya_speech
import numpy as np
from malaya_speech import Pipeline

`pyaudio` is not available, `malaya_speech.streaming.stream` is not able to use.

[3]:

import logging

logging.basicConfig(level=logging.INFO)

[4]:

import warnings
warnings.filterwarnings('default')

Install ctc-decoders#

From PYPI#

pip3 install ctc-decoders

But if you use linux, we unable to upload linux wheels to pypi repository, so download linux wheel at malaya-speech/ctc-decoders.

From source#

Check malaya-speech/ctc-decoders how to build from source incase there is no available wheel for your operating system.

Building from source should only take a few minutes.

Benefit#

  1. ctc-decoders faster than pyctcdecode, ~26x faster based on husein benchmark, but very slightly less accurate than pyctcdecode.

List available CTC model#

[5]:

malaya_speech.stt.ctc.available_transformer()

/home/husein/dev/malaya-speech/malaya_speech/stt/ctc.py:144: DeprecationWarning: `malaya.stt.ctc.available_transformer` is deprecated, use `malaya.stt.ctc.available_huggingface` instead
  warnings.warn(
INFO:malaya_speech.stt:for `malay-fleur102` language, tested on FLEURS102 `ms_my` test set, https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt
INFO:malaya_speech.stt:for `malay-malaya` language, tested on malaya-speech test set, https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt
INFO:malaya_speech.stt:for `singlish` language, tested on IMDA malaya-speech test set, https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt

[5]:
Size (MB) Quantized Size (MB) malay-malaya Language
hubert-conformer-tiny 36.6 10.3 {'WER': 0.238714008166, 'CER': 0.060899814, 'W... [malay]
hubert-conformer 115 31.1 {'WER': 0.2387140081, 'CER': 0.06089981404, 'W... [malay]
hubert-conformer-large 392 100 {'WER': 0.2203140421, 'CER': 0.0549270416, 'WE... [malay]

Load CTC model#

def transformer(
    model: str = 'hubert-conformer',
    quantized: bool = False,
    **kwargs,
):
    """
    Load Encoder-CTC ASR model.

    Parameters
    ----------
    model : str, optional (default='hubert-conformer')
        Check available models at `malaya_speech.stt.ctc.available_transformer()`.
    quantized : bool, optional (default=False)
        if True, will load 8-bit quantized model.
        Quantized model not necessary faster, totally depends on the machine.

    Returns
    -------
    result : malaya_speech.model.wav2vec.Wav2Vec2_CTC class
    """

[6]:

model = malaya_speech.stt.ctc.transformer(model = 'hubert-conformer')

/home/husein/dev/malaya-speech/malaya_speech/stt/ctc.py:181: DeprecationWarning: `malaya.stt.ctc.transformer` is deprecated, use `malaya.stt.ctc.huggingface` instead
  warnings.warn(
2023-02-01 11:34:40.091378: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-01 11:34:40.096938: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-02-01 11:34:40.096964: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: husein-MS-7D31
2023-02-01 11:34:40.096968: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: husein-MS-7D31
2023-02-01 11:34:40.097057: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program
2023-02-01 11:34:40.097083: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 470.161.3

Load sample#

[7]:

ceramah, sr = malaya_speech.load('speech/khutbah/wadi-annuar.wav')
record1, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-36-06_294832.wav')
record2, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-40-56_929661.wav')

[8]:

import IPython.display as ipd

ipd.Audio(ceramah, rate = sr)

[8]:

As we can hear, the speaker speaks in kedahan dialects plus some arabic words, let see how good our model is.

[9]:

ipd.Audio(record1, rate = sr)

[9]:

[10]:

ipd.Audio(record2, rate = sr)

[10]:

As you can see, below is the output from beam decoder without language model,

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah ma ini',
 'helo nama saya esin saya tak suka mandi ketak saya masak',
 'helo nama saya musin saya suka mandi saya mandi titiap hari']

Predict logits#

def predict_logits(self, inputs):
    """
    Predict logits from inputs.

    Parameters
    ----------
    input: List[np.array]
        List[np.array] or List[malaya_speech.model.frame.Frame].


    Returns
    -------
    result: List[np.array]
    """

[8]:

%%time

logits = model.predict_logits([ceramah, record1, record2])

CPU times: user 26.5 s, sys: 10.4 s, total: 36.9 s
Wall time: 20.3 s

[9]:

logits[0].shape

[9]:

(499, 39)

Load ctc-decoders#

I will use dump-combined for this example.

[10]:

from ctc_decoders import Scorer
from ctc_decoders import ctc_beam_search_decoder
from malaya_speech.utils.char import CTC_VOCAB

[11]:

lm = malaya_speech.language_model.kenlm(model = 'dump-combined')

[12]:

scorer = Scorer(0.5, 1.0, lm, CTC_VOCAB)

[13]:

o = ctc_beam_search_decoder(logits[0], CTC_VOCAB, 20, ext_scoring_func = scorer)[0][1]
o

[13]:

'jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allah maini'

[14]:

o = ctc_beam_search_decoder(logits[1], CTC_VOCAB, 20, ext_scoring_func = scorer)[0][1]
o

[14]:

'helo nama saya mesin saya tak suka mandi ketat saya masak'

[15]:

o = ctc_beam_search_decoder(logits[2], CTC_VOCAB, 20, ext_scoring_func = scorer)[0][1]
o

[15]:

'helo nama saya mesin saya suka mandi saya mandi titik hari'