Speech-to-Text RNNT

Encoder model + RNNT loss

This tutorial is available as an IPython notebook at malaya-speech/example/stt-transducer-model.

This module is not language independent, so it not save to use on different languages. Pretrained models trained on hyperlocal languages.

import os

os.environ['CUDA_VISIBLE_DEVICES'] = ''

import malaya_speech
import numpy as np
from malaya_speech import Pipeline

`pyaudio` is not available, `malaya_speech.streaming.stream` is not able to use.

import logging

logging.basicConfig(level=logging.INFO)

List available RNNT model

malaya_speech.stt.transducer.available_transformer()

INFO:malaya_speech.stt:for `malay-fleur102` language, tested on FLEURS102 `ms_my` test set, https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt
INFO:malaya_speech.stt:for `malay-malaya` language, tested on malaya-speech test set, https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt
INFO:malaya_speech.stt:for `singlish` language, tested on IMDA malaya-speech test set, https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt

	Size (MB)	Quantized Size (MB)	malay-malaya	malay-fleur102	Language	singlish
tiny-conformer	24.4	9.14	{'WER': 0.2128108, 'CER': 0.08136871, 'WER-LM'...	{'WER': 0.2682816, 'CER': 0.13052725, 'WER-LM'...	[malay]	NaN
small-conformer	49.2	18.1	{'WER': 0.19853302, 'CER': 0.07449528, 'WER-LM...	{'WER': 0.23412149, 'CER': 0.1138314813, 'WER-...	[malay]	NaN
conformer	125	37.1	{'WER': 0.16340855635999124, 'CER': 0.05897205...	{'WER': 0.20090442596, 'CER': 0.09616901, 'WER...	[malay]	NaN
large-conformer	404	107	{'WER': 0.1566839, 'CER': 0.0619715, 'WER-LM':...	{'WER': 0.1711028238, 'CER': 0.077953559, 'WER...	[malay]	NaN
conformer-stack-2mixed	130	38.5	{'WER': 0.1889883954, 'CER': 0.0726845531, 'WE...	{'WER': 0.244836948, 'CER': 0.117409327, 'WER-...	[malay, singlish]	{'WER': 0.08535878149, 'CER': 0.0452357273822,...
small-conformer-singlish	49.2	18.1	NaN	NaN	[singlish]	{'WER': 0.087831, 'CER': 0.0456859, 'WER-LM': ...
conformer-singlish	125	37.1	NaN	NaN	[singlish]	{'WER': 0.07779246, 'CER': 0.0403616, 'WER-LM'...
large-conformer-singlish	404	107	NaN	NaN	[singlish]	{'WER': 0.07014733, 'CER': 0.03587201, 'WER-LM...

malaya_speech.stt.google_accuracy

{'malay-malaya': {'WER': 0.16477548774, 'CER': 0.05973209121},
 'malay-fleur102': {'WER': 0.109588779, 'CER': 0.047891527},
 'singlish': {'WER': 0.4941349, 'CER': 0.3026296}}

Load RNNT model

def transformer(
    model: str = 'conformer',
    quantized: bool = False,
    **kwargs,
):
    """
    Load Encoder-Transducer ASR model.

    Parameters
    ----------
    model : str, optional (default='conformer')
        Check available models at `malaya_speech.stt.transducer.available_transformer()`.
    quantized : bool, optional (default=False)
        if True, will load 8-bit quantized model.
        Quantized model not necessary faster, totally depends on the machine.

    Returns
    -------
    result : malaya_speech.model.transducer.Transducer class
    """

small_model = malaya_speech.stt.transducer.transformer(model = 'small-conformer')
model = malaya_speech.stt.transducer.transformer(model = 'conformer')

2023-02-01 11:53:39.010188: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-01 11:53:39.015470: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-02-01 11:53:39.015489: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: husein-MS-7D31
2023-02-01 11:53:39.015496: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: husein-MS-7D31
2023-02-01 11:53:39.015568: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program
2023-02-01 11:53:39.015587: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 470.161.3

Load Quantized deep model

To load 8-bit quantized model, simply pass quantized = True, default is False.

We can expect slightly accuracy drop from quantized model, and not necessary faster than normal 32-bit float model, totally depends on machine.

quantized_small_model = malaya_speech.stt.transducer.transformer(model = 'small-conformer', quantized = True)
quantized_model = malaya_speech.stt.transducer.transformer(model = 'conformer', quantized = True)

Load sample

ceramah, sr = malaya_speech.load('speech/khutbah/wadi-annuar.wav')
record1, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-36-06_294832.wav')
record2, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-40-56_929661.wav')
shafiqah_idayu, sr = malaya_speech.load('speech/example-speaker/shafiqah-idayu.wav')
mas_aisyah, sr = malaya_speech.load('speech/example-speaker/mas-aisyah.wav')
khalil, sr = malaya_speech.load('speech/example-speaker/khalil-nooh.wav')

import IPython.display as ipd

ipd.Audio(ceramah, rate = sr)

As we can hear, the speaker speaks in kedahan dialects plus some arabic words, let see how good our model is.

ipd.Audio(record1, rate = sr)

ipd.Audio(record2, rate = sr)

ipd.Audio(shafiqah_idayu, rate = sr)

ipd.Audio(mas_aisyah, rate = sr)

ipd.Audio(khalil, rate = sr)

Predict using greedy decoder

def greedy_decoder(self, inputs):
    """
    Transcribe inputs using greedy decoder.

    Parameters
    ----------
    inputs: List[np.array]
        List[np.array] or List[malaya_speech.model.frame.Frame].

    Returns
    -------
    result: List[str]
    """

%%time

small_model.greedy_decoder([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil])

CPU times: user 7.09 s, sys: 1.8 s, total: 8.88 s
Wall time: 6.05 s

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allah maha ini',
 'helo nama saya husin saya tak suka mandi ketat saya masak',
 'helo nama saya husin saya suka mandi saya mandi tetek hari',
 'nama saya syafiqah hidayah',
 'sebut perkataan uncle',
 'tolong sebut anti kata']

%%time

model.greedy_decoder([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil])

CPU times: user 11.3 s, sys: 3.47 s, total: 14.8 s
Wall time: 11.2 s

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah maaf ini',
 'helo nama saya send saya tak suka mandi ke tak saya masam',
 'helo nama saya husin saya suka mandi saya mandi setiap hari',
 'nama saya syafiqah idayu',
 'sebut perkataan angka',
 'tolong sebut antika']

%%time

quantized_small_model.greedy_decoder([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil])

CPU times: user 6.58 s, sys: 1.34 s, total: 7.92 s
Wall time: 5.24 s

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allah maha ini',
 'helo nama saya husin saya tak suka mandi ketat saya masak',
 'helo nama saya husin saya suka mandi saya mandi tetek hari',
 'nama saya syafiqah hidayah',
 'sebut perkataan uncle',
 'tolong sebut anti kata']

%%time

quantized_model.greedy_decoder([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil])

CPU times: user 10.8 s, sys: 3.02 s, total: 13.8 s
Wall time: 8.91 s

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah maaf ini',
 'helo nama saya send saya tak suka mandi ke tak saya masam',
 'helo nama saya pusing saya suka mandi saya mandi setiap hari',
 'nama saya syafiqah idayu',
 'sebut perkataan angka',
 'tolong sebut antika']

Predict using beam decoder

def beam_decoder(self, inputs, beam_width: int = 5,
                 temperature: float = 0.0,
                 score_norm: bool = True):
    """
    Transcribe inputs using beam decoder.

    Parameters
    ----------
    inputs: List[np.array]
        List[np.array] or List[malaya_speech.model.frame.Frame].
    beam_width: int, optional (default=5)
        beam size for beam decoder.
    temperature: float, optional (default=0.0)
        apply temperature function for logits, can help for certain case,
        logits += -np.log(-np.log(uniform_noise_shape_logits)) * temperature
    score_norm: bool, optional (default=True)
        descending sort beam based on score / length of decoded.

    Returns
    -------
    result: List[str]
    """

%%time

small_model.beam_decoder([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil], beam_width = 5)

CPU times: user 11.2 s, sys: 1.97 s, total: 13.2 s
Wall time: 8.14 s

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allah maha ini',
 'helo nama saya pusing saya tak suka mandi ketat saya masak',
 'helo nama saya husin saya suka mandi saya mandi tetek hari',
 'nama saya syafiqah hidayah',
 'sebut perkataan uncle',
 'tolong sebut anti kata']

%%time

model.beam_decoder([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil], beam_width = 5)

CPU times: user 21.3 s, sys: 3.23 s, total: 24.6 s
Wall time: 13.6 s

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah maaf ini',
 'helo nama saya pusing saya tak suka mandi ke tak saya masam',
 'helo nama saya husin saya suka mandi saya mandi tiap tiap hari',
 'nama saya syafiqah idayu',
 'sebut perkataan angka',
 'tolong sebut antika']

%%time

quantized_small_model.beam_decoder([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil], beam_width = 5)

CPU times: user 10.6 s, sys: 1.71 s, total: 12.3 s
Wall time: 7.53 s

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allah maha ini',
 'helo nama saya pusing saya tak suka mandi ketat saya masak',
 'helo nama saya husin saya suka mandi saya mandi tetek hari',
 'nama saya syafiqah hidayah',
 'sebut perkataan uncle',
 'tolong sebut anti kata']

%%time

quantized_model.beam_decoder([ceramah, record1, record2, shafiqah_idayu, mas_aisyah, khalil], beam_width = 5)

CPU times: user 16.8 s, sys: 1.67 s, total: 18.5 s
Wall time: 6.45 s

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni alah maaf ini',
 'helo nama saya pusing saya tak suka mandi ke tak saya masam',
 'helo nama saya pusing saya suka mandi saya mandi tiap tiap hari',
 'nama saya syafiqah idayu',
 'sebut perkataan angka',
 'tolong sebut antika']

RNNT model beam decoder not able to utilise batch processing, if feed a batch, it will process one by one.

Predict alignment

We want to know when the speakers speak certain words, so we can use predict_timestamp,

def predict_alignment(self, input, combined = True):
    """
    Transcribe input and get timestamp, only support greedy decoder.

    Parameters
    ----------
    input: np.array
        np.array or malaya_speech.model.frame.Frame.
    combined: bool, optional (default=True)
        If True, will combined subwords to become a word.

    Returns
    -------
    result: List[Dict[text, start, end]]
    """

%%time

small_model.predict_alignment(shafiqah_idayu)

CPU times: user 3.7 s, sys: 784 ms, total: 4.48 s
Wall time: 4.11 s

[{'text': 'nama', 'start': 0.28, 'end': 0.57},
 {'text': 'saya', 'start': 0.68, 'end': 0.97},
 {'text': 'syafiqah', 'start': 1.28, 'end': 1.69},
 {'text': 'idri', 'start': 1.8, 'end': 2.01}]

%%time

small_model.predict_alignment(shafiqah_idayu, combined = False)

CPU times: user 405 ms, sys: 84.8 ms, total: 489 ms
Wall time: 128 ms

[{'text': 'nam', 'start': 0.28, 'end': 0.29},
 {'text': 'a_', 'start': 0.56, 'end': 0.57},
 {'text': 'say', 'start': 0.68, 'end': 0.69},
 {'text': 'a_', 'start': 0.96, 'end': 0.97},
 {'text': 'sya', 'start': 1.28, 'end': 1.29},
 {'text': 'fi', 'start': 1.44, 'end': 1.45},
 {'text': 'q', 'start': 1.52, 'end': 1.53},
 {'text': 'ah_', 'start': 1.68, 'end': 1.69},
 {'text': 'id', 'start': 1.8, 'end': 1.81},
 {'text': 'ri', 'start': 2.0, 'end': 2.01}]

%%time

small_model.predict_alignment(ceramah)

CPU times: user 1.25 s, sys: 324 ms, total: 1.58 s
Wall time: 348 ms

[{'text': 'jadi', 'start': 0.36, 'end': 0.53},
 {'text': 'dalam', 'start': 0.6, 'end': 0.73},
 {'text': 'perjalanan', 'start': 0.84, 'end': 1.33},
 {'text': 'ini', 'start': 1.4, 'end': 1.41},
 {'text': 'dunia', 'start': 2.44, 'end': 2.65},
 {'text': 'yang', 'start': 2.76, 'end': 2.81},
 {'text': 'susah', 'start': 2.88, 'end': 3.13},
 {'text': 'ini', 'start': 3.24, 'end': 3.25},
 {'text': 'ketika', 'start': 5.64, 'end': 5.85},
 {'text': 'nabi', 'start': 6.12, 'end': 6.37},
 {'text': 'mengajar', 'start': 6.44, 'end': 6.81},
 {'text': 'muaz', 'start': 6.96, 'end': 7.21},
 {'text': 'bin', 'start': 7.28, 'end': 7.29},
 {'text': 'jabal', 'start': 7.44, 'end': 7.73},
 {'text': 'tadi', 'start': 7.84, 'end': 8.05},
 {'text': 'ni', 'start': 8.12, 'end': 8.13},
 {'text': 'allah', 'start': 8.52, 'end': 8.69},
 {'text': 'maha', 'start': 8.8, 'end': 9.01},
 {'text': 'ini', 'start': 9.4, 'end': 9.41}]

%%time

model.predict_alignment(shafiqah_idayu)

CPU times: user 5.89 s, sys: 2.04 s, total: 7.93 s
Wall time: 7.37 s

[{'text': 'nama', 'start': 0.28, 'end': 0.57},
 {'text': 'saya', 'start': 0.64, 'end': 0.97},
 {'text': 'syafiqah', 'start': 1.28, 'end': 1.69},
 {'text': 'idayu', 'start': 1.8, 'end': 2.05}]

%%time

model.predict_alignment(ceramah)

CPU times: user 2.35 s, sys: 515 ms, total: 2.87 s
Wall time: 593 ms

[{'text': 'jadi', 'start': 0.36, 'end': 0.53},
 {'text': 'dalam', 'start': 0.6, 'end': 0.73},
 {'text': 'perjalanan', 'start': 0.8, 'end': 1.29},
 {'text': 'ini', 'start': 1.4, 'end': 1.41},
 {'text': 'dunia', 'start': 2.44, 'end': 2.65},
 {'text': 'yang', 'start': 2.72, 'end': 2.81},
 {'text': 'susah', 'start': 2.88, 'end': 3.13},
 {'text': 'ini', 'start': 3.24, 'end': 3.25},
 {'text': 'ketika', 'start': 5.64, 'end': 5.85},
 {'text': 'nabi', 'start': 6.12, 'end': 6.37},
 {'text': 'mengajar', 'start': 6.44, 'end': 6.81},
 {'text': 'muaz', 'start': 6.96, 'end': 7.21},
 {'text': 'bin', 'start': 7.28, 'end': 7.29},
 {'text': 'jabal', 'start': 7.44, 'end': 7.73},
 {'text': 'tadi', 'start': 7.84, 'end': 8.05},
 {'text': 'ni', 'start': 8.12, 'end': 8.13},
 {'text': 'alah', 'start': 8.52, 'end': 8.69},
 {'text': 'maaf', 'start': 8.8, 'end': 9.01}]

%%time

quantized_small_model.predict_alignment(shafiqah_idayu)

CPU times: user 3.82 s, sys: 743 ms, total: 4.56 s
Wall time: 4.19 s

[{'text': 'nama', 'start': 0.28, 'end': 0.57},
 {'text': 'saya', 'start': 0.68, 'end': 0.97},
 {'text': 'syafiqah', 'start': 1.28, 'end': 1.69},
 {'text': 'idri', 'start': 1.8, 'end': 2.01}]

%%time

quantized_model.predict_alignment(shafiqah_idayu)

CPU times: user 5.59 s, sys: 1.85 s, total: 7.44 s
Wall time: 6.8 s

[{'text': 'nama', 'start': 0.28, 'end': 0.57},
 {'text': 'saya', 'start': 0.64, 'end': 0.97},
 {'text': 'syafiqah', 'start': 1.28, 'end': 1.69},
 {'text': 'id', 'start': 1.8, 'end': 1.81}]