Speech-to-Text RNNT Malay + Singlish#

Encoder model + RNNT loss for Malay + Singlish

This tutorial is available as an IPython notebook at malaya-speech/example/stt-transducer-model-2mixed.

This module is not language independent, so it not save to use on different languages. Pretrained models trained on hyperlocal languages.

This is an application of malaya-speech Pipeline, read more about malaya-speech Pipeline at malaya-speech/example/pipeline.

[1]:

import malaya_speech
import numpy as np
from malaya_speech import Pipeline

List available RNNT model#

[2]:

malaya_speech.stt.available_transducer()

[2]:

	Size (MB)	Quantized Size (MB)	WER	CER	WER-LM	CER-LM	Language
tiny-conformer	24.4	9.14	0.212811	0.081369	0.199683	0.077004	[malay]
small-conformer	49.2	18.1	0.198533	0.074495	0.185361	0.071143	[malay]
conformer	125	37.1	0.163602	0.058744	0.156182	0.05719	[malay]
large-conformer	404	107	0.156684	0.061971	0.148622	0.05901	[malay]
conformer-stack-2mixed	130	38.5	0.103608	0.050069	0.102911	0.050201	[malay, singlish]
conformer-stack-3mixed	130	38.5	0.234768	0.133944	0.229241	0.130702	[malay, singlish, mandarin]
small-conformer-singlish	49.2	18.1	0.087831	0.045686	0.087333	0.045317	[singlish]
conformer-singlish	125	37.1	0.077792	0.040362	0.077186	0.03987	[singlish]
large-conformer-singlish	404	107	0.070147	0.035872	0.069812	0.035723	[singlish]

Lower is better. Test set can get at https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt

Malay trained on Malaya Speech dataset, singlish trained on IMDA dataset.

Google Speech-to-Text only supported monolanguage, so we are not able to compare the accuracy.

Load RNNT model#

def deep_transducer(
    model: str = 'conformer', quantized: bool = False, **kwargs
):
    """
    Load Encoder-Transducer ASR model.

    Parameters
    ----------
    model : str, optional (default='conformer')
        Model architecture supported. Allowed values:

        * ``'tiny-conformer'`` - TINY size Google Conformer.
        * ``'small-conformer'`` - SMALL size Google Conformer.
        * ``'conformer'`` - BASE size Google Conformer.
        * ``'large-conformer'`` - LARGE size Google Conformer.
        * ``'conformer-stack-2mixed'`` - BASE size Stacked Google Conformer for (Malay + Singlish) languages.
        * ``'conformer-stack-3mixed'`` - BASE size Stacked Google Conformer for (Malay + Singlish + Mandarin) languages.
        * ``'small-conformer-singlish'`` - SMALL size Google Conformer for singlish language.
        * ``'conformer-singlish'`` - BASE size Google Conformer for singlish language.
        * ``'large-conformer-singlish'`` - LARGE size Google Conformer for singlish language.

    quantized : bool, optional (default=False)
        if True, will load 8-bit quantized model.
        Quantized model not necessary faster, totally depends on the machine.

    Returns
    -------
    result : malaya_speech.model.tf.Transducer class
    """

[20]:

model = malaya_speech.stt.deep_transducer(model = 'conformer-stack-2mixed')

Load Quantized deep model#

To load 8-bit quantized model, simply pass quantized = True, default is False.

We can expect slightly accuracy drop from quantized model, and not necessary faster than normal 32-bit float model, totally depends on machine.

[21]:

quantized_model = malaya_speech.stt.deep_transducer(model = 'conformer-stack-2mixed', quantized = True)

Load sample#

[6]:

ceramah, sr = malaya_speech.load('speech/khutbah/wadi-annuar.wav')
record1, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-36-06_294832.wav')
record2, sr = malaya_speech.load('speech/record/savewav_2020-11-26_22-40-56_929661.wav')
singlish0, sr = malaya_speech.load('speech/singlish/singlish0.wav')
singlish1, sr = malaya_speech.load('speech/singlish/singlish1.wav')
singlish2, sr = malaya_speech.load('speech/singlish/singlish2.wav')

[7]:

import IPython.display as ipd

ipd.Audio(ceramah, rate = sr)

[7]:

[8]:

ipd.Audio(record1, rate = sr)

[8]:

[9]:

ipd.Audio(record2, rate = sr)

[9]:

[10]:

ipd.Audio(singlish0, rate = sr)

[10]:

[11]:

ipd.Audio(singlish1, rate = sr)

[11]:

[12]:

ipd.Audio(singlish2, rate = sr)

[12]:

Predict using greedy decoder#

def greedy_decoder(self, inputs):
    """
    Transcribe inputs using greedy decoder.

    Parameters
    ----------
    inputs: List[np.array]
        List[np.array] or List[malaya_speech.model.frame.Frame].

    Returns
    -------
    result: List[str]
    """

[7]:

%%time

model.greedy_decoder([ceramah, record1, record2, singlish0, singlish1, singlish2])

CPU times: user 10.9 s, sys: 3.22 s, total: 14.2 s
Wall time: 9.24 s

[7]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allahu',
 'helo nama saya mesin saya tak suka mandi ke tak saya masak',
 'helo nama saya husin saya suka mandi saya mandi titik hari',
 'and then see how they roll it in film okay actually',
 'then you talk to your eyes',
 'surprise in ma']

[13]:

%%time

quantized_model.greedy_decoder([ceramah, record1, record2, singlish0, singlish1, singlish2])

CPU times: user 11 s, sys: 3.28 s, total: 14.3 s
Wall time: 9.53 s

[13]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allah mah',
 'hello nama saya usin saya tak suka mandi ketat saya masam',
 'hello nama saya husin saya suka mandi saya mandi titik hari',
 'and then see how they roll it in film okay actually',
 'then you tell to your eyes',
 'seven seven in mall']

Predict using beam decoder#

def beam_decoder(self, inputs, beam_width: int = 5,
                 temperature: float = 0.0,
                 score_norm: bool = True):
    """
    Transcribe inputs using beam decoder.

    Parameters
    ----------
    inputs: List[np.array]
        List[np.array] or List[malaya_speech.model.frame.Frame].
    beam_width: int, optional (default=5)
        beam size for beam decoder.
    temperature: float, optional (default=0.0)
        apply temperature function for logits, can help for certain case,
        logits += -np.log(-np.log(uniform_noise_shape_logits)) * temperature
    score_norm: bool, optional (default=True)
        descending sort beam based on score / length of decoded.

    Returns
    -------
    result: List[str]
    """

[15]:

%%time

model.beam_decoder([ceramah, record1, record2, singlish0, singlish1, singlish2], beam_width = 5)

CPU times: user 22.3 s, sys: 3.5 s, total: 25.8 s
Wall time: 13.8 s

[15]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allah mah',
 'helo nama saya usin saya tak suka mandi ketat saya masam',
 'helo nama saya husin saya suka mandi saya mandi titik hari',
 'and then see how they roll it in film okay actually',
 'then you tell to your eyes',
 'some person in mao']

[17]:

%%time

quantized_model.beam_decoder([ceramah, record1, record2, singlish0, singlish1, singlish2], beam_width = 5)

CPU times: user 21.2 s, sys: 3.14 s, total: 24.3 s
Wall time: 12.6 s

[17]:

['jadi dalam perjalanan ini dunia yang susah ini ketika nabi mengajar muaz bin jabal tadi ni allah mak',
 'helo nama saya husin saya tak suka mandi ketat saya masam',
 'helo nama saya husin saya suka mandi saya mandi titik hari',
 'and then see how they roll it in film okay actually',
 'then you tell to your eyes',
 'some person in mao']

RNNT model beam decoder not able to utilise batch programming, if feed a batch, it will process one by one.

Predict alignment#

We want to know when the speakers speak certain words, so we can use predict_timestamp,

def predict_alignment(self, input, combined = True):
    """
    Transcribe input and get timestamp, only support greedy decoder.

    Parameters
    ----------
    input: np.array
        np.array or malaya_speech.model.frame.Frame.
    combined: bool, optional (default=True)
        If True, will combined subwords to become a word.

    Returns
    -------
    result: List[Dict[text, start, end]]
    """

[18]:

%%time

model.predict_alignment(singlish0)

CPU times: user 6.09 s, sys: 2.16 s, total: 8.25 s
Wall time: 7.3 s

[18]:

[{'text': 'and', 'start': 0.2, 'end': 0.21},
 {'text': 'then', 'start': 0.36, 'end': 0.45},
 {'text': 'see', 'start': 0.6, 'end': 0.77},
 {'text': 'how', 'start': 0.88, 'end': 1.09},
 {'text': 'they', 'start': 1.36, 'end': 1.49},
 {'text': 'roll', 'start': 1.92, 'end': 2.09},
 {'text': 'it', 'start': 2.2, 'end': 2.21},
 {'text': 'in', 'start': 2.4, 'end': 2.41},
 {'text': 'film', 'start': 2.6, 'end': 2.85},
 {'text': 'okay', 'start': 3.68, 'end': 3.85},
 {'text': 'actually', 'start': 3.92, 'end': 4.21}]

[19]:

%%time

model.predict_alignment(singlish0, combined = False)

CPU times: user 1.07 s, sys: 165 ms, total: 1.24 s
Wall time: 284 ms

[19]:

[{'text': 'and', 'start': 0.2, 'end': 0.21},
 {'text': ' ', 'start': 0.28, 'end': 0.29},
 {'text': 'the', 'start': 0.36, 'end': 0.37},
 {'text': 'n_', 'start': 0.44, 'end': 0.45},
 {'text': 'se', 'start': 0.6, 'end': 0.61},
 {'text': 'e_', 'start': 0.76, 'end': 0.77},
 {'text': 'ho', 'start': 0.88, 'end': 0.89},
 {'text': 'w_', 'start': 1.08, 'end': 1.09},
 {'text': 'the', 'start': 1.36, 'end': 1.37},
 {'text': 'y_', 'start': 1.48, 'end': 1.49},
 {'text': 'ro', 'start': 1.92, 'end': 1.93},
 {'text': 'll_', 'start': 2.08, 'end': 2.09},
 {'text': 'it_', 'start': 2.2, 'end': 2.21},
 {'text': 'in_', 'start': 2.4, 'end': 2.41},
 {'text': 'fi', 'start': 2.6, 'end': 2.61},
 {'text': 'l', 'start': 2.76, 'end': 2.77},
 {'text': 'm_', 'start': 2.84, 'end': 2.85},
 {'text': 'oka', 'start': 3.68, 'end': 3.69},
 {'text': 'y_', 'start': 3.84, 'end': 3.85},
 {'text': 'ac', 'start': 3.92, 'end': 3.93},
 {'text': 'tu', 'start': 4.04, 'end': 4.05},
 {'text': 'all', 'start': 4.16, 'end': 4.17},
 {'text': 'y', 'start': 4.2, 'end': 4.21}]

Speech-to-Text RNNT Malay + Singlish

Contents

Speech-to-Text RNNT Malay + Singlish#

List available RNNT model#

Load RNNT model#

Load Quantized deep model#

Load sample#

Predict using greedy decoder#

Predict using beam decoder#

Predict alignment#