Gender Detection#

This tutorial is available as an IPython notebook at malaya-speech/example/gender.

This module is language independent, so it save to use on different languages. Pretrained models trained on multilanguages.

This is an application of malaya-speech Pipeline, read more about malaya-speech Pipeline at malaya-speech/example/pipeline.

Dataset#

Trained on Gender column from Common Voice Mozilla Dataset and VoxCeleb 1 with augmented noises, https://commonvoice.mozilla.org/, http://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html

[1]:

import malaya_speech
import numpy as np
from malaya_speech import Pipeline

[2]:

y, sr = malaya_speech.load('speech/video/The-Singaporean-White-Boy.wav')
len(y), sr

[2]:

(1634237, 16000)

[3]:

# just going to take 30 seconds
y = y[:sr * 30]

[4]:

import IPython.display as ipd
ipd.Audio(y, rate = sr)

[4]:

This audio extracted from https://www.youtube.com/watch?v=HylaY5e1awo&t=2s

Supported genders#

[5]:

malaya_speech.gender.labels

[5]:

['female', 'male', 'not a gender']

List available deep model#

[6]:

malaya_speech.gender.available_model()

INFO:root:last accuracy during training session before early stopping.

[6]:
Size (MB) Quantized Size (MB) Accuracy
vggvox-v2 31.1 7.92 0.9756
deep-speaker 96.9 24.40 0.9455

Load deep model#

def deep_model(model: str = 'vggvox-v2', quantized: bool = False, **kwargs):
    """
    Load gender detection deep model.

    Parameters
    ----------
    model : str, optional (default='vggvox-v2')
        Model architecture supported. Allowed values:

        * ``'vggvox-v2'`` - finetuned VGGVox V2.
        * ``'deep-speaker'`` - finetuned Deep Speaker.
    quantized : bool, optional (default=False)
        if True, will load 8-bit quantized model.
        Quantized model not necessary faster, totally depends on the machine.

    Returns
    -------
    result : malaya_speech.supervised.classification.load function
    """

[7]:

vggvox_v2 = malaya_speech.gender.deep_model(model = 'vggvox-v2')
deep_speaker = malaya_speech.gender.deep_model(model = 'deep-speaker')

INFO:root:running gender/vggvox-v2 using device /device:CPU:0
INFO:root:running gender/deep-speaker using device /device:CPU:0

Load Quantized deep model#

To load 8-bit quantized model, simply pass quantized = True, default is False.

We can expect slightly accuracy drop from quantized model, and not necessary faster than normal 32-bit float model, totally depends on machine.

[8]:

quantized_vggvox_v2 = malaya_speech.gender.deep_model(model = 'vggvox-v2', quantized = True)

WARNING:root:Load quantized model will cause accuracy drop.
INFO:root:running gender/vggvox-v2-quantized using device /device:CPU:0

How to classify genders in an audio sample#

So we are going to use VAD to help us. Instead we are going to classify as a whole sample, we chunk it into multiple small samples and classify it.

[9]:

vad = malaya_speech.vad.deep_model(model = 'vggvox-v2')

INFO:root:running vad/vggvox-v2 using device /device:CPU:0

[10]:

%%time

frames = list(malaya_speech.utils.generator.frames(y, 30, sr))

CPU times: user 1.74 ms, sys: 191 µs, total: 1.94 ms
Wall time: 2.09 ms

[11]:

p = Pipeline()
pipeline = (
    p.batching(5)
    .foreach_map(vad.predict)
    .flatten()
)
p.visualize()

[11]:
_images/load-gender_22_0.png 
[12]:

%%time

result = p.emit(frames)
result.keys()

/Users/huseinzolkepli/Documents/tf-1.15/env/lib/python3.7/site-packages/librosa/core/spectrum.py:224: UserWarning: n_fft=512 is too small for input signal of length=480
  n_fft, y.shape[-1]

CPU times: user 33.1 s, sys: 6.5 s, total: 39.6 s
Wall time: 9.41 s

[12]:

dict_keys(['batching', 'predict', 'flatten'])

[13]:

frames_vad = [(frame, result['flatten'][no]) for no, frame in enumerate(frames)]
grouped_vad = malaya_speech.utils.group.group_frames(frames_vad)
grouped_vad = malaya_speech.utils.group.group_frames_threshold(grouped_vad, threshold_to_stop = 0.3)

[14]:

malaya_speech.extra.visualization.visualize_vad(y, grouped_vad, sr, figsize = (15, 3))
_images/load-gender_25_0.png 
[15]:

p_vggvox_v2 = Pipeline()
pipeline = (
    p_vggvox_v2.foreach_map(vggvox_v2)
    .flatten()
)
p_vggvox_v2.visualize()

[15]:
_images/load-gender_26_0.png 
[16]:

p_deep_speaker = Pipeline()
pipeline = (
    p_deep_speaker.foreach_map(deep_speaker)
    .flatten()
)
p_deep_speaker.visualize()

[16]:
_images/load-gender_27_0.png 
[17]:

%%time

samples_vad = [g[0] for g in grouped_vad]
result_vggvox_v2 = p_vggvox_v2.emit(samples_vad)
result_vggvox_v2.keys()

CPU times: user 5.31 s, sys: 1.32 s, total: 6.63 s
Wall time: 2.29 s

[17]:

dict_keys(['gender', 'flatten'])

[18]:

%%time

samples_vad = [g[0] for g in grouped_vad]
result_deep_speaker = p_deep_speaker.emit(samples_vad)
result_deep_speaker.keys()

CPU times: user 6.2 s, sys: 1.74 s, total: 7.94 s
Wall time: 3.79 s

[18]:

dict_keys(['gender', 'flatten'])

[19]:

samples_vad_vggvox_v2 = [(frame, result_vggvox_v2['flatten'][no]) for no, frame in enumerate(samples_vad)]
samples_vad_vggvox_v2

[19]:

[(<malaya_speech.model.frame.Frame at 0x15953de50>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x1595503d0>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x159550390>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550450>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550490>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550510>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550550>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550590>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550610>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x1595505d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x1595504d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550650>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x159550690>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x1595506d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550710>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x159550790>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550750>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550810>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550850>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x1595507d0>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x1595508d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550890>, 'male')]

[20]:

samples_vad_deep_speaker = [(frame, result_deep_speaker['flatten'][no]) for no, frame in enumerate(samples_vad)]
samples_vad_deep_speaker

[20]:

[(<malaya_speech.model.frame.Frame at 0x15953de50>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x1595503d0>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x159550390>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550450>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550490>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550510>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550550>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550590>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550610>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x1595505d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x1595504d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550650>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x159550690>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x1595506d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550710>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550790>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550750>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550810>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550850>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x1595507d0>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x1595508d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550890>, 'male')]

[21]:

import matplotlib.pyplot as plt

[22]:

nrows = 3
fig, ax = plt.subplots(nrows = nrows, ncols = 1)
fig.set_figwidth(20)
fig.set_figheight(nrows * 3)
malaya_speech.extra.visualization.visualize_vad(y, grouped_vad, sr, ax = ax[0])
malaya_speech.extra.visualization.plot_classification(samples_vad_vggvox_v2,
                                                      'emotion detection vggvox v2', ax = ax[1])
malaya_speech.extra.visualization.plot_classification(samples_vad_deep_speaker,
                                                      'emotion detection deep speaker', ax = ax[2])
fig.tight_layout()
plt.show()
_images/load-gender_33_0.png 
[23]:

p_quantized_vggvox_v2 = Pipeline()
pipeline = (
    p_quantized_vggvox_v2.foreach_map(quantized_vggvox_v2)
    .flatten()
)
p_quantized_vggvox_v2.visualize()

[23]:
_images/load-gender_34_0.png 
[24]:

%%time

samples_vad = [g[0] for g in grouped_vad]
result_quantized_vggvox_v2 = p_quantized_vggvox_v2.emit(samples_vad)
result_quantized_vggvox_v2.keys()

CPU times: user 4.07 s, sys: 1.41 s, total: 5.48 s
Wall time: 2.1 s

[24]:

dict_keys(['gender', 'flatten'])

[25]:

samples_vad_quantized_vggvox_v2 = [(frame, result_quantized_vggvox_v2['flatten'][no]) for no, frame in enumerate(samples_vad)]
samples_vad_quantized_vggvox_v2

[25]:

[(<malaya_speech.model.frame.Frame at 0x15953de50>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x1595503d0>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x159550390>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550450>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550490>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550510>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550550>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550590>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x159550610>, 'female'),
 (<malaya_speech.model.frame.Frame at 0x1595505d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x1595504d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550650>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x159550690>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x1595506d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550710>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x159550790>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550750>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550810>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550850>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x1595507d0>, 'not a gender'),
 (<malaya_speech.model.frame.Frame at 0x1595508d0>, 'male'),
 (<malaya_speech.model.frame.Frame at 0x159550890>, 'male')]

[26]:

nrows = 3
fig, ax = plt.subplots(nrows = nrows, ncols = 1)
fig.set_figwidth(20)
fig.set_figheight(nrows * 3)
malaya_speech.extra.visualization.visualize_vad(y, grouped_vad, sr, ax = ax[0])
malaya_speech.extra.visualization.plot_classification(samples_vad_vggvox_v2,
                                                      'gender detection vggvox v2', ax = ax[1])
malaya_speech.extra.visualization.plot_classification(samples_vad_quantized_vggvox_v2,
                                                      'gender detection quantized vggvox v2', ax = ax[2])
fig.tight_layout()
plt.show()
_images/load-gender_37_0.png

Reference#

  1. The Singaporean White Boy - The Shan and Rozz Show: EP7, https://www.youtube.com/watch?v=HylaY5e1awo&t=2s&ab_channel=Clicknetwork

  2. Common Voice dataset, https://commonvoice.mozilla.org/