Pathological Voice Detection with Nkululeko

I tried to add a recipe to Nkululeko for pathological voice detection using the Saarbrueken Voice Database (SVD, the dataset currently perhaps cannot be downloaded due to a server problem). Using Nkululeko is easy; I just need two working days (with a lot of play and others) to process the data and get the initial result. The evaluation was mainly measured with F1-Score (Macro). The initial result is an F1-score of 71% obtained using open smile ('os') and SVM without any customization.

On the third day, I made some modifications, but I am still using the 'os' feature (so I don't need to extract the feature again, using the same 'exp' directory to boost my experiments). The best F1 score I achieved was 76% (macro). This time, the modifications were: without feature scaling, feature balancing with smote, and using XGB as a classifier. My goal is to obtain an F1 score of 80% before the end of the fiscal year (March 2025).

Here is the configuration (INI) file and a sample of outputs (terminal).

Configuration file:

[EXP]
root = /tmp/results/
name = exp_os
[DATA]
databases = ['train', 'dev', 'test']
train = ./data/svd/svd_a_train.csv
train.type = csv
train.absolute_path = True
train.split_strategy = train
train.audio_path = /data/SaarbrueckenVoiceDatabase/export_16k
dev = ./data/svd/svd_a_dev.csv
dev.type = csv
dev.absolute_path = True
dev.split_strategy = train
dev.audio_path = /data/SaarbrueckenVoiceDatabase/export_16k
test = ./data/svd/svd_a_test.csv
test.type = csv
test.absolute_path = True
test.split_strategy = test
test.audio_path = /data/SaarbrueckenVoiceDatabase/export_16k
target = label
; no_reuse = True
; labels = ['angry', 'calm', 'sad']
; get the number of classes from the target column automatically
[FEATS]
; type = ['wav2vec2']
; type = ['hubert-large-ll60k']
; type = []
type = ['os']
; scale = standard
balancing = smote
; no_reuse = True
[MODEL]
type = xgb

Outputs:

$ python3 -m nkululeko.nkululeko --config exp_svd/exp_os.ini 
DEBUG: nkululeko: running exp_os from config exp_svd/exp_os.ini, nkululeko version 0.88.12
DEBUG: dataset: loading train
DEBUG: dataset: Loaded database train with 1650 samples: got targets: True, got speakers: False (0), got sexes: False, got age: False
DEBUG: dataset: converting to segmented index, this might take a while...
DEBUG: dataset: loading dev
DEBUG: dataset: Loaded database dev with 192 samples: got targets: True, got speakers: False (0), got sexes: False, got age: False
DEBUG: dataset: converting to segmented index, this might take a while...
DEBUG: dataset: loading test
DEBUG: dataset: Loaded database test with 190 samples: got targets: True, got speakers: False (0), got sexes: False, got age: False
DEBUG: dataset: converting to segmented index, this might take a while...
DEBUG: experiment: target: label
DEBUG: experiment: Labels (from database): ['n', 'p']
DEBUG: experiment: loaded databases train,dev,test
DEBUG: experiment: reusing previously stored /tmp/results/exp_os/./store/testdf.csv and /tmp/results/exp_os/./store/traindf.csv
DEBUG: experiment: value for type is not found, using default: dummy
DEBUG: experiment: Categories test (nd.array): ['n' 'p']
DEBUG: experiment: Categories train (nd.array): ['n' 'p']
DEBUG: nkululeko: train shape : (1842, 3), test shape:(190, 3)
DEBUG: featureset: value for n_jobs is not found, using default: 8
DEBUG: featureset: reusing extracted OS features: /tmp/results/exp_os/./store/train_dev_test_os_train.pkl.
DEBUG: featureset: value for n_jobs is not found, using default: 8
DEBUG: featureset: reusing extracted OS features: /tmp/results/exp_os/./store/train_dev_test_os_test.pkl.
DEBUG: experiment: All features: train shape : (1842, 88), test shape:(190, 88)
DEBUG: experiment: scaler: False
DEBUG: runmanager: value for runs is not found, using default: 1
DEBUG: runmanager: run 0 using model xgb
DEBUG: modelrunner: balancing the training features with: smote
DEBUG: modelrunner: balanced with: smote, new size: 2448 (was 1842)
DEBUG: modelrunner: {'n': 1224, 'p': 1224})
DEBUG: model: value for n_jobs is not found, using default: 8
DEBUG: modelrunner: value for epochs is not found, using default: 1
DEBUG: modelrunner: run: 0 epoch: 0: result: test: 0.771 UAR
DEBUG: modelrunner: plotting confusion matrix to train_dev_test_label_xgb_os_balancing-smote_0_000_cnf
DEBUG: reporter: Saved confusion plot to /tmp/results/exp_os/./images/run_0/train_dev_test_label_xgb_os_balancing-smote_0_000_cnf.png
DEBUG: reporter: Best score at epoch: 0, UAR: .77, (+-.704/.828), ACC: .773
DEBUG: reporter: 
               precision    recall  f1-score   support

           n       0.65      0.76      0.70        67
           p       0.86      0.78      0.82       123

    accuracy                           0.77       190
   macro avg       0.76      0.77      0.76       190
weighted avg       0.79      0.77      0.78       190

DEBUG: reporter: labels: ['n', 'p']
DEBUG: reporter: Saved ROC curve to /tmp/results/exp_os/./results/run_0/train_dev_test_label_xgb_os_balancing-smote_0_roc.png
DEBUG: reporter: auc: 0.771, pauc: 0.560 from epoch: 0
DEBUG: reporter: result per class (F1 score): [0.703, 0.817] from epoch: 0
DEBUG: experiment: Done, used 11.065 seconds
DONE

Update 2024/08/30: 

• Using praat + xgb (no scaling, balancing: smote) achieves higher F1-score, i.e.,  78% (macro) and 79% (weighted)
   

Ekstraksi fitur MFCC dan zero paddingnya dengan library LIBROSA

Mel frequency cepstral coefficient, disingkat MFCC, masih merupakan fitur yang paling banyak digunakan pada pemrosesan sinyal wicara, khususnya pengenalan sinyal wicara. Untuk mengekstrak fitur MFCC, salah satu tool yang paling banyak digunakan adalah librosa. Tulisan berikut merupakan paparan singkat untuk mengekstrak fitur MFCC dari set sinyal wicara dalam sebuah direktori.

Instalasi

pip3 install --user librosa

Workflow
Misalkan dalam direktory saat ini (`./`), kita memiliki 30 file wav sinyal wicara. Dari 30 file tersebut, tiap file akan kita ekstrak 20 MFCC per frame/window (dari default librosa). Karena panjang tiap file berbeda, maka bentuk/ukuran variabel MFCC untuk tiap file tersebut berbeda, misalnya: (20, 44), (20, 193) dan (20, 102).  Karena default window size atau hopsize (n_fft) dari librosa adalah 2048 (25ms) dengan 512 hop length /stride (10 ms overlap), maka untuk file suara dengan durasi 2 detik banyaknya frame mffc yang didapat adalah,

output _length = (seconds) * (sample rate) / (hop_length)
output_length = 2 * 22050/512 = 87 --> (20, 87)

Jika panjang file 3 detik maka panjang output_length adalah,

output_length = 3 * 22050/512 = 130 --> (20, 130)

Variabel lebar window and stride untuk memproses MFCC pada librosa bisa dikontrol dengan argumen n_fft dan hop_length. Misal n_fft = int(sr/40), hop_lenghth=int(sr/100) untuk lebar window 25 ms dan hop length (geser ke kanan) sebesar 10 ms. Perhatikan gambar berikut untuk lebih jelasnya, K adalah lebar window (n_fft) dan Q adalah stride atau hop length (disebut juga hop size). Pada beberapa literatur sinyal sistem, variabel pergeseran antar window memakai nilai % overlap yang pada gambar dibawah disimbolkan dengan 0. Jika default hop_length=512 dan n_fft = 2048, maka overlap defaultnya adalah 75%.

Windowing: window length, hop size, and its overlap.

Jadi, by default MFCC yang dihasilkan oleh librosa berbeda ukurannya bergantung pada ukuran file input. Padahal, agar bisa diproses oleh deep learning, kita inginkan panjang semua variabel MFCC tersebut sama, misal (20, 100). Jika kurang kita zero-paddingkan, jika lebih kita potong sampai 100 sample sehingga menjadi (20, 100). Karena input deeplearning berupa (sample, fitur), bentuk (20, 100) kita transpose menjadi (100, 20). Tujuan akhir kita adalah memperoleh bentuk dari variabel fitur berupa (30, 100, 20) yang siap dimasukkan pada arsitektur deep learning.

Berikut caranya:

import os
import librosa

lenmin = 100

mfcc =[]
for i in os.listdir('.'):
    x_i, sr = librosa.load(i, sr=None)
    mfcc_i = librosa.feature.mfcc(x_i, sr)
    if mfcc_i.shape[1] < lenmin:
        mfcc_i = np.hstack((mfcc_i, np.zeros((mfcc_i.shape[0], lenmin-mfcc_i.shape[1]))))
    mfcc_i = mfcc_i[:,:lenmin]
    mfcc.append(mfcc_i.T)

mfcc = np.array(mfcc)

Cek ukuran variable mfcc dengan:

In [1]: np_mfcc.shape
Out[1]: (30, 100, 20)

Mengekstrak Delta dan Delta-delta MFCC
Pada banyak aplikasi pemrosesan sinyal wicara, tidak hanya MFCC yang dipakai sebagai fitur, namun juga perbedaan antar koefisien MFCC (delta) dan perbedaan antar delta MFCC (delta-delta). Librosa menyediakan fungsi untuk mengekstrak kedua fitur tersebut.

Untuk mengekstrak delta MFCC dari MFCC:

delta = librosa.feature.delta(mfcc)

Untuk mengekstrak delta-delta dari MFCC:

deltad = librosa.feature.delta(MFCC, order=2)

Zero padding dengan Tensorflow 

Ada cara yang lebih singkat untuk zero padding, yakni dengan tensorflow. Dengan tensorflow, kita hanya butuh satu perintah (dan satu baris jika memungkinkan). Contoh jika kita mempunyai list MFCC dengan ukuran seperti [(100, 20), (28, 20), (53,20), ...]. Semua MFCC tersebut kita rubah menjadi ukuran (100, 200) dengan cara berikut.

import tensorflow as tf
mfcc_padded = tf.keras.preprocessing.sequence.pad_sequence(mfcc, maxlen=100, dtype=float)

Dimana mfcc_list merupakan list MFCC yang berbeda-beda ukuran dan hasilnya, mfcc_padded, merupakan list MFCC yang kesemuanya berukuran (100, 20) melalui teknik zero padding.

SPTK basic operation: .wav, .raw., .short

The Speech Signal Processing Toolkit (SPTK) is a suite of speech signal processing tools for UNIX environments, e.g., LPC analysis, PARCOR analysis, LSP analysis, PARCOR synthesis filter, LSP synthesis filter, vector quantization techniques, and other extended versions of them. It is developed by Prof. Imai and Prof. Kobayashi of Tokyo Inst. of Tech, and currently maintained by Nagoya Inst. of Tech. To install SPTK in Linux and Unix based-OS is easy, download the source file, extract and do the following,

./configure
make
sudo make install

That's all installation process, it will install SPTK in /usr/local/SPTK by default.

Add SPTK path to .bashrc

Now SPTK is installed in our machine, but by default it is (SPTK command) not searchable in our shell/terminal. To make it searchable (instead of use /usr/local/SPTK/bin/command everytime), we can add the path to .bashrc. Open .bashrc with editor (vim/gedit/other) and add the following,

export PATH=$PATH:/usr/local/SPTK/bin

And it's added to our command path, check it by using "impulse -h" in terminal. If done, it will show man pages of "impulse" command.

Remove Header : Convert Wav to Raw

SPTK provide utilities to remove header file in sound recording process by converting wav files to raw files (and we can convert it back to wav file). Here is how,

wav2raw +s data.wav

In current directory, there will be new file data.raw with header file removed. Argumen +s is for short data type, you can check full argument by wav2raw -h. To convert back in wav, use "raw2wav" command.

Convert to Short

To convert data from raw, wav or other format to .short (because mainly processing in SPTK is in .short format) we use command "x2x",

x2x +s data.raw > data.short 

It will covert raw to short, you can try to change other format.

Plot Waveform

Plot waveform in SPTK