Menginstall ESPNET via Conda

Tulisan berikut merupakan catatan singkat instalasi ESPNET dengan Conda (OS: Ubuntu 20.04~).

Dari dokumentasi ESPNET, cara yang disarankan untuk menginsall ESPNET adalah melalui Conda.

./setup_anaconda.sh miniconda espnet 3.8

Namun dengan cara ini, environment conda menjadi tidak bernama sehingga kita perlu me-load conda dengan fullpath. Solusinya adalah dengan memberikan argumen yang tepat untuk `setup_anaconda.sh`, yakni $CONDA_ROOT. Pada Ubuntu, default CONDA_ROOT ada di `/home/$USER/miniconda3` (Setelah menginstall miniconda). Contohnya adalah sebagai berikut.

./setup_anaconda.sh /home/bagus/miniconda3/ espnet 3.9

Dengan cara ini kita bisa berpindah ke perintah instalasi selanjutnya, yakni `make`. Setelah terinstall, kita bisa mengaktifkan ESPNET dengan `conda activate espnet`. 

 

Catatan:

- Sebelum menginstall ESPNET, kita perlu menginstall cmake, python3-dev, sox, flac, dan build-essential via apt-get.

Konversi Fail Stereo ke Mono Dari Direktori Berisi Banyak Fail Suara

Skrip berikut dapat merubah fail suara stereo ke mono dari suatu direktori (termasuk subdirektori di dalamnya) yang diberikan.
Input: Direktori/folder
Output: Fail berakhiran "_mono" dengan direktori yang sama terhadap input (termasuk subdirektori)

#!/usr/bin/env python3
import os
import argparse
import glob
from pydub import AudioSegment

def stereo2mono(files):
    """Convert all files from stereo to mono.
    Note: this would effectively also create a copy of files that were already in a mono format
    Parameters
    ----------
    files : iterable
        Sequence of files
    Example use:
    ```
    $ python3 stereo2mono.py -f /path/to/audio/files/
    ```
    Then you may remove the files that do not contain the '_mono' tag with: 
    $ find . -name "*-??.wav" -delete # for emovo
    """
    for f in files:
        print(f"Converting {f}")
        # Load audio
        sound = AudioSegment.from_wav(f)
        # Convert to mono
        sound = sound.set_channels(1)
        # Save file
        stem, ext = os.path.splitext(f)
        sound.export(f'{stem}_mono{ext}', format='wav')


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Convert stereo to mono')
    parser.add_argument('-f', '--folder', 
        type=str, help='Path to wavfiles')
    args = parser.parse_args()
    files = glob.glob(args.folder + '**/*.wav', recursive=True)
    stereo2mono(files)

Contoh penggunaan

# Konversi ke mono
python3 stereo2mono.py -f tensorflow_datasets/downloads/extracted/ZIP.emovo.zip/EMOVO/
# menghapus file yang bukan mono
$ pwd
/home/bagus/tensorflow_datasets/downloads/extracted/ZIP.emovo.zip/EMOVO/
$ find . -name "*-??.wav" -delete

Referensi gist:
1. https://gist.github.com/bagustris/40b406d99820207bc804a020db169f7e

Acoustic Feature Extraction with Transformers

The example in Transformers' documentation here shows how to use the wav2vec 2.0 model for automatic speech recognition. However, there are two crucial issues in that example. First, we usually use our data (set) instead of their (available) dataset. Second, we need to extract acoustic features (the last hidden states instead of logits). The following is my example of adapting Transformers to extract acoustic embedding given any audio file (WAVE) using several models. It includes the pooling average from frame-based processing to utterance-based processing for given any audio file. You don't need to perform the pooling average if you want to process your audio file in frame-based processing (remove the `.mean(axis=0)` in the variable `last_hidden_states`).

Basic syntax: wav2vec2 base model

This is the example from the documentation. I replaced the use of the dataset with the defined path of the audio file ('00001.wav').

from transformers import Wav2Vec2Processor, Wav2Vec2Model
import torchaudio
import torch
# load model
processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")
model = Wav2Vec2Model.from_pretrained("facebook/wav2vec2-base-960h")

# audio file is decoded on the fly
array, fs = torchaudio.load("/data/A-VB/audio/wav/00001.wav")
input = processor(array.squeeze(), sampling_rate=fs, return_tensors="pt")

# apply the model to the input array from wav
with torch.no_grad():
    outputs = model(**input)

# extract last hidden state, compute average, convert to numpy
last_hidden_states = outputs.last_hidden_state.squeeze().mean(axis=0).numpy()

# print shape
print(f"Hidden state shape: {last_hidden_states.shape}")
# Hidden state shape: (768,)

The syntax for the wav2vec2 large and robust model

In this second example, I replace the base model with the large and robust model without finetuning. This example is adapted from here. Note that I replaced 'Wav2Vec2ForCTC' with 'wav2vec2Model'. The former is used when we want to obtain the logits (for speech-to-text transcription) instead of obtaining the hidden states.

from transformers import Wav2Vec2Processor, Wav2Vec2Model
import torch
import torchaudio

# load model
processor = Wav2Vec2Processor.from_pretrained(
    "facebook/wav2vec2-large-robust-ft-swbd-300h")
model = Wav2Vec2Model.from_pretrained(
    "facebook/wav2vec2-large-robust-ft-swbd-300h")

# audio file is decoded on the fly
array, fs = torchaudio.load("/data/A-VB/audio/wav/00001.wav")
input = processor(array.squeeze(), sampling_rate=fs, return_tensors="pt")

with torch.no_grad():
    outputs = model(**input)

last_hidden_states = outputs.last_hidden_state.squeeze().mean(axis=0).numpy()
# printh shape
print(f"Hidden state shape: {last_hidden_states.shape}")

You can replace "facebook/wav2vec2-large-robust-ft-swbd-300h" with "facebook/wav2vec2-large-robust-ft-libri-960h" for the larger fine-tuned model.

 For other models, you may need to change `Wav2Vec2Processor` with `Wav2Vec2FeatureExtractor` for processor variable. In my case, this is needed for the following models:

• facebook/wav2vec2-large-robust
• facebook/wav2vec2-large-xlsr-53

The syntax for the custom model (wav2vec-R-emo-vad)

The last one is the example of the custom model. The model is wav2vec 2.0 fine-tuned on the MSP-Podcast dataset for speech emotion recognition. This last example differs from the previous one since the configuration is given by the authors of the model (read the code thoroughly to inspect the details). I replaced the dummy audio file with the real audio file. It is assumed to process in batch (with batch_size=2) by replicating the same audio file.

import torch
import torch.nn as nn
from transformers import Wav2Vec2Processor
from transformers.models.wav2vec2.modeling_wav2vec2 import (
    Wav2Vec2Model,
    Wav2Vec2PreTrainedModel,
)
import torchaudio


class RegressionHead(nn.Module):
    r"""Classification head."""

    def __init__(self, config):

        super().__init__()

        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.dropout = nn.Dropout(config.final_dropout)
        self.out_proj = nn.Linear(config.hidden_size, config.num_labels)

    def forward(self, features, **kwargs):

        x = features
        x = self.dropout(x)
        x = self.dense(x)
        x = torch.tanh(x)
        x = self.dropout(x)
        x = self.out_proj(x)

        return x


class EmotionModel(Wav2Vec2PreTrainedModel):
    r"""Speech emotion classifier."""

    def __init__(self, config):

        super().__init__(config)

        self.config = config
        self.wav2vec2 = Wav2Vec2Model(config)
        self.classifier = RegressionHead(config)
        self.init_weights()

    def forward(
            self,
            input_values,
    ):

        outputs = self.wav2vec2(input_values)
        hidden_states = outputs[0]
        hidden_states = torch.mean(hidden_states, dim=1)
        logits = self.classifier(hidden_states)

        return hidden_states, logits


def process_func(
    wavs,
    sampling_rate: int
    # embeddings: bool = False,
):
    r"""Predict emotions or extract embeddings from raw audio signal."""

    # run through processor to normalize signal
    # always returns a batch, so we just get the first entry
    # then we put it on the device
    # wavs = pad_sequence(wavs, batch_first=True)
    # load model from hub
    device = 'cpu'
    model_name = 'audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim'
    processor = Wav2Vec2Processor.from_pretrained(model_name)
    model = EmotionModel.from_pretrained(model_name)

    y = processor([wav.cpu().numpy() for wav in wavs],          
                   sampling_rate=sampling_rate,
                   return_tensors="pt",
                   padding="longest"
        )
    y = y['input_values']
    y = y.to(device)


    y = model(y)

    return {
        'hidden_states': y[0],
        'logits': y[1],
    }


## test to an audiofile
sampling_rate = 16000
signal = [torchaudio.load('train_001.wav')[0].squeeze().to('cpu') for _ in range(2)]

# extract hidden states
with torch.no_grad():
    hs = process_func(signal, sampling_rate)['hidden_states']
print(f"Hidden states shape={hs.shape}")

Please note for all models, the audio file must be sampled with 16000 Hz, otherwise, you must resample it before extracting acoustic embedding using the methods above. It may not throw an error even if the sampling rate is not 16000 Hz but the results, hence, is not valid since all models were generated based on 16 kHz of sampling rate speech datasets. 

You may also want to extract acoustic features using the opensmile toolkit. The tutorial for Windows users using WSL is available here: http://bagustris.blogspot.com/2021/08/extracting-emobase-feature-using-python.html.

Happy reading. Don't wait for more time to apply these methods to your own audio file.

Basic Audio Manipulation With Torchaudio

Recently, I moved my audio processing toolkit from librosa (and others) to Torchaudio. This short writing documented the very basics of torchaudio for audio manipulation: read, resample, and write an audiofile.

Load audio file (read)

The process of loading (reading) an audio file is straightforward, just pass the audio path to `torchaudio.load`. We need to import the needed modules first. Most audio files can be loaded by torchaudio (WAV, OGG, MP3, etc.).

import torchaudio
import torchaudio.transforms as T
wav0, sr0 = torchaudio.load("old_file_48k.wav", normalize=True) 

where wav0 is the output tensor (array) and sr0 is the original sampling rate. Argument `normalize=True` is optional to normalize the waveform. Note that one of my colleagues (a student) found that using `librosa.util.normalize()` resulted in better normalization (peak to peak waveform is -1 to 1) than this torchaudio normalization.

 

Resample

Resample a sampling rate to another sampling rate is done by a Class; the output is a function. Hence, we need to pass the old tensor to the resampler function. Here is an example to convert 48k tensor to 16k tensor.

sr1 = 16000
resampler = T.Resample(sr0, sr1)
wav1 = resampler(wav0)

Save as a new audio file (write)

The process of saving files is also straightforward, just pass the file name, tensor, and sampling rate in order.

torchaudio.save('new_file_16k.wav', wav1, sr1)

Then the new audio file appeared in the current directory. Just set the path and file name if you want to save it in another directory.

 

Reference:

[1] https://pytorch.org/tutorials/beginner/audio_preprocessing_tutorial.html

How-to: Install Numba and LibROSA in Jetson AGX Xavier (Ubuntu 18.04)

Installing Numba and Librosa in Jetson AGX Xavier is very painful. To document my experience, I wrote this note. Here is how-to install Numba and Librosa on AGX Xavier.

Device: AGX Xavier Dev Kit (32GB RAM)
Software
OS: Ubuntu 18.04
nvidia-jetpack: Version: 4.4-b186

python: Version 3.6.9

llvm: Version 9.0.1 (and 7.1.0)

First, we need to install llvmlite. We can install either from source or via apt. I installed it from the source as shown below.

cd /tmp
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/llvm-9.0.1.src.tar.xz
tar -xvf llvm-9.0.1.src.tar.xz 
cd llvm-9.0.1.src/
mkdir llvm_build_dir
cd llvm_build_dir
cmake ../ -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="ARM;X86;AArch64"
make -j4
sudo make install
cd bin/
echo "export LLVM_CONFIG=\""`pwd`"/llvm-config\"" >> ~/.bashrc
echo "alias llvm='"`pwd`"/llvm-lit'" >> ~/.bashrc
source ~/.bashrc
python3.6 -m pip install --user -U llvmlite==0.31

If you want to install via apt, do it as follows. I use llvm-7 from Ubuntu repository (llvm-9 fails installing llvmlite 0.31 but success for 0.33).

sudo apt install llvm-7

If you install llvm via apt, you need to specify where llvm-config is located in .bashrc. For example here is mine. First, locate llvm-config.

s1820002@s1820002-desktop:~$ locate llvm-config
/usr/bin/llvm-config-7
/usr/include/llvm-7/llvm/Config/llvm-config.h
/usr/lib/llvm-7/bin/llvm-config

Then add the location of llvm-config to .bashrc.

export LLVM_CONFIG=/usr/lib/llvm-7/bin/llvm-config

Please be remembered that you need to `source .bashrc` after you edit your .bashrc.

As an alternative to using .bashrc, you can make softlink by mapping llvm-config-9 to llvm-config.

$ cd /usr/bin
$ sudo ln llvm-config-9 llvm-config

The installation of llvm must success before installing llvmlite. If installation of llvmlite by the last line on the script above success, it shows like this one,

Collecting llvmlite
Installing collected packages: llvmlite
Successfully installed llvmlite-0.33.0

If you face an error regarding `setuptools`, you may upgrade your setuptools as follows.

python3.6 -m pip install --user -U setuptools

Now, we can install numba. We need numba version 0.48 to install librosa 0.7.2. To install Numba, first, we must disable tbb.h according to this.

sudo mv /usr/include/tbb/tbb.h /usr/include/tbb/tbb.h.bak

Then, install numba as the following:

bagus@s1820002:~$ python3.6 -m pip install --user -U numba==0.48
Collecting numba==0.48
Collecting setuptools (from numba==0.48) Using cached
 https://files.pythonhosted.org/packages/41/fa/60888a1d591db07bc9c17dce2bcfb9f00ac507c0a23ecb827e76feb8f816/setuptools-49.1.0-py3-none-any.whl
Collecting numpy>=1.15 (from numba==0.48)
Collecting llvmlite<0 .32.0="">=0.31.0dev0 (from numba==0.48)
Installing collected packages: setuptools, numpy, llvmlite, numba
Successfully installed llvmlite-0.33.0 numba-0.50.1 numpy-1.19.0 setuptools-49.1.0

Again, it must install numba without fail to be able to install Librosa.
Before installing librosa, the following packages must be installed via apt.

sudo apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran

Finally, we can install librosa via pip as below.

bagus@s1820002:~$ python3.6 -m pip install --user -U librosa
Collecting librosa
Collecting resampy>=0.2.2 (from librosa)
Collecting numpy>=1.15.0 (from librosa)
Collecting numba>=0.43.0 (from librosa)
Collecting scikit-learn!=0.19.0,>=0.14.0 (from librosa)
Collecting joblib>=0.12 (from librosa) Using cached
 https://files.pythonhosted.org/packages/51/dd/0e015051b4a27ec5a58b02ab774059f3289a94b0906f880a3f9507e74f38/joblib-0.16.0-py3-none-any.whl
Collecting scipy>=1.0.0 (from librosa)
Collecting soundfile>=0.9.0 (from librosa) Using cached
 https://files.pythonhosted.org/packages/eb/f2/3cbbbf3b96fb9fa91582c438b574cff3f45b29c772f94c400e2c99ef5db9/SoundFile-0.10.3.post1-py2.py3-none-any.whl
Collecting decorator>=3.0.0 (from librosa) Using cached
 https://files.pythonhosted.org/packages/ed/1b/72a1821152d07cf1d8b6fce298aeb06a7eb90f4d6d41acec9861e7cc6df0/decorator-4.4.2-py2.py3-none-any.whl
Collecting six>=1.3 (from librosa) Using cached
 https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting audioread>=2.0.0 (from librosa)
Collecting llvmlite<0 .34="">=0.33.0.dev0 (from numba>=0.43.0->librosa)
Collecting setuptools (from numba>=0.43.0->librosa) Using cached
 https://files.pythonhosted.org/packages/41/fa/60888a1d591db07bc9c17dce2bcfb9f00ac507c0a23ecb827e76feb8f816/setuptools-49.1.0-py3-none-any.whl
Collecting threadpoolctl>=2.0.0 (from scikit-learn!=0.19.0,>=0.14.0->librosa) Using cached
 https://files.pythonhosted.org/packages/f7/12/ec3f2e203afa394a149911729357aa48affc59c20e2c1c8297a60f33f133/threadpoolctl-2.1.0-py3-none-any.whl
Collecting cffi>=1.0 (from soundfile>=0.9.0->librosa)
Collecting pycparser (from cffi>=1.0->soundfile>=0.9.0->librosa) Using cached
 https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
Installing collected packages: six, numpy, scipy, llvmlite, setuptools, numba, resampy, threadpoolctl, joblib, scikit-learn, pycparser, cffi, soundfile, decorator, audioread, librosa
Successfully installed audioread-2.1.8 cffi-1.14.0 decorator-4.4.2 joblib-0.16.0 librosa-0.7.2 llvmlite-0.33.0 numba-0.50.1 numpy-1.19.0 pycparser-2.20 resampy-0.2.2 scikit-learn-0.23.1 scipy-1.5.1 setuptools-49.1.0 six-1.15.0 soundfile-0.10.3.post1 threadpoolctl-2.1.0

Now everything seems good. We can do audio processing in Jetson AGX. Librosa is the best audio library in python so far. Having it installed on Jetson is a basic requirement.

BONUS: Installing TensorFlow
Step-by-step:
1. Install the following packages.

sudo apt-get install libhdf5-serial-dev hdf5-tools libhdf5-dev zlib1g-dev zip libjpeg8-dev

2. Install/upgrade testresources

python3.6 -m pip install --user -U testresources

3. Installing other python dependencies

python3.6 -m pip future==0.17.1 mock==3.0.5 h5py==2.9.0 keras_preprocessing==1.0.5 keras_applications==1.0.8 gast==0.2.2 futures protobuf pybind11

4. Install tensorflow.

python3.6 -m pip install --user --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 'tensorflow<2' 

output:

Successfully installed absl-py-0.9.0 astor-0.8.1 google-pasta-0.2.0 grpcio-1.30.0 importlib-metadata-1.7.0 markdown-3.2.2 opt-einsum-3.2.1 tensorboard-1.15.0 tensorflow-1.15.2+nv20.3.tf1 tensorflow-estimator-1.15.1 termcolor-1.1.0 werkzeug-1.0.1 wrapt-1.12.1 zipp-3.1.0

5. Test the GPU device.

import tensorflow as tf
tf.test.gpu_device_name()

output:

>>> tf.test.gpu_device_name()
2020-07-10 16:01:36.242822: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-07-10 16:01:36.244367: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x37b6c3f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-10 16:01:36.244536: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-07-10 16:01:36.259171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-07-10 16:01:36.432077: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-07-10 16:01:36.432674: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x37a81480 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-07-10 16:01:36.432768: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Xavier, Compute Capability 7.2
2020-07-10 16:01:36.433580: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-07-10 16:01:36.433892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Xavier major: 7 minor: 2 memoryClockRate(GHz): 1.377
pciBusID: 0000:00:00.0
2020-07-10 16:01:36.433987: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-07-10 16:01:36.467839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-07-10 16:01:36.496813: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-07-10 16:01:36.534413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-07-10 16:01:36.576197: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-07-10 16:01:36.600464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-07-10 16:01:36.688185: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-07-10 16:01:36.688573: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-07-10 16:01:36.690082: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-07-10 16:01:36.690253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-07-10 16:01:36.690456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-07-10 16:01:38.484495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-10 16:01:38.484668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-07-10 16:01:38.484715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-07-10 16:01:38.485215: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-07-10 16:01:38.485695: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-07-10 16:01:38.485953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/device:GPU:0 with 23699 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
'/device:GPU:0

Source:
[1] https://github.com/jefflgaol/Install-Packages-Jetson-ARM-Family
[2] https://forums.developer.nvidia.com/t/install-python-packages-librosa-jetson-tx2-developer-kit-problem/126337/5
[3] https://learninone209186366.wordpress.com/2019/07/24/how-to-install-the-librosa-library-in-jetson-nano-or-aarch64-module/

Pengolahan sinyal auditori [2]

Ini adalah catatan kuliah yang saya transkrip ketika kuliah berlangsung: I656 - Human Perceptual Systems and its Models, pertemuan kedua.

Sistem auditori merupakan sistem pendengaran manusia.
Video tentang sistem auditori berikut memvisualisasikan tulisan di bawah ini:

Telinga manusia terbagi menjadi 3:
1. telinga bagian luar
2. telinga bagian tengah
3. telinga bagian dalam

Gambar berikut menjelaskan bagian-bagian telinga tersebut yang akan dijelaskan lebih detil di bawahnya.

Speech enhancement on smartphone Voice Recording

Abstract

Speech enhancement is a challenging task in audio signal processing to enhance the quality of targeted speech signal while suppressing other noises. In the beginning, the speech enhancement algorithm growth rapidly from spectral subtraction, Wiener filtering, spectral amplitude MMSE estimator to Non-negative Matrix Factorization (NMF). Smartphone as a revolutionary device now is being used in all aspect of life including journalism; personally and professionally. Although many smartphones have two microphones (main and rear) the only main microphone is widely used for voice recording. This is why the NMF algorithm widely used for this purpose of speech enhancement. This paper evaluates speech enhancement on smartphone voice recording by using some algorithms mentioned previously. We also extend the NMF algorithm to Kulback-Leibler NMF with supervised separation. The last algorithm shows improved result compared to others by spectrogram and PESQ score evaluation.

For full paper please download here (submitted to ICOPIA 2016). For Octave code (obtained from single source separation: mini tutorial), and latex code please refer to this link.

Forensik Suara di Indonesia: Past, Present and Future

Berikut adalah resume singkat saya saat mengikuti "Workshop Forensik Suara Ucap di Indonesia: Pas, Present and Future" yang diorganisir oleh Kelompok Keahlian Instrumentasi dan Kontrol, Teknik Fisika ITB. Workshop ini dibagi menjadi tiga sesi, sesi dari Komisi Pemberantasan Korupsi, sesi dari Puslabfor POLRI, dan sesi dari akademisi ITB.

Kebutuhan Penyidik akan Forensik Suara Ucap

Forensik suara ucap → Proses untuk menentukan apakah contoh dari suara seseorang (known sample) merupakan sumber dari suara yang diselidiki (unknown sample). Jenis alat bukti forensik suara ucap adalah sbb:

• Rekaman suara 
• Laporan forensik suara 
• Pendapat ahli

Dari ketiga jenis alat bukti forensik suara diatas, jelas peran forensik sangat vital untuk bisa dijadikan referensi dalam menuntut tersangka/terdakwa dengan hukuman semaksimal mungkin. Didukung dengan barang bukti yang sah (real evidence), maka tugas KPK untuk menjerat koruptor akan semakin realistis. Dalam KUHAP Pasal 184 alat bukti yang sah adalah: keterangan saksi, keterangan ahli, surat, petunjuk dan keterangan terdakwa. Perluasan alat bukti (forms of evidence) ini mencakup alat bukti elektronik yang mencangkup rekaman suara maupun data pendukungnya. Trend penggunaan forensik suara dalam hukum semakin meningkat seiring berkembangnya teknologi. Dalam film-film spy Hollywood, penggunaan teknologi suara sudah di-imajinasikan sangat canggih sehingga kita bisa mengetahui posisi seseorang hanya dari suaranya saja, misal dalam film Bourne, Mission Impossible atau 007.

Welcome to Week #1!

Signal Enhancement By Single Channel Source Separation

Most gadgets and electronics devices are commonly equipped with single microphone only. This is difficult task in source separation world which traditionally required more sensors than sources to achieve better performance. In this paper we evaluated single channel source separation to enhance target signal from inteferred noise. The method we used is non-negative matrix factorization (NMF) that decompose signal into its components and find the matched signal to target speaker. As objective evaluation, coherence score is used to measure the perceptual similarity from enhanced to original one. It show the extracted has 0.5 of average coherence that shows medium correlation between both signals.

The following slides talk a bit about signal Signal enhancement by single channel source separation principle. You can grab the full paper here.

Extracting Sound From Multiple Sources in Anechoic Room

Ini adalah revisi dari poster saya sebelumnya. What I learned? Apa yang saya perbaiki?

• Judul harus menarik dan spesifik, tidak lebih dari 10 kata
• Tambahkan ABSTRAK
• Hindari Jargon (kata-kata spesifik) dan akronim
• Keep it Simple!
• Tunjukkan the BIG idea, the big deal!
• Tunjukkan apa yang menarik dan kenapa
• Next Step, apa?
• Font minimal 28, idealnya 36 
• Perhatikan flow (kiri ke kanan) dan balance layout 
• Keywords! orang menemukan poster/paper kita dengan bantuan keyword (termasuk anda ketika menemukan tulisan ini)

 Have a good poster and snapshot!

SPTK basic operation: .wav, .raw., .short

The Speech Signal Processing Toolkit (SPTK) is a suite of speech signal processing tools for UNIX environments, e.g., LPC analysis, PARCOR analysis, LSP analysis, PARCOR synthesis filter, LSP synthesis filter, vector quantization techniques, and other extended versions of them. It is developed by Prof. Imai and Prof. Kobayashi of Tokyo Inst. of Tech, and currently maintained by Nagoya Inst. of Tech. To install SPTK in Linux and Unix based-OS is easy, download the source file, extract and do the following,

./configure
make
sudo make install

That's all installation process, it will install SPTK in /usr/local/SPTK by default.

Add SPTK path to .bashrc

Now SPTK is installed in our machine, but by default it is (SPTK command) not searchable in our shell/terminal. To make it searchable (instead of use /usr/local/SPTK/bin/command everytime), we can add the path to .bashrc. Open .bashrc with editor (vim/gedit/other) and add the following,

export PATH=$PATH:/usr/local/SPTK/bin

And it's added to our command path, check it by using "impulse -h" in terminal. If done, it will show man pages of "impulse" command.

Remove Header : Convert Wav to Raw

SPTK provide utilities to remove header file in sound recording process by converting wav files to raw files (and we can convert it back to wav file). Here is how,

wav2raw +s data.wav

In current directory, there will be new file data.raw with header file removed. Argumen +s is for short data type, you can check full argument by wav2raw -h. To convert back in wav, use "raw2wav" command.

Convert to Short

To convert data from raw, wav or other format to .short (because mainly processing in SPTK is in .short format) we use command "x2x",

x2x +s data.raw > data.short 

It will covert raw to short, you can try to change other format.

Plot Waveform

Plot waveform in SPTK