Feature extractors

(...under construction...)

1. Feature extractors
2. Post-processing
3. Serialization
4. Time-domain feature extractor
5. Spectral feature extractor
6. Pitch
7. MPEG-7 feature extractor
8. MFCC/PNCC
9. LPC/LPCC
10. AMS
11. Creating your own feature extractor

Feature extractors

All feature extractors are inherited from abstract class FeatureExtractor. It's relatively complex, so first we'll consider only the most important part. The rest will be discussed in section Creating your own feature extractor.

Each feature extractor has method ComputeFrom() that takes float[] array or DiscreteSignal as input (and optionally starting and ending positions for analysis). Extractors can and must be reused: create extractor object once and call ComputeFrom() method every time there's a new portion of data and new features should be calculated.

All feature extractors decompose signal or array of samples into sequence of frames with certain hop. So in constructor of any feature extractor at least (1) sampling rate, (2) number of features, (3) frame duration and (4) hop duration (in seconds) must be specified.

Properties:

• abstract int FeatureCount
• abstract List<string> FeatureDescriptions
• virtual List<string> DeltaFeatureDescriptions
• virtual List<string> DeltaDeltaFeatureDescriptions
• double FrameDuration (in seconds)
• double HopDuration (in seconds)
• int FrameSize (in samples)
• int HopSize (in samples)
• int SamplingRate

Methods:

• abstract List<FeatureVector> ComputeFrom()
• virtual List<FeatureVector> ParallelComputeFrom()
• virtual bool IsParallelizable()

Example:

var mfccExtractor = new MfccExtractor(signal.SamplingRate, 13, 0.025, 0.01);
var mfccVectors = mfccExtractor.ComputeFrom(signal);

// let's say we have some samples from external source
// process them with same extractor
// (for demo, let's also ignore first 100 samples):

var newVectors = mfccExtractor.ComputeFrom(samples, 100, samples.Length);

// (supposing that signal sampling rate is 16000 Hz)
// properties:

var count = mfccExtractor.FeatureCount;          // 13
var names = mfccExtractor.FeatureDescriptions;   // { "mfcc0", "mfcc1", ..., "mfcc12" }
var frameDuration = mfccExtractor.FrameDuration; // 0.025
var hopDuration = mfccExtractor.HopDuration;     // 0.010
var frameSize = mfccExtractor.FrameSize;         // 400 = 0.025 * 16000
var hopSize = mfccExtractor.HopSize;             // 160 = 0.010 * 16000

Feature extractors return collection of feature vectors (List<FeatureVector>). Each feature vector contains time position (in seconds) and array of features:

• double TimePosition
• float[] Features

Also, there is method featureVector.Statistics() returning Dictionary with keys:

var stats = featureVector.Statistics();

var min = stats["min"];
var max = stats["max"];
var avg = stats["mean"];

Feature extractors can be parallelizable, i.e. they can create their internal clones which will simultaneously process different parts of signal. In this case property IsParallelizable will return true. All available extractors in NWaves are parallelizable, except PnccExtractor. For example, LPCExtractor:

var lpcExtractor = new LpcExtractor(signal.SamplingRate, 16, 0.032, 0.015);
var lpcVectors = lpcExtractor.ParallelComputeFrom(signal);

Note. ParallelComputeFrom() method involves all available CPU cores and this behavior can't be changed. That's just because in version 0.9.0 I simply forgot to add threadCount parameter to the method ((. In later versions this will be fixed, and there will be possibility to tweak the number of cores for computations. So far there's still a workaround: using public helper-method ParallelChunksComputeFrom(signal, threadCount) that actually allows changing the number of cores. However, it's slightly inconvenient, since it returns not the list of feature vectors, but the array of List<FeatureVector> calculated in each thread.

Post-processing

• Mean subtraction
• Variance normalization
• Adding deltas and delta-deltas to existing feature vector
• Joining (merging feature vectors into one longer vector)
• Extracting the featuregram

FeaturePostProcessing.NormalizeMean(mfccVectors);
FeaturePostProcessing.NormalizeVariance(mfccVectors, bias: 0);

FeaturePostProcessing.AddDeltas(mfccs, includeDeltaDelta: false);

var totalVectors = FeaturePostProcessing.Join(mfccs, lpcs, lpccs);

IEnumerable<float[]> featureGram = mfccs.Featuregram();

The bias parameter in NormalizeVariance() method is by default equal to 1 (so estimate of variance is unbiased). This parameter is present in formula:

Method AddDeltas() extends each feature vector in the list. Deltas are computed according to formula:

Since in NWaves N is fixed and N=2, the formula reduces to:

As can be seen, we need to account for marginal cases (at the beginning and at the end of the list). Method AddDeltas() by default adds two zero vectors at the beginning and two zero vectors at the end of the list (and that's perfectly fine in most cases). You can also prepend and append your own collections of feature vectors to specify marginal 'previous' and 'next' sets (there must be at least two vectors in each set; method will unite all lists and calculate deltas in united list starting from the third vector and ending with the third from the end):

FeatureVector[] previous = new FeatureVector[2]; // for two previous vectors
FeatureVector[] next = new FeatureVector[2];     // for two vectors after the last one

// fill 'previous' and 'next' with values

FeaturePostProcessing.AddDeltas(mfccs, previous, next);

Featuregram is an analog of stft.Spectrogram. It simply skips the time positions and yields 2D representation of feature values (float[]) from List<FeatureVector>. Note, it returns IEnumerable.

Serialization

var mfccs = new MfccExtractor(signal.SamplingRate, 13).ComputeFrom(signal);

using (var csvFile = new FileStream("mfccs.csv", FileMode.Create))
{
    var serializer = new CsvFeatureSerializer(mfccs);
    await serializer.SerializeAsync(csvFile);
}

Time-domain feature extractor

TimeDomainFeaturesExtractor class is the first representative of the multi-extractors family:

• TimeDomainFeaturesExtractor
• SpectralFeaturesExtractor
• Mpeg7SpectralFeaturesExtractor

These extractors compute several features at once using different formulae/routines. They accept the string containing the list of feature names enumerated with any separator (',', '+', '-', ';', ':'). If the string "all" or "full" is specified then multi-extractor will compute all pre-defined features (returned by FeatureSet public property).

TimeDomainFeaturesExtractor, as the name suggests, computes time-domain features, such as: energy, RMS, ZCR and entropy (all these methods are contained in DiscreteSignal class):

var tdExtractor = new TimeDomainFeaturesExtractor(signal.SamplingRate, "all", 0.032, 0.02);
var tdVectors = tdExtractor.ComputeFrom(signal);

// compute only RMS and ZCR at each step:

tdExtractor = new TimeDomainFeaturesExtractor(signal.SamplingRate, "rms, zcr", 0.032, 0.02);
tdVectors = tdExtractor.ComputeFrom(signal);

// let's examine what is available:

var names = tdExtractor.FeatureSet;  // { "energy, rms, zcr, entropy" }

Recognized keywords are:

• Energy: "e", "en", "energy"
• RMS: "rms"
• ZCR: "zcr", "zero-crossing-rate"
• Entropy: "entropy"

Keywords are case-insensitive.

You can also add your own feature with corresponding function for its calculation. This function must accept three parameters: 1) signal, 2) start position, 3) end position (exclusive). Code example:

var tdExtractor = new TimeDomainFeaturesExtractor(sr, "RMS", frameDuration, hopDuration);

// let's add two features:
// 1) "positives": percentage of samples with positive values
// 2) "avgStartEnd": just the average of start and end sample

tdExtractor.AddFeature("positives", CountPositives);
tdExtractor.AddFeature("avgStartEnd", (s, start, end) => { return (s[start] + s[end - 1]) / 2; } );

var count = tdExtractor.FeatureCount;        // 3
var names = tdExtractor.FeatureDescriptions; // { "rms", "positives", "avgStartEnd" }

// ...
float CountPositives(DiscreteSignal signal, int start, int end)
{
    var count = 0;
    for (var i = start; i < end; i++)
    {
        if (signal[i] >= 0) count++;
    }
    return (float)count / (end - start);
}

Spectral feature extractor

SpectralFeaturesExtractor computes spectral features, such as: centroid, spread, flatness, etc. All these methods are taken from static class Spectral and can be calculated separately for one particular spectrum, without creating any extractor:

using NWaves.Features;

// prepare array of frequencies
// (simply spectral frequencies: 0, resolution, 2*resolution, ...)

var resolution = (float)samplingRate / fftSize;

var frequencies = Enumerable.Range(0, fftSize / 2 + 1)
                            .Select(f => f * resolution)
                            .ToArray();

var spectrum = new Fft(fftSize).MagnitudeSpectrum(signal);

// compute various spectral features
// (spectrum has length fftSize/2+1)

var centroid = Spectral.Centroid(spectrum, frequencies);
var spread = Spectral.Spread(spectrum, frequencies);
var flatness = Spectral.Flatness(spectrum, minLevel);
var noiseness = Spectral.Noiseness(spectrum, frequencies, noiseFreq);
var rolloff = Spectral.Rolloff(spectrum, frequencies, rolloffPercent);
var crest = Spectral.Crest(spectrum);
var decrease = Spectral.Decrease(spectrum);
var entropy = Spectral.Entropy(spectrum);
var contrast1 = Spectral.Contrast(spectrum, frequencies, 1);
//...
var contrast6 = Spectral.Contrast(spectrum, frequencies, 6);

Usually, the spectral frequencies are involved in calculations, but you can specify any frequency array you want:

var freqs = new float[] { 200, 300, 500, 800, 1200, 1600, 2500, 5000/*Hz*/ };

var centroid = Spectral.Centroid(spectrum, freqs);
var spread = Spectral.Spread(spectrum, freqs);

SpectralFeaturesExtractor usage example:

var extractor = new SpectralFeaturesExtractor(signal.SamplingRate, "all", 0.032, 0.02);
var vectors = extractor.ComputeFrom(signal);

// let's examine what is available:

var names = extractor.FeatureSet;
// { "centroid, spread, flatness, noiseness, rolloff, crest, entropy, decrease, c1+c2+c3+c4+c5+c6" }

Recognized keywords are:

• Spectral Centroid: "sc", "centroid"
• Spectral Spread: "ss", "spread"
• Spectral Flatness: "sfm", "flatness"
• Spectral Noiseness: "sn", "noiseness"
• Spectral Rolloff: "rolloff"
• Spectral Entropy: "ent", "entropy"
• Spectral Decrease: "sd", "decrease"
• Spectral Contrast in band 1,2,...: "c1", "c2", ...

Keywords are case-insensitive.

You can also add your own spectral feature with corresponding function for its calculation. This function must accept two parameters: 1) array of samples, 2) array of frequencies. Code example:

var extractor = new SpectralFeaturesExtractor(sr, "sc", frameDuration, hopDuration);

// let's add new feature: relative position of the first peak
extractor.AddFeature("peakPos", FirstPeakPosition);

// ...
// in our case 'frequencies' array will be ignored

float FirstPeakPosition(float[] spectrum, float[] frequencies)
{
    for (var i = 2; i < spectrum.Length - 2; i++)
    {
        if (spectrum[i] > spectrum[i - 2] && spectrum[i] > spectrum[i - 1] && 
            spectrum[i] > spectrum[i + 2] && spectrum[i] > spectrum[i + 1]) 
        {
            return (float) i / spectrum.Length;
        }
    }
    return 0;
}

Full list of constructor parameters:

• int samplingRate
• string featureList
• double frameDuration (0.0256 seconds by default)
• double hopDuration (0.010 seconds by default)
• int fftSize (by default 0, i.e. it will be calculated automatically as the closest power of 2 to FrameSize)
• float[] frequencies (by default null, i.e. spectral frequencies will be derived automatically)
• WindowTypes window (by default, WindowTypes.Hamming)
• IReadOnlyDictionary<string, object> parameters (by default, null)

Dictionary of parameters may contain following keys:

• "noiseFrequency" (used for computing Spectral.Noiseness; by default 3000)
• "rolloffPercent" (used for computing Spectral.Rolloff; by default 0.85f)

Note. Spectral noiseness is unconventional parameter and is calculated as a ratio of spectral energy in high-frequency region and total spectral energy, by simple formula:

Pitch

MPEG-7 feature extractor

MFCC/PNCC

MfccExtractor constructor has a lot of parameters, and there are broad possibilities for customization: you can pass your own filter bank, for instance bark bank, then the algorithm will become BFCC, or the gammatone bank, then it'll become GFCC, etc.

Full list of constructor parameters:

• int samplingRate
• int featureCount (number of MFCC coefficients)
• double frameDuration (0.0256 seconds by default)
• double hopDuration (0.010 seconds by default)
• int filterbankSize (20 by default)
• double lowFreq (0 by default, filter bank lower frequency)
• double highFreq (samplingRate / 2 by default, filter bank upper frequency)
• int fftSize (by default 0, i.e. it will be calculated automatically as the closest power of 2 to FrameSize)
• float[] filterbank (by default null, i.e. filterbank will be generated from parameters above)
• int lifterSize (22 by default)
• double preEmphasis (positive pre-emphasis coefficient, by default 0 - no pre-emphasis)
• WindowTypes window (by default, WindowTypes.Hamming)

Constructor of PNCC and SPNCC extractors has one additional parameter:

• int power (by default 15)

If power is set to 0 then the Log10(x) operation will be applied to spectrum before doing DCT-II. Otherise the operation Pow(x, 1/power) will be applied.

var sr = signal.SamplingRate;

var mfccExtractor = new MfccExtractor(sr, 13, filterbankSize: 24, preEmphasis: 0.95);
var mfccVectors = mfccExtractor.ParallelComputeFrom(signal);

/* the following lines of code are equivalent to previous lines,
   except that in this case intermediate signal will be created after
   processing original signal with pre-emphasis filter:

var mfccExtractor = new MfccExtractor(sr, 13, filterbankSize: 24);
var preEmphasis = new PreEmphasisFilter(0.95);
var mfccVectors = mfccExtractor.ParallelComputeFrom(preEmphasis.ApplyTo(signal));
*/

var pnccExtractor = new PnccExtractor(sr, 13);
var pnccVectors = pnccExtractor.ComputeFrom(signal, /*from*/1000, /*to*/10000 /*sample*/);
FeaturePostProcessing.NormalizeMean(pnccVectors);

Default filter bank is triangular overlapping mel-bands. If you specify your own filter bank, then parameters lowFreq, highFreq and filterbankSize will be ignored. For example, let's change mel-bands to bark-bands and obtain actually a BFCC extractor:

var barkbands = FilterBanks.BarkBands(16, 512, sr, 100/*Hz*/, 6500/*Hz*/, overlap: true);
var barkbank = FilterBanks.Triangular(512, sr, barkbands);

var bfccExtractor = new MfccExtractor(sr, 13, filterbank: barkbank, preEmphasis: 0.95);
var bfccVectors = bfccExtractor.ParallelComputeFrom(signal);

LPC/LPCC

Full list of constructor parameters:

• int samplingRate
• int order (LPC order, i.e. number of LPC coefficients)
• double frameDuration (0.0256 seconds by default)
• double hopDuration (0.010 seconds by default)
• double preEmphasis (positive pre-emphasis coefficient, by default 0 - no pre-emphasis)
• WindowTypes window (by default, WindowTypes.Rectangular)

Constructor of LPCC extractor has one additional parameter:

• int lifterSize (by default 22)

var lpcExtractor = new LpcExtractor(sr, 16, 0.050, 0.020, 0.95);
var lpccExtractor = new LpccExtractor(sr, 16, 0.050, 0.020, 26, 0.95);

AMS

Amplutide Modulation Spectra extractor.

Full list of constructor parameters:

• int samplingRate
• double frameDuration (0.0256 seconds by default)
• double hopDuration (0.010 seconds by default)
• int modulationFftSize (64 by default)
• int modulationHopSize (4 by default)
• int fftSize (by default 0, i.e. it will be calculated automatically as the closest power of 2 to FrameSize)
• IEnumerable<float[]> featuregram (null by default)
• float[][] filterbank (null by default)
• double preEmphasis (positive pre-emphasis coefficient, by default 0 - no pre-emphasis)
• WindowTypes window (WindowTypes.Rectangular by default)

If the filterbank is specified, than it will be used for calculations:

var extractor = new AmsExtractor(signal.SamplingRate, 0.0625, 0.02, 64, 16, filterbank);
var features = extractor.ComputeFrom(signal);

You can also specify featuregram (i.e. compute AMS for various featuregrams, not only spectrograms):

var frameSize = 0.032;
var hopSize = 0.02;

var mfccExtractor = new MfccExtractor(signal.SamplingRate, 13, frameSize, hopSize);
var vectors = mfccExtractor.ComputeFrom(signal);
FeaturePostProcessing.NormalizeMean(vectors);

var featuregram = vectors.Featuregram();

var extractor = new AmsExtractor(signal.SamplingRate, frameSize, hopSize, 64, 16, featuregram);

If neither filterbank, nor featuregram are specified then the filterbank is generated automatically in AmsExtractor as 12 overlapping mel-bands covering frequency range from 100 to 3200 Hz.

Creating your own feature extractor