This piece was created with AI assistance and may still contain mistakes or unclear bits. If you spot anything that should be corrected, please email ayushdeva97@gmail.com.

Audio Features · A Field Guide

Handcrafted audio features

Before models learned their own representations from raw waveform, we engineered features by hand. This is a tour of the classic ones — what each measures, what it is blind to, and when it is the right tool. The same descriptors still show up everywhere: as quick baselines, as CNN inputs, and as interpretable signals next to deep models.

The idea is simple: instead of feeding a raw waveform into a model, we extract meaningful acoustic descriptors first. Each feature throws away most of the signal and keeps one specific property — loudness, brightness, pitch class, spectral shape — and the art is knowing which property your task actually depends on.

The conceptual progression worth holding in your head: Waveform → STFT / Spectrogram → Mel Spectrogram → MFCC → Delta features. That arc mirrors how audio feature engineering actually evolved — from raw signal, toward perceptually meaningful representations.

Time domain

Read straight off the waveform

Zero-Crossing Rate, RMS Energy, and Amplitude Envelope. Cheap, no transform needed — they capture noisiness, loudness, and peaks.

Spectral shape

Summarise the spectrum

Centroid, Bandwidth, Rolloff. One number each describing where the energy sits and how it spreads.

Time–frequency

Full perceptual maps

Chroma and the Mel Spectrogram. 2D representations — pitch class over time, or perceptual energy over time.

Cepstral & dynamics

Envelope and its motion

MFCCs plus Delta and Delta-Delta. The spectral envelope (timbre) and how fast it changes.

The mental model

From waveform to features

Most of these features are different ways of compressing the same Short-Time Fourier Transform. Knowing where each one sits in the pipeline tells you what it can and cannot see.

Raw waveform ── time-domain ──► ZCR, RMS, Amplitude Envelope │ ▼ STFT (windowed FFT) Magnitude spectrum ── spectral shape ──► Centroid, Bandwidth, Rolloff │ ── pitch folding ──► Chroma (12 bins) ▼ Mel filterbank Mel spectrogram ── perceptual map ──► (CNN input) │ ▼ log + DCT MFCC ── envelope / timbre ──► + Δ (velocity) + ΔΔ (acceleration)

Rule of thumb: time-domain features are cheap and crude, spectral features summarise shape, Mel representations are what you hand a neural net, and MFCCs are the compact, decorrelated envelope that classical speech systems were built on.

Time-Domain Features

Zero-Crossing Rate (ZCR)

Captures: noisiness & rough frequency content

zcr = librosa.feature.zero_crossing_rate(y)[0]

Definition

How often the signal changes sign — i.e. crosses zero — per frame.

ZCR = (1 / (N−1)) · Σ 𝟙[ sign(xₙ) ≠ sign(xₙ₋₁) ] summed over n = 1 … N−1; 𝟙 is the indicator function

Intuition

Picture the waveform. Low-frequency sounds wander across zero slowly; high-frequency and noisy sounds cross constantly.

Low frequency High frequency / noise ~~~ ~~~ ~ ~ ~ ~ ~ ~ ~ ~~~~~~ (many crossings) (few crossings)

What it captures

Noisiness and rough frequency content
Voiced vs unvoiced speech — vowels (aaa, ooo) have low ZCR; fricatives (sss, fff) have high ZCR
White noise sits very high

When to use it

Quick voiced/unvoiced gating in speech
Separating percussion from harmonic instruments in music
As a cheap noisiness signal alongside richer features

Limitation: extremely crude. Two signals with identical ZCR can have completely different spectra — it sees crossing count, not shape.

RMS Energy

Captures: loudness / signal strength

rms = librosa.feature.rms(y=y)[0]

Definition

The root-mean-square amplitude of each frame — a robust measure of loudness.

RMS = √( (1 / N) · Σ xᵢ² )

Intuition

Big swings in the waveform mean high energy; a flat, small waveform means low energy. It measures strength, and nothing about frequency.

High RMS Low RMS ^^^^^^^^^^ -- -- --

When to use it

Voice-activity detection: speech vs silence
Beat tracking and dynamics analysis in music
Event onset detection in environmental audio

Limitation: energy only. A loud dog bark and a loud car horn can have nearly identical RMS — it can't tell them apart.

Amplitude Envelope

Captures: peak loudness outline / transients

import numpy as np def amplitude_envelope(y, frame_length=2048, hop_length=512): return np.array([ np.max(np.abs(y[i:i + frame_length])) for i in range(0, len(y), hop_length) ]) ae = amplitude_envelope(y) # librosa has no built-in for this

Definition

The maximum absolute amplitude within each frame — the outer "shape" traced over the waveform's peaks.

AEₜ = max |x(n)| maximum taken over all samples n in frame t

Intuition

Where RMS averages energy across a frame, the amplitude envelope keeps only the loudest sample. That makes it trace the silhouette of the signal and react sharply to sudden attacks — a drum hit, a consonant burst, a door slam.

waveform ╱╲ ╱╲╲ envelope ───► ╱ ╲___╱ ╲___ (follows the peaks)

Amplitude Envelope vs RMS

Amplitude Envelope → peak per frame (spiky, reacts to onsets) RMS Energy → energy per frame (smoother, more robust)

When to use it

Onset and transient detection — sharp attacks stand out
Segmenting music or audio into events / notes
A quick loudness outline when you care about peaks more than average level

Limitation: a single loud sample sets the whole frame, so it is sensitive to outliers and noise spikes. For steady loudness, RMS is the more robust choice.

Spectral Shape Features

Spectral Centroid

Captures: brightness

spec_cent = librosa.feature.spectral_centroid(y=y, sr=sr)[0]

Definition

The center of mass of the spectrum — the magnitude-weighted average frequency.

Centroid = ( Σₖ fₖ · Sₖ ) / ( Σₖ Sₖ ) fₖ = frequency of bin k, Sₖ = its magnitude

Intuition

Think of balancing weights on a ruler. If energy concentrates at high frequencies, the balance point — the centroid — moves upward. This is what we perceive as "brightness."

Bass guitar → most energy low → low centroid
Cymbal → most energy high → high centroid

When to use it

Brightness estimation in music information retrieval
Instrument classification (flute vs violin vs cymbal)
Emotion recognition — angry speech often has a higher centroid

Limitation: a single number; it ignores the detailed shape of the spectrum around that balance point.

Spectral Bandwidth

Captures: spread of frequencies

spec_bw = librosa.feature.spectral_bandwidth(y=y, sr=sr)[0]

Definition

How widely the spectrum spreads around the centroid — a weighted standard deviation of frequency.

Bandwidth = √( ( Σₖ Sₖ · (fₖ − Centroid)² ) / ( Σₖ Sₖ ) )

Intuition

Is the energy packed into a narrow band or smeared across the whole range?

Pure tone (narrow) Noise (wide) | |||||||||||||| | (low bandwidth) (high bandwidth)

When to use it

Voice quality analysis in speech
Telling tonal instruments from noisy ones
Noise detection in sound classification, usually paired with the centroid

Spectral Rolloff

Captures: the high-frequency edge

rolloff = librosa.feature.spectral_rolloff(y=y, sr=sr)[0]

Definition

The frequency below which a fixed percentage — typically 85% (sometimes 95%) — of the total spectral energy lies.

smallest f_r such that Σ (Sₖ for fₖ ≤ f_r) ≥ 0.85 · Σₖ Sₖ

Intuition

"How far up do the frequencies actually extend?" If 85% of the energy lives below 3000 Hz, then rolloff = 3000 Hz. It marks the edge of the spectrum rather than its center.

Centroid vs Rolloff

Centroid → average location of energy Rolloff → edge location of energy

When to use it

Voiced/unvoiced separation in speech
Brightness estimation in music
General audio classification — strong when combined with the centroid

Time–Frequency Representations

Chroma Features

Captures: musical pitch class

chroma = librosa.feature.chroma_stft(y=y, sr=sr) # shape: 12 × T

Definition

Energy folded into the 12 musical pitch classes, regardless of octave.

C C# D D# E F F# G G# A A# B

Intuition

Humans hear notes an octave apart as "the same" note. Chroma builds that in — every C collapses to one bin.

C3 = 130 Hz ┐ C4 = 261 Hz ├──► C C5 = 523 Hz ┘

A song in C major piles most of its energy into C, E, and G.

When to use it

Chord recognition and key detection
Cover-song retrieval and music recommendation
Any harmony-aware music task

Rarely useful for speech or environmental audio — pitch class isn't a meaningful concept there.

Mel Spectrogram

Captures: perceptual time–frequency energy

mel = librosa.feature.melspectrogram(y=y, sr=sr, n_mels=40) # shape: 40 × T

Definition

A spectrogram projected onto Mel-spaced frequency bins.

Waveform → STFT → Power spectrum → Mel filterbank → Mel spectrogram

Why Mel?

Human hearing is nonlinear. We easily distinguish 100 Hz from 200 Hz, but barely hear the difference between 7100 Hz and 7200 Hz. The Mel scale compresses frequency to mirror that — fine resolution low down, coarse resolution up high.

Output

A 40 × T matrix: each row is a Mel band, each column a time frame. In practice you usually take the log of it (log-Mel) before feeding a model.

When to use it

Almost everything in modern audio: ASR, speaker recognition, keyword spotting
Audio classification benchmarks (ESC-50, UrbanSound8K)
Deep learning — CNNs treat the log-Mel spectrogram as an image. This is the default input for most learned models, including Whisper.

Cepstral & Dynamic Features

MFCCs

Captures: spectral envelope / timbre

mfcc = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=20)

The classic workhorse

Mel-Frequency Cepstral Coefficients were the dominant speech feature for decades. They describe the shape of the spectrum compactly.

Waveform → STFT → Mel filterbank → Log → DCT → MFCC

Why the DCT?

Adjacent Mel bands are highly correlated. The Discrete Cosine Transform decorrelates them and packs the information into a handful of coefficients, so you keep ~13–20 numbers instead of 40 bands.

Intuition

MFCCs describe the overall spectral envelope, not exact frequencies. For speech, that envelope is essentially the shape of the vocal tract:

speech = voice source + vocal-tract filter MFCCs capture the filter (the "shape of the tube")

Reading the coefficients

Coefficient 0 — overall energy (often dropped, since RMS covers it)
Lower coefficients — broad spectral slope / tilt
Higher coefficients — progressively finer spectral detail

When to use it

Classical speech recognition and speaker identification
Emotion recognition and general audio classification
Any time you want a compact, model-friendly summary of timbre — especially with non-deep classifiers (SVMs, gradient boosting)

Delta MFCC

Captures: rate of spectral change (velocity)

mfcc_delta = librosa.feature.delta(mfcc)

Definition

The first derivative of the MFCCs over time — how fast the spectral envelope is changing.

Δ MFCC = d(MFCC) / dt

Intuition

A steady vowel barely moves; a transition between sounds moves a lot.

aaaaaaa → small delta aaaEEEooo → large delta

When to use it

Capturing speech dynamics and phoneme transitions
Emotion recognition, where movement carries affect
Alongside static MFCCs whenever timing matters

Delta-Delta MFCC

Captures: acceleration of spectral change

mfcc_delta2 = librosa.feature.delta(mfcc, order=2)

Definition

The second derivative — the acceleration of the spectral envelope.

Δ² MFCC = d²(MFCC) / dt²

Intuition

The classic motion analogy makes it click:

MFCC → position Δ MFCC → velocity Δ² MFCC → acceleration

Why bother?

Speech is never static, and stacking statics with their dynamics usually beats statics alone. The canonical classical feature vector was exactly this:

13 MFCC + 13 Δ + 13 ΔΔ = 39-dimensional feature

That 39-dim vector powered a generation of HMM-based speech recognizers.

The Big Picture

Each feature keeps one property and discards the rest. Picking features is really picking which property your task depends on.

Feature	Captures	Family
Zero-Crossing Rate	Noisiness, rough frequency content	Time domain
RMS Energy	Loudness / signal strength	Time domain
Amplitude Envelope	Peak loudness outline, transients	Time domain
Spectral Centroid	Brightness (center of energy)	Spectral shape
Spectral Bandwidth	Spread of frequencies	Spectral shape
Spectral Rolloff	High-frequency edge	Spectral shape
Chroma	Musical pitch classes	Time–frequency
Mel Spectrogram	Perceptual time–frequency map	Time–frequency
MFCC	Spectral envelope / timbre	Cepstral
Delta MFCC	Temporal change (velocity)	Dynamics
Delta-Delta MFCC	Temporal acceleration	Dynamics

When to Use What

A task-first cheat sheet. Start from what you're trying to predict, then reach for the matching features.

If you're doing…	Reach for	Why
Speech vs silence (VAD)	RMS, ZCR	Loudness plus a noisiness gate, both nearly free to compute
Onset / transient detection	Amplitude Envelope, RMS	The peak outline reacts sharply to sudden attacks
Voiced vs unvoiced	ZCR, Spectral Rolloff	Fricatives push crossing rate and the spectral edge up
Classical ASR / speaker ID	MFCC + Δ + ΔΔ	Compact, decorrelated timbre plus its motion — the proven 39-dim vector
Emotion / affect	MFCC, Centroid, Δ features	Affect lives in timbre, brightness, and how they move over time
Chords / key / cover songs	Chroma	Octave-invariant pitch class is exactly the harmonic signal
Instrument / timbre ID	Centroid, Bandwidth, MFCC	Brightness and spectral shape separate instrument families
Percussion vs harmonic	ZCR, Centroid	Percussive hits are noisier and brighter
Environmental sound (CNN)	Log-Mel Spectrogram	A perceptual 2D map that convnets read as an image
Any deep-learning model	Log-Mel Spectrogram	The default learned-model input; everything else is mostly classical baselines

From Handcrafted to Learned

These features are the "before" picture. Each one bakes a human assumption about sound into the pipeline — and that was also their ceiling. The whole arc of modern audio modeling is about letting the model discover its own representation instead.

A practical split still holds: time-domain features are cheap gates, spectral features summarise shape, the log-Mel spectrogram is what you hand a neural net, and MFCCs + deltas are the compact classical envelope. Even today, these make excellent fast baselines and interpretable signals to sit beside a deep model.

For where the story goes next — from these engineered features to self-supervised and multimodal systems — see The Story of Modern Audio Models and the Audio Model Explorer.