The story of modern audio models

Earlier speech systems usually did not learn directly from raw audio. We first converted the waveform into more digestible representations such as spectrograms, log-Mel features, MFCCs, pitch tracks, or other engineered descriptors. Then we trained CNNs, RNNs, or hybrid systems on top.

This worked, and it worked well enough to define the field for years. But there was a hidden limitation: a lot of domain knowledge had to be manually baked into the representation. The model was learning on processed views of sound rather than discovering its own representation from raw audio.

Then BERT happened. In language, a model could hide tokens, use context on both sides, and learn rich representations without explicit labels. That result was so powerful that naturally people asked: can we do the same thing for audio?

The obstacle was immediate. Text has discrete tokens. Audio is continuous. If you hide part of a waveform, what exactly should the model predict? The raw sample values? A spectrogram patch? Some latent symbol we do not yet know how to define?

Wav2Vec 2.0 is the first moment where the answer really clicks. Instead of forcing audio into fixed human-designed units up front, it learns continuous representations from raw waveform and then creates discrete targets through quantization. That gives the model something token-like to predict.

In the standard speech setup, that raw waveform is typically normalized to 16 kHz mono first. So when we talk about canonical Wav2Vec 2.0 or HuBERT checkpoints, we are usually talking about models trained around a 16 kHz speech pipeline rather than arbitrary sample rates.

Conceptually, the model says: first learn a dense, local encoding of sound; then convert those encodings into discrete codebook choices; then mask some positions and make a Transformer guess which discrete target belongs there from context.

Issues with Wav2Vec 2.0

• Training instability: the model is learning the continuous representation space and the discrete target system at the same time, which makes optimization more brittle than HuBERT's offline-target setup.
• Codebook collapse risk: without explicit pressure to use the codebooks well, the quantizer can overuse a small subset of entries instead of spreading assignments across the vocabulary. This is why the method includes a diversity term.
• Target quality shifts during training: because the quantized targets are learned online, the "thing to predict" is itself moving while the model trains.
• False negatives in contrastive learning: some distractor Q vectors sampled from other time steps may actually be acoustically or linguistically similar to the correct target, which makes the objective noisier.

HuBERT starts from a practical observation: in Wav2Vec 2.0, two hard things are happening at once. The model is trying to learn good speech representations while also learning the discrete target system that those representations should match.

HuBERT separates these concerns. Instead of learning the targets online through a jointly trained quantizer, it generates pseudo-labels offline using clustering. The model then predicts those cluster assignments for masked regions. In each round, k-means chooses k clusters, which means HuBERT is learning to classify each masked frame into one of k hidden-unit labels. Later rounds can recluster using better learned features, making the targets progressively stronger.

In the original recipe, the first round uses MFCC features with k = 100, giving a coarse 100-way pseudo-label vocabulary. A later round reclusters HuBERT's own learned representations with k = 500, creating a finer 500-way target space. These are frame-level acoustic units rather than text tokens, and that progressive refinement is what makes HuBERT's training unusually stable.

If Wav2Vec 2.0 and HuBERT are about unsupervised or self-supervised representation learning, Whisper is a different kind of breakthrough. It shows what happens when you train a large encoder-decoder Transformer on 680,000 hours of noisy, multilingual paired audio-text data — hundreds of times larger than the 960-hour LibriSpeech scale that defined many earlier speech benchmarks.

Whisper matters because it changes the field’s confidence in a practical way. It is not just a clever pretraining objective. It is a general speech model that works surprisingly well in the wild: accents, background noise, code-switching, translation, and messy internet audio.

The problem Whisper really tries to solve is not "how do we invent better speech units?" but "how do we build one speech system that is robust across the internet's actual mess?" Earlier systems were often strong only when the audio and labels looked like the benchmark. Whisper bets that if you train directly on enough real paired audio and text, robustness can emerge from scale and diversity rather than from a delicate self-supervised objective.

The training corpus is the other big story. Whisper was trained on about 680,000 hours of web-scale paired audio and text: English ASR pairs, non-English ASR pairs across 96 other languages, and X→English translation pairs. These labels were available, but they were noisy — subtitles can drift, paraphrase, or summarize — so the paper frames the setup as weakly supervised, not carefully hand-aligned supervised speech data.

That detail matters because it explains both Whisper's strength and its failure mode. The model becomes unusually robust to accents, microphones, and domain shift because it has seen a huge variety of imperfect real-world data. But the same noisy supervision also helps explain why Whisper can hallucinate on silence, music, or badly segmented input: it learned from a world where the text and the audio were often only approximately aligned.

How Whisper encodes time

One elegant consequence of the unified token stream is that Whisper can produce timestamps without any separate alignment module or timing head. Time is simply more tokens in the same vocabulary. The tokenizer reserves a contiguous block of special timestamp tokens, and the decoder emits them inline with the text exactly like any other token — so timing is learned as part of next-token prediction rather than bolted on afterwards.

Each 30-second window is divided into 0.02s (20 ms) bins, which gives 1,501 timestamp tokens running from <|0.00|> through <|30.00|>. A spoken segment is then bracketed by a start and an end timestamp token, with the text tokens sitting in between:

Because those timestamps are measured from the start of the current window, long-form audio has to add each window's offset back to recover absolute time — the same chunk-and-stitch logic that already drives Whisper's 30-second processing. This built-in timing is segment-level: it brackets phrases rather than individual words, and it can drift on long stitched recordings.

For word-level timestamps the inline tokens are too coarse, so the standard trick is cross-attention alignment — running dynamic time warping over the decoder-to-encoder attention weights of specific heads to line each word up with an audio frame. That is what tools like WhisperX, faster-whisper, and whisper.cpp do under the hood. In the Hugging Face transformers pipeline the whole choice collapses to one flag: return_timestamps=True for segment-level timing and return_timestamps="word" for per-word timing.

• Wav2Vec 2.0 asks: how can audio learn internal units without labels?
• HuBERT asks: how can we make that process more stable?
• Whisper asks: what if we simply scale paired audio-text learning until it becomes broadly useful?

The newest part of the story is that audio is no longer treated only as a signal to transcribe. In multimodal systems like Qwen Audio, audio becomes another modality that can be fed into a general instruction-following model.

That means the goal is no longer just “what was said?” but also “what is happening?”, “what should I infer from this sound?”, or “how does this audio relate to the rest of the context?” Audio moves from a narrow speech pipeline into a broader reasoning stack.

This section is really about a change in problem definition. Whisper is still fundamentally a speech-to-text system, even if it is multilingual and robust. Qwen Audio asks for something broader: can one model answer questions about speech, music, environmental sound, and mixed acoustic scenes in natural language, while also behaving like an instruction-following assistant?

There is also an architectural evolution inside the family. The earlier Qwen-Audio recipe uses a Whisper-large-v2 encoder plus a projection layer into Qwen-7B. Qwen2-Audio still initializes its audio encoder from Whisper (the newer large-v3), but trains it more broadly for general audio understanding rather than leaving a speech-first encoder untouched. That shift is important because speech encoders are excellent at transcription, but they are not automatically ideal for music, sound events, or open-ended audio reasoning.

The paper-level contribution here is not one isolated trick. It is the combination: a multimodal architecture, a broad multi-dataset training mixture, and instruction tuning that teaches the model to respond in natural language. The result is that audio stops being a narrow pipeline and becomes another input channel for a general-purpose assistant.