G: Alright, let’s do this in two parts.

Part 1: How MP3 (or similar) compression works at a basic technical level

Think of audio compression like this:

Raw audio is just a stream of amplitudes (like how far a speaker cone moves in or out) sampled many thousands of times per second. That’s a ton of data.
Compression takes advantage of the fact that human ears don’t notice everything.
- The ear is less sensitive to certain frequencies (very low or very high).
- If a loud sound is present, quieter sounds nearby in frequency or time can be masked (psychoacoustic masking).
The encoder converts the waveform into frequency components (using transforms like the MDCT). Instead of storing every sample directly, it stores how strong each frequency is over short windows of time.
Quantization and bit allocation: frequencies judged as less important to human perception are stored with fewer bits (rougher detail) or sometimes dropped entirely. Important frequencies get more detail.
Entropy coding then packs those numbers efficiently, kind of like zip compression.

The result: instead of thousands of numbers per second per channel, you get a compact description of “what the ear would hear,” not the literal waveform.

Part 2: Relation to video pixel compression

Yes—there’s a conceptual relationship, though the mechanics differ.

In video compression (e.g., H.264), a pixel’s exact color value isn’t usually stored frame by frame. Instead:
- Frames are broken into blocks.
- Those blocks are transformed into frequency components (like with audio—DCT is common).
- High-frequency details (tiny changes in color or brightness the eye barely notices) are stored with fewer bits or discarded.
- Many blocks are predicted from previous frames (temporal compression) so not every pixel is stored at all.
If you imagine a pixel’s possible values as a wave, compression is less about reshaping that single wave and more about saying:
- “This pixel looks mostly like its neighbors or like the same pixel in the last frame, plus a small correction.”
- The correction is stored in frequency space, just like audio frequencies.
- The same human perception principle applies: the eye is less sensitive to fine detail in certain situations (e.g., tiny color shifts in high-motion areas), so compression drops them.

TLDR

Audio (MP3) compression: turn waveform → frequencies → drop/muffle inaudible stuff → pack efficiently.
Video compression: turn pixel blocks → frequencies → drop/blur imperceptible details → predict across frames.
Both rely on transform coding + perceptual limits.
Your analogy of “a pixel’s settings as a waveform” isn’t exactly literal, but conceptually sound: both systems describe information in terms of frequencies and perceptual relevance rather than storing every raw value.

Do you want me to give you a side-by-side analogy chart (audio vs video) so you can use it as a mental shortcut when explaining to other tech folks?

G: Got it — thanks for clarifying. Let me break this into two clean pieces:

1. Why MP3 was a breakthrough

MP3 wasn’t just “we compressed audio” — that had been done before with ADPCM and such. The leap was:

Psychoacoustics baked into the algorithm: Instead of only finding statistical redundancy, MP3 uses a perceptual model of human hearing to decide which errors don’t matter.
→ This let it throw away huge amounts of data while keeping the illusion of fidelity.
Transform coding (MDCT): Audio is converted into overlapping frequency-domain chunks, which are more compact and easier to quantize in a perceptually meaningful way.
Bit reservoir trick: Instead of forcing every short window to use the same number of bits, MP3 lets frames borrow space from neighbors — smoothing out complex vs. simple sections without audible glitches.

The net effect: order-of-magnitude compression without unbearable quality loss, and feasible on 1990s CPUs. That’s why it exploded.

2. Has anyone tried “MP3-style per-pixel waveform compression” for video?

Short answer: Yes, the idea has been floated, but it’s not used in practice. Here’s why:

Concept: If you treat each pixel’s time evolution as a waveform, you could Fourier/MDCT transform it and apply psycho-visual masking the way MP3 does psychoacoustics. In theory, each pixel is like its own audio channel.
Problem 1 – Noisy signals: Individual pixel timelines are very noisy. Video compression works because pixels are correlated with neighbors (spatial redundancy) and past/future frames (temporal redundancy). A pixel-alone transform doesn’t capture that correlation, so you lose the biggest compression lever.
Problem 2 – Human vision masking is spatial-temporal, not just temporal: MP3 can drop a faint sound masked by a loud one nearby in frequency/time. For vision, what gets masked depends on patterns across areas of the image, not a single pixel’s history. Looking only at pixel waveforms misses that.
Problem 3 – Efficiency: You’d still need to store millions of tiny “pixel MP3s.” The bit overhead per pixel would kill you compared to block-based transforms that already compress shared patterns.

Closest real-world analogues

3D Wavelet/Transform codecs: Some experimental codecs (e.g., early 2000s wavelet-based video) treat video as a 3D signal (2D space + time) and transform it that way. That’s the mathematical cousin of your “pixel-as-waveform” idea, but at block or volume level instead of per-pixel.
MPEG perceptual video coding: Modern codecs already borrow psycho-visual models (e.g., H.265 has adaptive quantization that reduces detail in areas the eye won’t notice, similar to MP3 dropping masked frequencies).
Research ideas: People have toyed with “motion-compensated temporal filtering” and pixel-frequency-domain experiments, but mainstream codecs settled on block transforms + motion prediction because they dominate in compression efficiency.

TLDR

MP3’s breakthrough: it married transform coding with psychoacoustics and clever bit-allocation tricks.
Your idea (pixel-as-waveform MP3 compression) has been considered, but vision masking is too spatial and the redundancy lies in blocks/motion, not in pixel histories.
Closest real-world cousins: 3D wavelet codecs and perceptual quantization in H.265/AV1.

Would you like me to sketch what a hybrid codec based on your idea might look like (e.g., use normal block/motion compression but swap in MP3-like temporal coding for residuals) so you can see how it could be tested in principle?