Browser Video Playback Principles: From video Tag to MSE, EME, WebCodecs

Why can't Chrome play M3U8? Why is HLS slower than MP4? How can Bilibili play FLV? Why can't Netflix videos be pirated? The answers lie in browser video playback technology. This guide covers everything from basic video tags to MSE, EME, WebCodecs.

Evolution of browser video playback

Browser video has gone through three generations:

Generation 1: HTML5 video tag (pre-2010)

Simplest form:

<video src="video.mp4" controls></video>

Browser native support, plays MP4/WebM single files directly. Limitations:

• Only "complete files," no streaming
• Only browser-native codecs (H.264, VP8/9)
• No encrypted content
• No live streaming

Flash plugin filled these gaps, but Flash is dead.

Generation 2: MSE + EME (2013-present)

Media Source Extensions (MSE) solved streaming playback:

const video = document.querySelector('video');
const mediaSource = new MediaSource();
video.src = URL.createObjectURL(mediaSource);

mediaSource.addEventListener('sourceopen', () => {
  const sourceBuffer = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.42E01E, mp4a.40.2"');
  
  // Dynamically append segment data
  fetch('chunk-1.m4s')
    .then(res => res.arrayBuffer())
    .then(data => sourceBuffer.appendBuffer(data));
});

Encrypted Media Extensions (EME) solved DRM encryption, enabling Netflix etc. to play copyrighted content in browsers.

Generation 3: WebCodecs (2021-present)

WebCodecs API lets JS directly call browser hardware codecs, breaking free of video tag limitations:

// Decode video frames
const decoder = new VideoDecoder({
  output: (frame) => {
    // Got raw VideoFrame, can apply effects
    canvas.getContext('2d').drawImage(frame, 0, 0);
    frame.close();
  },
  error: (e) => console.error(e),
});

decoder.configure({
  codec: 'avc1.42E01E',
  hardwareAcceleration: 'prefer-hardware',
});

// Feed encoded data
decoder.decode(new EncodedVideoChunk({
  type: 'key',
  timestamp: 0,
  data: h264Data,
}));

WebCodecs enables screen recording, streaming, video effects, custom players — things previously impossible.

MSE: Core of streaming playback

MSE is the foundation of modern web video. All major player libraries (hls.js, dash.js, flv.js, Shaka Player) build on it.

How MSE works

1. Create MediaSource object: JS creates a virtual "media source"
2. Bind to video tag: Generate blob URL via URL.createObjectURL(), assign to video.src
3. Create SourceBuffer: Independent buffer for each stream (video, audio)
4. Append segment data: JS downloads segments, converts to ArrayBuffer, calls sourceBuffer.appendBuffer()
5. Browser decodes and plays: video tag pulls data from SourceBuffer, decodes and renders

MSE limitations

• Must use fMP4 or WebM: Native MSE doesn't support TS segments — hls.js converts TS to fMP4 before feeding MSE
• Codec must be declared upfront: Codec specified when creating SourceBuffer, can't change mid-stream
• Memory management: Appended data must be manually removed to free memory
• Strict ordering: Must append in timestamp order, otherwise errors

MSE codec support

Codec	Chrome	Firefox	Safari	Edge
H.264 (avc1)	✅	✅	✅	✅
H.265 (hevc)	🟡	❌	✅	🟡
VP9	✅	✅	✅ (iOS 14+)	✅
AV1	✅ (85+)	✅ (68+)	✅ (16.4+)	✅
AAC	✅	✅	✅	✅
Opus	✅	✅	✅ (partial)	✅

How hls.js, dash.js, flv.js work

hls.js: Playing HLS in non-Safari browsers

hls.js is a typical MSE application:

1. Download M3U8: fetch requests M3U8 file
2. Parse M3U8: JS parses text format, extracts segment URLs
3. Download TS segments: Sequentially fetch each .ts file
4. Convert TS to fMP4: Convert MPEG-TS container to fMP4 (key step, MSE doesn't accept TS)
5. Append to MSE: Call sourceBuffer.appendBuffer() to feed video
6. Adaptive bitrate: Monitor download speed and buffer, switch M3U8 quality

The whole process is transparent to developers. After importing hls.js, just a few lines:

if (Hls.isSupported()) {
  const hls = new Hls();
  hls.loadSource('https://example.com/stream.m3u8');
  hls.attachMedia(video);
}

For details, see our HLS.js complete guide.

dash.js: Playing MPEG-DASH in browsers

dash.js works similarly to hls.js, differences:

• Parses MPD (XML) instead of M3U8 (text)
• Segments typically fMP4, not TS — no container conversion needed
• Directly appends fMP4 to MSE

flv.js: Playing FLV in browsers

Bilibili's open-source flv.js lets Chrome play FLV streams:

1. Parse FLV container header
2. Extract AAC audio and H.264 video streams from FLV
3. Repackage as fMP4
4. Append to MSE for playback

For details, see our flv.js RTMP/FLV player guide.

EME: DRM encrypted playback

Netflix, Disney+ videos can't be pirated thanks to EME (Encrypted Media Extensions) + DRM systems.

EME workflow

1. Browser requests encrypted content: Video segments are encrypted, can't decode directly
2. JS calls EME API: Create DRM session via MediaKeySystemAccess.createMediaKeys()
3. Generate license request: Browser requests decryption key from DRM license server
4. Server verifies user: Confirms user has rights via Cookie, Token, etc.
5. Deliver key: Server returns encrypted key
6. Browser decrypts internally: In secure area (CDM, Content Decryption Module)
7. Decode and play: Decrypted data goes to decoder, normal playback

Three major DRM systems

DRM	Vendor	Browser support	Use case
Widevine	Google	Chrome, Firefox, Edge, Android	Web default
PlayReady	Microsoft	Edge, IE, Xbox	Windows ecosystem
FairPlay	Apple	Safari, iOS	Apple ecosystem

Commercial platforms need to support all three DRMs to cover all browsers. Open-source reference: Shaka Player DRM implementation.

WebCodecs: New era of browser video processing

WebCodecs lets JS directly control encoding/decoding, breaking video tag limitations.

What WebCodecs can do

1. Screen recording and sharing: Capture screen frames, encode to H.264/VP9, stream or save
2. Video effects: Decode video frames to Canvas, apply filters, re-encode
3. Low-latency playback: Custom players, bypassing video tag overhead
4. Video transcoding: Transcode in browser, no server needed
5. AI video processing: Feed video frames to TensorFlow.js for recognition

Screen recording streaming example

// Capture screen
const stream = await navigator.mediaDevices.getDisplayMedia({ video: true });

// Encode with WebCodecs
const track = stream.getVideoTracks()[0];
const processor = new MediaStreamTrackProcessor({ track });
const reader = processor.readable.getReader();

const encoder = new VideoEncoder({
  output: (chunk, meta) => {
    // Send encoded H.264 data via WebSocket/WebRTC
    sendChunk(chunk);
  },
  error: (e) => console.error(e),
});

encoder.configure({
  codec: 'avc1.42E01E',
  width: 1920,
  height: 1080,
  bitrate: 4_000_000,
  framerate: 30,
});

while (true) {
  const { done, value: frame } = await reader.read();
  if (done) break;
  encoder.encode(frame, { keyFrame: false });
  frame.close();
}

Root causes of video playback stuttering

Understanding principles helps pinpoint stuttering causes:

1. Network bottleneck

• Download speed < video bitrate: Buffer depletes, stuttering inevitable
• Diagnosis: Check video.buffered for buffer duration, dangerous if <2 seconds
• Solution: Lower bitrate, use ABR for lower quality, CDN acceleration

2. Decoding bottleneck

• High CPU/GPU usage: Decode speed can't keep up with playback
• Diagnosis: Chrome DevTools Performance panel
• Solution: Hardware decoding, lower resolution, lower frame rate

3. MSE buffer management

• SourceBuffer full: Append fails, playback interrupts
• Diagnosis: Monitor sourceBuffer.updateend event, check sourceBuffer.buffered
• Solution: Regularly sourceBuffer.remove() to clean played content

4. Main thread blocking

• JS execution blocks video rendering: Large computations, long tasks block main thread
• Diagnosis: Performance panel for long tasks
• Solution: Move computation to Web Worker, use requestIdleCallback

5. GC pauses

• Large ArrayBuffers trigger GC: Appending segments allocates lots of memory
• Diagnosis: Performance panel Memory view
• Solution: Reuse ArrayBuffers, control segment size

Browser video technology boundaries

What can web video playback achieve?

Capability	video tag	MSE	WebCodecs
Play MP4 single file	✅	✅	✅
Play HLS stream	❌	✅ (hls.js)	✅
Play DASH stream	❌	✅ (dash.js)	✅
Play DRM content	❌	✅ (EME)	✅
Real-time screen streaming	❌	❌	✅
Video frame effects	❌	❌	✅
Custom codec	❌	❌	✅
Hardware accelerated decoding	✅	✅	✅

MP4 native vs HLS playback

Why is HLS slower than MP4? From principle:

Dimension	MP4 native	HLS (hls.js)
Startup requests	1 (direct MP4)	2 (M3U8 + first segment)
Startup latency	Near 0 (progressive)	2-10 seconds (wait for segment)
Buffer strategy	Browser internal	JS manual MSE management
Main thread overhead	Minimal	Significant (parsing, remuxing)
ABR support	❌	✅
Live support	❌	✅
Encryption support	❌	✅ (DRM)

Simple scenarios use MP4 native for fastest, most stable playback. Complex scenarios (live, ABR, DRM) require MSE solutions.

Summary

Browser video has evolved from simple video tags to MSE+EME+WebCodecs, capable of almost everything desktop players can do.

• video tag: Play local MP4/WebM, simplest
• MSE: Streaming playback core, hls.js/dash.js/flv.js foundation
• EME: DRM encrypted playback, used by Netflix/Disney+
• WebCodecs: Direct codec control, new era for screen recording, streaming, effects

Understanding this technology stack helps pinpoint playback issues and avoid pitfalls in product tech choices.

Quick reference:

• video tag: MP4 single file playback
• MSE: Streaming core, hls.js/dash.js/flv.js foundation
• EME: DRM encrypted playback
• WebCodecs: New era of browser codec control
• Playback stuttering: Network/decoding/MSE buffer/main thread/GC — five causes

References

• Media Source Extensions — MDN
• Encrypted Media Extensions — MDN
• WebCodecs API — MDN
• HLS.js — GitHub
• Web Video Technology — Google Developers