Skip to content

Proxy-System.md

Latest commit

 

History

History
161 lines (102 loc) · 5.59 KB

Proxy-System.md

File metadata and controls

161 lines (102 loc) · 5.59 KB

Proxy System

Back to Index

JPEG image proxy generation and playback for smoother scrubbing of large video files.

Overview

Proxies are stored inside the project folder and are used only when proxy mode is enabled. The current implementation does not generate a separate proxy folder picker or a detached proxy library.

Current Behavior

  • Proxy mode mutes and pauses the original video elements when enabled.
  • Video proxies are stored as packed JPEG frame data in the project folder.
  • The editor falls back to the original media when proxy data is missing.
  • Audio proxy files are optional and non-fatal.
  • The all-intra MP4 proxy path remains in the codebase for quick reactivation, but it is not the active generation or playback path.
  • Existing mp4-all-intra proxy status does not count as complete for the active JPEG path; enabling proxy mode can regenerate the JPEG proxy frames.

Proxy Generation

Proxy generation is handled by ProxyGeneratorWebCodecs.

Current Pipeline

  1. MP4Box parses the source file.
  2. Codec configuration is extracted from the sample entry (avcC, hvcC, vpcC, or av1C) and passed to WebCodecs.
  3. WebCodecs VideoDecoder decodes frames.
  4. Decoded VideoFrame objects are transferred to a bounded Dedicated Worker pool.
  5. Each worker resizes on OffscreenCanvas and encodes a JPEG frame.
  6. JPEG frames are saved into project proxy pack files plus an index.

Current Settings

  • Maximum width: 1280 px
  • Proxy frame rate: 30 fps
  • Decode batch size: 30 samples
  • Image format: JPEG
  • JPEG quality: 0.82
  • Worker encoder pool: up to 8 Dedicated Workers, leaving 2 hardware threads reserved when possible

Queue Support

  • Enabling proxy mode starts the next missing video proxy immediately.
  • When proxy mode is already enabled, newly imported videos are added to the proxy generation flow as soon as import finishes.
  • The timeline proxy button shows the active queue position while generating, for example Generating 1/5.

Completion Rule

  • A proxy is marked ready when the generated JPEG frame index contains at least 98 percent of the expected frame indices.

Resource Limit

  • Only one proxy generation runs at a time.
  • Additional videos are processed sequentially by the proxy generation queue.

Storage

Proxies are stored in the project folder under Proxy/{mediaId}/.

Current On-Disk Layout

  • Video proxies are written as packed JPEG data: Proxy/{mediaId}/frames_0000.pack, Proxy/{mediaId}/frames_0001.pack, and Proxy/{mediaId}/frames.index.json.
  • The index maps each frame index to a pack filename, byte offset, byte size, and MIME type.
  • Older frame_000000.jpg and .webp frame files are still readable for project compatibility, but new active generation writes pack files.
  • Audio proxies are written as WAV files under the project audio-proxy folder, using a sanitized storage-key filename such as <mediaId>.wav. Older Proxy/{mediaId}/audio.wav and Proxy/{mediaId}/audio.m4a files are still read for compatibility.

Backend Caveat

  • Image proxy storage currently uses the File System Access project handle path.

Deduplication

  • Storage is keyed by fileHash when available.
  • If no file hash is available, the media file ID is used.

Proxy Playback

proxyFrameCache loads JPEG frame blobs on demand and keeps decoded HTMLImageElement objects in memory for scrubbing.

Current Behavior

  • Exact image-frame lookups are cached in memory.
  • Nearest-frame and held-frame fallbacks smooth scrubbing while requested frames are still loading.
  • Playback can use proxy audio when it exists.
  • Missing proxy frames fall back to the original source media.

Cache Limits

  • Image-frame cache size: 900 frames
  • Scrubbing preload window: 90 frames around the scrub position in active scrubs
  • Parallel preload batch size: 16

Limitation

  • The proxy cache reads image frames from the project folder. It does not use IndexedDB as an alternate store.

Warmup

The warmup button in the proxy cache path does not generate proxy files.

What It Does

  • It seeks the source video elements in 0.5 second steps.
  • It is meant to warm browser decode and cache state.
  • It includes nested composition clips.

What It Does Not Do

  • It does not create new proxy frames.
  • It does not convert media into proxy format.

Audio Proxies

After the video frames finish, the code attempts to extract audio in the background.

Current Behavior

  • Audio extraction is non-blocking after the JPEG proxy frames complete.
  • Audio proxy failures are treated as non-fatal.
  • If extraction succeeds, the current audio proxy is saved as WAV. Legacy audio.m4a proxy files remain readable.
  • Scrub audio uses decoded WAV/AudioBuffer data and schedules pitch-stable short grains with minimal overlap.
  • Fast scrub jumps fade out older grains before scheduling the new position so stale audio does not stack up.

Limitation

  • Proxy audio is best-effort. The editor keeps working even if audio extraction fails.

Current Limitations

  • Native Helper-backed projects do not currently persist image proxy files through the same native path.
  • Proxy generation is browser-session based and relies on WebCodecs and OffscreenCanvas support.
  • Only one generation can run at a time.

Sources

Key implementation files:

  • src/services/proxyGenerator.ts
  • src/workers/proxyFrameEncodeWorker.ts
  • src/services/proxyFrameCache.ts
  • src/stores/mediaStore/slices/proxySlice.ts
  • src/stores/timeline/proxyCacheSlice.ts
  • src/services/project/ProjectFileService.ts
  • src/services/project/domains/ProxyStorageService.ts