kempnerforge.data.video_io

Video frame sampling and decoding for the VLM video path.

A clip is reduced to an ordered set of still frames that the VLM pipeline treats like a sequence of images. Two concerns live here:

  1. sample_timestamps — which timestamps to sample. This is the policy from the Molmo2 paper (§3.1, §A): sample at a target frame-rate fps, cap the total at max_frames (uniformly subsampling longer clips), and always include the first and last frame. Sampling is expressed in seconds rather than frame indices so it is robust to variable-fps video. This function is pure (no decoder dependency) and unit-tested directly.

  2. decode_video_frames — how to read those frames. Decoding uses PyAV (av), whose manylinux wheel bundles FFmpeg, so no system FFmpeg or matching CUDA libraries are required (torchcodec needs both). av is imported lazily so this module imports cleanly without it; only actual decoding requires the package. Decoding seeks over the keyframe groups that hold no sampled timestamp instead of decoding the whole clip, so cost scales with the frames kept rather than clip length; streams that cannot seek reliably fall back to a single serial pass with identical selection.

Returned frames are PIL.Image objects so the caller can reuse the exact image preprocessing (pil_to_tensor) used on the single-image path.

Functions

decode_video_frames(path, *, fps, ...[, ...])

Decode a clip into a list of sampled PIL.Image frames (RGB).

sample_timestamps(duration_s, fps, ...)

Timestamps (seconds) to sample from a clip of length duration_s.
kempnerforge.data.video_io.sample_timestamps(duration_s, fps, min_frames, max_frames)[source]

Timestamps (seconds) to sample from a clip of length duration_s.

Policy (Molmo2 §3.1/§A): aim for fps frames per second, clamp the count to [min_frames, max_frames], and lay the samples out uniformly over [0, duration_s] so the first frame (0.0) and last frame (duration_s) are always included. A non-positive duration (unknown or instantaneous) yields a single timestamp at the start.

Returns a strictly increasing list of length in [1, max_frames].

Parameters:
Return type:

list[float]

kempnerforge.data.video_io.decode_video_frames(path, *, fps, min_frames, max_frames, sampling_policy='uniform')[source]

Decode a clip into a list of sampled PIL.Image frames (RGB).

Frames are chosen by the registered sampling_policy (default "uniform" = sample_timestamps) and read in a single decode pass: each target timestamp is mapped to the first decoded frame at or after it (timestamps past the last frame map to the last frame, so the final frame is always returned). Frame times and the sampled span come from the video stream’s packet timestamps, read from the same open container (_video_extent): times count from the stream’s start, so a stream whose timestamps start after zero is sampled like one starting at zero, and the span ends where its last frame ends. Where the last packet carries no duration, the container’s duration supplies that end, read as a length or as an end time, so a duration that covers other streams can extend it; for a pipe, which can be read only once, and a stream without timestamps (a raw elementary stream), the whole span comes from the container metadata and times count from the first frame that has a timestamp; a frame without a timestamp counts as time zero. The returned list has length equal to the number of sampled timestamps (<= max_frames), or is empty when the file has no decodable video stream.

Decoding seeks over the keyframe groups that hold no target (_decode_seek), so cost scales with frames kept rather than clip length. If a seek cannot guarantee the same selection, the stream is replayed from its first packet in the same container and decoded in a single pass (_decode_serial), so seeking changes which frames are decoded, never which are returned. A stream that cannot be replayed, a pipe or one that starts off a keyframe, is decoded serially from the start without seeking. One consequence of seeking: damage confined to a skipped stretch is never decoded, so such a clip returns frames where a serial pass would raise; damage in a group that holds a target still raises.

Raises whatever av raises on a missing/corrupt file, and RuntimeError if the container cannot seek back to the stream’s first packet after reading its end; callers that train over noisy data should catch and substitute an empty clip.

Parameters:
Return type:

list[PILImage]