moq-video

The native video stack: grab pictures from a camera or a screen, encode them with a hardware codec, publish them as a hang track, and do the same in reverse down to a texture on screen.

This is what the browser gets from WebCodecs and getUserMedia. A native application has no such thing, so moq-video provides it: no ffmpeg, no GStreamer, no system codec to install. Just the platform APIs, plus a vendored statically-linked openh264 so a build never depends on what the host happens to have.

Overview

Four role modules, symmetric on both ends of the wire:

Module	Does	Platform
capture	Camera, display, window, or application frames	AVFoundation + ScreenCaptureKit (macOS), V4L2 + PipeWire (Linux), Media Foundation + DXGI (Windows)
encode	Raw frames to H.264/H.265, published through moq-mux	VideoToolbox, Media Foundation, NVENC, VAAPI, openh264
decode	A subscribed track back to raw frames	VideoToolbox, Media Foundation/DXVA, NVDEC, openh264
render	A frame drawn on the GPU, handed back as a wgpu texture	wgpu, with zero-copy Metal and Vulkan imports

A picture is a Frame wherever it crosses the API: a moq_net::Timestamp and a Surface holding the pixels. Capture and decode produce them, encode and render consume them.

Backend selection is automatic and ordered: platform hardware first, then the software fallback. The Linux hardware libraries are loaded at runtime (dlopen), so a binary built with NVENC still links on a GPU-less builder and starts on a machine with no NVIDIA driver, falling through to the next backend instead of failing to load. Falling past a hardware backend that was compiled in logs a warning naming what refused and why, so a host encoding on the CPU when it shouldn't be is visible without turning on debug logging. No public type, function, or error variant names a backend, so swapping one is never a breaking change for you.

openh264 is the fallback for H.264 only. It is statically linked and always compiled in, so H.264 encodes and decodes on any machine. H.265 is hardware-only: with no usable platform backend you get NoEncoder / NoDecoder rather than a slow path. AV1 is decode-only, via NVDEC.

Installation

cargo add moq-video

Capture and NVIDIA hardware codecs are on by default. VAAPI is not, since that backend has never been validated on real hardware, and neither are rendering and PipeWire screen capture, which pull in a graphics stack and a libpipewire build dependency respectively:

cargo add moq-video --features render,pipewire

Feature	Default	Pulls in
capture	yes	Native device capture (v4l and zune-jpeg on Linux)
nvidia	yes	NVENC encode and NVDEC decode on Linux (cudarc, moq-nvenc)
vaapi	no	Intel/AMD encode on Linux (moq-vaapi), unvalidated on hardware
render	no	wgpu, the GPU renderer, and Linux DMA-BUF support
pipewire	no	Wayland/X11 screen capture via xdg-desktop-portal and DMA-BUF

--no-default-features gives a codec-only build that still encodes and decodes H.264 with openh264 but omits native capture and the Linux GPU dependencies. A relay or language binding that only handles supplied frames needs none of them.

Publishing

encode::publish_capture is the turnkey path: it advertises the track and catalog up front, then opens the camera only while somebody is watching and releases it when the last subscriber leaves.

use moq_video::{capture, encode};

// Defaults to the default camera; pick a specific one by id.
let mut config = capture::Config::default();
if let Some(camera) = capture::cameras().await?.first() {
    config.source = camera.source();
}

encode::publish_capture(
    broadcast,
    catalog,
    config,
    encode::Options::default(),
    clock,
).await?;

To encode frames you produced yourself (from a decoder, a game engine, or your own pixels), drive encode::Encoder and publish the results with encode::Producer. Keyframes are the encoder's business: it inserts them per encode::Config::gop, and Encoder::keyframe is there for the rarer case where you need one at a specific frame.

encode::Sink is the same encoder with a thread of its own, and an async API on top. Reach for it when the codec outlives a single thread's stack: an object shared between threads, a handle behind an FFI boundary, or a task that migrates between executor workers. Hardware codecs are not all thread-agnostic (a Media Foundation MFT's COM apartment is per-thread, so building it on one thread and dropping it on another corrupts COM state), and the sink confines the whole encoder lifetime to one thread so callers do not have to. Awaiting rather than blocking is the point: the executor keeps its worker while a slow hardware encoder works through a frame. A plain Encoder you build, drive, and drop inside one function needs none of this.

Subscribing

decode::Consumer is the mirror. It reads the rendition's catalog entry to pick a decoder, then hands back one Frame per call:

use moq_video::decode;

// `rendition` is the hang catalog's VideoConfig for the track you want.
let mut video = decode::Consumer::new(&broadcast, &rendition, "video", decode::Config::default()).await?;

while let Some(frame) = video.read().await? {
    // frame.timestamp, frame.surface
}

Zero-copy

Surface is a #[non_exhaustive] enum naming what actually holds a frame's pixels: a CVPixelBuffer on macOS, a Direct3D 11 texture on Windows, CUDA or a DMA-BUF on Linux, or plain I420 anywhere. Keeping a frame in one of the native representations avoids a round trip through system memory on every frame.

How far that gets today depends on the platform, so here is the honest matrix rather than a blanket promise:

Platform	Decode output	Zero-copy transcode	Zero-copy render
macOS	PixelBuffer (VideoToolbox)	yes	yes, via CVMetalTextureCache
Linux	Cuda (NVDEC)	yes, straight into NVENC	decoded CUDA frames: no; packed PipeWire DMA-BUF capture: yes, via Vulkan
Windows	Texture (Media Foundation / DXVA)	yes, through the Direct3D11 video processor	no, downloaded to I420 first

Frame::resize stays on the GPU through a VTPixelTransferSession, CUDA kernel, or Direct3D11 video processor. Call Frame::resize_with with resize::Acceleration::Cpu to force a download and CPU resize. A driver that rejects GPU resizing returns to CPU scaling and warns once. Linux Vulkan can import packed RGB DMA-BUF screen frames. Multi-plane NV12 import and retiling a modifier that Vulkan rejects remain tracked in #2819.

Matching on Surface stays portable because every variant has a universal fallback in Surface::into_i420(): take the fast path you recognize and let the _ arm download. The renderer does exactly this, and an import path that keeps failing retires itself after a few frames instead of paying for the attempt forever. Set render::Config::zero_copy to false to force the download path when comparing output or working around a driver.

Rendering

render::Renderer takes a wgpu device and queue and hands back a wgpu::Texture per frame. That texture is the entire integration seam: present it to a window, feed it to egui or bevy, or copy it back. The module carries no windowing or UI dependency and never picks a surface format for you.

use moq_video::render::{Config, Renderer};

let mut renderer = Renderer::new(&device, &queue, Config::new())?;

while let Some(frame) = video.read().await? {
    let texture = renderer.render(&frame)?;
    // ... present it ...
}

The wgpu version this was built against is re-exported as moq_video::render::wgpu, so you name the exact version rather than guessing at a compatible one.

On Linux, request wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF when creating the device to activate the PipeWire DMA-BUF fast path. The renderer still works without it and falls back to a CPU upload for linear allocations.

Color names the matrix and range (BT.601 or BT.709, limited or full), and the shader converts per frame rather than assuming one space. A capture labels what it produced, so a locally captured frame renders correctly with no help from you.

A decoded frame is the case to watch. The authoritative answer lives in the bitstream's VUI and does not survive decoding, and a Windows or CUDA surface carries no color metadata at all, so the renderer falls back to inferring from resolution (BT.601 at 576 lines or fewer, BT.709 above). That is a good guess, not a correct one: full-range content, or an unusual resolution, will render with the wrong range or matrix. Set render::Config::color when you know the stream's color space and the frame does not.

Devices

Each enumerator returns ids that go straight back into capture::Config::source:

moq_video::capture::cameras().await?;   // webcams
moq_video::capture::displays().await?;  // monitors
moq_video::capture::windows().await?;   // single windows (macOS)
moq_video::capture::apps().await?;      // every window of an app (macOS)

The moq devices subcommand prints the same lists from the command line.

API Reference

Full API documentation: docs.rs/moq-video

Next Steps

• Pair it with moq-audio for the other half of a call
• Publish through hang catalogs and moq-mux containers
• Capture and publish from the command line with moq-cli