Media over QUIC - Hang

INFO

Rendered from the Internet-Draft source in this repository. Submitted versions are on the IETF datatracker.

Abstract

Hang is a real-time conferencing protocol built on top of moq-lite. A room consists of multiple participants who publish media tracks. All updates are live, such as a change in participants or media tracks.

Conventions and Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 RFC2119 RFC8174 when, and only when, they appear in all capitals, as shown here.

Terminology

Hang is built on top of moq-lite draft-lcurley-moq-lite and uses much of the same terminology. A quick recap:

• Broadcast: A collection of Tracks from a single publisher.
• Track: A series of Groups, each of which can be delivered and decoded out-of-order.
• Group: A series of Frames, each of which must be delivered and decoded in-order.
• Frame: A sized payload of bytes representing a single moment in time.

Hang introduces additional terminology:

• Room: A collection of participants, publishing under a common prefix.
• Participant: A moq-lite broadcaster that may produce any number of media tracks.
• Catalog: A JSON document that describes each available media track, supporting live updates.
• Container: A tiny header in front of each media payload containing the timestamp.

Discovery

The first requirement for a real-time conferencing application is to discover other participants in the same room. Hang does this using moq-lite's ANNOUNCE capabilities.

A room consists of a path. Any participants within the room MUST publish a broadcast with the room path as a prefix which SHOULD end with the .hang suffix.

For example:

/room123/alice.hang
/room123/bob.hang
/room456/zoe.hang

A participant issues an ANNOUNCE_PLEASE message to discover any other participants in the same room. The server (relay) will then respond with an ANNOUNCE message for any matching broadcasts, including their own.

For example:

ANNOUNCE_PLEASE prefix=/room/
ANNOUNCE suffix=alice.hang active=true
ANNOUNCE suffix=bob.hang   active=true

If a participant leaves or is disconnected, their broadcast is unannounced. Publishers and subscribers SHOULD terminate any subscriptions once a participant is unannounced.

ANNOUNCE suffix=alice.hang active=false

Catalog

The catalog describes the available media tracks for a single participant. It's a JSON document that extends the W3C WebCodecs specification WebCodecs.

The catalog is published as a catalog.json track within the broadcast so it can be updated live as the participant's media tracks change. A participant MAY forgo publishing a catalog if it does not wish to publish any media tracks now and in the future.

The catalog track consists of multiple groups, one for each update. Each group contains a single frame with UTF-8 JSON.

A publisher MUST NOT write multiple frames to a group until a future specification includes a delta-encoding mechanism (via JSON Patch most likely).

A publisher SHOULD also serve the catalog as a catalog.json.z track: the identical JSON under the same group and frame rules, differing only by compression (Compression). A consumer reads whichever of the two tracks it prefers.

Root

The root of the catalog is a JSON document with the following schema:

type Catalog = {
  "audio": AudioSchema | undefined,
  "video": VideoSchema | undefined,
  // ... any custom fields ...
}

Additional fields MAY be added based on the application. The catalog SHOULD be mostly static, delegating any dynamic content to other tracks.

For example, a "chat" section should include the name of a chat track, not individual chat messages. This way catalog updates are rare and a client MAY choose to not subscribe.

This specification currently only defines audio and video tracks.

Video

A video track contains the necessary information to decode a video stream.

type VideoSchema = {
  "renditions": Map<TrackName, VideoDecoderConfig>,
  "display": {
    "width": number,
    "height": number,
  } | undefined,
  "rotation": number | undefined,
  "flip": boolean | undefined,
}

The renditions field contains a map of track names to video decoder configurations. See the WebCodecs specification for specifics and registered codecs. Any field carrying raw bytes, notably description, is a hex string (Binary Fields).

The display field is the size to render the video at, in pixels. It is separate from a rendition's displayAspectWidth/displayAspectHeight because changing it does not require reinitializing the decoder.

In addition to the WebCodecs fields, each rendition MAY carry the fields common to audio and video (Common Rendition Fields) plus:

type VideoDecoderConfigExtensions = {
  "displayAspectWidth": number | undefined,
  "displayAspectHeight": number | undefined,
  "stalled": boolean | undefined,
}

displayAspectWidth and displayAspectHeight give the display aspect ratio of the media, stretching or shrinking the coded pixels. A consumer that understands neither field MUST assume square pixels, a 1:1 ratio. Both MUST be present together; a consumer that sees only one MUST ignore it.

stalled indicates that the publisher recommends temporarily avoiding the rendition. The track remains available when stalled is true. A consumer SHOULD select an unstalled rendition when it supports one, but MAY select a stalled rendition when no unstalled rendition is suitable. If absent, stalled defaults to false.

For example:

{
  "renditions": {
    "720p": {
      "codec": "avc1.64001f",
      "container": { "kind": "legacy" },
      "codedWidth": 1280,
      "codedHeight": 720,
      "bitrate": 6000000,
      "stalled": true,
      "framerate": 30.0,
      "jitter": 33
    },
    "480p": {
      "codec": "avc1.64001e",
      "container": { "kind": "legacy" },
      "codedWidth": 848,
      "codedHeight": 480,
      "bitrate": 2000000,
      "framerate": 30.0,
      "jitter": 33
    }
  },
  "display": {
    "width": 1280,
    "height": 720
  },
  "rotation": 0,
  "flip": false,
}

Audio

An audio track contains the necessary information to decode an audio stream.

type AudioSchema = {
  "renditions": Map<TrackName, AudioDecoderConfig>,
}

The renditions field contains a map of track names to audio decoder configurations. See the WebCodecs specification for specifics and registered codecs. Any field carrying raw bytes, notably description, is a hex string (Binary Fields).

In addition to the WebCodecs fields, each rendition MAY carry the fields common to audio and video (Common Rendition Fields).

PCM

Hang defines the "pcm" audio codec for uncompressed samples. The sampleRate and numberOfChannels fields MUST be present and greater than zero. The description field MUST NOT be present. If bitrate is present, it MUST equal sampleRate * numberOfChannels * 32.

Each codec payload consists of interleaved IEEE 754 binary32 samples in little-endian byte order. Samples are ordered by sample frame, then by ascending channel index within each frame. The payload length MUST be a non-zero multiple of 4 * numberOfChannels. The frame timestamp identifies the presentation time of its first sample. The frame duration in seconds is the payload length divided by 4 * numberOfChannels * sampleRate.

For example:

{
  "renditions": {
    "stereo": {
      "codec": "opus",
      "container": { "kind": "legacy" },
      "sampleRate": 48000,
      "numberOfChannels": 2,
      "bitrate": 128000,
      "jitter": 20
    },
    "mono": {
      "codec": "opus",
      "container": { "kind": "legacy" },
      "sampleRate": 48000,
      "numberOfChannels": 1,
      "bitrate": 64000,
      "jitter": 20
    }
  },
}

Binary Fields

A decoder config field carrying raw bytes, notably description (an AllowSharedBufferSource in WebCodecs), is carried in the catalog as a hex string (RFC 4648, Section 8). A publisher SHOULD emit lowercase hexadecimal characters and MUST NOT emit a 0x prefix or any separators. A consumer MUST accept either case.

Note that this differs from the cmaf container's init field (Container), which is base64 (RFC 4648, Section 4); the two alphabets overlap, so the encoding cannot be detected and must be specified.

Common Rendition Fields

Audio and video renditions share the following fields, extending the WebCodecs decoder config:

type CommonExtensions = {
  "broadcast": string | undefined,
  "container": Container,
  "jitter": number | undefined,
  "timeline": Timeline | undefined,
}

broadcast

By default a rendition's track lives in the same broadcast that served the catalog. The broadcast field overrides that, naming a different broadcast that publishes the track.

The value is a relative path, resolved against the path of the broadcast that served the catalog. It uses relative reference resolution (RFC 3986, Section 5.2): a non-empty reference replaces the catalog broadcast's last path segment before applying . and .. segments. For example, ./source in a catalog served by room/transcode resolves to room/source, while . resolves to room. An empty reference resolves to the catalog broadcast itself. A publisher MUST NOT use an absolute path, and a consumer MUST ignore a rendition whose broadcast escapes above the root.

This lets a publisher author a catalog that points at tracks it does not republish. For example, a transcoder produces a catalog listing its own downstream renditions alongside the untouched source rendition, referencing the latter in the source broadcast rather than copying the bytes through.

A consumer subscribes to such a rendition in the referenced broadcast, using the rendition's track name unchanged.

container

The container used to frame this rendition's media, as described in Container. If absent, it defaults to { "kind": "legacy" }.

jitter

The maximum delay, in milliseconds, between a frame being ready and the publisher flushing it. A consumer's jitter buffer SHOULD be at least this large to avoid stalling. If absent, a consumer SHOULD assume each frame is flushed immediately.

For example:

• If each frame is flushed immediately, a video track's jitter is 1000/framerate.
• If up to 3 B-frames may be emitted in a row, it is 3 * 1000/framerate.
• If frames are buffered into 2 second segments, it is 2000.

An audio frame's duration is codec dependent. AAC often uses 1024 samples per frame, so at 44100Hz an immediately-flushed track's jitter is 23.

timeline

The companion timeline track indexing this rendition's groups (Timeline), if the publisher offers one.

Container

Audio and video tracks use a container to encapsulate the media payload. A rendition declares its container via the container field of its catalog entry (Common Rendition Fields):

type Container =
  { "kind": "legacy" } |
  { "kind": "cmaf", "init": string } |
  { "kind": "loc" }

The kind field selects the framing; a consumer MUST ignore a rendition whose kind it does not recognize. Every container shares the same group rules:

Each moq-lite group MUST start with a keyframe. If the codec does not support delta frames (e.g. audio), a group MAY consist of multiple keyframes. Otherwise, a group MUST consist of a single keyframe followed by zero or more delta frames.

An empty group declares a discontinuity between codec epochs. A consumer MUST reset codec state before decoding the next non-empty group, including reapplying any codec startup delay or pre-skip. This applies whether the resumed timestamps move backward or forward.

legacy

The default, used when the container field is absent.

Each frame starts with a timestamp, a QUIC variable-length integer (62-bit max) encoded in microseconds. The remainder of the payload is codec specific; see the WebCodecs specification for specifics.

A frame with an empty codec payload is an end marker, not media. Its timestamp is the exclusive endpoint of the source media. When a codec must receive additional packets to emit buffered source samples, the marker MUST precede those terminal packets. A consumer MUST NOT submit the marker to the codec decoder, MUST decode the terminal packets, and MUST discard decoded samples at or after the endpoint.

For example, h.264 with no description field would be annex.b encoded, while h.264 with a description field would be AVCC encoded.

cmaf

Each frame is a complete fragmented MP4 fragment (moof+mdat), carrying its own timestamps.

The init field is the initialization segment (ftyp+moov) for the track, base64-encoded (RFC 4648, Section 4). A consumer MUST feed init to the decoder before the first frame.

loc

Each frame is a Low Overhead Container frame draft-ietf-moq-loc: a property block, carrying the timestamp among other properties, followed by the codec payload.

Compression

Some metadata tracks are compressed, conventionally marked with a .z suffix on the track name.

Each group is one raw DEFLATE stream (RFC 1951), sync-flushed at each frame boundary. Each frame is therefore a self-delimited, byte-aligned slice, while later frames compress against the earlier ones in the same group. A consumer MUST decompress a group's frames in order, starting from the first.

A sync flush ends with the empty-block marker 0x00 0x00 0xff 0xff. A publisher MUST omit this trailing marker from each frame and a consumer MUST append it before decompressing, the same trick as permessage-deflate (RFC 7692, Section 7.2.1).

Timeline

A media track MAY have a companion timeline track: an ordered log mapping the media track's groups to their start timestamps. On the wire a group carries only a sequence number; the timestamps live inside the media frames. The timeline republishes that mapping as metadata, so a consumer can determine which group covers a given time, and where the live edge is, without downloading the media itself. This is the primitive used to seek, or to serve HLS/DASH playlists.

A rendition advertises its timeline via the timeline field (Common Rendition Fields):

type Timeline = {
  "track": string,
  "timescale": number | undefined,
  "wall": number | undefined,
}

The track field names the timeline track, published in the same broadcast as the media track it indexes: the rendition's broadcast field (broadcast), when present, relocates the timeline along with the media. The conventional name is the media track's name with a .timeline.z suffix, but a consumer MUST use the name from the catalog rather than deriving it. Renditions MAY share a timeline track when their group boundaries are aligned, for example a transcode ladder mirroring the source's groups. Audio and video groups have different durations, so each declares its own timeline.

The timescale field is the number of timestamp units per second, a positive integer defaulting to 1000 (milliseconds). A value of 0 is invalid; a consumer MUST reject a timeline that declares it.

The wall field anchors the timeline to the wall clock, if known: the wall-clock time of timestamp 0, in timescale units since the moq epoch, 2020-01-01T00:00:00Z (1577836800 Unix seconds). Measuring from 2020 rather than 1970 keeps the values small. A consumer derives the wall-clock time of any group as wall + pts.

Timeline integers (timescale, wall, and each record's pts) MUST NOT exceed 2^53 - 1, so they survive JSON consumers that parse numbers as IEEE 754 doubles. A publisher chooses a timescale coarse enough to honor this bound.

Timeline Track

The timeline track is a single compressed (Compression) group that is never rolled. Each frame is one UTF-8 JSON record:

type Record = {
  "group": number,
  "pts": number,
  // ... any custom fields ...
}

The group field is the sequence number of the media track's group, as used by subscriptions and fetches. The pts field is the group's start, its first frame's presentation timestamp, re-expressed in the timeline's timescale (rounding down when the media clock is finer). Additional fields MAY be added based on the application; a consumer MUST ignore fields it does not recognize.

A record is appended when its group opens, so the live edge of the timeline is the live edge of the media. A publisher MAY throttle records to a granularity, emitting at most one record per some amount of media time; video keyframes are usually at least that far apart, while short audio groups are thinned out.

A record therefore spans from its pts until the next record's pts: the named group plus any unrecorded groups that opened in between. The last record's span extends to the live edge, its duration unknown until the next record arrives. A consumer looks up a time by finding the record whose span covers it, which names the group to fetch first. To locate an individual unrecorded group within a span, a consumer MAY extrapolate from the surrounding records when the media track's group sequence numbers are contiguous, or inspect the fetched media itself.

Security Considerations

TODO Security

IANA Considerations

This document has no IANA actions.

Normative References

• draft-lcurley-moq-lite
• WebCodecs

Acknowledgments

TODO acknowledge.