MoQ Cluster Extension
INFO
Rendered from the Internet-Draft source in this repository. Submitted versions are on the IETF datatracker.
Abstract
This document defines a clustering extension for MoQ Transport draft-ietf-moq-transport, used to build a mesh of relays. Each namespace advertisement carries the list of Hop IDs it has passed through, starting with the original publisher, and the accumulated cost of that path. A receiver uses the list to detect loops and to tell which advertisements come from the same publisher, and the cost to choose between paths. Each endpoint declares its own Hop ID at setup, so a peer never advertises or serves it a path that already passed through it.
Note to Readers
This document was generated by an AI model from the implementation at github.com/moq-dev/moq and is maintained alongside it. Submit an issue or PR if this spec sucks and you want to fix anything.
Conventions and Definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 RFC2119 RFC8174 when, and only when, they appear in all capitals, as shown here.
Upstream and downstream are relative to the flow of an advertisement, not to the endpoints: the peer that sends an advertisement is upstream, the one that receives it is downstream. The same pair of relays can be upstream of each other for different namespaces.
Introduction
draft-ietf-moq-transport is designed to deliver content through a mesh of relays but does not say how to build one, and the base protocol does not carry enough information to do so. Relays that simply forward PUBLISH_NAMESPACE to each other break down: advertisements loop forever, and a relay that hears one namespace from two peers has no basis for choosing where to send a SUBSCRIBE.
This extension adds two parameters to PUBLISH_NAMESPACE and NAMESPACE. HOP_PATH lists every endpoint an advertisement has passed through, starting with the original publisher, which breaks loops and lets paths be compared. ROUTE_COST is the accumulated price of the path: the publisher seeds it, and each hop adds the RELAY_COST its upstream declared at setup, so an unpriced mesh ranks by hop count. A relay that already carries a namespace advertises a lower cost, steering subscribers toward its warm copy.
Each endpoint also declares its own Hop ID at setup, so a peer can leave it out of every path it advertises or serves to it, even across several connections between the same two relays. An advertisement is one path, so a relay forwards only the best path it knows per namespace and serves a subscription from one source at a time (Several Publishers of One Namespace).
Setup Negotiation
Hop ID
The extension is negotiated during SETUP (draft-ietf-moq-transport Section 10.3). An endpoint offers it by declaring its own Hop ID:
HOP_ID Setup Option {
Option Key (vi64) = 0x40B54
Hop ID (vi64)
}Negotiation is per session; a relay MUST NOT assume that because one session negotiated the extension, another did. On a session that did, every PUBLISH_NAMESPACE and NAMESPACE MUST carry HOP_PATH, NAMESPACE takes the extended form in Namespace Advertisements, and a receiver MUST close the session with a PROTOCOL_VIOLATION if either arrives without HOP_PATH.
Relay Cost
An endpoint MAY declare what it charges for sending content:
RELAY_COST Setup Option {
Option Key (vi64) = 0x40B56
Option Value (vi64)
}The value prices the sender's own egress, so each endpoint declares its own and the two need not match, as OSPF prices each router's own output interfaces (RFC 2328, Section 9). A receiver adds it to the ROUTE_COST of every advertisement that peer forwards (Accumulating Cost). Absent means 1, so an unpriced mesh ranks by hop count. 0 is distinct from absent: it makes the link free, which is how to describe two relays in the same datacenter.
A declared cost is an assertion, not an instruction: a receiver MAY charge a locally configured value instead, so a peer cannot make itself cheap by saying so.
The cost is one dimensionless integer, as in every deployed routing metric: RIP's hop count (RFC 2453, Section 3.5), OSPF's interface cost, and IS-IS's default metric, whose delay, expense, and error metrics went unimplemented (RFC 5305, Section 3), as did OSPF's per-type-of-service metrics (RFC 2178, Appendix G.10). A deployment that weighs latency, hop count, and price folds them into the one value. Like BGP's MULTI_EXIT_DISC (RFC 4271, Section 5.1.4), the value only means something within the deployment that chose its units, so a trust boundary clamps or replaces it (Security Considerations).
Hop IDs
A Hop ID is a variable-length integer naming one endpoint in a path.
Hop IDs SHOULD be unique among the endpoints an advertisement can traverse. An endpoint MAY pick one at random, since collisions in a 64-bit space are unlikely, or use a configured identifier that survives restarts.
Loops and origins are detected by comparing Hop IDs for equality, so two endpoints sharing one are indistinguishable. Redundant publishers of interchangeable content MAY share one deliberately, so the mesh treats their paths as failover options for the same content (Path Selection).
The Reserved Hop ID 0
0 means "no identity" and is reserved. It stands for an endpoint that did not negotiate this extension, and an endpoint MAY declare it to withhold its identity.
Since any number of endpoints can be 0, it identifies nothing:
- Loop detection: 0 in a HOP_PATH is never a loop. A receiver whose own Hop ID is 0 cannot detect loops through itself and MUST NOT discard an advertisement merely because the path contains 0.
- Origin identity: an advertisement whose first entry is 0 has an unknown publisher, which a relay names by putting a stamp of its own in front (Stamping an Unknown Publisher). A receiver MUST NOT treat two unstamped ones as interchangeable (Path Selection).
- Filtering: a peer that declared 0 gave the receiver nothing to filter that session on. The receiver MAY assign an ID of its own (Assigned Identities) as local selection state and MUST NOT write it into HOP_PATH.
Duplicate non-zero Hop IDs in one HOP_PATH are a loop; duplicate zeros are not. Declaring 0 trades loop detection and failover for anonymity, except against a receiver that assigns an identity of its own.
Assigned Identities
A receiver MAY assign a Hop ID to a peer that declared none, whether by declaring 0 or by not negotiating the extension. It uses that ID as local selection state: as what it filters that session on, including for advertisements that arrived carrying their own HOP_PATH.
The ID is the receiver's own, not the peer's, and MUST NOT be forwarded. An advertisement that arrives with its own HOP_PATH already names the sender there, as 0 if withheld. An unnamed publisher is stamped instead (Stamping an Unknown Publisher), which is not an assigned identity.
An assigned ID MUST NOT be shared between peers not known to be the same endpoint. Sharing one suppresses each one's advertisements to the other, so two unrelated peers would starve each other of routes.
A peer the receiver authenticated, or dialed and therefore chose, SHOULD get one stable ID, so its reconnects and redundant sessions are filtered as one peer; a fresh ID per connection would make one peer look like several. An anonymous accepted session cannot be correlated with anything, so it SHOULD get a distinct ID per session: not an identity, but enough to keep routes learned from it from being advertised back to it, which is the loop 0 cannot prevent.
Stamping an Unknown Publisher
A relay that records an advertisement whose first HOP_PATH entry is 0 MUST insert a random non-zero Hop ID in front of it, picked once for the session the advertisement arrived on. One that arrived with no HOP_PATH (Bridging) becomes that stamp followed by 0.
Nothing on the wire says whether an unnamed publisher that reconnects is the same one, so a fresh stamp per session makes its reconnect a change of publisher downstream (Updating an Advertisement), while an update on the same session keeps the stamp and applies in place. The 0 behind the stamp keeps the path ranked below fully identified ones (Path Selection), since the unnamed hop may hide any depth. A stamp names a session, not a peer: it MUST NOT be reused across sessions or taken as the peer's identity.
Namespace Advertisements
HOP_PATH and ROUTE_COST are Key-Value-Pair parameters (draft-ietf-moq-transport Section 2.5). PUBLISH_NAMESPACE (draft-ietf-moq-transport Section 10.15) already carries parameters. NAMESPACE (draft-ietf-moq-transport Section 10.16) does not, and a subscriber-driven mesh propagates advertisements as NAMESPACE, so this extension appends a parameter block to it:
NAMESPACE Message (Cluster) {
Type (vi64) = 0x8,
Length (16),
Track Namespace Suffix (..),
Number of Parameters (vi64),
Parameters (..) ...
}The added fields are encoded exactly as in PUBLISH_NAMESPACE. Negotiating this extension enables the block on every NAMESPACE, with a parameter count of 0 when it is empty; when another extension defines the same block an endpoint appends one block holding the parameters of both, not two blocks. An endpoint MUST NOT append the block when nothing negotiated it, and MUST NOT include HOP_PATH or ROUTE_COST unless this extension is.
NAMESPACE_DONE (draft-ietf-moq-transport Section 10.17) carries no state from this extension.
An advertisement claims capability, not inventory: namespaces beneath the advertised one can be served, not that any exists. A publisher that serves only some of them advertises the covering namespace and refuses the requests it will not serve (Path Selection).
HOP_PATH Parameter
HOP_PATH is the ordered list of Hop IDs an advertisement has passed through, from the original publisher to the peer sending it:
HOP_PATH Parameter {
Type (vi64) = 0x40B57
Length (vi64)
Hop ID (vi64) ...
}The list always has at least one entry, the original publisher, 0 if unknown (The Reserved Hop ID 0). A receiver MUST close the session with a PROTOCOL_VIOLATION if the list is empty, if the entries do not exactly fill Length, or if a non-zero Hop ID appears twice.
ROUTE_COST Parameter
ROUTE_COST is the marginal cost of subscribing through this advertisement: the price of the transfers a new subscription would cause.
ROUTE_COST Parameter {
Type (vi64) = 0x40B58
Value (vi64)
}It is OPTIONAL and absent means 0. Costs still accumulate across a mesh that sends none, because each receiver adds the RELAY_COST of the link it received over (Accumulating Cost).
The original publisher seeds the value with its production cost: 0 for content it already produces, higher for content it would have to start on demand, such as a standby transcoder advertising everything it could serve.
A standby seed only ranks last if no live path can accumulate past it, which is a property of the deployment, not of the number. A deployment relying on standby ordering within one specificity tier (Path Selection) MUST bound the charged links on an admitted path by H and each link's cost by C, including the receiving link, and MUST enforce both when admitting paths and links. Its live publishers MUST seed 0 and its standby publishers MUST seed above H * C and below saturation; 2^32 is RECOMMENDED where H * C < 2^32. Unknown, out-of-budget, and saturated routes are outside the guarantee: a receiver MUST NOT rank them above standby capacity on the guess that they already carry content.
Relay Behavior
A relay forwarding an advertisement MUST append its own Hop ID to the HOP_PATH it received, so its ID is always the last entry. A received 0 is forwarded unchanged, behind a stamp when it is the first entry (Stamping an Unknown Publisher).
A relay MUST discard an advertisement whose HOP_PATH already contains its own non-zero Hop ID: forwarding it would extend a loop, and subscribing through it would route the relay back to itself. This check catches loops of any length and is the only loop defense required. A conforming sender never sends one (Path Selection), so a receiver MAY close the session with a PROTOCOL_VIOLATION instead; discarding is what keeps the mesh working when one member does not conform.
Bridging
An upstream that did not negotiate the extension sends no HOP_PATH. The relay creates one holding its stamp for that session followed by 0 (Stamping an Unknown Publisher), then appends its own Hop ID. The identity a receiver assigned that upstream (Assigned Identities) is local selection state and MUST NOT appear in HOP_PATH.
Accumulating Cost
Before forwarding or acting on an advertisement, a relay MUST add the RELAY_COST the sender declared (Relay Cost) to the ROUTE_COST it received. The addition MUST saturate rather than wrap, so an absurd value ranks last instead of overflowing to best.
A relay actively carrying the namespace (a live subscription exists for at least one of its tracks) SHOULD advertise 0 instead: its ingress is already paid for, so another subscriber costs only the links below it. This is what lets a cluster converge on a warm copy. The discount applies only to the path it actually serves from; a standby path keeps its accumulated value, since serving from it means opening a fresh ingest. When it stops carrying the namespace it SHOULD restore the accumulated value, optionally after a grace period so brief churn does not flap routing.
Two relays that each begin carrying the same namespace would each see the other's 0 as cheaper than its own source, and if both switched at once the namespace would have no source. Before re-parenting onto a 0-cost advertisement from another actively-carrying relay (one whose HOP_PATH has two or more entries), a relay SHOULD apply a deterministic tie-break, such as comparing a hash of the namespace and each Hop ID, so exactly one side moves. Equal Hop IDs, including two relays that both declared 0, cannot be ordered, and neither side SHOULD move. Cheaper advertisements from anything else carry no such hazard and SHOULD be adopted at once.
Updating an Advertisement
An endpoint updates a PUBLISH_NAMESPACE with REQUEST_UPDATE (draft-ietf-moq-transport Section 9.5) on its request stream, carrying the HOP_PATH or ROUTE_COST that changed. An omitted parameter keeps its value, so a relay that starts carrying a namespace sends an explicit ROUTE_COST of 0. The receiver answers REQUEST_OK, or REQUEST_ERROR and closes the stream, which withdraws the advertisement.
NAMESPACE has no REQUEST_UPDATE, so an endpoint updates one by re-sending it with new parameters on the same SUBSCRIBE_NAMESPACE response stream. A receiver MUST NOT treat the repeat as a duplicate or a protocol violation.
An advertisement lives as long as its stream, so an update on a new stream would leave two streams claiming one namespace. An endpoint MUST NOT open a second stream for an advertisement it already maintains on the session.
An update replaces the old parameters atomically, so a receiver MUST NOT tear down subscriptions or drop cached state because one arrived. If the first HOP_PATH entry is unchanged the content is continuous and subscriptions MAY resume on the new route at a group boundary. If the first entry changed, the publisher changed, and the endpoint still sends an ordinary update. The receiver keeps each subscription it is already serving on the old source until that source ends, MUST NOT resume or splice it onto the updated route, and serves new requests from the updated route without state cached from the old publisher.
The expected update is a ROUTE_COST change, which is how a relay signals that it started or stopped carrying the namespace.
Path Selection
A receiver resolving a SUBSCRIBE, FETCH, or track-status request consults only the most specific advertisements covering it: the longest prefix. A refusal never falls through to a less specific tier.
Within that tier, a receiver SHOULD prefer a HOP_PATH that contains no 0 entry over one that does, then the lowest ROUTE_COST, breaking ties toward the shorter HOP_PATH and then toward the most recently received. This is advisory: a receiver MAY apply local policy, such as measured RTT, instead.
NO_CAPACITY (IANA Considerations) refuses a request the publisher could serve but has no capacity for now. It permits ONE re-resolution within the same tier, excluding the refusing advertiser: every route with its non-zero first Hop ID, or its session when that ID is 0. A receiver that has spent its retry, or has no other candidate, MUST refuse downstream with a code other than NO_CAPACITY, so retries cannot compound hop by hop. Every other refusal, including an unrecognized code, is terminal. A receiver SHOULD NOT cache refusals.
A relay MUST NOT advertise a namespace merely because it resolved it: the covering advertisement stays the only one until the publisher advertises the concrete namespace, which it SHOULD do once producing, so a later request finds the running content by its exact namespace instead of resolving a second producer.
Two advertisements whose HOP_PATH begins with the same non-zero Hop ID come from the same publisher and carry interchangeable content: a receiver MAY hold them as redundant paths and fail an active subscription over to the survivor at a group boundary. If the first entries differ, or either is 0, they are distinct publishers reusing a namespace (Several Publishers of One Namespace).
An endpoint MUST NOT advertise a path whose HOP_PATH contains the Hop ID the peer declared: the peer could only discard it, and acting on it would form a loop. Of the paths that remain it SHOULD advertise the best, and advertises nothing when every path contains that Hop ID. Because selection is per session, a peer that the serving path runs through still receives the best standby, which is what lets it fail over if its own copy dies.
An endpoint MUST select the source for a subscription by the same rule. If only excluded sources remain the subscription is unroutable, since serving it would hand the subscriber data that already flowed through itself. One rule for advertisement and dispatch keeps advertised paths truthful and prevents subscription cycles of any length.
Several Publishers of One Namespace
draft-ietf-moq-transport lets several publishers advertise one namespace and leaves to the relay how it serves a SUBSCRIBE among them. Under this extension an advertisement is a path, so a session advertises a namespace at most once, a relay forwards only the best path it knows (Path Selection), and a subscription is served from one source at a time.
A receiver MAY still hold paths to several publishers of one namespace and choose between them as it sees fit: serve from the cheapest and move to the next when it fails. A refusal moves to another publisher only as Path Selection allows: once, and only for NO_CAPACITY. A relay that moves to another publisher MUST update its advertisement to the new path (Updating an Advertisement), so the first Hop ID downstream names the publisher that new subscriptions reach. Moving between distinct publishers is a discontinuity: their groups are not one sequence, so a subscriber sees an unrelated Location, and a FETCH that succeeds against one may fail against the other.
Redundant publishers of the same content avoid this by sharing a Hop ID (Hop IDs), which makes their paths interchangeable and lets a subscription fail over at a group boundary. Publishers that do not share one are treated as reusing a name.
Security Considerations
A Hop ID reveals nothing beyond what its operator encodes in it; a deployment that considers its identifiers sensitive can use random values or declare 0 (The Reserved Hop ID 0). Declaring 0 hides an identity from the mesh; a peer MAY assign one as local selection state (Assigned Identities) and MUST NOT forward it. A HOP_PATH does reveal how many hops an advertisement crossed, which hints at the size of a deployment; a relay MAY collapse its internal hops into one entry, or strip HOP_PATH, before forwarding across a trust boundary.
Because a relay only appends to HOP_PATH, it cannot make a competing path look shorter than it is; the worst it can do is under-report its own upstream portion to win an advisory tie-break. ROUTE_COST has no such protection: it is a single value the sender chooses, so a relay can advertise 0 for content it is not carrying and attract subscriptions it then has to fetch. Both cost only a suboptimal path choice, and the latter is self-limiting, since the traffic won this way must then be served.
Implementations SHOULD bound the work started by requests beneath a broad advertisement, using NO_CAPACITY when capacity is exhausted.
A receiver MUST NOT make security decisions based on Hop IDs, and a deployment spanning a trust boundary SHOULD treat a peer's ROUTE_COST as a hint to clamp or ignore rather than an accounting figure.
IANA Considerations
This document requests the following registrations. High, distinctive values are requested to avoid the low ranges reserved by draft-ietf-moq-transport and to minimize collisions with provisional registrations by other extensions.
MOQT Setup Options
This document requests two registrations in the "MOQT Setup Options" registry (draft-ietf-moq-transport Section 15.4), whose policy is Specification Required.
| Value | Name | Reference |
|---|---|---|
| 0x40B54 | HOP_ID | This Document |
| 0x40B56 | RELAY_COST | This Document |
MOQT Message Parameters
This document requests two registrations in the "MOQT Message Parameters" registry (draft-ietf-moq-transport Section 15.7). Both are carried in PUBLISH_NAMESPACE, in REQUEST_UPDATE of a PUBLISH_NAMESPACE (Updating an Advertisement), and in the extended NAMESPACE message (Namespace Advertisements).
| Value | Name | Carried In | Reference |
|---|---|---|---|
| 0x40B57 | HOP_PATH | PUBLISH_NAMESPACE, REQUEST_UPDATE, NAMESPACE | This Document |
| 0x40B58 | ROUTE_COST | PUBLISH_NAMESPACE, REQUEST_UPDATE, NAMESPACE | This Document |
The Key-Value-Pair parity is load-bearing: HOP_PATH is odd, so its value is a length-prefixed byte string, while HOP_ID, RELAY_COST, and ROUTE_COST are even, so their values are bare varints.
MOQT Error Codes
This document requests one registration in the "REQUEST_ERROR Codes" registry.
| Value | Name | Reference |
|---|---|---|
| 0x40B5A | NO_CAPACITY | This Document |
Normative References
Appendix A: Changelog
moq-cluster-02
- Defined request resolution against the longest covering prefix and the NO_CAPACITY refusal with its single re-resolution; any other refusal is terminal, including between several publishers of one namespace.
- A relay does not advertise a namespace because it resolved it; the publisher advertises the concrete namespace once producing.
- A change of original publisher is an ordinary update instead of a withdrawal and a new advertisement. Subscriptions already served drain the old source; new requests take the updated route.
- A relay puts a random Hop ID, picked per session, in front of an advertisement whose first HOP_PATH entry is 0, and writes that stamp followed by 0 for one with no HOP_PATH.
moq-cluster-01
- Assigned identities are local selection state and MUST NOT be forwarded.
- Bridging an upstream that sent no HOP_PATH writes 0 for that hop; a received 0 is forwarded unchanged.
- Path selection prefers a HOP_PATH with no 0 entry before comparing ROUTE_COST.
- Renamed the RELAY_HOPS Setup Option to HOP_ID and moved it to the even key 0x40B54, so its value is a bare varint rather than a length-prefixed one.
- A PUBLISH_NAMESPACE is updated with REQUEST_UPDATE on its request stream instead of a repeated PUBLISH_NAMESPACE; HOP_PATH and ROUTE_COST are registered for REQUEST_UPDATE. A NAMESPACE is still re-sent on its stream.
- A session advertises a namespace at most once and a subscription is served from one source at a time. A receiver chooses among several publishers of one namespace; moving between them is a discontinuity unless they share a Hop ID.
- Named the routing protocols whose single per-direction metric RELAY_COST follows.
- Path selection consults the most specific advertisement first, the longest prefix; an advertisement is always a prefix, and a request beneath it that the advertiser will not serve is refused. Standby seeds are bounded by deployment limits.