How Mozilla Hubs Uses WebRTC

Producer Example Usage - Client Context

From hubs/naf-dialog-adapter.js > DialogAdapter > setLocalMediaStream():

async setLocalMediaStream(stream) {
    if (!this._sendTransport) {
        console.error("...");
        return;
    }
    ...
    await Promise.all(stream.getTracks().map(async track => {
        if (track.kind === "audio") {
            ...
            if (this._micProducer) {
                ...
            } else {
                this._micProducer = await this._sendTransport.produce({
                    track,
                    ...
                });
                ...
            }
        }
    }))
}

Translation: setLocalMediaStream() is called by the client in a few cases, including when the client first joins a Dialog room, and when the client's audio input device changes. The stream variable passed to the function is a MediaStream object, such as that returned by getUserMedia()

First, we must make sure we've already created a "Send Transport" locally. This Transport is created when initially joining a Dialog Room.

await Promise.all(...) means "fulfill this one Promise when all of the passed Promises have fulfilled, or reject this one Promise when any one of the passed Promises has rejected."

Next, we split the input stream into its constituent tracks, which are of type MediaStreamTrack. In most cases, the MediaStream associated with a person's audio input device consists of just one track.

If the "kind" of track we're processing is an "audio" track (as opposed to a "video" track), and we've previously created a Mediasoup Producer, we execute a special case of code.

However, if this is an "audio" track and this is the first time we're producing, we have to create a new Producer by calling this._sendTransport.produce(). Calling this function tells the Transport to begin sending the specified audio track to the remote Mediasoup Router as per the specified options.

We can later pause, resume, and close the resultant Producer.

Critically, none of the Mediasoup-related code above has anything to do with signaling (which is handled by Protoo). This signaling component is necessary for getting the media stream from Peer A to all other clients connected to a Dialog Room. Let's quickly follow through the client side of that signaling piece here:

From hubs/naf-dialog-adapter.js > DialogAdapter > createSendTransport():

this._sendTransport.on("produce", async ({ kind, rtpParameters, appData }, callback, errback) => {
    ...
    try {
        const { id } = await this._protoo.request("produce", {
            transportId: this._sendTransport.id,
            kind,
            rtpParameters,
            appData
        });
        callback({ id });
    } catch (error) {
        ...
        errback(error);
    }
});

This code snippet says: "When the Mediasoup _sendTransport starts producing, let the Protoo signaling server know that this client has started producing." Below, as we discuss the server-side Producer, we'll make more sense of what that means.

Producer Example Usage - Server Context

From dialog/lib/Room.js > Room > _handleProtooRequest():

async _handleProtooRequest(peer, request, accept, reject) {
    const router = this._mediasoupRouters.get(peer.data.routerId);
    switch (request.method) {
    ...
    case 'produce':
    ...
    const transport = peer.data.transports.get(transportId);
    if (!transport) throw new Error('...');
    ...
    const producer = await transport.produce(
        {
            kind,
            ...
        });
    for (const [ routerId, targetRouter ] of this._mediasoupRouters)
    {
        if (routerId === peer.data.routerId) {
            continue;
        }
        await router.pipeToRouter({
            producerId : producer.id,
            router     : targetRouter
        });
    }
    peer.data.producers.set(producer.id, producer);
    ...
    for (const otherPeer of this._getJoinedPeers({ excludePeer: peer }))
    {
        this._createConsumer(
            {
                consumerPeer : otherPeer,
                producerPeer : peer,
                producer
            });
    }
        break;
    }
}

Translation: _handleProtooRequest() is called on the Dialog server when a Protoo signaling request is received. Dialog will always formulate and send a response to Protoo requests.

When the Dialog server receives a Protoo signaling request with method "produce" - as will occur when a Peer begins producing media from the client - we'll run the code above within the switch statement.

First, the Dialog server must obtain the "Send Transport" it previously created and stored associated with this Peer. This Transport was created at the client's signaling request when the client joined the Dialog room. If we don't find that Send Transport, we fail and return early.

We then create a new Producer on the server associated with that Send Transport.

In the current iteration of Hubs and Dialog, there's only ever a single element in the this._mediasoupRouters Map. So, we will never pipe any data between Routers; we'll hit that continue statement.

Dialog then keeps track of the Producer objects associated with a given Protoo Peer's ID in a Map. This is necessary for pausing, resuming, and closing that Producer later.

We then create Dialog Consumers on the server associated with every other Peer who is marked as having joined the Dialog room. We'll learn about Consumers below.

Consumer Example Usage - Client Context

From hubs/naf-dialog-adapter.js > DialogAdapter > connect() > _protoo.on("request") callback:

this._protoo.on("request", async (request, accept, reject) => {
    switch (request.method) {
        case "newConsumer":
            const { peerId, producerId, id, kind, rtpParameters, appData, ... } =
            request.data;

            try {
                const consumer = await this._recvTransport.consume({
                    id,
                    producerId,
                    kind,
                    rtpParameters,
                    appData: { ...appData, peerId }
                });

                this._consumers.set(consumer.id, consumer);
                ...
                accept();

                this.resolvePendingMediaRequestForTrack(peerId, consumer.track);

                this.emit("stream_updated", peerId, kind);
            } catch (err) {
                error('...');
                throw err;
            }
            break;
    }
}

Translation: We call _createConsumer() in a couple of cases, primarily when the Dialog server receives a Protoo signaling request with method "produce". This happens when a Peer begins newly producing, as in the Producer code we walked through above.

First, the Dialog server must obtain the "Receive Transport" it previously created and stored associated with this Peer. This Transport was created at the client's signaling request when the client joined the Dialog room. If we don't find that Receive Transport, we fail and return early.

Next, we initialize a paused Consumer on top of that Transport, given the RTP capabilities of the Consumer. "RTP" stands for "Real-time Transport Protocol", and "RTP capabilities" refer to the media codecs and types that a consumer can consume. They are set by the remote Peer when joining the Dialog Room.

We then store data about that new Consumer in two Maps, which are used in cases such as pausing/resuming the Consumer, or blocking other Peers.

We then try to signal to the relevant Protoo Peer via a "newConsumer" request that the Peer needs to begin consuming some media. In this context, that Peer is a Hubs client.

After we get a response to that "newConsumer" request, we resume this consumer. That'll trigger the remote endpoint to begin receiving the first RTP media packet.

Consumer Example Usage - Server Context

From dialog/lib/Room.js > Room > _createConsumer():

async _createConsumer({ consumerPeer, producerPeer, producer }) {
    const transport = Array.from(consumerPeer.data.transports.values()).find((t) => t.appData.consuming);
    if (!transport) { return; }

    let consumer;
    try {
        consumer = await transport.consume(
        {
            producerId      : producer.id,
            rtpCapabilities : consumerPeer.data.rtpCapabilities,
            paused          : true
        });
    } catch (error) {
        return;
    }

    consumerPeer.data.consumers.set(consumer.id, consumer);
    consumerPeer.data.peerIdToConsumerId.set(producerPeer.id, consumer.id);
    ...
    // Setup Consumer event handlers...
    ...
    try {
    await consumerPeer.request(
    'newConsumer',
    {
        peerId         : producerPeer.id,
        producerId     : producer.id,
        id             : consumer.id,
        kind           : consumer.kind,
        rtpParameters  : consumer.rtpParameters,
        type           : consumer.type,
        appData        : producer.appData,
        producerPaused : consumer.producerPaused
    });

    await consumer.resume();
    ...
    } catch (error) {
        return;
    }
}

Translation: We call _createConsumer() in a couple of cases, primarily when the Dialog server receives a Protoo signaling request with method "produce". This happens when a Peer begins newly producing, as in the Producer code we walked through above.

First, the Dialog server must obtain the "Receive Transport" it previously created and stored associated with this Peer. This Transport was created at the client's signaling request when the client joined the Dialog room. If we don't find that Receive Transport, we fail and return early.

Next, we initialize a paused Consumer on top of that Transport, given the RTP capabilities of the Consumer. "RTP" stands for "Real-time Transport Protocol", and "RTP capabilities" refer to the media codecs and types that a consumer can consume. They are set by the remote Peer when joining the Dialog Room.

We then store data about that new Consumer in two Maps, which are used in cases such as pausing/resuming the Consumer, or blocking other Peers.

We then try to signal to the relevant Protoo Peer via a "newConsumer" request that the Peer needs to begin consuming some media. In this context, that Peer is a Hubs client.

After we get a response to that "newConsumer" request, we resume this consumer. That'll trigger the remote endpoint to begin receiving the first RTP media packet.