jitsi-videobridge: Colibri Message ReceiverVideoConstraintsChangedEvent is considered unexpected

Description

We are trying to make use of the ReceiverVideoConstraintsChangedEvent message, so that our client code (using lib-jitsi-meet) can specify the resolution constraint for each of the remote participants by ID. Currently lib-jitsi-meet does not support sending this message, but we are adding that code with the goal of creating a PR in lib-jitsi-meet, following the ideas of https://github.com/jitsi/lib-jitsi-meet/issues/1175#issuecomment-634765227.

Current behavior

When we send a message like this one:

{
     colibriClass: 'ReceiverVideoConstraintsChangedEvent',
     videoConstraints: [
         { "id": "ccb8ecc5", "idealHeight": 720 },
         { "id": "022d9fa9", "idealHeight": 360 },
         { "id": "b42af35f", "idealHeight": 360 }, 
         { "id": "28ab2220", "idealHeight": 180 }
     ]
}

We get this message in the videobridge log:

WARNING: Received a message with an unexpected type: ReceiverVideoConstraintsChangedEvent

Expected Behavior

We would like if the message could be honored so that the client defines the quality to receive for each remote endpoint.

Possible Solution

Possibly this could be solved by adding a new method in EndpointMessageTransport class, which would handle this message.

Steps to reproduce

Using the code of this PR jitsi/lib-jitsi-meet#1324, send the message ReceiverVideoConstraintsChangedEvent as specified in the current behavior sample. This can be done through the new public method JitsiConference.setReceiverVideoConstraints added in the PR.

Environment details

Self-hosted deployment, using jitsi-videobridge version 2.1-304-g8488f77d-1, custom web application using lib-jitsi-meet with the mentioned PR code.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 22 (6 by maintainers)

Most upvoted comments

Thanks very much for the quick reply @bgrozev and for explaining the plans, we really appreciate it.

Regarding the LastN issue, here is the description: We have a custom UI with lib-jitsi-meet. In this UI we have a stage where we can have from 1 to 4 participants, and we also have a collapsible panel to view all participants in low resolution. The stage is shared between all participants, meaning they all see the same endpoints there. When a participant closes the panel and has only stage, we use LastN and selectParticipants for that client to save bandwidth and only get the video feeds that are on stage.

We achieve that by doing these two calls in lib-jitsi-meet:

conference.setLastN(numberOfParticipantsOnStage);
conference.selectParticipants(arrayOfStageParticipantIds);

This was working well for us. Now after this update (we went from Jitsi Videobridge 2.1-273-g072dd44b to 2.1-376-g9f12bfe2) the problem we have is as follows: Let’s say that we have participants with Endpoint IDs (real example): 49d41a92, 7222785b, b13094d2, 44f76001. Now 49d41a92 and 7222785b are sent to the stage. b13094d2 client is watching only stage so he calls setLastN with 2, and selectParticipants with IDs 49d41a92,7222785b (as shown in these client logs):

2020-10-21T20:04:11.264Z [modules/RTC/BridgeChannel.js] <l.sendSelectedEndpointsMessage>:  Sending selected endpoints: 49d41a92,7222785b.
Logger.js:154 2020-10-21T20:04:11.265Z [modules/RTC/BridgeChannel.js] <l.sendSetLastNMessage>:  Sending lastN=2.

That works well. But now 44f76001 starts talking, and he becomes dominant speaker:

2020-10-21T20:07:37.870Z [modules/RTC/BridgeChannel.js] <WebSocket.e.onmessage [as __zone_symbol__ON_PROPERTYmessage]>:  New dominant speaker: 44f76001.
Logger.js:154 2020-10-21T20:07:37.872Z [modules/RTC/BridgeChannel.js] <WebSocket.e.onmessage [as __zone_symbol__ON_PROPERTYmessage]>:  New forwarded endpoints: 44f76001,49d41a92

From that moment on, b13094d2 sees 7222785b as frozen. As shown in the logs, the Videobridge has now decided to forward 44f76001 and 49d41a92, where 44f76001 has replaced 7222785b in our original participant list. The 7222785b Participant Connection Status changes from active to inactive for b13094d2. This is easy to replicate for us and happens also when there is only one participant on stage. The dominant speaker replaces the forwarded endpoints when using LastN to get an exact amount of videos, even if we have called to select specific endpoints.

We would like to be able to call pinParticipant which might solve this problem but lib-jitsi-meet allows to pin only one endpoint and our stage can have multiple ones.

I hope the explanation of the issue makes sense. If there is any way we can help through PRs or testing, if you point us in the right direction we are available to contribute code and test. We would like to contribute to this amazing project if we could be of any help.

iencotech on Oct 22, 2020

Amazing thanks @bgrozev! I see the new message format, BandwidthAllocationSettings looks great. The documentation is very clear and useful.

@jallamsetty1 If it is of any help, I am open to work in the lib-jitsi-meet part of it, perhaps adjusting the original PR I had done jitsi/lib-jitsi-meet#1324 or starting it from scratch.

Thanks for offering to help @jienco. Like Boris said, the new signaling format is still under discussion. Once we have decided on the format we can start the client side work.

jallamsetty1 on Dec 17, 2020

@bgrozev Regarding this refactor, with the idea of use the new messaging for signaling receiver video constraints per each endpoint, is there any plans to work on it in the short term future? Just for us to know

Yes, I am currently working on it. Progress has been slow because high priority tasks kept coming up, but I hope to have something soon.

bgrozev on Nov 19, 2020