RootEncoder: Adaptive bitrate doesn't work

I took a deep dive into the adaptive bitrate in this library (latest version, 2.1.4), and I’ve been able to fix most of the adaptive bitrate problems. Here’s what I found:

Problems with default BitrateAdapter

• Average was not properly calculated
• Bandwidth in the last second (onNewBitrateRtmp) is calculated as the total data size send to the socket, with packet headers and audio packets. Yet bitrate is set for video encoder only (audio bitrate set separately).
• If there is congestion, need to immediately reduce bitrate, to minimize drop frames and even worse congestion
• Keep the cache size (queue length) to the minimum, so congestion can be quickly discovered (it’s 20% of the cache size), and so the backlog of old frames is easier to push out. I suggest 30.
• It is necessary to account for what was in the queue for bandwidth calculation before raising bitrate, because of the bitrate adjustments. Wait for a few cycles after lowering to raise again.

Notice: All of this makes sense with Constant Bitrate only! Here’s my version of BitrateAdapter that accounts for audio bitrate, packet overhead, and reacts faster to congestion.

package pro.eventlive;

import android.util.Log;

public class BitrateAdapter {

  public interface Listener {
    void onBitrateAdapted(int bitrate);
  }

  private int maxBitrate;
  private int minBitrate = 100 * 1024;
  private int audioBitrate = 96 * 1000; // TODO: This should be passed into the class
  private int oldBitrate;
  private int averageBitrate;
  private int cont;
  private int cyclesSinceReduced = 0;
  private Listener listener;
  private float decreaseRange = 0.8f; //20%
  private float increaseRange = 1.2f; //20%

  private int decreaseBy = 256 * 1024; 
  private int increaseBy = 128 * 1024; 
  private int packetOverhead = 15 * 1024; // Magic number

  public BitrateAdapter(Listener listener) {
    this.listener = listener;
    reset();
  }

  public void setMaxBitrate(int bitrate) {
    this.maxBitrate = bitrate;
    this.oldBitrate = bitrate;
    reset();
  }

  public void adaptBitrate(long actualBitrate) {
    averageBitrate += actualBitrate;
    if (cont > 0) {
        averageBitrate /= 2;
    }
    cont++;
    if (cont >= 3) { // lowered the measurement interval from 5s to 3s
      if (listener != null && maxBitrate != 0) {
        listener.onBitrateAdapted(getBitrateAdapted(averageBitrate));
        reset();
      }
    }
  }

  /**
   * Adapt bitrate on fly based on queue state.
   */
  public void adaptBitrate(long actualBitrate, boolean hasCongestion) {
    
    averageBitrate += actualBitrate;
    if (cont > 0) {
        averageBitrate /= 2;
    }
    cont++;

    if (hasCongestion) {
        // Immediately react with adjustments
        listener.onBitrateAdapted(getBitrateAdapted(averageBitrate, hasCongestion));
        reset();
    }

    if (cont >= 3) { // lowered the measurement interval from 5s to 3s
      if (listener != null && maxBitrate != 0) {
        cyclesSinceReduced++;
        listener.onBitrateAdapted(getBitrateAdapted(averageBitrate, hasCongestion));
        reset();
      }
    }
  }

  // Version that doesn't take the queue size into account
  private int getBitrateAdapted(int averageBw) { 
    if (averageBw >= maxBitrate) { //You have high speed and max bitrate. Keep max speed
      oldBitrate = maxBitrate;
    } else if (averageBw <= oldBitrate * 0.9f) { //You have low speed and bitrate too high. Reduce bitrate
      oldBitrate = Math.max(averageBw - audioBitrate - decreaseBy, minBitrate);
    } else if (averageBw >= oldBitrate) { //You have high speed and bitrate too low. Increase bitrate 
      oldBitrate = Math.min(oldBitrate + increaseBy, maxBitrate);
    }
    // keep it otherwise
    return oldBitrate;
  }

  private int getBitrateAdapted(int averageBw, boolean hasCongestion) { 
    // what we expect should have been sent over the network
    int expectedBandwidth = oldBitrate + audioBitrate + packetOverhead;
    // Explicitly has congestion in the queue or average bandwidth 90% less than expected
    if (hasCongestion || (averageBw <= expectedBandwidth * 0.9f)) {
        // Reduce! decreaseBy is added for when there's congestion but bitrate - audioBitrate is within current bitrate
        oldBitrate = Math.max(averageBw - audioBitrate - decreaseBy, minBitrate);
        cyclesSinceReduced = 0; // wait a few cycles to let the higher bitrate frames in the queue pass through
    } else if (averageBw >= expectedBandwidth&& cyclesSinceReduced >= 3) { 
        // When fully recovered, attempt to increase by a bit
        oldBitrate = Math.min(oldBitrate + increaseBy, maxBitrate);
    }

    // keep it otherwise
    return oldBitrate;
  }

  public void reset() {
    averageBitrate = 0;
    cont = 0;
  }

  public float getDecreaseRange() {
    return decreaseRange;
  }

  /**
   * @param decreaseRange in percent. How many bitrate will be reduced based on oldBitrate.
   * valid values:
   * 0 to 100 not included
   */
  public void setDecreaseRange(float decreaseRange) {
    if (decreaseRange > 0f && decreaseRange < 100f) {
      this.decreaseRange = 1f - (decreaseRange / 100f);
    }
  }

  public float getIncreaseRange() {
    return increaseRange;
  }

  /**
   * @param increaseRange in percent. How many bitrate will be increment based on oldBitrate.
   * valid values:
   * 0 to 100
   */
  public void setIncreaseRange(float increaseRange) {
    if (increaseRange > 0f && increaseRange < 100f) {
      this.increaseRange = 1f + (increaseRange / 100f);
    }
  }
}

Suggestions to improve further:

• If the queue is congested and not sending the frames, then calculateBitrate will not be invoked and onNewBitrateRtmp will not be called and bitrate would not be adjusted. Instead, I suggest using a timer or callback for congestion.
• When the queue is full we should not discard audio packets because they’re small compared to video. Ideally, we should not discard Keyframes because without them the picture falls apart. How to do it? Two separate queues? Or queue clean up? It is better to miss some video frames than audio - because you will hear pauses in the audio but may not notice low frame rate.

@marcin-adamczewski Actually I end up not relying on upload speed calculation at all in adaptive bitrate. What we do now is - if there is congestion (i.e. queue is 15% full) then we immediately drop the target bitrate by set amount, then re-evaluate every 2s, if there’s no congestion raise by lower amount. Basically adding and subtracting to the same target, instead of manipulating the value received from upload speed calculation.

Yes, I will do it soon. I think this monday