VideoProcessingFramework: Unexpected seeking behavior at the beginning of a video

Describe the bug Seeking seems to start at the wrong frame for just the beginning of the video. Maybe related to the negative dts? Probably not a huge deal operationally, but thought I would add this observation anyway.

To Reproduce Run SampleDecode.py with self.seek_mode = nvc.SeekMode.EXACT_FRAME and verbose logs on. Using the STANDALONE decoder as configured by default. Test video : https://github.com/NVIDIA/DALI_extra/blob/main/db/video/test_label.mp4

Results with varying sk_frm and printing only the first 5 for brevity.

self.sk_frm = -1: (I thought this was deprecated from https://github.com/NVIDIA/VideoProcessingFramework/issues/178#issue-839705819, but it still works.)

Decoding on GPU 0
frame pts (decode order)      : 0
frame dts (decode order)      : -1024
frame pos (decode order)      : 48
frame duration (decode order) : 512

frame pts (decode order)      : 2048
frame dts (decode order)      : -512
frame pos (decode order)      : 1669
frame duration (decode order) : 512

frame pts (decode order)      : 1024
frame dts (decode order)      : 0
frame pos (decode order)      : 1954
frame duration (decode order) : 512

frame pts (decode order)      : 512
frame dts (decode order)      : 512
frame pos (decode order)      : 2121
frame duration (decode order) : 512

frame pts (decode order)      : 1536
frame dts (decode order)      : 1024
frame pos (decode order)      : 2278
frame duration (decode order) : 512

self.sk_frm = 0: Shouldn’t this start at pts=0?

Decoding on GPU 0
Seeking for the frame  0
We are at frame with pts 1024
frame pts (decode order)      : 1024
frame dts (decode order)      : 0
frame pos (decode order)      : 1954
frame duration (decode order) : 512

frame pts (decode order)      : 512
frame dts (decode order)      : 512
frame pos (decode order)      : 2121
frame duration (decode order) : 512

frame pts (decode order)      : 1536
frame dts (decode order)      : 1024
frame pos (decode order)      : 2278
frame duration (decode order) : 512

frame pts (decode order)      : 4096
frame dts (decode order)      : 1536
frame pos (decode order)      : 2456
frame duration (decode order) : 512

frame pts (decode order)      : 3072
frame dts (decode order)      : 2048
frame pos (decode order)      : 2746
frame duration (decode order) : 512

self.sk_frm = 1: This starts before sk_frm=0?

Decoding on GPU 0
Seeking for the frame  1
We are at frame with pts 512
frame pts (decode order)      : 512
frame dts (decode order)      : 512
frame pos (decode order)      : 2121
frame duration (decode order) : 512

frame pts (decode order)      : 1536
frame dts (decode order)      : 1024
frame pos (decode order)      : 2278
frame duration (decode order) : 512

frame pts (decode order)      : 4096
frame dts (decode order)      : 1536
frame pos (decode order)      : 2456
frame duration (decode order) : 512

frame pts (decode order)      : 3072
frame dts (decode order)      : 2048
frame pos (decode order)      : 2746
frame duration (decode order) : 512

frame pts (decode order)      : 2560
frame dts (decode order)      : 2560
frame pos (decode order)      : 2961
frame duration (decode order) : 512

self.sk_frm = 2:

Decoding on GPU 0
Seeking for the frame  2
We are at frame with pts 1536
frame pts (decode order)      : 1536
frame dts (decode order)      : 1024
frame pos (decode order)      : 2278
frame duration (decode order) : 512

frame pts (decode order)      : 4096
frame dts (decode order)      : 1536
frame pos (decode order)      : 2456
frame duration (decode order) : 512

frame pts (decode order)      : 3072
frame dts (decode order)      : 2048
frame pos (decode order)      : 2746
frame duration (decode order) : 512

frame pts (decode order)      : 2560
frame dts (decode order)      : 2560
frame pos (decode order)      : 2961
frame duration (decode order) : 512

frame pts (decode order)      : 3584
frame dts (decode order)      : 3072
frame pos (decode order)      : 3124
frame duration (decode order) : 512

Decoding on GPU 0
Seeking for the frame  5
We are at frame with pts 2560
frame pts (decode order)      : 2560
frame dts (decode order)      : 2560
frame pos (decode order)      : 2961
frame duration (decode order) : 512

frame pts (decode order)      : 3584
frame dts (decode order)      : 3072
frame pos (decode order)      : 3124
frame duration (decode order) : 512

frame pts (decode order)      : 6144
frame dts (decode order)      : 3584
frame pos (decode order)      : 3274
frame duration (decode order) : 512

frame pts (decode order)      : 5120
frame dts (decode order)      : 4096
frame pos (decode order)      : 3639
frame duration (decode order) : 512

frame pts (decode order)      : 4608
frame dts (decode order)      : 4608
frame pos (decode order)      : 3816
frame duration (decode order) : 512

Decoding on GPU 0
Seeking for the frame  10
We are at frame with pts 5632
frame pts (decode order)      : 5632
frame dts (decode order)      : 5120
frame pos (decode order)      : 3961
frame duration (decode order) : 512

frame pts (decode order)      : 8192
frame dts (decode order)      : 5632
frame pos (decode order)      : 4121
frame duration (decode order) : 512

frame pts (decode order)      : 7168
frame dts (decode order)      : 6144
frame pos (decode order)      : 4411
frame duration (decode order) : 512

frame pts (decode order)      : 6656
frame dts (decode order)      : 6656
frame pos (decode order)      : 4615
frame duration (decode order) : 512

frame pts (decode order)      : 7680
frame dts (decode order)      : 7168
frame pos (decode order)      : 4762
frame duration (decode order) : 512

I checked a few more after sk_frm=10, and the seeked frame aligns with pts as expected.

Desktop (please complete the following information):

OS: Ubuntu 18.04
Nvidia driver version 440.33.01
CUDA Version 10.2
Video Codec SDK Version Video_Codec_SDK_10.0.26
Python Version 3.6.9

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (9 by maintainers)

Commits related to this issue

Issue #218: checking if seek frame number is negative — committed to NVIDIA/VideoProcessingFramework by rarzumanyan 3 years ago
Issue #218: remove decoder flush to prevent SPS & PPS preemtion — committed to NVIDIA/VideoProcessingFramework by rarzumanyan 3 years ago

Most upvoted comments

@SiftingSands

One more update: Compressed frames between 461 and 512 are demuxed from container and sent to decoder although HandlePictureDecode() and HandlePictureDisplay() parser callbacks are never called on them. So I assume it’s Annex.B parser bug which makes it skip all the frames between 461 and 512. Probably parser expects frame 461 to be IDR frame which it’s not (it has sclie header type of I and not I, IDR).

I’ll inform our Video Codec SDK team about this parser behavior. As soon as I get word back from them I’ll update you in this thread.

rarzumanyan on Jun 9, 2021

Hi @SiftingSands More updates on this case:

Demuxer actually seeks for the frame 461 as you ask it to seek. The issue lies on decoder side - it struggles to return decoded frames between recovery point (frame 461) and next key frame (512). I’ve modified the code slightly to see how many frames are kicked off to HW before we get a frame ready for display:

Surface *PyNvDecoder::getDecodedSurface(NvdecDecodeFrame *decoder,
                                        DemuxFrame *demuxer,
                                        SeekContext &seek_ctx,
                                        bool needSEI) {
  Surface *surface = nullptr;
  auto num_runs = 0U;
  do {
    auto elementaryVideo = getElementaryVideo(demuxer, seek_ctx, needSEI);
    auto pktData = (Buffer *)demuxer->GetOutput(3U);

    decoder->SetInput(elementaryVideo, 0U);
    decoder->SetInput(pktData, 1U);
    if (TASK_EXEC_FAIL == decoder->Execute()) {
      break;
    }

    surface = (Surface *)decoder->GetOutput(0U);
    num_runs++;
  } while (!surface);

  cout << num_runs
       << " frames kicked off for decoding before we got a frame ready for "
          "display."
       << endl;
  return surface;
};

As I seek for frame 461 this is the output I get:

Decoding on GPU 0
Seeking for the frame  461
55 frames kicked off for decoding before we got a frame ready for display.

Decoder can’t get frames between 461 and 512, then it faces a usual delay of 4 frames and returns frame 512. I assume there are some specific actions needed to be taken to maintain recovery point SEI on Nvdec side. I’ll update you later as I find something on this topic.

rarzumanyan on Jun 8, 2021