## An ffmpeg and SDL TutorialPage 1 2 3 4 5 6 7 8 End Prev Home NextPrintable version Text version## Tutorial 05: Synching VideoCode: tutorial05.c### How Video SyncsSo this whole time, we've had an essentially useless movie player. It playsthe video, yeah, and it plays the audio, yeah, but it's not quite yet what wewould call a _movie_. So what do we do?#### PTS and DTSFortunately, both the audio and video streams have the information about howfast and when you are supposed to play them inside of them. Audio streams havea sample rate, and the video streams have a frames per second value. However,if we simply synced the video by just counting frames and multiplying by framerate, there is a chance that it will go out of sync with the audio. Instead,packets from the stream might have what is called a **decoding time stamp(DTS)** and a **presentation time stamp (PTS)**. To understand these twovalues, you need to know about the way movies are stored. Some formats, likeMPEG, use what they call "B" frames (B stands for "bidirectional雙向的雙通道的"). The twoother kinds of frames are called "I" frames and "P" frames ("I" for "intra在內的"and "P" for "predicted預言預告"). I frames contain a full image. P frames depend uponprevious I and P frames and are like diffs or deltas. B frames are the same asP frames, but depend upon information found in frames that are displayed both_before_ and _after_ them! This explains why we might not have a finishedframe after we call avcodec_decode_video.So let's say we had a movie, and the frames were displayed like: I B B P. Now,we need to know the information in P before we can display either B frame.Because of this, the frames might be stored like this: I P B B. This is why wehave a decoding timestamp and a presentation timestamp on each frame. Thedecoding timestamp tells us when we need to decode something, and thepresentation time stamp tells us when we need to display something. So, inthis case, our stream might look like this:
       PTS: 1 4 2 3
       DTS: 1 2 3 4
    Stream: I P B B
Generally the PTS and DTS will only differ when the stream we are playing hasB frames in it.When we get a packet from av_read_frame(), it will contain the PTS and DTSvalues for the information inside that packet. But what we really want is thePTS of our newly decoded raw frame, so we know when to display it. However,the frame we get from avcodec_decode_video() gives us an AVFrame, whichdoesn't contain a useful PTS value. (Warning: AVFrame does contain a ptsvariable, but this will not always contain what we want when we get a frame.)However, ffmpeg reorders the packets so that the DTS of the packet beingprocessed by avcodec_decode_video() will _always be the same_ as the PTS ofthe frame it returns. But, another warning: we won't always get thisinformation, either.Not to worry, because there's another way to find out the PTS of a frame, andwe can have our program reorder the packets by itself. We save the PTS of thefirst packet of a frame: this will be the PTS of the finished frame. So whenthe stream doesn't give us a DTS, we just use this saved PTS. We can figureout which packet is the first packet of a frame by lettingavcodec_decode_video() tell us. How? Whenever a packet starts a frame, theavcodec_decode_video() will call a function to allocate a buffer in our frame.And of course, ffmpeg allows us to redefine what that allocation function is.So we'll make a new function that saves the pts of the packet.Of course, even then, we might not get a proper pts. We'll deal with thatlater.#### SynchingNow, while it's all well and good to know when we're supposed to show aparticular video frame, but how do we actually do so? Here's the idea: afterwe show a frame, we figure out when the next frame should be shown. Then wesimply set a new timeout to refresh the video again after that amount of time.As you might expect, we check the value of the PTS of the next frame againstthe system clock to see how long our timeout should be. This approach works,but there are two issues that need to be dealt with.First is the issue of knowing when the next PTS will be. Now, you might thinkthat we can just add the video rate to the current PTS -- and you'd be mostlyright. However, some kinds of video call for frames to be repeated. This meansthat we're supposed to repeat the current frame a certain number of times.This could cause the program to display the next frame too soon. So we need toaccount for that.The second issue is that as the program stands now, the video and the audiochugging away happily, not bothering to sync at all. We wouldn't have to worryabout that if everything worked perfectly. But your computer isn't perfect,and a lot of video files aren't, either. So we have three choices: sync theaudio to the video, sync the video to the audio, or sync both to an externalclock (like your computer). For now, we're going to sync the video to theaudio.### Coding it: getting the frame PTSNow let's get into the code to do all this. We're going to need to add somemore members to our big struct, but we'll do this as we need to. First let'slook at our video thread. Remember, this is where we pick up the packets thatwere put on the queue by our decode thread. What we need to do in this part ofthe code is get the PTS of the frame given to us by avcodec_decode_video. Thefirst way we talked about was getting the DTS of the last packet processed,which is pretty easy:
      double pts;

      for(;;) {
        if(packet_queue_get(&is->videoq, packet, 1) < 0) {
          // means we quit getting packets
          break;
        }
        pts = 0;
        // Decode video frame
        len1 = avcodec_decode_video(is->video_st->codec,
                                    pFrame, &frameFinished,
    				packet->data, packet->size);
        if(packet->dts != AV_NOPTS_VALUE) {
          pts = packet->dts;
        } else {
          pts = 0;
        }
        pts *= av_q2d(is->video_st->time_base);
We set the PTS to 0 if we can't figure out what it is.Well, that was easy. But we said before that if the packet's DTS doesn't helpus, we need to use the PTS of the first packet that was decoded for the frame.We do this by telling ffmpeg to use our custom functions to allocate a frame.Here's the types of the functions:
    int get_buffer(struct AVCodecContext *c, AVFrame *pic);
    void release_buffer(struct AVCodecContext *c, AVFrame *pic);
The get function doesn't tell us anything about packets, so what we'll do issave the PTS to a global variable each time we get a packet, and our getfunction can just read that. Then, we can store the value in the AVFramestructure's opaque variable. This is a user-defined variable, so we can use itfor whatever we want. So first, here are our functions:
    uint64_t global_video_pkt_pts = AV_NOPTS_VALUE;

    /* These are called whenever we allocate a frame
     * buffer. We use this to store the global_pts in
     * a frame at the time it is allocated.
     */
    int our_get_buffer(struct AVCodecContext *c, AVFrame *pic) {
      int ret = avcodec_default_get_buffer(c, pic);
      uint64_t *pts = av_malloc(sizeof(uint64_t));
      *pts = global_video_pkt_pts;
      pic->opaque = pts;
      return ret;
    }
    void our_release_buffer(struct AVCodecContext *c, AVFrame *pic) {
      if(pic) av_freep(&pic->opaque);
      avcodec_default_release_buffer(c, pic);
    }
avcodec_default_get_buffer and avcodec_default_release_buffer are the defaultfunctions that ffmpeg uses for allocation. av_freep is a memory managementfunction that not only frees the memory pointed to, but sets the pointer toNULL.Now going way down to our stream open function (stream_component_open), we addthese lines to tell ffmpeg what to do:
        codecCtx->get_buffer = our_get_buffer;
        codecCtx->release_buffer = our_release_buffer;
Now we have to add code to save the PTS to the global variable and then usethe stored PTS when necessary. Our code now looks like this:
      for(;;) {
        if(packet_queue_get(&is->videoq, packet, 1) < 0) {
          // means we quit getting packets
          break;
        }
        pts = 0;

        // Save global pts to be stored in pFrame in first call
        global_video_pkt_pts = packet->pts;
        // Decode video frame
        len1 = avcodec_decode_video(is->video_st->codec, pFrame,&frameFinished,
    				packet->data, packet->size);
        if(packet->dts == AV_NOPTS_VALUE
           && pFrame->opaque && *(uint64_t*)pFrame->opaque != AV_NOPTS_VALUE){
          pts = *(uint64_t *)pFrame->opaque;
        } else if(packet->dts != AV_NOPTS_VALUE) {
          pts = packet->dts;
        } else {
          pts = 0;
        }
        pts *= av_q2d(is->video_st->time_base);
A technical note: You may have noticed we're using int64 for the PTS. This isbecause the PTS is stored as an integer. This value is a timestamp thatcorresponds to a measurement of time in that stream's time_base unit. Forexample, if a stream has 24 frames per second, a PTS of 42 is going toindicate that the frame should go where the 42nd frame would be if there wehad a frame every 1/24 of a second (certainly not necessarily true).We can convert this value to seconds by dividing by the framerate. Thetime_base value of the stream is going to be 1/framerate (for fixed-fpscontent), so to get the PTS in seconds, we multiply by the time_base.### Coding: Synching and using the PTSSo now we've got our PTS all set. Now we've got to take care of the twosynchronization problems we talked about above. We're going to define afunction called synchronize_video that will update the PTS to be in sync witheverything. This function will also finally deal with cases where we don't geta PTS value for our frame. At the same time we need to keep track of when thenext frame is expected so we can set our refresh rate properly. We canaccomplish this by using an internal video_clock value which keeps track ofhow much time has passed according to the video. We add this value to our bigstruct.
    typedef struct VideoState {
      double          video_clock; ///Here's the synchronize_video function, which is pretty self-explanatory:
    double synchronize_video(VideoState *is, AVFrame *src_frame, double pts){

      double frame_delay;

      if(pts != 0) {
        /* if we have pts, set video clock to it */
        is->video_clock = pts;