ProTools and H.264 Video
There seems to be plenty of curiosity about video in ProTools, especially the H.264 codec. I’ve previously identified problems using H.264 in ProTools, but let’s take a wider look at how these two interact.
TASTES GREAT / LESS FILLING
If you’ve ever compared video codecs you know H.264 is compact for the quality level it affords. That explains why it’s so popular. The trick is Variable Bit Rate, meaning that each successive frame of picture isn’t fully represented in the data. Most of the time the percentage of stuff that changes frame to frame isn’t high, so the codec just saves the difference. On a static video shot the amount of data needed over time is relatively low. Lots of cuts and motion will require more data. The point is: the amount of memory varies from frame to frame. This can be very efficient in terms of storage, but not in terms of processing overhead. By comparison, Motion JPEG-A is basically the same amount of data for every frame. The files are much larger but it’s easier for your computer to render the image because the data stream is consistent. H.264 requires your computer to work harder, taking more horsepower.
SOUND WITH PICTURE
If that Variable Bit Rate trick sounds familiar it should. AAC (also known as MP4 audio) can vary the bit rate to help keep things smaller. Now take those two codecs and put them together. If the picture is H.264 and the audio is AAC your computer has the increased overhead of decoding two complex streams of data AND it has to keep them in sync! If you have experienced problems with sluggish H.264 video in ProTools, definitely keep away from AAC encoded audio.
WHAT’S A FRAME BETWEEN FRIENDS?
Because the data for a given frame is spread across several other frames, random access of H.264 gets tricky. If you drop somewhere in the ProTools timeline that’s in the middle of an H.264 stream the computer might not have enough data for that frame. So your computer looks back (or maybe forward) to pick up the missing data. That makes it tough for the computer to know exactly which frame to render at any given location. For that reason you may see the wrong frame when you drop the cursor interframe. Let’s say you’re spotting a face slap sound effect. You probably don’t want to be a frame off sync, but that’s exactly what can happen. People who are serious about sync — notably film post and game audio professionals — find H.264 a deal breaker for this reason.
PROTOOLS AND VIDEO
Why is it a video will play fine on it’s own, then die horribly in ProTools? Tough question, but the heart of the issue is how ProTools deals with video.
The ability to play video is built into your OS, not ProTools. So when you play video, ProTools hands off the task. If that task is heavy because you’ve got a big, high resolution video with a variable bit rate, your machine will get busy. When you’re not running ProTools, the video might play fine. But ProTools is a resource hog. If you ask your machine to run ProTools (which tends to tax it) and then ProTools hands off the video playback task (further taxing it), your ProTools may get sluggish. Faster processors, efficient code and tons of memory help video that used to choke machines a few years ago run pretty smoothly today. Moore’s Law means this kind of problem will be less of an issue in the future, but understanding why ProTools bogs down can help you problem solve when you have video trouble.
BEST PRACTICES
By all means, use H.264 to encode and throw those tiny files around to everyone one who needs video. It’s great quality at a low memory price, for quicker up/downloads.
If you need to be frame accurate, convert H.264 to something else before importing into ProTools. I like Motion JPEG-A.
If you use H.264 video and it makes your ProTools rig too slow, first check to see if the audio is AAC. If it is, convert your video or ask for new video with constant bit rate audio.
Slow ProTools and H.264 may simply indicate the limits of your setup. Convert or ask for new video with a smaller frame size, lower quality and/or constrained data rate. In other words, lower the complexity of the stream.
If you’re still having problems with H.264 video after all of that, I say give up. Go ahead and convert to another codec, such as Motion JPEG-A. MPEG Streamclip is free for Mac and Windows, reliable, and can batch convert. (More video conversion options here.)
UPDATE Aug 16, 2012
A video codec based on H.264, called High Efficiency Video Coding (HEVC), is expected from the Motion Pictures Expert Group in 2013.
See also- ProTools + H.264 video = Problem, ProTools: Bounce to Quicktime Movie with Full Options, Conflict: ProTools and Spotlight Indexing, ProTools Sync: The Short Video Problem
Do you have a Pinterest board to collect articles like this one? Cool. 
Randy Coppinger 
Hi Randy,
Thanks for a great article, I think it’s really important to be aware of this since H.264 is so prevalent these days!
There’s just one thing that I believe needs to be slightly corrected in the article – the technique where “each successive frame of picture isn’t fully represented in the data” is inter-frame compression. It is possible to encode a Constant Bit Rate H.264 file, and still potentially run into sync issues due to inter-frame compression. Codecs such as Motion JPEG, on the other hand, use intra-frame compression, which – as you rightly point out – avoids unreliable sync as all of the information for each individual frame is independently encoded.
I understand that variable bit rate encoding of audio (e.g. AAC) can cause audio sync issues, but for video files the primary factor affecting sync accuracy is intra-frame vs inter-frame compression. I hope this information helps to clarify your point.
Best,
James
I think I’m wrapping my head around your point James: the issue isn’t that the bit rate varies, the issue is that data reduction that relies on bits from other frames confuses the exact location of a given frame when it is displayed out of sequence. Correct?
The issue is the data reduction relying on information across several frames, and then the CPU having to compute this during playback, which for one thing can cause lagging video playback.
My understanding is that inter-frame compression generally causes issues in playback and scrubbing, rather than when pausing on a frame, so the issue doesn’t necessarily relate to jumping to a frame out of sequence. Does that make sense?
I wouldn’t be surprised if variable bit rates exacerbate the issue, though that is conjecture on my part.
That’s not been my experience, James. If I place the face slap sound in ProTools against H.264 picture it will play in the same timing relation to the picture slap for a given start point. Repeated playings from the same start give consistent results. But when you change the location of the start everything may shift and it’s never more than a frame out of sync. Which leads me to believe that VBR is the issue here, not simply a distribution of the data across several frames.
Now if ProTools and the OS couldn’t work together – if they couldn’t agree when to start simultaneously – then the timing difference between picture and sound could be ANY value greater than zero, with larger timing differences for computers with low processing power. The sync problem with H.264 in ProTools is limited to under a whole frame. Big deal? Sound that is earlier than picture by 99% of a frame (roughly 40 ms at 24fps) can be pretty obvious for a hard effect, like a face slap. Early sound always feels more obvious because sound never precedes vision in nature.
I see what you’re saying about scrubbing, but no sound for picture professional relies on scrubbing to judge timing. At least no one I know.
I appreciate your insights and discussion. If I need further explanation on this issue please continue to inform me. Thanks.
Thanks a bunch! Very helpful!
Thanks for reading.
Great article .. just a tip .. using quicktime pro 7 exporting h.264 with “all” frames as a key frames seems to fix this problem for me. sadly mpeg streamclip doesn’t have a keyframe option
Interesting solution, Joel. I’ve recently started using Xilisoft Video Converter and can’t find a keyframe option in it either.
What is the file size difference between “auto” and “all” keyframes?
I can’t even see where MPEG Streamclip gives you the option of converting a video file to Apple Motion JPEG A! I am in dire straits with my ProTools gig – it seems utterly incapable of playing video files, seemingly no matter what! Help!!!
@Andy Pro Tools 11 and 12 play much better with DNxHD – if you’re running one of those versions, that may help. The codecs need to be installed on your system separately from MPEG Streamclip though. Alternatively, run the video through a synced application such as Video Slave.
>