What Is a Codec? (And Which One Should I Use?)William Gadea 06.26.2018
What is a codec? If we defined the color of every pixel in a video, the file size would be enormous (nearly 4 gigs per minute for 720p video.) if you’re interested, here is the math:
1280 pixels wide x
720 pixels tall x
24 bits of color info per pixel x
24 frames per second x
60 seconds per minute =
31,850,496,000 bits or
Since this is a difficult size to deal with, computer scientists have devised protocols for reducing that file size while maintaining picture quality. These protocols are called codecs. (Codec stands for COmpressor-DECompressor.)
Before we start talking about codecs, let’s distinguish them from video formats. What are video formats? If a codec is the language a letter is written, then the format is the envelope that the letter is carried in. That is why the same codecs can often live happily within many different formats. Video formats are represented by the extension on your video file:
- Quicktime (.mov)
- MP4 (.mp4)
- Window Media Video (.wmv)
On the other hand, you will probably need a software reader to discover what codec a file uses. Here are some common ones:
- Sorenson (older)
What is the best codec to use?
The answer to this question will depend on what your purpose is. Different codecs are used at different places in the video-making/distributing process. In the infographic below, we are testing these codecs:
- Shooting (RAW)
- Editing (Pro Res 4:2:2 and Uncompressed 10-bit)
- Animation and Special Effects (Animation)
- Mastering and archiving (Pro Res 4:4:4)
- Streaming or Distributing (H.264)
Usually you are best off going with the recommended codec of your chosen hardware (or the specs of your video host, if you are uploading something to stream.)
How do codecs work?
Codecs use different techniques to reduce file size.
Run-Length Encoding: Say you have a big patch of your video image that is the same color. It would not be economical to repeat the color information for every pixel in the frame. Rather, it would make more sense to say “this pixel is x color, and the following 35 pixels are the same color too.” An advantage of this method is that it is lossless, meaning no picture quality is lost at all.
The Animation codec that is tested below is a lossless codec that uses Run-Length Encoding. Notice that for live action footage, like the tested image, this method is not very efficient, meaning it reduces the file size only slightly. However, if we were rendering animation or motion graphics with large areas that are the same color, then this codec would produce much smaller files.
Bit Depth: Usually in video we use 8 bits (or 8 binary numbers) to define a color. That results in 256 gradations of color, which usually gives us a pretty good image, but if we want more definition we can apply 10 bits to the color of each pixel (1,024 gradations.) This is what the 10-bit Uncompressed video does, and it actually makes the file a bit bigger. Likewise, we can reduce file size by communicating color information with fewer binary slots than 8. The tell-tale sign that a file is being compressed this way is that we start to see banding on the gradated areas of the image.
Chroma Sub-sampling: Our eyes are designed to be very sensitive to brightness, but not that sensitive to color. (In our retinas we have nearly 20 times the number of brightness-seeing Rod photoreceptors as color-seeing Cone receptors.) Codecs can take advantage of this human characteristic by keeping brightness information but throwing away some color information.
Rather than expressing a color value with three numbers representing colors (Red, Green, Blue,) we can express a color value as Y’CbCr (Brightness, Blue, Red) and derive the missing color (green) by math, since brightness is just a combination of the three primaries. The power of this is that it lets us make the color information chunkier – one pixel of information may represent two or four pixels in the image. Because the brightness is being varied for every pixel, however, the eye has a very hard time spotting the difference.
When we see three numbers separated by colons, as we do with the Pro Res codecs below, they represent how much information is being applied to the Brightness-Blue-Red channels. For instance, a 4:4:4 codec would have no Chroma Sub-sampling; brightness and color information is provided for every pixel. The Pro Res 4444 is a 4:4:4 codec; the additional 4 refers to the alpha channel (transparency information) that is sometimes used with this codec.
In a 4:2:2 codec (such as the Pro Res 422 we have tested below) every pixel has brightness information, but out of every four pixels we only have two pixels of color information in the blue and red channels. This process can reduce file sizes considerably and be barely perceptible.
Spatial compression: We have talked about how Run-Length Encoding can bunch together the same color and indicate that it is being repeated. Spatial Compression is like that, but it fudges a little bit. If two colors are similar, then it reproduces them as being the same. This can cause the same sort of banding that you see in low bit-depth images.
Temporal compression: Imagine a talking heads shot. On camera, we have a person talking and moving around a bit, but if the camera is on a tripod, the background isn’t moving. Why repeat the image information for the background areas if you can say “let’s repeat these patches of the image that don’t move for the next 50 frames.” When a Codec does this, it is applying Temporal Compression.
We have talked about five different techniques for reducing file size:
- Run-Length Encoding
- Bit Depth
- Chroma Sub-sampling
- Spatial Compression, and
- Temporal Compression
While these are common ways Codecs reduce file size while maximizing quality, many of the most effective codecs use a combination of these methods, or others we have not broached.
What is bit rate?
Finally, let’s talk about a measure of how much a Codec has compressed a video clip: the bit rate, usually expressed as Data Amount/Second. A Codec that outputs at 8 Mbps would give us a file that is 8 megabits large (or 1 megabyte, to use a more common measure) for every second of video.
A mistake that many people make is to confuse bit rate with quality. Some codecs can produce excellent images at the same bit rate that would produce a chunky mess in older, inferior codecs.
If you are rendering out a video, your chosen Codec will often ask you what bit rate you would like to output at. When you set it, you should remember that codecs nearly always have a sweet range where they are producing the best images at the smallest file sizes. Larger bit rates usually produce only very marginal improvement, while smaller bit rates can quickly deteriorate the image. Experimentation (or just experience) will tell you where a Codec’s sweet range is.