The casual user might only use the FFmpeg multimedia framework for converting from one audio or video format to another, but FFmpeg can do much more than that. I'll take a look at one of FFmpeg's most powerful secret weapons: the lavfi virtual device.

Using lavfi

FFmpeg's lavfi (short for libavfilter) virtual device sounds more complicated than it really is. Instead of using prerecorded video or audio files as streams, lavfi lets you create streams out of thin air. You can use these streams on the fly and combine them with clips and other dynamically generated streams (e.g., from a webcam or microphone) to create your output file.

Maybe it is better explained with an example:

ffmpeg -f lavfi -i color=c=red:size=640x480:rate=30 -t 10 red.webm

Most basic ffmpeg instructions you have seen probably have at least one input file, but the instruction above has none because "input" is coming from the lavfi virtual device, not from a file. The content the virtual device generates is described by what goes after the -i parameter, that is, color=c=red:size=640x480:rate=30. Broken up, color tells FFmpeg what sort of stream it should expect – in this case, a simple color video stream. After the first equals sign comes the color parameters:

• c sets the color of the frame; you can browse a complete list of colors that FFmpeg understands [1] and use one of those (e.g., red here), or you can use the #RRGGBB[AA] notation.
• size establishes the frame size.
• rate sets the frames per second.

The -t 10 parameter makes the resulting clip 10 seconds long.

Making a 10-second video of a flat color is probably the simplest thing you can do with lavfi, but making a video of a test pattern is also easy:

ffmpeg -f lavfi -i testsrc=s=640x480 -t 10 testscr.webm

Combining lavfi-generated streams with other streams is what makes it really powerful. As you can see in Listing 1, lavfi streams can be used the same way you use regular streams, as described in last month's issue [2]. In this case, you are blending a generated test card stream with a clip taken from the Big Buck Bunny [3] movie (Figure 1). If you would like to see what it will look like before rendering a new video file, check out the "Going Live" box and the ffplay utility.

Figure 1: You can combine lavfi-generated streams to achieve all sorts of interesting effects.

ffplay -f lavfi -i "color=c=red:size=640x480:rate=30"

ffplay -f lavfi -i "testsrc=s=640x480"

Meme Factory

All of the above is well and good, but lavfi-generated streams do have some real-world applications, such as in animated GIF building.

GIFs are a really old and not very efficient graphical format, but they have become very popular in the last years. Twitter, Facebook, WhatsApp, and Telegram all implement ways for users to convey their deepest thoughts using animated clips of celebrity facepalms and cats falling off of things.

However, not all GIFs are equal; in fact, it is easy to distinguish a "professional" GIF artisan from a wannabe artist by the caliber of their pixels. Let me explain with Figure 2.

Figure 2: Smooth pixels make a high-quality GIF animation.

You are looking at a compound image of two different GIFs, each generated using a different technique. On the left, the image looks smooth, with a gradual gradient between each shade of gray. On the right, you can clearly see the pixels, and the transition between areas with different shades of gray is abrupt and jagged. Believe it or not, both are the exact same resolution, and the smooth, good-looking version on the left weighs a whole 2MB less than the one on the right.

To understand why this happens, you must look at how GIF uses palettes. GIF images use a palette of only 256 colors, which is not enough to cover the whole spectrum of a full-color video and make it look good; at least, it is not good enough by today's standards. The easiest and laziest way of generating a GIF from a clip is to enter:

ffmpeg -i clip.mp4 image.gif

However, this is not optimum by a long shot. All of the colors in all of the frames in your clip must be matched to one of the colors in the default palette used by FFmpeg (Figure 3). Not surprisingly, the final result can look grainy.

Figure 3: The default FFmpeg palette used when generating GIFs.

However, you can create a palette that draws its colors from the clip you are converting itself:

ffmpeg -i clip.mp4 -filter_complex "fps=15,scale=320:-1:flags=lanczos,palettegen" clip.palette.png

Here, FFmpeg creates a palette called clip.palette.png (Figure 4) with the colors from clip.mp4. Notice how the colors are softer and more pastel than those of the default palette and how they match the general palette of the clip much more closely.

Figure 4: A palette taken from a five-second clip lifted out of Big Buck Bunny.

Once you have your custom palette, you can then use it to generate your GIF:

ffmpeg -i clip.mp4 -i clip.palette.png -filter_complex "paletteuse,fps=10" clip.gif

Apart from using the palette, you are also reducing the frame rate (fps=10) to make a smaller, lighter clip.

If your clip still doesn't look very good, remember you still only have 256 colors. The palette in Figure 4 does not look like a uniform gradient, which is a sure sign your GIF will probably look splotchy. If your original clip has a large variety of colors (e.g., with frames shot at night and bright, colorful frames shot during the day), your custom palette will be more uneven and your GIF will look grainier.

Filters and algorithms let you compensate for this effect, but none are simple, nor do they tend to improve the quality that much. Instead, "professional" GIF creators have come up with something else.

256 Shades of Gray

You might have noticed that many GIFs are rendered in shades of gray. This is because it is easier to get smooth transitions from one shade of gray to the next with a palette of 256 grays than with a palette trying to cover a wider array of colors.

To prepare a clip for the "GIFification" process, you need to reduce it to shades of gray:

ffmpeg -i clip.mp4 -filter_complex "format=gray,scale=640:-1" clip_gray.mp4

It's important to notice how you should resize your clip to the size you want your GIF to be (scale=640:-1). If you mess with the size when converting to GIF, it will mess with your palette.

By converting your clip to gray, you would seemingly have made matters worse. The default palette in Figure 3, has very little in the way of gray. Instead of being able to pick from 256 colors, if FFmpeg were to use the default palette, it would have to choose from 10 or 15 colors, making the splotchiness even worse than if you left the clip in full color.

However, if you generate a palette with the following,

ffmpeg -i clip_gray.mp4 -filter_complex "fps=15,scale=320:-1: flags=lanczos,palettegen" clip_gray.palette.png

you'll see something like Figure 5.

Figure 5: A grayscale palette generated from a black-and-white clip taken from Big Buck Bunny.

If you use this palette to generate your GIF,

ffmpeg -i clip_gray.mp4 -i clip_gray.palette.png -filter_complex "paletteuse,fps=10" clip_gray.gif

open clip_gray.gif in an image visualizer and zoom, you will notice how the transitions between colors are much smoother.