Tutorials – Natron

Below is the video associated with this article. 

Motion tracking is used extensively in movies and TVs nowadays, not only to turn actors into all sorts of fantastical creatures, but also, more subliminally, to create interesting scenery for what would otherwise be a mundane backdrop. It doesn't have to be fantastical settings either: Actors may seem to be walking down an avenue with a view of the Manhattan skyline or dining in a fancy restaurant, when, in reality, they are just strolling on a regular street of an anonymous Canadian city or sitting in a blank room with a few props in front of a green screen.

However, when the camera moves, the background moves coherently with it, in such a way a painted backdrop wouldn't. This is achieved with motion tracking and can help your film look like it has a much higher budget than it really has, while at the same time saving money on locations.

After our excursion into using motion tracking on an object for stabilization in the last issue of Linux Magazine [1], I'll show how tracking can be used for integrating backgrounds and foregrounds into your shots.

Tracking

I'll use Natron [2] to project one clip onto another. In this example, I have a clip showing a notebook I move around. I will project a clip taken from Tears of Steel, a film made by the Blender team, onto a clip showing a page of a notebook I filmed myself. The aim is to make the Tears of Steel clip look like it is part of the notebook's page. Tears of Steel is downloadable from the Project Mango page [3]. You can get the notebook clip from [4], although you may find it more fun to film your own clip and play with that.

Open your background clip (Image | Read), the one onto which you are going to project the effect, and choose the items you want to track. I did a full explanation on how to do this in last month's issue [1]. Since the clip you are going to project is an oblong, you need to track four points on the notebook's page to do this correctly. I have drawn some reference marks on the page as a guide. In professional production, these marks would later be erased using the tracking information. If you're filming out in the street, and you can't paint marks on things like walls and buildings, you can use windows, doors, etc. as references to track.

Start by adding a Tracker node between your video and your Viewer nodes; go to the node's property box and look for the list of trackers at the bottom of the box. It will be empty to start with, but you can add a new tracker by clicking the + button below the list. The tracker (which looks like a square bullseye) will appear in the middle of your clip's preview.

Make sure you are on the first frame of your clip. Click in the middle of the tracker and drag it over to the first reference point. This will automatically create a keyframe for the tracker on your first frame.

In last month's issue, I said that you could add keyframes at regular intervals along the timeline to guide the tracking smoothly. You would do this by moving from frame 1 to frame 20, for example, and dragging the tracker to wherever the reference point had moved. Then you would do the same on frames 40, 60, and so on, setting keyframes every 20 frames (Figure 1).

Figure 1: Setting a keyframe for the tracker every 20 frames.

Another way of guiding the tracker is by processing frames in batches and backtracking when the tracker loses its way. To do this, make sure you are on the first frame, place the tracker on your reference point as explained above, and locate the trackRange button in the tracker's toolbar (the set of buttons with blue icons above the clip's preview in Figure 1). The trackRange button is the sixth button from the left and shows a blue arrow between square brackets. When you click trackRange, a small dialog will pop up that lets you set the range to track. A manageable range is 50 frames, so in the First Frame box put 1 and in the Last Frame box put 50. Then hit the trackFW button (immediately to the left of the trackRange button) and Natron will start tracking.

It is a good idea to activate the showError toggle button – the third button from the right in the tracking toolbar. This will show you the "health" of the track. If the tracker is on course, the nodes on the track will show up green. If the tracker thinks it is deviating off course, the nodes on the track will show up orange or red.

When you are tracking in batches and notice the tracker deviating off course, it is relatively easy to backtrack to the frame where the tracker first got confused, relocate the tracker onto the reference point, and start tracking again from that point onwards.

Be warned that problematic frames usually occur in clumps, so it is normal to have hiccups on, say, frames 80, 81, 82, 83, and so on. Then you will have whole stretches of frames where the tracker has no problem at all. Tracking requires patience.

Once you have reached the 50-frame mark, click on the trackRange button again and process frames 50 to 100 as above.

Whichever method you use, you will get the best results with footage that is of high resolution, high contrast, stable, and filmed at a high framerate. Shakiness and fast camera moves makes things blur in individual frames, throwing off the tracker. If you are wondering how the professionals manage to track stuff in, say, a frantic, high-speed car chase, the secret is in the magic of post-production: They film slow and smooth, do their tracking, and then add speed and shaking later.

When you are happy with the tracks, it is time to prepare for your second clip.

Figure 2: The tracker toolbar explained.

Projections

Although you have four tracks, you haven't told Natron what you want to do with them. Let's do so now.

First disconnect the Tracker node from the Viewer, otherwise a bug in Natron will show confusing information that will make the next steps more difficult. As to better see what you are doing, connect the Read node containing your background clip to the Viewer node, as shown in Figure 3.

Figure 3: Connect your background clip to the Viewer node.

Double-click on the Tracker node to bring its property box to the top of the pile, and, within the box, click on the Transform tab. Click on the Motion Type drop-down and pick Match Move from the list. This tells Natron that you want to match the movement of an element (in this case, another clip that you will bring in later) to the movement of the trackers.

From the Transform Type drop-down, choose Corner Pin. This tells Natron that you want it to connect the corners of the element you want to move to the trackers – that is, you want to pin the corners of your second clip to the trackers.

Before continuing, uncheck the Robust Model checkbox. It should not apply to your video and can cause problems down the road.

Next, you have to set up the area in which you are going to insert the projected video clip. Look at the Corner Pin controls section a bit lower down and make sure the Disable CornerPin checkbox is unmarked.

In the tabbed box below, click on the from tab. It should show the coordinates of your clip's corners. In the example, from1 shows x=0.0 and y=0.0 (lower left-hand corner), from2 shows x=1920.0 and y=0.0 (lower right-hand corner), from3 shows x=1920.0 and y=1080.0 (upper right-hand corner), and finally from4 shows x=0.0 and y=1080.0 (upper left-hand corner). If you are familiar with standard video formats, this should ring a bell: The clip is filmed at 1080p.

This is the default starting point for the frame that will enclose your projected clip. However, you don't want the corners of your projected clip to be connected to the corners of the background clip. Instead, you want the corners of your connected clip to coincide with the trackers.

Notice in the Viewer how there is now a new white frame around the outside of the clip. The corners are labeled to1, to2, to3, and to4, anticlockwise from bottom left. Go back to the Tracker's property box and locate the Overlay Points drop-down. Click on that and select From. The labels on the frame will change to from1, from2, from3, and from4.

Make sure you are on the first frame of your clip and locate the little square handles on each of the corners that allow you to drag around the From frame. Drag each corner over its nearest tracker (Figure 4). Notice that you can get the from nodes quite near the trackers, but not right on top of them – for some reason Natron doesn't allow you to deposit the corners right on the trackers. To get them as close as possible, hover your cursor over the viewer, roll your mouse wheel, and zoom in. To pan around within the viewer, click and hold the middle button and drag.

Figure 4: Relocating the From frame onto the trackers.

Once you have placed all four corners of the frame, click on the Overlay Points drop-down again and choose To from the list. Find the Compute button within the Tracker's property box and click it. This calculates the position of all the to nodes, matching them up with the position of the trackers throughout the whole clip. If you run through the footage, you will see the to frame following the trackers around as they move.

Look at the bottom of the Transform tab in the Tracker's property box, and you will see a button labeled Export. Make sure the Link checkbox to its left is checked and then click the Export button. This creates a new CornerPin node in your Node Graph containing the information from the transformation you have just made.

By checking the Link checkbox, you make sure that any modifications you make to, say, correct the trackers will be immediately inherited into the CornerPin node. If you uncheck the Link checkbox, Natron dumps a static copy from the trackers when you press the Export button. If you then modify a tracker's position, you will have to export all the data again to a new CornerPin node and get rid of the old one. For your purposes, linking is much more convenient.

Fitting In

Time (to at last!) bring in your second projected clip.

Add a new Read node (Image | Read) to pull in the clip you want to project onto the first clip. In this case, as mentioned above, I am using a clip from Tears of Steel, a free film created by the Blender Foundation.

Connect the clip's Read node to the CornerPin node and then create a Merge node (Merge | Merge). From the Merge node's property box, you can choose a wide variety of merges from the Operation drop-down list. The default over operation is good enough for now, since it will allow you to see both clips, with the projected clip overlaid and semitransparent on the background clip. To pull the merged clips onto the viewer, connect the CornerPin node to Merge's A connector and your background clip to Merge's B connector. Then connect the Merge node to the Viewer (Figure 5).

Figure 5: Connecting both clips to the Viewer.

You will see the projected clip overlaid on your background clip. The inclination and angle of the overlaid clip look good, but it is not placed between the nodes (Figure 6), which is where it should be.

Figure 6: The overlaid clip needs relocating.

What is happening is that Natron is projecting your clip onto a plane that has the same dimensions as the background clip. Since in this example the projected clip is smaller (1280x534) than the background clip (1920x1080), it appears in the corner of this invisible bigger plane, in its lower left corner, where the coordinates' origin is. What you want to tell Natron is to adjust the size of the plane to the size of the projected clip – that is, instead of using a 1920x1080 plane, Natron must use a 1280x534-sized plane.

You do this by adjusting the CornerPin node's own From parameters. Double-click the CornerPin node to bring its property box to the top of the stack, and locate the to and from tabs for the node. Click on the from tab; at the bottom of the tab, you will see three buttons: Set to input rod ("rod" stands for "region of definition"), Copy "to", and Copy "to" (Single). You want to click on Set to input rod, because the "rod" is what Natron calls the size of the source clip.

Once you do that, the projected clip will snap between the trackers (Figure 7), and your task is finished.

Figure 7: The final clip, with the projection in place between the markers.