Any visual effects artist would probably agree that masking is one of the most tedious tasks to have to sit through in post. Although the process allows us to isolate objects from their backgrounds, going frame-by-frame to perfect a mask could range from a few minutes to hours or sometimes even days. This is especially true for shots that involve camera or subject movement.
Over the years, however, Adobe has included an alternative method of masking in After Effects which many novices may still not know about. It’s called rotoscoping, which essentially performs the same function as masking but is easier to pull off, as explained in a video by Max Novak below.
To begin, import your footage into After Effects CC, create a new composition and add the image to it. Then double-click on your footage in the comp and select the Roto Brush tool which you’ll find in the toolbar.
The next step is to take your cursor and hold and draw around the subject you want to root out of the scene (isolate from the background). As you draw, you will notice a green streak follow your path, helping you indicate the areas you’ve drawn. Keep in mind that you don’t have to be very intricate with your brush stroke on the first go.
When you let go of the mouse, After Effects will calculate and outline your subject with a pink/purple line, showing you what the software thinks you want to isolate in the scene. If there are large areas that are selected in the rotoscope, you can easily remove this by holding the Alt key and then drawing around the unwanted area. If there are items that were left out of the rotoscope, simply draw over the area and the computer will add the selected region to the mask.
To move back and forth between frames, use the Page Up/Page Down keys to scrub through, making corrections as needed. When you’re finished with the rotoscope, you need to make sure that the duration of your edits is selected.
On the bottom of the preview window, just below the mini-timeline, you should see a grey bar. Adjusting this will tell the software what the stop/start points of the rotoscope effect are. Tweak the gray bar accordingly to the length of your clip/effect. When you’ve done so, click the Freeze button on the lower right corner of the preview window and the computer will begin rotoscoping your subject.
Once the process is completed, you’ll be able to add any background elements you desire. Keep in mind, though, that unlike the Mask tool, rotoscoping doesn’t offer features such as path animations, expansions, or feathering (or else the Mask tool would be obsolete). The rotoscope tool is mainly used for quick background replacements since it’s much easier to draw the mask path then manually click and adjust nodes and points.