Blackmagic Fusion: Clean Plates

Painting clean plates is another task commonly assigned to junior compositors. While roto is an easy and tedious job, paint can be very challenging. Every shot brings unique challenges, and many will require you to use a variety of techniques. A shot might require paint work in order to remove wires or other rigs that were necessary for the shot but shouldn't appear in the finished work. Sometimes lighting or sound equipment is visible accidentally and needs to be removed, or it was impossible to shoot a fantasy piece in the countryside without telephone wires visible in the background. Perhaps an effect calls for a character to dissolve into mist, and you need to create whatever set should have been behind them.

Although I use the term "paint," the Paint tool is usually not the first tool I reach for when doing this kind of work. If you attempt to touch up an image frame-by-frame by painting on it, you will almost certainly introduce chatter—pixels that dance and flicker—because it is virtually impossible to repeat a paint stroke from frame to frame, particularly if the image being painted is also moving. Transforms, warps, image filtering, and other methods are all used to supplement the Paint tool when the time comes to remove something from the frame. Let's get to work.Continue reading →

Blackmagic Fusion: Rotoscoping and Keying

The creation of mattes from moving footage is a crucial, though sometimes tedious, part of making visual effects. It is a rare shot that does not require a matte of some kind. Let's first talk about what exactly a matte is and a few methods that can be used to generate them.

First, the words matte and mask are used mostly interchangeably, although matte is not often used as a verb. A matte is a single-channel image that is used to isolate a part of an image, either to prevent it from being changed or to restrict changes to only that area. You might come across the term "traveling matte," which simply means the matte image moves, probably to follow whatever object in the scene it is meant to isolate. You might also run across reference to an "articulated" matte or roto, which usually means a matte made up of several pieces that can move independently to cover something complex, like a person.

Rotoscoping (roto, for short) uses splines, like the Polygon tool we used in the Basics lesson, to create outlines around a subject. It is a time-consuming job that is frequently assigned to junior compositors or dedicated roto artists in order to save the mid- and senior-level artists (and their higher salaries) for more demanding tasks. As such, it's an important skill to learn to do well and quickly if you want to show your worth to a visual effects studio.

lightsabergreenscreenKeying uses the color values in the image to try to separate the subject from the background. Although the most obvious use for keying is isolating actors on a green or blue screen stage, it can also be used on any object that is of a different color than whatever surrounds it. In the previous lesson, we could have used a keying operation to make a matte for the orange gun barrel and possibly skipped the tracking step entirely.

bunkspeed_camaro_2You might also receive mattes rendered from the CG department. This image shows a so-called "clown pass," in which each piece of the Camaro is assigned a different flat color, allowing the compositor to easily affect one piece of the car without touching the others (although making the hood stripes and the windows both yellow was a bad move. Hopefully there's also a refractions pass that can be used to separate the glass.)

Rotoscoping

For this lesson, I have prepared a short greenscreen clip from another of my student projects. I can't for the life of me remember either of these two gentlemen's names, but I do remember that they were a delight to work with. To start, we'll rotoscope the man on camera right (we specify camera, screen or stage right in order to make it clear that we don't mean the actor's right. It's easy to completely change the action of a shot with that kind of misunderstanding), then we'll key the man on camera left. Download the footage here: 004-mattes_plate.zip (95.8 MB)

004_mattes_001A human being is a very complex shape, with many moving parts that deform and can move independently. As someone turns around, their silhouette can change completely. Many beginning rotoscopers start by trying to draw a single spline around an actor, as shown in the image to the right.

As soon as you start playing through the sequence, though, you'll quickly realize how much work it will be to manage that shape. Instead, try to analyze the actor's body and accessories. Determine which parts of him travel together and are unlikely to change shape much.

The lower legs, the thighs, the torso, the upper arms, the lower arms, the hands, the head, and the suitcase are each much simpler shapes that could be more easily managed. By splitting the subject into piece like this (articulating it), you can let the computer do more of the work interpolating between extremes.

004_mattes_002Pay attention to how the subject moves. Find the places where a particular piece of him changes direction, or starts or stops moving. Find the apex of a curved motion or the place where the hand is most open. In this case, I elected to draw my splines for the extended hand when it was most open and complex. I made a separate shape for the thumb because I saw that it closed down over the other man's hand and therefore moved a little differently than the rest of the hand.

Fusion automatically sets keyframes on the splines you create. You can manipulate the animation, setting keys or removing it, by right-clicking the bottom most line in the Tools panel. It will also set a keyframe on the entire shape any time you move a point. That's another reason to use smaller shapes—you may not need keyframes on the head in the same place you want them for the hands, but if you have just one big shape, you don't have a choice about it. The points don't get individual keys.

Once you've created the initial position for a spline, move down the timeline until you find the other extreme of whatever motion you observed. Drag a selection around your spline and move it to the new position. Fusion will interpolate between the old position and the new, hopefully keeping the spline fairly close to whatever it's intended to follow.

004_mattes_003

In this image, I have placed keys at frames 1 and 16. Frame 7 is Fusion's interpolation between the two. To continue the task, I select the points at Frame 7 and move them back to the head. You may notice that this spline doesn't have the Bezier handles that you saw on the Polygon. This is a different type of spline, called the B-Spline in both Fusion and Nuke. The After Effects equivalent is the Roto-Bezier, and some software, like Mocha, calls it an X-spline. I prefer it for organic shapes because it has fewer controls to keep track of, although it tends to require more points.

Try not to move individual points if you can help it. The matte will tend to "crawl" if the points change position relative to one another. If you need to rotate the entire spline, hold the T key ("twist") and click-drag with your mouse. You can scale by holding S, or scale along just one axis by holding X or Y while dragging. In a B-Spline, you can also change a point's smoothness—how much of an angle it has, by holding W while dragging.

Set a keyframe at each extreme of motion you see, then go back and check the frames in between, adjusting the spline as needed, then check between the original and the new keyframe, and so on, until the spline is correct on every frame. Hopefully you won't need to set keys on every frame, but in some cases it's necessary. My very first job was rotoscoping a woman with long, fine hair and loose clothing standing in front of a wind machine. There was nothing to do but put a key on every single frame.

Sometimes it's difficult to see exactly where your subject ends. Blur and anti-aliasing make some edges vague. If the edge is reasonably uniform in blurriness, the Soft Edge and Border Width controls can take care of handling the blend. If there is motion blur, then Fusion's Motion Blur features can usually match it. In those cases, decide where in the blur you want your spline to fall and stick to that decision all the way around and throughout the entire clip. I usually use whatever I determine is the first fully opaque pixel. If you need a varying amount of blur around an object—if you know an effect needs to fade off, for instance, or you have an extremely shallow depth of field (the blur gets stronger based on distance from the camera)—you can turn on the double spline mode in your tool to create a smooth falloff between the main spline and a secondary shape that you pull off from it.

Roto is a simple task, but it's time-consuming. Don't get discouraged if it takes you several hours to roto even a simple shot. You'll get faster with practice, but some shots will simply take a long time. If they didn't, the seniors would just do the work themselves. Remember, though, that you should take every opportunity you can to simplify matters for yourself. Apply what you know about tracking to automate moving splines. There are also some specialized tools, like Mocha and Silhouette, that can make roto easier and faster.

To conclude this part of the lesson, here are a couple of things to keep in mind: It is far easier to rotoscope a positive shape than a negative one. That is, if you are tasked to roto a window behind an actor, don't try to make a shape for the left side of the window and the actor's left profile and another for the right side and the actor's right profile. Instead, roto the actor and use that to cut a hole in the shape you make for the window.

When you are finished with a spline, use its Level control to turn it off instead of sending it off-screen. The compositor who uses your mattes may wish to turn on motion blur, and when the spline suddenly flies off the edge of the screen, it leaves a streak behind. You can move it once it's remained still and invisible for a frame or two.

Like all of Fusion's tools and workflows, there is a lot more to learn about the Polygon and B-Spline tools, and about the process of rotoscoping. We'll return to the topic several times during future lessons.

Keying

The second method to create a matte is to use contrasts in the image's color information to generate it automatically. This is known as keying. When a green or bluescreen is used, it is more specifically called chroma-keying. Fusion comes with five keying tools, although one, the Difference Keyer, is not useful for this purpose.

Luma Keyer

We'll start with the Luma Keyer. In its default mode, the Luma Key attempts to create the matte by making the brightest parts of the image white and the darkest parts black. Create a Luma Keyer, connect it to the Loader, and put it in your Viewer. Click the Viewer's Channels button to look at the alpha. Like the Tracker and Color Corrector, the Luma Keyer works best in a gamma corrected space, so I'm going to leave my image in sRGB for the time being. I also toggled the Invert control because we want the actor's matte to be white on black rather than the reverse.

004_mattes_004

The Luminance mode isn't much good in this case because our subject's collar is brighter than the background, and his face is about the same brightness. Even after I have adjusted the Low and High threshold values to get purer blacks and whites, it is clear that the face and collar will not be part of the matte.

If you look at each of the color channels, though, by selecting the Viewer and hitting the R, G, and B keys, you can see that the green channel has a great deal of contrast with everything except that collar. Change the Channel control to Green and adjust the Low and High thresholds again to get a somewhat better result.

004_mattes_005

The collar is still a problem. There are a few other modes we can try, though. The consequence of shooting on a screen that is such a pure green is that the screen is quite likely the most saturated thing in the image (saturation is measured as the difference between the brightest channel and the other two. Since the screen is almost white in the green channel and black in the red and blue, it is highly saturated. We'll hit some color theory eventually, too, where we can go more deeply into that topic.)

004_mattes_006Switch the tool to Saturation mode and try adjusting the thresholds again. Unfortunately, this is even worse than the green. The doofus who set up this camera (me) evidently forgot to turn off the internal sharpening filter. You probably saw the dramatic ringing in the red and green channels. That bright outline is causing the saturation key to break down.

So it looks like the Luma Keyer isn't going to get us where we need to be. In fact, that sharpening is probably going to stymie even our best efforts. Nevertheless, we'll carry on and see if we can't salvage something from this footage.

Chroma Keyer

The next weapon in our arsenal is the Chroma Keyer. This tool is designed specifically to deal with screens of this sort. Maybe it will give us better results. The Chroma Keyer works by sampling the screen color and setting thresholds on all three color channels plus the luminance simultaneously to try to make just that color transparent. Sampling the color is done simply by clicking on the green screen in the Viewport. Clicking just one pixel probably won't do much. The Chroma Keyer lets you drag out a selection box around a group of pixels, though, and all of those pixels are considered for the thresholds.

004_mattes_007You can make several such selections, and each time you do, more colors are added to the transparency. After getting rid of most of the large blocks of green and pushing the Soft Range all the way up, we start to get something that might just be passable. There is still more work to be done, of course, but the Chroma Keyer has certainly done a much better job than the Luma Keyer did. In the screenshot, you can see that I have spun down the Color Range controls, so we can see all of the thresholds that have been set. Any pixel that falls into all four ranges shown is considered transparent. The Soft Range control blurs the ends of the thresholds, allowing pixels near the boundaries to become translucent.

In order to clean up the green fringe, we'll use a Matte Control tool. Matte Control is a very useful tool for dealing with alpha channels. It can combine them in various ways, blur them, erode and dilate them, change their falloff, and, most importantly for this image, it can despill and defringe. Make a Matte Control and place it after the Chroma Keyer. Switch over to the Spill tab to see what options we have for controlling that fringe.

The very first thing to do is to switch the Spill Color from Blue to Green. Obviously, we have a green screen, so our spill is also going to be green. "Spill" is the reflected colored light from the screen falling on the actor. It will be present to some degree in any chroma key shot, but Fusion's Matte Control is a superb means of getting rid of it. The Spill Suppression control determines how strongly the despill operation is applied. The Spill Method is a selection of four humorously-named algorithms, from least-dramatic to most-dramatic. Rare and Medium are usually most useful for blue screens, which produce comparatively less spill. Well Done is typically the best for green screens, and Burnt can kill even the strongest spill, but it tends to add too much of the complementary color (magenta is complementary to green and yellow to blue).

004_mattes_008The Fringe controls are a simple color corrector that is applied only to pixels that are translucent. The Fringe Size can push it into neighboring pixels (both fully opaque and fully transparent) if you need to blend the fringe with the rest of the picture. With a combination of the Spill Suppression and the Fringe controls, I can get rid of the green halo, but that black line is just not going to go away. At least, not with these controls.

Still, given the very poor quality of the footage, this is a result I might be able to live with. I put it in front of an actual background just to see how it holds up on something other than the checkerboard. Surprisingly well, actually. If I put them in front of a bright background like this one and let the lightwrap and edge blur take care of the black line, it might even work. I still want to look at that last keyer, though, and see if it can get even better.

UltraKeyer

UltraKeyer is a third-party keying plug-in that is included with Fusion. It functions much like the Chroma Keyer, but it has a couple of other controls. I am not 100% sure what the additional controls are doing, mathematically, but I can guess. Background Correction is probably applying a hue shift to try to compensate for screens that are not pure blue or green. Matte Separation is likely a saturation adjustment to further push the greenscreen's levels away from the foreground's. The PreMatte Range works exactly like the Chroma Keyer, except that if you are looking at UltraKeyer's output while you select colors, you're going to get a tainted selection. UK has already started to make the matte as soon as it is added to the flow, so you have to pick your screen colors from a Viewer that is displaying a tool upstream of the keyer. I usually put the footage in Viewer 2 and the keyer in Viewer 1 so I can easily see the results of my color picking.

Ultra Keyer also has a Matte Control tool built-in, so you don't need a separate node. After a few minutes of experimentation, I decided that Ultra Keyer wasn't going to do any better than the Chroma Keyer and Matte Control I'd already set up.

Primatte

There is one final color keying tool available in Fusion that doesn't show up if you Ctrl-Space and search for Key, but you may have seen it if you searched the Add Tool menu for the others (it's in the Matte folder). Primatte is another third-party tool that operates on an entirely different principle than the Chroma and Ultra Keyers. It also has a much more intimidating interface.

Most of the time, I really only use a few of the features in Primatte. I have to admit that I am not an expert with it, preferring to do most of my matte refinement outside of the keyer. So the first thing to try is the Auto Compute button at the top of the interface.

004_mattes_009

Impressive! One click, and already we have a nicer matte than the Chroma Keyer made. The spill suppression is too strong, though; it's created a magenta cast in the actor's suit. Let's back up and actually do the work instead of letting the automagical algorithm make all the decisions.

As with the other keyers, the first step is to select the background color. Primatte operates by using the color picker in several different modes. When you first add it to the comp, you'll be selecting the background. Primatte doesn't use a selection box, but a paint stroke. Click drag across a range of background colors, and you will see the screen disappear. Just do this once. To get rid of the remaining background, we'll switch to Clean Background Noise.

004_mattes_010

As you can see here, the matte is a little harsh, but we haven't introduced too much magenta into the image yet. Drag your sampling line across some of the pixels in the background that are still visible. Don't worry about anything that's more than a few pixels outside the guy's outline; we can mask that out with a garbage matte. We're mostly concerned with the edge here.

With the background gone, we can switch to Clean Foreground Noise and sample any pixels in the interior of the matte that are transparent. In my case, Primatte did a pretty good job, and I don't see any foreground noise.

004_mattes_011

Switch back and forth frequently between looking at the color image and the alpha when you're keying. Some problems will show up in one that don't show in the other. For instance, it's often difficult to see noise in the matte if you're just looking at the color, but you won't see that dark fringe if you only look at the alpha.

I could keep noodling at the matte for a while, resetting and trying different sample points, but I think this is a reasonable result for my purposes. I switched to the Spill Sponge to get rid of the green and, like the Auto Compute button, it added far too much magenta. In order to get back some of the real color, use the Spill (+) mode. Sample a spot that is too magenta, and some of the spill suppression will be removed. You will probably have to do this several times in several areas to get the image back to a pleasing color. If you take it too far, you can switch to Spill (-) to gradually remove the green you've reintroduced.

I seldom use the Matte Sponge mode because it usually adds far too many colors back to the foreground, destroying the work I've done. Restore Detail can be used to get back edge pixels that have been removed from the matte, making them translucent. Generally, I find that Primatte is not the best tool for soft mattes on hair or fine detail. It's better than the other built-in tools, but Fusion really needs a decent new keyer.

Other Options

The good news is that it is possible to construct your own keyer and then save it as a Macro or, if you're good with programming, a Fuse. A Macro is a group of nodes that has been wrapped into a single node, frequently with a customized control panel. A Fuse is a custom node written in the scripting language Lua. Fuses and Macros are not plug-ins, so they work in the free version of Fusion, not just in Fusion Studio. In the Macros chapter, I'll show you how to construct a simple color difference keyer (similar to Keylight for After Effects and Nuke), but in the meantime, check out the Kick Ass Keyer (KAK) by Pieter Van Houte, aka SecondMan on the We Suck Less Forums.

We Suck Less is currently the most useful community of Fusion users I am aware of. They're friendly and enthusiastic, and many of them have been Fusion users for a very long time. They also archived and maintain the old VFXpedia wiki, which was the only source of documentation for Fusion before it was purchased by Blackmagic Design. VFXpedia contains numerous articles, tips, and downloadable tools for Fusion 6 and earlier, most of which work in later versions. Most of the scripts found there will need some updates, but the Macros and Fuses all work fine.

Next time we'll look at Fusion's Paint tools and learn some ways to create clean plates and remove unwanted elements from a clip.

Blackmagic Fusion: The Interface and Tracking

The Viewer Panels

Before we get into the lesson on Motion Tracking, let's take a walk through more of Fusion's interface. We have already had an overview of the Viewers, the Flow view, the Tools view, the toolbar, and the playback controls. There are quite a few other views and functions that you will find very useful as you grow as a compositor. Let's start with all of those buttons at the bottom of the Viewer:

Starting with the upper-left, we have the SubView button, labeled "SubV." This button turns on an information panel overlaid on the Viewer, as you can see in the upper-right corner of this image. Right now, it's set to display a waveform monitor, a very useful tool for viewing the luminance range of your image. The small triangle next to the button activates a pop-up menu, which you can use to choose the specific SubView you want to see.Continue reading →

The Student Loan Problem

As of the publication of this post, I have $81,411.15 in student loan debt, courtesy of the Art Institute of Colorado (whose parent organization, Education Management Corporation, has settled at least one large lawsuit alleging student loan fraud.) The debt paid for two years of tuition, supplies, room and board at a private, for-profit art school and was supplemented a bit by grants and scholarships. My back-of-an-envelope calculations suggest that a four-year degree at a state University would cost a similar amount. A student attending an in-state public University and living with their parents would spend about $36,000. A student living in a dorm at an out-of-state public University could spend upward of $135,000¹. So $80,000 does not represent an atypical debt for a student graduating within the last five years. I therefore consider myself well situated to talk about this topic.

It might be expected that I would be all for any plan to forgive student debt. Who wouldn't want $80,000 added to their net worth in one swell foop? The trouble is that such a move is just moving beans from one bowl to another. It relieves borrowers of their responsibility for their choices (bad, in my opinion) and potentially serves to increase the federal budget deficit (bad, in almost everyone's opinion). It would also spur an even bigger money grab by certain Universities. It's a bad idea, and not one that I see ever making it through the legislature, even if the Democrats had locked away the election.

I think, however, that there are some very fair ways of helping to reduce the burden on students.Continue reading →

Blackmagic Fusion: The Basics

The following article is a chapter in a forthcoming compositing textbook. There are references to chapters and appendices that have not yet been written. As those chapters are completed, I will link to them. For now, please be patient.


Last time we covered importing footage with Loaders, handling color space with the Gamut and CineonLog tools, the Viewports and Time Ruler, and rendering out the finished product with a Saver. There wasn't much in the way of actual finished product to be had, though, so this time around we'll construct a simple composite and learn about some of the most important tools in the compositor's kit: Merges, Color Corrects, Transforms, and Masks.Continue reading →

Anatomy of an Image

I am not entirely certain whether this information should be a chapter of its own, an appendix, or be sprinkled throughout the rest of the text. For now, I'm going to treat it as an appendix to Chapter One. Maybe it will just remain as an article on the blog to which the main text will refer by link and reference.


It's tempting when someone starts going on about LUTs and gamma and gamuts to let your eyes just glaze over and then ignore them until they go away. Unless you're one of the nerdiest nerds who loves to talk about bit-rates and logarithmic response curves, you're probably eager to skip all off the color management stuff and jump straight to the Art.

Bad news: You're going to have to learn at least some of the tech stuff if you're going to be a good compositor. You can be an adequate compositor without all of this, but who wants to settle for merely adequate? You don't have to learn it all in one huge bite like I'm giving it to you here. Read through it, but if you don't understand something right now, don't worry about it—you can come back and read pieces of it again once you've got some better context.Continue reading →

Card Wipe Transition for Fusion – Developer's Diary

The Glitch Tools macro collection I am working on is, in large part, based on a template for After Effects that we have at work. I am not sure where the template came from, probably VideoHive. In any case, one of the glitches depends on After Effects' Card Wipe transition. That's the kind of thing that Fusion doesn't have right out of the box, but that I think I can reconstruct using the 3d system. Since I have had a number of questions recently about macros and expressions, I thought it might be instructive to document my process on creating this thing.Continue reading →

Blackmagic Fusion: Getting Started

Getting Started in Blackmagic Fusion

These articles are mostly based on Fusion 8.2 build 2 running on Windows 7 with an nVidia GPU. There are some slight differences in the interface between Fusion 8 and previous versions. I will do my best to highlight them as they come up throughout this series. For the first couple of lessons I was using Fusion 7, though, so some of the screenshots will look a little different from your interface, regardless of which version you are using.

A note for Mac users: As is typical with most software, the Windows Control key is Mac's Command key. As I am sure you know that already, I won't continue to insult your intelligence by pointing it out every three paragraphs.

This lesson covers importing footage into Fusion, converting it to linear color, viewing and playing it, and exporting it back out in the file format of your choice. Where Fusion differs greatly from Nuke and After Effects, I will make note of it. Continue reading →

Blackmagic Fusion: Expressions

In order to get some early feedback on this chapter of a forthcoming book, I am publishing it early. There are references to other chapters that have been written but are not currently published. 


 

Computer Graphics is half art and half science. Some practitioners lean more one way or the other, but all of us blend both disciplines in our work to some extent. Expressions are one way in which you can bring a little bit of engineering into your compositing workflow. They allow you to automate animation, to connect the properties of one node to another, and to use programming techniques to make your tools more responsive and interactive.

In this article, I will take you through several applications of expressions and start to set up a foundation on which we will build a solid understanding of tool development, scripting, and automation in Fusion.Continue reading →

Face Replacement with PFTrack, 3DS Max and Fusion

The first year of my time at Muse VFX was spent almost exclusively on the Disney's Dog With a Blog. For those not familiar with the show, it's about a talking dog that (surprise!) has a blog. In a half-hour sitcom, there were typically 30 – 40 shots of Stan with a CG face. By the time Season 3 came around, we had a pretty bullet-proof and fast pipeline in place.

1280x720-ODd
G. Hannelius as Avery Jennings and Mick as Stan the Dog. Copyright It's a Laugh Productions and Disney Channels Worldwide. Stan's brows, eyes, muzzle and jaw are CGI. His ears, nose, and parts of his cheek are real.

I won't give you all of the pieces here, but I would like to describe one particular technique that made many other things possible.Continue reading →