SENSOR ARTIFACTS AND CMOS ROLLING SHUTTER
by Barry Green

While most of the camcorders on the market have traditionally used CCD sensors, many new camcorders are coming on the market are now using CMOS sensors. While CMOS and CCD both have unique properties that make each one suitable for one type of job or another, there are unique characteristics to each that need to be understood in order for a user to know whether a CMOS or CCD camera is more appropriate for the type of shooting they’ll be doing.

All Sensors Are Not Created Equal

Choosing a CCD or CMOS system can have a very real impact on the type of footage you can gather, as well as ancillary impact such as the battery life of your camcorder (CMOS sensors are typically quite a bit more energy efficient, needing less electricity to power them and generating less heat than a comparable CCD.) CMOS sensors can scan and offload their footage quicker, making CMOS a more appropriate choice for high-speed cameras, for example. While CMOS and CCD sensors do the same basic job (gathering light and turning it into a video image), they go about it in different ways, and the differences can have very significant impact on your footage.

A full discussion of how the two differ is not in the scope of this article; in this article we’re going to instead focus on the different artifacts that each sensor causes. There are four main sensor artifacts we’re going to explore: Smear, Skew, Wobble, and Partial Exposure. Your type of shooting may or may not encounter these, but any prudent shooter should be educated as to what they are, how they affect your footage, and what circumstances cause them to happen. “Forewarned is forearmed,” and having an understanding of typical sensor artifacts may keep you from potentially ruining footage that may otherwise have turned out well.

Smear, Skew, Wobble, and Partial Exposure

There are four main characteristics sensors may exhibit. CCDs can suffer from vertical smearing on bright light sources, while CMOS sensors are immune to that artifact. But CMOS sensors (equipped with “rolling shutters”) can exhibit skew, wobble, and partial exposure; CCD sensors are immune to those effects. And a CMOS sensor with a “global shutter” would also be immune to them, but since no current CMOS camcorders are equipped with global shutters, a camcorder buyer needs to be aware of what the implications of a rolling shutter would be.

Before we go into what the various artifacts are, let’s first examine the concept of a “rolling shutter” versus a “global shutter” and why it makes such a difference in the images that each produces. (For point of clarification, CCDs use a “global shutter”; CMOS can use either “rolling” or “global” but as a practical matter all the camcorders on the market use the “rolling shutter” technology).

A GLOBAL SHUTTER exposes the entire imager simultaneously. The entire frame is exposed and begins gathering light; when the predetermined “shutter speed” has elapsed, the sensor stops gathering light and turns its current exposure into an electronic image. There is no physical “shutter” that covers and uncovers the sensor; it’s all done with timing. At the start of exposure the entire sensor starts gathering light; at the end of exposure the light-gathering circuitry is turned off and the contents of the sensor are then “read out” to become an image.

Here’s a graphical illustration of basically how a Global Shutter would expose an image (of course, this example is in extreme slow motion; in reality the entire process would happen in typically 1/60th of a second).

A ROLLING SHUTTER is very different. The rolling shutter actually exposes different portions of the frame at different points in time, “rolling” through the frame. Again, it’s not an actual physical moving shutter that’s doing this (as opposed to a movie camera, which actually has a moving physical shutter). Instead, the sensor is telling different portions to become light-sensitive at different moments in time, and as this process proceeds down the course of the full frame, until the entire frame is exposed. Here’s a rough simulation of how a Rolling Shutter system works:

(again, this is of course shown in extreme slow motion; in actual practice the entire process would occur in usually 1/48th to 1/60th of a second).

In order to understand the various image artifacts that arrive, the important thing to note here is that in the rolling shutter, different portions of the frame are exposed at different times than other portions. If the subject or the camera were to move during the exposure, the result would be reflected in the frame as one of the three Rolling Shutter Artifacts (Skew, Wobble, or Partial Exposure). For example, here’s a simulation of what would happen if a rolling-shutter camera were to pan horizontally during exposure:

A Global Shutter avoids that problem; instead with a global shutter any motion during the frame would result in blurring of that motion.The amount of blur in a global shutter is dependent on the shutter speed; blur can only occur if the shutter’s speed is slow enough that the object actually can move during one shutter cycle, and with a high shutter speed that’s typically not possible and any motion will be frozen in time, resulting in crisp blur-less motion. Skew and other rolling shutter artifacts aren't affected by shutter speed, they're affected by the frame rate. The faster the frame rate, the less skew you'll see; the slower the frame rate, the more exaggerated the rolling shutter artifacts wil be."

Jason Rodriguez, of the engineering team behind the Silicon Imaging SI-2K CMOS camera, gives a great rundown of the differences between how a global shutter scans and how a rolling shutter scans, and explains why lower-cost cameras are more likely to have rolling shutters, HERE

Smear
Smear occurs when a very bright portion of an image causes an entire column of pixels to overload and bloom to white. Here’s an example of vertical smear.

Any bright point of light can potentially cause smear; common offenders include street lights and car headlights; it can also happen if the camera is shooting footage of a camera flash, or even the sun. Avoiding smear involves lowering the exposure enough that the bright lights don’t bloom and trigger a column of smearing; stopping down the iris to bring down the brightness of the bright lights can eliminate smear entirely, but also may cause the overall picture to be too dark. Smear is also one of the “dead giveaways” that your production was shot on video rather than on film; film doesn’t “smear” like this. Controlled lighting can eliminate all traces of smear, but in uncontrolled circumstances it’s going to happen. CMOS sensors function differently and are immune to smear.

Here’s an example picture of a chandelier, where the lights are blown out to the point of provoking vertical smear. In the CCD picture you can see greenish lines running from the lights through to the bottom of the frame. CMOS sensors are immune to this smear artifact, so in the CMOS picture you’ll notice no such lines.

CMOS
CCD


Skew
The easiest way to see Skew is to think about taking a photo of a tall building. As the sensor scans down the frame, the building is “drawn” onto the chips and is represented accurately. But if the camera were to pan horizontally while the sensor was still exposing, what would happen? The top of the building would be on one side of the frame, and the bottom of the building would be on the other. The whole building would look like the Leaning Tower Of Pisa! This is called “Skew” and is one of the more prevalent Rolling Shutter effects. With horizontal camera motion, vertical objects can look like they’re “leaning” one way or the other depending on which way the camera was moving.

HERE (click to view) are some still photos of the Skew effect

For motion examples, you can watch THIS (right click to download).

Skew is probably the least objectionable of the rolling shutter artifacts, since it typically only exhibits during a fast pan or with a slower frame rate. However, skew can play havoc with motion tracking for special effects compositing, so if you plan on doing compositing it’s vital to be aware of the effect that skew can have.

Wobble
Probably the most-talked-about rolling shutter effect is “wobble.” Wobble is related to Skew, and is caused by the same root causes, but Wobble is far more troublesome. Whereas Skew represents a leaning of vertical objects, Wobble is a stretchy/rubbery look that happens to the video. Wobble is more likely to occur in handheld footage, or in situations where the camcorder is subject to vibration or sudden motion.

To imagine how Wobble might affect the scanning of an image, picture the idea of shooting a scene while tilting the camera upwards. If the camera moved fast enough, it may be possible for you to “race” the rolling shutter down the frame, and thus the camera would “stretch” any object that was being filmed – it’d keep repeating that same section over and over, pulling and stretching the object down the frame. Or when moving down, the image might become artificially shortened or compressed. Now, let’s throw in the concept of handheld motion (a little up, a little down, some tilting or diagonal movement, etc) and you can see where the “wobble” comes in: as the camcorder is moving downward the image becomes stretched out; when it moves upward the image gets scrunched down, and as it moves side to side it leans one way or the other. The result is that your entire frame becomes “wobbly”; a way to describe it might be to say that it looks like you’re shooting through a layer of Jell-O™ .

For a relatively static shot, this is obviously not a big deal. For a camera mounted on a tripod it shouldn’t be a problem. But the more motion that’s involved, and especially motion that reverses itself (like a bump or jolt), the worse it can become – or, for unintentional motion it can become a problem. HERE (right click to download) is an example of a shot on a tripod, where the tripod is occasionally “jiggled” by wind.

A sturdier tripod might help avoid that wobble. Another example is far more concerning; this is an example of the vibration caused by shooting from a helicopter. If you can imagine the stretching/shrinking effect, combined with the Skew effect, all running at very high frequency due to the fast vibration of a helicopter, then you can see why the footage turned out like THIS (right click to download).

This is perhaps the most extreme Wobble anyone’s encountered. The shooter is Russ Andersson, of Andersson Technologies, makers of SynthEyes. He re-shot this footage using a CCD-based Canon XHA1 and the same mounting technique, and he says “the results are watchable even before stabilization.”

Can wobble be corrected in post? Unfortunately, no. Regular Skew probably could, but the stretchy wobble can’t be automatically compensated for. HERE (right click to download) is an example of wobbly handheld footage shot by user “Norbert” and the results of processing the footage through a post-production image stabilization product, SteadyHand.

SteadyHand can stabilize a lot of normal footage shake, but it can’t deal with the stretchy/squishy effect of the wobble.

Chia-Kai Liang wrote a research paper attempting to correct for rolling shutter issues; an abstract and examples are HERE (click to view).

His CONCLUSION (click to view) is that it can’t be compensated for automatically.

Obviously not all footage is ruined by excessive wobbling; it depends largely on the type of motion of the camera and the speed and direction of that motion; direct linear motion (such as a tripod pan) shouldn’t cause any wobble at all (just skew), but handheld footage or footage subject to vibration is just going to wobble. Slower frame rates seem to accentuate the wobble, faster frame rates seem to minimize it. Higher shutter speeds make it more distinct, and slower shutter speeds mask the wobble under motion blur, but the wobbling effect is equivalent regardless of shutter speed. The amount of wobble you get all depends on how the camera moves during a shot, and how quickly it moves (or, how quickly the subject is moving; the wobble doesn’t distinguish between camera movement and subject movement!) Under most circumstances you shouldn’t see or notice wobble at all, but as the helicopter footage shows, there are just some types of shots that wobble will ruin.

Where wobble can be most troublesome would be for those wanting to use a CMOS camera for special effects work, particularly motion tracking or motion capture (hence Russ Andersson’s exhortation that if you want to track objects, you should completely forget about using CMOS/rolling shutter cameras, HERE [click to view]).

How can you control wobble? That’s a tough one, because the skew and wobble is inherent to the rolling-shutter technology. Faster frame rates will exhibit less wobble than slower ones (which means you’ll see more wobble at 24P than you will at 60i, for example, and it becomes very obviously noticeable if you attempt to slow down your footage using slow-motion processing in post). But the stretching, skewing, scrunching and leaning are endemic to the nature of the rolling shutter. To eliminate wobble you would need to either a) eliminate all motion, or b) use a global shutter system. Since neither of these are practical options for a video camera using a CMOS sensor, you will have to determine if the amount of wobbling that happens in video is acceptable to you. If it is, and serves your purposes, then CMOS would be a viable option. If it’s not acceptable to you, you would be better served looking at a CCD-based product.

Partial Exposure
Partial Exposure is perhaps one of the more pernicious but least-encountered image artifacts. It usually occurs under circumstances where there’s a flash of light that significantly alters the exposure – a camera flash, a bolt of lightning, fireworks, etc.

Here’s what it looks like:

In these cases, if you look at how the rolling shutter system works, you can easily see how part of the image would be dark and the rest of the image would be bright – as the shutter “rolls” through the frame, it will be exposing portions of the frame at the prevailing light conditions, and then when the “flash” occurs, the current portion of the frame that the shutter’s exposing will be brightly lit. The result is that when you use a rolling-shutter camera in a scenario where flash photography is taking place (such as a wedding, a press conference, a red-carpet premiere, etc) it’s possible that you’ll encounter black or dark bands, or even bright bands, in your video.

Here are some more examples of flash-induced Partial Exposure artifacts:
DVINFO (click to view)
DVINFO (click to view)

Fortunately, this black-banding artifact will only affect one frame of video at a time. Unfortunately, there’s not a thing you can do about it – you’re at the mercy of the flashes. The longer the flash duration, the more even the exposure will be; the shorter the flash, the more banding will occur. If it’s an occasional flash, it may not be much of an issue. But if you’re at a red-carpet event where many flashes are going off frequently, or shooting in a nightclub under flashing lights, it may be more problematic.

Another dangerous situation for rolling shutters is slow-refresh fluorescent lighting. Modern fluorescent lighting uses high-frequency ballasts and produces a largely continuous stream of light, but older fluorescents used quite slow ballasts and as such, the intensity of the light could vary significantly over the course of 1/60th of a second. This varying intensity can play havoc with a rolling shutter camcorder. Be very cautious about shooting under fluorescent lighting with a rolling-shutter camcorder, and try to use only high-frequency-ballast fixtures with a rolling-shutter camcorder.

Slow ballasts could also prove problematic for global shutters, if the shutter speed is not in phase with the cycle of the light; that’s why it’s recommended to shoot using the same shutter speed as the lights (i.e., in PAL territories you’d use 1/50th exposure to match the 50Hz lights, in NTSC territories you’d use 1/60th to match the 60Hz lights). A Global Shutter system running with a shutter speed that’s out of phase with the lights (50Hz shutter under 60Hz lights, for example) can exhibit some general flickering or color shifting under fluorescent lights. The Rolling Shutter system exhibits a very different artifact – bands of shifting color scrolling in the image, or darker lines flickering in and out on the image. It’s easy to correct for this circumstance with a global shutter – just match the shutter speed to the light frequency. But it’s not possible to do so with a rolling shutter system – instead, you’d have to turn off the offending lights and substitute in your own, high-frequency or constant-source lighting (constant source being a non-ballasted light system, such as tungsten lighting).

Here’s an example video of how a rolling shutter system deals with a flickery fluorescent light:

Summary
The CCD artifact of Smear is well known, and for the most part shooters know how to deal with it and minimize or avoid it. But the new artifacts/characteristics that come from the new CMOS technology are likely to provide some nasty surprises for unprepared shooters. And camera manufacturers are likely going to continue to offer more and more CMOS camcorders. Which means we’re likely to see more and more rolling shutter issues crop up, as the technology becomes more widespread. Shooters, educate yourselves as to what the issues are, how they affect the footage, and prepare yourselves for whatever steps you need to take to make sure that there are no unwelcome surprises in your footage.

For further reading, Point Grey Research has published an ARTICLE (click to view) describing the differences between the rolling shutter used in CMOS sensors and the global shutter used in CCD sensors.

Dalsa has also published some excellent articles discussing CMOS and CCD sensor technology:
ARTICLE ONE (click to view)
ARTICLE TWO(click to view)
ARTICLE THREE(click to view)

Discuss this in the Forum