April 2011

Abstract

As a final project for my Computational Photography class at Georgia Tech, I created an application that uses a single video camera to create tonemapped HDR images in real time. This project’s inspiration comes from Soviet Montage Production’s DSLR HDR video.

The app runs on Linux PCs (using USB or Firewire cameras) and Android phones (using the built-in camera). Custom OpenCV code manages the different exposure images, and generates a basic HDR image. The HDR image is fed to a tonemapping algorithm by Mantiuk et al, creating either a ‘ghostly’ or ‘painterly’ effect.

An example of the Mantiuk tonemapping effect made with a DSLR camera and Luminance-HDR can be found here.

Some example project result images and videos can be seen below.

Videos

*Full Length Demo*

Contrast Equalization

Contrast Mapping

Sample Images

contrast mapping

contrast equalization

Android Camera Screen Capture-- Mid-exposure Image (top) | Contrast Mapped HDR (bottom)

Android Camera Screen Capture-- Mid-exposure Image (top) | Contrast Mapped HDR (bottom)

Implementation Details

Three images are captured from a single camera, each with varying exposures (low, medium, high). These are then added into a single HDR image by adding the logarithm of each image. The HDR image is fed to a tonemapping algorithm by Mantiuk et al, taken from the Luminance HDR project, and modified for this application.

Mantuik et al’s tonemapping operator can work in different two modes: contrast mapping, or contrast equalization. Both methods are fairly computationally intensive, requiring severe down-scaling of the raw camera images to keep processing time reasonably fast.

On Linux, USB cameras are supported (and captured) by OpenCV, while Firewire cameras are handled by a custom libdc1394 wrapper. Most USB cameras only support changing the brightness (not exposure), which generates a faux-HDR image that then gets tonemapped. An AVT Guppy machine vision firewire camera was also used for testing, because it allows changing the shutter speed and adjusting gain. This camera produced much better results than any webcam tested.

On Android, the built-in camera was controlled via Android camera APIs in Java. Unfortunately, there is a massive delay between setting the exposure, and when the camera actually gets to that exposure. After each exposure change, an arbitrary number of dummy frames are discarded before grabbing an image, in attempt to give the camera time to adjust. This waiting for the camera’s exposure change takes up about as much time as the actual processing, exaggerating an already slow image processing loop.

Application Details

Pros / Features:

• Single camera, live High Dynamic Range Image viewing
• Mantuik tone mapping operators
  • Contrast Mapping (faster, exaggerates shadows, darker)
  • Contrast Equalization (slower, exaggerates colors, brighter)
• Cross platform (Android / Linux / with a little work, Windows)
• Various camera support (USB / Firewire / Android)
• No image alignment pre-processing needed (assuming little camera movement)
• OpenCV + OpenMP

Cons / TODOs:

• Very low resolution
• Low frame rate (exposure change time limits frame rate)
• Android’s camera exposure change is terribly slow
• No fancy GUI
• Manual adjustment of camera settings required (trial-and-error based)
• Results are extremely dependent on quality/extent of the camera’s exposure changes (quality = actually getting to the desired +/-2 EV)
• Port code to GPU
• Use a faster tone-mapping-operator

Resources

All the code for this project can be found in my Google Code repository:
ViewerCV (Android)  |  rttmo (Linux)

Download ViewerCV on Android Market.

Related Android computer vision post.

Class presentation slides.