Apollo Hosting- Company Blog

Rushing back in on familiar ground, design. Over my little sabbatical I discovered a couple really amazing projects in the works that, some day, will make bloody awesome Photoshop plug-ins. In development at Carnegie Mellon University, these two complimentary technologies promise to make short work of previously painstaking photo alteration tasks in Photoshop. The original article hails from Science Daily (which certainly appears to be an "ad-supported property"), but Carnegie Mellon hosts websites for each.

Scene Completion

Getting rid of unwanted image elements- trees, cars, your thumb, ugly relatives- is a common task that falls under the "retouching" umbrella. The slang term would be "Photoshopping", but one certainly doesn't have to use Photoshop to do it. Like most things the 500lbs. gorilla does, Photoshop does it pretty well, though. Accomplishing this kind of retouch is a fairly straightforward task that involves removing the unwanted elements via replacing them with other elements from the same photo, or a very similar photo.

Though the task is straightforward, accomplishing it is where the talent of the individual artist plays a part. Selecting and removing elements in a highly convincing manner is something that takes time and focus. Carnegie Mellon's researchers hope to alleviate us of this tedium with their Scene Completion project. Using a database of a millions images drawn from your favorite and mine, the web, their algorithm uses basic masks to fill in content over areas the user wishes to eliminate.

The image selection set examples provided on the site indicate these masks don't have to very precise, either. Precise masking is yet other one of the retoucher's trades, but Scene Completion seems to run on what any retoucher worth their salt would consider heinously loose masks. The system apparently requires no human input outside the mask, as the images in the library are understood semantically.

Experienced retouchers know that choosing the right photos to import elements from is very important. Trying to combine elements from images with different lighting and composition will result in a lackluster end product. For instance, trying to add elements from a scene photographed outside to an interior shot makes a convincing composite very difficult. Scene Completion understands this and chooses elements based on their suitability to be added to the final image.

One of the more fascinating aspects of the project is the ability to create a variety of different alternative replacements for the mask. Perhaps a future version with a large database from which to choose could literally have a randomization function, or the ability to specify a preferred type of replacement. Want to replace those trees with building, a different set of trees, or your ugly relative?

Suffice to say, very exciting stuff, even if the rather equation-y parts of their PDF file made me a little queasy. What's a guy gotta do to get away from calculus these days?

Photo Clip Art

The second project, Photo Clip Art, is also focused on image touching. Instead of replacing elements in an image, it is designed to add them. Adding scene elements can be just as arduous a task as changing or removing them. The same rules apply, for best results, the source material and the destination image must have similarity of light and composition in order to create the most convincing product. Here masks matter a lot, as the more detailed a mask you use on the image element to be composited will determine how convincingly is meshes with the target image.

The Photo Clip Art project takes the task out of our hands. Built on a 3d interface that allows users to simply point and click where they want elements to appear, Photo Clip Art presents the user only with a selection of pre -masked elements suitable to be placed in the image. The user doesn't have to hunt down images with similar composition or alter his inserted element to meet the requirements of the final composite.

The site examples confine themselves to "populating" mostly empty streets with new inhabitants, most of them a good match for the image into which they were inserted. Were I to pick nits, I would say the masking on the inserted elements could use some work. That is a minor concern, though, as the point of the exercise wasn't to display excellence in masking technique, but to show the individual elements worked within the context of the images into which they were placed. Though more hit or miss than the Scene Completion examples, the results still showed promise.

If the technology can be exploited to go beyond "people population" it will make for a great tool. Adding in elements like trees or boats to large outdoor scenes, or small elements like books or knick-nacks to internal scenes would make for an entertaining way to touch up photos. Instead of replacing your ugly relative with Scene Completion, just drop a tree (or a building if she's really ugly) on top of her with Photo Clip Art and call it a day.

Tech Upgrade, Fun Downgrade?

The technology in both these project is amazing, though, as someone skilled in the "retouching arts," I wonder if they would eventually take the fun out of it. Sure, I might have spoken a bit disparagingly about the difficulty of creating a good composite, but I'll admit, the challenge can be part of the fun. It's fairly common to see retouching approached like art, and much of it done for that purpose entirely, be it to humorous or prurient effect. Tools like this could bring very high quality work to the "masses", who would ordinarily be going "woo" at my amazing Jedi Master Photoshop Skill.

On the other hand, technology has repeatedly proven a poor substitute for the human hand in the arts, electronic or otherwise. Witness Adobe's continued efforts to build "smart" selection tools that still just don't work like you want them to. Hammering out the perfect mask from channels, filters, and manual touch-ups is a long, painstaking process, but the end result is better than anything the software could come up with.

Even in what could be considered a "pre-alpha" state, these tools are impressive, and have a lot of room for improvement. Still, I don't think they'll completely run the retouching community out of business. Humans will probably still do it a little better. I'll certainly want to try them out. The creative possibilities are very enticing, and doing a little hand touch-up of the filter's work will probably produce great results. Just guessing on the 2010 "street date", of course. With any luck it'll be even sooner.