AOL Image Organizer Applies Algorithms to Everyday Pictures

I was wandering through the AOL Greenhouse this afternoon, and I came across a "Freshly Planted" application that immediately caught my attention. The working title for the app is "Image Organizer." Image Organizer was developed by the "Image Competency Team," certainly an interesting title for an organization. Here's the application description:

The Image Organizer helps to automatically organize your photos based on which individuals are in them

This, of course, could mean a lot of things, from the mundane (you manually identify who's in each picture) to the truly exciting (an image pattern recognition application). A lot of scientific programming work I've done has applied pattern recognition techniques to various incoming data streams, and I've been waiting a long time for serious advances in this area with respect to arbitrary everyday images.

Pattern Recognition and Artificial Intelligence

The Image Organizer download page doesn't provide a lot of information about how the application works, but the summary blurb makes it clear that the application is applying some advanced pattern recognition and artificial intelligence techniques to automate identification of individuals in images:

Once you select an album to organize, the Image Organizer will automatically group your images into sets based on whether the images seem to contain the same person. You then can label a few examples of different people, and the Image Organizer will label automatically other faces in photos that it determines are the same people as those that have been labeled previously. It also will sort automatically these photos into different virtual "face" folders.

Sounds a bit like science fiction, or the kind of application national intelligence and defense agencies might be working on, doesn't it?

Can Software Really Recognize Faces?

A big question, of course, is: "Is this possible?"

Having worked for many years on scientific data analysis, it's easy for me to predict that the application will work great in some cases, less great in others, and that it will fail in some cases as well. It's a problem of signal and noise. The "signal" is the portion of the image that is recognizable as the person's face. The "noise" is anything that detracts from the clarity of the signal.

A fuzzy border between the person's face and the background will also be a problem for an algorithm that is attempting to identify a face. The algorithm must find and identify a "object" that "stands out" from what may be a very cluttered background. If the algorithm can't identify any "face-like" objects in an image, then it has nothing to work with for comparing that image with others.

In other words, the task that Image Organizer attempts to accomplish is very complex. Machine image recognition has proceeded at an almost unexpectedly slow pace compared with other computer-related developments precisely because the task is so complex. Human visual perception almost instantaneously analyzes the incoming visual data and snaps it into a collection of defined objects (see Owen Barfield's book Saving the Appearances if you're interested in this subject). This happens due to a process that begins in infancy as soon as we open our eyes (at which point there are probably no objects, just a lot of strange sensations).

A computer program "observing" an image is like the infant. The data is all meaningless, unless some organizing principle is applied. The organizing principles are mathematical algorithms.

Getting and Running Image Organizer

We can't pop the hood and look at the mathematical methods that are integrated into Image Organizer -- but we can certainly give it a test run.

The download and installation is quick and painless. Launching the application brings up a window that displays "AOL pictures" on the top bar. So I guess you can use this with your AOL Pictures albums. I tried it out with some directories of digital pictures I have on my laptop.

I initially selected a directory that contained 48 pictures from a recent vacation. Image Organizer popped up a dialog window letting me know that it might take about 5 minutes to process every 100 images the first time (depending on image size). I clicked OK and let the program run.

A process time line is displayed as the software analyzes each image, with a label that estimates how much time is remaining for the processing. It took about the predicted amount of time to process my 48 relatively large (2048 x 1536 pixel) images.

Image Categorization Results

Then the application showed pictures of 11 faces it distinguished from among the entire batch of 48 pictures. In fact, there were only four people in the pictures, plus a chihuahua. There were 3 faces where the application said "Don't Know," meaning perhaps that it wasn't sure if saw a face or not.

There was also a "Never Label" category, which had all the images where the algorithm did not see a face. These included all of our nature shots, some back views of people, images of our chihuahua, and a few images where a person was facing the camera but from a long distance. The algorithm also seemed fooled by sideways images of people.

Clicking on one of the identified faces brings up thumbnails of the actual images in which that face was identified. In my testing, there was never more than one image per group. I wondered if this meant that the application does not attempt to match faces within the batch of pictures.

Partly Automated, Partly Manual

I think this much actually be the case, because the run time would increase exponentially if each face was compared with each other face identified within the image set. Whereas, in fact, the processing time for Image Organizer increases linearly with the number of pictures in the batch.

Within a single album, Image Organizer finds faces in pictures, but it doesn't attempt to match faces that are in different images. Instead, it provides an interface where the user can name each face. Then Image Organizer groups the pictures using the name tags supplied by the user. So, its categorization is partly automated and partly manual.

But wait, there's more!

But wait! When I started flipping back and forth between picture albums (directories), Image Organizer flashed "New!" and seemed to be telling me it had found another picture of a previously identified person. This suggests that there is indeed some form of image comparison and reprocessing that happens after you identify a set of pictures as being the same person.

It looks like if you've already identified some faces and given them names, then when Image Organizer processes (or reprocesses) a directory/album of pictures, it does indeed compare the faces in those pictures with the faces you've identified. As I said above, this is a very difficult task. Hence, I wasn't surprised to see a few humorous results (such as when the program told me it had found a new picture of one of the people I had identified, and it showed me a picture of a rock instead).

It would take too long for me to try to definitively reverse-engineer everything Image Organizer does, or attempts to do. Suffice to say, it's a nifty and quite intelligent little application. It's certainly a lot of fun to play with and use!

-- Kevin Farnham
O'Reilly Media