2007 TopCoder Open: Algorithmic Dreams

I've been thinking about the upcoming 2007 TopCoder Open competition over the past week or so. Registration is open now. The 80 competition finalists will spend June 26-29 at the Mirage Hotel and Casino in Las Vegas, Nevada. I trust that the winners--especially of the algorithm contest--will clearly understand who the odds favor, should they decide to wander into the Casino after the competition.

Algorithm Competition Tops the Line-Up

The headline TopCoder competition seems to be the Algorithm Competition. I watched the 2006 TopCoder Collegiate Challenge finals online this past November. The Algorithm Competition was shown live, and it was a remarkably interesting event, with almost an Olympic-like quality (induced partly through the manner in which the event was covered by the TopCoder "sports-casting" crew).

You can find links to video clips from the 2006 TopCoder Collegiate Challenge finals by going to the AOL Developer Network Search page and searching for TopCoder. Right now, almost everything that's returned will be in reference to November's Collegiate event.

Who Thinks TopCoder Is Important?

The TopCoder sponsor list always inspires me. For the 2007 TopCoder Open, there are five major sponsors. Three of the sponsors were also sponsors of the 2006 Collegiate Challenge (which had four primary sponsors). The organizations who sponsored both events are:

• AOL: "a global Web services company that operates some of the most popular Web destinations and offers a comprehensive suite of free software and services."
• National Security Agency: "Our adversaries do their best to keep their plans a secret. We work to uncover those secrets, and keep our own secrets safe."
• UBS: "one of the largest financial services firms in the world, a leader in equities, corporate finance, foreign exchange, derivatives and risk management."

I like thinking about those three: AOL, NSA, and UBS. They are all so different in their mission, in where they are "coming from," yet all three have decided that TopCoder is well worth their investment dollars. All of them are actively seeking to employ the type of people who compete at TopCoder, and especially, of course, the top competitors.

The new sponsors for the 2007 TopCoder Open are:

• DRW Trading Group: "an aggressive, top-tier proprietary trading firm striving to optimize discovery and capture edge in the capital markets."
• VeriSign: "operates intelligent infrastructure services that enable and protect billions of interactions every day across the world's voice and data networks."

No slouches there, either.

So, if you look at the five sponsors, you've got a company that has one of the largest footprints on the Internet (AOL), two financial securities companies (UBS and DRW), a company focused on financial transactional security (VeriSign), and an organization that is focused on National security (NSA).

The Data Blur

How can it be that the ability to solve problems involving algorithms is so useful to this somewhat surprising and genuinely diverse community of organizations? I have quite a lot of personal insight into this question, since I've worked with algorithms for the entirety of my career as a software engineer. About two thirds of my work has been scientific programming (mathematical modeling, simulation, automated data analysis), and about one third was corporate programming (including financial market analysis and document indexing). Almost all of my work for the past 15 years has involved developing applications capable of analyzing very large volumes of data, categorizing that data, identifying significant events within the data, etc.

Today, moreso than any point in the past, we are faced with enormous amounts of data. As computers have become more powerful, as instrumentation has become more capable of gathering more data faster, as the creation of data by humans on the Internet has exploded, the data that exists for potential analysis looks like a blur to the human eye. How can anyone detect what is significant amid the endless streams? This is the case whether you're talking about a constellation of satellites each capable of gathering terabytes of raw data hourly, or about billions of instant messages, emails, and web pages of content in forms that include text, images, audio files, and videos. It is a blur that will leave any human individual dumbfounded if they are asked to "sort it out and tell me what's important."

Why Algorithmic Problem-Solving Is Important

So, how can organizations approach this blur of data and attempt make sense of it? What happens if they can't make sense of it?

• At UBS and DRW, the financial consequences of failure are dire. The incoming financial data must be properly analyzed, the "signal" must be detected within the blur of "noise."
• At the NSA, failure to notice and properly decode messages between terrorists could result in the deaths of thousands or even millions of people as they were simply going about their daily business at their jobs or at schools and universities.
• A tiny, brief security breach at VeriSign could quickly compromise the privacy of hundreds of thousands of people and necessitate cancellation of millions of transactions.
• AOL, meanwhile, is competing with powerhouse companies like Google and Microsoft. Failure to sufficiently "understand" the nature of the Internet means the end of relevancy.

Google is the epitomizing example of a company that becomes dominant through creation of an algorithm that finds a signal in an unstructured blur of data (the Web) that no one previously could detect. AOL, Microsoft, Yahoo! -- they do not intend to be left behind.

In fact -- isn't that the key? None of these organizations can afford to be "left behind." If they are left behind, the consequences, in all cases, potentially include the destruction of the organization and of the assets and/or community it sought to advance or protect.

Algorithms Are the Key to Understanding

Algorithms are the key to understanding, to differentiating what's meaningful from what doesn't really matter, distinguishing the significant (but sometimes very faint) patterns within the ocean of available information. No human can look at it all. The best we can do is to design methods that "study" the data and notify us when a pattern we have identified as being significant is discerned within the data.

TopCoder brings together a group of people who have proven themselves very adept at this type of problem solving. That's why these five organizations have decided that the TopCoder competitions are very important events, support of which increases their ability to accomplish their unique strategic objectives. In TopCoder, they see a structure that brings together, and ultimately offers, hands and minds that may unlock the secrets hidden within tomorrow's data blur -- which will surely be more like a horrific blizzard from today's point of view: far more dense, dynamic, and chaotic than anything we've seen thus far...

-- Kevin Farnham
O'Reilly Media