You don't really, truly understand Brewster Kahle until you've had him show you the server farm in Alexa Internet's basement. Walk down a flight of outdoor steps at the side of a old military personnel processing building in San Francisco's Presido, and you'll see an entire universe of data -- or at least a bank of dark-toned Linux servers arrayed along a twenty-foot wall. The room itself -- moldy concrete, with a few spare windows gazing out at foot-level -- might have held a lawn mower and some spare file cabinets a few decades ago. Now it houses what may well be the most accurate snapshot of The Collective Intelligence anywhere in the world: thirty terabytes of data, archiving both the web itself, and the patterns of traffic flowing through it.
As the creator of the WAIS (Wide Area Information Server) system, Kahle was already an Internet legend when he launched Alexa in 1996. Described as a "surf engine," the Alexa software used collaborative-filtering-like technology to build connections between sites based on user traffic. The results from its technology are showcased in the "Related sites" menu option found in most browsers today. Amazon.com acquired Alexa Internet in 1999, but the company remains happily ensconced in its low-tech Presidio offices, WWII temporary structures filled with the smell of the nearby eucalyptus trees.
During our half-hour conversation in Alexa's makeshift conference room, Kahle jumps up excitedly at several points to sketch a graph out on the whiteboard. We speak about the large-scale trends in Web traffic, the history of libraries, and how to build a business model for small publishers. After our talk, he takes me down to the basement to see the servers. "In just three years we got bigger than the Library of Congress, the biggest library on the planet," he says, arms outstretched, smiling. "So the question is: What do we do now?"
-- Steven Johnson
