Open-Source Spying

http://www.nytimes.com/
By CLIVE THOMPSON
Published: December 3, 2006

Excerpt from "Open-Source Spying"

. . . Intelligence heads wanted to try to find some new answers to this problem. So the C.I.A. set up a competition, later taken over by the D.N.I., called the Galileo Awards: any employee at any intelligence agency could submit an essay describing a new idea to improve information sharing, and the best ones would win a prize. The first essay selected was by Calvin Andrus, chief technology officer of the Center for Mission Innovation at the C.I.A. In his essay, “The Wiki and the Blog: Toward a Complex Adaptive Intelligence Community,” Andrus posed a deceptively simple question: How did the Internet become so useful in helping people find information?

Andrus argued that the real power of the Internet comes from the boom in self-publishing: everyday people surging online to impart their thoughts and views. He was particularly intrigued by Wikipedia, the “reader-authored” encyclopedia, where anyone can edit an entry or create a new one without seeking permission from Wikipedia’s owners. This open-door policy, as Andrus noted, allows Wikipedia to cover new subjects quickly. The day of the London terrorist bombings, Andrus visited Wikipedia and noticed that barely minutes after the attacks, someone had posted a page describing them. Over the next hour, other contributors — some physically in London, with access to on-the-spot details — began adding more information and correcting inaccurate news reports. “You could just sit there and hit refresh, refresh, refresh, and get a sort of ticker-tape experience,” Andrus told me. What most impressed Andrus was Wikipedia’s self-governing nature. No central editor decreed what subjects would be covered. Individuals simply wrote pages on subjects that interested them — and then like-minded readers would add new facts or fix errors. Blogs, Andrus noted, had the same effect: they leveraged the wisdom of the crowd. When a blogger finds an interesting tidbit of news, he posts a link to it, along with a bit of commentary. Then other bloggers find that link and, if they agree it’s an interesting news item, post their own links pointing to it. This produces a cascade effect. Whatever the first blogger pointed toward can quickly amass so many links pointing in its direction that it rockets to worldwide notoriety in a matter of hours.

Spies, Andrus theorized, could take advantage of these rapid, self-organizing effects. If analysts and agents were encouraged to post personal blogs and wikis on Intelink — linking to their favorite analyst reports or the news bulletins they considered important — then mob intelligence would take over. In the traditional cold-war spy bureaucracy, an analyst’s report lived or died by the whims of the hierarchy. If he was in the right place on the totem pole, his report on Soviet missiles could be pushed up higher; if a supervisor chose to ignore it, the report essentially vanished. Blogs and wikis, in contrast, work democratically. Pieces of intel would receive attention merely because other analysts found them interesting. This grass-roots process, Andrus argued, suited the modern intelligence challenge of sifting through thousands of disparate clues: if a fact or observation struck a chord with enough analysts, it would snowball into popularity, no matter what their supervisors thought.

A profusion of spy blogs and wikis would have another, perhaps even more beneficial impact. It would drastically improve the search engines of Intelink. In a paper that won an honorable mention in the Galileo Awards, Matthew Burton — the young former D.I.A. analyst — made this case. He pointed out that the best Internet search engines, including Google, all use “link analysis” to measure the authority of documents. When you type the search “Afghanistan” into Google, it finds every page that includes that word. Then it ranks the pages in part by how many links point to the page — based on the idea that if many bloggers and sites have linked to a page, it must be more useful than others. To do its job well, Google relies on the links that millions of individuals post online every day.

This, Burton pointed out, is precisely the problem with Intelink. It has no links, no social information to help sort out which intel is significant and which isn’t. When an analyst’s report is posted online, it does not include links to other reports, even ones it cites. There’s no easy way for agents to link to a report or post a comment about it. Searching Intelink thus resembles searching the Internet before blogs and Google came along — a lot of disconnected information, hard to sort through. If spies were encouraged to blog on Intelink, Burton reasoned, their profuse linking could mend that situation. “

Imagine having tools that could spot emerging patterns for you and guide you to documents that might be the missing pieces of evidence you’re looking for,” Burton wrote in his Galileo paper. “Analytical puzzles, like terror plots, are often too piecemeal for individual brains to put together. Having our documents aware of each other would be like hooking several brains up in a line, so that each one knows what the others know, making the puzzle much easier to solve.”

With Andrus and Burton’s vision in mind, you can almost imagine how 9/11 might have played out differently. In Phoenix, the F.B.I. agent Kenneth Williams might have blogged his memo noting that Al Qaeda members were engaging in flight-training activity. The agents observing a Qaeda planning conference in Malaysia could have mentioned the attendance of a Saudi named Khalid al-Midhar; another agent might have added that he held a multi-entry American visa. The F.B.I. agents who snared Zacarias Moussaoui in Minnesota might have written about their arrest of a flight student with violent tendencies. Other agents and analysts who were regular readers of these blogs would have found the material interesting, linked to it, pointed out connections or perhaps entered snippets of it into a wiki page discussing this new trend of young men from the Middle East enrolling in pilot training.

Full Story