Meet The Deep Web: Inside The Hidden Internet That Lies Beyond Google

The entire expanse of the Internet, going well beyond what you surf every day, is only the tip of a massive iceberg, the size of which is too vast to be known. Far beyond searching the Web with Google or chatting with friends on Facebook, there's a dense maze of websites that exist out of view, though for how long that will be true is unclear.

When most users log on to the Internet, they visit their bookmarked pages or use Google to search for sites that can provide the information they're looking for. That Internet, used by billions around the world every day, is sometimes known as the Surface Web, or the Clearnet, as coined by Tor and other anonymous online users.

The Deep Web, simply put, is everything else. It’s made up of tens of billions of sites that are hidden within a universe of code -- various estimates have put the Deep Web at anywhere from five to 500,000 times the size of the Surface Web.

Not a lot of the Deep Web is particularly interesting, as it's mostly massive databases ranging from NASA's to LexisNexis -- and those are just the oldest examples of the Deep Web. Other parts of the Deep Web include internal networks used by companies and academic institutes. Basically every page, website and internal network not indexed by search engines can be classified as part of the Deep Web.

Military origins

Like other areas of the Internet, the Deep Web began to grow with help from the U.S. military, which sought a way to communicate with intelligence assets and Americans stationed abroad without being detected. Paul Syverson, David Goldschlag and Michael Reed, mathematicians at the Naval Research Laboratory, began working on the concept of "onion routing" in 1995. Their research soon developed into The Onion Router project, better known as Tor, in 1997 with funding from U.S. Department of Defense's Defense Advanced Research Projects Agency (DARPA).

All of this led up to the Free Haven Project, formed in 1999 as a research project by a group of students from the Massachusetts Institute of Technology. The MIT students, including Roger Dingledine and Nick Mathewson, worked with the Naval Research Laboratory mathematicians to develop the first instance of Tor that we're familiar with today.

The next generation of Tor was presented at the “Proceedings of the 13th USENIX Security Symposium" in 2004. As the paper explains, there were other anonymous systems too complicated to pursue as the process would involve the encryption of data followed by its decryption and then sending that message forward. This process would repeat each step of the way until it reached its destination, which meant a much longer lag in the communication. The U.S. Navy released the Tor code to the public in 2004, and in 2006 Dingledine, Mathewson and several others formed the Tor Project and released the service currently in use.

How Tor works

Tor works like a circuit, taking an individual’s Internet connection and redirecting it through different connection points (or “nodes”) all over the world before delivering it to the desired website. Peeling back the layers of that “onion” becomes more difficult with every additional node, so both Tor’s strength and popularity soared when the Electronic Frontier Foundation released a public version of the software in 2004.

Tor software isn’t the only way to access the Deep Web, but it’s by far the most efficient and user-friendly. Its growth has also made the Deep Web more accessible, giving users who simply sought anonymity an unexpected level of access to everything from drug bazaars to crisis counseling.

“It’s wedded to the idea of the origins of the Internet, around the intertwining of the cryptography and open-source ethos and philosophy,” said Alex Winter, the director behind the 2013 documentary “Downloaded” and an upcoming Deep Web film. “It’s largely just what’s under the hood of the Internet. That space has been penetrated by people who want to remain anonymous and hide their identity online.”

Into the 'Darknet'

Perhaps it shouldn’t be a surprise, then, that where so many people are trying to hide their identity, law enforcement isn’t far behind. Within the larger Deep Web exists the Darknet, a loosely affiliated community of forums where everything from bomb recipes and drugs to child porn and human services are peddled.

While enabling free speech and guaranteeing a high level of anonymity, Tor has also been accused of facilitating illegal activity by cloaking its users. Yet, like BitTorrent and Napster before it, Tor’s operators have found that when such a powerful new technology is introduced, it will almost certainly be subverted for criminal gain.

“Our answer to that is, we don’t condone illegal activity,” Andrew Lewman, executive director of the Tor project, said. “We’ve worked with law enforcement and with the FBI and Interpol to stop things like child trafficking and these drug markets. Most of that is just explaining how Tor works, because old-fashioned police work still works really well.”

The Silk Road

Most people first became familiar with this hub of criminal activity in October 2013 when the FBI shut down the Silk Road and arrested its alleged owner Ross Ulbricht (who referred to himself as Dread Pirate Roberts). Federal prosecutors argued that in its nearly three years of operation on Tor, the Silk Road hosted approximately 1.5 million transactions worth roughly $1.2 billion. A multitude of sites rushed to fill the vacuum created when the Silk Road was shut down, like Silk Road 2.0, which the FBI shut down in October.

Other outgrowths of the original Silk Road included a number of support forums for addicts anonymously seeking advice on how to overcome drug addiction. One doctor who learned about the Silk Road from a patient began posting on a forum under the moniker Dr. X, issuing free harm-reduction advice in a corner of the Internet where dealers had previously been all too happy to sell drugs while looking the other way on addiction.

"What started to happen was a lot of people saying, 'I’m addicted to heroin. How do I stop this?'" Lewman said. "He started giving nonjudgmental advice about drugs and drug interaction. Then a bunch of other people got on, social workers and people like that, saying ‘Hey, these are ways to treat your addiction.'"

From police to doctors to drug dealers, it’s that Wild West mentality that has led experts to suggest that the Deep Web will foster innovation. Winter, the documentarian, compared the current state of the Deep Web to Napster, the file-sharing platform that would go on to upend the music industry.

“It’s a provocative new kind of technology that gets clobbered by law enforcement, and then [spawns] many more copycat technologies,” he said. “We’re in a large phase of transition, but make no mistake, what we’re experiencing is a curve that’s going up, not down.”