There is another Internet out there, one that everyday users never get involved with. It is used for any number of reasons, some of them legal, some of them distinctly not. A citizen of a repressive regime may wish to find a way to communicate free from state interference, or a national security agent may need to communicate untraceably; this can only be achieved under conditions of anonymity. A seller of controlled substances such as drugs, images or weapons will be equally concerned with hiding their identity, for less acceptable reasons. This race to invisibility has directly contributed to the development and widespread growth of the “hidden Internet.” There is no accurate estimate of the size of this other online world but the names evoke an impression of vast scale; the Deep Web (as opposed to the Surface Web), the Darknet or Darkweb. These services are most commonly used for file-sharing, sale or distribution of controlled goods and substances and avoidance of personal or national surveillance, and they already have a history.
Evolution of the Darknet
The first attempts to illegally share copyrighted content online came in the form of single, publicly visible servers hosting large amounts of data. These servers, being so visible and so popular, were very easy to identify and shut down, mostly through legal or regulatory channels and so their popularity was very short lived.
The next evolutionary step was an online explosion and it happened in 1999. Anyone who was online at that time still remembers the early days of Napster. The Napster model avoided the centralised storage of data by employing a distributed network of “peers” or client computers, each sharing its own content. The service provision and the “index” of content however remained centralised and Napster very quickly came under pressure from both artists and record labels to pursue copyright infringements. The conversion of Napster to a paid-for service subsequently saw the rise of Gnutella. Gnutella is an open source project, has a fully distributed database and does not rely on any centralised service; it is a fully peer-to-peer (P2P) network and to this day still has a large user base, and has not been as successfully suppressed as Napster was. These early P2P incarnations all had one overriding shortcoming, especially to those whose intentions were illegal or more seriously criminal; a lack of anonymity.
Freenet and the invisible Onion
The late nineties to mid-noughties witnessed two important developments in “underweb” communications; Freenet and Tor.
Access to the Freenet network is gained by simply installing a client, the installation is disarmingly simple, asking only whether you require low security – “an attacker with moderate resources may be able to trace your activity on Freenet back to you,” or high security – “this setting allows you to create your own Freenet darknet for vastly improved security.” Nevertheless, one of the key defining characteristics of the network is anonymity, both for the customer and for the information provider. Content is not stored on one single server, as is the case with the traditional world-wide web. Rather content is broken down into several chunks and split across many computers. Each user requesting content will pass through several proxies, and no individual node knows who requested the data or who was storing it. Each member of the Freenet devotes a certain amount of space and bandwidth to storing and processing requests from other members. Freenet is searchable to a degree and large indices of content are available.
Tor may be more widely known as a means to avoid proxies or to hide your location when accessing worldwide web content but it is also widely used to offer “hidden services.” Similarly to Freenet, these are destinations that cannot be reached on the wider Internet and can only be accessed through the Tor network. Hidden services on the Tor network have an unusual address format, such as http://dppmftaacucpuzpc.onion/ (example only). This pseudo top-level domain (.onion) exists only within the Tor network and is not a part of the wider DNS used by the visible worldwide web so it cannot be resolved to an IP address through standard means. It is only understood by the Tor network itself and allows traffic to be routed between source and destination without either party needing to reveal their IP address and thus their identity.
These hidden services and darknets are often abused by criminals for the distribution and sale of illegal material from hard drugs to child exploitation; they are used to distribute WikiLeaks-type content anonymously and also to facilitate freedom of speech. This phenomenon, coupled with the rise in anonymous online currencies, such as Bitcoin and WebMoney, is already causing new frustrations for law enforcement, presenting very complex challenges to effective identification of online criminals.
Attempts to quantify the amount of information available on this deep web are of course complex, but as far back as September 2001, Michael K. Bergman released a paper entitled, “The Deep Web: Surfacing Hidden Value,” where he concluded that “Public information on the deep Web is currently 400 to 550 times larger than the commonly defined World Wide Web.” There is no reason to assume that level of disparity has changed over time.
Do not venture into the darknet unprepared, forewarned is forearmed. It is highly likely that much of the content you may find on a darknet may be illegal in your own jurisdiction and also potentially harmful, given the nature of many of the “merchants.” Unfortunately, you will not be able to rely on traditional web filtering to keep you safe from harmful content, as the absence of a publicly accessible URL makes this very difficult. Traditional malicious file detection is becoming simple for criminals to avoid, so stripping away this important layer of protection is highly damaging to your security.
The Internet we can see is vast and growing exponentially, I’d recommend you stick to the light side. To paraphrase Douglas Adams; The Internet is big. You just won’t believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it’s a long way down the road to the chemist’s, but that’s just peanuts to the Internet.