Tracking compromised machines can be difficult. Security solutions often don't scale to the size of larger networks. Technologies such as IDS are flawed, producing copious false positives. When solutions are scaled to fit the larger providers, they often require considerable care and feeding, thus taking time away from problem mitigation. There must be a better way!
Enter the Darknet! A Darknet is a portion of routed, allocated IP space in which no active services or servers reside. These are "dark" because there is, seemingly, nothing within these networks.
A Darknet does in fact include at least one server, designed as a packet vacuum. This server gathers the packets and flows that enter the Darknet, useful for real-time analysis or post-event network forensics.
Any packet that enters a Darknet is by its presence aberrant. No legitimate packets should be sent to a Darknet. Such packets may have arrived by mistake or misconfiguration, but the majority of such packets are sent by malware. This malware, actively scanning for vulnerable devices, will send packets into the Darknet, and this is exactly what we want.
Darknets have multiple uses. These can be used to host flow collectors, backscatter detectors, packet sniffers, and IDS boxes. The elegance of the Darknet is that it cuts down considerably on the false positives for any device or technology.
The goals of the Darknet are simple - to increase awareness, and to ease mitigation. With a Darknet in place, it is far easier to determine the amount of naughty traffic on a network, as well as the sources of said traffic.
The first step in deploying a Darknet is to locate a suitable home. We recommend first determining the amount of address space you plan to allocate to the Darknet. At a minimum we recommend a /24, though smaller address space - even up to a /32 - will work. You could even allocate a single /32 from each of several /24s to make the Darknet very difficult to detect. The more address space you allocate to the Darknet, the greater your visibility and statistical sampling.
Remember the rule of routing - the most specific prefix wins. This is a great advantage for a Darknet. If you have a /16 for your enterprise, you can route the entire /16 to your Darknet, with more specific routes in your IGP carrying production traffic to the proper places. As with all things, carefully ponder and test this configuration prior to deployment.
Do not use bogon address space. Bogon space is avoided by many scanners and malware. Because bogon space does not include vulnerable hosts, there is no return on the investment for the malware. Use only allocated, routed space for the Darknet.
Note: While this document uses IPv4 exclusively, an IPv6 Darknet is also a viable solution. Future versions of this document may include IPv6 examples, depending largely on Rob's caffeine intake.
A /32, not within the allocated Darknet space, needs to be assigned for both the management IP and the sniffer IP.
The next step is to set aside the physical and logical space for the Darknet. You will need:
We strongly recommend against putting the Darknet in the same collision domain or VLAN as other subnets. The goal of the Darknet is to provide credible data, thus it is critical to avoid "poisoning" the Darknet with legitimate traffic.
We recommend against putting the Darknet IPs into your publicly visible DNS. DNS RRs such as darknet-sniffer.your.net are just a bit obvious. ;)
For our example, we'll use the following (these are BOGONS, so do not use them!):
Darknet prefix = 10.18.18.0/24 Sniffer IP = 172.16.18.2 Management IP = 192.168.18.2 Darknet VLAN = 855 Management VLAN = 222
An upstream router interface, or entire router if you have the luxury, should be selected and configured for the Darknet. The router should be configured to protect itself; the Secure IOS Template or the Secure JunOS Template can be of use here. Otherwise the router should permit all packets to pass into the Darknet. The router can be configured to block all outbound packets from the Darknet.
The router should be configured for SNMP, and at a minimum traffic statistics should be collected from the interface facing the Darknet server. This is done for two reasons:
The Witty worm is an example where our Darknets alerted us within minutes of the release of the worm. This set of graphics from three distinct Darknets makes it quite clear something was amiss:
You can see the traffic statistics from a small subset of our Darknet pods at http://www.cymru.com/Reach/darknet.html.
There are myriad packages designed to collect router statistics, to include MRTG. The router can also be configured to export flows to a package of your choice. While this may be redundant for a Darknet, it may also provide a seamless way to integrate Darknet data into existing network monitoring infrastructure. We like the following packages, though your mileage may vary:
This router could also carry the Darknet management traffic. Your call here.
The router should be configured to send all Darknet prefix traffic to the SNIFFER NIC on the Darknet server. On a Cisco router, configure the route thusly:
router#conf t router(config)# ip route 10.18.18.0 255.255.255.0 172.16.18.2 router(config)# ^Z router# wr
The router, or an upstream router, should inject the Darknet prefix into your IGP. Details on how this is done is network dependent, but please contact Team Cymru if we can assist you with this task.
For our continuing example, we'll use 172.16.18.1 and 192.168.18.1 as the router IPs. The topology is now, in fine ASCII art:
_______________________________________________ ____________________________________ WAN <-| some.internal.ip.here - ROUTER - 172.16.18.1 |-> VLAN 855 <-| 172.16.18.2 DARKNET-SERVER-SNIFFER | |_______________________________________________| | | | DARKNET | | SERVER | _______________________________________________ | | WAN <-| some.internal.ip.here - ROUTER - 192.168.18.1 |-> VLAN 222 <-| 192.168.18.2 DARKNET-SERVER-MGMT | |_______________________________________________| |____________________________________|
Everyone has a favorite server platform and operating system. We provide these suggestions as a guideline, and you should feel free to modify them to fit your environment. The goal of a Darknet is to increase awareness, not to increase workload.
The server must have two NICs. One is for sniffing, and the other is for management. Again it is imperative that we not contaminate our Darknet with legitimate traffic, to include our own management traffic. In our example we pick FreeBSD 4.8 and assign the two NICs thusly:
em0 - SNIFFER NIC - 172.16.18.2 fxp0 - MGMT NIC - 192.168.18.2
The default route for the box, if any, should point to the gateway on the management subnet, e.g. the gateway upstream from the fxp0 interface in our example. Do not point any routes out of the SNIFFER NIC. If you can go with static routes and no default, all the better to protect the Darknet server. Default routes are not always necessary.
route add -net <mgmt-subnet> 192.168.18.1
We also recommend adding a null route on the box for the allocated Darknet prefix. This adds a bit of extra protection, and ensures that the Darknet server can not answer packets destined for the Darknet prefix. Both Solaris and FreeBSD support blackhole routes, which are implemented thusly:
route add -net 10.18.18.0/24 127.0.0.1 -blackhole
Remember to update the boot files so that this route is implemented at reboot.
We recommend a single CPU box, at least 1.8GHz for Intel, with at least 200GB of hard drive space. This depends entirely on the amount of data you choose to log. In one of our Darknet pods, we can log a gigabyte per hour from a single /16. Ultimately disk space and I/O are likely to be your strictest bottlenecks. Ensure that you monitor the disk space utilization and I/O throughput on the Darknet server.
We recommend using UTC (aka GMT) as the timezone on all of your Darknet servers. A standard timezone will make cross-Darknet correlation a much simpler task. This can be accomplished by copying the proper timezone file, usually named "GMT," to /etc/localtime. Don't forget to synchronize time with two or more NTP servers. In our example, the management servers 192.168.1.9 and 192.168.2.9 will be our NTP servers. There are several features for NTP configuration, but at a minimum you want the following in your ntp.conf file:
# Our NTP servers. server 192.168.1.9 server 192.168.2.9
You likely want DNS running on your Darknet server. In our example we will use the management servers 192.168.1.9 and 192.168.2.9 as our DNS name servers (they also serve NTP). Be careful with DNS and sniffing! Be sure to use the -n flag with tcpdump so that your Darknet server doesn't attempt to resolve each IP in your sniffer output. Our /etc/resolv.conf file should now include:
search first.domain.here second.domain.here so.on.and.so.on nameserver 192.168.1.9 nameserver 192.168.2.9
We're now going to add some software to our Darknet server. We use and recommend the following:
A discourse on how to build and implement each is beyond the scope of this treatise. Each comes with ample build and installation documentation. We will cover only certain tweaks here.
Argus and tcpdump should be configured to listen on the SNIFFER NIC only, in our example em0. The remote management daemon for Argus should be bound to the MGMT NIC only, e.g. fxp0, on port 2002 in our example. It should not be bound to the SNIFFER NIC.
Argus should be run as a daemon, and ideally started at boot. Here is a suggested template for your /etc/argus.conf file.
# /etc/argus.conf # 01 MAY 2004 email@example.com # # Run Argus as a daemon ARGUS_DAEMON=yes # # Now select a _unique_ Argus ID for this Darknet server. # Configure all Darknet servers with their own, unique # Argus ID. ARGUS_MONITOR_ID=5 # # Select a port on which the Argus daemon will listen for # remote management and queries. We've already chosen # TCP 2002 as our port. ARGUS_ACCESS_PORT=2002 # # We only want to sniff the em0 port. ARGUS_INTERFACE=em0 # # Store the flows in an output file. Rotate this file # regularly with a script. ARGUS_OUTPUT_FILE=/var/log/argus/argus-id5-log # # Log the PID of the Argus daemon. ARGUS_SET_PID=yes # # Set to promiscuous mode. ARGUS_GO_PROMISCUOUS=yes # # Decision time. How often do you wish to receive reports # on long-lived flows? A small number here is safe, because # there should be _NO_ long lived flows in a Darknet. The # Darknet isn't supposed to actually complete a connection # request, after all. We'll set this to five seconds. ARGUS_FLOW_STATUS_INTERVAL=5 # # It's wise to regularly output the Argus daemon stats. # These can be very helpful when debugging problems. We'll # set this to 60 seconds. ARGUS_MAR_STATUS_INTERVAL=60 # # While we don't use these statistics regularly, they can # be handy at times. Go ahead and collect them. ARGUS_GENERATE_RESPONSE_TIME_DATA=yes ARGUS_GENERATE_JITTER_DATA=yes # # MAC address information should be uninteresting, because # all packets will have the source MAC of the upstream # router. ARGUS_GENERATE_MAC_DATA=no # # Capturing only headers can hinder analysis efforts. If # you have the CPU and disk space, gather all the bits in # each packet. ARGUS_CAPTURE_DATA_LEN=1518 # # Optimize the libpcap filters. ARGUS_FILTER_OPTIMIZER=yes # # Do not filter the packets; grab them all. ARGUS_FILTER="" # # The ability to run tcpdump under the Aegis of Argus is # one of its most powerful features. This combines the # flow logging with packet payload. If you have the CPU # and disk capacity, enable this feature. Remember to # rotate these logfiles! ARGUS_PACKET_CAPTURE_FILE="/var/log/argus/argus-id5-tcpdump" #
IP Filter should be configured to block everything in and out of the SNIFFER NIC interface. In our example this is the em0 interface, IP address 172.16.18.2. You may wish to log any hits on the block rules so that debugging and alerts are easy to accomplish; it's not a bad idea to do the same on the pass rules as well. If you choose to log the rule hits, remember to start ipmon with the proper flags.
Modify /etc/ipf.rules (/etc/opt/ipf/ipf.conf in Solaris) thusly:
# Prevent any traffic from entering and exiting the em0 # SNIFFER NIC. block in log quick on em0 from any to any block out log quick on em0 from any to any #
But won't this block the traffic we're attempting to log? No, both Argus and tcpdump insert themselves beneath ipf, gathering all the packets and flows before ipf blocks the traffic.
The Darknet server is now configured with SSH on TCP 22 and an Argus remote flow collection port on TCP 2002. Both should be configured to bind only to the MGMT NIC, not the SNIFFER NIC. These should also be protected using ipf. Remember we are managing this device out of band from the Darknet network.
# Lock down management access to the fxp0 MGMT NIC. pass in quick on fxp0 proto tcp from <mgmt-subnet> to 192.168.18.2 port = 22 keep state pass in quick on fxp0 proto tcp from <mgmt-subnet> to 192.168.18.2 port = 2002 keep state block in log quick on fxp0 from any to any #
Depending on the functions and management methods of your Darknet server, you may wish to permit outbound connections from the MGMT NIC, permit inbound ICMP, etc. Here is an example of a complete FreeBSD Darknet server /etc/ipf.rules (Solaris /etc/opt/ipf/ipf.conf) file. In this example the management subnets are 192.168.1.0/24 and 192.168.2.0/24. For guidelines on permissable ICMP messages, please see our ICMP Packet Filtering guide. It is wise to test new filter sets from an out of band management port, e.g. the console, to avoid locking yourself out of the Darknet server. Remember to modify this filter set to fit your environment and adhere to applicable security policies.
# /etc/ipf.rules # 01 MAY 2004 firstname.lastname@example.org # # em0 172.16.18.2 SNIFFER NIC # fxp0 192.168.18.2 MGMT NIC # # 192.168.1.0/24 ORD NOC # 192.168.2.0/24 LHR NOC # # Prevent any traffic from entering and exiting the em0 # SNIFFER NIC. block in log quick on em0 from any to any block out log quick on em0 from any to any # # Lock down SSH, Argus, and SNMP access to the fxp0 MGMT NIC. pass in quick on fxp0 proto tcp from 192.168.1.0/24 to 192.168.18.2 port = 22 keep state pass in quick on fxp0 proto tcp from 192.168.2.0/24 to 192.168.18.2 port = 22 keep state pass in quick on fxp0 proto tcp from 192.168.1.0/24 to 192.168.18.2 port = 2002 keep state pass in quick on fxp0 proto tcp from 192.168.2.0/24 to 192.168.18.2 port = 2002 keep state pass in quick on fxp0 proto udp from 192.168.1.0/24 to 192.168.18.2 port = 161 keep state pass in quick on fxp0 proto udp from 192.168.2.0/24 to 192.168.18.2 port = 161 keep state # # Permit SSH outbound to the management subnets. pass out quick on fxp0 proto tcp from 192.168.18.2 to 192.168.1.0/24 port = 22 keep state pass out quick on fxp0 proto tcp from 192.168.18.2 to 192.168.2.0/24 port = 22 keep state # Permit syslog outbound to the management subnets. pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.1.0/24 port = 514 keep state pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.2.0/24 port = 514 keep state # # Permit DNS outbound to the DNS name servers. pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.1.9 port = 53 keep state pass out quick on fxp0 proto tcp from 192.168.18.2 to 192.168.1.9 port = 53 keep state pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.2.9 port = 53 keep state pass out quick on fxp0 proto tcp from 192.168.18.2 to 192.168.2.9 port = 53 keep state # # Permit NTP outbound to the NTP servers. pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.1.9 port = 123 keep state pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.2.9 port = 123 keep state # # Permit some ICMP from the management subnets and the upstream router. pass in quick on fxp0 proto icmp from 192.168.1.0/24 to 192.168.18.2 icmp-type echo pass in quick on fxp0 proto icmp from 192.168.2.0/24 to 192.168.18.2 icmp-type echo pass in quick on fxp0 proto icmp from 192.168.18.1 to 192.168.18.2 icmp-type echo pass in quick on fxp0 proto icmp from 192.168.1.0/24 to 192.168.18.2 icmp-type echorep pass in quick on fxp0 proto icmp from 192.168.2.0/24 to 192.168.18.2 icmp-type echorep pass in quick on fxp0 proto icmp from 192.168.18.1 to 192.168.18.2 icmp-type echorep pass in quick on fxp0 proto icmp from 192.168.1.0/24 to 192.168.18.2 icmp-type timex pass in quick on fxp0 proto icmp from 192.168.2.0/24 to 192.168.18.2 icmp-type timex pass in quick on fxp0 proto icmp from 192.168.1.0/24 to 192.168.18.2 icmp-type unreach pass in quick on fxp0 proto icmp from 192.168.2.0/24 to 192.168.18.2 icmp-type unreach # # Permit some outbound ICMP for debugging purposes. pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.1.0/24 icmp-type echo pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.2.0/24 icmp-type echo pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.18.1 icmp-type echo pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.1.0/24 icmp-type echorep pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.2.0/24 icmp-type echorep pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.18.1 icmp-type echorep # # Block everything else. block in log quick on fxp0 from any to any block out log quick on fxp0 from any to any #
Once you have the desired filter set in place, reload your IP Filter firewall with:
ipf -f /etc/ipf.rules
Now fire up tcpdump on the em0 interface to see if the packets to our Darknet prefix, 10.18.18.0/24, are arriving. After launching tcpdump, send a few packets to sundry addresses within 10.18.18.0/24. If your chosen Darknet prefix is accessible from the Internet, it won't take long before the naughty packets arrive.
tcpdump -n -i em0
Next test that you can connect to the Argus port from your remote management stations.
ra -zn -S 192.168.18.2 -P 2002
If you see packets with tcpdump and flows with Argus, congratulations! You now have an operational Darknet. :)
A clever alternative is to use a host-based system. One suggestion is to use Linux and iptables. Route a small prefix (e.g. a /25) to a Linux box that is a "log and drop" iptables firewall. The data can be pulled from the logs to produce results similar to the Darknet configuration detailed here.
Managing a Darknet shouldn't be an arduous task. There is the usual health checking of both the router and the Darknet server. We recommend using SNMP on both to monitor interface usage. It is unlikely to ever be zero bps or pps. We also strongly recommend monitoring the processes on the Darknet server, to include cron, syslog, Argus, and tcpdump. Keep a close eye on the disk space on the Darknet server, as that can fill up rapidly during periods of high network stress.
We recommend rotating the created logfiles, as they can become quite large. tcpdump can be told to rotate the log file when it reaches a set size with the -C flag. This example rotates the file and names it "darkdumpN" (where N is an incrementing number) when it reaches 150MB in size.
tcpdump -i em0 -n -w darkdump -C150
Scripts can also be used to rotate log files.
As with all things keep patches and code current and secure. Regularly audit your Darknet. This is not the host you want compromised when a nasty bit of malware visits your network.
Patrick Green of the OxCERT team at Oxford University, UK, has put together a great paper on the practical use of a Darknet. Our thanks to Pat for authoring the paper and permitting us to host it here. You will find the paper at http://www.team-cymru.org/documents/dark-watcher.pdf.
So you have a Darknet - now what do you do with it? There are a variety of ways to peruse the collected data, and myriad uses for that data.
Perhaps you've heard that there is a bot that exploits Microsoft Windows 2000 open shares. This bot scans on TCP port 445, and you wish to know how much of that traffic is reaching (or sourced within) your network. A quick Argus query to your Darknet will help to answer that question. The query is constructed using tcpdump pattern matching syntax. See the ra man page for more examples.
host$ ra -n -S 192.168.18.2 -P 2002 tcp dst port 445 ra: Trying 192.168.18.2 port 2002 ra: connected Argus Version 1.8 30 Apr 04 17:37:11 tcp x.220.127.116.1101 o> y.10.73.47.445 TIM 30 Apr 04 17:37:11 tcp x.18.104.22.16835 o> y.10.12.121.445 TIM 30 Apr 04 17:37:11 tcp x.22.214.171.12436 o> y.10.12.121.445 TIM 30 Apr 04 17:37:11 tcp x.126.96.36.19900 o> y.10.12.121.445 TIM 30 Apr 04 17:37:11 tcp x.188.8.131.521 o> y.10.12.121.445 TIM 30 Apr 04 17:37:11 s tcp x.184.108.40.2066 -> y.10.66.129.445 REQ 30 Apr 04 17:37:11 s tcp x.220.127.116.1171 -> y.10.76.251.445 REQ 30 Apr 04 17:37:11 s tcp x.18.104.22.16899 -> y.10.12.121.445 REQ 30 Apr 04 17:37:11 s tcp x.22.214.171.12453 -> y.10.11.95.445 REQ 30 Apr 04 17:37:11 s tcp x.126.96.36.19957 -> y.10.111.235.445 REQ 30 Apr 04 17:37:11 s tcp x.240.143.78.22232 -> y.10.78.235.445 REQ 30 Apr 04 17:37:11 s tcp x.188.8.131.521 -> y.10.101.71.445 REQ 30 Apr 04 17:37:11 s tcp x.184.108.40.2064 -> y.10.103.232.445 REQ
Sure enough there is a fair bit of TCP 445 scanning occuring. The first column of IPs are the sources; these can now be reported to the system owners. What has compromised these hosts? Unclear, but what is clear is that something is wrong - remember, no legitimate packets should ever enter a Darknet.
Slammer was a worm that sent packets to UDP 1434. Argus and a Darknet can spot these easily enough.
host$ ra -n -S 192.168.18.2 -P 2002 udp and dst port 1434 ra: Trying 192.168.18.2 port 2002 ra: connected Argus Version 1.8 30 Apr 04 17:29:28 udp x.220.127.116.117 -> y.10.19.234.1434 TIM 30 Apr 04 17:29:34 udp x.18.104.22.16881 -> y.10.103.192.1434 TIM 30 Apr 04 17:29:34 udp x.22.214.171.1242 -> y.10.87.15.1434 TIM 30 Apr 04 17:29:35 udp x.126.96.36.19954 -> y.10.66.161.1434 TIM 30 Apr 04 17:29:35 udp x.188.8.131.5279 -> y.10.71.103.1434 TIM
The data files can also be perused in batch mode with custom shell and Perl scripts.
One use is to match up the flows with known malware signatures.
You can use this data to alert customers to problems within their networks. For example, if you flag 300 Slammer scans from a customer network, you can contact them with the source IP and UTC timestamp of the scan. They can then take action to clean the reported hosts. If you're reporting infected hosts to peers or other providers, you can run the list through the Team Cymru WHOIS server to make notification list easier to assemble.
You can use the data for statistical analysis. This is an excellent method by which you can begin to quantify the amount of "trouble" on your network. You can even create a simple "garbage meter" based on the traffic entering your Darknets.
Customers and management will appreciate the proactive nature of Darknets, meaning you can "spot bot" and other malware before it is used for spam or DDoS.
You could readily host IDS boxes in your Darknet, thus matching on the known fingerprints.
Best of all you can use your Darknet deployment and data to convince others to do the same. Darknets aren't just for providers. They will fit nicely in enterprise and (supposedly) disconnected networks.
The Darknet concept is not new or the creation of Team Cymru. Lots of brilliant folks are doing similar things, and we appreciate their contributions. The folks at CAIDA have their Network Telescope project. The folks at the University of Michigan have their (now inactive) Internet Motion Sensor project (2004 paper describing IMS [PDF]).
We hope this page and project are useful to you. Please feel free to share suggestions, comments, and references with Team Cymru. Direct your comments to email@example.com.