Protecting your servers, workstations and networks can only go so far. Attacks which consume your availible Internet-facing bandwidth, or overpower your router's CPU can still take you offline. This paper is aimed at mitigating the effects of such attacks, and guiding you in what you should do if you are attacked.
Different types of attacks
Denial of Service attacks as their name implies, set out to remove a service from functional use by its clients. Web servers will stop serving web pages, e-mail servers will stop accepting or delivering e-mail, and routers will go dark taking you off the Internet all together.
Denial of a particular service will come in one of two forms:
* Complete consumption of a resource such as bandwidth, memory, CPU, file handles, or any other finite asset.
* Exploiting a weakness in the service to stop it functioning or causing the service to crash.
Over the last few years, attackers have refined their methods. As developers make software more reliable and more resiliant to DoS, the attack vectors have changed to target hard to secure parts of a service. In this paper we will discuss the first type of attack, and what we can do to protect our services from it.
Getting the most out of your service
Protecting your services from attack follows many of the same idiologies as tuning your services for maximum performance. The greater the load you can handle, the more resilliant you are. Things change slightly when the attack changes the profile of your service.
For example, if you have a webserver tuned to transfer mostly large file transfer and your attack forces through a lot of small, shortlived transactions, you could find you run out of network memory buffers very quickly. I would recommend starting by reading the papers on tuning FreeBSD for different applications: http://silverwraith.com/papers/freebsd-tuning.php
The paper describes good ways to start tuning your servers. Also, the tuning(7) man page is an excellent resource on performance improvements.
Analyzing and Blocking Denial of Service
The first step to protecting yourself from an attack is to understand the nature of different types of attacks. As we said earlier, resource consumption attacks target your system in places which can cause bottlenecks. The most popular targets are network bandwidth, system memory, network stack memory, disk I/O, operating system limitations such as a limit on the number of open file handles, and the CPU. These bottlenecks can either be on your systems or in your network hardware.
Attacks on bandwidth
Attacks against your network bandwidth are amongst the hardest to defend, and how you deal with them depends heavily on your network topology and how helpful your ISP is. To start with ask yourself the following questions:
* Is the attack against a single host, or multiple hosts?
* Is the attacker hitting a small set of ports, or randomly hitting many ports?
* Does the attack consists of protocols which should normally not be used with the attacked servers?
We are fortunate today, that most attacks are simple in their nature. They choose one or two styles of attack and at most a small number of IP addresses. This makes sense - bandwidth is as hard for attackers to acquire as it is for us to defend. If your Internet peering bandwidth is not saturated, the accepted approach is to block traffic to the attacked host(s) at your gateway. It is a good idea to run tcpdump on the attacked servers if you can, to see what kind of attack is taking place. Look for floods of very similar packets - all TCP SYN, UDP or ICMP. Look for packets all going to a particular port. If you find the number of source IP addresses is reasonably small, blocking out the packets based on source address may be possible. However if the source addresses are highly volatile in addressing, this can be an indicator that they are spoofed (forged). When this is the case, you may need to look for other similarities in the attack such as packet size, window size, fragmentation, etc. If you have the ability to block based on these less common criteria you may want to investigate here further. In today's multi-gigabit networks, it is not unusual for an Internet connection to have more bandwidth than the local LAN, and so it may be possible for you to block the attack at your Internet gateway.
More often than not though, this does not apply, and having your Internet bandwidth consumed can be a tiring and frustrating ordeal. This might be the right time to call your ISP, if you have one who is willing to work with you on these problems. Before you make the call, try to analyze the attack. This is most certainly help your ISP in selectively filtering the attack off your network. If filtering is possible, one of two common options will usually be available, selectively filtering out the attacking systems, or dropping all packets to the attacked servers. The latter is often preferred as it is easier to manage and would be more effective in the event that the attack profile changes against those hosts.
If you run Border Gateway Protocol (BGP) at your Internet gateway to announce your IP space to the Internet you may have a third option, one which is becoming popular with a lot of ISP's. UUNet, C&W, XO, and many others are allowing users to export routes as small as /32, with a special community string which causes all incoming data for the route to be dropped at the ISP's border. This is a highly effective method of dropping an attack on the floor with the least damage to yourself and your ISP. Of course this only works well if the number of hosts being attacked is small, or if your ISP offers such functionality. Contact your ISP to find out. The obvious downside of this is that the IP addresses you export in this fashion will lose ALL connectivity to the Internet.
In general is it a good idea to keep your network clean; only allow traffic on to it which is needed for your services to operate. Allow TCP to ports 80 and 443 on your servers and allow UDP to your game servers. Allow SSH connections only from trusted hosts. All of these limit the options of the attackers when they come to visit.
Attacks on Systems and Services
If your bandwidth is not saturated, it is most likely that the attack is against your systems and the services they host, rather than your entire network. Again, the remedy depends on the nature of the attack. Systems contain all of the possible bottlenecks which can be targeted, and you may find that more than one bottleneck is exposed at any one time. Attacks on systems and their services generally fall into the following categories:
* Network subsystem limitations (very high number of packets per second)
* OS or application memory limitations (memory consumption)
* Disk or CPU limitations (large numbers of valid requests)
System-targetting attacks can be some of the most frustrating as people have a hard time defending against them. There is however some special magic which FreeBSD lets us use to help out.
By default, each time your network card receives a packets, it generates an interrupt to the CPU along its IRQ. The CPU will catch this and dedicate an amount of time to fetch this packet from the interface. Under normal operations this can happen several thousand times per second which is well within the capabilities of even low end CPU's. It is quite likely with older CPU's that you will start to see performance impacts at around 25,000 to 50,000 packets per second. With packet sizes of 1500 bytes, this works out to around 40Mbytes/sec to 75Mbytes/sec, which is quite a lot for most older CPU's to serve anyway. Most 1Ghz systems will begin to feel pressure around 75,000 packets per second. The problem is exasperated by two factors:
* If the packet flood is from TCP SYN packets, these packets must be fully processed and then SYN ACK packets sent back to the source address of the original SYN packets. This is a reasonably expensive operation in itself. Other TCP and UDP packets to closed ports, and also ICMP packets, need to be similarly processed and have the appropriate TCP or ICMP replies sent back. While not as expensive as SYN processing, this still takes time and consumes outbound bandwidth.
* Packet size also plays an important factor. You can get more small packets in a particular amount of bandwidth than you can large packets. The more packets you take in, the more CPU time is required to process them, no matter what type of packets they are.
As we discussed previously, each time an IRQ is generated some CPU time is taken. If enough IRQ's can be generated, the CPU will have no time to do anything other than serve the interrupts. Inbound packets do not get processed, applications get no CPU time, and your system is effectively dead in the water. This is known as "Live-lock". Your system is still live, in so much that it has not crashed, but it is locked from performing any useful functions. Once packets stop coming in to the interface, the CPU starts to process all of the backlogged packets it has already accepted. This can take anything from a few minutes to several hours.
There are several things you can do to prevent or mitigate the effects of a high rate of packets, before you need to go out and buy any hardware upgrades. All of these are performed using FreeBSD's sysctl(8) command. Here are the settings you will need, you can place them in /etc/sysctl.conf:
* net.inet.tcp.msl=7500
net.inet.tcp.msl defines the Maximum Segment Life. This is the maximum amount of time to wait for an ACK in reply to a SYN-ACK or FIN-ACK, in milliseconds. If an ACK is not received in this time, the segment can be considered "lost" and the network connection is freed.
There are two implications for this. When you are trying to close a connection, if the final ACK is lost or delayed, the socket will still close, and more quickly. However if a client is trying to open a connection to you and their ACK is delayed more than 7500ms, the connection will not form. RFC 753 defines the MSL as 120 seconds (120000ms), however this was written in 1979 and timing issues have changed slightly since then. Today, FreeBSD's default is 30000ms. This is sufficient for most conditions, but for stronger DoS protection you will want to lower this to 7500, or maybe even less.
* net.inet.tcp.blackhole=2
net.inet.tcp.blackhole defines what happens when a TCP packet is received on a closed port. When set to '1', SYN packets arriving on a closed port will be dropped without a RST packet being sent back. When set to '2', all packets arriving on a closed port are dropped without an RST being sent back. This saves both CPU time because packets don't need to be processed as much, and outbound bandwidth as packets are not sent out.
* net.inet.udp.blackhole=1
net.inet.udp.blackhole is similar to net.inet.tcp.blackhole in its function. As the UDP protocol does not have states like TCP, there is only a need for one choice when it comes to dropping UDP packets. When net.inet.udp.blackhole is set to '1', all UDP packets arriving on a closed port will be dropped.
* net.inet.icmp.icmplim=50
The name 'net.inet.icmp.icmplim' is somewhat misleading. This sysctl controls the maximum number of ICMP "Unreachable" and also TCP RST packets that will be sent back every second. It helps curb the effects of attacks which generate a lot of reply packets.
* kern.ipc.somaxconn=32768
kern.ipc.somaxconn limits the maximum number of sockets that can be open at any one time. The default here is just 128. If an attacker can flood you with a sufficiently high number of SYN packets in a short enough period of time, all of your possible network connections will be used up, thus successfully denying your users access to the service.
You may find these settings to either be too aggressive, or not aggressive enough. You should tune them until you receive satisfactory results.
Finally, if you are blessed enough to own one of the following network cards you can enable a kernel feature call DEVICE_POLLING:
* dc
* em
* fxp
* nge
* rl
* sis
DEVICE_POLLING changed the way that interrupts are handled. Actually with DEVICE_POLLING, they are not handled at all! DEVICE_POLLING causes interrupts to be effectively ignored. Instead, at certain times, the CPU will poll the network card, and pick up an packets that are waiting for processing. This can significantly reduce the amount of CPU time used in processing inbound traffic, but only the above cards are supported as the drivers have to be written to support DEVICE_POLLING. The FXP cards generally work best with the feature as their drivers are very well developed, as is their hardware. The hardware design and quality of RL cards is a lot lower - without sufficient CPU (usually around 1Ghz), they have a hard time achieving the full 100MB/s at all. If you are looking for a new network card, you will get what you pay for!
You can learn more about DEVICE_POLLING at the author's home page:
http://info.iet.unipi.it/~luigi/polling/. You can also find good installation and tuning instructions there, as well as some statitics from comparative tests with DEVICE_POLLING enabled and disabled.
Tracking the source of the attack
Attacks can come from inside and outside your network and obviously one is easier to isolate than the other. Tracking the sources of attacks requires some familiarity with packet sniffing tools such as TCPDump, ngrep, or ethereal. Unless you have spent several months, carefully profiling your network traffic and set up monitoring specifically to alert you of anomalies, the chances of discovering you are under Denial of Service conditions before someone else does are slim. More often than not it is complaints such as "The Internet is slow", or "I can't get my e-mail" that lead us to find the truth. It is important to realize two things:
* Attacks can comes from inside and outside your network
* Not all service-denying events constitute a denial of service attack, and not all denial of service attacks constitute a service-denying event
What does this mean to you? It means that when you start to look for why "your Internet is slow" or why people cannot get their e-mail, remember that the source of the problem could be from any machine on your network or the Internet, and that the denial may not be deliberate.
A good place to start is the point of bottleneck. This could be the CPU on your HTTP proxy, or maybe your Internet gateway. If your bottleneck is a system process such as a proxy server, examine the logs for this. Is there a single system or small number of systems making an unusually large number of requests, or using more resources than they should? If your bottleneck is your Internet gateway (which we assume is running FreeBSD), you can use a command like this to view what IP packets are passing through your gateway:
router# tcpdump -n -i
This command will display a summary of the first 100 packets (-c 100) it sees, on the
04:59:53.915324 192.168.0.3.2327 > 192.168.0.10.1214: S 3199611726:3199611726(0) win 16384
Let us look at the first few parts of this output which can be useful to us.
* 04:59:53.915324
This is the time stamp of when the packet was processed.
* 192.168.0.3.2327
This is the source IP address. The numbers after the last octet, 2327, indicate the source port number the packet was sent from.
* 192.168.0.10.1214
This is the destination IP address. The numbers after the last octet, 1214, indicate the destination port number to which the packet is going.
* S
This indicated the type of packet, in this case a SYN packet. The process and life of a TCP connection, and the types of packets you would see here can be learnt about from Daryl's TCP/IP Primer at: http://www.ipprimer.com/.
What you may see during an actual attack is hard to predict, as denial of service attacks come in so many shapes and sizes. A typical attack involves flooding a listening port on your server, with SYN packets. The idea is to make your system so busy processing the new connections that it cannot do anything else. Here you may be a large number of SYN packets. Usually these should be well balanced with packets of other types.
Tidak ada komentar:
Posting Komentar