|
|
 
In this article we will show how to monitor network health from the client perspective using our AreWeDown tool. We will then disrupt communication from the client perspective to the server by using a ping flood, and will solve the problem using traffic shaping.
Let's start out with a healthy network:
| 2005-08-06 08:13:50 | are@10.50.100.190 | 101 |
| 2005-08-06 08:14:12 | are@10.10.10.11 | 100 |
| 2005-08-06 08:14:12 | are@10.10.10.11 | 101 |
| 2005-08-06 08:14:20 | are@10.50.100.190 | 100 |
| 2005-08-06 08:14:20 | are@10.50.100.190 | 101 |
| 2005-08-06 08:14:42 | are@10.10.10.11 | 100 |
| 2005-08-06 08:14:42 | are@10.10.10.11 | 101 |
| 2005-08-06 08:14:50 | are@10.50.100.190 | 100 |
| 2005-08-06 08:14:50 | are@10.50.100.190 | 101 |
| 2005-08-06 08:15:12 | are@10.10.10.11 | 100 |
| 2005-08-06 08:15:12 | are@10.10.10.11 | 101 |
| 2005-08-06 08:15:20 | are@10.50.100.190 | 100 |
| 2005-08-06 08:15:20 | are@10.50.100.190 | 101 |
|
See this article for information on the utility we are using to test with. Basically, this measures network health by showing how long it takes to make two consecutive TCP requests from the client's perspective. The 101 entry is sent right after the 100 entry, so they should be pretty much happening at the same time.
A ping looks like this:
[root@srv-1 usr-1]# ping 10.10.10.11
PING 10.10.10.11 (10.10.10.11) 56(84) bytes of data.
64 bytes from 10.10.10.11: icmp_seq=0 ttl=127 time=18.6 ms
64 bytes from 10.10.10.11: icmp_seq=1 ttl=127 time=18.7 ms
64 bytes from 10.10.10.11: icmp_seq=2 ttl=127 time=18.6 ms
64 bytes from 10.10.10.11: icmp_seq=3 ttl=127 time=18.7 ms
64 bytes from 10.10.10.11: icmp_seq=4 ttl=127 time=18.6 ms
--- 10.10.10.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 18.623/18.697/18.762/0.053 ms, pipe 2
[root@srv-1 usr-1]#
|
Our router stats look like this:
router#show interfaces Async 5
Async5 is up, line protocol is up
Hardware is Async Serial
Internet address is 10.10.10.10/24
MTU 1500 bytes, BW 9 Kbit, DLY 100000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, loopback not set
Keepalive not set
DTR is pulsed for 5 seconds on reset
LCP Open
Open: IPCP
Last input 00:00:09, output 00:00:09, output hang never
Last clearing of "show interface" counters 00:29:33
Input queue: 1/75/0 (size/max/drops); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/16 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
615 packets input, 45313 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
565 packets output, 35812 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
router#
|
This is running a workstation via PPP through the Aux port on a Cisco 1720 router we have in our lab. More information on this configuration is available in this article.
Now, let's kill the network connection with a ping flood:
[root@srv-1 usr-1]# ping -f -s 1000 10.10.10.11
PING 10.10.10.11 (10.10.10.11) 1000(1028) bytes of data.
................................................................................
|
You can see the ping stats climb:
64 bytes from 10.10.10.11: icmp_seq=3 ttl=127 time=18.6 ms
64 bytes from 10.10.10.11: icmp_seq=4 ttl=127 time=18.6 ms
64 bytes from 10.10.10.11: icmp_seq=5 ttl=127 time=4882 ms
64 bytes from 10.10.10.11: icmp_seq=34 ttl=127 time=6474 ms
64 bytes from 10.10.10.11: icmp_seq=47 ttl=127 time=6697 ms
64 bytes from 10.10.10.11: icmp_seq=53 ttl=127 time=6787 ms
64 bytes from 10.10.10.11: icmp_seq=68 ttl=127 time=7011 ms
64 bytes from 10.10.10.11: icmp_seq=69 ttl=127 time=6935 ms
64 bytes from 10.10.10.11: icmp_seq=87 ttl=127 time=7327 ms
64 bytes from 10.10.10.11: icmp_seq=88 ttl=127 time=7252 ms
|
We are starting to see some delays between 100 and 101 on the AreWeDown tool:
| 2005-08-06 08:25:17 | are@10.10.10.11 | 100 |
| 2005-08-06 08:25:18 | are@10.10.10.11 | 101 |
| 2005-08-06 08:25:21 | are@10.50.100.190 | 100 |
| 2005-08-06 08:25:21 | are@10.50.100.190 | 101 |
| 2005-08-06 08:25:49 | are@10.10.10.11 | 100 |
| 2005-08-06 08:25:50 | are@10.50.100.190 | 100 |
| 2005-08-06 08:25:50 | are@10.50.100.190 | 101 |
| 2005-08-06 08:25:51 | are@10.10.10.11 | 101 |
|
Our output queue is at the drop threshold on the router:
router#show interfaces Async 5
Async5 is up, line protocol is up
Hardware is Async Serial
Internet address is 10.10.10.10/24
MTU 1500 bytes, BW 9 Kbit, DLY 100000 usec,
reliability 255/255, txload 141/255, rxload 140/255
Encapsulation PPP, loopback not set
Keepalive not set
DTR is pulsed for 5 seconds on reset
LCP Open
Open: IPCP
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:35:37
Input queue: 1/75/0 (size/max/drops); Total output drops: 7997
Queueing strategy: weighted fair
Output queue: 64/1000/64/7997 (size/max total/threshold/drops)
Conversations 1/2/16 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
5 minute input rate 49000 bits/sec, 11 packets/sec
5 minute output rate 50000 bits/sec, 11 packets/sec
2347 packets input, 1714050 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
3 input errors, 2 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort
2324 packets output, 1734040 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
router#
|
Things are getting worse:
| 2005-08-06 08:27:26 | are@10.10.10.11 | 100 |
| 2005-08-06 08:27:29 | are@10.10.10.11 | 101 |
| 2005-08-06 08:27:51 | are@10.50.100.190 | 100 |
| 2005-08-06 08:27:51 | are@10.50.100.190 | 101 |
| 2005-08-06 08:27:58 | are@10.10.10.11 | 100 |
| 2005-08-06 08:28:02 | are@10.10.10.11 | 101 |
|
Four seconds, now, between 100 and 101. Our ping:
64 bytes from 10.10.10.11: icmp_seq=209 ttl=127 time=8998 ms
64 bytes from 10.10.10.11: icmp_seq=215 ttl=127 time=9254 ms
64 bytes from 10.10.10.11: icmp_seq=217 ttl=127 time=9186 ms
64 bytes from 10.10.10.11: icmp_seq=262 ttl=127 time=9989 ms
|
We are now unresponsive:
| 2005-08-06 08:29:36 | are@10.10.10.11 | 100 |
| 2005-08-06 08:29:41 | are@10.10.10.11 | 101 |
| 2005-08-06 08:29:51 | are@10.50.100.190 | 100 |
| 2005-08-06 08:29:51 | are@10.50.100.190 | 101 |
| 2005-08-06 08:30:21 | are@10.50.100.190 | 100 |
| 2005-08-06 08:30:21 | are@10.50.100.190 | 101 |
|
We should see an entry every 30 seconds, but .11 has stopped sending requests. Our ping responses are climbing as well:
64 bytes from 10.10.10.11: icmp_seq=385 ttl=127 time=11831 ms
64 bytes from 10.10.10.11: icmp_seq=386 ttl=127 time=11756 ms
64 bytes from 10.10.10.11: icmp_seq=387 ttl=127 time=11680 ms
64 bytes from 10.10.10.11: icmp_seq=388 ttl=127 time=11604 ms
64 bytes from 10.10.10.11: icmp_seq=406 ttl=127 time=12122 ms
|
Our router stats:
Async5 is up, line protocol is up
Hardware is Async Serial
Internet address is 10.10.10.10/24
MTU 1500 bytes, BW 9 Kbit, DLY 100000 usec,
reliability 255/255, txload 252/255, rxload 255/255
Encapsulation PPP, loopback not set
Keepalive not set
DTR is pulsed for 5 seconds on reset
LCP Open
Open: IPCP
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:40:27
Input queue: 1/75/0 (size/max/drops); Total output drops: 23735
Queueing strategy: weighted fair
Output queue: 64/1000/64/23735 (size/max total/threshold/drops)
Conversations 1/2/16 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
5 minute input rate 88000 bits/sec, 10 packets/sec
5 minute output rate 91000 bits/sec, 11 packets/sec
5632 packets input, 4974777 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
3 input errors, 2 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort
5652 packets output, 5042433 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
router#
|
Our output queue is still at the threshold and we are dropping a lot of packets. One fix would be to disallow ICMP. But, we are going to assume that we want ICMP allowed. Another fix for this is to apply traffic shaping:
router#conf term
Enter configuration commands, one per line. End with CNTL/Z.
router(config)#int Async 5
router(config-if)#traffic-shape rate 80000
router(config-if)#exit
router(config)#exit
router#
|
Our output queue is back down:
router#show interfaces Async 5
Async5 is up, line protocol is up
Hardware is Async Serial
Internet address is 10.10.10.10/24
MTU 1500 bytes, BW 9 Kbit, DLY 100000 usec,
reliability 255/255, txload 18/255, rxload 255/255
Encapsulation PPP, loopback not set
Keepalive not set
DTR is pulsed for 5 seconds on reset
LCP Open
Open: IPCP
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:43:08
Input queue: 1/75/0 (size/max/drops); Total output drops: 32551
Queueing strategy: weighted fair
Output queue: 0/1000/64/30046 (size/max total/threshold/drops)
Conversations 0/2/16 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
5 minute input rate 89000 bits/sec, 11 packets/sec
5 minute output rate 86000 bits/sec, 10 packets/sec
7439 packets input, 6773226 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
11 input errors, 10 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort
7475 packets output, 6851326 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
router#
|
Our client can now talk again:
| 2005-08-06 08:33:21 | are@10.50.100.190 | 101 |
| 2005-08-06 08:33:51 | are@10.50.100.190 | 100 |
| 2005-08-06 08:33:51 | are@10.50.100.190 | 101 |
| 2005-08-06 08:34:21 | are@10.50.100.190 | 100 |
| 2005-08-06 08:34:21 | are@10.50.100.190 | 101 |
| 2005-08-06 08:34:51 | are@10.50.100.190 | 100 |
| 2005-08-06 08:34:51 | are@10.50.100.190 | 101 |
| 2005-08-06 08:35:08 | are@10.10.10.11 | 100 |
| 2005-08-06 08:35:10 | are@10.10.10.11 | 101 |
| 2005-08-06 08:35:20 | are@10.10.10.11 | 100 |
| 2005-08-06 08:35:21 | are@10.50.100.190 | 100 |
| 2005-08-06 08:35:21 | are@10.50.100.190 | 101 |
| 2005-08-06 08:35:21 | are@10.10.10.11 | 101 |
| 2005-08-06 08:35:43 | are@10.10.10.11 | 100 |
| 2005-08-06 08:35:43 | are@10.10.10.11 | 101 |
|
Now, the ping times from the flood machine are still high:
64 bytes from 10.10.10.11: icmp_seq=538 ttl=127 time=14117 ms
64 bytes from 10.10.10.11: icmp_seq=557 ttl=127 time=14413 ms
64 bytes from 10.10.10.11: icmp_seq=558 ttl=127 time=14338 ms
64 bytes from 10.10.10.11: icmp_seq=559 ttl=127 time=18114 ms
|
But, from another machine:
[root@pippi ~]# ping 10.10.10.11
PING 10.10.10.11 (10.10.10.11) 56(84) bytes of data.
64 bytes from 10.10.10.11: icmp_seq=0 ttl=54 time=80.3 ms
64 bytes from 10.10.10.11: icmp_seq=1 ttl=54 time=81.5 ms
64 bytes from 10.10.10.11: icmp_seq=2 ttl=54 time=81.5 ms
64 bytes from 10.10.10.11: icmp_seq=3 ttl=54 time=82.8 ms
|
If we then ping flood from that same host, it will eventually drop most of the packets:
64 bytes from 10.10.10.11: icmp_seq=2 ttl=54 time=774 ms
64 bytes from 10.10.10.11: icmp_seq=8 ttl=54 time=763 ms
--- 10.10.10.11 ping statistics ---
27 packets transmitted, 2 received, 92% loss, time 26204ms
rtt min/avg/max/mdev = 763.927/769.305/774.684/5.449 ms
[root@mondo root]#
|
This is as it should be.
For more information on Cisco traffic shaping, see this article.
|
|
