在所有的子系統監控中,網(wǎng)絡(luò )是最困難的.這主要是由于網(wǎng)絡(luò )概念很抽象.當監控系統上的網(wǎng)絡(luò )性能,這有太多因素.這些因素包括了延遲,沖突,擁擠和數據包丟失.
這個(gè)章節討論怎么樣檢查Ethernet(譯注:網(wǎng)卡),IP,TCP的性能.
8.1 Ethernet Configuration Settings(譯注:網(wǎng)卡配置的設置)
除非很明確的指定,幾乎所有的網(wǎng)卡都是自適應網(wǎng)絡(luò )速度.當一個(gè)網(wǎng)絡(luò )中有很多不同的網(wǎng)絡(luò )設備時(shí),會(huì )各自采用不同的速率和工作模式.
多數商業(yè)網(wǎng)絡(luò )都運行在100 或 1000BaseTX.使用ethtool 可以確定這個(gè)系統是處于那種速率.
以下的例子中,是一個(gè)有100BaseTX 網(wǎng)卡的系統,自動(dòng)協(xié)商適應至10BaseTX 的情況.
# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: Yes
Speed: 10Mb/s
Duplex: Half
Port: MII
PHYAD: 32
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: d
Current message level: 0×00000007 (7)
Link detected: yes
以下示范例子中,如何強制網(wǎng)卡速率調整至100BaseTX:
# ethtool -s eth0 speed 100 duplex full autoneg off
# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: No
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 32
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: pumbg
Wake-on: d
Current message level: 0×00000007 (7)
Link detected: yes
8.2 Monitoring Network Throughput(譯注:網(wǎng)絡(luò )吞吐量監控)
接口之間的同步并不意味著(zhù)僅僅有帶寬問(wèn)題.重要的是,如何管理并優(yōu)化,這2臺主機之間的交換機,網(wǎng)線(xiàn),或者路由器.測試網(wǎng)絡(luò )吞吐量最好的方式就是,在這2個(gè)系統之間互相發(fā)送數據傳輸并統計下來(lái),比如延遲和速度.
8.2.0 使用iptraf 查看本地吞吐量
iptraf 工具(http://iptraf.seul.org),提供了每個(gè)網(wǎng)卡吞吐量的儀表盤(pán).
#iptraf -d eth0
Figure 1: Monitoring for Network Throughput
從輸出中可看到,該系統發(fā)送傳輸率(譯注:Outgoing rates)為 61 mbps,這對于100 mbps網(wǎng)絡(luò )來(lái)說(shuō),有點(diǎn)慢.
8.2.1 使用netperf 查看終端吞吐量
不同于iptraf 被動(dòng)的在本地監控流量,netperf 工具可以讓管理員,執行更加可控的吞吐量監控.對于確定從客戶(hù)端工作站到一個(gè)高負荷的服務(wù)器端(比如file 或web server),它們之間有多少吞吐量是非常有幫助的.netperf 工具運行的是client/server 模式.
完成一個(gè)基本可控吞吐量測試,首先netperf server 必須運行在服務(wù)器端系統上:
server# netserver
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC
netperf 工具可能需要進(jìn)行多重采樣.多數基本測試就是一次標準的吞吐量測試.以下例子就是,一個(gè)LAN(譯注:局域網(wǎng)) 環(huán)境下,從client 上執行一次30秒的TCP 吞吐量采樣:
從輸出可看出,該網(wǎng)絡(luò )的吞吐量大致在89 mbps 左右.server(192.168.1.215) 與client 在同一LAN 中.這對于100 mbps網(wǎng)絡(luò )來(lái)說(shuō),性能非常好.
client# netperf -H 192.168.1.215 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.230 (192.168.1.230) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 30.02 89.46
從LAN 切換到具備54G(譯注:Wireless-G是未來(lái)54Mbps無(wú)線(xiàn)網(wǎng)聯(lián)網(wǎng)標準)無(wú)線(xiàn)網(wǎng)絡(luò )路由器中,并在10 英尺范圍內測試時(shí).該吞吐量就急劇的下降.在最大就為54 MBits的可能下,筆記本電腦可實(shí)現總吞吐量就為14 MBits.
client# netperf -H 192.168.1.215 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.215 (192.168.1.215) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 30.10 14.09
如果在50英尺范圍內呢,則進(jìn)一步會(huì )下降至5 MBits.
# netperf -H 192.168.1.215 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.215 (192.168.1.215) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 30.64 5.05
如果從LAN 切換到互聯(lián)網(wǎng)上,則吞吐量跌至1 Mbits下了.
# netperf -H litemail.org -p 1500 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
litemail.org (72.249.104.14
port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 31.58 0.93
最后是一個(gè)VPN 連接環(huán)境,這是所有網(wǎng)絡(luò )環(huán)境中最槽糕的吞吐量了.
# netperf -H 10.0.1.129 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.0.1.129 (10.0.1.129) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 31.99 0.51
另外,netperf 可以幫助測試每秒總計有多少的TCP 請求和響應數.通過(guò)建立單一TCP 連接并順序地發(fā)送多個(gè)請求/響應(ack 包來(lái)回在1個(gè)byte 大小).有點(diǎn)類(lèi)似于RDBMS 程序在執行多個(gè)交易或者郵件服務(wù)器在同一個(gè)連接管道中發(fā)送郵件.
以下例子在30 秒的持續時(shí)間內,模擬TCP 請求/響應:
client# netperf -t TCP_RR -H 192.168.1.230 -l 30
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 192.168.1.230 (192.168.1.230) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 30.00 4453.80
16384 87380
在輸出中看出,這個(gè)網(wǎng)絡(luò )支持的處理速率為每秒4453 psh/ack(包大小為1 byte).這其實(shí)是理想狀態(tài)下,因為實(shí)際情況時(shí),多數requests(譯注:請求),特別是responses(譯注:響應),都大于1 byte.
現實(shí)情況下,netperf 一般requests 默認使用2K大小,responses 默認使用32K大小:
client# netperf -t TCP_RR -H 192.168.1.230 -l 30 — -r 2048,32768
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.230 (192.168.1.230) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
16384 87380 2048 32768 30.00 222.37
16384 87380
這個(gè)處理速率減少到了每秒222.
8.2.2 使用iperf 評估網(wǎng)絡(luò )效率
基于都是需要在2端檢查連接情況下,iperf 和netperf 很相似.不同的是,iperf 更深入的通過(guò)windows size和QOS 設備來(lái)檢查T(mén)CP/UDP 的效率情況.這個(gè)工具,是給需要優(yōu)化TCP/IP stacks以及測試這些stacks 效率的管理員們量身定做的.
iperf 作為一個(gè)二進(jìn)制程序,可運行在server 或者client 任一模式下.默認使用50001 端口.
首先啟動(dòng)server 端(192.168.1.215):
server# iperf -s -D
Running Iperf Server as a daemon
The Iperf daemon process ID : 3655
————————————————————
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
————————————————————
在以下例子里,一個(gè)無(wú)線(xiàn)網(wǎng)絡(luò )環(huán)境下,其中client 端重復運行iperf,用于測試網(wǎng)絡(luò )的吞吐量情況.這個(gè)環(huán)境假定處于被充分利用狀態(tài),很多主機都在下載ISO images文件.
首先client 端連接到server 端(192.168.1.215),并在總計60秒時(shí)間內,每5秒進(jìn)行一次帶寬測試的采樣.
client# iperf -c 192.168.1.215 -t 60 -i 5
————————————————————
Client connecting to 192.168.1.215, TCP port 5001
TCP window size: 25.6 KByte (default)
————————————————————
[ 3] local 192.168.224.150 port 51978 connected with
192.168.1.215 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 5.0 sec 6.22 MBytes 10.4 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 5.0-10.0 sec 6.05 MBytes 10.1 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 10.0-15.0 sec 5.55 MBytes 9.32 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 15.0-20.0 sec 5.19 MBytes 8.70 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 20.0-25.0 sec 4.95 MBytes 8.30 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 25.0-30.0 sec 5.21 MBytes 8.74 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 30.0-35.0 sec 2.55 MBytes 4.29 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 35.0-40.0 sec 5.87 MBytes 9.84 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 40.0-45.0 sec 5.69 MBytes 9.54 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 45.0-50.0 sec 5.64 MBytes 9.46 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 50.0-55.0 sec 4.55 MBytes 7.64 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 55.0-60.0 sec 4.47 MBytes 7.50 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 61.9 MBytes 8.66 Mbits/sec
這臺主機的其他網(wǎng)絡(luò )傳輸,也會(huì )影響到這部分的帶寬采樣.所以可以看到總計60秒時(shí)間內,都在4 - 10 MBits 上下起伏.
除了TCP 測試之外,iperf 的UDP 測試主要是評估包丟失和抖動(dòng).
接下來(lái)的iperf 測試,是在同樣的54Mbit G標準無(wú)線(xiàn)網(wǎng)絡(luò )中.在早期的示范例子中,目前的吞吐量只有9 Mbits.
# iperf -c 192.168.1.215 -b 10M
WARNING: option -b implies udp testing
————————————————————
Client connecting to 192.168.1.215, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 107 KByte (default)
————————————————————
[ 3] local 192.168.224.150 port 33589 connected with 192.168.1.215 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.8 MBytes 9.90 Mbits/sec
[ 3] Sent 8420 datagrams
[ 3] Server Report:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 6.50 MBytes 5.45 Mbits/sec 0.480 ms 3784/ 8419 (45%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
從輸出中可看出,在嘗試傳輸10M 的數據時(shí),實(shí)際上只產(chǎn)生了5.45M.卻有45% 的包丟失.
8.3 Individual Connections with tcptrace
tcptrace 工具提供了對于某一具體連接里,詳細的TCP 相關(guān)信息.該工具使用libcap 來(lái)分析某一具體TCP sessions.該工具匯報的信息,有時(shí)很難在某一TCP stream被發(fā)現.這些信息
包括了有:
1,TCP Retransmissions(譯注:IP 轉播) - 所有數據大小被發(fā)送所需的包總額
2,TCP Windows Sizes - 連接速度慢與小的windows sizes 有關(guān)
3,Total throughput of the connection - 連接的吞吐量
4,Connection duration - 連接的持續時(shí)間
8.3.1 案例學(xué)習 - 使用tcptrace
tcptrace 工具可能已經(jīng)在部分Linux 發(fā)布版中有安裝包了,該文作者通過(guò)網(wǎng)站,下載的是源碼安裝包:http://dag.wieers.com/rpm/packages /tcptrace.tcptrace 需要libcap 基于文件輸入方式使用.在tcptrace 沒(méi)有選項的情況下,默認每個(gè)唯一的連接過(guò)程都將被捕獲.
以下例子是,使用libcap 基于輸入文件為bigstuff:
# tcptrace bigstuff
1 arg remaining, starting with ‘bigstuff’
Ostermann’s tcptrace — version 6.6.7 — Thu Nov 4, 2004
146108 packets seen, 145992 TCP packets traced
elapsed wallclock time: 0:00:01.634065, 89413 pkts/sec analyzed
trace file elapsed time: 0:09:20.358860
TCP connection info:
1: 192.168.1.60:pcanywherestat - 192.168.1.102:2571 (a2b) 404> 450< 2: 192.168.1.60:3356 - ftp.strongmail.net:21 (c2d) 35> 21< 3: 192.168.1.60:3825 - ftp.strongmail.net:65023 (e2f) 5> 4< (complete) 4: 192.168.1.102:1339 - 205.188.8.194:5190 (g2h) 6> 6< 5: 192.168.1.102:1490 - cs127.msg.mud.yahoo.com:5050 (i2j) 5> 5< 6: py-in-f111.google.com:993 - 192.168.1.102:3785 (k2l) 13> 14<
上面的輸出中,每個(gè)連接都有對應的源主機和目的主機.tcptrace 使用-l 和-o 選項可查看某一連接更詳細的數據.
以下的結果,就是在bigstuff 文件中,#16 連接的相關(guān)統計數據:
# tcptrace -l -o1 bigstuff
1 arg remaining, starting with ‘bigstuff’
Ostermann’s tcptrace — version 6.6.7 — Thu Nov 4, 2004
146108 packets seen, 145992 TCP packets traced
elapsed wallclock time: 0:00:00.529361, 276008 pkts/sec analyzed
trace file elapsed time: 0:09:20.358860
TCP connection info:
32 TCP connections traced:
TCP connection 1:
host a: 192.168.1.60:pcanywherestat
host b: 192.168.1.102:2571
complete conn: no (SYNs: 0) (FINs: 0)
first packet: Sun Jul 20 15:58:05.472983 2008
last packet: Sun Jul 20 16:00:04.564716 2008
elapsed time: 0:01:59.091733
total packets: 854
filename: bigstuff
a->b: b->a:
total packets: 404 total packets: 450
ack pkts sent: 404 ack pkts sent: 450
pure acks sent: 13 pure acks sent: 320
sack pkts sent: 0 sack pkts sent: 0
dsack pkts sent: 0 dsack pkts sent: 0
max sack blks/ack: 0 max sack blks/ack: 0
unique bytes sent: 52608 unique bytes sent: 10624
actual data pkts: 391 actual data pkts: 130
actual data bytes: 52608 actual data bytes: 10624
rexmt data pkts: 0 rexmt data pkts: 0
rexmt data bytes: 0 rexmt data bytes: 0
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 0 outoforder pkts: 0
pushed data pkts: 391 pushed data pkts: 130
SYN/FIN pkts sent: 0/0 SYN/FIN pkts sent: 0/0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 0 bytes mss requested: 0 bytes
max segm size: 560 bytes max segm size: 176 bytes
min segm size: 48 bytes min segm size: 80 bytes
avg segm size: 134 bytes avg segm size: 81 bytes
max win adv: 19584 bytes max win adv: 65535 bytes
min win adv: 19584 bytes min win adv: 64287 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 19584 bytes avg win adv: 64949 bytes
initial window: 160 bytes initial window: 0 bytes
initial window: 2 pkts initial window: 0 pkts
ttl stream length: NA ttl stream length: NA
missed data: NA missed data: NA
truncated data: 36186 bytes truncated data: 5164 bytes
truncated packets: 391 pkts truncated packets: 130 pkts
data xmit time: 119.092 secs data xmit time: 116.954 secs
idletime max: 441267.1 ms idletime max: 441506.3 ms
throughput: 442 Bps throughput: 89 Bps
8.3.2 案例學(xué)習 - 計算轉播率
幾乎不可能確定說(shuō)哪個(gè)連接會(huì )有嚴重不足的轉播問(wèn)題,只是需要分析,使用tcptrace 工具可以通過(guò)過(guò)濾機制和布爾表達式來(lái)找出出問(wèn)題的連接.一個(gè)很繁忙的網(wǎng)絡(luò )中,會(huì )有很多的連接,幾乎所有的連接都會(huì )有轉播.找出其中最多的一個(gè),這就是問(wèn)題的關(guān)鍵.
下面的例子里,tcptrace 將找出那些轉播大于100 segments(譯注:分段數)的連接:
# tcptrace -f’rexmit_segs>100′ bigstuff
Output filter: ((c_rexmit_segs>100)OR(s_rexmit_segs>100))
1 arg remaining, starting with ‘bigstuff’
Ostermann’s tcptrace — version 6.6.7 — Thu Nov 4, 2004
146108 packets seen, 145992 TCP packets traced
elapsed wallclock time: 0:00:00.687788, 212431 pkts/sec analyzed
trace file elapsed time: 0:09:20.358860
TCP connection info:
16: ftp.strongmail.net:65014 - 192.168.1.60:2158 (ae2af) 18695> 9817< 在這個(gè)輸出中,是#16 這個(gè)連接里,超過(guò)了100 轉播.現在,使用以下命令查看關(guān)于這個(gè)連接的其他信息: # tcptrace -l -o16 bigstuff arg remaining, starting with ‘bigstuff’ Ostermann’s tcptrace — version 6.6.7 — Thu Nov 4, 2004 146108 packets seen, 145992 TCP packets traced elapsed wallclock time: 0:00:01.355964, 107752 pkts/sec analyzed trace file elapsed time: 0:09:20.358860 TCP connection info: 32 TCP connections traced: ================================ TCP connection 16: host ae: ftp.strongmail.net:65014 host af: 192.168.1.60:2158 complete conn: no (SYNs: 0) (FINs: 1) first packet: Sun Jul 20 16:04:33.257606 2008 last packet: Sun Jul 20 16:07:22.317987 2008 elapsed time: 0:02:49.060381 total packets: 28512 filename: bigstuff ae->af: af->ae:
unique bytes sent: 25534744 unique bytes sent: 0
actual data pkts: 18695 actual data pkts: 0
actual data bytes: 25556632 actual data bytes: 0
rexmt data pkts: 1605 rexmt data pkts: 0
rexmt data bytes: 2188780 rexmt data bytes: 0
計算轉播率:
rexmt/actual * 100 = Retransmission rate
1605/18695* 100 = 8.5%
這個(gè)慢連接的原因,就是因為它有8.5% 的轉播率.
8.3.3 案例學(xué)習 - 計算轉播時(shí)間
tcptrace 工具有一系列的模塊展示不同的數據,按照屬性,其中就有protocol(譯注:協(xié)議),port(譯注:端口),time等等.Slice module使得你可觀(guān)察在一段時(shí)間內的TCP 性能.你可以在一系列的轉發(fā)過(guò)程中,查看其他性能數據,以確定找出瓶頸.
以下例子示范了,tcptrace 是怎樣使用slice 模式的:
# tcptrace –xslice bigfile
以上命令會(huì )創(chuàng )建一個(gè)slice.dat 文件在現在的工作目錄中.這個(gè)文件內容,包含是每15秒間隔內轉播的相關(guān)信息:
# ls -l slice.dat
-rw-r–r– 1 root root 3430 Jul 10 22:50 slice.dat
# more slice.dat
date segs bytes rexsegs rexbytes new active
————— ——– ——– ——– ——– ——– ——–
22:19:41.913288 46 5672 0 0 1 1
22:19:56.913288 131 25688 0 0 0 1
22:20:11.913288 0 0 0 0 0 0
22:20:26.913288 5975 4871128 0 0 0 1
22:20:41.913288 31049 25307256 0 0 0 1
22:20:56.913288 23077 19123956 40 59452 0 1
22:21:11.913288 26357 21624373 5 7500 0 1
22:21:26.913288 20975 17248491 3 4500 12 13
22:21:41.913288 24234 19849503 10 15000 3 5
22:21:56.913288 27090 22269230 36 53999 0 2
22:22:11.913288 22295 18315923 9 12856 0 2
22:22:26.913288 8858 7304603 3 4500 0 1
8.4 結論
監控網(wǎng)絡(luò )性能由以下幾個(gè)部分組成:
1,檢查并確定所有網(wǎng)卡都工作在正確的速率.
2,檢查每塊網(wǎng)卡的吞吐量,并確認其處于服務(wù)時(shí)的網(wǎng)絡(luò )速度.
