天天看点

园区间业务系统访问故障处理

故障现象:

客户反映园区仓库访问NC系统速度很慢,从园区ping NC服务器地址时有严重的丢包现象。如下图所示:

<a href="http://s3.51cto.com/wyfs02/M00/8A/8C/wKiom1gz1DKi117aAAKQFNfloAw338.png-wh_500x0-wm_3-wmp_4-s_1579186669.png" target="_blank"></a>

<a href="http://s1.51cto.com/wyfs02/M02/8A/8C/wKiom1gz1DOx2rwHAAFuqD6gfac287.png-wh_500x0-wm_3-wmp_4-s_2366122665.png" target="_blank"></a>

处理过程:

登录用户两个园区的核心发现,两个园区的portchannel连接正常,但是其中一个园区portchannel成员端口下存在严重的CRC校验错误,如下图所示:

<a href="http://s1.51cto.com/wyfs02/M02/8A/89/wKioL1gz08HDzqxmAALCiT0OPh0626.png-wh_500x0-wm_3-wmp_4-s_3053852100.png" target="_blank"></a>

此时,关闭端口ten1/5/1,数据转发从另一个端口走,关闭后再次PING NC服务器,数据包正常

<a href="http://s3.51cto.com/wyfs02/M02/8A/89/wKioL1gz1KDgfrB5AAFB4YDNDbs447.png-wh_500x0-wm_3-wmp_4-s_1719984639.png" target="_blank"></a>

<a href="http://s3.51cto.com/wyfs02/M00/8A/8C/wKiom1gz1KCTa0kNAAFlB_3Ct-w146.png-wh_500x0-wm_3-wmp_4-s_3799085379.png" target="_blank"></a>

为进一步确认该故障,再次对ten1/5/1进行开启操作,但打开后发现,问题立即出现

SW#ping 192.168.x.x repeat 1000 source 192.168.x.x

Type escape sequence to abort.

Sending 1000, 100-byte ICMP Echos to 192.168.x.x, timeout is 2 seconds:

Packet sent with a source address of 192.168.x.x

!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!.!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!

!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!.!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!

!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 97 percent (978/1000), round-trip min/avg/max = 1/2/20 ms

SW#sh int ten1/5/1

TenGigabitEthernet1/5/1 is up, line protocol is up (connected)

  Hardware is Ten Gigabit Ethernet Port, address is 78da.6e07.7b7a (bia 78da.6e07.7b7a)

  Description: connection to laishanchangqu

  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,

     reliability 245/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-ER

  input flow-control is on, output flow-control is on

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:03, output never, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 3209000 bits/sec, 358 packets/sec

  5 minute output rate 926000 bits/sec, 685 packets/sec

     4935914010 packets input, 5278619447870 bytes, 0 no buffer

     Received 87068 broadcasts (87050 multicasts)

     0 runts, 0 giants, 0 throttles

     277020288 input errors, 277020288 CRC, 0 frame, 0 overrun, 0 ignored

     0 input packets with dribble condition detected

     9064745454 packets output, 3287137727436 bytes, 0 underruns

     0 output errors, 0 collisions, 4 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

SW#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

SW(config)#int ten1/5/1

SW(config-if)#shut

SW(config-if)#do sh run int ten1/5/1

Building configuration...

Current configuration : 171 bytes

!

interface TenGigabitEthernet1/5/1

 switchport access vlan 133

 switchport mode access

 shutdown

 channel-group 10 mode on

end

SW(config-if)#exit

SW(config)#exit

SW#wr

Compressed configuration from 21267 bytes to 7568 bytes[OK]

Sending 1000, 100-byte ICMP Echos to 192.168.x.x timeout is 2 seconds:

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/2/20 ms

经最终确认,NC访问问题主要是由于该端口的CRC校验错误引起。

解决建议:

1、检查两厂区之间的链路

2、检查模块

3、检查尾纤

                                                               *********************

最终:

       第二天测试两个园区之间的链路,发现一根光纤的衰减在-21左右,而该接口上的万兆模块可接受的衰减最小值是-15.8,后来运营商对线路进行调优检测,最终恢复正常,测试,网络访问正常

本文转自 弯月楼主 51CTO博客,原文链接:http://blog.51cto.com/05wylz/1875378,如需转载请自行联系原作者