怀化主要网站,网站建设 自查表,怎么用网站开发者工具更换网页,网上注册公司审核需要多久文章目录 1. 前言2. 问题描述3. 问题简析 1. 前言
限于作者能力水平#xff0c;本文可能存在谬误#xff0c;因此而给读者带来的损失#xff0c;作者不做任何承诺。
2. 问题描述
在 TI 的 AM335x 平台适配了一个 1000Mbps 的 PHY 芯片#xff0c;通过 iperf 打流#x… 文章目录 1. 前言2. 问题描述3. 问题简析 1. 前言
限于作者能力水平本文可能存在谬误因此而给读者带来的损失作者不做任何承诺。
2. 问题描述
在 TI 的 AM335x 平台适配了一个 1000Mbps 的 PHY 芯片通过 iperf 打流TCP 带宽测试结果如下
# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.201, port 55944
[ 5] local 192.168.1.88 port 5201 connected to 192.168.1.201 port 55945
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 51.5 MBytes 431 Mbits/sec
[ 5] 1.00-2.00 sec 53.0 MBytes 444 Mbits/sec
[ 5] 2.00-3.00 sec 52.1 MBytes 438 Mbits/sec
[ 5] 3.00-4.00 sec 52.6 MBytes 442 Mbits/sec
[ 5] 4.00-5.00 sec 53.0 MBytes 443 Mbits/sec
[ 5] 5.00-6.00 sec 52.4 MBytes 441 Mbits/sec
[ 5] 6.00-7.00 sec 52.0 MBytes 435 Mbits/sec
[ 5] 7.00-8.00 sec 52.1 MBytes 438 Mbits/sec
[ 5] 8.00-9.00 sec 52.2 MBytes 439 Mbits/sec
[ 5] 9.00-10.00 sec 53.0 MBytes 444 Mbits/sec
[ 5] 10.00-11.00 sec 52.9 MBytes 442 Mbits/sec
[ 5] 11.00-12.00 sec 52.9 MBytes 445 Mbits/sec
[ 5] 12.00-13.00 sec 52.4 MBytes 439 Mbits/sec
[ 5] 13.00-14.00 sec 52.6 MBytes 441 Mbits/sec
[ 5] 14.00-15.00 sec 51.6 MBytes 433 Mbits/sec
[ 5] 15.00-16.00 sec 52.5 MBytes 442 Mbits/sec
[ 5] 16.00-17.00 sec 52.2 MBytes 438 Mbits/sec
[ 5] 17.00-18.00 sec 52.1 MBytes 437 Mbits/sec
[ 5] 18.00-19.00 sec 52.1 MBytes 438 Mbits/sec
[ 5] 19.00-19.54 sec 27.8 MBytes 431 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-19.54 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-19.54 sec 1023 MBytes 439 Mbits/sec receiver
-----------------------------------------------------------# iperf3 -c 192.168.1.201 -p 5201 -n 2G
Connecting to host 192.168.1.201, port 5201
[ 4] local 192.168.1.88 port 45292 connected to 192.168.1.201 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.02 sec 45.0 MBytes 371 Mbits/sec 0 130 KBytes
[ 4] 1.02-2.01 sec 43.8 MBytes 370 Mbits/sec 0 130 KBytes
[ 4] 2.01-3.02 sec 45.0 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 3.02-4.01 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 4.01-5.02 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 5.02-6.01 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 6.01-7.02 sec 45.0 MBytes 374 Mbits/sec 0 130 KBytes
[ 4] 7.02-8.00 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 8.00-9.01 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 9.01-10.03 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 10.03-11.01 sec 43.8 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 11.01-12.02 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 12.02-13.00 sec 43.8 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 13.00-14.02 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 14.02-15.00 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 15.00-16.02 sec 45.0 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 16.02-17.00 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 17.00-18.02 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 18.02-19.00 sec 43.8 MBytes 371 Mbits/sec 0 130 KBytes
[ 4] 19.00-20.01 sec 45.0 MBytes 374 Mbits/sec 0 130 KBytes
[ 4] 20.01-21.03 sec 45.0 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 21.03-22.01 sec 43.8 MBytes 374 Mbits/sec 0 130 KBytes
[ 4] 22.01-23.02 sec 45.0 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 23.02-24.01 sec 43.8 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 24.01-25.03 sec 45.0 MBytes 371 Mbits/sec 0 130 KBytes
[ 4] 25.03-26.01 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 26.01-27.00 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 27.00-28.01 sec 45.0 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 28.01-29.00 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 29.00-30.01 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 30.01-31.00 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 31.00-32.02 sec 45.0 MBytes 371 Mbits/sec 0 130 KBytes
[ 4] 32.02-33.01 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 33.01-34.02 sec 45.0 MBytes 374 Mbits/sec 0 130 KBytes
[ 4] 34.02-35.03 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 35.03-36.01 sec 43.8 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 36.01-37.03 sec 45.0 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 37.03-38.02 sec 43.8 MBytes 371 Mbits/sec 0 130 KBytes
[ 4] 38.02-39.00 sec 43.8 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 39.00-40.01 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 40.01-41.02 sec 45.0 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 41.02-42.01 sec 43.8 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 42.01-43.03 sec 45.0 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 43.03-44.02 sec 43.8 MBytes 371 Mbits/sec 0 130 KBytes
[ 4] 44.02-45.00 sec 43.8 MBytes 373 Mbits/sec 0 130 KBytes
[ 4] 45.00-46.02 sec 45.0 MBytes 372 Mbits/sec 0 130 KBytes
[ 4] 46.02-46.16 sec 6.25 MBytes 368 Mbits/sec 0 130 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-46.16 sec 2.00 GBytes 372 Mbits/sec 0 sender
[ 4] 0.00-46.16 sec 2.00 GBytes 372 Mbits/sec receiveriperf Done.可以看到TCP 带宽测试接收(RX)方向的平均带宽为 439 Mbits/sec发送方向的平均带宽为 372 Mbits/sec无论哪个方向都离 1000Mbps 的很远。 进行 UDP 带宽测试在 TI AM335x 一侧运行的 iperf 作为 server 时发现有极高的丢包率。
3. 问题简析
先交代下背景TI 的 AM335x CPU 是单核的最高频率为 1GHz系统 RAM 为 256MB。另外测试当中去掉所有无干或可能产生对测试产生干扰的程序。首先排除硬件相关的问题譬如网线、变压器、电阻电容以及上电、复位时序都有可能造成网络通信异常但不属于本文案例中的情形。既然排查了硬件问题剩下的自然是排查软件问题。在进行 TCP 带宽测试时(无论是 server 还是 client 模式运行)首先用 top 观察了基础状况
# top
Mem: 25632K used, 220844K free, 180K shrd, 0K buff, 1528K cached
CPU: 0% usr 63% sys 0% nic 0% idle 0% io 0% irq 36% sirq
Load average: 0.45 0.17 0.06 3/51 133PID PPID USER STAT VSZ %VSZ %CPU COMMAND132 1 root R 2084 1% 97% iperf3 -s -D9 2 root SW 0 0% 2% [ksoftirqd/0]10 2 root RW 0 0% 1% [rcu_preempt]133 126 root R 2076 1% 0% top16 2 root IW 0 0% 0% [kworker/0:1-eve]7 2 root IW 0 0% 0% [kworker/u2:0-ev]126 1 root S 2076 1% 0% -sh124 1 root S 2072 1% 0% inetd1 0 root S 1976 1% 0% init71 1 root S 1976 1% 0% /sbin/syslogd -n72 1 root S 1976 1% 0% /sbin/klogd -n45 2 root IW 0 0% 0% [kworker/u2:1-ev]13 2 root SW 0 0% 0% [kdevtmpfs]56 2 root SW 0 0% 0% [ubi_bgt0d]57 2 root SW 0 0% 0% [ubifs_bgt0_0]2 0 root SW 0 0% 0% [kthreadd]3 2 root IW 0 0% 0% [rcu_gp]4 2 root IW 0 0% 0% [rcu_par_gp]5 2 root IW 0 0% 0% [kworker/0:0-pm]从上面看到RAM 使用上毫无压力但 iperf 程序的 CPU 的占用率已经高达97%。进一步从 CPU 那一行的信息
CPU: 0% usr 63% sys 0% nic 0% idle 0% io 0% irq 36% sirq可以看到CPU 消耗在内核空间的时间 sys 占 63%同时 softirq 占据 36%。我们知道网络子系统收发包都会经由 softirq 处理所以一部分时间消耗在 softirq 上而另外一部分时间消耗在 Linux 网络协议栈的处理过程中经过进一步通过 ftrace 分析网络协议栈数据传输流程验证了这一点。由于某些原因这部分验证过程的相关信息没能在本文中有所系统。 另一方面也排查了相关 MAC 和 PHY 驱动PHY 的排查是通过厂商提供可读调试寄存器在网络通信后读取其中的调试信息没有发现有问题出现而对 MAC 的排查除了对其驱动代码进行检验外也通过 ethtool 导出数据通信后的信息如下(只取了某侧测试的结果)
# ethtool -S eth0
NIC statistics:Good Rx Frames: 1108831Broadcast Rx Frames: 63Multicast Rx Frames: 2762Pause Rx Frames: 0Rx CRC Errors: 0Rx Align/Code Errors: 0Oversize Rx Frames: 0Rx Jabbers: 0Undersize (Short) Rx Frames: 0Rx Fragments: 0Rx Octets: 1141776072Good Tx Frames: 1104727Broadcast Tx Frames: 2Multicast Tx Frames: 20Pause Tx Frames: 0Deferred Tx Frames: 0Collisions: 0Single Collision Tx Frames: 0Multiple Collision Tx Frames: 0Excessive Collisions: 0Late Collisions: 0Tx Underrun: 0Carrier Sense Errors: 0Tx Octets: 1141221834Rx Tx 64 Octet Frames: 370190Rx Tx 65-127 Octet Frames: 369612Rx Tx 128-255 Octet Frames: 198Rx Tx 256-511 Octet Frames: 34Rx Tx 512-1023 Octet Frames: 6Rx Tx 1024-Up Octet Frames: 1473518Net Octets: 2282997906Rx Start of Frame Overruns: 0Rx Middle of Frame Overruns: 0Rx DMA Overruns: 0Rx DMA chan 0: head_enqueue: 1Rx DMA chan 0: tail_enqueue: 1106205Rx DMA chan 0: pad_enqueue: 0Rx DMA chan 0: misqueued: 0Rx DMA chan 0: desc_alloc_fail: 0Rx DMA chan 0: pad_alloc_fail: 0Rx DMA chan 0: runt_receive_buf: 0Rx DMA chan 0: runt_transmit_bu: 0Rx DMA chan 0: empty_dequeue: 0Rx DMA chan 0: busy_dequeue: 1071101Rx DMA chan 0: good_dequeue: 1106078Rx DMA chan 0: requeue: 0Rx DMA chan 0: teardown_dequeue: 0Tx DMA chan 0: head_enqueue: 369476Tx DMA chan 0: tail_enqueue: 735251Tx DMA chan 0: pad_enqueue: 0Tx DMA chan 0: misqueued: 17Tx DMA chan 0: desc_alloc_fail: 0Tx DMA chan 0: pad_alloc_fail: 0Tx DMA chan 0: runt_receive_buf: 0Tx DMA chan 0: runt_transmit_bu: 368846Tx DMA chan 0: empty_dequeue: 369476Tx DMA chan 0: busy_dequeue: 447896Tx DMA chan 0: good_dequeue: 1104727Tx DMA chan 0: requeue: 3Tx DMA chan 0: teardown_dequeue: 0从上面的信息也没有看到有问题的地方。每家 MAC 驱动的输出信息都会有所不同具体的含义要结合驱动代码来看。 通过一系列的排查最终将问题定为 CPU 资源不够导致。在单核系统下网卡多队列、RPS、RFS 等等内核内置的优化策略都毫无意义因为只有一个 CPU 核。
