Before talking about the bandwidth of Linux NICs, let's talk about the concept of network bandwidth.
1. Network bandwidth
1.1 Concept
Network bandwidth refers to the amount of data that can be transmitted per unit of time (usually 1 second). The network is similar to a highway, the greater the bandwidth, the more lanes that resemble the highway, and the higher its capacity. As an important indicator to measure network characteristics, network bandwidth has attracted more and more attention. It is not only an important basis for the government or units to formulate network communication development strategies, but also one of the main factors for Internet users and units to choose Internet access service providers.
Bandwidth is a very useful concept and plays an important role in network communication. The actual meaning of bandwidth is the maximum number of bits of data that flow through a particular area at a given time, for example. While its concept is a bit abstract, it can be used metaphorically to help understand what bandwidth means. We use the analogy of the city's water supply network, the diameter of the water supply pipe can measure the ability to carry water, the diameter of the main water pipe may be 2 meters, and the diameter of the main water pipe may be only 2 cm. In this analogy, the diameter of a water pipe is like a bandwidth, and water is like a quantity of information. The use of thick tubes means a wider bandwidth, which means greater information transport capacity.
1.2 Calculation Method
The basic unit of bandwidth is "bit", abbreviated as the lowercase letter "b", and the larger units are: Kb, Mb, Gb, etc.; The basic unit of network speed is "byte", abbreviated as the capital letter "B", and the larger units are: KB, MB, GB, etc.
1 byte = 8 bits
The advance rate of bandwidth and network speed is 1024:
1Kb=1024b
1Mb=1024Kb
1Gb=1024Mb
1KB=1024B
1MB=1024KB
1GB=1024MB
Bandwidth is converted to network speed divided by 8; The speed is converted into a bandwidth multiplied by 8.
In computer networks and IDC computer rooms, the unit of network transmission rate is expressed in b/s (bits per second) (or bit/s, sometimes written as bps, that is, bit per second).
In real network applications, when downloading software, you often see the download speed displayed as 176KB/s, 103KB/s and other broadband speed sizes, because the unit of line bandwidth provided by the ISP is bit, while the general download software displays bytes, so the actual value can be obtained through conversion.
Let's take 1M broadband as an example and convert it according to the conversion formula:
1Mb/s=1024Kb/s=1024/8KB/s=128KB/s
Theoretically: the theoretical rate of 2M (i.e. 2Mb/s) broadband is: 256KB/s, and the actual rate is about 150~240KB/s; (The reason is caused by the user's computer performance, network equipment quality, resource usage, network peak period, website service capacity, line attenuation, signal attenuation and other factors). The theoretical rate of 4M (i.e. 4Mb/s) broadband is 512KB/s, and the actual rate is about 200~440KB/s. The uplink rate refers to the data transmission rate when the user's computer sends information to the network, and the downlink rate refers to the transmission rate when the network sends information to the user's computer. For example, if you use FTP to upload files to the Internet, the upload speed will affect the "uplink rate". Downloading files from the Internet affects the download speed is the "downlink rate". Of course, during the actual upload and download process, the quality of the line, equipment (including computers and other equipment), etc., will also affect the speed to a greater or lesser extent.
2. NIC bandwidth
Network bandwidth is the ability of a network card to receive or send data per second, measured in Mbps.
2.1 Check the bandwidth of the specified NIC on the server
Method 1: Run the ethtool command to view the bandwidth of the specified NIC on the server
The ethtool command is used to query and control network device drivers and hardware settings, especially the name of the Wired Ethernet device and the devname of the NIC.
In the first method, we will use the ethtool utility to check the Network Interface Card (NIC) speed. However, if you don't already have this utility installed on your Linux system, you must first install it with the following command:
sudo apt install ethtool
Once you have installed this utility on your system, you need to check the interface name of the network interface whose speed you want to find out. To do this, you will have to run the command given below:
IP A
This command will display information about all the network interfaces of the system, as shown below, we have selected the network interface named enp129s0f0.
|
Now, to check the speed of the specified NIC, we will execute a command like this:
ubuntu@ubun:~$ ethtool enp129s0f0
Settings for enp129s0f0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
ubuntu@ubun:~$ Speed: 1000Mb/s is the bandwidth of this NIC. However, you can see that the output of the above command is a bit too long. Therefore, if you wish to limit this output to NIC speed only, you must modify it as follows:
|
注意：换算成网速为1000Mb/s = 1000/8 MB/s = 125 MB/s。
Method 2: Run the dmesg command to check the bandwidth of the specified NIC on the server
DMESG is used to display the contents of the kernel's ring buffer, where the kernel stores various messages.
In this method, we will use the built-in dmesg command in Linux to check the NIC speed as follows:
|
Method 3: Run the mii-tool command to view the bandwidth of the specified NIC on the server
The mii-tool command is used to view and manage the status of the network interface of the media, and this package needs to be installed on the system first:
sudo apt install net-tools
Run the following command to check the NIC speed (1000baseT-FD):
ubuntu@ubun:~$ sudo mii-tool -v enp129s0f0
enp129s0f0: negotiated 1000baseT-FD flow-control, link ok
product info: vendor 00:55:00, model 59 rev 1
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
ubuntu@ubun:~$ 3. The NIC bandwidth of the virtual machine
Chapter 2 is all about the bandwidth of the bare metal NIC , but in fact, in order to maximize the use of resources, we will use virtualization software to create virtual machines as servers, so we generally use commands to view the NIC bandwidth of the virtual machine, and the bandwidth of the virtual machine NIC is limited by the bandwidth of the NIC of the bare metal to which it belongs.
If a VM is created by virtualization software such as VMware ESXi, the NIC bandwidth of the VM is 10,000 Mbps.
|
On the ESXi management page, you can see that the BIS NIC has a bandwidth of 1,000 Mbit/s.
This is because the virtual network adapter selected for creating a virtual machine is VMXNET3, which simulates a 10 Gigabit network card, and this virtual network adapter can not correspond to a physical network card (ordinary bare metal is generally a Gigabit network card), that is, it does not need to use the underlying hardware network card, and by optimizing the performance in the virtual machine, the network exchange between virtual machines on the same bare metal is not limited by the underlying network card.
For more information, please refer to: In VMWare ESXi, the performance of different virtual NICs can be three times different!
Principle: The bandwidth of a virtual machine created on the same bare metal is managed by a virtual switch without passing through a NIC (physical network card), while the bandwidth of a virtual machine accessing the Internet is limited by the NIC.
For more information, see Network Virtualization and Virtual Networking
Summary:
- Virtual machines created on the same bare metal communicate through virtual switches and do not go through NICs, so we can deploy components with large correlation on different virtual machines created on the same bare metal, such as Redis clusters. When a virtual machine accesses an external network (a network that accesses another virtual machine on a bare metal other than the current virtual machine), it uses a NIC, and its bandwidth is limited by the NIC.
- Before you check the bandwidth of a server's NIC, you must know whether the server you are checking is a bare metal or a virtual machine, and if it is a virtual machine, you must also know the bandwidth of the NIC of the bare metal to which it belongs.
4. Use the iperf3 command to test the real network bandwidth between servers
The network bandwidth mentioned in the above sections of this article is a theoretical value, and its real bandwidth will be less than the bandwidth of the network card, which is caused by the user's computer performance, network equipment quality, resource usage, network peak period, website service capability, line attenuation, signal attenuation and other factors.
Note: Generally, we will proxy the server program through the proxy, the general link of the client to the server is: client-> proxy-> server, and iperf3 can only test one of the real network bandwidth, in order to test the real network bandwidth from the client to the server, you need to test the real network bandwidth from the client to the proxy, and then test the real network bandwidth from the proxy to the server, and then test the real network bandwidth from the proxy to the server.
4.1 iperf3 Overview
iperf3 is a network performance testing tool. iperf can test TCP and UDP bandwidth quality. iperf can measure the maximum TCP bandwidth with a variety of parameters and UDP features. iperf can report bandwidth, delay, jitter, and packet loss.
iperf3 is a tool for actively measuring the maximum bandwidth achievable on an IP network, and to perform iperf3 testing, users must establish both a server and a client.
4.2 Install iperf3 on the server
yum install -y iperf3
4.3 Introduction to the iperf3 command
Iperf3 is divided into server and client, for example, if you want to test the bandwidth of server A requesting server B, then server B is the server and A is the client.
Server:
iperf3 -s
Client:
| iperf3 -c server IP address |
iperf3 has many parameters, the commonly used ones are:
-c server address
-i indicates how often the data is printed
-t test time
-n The size of the packet sent
-p Number of threads sent
4.4 Examples
Note: The following test results are only for the current test room, and the test results are different in different environments.
Example 1: Test the bandwidth of the real NIC from 10.20.30.28 (client) to 10.20.30.30 (server) in the same cabinet on the same LAN
Execute the iperf3 -s command at 10.20.30.30 and the following command on 10.20.30.28 machine:
[root@ob1 ~]# iperf3 -c 10.20.30.31 -t 60
Connecting to host 10.20.30.31, port 5201
[ 4] local 10.20.30.28 port 52762 connected to 10.20.30.31 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 9 373 KBytes
.......
[ 4] 59.00-60.00 sec 112 MBytes 938 Mbits/sec 0 495 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 6.53 GBytes 935 Mbits/sec 620 sender
[ 4] 0.00-60.00 sec 6.53 GBytes 935 Mbits/sec receiver According to the results, the bandwidth of the real NIC between the bare metal of two Gigabit NICs in the same cabinet is 935.
Example 2: Test the bandwidth of the real NICs from 10.20.30.173 (client) to 10.20.30.30 (server) in different cabinets within the same LAN
Execute the iperf3 -s command at 10.20.30.30 and the following command on 10.20.30.173 machine:
|
According to the results, the bandwidth of the real NIC between the two Gigabit NICs is 913 in different cabinets.
Example 3: Test the bandwidth of the NICs from 10.20.31.118 (client) to 10.20.30.30 (server) in different cabinets within the same LAN
Execute the iperf3 -s command at 10.20.30.30 and the following command on 10.20.31.118 machine:
|
According to the results, the bandwidth of the real NIC between the virtual machine (the bare metal where the Gigabit NIC resides) and the bare metal is 867 in different cabinets.
Summary:
- The actual bandwidth of the server is less than the bandwidth of the NIC.
- The real network bandwidth between VMs and BMS in the same environment is higher than that between VMs and BMS (due to many factors, the most important of which is that all VMs on the same BMS share the physical NIC of the BMS).
5. Determine network performance bottlenecks
Determine the direction of data flow, that is, determine the links that network requests go through after they are sent from the client and finally reach the server. What is the process through which the response data is returned to the client? And what is the upper limit of the network in each link, and finally the upper limit of the traffic of our entire network loop depends on the lowest upper limit value in the loop (barrel principle).
For example, as shown in the following figure:
The network request data is first sent from the client's network card (Gigabit network card), and then the network bandwidth will also be limited (100Mb/s), and may also pass through some network devices, such as routers, switches, firewalls, proxies, etc., these devices also have network limits, if they are all 10000Mb/s, and finally arrive at the network card (Gigabit network card) on the server. Therefore, judging from the above network architecture, the overall link supports up to 100Mb/s data traffic, and the network speed is 100/8=12.5MB/s.
If you are in the LAN, you almost don't need to consider the problem of network bandwidth and routers, and the firewall in the general LAN is a 10 Gigabit network card, so the upper limit of the server network card in the LAN is the network upper limit of the entire link.