网卡型号: MT27800，2 port，100G
1.BIOS里开启SRIOV，CPU开启VT以及安装网卡对应的驱动等过程略
2.修改启动菜单参数
执行命令vi /etc/default/grub，按“i”进入编辑模式，在linuxefi行后添加启动参数“intel_iommu=on iommu=pt pci=assign-busses pci=realloc”。
添加完参数之后的启动参数如下：
GRUB_CMDLINE_LINUX=“crashkernel=auto spectre_v2=retpoline rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet intel_iommu=on iommu=pt pci=assign-busses pci=realloc”
修改完后重新生成启动菜单文件：grub2-mkconfig >/boot/efi/EFI/centos/grub.cfg
注意：启动方式为UEFI时， 生成grub.cfg的路径是 /boot/efi/EFI/centos/grub.cfg
在非UEFI启动里的启动菜单路径在 /boot/grub2/grub.cfg
检查上述修改是否生效：cat /boot/efi/EFI/centos/grub.cfg 查看是否有 intel_iommu 几个参数
-
修改完以上参数后，重启一下服务器。
-
执行 lspci |grep -i MT27800,找到网卡在系统里的编号，下面加粗的数字
[root@101 ~]# lspci |grep -i MT27800
18:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
18:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
6.执行 find / -name “sriov_numvfs”|grep -i “18:00”，找到这几张网卡的sriov_numvfs的配置文件
[root@101 ~]# find / -name “sriov_numvfs” |grep -i “18:00”
/sys/devices/pci0000:17/0000:17:00.0/0000:18:00.0/sriov_numvfs
/sys/devices/pci0000:17/0000:17:00.0/0000:18:00.1/sriov_numvfs
- 执行echo [num] > [文件路径]/sriov_numvfs命令，使该路径对应的网卡可以虚拟出[num]个VF。
命令如下：
echo 8 >/sys/devices/pci0000:17/0000:17:00.0/0000:18:00.0/sriov_numvfs
echo 8 >/sys/devices/pci0000:17/0000:17:00.0/0000:18:00.1/sriov_numvfs
注意: 上面的命令只是临时写入的数值，在机器重启之后上面的值会恢复为0，需要重新设置。解决的办法是将上面的指令写入到 /etc/rc.d/rc.local里
- 执行lspci|grep -i "virtual function"命令查看到新增VF对应的PCIe设备。
[root@101 ~]# lspci|grep -i “virtual function”
18:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
ip a | grep enp8s0f0v | wc -l
ip a | grep enp8s0f0_ | wc -l
查看可用网卡
mst status
设置网卡固件VF可用数量
mlxconfig -d /dev/mst/dev/mst/mt4125_pciconf0 -y set NUM_OF_VFS=127
上面两个参数一个是虚拟机出来的VF数量，一个是开启sriov 功能的开关；
mst start //mlx管理工具 mst status
mlxfwreset --device /dev/mst/mt41686_pciconf0 reset