如果大家用的万兆网卡跑linux或者nginx做大规模的负载均衡,那么肯定会遇到网卡中断占耗尽一个CPU的情况,会发现有一个ksoftirqd进程耗CPU非常厉害。这个时候就需要把万兆网卡的多个队列分别绑定到不同的核上。简单的在自己的笔记本上测试一下把单个中断绑定到指定CPU的方式。
1.先查看中断号
root@debian:/home/pm/downloads# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 55 0 0 0 IO-APIC-edge timer 1: 9828 0 0 0 IO-APIC-edge i8042 8: 1 0 0 0 IO-APIC-edge rtc0 9: 64153 0 0 0 IO-APIC-fasteoi acpi 12: 12812 0 0 0 IO-APIC-edge i8042 16: 50287 0 0 0 IO-APIC-fasteoi mmc0, ehci_hcd:usb1 23: 32 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2 40: 2053 0 0 0 PCI-MSI-edge eth0 41: 84430 0 0 0 PCI-MSI-edge ahci 42: 540 0 0 0 PCI-MSI-edge snd_hda_intel 43: 189429 3306 1140 185873 PCI-MSI-edge iwlwifi 44: 367098 96565 0 0 PCI-MSI-edge i915 NMI: 389 212 369 216 Non-maskable interrupts LOC: 686664 448243 658281 387582 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 389 212 369 216 Performance monitoring interrupts IWI: 0 0 0 0 IRQ work interrupts RES: 1618241 979307 1837671 950705 Rescheduling interrupts CAL: 393 803 859 871 Function call interrupts TLB: 68943 88659 91389 45605 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 15 15 15 15 Machine check polls
2.分别测试把中断号43绑定到不同的CPU,各个 CPU 用二进制和十六进制表示就是: CPU Binary oct
CPU 0 00000001 1
CPU 1 00000010 2
CPU 2 00000100 4
CPU 3 00001000 8
所以
echo 8 > /proc/irq/43/smp_affinity就会把网卡中断绑定到CPU3上,同理1,2,8会把中断绑定到CPU0,CPU1,CPU3上。但把中断绑定到制定的核后,我们再cat /proc/interrupts查看的话会发现对于中断号43,只有CPU3对应的数字在增加。
root@debian:/home/pm/downloads# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 57 0 0 0 IO-APIC-edge timer 1: 12224 0 0 0 IO-APIC-edge i8042 8: 1 0 0 0 IO-APIC-edge rtc0 9: 72086 0 0 0 IO-APIC-fasteoi acpi 12: 13634 0 0 0 IO-APIC-edge i8042 16: 61911 0 0 0 IO-APIC-fasteoi mmc0, ehci_hcd:usb1 23: 32 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2 40: 2310 0 0 0 PCI-MSI-edge eth0 41: 89086 0 0 0 PCI-MSI-edge ahci 42: 540 0 0 0 PCI-MSI-edge snd_hda_intel
43: 189429 3306 1140 279511 PCI-MSI-edge iwlwifi
44: 367098 146592 0 0 PCI-MSI-edge i915 NMI: 459 262 442 272 Non-maskable interrupts LOC: 830429 549702 798508 469421 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 459 262 442 272 Performance monitoring interrupts IWI: 0 0 0 0 IRQ work interrupts RES: 1992918 1266807 2245210 1188155 Rescheduling interrupts CAL: 393 804 860 872 Function call interrupts TLB: 94270 118496 115458 47381 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 17 17 17 17 Machine check polls ERR: 0 MIS: 0
另外测试了一下直接把一个中断号绑定在2个核上比如 echo 6 >/proc/irq/43/smp_affinity , 但是结果表明不能同时把这个中断绑定到2个核。linux下有个软中断负载平衡的服务irqbalance。