* Observing Softlockup's while running heavy IOs
@ 2016-08-18 5:55 Sreekanth Reddy
2016-08-18 14:59 ` Bart Van Assche
2016-08-18 21:08 ` Elliott, Robert (Persistent Memory)
0 siblings, 2 replies; 16+ messages in thread
From: Sreekanth Reddy @ 2016-08-18 5:55 UTC (permalink / raw)
To: linux-scsi, linux-kernel, irqbalance
Cc: Kashyap Desai, Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
Hi,
Problem statement:
Observing softlockups while running heavy IOs on 8 SSD drives
connected behind our LSI SAS 3004 HBA.
System configuration:
OS & kernel version: Fedora 23, v4.2.3-300.fc23.x86_64
NUMA : disabled,
CPUs : 24 logical cpus,
SCSI_MQ: enabled
Driver : mpt3sas,
MSIx vectors: we have enabled only 2 MSIx vectors using driver module
parameter 'max_msix_vectors' set to 2,
IRQbalance version: v1.0.9
IRQbalance policy : 'subset'
rq_affinity : 2
mpt3sas IRQs info:
irq number = 219, smp_affinity = 000040 affinity_hint = 000fff
irq number = 220, smp_affinity = 001000 affinity_hint = fff000
issued IOs using fio tool with below parameters,
iodepth=128
direct=1
runtime=300
group_reporting
ioengine=libaio
cpus_allowed=6,7,8,9,10,11,18,19,20,21,22,23
time_based
[Ran-Read-4k-IOPs]
numjobs=24
rw=randread
bs=4k
My Analysis:
Observing a loop in the IO path, i.e only one CPU is busy with
processing the interrupts and other CPUs (in the affinity_hint mask)
are busy with sending the IOs (these CPUs are not yet all receiving
any interrupts). For example, only CPU6 is busy with processing the
interrupts from IRQ 219 and remaining CPUs i.e CPU 7,8,9,10 & 11 are
just busy with pumping the IOs and they never processed any IO
interrupts from IRQ 219. So we are observing softlockups due to
existence this loop in the IO Path.
We may not observe these softlockups if irqbalancer might have
balanced the interrupts among the CPUs enabled in the particular irq's
affinity_hint mask. so that all the CPUs are equaly busy with send IOs
and processing the interrupts. I am not sure how irqbalancer balance
the load among the CPUs, but here I see only one CPU from irq's
affinity_hint mask is busy with interrupts and remaining CPUs won't
receive any interrupts from this IRQ.
Please help me with any suggestions/recomendations to slove/limit
these kind of softlockups. Also please let me known if I have missed
any setting in the irqbalance.
Here is my 'sar -u ALL -P ALL 1' command output when softlock occurred,
12:16:11 PM CPU %usr %nice %sys %iowait %steal
%irq %soft %guest %gnice %idle
12:16:12 PM all 6.31 0.00 31.62 4.49 0.00
0.00 2.59 0.00 0.00 54.99
12:16:12 PM 0 4.08 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 95.92
12:16:12 PM 1 4.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 96.00
12:16:12 PM 2 2.97 0.00 1.98 0.00 0.00
0.00 0.00 0.00 0.00 95.05
12:16:12 PM 3 7.00 0.00 1.00 0.00 0.00
0.00 0.00 0.00 0.00 92.00
12:16:12 PM 4 4.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 96.00
12:16:12 PM 5 4.04 0.00 1.01 0.00 0.00
0.00 0.00 0.00 0.00 94.95
12:16:12 PM 6 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.00 0.00 0.00
12:16:12 PM 7 17.02 0.00 58.51 0.00 0.00
0.00 0.00 0.00 0.00 24.47
12:16:12 PM 8 14.13 0.00 61.96 23.91 0.00
0.00 0.00 0.00 0.00 0.00
12:16:12 PM 9 5.21 0.00 14.58 80.21 0.00
0.00 0.00 0.00 0.00 0.00
12:16:12 PM 10 4.21 0.00 14.74 0.00 0.00
0.00 0.00 0.00 0.00 81.05
12:16:12 PM 11 13.04 0.00 61.96 0.00 0.00
0.00 0.00 0.00 0.00 25.00
12:16:12 PM 12 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.00 0.00 0.00
12:16:12 PM 13 4.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 96.00
12:16:12 PM 14 4.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 96.00
12:16:12 PM 15 3.03 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 96.97
12:16:12 PM 16 4.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 96.00
12:16:12 PM 17 4.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 96.00
12:16:12 PM 18 0.00 0.00 100.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
12:16:12 PM 19 16.84 0.00 69.47 0.00 0.00
0.00 0.00 0.00 0.00 13.68
12:16:12 PM 20 16.13 0.00 69.89 0.00 0.00
0.00 0.00 0.00 0.00 13.98
12:16:12 PM 21 0.00 0.00 100.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
12:16:12 PM 22 0.00 0.00 100.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
12:16:12 PM 23 15.22 0.00 69.57 0.00 0.00
0.00 0.00 0.00 0.00 15.22
And here is the softlockup call-trace:
[root@dhcp-135-24-192-127 soft_lockup_issue]# cat /opt/softlockup_calltrace.txt
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: NMI watchdog: BUG: soft
lockup - CPU#6 stuck for 23s! [fio:32228]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Modules linked in:
mpt3sas(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge
stp llc ebtable_filter ebtables ip6table_filter ip6_tables intel_rapl
iosf_mbi raid456 x86_pkg_temp_thermal async_raid6_recov async_memcpy
async_pq async_xor coretemp joydev xor async_tx ast i2c_algo_bit
raid6_pq kvm ttm drm_kms_helper drm sb_edac crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel raid_class scsi_transport_sas
iTCO_wdt iTCO_vendor_support i2c_i801 edac_core ipmi_ssif lpc_ich
mei_me mei ipmi_si ipmi_msghandler ioatdma tpm_tis shpchp tpm wmi
acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc
xfs libcrc32c ixgbe crc32c_intel mdio vxlan ip6_udp_tunnel udp_tunnel
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ptp nvme pps_core dca
[last unloaded: mpt3sas]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CPU: 6 PID: 32228 Comm:
fio Tainted: G W OEL 4.2.3-300.fc23.x86_64 #1
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Hardware name: Supermicro
SYS-2028U-TNRT+/X10DRU-i+, BIOS 1.1 07/22/2015
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: task: ffff883f5b8c8000 ti:
ffff883ecd918000 task.ti: ffff883ecd918000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RIP:
0010:[<ffffffff810a27db>] [<ffffffff810a27db>]
__do_softirq+0x7b/0x290
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RSP: 0018:ffff883f7f183f08
EFLAGS: 00000286
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RAX: ffff883ecd91c000 RBX:
ffff883f7f190080 RCX: 00000000000006e0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RDX: 0000000000050109 RSI:
000000002a60bb93 RDI: ffff883f5b8c8000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RBP: ffff883f7f183f68 R08:
000501092a0e06ad R09: ffff883f7f183d80
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: R10: 0000000000000000 R11:
0000000000000004 R12: ffff883f7f183e78
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: R13: ffffffff8177984b R14:
ffff883f7f183f68 R15: 0000000000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: FS:
00007fa884206740(0000) GS:ffff883f7f180000(0000)
knlGS:0000000000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CS: 0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CR2: 00007fa87650d090 CR3:
0000003edf8ee000 CR4: 00000000001406e0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Stack:
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883f7f18fc40
004041407f18fce8 ffff883ecd91c000 0000000118aaea84
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: 000178ab0000000a
000000827f18fc58 ffff883f7f183f58 0000000000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883ecd91b7d8
ffffe8ffff985300 ffff883ecd9a7100 ffff883e5d0c8080
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Call Trace:
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: <IRQ>
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff810a2c09>]
irq_exit+0x119/0x120
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8177b6b6>]
smp_apic_timer_interrupt+0x46/0x60
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8177984b>]
apic_timer_interrupt+0x6b/0x70
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: <EOI>
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137c415>] ?
__bt_get.isra.5+0xe5/0x1c0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff810e66cc>] ?
queued_spin_lock_slowpath+0xdc/0x180
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81778630>]
_raw_spin_lock+0x20/0x30
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81379e07>]
__blk_mq_run_hw_queue+0x157/0x370
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81378546>] ?
__blk_mq_alloc_request+0x156/0x1f0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137a2cb>]
blk_mq_map_request+0x13b/0x1e0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137b1f4>]
blk_sq_make_request+0x84/0x2b0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff811a571d>] ?
mempool_alloc_slab+0x1d/0x30
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8136dca6>]
generic_make_request+0xd6/0x110
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8136dd56>]
submit_bio+0x76/0x170
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff811adcce>] ?
set_page_dirty_lock+0x2e/0x40
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff813683bb>] ?
bio_set_pages_dirty+0x4b/0x70
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8125b797>]
do_blockdev_direct_IO+0x22e7/0x2b20
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff813c55cd>] ?
swiotlb_map_sg_attrs+0x6d/0x130
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffffa05d237f>] ?
_base_build_sg_scmd_ieee+0x25f/0x5f0 [mpt3sas]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81256760>] ?
I_BDEV+0x20/0x20
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8125c013>]
__blockdev_direct_IO+0x43/0x50
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81379e80>] ?
__blk_mq_run_hw_queue+0x1d0/0x370
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8125706c>]
blkdev_direct_IO+0x4c/0x50
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff811a42cb>]
generic_file_read_iter+0x48b/0x5c0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81257c60>] ?
blkdev_write_iter+0x120/0x120
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81257c95>]
blkdev_read_iter+0x35/0x40
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126994a>]
aio_run_iocb+0x13a/0x2e0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff812691df>] ?
read_events+0x12f/0x340
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff812000c6>] ?
kmem_cache_alloc+0x1d6/0x210
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126a94a>]
do_io_submit+0x23a/0x4d0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff817742f1>] ?
__schedule+0x371/0x950
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81774907>] ?
schedule+0x37/0x80
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126abf0>]
SyS_io_submit+0x10/0x20
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff817789ae>]
entry_SYSCALL_64_fastpath+0x12/0x71
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Code: 7e 00 01 00 00 65 48
8b 04 25 c4 3c 01 00 c7 45 c0 0a 00 00 00 48 89 45 b0 65 c7 05 2c 14
f7 7e 00 00 00 00 fb 66 0f 1f 44 00 00 <b8> ff ff ff ff 49 c7 c4 c0 a0
c0 81 0f bc 45 cc 83 c0 01 89 45
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: NMI watchdog: BUG: soft
lockup - CPU#18 stuck for 23s! [fio:32235]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Modules linked in:
mpt3sas(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge
stp llc ebtable_filter ebtables ip6table_filter ip6_tables intel_rapl
iosf_mbi raid456 x86_pkg_temp_thermal async_raid6_recov async_memcpy
async_pq async_xor coretemp joydev xor async_tx ast i2c_algo_bit
raid6_pq kvm ttm drm_kms_helper drm sb_edac crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel raid_class scsi_transport_sas
iTCO_wdt iTCO_vendor_support i2c_i801 edac_core ipmi_ssif lpc_ich
mei_me mei ipmi_si ipmi_msghandler ioatdma tpm_tis shpchp tpm wmi
acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc
xfs libcrc32c ixgbe crc32c_intel mdio vxlan ip6_udp_tunnel udp_tunnel
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ptp nvme pps_core dca
[last unloaded: mpt3sas]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CPU: 18 PID: 32235 Comm:
fio Tainted: G W OEL 4.2.3-300.fc23.x86_64 #1
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Hardware name: Supermicro
SYS-2028U-TNRT+/X10DRU-i+, BIOS 1.1 07/22/2015
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: task: ffff883f3f445880 ti:
ffff883edf894000 task.ti: ffff883edf894000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RIP:
0010:[<ffffffff810e6703>] [<ffffffff810e6703>]
queued_spin_lock_slowpath+0x113/0x180
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RSP: 0018:ffff883edf897c18
EFLAGS: 00000246
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RAX: 0000000000000000 RBX:
ffff883ecf7bc000 RCX: ffff883f7f497340
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RDX: ffff883f7f597340 RSI:
00000000004c0000 RDI: ffff883f51364400
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RBP: ffff883edf897c18 R08:
0000000000000101 R09: 0000000000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: R10: 0000000000001000 R11:
0000000000000000 R12: ffff883f51364400
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: R13: 0000000000000000 R14:
0000000017145a7c R15: 0000000000000246
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: FS:
00007fa884206740(0000) GS:ffff883f7f480000(0000)
knlGS:0000000000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CS: 0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CR2: 00007fa876512024 CR3:
0000003e5d781000 CR4: 00000000001406e0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Stack:
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883edf897c28
ffffffff81778630 ffff883edf897cc8 ffffffff81379f09
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883edf897c68
ffff883f51334d58 ffff883f51364408 0000001000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883e5882af00
ffff883e5882af00 ffff883edf897c68 ffff883edf897c68
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Call Trace:
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81778630>]
_raw_spin_lock+0x20/0x30
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81379f09>]
__blk_mq_run_hw_queue+0x259/0x370
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81379c91>]
blk_mq_run_hw_queue+0x91/0xb0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137b11c>]
blk_mq_insert_requests+0xbc/0x110
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137bc32>]
blk_mq_flush_plug_list+0x132/0x160
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff813718b6>]
blk_flush_plug_list+0xb6/0x220
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81371d84>]
blk_finish_plug+0x34/0x50
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126a9aa>]
do_io_submit+0x29a/0x4d0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff817742f1>] ?
__schedule+0x371/0x950
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81774907>] ?
schedule+0x37/0x80
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126abf0>]
SyS_io_submit+0x10/0x20
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff817789ae>]
entry_SYSCALL_64_fastpath+0x12/0x71
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Code: 12 48 c1 ea 0c 83 e8
01 83 e2 30 48 98 48 81 c2 40 73 01 00 48 03 14 c5 80 cb d2 81 48 89
0a 8b 41 08 85 c0 75 0d f3 90 8b 41 08 <85> c0 74 f7 eb 02 f3 90 8b 17
66 85 d2 75 f7 39 f2 75 0f 89 d0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: NMI watchdog: BUG: soft
lockup - CPU#21 stuck for 23s! [fio:32224]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: NMI watchdog: BUG: soft
lockup - CPU#22 stuck for 23s! [fio:32220]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Modules linked in:
mpt3sas(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge
stp llc ebtable_filter ebtables ip6table_filter ip6_tables intel_rapl
iosf_mbi raid456 x86_pkg_temp_thermal async_raid6_recov async_memcpy
async_pq async_xor coretemp joydev xor async_tx ast i2c_algo_bit
raid6_pq kvm ttm drm_kms_helper drm sb_edac crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel raid_class scsi_transport_sas
iTCO_wdt iTCO_vendor_support i2c_i801 edac_core ipmi_ssif lpc_ich
mei_me mei ipmi_si ipmi_msghandler ioatdma tpm_tis shpchp tpm wmi
acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc
xfs libcrc32c ixgbe crc32c_intel mdio vxlan ip6_udp_tunnel udp_tunnel
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ptp nvme pps_core dca
[last unloaded: mpt3sas]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CPU: 22 PID: 32220 Comm:
fio Tainted: G W OEL 4.2.3-300.fc23.x86_64 #1
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Hardware name: Supermicro
SYS-2028U-TNRT+/X10DRU-i+, BIOS 1.1 07/22/2015
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: task: ffff883f5c1b8000 ti:
ffff883eb9300000 task.ti: ffff883eb9300000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RIP:
0010:[<ffffffff810e6703>] [<ffffffff810e6703>]
queued_spin_lock_slowpath+0x113/0x180
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RSP: 0018:ffff883eb9303c18
EFLAGS: 00000246
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RAX: 0000000000000000 RBX:
ffff883ecf7bc000 RCX: ffff883f7f597340
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RDX: ffff883f7f557340 RSI:
00000000005c0000 RDI: ffff883f51364400
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RBP: ffff883eb9303c18 R08:
0000000000000101 R09: 0000000000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: R10: 0000000000001000 R11:
0000000000000000 R12: ffff883f51364400
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: R13: 0000000000000000 R14:
000000008327c0c7 R15: 0000000000000246
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: FS:
00007fa884206740(0000) GS:ffff883f7f580000(0000)
knlGS:0000000000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CS: 0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CR2: 00007fa8841df000 CR3:
0000003ecf698000 CR4: 00000000001406e0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Stack:
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883eb9303c28
ffffffff81778630 ffff883eb9303cc8 ffffffff81379f09
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883eb9303c68
ffff883f51334d58 ffff883f51364408 0000001000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883ea28e4600
ffff883ea28e4600 ffff883eb9303c68 ffff883eb9303c68
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Call Trace:
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81778630>]
_raw_spin_lock+0x20/0x30
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81379f09>]
__blk_mq_run_hw_queue+0x259/0x370
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81379c91>]
blk_mq_run_hw_queue+0x91/0xb0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137b11c>]
blk_mq_insert_requests+0xbc/0x110
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137bc32>]
blk_mq_flush_plug_list+0x132/0x160
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff813718b6>]
blk_flush_plug_list+0xb6/0x220
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81371d84>]
blk_finish_plug+0x34/0x50
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126a9aa>]
do_io_submit+0x29a/0x4d0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff817742f1>] ?
__schedule+0x371/0x950
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81774907>] ?
schedule+0x37/0x80
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126abf0>]
SyS_io_submit+0x10/0x20
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff817789ae>]
entry_SYSCALL_64_fastpath+0x12/0x71
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Code: 12 48 c1 ea 0c 83 e8
01 83 e2 30 48 98 48 81 c2 40 73 01 00 48 03 14 c5 80 cb d2 81 48 89
0a 8b 41 08 85 c0 75 0d f3 90 8b 41 08 <85> c0 74 f7 eb 02 f3 90 8b 17
66 85 d2 75 f7 39 f2 75 0f 89 d0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Modules linked in:
mpt3sas(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge
stp llc ebtable_filter ebtables ip6table_filter ip6_tables intel_rapl
iosf_mbi raid456 x86_pkg_temp_thermal async_raid6_recov async_memcpy
async_pq async_xor coretemp joydev xor async_tx ast i2c_algo_bit
raid6_pq kvm ttm drm_kms_helper drm sb_edac crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel raid_class scsi_transport_sas
iTCO_wdt iTCO_vendor_support i2c_i801 edac_core ipmi_ssif lpc_ich
mei_me mei ipmi_si ipmi_msghandler ioatdma tpm_tis shpchp tpm wmi
acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc
xfs libcrc32c ixgbe crc32c_intel mdio vxlan ip6_udp_tunnel udp_tunnel
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ptp nvme pps_core dca
[last unloaded: mpt3sas]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CPU: 21 PID: 32224 Comm:
fio Tainted: G W OEL 4.2.3-300.fc23.x86_64 #1
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Hardware name: Supermicro
SYS-2028U-TNRT+/X10DRU-i+, BIOS 1.1 07/22/2015
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: task: ffff883f5a3b8000 ti:
ffff883ecfb58000 task.ti: ffff883ecfb58000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RIP:
0010:[<ffffffff810e6703>] [<ffffffff810e6703>]
queued_spin_lock_slowpath+0x113/0x180
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RSP: 0018:ffff883ecfb5b698
EFLAGS: 00000246
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RAX: 0000000000000000 RBX:
ffff883ecf7bc000 RCX: ffff883f7f557340
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RDX: ffff883f7f197340 RSI:
0000000000580000 RDI: ffff883f51364400
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: RBP: ffff883ecfb5b698 R08:
0000000000000101 R09: 0000000000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: R10: 0000000118aaea81 R11:
0000000000000006 R12: ffff883f51364400
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: R13: ffff883ecfb5b648 R14:
000000006c0b2c7f R15: 0000000000000246
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: FS:
00007fa884206740(0000) GS:ffff883f7f540000(0000)
knlGS:0000000000000000
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CS: 0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: CR2: 00007f8d99882000 CR3:
0000003eb901c000 CR4: 00000000001406e0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Stack:
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883ecfb5b6a8
ffffffff81778630 ffff883ecfb5b748 ffffffff81379f09
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883ecfb5b6c8
ffff883f51334d58 ffff883f51364408 0000000800000296
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: ffff883eae160000
ffff883eae092300 ffff883ecfb5b6e8 ffff883ecfb5b6e8
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Call Trace:
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81778630>]
_raw_spin_lock+0x20/0x30
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81379f09>]
__blk_mq_run_hw_queue+0x259/0x370
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81379c91>]
blk_mq_run_hw_queue+0x91/0xb0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137c615>]
bt_get+0x125/0x1d0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff810df960>] ?
wake_atomic_t_function+0x70/0x70
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137ca30>]
blk_mq_get_tag+0xb0/0xe0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137840b>]
__blk_mq_alloc_request+0x1b/0x1f0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137a34d>]
blk_mq_map_request+0x1bd/0x1e0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8137b1f4>]
blk_sq_make_request+0x84/0x2b0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff811a571d>] ?
mempool_alloc_slab+0x1d/0x30
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8136dca6>]
generic_make_request+0xd6/0x110
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8136dd56>]
submit_bio+0x76/0x170
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff811adcce>] ?
set_page_dirty_lock+0x2e/0x40
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff813683bb>] ?
bio_set_pages_dirty+0x4b/0x70
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8125b797>]
do_blockdev_direct_IO+0x22e7/0x2b20
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff813c55cd>] ?
swiotlb_map_sg_attrs+0x6d/0x130
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffffa05d237f>] ?
_base_build_sg_scmd_ieee+0x25f/0x5f0 [mpt3sas]
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81256760>] ?
I_BDEV+0x20/0x20
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8125c013>]
__blockdev_direct_IO+0x43/0x50
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81379e80>] ?
__blk_mq_run_hw_queue+0x1d0/0x370
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8125706c>]
blkdev_direct_IO+0x4c/0x50
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff811a42cb>]
generic_file_read_iter+0x48b/0x5c0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81257c60>] ?
blkdev_write_iter+0x120/0x120
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81257c95>]
blkdev_read_iter+0x35/0x40
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126994a>]
aio_run_iocb+0x13a/0x2e0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff812691df>] ?
read_events+0x12f/0x340
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff812000c6>] ?
kmem_cache_alloc+0x1d6/0x210
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126a94a>]
do_io_submit+0x23a/0x4d0
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff817742f1>] ?
__schedule+0x371/0x950
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff81774907>] ?
schedule+0x37/0x80
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff8126abf0>]
SyS_io_submit+0x10/0x20
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: [<ffffffff817789ae>]
entry_SYSCALL_64_fastpath+0x12/0x71
Aug 17 12:16:12 dhcp-135-24-192-127 kernel: Code: 12 48 c1 ea 0c 83 e8
01 83 e2 30 48 98 48 81 c2 40 73 01 00 48 03 14 c5 80 cb d2 81 48 89
0a 8b 41 08 85 c0 75 0d f3 90 8b 41 08 <85> c0 74 f7 eb 02 f3 90 8b 17
66 85 d2 75 f7 39 f2 75 0f 89 d0
Thanks,
Sreekanth
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-08-18 5:55 Observing Softlockup's while running heavy IOs Sreekanth Reddy
@ 2016-08-18 14:59 ` Bart Van Assche
2016-08-18 21:08 ` Elliott, Robert (Persistent Memory)
1 sibling, 0 replies; 16+ messages in thread
From: Bart Van Assche @ 2016-08-18 14:59 UTC (permalink / raw)
To: Sreekanth Reddy, linux-scsi, linux-kernel, irqbalance
Cc: Kashyap Desai, Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
[-- Attachment #1: Type: text/plain, Size: 404 bytes --]
On 08/17/16 22:55, Sreekanth Reddy wrote:
> Observing softlockups while running heavy IOs on 8 SSD drives
> connected behind our LSI SAS 3004 HBA.
Hello Sreekanth,
This means that more than 23s was spent before the scheduler was
invoked, probably due to a loop. Can you give the attached (untested)
patch a try to see whether it is the loop in __blk_mq_run_hw_queue()?
Thanks,
Bart.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-block-Measure-__blk_mq_run_hw_queue-execution-time.patch --]
[-- Type: text/x-patch; name="0001-block-Measure-__blk_mq_run_hw_queue-execution-time.patch", Size: 1132 bytes --]
From 4da94f2ec37ee5d1b4a5f1ce2886bdafd5cd394c Mon Sep 17 00:00:00 2001
From: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Thu, 18 Aug 2016 07:51:49 -0700
Subject: [PATCH] block: Measure __blk_mq_run_hw_queue() execution time
Note: the "max_elapsed" variable can be modified by multiple threads
concurrently.
---
block/blk-mq.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index e931a0e..6d0961c 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -792,6 +792,9 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
LIST_HEAD(driver_list);
struct list_head *dptr;
int queued;
+ static long max_elapsed = -1;
+ unsigned long start = jiffies;
+ long elapsed;
WARN_ON(!cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask));
@@ -889,6 +892,13 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
**/
blk_mq_run_hw_queue(hctx, true);
}
+
+ elapsed = jiffies - start;
+ if (elapsed > max_elapsed) {
+ max_elapsed = elapsed;
+ pr_info("%s() finished after %d ms\n", __func__,
+ jiffies_to_msecs(elapsed));
+ }
}
/*
--
2.9.2
^ permalink raw reply related [flat|nested] 16+ messages in thread
* RE: Observing Softlockup's while running heavy IOs
2016-08-18 5:55 Observing Softlockup's while running heavy IOs Sreekanth Reddy
2016-08-18 14:59 ` Bart Van Assche
@ 2016-08-18 21:08 ` Elliott, Robert (Persistent Memory)
2016-08-19 11:44 ` Sreekanth Reddy
1 sibling, 1 reply; 16+ messages in thread
From: Elliott, Robert (Persistent Memory) @ 2016-08-18 21:08 UTC (permalink / raw)
To: Sreekanth Reddy, linux-scsi, linux-kernel, irqbalance
Cc: Kashyap Desai, Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> owner@vger.kernel.org] On Behalf Of Sreekanth Reddy
> Sent: Thursday, August 18, 2016 12:56 AM
> Subject: Observing Softlockup's while running heavy IOs
>
> Problem statement:
> Observing softlockups while running heavy IOs on 8 SSD drives
> connected behind our LSI SAS 3004 HBA.
>
...
> Observing a loop in the IO path, i.e only one CPU is busy with
> processing the interrupts and other CPUs (in the affinity_hint mask)
> are busy with sending the IOs (these CPUs are not yet all receiving
> any interrupts). For example, only CPU6 is busy with processing the
> interrupts from IRQ 219 and remaining CPUs i.e CPU 7,8,9,10 & 11 are
> just busy with pumping the IOs and they never processed any IO
> interrupts from IRQ 219. So we are observing softlockups due to
> existence this loop in the IO Path.
>
> We may not observe these softlockups if irqbalancer might have
> balanced the interrupts among the CPUs enabled in the particular
> irq's
> affinity_hint mask. so that all the CPUs are equaly busy with send
> IOs
> and processing the interrupts. I am not sure how irqbalancer balance
> the load among the CPUs, but here I see only one CPU from irq's
> affinity_hint mask is busy with interrupts and remaining CPUs won't
> receive any interrupts from this IRQ.
>
> Please help me with any suggestions/recomendations to slove/limit
> these kind of softlockups. Also please let me known if I have missed
> any setting in the irqbalance.
>
The CPUs need to be forced to self-throttle by processing interrupts for
their own submissions, which reduces the time they can submit more IOs.
See https://lkml.org/lkml/2014/9/9/931 for discussion of this
problem when blk-mq was added.
---
Robert Elliott, HPE Persistent Memory
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-08-18 21:08 ` Elliott, Robert (Persistent Memory)
@ 2016-08-19 11:44 ` Sreekanth Reddy
2016-08-19 15:56 ` Bart Van Assche
2016-08-19 21:27 ` Elliott, Robert (Persistent Memory)
0 siblings, 2 replies; 16+ messages in thread
From: Sreekanth Reddy @ 2016-08-19 11:44 UTC (permalink / raw)
To: Elliott, Robert (Persistent Memory)
Cc: linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
First of all thanks for Robert and Bart for reply.
Robert,
Thanks for the URL, I have gone though this URL.
Yes I am also observing that all the interrupts are routed to one CPU.
But still I observing softlockups (sometime hardlockups) even when I
set rq_affinity to 2.
Is their any way to route the interrupts the same CPUs which has
submitted the corresponding IOs?
or
Is their any way/option in the irqbalance/kernel which can route
interrupts to CPUs (enabled in affinity_hint) in round robin manner
after specific time period.
Bart,
I have tried with your path and here is logs,
[Aug19 13:48] __blk_mq_run_hw_queue() finished after 1 ms
[ +1.196454] __blk_mq_run_hw_queue() finished after 11 ms
[Aug19 13:49] __blk_mq_run_hw_queue() finished after 20 ms
[ +14.173018] __blk_mq_run_hw_queue() finished after 278 ms
[ +14.066448] __blk_mq_run_hw_queue() finished after 588 ms
[ +5.394698] __blk_mq_run_hw_queue() finished after 1360 ms
[Aug19 13:51] __blk_mq_run_hw_queue() finished after 1539 ms
[Aug19 13:54] __blk_mq_run_hw_queue() finished after 1762 ms
[Aug19 13:55] __blk_mq_run_hw_queue() finished after 2087 ms
[Aug19 13:57] __blk_mq_run_hw_queue() finished after 2915 ms
[Aug19 14:06] perf interrupt took too long (2509 > 2500), lowering
kernel.perf_event_max_sample_rate to 50000
[Aug19 14:10] __blk_mq_run_hw_queue() finished after 3266 ms
[Aug19 14:15] __blk_mq_run_hw_queue() finished after 3953 ms
[Aug19 14:22] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s!
[llvmpipe-9:3152]
[ +0.000439] __blk_mq_run_hw_queue() finished after 10058 ms
[ +0.007206] Modules linked in: mpt3sas(OE) xt_CHECKSUM
iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat
nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack
nf_conntrack tun bridge stp llc ebtable_filter ebtables
ip6table_filter ip6_tables raid456 async_raid6_recov async_memcpy
async_pq async_xor xor intel_rapl async_tx iosf_mbi
x86_pkg_temp_thermal raid6_pq coretemp kvm crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel mei_me joydev pcspkr sb_edac i2c_i801
mei iTCO_wdt iTCO_vendor_support edac_core lpc_ich ipmi_ssif ipmi_si
ipmi_msghandler shpchp tpm_tis ioatdma acpi_power_meter tpm wmi
acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c ast
i2c_algo_bit drm_kms_helper ttm ixgbe drm mdio vxlan ip6_udp_tunnel
udp_tunnel crc32c_intel raid_class ptp scsi_transport_sas
[ +0.000063] pps_core nvme dca [last unloaded: mpt3sas]
[ +0.000007] CPU: 6 PID: 3152 Comm: llvmpipe-9 Tainted: G W
OE 4.2.0 #1
[ +0.000002] Hardware name: Supermicro SYS-2028U-TNRT+/X10DRU-i+,
BIOS 1.1 07/22/2015
[ +0.000003] task: ffff883f5cf557c0 ti: ffff883f5afd8000 task.ti:
ffff883f5afd8000
[ +0.000002] RIP: 0010:[<ffffffff810a159b>] [<ffffffff810a159b>]
__do_softirq+0x7b/0x290
[ +0.000008] RSP: 0000:ffff883f7f183f08 EFLAGS: 00000206
[ +0.000002] RAX: ffff883f5afdc000 RBX: ffff883f7f190080 RCX: 00000000000006e0
[ +0.000002] RDX: 0000000000003508 RSI: 0000000071c139c0 RDI: ffff883f5cf557c0
[ +0.000002] RBP: ffff883f7f183f68 R08: 00003508717f8da4 R09: ffff883f7f183d80
[ +0.000002] R10: 0000000000000000 R11: 0000000000000004 R12: ffff883f7f183e78
[ +0.000001] R13: ffffffff8177304b R14: ffff883f7f183f68 R15: 0000000000000000
[ +0.000003] FS: 00007fa76b7f6700(0000) GS:ffff883f7f180000(0000)
knlGS:0000000000000000
Message from syslogd@dhcp-135-24-192-127 at Aug 19 14:22:42 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [llvmpipe-9:3152]
[ +0.000002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ +0.000002] CR2: 00007fa03a2d90c0 CR3: 0000003f61fe4000 CR4: 00000000001406e0
[ +0.000002] Stack:
[ +0.000001] ffff883f7f18fc40 404040407f18fce8 ffff883f5afdc000
0000000100aecc1a
[ +0.000004] 00000ab10000000a 000002027f18fc58 ffff883f7f183f48
0000000000000000
[ +0.000003] ffff883f7f1967c0 ffff883f448d57c0 0000000000000000
0000000000000001
[ +0.000003] Call Trace:
[ +0.000002] <IRQ>
[ +0.000005] [<ffffffff810a19c6>] irq_exit+0x116/0x120
[ +0.000005] [<ffffffff81774eb6>] smp_apic_timer_interrupt+0x46/0x60
[ +0.000005] [<ffffffff8177304b>] apic_timer_interrupt+0x6b/0x70
[ +0.000002] <EOI>
[ +0.000005] [<ffffffff810c392b>] ? finish_task_switch+0x6b/0x200
[ +0.000006] [<ffffffff8176dabc>] __schedule+0x36c/0x950
[ +0.000002] [<ffffffff8176e0d7>] schedule+0x37/0x80
[ +0.000006] [<ffffffff81116a1c>] futex_wait_queue_me+0xbc/0x120
[ +0.000004] [<ffffffff81116da9>] futex_wait+0x119/0x270
[ +0.000004] [<ffffffff81116800>] ? futex_wake+0x90/0x180
[ +0.000003] [<ffffffff81118d6b>] do_futex+0x12b/0xb00
[ +0.000005] [<ffffffff810d348e>] ? set_next_entity+0x23e/0x440
[ +0.000007] [<ffffffff810136f1>] ? __switch_to+0x261/0x4b0
[ +0.000004] [<ffffffff811197c1>] SyS_futex+0x81/0x180
[ +0.000002] [<ffffffff8176e0d7>] ? schedule+0x37/0x80
[ +0.000004] [<ffffffff817721ae>] entry_SYSCALL_64_fastpath+0x12/0x71
[ +0.000001] Code: 7e 00 01 00 00 65 48 8b 04 25 c4 3c 01 00 c7 45 c0
0a 00 00 00 48 89 45 b0 65 c7 05 6c 26 f7 7e 00 00 00 00 fb 66 0f 1f
44 00 00 <b8> ff ff ff ff 49 c7 c4 c0 a0 c0 81 0f bc 45 cc 83 c0 01 89
45
and here is the 'sar -u ALL 1 -P ALL 1'
02:22:43 PM CPU %usr %nice %sys %iowait %steal
%irq %soft %guest %gnice %idle
02:22:44 PM all 5.73 0.60 29.31 14.90 0.00
0.00 1.57 0.00 0.00 47.89
02:22:44 PM 0 8.82 0.00 48.53 38.24 0.00
0.00 0.00 0.00 0.00 4.41
02:22:44 PM 1 10.29 0.00 63.24 26.47 0.00
0.00 0.00 0.00 0.00 0.00
02:22:44 PM 2 8.96 0.00 65.67 25.37 0.00
0.00 0.00 0.00 0.00 0.00
02:22:44 PM 3 11.94 0.00 61.19 26.87 0.00
0.00 0.00 0.00 0.00 0.00
02:22:44 PM 4 8.96 0.00 64.18 26.87 0.00
0.00 0.00 0.00 0.00 0.00
02:22:44 PM 5 7.46 0.00 67.16 25.37 0.00
0.00 0.00 0.00 0.00 0.00
02:22:44 PM 6 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.00 0.00 0.00
02:22:44 PM 7 1.33 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 98.67
02:22:44 PM 8 5.19 0.00 1.30 0.00 0.00
0.00 0.00 0.00 0.00 93.51
02:22:44 PM 9 2.67 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 97.33
02:22:44 PM 10 2.67 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 97.33
02:22:44 PM 11 3.95 0.00 1.32 1.32 0.00
0.00 0.00 0.00 0.00 93.42
02:22:44 PM 12 11.11 0.00 83.33 5.56 0.00
0.00 0.00 0.00 0.00 0.00
02:22:44 PM 13 8.70 0.00 52.17 36.23 0.00
0.00 0.00 0.00 0.00 2.90
02:22:44 PM 14 6.15 0.00 53.85 38.46 0.00
0.00 0.00 0.00 0.00 1.54
02:22:44 PM 15 5.97 0.00 55.22 37.31 0.00
0.00 0.00 0.00 0.00 1.49
02:22:44 PM 16 8.70 0.00 52.17 36.23 0.00
0.00 0.00 0.00 0.00 2.90
02:22:44 PM 17 7.35 0.00 55.88 35.29 0.00
0.00 0.00 0.00 0.00 1.47
02:22:44 PM 18 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 100.00
02:22:44 PM 19 2.63 0.00 1.32 2.63 0.00
0.00 0.00 0.00 0.00 93.42
02:22:44 PM 20 2.67 0.00 1.33 0.00 0.00
0.00 0.00 0.00 0.00 96.00
02:22:44 PM 21 7.89 5.26 2.63 0.00 0.00
0.00 0.00 0.00 0.00 84.21
02:22:44 PM 22 1.32 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 98.68
02:22:44 PM 23 4.05 8.11 2.70 4.05 0.00
0.00 0.00 0.00 0.00 81.08
Still I am continuing my investigation on this.
Note:
I am taking vacation next week, please expect some delay for response.
Thanks,
Sreekanth
On Fri, Aug 19, 2016 at 2:38 AM, Elliott, Robert (Persistent Memory)
<elliott@hpe.com> wrote:
>
>
>> -----Original Message-----
>> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
>> owner@vger.kernel.org] On Behalf Of Sreekanth Reddy
>> Sent: Thursday, August 18, 2016 12:56 AM
>> Subject: Observing Softlockup's while running heavy IOs
>>
>> Problem statement:
>> Observing softlockups while running heavy IOs on 8 SSD drives
>> connected behind our LSI SAS 3004 HBA.
>>
> ...
>> Observing a loop in the IO path, i.e only one CPU is busy with
>> processing the interrupts and other CPUs (in the affinity_hint mask)
>> are busy with sending the IOs (these CPUs are not yet all receiving
>> any interrupts). For example, only CPU6 is busy with processing the
>> interrupts from IRQ 219 and remaining CPUs i.e CPU 7,8,9,10 & 11 are
>> just busy with pumping the IOs and they never processed any IO
>> interrupts from IRQ 219. So we are observing softlockups due to
>> existence this loop in the IO Path.
>>
>> We may not observe these softlockups if irqbalancer might have
>> balanced the interrupts among the CPUs enabled in the particular
>> irq's
>> affinity_hint mask. so that all the CPUs are equaly busy with send
>> IOs
>> and processing the interrupts. I am not sure how irqbalancer balance
>> the load among the CPUs, but here I see only one CPU from irq's
>> affinity_hint mask is busy with interrupts and remaining CPUs won't
>> receive any interrupts from this IRQ.
>>
>> Please help me with any suggestions/recomendations to slove/limit
>> these kind of softlockups. Also please let me known if I have missed
>> any setting in the irqbalance.
>>
>
> The CPUs need to be forced to self-throttle by processing interrupts for
> their own submissions, which reduces the time they can submit more IOs.
>
> See https://lkml.org/lkml/2014/9/9/931 for discussion of this
> problem when blk-mq was added.
>
>
> ---
> Robert Elliott, HPE Persistent Memory
>
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-08-19 11:44 ` Sreekanth Reddy
@ 2016-08-19 15:56 ` Bart Van Assche
2016-09-01 10:31 ` Sreekanth Reddy
2016-08-19 21:27 ` Elliott, Robert (Persistent Memory)
1 sibling, 1 reply; 16+ messages in thread
From: Bart Van Assche @ 2016-08-19 15:56 UTC (permalink / raw)
To: Sreekanth Reddy, Elliott, Robert (Persistent Memory)
Cc: linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
On 08/19/2016 04:44 AM, Sreekanth Reddy wrote:
> [ +0.000439] __blk_mq_run_hw_queue() finished after 10058 ms
> [ ... ]
> [ +0.000005] [<ffffffff810c392b>] ? finish_task_switch+0x6b/0x200
> [ +0.000006] [<ffffffff8176dabc>] __schedule+0x36c/0x950
> [ +0.000002] [<ffffffff8176e0d7>] schedule+0x37/0x80
> [ +0.000006] [<ffffffff81116a1c>] futex_wait_queue_me+0xbc/0x120
> [ +0.000004] [<ffffffff81116da9>] futex_wait+0x119/0x270
> [ +0.000004] [<ffffffff81116800>] ? futex_wake+0x90/0x180
> [ +0.000003] [<ffffffff81118d6b>] do_futex+0x12b/0xb00
> [ +0.000005] [<ffffffff810d348e>] ? set_next_entity+0x23e/0x440
> [ +0.000007] [<ffffffff810136f1>] ? __switch_to+0x261/0x4b0
> [ +0.000004] [<ffffffff811197c1>] SyS_futex+0x81/0x180
> [ +0.000002] [<ffffffff8176e0d7>] ? schedule+0x37/0x80
> [ +0.000004] [<ffffffff817721ae>] entry_SYSCALL_64_fastpath+0x12/0x71
Hello Sreekanth,
If a "soft lockup" is reported that often means that kernel code is
iterating too long in a loop without giving up the CPU. Inserting a
cond_resched() call in such loops usually resolves these soft lockup
complaints. However, your latest e-mail shows that the soft lockup
complaint was reported on other code than __blk_mq_run_hw_queue(). I'm
afraid this means that the CPU on which the soft lockup was reported is
hammered so hard with interrupts that hardly any time remains for the
scheduler to run code on that CPU. You will have to follow Robert
Elliott's advice and reduce the time that is spent per CPU in interrupt
context.
Bart.
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: Observing Softlockup's while running heavy IOs
2016-08-19 11:44 ` Sreekanth Reddy
2016-08-19 15:56 ` Bart Van Assche
@ 2016-08-19 21:27 ` Elliott, Robert (Persistent Memory)
2016-08-23 9:52 ` Kashyap Desai
1 sibling, 1 reply; 16+ messages in thread
From: Elliott, Robert (Persistent Memory) @ 2016-08-19 21:27 UTC (permalink / raw)
To: Sreekanth Reddy
Cc: linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
> -----Original Message-----
> From: Sreekanth Reddy [mailto:sreekanth.reddy@broadcom.com]
> Sent: Friday, August 19, 2016 6:45 AM
> To: Elliott, Robert (Persistent Memory) <elliott@hpe.com>
> Subject: Re: Observing Softlockup's while running heavy IOs
>
...
> Yes I am also observing that all the interrupts are routed to one
> CPU. But still I observing softlockups (sometime hardlockups)
> even when I set rq_affinity to 2.
That'll ensure the block layer's completion handling is done there,
but not your driver's interrupt handler (which precedes the block
layer completion handling).
> Is their any way to route the interrupts the same CPUs which has
> submitted the corresponding IOs?
> or
> Is their any way/option in the irqbalance/kernel which can route
> interrupts to CPUs (enabled in affinity_hint) in round robin manner
> after specific time period.
Ensure your driver creates one MSIX interrupt per CPU core, uses
that interrupt for all submissions from that core, and reports
that it would like that interrupt to be serviced by that core
in /proc/irq/nnn/affinity_hint.
Even with hyperthreading, this needs to be based on the logical
CPU cores, not just the physical core or the physical socket.
You can swamp a logical CPU core as easily as a physical CPU core.
Then, provide an irqbalance policy script that honors the
affinity_hint for your driver, or turn off irqbalance and
manually set /proc/irq/nnn/smp_affinity to match the
affinity_hint.
Some versions of irqbalance honor the hints; some purposely
don't and need to be overridden with a policy script.
---
Robert Elliott, HPE Persistent Memory
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: Observing Softlockup's while running heavy IOs
2016-08-19 21:27 ` Elliott, Robert (Persistent Memory)
@ 2016-08-23 9:52 ` Kashyap Desai
0 siblings, 0 replies; 16+ messages in thread
From: Kashyap Desai @ 2016-08-23 9:52 UTC (permalink / raw)
To: Elliott, Robert (Persistent Memory), Sreekanth Reddy
Cc: linux-scsi, linux-kernel, irqbalance, Sathya Prakash Veerichetty,
Chaitra Basappa, Suganath Prabu Subramani
> -----Original Message-----
> From: Elliott, Robert (Persistent Memory) [mailto:elliott@hpe.com]
> Sent: Saturday, August 20, 2016 2:58 AM
> To: Sreekanth Reddy
> Cc: linux-scsi@vger.kernel.org; linux-kernel@vger.kernel.org;
> irqbalance@lists.infradead.org; Kashyap Desai; Sathya Prakash Veerichetty;
> Chaitra Basappa; Suganath Prabu Subramani
> Subject: RE: Observing Softlockup's while running heavy IOs
>
>
>
> > -----Original Message-----
> > From: Sreekanth Reddy [mailto:sreekanth.reddy@broadcom.com]
> > Sent: Friday, August 19, 2016 6:45 AM
> > To: Elliott, Robert (Persistent Memory) <elliott@hpe.com>
> > Subject: Re: Observing Softlockup's while running heavy IOs
> >
> ...
> > Yes I am also observing that all the interrupts are routed to one CPU.
> > But still I observing softlockups (sometime hardlockups) even when I
> > set rq_affinity to 2.
How about below scenario ? For simplicity. HBA with single MSI-x vector.
(Whenever HBA supports less MSI-x and logical CPUs are more on system, we
can see chance of these issue frequently..)
Assume we have 32 logical CPU (4 socket, each with 8 logical CPU). CPU-0 is
not participating in IO.
Remaining CPU range from 1 to 31 is submitting IO. In such a scenario
rq_affinity=2 and irqbalance supporting *exact* smp_affinity_hint will not
help.
We may see soft/hard lockup on CPU-0.. Are we going to resolve such issue or
it is very rare to happen on field ?
>
> That'll ensure the block layer's completion handling is done there, but
> not your
> driver's interrupt handler (which precedes the block layer completion
> handling).
>
>
> > Is their any way to route the interrupts the same CPUs which has
> > submitted the corresponding IOs?
> > or
> > Is their any way/option in the irqbalance/kernel which can route
> > interrupts to CPUs (enabled in affinity_hint) in round robin manner
> > after specific time period.
>
> Ensure your driver creates one MSIX interrupt per CPU core, uses that
> interrupt
> for all submissions from that core, and reports that it would like that
> interrupt to
> be serviced by that core in /proc/irq/nnn/affinity_hint.
>
> Even with hyperthreading, this needs to be based on the logical CPU cores,
> not
> just the physical core or the physical socket.
> You can swamp a logical CPU core as easily as a physical CPU core.
>
> Then, provide an irqbalance policy script that honors the affinity_hint
> for your
> driver, or turn off irqbalance and manually set /proc/irq/nnn/smp_affinity
> to
> match the affinity_hint.
>
> Some versions of irqbalance honor the hints; some purposely don't and need
> to
> be overridden with a policy script.
>
>
> ---
> Robert Elliott, HPE Persistent Memory
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-08-19 15:56 ` Bart Van Assche
@ 2016-09-01 10:31 ` Sreekanth Reddy
2016-09-01 23:04 ` Bart Van Assche
0 siblings, 1 reply; 16+ messages in thread
From: Sreekanth Reddy @ 2016-09-01 10:31 UTC (permalink / raw)
To: Bart Van Assche
Cc: Elliott, Robert (Persistent Memory),
linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
On Fri, Aug 19, 2016 at 9:26 PM, Bart Van Assche
<bart.vanassche@sandisk.com> wrote:
> On 08/19/2016 04:44 AM, Sreekanth Reddy wrote:
>>
>> [ +0.000439] __blk_mq_run_hw_queue() finished after 10058 ms
>> [ ... ]
>> [ +0.000005] [<ffffffff810c392b>] ? finish_task_switch+0x6b/0x200
>> [ +0.000006] [<ffffffff8176dabc>] __schedule+0x36c/0x950
>> [ +0.000002] [<ffffffff8176e0d7>] schedule+0x37/0x80
>> [ +0.000006] [<ffffffff81116a1c>] futex_wait_queue_me+0xbc/0x120
>> [ +0.000004] [<ffffffff81116da9>] futex_wait+0x119/0x270
>> [ +0.000004] [<ffffffff81116800>] ? futex_wake+0x90/0x180
>> [ +0.000003] [<ffffffff81118d6b>] do_futex+0x12b/0xb00
>> [ +0.000005] [<ffffffff810d348e>] ? set_next_entity+0x23e/0x440
>> [ +0.000007] [<ffffffff810136f1>] ? __switch_to+0x261/0x4b0
>> [ +0.000004] [<ffffffff811197c1>] SyS_futex+0x81/0x180
>> [ +0.000002] [<ffffffff8176e0d7>] ? schedule+0x37/0x80
>> [ +0.000004] [<ffffffff817721ae>] entry_SYSCALL_64_fastpath+0x12/0x71
>
>
> Hello Sreekanth,
>
> If a "soft lockup" is reported that often means that kernel code is
> iterating too long in a loop without giving up the CPU. Inserting a
> cond_resched() call in such loops usually resolves these soft lockup
> complaints. However, your latest e-mail shows that the soft lockup complaint
> was reported on other code than __blk_mq_run_hw_queue(). I'm afraid this
> means that the CPU on which the soft lockup was reported is hammered so hard
> with interrupts that hardly any time remains for the scheduler to run code
> on that CPU. You will have to follow Robert Elliott's advice and reduce the
> time that is spent per CPU in interrupt context.
>
Sorry for delay in response as I was on Vacation.
Bart,
I reduced the ISR workload by one third in-order to reduce the time
that is spent per CPU in interrupt context, even then I am observing
softlockups.
As I mentioned before only same single CPU in the set of CPUs(enabled
in affinity_hint) is busy with handling the interrupts from
corresponding IRQx. I have done below experiment in driver to limit
these softlockups/hardlockups. But I am not sure whether it is
reasonable to do this in driver,
Experiment:
If the CPUx is continuously busy with handling the remote CPUs
(enabled in the corresponding IRQ's affinity_hint) IO works by 1/4th
of the HBA queue depth in the same ISR context then enable a flag
called 'change_smp_affinity' for this IRQ. Also created a thread with
will poll for this flag for every IRQ's (enabled by driver) for every
second. If this thread see that this flag is enabled for any IRQ then
it will write next CPU number from the CPUs enabled in the IRQ's
affinity_hint to the IRQ's smp_affinity procfs attribute using
'call_usermodehelper()' API.
This to make sure that interrupts are not processed by same single CPU
all the time and to make the other CPUs to handle the interrupts if
the current CPU is continuously busy with handling the other CPUs IO
interrupts.
For example consider a system which has 8 logical CPUs and one MSIx
vector enabled (called IRQ 120) in driver, HBA queue depth as 8K.
then IRQ's procfs attributes will be
IRQ# 120, affinity_hint=0xff, smp_affinity=0x00
After starting heavy IOs, we will observe that only CPU0 will be busy
with handling the interrupts. This experiment driver will change the
smp_affinity to next CPU number i.e. 0x01 (using cmd 'echo 0x01 >
/proc/irq/120/smp_affinity', driver issue's this cmd using
call_usermodehelper() API) if it observes that CPU0 is continuously
processing more than 2K of IOs replies of other CPUs i.e from CPU1 to
CPU7.
Whether doing this kind of stuff in driver is ok?
Thanks,
Sreekanth
> Bart.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-09-01 10:31 ` Sreekanth Reddy
@ 2016-09-01 23:04 ` Bart Van Assche
[not found] ` <CAK=zhgrLL22stCfwKdpJkN=PkxPVxL=K9RgpP1USEbg_xx5TEg@mail.gmail.com>
0 siblings, 1 reply; 16+ messages in thread
From: Bart Van Assche @ 2016-09-01 23:04 UTC (permalink / raw)
To: Sreekanth Reddy
Cc: Elliott, Robert (Persistent Memory),
linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
On 09/01/2016 03:31 AM, Sreekanth Reddy wrote:
> I reduced the ISR workload by one third in-order to reduce the time
> that is spent per CPU in interrupt context, even then I am observing
> softlockups.
>
> As I mentioned before only same single CPU in the set of CPUs(enabled
> in affinity_hint) is busy with handling the interrupts from
> corresponding IRQx. I have done below experiment in driver to limit
> these softlockups/hardlockups. But I am not sure whether it is
> reasonable to do this in driver,
>
> Experiment:
> If the CPUx is continuously busy with handling the remote CPUs
> (enabled in the corresponding IRQ's affinity_hint) IO works by 1/4th
> of the HBA queue depth in the same ISR context then enable a flag
> called 'change_smp_affinity' for this IRQ. Also created a thread with
> will poll for this flag for every IRQ's (enabled by driver) for every
> second. If this thread see that this flag is enabled for any IRQ then
> it will write next CPU number from the CPUs enabled in the IRQ's
> affinity_hint to the IRQ's smp_affinity procfs attribute using
> 'call_usermodehelper()' API.
>
> This to make sure that interrupts are not processed by same single CPU
> all the time and to make the other CPUs to handle the interrupts if
> the current CPU is continuously busy with handling the other CPUs IO
> interrupts.
>
> For example consider a system which has 8 logical CPUs and one MSIx
> vector enabled (called IRQ 120) in driver, HBA queue depth as 8K.
> then IRQ's procfs attributes will be
> IRQ# 120, affinity_hint=0xff, smp_affinity=0x00
>
> After starting heavy IOs, we will observe that only CPU0 will be busy
> with handling the interrupts. This experiment driver will change the
> smp_affinity to next CPU number i.e. 0x01 (using cmd 'echo 0x01 >
> /proc/irq/120/smp_affinity', driver issue's this cmd using
> call_usermodehelper() API) if it observes that CPU0 is continuously
> processing more than 2K of IOs replies of other CPUs i.e from CPU1 to
> CPU7.
>
> Whether doing this kind of stuff in driver is ok?
Hello Sreekanth,
To me this sounds like something that should be implemented in the I/O
chipset on the motherboard. If you have a look at the Intel Software
Developer Manuals then you will see that logical destination mode
supports round-robin interrupt delivery. However, the Linux kernel
selects physical destination mode on systems with more than eight
logical CPUs (see also arch/x86/kernel/apic/apic_flat_64.c).
I'm not sure the maintainers of the interrupt subsystem would welcome
code that emulates round-robin interrupt delivery. So your best option
is probably to minimize the amount of work that is done in interrupt
context and to move as much work as possible out of interrupt context in
such a way that it can be spread over multiple CPU cores, e.g. by using
queue_work_on().
Bart.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
[not found] ` <CAK=zhgrLL22stCfwKdpJkN=PkxPVxL=K9RgpP1USEbg_xx5TEg@mail.gmail.com>
@ 2016-09-06 15:06 ` Neil Horman
2016-09-07 6:00 ` Sreekanth Reddy
0 siblings, 1 reply; 16+ messages in thread
From: Neil Horman @ 2016-09-06 15:06 UTC (permalink / raw)
To: Sreekanth Reddy
Cc: Bart Van Assche, Elliott, Robert (Persistent Memory),
linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote:
> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche
> <bart.vanassche@sandisk.com> wrote:
> > On 09/01/2016 03:31 AM, Sreekanth Reddy wrote:
> >>
> >> I reduced the ISR workload by one third in-order to reduce the time
> >> that is spent per CPU in interrupt context, even then I am observing
> >> softlockups.
> >>
> >> As I mentioned before only same single CPU in the set of CPUs(enabled
> >> in affinity_hint) is busy with handling the interrupts from
> >> corresponding IRQx. I have done below experiment in driver to limit
> >> these softlockups/hardlockups. But I am not sure whether it is
> >> reasonable to do this in driver,
> >>
> >> Experiment:
> >> If the CPUx is continuously busy with handling the remote CPUs
> >> (enabled in the corresponding IRQ's affinity_hint) IO works by 1/4th
> >> of the HBA queue depth in the same ISR context then enable a flag
> >> called 'change_smp_affinity' for this IRQ. Also created a thread with
> >> will poll for this flag for every IRQ's (enabled by driver) for every
> >> second. If this thread see that this flag is enabled for any IRQ then
> >> it will write next CPU number from the CPUs enabled in the IRQ's
> >> affinity_hint to the IRQ's smp_affinity procfs attribute using
> >> 'call_usermodehelper()' API.
> >>
> >> This to make sure that interrupts are not processed by same single CPU
> >> all the time and to make the other CPUs to handle the interrupts if
> >> the current CPU is continuously busy with handling the other CPUs IO
> >> interrupts.
> >>
> >> For example consider a system which has 8 logical CPUs and one MSIx
> >> vector enabled (called IRQ 120) in driver, HBA queue depth as 8K.
> >> then IRQ's procfs attributes will be
> >> IRQ# 120, affinity_hint=0xff, smp_affinity=0x00
> >>
> >> After starting heavy IOs, we will observe that only CPU0 will be busy
> >> with handling the interrupts. This experiment driver will change the
> >> smp_affinity to next CPU number i.e. 0x01 (using cmd 'echo 0x01 >
> >> /proc/irq/120/smp_affinity', driver issue's this cmd using
> >> call_usermodehelper() API) if it observes that CPU0 is continuously
> >> processing more than 2K of IOs replies of other CPUs i.e from CPU1 to
> >> CPU7.
> >>
> >> Whether doing this kind of stuff in driver is ok?
> >
> >
> > Hello Sreekanth,
> >
> > To me this sounds like something that should be implemented in the I/O
> > chipset on the motherboard. If you have a look at the Intel Software
> > Developer Manuals then you will see that logical destination mode supports
> > round-robin interrupt delivery. However, the Linux kernel selects physical
> > destination mode on systems with more than eight logical CPUs (see also
> > arch/x86/kernel/apic/apic_flat_64.c).
> >
> > I'm not sure the maintainers of the interrupt subsystem would welcome code
> > that emulates round-robin interrupt delivery. So your best option is
> > probably to minimize the amount of work that is done in interrupt context
> > and to move as much work as possible out of interrupt context in such a way
> > that it can be spread over multiple CPU cores, e.g. by using
> > queue_work_on().
> >
> > Bart.
>
> Bart,
>
> Thanks a lot for providing lot of inputs and valuable information on this issue.
>
> Today I got one more observation. i.e. I am not observing any lockups
> if I use 1.0.4-6 versioned irqbalance.
> Since this versioned irqbalance is able to shift the load to other CPU
> when one CPU is heavily loaded.
>
This isn't happening because irqbalance is no longer able to shift load between
cpus, its happening because of commit 996ee2cf7a4d10454de68ac4978adb5cf22850f8.
irqs with higher interrupt volumes sould be balanced to a specific cpu core,
rather than to a cache domain to maximize cpu-local cache hit rates. Prior to
that change we balanced to a cache domain and your workload didn't have to
serialize multiple interrupts to a single core. My suggestion to you is to use
the --policyscript option to make your storage irqs get balanced to the cache
level, rather than the core level. That should return the behavior to what you
want.
Neil
> while running heavy IOs, for first few seconds here is my driver irq's
> attributes,
> --------------------------------------------------------------------------------------------------------------------
> ioc number = 0
> number of core processors = 24
> msix vector count = 2
> number of cores per msix vector = 16
>
>
> msix index = 0, irq number = 50, smp_affinity = 000040
> affinity_hint = 000fff
> msix index = 1, irq number = 51, smp_affinity = 001000
> affinity_hint = fff000
>
> We have set affinity for 2 msix vectors and 24 core processors
> ----------------------------------------------------------------------------------------------------------------------
>
> After few seconds it observed that CPU12 is heavily loaded for IRQ 51
> and it changed the smp_affinity to CPU21
> --------------------------------------------------------------------------------------------------------------------
> ioc number = 0
> number of core processors = 24
> msix vector count = 2
> number of cores per msix vector = 16
>
>
> msix index = 0, irq number = 50, smp_affinity = 000040
> affinity_hint = 000fff
> msix index = 1, irq number = 51, smp_affinity = 200000
> affinity_hint = fff000
>
> We have set affinity for 2 msix vectors and 24 core processors
> ---------------------------------------------------------------------------------------------------------------------
>
> Where as irqblanance versioned 1.0.9 is not able to shift the load to
> the other CPUs enabled in the affinity_hint (even when subset policy
> is enabled) and so I was observing the softlocks/hardlockups.
>
> Here I have attached irqbalance logs with debug enabled for both versions.
>
> Thanks,
> Sreekanth
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-09-06 15:06 ` Neil Horman
@ 2016-09-07 6:00 ` Sreekanth Reddy
2016-09-07 13:24 ` Neil Horman
0 siblings, 1 reply; 16+ messages in thread
From: Sreekanth Reddy @ 2016-09-07 6:00 UTC (permalink / raw)
To: Neil Horman
Cc: Bart Van Assche, Elliott, Robert (Persistent Memory),
linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
[-- Attachment #1: Type: text/plain, Size: 7068 bytes --]
On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote:
>> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche
>> <bart.vanassche@sandisk.com> wrote:
>> > On 09/01/2016 03:31 AM, Sreekanth Reddy wrote:
>> >>
>> >> I reduced the ISR workload by one third in-order to reduce the time
>> >> that is spent per CPU in interrupt context, even then I am observing
>> >> softlockups.
>> >>
>> >> As I mentioned before only same single CPU in the set of CPUs(enabled
>> >> in affinity_hint) is busy with handling the interrupts from
>> >> corresponding IRQx. I have done below experiment in driver to limit
>> >> these softlockups/hardlockups. But I am not sure whether it is
>> >> reasonable to do this in driver,
>> >>
>> >> Experiment:
>> >> If the CPUx is continuously busy with handling the remote CPUs
>> >> (enabled in the corresponding IRQ's affinity_hint) IO works by 1/4th
>> >> of the HBA queue depth in the same ISR context then enable a flag
>> >> called 'change_smp_affinity' for this IRQ. Also created a thread with
>> >> will poll for this flag for every IRQ's (enabled by driver) for every
>> >> second. If this thread see that this flag is enabled for any IRQ then
>> >> it will write next CPU number from the CPUs enabled in the IRQ's
>> >> affinity_hint to the IRQ's smp_affinity procfs attribute using
>> >> 'call_usermodehelper()' API.
>> >>
>> >> This to make sure that interrupts are not processed by same single CPU
>> >> all the time and to make the other CPUs to handle the interrupts if
>> >> the current CPU is continuously busy with handling the other CPUs IO
>> >> interrupts.
>> >>
>> >> For example consider a system which has 8 logical CPUs and one MSIx
>> >> vector enabled (called IRQ 120) in driver, HBA queue depth as 8K.
>> >> then IRQ's procfs attributes will be
>> >> IRQ# 120, affinity_hint=0xff, smp_affinity=0x00
>> >>
>> >> After starting heavy IOs, we will observe that only CPU0 will be busy
>> >> with handling the interrupts. This experiment driver will change the
>> >> smp_affinity to next CPU number i.e. 0x01 (using cmd 'echo 0x01 >
>> >> /proc/irq/120/smp_affinity', driver issue's this cmd using
>> >> call_usermodehelper() API) if it observes that CPU0 is continuously
>> >> processing more than 2K of IOs replies of other CPUs i.e from CPU1 to
>> >> CPU7.
>> >>
>> >> Whether doing this kind of stuff in driver is ok?
>> >
>> >
>> > Hello Sreekanth,
>> >
>> > To me this sounds like something that should be implemented in the I/O
>> > chipset on the motherboard. If you have a look at the Intel Software
>> > Developer Manuals then you will see that logical destination mode supports
>> > round-robin interrupt delivery. However, the Linux kernel selects physical
>> > destination mode on systems with more than eight logical CPUs (see also
>> > arch/x86/kernel/apic/apic_flat_64.c).
>> >
>> > I'm not sure the maintainers of the interrupt subsystem would welcome code
>> > that emulates round-robin interrupt delivery. So your best option is
>> > probably to minimize the amount of work that is done in interrupt context
>> > and to move as much work as possible out of interrupt context in such a way
>> > that it can be spread over multiple CPU cores, e.g. by using
>> > queue_work_on().
>> >
>> > Bart.
>>
>> Bart,
>>
>> Thanks a lot for providing lot of inputs and valuable information on this issue.
>>
>> Today I got one more observation. i.e. I am not observing any lockups
>> if I use 1.0.4-6 versioned irqbalance.
>> Since this versioned irqbalance is able to shift the load to other CPU
>> when one CPU is heavily loaded.
>>
>
> This isn't happening because irqbalance is no longer able to shift load between
> cpus, its happening because of commit 996ee2cf7a4d10454de68ac4978adb5cf22850f8.
> irqs with higher interrupt volumes sould be balanced to a specific cpu core,
> rather than to a cache domain to maximize cpu-local cache hit rates. Prior to
> that change we balanced to a cache domain and your workload didn't have to
> serialize multiple interrupts to a single core. My suggestion to you is to use
> the --policyscript option to make your storage irqs get balanced to the cache
> level, rather than the core level. That should return the behavior to what you
> want.
>
> Neil
Hi Neil,
Thanks for reply.
Today I tried with setting balance_level to 'cache' for mpt3sas driver
IRQ's using below policy script and used 1.0.9 versioned irqbalance,
----------------------------------------------------------------------------------------------
#!/bin/bash
# Header
# Linux Shell Scripting for Irq Balance Policy select for mpt3sas driver
#
# Command Line Args
#IRQ_PATH -> PATH
#IRQ_NUMBER -> IRQ Number
declare -r IRQ_PATH=$1
declare -r IRQ_NUMBER=$2
if [ -d /proc/irq/$IRQ_NUMBER ]; then
mpt3sas_irq=(`ls /proc/irq/$IRQ_NUMBER/ | grep mpt3sas | wc -l`)
if [ $mpt3sas_irq == 1 ]; then
echo "hintpolicy=subset"
echo "balance_level=cache"
fi
fi
-----------------------------------------------------------------------------------------------
But still I don't see any load shift happening between the CPUs and
still observing hardlockups.
Here I have attached the irqbalance logs.
Thanks,
Sreekanth
>
>> while running heavy IOs, for first few seconds here is my driver irq's
>> attributes,
>> --------------------------------------------------------------------------------------------------------------------
>> ioc number = 0
>> number of core processors = 24
>> msix vector count = 2
>> number of cores per msix vector = 16
>>
>>
>> msix index = 0, irq number = 50, smp_affinity = 000040
>> affinity_hint = 000fff
>> msix index = 1, irq number = 51, smp_affinity = 001000
>> affinity_hint = fff000
>>
>> We have set affinity for 2 msix vectors and 24 core processors
>> ----------------------------------------------------------------------------------------------------------------------
>>
>> After few seconds it observed that CPU12 is heavily loaded for IRQ 51
>> and it changed the smp_affinity to CPU21
>> --------------------------------------------------------------------------------------------------------------------
>> ioc number = 0
>> number of core processors = 24
>> msix vector count = 2
>> number of cores per msix vector = 16
>>
>>
>> msix index = 0, irq number = 50, smp_affinity = 000040
>> affinity_hint = 000fff
>> msix index = 1, irq number = 51, smp_affinity = 200000
>> affinity_hint = fff000
>>
>> We have set affinity for 2 msix vectors and 24 core processors
>> ---------------------------------------------------------------------------------------------------------------------
>>
>> Where as irqblanance versioned 1.0.9 is not able to shift the load to
>> the other CPUs enabled in the affinity_hint (even when subset policy
>> is enabled) and so I was observing the softlocks/hardlockups.
>>
>> Here I have attached irqbalance logs with debug enabled for both versions.
>>
>> Thanks,
>> Sreekanth
>
>
>
[-- Attachment #2: irqbalance_1.0.9_with_set_policy_logs --]
[-- Type: application/octet-stream, Size: 348160 bytes --]
Isolated CPUs: 00000000
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
CPU number 12 numa_node is 0 (load 0)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
CPU number 13 numa_node is 0 (load 0)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
CPU number 14 numa_node is 0 (load 0)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
CPU number 15 numa_node is 0 (load 0)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
CPU number 16 numa_node is 0 (load 0)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
CPU number 17 numa_node is 0 (load 0)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
CPU number 18 numa_node is 0 (load 0)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
CPU number 19 numa_node is 0 (load 0)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
CPU number 20 numa_node is 0 (load 0)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
CPU number 21 numa_node is 0 (load 0)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
CPU number 22 numa_node is 0 (load 0)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
CPU number 23 numa_node is 0 (load 0)
Adding IRQ 25 to database
Adding IRQ 27 to database
Adding IRQ 28 to database
Adding IRQ 30 to database
Adding IRQ 31 to database
Adding IRQ 32 to database
Adding IRQ 33 to database
Adding IRQ 206 to database
Adding IRQ 208 to database
Adding IRQ 209 to database
Adding IRQ 210 to database
Adding IRQ 211 to database
Adding IRQ 212 to database
Adding IRQ 213 to database
Adding IRQ 214 to database
Adding IRQ 44 to database
Adding IRQ 46 to database
Adding IRQ 11 to database
Adding IRQ 10 to database
Adding IRQ 18 to database
Adding IRQ 34 to database
Adding IRQ 35 to database
Adding IRQ 45 to database
Adding IRQ 5 to database
Adding IRQ 58 to database
Adding IRQ 59 to database
Adding IRQ 60 to database
Adding IRQ 61 to database
Adding IRQ 62 to database
Adding IRQ 63 to database
Adding IRQ 64 to database
Adding IRQ 65 to database
Adding IRQ 66 to database
Adding IRQ 67 to database
Adding IRQ 68 to database
Adding IRQ 69 to database
Adding IRQ 70 to database
Adding IRQ 71 to database
Adding IRQ 72 to database
Adding IRQ 73 to database
Adding IRQ 74 to database
Adding IRQ 75 to database
Adding IRQ 76 to database
Adding IRQ 77 to database
Adding IRQ 78 to database
Adding IRQ 79 to database
Adding IRQ 80 to database
Adding IRQ 81 to database
Adding IRQ 82 to database
Adding IRQ 84 to database
Adding IRQ 85 to database
Adding IRQ 86 to database
Adding IRQ 87 to database
Adding IRQ 88 to database
Adding IRQ 89 to database
Adding IRQ 90 to database
Adding IRQ 91 to database
Adding IRQ 92 to database
Adding IRQ 93 to database
Adding IRQ 94 to database
Adding IRQ 95 to database
Adding IRQ 96 to database
Adding IRQ 97 to database
Adding IRQ 98 to database
Adding IRQ 99 to database
Adding IRQ 100 to database
Adding IRQ 101 to database
Adding IRQ 102 to database
Adding IRQ 103 to database
Adding IRQ 104 to database
Adding IRQ 105 to database
Adding IRQ 106 to database
Adding IRQ 107 to database
Adding IRQ 108 to database
Adding IRQ 109 to database
Adding IRQ 110 to database
Adding IRQ 111 to database
Adding IRQ 112 to database
Adding IRQ 113 to database
Adding IRQ 114 to database
Adding IRQ 115 to database
Adding IRQ 116 to database
Adding IRQ 117 to database
Adding IRQ 118 to database
Adding IRQ 119 to database
Adding IRQ 120 to database
Adding IRQ 121 to database
Adding IRQ 122 to database
Adding IRQ 123 to database
Adding IRQ 124 to database
Adding IRQ 125 to database
Adding IRQ 126 to database
Adding IRQ 127 to database
Adding IRQ 128 to database
Adding IRQ 129 to database
Adding IRQ 130 to database
Adding IRQ 131 to database
Adding IRQ 132 to database
Adding IRQ 181 to database
Adding IRQ 182 to database
Adding IRQ 183 to database
Adding IRQ 184 to database
Adding IRQ 185 to database
Adding IRQ 186 to database
Adding IRQ 187 to database
Adding IRQ 188 to database
Adding IRQ 189 to database
Adding IRQ 190 to database
Adding IRQ 191 to database
Adding IRQ 192 to database
Adding IRQ 193 to database
Adding IRQ 194 to database
Adding IRQ 195 to database
Adding IRQ 196 to database
Adding IRQ 197 to database
Adding IRQ 198 to database
Adding IRQ 199 to database
Adding IRQ 200 to database
Adding IRQ 201 to database
Adding IRQ 202 to database
Adding IRQ 203 to database
Adding IRQ 204 to database
Adding IRQ 157 to database
Adding IRQ 158 to database
Adding IRQ 159 to database
Adding IRQ 160 to database
Adding IRQ 161 to database
Adding IRQ 162 to database
Adding IRQ 163 to database
Adding IRQ 164 to database
Adding IRQ 165 to database
Adding IRQ 166 to database
Adding IRQ 167 to database
Adding IRQ 168 to database
Adding IRQ 169 to database
Adding IRQ 170 to database
Adding IRQ 171 to database
Adding IRQ 172 to database
Adding IRQ 173 to database
Adding IRQ 174 to database
Adding IRQ 175 to database
Adding IRQ 176 to database
Adding IRQ 177 to database
Adding IRQ 178 to database
Adding IRQ 179 to database
Adding IRQ 180 to database
Adding IRQ 133 to database
Adding IRQ 134 to database
Adding IRQ 135 to database
Adding IRQ 136 to database
Adding IRQ 137 to database
Adding IRQ 138 to database
Adding IRQ 139 to database
Adding IRQ 140 to database
Adding IRQ 141 to database
Adding IRQ 142 to database
Adding IRQ 143 to database
Adding IRQ 144 to database
Adding IRQ 145 to database
Adding IRQ 146 to database
Adding IRQ 147 to database
Adding IRQ 148 to database
Adding IRQ 149 to database
Adding IRQ 150 to database
Adding IRQ 151 to database
Adding IRQ 152 to database
Adding IRQ 153 to database
Adding IRQ 154 to database
Adding IRQ 155 to database
Adding IRQ 156 to database
Adding IRQ 37 to database
Adding IRQ 39 to database
Adding IRQ 41 to database
Adding IRQ 43 to database
Adding IRQ 216 to database
Adding IRQ 218 to database
Adding IRQ 219 to database
Adding IRQ 220 to database
Adding IRQ 221 to database
Adding IRQ 222 to database
Adding IRQ 223 to database
Adding IRQ 224 to database
IRQ 50: Override hintpolicy to subset
IRQ 50: Override balance_level to cache
Adding IRQ 50 to database
IRQ 51: Override hintpolicy to subset
IRQ 51: Override balance_level to cache
Adding IRQ 51 to database
PCI: can't open file:/sys/vendor
Adding IRQ 0 to database
PCI: can't open file:/sys/vendor
Adding IRQ 4 to database
PCI: can't open file:/sys/vendor
Adding IRQ 8 to database
PCI: can't open file:/sys/vendor
Adding IRQ 9 to database
PCI: can't open file:/sys/vendor
Adding IRQ 29 to database
PCI: can't open file:/sys/vendor
Adding IRQ 47 to database
PCI: can't open file:/sys/vendor
Adding IRQ 48 to database
PCI: can't open file:/sys/vendor
Adding IRQ 49 to database
NUMA NODE NUMBER: -1
LOCAL CPU MASK: ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
NUMA NODE NUMBER: 0
LOCAL CPU MASK: 00ffffff
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/0)
Interrupt 174 node_num is 0 (storage/0)
Interrupt 198 node_num is 0 (storage/0)
Interrupt 126 node_num is 0 (storage/0)
Interrupt 102 node_num is 0 (ethernet/0)
Interrupt 77 node_num is 0 (ethernet/0)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/0)
Interrupt 162 node_num is 0 (storage/0)
Interrupt 186 node_num is 0 (storage/0)
Interrupt 114 node_num is 0 (storage/0)
Interrupt 90 node_num is 0 (ethernet/0)
Interrupt 65 node_num is 0 (ethernet/0)
Interrupt 51 node_num is -1 (storage/0)
Interrupt 31 node_num is 0 (legacy/0)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/0)
Interrupt 176 node_num is 0 (storage/0)
Interrupt 200 node_num is 0 (storage/0)
Interrupt 128 node_num is 0 (storage/0)
Interrupt 104 node_num is 0 (ethernet/0)
Interrupt 79 node_num is 0 (ethernet/0)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/0)
Interrupt 164 node_num is 0 (storage/0)
Interrupt 188 node_num is 0 (storage/0)
Interrupt 116 node_num is 0 (storage/0)
Interrupt 92 node_num is 0 (ethernet/0)
Interrupt 67 node_num is 0 (ethernet/0)
Interrupt 33 node_num is 0 (legacy/0)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/0)
Interrupt 178 node_num is 0 (storage/0)
Interrupt 202 node_num is 0 (storage/0)
Interrupt 130 node_num is 0 (storage/0)
Interrupt 106 node_num is 0 (ethernet/0)
Interrupt 81 node_num is 0 (ethernet/0)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/0)
Interrupt 166 node_num is 0 (storage/0)
Interrupt 190 node_num is 0 (storage/0)
Interrupt 118 node_num is 0 (storage/0)
Interrupt 94 node_num is 0 (ethernet/0)
Interrupt 69 node_num is 0 (ethernet/0)
Interrupt 46 node_num is 0 (legacy/0)
Interrupt 34 node_num is 0 (legacy/0)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/0)
Interrupt 180 node_num is 0 (storage/0)
Interrupt 204 node_num is 0 (storage/0)
Interrupt 132 node_num is 0 (storage/0)
Interrupt 108 node_num is 0 (ethernet/0)
Interrupt 84 node_num is 0 (ethernet/0)
Interrupt 59 node_num is 0 (ethernet/0)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/0)
Interrupt 168 node_num is 0 (storage/0)
Interrupt 192 node_num is 0 (storage/0)
Interrupt 120 node_num is 0 (storage/0)
Interrupt 96 node_num is 0 (ethernet/0)
Interrupt 71 node_num is 0 (ethernet/0)
Interrupt 45 node_num is 0 (storage/0)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/0)
Interrupt 170 node_num is 0 (storage/0)
Interrupt 194 node_num is 0 (storage/0)
Interrupt 122 node_num is 0 (storage/0)
Interrupt 98 node_num is 0 (ethernet/0)
Interrupt 73 node_num is 0 (ethernet/0)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/0)
Interrupt 158 node_num is 0 (storage/0)
Interrupt 182 node_num is 0 (storage/0)
Interrupt 110 node_num is 0 (storage/0)
Interrupt 86 node_num is 0 (ethernet/0)
Interrupt 61 node_num is 0 (ethernet/0)
Interrupt 39 node_num is -1 (legacy/0)
Interrupt 25 node_num is 0 (legacy/0)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/0)
Interrupt 172 node_num is 0 (storage/0)
Interrupt 196 node_num is 0 (storage/0)
Interrupt 124 node_num is 0 (storage/0)
Interrupt 100 node_num is 0 (ethernet/0)
Interrupt 75 node_num is 0 (ethernet/0)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/0)
Interrupt 160 node_num is 0 (storage/0)
Interrupt 184 node_num is 0 (storage/0)
Interrupt 112 node_num is 0 (storage/0)
Interrupt 88 node_num is 0 (ethernet/0)
Interrupt 63 node_num is 0 (ethernet/0)
Interrupt 43 node_num is -1 (legacy/0)
Interrupt 28 node_num is 0 (legacy/0)
Interrupt 224 node_num is -1 (other/0)
Interrupt 222 node_num is -1 (other/0)
Interrupt 220 node_num is -1 (other/0)
Interrupt 218 node_num is -1 (other/0)
Interrupt 214 node_num is 0 (other/0)
Interrupt 212 node_num is 0 (other/0)
Interrupt 210 node_num is 0 (other/0)
Interrupt 208 node_num is 0 (other/0)
Interrupt 10 node_num is 0 (other/0)
Interrupt 49 node_num is -1 (other/0)
Interrupt 47 node_num is -1 (other/0)
Interrupt 9 node_num is -1 (other/0)
Interrupt 4 node_num is -1 (other/0)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/0)
Interrupt 173 node_num is 0 (storage/0)
Interrupt 197 node_num is 0 (storage/0)
Interrupt 125 node_num is 0 (storage/0)
Interrupt 101 node_num is 0 (ethernet/0)
Interrupt 76 node_num is 0 (ethernet/0)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/0)
Interrupt 161 node_num is 0 (storage/0)
Interrupt 185 node_num is 0 (storage/0)
Interrupt 113 node_num is 0 (storage/0)
Interrupt 89 node_num is 0 (ethernet/0)
Interrupt 64 node_num is 0 (ethernet/0)
Interrupt 50 node_num is -1 (storage/0)
Interrupt 30 node_num is 0 (legacy/0)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/0)
Interrupt 175 node_num is 0 (storage/0)
Interrupt 199 node_num is 0 (storage/0)
Interrupt 127 node_num is 0 (storage/0)
Interrupt 103 node_num is 0 (ethernet/0)
Interrupt 78 node_num is 0 (ethernet/0)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/0)
Interrupt 163 node_num is 0 (storage/0)
Interrupt 187 node_num is 0 (storage/0)
Interrupt 115 node_num is 0 (storage/0)
Interrupt 91 node_num is 0 (ethernet/0)
Interrupt 66 node_num is 0 (ethernet/0)
Interrupt 32 node_num is 0 (legacy/0)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/0)
Interrupt 177 node_num is 0 (storage/0)
Interrupt 201 node_num is 0 (storage/0)
Interrupt 129 node_num is 0 (storage/0)
Interrupt 105 node_num is 0 (ethernet/0)
Interrupt 80 node_num is 0 (ethernet/0)
Interrupt 44 node_num is 0 (storage/0)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/0)
Interrupt 165 node_num is 0 (storage/0)
Interrupt 189 node_num is 0 (storage/0)
Interrupt 117 node_num is 0 (storage/0)
Interrupt 93 node_num is 0 (ethernet/0)
Interrupt 68 node_num is 0 (ethernet/0)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/0)
Interrupt 179 node_num is 0 (storage/0)
Interrupt 203 node_num is 0 (storage/0)
Interrupt 131 node_num is 0 (storage/0)
Interrupt 107 node_num is 0 (ethernet/0)
Interrupt 82 node_num is 0 (ethernet/0)
Interrupt 58 node_num is 0 (ethernet/0)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/0)
Interrupt 167 node_num is 0 (storage/0)
Interrupt 191 node_num is 0 (storage/0)
Interrupt 119 node_num is 0 (storage/0)
Interrupt 95 node_num is 0 (ethernet/0)
Interrupt 70 node_num is 0 (ethernet/0)
Interrupt 18 node_num is 0 (legacy/0)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/0)
Interrupt 169 node_num is 0 (storage/0)
Interrupt 193 node_num is 0 (storage/0)
Interrupt 121 node_num is 0 (storage/0)
Interrupt 97 node_num is 0 (ethernet/0)
Interrupt 72 node_num is 0 (ethernet/0)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/0)
Interrupt 157 node_num is 0 (storage/0)
Interrupt 181 node_num is 0 (storage/0)
Interrupt 109 node_num is 0 (storage/0)
Interrupt 85 node_num is 0 (ethernet/0)
Interrupt 60 node_num is 0 (ethernet/0)
Interrupt 37 node_num is -1 (legacy/0)
Interrupt 35 node_num is 0 (legacy/0)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/0)
Interrupt 171 node_num is 0 (storage/0)
Interrupt 195 node_num is 0 (storage/0)
Interrupt 123 node_num is 0 (storage/0)
Interrupt 99 node_num is 0 (ethernet/0)
Interrupt 74 node_num is 0 (ethernet/0)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/0)
Interrupt 159 node_num is 0 (storage/0)
Interrupt 183 node_num is 0 (storage/0)
Interrupt 111 node_num is 0 (storage/0)
Interrupt 87 node_num is 0 (ethernet/0)
Interrupt 62 node_num is 0 (ethernet/0)
Interrupt 41 node_num is -1 (legacy/0)
Interrupt 27 node_num is 0 (legacy/0)
Interrupt 223 node_num is -1 (other/0)
Interrupt 221 node_num is -1 (other/0)
Interrupt 219 node_num is -1 (other/0)
Interrupt 216 node_num is -1 (other/0)
Interrupt 213 node_num is 0 (other/0)
Interrupt 211 node_num is 0 (other/0)
Interrupt 209 node_num is 0 (other/0)
Interrupt 206 node_num is 0 (other/0)
Interrupt 5 node_num is 0 (other/0)
Interrupt 11 node_num is 0 (other/0)
Interrupt 48 node_num is -1 (other/0)
Interrupt 29 node_num is -1 (other/0)
Interrupt 8 node_num is -1 (other/0)
Interrupt 0 node_num is -1 (other/0)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Selecting irq 101 for rebalancing
Selecting irq 76 for rebalancing
Package 0: numa_node is 0 cpu mask is 0003f03f (load 2688)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 1344)
CPU number 1 numa_node is 0 (load 1344)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1344)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 1344)
CPU number 3 numa_node is 0 (load 1344)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1344)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 833333)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 5000000)
CPU number 6 numa_node is 0 (load 10000000)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/4993464)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/104166)
Interrupt 221 node_num is -1 (other/104166)
Interrupt 219 node_num is -1 (other/104166)
Interrupt 216 node_num is -1 (other/104166)
Interrupt 213 node_num is 0 (other/104166)
Interrupt 211 node_num is 0 (other/104166)
Interrupt 209 node_num is 0 (other/104166)
Interrupt 206 node_num is 0 (other/104166)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 833333)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 5000000)
CPU number 6 numa_node is 0 (load 10000000)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/4989390)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Interrupt 74 node_num is 0 (ethernet/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/104166)
Interrupt 221 node_num is -1 (other/104166)
Interrupt 219 node_num is -1 (other/104166)
Interrupt 216 node_num is -1 (other/104166)
Interrupt 213 node_num is 0 (other/104166)
Interrupt 211 node_num is 0 (other/104166)
Interrupt 209 node_num is 0 (other/104166)
Interrupt 206 node_num is 0 (other/104166)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Selecting irq 99 for rebalancing
Package 0: numa_node is 0 cpu mask is 0003f03f (load 925925)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 925925)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 925925)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/925925)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 833333)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 5000000)
CPU number 11 numa_node is 0 (load 10000000)
Interrupt 74 node_num is 0 (ethernet/9074065)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Selecting irq 104 for rebalancing
Selecting irq 79 for rebalancing
Selecting irq 76 for rebalancing
Selecting irq 9 for rebalancing
Package 0: numa_node is 0 cpu mask is 0003f03f (load 1626983)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 5000000)
CPU number 1 numa_node is 0 (load 10000000)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 793650)
CPU number 3 numa_node is 0 (load 793650)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/793650)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 4 node_num is -1 (other/801523)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 1618958)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 793650)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 793650)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/793650)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 793650)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 793650)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 104 node_num is 0 (ethernet/793650)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/31658)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 92 node_num is 0 (ethernet/1)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 4 node_num is -1 (other/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Selecting irq 92 for rebalancing
Package 0: numa_node is 0 cpu mask is 0003f03f (load 833333)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 5000000)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
CPU number 13 numa_node is 0 (load 10000000)
Interrupt 67 node_num is 0 (ethernet/8717940)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 1282050)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 1282050)
CPU number 9 numa_node is 0 (load 1282050)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
Interrupt 92 node_num is 0 (ethernet/1282050)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 833333)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 5000000)
CPU number 1 numa_node is 0 (load 10000000)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
Interrupt 92 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 73 node_num is 0 (ethernet/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 224 node_num is -1 (other/1)
Interrupt 222 node_num is -1 (other/1)
Interrupt 220 node_num is -1 (other/1)
Interrupt 218 node_num is -1 (other/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
Interrupt 92 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
-----------------------------------------------------------------------------
Selecting irq 98 for rebalancing
Selecting irq 224 for rebalancing
Selecting irq 222 for rebalancing
Selecting irq 220 for rebalancing
Selecting irq 218 for rebalancing
Package 0: numa_node is 0 cpu mask is 0003f03f (load 833333)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 5000000)
CPU number 4 numa_node is 0 (load 10000000)
Interrupt 73 node_num is 0 (ethernet/7499988)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 214 node_num is 0 (other/104166)
Interrupt 212 node_num is 0 (other/104166)
Interrupt 210 node_num is 0 (other/104166)
Interrupt 208 node_num is 0 (other/104166)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 2499994)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 2083330)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
Interrupt 92 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 2083330)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 98 node_num is 0 (ethernet/2083330)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
CPU number 23 numa_node is 0 (load 0)
Interrupt 135 node_num is 0 (storage/1)
Interrupt 159 node_num is 0 (storage/1)
Interrupt 183 node_num is 0 (storage/1)
Interrupt 111 node_num is 0 (storage/1)
Interrupt 87 node_num is 0 (ethernet/1)
Interrupt 62 node_num is 0 (ethernet/1)
Interrupt 41 node_num is -1 (legacy/1)
Interrupt 27 node_num is 0 (legacy/1)
Interrupt 223 node_num is -1 (other/1)
Interrupt 221 node_num is -1 (other/1)
Interrupt 219 node_num is -1 (other/1)
Interrupt 216 node_num is -1 (other/1)
Interrupt 213 node_num is 0 (other/1)
Interrupt 211 node_num is 0 (other/1)
Interrupt 209 node_num is 0 (other/1)
Interrupt 206 node_num is 0 (other/1)
Interrupt 5 node_num is 0 (other/1)
Interrupt 11 node_num is 0 (other/1)
Interrupt 48 node_num is -1 (other/1)
Interrupt 29 node_num is -1 (other/1)
Interrupt 8 node_num is -1 (other/1)
Interrupt 0 node_num is -1 (other/1)
Interrupt 9 node_num is -1 (other/1)
Interrupt 224 node_num is -1 (other/104166)
Interrupt 222 node_num is -1 (other/104166)
Interrupt 220 node_num is -1 (other/104166)
Interrupt 218 node_num is -1 (other/104166)
-----------------------------------------------------------------------------
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
Cache domain 1: numa_node is 0 cpu mask is 00002002 (load 0)
CPU number 1 numa_node is 0 (load 0)
Interrupt 152 node_num is 0 (storage/1)
Interrupt 176 node_num is 0 (storage/1)
Interrupt 200 node_num is 0 (storage/1)
Interrupt 128 node_num is 0 (storage/1)
CPU number 13 numa_node is 0 (load 0)
Interrupt 67 node_num is 0 (ethernet/1)
Interrupt 140 node_num is 0 (storage/1)
Interrupt 164 node_num is 0 (storage/1)
Interrupt 188 node_num is 0 (storage/1)
Interrupt 116 node_num is 0 (storage/1)
Interrupt 33 node_num is 0 (legacy/1)
Cache domain 2: numa_node is 0 cpu mask is 00004004 (load 0)
CPU number 2 numa_node is 0 (load 0)
Interrupt 154 node_num is 0 (storage/1)
Interrupt 178 node_num is 0 (storage/1)
Interrupt 202 node_num is 0 (storage/1)
Interrupt 130 node_num is 0 (storage/1)
Interrupt 106 node_num is 0 (ethernet/1)
Interrupt 81 node_num is 0 (ethernet/1)
CPU number 14 numa_node is 0 (load 0)
Interrupt 142 node_num is 0 (storage/1)
Interrupt 166 node_num is 0 (storage/1)
Interrupt 190 node_num is 0 (storage/1)
Interrupt 118 node_num is 0 (storage/1)
Interrupt 94 node_num is 0 (ethernet/1)
Interrupt 69 node_num is 0 (ethernet/1)
Interrupt 46 node_num is 0 (legacy/1)
Interrupt 34 node_num is 0 (legacy/1)
Cache domain 3: numa_node is 0 cpu mask is 00008008 (load 0)
CPU number 3 numa_node is 0 (load 0)
Interrupt 156 node_num is 0 (storage/1)
Interrupt 180 node_num is 0 (storage/1)
Interrupt 204 node_num is 0 (storage/1)
Interrupt 132 node_num is 0 (storage/1)
Interrupt 108 node_num is 0 (ethernet/1)
Interrupt 84 node_num is 0 (ethernet/1)
Interrupt 59 node_num is 0 (ethernet/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 15 numa_node is 0 (load 0)
Interrupt 144 node_num is 0 (storage/1)
Interrupt 168 node_num is 0 (storage/1)
Interrupt 192 node_num is 0 (storage/1)
Interrupt 120 node_num is 0 (storage/1)
Interrupt 96 node_num is 0 (ethernet/1)
Interrupt 71 node_num is 0 (ethernet/1)
Interrupt 45 node_num is 0 (storage/1)
Interrupt 99 node_num is 0 (ethernet/1)
Cache domain 4: numa_node is 0 cpu mask is 00010010 (load 0)
CPU number 4 numa_node is 0 (load 0)
Interrupt 73 node_num is 0 (ethernet/1)
Interrupt 146 node_num is 0 (storage/1)
Interrupt 170 node_num is 0 (storage/1)
Interrupt 194 node_num is 0 (storage/1)
Interrupt 122 node_num is 0 (storage/1)
CPU number 16 numa_node is 0 (load 0)
Interrupt 134 node_num is 0 (storage/1)
Interrupt 158 node_num is 0 (storage/1)
Interrupt 182 node_num is 0 (storage/1)
Interrupt 110 node_num is 0 (storage/1)
Interrupt 86 node_num is 0 (ethernet/1)
Interrupt 61 node_num is 0 (ethernet/1)
Interrupt 39 node_num is -1 (legacy/1)
Interrupt 25 node_num is 0 (legacy/1)
Cache domain 5: numa_node is 0 cpu mask is 00020020 (load 0)
CPU number 5 numa_node is 0 (load 0)
Interrupt 148 node_num is 0 (storage/1)
Interrupt 172 node_num is 0 (storage/1)
Interrupt 196 node_num is 0 (storage/1)
Interrupt 124 node_num is 0 (storage/1)
Interrupt 100 node_num is 0 (ethernet/1)
Interrupt 75 node_num is 0 (ethernet/1)
CPU number 17 numa_node is 0 (load 0)
Interrupt 136 node_num is 0 (storage/1)
Interrupt 160 node_num is 0 (storage/1)
Interrupt 184 node_num is 0 (storage/1)
Interrupt 112 node_num is 0 (storage/1)
Interrupt 88 node_num is 0 (ethernet/1)
Interrupt 63 node_num is 0 (ethernet/1)
Interrupt 43 node_num is -1 (legacy/1)
Interrupt 28 node_num is 0 (legacy/1)
Interrupt 214 node_num is 0 (other/1)
Interrupt 212 node_num is 0 (other/1)
Interrupt 210 node_num is 0 (other/1)
Interrupt 208 node_num is 0 (other/1)
Interrupt 10 node_num is 0 (other/1)
Interrupt 49 node_num is -1 (other/1)
Interrupt 47 node_num is -1 (other/1)
Interrupt 4 node_num is -1 (other/1)
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
Interrupt 30 node_num is 0 (legacy/1)
Cache domain 7: numa_node is 0 cpu mask is 00080080 (load 0)
CPU number 7 numa_node is 0 (load 0)
Interrupt 151 node_num is 0 (storage/1)
Interrupt 175 node_num is 0 (storage/1)
Interrupt 199 node_num is 0 (storage/1)
Interrupt 127 node_num is 0 (storage/1)
Interrupt 103 node_num is 0 (ethernet/1)
Interrupt 78 node_num is 0 (ethernet/1)
CPU number 19 numa_node is 0 (load 0)
Interrupt 139 node_num is 0 (storage/1)
Interrupt 163 node_num is 0 (storage/1)
Interrupt 187 node_num is 0 (storage/1)
Interrupt 115 node_num is 0 (storage/1)
Interrupt 91 node_num is 0 (ethernet/1)
Interrupt 66 node_num is 0 (ethernet/1)
Interrupt 32 node_num is 0 (legacy/1)
Cache domain 8: numa_node is 0 cpu mask is 00100100 (load 0)
CPU number 8 numa_node is 0 (load 0)
Interrupt 153 node_num is 0 (storage/1)
Interrupt 177 node_num is 0 (storage/1)
Interrupt 201 node_num is 0 (storage/1)
Interrupt 129 node_num is 0 (storage/1)
Interrupt 105 node_num is 0 (ethernet/1)
Interrupt 80 node_num is 0 (ethernet/1)
Interrupt 44 node_num is 0 (storage/1)
CPU number 20 numa_node is 0 (load 0)
Interrupt 141 node_num is 0 (storage/1)
Interrupt 165 node_num is 0 (storage/1)
Interrupt 189 node_num is 0 (storage/1)
Interrupt 117 node_num is 0 (storage/1)
Interrupt 93 node_num is 0 (ethernet/1)
Interrupt 68 node_num is 0 (ethernet/1)
Interrupt 79 node_num is 0 (ethernet/1)
Cache domain 9: numa_node is 0 cpu mask is 00200200 (load 0)
CPU number 9 numa_node is 0 (load 0)
Interrupt 155 node_num is 0 (storage/1)
Interrupt 179 node_num is 0 (storage/1)
Interrupt 203 node_num is 0 (storage/1)
Interrupt 131 node_num is 0 (storage/1)
Interrupt 107 node_num is 0 (ethernet/1)
Interrupt 82 node_num is 0 (ethernet/1)
Interrupt 58 node_num is 0 (ethernet/1)
Interrupt 92 node_num is 0 (ethernet/1)
CPU number 21 numa_node is 0 (load 0)
Interrupt 143 node_num is 0 (storage/1)
Interrupt 167 node_num is 0 (storage/1)
Interrupt 191 node_num is 0 (storage/1)
Interrupt 119 node_num is 0 (storage/1)
Interrupt 95 node_num is 0 (ethernet/1)
Interrupt 70 node_num is 0 (ethernet/1)
Interrupt 104 node_num is 0 (ethernet/1)
Interrupt 98 node_num is 0 (ethernet/1)
Interrupt 18 node_num is 0 (legacy/1)
Cache domain 10: numa_node is 0 cpu mask is 00400400 (load 0)
CPU number 10 numa_node is 0 (load 0)
Interrupt 145 node_num is 0 (storage/1)
Interrupt 169 node_num is 0 (storage/1)
Interrupt 193 node_num is 0 (storage/1)
Interrupt 121 node_num is 0 (storage/1)
Interrupt 97 node_num is 0 (ethernet/1)
Interrupt 72 node_num is 0 (ethernet/1)
CPU number 22 numa_node is 0 (load 0)
Interrupt 133 node_num is 0 (storage/1)
Interrupt 157 node_num is 0 (storage/1)
Interrupt 181 node_num is 0 (storage/1)
Interrupt 109 node_num is 0 (storage/1)
Interrupt 85 node_num is 0 (ethernet/1)
Interrupt 60 node_num is 0 (ethernet/1)
Interrupt 37 node_num is -1 (legacy/1)
Interrupt 35 node_num is 0 (legacy/1)
Cache domain 11: numa_node is 0 cpu mask is 00800800 (load 0)
CPU number 11 numa_node is 0 (load 0)
Interrupt 74 node_num is 0 (ethernet/1)
Interrupt 147 node_num is 0 (storage/1)
Interrupt 171 node_num is 0 (storage/1)
Interrupt 195 node_num is 0 (storage/1)
Interrupt 123 node_num is 0 (storage/1)
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-09-07 6:00 ` Sreekanth Reddy
@ 2016-09-07 13:24 ` Neil Horman
2016-09-08 5:42 ` Sreekanth Reddy
0 siblings, 1 reply; 16+ messages in thread
From: Neil Horman @ 2016-09-07 13:24 UTC (permalink / raw)
To: Sreekanth Reddy
Cc: Bart Van Assche, Elliott, Robert (Persistent Memory),
linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote:
> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> > On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote:
> >> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche
> >> <bart.vanassche@sandisk.com> wrote:
> >> > On 09/01/2016 03:31 AM, Sreekanth Reddy wrote:
> >> >>
> >> >> I reduced the ISR workload by one third in-order to reduce the time
> >> >> that is spent per CPU in interrupt context, even then I am observing
> >> >> softlockups.
> >> >>
> >> >> As I mentioned before only same single CPU in the set of CPUs(enabled
> >> >> in affinity_hint) is busy with handling the interrupts from
> >> >> corresponding IRQx. I have done below experiment in driver to limit
> >> >> these softlockups/hardlockups. But I am not sure whether it is
> >> >> reasonable to do this in driver,
> >> >>
> >> >> Experiment:
> >> >> If the CPUx is continuously busy with handling the remote CPUs
> >> >> (enabled in the corresponding IRQ's affinity_hint) IO works by 1/4th
> >> >> of the HBA queue depth in the same ISR context then enable a flag
> >> >> called 'change_smp_affinity' for this IRQ. Also created a thread with
> >> >> will poll for this flag for every IRQ's (enabled by driver) for every
> >> >> second. If this thread see that this flag is enabled for any IRQ then
> >> >> it will write next CPU number from the CPUs enabled in the IRQ's
> >> >> affinity_hint to the IRQ's smp_affinity procfs attribute using
> >> >> 'call_usermodehelper()' API.
> >> >>
> >> >> This to make sure that interrupts are not processed by same single CPU
> >> >> all the time and to make the other CPUs to handle the interrupts if
> >> >> the current CPU is continuously busy with handling the other CPUs IO
> >> >> interrupts.
> >> >>
> >> >> For example consider a system which has 8 logical CPUs and one MSIx
> >> >> vector enabled (called IRQ 120) in driver, HBA queue depth as 8K.
> >> >> then IRQ's procfs attributes will be
> >> >> IRQ# 120, affinity_hint=0xff, smp_affinity=0x00
> >> >>
> >> >> After starting heavy IOs, we will observe that only CPU0 will be busy
> >> >> with handling the interrupts. This experiment driver will change the
> >> >> smp_affinity to next CPU number i.e. 0x01 (using cmd 'echo 0x01 >
> >> >> /proc/irq/120/smp_affinity', driver issue's this cmd using
> >> >> call_usermodehelper() API) if it observes that CPU0 is continuously
> >> >> processing more than 2K of IOs replies of other CPUs i.e from CPU1 to
> >> >> CPU7.
> >> >>
> >> >> Whether doing this kind of stuff in driver is ok?
> >> >
> >> >
> >> > Hello Sreekanth,
> >> >
> >> > To me this sounds like something that should be implemented in the I/O
> >> > chipset on the motherboard. If you have a look at the Intel Software
> >> > Developer Manuals then you will see that logical destination mode supports
> >> > round-robin interrupt delivery. However, the Linux kernel selects physical
> >> > destination mode on systems with more than eight logical CPUs (see also
> >> > arch/x86/kernel/apic/apic_flat_64.c).
> >> >
> >> > I'm not sure the maintainers of the interrupt subsystem would welcome code
> >> > that emulates round-robin interrupt delivery. So your best option is
> >> > probably to minimize the amount of work that is done in interrupt context
> >> > and to move as much work as possible out of interrupt context in such a way
> >> > that it can be spread over multiple CPU cores, e.g. by using
> >> > queue_work_on().
> >> >
> >> > Bart.
> >>
> >> Bart,
> >>
> >> Thanks a lot for providing lot of inputs and valuable information on this issue.
> >>
> >> Today I got one more observation. i.e. I am not observing any lockups
> >> if I use 1.0.4-6 versioned irqbalance.
> >> Since this versioned irqbalance is able to shift the load to other CPU
> >> when one CPU is heavily loaded.
> >>
> >
> > This isn't happening because irqbalance is no longer able to shift load between
> > cpus, its happening because of commit 996ee2cf7a4d10454de68ac4978adb5cf22850f8.
> > irqs with higher interrupt volumes sould be balanced to a specific cpu core,
> > rather than to a cache domain to maximize cpu-local cache hit rates. Prior to
> > that change we balanced to a cache domain and your workload didn't have to
> > serialize multiple interrupts to a single core. My suggestion to you is to use
> > the --policyscript option to make your storage irqs get balanced to the cache
> > level, rather than the core level. That should return the behavior to what you
> > want.
> >
> > Neil
>
> Hi Neil,
>
> Thanks for reply.
>
> Today I tried with setting balance_level to 'cache' for mpt3sas driver
> IRQ's using below policy script and used 1.0.9 versioned irqbalance,
> ----------------------------------------------------------------------------------------------
> #!/bin/bash
> # Header
> # Linux Shell Scripting for Irq Balance Policy select for mpt3sas driver
> #
>
> # Command Line Args
> #IRQ_PATH -> PATH
> #IRQ_NUMBER -> IRQ Number
> declare -r IRQ_PATH=$1
> declare -r IRQ_NUMBER=$2
>
> if [ -d /proc/irq/$IRQ_NUMBER ]; then
> mpt3sas_irq=(`ls /proc/irq/$IRQ_NUMBER/ | grep mpt3sas | wc -l`)
> if [ $mpt3sas_irq == 1 ]; then
> echo "hintpolicy=subset"
> echo "balance_level=cache"
> fi
> fi
> -----------------------------------------------------------------------------------------------
>
> But still I don't see any load shift happening between the CPUs and
> still observing hardlockups.
>
> Here I have attached the irqbalance logs.
>
> Thanks,
> Sreekanth
Hey there-
So, looking at your logs, your script is working correctly:
Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
CPU number 0 numa_node is 0 (load 0)
Interrupt 150 node_num is 0 (storage/1)
Interrupt 174 node_num is 0 (storage/1)
Interrupt 198 node_num is 0 (storage/1)
Interrupt 126 node_num is 0 (storage/1)
Interrupt 102 node_num is 0 (ethernet/1)
Interrupt 77 node_num is 0 (ethernet/1)
CPU number 12 numa_node is 0 (load 0)
Interrupt 138 node_num is 0 (storage/1)
Interrupt 162 node_num is 0 (storage/1)
Interrupt 186 node_num is 0 (storage/1)
Interrupt 114 node_num is 0 (storage/1)
Interrupt 90 node_num is 0 (ethernet/1)
Interrupt 65 node_num is 0 (ethernet/1)
Interrupt 51 node_num is -1 (storage/1)
Interrupt 31 node_num is 0 (legacy/1)
...
Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
CPU number 6 numa_node is 0 (load 0)
Interrupt 149 node_num is 0 (storage/1)
Interrupt 173 node_num is 0 (storage/1)
Interrupt 197 node_num is 0 (storage/1)
Interrupt 125 node_num is 0 (storage/1)
Interrupt 101 node_num is 0 (ethernet/1)
Interrupt 76 node_num is 0 (ethernet/1)
CPU number 18 numa_node is 0 (load 0)
Interrupt 137 node_num is 0 (storage/1)
Interrupt 161 node_num is 0 (storage/1)
Interrupt 185 node_num is 0 (storage/1)
Interrupt 113 node_num is 0 (storage/1)
Interrupt 89 node_num is 0 (ethernet/1)
Interrupt 64 node_num is 0 (ethernet/1)
Interrupt 50 node_num is -1 (storage/1)
irqbalance correctly decided to balance irqs 50 and 51 to the cache level, which
is good. The only other thing I would check though is the affinity_hint those
irqs are exporting. With an affinity hint set to subset, if the exported hint
only intersects the cache domain cpu set at one cpu, you will still only get
affinity for that one cpu. You may want to consider changing the hintpolicy for
those interrupts to ignore, to ensure that you have affinity for two cpus.
Beyond that though, the kernel is in control of irq delivery. Normally the
configured hardware delivery policy is to select the highest priority cpu that
isn't already servicing an interrupt (to maximize cache hit rates). If the irq
rate is sufficiently slow however, it will always hit the same cpu, because it
isn't blocked by another interrupt.
Best
Neil
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-09-07 13:24 ` Neil Horman
@ 2016-09-08 5:42 ` Sreekanth Reddy
2016-09-08 13:39 ` Neil Horman
0 siblings, 1 reply; 16+ messages in thread
From: Sreekanth Reddy @ 2016-09-08 5:42 UTC (permalink / raw)
To: Neil Horman
Cc: Bart Van Assche, Elliott, Robert (Persistent Memory),
linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote:
>> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
>> > On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote:
>> >> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche
>> >> <bart.vanassche@sandisk.com> wrote:
>> >> > On 09/01/2016 03:31 AM, Sreekanth Reddy wrote:
>> >> >>
>> >> >> I reduced the ISR workload by one third in-order to reduce the time
>> >> >> that is spent per CPU in interrupt context, even then I am observing
>> >> >> softlockups.
>> >> >>
>> >> >> As I mentioned before only same single CPU in the set of CPUs(enabled
>> >> >> in affinity_hint) is busy with handling the interrupts from
>> >> >> corresponding IRQx. I have done below experiment in driver to limit
>> >> >> these softlockups/hardlockups. But I am not sure whether it is
>> >> >> reasonable to do this in driver,
>> >> >>
>> >> >> Experiment:
>> >> >> If the CPUx is continuously busy with handling the remote CPUs
>> >> >> (enabled in the corresponding IRQ's affinity_hint) IO works by 1/4th
>> >> >> of the HBA queue depth in the same ISR context then enable a flag
>> >> >> called 'change_smp_affinity' for this IRQ. Also created a thread with
>> >> >> will poll for this flag for every IRQ's (enabled by driver) for every
>> >> >> second. If this thread see that this flag is enabled for any IRQ then
>> >> >> it will write next CPU number from the CPUs enabled in the IRQ's
>> >> >> affinity_hint to the IRQ's smp_affinity procfs attribute using
>> >> >> 'call_usermodehelper()' API.
>> >> >>
>> >> >> This to make sure that interrupts are not processed by same single CPU
>> >> >> all the time and to make the other CPUs to handle the interrupts if
>> >> >> the current CPU is continuously busy with handling the other CPUs IO
>> >> >> interrupts.
>> >> >>
>> >> >> For example consider a system which has 8 logical CPUs and one MSIx
>> >> >> vector enabled (called IRQ 120) in driver, HBA queue depth as 8K.
>> >> >> then IRQ's procfs attributes will be
>> >> >> IRQ# 120, affinity_hint=0xff, smp_affinity=0x00
>> >> >>
>> >> >> After starting heavy IOs, we will observe that only CPU0 will be busy
>> >> >> with handling the interrupts. This experiment driver will change the
>> >> >> smp_affinity to next CPU number i.e. 0x01 (using cmd 'echo 0x01 >
>> >> >> /proc/irq/120/smp_affinity', driver issue's this cmd using
>> >> >> call_usermodehelper() API) if it observes that CPU0 is continuously
>> >> >> processing more than 2K of IOs replies of other CPUs i.e from CPU1 to
>> >> >> CPU7.
>> >> >>
>> >> >> Whether doing this kind of stuff in driver is ok?
>> >> >
>> >> >
>> >> > Hello Sreekanth,
>> >> >
>> >> > To me this sounds like something that should be implemented in the I/O
>> >> > chipset on the motherboard. If you have a look at the Intel Software
>> >> > Developer Manuals then you will see that logical destination mode supports
>> >> > round-robin interrupt delivery. However, the Linux kernel selects physical
>> >> > destination mode on systems with more than eight logical CPUs (see also
>> >> > arch/x86/kernel/apic/apic_flat_64.c).
>> >> >
>> >> > I'm not sure the maintainers of the interrupt subsystem would welcome code
>> >> > that emulates round-robin interrupt delivery. So your best option is
>> >> > probably to minimize the amount of work that is done in interrupt context
>> >> > and to move as much work as possible out of interrupt context in such a way
>> >> > that it can be spread over multiple CPU cores, e.g. by using
>> >> > queue_work_on().
>> >> >
>> >> > Bart.
>> >>
>> >> Bart,
>> >>
>> >> Thanks a lot for providing lot of inputs and valuable information on this issue.
>> >>
>> >> Today I got one more observation. i.e. I am not observing any lockups
>> >> if I use 1.0.4-6 versioned irqbalance.
>> >> Since this versioned irqbalance is able to shift the load to other CPU
>> >> when one CPU is heavily loaded.
>> >>
>> >
>> > This isn't happening because irqbalance is no longer able to shift load between
>> > cpus, its happening because of commit 996ee2cf7a4d10454de68ac4978adb5cf22850f8.
>> > irqs with higher interrupt volumes sould be balanced to a specific cpu core,
>> > rather than to a cache domain to maximize cpu-local cache hit rates. Prior to
>> > that change we balanced to a cache domain and your workload didn't have to
>> > serialize multiple interrupts to a single core. My suggestion to you is to use
>> > the --policyscript option to make your storage irqs get balanced to the cache
>> > level, rather than the core level. That should return the behavior to what you
>> > want.
>> >
>> > Neil
>>
>> Hi Neil,
>>
>> Thanks for reply.
>>
>> Today I tried with setting balance_level to 'cache' for mpt3sas driver
>> IRQ's using below policy script and used 1.0.9 versioned irqbalance,
>> ----------------------------------------------------------------------------------------------
>> #!/bin/bash
>> # Header
>> # Linux Shell Scripting for Irq Balance Policy select for mpt3sas driver
>> #
>>
>> # Command Line Args
>> #IRQ_PATH -> PATH
>> #IRQ_NUMBER -> IRQ Number
>> declare -r IRQ_PATH=$1
>> declare -r IRQ_NUMBER=$2
>>
>> if [ -d /proc/irq/$IRQ_NUMBER ]; then
>> mpt3sas_irq=(`ls /proc/irq/$IRQ_NUMBER/ | grep mpt3sas | wc -l`)
>> if [ $mpt3sas_irq == 1 ]; then
>> echo "hintpolicy=subset"
>> echo "balance_level=cache"
>> fi
>> fi
>> -----------------------------------------------------------------------------------------------
>>
>> But still I don't see any load shift happening between the CPUs and
>> still observing hardlockups.
>>
>> Here I have attached the irqbalance logs.
>>
>> Thanks,
>> Sreekanth
>
> Hey there-
> So, looking at your logs, your script is working correctly:
> Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
> Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
> CPU number 0 numa_node is 0 (load 0)
> Interrupt 150 node_num is 0 (storage/1)
> Interrupt 174 node_num is 0 (storage/1)
> Interrupt 198 node_num is 0 (storage/1)
> Interrupt 126 node_num is 0 (storage/1)
> Interrupt 102 node_num is 0 (ethernet/1)
> Interrupt 77 node_num is 0 (ethernet/1)
> CPU number 12 numa_node is 0 (load 0)
> Interrupt 138 node_num is 0 (storage/1)
> Interrupt 162 node_num is 0 (storage/1)
> Interrupt 186 node_num is 0 (storage/1)
> Interrupt 114 node_num is 0 (storage/1)
> Interrupt 90 node_num is 0 (ethernet/1)
> Interrupt 65 node_num is 0 (ethernet/1)
> Interrupt 51 node_num is -1 (storage/1)
> Interrupt 31 node_num is 0 (legacy/1)
> ...
> Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
> Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
> CPU number 6 numa_node is 0 (load 0)
> Interrupt 149 node_num is 0 (storage/1)
> Interrupt 173 node_num is 0 (storage/1)
> Interrupt 197 node_num is 0 (storage/1)
> Interrupt 125 node_num is 0 (storage/1)
> Interrupt 101 node_num is 0 (ethernet/1)
> Interrupt 76 node_num is 0 (ethernet/1)
> CPU number 18 numa_node is 0 (load 0)
> Interrupt 137 node_num is 0 (storage/1)
> Interrupt 161 node_num is 0 (storage/1)
> Interrupt 185 node_num is 0 (storage/1)
> Interrupt 113 node_num is 0 (storage/1)
> Interrupt 89 node_num is 0 (ethernet/1)
> Interrupt 64 node_num is 0 (ethernet/1)
> Interrupt 50 node_num is -1 (storage/1)
>
>
> irqbalance correctly decided to balance irqs 50 and 51 to the cache level, which
> is good. The only other thing I would check though is the affinity_hint those
> irqs are exporting. With an affinity hint set to subset, if the exported hint
> only intersects the cache domain cpu set at one cpu, you will still only get
> affinity for that one cpu. You may want to consider changing the hintpolicy for
> those interrupts to ignore, to ensure that you have affinity for two cpus.
Hi Neil,
I changed the hint policy to ignore for these IRQs but still I observe
only one CPU
is busy with interrupt processing and eventually I am observe softlockups.
Thanks,
Sreekanth
>
> Beyond that though, the kernel is in control of irq delivery. Normally the
> configured hardware delivery policy is to select the highest priority cpu that
> isn't already servicing an interrupt (to maximize cache hit rates). If the irq
> rate is sufficiently slow however, it will always hit the same cpu, because it
> isn't blocked by another interrupt.
>
> Best
> Neil
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-09-08 5:42 ` Sreekanth Reddy
@ 2016-09-08 13:39 ` Neil Horman
2016-09-12 8:18 ` Sreekanth Reddy
0 siblings, 1 reply; 16+ messages in thread
From: Neil Horman @ 2016-09-08 13:39 UTC (permalink / raw)
To: Sreekanth Reddy
Cc: Bart Van Assche, Elliott, Robert (Persistent Memory),
linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
On Thu, Sep 08, 2016 at 11:12:40AM +0530, Sreekanth Reddy wrote:
> On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> > On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote:
> >> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> >> > On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote:
> >> >> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche
> >> >> <bart.vanassche@sandisk.com> wrote:
> >> >> > On 09/01/2016 03:31 AM, Sreekanth Reddy wrote:
> >> >> >>
> >> >> >> I reduced the ISR workload by one third in-order to reduce the time
> >> >> >> that is spent per CPU in interrupt context, even then I am observing
> >> >> >> softlockups.
> >> >> >>
> >> >> >> As I mentioned before only same single CPU in the set of CPUs(enabled
> >> >> >> in affinity_hint) is busy with handling the interrupts from
> >> >> >> corresponding IRQx. I have done below experiment in driver to limit
> >> >> >> these softlockups/hardlockups. But I am not sure whether it is
> >> >> >> reasonable to do this in driver,
> >> >> >>
> >> >> >> Experiment:
> >> >> >> If the CPUx is continuously busy with handling the remote CPUs
> >> >> >> (enabled in the corresponding IRQ's affinity_hint) IO works by 1/4th
> >> >> >> of the HBA queue depth in the same ISR context then enable a flag
> >> >> >> called 'change_smp_affinity' for this IRQ. Also created a thread with
> >> >> >> will poll for this flag for every IRQ's (enabled by driver) for every
> >> >> >> second. If this thread see that this flag is enabled for any IRQ then
> >> >> >> it will write next CPU number from the CPUs enabled in the IRQ's
> >> >> >> affinity_hint to the IRQ's smp_affinity procfs attribute using
> >> >> >> 'call_usermodehelper()' API.
> >> >> >>
> >> >> >> This to make sure that interrupts are not processed by same single CPU
> >> >> >> all the time and to make the other CPUs to handle the interrupts if
> >> >> >> the current CPU is continuously busy with handling the other CPUs IO
> >> >> >> interrupts.
> >> >> >>
> >> >> >> For example consider a system which has 8 logical CPUs and one MSIx
> >> >> >> vector enabled (called IRQ 120) in driver, HBA queue depth as 8K.
> >> >> >> then IRQ's procfs attributes will be
> >> >> >> IRQ# 120, affinity_hint=0xff, smp_affinity=0x00
> >> >> >>
> >> >> >> After starting heavy IOs, we will observe that only CPU0 will be busy
> >> >> >> with handling the interrupts. This experiment driver will change the
> >> >> >> smp_affinity to next CPU number i.e. 0x01 (using cmd 'echo 0x01 >
> >> >> >> /proc/irq/120/smp_affinity', driver issue's this cmd using
> >> >> >> call_usermodehelper() API) if it observes that CPU0 is continuously
> >> >> >> processing more than 2K of IOs replies of other CPUs i.e from CPU1 to
> >> >> >> CPU7.
> >> >> >>
> >> >> >> Whether doing this kind of stuff in driver is ok?
> >> >> >
> >> >> >
> >> >> > Hello Sreekanth,
> >> >> >
> >> >> > To me this sounds like something that should be implemented in the I/O
> >> >> > chipset on the motherboard. If you have a look at the Intel Software
> >> >> > Developer Manuals then you will see that logical destination mode supports
> >> >> > round-robin interrupt delivery. However, the Linux kernel selects physical
> >> >> > destination mode on systems with more than eight logical CPUs (see also
> >> >> > arch/x86/kernel/apic/apic_flat_64.c).
> >> >> >
> >> >> > I'm not sure the maintainers of the interrupt subsystem would welcome code
> >> >> > that emulates round-robin interrupt delivery. So your best option is
> >> >> > probably to minimize the amount of work that is done in interrupt context
> >> >> > and to move as much work as possible out of interrupt context in such a way
> >> >> > that it can be spread over multiple CPU cores, e.g. by using
> >> >> > queue_work_on().
> >> >> >
> >> >> > Bart.
> >> >>
> >> >> Bart,
> >> >>
> >> >> Thanks a lot for providing lot of inputs and valuable information on this issue.
> >> >>
> >> >> Today I got one more observation. i.e. I am not observing any lockups
> >> >> if I use 1.0.4-6 versioned irqbalance.
> >> >> Since this versioned irqbalance is able to shift the load to other CPU
> >> >> when one CPU is heavily loaded.
> >> >>
> >> >
> >> > This isn't happening because irqbalance is no longer able to shift load between
> >> > cpus, its happening because of commit 996ee2cf7a4d10454de68ac4978adb5cf22850f8.
> >> > irqs with higher interrupt volumes sould be balanced to a specific cpu core,
> >> > rather than to a cache domain to maximize cpu-local cache hit rates. Prior to
> >> > that change we balanced to a cache domain and your workload didn't have to
> >> > serialize multiple interrupts to a single core. My suggestion to you is to use
> >> > the --policyscript option to make your storage irqs get balanced to the cache
> >> > level, rather than the core level. That should return the behavior to what you
> >> > want.
> >> >
> >> > Neil
> >>
> >> Hi Neil,
> >>
> >> Thanks for reply.
> >>
> >> Today I tried with setting balance_level to 'cache' for mpt3sas driver
> >> IRQ's using below policy script and used 1.0.9 versioned irqbalance,
> >> ----------------------------------------------------------------------------------------------
> >> #!/bin/bash
> >> # Header
> >> # Linux Shell Scripting for Irq Balance Policy select for mpt3sas driver
> >> #
> >>
> >> # Command Line Args
> >> #IRQ_PATH -> PATH
> >> #IRQ_NUMBER -> IRQ Number
> >> declare -r IRQ_PATH=$1
> >> declare -r IRQ_NUMBER=$2
> >>
> >> if [ -d /proc/irq/$IRQ_NUMBER ]; then
> >> mpt3sas_irq=(`ls /proc/irq/$IRQ_NUMBER/ | grep mpt3sas | wc -l`)
> >> if [ $mpt3sas_irq == 1 ]; then
> >> echo "hintpolicy=subset"
> >> echo "balance_level=cache"
> >> fi
> >> fi
> >> -----------------------------------------------------------------------------------------------
> >>
> >> But still I don't see any load shift happening between the CPUs and
> >> still observing hardlockups.
> >>
> >> Here I have attached the irqbalance logs.
> >>
> >> Thanks,
> >> Sreekanth
> >
> > Hey there-
> > So, looking at your logs, your script is working correctly:
> > Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
> > Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
> > CPU number 0 numa_node is 0 (load 0)
> > Interrupt 150 node_num is 0 (storage/1)
> > Interrupt 174 node_num is 0 (storage/1)
> > Interrupt 198 node_num is 0 (storage/1)
> > Interrupt 126 node_num is 0 (storage/1)
> > Interrupt 102 node_num is 0 (ethernet/1)
> > Interrupt 77 node_num is 0 (ethernet/1)
> > CPU number 12 numa_node is 0 (load 0)
> > Interrupt 138 node_num is 0 (storage/1)
> > Interrupt 162 node_num is 0 (storage/1)
> > Interrupt 186 node_num is 0 (storage/1)
> > Interrupt 114 node_num is 0 (storage/1)
> > Interrupt 90 node_num is 0 (ethernet/1)
> > Interrupt 65 node_num is 0 (ethernet/1)
> > Interrupt 51 node_num is -1 (storage/1)
> > Interrupt 31 node_num is 0 (legacy/1)
> > ...
> > Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
> > Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
> > CPU number 6 numa_node is 0 (load 0)
> > Interrupt 149 node_num is 0 (storage/1)
> > Interrupt 173 node_num is 0 (storage/1)
> > Interrupt 197 node_num is 0 (storage/1)
> > Interrupt 125 node_num is 0 (storage/1)
> > Interrupt 101 node_num is 0 (ethernet/1)
> > Interrupt 76 node_num is 0 (ethernet/1)
> > CPU number 18 numa_node is 0 (load 0)
> > Interrupt 137 node_num is 0 (storage/1)
> > Interrupt 161 node_num is 0 (storage/1)
> > Interrupt 185 node_num is 0 (storage/1)
> > Interrupt 113 node_num is 0 (storage/1)
> > Interrupt 89 node_num is 0 (ethernet/1)
> > Interrupt 64 node_num is 0 (ethernet/1)
> > Interrupt 50 node_num is -1 (storage/1)
> >
> >
> > irqbalance correctly decided to balance irqs 50 and 51 to the cache level, which
> > is good. The only other thing I would check though is the affinity_hint those
> > irqs are exporting. With an affinity hint set to subset, if the exported hint
> > only intersects the cache domain cpu set at one cpu, you will still only get
> > affinity for that one cpu. You may want to consider changing the hintpolicy for
> > those interrupts to ignore, to ensure that you have affinity for two cpus.
>
> Hi Neil,
>
> I changed the hint policy to ignore for these IRQs but still I observe
> only one CPU
> is busy with interrupt processing and eventually I am observe softlockups.
>
> Thanks,
> Sreekanth
>
Then it seems that something else is going on. If you cat
/proc/irq/50/smp_affinity to confirm that your affinity mask has at least 2 cpus
set, then the only reason you would be getting irqs processed on only one cpu is
because the highest priority cpu in hardware (usually the lowest numbered one),
is free to handle the irq every time its asserted.
Neil
> >
> > Beyond that though, the kernel is in control of irq delivery. Normally the
> > configured hardware delivery policy is to select the highest priority cpu that
> > isn't already servicing an interrupt (to maximize cache hit rates). If the irq
> > rate is sufficiently slow however, it will always hit the same cpu, because it
> > isn't blocked by another interrupt.
> >
> > Best
> > Neil
> >
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-09-08 13:39 ` Neil Horman
@ 2016-09-12 8:18 ` Sreekanth Reddy
2016-09-12 12:03 ` Neil Horman
0 siblings, 1 reply; 16+ messages in thread
From: Sreekanth Reddy @ 2016-09-12 8:18 UTC (permalink / raw)
To: Neil Horman
Cc: Bart Van Assche, Elliott, Robert (Persistent Memory),
linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
On Thu, Sep 8, 2016 at 7:09 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> On Thu, Sep 08, 2016 at 11:12:40AM +0530, Sreekanth Reddy wrote:
>> On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
>> > On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote:
>> >> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
>> >> > On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote:
>> >> >> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche
>> >> >> <bart.vanassche@sandisk.com> wrote:
>> >> >> > On 09/01/2016 03:31 AM, Sreekanth Reddy wrote:
>> >> >> >>
>> >> >> >> I reduced the ISR workload by one third in-order to reduce the time
>> >> >> >> that is spent per CPU in interrupt context, even then I am observing
>> >> >> >> softlockups.
>> >> >> >>
>> >> >> >> As I mentioned before only same single CPU in the set of CPUs(enabled
>> >> >> >> in affinity_hint) is busy with handling the interrupts from
>> >> >> >> corresponding IRQx. I have done below experiment in driver to limit
>> >> >> >> these softlockups/hardlockups. But I am not sure whether it is
>> >> >> >> reasonable to do this in driver,
>> >> >> >>
>> >> >> >> Experiment:
>> >> >> >> If the CPUx is continuously busy with handling the remote CPUs
>> >> >> >> (enabled in the corresponding IRQ's affinity_hint) IO works by 1/4th
>> >> >> >> of the HBA queue depth in the same ISR context then enable a flag
>> >> >> >> called 'change_smp_affinity' for this IRQ. Also created a thread with
>> >> >> >> will poll for this flag for every IRQ's (enabled by driver) for every
>> >> >> >> second. If this thread see that this flag is enabled for any IRQ then
>> >> >> >> it will write next CPU number from the CPUs enabled in the IRQ's
>> >> >> >> affinity_hint to the IRQ's smp_affinity procfs attribute using
>> >> >> >> 'call_usermodehelper()' API.
>> >> >> >>
>> >> >> >> This to make sure that interrupts are not processed by same single CPU
>> >> >> >> all the time and to make the other CPUs to handle the interrupts if
>> >> >> >> the current CPU is continuously busy with handling the other CPUs IO
>> >> >> >> interrupts.
>> >> >> >>
>> >> >> >> For example consider a system which has 8 logical CPUs and one MSIx
>> >> >> >> vector enabled (called IRQ 120) in driver, HBA queue depth as 8K.
>> >> >> >> then IRQ's procfs attributes will be
>> >> >> >> IRQ# 120, affinity_hint=0xff, smp_affinity=0x00
>> >> >> >>
>> >> >> >> After starting heavy IOs, we will observe that only CPU0 will be busy
>> >> >> >> with handling the interrupts. This experiment driver will change the
>> >> >> >> smp_affinity to next CPU number i.e. 0x01 (using cmd 'echo 0x01 >
>> >> >> >> /proc/irq/120/smp_affinity', driver issue's this cmd using
>> >> >> >> call_usermodehelper() API) if it observes that CPU0 is continuously
>> >> >> >> processing more than 2K of IOs replies of other CPUs i.e from CPU1 to
>> >> >> >> CPU7.
>> >> >> >>
>> >> >> >> Whether doing this kind of stuff in driver is ok?
>> >> >> >
>> >> >> >
>> >> >> > Hello Sreekanth,
>> >> >> >
>> >> >> > To me this sounds like something that should be implemented in the I/O
>> >> >> > chipset on the motherboard. If you have a look at the Intel Software
>> >> >> > Developer Manuals then you will see that logical destination mode supports
>> >> >> > round-robin interrupt delivery. However, the Linux kernel selects physical
>> >> >> > destination mode on systems with more than eight logical CPUs (see also
>> >> >> > arch/x86/kernel/apic/apic_flat_64.c).
>> >> >> >
>> >> >> > I'm not sure the maintainers of the interrupt subsystem would welcome code
>> >> >> > that emulates round-robin interrupt delivery. So your best option is
>> >> >> > probably to minimize the amount of work that is done in interrupt context
>> >> >> > and to move as much work as possible out of interrupt context in such a way
>> >> >> > that it can be spread over multiple CPU cores, e.g. by using
>> >> >> > queue_work_on().
>> >> >> >
>> >> >> > Bart.
>> >> >>
>> >> >> Bart,
>> >> >>
>> >> >> Thanks a lot for providing lot of inputs and valuable information on this issue.
>> >> >>
>> >> >> Today I got one more observation. i.e. I am not observing any lockups
>> >> >> if I use 1.0.4-6 versioned irqbalance.
>> >> >> Since this versioned irqbalance is able to shift the load to other CPU
>> >> >> when one CPU is heavily loaded.
>> >> >>
>> >> >
>> >> > This isn't happening because irqbalance is no longer able to shift load between
>> >> > cpus, its happening because of commit 996ee2cf7a4d10454de68ac4978adb5cf22850f8.
>> >> > irqs with higher interrupt volumes sould be balanced to a specific cpu core,
>> >> > rather than to a cache domain to maximize cpu-local cache hit rates. Prior to
>> >> > that change we balanced to a cache domain and your workload didn't have to
>> >> > serialize multiple interrupts to a single core. My suggestion to you is to use
>> >> > the --policyscript option to make your storage irqs get balanced to the cache
>> >> > level, rather than the core level. That should return the behavior to what you
>> >> > want.
>> >> >
>> >> > Neil
>> >>
>> >> Hi Neil,
>> >>
>> >> Thanks for reply.
>> >>
>> >> Today I tried with setting balance_level to 'cache' for mpt3sas driver
>> >> IRQ's using below policy script and used 1.0.9 versioned irqbalance,
>> >> ----------------------------------------------------------------------------------------------
>> >> #!/bin/bash
>> >> # Header
>> >> # Linux Shell Scripting for Irq Balance Policy select for mpt3sas driver
>> >> #
>> >>
>> >> # Command Line Args
>> >> #IRQ_PATH -> PATH
>> >> #IRQ_NUMBER -> IRQ Number
>> >> declare -r IRQ_PATH=$1
>> >> declare -r IRQ_NUMBER=$2
>> >>
>> >> if [ -d /proc/irq/$IRQ_NUMBER ]; then
>> >> mpt3sas_irq=(`ls /proc/irq/$IRQ_NUMBER/ | grep mpt3sas | wc -l`)
>> >> if [ $mpt3sas_irq == 1 ]; then
>> >> echo "hintpolicy=subset"
>> >> echo "balance_level=cache"
>> >> fi
>> >> fi
>> >> -----------------------------------------------------------------------------------------------
>> >>
>> >> But still I don't see any load shift happening between the CPUs and
>> >> still observing hardlockups.
>> >>
>> >> Here I have attached the irqbalance logs.
>> >>
>> >> Thanks,
>> >> Sreekanth
>> >
>> > Hey there-
>> > So, looking at your logs, your script is working correctly:
>> > Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
>> > Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
>> > CPU number 0 numa_node is 0 (load 0)
>> > Interrupt 150 node_num is 0 (storage/1)
>> > Interrupt 174 node_num is 0 (storage/1)
>> > Interrupt 198 node_num is 0 (storage/1)
>> > Interrupt 126 node_num is 0 (storage/1)
>> > Interrupt 102 node_num is 0 (ethernet/1)
>> > Interrupt 77 node_num is 0 (ethernet/1)
>> > CPU number 12 numa_node is 0 (load 0)
>> > Interrupt 138 node_num is 0 (storage/1)
>> > Interrupt 162 node_num is 0 (storage/1)
>> > Interrupt 186 node_num is 0 (storage/1)
>> > Interrupt 114 node_num is 0 (storage/1)
>> > Interrupt 90 node_num is 0 (ethernet/1)
>> > Interrupt 65 node_num is 0 (ethernet/1)
>> > Interrupt 51 node_num is -1 (storage/1)
>> > Interrupt 31 node_num is 0 (legacy/1)
>> > ...
>> > Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
>> > Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
>> > CPU number 6 numa_node is 0 (load 0)
>> > Interrupt 149 node_num is 0 (storage/1)
>> > Interrupt 173 node_num is 0 (storage/1)
>> > Interrupt 197 node_num is 0 (storage/1)
>> > Interrupt 125 node_num is 0 (storage/1)
>> > Interrupt 101 node_num is 0 (ethernet/1)
>> > Interrupt 76 node_num is 0 (ethernet/1)
>> > CPU number 18 numa_node is 0 (load 0)
>> > Interrupt 137 node_num is 0 (storage/1)
>> > Interrupt 161 node_num is 0 (storage/1)
>> > Interrupt 185 node_num is 0 (storage/1)
>> > Interrupt 113 node_num is 0 (storage/1)
>> > Interrupt 89 node_num is 0 (ethernet/1)
>> > Interrupt 64 node_num is 0 (ethernet/1)
>> > Interrupt 50 node_num is -1 (storage/1)
>> >
>> >
>> > irqbalance correctly decided to balance irqs 50 and 51 to the cache level, which
>> > is good. The only other thing I would check though is the affinity_hint those
>> > irqs are exporting. With an affinity hint set to subset, if the exported hint
>> > only intersects the cache domain cpu set at one cpu, you will still only get
>> > affinity for that one cpu. You may want to consider changing the hintpolicy for
>> > those interrupts to ignore, to ensure that you have affinity for two cpus.
>>
>> Hi Neil,
>>
>> I changed the hint policy to ignore for these IRQs but still I observe
>> only one CPU
>> is busy with interrupt processing and eventually I am observe softlockups.
>>
>> Thanks,
>> Sreekanth
>>
>
> Then it seems that something else is going on. If you cat
> /proc/irq/50/smp_affinity to confirm that your affinity mask has at least 2 cpus
> set, then the only reason you would be getting irqs processed on only one cpu is
> because the highest priority cpu in hardware (usually the lowest numbered one),
> is free to handle the irq every time its asserted.
Yes Neil, I have observed that two CPU's are enabled in smp_affinity for IRQ#50
through 'cat /proc/irq/50/smp_affinity' command output.
Thanks,
Sreekanth
>
> Neil
>
>> >
>> > Beyond that though, the kernel is in control of irq delivery. Normally the
>> > configured hardware delivery policy is to select the highest priority cpu that
>> > isn't already servicing an interrupt (to maximize cache hit rates). If the irq
>> > rate is sufficiently slow however, it will always hit the same cpu, because it
>> > isn't blocked by another interrupt.
>> >
>> > Best
>> > Neil
>> >
>>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Observing Softlockup's while running heavy IOs
2016-09-12 8:18 ` Sreekanth Reddy
@ 2016-09-12 12:03 ` Neil Horman
0 siblings, 0 replies; 16+ messages in thread
From: Neil Horman @ 2016-09-12 12:03 UTC (permalink / raw)
To: Sreekanth Reddy
Cc: Bart Van Assche, Elliott, Robert (Persistent Memory),
linux-scsi, linux-kernel, irqbalance, Kashyap Desai,
Sathya Prakash Veerichetty, Chaitra Basappa,
Suganath Prabu Subramani
On Mon, Sep 12, 2016 at 01:48:39PM +0530, Sreekanth Reddy wrote:
> On Thu, Sep 8, 2016 at 7:09 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> > On Thu, Sep 08, 2016 at 11:12:40AM +0530, Sreekanth Reddy wrote:
> >> On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> >> > On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote:
> >> >> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> >> >> > On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote:
> >> >> >> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche
> >> >> >> <bart.vanassche@sandisk.com> wrote:
> >> >> >> > On 09/01/2016 03:31 AM, Sreekanth Reddy wrote:
> >> >> >> >>
> >> >> >> >> I reduced the ISR workload by one third in-order to reduce the time
> >> >> >> >> that is spent per CPU in interrupt context, even then I am observing
> >> >> >> >> softlockups.
> >> >> >> >>
> >> >> >> >> As I mentioned before only same single CPU in the set of CPUs(enabled
> >> >> >> >> in affinity_hint) is busy with handling the interrupts from
> >> >> >> >> corresponding IRQx. I have done below experiment in driver to limit
> >> >> >> >> these softlockups/hardlockups. But I am not sure whether it is
> >> >> >> >> reasonable to do this in driver,
> >> >> >> >>
> >> >> >> >> Experiment:
> >> >> >> >> If the CPUx is continuously busy with handling the remote CPUs
> >> >> >> >> (enabled in the corresponding IRQ's affinity_hint) IO works by 1/4th
> >> >> >> >> of the HBA queue depth in the same ISR context then enable a flag
> >> >> >> >> called 'change_smp_affinity' for this IRQ. Also created a thread with
> >> >> >> >> will poll for this flag for every IRQ's (enabled by driver) for every
> >> >> >> >> second. If this thread see that this flag is enabled for any IRQ then
> >> >> >> >> it will write next CPU number from the CPUs enabled in the IRQ's
> >> >> >> >> affinity_hint to the IRQ's smp_affinity procfs attribute using
> >> >> >> >> 'call_usermodehelper()' API.
> >> >> >> >>
> >> >> >> >> This to make sure that interrupts are not processed by same single CPU
> >> >> >> >> all the time and to make the other CPUs to handle the interrupts if
> >> >> >> >> the current CPU is continuously busy with handling the other CPUs IO
> >> >> >> >> interrupts.
> >> >> >> >>
> >> >> >> >> For example consider a system which has 8 logical CPUs and one MSIx
> >> >> >> >> vector enabled (called IRQ 120) in driver, HBA queue depth as 8K.
> >> >> >> >> then IRQ's procfs attributes will be
> >> >> >> >> IRQ# 120, affinity_hint=0xff, smp_affinity=0x00
> >> >> >> >>
> >> >> >> >> After starting heavy IOs, we will observe that only CPU0 will be busy
> >> >> >> >> with handling the interrupts. This experiment driver will change the
> >> >> >> >> smp_affinity to next CPU number i.e. 0x01 (using cmd 'echo 0x01 >
> >> >> >> >> /proc/irq/120/smp_affinity', driver issue's this cmd using
> >> >> >> >> call_usermodehelper() API) if it observes that CPU0 is continuously
> >> >> >> >> processing more than 2K of IOs replies of other CPUs i.e from CPU1 to
> >> >> >> >> CPU7.
> >> >> >> >>
> >> >> >> >> Whether doing this kind of stuff in driver is ok?
> >> >> >> >
> >> >> >> >
> >> >> >> > Hello Sreekanth,
> >> >> >> >
> >> >> >> > To me this sounds like something that should be implemented in the I/O
> >> >> >> > chipset on the motherboard. If you have a look at the Intel Software
> >> >> >> > Developer Manuals then you will see that logical destination mode supports
> >> >> >> > round-robin interrupt delivery. However, the Linux kernel selects physical
> >> >> >> > destination mode on systems with more than eight logical CPUs (see also
> >> >> >> > arch/x86/kernel/apic/apic_flat_64.c).
> >> >> >> >
> >> >> >> > I'm not sure the maintainers of the interrupt subsystem would welcome code
> >> >> >> > that emulates round-robin interrupt delivery. So your best option is
> >> >> >> > probably to minimize the amount of work that is done in interrupt context
> >> >> >> > and to move as much work as possible out of interrupt context in such a way
> >> >> >> > that it can be spread over multiple CPU cores, e.g. by using
> >> >> >> > queue_work_on().
> >> >> >> >
> >> >> >> > Bart.
> >> >> >>
> >> >> >> Bart,
> >> >> >>
> >> >> >> Thanks a lot for providing lot of inputs and valuable information on this issue.
> >> >> >>
> >> >> >> Today I got one more observation. i.e. I am not observing any lockups
> >> >> >> if I use 1.0.4-6 versioned irqbalance.
> >> >> >> Since this versioned irqbalance is able to shift the load to other CPU
> >> >> >> when one CPU is heavily loaded.
> >> >> >>
> >> >> >
> >> >> > This isn't happening because irqbalance is no longer able to shift load between
> >> >> > cpus, its happening because of commit 996ee2cf7a4d10454de68ac4978adb5cf22850f8.
> >> >> > irqs with higher interrupt volumes sould be balanced to a specific cpu core,
> >> >> > rather than to a cache domain to maximize cpu-local cache hit rates. Prior to
> >> >> > that change we balanced to a cache domain and your workload didn't have to
> >> >> > serialize multiple interrupts to a single core. My suggestion to you is to use
> >> >> > the --policyscript option to make your storage irqs get balanced to the cache
> >> >> > level, rather than the core level. That should return the behavior to what you
> >> >> > want.
> >> >> >
> >> >> > Neil
> >> >>
> >> >> Hi Neil,
> >> >>
> >> >> Thanks for reply.
> >> >>
> >> >> Today I tried with setting balance_level to 'cache' for mpt3sas driver
> >> >> IRQ's using below policy script and used 1.0.9 versioned irqbalance,
> >> >> ----------------------------------------------------------------------------------------------
> >> >> #!/bin/bash
> >> >> # Header
> >> >> # Linux Shell Scripting for Irq Balance Policy select for mpt3sas driver
> >> >> #
> >> >>
> >> >> # Command Line Args
> >> >> #IRQ_PATH -> PATH
> >> >> #IRQ_NUMBER -> IRQ Number
> >> >> declare -r IRQ_PATH=$1
> >> >> declare -r IRQ_NUMBER=$2
> >> >>
> >> >> if [ -d /proc/irq/$IRQ_NUMBER ]; then
> >> >> mpt3sas_irq=(`ls /proc/irq/$IRQ_NUMBER/ | grep mpt3sas | wc -l`)
> >> >> if [ $mpt3sas_irq == 1 ]; then
> >> >> echo "hintpolicy=subset"
> >> >> echo "balance_level=cache"
> >> >> fi
> >> >> fi
> >> >> -----------------------------------------------------------------------------------------------
> >> >>
> >> >> But still I don't see any load shift happening between the CPUs and
> >> >> still observing hardlockups.
> >> >>
> >> >> Here I have attached the irqbalance logs.
> >> >>
> >> >> Thanks,
> >> >> Sreekanth
> >> >
> >> > Hey there-
> >> > So, looking at your logs, your script is working correctly:
> >> > Package 0: numa_node is 0 cpu mask is 0003f03f (load 0)
> >> > Cache domain 0: numa_node is 0 cpu mask is 00001001 (load 0)
> >> > CPU number 0 numa_node is 0 (load 0)
> >> > Interrupt 150 node_num is 0 (storage/1)
> >> > Interrupt 174 node_num is 0 (storage/1)
> >> > Interrupt 198 node_num is 0 (storage/1)
> >> > Interrupt 126 node_num is 0 (storage/1)
> >> > Interrupt 102 node_num is 0 (ethernet/1)
> >> > Interrupt 77 node_num is 0 (ethernet/1)
> >> > CPU number 12 numa_node is 0 (load 0)
> >> > Interrupt 138 node_num is 0 (storage/1)
> >> > Interrupt 162 node_num is 0 (storage/1)
> >> > Interrupt 186 node_num is 0 (storage/1)
> >> > Interrupt 114 node_num is 0 (storage/1)
> >> > Interrupt 90 node_num is 0 (ethernet/1)
> >> > Interrupt 65 node_num is 0 (ethernet/1)
> >> > Interrupt 51 node_num is -1 (storage/1)
> >> > Interrupt 31 node_num is 0 (legacy/1)
> >> > ...
> >> > Package 1: numa_node is 0 cpu mask is 00fc0fc0 (load 0)
> >> > Cache domain 6: numa_node is 0 cpu mask is 00040040 (load 0)
> >> > CPU number 6 numa_node is 0 (load 0)
> >> > Interrupt 149 node_num is 0 (storage/1)
> >> > Interrupt 173 node_num is 0 (storage/1)
> >> > Interrupt 197 node_num is 0 (storage/1)
> >> > Interrupt 125 node_num is 0 (storage/1)
> >> > Interrupt 101 node_num is 0 (ethernet/1)
> >> > Interrupt 76 node_num is 0 (ethernet/1)
> >> > CPU number 18 numa_node is 0 (load 0)
> >> > Interrupt 137 node_num is 0 (storage/1)
> >> > Interrupt 161 node_num is 0 (storage/1)
> >> > Interrupt 185 node_num is 0 (storage/1)
> >> > Interrupt 113 node_num is 0 (storage/1)
> >> > Interrupt 89 node_num is 0 (ethernet/1)
> >> > Interrupt 64 node_num is 0 (ethernet/1)
> >> > Interrupt 50 node_num is -1 (storage/1)
> >> >
> >> >
> >> > irqbalance correctly decided to balance irqs 50 and 51 to the cache level, which
> >> > is good. The only other thing I would check though is the affinity_hint those
> >> > irqs are exporting. With an affinity hint set to subset, if the exported hint
> >> > only intersects the cache domain cpu set at one cpu, you will still only get
> >> > affinity for that one cpu. You may want to consider changing the hintpolicy for
> >> > those interrupts to ignore, to ensure that you have affinity for two cpus.
> >>
> >> Hi Neil,
> >>
> >> I changed the hint policy to ignore for these IRQs but still I observe
> >> only one CPU
> >> is busy with interrupt processing and eventually I am observe softlockups.
> >>
> >> Thanks,
> >> Sreekanth
> >>
> >
> > Then it seems that something else is going on. If you cat
> > /proc/irq/50/smp_affinity to confirm that your affinity mask has at least 2 cpus
> > set, then the only reason you would be getting irqs processed on only one cpu is
> > because the highest priority cpu in hardware (usually the lowest numbered one),
> > is free to handle the irq every time its asserted.
>
> Yes Neil, I have observed that two CPU's are enabled in smp_affinity for IRQ#50
> through 'cat /proc/irq/50/smp_affinity' command output.
>
Ok, so that tells me that irqbalance is doing what it can, its opting to allow
two cpus to service irq 50. Which cpu gets selected is the purview of the
kernel choice of hardware configuration, which is usually lowest priority (which
is to say, that if the lower numbered cpu isn't servicing an interrupt, its the
cpu that will service this irq). Assuming that /proc/interrupts, shows all irqs
for this device are getting serviced by one cpu, we can infer that the cpu is
sufficiently idle in terms of irq processing demand, and that the performance
bottleneck is elsewhere.
Neil
> Thanks,
> Sreekanth
> >
> > Neil
> >
> >> >
> >> > Beyond that though, the kernel is in control of irq delivery. Normally the
> >> > configured hardware delivery policy is to select the highest priority cpu that
> >> > isn't already servicing an interrupt (to maximize cache hit rates). If the irq
> >> > rate is sufficiently slow however, it will always hit the same cpu, because it
> >> > isn't blocked by another interrupt.
> >> >
> >> > Best
> >> > Neil
> >> >
> >>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2016-09-12 12:03 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-18 5:55 Observing Softlockup's while running heavy IOs Sreekanth Reddy
2016-08-18 14:59 ` Bart Van Assche
2016-08-18 21:08 ` Elliott, Robert (Persistent Memory)
2016-08-19 11:44 ` Sreekanth Reddy
2016-08-19 15:56 ` Bart Van Assche
2016-09-01 10:31 ` Sreekanth Reddy
2016-09-01 23:04 ` Bart Van Assche
[not found] ` <CAK=zhgrLL22stCfwKdpJkN=PkxPVxL=K9RgpP1USEbg_xx5TEg@mail.gmail.com>
2016-09-06 15:06 ` Neil Horman
2016-09-07 6:00 ` Sreekanth Reddy
2016-09-07 13:24 ` Neil Horman
2016-09-08 5:42 ` Sreekanth Reddy
2016-09-08 13:39 ` Neil Horman
2016-09-12 8:18 ` Sreekanth Reddy
2016-09-12 12:03 ` Neil Horman
2016-08-19 21:27 ` Elliott, Robert (Persistent Memory)
2016-08-23 9:52 ` Kashyap Desai
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).