* rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
@ 2021-02-05 7:29 Paul Menzel
2021-02-05 17:12 ` Paul E. McKenney
0 siblings, 1 reply; 10+ messages in thread
From: Paul Menzel @ 2021-02-05 7:29 UTC (permalink / raw)
To: Darrick J. Wong, linux-xfs, Paul E. McKenney, Josh Triplett
Cc: rcu, it+linux-rcu, LKML
[-- Attachment #1: Type: text/plain, Size: 2737 bytes --]
Dear Linux folks,
On a Dell PowerEdge T630/0NT78X, BIOS 2.8.0 05/23/2018 with Linux
5.4.57, we twice saw a self-detected stall on a CPU (October 27th, 2020,
January 18th, 2021).
Both times, the workqueue is `xfs-conv/md0 xfs_end_io`.
```
[ 0.000000] Linux version 5.4.57.mx64.340
(root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC)) #1 SMP Tue
Aug 11 13:20:33 CEST 2020
[…]
[48962.981257] rcu: INFO: rcu_sched self-detected stall on CPU
[48962.987511] rcu: 4-....: (20999 ticks this GP)
idle=fe6/1/0x4000000000000002 softirq=3630188/3630188 fqs=4696
[48962.998805] (t=21017 jiffies g=14529009 q=32263)
[48963.004074] Task dump for CPU 4:
[48963.007689] kworker/4:2 R running task 0 25587 2
0x80004008
[48963.015591] Workqueue: xfs-conv/md0 xfs_end_io
[48963.020570] Call Trace:
[48963.023311] <IRQ>
[48963.025560] sched_show_task+0x11e/0x150
[48963.029957] rcu_dump_cpu_stacks+0x70/0xa0
[48963.034545] rcu_sched_clock_irq+0x502/0x770
[48963.039322] ? tick_sched_do_timer+0x60/0x60
[48963.044106] update_process_times+0x24/0x60
[48963.048791] tick_sched_timer+0x37/0x70
[48963.053089] __hrtimer_run_queues+0x11f/0x2b0
[48963.057960] ? recalibrate_cpu_khz+0x10/0x10
[48963.062744] hrtimer_interrupt+0xe5/0x240
[48963.067235] smp_apic_timer_interrupt+0x6f/0x130
[48963.072407] apic_timer_interrupt+0xf/0x20
[48963.076994] </IRQ>
[48963.079347] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
[48963.085491] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea
5e ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56
9d <c3> 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
[48963.106524] RSP: 0018:ffffc9000738fd40 EFLAGS: 00000202 ORIG_RAX:
ffffffffffffff13
[48963.115003] RAX: ffffffff82407588 RBX: ffffffff82407580 RCX:
ffffffff82407588
[48963.122994] RDX: ffffffff82407588 RSI: 0000000000000202 RDI:
ffffffff82407580
[48963.130989] RBP: 0000000000000202 R08: ffffffff8203ea00 R09:
0000000000000001
[48963.138982] R10: ffffc9000738fbb8 R11: 0000000000000001 R12:
ffffffff82407588
[48963.146976] R13: ffffea005e7ae600 R14: ffff8897b7e5a040 R15:
ffffea005e7ae600
[48963.154971] wake_up_page_bit+0xe0/0x100
[48963.159366] xfs_destroy_ioend+0xce/0x1c0
[48963.163857] xfs_end_ioend+0xcf/0x1a0
[48963.167958] xfs_end_io+0xa4/0xd0
[48963.171672] process_one_work+0x1e5/0x410
[48963.176163] worker_thread+0x2d/0x3c0
[48963.180265] ? cancel_delayed_work+0x90/0x90
[48963.185048] kthread+0x117/0x130
[48963.188663] ? kthread_create_worker_on_cpu+0x70/0x70
[48963.194321] ret_from_fork+0x35/0x40
```
As it’s just log level INFO, is there anything what should be done, or
was the system probably just “overloaded”?
Kind regards,
Paul
[-- Attachment #2: salvadorthegunzerker-linux-5.4.57-messages-dmesg.txt --]
[-- Type: text/plain, Size: 157686 bytes --]
[ 0.000000] Linux version 5.4.57.mx64.340 (root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC)) #1 SMP Tue Aug 11 13:20:33 CEST 2020
[ 0.000000] Command line: BOOT_IMAGE=/boot/bzImage.x86_64 root=LABEL=root ro crashkernel=256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000098fff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000000099000-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007a288fff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007a289000-0x000000007af0afff] reserved
[ 0.000000] BIOS-e820: [mem 0x000000007af0b000-0x000000007b93afff] ACPI NVS
[ 0.000000] BIOS-e820: [mem 0x000000007b93b000-0x000000007bab5fff] ACPI data
[ 0.000000] BIOS-e820: [mem 0x000000007bab6000-0x000000007bae8fff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007bae9000-0x000000007bafefff] ACPI data
[ 0.000000] BIOS-e820: [mem 0x000000007baff000-0x000000007bafffff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007bb00000-0x000000008fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000feda8000-0x00000000fedabfff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000ff310000-0x00000000ffffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000187fffffff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: Dell Inc. PowerEdge T630/0NT78X, BIOS 2.8.0 05/23/2018
[ 0.000000] tsc: Fast TSC calibration using PIT
[ 0.000000] tsc: Detected 1698.051 MHz processor
[ 0.003203] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.003205] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.003212] last_pfn = 0x1880000 max_arch_pfn = 0x400000000
[ 0.003217] MTRR default type: uncachable
[ 0.003218] MTRR fixed ranges enabled:
[ 0.003219] 00000-9FFFF write-back
[ 0.003220] A0000-BFFFF uncachable
[ 0.003221] C0000-FFFFF write-protect
[ 0.003222] MTRR variable ranges enabled:
[ 0.003224] 0 base 000000000000 mask 3FFF80000000 write-back
[ 0.003225] 1 base 000100000000 mask 3FFF00000000 write-back
[ 0.003226] 2 base 000200000000 mask 3FFE00000000 write-back
[ 0.003227] 3 base 000400000000 mask 3FFC00000000 write-back
[ 0.003228] 4 base 000800000000 mask 3FF800000000 write-back
[ 0.003229] 5 base 001000000000 mask 3FF800000000 write-back
[ 0.003230] 6 base 001800000000 mask 3FFF80000000 write-back
[ 0.003231] 7 base 0000FF000000 mask 3FFFFF000000 write-protect
[ 0.003232] 8 disabled
[ 0.003233] 9 disabled
[ 0.003600] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.004029] last_pfn = 0x7bb00 max_arch_pfn = 0x400000000
[ 0.014673] Using GB pages for direct mapping
[ 0.015067] RAMDISK: [mem 0x375e7000-0x37aeafff]
[ 0.015072] ACPI: Early table checksum verification disabled
[ 0.015075] ACPI: RSDP 0x00000000000FE320 000024 (v02 DELL )
[ 0.015080] ACPI: XSDT 0x000000007BAB40E8 0000BC (v01 DELL PE_SC3 00000000 01000013)
[ 0.015086] ACPI: FACP 0x000000007BAB0000 0000F4 (v04 DELL PE_SC3 00000000 DELL 00000001)
[ 0.015093] ACPI: DSDT 0x000000007BA98000 010777 (v02 DELL PE_SC3 00000003 DELL 00000001)
[ 0.015096] ACPI: FACS 0x000000007B91B000 000040
[ 0.015100] ACPI: MCEJ 0x000000007BAB3000 000130 (v01 INTEL 00000001 INTL 0100000D)
[ 0.015103] ACPI: WD__ 0x000000007BAB2000 000134 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.015107] ACPI: SLIC 0x000000007BAB1000 000024 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.015110] ACPI: HPET 0x000000007BAAF000 000038 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.015114] ACPI: APIC 0x000000007BAAE000 000AFC (v02 DELL PE_SC3 00000000 DELL 00000001)
[ 0.015117] ACPI: MCFG 0x000000007BAAD000 00003C (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.015121] ACPI: MSCT 0x000000007BAAC000 000090 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.015124] ACPI: SLIT 0x000000007BAAB000 00006C (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.015128] ACPI: SRAT 0x000000007BAA9000 001130 (v03 DELL PE_SC3 00000001 DELL 00000001)
[ 0.015131] ACPI: SSDT 0x000000007B955000 1424A9 (v02 DELL PE_SC3 00004000 INTL 20121114)
[ 0.015135] ACPI: SSDT 0x000000007B952000 00217F (v02 DELL PE_SC3 00000002 INTL 20121114)
[ 0.015138] ACPI: SSDT 0x000000007B951000 00006E (v02 DELL PE_SC3 00000002 INTL 20121114)
[ 0.015142] ACPI: PRAD 0x000000007B950000 000132 (v02 DELL PE_SC3 00000002 INTL 20121114)
[ 0.015145] ACPI: DMAR 0x000000007BAFE000 000110 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.015149] ACPI: HEST 0x000000007BAFD000 00017C (v01 DELL PE_SC3 00000002 DELL 00000001)
[ 0.015152] ACPI: BERT 0x000000007BAFC000 000030 (v01 DELL PE_SC3 00000002 DELL 00000001)
[ 0.015156] ACPI: ERST 0x000000007BAFB000 000230 (v01 DELL PE_SC3 00000002 DELL 00000001)
[ 0.015159] ACPI: EINJ 0x000000007BAFA000 000150 (v01 DELL PE_SC3 00000002 DELL 00000001)
[ 0.015168] ACPI: Local APIC address 0xfee00000
[ 0.015201] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[ 0.015202] SRAT: PXM 1 -> APIC 0x10 -> Node 1
[ 0.015203] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[ 0.015204] SRAT: PXM 1 -> APIC 0x12 -> Node 1
[ 0.015205] SRAT: PXM 0 -> APIC 0x04 -> Node 0
[ 0.015207] SRAT: PXM 1 -> APIC 0x14 -> Node 1
[ 0.015208] SRAT: PXM 0 -> APIC 0x06 -> Node 0
[ 0.015209] SRAT: PXM 1 -> APIC 0x16 -> Node 1
[ 0.015210] SRAT: PXM 0 -> APIC 0x08 -> Node 0
[ 0.015211] SRAT: PXM 1 -> APIC 0x18 -> Node 1
[ 0.015212] SRAT: PXM 0 -> APIC 0x0a -> Node 0
[ 0.015213] SRAT: PXM 1 -> APIC 0x1a -> Node 1
[ 0.015223] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0xc7fffffff]
[ 0.015225] ACPI: SRAT: Node 1 PXM 1 [mem 0xc80000000-0x187fffffff]
[ 0.015229] NUMA: Initialized distance table, cnt=2
[ 0.015234] NODE_DATA(0) allocated [mem 0xc7fffc000-0xc7fffffff]
[ 0.015239] NODE_DATA(1) allocated [mem 0x187fff9000-0x187fffcfff]
[ 0.015251] Reserving 256MB of memory at 1696MB for crashkernel (System RAM: 98210MB)
[ 0.015354] Zone ranges:
[ 0.015355] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.015357] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.015359] Normal [mem 0x0000000100000000-0x000000187fffffff]
[ 0.015360] Movable zone start for each node
[ 0.015361] Early memory node ranges
[ 0.015363] node 0: [mem 0x0000000000001000-0x0000000000098fff]
[ 0.015364] node 0: [mem 0x0000000000100000-0x000000007a288fff]
[ 0.015366] node 0: [mem 0x000000007bab6000-0x000000007bae8fff]
[ 0.015367] node 0: [mem 0x000000007baff000-0x000000007bafffff]
[ 0.015368] node 0: [mem 0x0000000100000000-0x0000000c7fffffff]
[ 0.015373] node 1: [mem 0x0000000c80000000-0x000000187fffffff]
[ 0.015750] Zeroed struct page in unavailable ranges: 23979 pages
[ 0.015752] Initmem setup node 0 [mem 0x0000000000001000-0x0000000c7fffffff]
[ 0.015754] On node 0 totalpages: 12558933
[ 0.015756] DMA zone: 64 pages used for memmap
[ 0.015756] DMA zone: 21 pages reserved
[ 0.015757] DMA zone: 3992 pages, LIFO batch:0
[ 0.015855] DMA32 zone: 7755 pages used for memmap
[ 0.015856] DMA32 zone: 496317 pages, LIFO batch:63
[ 0.027087] Normal zone: 188416 pages used for memmap
[ 0.027089] Normal zone: 12058624 pages, LIFO batch:63
[ 0.214482] Initmem setup node 1 [mem 0x0000000c80000000-0x000000187fffffff]
[ 0.214486] On node 1 totalpages: 12582912
[ 0.214487] Normal zone: 196608 pages used for memmap
[ 0.214488] Normal zone: 12582912 pages, LIFO batch:63
[ 0.447974] ACPI: PM-Timer IO Port: 0x408
[ 0.447977] ACPI: Local APIC address 0xfee00000
[ 0.447996] ACPI: LAPIC_NMI (acpi_id[0x00] high level lint[0x1])
[ 0.447997] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
[ 0.447998] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
[ 0.447999] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
[ 0.448001] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
[ 0.448002] ACPI: LAPIC_NMI (acpi_id[0x05] high level lint[0x1])
[ 0.448003] ACPI: LAPIC_NMI (acpi_id[0x06] high level lint[0x1])
[ 0.448004] ACPI: LAPIC_NMI (acpi_id[0x07] high level lint[0x1])
[ 0.448005] ACPI: LAPIC_NMI (acpi_id[0x08] high level lint[0x1])
[ 0.448006] ACPI: LAPIC_NMI (acpi_id[0x09] high level lint[0x1])
[ 0.448008] ACPI: LAPIC_NMI (acpi_id[0x0a] high level lint[0x1])
[ 0.448009] ACPI: LAPIC_NMI (acpi_id[0x0b] high level lint[0x1])
[ 0.448010] ACPI: LAPIC_NMI (acpi_id[0x0c] high level lint[0x1])
[ 0.448011] ACPI: LAPIC_NMI (acpi_id[0x0d] high level lint[0x1])
[ 0.448012] ACPI: LAPIC_NMI (acpi_id[0x0e] high level lint[0x1])
[ 0.448013] ACPI: LAPIC_NMI (acpi_id[0x0f] high level lint[0x1])
[ 0.448014] ACPI: LAPIC_NMI (acpi_id[0x10] high level lint[0x1])
[ 0.448015] ACPI: LAPIC_NMI (acpi_id[0x11] high level lint[0x1])
[ 0.448016] ACPI: LAPIC_NMI (acpi_id[0x12] high level lint[0x1])
[ 0.448017] ACPI: LAPIC_NMI (acpi_id[0x13] high level lint[0x1])
[ 0.448018] ACPI: LAPIC_NMI (acpi_id[0x14] high level lint[0x1])
[ 0.448019] ACPI: LAPIC_NMI (acpi_id[0x15] high level lint[0x1])
[ 0.448021] ACPI: LAPIC_NMI (acpi_id[0x16] high level lint[0x1])
[ 0.448022] ACPI: LAPIC_NMI (acpi_id[0x17] high level lint[0x1])
[ 0.448023] ACPI: LAPIC_NMI (acpi_id[0x18] high level lint[0x1])
[ 0.448024] ACPI: LAPIC_NMI (acpi_id[0x19] high level lint[0x1])
[ 0.448025] ACPI: LAPIC_NMI (acpi_id[0x1a] high level lint[0x1])
[ 0.448026] ACPI: LAPIC_NMI (acpi_id[0x1b] high level lint[0x1])
[ 0.448027] ACPI: LAPIC_NMI (acpi_id[0x1c] high level lint[0x1])
[ 0.448028] ACPI: LAPIC_NMI (acpi_id[0x1d] high level lint[0x1])
[ 0.448029] ACPI: LAPIC_NMI (acpi_id[0x1e] high level lint[0x1])
[ 0.448030] ACPI: LAPIC_NMI (acpi_id[0x1f] high level lint[0x1])
[ 0.448031] ACPI: LAPIC_NMI (acpi_id[0x20] high level lint[0x1])
[ 0.448033] ACPI: LAPIC_NMI (acpi_id[0x21] high level lint[0x1])
[ 0.448034] ACPI: LAPIC_NMI (acpi_id[0x22] high level lint[0x1])
[ 0.448035] ACPI: LAPIC_NMI (acpi_id[0x23] high level lint[0x1])
[ 0.448036] ACPI: LAPIC_NMI (acpi_id[0x24] high level lint[0x1])
[ 0.448037] ACPI: LAPIC_NMI (acpi_id[0x25] high level lint[0x1])
[ 0.448038] ACPI: LAPIC_NMI (acpi_id[0x26] high level lint[0x1])
[ 0.448039] ACPI: LAPIC_NMI (acpi_id[0x27] high level lint[0x1])
[ 0.448040] ACPI: LAPIC_NMI (acpi_id[0x28] high level lint[0x1])
[ 0.448041] ACPI: LAPIC_NMI (acpi_id[0x29] high level lint[0x1])
[ 0.448042] ACPI: LAPIC_NMI (acpi_id[0x2a] high level lint[0x1])
[ 0.448043] ACPI: LAPIC_NMI (acpi_id[0x2b] high level lint[0x1])
[ 0.448044] ACPI: LAPIC_NMI (acpi_id[0x2c] high level lint[0x1])
[ 0.448046] ACPI: LAPIC_NMI (acpi_id[0x2d] high level lint[0x1])
[ 0.448047] ACPI: LAPIC_NMI (acpi_id[0x2e] high level lint[0x1])
[ 0.448048] ACPI: LAPIC_NMI (acpi_id[0x2f] high level lint[0x1])
[ 0.448049] ACPI: LAPIC_NMI (acpi_id[0x30] high level lint[0x1])
[ 0.448050] ACPI: LAPIC_NMI (acpi_id[0x31] high level lint[0x1])
[ 0.448051] ACPI: LAPIC_NMI (acpi_id[0x32] high level lint[0x1])
[ 0.448052] ACPI: LAPIC_NMI (acpi_id[0x33] high level lint[0x1])
[ 0.448053] ACPI: LAPIC_NMI (acpi_id[0x34] high level lint[0x1])
[ 0.448054] ACPI: LAPIC_NMI (acpi_id[0x35] high level lint[0x1])
[ 0.448055] ACPI: LAPIC_NMI (acpi_id[0x36] high level lint[0x1])
[ 0.448056] ACPI: LAPIC_NMI (acpi_id[0x37] high level lint[0x1])
[ 0.448057] ACPI: LAPIC_NMI (acpi_id[0x38] high level lint[0x1])
[ 0.448058] ACPI: LAPIC_NMI (acpi_id[0x39] high level lint[0x1])
[ 0.448059] ACPI: LAPIC_NMI (acpi_id[0x3a] high level lint[0x1])
[ 0.448061] ACPI: LAPIC_NMI (acpi_id[0x3b] high level lint[0x1])
[ 0.448062] ACPI: LAPIC_NMI (acpi_id[0x3c] high level lint[0x1])
[ 0.448063] ACPI: LAPIC_NMI (acpi_id[0x3d] high level lint[0x1])
[ 0.448064] ACPI: LAPIC_NMI (acpi_id[0x3e] high level lint[0x1])
[ 0.448065] ACPI: LAPIC_NMI (acpi_id[0x3f] high level lint[0x1])
[ 0.448066] ACPI: LAPIC_NMI (acpi_id[0x40] high level lint[0x1])
[ 0.448067] ACPI: LAPIC_NMI (acpi_id[0x41] high level lint[0x1])
[ 0.448068] ACPI: LAPIC_NMI (acpi_id[0x42] high level lint[0x1])
[ 0.448069] ACPI: LAPIC_NMI (acpi_id[0x43] high level lint[0x1])
[ 0.448070] ACPI: LAPIC_NMI (acpi_id[0x44] high level lint[0x1])
[ 0.448071] ACPI: LAPIC_NMI (acpi_id[0x45] high level lint[0x1])
[ 0.448072] ACPI: LAPIC_NMI (acpi_id[0x46] high level lint[0x1])
[ 0.448073] ACPI: LAPIC_NMI (acpi_id[0x47] high level lint[0x1])
[ 0.448074] ACPI: LAPIC_NMI (acpi_id[0x48] high level lint[0x1])
[ 0.448076] ACPI: LAPIC_NMI (acpi_id[0x49] high level lint[0x1])
[ 0.448077] ACPI: LAPIC_NMI (acpi_id[0x4a] high level lint[0x1])
[ 0.448078] ACPI: LAPIC_NMI (acpi_id[0x4b] high level lint[0x1])
[ 0.448079] ACPI: LAPIC_NMI (acpi_id[0x4c] high level lint[0x1])
[ 0.448080] ACPI: LAPIC_NMI (acpi_id[0x4d] high level lint[0x1])
[ 0.448081] ACPI: LAPIC_NMI (acpi_id[0x4e] high level lint[0x1])
[ 0.448082] ACPI: LAPIC_NMI (acpi_id[0x4f] high level lint[0x1])
[ 0.448083] ACPI: LAPIC_NMI (acpi_id[0x50] high level lint[0x1])
[ 0.448084] ACPI: LAPIC_NMI (acpi_id[0x51] high level lint[0x1])
[ 0.448085] ACPI: LAPIC_NMI (acpi_id[0x52] high level lint[0x1])
[ 0.448086] ACPI: LAPIC_NMI (acpi_id[0x53] high level lint[0x1])
[ 0.448087] ACPI: LAPIC_NMI (acpi_id[0x54] high level lint[0x1])
[ 0.448089] ACPI: LAPIC_NMI (acpi_id[0x55] high level lint[0x1])
[ 0.448090] ACPI: LAPIC_NMI (acpi_id[0x56] high level lint[0x1])
[ 0.448091] ACPI: LAPIC_NMI (acpi_id[0x57] high level lint[0x1])
[ 0.448092] ACPI: LAPIC_NMI (acpi_id[0x58] high level lint[0x1])
[ 0.448093] ACPI: LAPIC_NMI (acpi_id[0x59] high level lint[0x1])
[ 0.448094] ACPI: LAPIC_NMI (acpi_id[0x5a] high level lint[0x1])
[ 0.448095] ACPI: LAPIC_NMI (acpi_id[0x5b] high level lint[0x1])
[ 0.448096] ACPI: LAPIC_NMI (acpi_id[0x5c] high level lint[0x1])
[ 0.448097] ACPI: LAPIC_NMI (acpi_id[0x5d] high level lint[0x1])
[ 0.448098] ACPI: LAPIC_NMI (acpi_id[0x5e] high level lint[0x1])
[ 0.448100] ACPI: LAPIC_NMI (acpi_id[0x5f] high level lint[0x1])
[ 0.448101] ACPI: LAPIC_NMI (acpi_id[0x60] high level lint[0x1])
[ 0.448102] ACPI: LAPIC_NMI (acpi_id[0x61] high level lint[0x1])
[ 0.448103] ACPI: LAPIC_NMI (acpi_id[0x62] high level lint[0x1])
[ 0.448104] ACPI: LAPIC_NMI (acpi_id[0x63] high level lint[0x1])
[ 0.448105] ACPI: LAPIC_NMI (acpi_id[0x64] high level lint[0x1])
[ 0.448106] ACPI: LAPIC_NMI (acpi_id[0x65] high level lint[0x1])
[ 0.448107] ACPI: LAPIC_NMI (acpi_id[0x66] high level lint[0x1])
[ 0.448108] ACPI: LAPIC_NMI (acpi_id[0x67] high level lint[0x1])
[ 0.448109] ACPI: LAPIC_NMI (acpi_id[0x68] high level lint[0x1])
[ 0.448110] ACPI: LAPIC_NMI (acpi_id[0x69] high level lint[0x1])
[ 0.448111] ACPI: LAPIC_NMI (acpi_id[0x6a] high level lint[0x1])
[ 0.448112] ACPI: LAPIC_NMI (acpi_id[0x6b] high level lint[0x1])
[ 0.448113] ACPI: LAPIC_NMI (acpi_id[0x6c] high level lint[0x1])
[ 0.448115] ACPI: LAPIC_NMI (acpi_id[0x6d] high level lint[0x1])
[ 0.448116] ACPI: LAPIC_NMI (acpi_id[0x6e] high level lint[0x1])
[ 0.448117] ACPI: LAPIC_NMI (acpi_id[0x6f] high level lint[0x1])
[ 0.448118] ACPI: LAPIC_NMI (acpi_id[0x70] high level lint[0x1])
[ 0.448119] ACPI: LAPIC_NMI (acpi_id[0x71] high level lint[0x1])
[ 0.448120] ACPI: LAPIC_NMI (acpi_id[0x72] high level lint[0x1])
[ 0.448121] ACPI: LAPIC_NMI (acpi_id[0x73] high level lint[0x1])
[ 0.448122] ACPI: LAPIC_NMI (acpi_id[0x74] high level lint[0x1])
[ 0.448123] ACPI: LAPIC_NMI (acpi_id[0x75] high level lint[0x1])
[ 0.448124] ACPI: LAPIC_NMI (acpi_id[0x76] high level lint[0x1])
[ 0.448125] ACPI: LAPIC_NMI (acpi_id[0x77] high level lint[0x1])
[ 0.448126] ACPI: LAPIC_NMI (acpi_id[0x78] high level lint[0x1])
[ 0.448127] ACPI: LAPIC_NMI (acpi_id[0x79] high level lint[0x1])
[ 0.448128] ACPI: LAPIC_NMI (acpi_id[0x7a] high level lint[0x1])
[ 0.448130] ACPI: LAPIC_NMI (acpi_id[0x7b] high level lint[0x1])
[ 0.448131] ACPI: LAPIC_NMI (acpi_id[0x7c] high level lint[0x1])
[ 0.448132] ACPI: LAPIC_NMI (acpi_id[0x7d] high level lint[0x1])
[ 0.448133] ACPI: LAPIC_NMI (acpi_id[0x7e] high level lint[0x1])
[ 0.448134] ACPI: LAPIC_NMI (acpi_id[0x7f] high level lint[0x1])
[ 0.448135] ACPI: LAPIC_NMI (acpi_id[0x80] high level lint[0x1])
[ 0.448136] ACPI: LAPIC_NMI (acpi_id[0x81] high level lint[0x1])
[ 0.448137] ACPI: LAPIC_NMI (acpi_id[0x82] high level lint[0x1])
[ 0.448138] ACPI: LAPIC_NMI (acpi_id[0x83] high level lint[0x1])
[ 0.448139] ACPI: LAPIC_NMI (acpi_id[0x84] high level lint[0x1])
[ 0.448140] ACPI: LAPIC_NMI (acpi_id[0x85] high level lint[0x1])
[ 0.448141] ACPI: LAPIC_NMI (acpi_id[0x86] high level lint[0x1])
[ 0.448142] ACPI: LAPIC_NMI (acpi_id[0x87] high level lint[0x1])
[ 0.448143] ACPI: LAPIC_NMI (acpi_id[0x88] high level lint[0x1])
[ 0.448145] ACPI: LAPIC_NMI (acpi_id[0x89] high level lint[0x1])
[ 0.448146] ACPI: LAPIC_NMI (acpi_id[0x8a] high level lint[0x1])
[ 0.448147] ACPI: LAPIC_NMI (acpi_id[0x8b] high level lint[0x1])
[ 0.448148] ACPI: LAPIC_NMI (acpi_id[0x8c] high level lint[0x1])
[ 0.448149] ACPI: LAPIC_NMI (acpi_id[0x8d] high level lint[0x1])
[ 0.448150] ACPI: LAPIC_NMI (acpi_id[0x8e] high level lint[0x1])
[ 0.448151] ACPI: LAPIC_NMI (acpi_id[0x8f] high level lint[0x1])
[ 0.448152] ACPI: LAPIC_NMI (acpi_id[0x90] high level lint[0x1])
[ 0.448153] ACPI: LAPIC_NMI (acpi_id[0x91] high level lint[0x1])
[ 0.448154] ACPI: LAPIC_NMI (acpi_id[0x92] high level lint[0x1])
[ 0.448155] ACPI: LAPIC_NMI (acpi_id[0x93] high level lint[0x1])
[ 0.448157] ACPI: LAPIC_NMI (acpi_id[0x94] high level lint[0x1])
[ 0.448158] ACPI: LAPIC_NMI (acpi_id[0x95] high level lint[0x1])
[ 0.448159] ACPI: LAPIC_NMI (acpi_id[0x96] high level lint[0x1])
[ 0.448160] ACPI: LAPIC_NMI (acpi_id[0x97] high level lint[0x1])
[ 0.448161] ACPI: LAPIC_NMI (acpi_id[0x98] high level lint[0x1])
[ 0.448162] ACPI: LAPIC_NMI (acpi_id[0x99] high level lint[0x1])
[ 0.448163] ACPI: LAPIC_NMI (acpi_id[0x9a] high level lint[0x1])
[ 0.448164] ACPI: LAPIC_NMI (acpi_id[0x9b] high level lint[0x1])
[ 0.448165] ACPI: LAPIC_NMI (acpi_id[0x9c] high level lint[0x1])
[ 0.448166] ACPI: LAPIC_NMI (acpi_id[0x9d] high level lint[0x1])
[ 0.448168] ACPI: LAPIC_NMI (acpi_id[0x9e] high level lint[0x1])
[ 0.448169] ACPI: LAPIC_NMI (acpi_id[0x9f] high level lint[0x1])
[ 0.448170] ACPI: LAPIC_NMI (acpi_id[0xa0] high level lint[0x1])
[ 0.448171] ACPI: LAPIC_NMI (acpi_id[0xa1] high level lint[0x1])
[ 0.448172] ACPI: LAPIC_NMI (acpi_id[0xa2] high level lint[0x1])
[ 0.448173] ACPI: LAPIC_NMI (acpi_id[0xa3] high level lint[0x1])
[ 0.448174] ACPI: LAPIC_NMI (acpi_id[0xa4] high level lint[0x1])
[ 0.448175] ACPI: LAPIC_NMI (acpi_id[0xa5] high level lint[0x1])
[ 0.448176] ACPI: LAPIC_NMI (acpi_id[0xa6] high level lint[0x1])
[ 0.448177] ACPI: LAPIC_NMI (acpi_id[0xa7] high level lint[0x1])
[ 0.448178] ACPI: LAPIC_NMI (acpi_id[0xa8] high level lint[0x1])
[ 0.448179] ACPI: LAPIC_NMI (acpi_id[0xa9] high level lint[0x1])
[ 0.448180] ACPI: LAPIC_NMI (acpi_id[0xaa] high level lint[0x1])
[ 0.448182] ACPI: LAPIC_NMI (acpi_id[0xab] high level lint[0x1])
[ 0.448183] ACPI: LAPIC_NMI (acpi_id[0xac] high level lint[0x1])
[ 0.448184] ACPI: LAPIC_NMI (acpi_id[0xad] high level lint[0x1])
[ 0.448185] ACPI: LAPIC_NMI (acpi_id[0xae] high level lint[0x1])
[ 0.448186] ACPI: LAPIC_NMI (acpi_id[0xaf] high level lint[0x1])
[ 0.448187] ACPI: LAPIC_NMI (acpi_id[0xb0] high level lint[0x1])
[ 0.448188] ACPI: LAPIC_NMI (acpi_id[0xb1] high level lint[0x1])
[ 0.448189] ACPI: LAPIC_NMI (acpi_id[0xb2] high level lint[0x1])
[ 0.448190] ACPI: LAPIC_NMI (acpi_id[0xb3] high level lint[0x1])
[ 0.448191] ACPI: LAPIC_NMI (acpi_id[0xb4] high level lint[0x1])
[ 0.448192] ACPI: LAPIC_NMI (acpi_id[0xb5] high level lint[0x1])
[ 0.448193] ACPI: LAPIC_NMI (acpi_id[0xb6] high level lint[0x1])
[ 0.448194] ACPI: LAPIC_NMI (acpi_id[0xb7] high level lint[0x1])
[ 0.448195] ACPI: LAPIC_NMI (acpi_id[0xb8] high level lint[0x1])
[ 0.448197] ACPI: LAPIC_NMI (acpi_id[0xb9] high level lint[0x1])
[ 0.448198] ACPI: LAPIC_NMI (acpi_id[0xba] high level lint[0x1])
[ 0.448199] ACPI: LAPIC_NMI (acpi_id[0xbb] high level lint[0x1])
[ 0.448200] ACPI: LAPIC_NMI (acpi_id[0xbc] high level lint[0x1])
[ 0.448201] ACPI: LAPIC_NMI (acpi_id[0xbd] high level lint[0x1])
[ 0.448202] ACPI: LAPIC_NMI (acpi_id[0xbe] high level lint[0x1])
[ 0.448203] ACPI: LAPIC_NMI (acpi_id[0xbf] high level lint[0x1])
[ 0.448215] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
[ 0.448219] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47
[ 0.448224] IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71
[ 0.448229] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.448231] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.448234] ACPI: IRQ0 used by override.
[ 0.448236] ACPI: IRQ9 used by override.
[ 0.448240] Using ACPI (MADT) for SMP configuration information
[ 0.448242] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[ 0.448246] TSC deadline timer available
[ 0.448248] smpboot: Allowing 12 CPUs, 0 hotplug CPUs
[ 0.448267] [mem 0x90000000-0xfeda7fff] available for PCI devices
[ 0.448271] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[ 0.609290] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:12 nr_node_ids:2
[ 0.610434] percpu: Embedded 51 pages/cpu s171160 r8192 d29544 u262144
[ 0.610443] pcpu-alloc: s171160 r8192 d29544 u262144 alloc=1*2097152
[ 0.610444] pcpu-alloc: [0] 00 02 04 06 08 10 -- -- [1] 01 03 05 07 09 11 -- --
[ 0.610474] Built 2 zonelists, mobility grouping on. Total pages: 24748981
[ 0.610476] Policy zone: Normal
[ 0.610478] Kernel command line: BOOT_IMAGE=/boot/bzImage.x86_64 root=LABEL=root ro crashkernel=256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on
[ 0.610615] audit: disabled (until reboot)
[ 0.610782] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.977604] Memory: 98627896K/100567380K available (14340K kernel code, 1490K rwdata, 3556K rodata, 1616K init, 3072K bss, 1939484K reserved, 0K cma-reserved)
[ 0.977769] Kernel/User page tables isolation: enabled
[ 0.977789] ftrace: allocating 40645 entries in 159 pages
[ 0.998007] rcu: Hierarchical RCU implementation.
[ 0.998009] rcu: RCU event tracing is enabled.
[ 0.998011] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=12.
[ 0.998012] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[ 0.998014] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=12
[ 1.001984] NR_IRQS: 16640, nr_irqs: 1336, preallocated irqs: 16
[ 1.002240] random: crng done (trusting CPU's manufacturer)
[ 1.007658] Console: colour VGA+ 80x25
[ 1.066449] printk: console [tty0] enabled
[ 3.226148] printk: console [ttyS0] enabled
[ 3.231006] ACPI: Core revision 20190816
[ 3.236963] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns
[ 3.247432] APIC: Switch to symmetric I/O mode setup
[ 3.253152] DMAR: Host address width 46
[ 3.257605] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[ 3.263716] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 3.272912] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[ 3.279028] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 3.288234] DMAR: RMRR base: 0x0000007ae07000 end: 0x0000007af06fff
[ 3.295405] DMAR: ATSR flags: 0x0
[ 3.299272] DMAR: ATSR flags: 0x0
[ 3.303148] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0
[ 3.310514] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1
[ 3.317782] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1
[ 3.325049] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[ 3.331248] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 3.331248] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 3.348690] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 3.354782] x2apic: IRQ remapping doesn't support X2APIC mode
[ 3.361372] Switched APIC routing to physical flat.
[ 3.369479] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 3.380433] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x1879f6264ed, max_idle_ns: 440795254511 ns
[ 3.392449] Calibrating delay loop (skipped), value calculated using timer frequency.. 3396.10 BogoMIPS (lpj=1698051)
[ 3.393449] pid_max: default: 32768 minimum: 301
[ 3.394526] LSM: Security Framework initializing
[ 3.410020] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, vmalloc)
[ 3.418219] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, vmalloc)
[ 3.418687] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc)
[ 3.419662] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc)
[ 3.420864] mce: CPU0: Thermal monitoring enabled (TM1)
[ 3.421480] process: using mwait in idle threads
[ 3.422451] Last level iTLB entries: 4KB 128, 2MB 8, 4MB 8
[ 3.423449] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[ 3.424450] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[ 3.425450] Spectre V2 : Mitigation: Full generic retpoline
[ 3.426449] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[ 3.427449] Spectre V2 : Enabling Restricted Speculation for firmware calls
[ 3.428450] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[ 3.429450] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
[ 3.430450] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
[ 3.431449] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[ 3.432781] Freeing SMP alternatives memory: 40K
[ 3.434550] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (family: 0x6, model: 0x4f, stepping: 0x1)
[ 3.435640] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver.
[ 3.436451] ... version: 3
[ 3.437449] ... bit width: 48
[ 3.438449] ... generic registers: 8
[ 3.439450] ... value mask: 0000ffffffffffff
[ 3.440450] ... max period: 00007fffffffffff
[ 3.441449] ... fixed-purpose events: 3
[ 3.442449] ... event mask: 00000007000000ff
[ 3.443505] rcu: Hierarchical SRCU implementation.
[ 3.445425] smp: Bringing up secondary CPUs ...
[ 3.445659] x86: Booting SMP configuration:
[ 3.446453] .... node #1, CPUs: #1
[ 2.367262] smpboot: CPU 1 Converting physical 0 to logical die 1
[ 3.519602] .... node #0, CPUs: #2
[ 3.521453] .... node #1, CPUs: #3
[ 3.523450] .... node #0, CPUs: #4
[ 3.525451] .... node #1, CPUs: #5
[ 3.527451] .... node #0, CPUs: #6
[ 3.529451] .... node #1, CPUs: #7
[ 3.531450] .... node #0, CPUs: #8
[ 3.533451] .... node #1, CPUs: #9
[ 3.535450] .... node #0, CPUs: #10
[ 3.537450] .... node #1, CPUs: #11
[ 3.538452] smp: Brought up 2 nodes, 12 CPUs
[ 3.540450] smpboot: Max logical packages: 2
[ 3.541451] smpboot: Total of 12 processors activated (40793.34 BogoMIPS)
[ 3.547489] devtmpfs: initialized
[ 3.551743] PM: Registering ACPI NVS region [mem 0x7af0b000-0x7b93afff] (10682368 bytes)
[ 3.561839] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[ 3.573467] futex hash table entries: 4096 (order: 6, 262144 bytes, vmalloc)
[ 3.582635] xor: automatically using best checksumming function avx
[ 3.590646] NET: Registered protocol family 16
[ 3.596759] cpuidle: using governor menu
[ 3.601539] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[ 3.611452] ACPI: bus type PCI registered
[ 3.615505] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
[ 3.627469] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
[ 3.635469] PCI: Using configuration type 1 for base access
[ 3.642461] PCI: Dell System detected, enabling pci=bfsort.
[ 3.649451] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[ 3.652505] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 3.678140] raid6: avx2x4 gen() 15753 MB/s
[ 3.700224] raid6: avx2x4 xor() 9248 MB/s
[ 3.722228] raid6: avx2x2 gen() 13410 MB/s
[ 3.744237] raid6: avx2x2 xor() 8101 MB/s
[ 3.766229] raid6: avx2x1 gen() 11738 MB/s
[ 3.788180] raid6: avx2x1 xor() 7566 MB/s
[ 3.810225] raid6: sse2x4 gen() 8546 MB/s
[ 3.832237] raid6: sse2x4 xor() 5003 MB/s
[ 3.854230] raid6: sse2x2 gen() 7128 MB/s
[ 3.876227] raid6: sse2x2 xor() 4509 MB/s
[ 3.898223] raid6: sse2x1 gen() 6121 MB/s
[ 3.920233] raid6: sse2x1 xor() 4021 MB/s
[ 3.925450] raid6: using algorithm avx2x4 gen() 15753 MB/s
[ 3.931450] raid6: .... xor() 9248 MB/s, rmw enabled
[ 3.937450] raid6: using avx2x2 recovery algorithm
[ 3.943568] ACPI: Added _OSI(Module Device)
[ 3.948453] ACPI: Added _OSI(Processor Device)
[ 3.954450] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 3.960450] ACPI: Added _OSI(Processor Aggregator Device)
[ 3.966452] ACPI: Added _OSI(Linux-Dell-Video)
[ 3.972452] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[ 3.978451] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[ 4.257718] ACPI: 4 ACPI AML tables successfully acquired and loaded
[ 4.296349] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[ 4.310951] ACPI: Interpreter enabled
[ 4.315465] ACPI: (supports S0 S5)
[ 4.319451] ACPI: Using IOAPIC for interrupt routing
[ 4.325480] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 4.337314] ACPI: Enabled 2 GPEs in block 00 to 3F
[ 4.363306] ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff])
[ 4.370456] acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[ 4.381834] acpi PNP0A03:02: _OSC: OS now controls [PME AER PCIeCapability LTR]
[ 4.390451] acpi PNP0A03:02: FADT indicates ASPM is unsupported, using BIOS configuration
[ 4.400500] PCI host bridge to bus 0000:ff
[ 4.406452] pci_bus 0000:ff: root bus resource [bus ff]
[ 4.412464] pci 0000:ff:08.0: [8086:6f80] type 00 class 0x088000
[ 4.419526] pci 0000:ff:08.2: [8086:6f32] type 00 class 0x110100
[ 4.426514] pci 0000:ff:08.3: [8086:6f83] type 00 class 0x088000
[ 4.434538] pci 0000:ff:09.0: [8086:6f90] type 00 class 0x088000
[ 4.441510] pci 0000:ff:09.2: [8086:6f33] type 00 class 0x110100
[ 4.448512] pci 0000:ff:09.3: [8086:6f93] type 00 class 0x088000
[ 4.455530] pci 0000:ff:0b.0: [8086:6f81] type 00 class 0x088000
[ 4.463505] pci 0000:ff:0b.1: [8086:6f36] type 00 class 0x110100
[ 4.470506] pci 0000:ff:0b.2: [8086:6f37] type 00 class 0x110100
[ 4.477507] pci 0000:ff:0b.3: [8086:6f76] type 00 class 0x088000
[ 4.484509] pci 0000:ff:0c.0: [8086:6fe0] type 00 class 0x088000
[ 4.491505] pci 0000:ff:0c.1: [8086:6fe1] type 00 class 0x088000
[ 4.499505] pci 0000:ff:0c.2: [8086:6fe2] type 00 class 0x088000
[ 4.506505] pci 0000:ff:0c.3: [8086:6fe3] type 00 class 0x088000
[ 4.513506] pci 0000:ff:0c.4: [8086:6fe4] type 00 class 0x088000
[ 4.520503] pci 0000:ff:0c.5: [8086:6fe5] type 00 class 0x088000
[ 4.528510] pci 0000:ff:0f.0: [8086:6ff8] type 00 class 0x088000
[ 4.535505] pci 0000:ff:0f.1: [8086:6ff9] type 00 class 0x088000
[ 4.542511] pci 0000:ff:0f.4: [8086:6ffc] type 00 class 0x088000
[ 4.549508] pci 0000:ff:0f.5: [8086:6ffd] type 00 class 0x088000
[ 4.556508] pci 0000:ff:0f.6: [8086:6ffe] type 00 class 0x088000
[ 4.564507] pci 0000:ff:10.0: [8086:6f1d] type 00 class 0x088000
[ 4.571509] pci 0000:ff:10.1: [8086:6f34] type 00 class 0x110100
[ 4.578509] pci 0000:ff:10.5: [8086:6f1e] type 00 class 0x088000
[ 4.585506] pci 0000:ff:10.6: [8086:6f7d] type 00 class 0x110100
[ 4.593508] pci 0000:ff:10.7: [8086:6f1f] type 00 class 0x088000
[ 4.600509] pci 0000:ff:12.0: [8086:6fa0] type 00 class 0x088000
[ 4.607489] pci 0000:ff:12.1: [8086:6f30] type 00 class 0x110100
[ 4.614508] pci 0000:ff:12.2: [8086:6f70] type 00 class 0x088000
[ 4.622514] pci 0000:ff:13.0: [8086:6fa8] type 00 class 0x088000
[ 4.629559] pci 0000:ff:13.1: [8086:6f71] type 00 class 0x088000
[ 4.636527] pci 0000:ff:13.2: [8086:6faa] type 00 class 0x088000
[ 4.643525] pci 0000:ff:13.3: [8086:6fab] type 00 class 0x088000
[ 4.651526] pci 0000:ff:13.4: [8086:6fac] type 00 class 0x088000
[ 4.658528] pci 0000:ff:13.5: [8086:6fad] type 00 class 0x088000
[ 4.665525] pci 0000:ff:13.6: [8086:6fae] type 00 class 0x088000
[ 4.672508] pci 0000:ff:13.7: [8086:6faf] type 00 class 0x088000
[ 4.680511] pci 0000:ff:14.0: [8086:6fb0] type 00 class 0x088000
[ 4.687527] pci 0000:ff:14.1: [8086:6fb1] type 00 class 0x088000
[ 4.694525] pci 0000:ff:14.2: [8086:6fb2] type 00 class 0x088000
[ 4.701525] pci 0000:ff:14.3: [8086:6fb3] type 00 class 0x088000
[ 4.709527] pci 0000:ff:14.4: [8086:6fbc] type 00 class 0x088000
[ 4.716510] pci 0000:ff:14.5: [8086:6fbd] type 00 class 0x088000
[ 4.723511] pci 0000:ff:14.6: [8086:6fbe] type 00 class 0x088000
[ 4.730510] pci 0000:ff:14.7: [8086:6fbf] type 00 class 0x088000
[ 4.737511] pci 0000:ff:15.0: [8086:6fb4] type 00 class 0x088000
[ 4.745531] pci 0000:ff:15.1: [8086:6fb5] type 00 class 0x088000
[ 4.752531] pci 0000:ff:15.2: [8086:6fb6] type 00 class 0x088000
[ 4.759530] pci 0000:ff:15.3: [8086:6fb7] type 00 class 0x088000
[ 4.766533] pci 0000:ff:16.0: [8086:6f68] type 00 class 0x088000
[ 4.774524] pci 0000:ff:16.6: [8086:6f6e] type 00 class 0x088000
[ 4.781510] pci 0000:ff:16.7: [8086:6f6f] type 00 class 0x088000
[ 4.788509] pci 0000:ff:17.0: [8086:6fd0] type 00 class 0x088000
[ 4.796520] pci 0000:ff:17.4: [8086:6fb8] type 00 class 0x088000
[ 4.803509] pci 0000:ff:17.5: [8086:6fb9] type 00 class 0x088000
[ 4.810508] pci 0000:ff:17.6: [8086:6fba] type 00 class 0x088000
[ 4.817508] pci 0000:ff:17.7: [8086:6fbb] type 00 class 0x088000
[ 4.824516] pci 0000:ff:1e.0: [8086:6f98] type 00 class 0x088000
[ 4.832507] pci 0000:ff:1e.1: [8086:6f99] type 00 class 0x088000
[ 4.839506] pci 0000:ff:1e.2: [8086:6f9a] type 00 class 0x088000
[ 4.846507] pci 0000:ff:1e.3: [8086:6fc0] type 00 class 0x088000
[ 4.853492] pci 0000:ff:1e.4: [8086:6f9c] type 00 class 0x088000
[ 4.861511] pci 0000:ff:1f.0: [8086:6f88] type 00 class 0x088000
[ 4.878509] pci 0000:ff:1f.2: [8086:6f8a] type 00 class 0x088000
[ 4.885606] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus 7f])
[ 4.892453] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[ 4.903878] acpi PNP0A03:03: _OSC: OS now controls [PME AER PCIeCapability LTR]
[ 4.913451] acpi PNP0A03:03: FADT indicates ASPM is unsupported, using BIOS configuration
[ 4.923496] PCI host bridge to bus 0000:7f
[ 4.928452] pci_bus 0000:7f: root bus resource [bus 7f]
[ 4.934463] pci 0000:7f:08.0: [8086:6f80] type 00 class 0x088000
[ 4.941514] pci 0000:7f:08.2: [8086:6f32] type 00 class 0x110100
[ 4.949513] pci 0000:7f:08.3: [8086:6f83] type 00 class 0x088000
[ 4.956529] pci 0000:7f:09.0: [8086:6f90] type 00 class 0x088000
[ 4.963509] pci 0000:7f:09.2: [8086:6f33] type 00 class 0x110100
[ 4.970511] pci 0000:7f:09.3: [8086:6f93] type 00 class 0x088000
[ 4.978526] pci 0000:7f:0b.0: [8086:6f81] type 00 class 0x088000
[ 4.985506] pci 0000:7f:0b.1: [8086:6f36] type 00 class 0x110100
[ 4.992506] pci 0000:7f:0b.2: [8086:6f37] type 00 class 0x110100
[ 4.999505] pci 0000:7f:0b.3: [8086:6f76] type 00 class 0x088000
[ 5.006506] pci 0000:7f:0c.0: [8086:6fe0] type 00 class 0x088000
[ 5.014508] pci 0000:7f:0c.1: [8086:6fe1] type 00 class 0x088000
[ 5.021507] pci 0000:7f:0c.2: [8086:6fe2] type 00 class 0x088000
[ 5.028507] pci 0000:7f:0c.3: [8086:6fe3] type 00 class 0x088000
[ 5.035505] pci 0000:7f:0c.4: [8086:6fe4] type 00 class 0x088000
[ 5.043503] pci 0000:7f:0c.5: [8086:6fe5] type 00 class 0x088000
[ 5.050508] pci 0000:7f:0f.0: [8086:6ff8] type 00 class 0x088000
[ 5.057503] pci 0000:7f:0f.1: [8086:6ff9] type 00 class 0x088000
[ 5.064506] pci 0000:7f:0f.4: [8086:6ffc] type 00 class 0x088000
[ 5.072505] pci 0000:7f:0f.5: [8086:6ffd] type 00 class 0x088000
[ 5.079504] pci 0000:7f:0f.6: [8086:6ffe] type 00 class 0x088000
[ 5.086506] pci 0000:7f:10.0: [8086:6f1d] type 00 class 0x088000
[ 5.093505] pci 0000:7f:10.1: [8086:6f34] type 00 class 0x110100
[ 5.100506] pci 0000:7f:10.5: [8086:6f1e] type 00 class 0x088000
[ 5.108505] pci 0000:7f:10.6: [8086:6f7d] type 00 class 0x110100
[ 5.115505] pci 0000:7f:10.7: [8086:6f1f] type 00 class 0x088000
[ 5.122533] pci 0000:7f:12.0: [8086:6fa0] type 00 class 0x088000
[ 5.129508] pci 0000:7f:12.1: [8086:6f30] type 00 class 0x110100
[ 5.137509] pci 0000:7f:12.2: [8086:6f70] type 00 class 0x088000
[ 5.144517] pci 0000:7f:13.0: [8086:6fa8] type 00 class 0x088000
[ 5.151559] pci 0000:7f:13.1: [8086:6f71] type 00 class 0x088000
[ 5.158523] pci 0000:7f:13.2: [8086:6faa] type 00 class 0x088000
[ 5.166527] pci 0000:7f:13.3: [8086:6fab] type 00 class 0x088000
[ 5.173525] pci 0000:7f:13.4: [8086:6fac] type 00 class 0x088000
[ 5.180526] pci 0000:7f:13.5: [8086:6fad] type 00 class 0x088000
[ 5.187518] pci 0000:7f:13.6: [8086:6fae] type 00 class 0x088000
[ 5.195508] pci 0000:7f:13.7: [8086:6faf] type 00 class 0x088000
[ 5.202509] pci 0000:7f:14.0: [8086:6fb0] type 00 class 0x088000
[ 5.209526] pci 0000:7f:14.1: [8086:6fb1] type 00 class 0x088000
[ 5.216525] pci 0000:7f:14.2: [8086:6fb2] type 00 class 0x088000
[ 5.224525] pci 0000:7f:14.3: [8086:6fb3] type 00 class 0x088000
[ 5.231521] pci 0000:7f:14.4: [8086:6fbc] type 00 class 0x088000
[ 5.238508] pci 0000:7f:14.5: [8086:6fbd] type 00 class 0x088000
[ 5.245507] pci 0000:7f:14.6: [8086:6fbe] type 00 class 0x088000
[ 5.253509] pci 0000:7f:14.7: [8086:6fbf] type 00 class 0x088000
[ 5.260508] pci 0000:7f:15.0: [8086:6fb4] type 00 class 0x088000
[ 5.267524] pci 0000:7f:15.1: [8086:6fb5] type 00 class 0x088000
[ 5.274525] pci 0000:7f:15.2: [8086:6fb6] type 00 class 0x088000
[ 5.282522] pci 0000:7f:15.3: [8086:6fb7] type 00 class 0x088000
[ 5.289528] pci 0000:7f:16.0: [8086:6f68] type 00 class 0x088000
[ 5.296518] pci 0000:7f:16.6: [8086:6f6e] type 00 class 0x088000
[ 5.303510] pci 0000:7f:16.7: [8086:6f6f] type 00 class 0x088000
[ 5.311510] pci 0000:7f:17.0: [8086:6fd0] type 00 class 0x088000
[ 5.318516] pci 0000:7f:17.4: [8086:6fb8] type 00 class 0x088000
[ 5.325508] pci 0000:7f:17.5: [8086:6fb9] type 00 class 0x088000
[ 5.332512] pci 0000:7f:17.6: [8086:6fba] type 00 class 0x088000
[ 5.339509] pci 0000:7f:17.7: [8086:6fbb] type 00 class 0x088000
[ 5.347516] pci 0000:7f:1e.0: [8086:6f98] type 00 class 0x088000
[ 5.354503] pci 0000:7f:1e.1: [8086:6f99] type 00 class 0x088000
[ 5.361508] pci 0000:7f:1e.2: [8086:6f9a] type 00 class 0x088000
[ 5.368504] pci 0000:7f:1e.3: [8086:6fc0] type 00 class 0x088000
[ 5.376493] pci 0000:7f:1e.4: [8086:6f9c] type 00 class 0x088000
[ 5.383511] pci 0000:7f:1f.0: [8086:6f88] type 00 class 0x088000
[ 5.390510] pci 0000:7f:1f.2: [8086:6f8a] type 00 class 0x088000
[ 5.399826] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
[ 5.407454] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[ 5.418697] acpi PNP0A08:00: _OSC: platform does not support [LTR]
[ 5.426677] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]
[ 5.434451] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
[ 5.444547] PCI host bridge to bus 0000:00
[ 5.449452] pci_bus 0000:00: root bus resource [io 0x0000-0x03bb window]
[ 5.457451] pci_bus 0000:00: root bus resource [io 0x03bc-0x03df window]
[ 5.465451] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window]
[ 5.473451] pci_bus 0000:00: root bus resource [io 0x1000-0x7fff window]
[ 5.481451] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[ 5.491451] pci_bus 0000:00: root bus resource [mem 0x90000000-0xc7ffbfff window]
[ 5.500451] pci_bus 0000:00: root bus resource [mem 0x38000000000-0x3bfffffffff window]
[ 5.510451] pci_bus 0000:00: root bus resource [bus 00-7e]
[ 5.516497] pci 0000:00:00.0: [8086:6f00] type 00 class 0x060000
[ 5.523676] pci 0000:00:01.0: [8086:6f02] type 01 class 0x060400
[ 5.531566] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[ 5.538687] pci 0000:00:02.0: [8086:6f04] type 01 class 0x060400
[ 5.546565] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
[ 5.553655] pci 0000:00:03.0: [8086:6f08] type 01 class 0x060400
[ 5.560563] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[ 5.568660] pci 0000:00:05.0: [8086:6f28] type 00 class 0x088000
[ 5.575629] pci 0000:00:05.1: [8086:6f29] type 00 class 0x088000
[ 5.583658] pci 0000:00:05.2: [8086:6f2a] type 00 class 0x088000
[ 5.590621] pci 0000:00:05.4: [8086:6f2c] type 00 class 0x080020
[ 5.597461] pci 0000:00:05.4: reg 0x10: [mem 0x92214000-0x92214fff]
[ 5.605638] pci 0000:00:11.0: [8086:8d7c] type 00 class 0xff0000
[ 5.612702] pci 0000:00:11.4: [8086:8d62] type 00 class 0x010601
[ 5.620470] pci 0000:00:11.4: reg 0x10: [io 0x4078-0x407f]
[ 5.626458] pci 0000:00:11.4: reg 0x14: [io 0x408c-0x408f]
[ 5.633458] pci 0000:00:11.4: reg 0x18: [io 0x4070-0x4077]
[ 5.640458] pci 0000:00:11.4: reg 0x1c: [io 0x4088-0x408b]
[ 5.646458] pci 0000:00:11.4: reg 0x20: [io 0x4040-0x405f]
[ 5.653458] pci 0000:00:11.4: reg 0x24: [mem 0x92211000-0x922117ff]
[ 5.661536] pci 0000:00:11.4: PME# supported from D3hot
[ 5.667554] pci 0000:00:14.0: [8086:8d31] type 00 class 0x0c0330
[ 5.674472] pci 0000:00:14.0: reg 0x10: [mem 0x92200000-0x9220ffff 64bit]
[ 5.682553] pci 0000:00:14.0: PME# supported from D3hot D3cold
[ 5.689573] pci 0000:00:16.0: [8086:8d3a] type 00 class 0x078000
[ 5.697472] pci 0000:00:16.0: reg 0x10: [mem 0x3bffff03000-0x3bffff0300f 64bit]
[ 5.706558] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
[ 5.713573] pci 0000:00:16.1: [8086:8d3b] type 00 class 0x078000
[ 5.721472] pci 0000:00:16.1: reg 0x10: [mem 0x3bffff02000-0x3bffff0200f 64bit]
[ 5.730556] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
[ 5.737581] pci 0000:00:1a.0: [8086:8d2d] type 00 class 0x0c0320
[ 5.744472] pci 0000:00:1a.0: reg 0x10: [mem 0x92213000-0x922133ff]
[ 5.752578] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
[ 5.759576] pci 0000:00:1c.0: [8086:8d10] type 01 class 0x060400
[ 5.766582] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[ 5.774471] pci 0000:00:1c.0: Enabling MPC IRBNCE
[ 5.780451] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled
[ 5.788547] pci 0000:00:1c.1: [8086:8d12] type 01 class 0x060400
[ 5.795587] pci 0000:00:1c.1: PME# supported from D0 D3hot D3cold
[ 5.802468] pci 0000:00:1c.1: Enabling MPC IRBNCE
[ 5.808451] pci 0000:00:1c.1: Intel PCH root port ACS workaround enabled
[ 5.816597] pci 0000:00:1c.2: [8086:8d14] type 01 class 0x060400
[ 5.823589] pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold
[ 5.831467] pci 0000:00:1c.2: Enabling MPC IRBNCE
[ 5.836451] pci 0000:00:1c.2: Intel PCH root port ACS workaround enabled
[ 5.844591] pci 0000:00:1c.4: [8086:8d18] type 01 class 0x060400
[ 5.852596] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
[ 5.859468] pci 0000:00:1c.4: Enabling MPC IRBNCE
[ 5.865452] pci 0000:00:1c.4: Intel PCH root port ACS workaround enabled
[ 5.873660] pci 0000:00:1d.0: [8086:8d26] type 00 class 0x0c0320
[ 5.880473] pci 0000:00:1d.0: reg 0x10: [mem 0x92212000-0x922123ff]
[ 5.888582] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[ 5.895577] pci 0000:00:1f.0: [8086:8d44] type 00 class 0x060100
[ 5.902717] pci 0000:00:1f.2: [8086:8d02] type 00 class 0x010601
[ 5.910468] pci 0000:00:1f.2: reg 0x10: [io 0x4068-0x406f]
[ 5.916456] pci 0000:00:1f.2: reg 0x14: [io 0x4084-0x4087]
[ 5.923457] pci 0000:00:1f.2: reg 0x18: [io 0x4060-0x4067]
[ 5.930456] pci 0000:00:1f.2: reg 0x1c: [io 0x4080-0x4083]
[ 5.937457] pci 0000:00:1f.2: reg 0x20: [io 0x4020-0x403f]
[ 5.943457] pci 0000:00:1f.2: reg 0x24: [mem 0x92210000-0x922107ff]
[ 5.951530] pci 0000:00:1f.2: PME# supported from D3hot
[ 5.957603] pci 0000:03:00.0: [1000:005d] type 00 class 0x010400
[ 5.964463] pci 0000:03:00.0: reg 0x10: [io 0x3000-0x30ff]
[ 5.971458] pci 0000:03:00.0: reg 0x14: [mem 0x92100000-0x9210ffff 64bit]
[ 5.979458] pci 0000:03:00.0: reg 0x1c: [mem 0x92000000-0x920fffff 64bit]
[ 5.987460] pci 0000:03:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[ 5.995546] pci 0000:03:00.0: supports D1 D2
[ 6.001518] pci 0000:00:01.0: PCI bridge to [bus 03]
[ 6.007452] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff]
[ 6.014452] pci 0000:00:01.0: bridge window [mem 0x92000000-0x921fffff]
[ 6.022495] pci 0000:04:00.0: [8086:154d] type 00 class 0x020000
[ 6.029467] pci 0000:04:00.0: reg 0x10: [mem 0x91e00000-0x91efffff 64bit]
[ 6.037456] pci 0000:04:00.0: reg 0x18: [io 0x2020-0x203f]
[ 6.044463] pci 0000:04:00.0: reg 0x20: [mem 0x91f04000-0x91f07fff 64bit]
[ 6.052456] pci 0000:04:00.0: reg 0x30: [mem 0xfff80000-0xffffffff pref]
[ 6.060546] pci 0000:04:00.0: PME# supported from D0 D3hot
[ 6.067471] pci 0000:04:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
[ 6.076451] pci 0000:04:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
[ 6.088463] pci 0000:04:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
[ 6.097452] pci 0000:04:00.0: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
[ 6.110559] pci 0000:04:00.1: [8086:154d] type 00 class 0x020000
[ 6.127471] pci 0000:04:00.1: reg 0x10: [mem 0x91d00000-0x91dfffff 64bit]
[ 6.135456] pci 0000:04:00.1: reg 0x18: [io 0x2000-0x201f]
[ 6.142464] pci 0000:04:00.1: reg 0x20: [mem 0x91f00000-0x91f03fff 64bit]
[ 6.150456] pci 0000:04:00.1: reg 0x30: [mem 0xfff80000-0xffffffff pref]
[ 6.158565] pci 0000:04:00.1: PME# supported from D0 D3hot
[ 6.164469] pci 0000:04:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
[ 6.173451] pci 0000:04:00.1: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
[ 6.186460] pci 0000:04:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
[ 6.195451] pci 0000:04:00.1: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
[ 6.207593] pci 0000:00:02.0: PCI bridge to [bus 04]
[ 6.213453] pci 0000:00:02.0: bridge window [io 0x2000-0x2fff]
[ 6.221452] pci 0000:00:02.0: bridge window [mem 0x91d00000-0x91ffffff]
[ 6.229488] pci 0000:00:03.0: PCI bridge to [bus 02]
[ 6.235495] pci 0000:00:1c.0: PCI bridge to [bus 05]
[ 6.241825] pci 0000:06:00.0: [1912:001d] type 01 class 0x060400
[ 6.252080] pci 0000:06:00.0: PME# supported from D0 D3hot D3cold
[ 6.260774] pci 0000:00:1c.1: PCI bridge to [bus 06-0a]
[ 6.267455] pci 0000:00:1c.1: bridge window [mem 0x91000000-0x919fffff]
[ 6.275455] pci 0000:00:1c.1: bridge window [mem 0x90000000-0x90ffffff 64bit pref]
[ 6.285644] pci 0000:07:00.0: [1912:001d] type 01 class 0x060400
[ 6.297723] pci 0000:07:00.0: PME# supported from D0 D3hot D3cold
[ 6.307285] pci 0000:06:00.0: PCI bridge to [bus 07-0a]
[ 6.314644] pci 0000:06:00.0: bridge window [mem 0x91000000-0x919fffff]
[ 6.322581] pci 0000:06:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref]
[ 6.333723] pci 0000:08:00.0: [1912:001a] type 01 class 0x060400
[ 6.345949] pci 0000:07:00.0: PCI bridge to [bus 08-09]
[ 6.352683] pci 0000:07:00.0: bridge window [mem 0x91000000-0x918fffff]
[ 6.360604] pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref]
[ 6.371566] pci_bus 0000:09: extended config space not accessible
[ 6.378703] pci 0000:09:00.0: [102b:0534] type 00 class 0x030000
[ 6.386968] pci 0000:09:00.0: reg 0x10: [mem 0x90000000-0x90ffffff pref]
[ 6.394727] pci 0000:09:00.0: reg 0x14: [mem 0x91800000-0x91803fff]
[ 6.402728] pci 0000:09:00.0: reg 0x18: [mem 0x91000000-0x917fffff]
[ 6.414870] pci 0000:08:00.0: PCI bridge to [bus 09]
[ 6.420683] pci 0000:08:00.0: bridge window [mem 0x91000000-0x918fffff]
[ 6.428605] pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref]
[ 6.439325] pci 0000:01:00.0: [8086:1521] type 00 class 0x020000
[ 6.446483] pci 0000:01:00.0: reg 0x10: [mem 0x91b00000-0x91bfffff]
[ 6.454483] pci 0000:01:00.0: reg 0x1c: [mem 0x91c04000-0x91c07fff]
[ 6.461483] pci 0000:01:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[ 6.469614] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[ 6.477508] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:1c.2 (capable of 16.000 Gb/s with 5 GT/s x4 link)
[ 6.493582] pci 0000:01:00.1: [8086:1521] type 00 class 0x020000
[ 6.500483] pci 0000:01:00.1: reg 0x10: [mem 0x91a00000-0x91afffff]
[ 6.508482] pci 0000:01:00.1: reg 0x1c: [mem 0x91c00000-0x91c03fff]
[ 6.515483] pci 0000:01:00.1: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[ 6.523614] pci 0000:01:00.1: PME# supported from D0 D3hot D3cold
[ 6.531641] pci 0000:00:1c.2: PCI bridge to [bus 01]
[ 6.537455] pci 0000:00:1c.2: bridge window [mem 0x91a00000-0x91cfffff]
[ 6.545499] pci 0000:00:1c.4: PCI bridge to [bus 0b]
[ 6.551492] pci_bus 0000:00: on NUMA node 0
[ 6.551811] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 11 12 14 *15)
[ 6.560525] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 12 *14 15)
[ 6.569520] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
[ 6.578519] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 *6 7 9 10 11 12 14 15)
[ 6.587519] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
[ 6.597517] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
[ 6.608515] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
[ 6.618514] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
[ 6.628823] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
[ 6.636455] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[ 6.647825] acpi PNP0A08:01: _OSC: OS now controls [PME AER PCIeCapability LTR]
[ 6.656451] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
[ 6.666581] PCI host bridge to bus 0000:80
[ 6.672452] pci_bus 0000:80: root bus resource [io 0x8000-0xffff window]
[ 6.680451] pci_bus 0000:80: root bus resource [mem 0xc8000000-0xfbffbfff window]
[ 6.689451] pci_bus 0000:80: root bus resource [mem 0x3c000000000-0x3ffffffffff window]
[ 6.699451] pci_bus 0000:80: root bus resource [bus 80-fe]
[ 6.705496] pci 0000:80:00.0: [8086:6f01] type 01 class 0x060400
[ 6.712574] pci 0000:80:00.0: PME# supported from D0 D3hot D3cold
[ 6.720594] pci 0000:80:01.0: [8086:6f02] type 01 class 0x060400
[ 6.727573] pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
[ 6.735642] pci 0000:80:02.0: [8086:6f04] type 01 class 0x060400
[ 6.742571] pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
[ 6.749636] pci 0000:80:03.0: [8086:6f08] type 01 class 0x060400
[ 6.757572] pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
[ 6.764641] pci 0000:80:05.0: [8086:6f28] type 00 class 0x088000
[ 6.772616] pci 0000:80:05.1: [8086:6f29] type 00 class 0x088000
[ 6.779651] pci 0000:80:05.2: [8086:6f2a] type 00 class 0x088000
[ 6.786608] pci 0000:80:05.4: [8086:6f2c] type 00 class 0x080020
[ 6.794461] pci 0000:80:05.4: reg 0x10: [mem 0xc8400000-0xc8400fff]
[ 6.801651] pci 0000:80:00.0: PCI bridge to [bus 81]
[ 6.807499] pci 0000:82:00.0: [8086:154d] type 00 class 0x020000
[ 6.814469] pci 0000:82:00.0: reg 0x10: [mem 0xc8200000-0xc82fffff 64bit]
[ 6.823456] pci 0000:82:00.0: reg 0x18: [io 0x9020-0x903f]
[ 6.829464] pci 0000:82:00.0: reg 0x20: [mem 0xc8304000-0xc8307fff 64bit]
[ 6.837456] pci 0000:82:00.0: reg 0x30: [mem 0xfff80000-0xffffffff pref]
[ 6.845549] pci 0000:82:00.0: PME# supported from D0 D3hot
[ 6.852473] pci 0000:82:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
[ 6.861452] pci 0000:82:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
[ 6.873462] pci 0000:82:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
[ 6.882454] pci 0000:82:00.0: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
[ 6.895609] pci 0000:82:00.1: [8086:154d] type 00 class 0x020000
[ 6.902468] pci 0000:82:00.1: reg 0x10: [mem 0xc8100000-0xc81fffff 64bit]
[ 6.910457] pci 0000:82:00.1: reg 0x18: [io 0x9000-0x901f]
[ 6.917464] pci 0000:82:00.1: reg 0x20: [mem 0xc8300000-0xc8303fff 64bit]
[ 6.925456] pci 0000:82:00.1: reg 0x30: [mem 0xfff80000-0xffffffff pref]
[ 6.933548] pci 0000:82:00.1: PME# supported from D0 D3hot
[ 6.939469] pci 0000:82:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref]
[ 6.948452] pci 0000:82:00.1: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs)
[ 6.961461] pci 0000:82:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref]
[ 6.970451] pci 0000:82:00.1: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs)
[ 6.982598] pci 0000:80:01.0: PCI bridge to [bus 82]
[ 6.988452] pci 0000:80:01.0: bridge window [io 0x9000-0x9fff]
[ 6.996452] pci 0000:80:01.0: bridge window [mem 0xc8100000-0xc83fffff]
[ 7.004488] pci 0000:80:02.0: PCI bridge to [bus 83]
[ 7.010497] pci 0000:84:00.0: [9005:028f] type 00 class 0x010700
[ 7.017467] pci 0000:84:00.0: reg 0x10: [mem 0xc8000000-0xc8007fff 64bit]
[ 7.025466] pci 0000:84:00.0: reg 0x20: [io 0x8000-0x80ff]
[ 7.032461] pci 0000:84:00.0: reg 0x30: [mem 0xffe00000-0xffffffff pref]
[ 7.040540] pci 0000:84:00.0: supports D1
[ 7.045451] pci 0000:84:00.0: PME# supported from D0 D1 D3hot
[ 7.052556] pci 0000:80:03.0: PCI bridge to [bus 84]
[ 7.058452] pci 0000:80:03.0: bridge window [io 0x8000-0x8fff]
[ 7.065452] pci 0000:80:03.0: bridge window [mem 0xc8000000-0xc80fffff]
[ 7.073471] pci_bus 0000:80: on NUMA node 1
[ 7.073729] iommu: Default domain type: Translated
[ 7.079646] pci 0000:09:00.0: vgaarb: setting as boot VGA device
[ 7.080448] pci 0000:09:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[ 7.097455] pci 0000:09:00.0: vgaarb: bridge control possible
[ 7.104450] vgaarb: loaded
[ 7.107566] SCSI subsystem initialized
[ 7.112483] libata version 3.00 loaded.
[ 7.112498] ACPI: bus type USB registered
[ 7.117478] usbcore: registered new interface driver usbfs
[ 7.124461] usbcore: registered new interface driver hub
[ 7.130492] usbcore: registered new device driver usb
[ 7.136472] pps_core: LinuxPPS API ver. 1 registered
[ 7.142450] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 7.153453] PTP clock support registered
[ 7.158502] EDAC MC: Ver: 3.0.0
[ 7.163472] PCI: Using ACPI for IRQ routing
[ 7.175191] PCI: pci_cache_line_size set to 64 bytes
[ 7.175695] e820: reserve RAM buffer [mem 0x00099000-0x0009ffff]
[ 7.175697] e820: reserve RAM buffer [mem 0x7a289000-0x7bffffff]
[ 7.175698] e820: reserve RAM buffer [mem 0x7bae9000-0x7bffffff]
[ 7.175700] e820: reserve RAM buffer [mem 0x7bb00000-0x7bffffff]
[ 7.176552] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[ 7.183452] hpet0: 8 comparators, 64-bit 14.318180 MHz counter
[ 7.192517] clocksource: Switched to clocksource tsc-early
[ 7.227610] VFS: Disk quotas dquot_6.6.0
[ 7.232496] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 7.240712] FS-Cache: Loaded
[ 7.244494] CacheFiles: Loaded
[ 7.248407] pnp: PnP ACPI init
[ 7.253036] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
[ 7.253267] system 00:01: [io 0x0500-0x053f] has been reserved
[ 7.260362] system 00:01: [io 0x0400-0x047f] has been reserved
[ 7.267482] system 00:01: [io 0x0540-0x057f] has been reserved
[ 7.274576] system 00:01: [io 0x0600-0x061f] has been reserved
[ 7.281716] system 00:01: [io 0x0ca0-0x0ca5] has been reserved
[ 7.288813] system 00:01: [io 0x0880-0x0883] has been reserved
[ 7.295947] system 00:01: [io 0x0800-0x081f] has been reserved
[ 7.303041] system 00:01: [mem 0xfeda8000-0xfedcbfff] could not be reserved
[ 7.311344] system 00:01: [mem 0xff000000-0xffffffff] could not be reserved
[ 7.319604] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved
[ 7.327465] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved
[ 7.345485] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved
[ 7.353442] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved
[ 7.361322] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 7.361603] pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active)
[ 7.361860] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active)
[ 7.361947] system 00:04: [io 0x0ca8] has been reserved
[ 7.368361] system 00:04: [io 0x0cac] has been reserved
[ 7.374781] system 00:04: Plug and Play ACPI device, IDs IPI0001 PNP0c01 (active)
[ 7.374923] pnp: PnP ACPI: found 5 devices
[ 7.381673] thermal_sys: Registered thermal governor 'step_wise'
[ 7.381674] thermal_sys: Registered thermal governor 'user_space'
[ 7.393391] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[ 7.411478] pci 0000:03:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window
[ 7.423440] pci 0000:04:00.0: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window
[ 7.435401] pci 0000:04:00.1: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window
[ 7.447369] pci 0000:01:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window
[ 7.459330] pci 0000:01:00.1: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window
[ 7.471382] pci 0000:82:00.0: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window
[ 7.483349] pci 0000:82:00.1: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window
[ 7.495313] pci 0000:84:00.0: can't claim BAR 6 [mem 0xffe00000-0xffffffff pref]: no compatible bridge window
[ 7.507309] pci 0000:00:02.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 400000 add_align 100000
[ 7.521032] pci 0000:00:02.0: BAR 15: assigned [mem 0x38000000000-0x380003fffff 64bit pref]
[ 7.531247] pci 0000:03:00.0: BAR 6: no space for [mem size 0x00100000 pref]
[ 7.539597] pci 0000:03:00.0: BAR 6: failed to assign [mem size 0x00100000 pref]
[ 7.548743] pci 0000:00:01.0: PCI bridge to [bus 03]
[ 7.554765] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff]
[ 7.562051] pci 0000:00:01.0: bridge window [mem 0x92000000-0x921fffff]
[ 7.570122] pci 0000:04:00.0: BAR 6: assigned [mem 0x91f80000-0x91ffffff pref]
[ 7.579075] pci 0000:04:00.1: BAR 6: no space for [mem size 0x00080000 pref]
[ 7.587427] pci 0000:04:00.1: BAR 6: failed to assign [mem size 0x00080000 pref]
[ 7.596656] pci 0000:04:00.0: BAR 7: assigned [mem 0x38000000000-0x380000fffff 64bit pref]
[ 7.606782] pci 0000:04:00.0: BAR 10: assigned [mem 0x38000100000-0x380001fffff 64bit pref]
[ 7.617004] pci 0000:04:00.1: BAR 7: assigned [mem 0x38000200000-0x380002fffff 64bit pref]
[ 7.627124] pci 0000:04:00.1: BAR 10: assigned [mem 0x38000300000-0x380003fffff 64bit pref]
[ 7.637340] pci 0000:00:02.0: PCI bridge to [bus 04]
[ 7.643366] pci 0000:00:02.0: bridge window [io 0x2000-0x2fff]
[ 7.650654] pci 0000:00:02.0: bridge window [mem 0x91d00000-0x91ffffff]
[ 7.658721] pci 0000:00:02.0: bridge window [mem 0x38000000000-0x380003fffff 64bit pref]
[ 7.668924] pci 0000:00:03.0: PCI bridge to [bus 02]
[ 7.674954] pci 0000:00:1c.0: PCI bridge to [bus 05]
[ 7.680985] pci 0000:08:00.0: PCI bridge to [bus 09]
[ 7.687160] pci 0000:08:00.0: bridge window [mem 0x91000000-0x918fffff]
[ 7.695406] pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref]
[ 7.705152] pci 0000:07:00.0: PCI bridge to [bus 08-09]
[ 7.711615] pci 0000:07:00.0: bridge window [mem 0x91000000-0x918fffff]
[ 7.719777] pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref]
[ 7.729513] pci 0000:06:00.0: PCI bridge to [bus 07-0a]
[ 7.735963] pci 0000:06:00.0: bridge window [mem 0x91000000-0x919fffff]
[ 7.744109] pci 0000:06:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref]
[ 7.753962] pci 0000:00:1c.1: PCI bridge to [bus 06-0a]
[ 7.760286] pci 0000:00:1c.1: bridge window [mem 0x91000000-0x919fffff]
[ 7.768351] pci 0000:00:1c.1: bridge window [mem 0x90000000-0x90ffffff 64bit pref]
[ 7.777881] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00100000 pref]
[ 7.786235] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00100000 pref]
[ 7.795468] pci 0000:01:00.1: BAR 6: no space for [mem size 0x00100000 pref]
[ 7.803825] pci 0000:01:00.1: BAR 6: failed to assign [mem size 0x00100000 pref]
[ 7.812970] pci 0000:00:1c.2: PCI bridge to [bus 01]
[ 7.818995] pci 0000:00:1c.2: bridge window [mem 0x91a00000-0x91cfffff]
[ 7.827062] pci 0000:00:1c.4: PCI bridge to [bus 0b]
[ 7.833097] pci_bus 0000:00: resource 4 [io 0x0000-0x03bb window]
[ 7.840477] pci_bus 0000:00: resource 5 [io 0x03bc-0x03df window]
[ 7.847860] pci_bus 0000:00: resource 6 [io 0x03e0-0x0cf7 window]
[ 7.855242] pci_bus 0000:00: resource 7 [io 0x1000-0x7fff window]
[ 7.862624] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff window]
[ 7.870783] pci_bus 0000:00: resource 9 [mem 0x90000000-0xc7ffbfff window]
[ 7.878945] pci_bus 0000:00: resource 10 [mem 0x38000000000-0x3bfffffffff window]
[ 7.888185] pci_bus 0000:03: resource 0 [io 0x3000-0x3fff]
[ 7.894885] pci_bus 0000:03: resource 1 [mem 0x92000000-0x921fffff]
[ 7.902360] pci_bus 0000:04: resource 0 [io 0x2000-0x2fff]
[ 7.909060] pci_bus 0000:04: resource 1 [mem 0x91d00000-0x91ffffff]
[ 7.916543] pci_bus 0000:04: resource 2 [mem 0x38000000000-0x380003fffff 64bit pref]
[ 7.926077] pci_bus 0000:06: resource 1 [mem 0x91000000-0x919fffff]
[ 7.933554] pci_bus 0000:06: resource 2 [mem 0x90000000-0x90ffffff 64bit pref]
[ 7.942504] pci_bus 0000:07: resource 1 [mem 0x91000000-0x919fffff]
[ 7.949983] pci_bus 0000:07: resource 2 [mem 0x90000000-0x90ffffff 64bit pref]
[ 7.958921] pci_bus 0000:08: resource 1 [mem 0x91000000-0x918fffff]
[ 7.966399] pci_bus 0000:08: resource 2 [mem 0x90000000-0x90ffffff 64bit pref]
[ 7.975357] pci_bus 0000:09: resource 1 [mem 0x91000000-0x918fffff]
[ 7.982836] pci_bus 0000:09: resource 2 [mem 0x90000000-0x90ffffff 64bit pref]
[ 7.991792] pci_bus 0000:01: resource 1 [mem 0x91a00000-0x91cfffff]
[ 7.999376] pci 0000:80:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 82] add_size 400000 add_align 100000
[ 8.013089] pci 0000:80:01.0: BAR 15: assigned [mem 0x3c000000000-0x3c0003fffff 64bit pref]
[ 8.023300] pci 0000:80:00.0: PCI bridge to [bus 81]
[ 8.029329] pci 0000:82:00.0: BAR 6: assigned [mem 0xc8380000-0xc83fffff pref]
[ 8.038274] pci 0000:82:00.1: BAR 6: no space for [mem size 0x00080000 pref]
[ 8.046631] pci 0000:82:00.1: BAR 6: failed to assign [mem size 0x00080000 pref]
[ 8.055778] pci 0000:82:00.0: BAR 7: assigned [mem 0x3c000000000-0x3c0000fffff 64bit pref]
[ 8.065895] pci 0000:82:00.0: BAR 10: assigned [mem 0x3c000100000-0x3c0001fffff 64bit pref]
[ 8.076114] pci 0000:82:00.1: BAR 7: assigned [mem 0x3c000200000-0x3c0002fffff 64bit pref]
[ 8.086230] pci 0000:82:00.1: BAR 10: assigned [mem 0x3c000300000-0x3c0003fffff 64bit pref]
[ 8.096437] pci 0000:80:01.0: PCI bridge to [bus 82]
[ 8.102460] pci 0000:80:01.0: bridge window [io 0x9000-0x9fff]
[ 8.109747] pci 0000:80:01.0: bridge window [mem 0xc8100000-0xc83fffff]
[ 8.117802] pci 0000:80:01.0: bridge window [mem 0x3c000000000-0x3c0003fffff 64bit pref]
[ 8.127921] pci 0000:80:02.0: PCI bridge to [bus 83]
[ 8.133960] pci 0000:84:00.0: BAR 6: no space for [mem size 0x00200000 pref]
[ 8.142316] pci 0000:84:00.0: BAR 6: failed to assign [mem size 0x00200000 pref]
[ 8.151545] pci 0000:80:03.0: PCI bridge to [bus 84]
[ 8.157572] pci 0000:80:03.0: bridge window [io 0x8000-0x8fff]
[ 8.164856] pci 0000:80:03.0: bridge window [mem 0xc8000000-0xc80fffff]
[ 8.172927] pci_bus 0000:80: resource 4 [io 0x8000-0xffff window]
[ 8.180309] pci_bus 0000:80: resource 5 [mem 0xc8000000-0xfbffbfff window]
[ 8.188470] pci_bus 0000:80: resource 6 [mem 0x3c000000000-0x3ffffffffff window]
[ 8.197622] pci_bus 0000:82: resource 0 [io 0x9000-0x9fff]
[ 8.204324] pci_bus 0000:82: resource 1 [mem 0xc8100000-0xc83fffff]
[ 8.211807] pci_bus 0000:82: resource 2 [mem 0x3c000000000-0x3c0003fffff 64bit pref]
[ 8.221340] pci_bus 0000:84: resource 0 [io 0x8000-0x8fff]
[ 8.228037] pci_bus 0000:84: resource 1 [mem 0xc8000000-0xc80fffff]
[ 8.235699] NET: Registered protocol family 2
[ 8.241286] tcp_listen_portaddr_hash hash table entries: 65536 (order: 8, 1048576 bytes, vmalloc)
[ 8.252418] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc)
[ 8.263376] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, vmalloc)
[ 8.272811] TCP: Hash tables configured (established 524288 bind 65536)
[ 8.280838] UDP hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[ 8.289643] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[ 8.299389] pci 0000:00:05.0: disabled boot interrupts on device [8086:6f28]
[ 8.324577] pci 0000:00:1a.0: quirk_usb_early_handoff+0x0/0x690 took 15996 usecs
[ 8.334000] pci 0000:04:00.0: CLS mismatch (128 != 64), using 64 bytes
[ 8.342018] pci 0000:09:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[ 8.352336] pci 0000:09:00.0: pci_fixup_video+0x0/0x110 took 10295 usecs
[ 8.360413] pci 0000:80:05.0: disabled boot interrupts on device [8086:6f28]
[ 8.368935] Trying to unpack rootfs image as initramfs...
[ 8.492233] Freeing initrd memory: 5136K
[ 8.497502] DMAR: dmar0: Using Queued invalidation
[ 8.503349] DMAR: dmar1: Using Queued invalidation
[ 8.509737] pci 0000:00:00.0: Adding to iommu group 0
[ 8.516107] pci 0000:00:01.0: Adding to iommu group 1
[ 8.522481] pci 0000:00:02.0: Adding to iommu group 2
[ 8.528857] pci 0000:00:03.0: Adding to iommu group 3
[ 8.535209] pci 0000:00:05.0: Adding to iommu group 4
[ 8.541571] pci 0000:00:05.1: Adding to iommu group 5
[ 8.547998] pci 0000:00:05.2: Adding to iommu group 6
[ 8.554361] pci 0000:00:05.4: Adding to iommu group 7
[ 8.560717] pci 0000:00:11.0: Adding to iommu group 8
[ 8.567103] pci 0000:00:11.4: Adding to iommu group 9
[ 8.583789] pci 0000:00:14.0: Adding to iommu group 10
[ 8.590304] pci 0000:00:16.0: Adding to iommu group 11
[ 8.596575] pci 0000:00:16.1: Adding to iommu group 11
[ 8.603109] pci 0000:00:1a.0: Adding to iommu group 12
[ 8.609566] pci 0000:00:1c.0: Adding to iommu group 13
[ 8.616029] pci 0000:00:1c.1: Adding to iommu group 14
[ 8.622511] pci 0000:00:1c.2: Adding to iommu group 15
[ 8.628981] pci 0000:00:1c.4: Adding to iommu group 16
[ 8.635513] pci 0000:00:1d.0: Adding to iommu group 17
[ 8.643219] pci 0000:00:1f.0: Adding to iommu group 18
[ 8.649488] pci 0000:00:1f.2: Adding to iommu group 18
[ 8.655957] pci 0000:01:00.0: Adding to iommu group 19
[ 8.662430] pci 0000:01:00.1: Adding to iommu group 20
[ 8.668926] pci 0000:03:00.0: Adding to iommu group 21
[ 8.675387] pci 0000:04:00.0: Adding to iommu group 22
[ 8.681863] pci 0000:04:00.1: Adding to iommu group 23
[ 8.688353] pci 0000:06:00.0: Adding to iommu group 24
[ 8.694864] pci 0000:07:00.0: Adding to iommu group 25
[ 8.701176] pci 0000:08:00.0: Adding to iommu group 25
[ 8.707418] pci 0000:09:00.0: Adding to iommu group 25
[ 8.713953] pci 0000:7f:08.0: Adding to iommu group 26
[ 8.720235] pci 0000:7f:08.2: Adding to iommu group 26
[ 8.726708] pci 0000:7f:08.3: Adding to iommu group 27
[ 8.733254] pci 0000:7f:09.0: Adding to iommu group 28
[ 8.739534] pci 0000:7f:09.2: Adding to iommu group 28
[ 8.746006] pci 0000:7f:09.3: Adding to iommu group 29
[ 8.752610] pci 0000:7f:0b.0: Adding to iommu group 30
[ 8.758885] pci 0000:7f:0b.1: Adding to iommu group 30
[ 8.765163] pci 0000:7f:0b.2: Adding to iommu group 30
[ 8.771451] pci 0000:7f:0b.3: Adding to iommu group 30
[ 8.778128] pci 0000:7f:0c.0: Adding to iommu group 31
[ 8.784414] pci 0000:7f:0c.1: Adding to iommu group 31
[ 8.790689] pci 0000:7f:0c.2: Adding to iommu group 31
[ 8.796967] pci 0000:7f:0c.3: Adding to iommu group 31
[ 8.803248] pci 0000:7f:0c.4: Adding to iommu group 31
[ 8.809530] pci 0000:7f:0c.5: Adding to iommu group 31
[ 8.816174] pci 0000:7f:0f.0: Adding to iommu group 32
[ 8.822455] pci 0000:7f:0f.1: Adding to iommu group 32
[ 8.828736] pci 0000:7f:0f.4: Adding to iommu group 32
[ 8.835016] pci 0000:7f:0f.5: Adding to iommu group 32
[ 8.841296] pci 0000:7f:0f.6: Adding to iommu group 32
[ 8.847926] pci 0000:7f:10.0: Adding to iommu group 33
[ 8.854207] pci 0000:7f:10.1: Adding to iommu group 33
[ 8.860489] pci 0000:7f:10.5: Adding to iommu group 33
[ 8.866774] pci 0000:7f:10.6: Adding to iommu group 33
[ 8.873057] pci 0000:7f:10.7: Adding to iommu group 33
[ 8.879630] pci 0000:7f:12.0: Adding to iommu group 34
[ 8.885917] pci 0000:7f:12.1: Adding to iommu group 34
[ 8.892197] pci 0000:7f:12.2: Adding to iommu group 34
[ 8.898666] pci 0000:7f:13.0: Adding to iommu group 35
[ 8.905131] pci 0000:7f:13.1: Adding to iommu group 36
[ 8.911599] pci 0000:7f:13.2: Adding to iommu group 37
[ 8.918160] pci 0000:7f:13.3: Adding to iommu group 38
[ 8.924628] pci 0000:7f:13.4: Adding to iommu group 39
[ 8.931101] pci 0000:7f:13.5: Adding to iommu group 40
[ 8.937626] pci 0000:7f:13.6: Adding to iommu group 41
[ 8.943911] pci 0000:7f:13.7: Adding to iommu group 41
[ 8.950371] pci 0000:7f:14.0: Adding to iommu group 42
[ 8.956826] pci 0000:7f:14.1: Adding to iommu group 43
[ 8.963290] pci 0000:7f:14.2: Adding to iommu group 44
[ 8.969740] pci 0000:7f:14.3: Adding to iommu group 45
[ 8.976341] pci 0000:7f:14.4: Adding to iommu group 46
[ 8.982629] pci 0000:7f:14.5: Adding to iommu group 46
[ 8.989006] pci 0000:7f:14.6: Adding to iommu group 46
[ 8.995292] pci 0000:7f:14.7: Adding to iommu group 46
[ 9.001745] pci 0000:7f:15.0: Adding to iommu group 47
[ 9.008207] pci 0000:7f:15.1: Adding to iommu group 48
[ 9.014659] pci 0000:7f:15.2: Adding to iommu group 49
[ 9.021115] pci 0000:7f:15.3: Adding to iommu group 50
[ 9.027677] pci 0000:7f:16.0: Adding to iommu group 51
[ 9.033962] pci 0000:7f:16.6: Adding to iommu group 51
[ 9.040252] pci 0000:7f:16.7: Adding to iommu group 51
[ 9.046874] pci 0000:7f:17.0: Adding to iommu group 52
[ 9.053164] pci 0000:7f:17.4: Adding to iommu group 52
[ 9.059452] pci 0000:7f:17.5: Adding to iommu group 52
[ 9.065738] pci 0000:7f:17.6: Adding to iommu group 52
[ 9.072024] pci 0000:7f:17.7: Adding to iommu group 52
[ 9.078675] pci 0000:7f:1e.0: Adding to iommu group 53
[ 9.084968] pci 0000:7f:1e.1: Adding to iommu group 53
[ 9.091257] pci 0000:7f:1e.2: Adding to iommu group 53
[ 9.097545] pci 0000:7f:1e.3: Adding to iommu group 53
[ 9.103829] pci 0000:7f:1e.4: Adding to iommu group 53
[ 9.110359] pci 0000:7f:1f.0: Adding to iommu group 54
[ 9.116643] pci 0000:7f:1f.2: Adding to iommu group 54
[ 9.123094] pci 0000:80:00.0: Adding to iommu group 55
[ 9.129558] pci 0000:80:01.0: Adding to iommu group 56
[ 9.136013] pci 0000:80:02.0: Adding to iommu group 57
[ 9.142485] pci 0000:80:03.0: Adding to iommu group 58
[ 9.148934] pci 0000:80:05.0: Adding to iommu group 59
[ 9.155402] pci 0000:80:05.1: Adding to iommu group 60
[ 9.161856] pci 0000:80:05.2: Adding to iommu group 61
[ 9.168326] pci 0000:80:05.4: Adding to iommu group 62
[ 9.174767] pci 0000:82:00.0: Adding to iommu group 63
[ 9.181231] pci 0000:82:00.1: Adding to iommu group 64
[ 9.187684] pci 0000:84:00.0: Adding to iommu group 65
[ 9.194217] pci 0000:ff:08.0: Adding to iommu group 66
[ 9.200512] pci 0000:ff:08.2: Adding to iommu group 66
[ 9.206966] pci 0000:ff:08.3: Adding to iommu group 67
[ 9.213490] pci 0000:ff:09.0: Adding to iommu group 68
[ 9.219785] pci 0000:ff:09.2: Adding to iommu group 68
[ 9.226235] pci 0000:ff:09.3: Adding to iommu group 69
[ 9.232832] pci 0000:ff:0b.0: Adding to iommu group 70
[ 9.239127] pci 0000:ff:0b.1: Adding to iommu group 70
[ 9.245507] pci 0000:ff:0b.2: Adding to iommu group 70
[ 9.251800] pci 0000:ff:0b.3: Adding to iommu group 70
[ 9.258459] pci 0000:ff:0c.0: Adding to iommu group 71
[ 9.264752] pci 0000:ff:0c.1: Adding to iommu group 71
[ 9.271048] pci 0000:ff:0c.2: Adding to iommu group 71
[ 9.277343] pci 0000:ff:0c.3: Adding to iommu group 71
[ 9.283638] pci 0000:ff:0c.4: Adding to iommu group 71
[ 9.289934] pci 0000:ff:0c.5: Adding to iommu group 71
[ 9.296573] pci 0000:ff:0f.0: Adding to iommu group 72
[ 9.302865] pci 0000:ff:0f.1: Adding to iommu group 72
[ 9.309160] pci 0000:ff:0f.4: Adding to iommu group 72
[ 9.315457] pci 0000:ff:0f.5: Adding to iommu group 72
[ 9.321752] pci 0000:ff:0f.6: Adding to iommu group 72
[ 9.328391] pci 0000:ff:10.0: Adding to iommu group 73
[ 9.334689] pci 0000:ff:10.1: Adding to iommu group 73
[ 9.340992] pci 0000:ff:10.5: Adding to iommu group 73
[ 9.347289] pci 0000:ff:10.6: Adding to iommu group 73
[ 9.353587] pci 0000:ff:10.7: Adding to iommu group 73
[ 9.360137] pci 0000:ff:12.0: Adding to iommu group 74
[ 9.366434] pci 0000:ff:12.1: Adding to iommu group 74
[ 9.372736] pci 0000:ff:12.2: Adding to iommu group 74
[ 9.379194] pci 0000:ff:13.0: Adding to iommu group 75
[ 9.385636] pci 0000:ff:13.1: Adding to iommu group 76
[ 9.392100] pci 0000:ff:13.2: Adding to iommu group 77
[ 9.398554] pci 0000:ff:13.3: Adding to iommu group 78
[ 9.405017] pci 0000:ff:13.4: Adding to iommu group 79
[ 9.411467] pci 0000:ff:13.5: Adding to iommu group 80
[ 9.417989] pci 0000:ff:13.6: Adding to iommu group 81
[ 9.424298] pci 0000:ff:13.7: Adding to iommu group 81
[ 9.430750] pci 0000:ff:14.0: Adding to iommu group 82
[ 9.437229] pci 0000:ff:14.1: Adding to iommu group 83
[ 9.443698] pci 0000:ff:14.2: Adding to iommu group 84
[ 9.450142] pci 0000:ff:14.3: Adding to iommu group 85
[ 9.456750] pci 0000:ff:14.4: Adding to iommu group 86
[ 9.463054] pci 0000:ff:14.5: Adding to iommu group 86
[ 9.469345] pci 0000:ff:14.6: Adding to iommu group 86
[ 9.475652] pci 0000:ff:14.7: Adding to iommu group 86
[ 9.482109] pci 0000:ff:15.0: Adding to iommu group 87
[ 9.488571] pci 0000:ff:15.1: Adding to iommu group 88
[ 9.495028] pci 0000:ff:15.2: Adding to iommu group 89
[ 9.501576] pci 0000:ff:15.3: Adding to iommu group 90
[ 9.508137] pci 0000:ff:16.0: Adding to iommu group 91
[ 9.514443] pci 0000:ff:16.6: Adding to iommu group 91
[ 9.520748] pci 0000:ff:16.7: Adding to iommu group 91
[ 9.527395] pci 0000:ff:17.0: Adding to iommu group 92
[ 9.533700] pci 0000:ff:17.4: Adding to iommu group 92
[ 9.540001] pci 0000:ff:17.5: Adding to iommu group 92
[ 9.546304] pci 0000:ff:17.6: Adding to iommu group 92
[ 9.552609] pci 0000:ff:17.7: Adding to iommu group 92
[ 9.559253] pci 0000:ff:1e.0: Adding to iommu group 93
[ 9.565559] pci 0000:ff:1e.1: Adding to iommu group 93
[ 9.571861] pci 0000:ff:1e.2: Adding to iommu group 93
[ 9.578164] pci 0000:ff:1e.3: Adding to iommu group 93
[ 9.584473] pci 0000:ff:1e.4: Adding to iommu group 93
[ 9.590995] pci 0000:ff:1f.0: Adding to iommu group 94
[ 9.597297] pci 0000:ff:1f.2: Adding to iommu group 94
[ 9.603528] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 9.613463] RAPL PMU: API unit is 2^-32 Joules, 2 fixed counters, 655360 ms ovfl timer
[ 9.623207] RAPL PMU: hw unit of domain package 2^-14 Joules
[ 9.630102] RAPL PMU: hw unit of domain dram 2^-16 Joules
[ 9.638387] Initialise system trusted keyrings
[ 9.643924] workingset: timestamp_bits=40 max_order=25 bucket_order=0
[ 9.651894] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[ 9.661323] Key type asymmetric registered
[ 9.666389] Asymmetric key parser 'x509' registered
[ 9.672335] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[ 9.681554] io scheduler mq-deadline registered
[ 9.687099] io scheduler kyber registered
[ 9.692674] pcieport 0000:00:01.0: PME: Signaling with IRQ 25
[ 9.699661] pcieport 0000:00:01.0: AER: enabled with IRQ 25
[ 9.706650] pcieport 0000:00:02.0: PME: Signaling with IRQ 26
[ 9.713612] pcieport 0000:00:02.0: AER: enabled with IRQ 26
[ 9.720574] pcieport 0000:00:03.0: PME: Signaling with IRQ 27
[ 9.727524] pcieport 0000:00:03.0: AER: enabled with IRQ 27
[ 9.744634] pcieport 0000:00:1c.0: PME: Signaling with IRQ 28
[ 9.751761] pcieport 0000:00:1c.1: PME: Signaling with IRQ 29
[ 9.758887] pcieport 0000:00:1c.1: AER: enabled with IRQ 29
[ 9.765815] pcieport 0000:00:1c.2: PME: Signaling with IRQ 30
[ 9.772867] pcieport 0000:00:1c.2: AER: enabled with IRQ 30
[ 9.779786] pcieport 0000:00:1c.4: PME: Signaling with IRQ 31
[ 9.786832] pcieport 0000:00:1c.4: AER: enabled with IRQ 31
[ 9.797819] pcieport 0000:80:00.0: PME: Signaling with IRQ 35
[ 9.804801] pcieport 0000:80:00.0: AER: enabled with IRQ 35
[ 9.811769] pcieport 0000:80:01.0: PME: Signaling with IRQ 36
[ 9.818736] pcieport 0000:80:01.0: AER: enabled with IRQ 36
[ 9.825680] pcieport 0000:80:02.0: PME: Signaling with IRQ 37
[ 9.832653] pcieport 0000:80:02.0: AER: enabled with IRQ 37
[ 9.839600] pcieport 0000:80:03.0: PME: Signaling with IRQ 38
[ 9.846568] pcieport 0000:80:03.0: AER: enabled with IRQ 38
[ 9.853395] IPMI message handler: version 39.2
[ 9.858991] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[ 9.868196] ACPI: Power Button [PWRF]
[ 9.872893] Monitor-Mwait will be used to enter C-1 state
[ 9.874403] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 9.902531] 00:02: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
[ 9.932158] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 9.941794] lp: driver loaded but no devices found
[ 9.947756] Linux agpgart interface v0.103
[ 9.956048] brd: module loaded
[ 9.962719] loop: module loaded
[ 9.970151] drbd: initialized. Version: 8.4.11 (api:1/proto:86-101)
[ 9.977637] drbd: built-in
[ 9.981137] drbd: registered as block device major 147
[ 9.987931] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
[ 9.997019] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[ 10.005319] megasas: 07.710.50.00-rc1
[ 10.010057] megaraid_sas 0000:03:00.0: BAR:0x1 BAR's base_addr(phys):0x0000000092100000 mapped virt_addr:0x0000000023a08203
[ 10.010060] megaraid_sas 0000:03:00.0: FW now in Ready state
[ 10.016870] megaraid_sas 0000:03:00.0: 63 bit DMA mask and 32 bit consistent mask
[ 10.026650] megaraid_sas 0000:03:00.0: firmware supports msix : (96)
[ 10.034424] megaraid_sas 0000:03:00.0: requested/available msix 13/13
[ 10.042101] megaraid_sas 0000:03:00.0: current msix/online cpus : (13/12)
[ 10.050169] megaraid_sas 0000:03:00.0: RDPQ mode : (disabled)
[ 10.057064] megaraid_sas 0000:03:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 0
[ 10.068852] megaraid_sas 0000:03:00.0: Configured max firmware commands: 927
[ 10.081825] megaraid_sas 0000:03:00.0: Performance mode :Latency
[ 10.089015] megaraid_sas 0000:03:00.0: FW supports sync cache : No
[ 10.096400] megaraid_sas 0000:03:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[ 10.151455] megaraid_sas 0000:03:00.0: FW provided supportMaxExtLDs: 1 max_lds: 64
[ 10.160794] megaraid_sas 0000:03:00.0: controller type : MR(2048MB)
[ 10.168274] megaraid_sas 0000:03:00.0: Online Controller Reset(OCR) : Enabled
[ 10.176726] megaraid_sas 0000:03:00.0: Secure JBOD support : No
[ 10.183818] megaraid_sas 0000:03:00.0: NVMe passthru support : No
[ 10.191103] megaraid_sas 0000:03:00.0: FW provided TM TaskAbort/Reset timeout : 0 secs/0 secs
[ 10.201509] megaraid_sas 0000:03:00.0: JBOD sequence map support : No
[ 10.209189] megaraid_sas 0000:03:00.0: PCI Lane Margining support : No
[ 10.657698] megaraid_sas 0000:03:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[ 10.669272] megaraid_sas 0000:03:00.0: INIT adapter done
[ 10.675689] megaraid_sas 0000:03:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5660
[ 10.686566] megaraid_sas 0000:03:00.0: pci id : (0x1000)/(0x005d)/(0x1028)/(0x1f42)
[ 10.695460] tsc: Refined TSC clocksource calibration: 1698.032 MHz
[ 10.696107] megaraid_sas 0000:03:00.0: unevenspan support : yes
[ 10.703503] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1879e428d19, max_idle_ns: 440795273478 ns
[ 10.722660] megaraid_sas 0000:03:00.0: firmware crash dump : no
[ 10.729750] megaraid_sas 0000:03:00.0: JBOD sequence map : disabled
[ 10.737246] clocksource: Switched to clocksource tsc
[ 10.737284] scsi host0: Avago SAS based MegaRAID driver
[ 10.749986] mpt3sas version 31.100.00.00 loaded
[ 10.750637] scsi 0:0:0:0: Direct-Access ATA TOSHIBA MG03ACA1 FL2H PQ: 0 ANSI: 6
[ 10.755943] ahci 0000:00:11.4: version 3.0
[ 10.765658] ahci 0000:00:11.4: SSS flag set, parallel bus scan disabled
[ 10.768124] scsi 0:0:1:0: Direct-Access ATA TOSHIBA MG03ACA1 FL2H PQ: 0 ANSI: 6
[ 10.783628] ahci 0000:00:11.4: AHCI 0001.0300 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[ 10.793557] ahci 0000:00:11.4: flags: 64bit ncq stag led clo pio slum part ems apst
[ 10.796637] scsi 0:2:0:0: Direct-Access DELL PERC H730P Adp 4.29 PQ: 0 ANSI: 5
[ 10.811775] scsi host1: ahci
[ 10.816851] scsi host2: ahci
[ 10.820662] scsi host3: ahci
[ 10.824497] scsi host4: ahci
[ 10.826814] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 10.828242] ata1: SATA max UDMA/133 abar m2048@0x92211000 port 0x92211100 irq 53
[ 10.831576] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[ 10.834814] scsi 0:0:1:0: Attached scsi generic sg1 type 0
[ 10.837497] sd 0:0:1:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[ 10.843762] ata2: SATA max UDMA/133 abar m2048@0x92211000 port 0x92211180 irq 53
[ 10.843763] ata3: SATA max UDMA/133 abar m2048@0x92211000 port 0x92211200 irq 53
[ 10.843765] ata4: SATA max UDMA/133 abar m2048@0x92211000 port 0x92211280 irq 53
[ 10.853304] scsi 0:2:0:0: Attached scsi generic sg2 type 0
[ 10.860075] ahci 0000:00:1f.2: SSS flag set, parallel bus scan disabled
[ 10.869325] sd 0:2:0:0: [sdc] 58589184000 512-byte logical blocks: (30.0 TB/27.3 TiB)
[ 10.888552] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[ 10.896739] sd 0:2:0:0: [sdc] Write Protect is off
[ 10.903329] ahci 0000:00:1f.2: flags: 64bit ncq stag led clo pio slum part ems apst
[ 10.906149] sd 0:0:0:0: [sda] Write Protect is off
[ 10.906150] sd 0:0:0:0: [sda] Mode Sense: 9b 00 10 08
[ 10.906769] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[ 10.911200] sd 0:2:0:0: [sdc] Mode Sense: 1f 00 00 08
[ 10.911236] sd 0:2:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 10.918417] sd 0:0:1:0: [sdb] Write Protect is off
[ 10.918418] sd 0:0:1:0: [sdb] Mode Sense: 9b 00 10 08
[ 10.919036] sd 0:0:1:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[ 10.936910] scsi host5: ahci
[ 10.990127] sda: sda1 sda2
[ 10.993918] scsi host6: ahci
[ 11.001198] scsi host7: ahci
[ 11.005000] scsi host8: ahci
[ 11.008795] scsi host9: ahci
[ 11.012595] scsi host10: ahci
[ 11.016437] ata5: SATA max UDMA/133 abar m2048@0x92210000 port 0x92210100 irq 54
[ 11.016975] sd 0:2:0:0: [sdc] Attached SCSI disk
[ 11.025591] ata6: SATA max UDMA/133 abar m2048@0x92210000 port 0x92210180 irq 54
[ 11.040376] ata7: SATA max UDMA/133 abar m2048@0x92210000 port 0x92210200 irq 54
[ 11.049535] ata8: SATA max UDMA/133 abar m2048@0x92210000 port 0x92210280 irq 54
[ 11.058683] ata9: SATA max UDMA/133 abar m2048@0x92210000 port 0x92210300 irq 54
[ 11.065090] sd 0:0:0:0: [sda] Attached SCSI disk
[ 11.067831] ata10: SATA max UDMA/133 abar m2048@0x92210000 port 0x92210380 irq 54
[ 11.083543] tun: Universal TUN/TAP device driver, 1.6
[ 11.089906] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.713.36-0 (2014/02/10)
[ 11.094028] sd 0:0:1:0: [sdb] Attached SCSI disk
[ 11.101170] e100: Intel(R) PRO/100 Network Driver, 3.5.24-k2-NAPI
[ 11.113678] e100: Copyright(c) 1999-2006 Intel Corporation
[ 11.120337] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[ 11.128691] e1000: Copyright (c) 1999-2006 Intel Corporation.
[ 11.135639] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[ 11.142634] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[ 11.149789] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 11.158048] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 11.164580] ata1: SATA link down (SStatus 0 SControl 300)
[ 11.218063] igb 0000:01:00.0: added PHC on eth0
[ 11.223630] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 11.231790] igb 0000:01:00.0: eth0: (PCIe:5.0Gb/s:Width x1) d0:94:66:13:a9:a5
[ 11.240325] igb 0000:01:00.0: eth0: PBA No: 106500-000
[ 11.246546] igb 0000:01:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[ 11.308894] igb 0000:01:00.1: added PHC on eth1
[ 11.314438] igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 11.322600] igb 0000:01:00.1: eth1: (PCIe:5.0Gb/s:Width x1) d0:94:66:13:a9:a6
[ 11.331136] igb 0000:01:00.1: eth1: PBA No: 106500-000
[ 11.337355] igb 0000:01:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[ 11.346752] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[ 11.356383] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 11.363554] sky2: driver version 1.30
[ 11.368530] Fusion MPT base driver 3.04.20
[ 11.373587] Copyright (c) 1999-2008 LSI Corporation
[ 11.379523] Fusion MPT SPI Host driver 3.04.20
[ 11.385002] Fusion MPT FC Host driver 3.04.20
[ 11.388575] ata5: SATA link down (SStatus 0 SControl 300)
[ 11.390386] Fusion MPT SAS Host driver 3.04.20
[ 11.402359] Fusion MPT misc device (ioctl) driver 3.04.20
[ 11.408897] mptctl: Registered with Fusion MPT base driver
[ 11.415505] mptctl: /dev/mptctl @ (major,minor=10,220)
[ 11.431883] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 11.439660] ehci-pci: EHCI PCI platform driver
[ 11.445338] ehci-pci 0000:00:1a.0: EHCI Host Controller
[ 11.451659] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1
[ 11.460816] ehci-pci 0000:00:1a.0: debug port 2
[ 11.470274] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported
[ 11.476581] ata2: SATA link down (SStatus 0 SControl 300)
[ 11.478354] ehci-pci 0000:00:1a.0: irq 18, io mem 0x92213000
[ 11.498460] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00
[ 11.505514] hub 1-0:1.0: USB hub found
[ 11.510191] hub 1-0:1.0: 2 ports detected
[ 11.515496] ehci-pci 0000:00:1d.0: EHCI Host Controller
[ 11.521825] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2
[ 11.530989] ehci-pci 0000:00:1d.0: debug port 2
[ 11.540523] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported
[ 11.548599] ehci-pci 0000:00:1d.0: irq 18, io mem 0x92212000
[ 11.561462] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00
[ 11.568495] hub 2-0:1.0: USB hub found
[ 11.573168] hub 2-0:1.0: 2 ports detected
[ 11.578335] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 11.585731] ohci-pci: OHCI PCI platform driver
[ 11.591212] uhci_hcd: USB Universal Host Controller Interface driver
[ 11.598951] xhci_hcd 0000:00:14.0: xHCI Host Controller
[ 11.605281] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 3
[ 11.615540] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x0000000000009810
[ 11.626632] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
[ 11.634987] hub 3-0:1.0: USB hub found
[ 11.639672] hub 3-0:1.0: 15 ports detected
[ 11.646427] xhci_hcd 0000:00:14.0: xHCI Host Controller
[ 11.652751] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 4
[ 11.661914] xhci_hcd 0000:00:14.0: Host supports USB 3.0 SuperSpeed
[ 11.669532] hub 4-0:1.0: USB hub found
[ 11.674209] hub 4-0:1.0: 6 ports detected
[ 11.680100] usbcore: registered new interface driver usb-storage
[ 11.687335] i8042: PNP: No PS/2 controller found.
[ 11.693161] rtc_cmos 00:00: RTC can wake from S4
[ 11.699006] rtc_cmos 00:00: registered as rtc0
[ 11.704472] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[ 11.714060] intel_pstate: Intel P-state driver initializing
[ 11.721168] hidraw: raw HID events driver (C) Jiri Kosina
[ 11.727907] usbcore: registered new interface driver usbhid
[ 11.734616] usbhid: USB HID core driver
[ 11.739486] drop_monitor: Initializing network drop monitor service
[ 11.747050] Key type dns_resolver registered
[ 11.753090] microcode: sig=0x406f1, pf=0x1, revision=0xb00002e
[ 11.760143] microcode: Microcode Update Driver: v2.2.
[ 11.760147] IPI shorthand broadcast: enabled
[ 11.771521] sched_clock: Marking stable (9405247474, 2366262096)->(12073505150, -301995580)
[ 11.781788] registered taskstats version 1
[ 11.786844] Loading compiled-in X.509 certificates
[ 11.793874] Loaded X.509 cert 'Build time autogenerated kernel key: 28883aa8ecc2a860280dd26c8062136547a5af61'
[ 11.797588] ata3: SATA link down (SStatus 0 SControl 300)
[ 11.812372] rtc_cmos 00:00: setting system clock to 2020-10-26T11:21:46 UTC (1603711306)
[ 11.849455] usb 1-1: new high-speed USB device number 2 using ehci-pci
[ 11.897456] usb 2-1: new high-speed USB device number 2 using ehci-pci
[ 11.970454] usb 3-12: new high-speed USB device number 2 using xhci_hcd
[ 11.992970] hub 1-1:1.0: USB hub found
[ 11.997699] hub 1-1:1.0: 6 ports detected
[ 12.032968] hub 2-1:1.0: USB hub found
[ 12.037699] hub 2-1:1.0: 8 ports detected
[ 12.107383] hub 3-12:1.0: USB hub found
[ 12.112229] hub 3-12:1.0: 6 ports detected
[ 12.132579] ata4: SATA link down (SStatus 0 SControl 300)
[ 12.231457] usb 3-13: new full-speed USB device number 3 using xhci_hcd
[ 12.368067] input: Peppercon AG Multidevice as /devices/pci0000:00/0000:00:14.0/usb3/3-13/3-13:1.0/0003:14DD:0002.0001/input/input1
[ 12.433508] hid-generic 0003:14DD:0002.0001: input,hidraw0: USB HID v1.01 Keyboard [Peppercon AG Multidevice] on usb-0000:00:14.0-13/input0
[ 12.444570] ata6: SATA link down (SStatus 0 SControl 300)
[ 12.449094] input: Peppercon AG Multidevice as /devices/pci0000:00/0000:00:14.0/usb3/3-13/3-13:1.1/0003:14DD:0002.0002/input/input2
[ 12.469075] hid-generic 0003:14DD:0002.0002: input,hidraw1: USB HID v1.01 Mouse [Peppercon AG Multidevice] on usb-0000:00:14.0-13/input1
[ 12.764571] ata7: SATA link down (SStatus 0 SControl 300)
[ 13.076547] ata8: SATA link down (SStatus 0 SControl 300)
[ 13.388548] ata9: SATA link down (SStatus 0 SControl 300)
[ 13.700548] ata10: SATA link down (SStatus 0 SControl 300)
[ 13.708367] Freeing unused kernel image memory: 1616K
[ 13.714522] Write protecting the kernel read-only data: 20480k
[ 13.722253] Freeing unused kernel image memory: 2012K
[ 13.728579] Freeing unused kernel image memory: 540K
[ 13.734616] Run /init as init process
[ 14.082002] XFS (sda1): Mounting V5 Filesystem
[ 14.221801] XFS (sda1): Ending clean mount
[ 15.036925] systemd[1]: Inserted module 'autofs4'
[ 15.327286] NET: Registered protocol family 10
[ 15.333101] Segment Routing with IPv6
[ 15.364042] NET: Registered protocol family 1
[ 15.369428] systemd[1]: Inserted module 'unix'
[ 15.511635] systemd[1]: systemd 242 running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[ 15.548499] systemd[1]: Detected architecture x86-64.
[ 15.581724] systemd[1]: Set hostname to <salvadorthegunzerker.molgen.mpg.de>.
[ 16.580225] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[ 16.590967] systemd[1]: Listening on initctl Compatibility Named Pipe.
[ 16.610802] systemd[1]: Listening on RPCbind Server Activation Socket.
[ 16.619797] systemd[1]: Created slice User and Session Slice.
[ 16.686302] systemd[1]: Condition check resulted in Journal Audit Socket being skipped.
[ 16.701162] systemd[1]: Listening on udev Control Socket.
[ 16.708914] systemd[1]: Created slice system-getty.slice.
[ 17.220301] RPC: Registered named UNIX socket transport module.
[ 17.227400] RPC: Registered udp transport module.
[ 17.233136] RPC: Registered tcp transport module.
[ 17.238877] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 17.574665] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 17.616945] xfs filesystem being remounted at / supports timestamps until 2038 (0x7fffffff)
[ 17.740392] systemd-journald[225]: Received request to flush runtime journal from PID 1
[ 20.233966] ipmi_si: IPMI System Interface driver
[ 20.239742] ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS
[ 20.247333] ipmi_platform: ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10
[ 20.256396] ipmi_si: Adding SMBIOS-specified kcs state machine
[ 20.263523] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10
[ 20.374514] ipmi_si dmi-ipmi-si.0: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed.
[ 20.405511] ipmi_si dmi-ipmi-si.0: Using irq 10
[ 20.423272] ipmi_si dmi-ipmi-si.0: IPMI message handler: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20)
[ 20.455210] ipmi_si dmi-ipmi-si.0: IPMI kcs interface initialized
[ 20.532697] iTCO_vendor_support: vendor-support=0
[ 20.647798] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[ 20.654586] iTCO_wdt: Found a Wellsburg TCO device (Version=2, TCOBASE=0x0460)
[ 20.663785] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[ 20.838713] Microsemi PQI Driver (v1.2.8-026)
[ 20.844178] smartpqi 0000:84:00.0: Microsemi Smart Family Controller found
[ 20.994761] smartpqi 0000:84:00.0: Online Firmware Activation enabled
[ 21.002463] smartpqi 0000:84:00.0: Serial Management Protocol enabled
[ 21.010150] smartpqi 0000:84:00.0: New Soft Reset Handshake enabled
[ 21.017798] scsi host11: smartpqi
[ 21.085663] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[ 21.095137] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 21.264074] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[ 21.276636] ixgbe 0000:04:00.0: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ 21.286607] ixgbe 0000:04:00.0: MAC: 2, PHY: 20, SFP+: 5, PBA No: G73129-009
[ 21.294968] ixgbe 0000:04:00.0: b4:96:91:0e:5a:a0
[ 21.304170] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[ 21.312824] libphy: ixgbe-mdio: probed
[ 21.469795] ixgbe 0000:04:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[ 21.471830] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094a6000d Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 21.482359] ixgbe 0000:04:00.1: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ 21.505650] ixgbe 0000:04:00.1: MAC: 2, PHY: 20, SFP+: 6, PBA No: G73129-009
[ 21.506857] scsi 11:0:0:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 21.514018] ixgbe 0000:04:00.1: b4:96:91:0e:5a:a2
[ 21.525547] scsi 11:0:0:0: Attached scsi generic sg3 type 0
[ 21.527489] sd 11:0:0:0: [sdd] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 21.527491] sd 11:0:0:0: [sdd] 4096-byte physical blocks
[ 21.529290] sd 11:0:0:0: [sdd] Write Protect is off
[ 21.529292] sd 11:0:0:0: [sdd] Mode Sense: db 00 10 08
[ 21.532648] sd 11:0:0:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 21.533274] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection
[ 21.533305] libphy: ixgbe-mdio: probed
[ 21.536841] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094a35ced Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 21.564461] sd 11:0:0:0: [sdd] Attached SCSI disk
[ 21.621381] scsi 11:0:1:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 21.632864] sd 11:0:1:0: Attached scsi generic sg4 type 0
[ 21.634781] sd 11:0:1:0: [sde] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 21.639441] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500949d0d5d Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 21.649132] sd 11:0:1:0: [sde] 4096-byte physical blocks
[ 21.650932] sd 11:0:1:0: [sde] Write Protect is off
[ 21.675033] sd 11:0:1:0: [sde] Mode Sense: db 00 10 08
[ 21.678481] sd 11:0:1:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 21.737870] sd 11:0:1:0: [sde] Attached SCSI disk
[ 21.745827] scsi 11:0:2:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 21.756127] ixgbe 0000:82:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[ 21.768710] ixgbe 0000:82:00.0: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ 21.778696] ixgbe 0000:82:00.0: MAC: 2, PHY: 20, SFP+: 5, PBA No: G73129-009
[ 21.787075] ixgbe 0000:82:00.0: b4:96:91:0e:58:d8
[ 21.794107] sd 11:0:2:0: Attached scsi generic sg5 type 0
[ 21.795982] sd 11:0:2:0: [sdf] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 21.800683] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094a36edd Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 21.810402] sd 11:0:2:0: [sdf] 4096-byte physical blocks
[ 21.826299] ixgbe 0000:82:00.0: Intel(R) 10 Gigabit Network Connection
[ 21.832225] sd 11:0:2:0: [sdf] Write Protect is off
[ 21.838300] libphy: ixgbe-mdio: probed
[ 21.844243] sd 11:0:2:0: [sdf] Mode Sense: db 00 10 08
[ 21.849319] scsi 11:0:3:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 21.852779] sd 11:0:2:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 21.880369] sd 11:0:3:0: Attached scsi generic sg6 type 0
[ 21.882276] sd 11:0:3:0: [sdg] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 21.886942] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094a52c09 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 21.896648] sd 11:0:3:0: [sdg] 4096-byte physical blocks
[ 21.898006] sd 11:0:2:0: [sdf] Attached SCSI disk
[ 21.898437] sd 11:0:3:0: [sdg] Write Protect is off
[ 21.898439] sd 11:0:3:0: [sdg] Mode Sense: db 00 10 08
[ 21.901793] sd 11:0:3:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 21.919944] sd 11:0:3:0: [sdg] Attached SCSI disk
[ 21.921276] scsi 11:0:4:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 21.922716] sd 11:0:4:0: Attached scsi generic sg7 type 0
[ 21.924570] sd 11:0:4:0: [sdh] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 21.924572] sd 11:0:4:0: [sdh] 4096-byte physical blocks
[ 21.926379] sd 11:0:4:0: [sdh] Write Protect is off
[ 21.926381] sd 11:0:4:0: [sdh] Mode Sense: db 00 10 08
[ 21.929750] sd 11:0:4:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 21.938870] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094a5181d Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.016532] scsi 11:0:5:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.016889] sd 11:0:4:0: [sdh] Attached SCSI disk
[ 22.028018] sd 11:0:5:0: Attached scsi generic sg8 type 0
[ 22.029900] sd 11:0:5:0: [sdi] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.029902] sd 11:0:5:0: [sdi] 4096-byte physical blocks
[ 22.031697] sd 11:0:5:0: [sdi] Write Protect is off
[ 22.031698] sd 11:0:5:0: [sdi] Mode Sense: db 00 10 08
[ 22.035066] sd 11:0:5:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.038906] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094aedab1 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.066758] sd 11:0:5:0: [sdi] Attached SCSI disk
[ 22.096258] ixgbe 0000:82:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[ 22.108837] ixgbe 0000:82:00.1: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)
[ 22.118897] ixgbe 0000:82:00.1: MAC: 2, PHY: 20, SFP+: 6, PBA No: G73129-009
[ 22.127266] ixgbe 0000:82:00.1: b4:96:91:0e:58:da
[ 22.136562] ixgbe 0000:82:00.1: Intel(R) 10 Gigabit Network Connection
[ 22.144398] libphy: ixgbe-mdio: probed
[ 22.149089] scsi 11:0:6:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.175440] sd 11:0:6:0: Attached scsi generic sg9 type 0
[ 22.177272] sd 11:0:6:0: [sdj] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.182026] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094a34845 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.191720] sd 11:0:6:0: [sdj] 4096-byte physical blocks
[ 22.193500] sd 11:0:6:0: [sdj] Write Protect is off
[ 22.215666] scsi 11:0:7:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.217647] sd 11:0:6:0: [sdj] Mode Sense: db 00 10 08
[ 22.232293] sd 11:0:6:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.245098] sd 11:0:7:0: Attached scsi generic sg10 type 0
[ 22.251341] sd 11:0:7:0: [sdk] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.256759] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094969f09 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.261463] sd 11:0:7:0: [sdk] 4096-byte physical blocks
[ 22.264923] sd 11:0:7:0: [sdk] Write Protect is off
[ 22.287376] sd 11:0:7:0: [sdk] Mode Sense: db 00 10 08
[ 22.294549] sd 11:0:7:0: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.296773] scsi 11:0:8:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.316732] sd 11:0:8:0: Attached scsi generic sg11 type 0
[ 22.318647] sd 11:0:8:0: [sdl] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.323403] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094a444f5 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.330816] sd 11:0:6:0: [sdj] Attached SCSI disk
[ 22.333124] sd 11:0:8:0: [sdl] 4096-byte physical blocks
[ 22.333271] sd 11:0:7:0: [sdk] Attached SCSI disk
[ 22.334911] sd 11:0:8:0: [sdl] Write Protect is off
[ 22.334912] sd 11:0:8:0: [sdl] Mode Sense: db 00 10 08
[ 22.338262] sd 11:0:8:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.356398] sd 11:0:8:0: [sdl] Attached SCSI disk
[ 22.361606] scsi 11:0:9:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.398282] sd 11:0:9:0: Attached scsi generic sg12 type 0
[ 22.400167] sd 11:0:9:0: [sdm] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.404946] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094a36905 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.414645] sd 11:0:9:0: [sdm] 4096-byte physical blocks
[ 22.416444] sd 11:0:9:0: [sdm] Write Protect is off
[ 22.450891] sd 11:0:9:0: [sdm] Mode Sense: db 00 10 08
[ 22.451277] scsi 11:0:10:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.454344] sd 11:0:9:0: [sdm] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.462845] sd 11:0:10:0: Attached scsi generic sg13 type 0
[ 22.464703] sd 11:0:10:0: [sdn] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.464705] sd 11:0:10:0: [sdn] 4096-byte physical blocks
[ 22.466492] sd 11:0:10:0: [sdn] Write Protect is off
[ 22.466494] sd 11:0:10:0: [sdn] Mode Sense: db 00 10 08
[ 22.469843] sd 11:0:10:0: [sdn] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.487916] sd 11:0:10:0: [sdn] Attached SCSI disk
[ 22.488644] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500949d4ee1 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.490025] sd 11:0:9:0: [sdm] Attached SCSI disk
[ 22.544866] scsi 11:0:11:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.556438] sd 11:0:11:0: Attached scsi generic sg14 type 0
[ 22.558298] sd 11:0:11:0: [sdo] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.563207] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094af0e99 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.572985] sd 11:0:11:0: [sdo] 4096-byte physical blocks
[ 22.594839] sd 11:0:11:0: [sdo] Write Protect is off
[ 22.599896] scsi 11:0:12:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.600866] sd 11:0:11:0: [sdo] Mode Sense: db 00 10 08
[ 22.613556] EDAC sbridge: Seeking for: PCI ID 8086:6fa0
[ 22.613582] EDAC sbridge: Seeking for: PCI ID 8086:6fa0
[ 22.613605] EDAC sbridge: Seeking for: PCI ID 8086:6fa0
[ 22.613615] EDAC sbridge: Seeking for: PCI ID 8086:6f60
[ 22.613633] EDAC sbridge: Seeking for: PCI ID 8086:6fa8
[ 22.613641] EDAC sbridge: Seeking for: PCI ID 8086:6fa8
[ 22.613651] EDAC sbridge: Seeking for: PCI ID 8086:6fa8
[ 22.613655] EDAC sbridge: Seeking for: PCI ID 8086:6f71
[ 22.613664] EDAC sbridge: Seeking for: PCI ID 8086:6f71
[ 22.613674] EDAC sbridge: Seeking for: PCI ID 8086:6f71
[ 22.613678] EDAC sbridge: Seeking for: PCI ID 8086:6faa
[ 22.613686] EDAC sbridge: Seeking for: PCI ID 8086:6faa
[ 22.613696] EDAC sbridge: Seeking for: PCI ID 8086:6faa
[ 22.613700] EDAC sbridge: Seeking for: PCI ID 8086:6fab
[ 22.613709] EDAC sbridge: Seeking for: PCI ID 8086:6fab
[ 22.613719] EDAC sbridge: Seeking for: PCI ID 8086:6fab
[ 22.613723] EDAC sbridge: Seeking for: PCI ID 8086:6fac
[ 22.613732] EDAC sbridge: Seeking for: PCI ID 8086:6fac
[ 22.613742] EDAC sbridge: Seeking for: PCI ID 8086:6fac
[ 22.613745] EDAC sbridge: Seeking for: PCI ID 8086:6fad
[ 22.613754] EDAC sbridge: Seeking for: PCI ID 8086:6fad
[ 22.613764] EDAC sbridge: Seeking for: PCI ID 8086:6fad
[ 22.613768] EDAC sbridge: Seeking for: PCI ID 8086:6f68
[ 22.613777] EDAC sbridge: Seeking for: PCI ID 8086:6f79
[ 22.613795] EDAC sbridge: Seeking for: PCI ID 8086:6f6a
[ 22.613812] EDAC sbridge: Seeking for: PCI ID 8086:6f6b
[ 22.613830] EDAC sbridge: Seeking for: PCI ID 8086:6f6c
[ 22.613848] EDAC sbridge: Seeking for: PCI ID 8086:6f6d
[ 22.613866] EDAC sbridge: Seeking for: PCI ID 8086:6ffc
[ 22.613874] EDAC sbridge: Seeking for: PCI ID 8086:6ffc
[ 22.613883] EDAC sbridge: Seeking for: PCI ID 8086:6ffc
[ 22.613889] EDAC sbridge: Seeking for: PCI ID 8086:6ffd
[ 22.613896] EDAC sbridge: Seeking for: PCI ID 8086:6ffd
[ 22.613906] EDAC sbridge: Seeking for: PCI ID 8086:6ffd
[ 22.613911] EDAC sbridge: Seeking for: PCI ID 8086:6faf
[ 22.613920] EDAC sbridge: Seeking for: PCI ID 8086:6faf
[ 22.613930] EDAC sbridge: Seeking for: PCI ID 8086:6faf
[ 22.614049] EDAC MC0: Giving out device to module sb_edac controller Broadwell SrcID#0_Ha#0: DEV 0000:7f:12.0 (INTERRUPT)
[ 22.614656] sd 11:0:12:0: Attached scsi generic sg15 type 0
[ 22.616526] sd 11:0:12:0: [sdp] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.616528] sd 11:0:12:0: [sdp] 4096-byte physical blocks
[ 22.618317] sd 11:0:12:0: [sdp] Write Protect is off
[ 22.618319] sd 11:0:12:0: [sdp] Mode Sense: db 00 10 08
[ 22.621674] sd 11:0:12:0: [sdp] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.627211] sd 11:0:11:0: [sdo] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.627299] EDAC MC1: Giving out device to module sb_edac controller Broadwell SrcID#1_Ha#0: DEV 0000:ff:12.0 (INTERRUPT)
[ 22.627300] EDAC sbridge: Ver: 1.1.2
[ 22.633978] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500949cdf65 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.639716] sd 11:0:12:0: [sdp] Attached SCSI disk
[ 22.662028] sd 11:0:11:0: [sdo] Attached SCSI disk
[ 22.678906] scsi 11:0:13:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.732388] sd 11:0:13:0: Attached scsi generic sg16 type 0
[ 22.734267] sd 11:0:13:0: [sdq] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.739148] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094a4d8b1 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.748971] sd 11:0:13:0: [sdq] 4096-byte physical blocks
[ 22.769413] mgag200 0000:09:00.0: remove_conflicting_pci_framebuffers: bar 0: 0x90000000 -> 0x90ffffff
[ 22.770904] sd 11:0:13:0: [sdq] Write Protect is off
[ 22.775979] scsi 11:0:14:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.777390] sd 11:0:14:0: Attached scsi generic sg17 type 0
[ 22.777429] smartpqi 0000:84:00.0: added 11:0:-:- 5000c50094aeda21 Direct-Access SEAGATE ST8000NM0075 AIO+ qd=64
[ 22.779268] sd 11:0:14:0: [sdr] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.779270] sd 11:0:14:0: [sdr] 4096-byte physical blocks
[ 22.780712] mgag200 0000:09:00.0: remove_conflicting_pci_framebuffers: bar 1: 0x91800000 -> 0x91803fff
[ 22.780713] mgag200 0000:09:00.0: remove_conflicting_pci_framebuffers: bar 2: 0x91000000 -> 0x917fffff
[ 22.780720] mgag200 0000:09:00.0: vgaarb: deactivate vga console
[ 22.781068] sd 11:0:14:0: [sdr] Write Protect is off
[ 22.785205] scsi 11:0:15:0: Direct-Access SEAGATE ST8000NM0075 E003 PQ: 0 ANSI: 6
[ 22.786631] sd 11:0:15:0: Attached scsi generic sg18 type 0
[ 22.786667] smartpqi 0000:84:00.0: added 11:0:-:- 5000c5009557ec8d Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 22.786785] sd 11:0:13:0: [sdq] Mode Sense: db 00 10 08
[ 22.788504] sd 11:0:15:0: [sds] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.788506] sd 11:0:15:0: [sds] 4096-byte physical blocks
[ 22.790293] sd 11:0:15:0: [sds] Write Protect is off
[ 22.790295] sd 11:0:15:0: [sds] Mode Sense: db 00 10 08
[ 22.793644] sd 11:0:15:0: [sds] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.796922] sd 11:0:14:0: [sdr] Mode Sense: db 00 10 08
[ 22.801438] scsi 11:0:16:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 22.802805] scsi 11:0:16:0: Attached scsi generic sg19 type 0
[ 22.802844] smartpqi 0000:84:00.0: added 11:0:-:- 5000c5009564634d Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 22.804669] sd 11:0:16:0: [sdt] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.807556] sd 11:0:13:0: [sdq] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.811654] sd 11:0:15:0: [sds] Attached SCSI disk
[ 22.813210] scsi 11:0:17:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 22.814589] sd 11:0:17:0: Attached scsi generic sg20 type 0
[ 22.814625] smartpqi 0000:84:00.0: added 11:0:-:- 5000c5009557ea4d Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 22.815623] scsi 11:0:18:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 22.816413] sd 11:0:17:0: [sdu] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.817001] sd 11:0:18:0: Attached scsi generic sg21 type 0
[ 22.817041] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500956d30dd Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 22.817135] sd 11:0:17:0: [sdu] Write Protect is off
[ 22.817136] sd 11:0:17:0: [sdu] Mode Sense: db 00 10 08
[ 22.818419] sd 11:0:17:0: [sdu] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.820634] sd 11:0:14:0: [sdr] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.825590] sd 11:0:13:0: [sdq] Attached SCSI disk
[ 22.827384] sd 11:0:18:0: [sdv] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.827720] sd 11:0:16:0: [sdt] Write Protect is off
[ 22.827722] sd 11:0:16:0: [sdt] Mode Sense: db 00 10 08
[ 22.828978] sd 11:0:16:0: [sdt] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.831041] scsi 11:0:19:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 22.832431] sd 11:0:19:0: Attached scsi generic sg22 type 0
[ 22.832473] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a61e4529 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 22.834228] sd 11:0:19:0: [sdw] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.838747] sd 11:0:14:0: [sdr] Attached SCSI disk
[ 22.840546] sd 11:0:16:0: [sdt] Attached SCSI disk
[ 22.845031] sd 11:0:17:0: [sdu] Attached SCSI disk
[ 22.845407] scsi 11:0:20:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 22.845551] sd 11:0:18:0: [sdv] Write Protect is off
[ 22.845553] sd 11:0:18:0: [sdv] Mode Sense: db 00 10 08
[ 22.846778] sd 11:0:20:0: Attached scsi generic sg23 type 0
[ 22.846813] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500955c2ee1 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 22.846825] sd 11:0:18:0: [sdv] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.848645] sd 11:0:20:0: [sdx] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.849311] sd 11:0:20:0: [sdx] Write Protect is off
[ 22.849313] sd 11:0:20:0: [sdx] Mode Sense: db 00 10 08
[ 22.850556] sd 11:0:20:0: [sdx] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.856854] sd 11:0:19:0: [sdw] Write Protect is off
[ 22.858360] sd 11:0:18:0: [sdv] Attached SCSI disk
[ 22.862006] sd 11:0:20:0: [sdx] Attached SCSI disk
[ 22.862979] scsi 11:0:21:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 22.864373] sd 11:0:21:0: Attached scsi generic sg24 type 0
[ 22.866177] sd 11:0:21:0: [sdy] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 22.866844] sd 11:0:21:0: [sdy] Write Protect is off
[ 22.866846] sd 11:0:21:0: [sdy] Mode Sense: db 00 10 08
[ 22.868087] sd 11:0:21:0: [sdy] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.869481] sd 11:0:19:0: [sdw] Mode Sense: db 00 10 08
[ 22.870763] sd 11:0:19:0: [sdw] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 22.879535] sd 11:0:21:0: [sdy] Attached SCSI disk
[ 22.879666] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a61e5945 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 22.897823] sd 11:0:19:0: [sdw] Attached SCSI disk
[ 22.909530] scsi 11:0:22:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.368088] Console: switching to colour dummy device 80x25
[ 23.373996] sd 11:0:22:0: Attached scsi generic sg25 type 0
[ 23.375829] sd 11:0:22:0: [sdz] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.380579] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500adbde28d Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.390169] sd 11:0:22:0: [sdz] Write Protect is off
[ 23.407648] sd 11:0:22:0: [sdz] Mode Sense: db 00 10 08
[ 23.408936] sd 11:0:22:0: [sdz] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.417379] scsi 11:0:23:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.418774] sd 11:0:23:0: Attached scsi generic sg26 type 0
[ 23.420355] sd 11:0:22:0: [sdz] Attached SCSI disk
[ 23.420582] sd 11:0:23:0: [sdaa] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.421251] sd 11:0:23:0: [sdaa] Write Protect is off
[ 23.421252] sd 11:0:23:0: [sdaa] Mode Sense: db 00 10 08
[ 23.422487] sd 11:0:23:0: [sdaa] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.427548] [TTM] Zone kernel: Available graphics memory: 49351580 KiB
[ 23.433898] sd 11:0:23:0: [sdaa] Attached SCSI disk
[ 23.436721] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500955c35f5 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.442904] [TTM] Zone dma32: Available graphics memory: 2097152 KiB
[ 23.442906] [TTM] Initializing pool allocator
[ 23.459815] scsi 11:0:24:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.462939] [TTM] Initializing DMA pool allocator
[ 23.474131] sd 11:0:24:0: Attached scsi generic sg27 type 0
[ 23.475964] sd 11:0:24:0: [sdab] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.476629] sd 11:0:24:0: [sdab] Write Protect is off
[ 23.476631] sd 11:0:24:0: [sdab] Mode Sense: db 00 10 08
[ 23.477863] sd 11:0:24:0: [sdab] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.489204] sd 11:0:24:0: [sdab] Attached SCSI disk
[ 23.498241] smartpqi 0000:84:00.0: added 11:0:-:- 5000c5009565b809 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.573609] fbcon: mgag200drmfb (fb0) is primary device
[ 23.585959] scsi 11:0:25:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.587343] sd 11:0:25:0: Attached scsi generic sg28 type 0
[ 23.587380] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500956d422d Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.589161] sd 11:0:25:0: [sdac] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.589830] sd 11:0:25:0: [sdac] Write Protect is off
[ 23.589831] sd 11:0:25:0: [sdac] Mode Sense: db 00 10 08
[ 23.591078] sd 11:0:25:0: [sdac] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.602439] sd 11:0:25:0: [sdac] Attached SCSI disk
[ 23.604060] scsi 11:0:26:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.605441] sd 11:0:26:0: Attached scsi generic sg29 type 0
[ 23.605484] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500955ae315 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.607275] sd 11:0:26:0: [sdad] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.607947] sd 11:0:26:0: [sdad] Write Protect is off
[ 23.607949] sd 11:0:26:0: [sdad] Mode Sense: db 00 10 08
[ 23.609186] sd 11:0:26:0: [sdad] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.611536] scsi 11:0:27:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.612910] sd 11:0:27:0: Attached scsi generic sg30 type 0
[ 23.612945] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500954c439d Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.614735] sd 11:0:27:0: [sdae] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.615403] sd 11:0:27:0: [sdae] Write Protect is off
[ 23.615404] sd 11:0:27:0: [sdae] Mode Sense: db 00 10 08
[ 23.616640] sd 11:0:27:0: [sdae] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.620648] sd 11:0:26:0: [sdad] Attached SCSI disk
[ 23.624903] scsi 11:0:28:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.626282] sd 11:0:28:0: Attached scsi generic sg31 type 0
[ 23.626319] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500955c2765 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.628045] sd 11:0:27:0: [sdae] Attached SCSI disk
[ 23.628108] sd 11:0:28:0: [sdaf] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.628779] sd 11:0:28:0: [sdaf] Write Protect is off
[ 23.628780] sd 11:0:28:0: [sdaf] Mode Sense: db 00 10 08
[ 23.630018] sd 11:0:28:0: [sdaf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.636656] scsi 11:0:29:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.637462] Console: switching to colour frame buffer device 128x48
[ 23.638053] sd 11:0:29:0: Attached scsi generic sg32 type 0
[ 23.638092] smartpqi 0000:84:00.0: added 11:0:-:- 5000c5009565a989 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.639895] sd 11:0:29:0: [sdag] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.641462] sd 11:0:28:0: [sdaf] Attached SCSI disk
[ 23.645445] mgag200 0000:09:00.0: fb0: mgag200drmfb frame buffer device
[ 23.650445] scsi 11:0:30:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.651822] sd 11:0:30:0: Attached scsi generic sg33 type 0
[ 23.651858] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500954def99 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.653616] sd 11:0:30:0: [sdah] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.654283] sd 11:0:30:0: [sdah] Write Protect is off
[ 23.654284] sd 11:0:30:0: [sdah] Mode Sense: db 00 10 08
[ 23.655402] sd 11:0:29:0: [sdag] Write Protect is off
[ 23.655527] sd 11:0:30:0: [sdah] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.666849] sd 11:0:30:0: [sdah] Attached SCSI disk
[ 23.673683] sd 11:0:29:0: [sdag] Mode Sense: db 00 10 08
[ 23.683148] scsi 11:0:31:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 23.689781] sd 11:0:29:0: [sdag] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.690463] [drm] Initialized mgag200 1.0.0 20110418 for 0000:09:00.0 on minor 0
[ 23.700095] sd 11:0:31:0: Attached scsi generic sg34 type 0
[ 23.701894] sd 11:0:31:0: [sdai] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 23.702561] sd 11:0:31:0: [sdai] Write Protect is off
[ 23.702563] sd 11:0:31:0: [sdai] Mode Sense: db 00 10 08
[ 23.703805] sd 11:0:31:0: [sdai] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 23.713927] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a47e5 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 23.715245] sd 11:0:29:0: [sdag] Attached SCSI disk
[ 23.730734] sd 11:0:31:0: [sdai] Attached SCSI disk
[ 23.747641] scsi 11:0:32:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 24.159509] sd 11:0:32:0: Attached scsi generic sg35 type 0
[ 24.161327] sd 11:0:32:0: [sdaj] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 24.167096] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a412d Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 24.178102] sd 11:0:32:0: [sdaj] Write Protect is off
[ 24.198578] sd 11:0:32:0: [sdaj] Mode Sense: db 00 10 08
[ 24.199873] sd 11:0:32:0: [sdaj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 24.222963] sd 11:0:32:0: [sdaj] Attached SCSI disk
[ 24.252303] scsi 11:0:33:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 24.281229] sd 11:0:33:0: Attached scsi generic sg36 type 0
[ 24.283041] sd 11:0:33:0: [sdak] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 24.288813] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a35f5 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 24.299795] sd 11:0:33:0: [sdak] Write Protect is off
[ 24.320280] sd 11:0:33:0: [sdak] Mode Sense: db 00 10 08
[ 24.321581] sd 11:0:33:0: [sdak] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 24.344822] sd 11:0:33:0: [sdak] Attached SCSI disk
[ 24.365788] scsi 11:0:34:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 24.394632] sd 11:0:34:0: Attached scsi generic sg37 type 0
[ 24.396460] sd 11:0:34:0: [sdal] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 24.413528] sd 11:0:34:0: [sdal] Write Protect is off
[ 24.419591] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a3be9 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 24.420725] sd 11:0:34:0: [sdal] Mode Sense: db 00 10 08
[ 24.435949] sd 11:0:34:0: [sdal] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 24.459175] sd 11:0:34:0: [sdal] Attached SCSI disk
[ 24.487135] scsi 11:0:35:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 24.516003] sd 11:0:35:0: Attached scsi generic sg38 type 0
[ 24.517828] sd 11:0:35:0: [sdam] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 24.523594] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a4799 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 24.534597] sd 11:0:35:0: [sdam] Write Protect is off
[ 24.555089] sd 11:0:35:0: [sdam] Mode Sense: db 00 10 08
[ 24.556401] sd 11:0:35:0: [sdam] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 24.579517] sd 11:0:35:0: [sdam] Attached SCSI disk
[ 24.600635] scsi 11:0:36:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 24.629516] sd 11:0:36:0: Attached scsi generic sg39 type 0
[ 24.631324] sd 11:0:36:0: [sdan] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 24.637120] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a42c1 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 24.648108] sd 11:0:36:0: [sdan] Write Protect is off
[ 24.668329] sd 11:0:36:0: [sdan] Mode Sense: db 00 10 08
[ 24.669639] sd 11:0:36:0: [sdan] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 24.692891] sd 11:0:36:0: [sdan] Attached SCSI disk
[ 24.720819] scsi 11:0:37:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 24.749705] sd 11:0:37:0: Attached scsi generic sg40 type 0
[ 24.751514] sd 11:0:37:0: [sdao] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 24.757340] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a3965 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 24.768383] sd 11:0:37:0: [sdao] Write Protect is off
[ 24.789027] sd 11:0:37:0: [sdao] Mode Sense: db 00 10 08
[ 24.790335] sd 11:0:37:0: [sdao] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 24.813425] sd 11:0:37:0: [sdao] Attached SCSI disk
[ 24.834460] scsi 11:0:38:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 24.863381] sd 11:0:38:0: Attached scsi generic sg41 type 0
[ 24.865175] sd 11:0:38:0: [sdap] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 24.871022] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a3fd9 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 24.882068] sd 11:0:38:0: [sdap] Write Protect is off
[ 24.902753] sd 11:0:38:0: [sdap] Mode Sense: db 00 10 08
[ 24.904041] sd 11:0:38:0: [sdap] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 24.927194] sd 11:0:38:0: [sdap] Attached SCSI disk
[ 24.948370] scsi 11:0:39:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 24.977500] sd 11:0:39:0: Attached scsi generic sg42 type 0
[ 24.979311] sd 11:0:39:0: [sdaq] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 24.996511] sd 11:0:39:0: [sdaq] Write Protect is off
[ 25.002657] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a3b21 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 25.003787] sd 11:0:39:0: [sdaq] Mode Sense: db 00 10 08
[ 25.019090] sd 11:0:39:0: [sdaq] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 25.042327] sd 11:0:39:0: [sdaq] Attached SCSI disk
[ 25.053240] scsi 11:0:40:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 25.082317] sd 11:0:40:0: Attached scsi generic sg43 type 0
[ 25.084119] sd 11:0:40:0: [sdar] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 25.089965] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500ae054a91 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 25.101059] sd 11:0:40:0: [sdar] Write Protect is off
[ 25.121728] sd 11:0:40:0: [sdar] Mode Sense: db 00 10 08
[ 25.123031] sd 11:0:40:0: [sdar] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 25.146420] sd 11:0:40:0: [sdar] Attached SCSI disk
[ 25.167715] scsi 11:0:41:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 25.196818] sd 11:0:41:0: Attached scsi generic sg44 type 0
[ 25.198635] sd 11:0:41:0: [sdas] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 25.204486] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a3e29 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 25.215571] sd 11:0:41:0: [sdas] Write Protect is off
[ 25.236301] sd 11:0:41:0: [sdas] Mode Sense: db 00 10 08
[ 25.237618] sd 11:0:41:0: [sdas] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 25.261020] sd 11:0:41:0: [sdas] Attached SCSI disk
[ 25.282100] scsi 11:0:42:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 25.311132] sd 11:0:42:0: Attached scsi generic sg45 type 0
[ 25.312936] sd 11:0:42:0: [sdat] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 25.318796] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a3805 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 25.329876] sd 11:0:42:0: [sdat] Write Protect is off
[ 25.350594] sd 11:0:42:0: [sdat] Mode Sense: db 00 10 08
[ 25.351898] sd 11:0:42:0: [sdat] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 25.374927] sd 11:0:42:0: [sdat] Attached SCSI disk
[ 25.396122] scsi 11:0:43:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 25.425120] sd 11:0:43:0: Attached scsi generic sg46 type 0
[ 25.426931] sd 11:0:43:0: [sdau] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 25.444150] sd 11:0:43:0: [sdau] Write Protect is off
[ 25.450181] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a685f0a5 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 25.451492] sd 11:0:43:0: [sdau] Mode Sense: db 00 10 08
[ 25.452762] sd 11:0:43:0: [sdau] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 25.488821] sd 11:0:43:0: [sdau] Attached SCSI disk
[ 25.498526] scsi 11:0:44:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 25.527503] sd 11:0:44:0: Attached scsi generic sg47 type 0
[ 25.529307] sd 11:0:44:0: [sdav] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 25.535163] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a3d1d Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 25.546247] sd 11:0:44:0: [sdav] Write Protect is off
[ 25.566941] sd 11:0:44:0: [sdav] Mode Sense: db 00 10 08
[ 25.568236] sd 11:0:44:0: [sdav] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 25.591431] sd 11:0:44:0: [sdav] Attached SCSI disk
[ 25.612492] scsi 11:0:45:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 25.658492] sd 11:0:45:0: Attached scsi generic sg48 type 0
[ 25.660295] sd 11:0:45:0: [sdaw] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 25.677469] sd 11:0:45:0: [sdaw] Write Protect is off
[ 25.677471] sd 11:0:45:0: [sdaw] Mode Sense: db 00 10 08
[ 25.678721] sd 11:0:45:0: [sdaw] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 25.683539] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a41d5 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 25.719043] scsi 11:0:46:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 25.723346] sd 11:0:45:0: [sdaw] Attached SCSI disk
[ 25.764974] sd 11:0:46:0: Attached scsi generic sg49 type 0
[ 25.766787] sd 11:0:46:0: [sdax] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 25.783976] sd 11:0:46:0: [sdax] Write Protect is off
[ 25.789996] smartpqi 0000:84:00.0: added 11:0:-:- 5000c500a68a4189 Direct-Access SEAGATE ST8000NM0065 AIO+ qd=64
[ 25.791251] sd 11:0:46:0: [sdax] Mode Sense: db 00 10 08
[ 25.806550] sd 11:0:46:0: [sdax] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 25.829598] sd 11:0:46:0: [sdax] Attached SCSI disk
[ 25.840410] scsi 11:0:47:0: Direct-Access SEAGATE ST8000NM0065 K004 PQ: 0 ANSI: 6
[ 25.869499] sd 11:0:47:0: Attached scsi generic sg50 type 0
[ 25.871323] sd 11:0:47:0: [sday] 1953506646 4096-byte logical blocks: (8.00 TB/7.28 TiB)
[ 25.877145] smartpqi 0000:84:00.0: added 11:0:-:- 50015b21401203bd Enclosure AIC 12G 3U16SAS3swap AIO-
[ 25.888210] sd 11:0:47:0: [sday] Write Protect is off
[ 25.907806] sd 11:0:47:0: [sday] Mode Sense: db 00 10 08
[ 25.909094] sd 11:0:47:0: [sday] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 25.932239] sd 11:0:47:0: [sday] Attached SCSI disk
[ 25.953230] scsi 11:0:48:0: Enclosure AIC 12G 3U16SAS3swap 0c01 PQ: 0 ANSI: 5
[ 25.981766] ses 11:0:48:0: Attached Enclosure device
[ 25.988724] ses 11:0:48:0: Attached scsi generic sg51 type 13
[ 26.033172] smartpqi 0000:84:00.0: added 11:0:-:- 50015b214011eb7d Enclosure AIC 12G 3U16SAS3swap AIO-
[ 26.064040] scsi 11:0:49:0: Enclosure AIC 12G 3U16SAS3swap 0c01 PQ: 0 ANSI: 5
[ 26.092467] ses 11:0:49:0: Attached Enclosure device
[ 26.099401] ses 11:0:49:0: Attached scsi generic sg52 type 13
[ 26.143794] smartpqi 0000:84:00.0: added 11:0:-:- 50015b214011ffbd Enclosure AIC 12G 3U16SAS3swap AIO-
[ 26.174638] scsi 11:0:50:0: Enclosure AIC 12G 3U16SAS3swap 0c01 PQ: 0 ANSI: 5
[ 26.203221] ses 11:0:50:0: Attached Enclosure device
[ 26.210156] ses 11:0:50:0: Attached scsi generic sg53 type 13
[ 26.254750] smartpqi 0000:84:00.0: added 11:0:-:- 50000d1e0018a818 Enclosure Adaptec Smart Adapter AIO-
[ 26.286604] scsi 11:0:51:0: Enclosure Adaptec Smart Adapter 2.62 PQ: 0 ANSI: 5
[ 26.314756] ses 11:0:51:0: Attached Enclosure device
[ 26.321684] ses 11:0:51:0: Attached scsi generic sg54 type 13
[ 26.364552] smartpqi 0000:84:00.0: added 11:2:0:0 0000000000000000 RAID Adaptec 1100-8e
[ 26.395184] scsi 11:2:0:0: RAID Adaptec 1100-8e 2.62 PQ: 0 ANSI: 5
[ 26.423224] scsi 11:2:0:0: Attached scsi generic sg55 type 12
[ 27.432392] igb 0000:01:00.0 net00: renamed from eth0
[ 27.495900] igb 0000:01:00.1 net01: renamed from eth1
[ 27.532748] ixgbe 0000:04:00.0 net02: renamed from eth2
[ 27.567702] ixgbe 0000:04:00.1 net03: renamed from eth3
[ 27.599774] ixgbe 0000:82:00.0 net04: renamed from eth4
[ 27.630874] ixgbe 0000:82:00.1 net05: renamed from eth5
[ 29.829764] ixgbe 0000:04:00.0: registered PHC device on net02
[ 30.009655] ixgbe 0000:04:00.0 net02: detected SFP+: 5
[ 30.263869] ixgbe 0000:04:00.0 net02: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 30.290561] md: md0 stopped.
[ 30.304028] md/raid:md0: device sdu operational as raid disk 0
[ 30.311351] md/raid:md0: device sdai operational as raid disk 15
[ 30.318824] md/raid:md0: device sdaa operational as raid disk 14
[ 30.326280] md/raid:md0: device sdw operational as raid disk 13
[ 30.333643] md/raid:md0: device sdah operational as raid disk 12
[ 30.341102] md/raid:md0: device sdae operational as raid disk 11
[ 30.348558] md/raid:md0: device sdad operational as raid disk 10
[ 30.355989] md/raid:md0: device sdz operational as raid disk 9
[ 30.363236] md/raid:md0: device sdv operational as raid disk 8
[ 30.370486] md/raid:md0: device sdaf operational as raid disk 7
[ 30.377820] md/raid:md0: device sdab operational as raid disk 6
[ 30.385141] md/raid:md0: device sdx operational as raid disk 5
[ 30.392379] md/raid:md0: device sdt operational as raid disk 4
[ 30.399598] md/raid:md0: device sdac operational as raid disk 3
[ 30.406924] md/raid:md0: device sdag operational as raid disk 2
[ 30.407086] 8021q: 802.1Q VLAN Support v1.8
[ 30.414240] md/raid:md0: device sdy operational as raid disk 1
[ 30.419620] 8021q: adding VLAN 0 to HW filter on device net02
[ 30.428066] md/raid:md0: raid level 6 active with 16 out of 16 devices, algorithm 2
[ 30.477101] md0: detected capacity change from 0 to 112020001128448
[ 30.530461] md: md1 stopped.
[ 30.544252] md/raid:md1: device sdg operational as raid disk 0
[ 30.551570] md/raid:md1: device sdm operational as raid disk 15
[ 30.558929] md/raid:md1: device sdq operational as raid disk 14
[ 30.566266] md/raid:md1: device sdp operational as raid disk 13
[ 30.573612] md/raid:md1: device sdl operational as raid disk 12
[ 30.580945] md/raid:md1: device sdj operational as raid disk 11
[ 30.588283] md/raid:md1: device sdf operational as raid disk 10
[ 30.595605] md/raid:md1: device sdh operational as raid disk 9
[ 30.602815] md/raid:md1: device sdn operational as raid disk 8
[ 30.610025] md/raid:md1: device sdd operational as raid disk 7
[ 30.617210] md/raid:md1: device sdr operational as raid disk 6
[ 30.624416] md/raid:md1: device sdi operational as raid disk 5
[ 30.631620] md/raid:md1: device sde operational as raid disk 4
[ 30.638819] md/raid:md1: device sds operational as raid disk 3
[ 30.646027] md/raid:md1: device sdo operational as raid disk 2
[ 30.653193] md/raid:md1: device sdk operational as raid disk 1
[ 30.661613] md/raid:md1: raid level 6 active with 16 out of 16 devices, algorithm 2
[ 30.705932] md1: detected capacity change from 0 to 112020001128448
[ 30.753238] md: md2 stopped.
[ 30.767090] md/raid:md2: device sday operational as raid disk 0
[ 30.774449] md/raid:md2: device sdak operational as raid disk 15
[ 30.781849] md/raid:md2: device sdaj operational as raid disk 14
[ 30.781852] md/raid:md2: device sdal operational as raid disk 13
[ 30.781853] md/raid:md2: device sdam operational as raid disk 12
[ 30.781856] md/raid:md2: device sdan operational as raid disk 11
[ 30.811425] md/raid:md2: device sdao operational as raid disk 10
[ 30.811426] md/raid:md2: device sdap operational as raid disk 9
[ 30.811427] md/raid:md2: device sdaq operational as raid disk 8
[ 30.811428] md/raid:md2: device sdar operational as raid disk 7
[ 30.811428] md/raid:md2: device sdas operational as raid disk 6
[ 30.811430] md/raid:md2: device sdat operational as raid disk 5
[ 30.854881] md/raid:md2: device sdav operational as raid disk 4
[ 30.854882] md/raid:md2: device sdaw operational as raid disk 3
[ 30.854883] md/raid:md2: device sdau operational as raid disk 2
[ 30.854884] md/raid:md2: device sdax operational as raid disk 1
[ 30.856118] md/raid:md2: raid level 6 active with 16 out of 16 devices, algorithm 2
[ 30.931670] md2: detected capacity change from 0 to 112020001128448
[ 31.001836] ixgbe 0000:04:00.0 net02: NIC Link is Down
[ 31.208491] XFS (sda2): Mounting V5 Filesystem
[ 31.382076] XFS (sda2): Ending clean mount
[ 31.409872] xfs filesystem being mounted at /amd/salvadorthegunzerker/0 supports timestamps until 2038 (0x7fffffff)
[ 31.434848] XFS (sdb): Mounting V5 Filesystem
[ 31.615088] XFS (sdb): Ending clean mount
[ 31.646164] xfs filesystem being mounted at /amd/salvadorthegunzerker/1 supports timestamps until 2038 (0x7fffffff)
[ 31.760377] XFS (sdc): Mounting V5 Filesystem
[ 32.007365] XFS (sdc): Ending clean mount
[ 32.352666] xfs filesystem being mounted at /amd/salvadorthegunzerker/2 supports timestamps until 2038 (0x7fffffff)
[ 32.519470] ixgbe 0000:04:00.0 net02: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 32.958228] XFS (md0): Mounting V5 Filesystem
[ 33.319013] XFS (md0): Ending clean mount
[ 35.433740] xfs filesystem being mounted at /amd/salvadorthegunzerker/M/M8017 supports timestamps until 2038 (0x7fffffff)
[ 35.495025] XFS (md1): Mounting V5 Filesystem
[ 35.912811] XFS (md1): Ending clean mount
[ 37.848827] xfs filesystem being mounted at /amd/salvadorthegunzerker/M/M8018 supports timestamps until 2038 (0x7fffffff)
[ 37.883054] XFS (md2): Mounting V5 Filesystem
[ 38.223659] XFS (md2): Ending clean mount
[ 38.909888] NFSD: Using UMH upcall client tracking operations.
[ 38.917414] NFSD: starting 90-second grace period (net f00000a8)
[ 40.395963] xfs filesystem being mounted at /amd/salvadorthegunzerker/M/M8023 supports timestamps until 2038 (0x7fffffff)
[ 1090.643456] FS-Cache: Netfs 'nfs' registered for caching
[ 1091.009271] NFS: Registering the id_resolver key type
[ 1091.015311] Key type id_resolver registered
[ 1091.020273] Key type id_legacy registered
[48962.981257] rcu: INFO: rcu_sched self-detected stall on CPU
[48962.987511] rcu: 4-....: (20999 ticks this GP) idle=fe6/1/0x4000000000000002 softirq=3630188/3630188 fqs=4696
[48962.998805] (t=21017 jiffies g=14529009 q=32263)
[48963.004074] Task dump for CPU 4:
[48963.007689] kworker/4:2 R running task 0 25587 2 0x80004008
[48963.015591] Workqueue: xfs-conv/md0 xfs_end_io
[48963.020570] Call Trace:
[48963.023311] <IRQ>
[48963.025560] sched_show_task+0x11e/0x150
[48963.029957] rcu_dump_cpu_stacks+0x70/0xa0
[48963.034545] rcu_sched_clock_irq+0x502/0x770
[48963.039322] ? tick_sched_do_timer+0x60/0x60
[48963.044106] update_process_times+0x24/0x60
[48963.048791] tick_sched_timer+0x37/0x70
[48963.053089] __hrtimer_run_queues+0x11f/0x2b0
[48963.057960] ? recalibrate_cpu_khz+0x10/0x10
[48963.062744] hrtimer_interrupt+0xe5/0x240
[48963.067235] smp_apic_timer_interrupt+0x6f/0x130
[48963.072407] apic_timer_interrupt+0xf/0x20
[48963.076994] </IRQ>
[48963.079347] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
[48963.085491] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3> 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
[48963.106524] RSP: 0018:ffffc9000738fd40 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[48963.115003] RAX: ffffffff82407588 RBX: ffffffff82407580 RCX: ffffffff82407588
[48963.122994] RDX: ffffffff82407588 RSI: 0000000000000202 RDI: ffffffff82407580
[48963.130989] RBP: 0000000000000202 R08: ffffffff8203ea00 R09: 0000000000000001
[48963.138982] R10: ffffc9000738fbb8 R11: 0000000000000001 R12: ffffffff82407588
[48963.146976] R13: ffffea005e7ae600 R14: ffff8897b7e5a040 R15: ffffea005e7ae600
[48963.154971] wake_up_page_bit+0xe0/0x100
[48963.159366] xfs_destroy_ioend+0xce/0x1c0
[48963.163857] xfs_end_ioend+0xcf/0x1a0
[48963.167958] xfs_end_io+0xa4/0xd0
[48963.171672] process_one_work+0x1e5/0x410
[48963.176163] worker_thread+0x2d/0x3c0
[48963.180265] ? cancel_delayed_work+0x90/0x90
[48963.185048] kthread+0x117/0x130
[48963.188663] ? kthread_create_worker_on_cpu+0x70/0x70
[48963.194321] ret_from_fork+0x35/0x40
[49120.136542] md: data-check of RAID array md1
[49120.207746] md: data-check of RAID array md2
[49124.782519] md: data-check of RAID array md0
[63521.593773] md: md0: data-check interrupted.
[63521.949767] md: md1: data-check interrupted.
[63522.042767] md: md2: data-check interrupted.
[135519.340478] md: data-check of RAID array md0
[135519.431284] md: data-check of RAID array md1
[135519.469736] md: data-check of RAID array md2
[149920.399175] md: md0: data-check interrupted.
[149920.858323] md: md1: data-check interrupted.
[149921.302314] md: md2: data-check interrupted.
[221918.809782] md: data-check of RAID array md0
[221918.843228] md: data-check of RAID array md1
[221918.873674] md: data-check of RAID array md2
[228830.899827] md: md2: data-check done.
[228953.183890] md: md1: data-check done.
[236319.515530] md: md0: data-check interrupted.
[308318.205528] md: data-check of RAID array md0
[314052.860139] md: md0: data-check done.
[2381905.982615] md: data-check of RAID array md0
[2381906.015683] md: data-check of RAID array md1
[2381906.056985] md: data-check of RAID array md2
[2396307.580411] md: md0: data-check interrupted.
[2396307.866401] md: md1: data-check interrupted.
[2396308.318396] md: md2: data-check interrupted.
[2468305.675625] md: data-check of RAID array md0
[2468305.705955] md: data-check of RAID array md1
[2468305.767082] md: data-check of RAID array md2
[2482707.087731] md: md0: data-check interrupted.
[2482707.569726] md: md1: data-check interrupted.
[2482707.822728] md: md2: data-check interrupted.
[2554704.700909] md: data-check of RAID array md0
[2554704.735312] md: data-check of RAID array md1
[2554704.778952] md: data-check of RAID array md2
[2569105.815166] md: md0: data-check interrupted.
[2569106.504161] md: md1: data-check interrupted.
[2569107.221160] md: md2: data-check interrupted.
[2641104.984749] md: data-check of RAID array md0
[2641105.017034] md: data-check of RAID array md1
[2641105.059236] md: data-check of RAID array md2
[2655506.334514] md: md0: data-check interrupted.
[2655506.987500] md: md1: data-check interrupted.
[2655507.586493] md: md2: data-check interrupted.
[2727503.944755] md: data-check of RAID array md0
[2727503.990456] md: data-check of RAID array md1
[2727504.028695] md: data-check of RAID array md2
[2741904.808855] md: md0: data-check interrupted.
[2741906.568907] md: md1: data-check interrupted.
[2741907.285031] md: md2: data-check interrupted.
[2813903.936546] md: data-check of RAID array md0
[2813903.977525] md: data-check of RAID array md1
[2813904.016756] md: data-check of RAID array md2
[2828305.476214] md: md0: data-check interrupted.
[2828305.952433] md: md1: data-check interrupted.
[2828306.168254] md: md2: data-check interrupted.
[2900303.188322] md: data-check of RAID array md0
[2900303.235502] md: data-check of RAID array md1
[2900303.268533] md: data-check of RAID array md2
[2911377.681079] md: md1: data-check done.
[2911861.547499] md: md2: data-check done.
[2913794.332966] md: md0: data-check done.
[4973891.015481] md: data-check of RAID array md0
[4973891.055935] md: data-check of RAID array md1
[4973891.092980] md: data-check of RAID array md2
[4988291.880061] md: md0: data-check interrupted.
[4988292.314104] md: md1: data-check interrupted.
[4988292.837857] md: md2: data-check interrupted.
[5060290.657706] md: data-check of RAID array md0
[5060290.687909] md: data-check of RAID array md1
[5060290.736589] md: data-check of RAID array md2
[5074691.687475] md: md0: data-check interrupted.
[5074692.316473] md: md1: data-check interrupted.
[5074692.997466] md: md2: data-check interrupted.
[5146689.581013] md: data-check of RAID array md0
[5146689.617846] md: data-check of RAID array md1
[5146689.672307] md: data-check of RAID array md2
[5161090.682149] md: md0: data-check interrupted.
[5161091.303148] md: md1: data-check interrupted.
[5161091.542144] md: md2: data-check interrupted.
[5233089.235845] md: data-check of RAID array md0
[5233089.286765] md: data-check of RAID array md1
[5233089.329845] md: data-check of RAID array md2
[5247490.844776] md: md0: data-check interrupted.
[5247491.022073] md: md1: data-check interrupted.
[5247491.580776] md: md2: data-check interrupted.
[5319489.310303] md: data-check of RAID array md0
[5319489.365708] md: data-check of RAID array md1
[5319489.402428] md: data-check of RAID array md2
[5333891.223820] md: md0: data-check interrupted.
[5333892.490824] md: md1: data-check interrupted.
[5333893.029810] md: md2: data-check interrupted.
[5405888.468376] md: data-check of RAID array md0
[5405888.509074] md: data-check of RAID array md1
[5405888.548773] md: data-check of RAID array md2
[5420289.492532] md: md0: data-check interrupted.
[5420290.250529] md: md1: data-check interrupted.
[5420290.718523] md: md2: data-check interrupted.
[5492287.964148] md: data-check of RAID array md0
[5492287.997414] md: data-check of RAID array md1
[5492288.048891] md: data-check of RAID array md2
[5503157.917519] md: md0: data-check done.
[5503490.042212] md: md1: data-check done.
[5504032.739464] md: md2: data-check done.
[7164019.947477] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7164019.956934] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7164019.966461] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7164019.975678] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7164019.985040] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7164019.994414] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7164020.003667] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7164020.013028] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7164020.022369] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7164020.031658] NFSD: client 141.14.26.78 testing state ID with incorrect client ID
[7284740.237483] rcu: INFO: rcu_sched self-detected stall on CPU
[7284740.244490] rcu: 1-....: (20999 ticks this GP) idle=d0a/1/0x4000000000000002 softirq=212838113/212838113 fqs=4849
[7284740.256793] (t=21019 jiffies g=1921397141 q=42868)
[7284740.262891] Task dump for CPU 1:
[7284740.267132] kworker/1:0 R running task 0 16950 2 0x80004008
[7284740.275667] Workqueue: xfs-conv/md0 xfs_end_io
[7284740.281292] Call Trace:
[7284740.284692] <IRQ>
[7284740.287597] sched_show_task+0x11e/0x150
[7284740.292651] rcu_dump_cpu_stacks+0x70/0xa0
[7284740.297905] rcu_sched_clock_irq+0x502/0x770
[7284740.303350] ? tick_sched_do_timer+0x60/0x60
[7284740.308802] update_process_times+0x24/0x60
[7284740.314166] tick_sched_timer+0x37/0x70
[7284740.319140] __hrtimer_run_queues+0x11f/0x2b0
[7284740.324708] ? recalibrate_cpu_khz+0x10/0x10
[7284740.330188] hrtimer_interrupt+0xe5/0x240
[7284740.335377] smp_apic_timer_interrupt+0x6f/0x130
[7284740.341256] apic_timer_interrupt+0xf/0x20
[7284740.346559] </IRQ>
[7284740.349625] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
[7284740.356497] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3> 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
[7284740.378890] RSP: 0018:ffffc900074c7d40 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[7284740.388194] RAX: ffffffff82407f18 RBX: ffffffff82407f10 RCX: ffffffff82407f18
[7284740.397046] RDX: ffffffff82407f18 RSI: 0000000000000202 RDI: ffffffff82407f10
[7284740.405877] RBP: 0000000000000202 R08: ffffffff8203ea00 R09: 0000000000000001
[7284740.414726] R10: ffffc900074c7bb8 R11: 0000000000000001 R12: ffffffff82407f18
[7284740.423593] R13: ffffea002d47b980 R14: ffff888e1c2c1140 R15: ffffea002d47b980
[7284740.432469] wake_up_page_bit+0xe0/0x100
[7284740.437763] xfs_destroy_ioend+0xce/0x1c0
[7284740.443151] xfs_end_ioend+0xcf/0x1a0
[7284740.448157] xfs_end_io+0xa4/0xd0
[7284740.452778] process_one_work+0x1e5/0x410
[7284740.458183] worker_thread+0x2d/0x3c0
[7284740.463197] ? cancel_delayed_work+0x90/0x90
[7284740.468901] kthread+0x117/0x130
[7284740.473444] ? kthread_create_worker_on_cpu+0x70/0x70
[7284740.480019] ret_from_fork+0x35/0x40
[7284772.752302] rcu: INFO: rcu_sched self-detected stall on CPU
[7284772.759602] rcu: 1-....: (20999 ticks this GP) idle=d0a/1/0x4000000000000002 softirq=212838122/212838122 fqs=4827
[7284772.772284] (t=21020 jiffies g=1921397149 q=85713)
[7284772.778770] Task dump for CPU 1:
[7284772.783420] kworker/1:0 R running task 0 16950 2 0x80004008
[7284772.792373] Workqueue: xfs-conv/md0 xfs_end_io
[7284772.798426] Call Trace:
[7284772.802238] <IRQ>
[7284772.805578] sched_show_task+0x11e/0x150
[7284772.811038] rcu_dump_cpu_stacks+0x70/0xa0
[7284772.816667] rcu_sched_clock_irq+0x502/0x770
[7284772.822465] ? tick_sched_do_timer+0x60/0x60
[7284772.828257] update_process_times+0x24/0x60
[7284772.833947] tick_sched_timer+0x37/0x70
[7284772.839239] __hrtimer_run_queues+0x11f/0x2b0
[7284772.845114] ? recalibrate_cpu_khz+0x10/0x10
[7284772.850899] hrtimer_interrupt+0xe5/0x240
[7284772.856383] smp_apic_timer_interrupt+0x6f/0x130
[7284772.862559] apic_timer_interrupt+0xf/0x20
[7284772.868152] </IRQ>
[7284772.871509] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
[7284772.878664] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3> 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
[7284772.901591] RSP: 0018:ffffc900074c7d40 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[7284772.911150] RAX: ffffffff824073d8 RBX: ffffffff824073d0 RCX: ffffffff824073d8
[7284772.920232] RDX: ffffffff824073d8 RSI: 0000000000000202 RDI: ffffffff824073d0
[7284772.929313] RBP: 0000000000000202 R08: ffffffff8203ea00 R09: 0000000000000001
[7284772.938400] R10: ffffc900074c7bb8 R11: 0000000000000001 R12: ffffffff824073d8
[7284772.947496] R13: ffffea004be32d00 R14: ffff888b8cb1be40 R15: ffffea004be32d00
[7284772.956602] wake_up_page_bit+0xe0/0x100
[7284772.962089] xfs_destroy_ioend+0xce/0x1c0
[7284772.967646] xfs_end_ioend+0xcf/0x1a0
[7284772.972791] xfs_end_io+0xa4/0xd0
[7284772.977534] process_one_work+0x1e5/0x410
[7284772.983054] worker_thread+0x2d/0x3c0
[7284772.988174] ? cancel_delayed_work+0x90/0x90
[7284772.993965] kthread+0x117/0x130
[7284772.998582] ? kthread_create_worker_on_cpu+0x70/0x70
[7284773.005234] ret_from_fork+0x35/0x40
[7652275.737662] md: data-check of RAID array md0
[7652275.766162] md: data-check of RAID array md1
[7652275.801760] md: data-check of RAID array md2
[7666676.850323] md: md0: data-check interrupted.
[7666677.331308] md: md1: data-check interrupted.
[7666678.222848] md: md2: data-check interrupted.
[8085907.815945] nfsd4_validate_stateid: 50 callbacks suppressed
[8085907.815946] NFSD: client 141.14.26.40 testing state ID with incorrect client ID
[8085907.837911] NFSD: client 141.14.26.40 testing state ID with incorrect client ID
[8085907.856716] NFSD: client 141.14.26.40 testing state ID with incorrect client ID
[8085907.881038] NFSD: client 141.14.26.40 testing state ID with incorrect client ID
[8085907.882688] NFSD: client 141.14.17.230 testing state ID with incorrect client ID
[8085907.890694] NFSD: client 141.14.26.40 testing state ID with incorrect client ID
[8085907.907041] NFSD: client 141.14.17.230 testing state ID with incorrect client ID
[8085907.920654] NFSD: client 141.14.26.40 testing state ID with incorrect client ID
[8085907.924620] NFSD: client 141.14.17.230 testing state ID with incorrect client ID
[8085907.932357] NFSD: client 141.14.26.40 testing state ID with incorrect client ID
[8086183.796202] nfsd4_validate_stateid: 177 callbacks suppressed
[8086183.796203] NFSD: client 141.14.26.34 testing state ID with incorrect client ID
[8086184.520616] NFSD: client 141.14.26.36 testing state ID with incorrect client ID
[8086184.527355] NFSD: client 141.14.26.34 testing state ID with incorrect client ID
[8086185.184430] NFSD: client 141.14.26.36 testing state ID with incorrect client ID
[8086185.193802] NFSD: client 141.14.26.34 testing state ID with incorrect client ID
[8086185.229706] NFSD: client 141.14.26.36 testing state ID with incorrect client ID
[8086185.229714] NFSD: client 141.14.26.34 testing state ID with incorrect client ID
[8086185.289349] NFSD: client 141.14.26.36 testing state ID with incorrect client ID
[8086185.638580] NFSD: client 141.14.26.34 testing state ID with incorrect client ID
[8086185.729928] NFSD: client 141.14.26.36 testing state ID with incorrect client ID
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
2021-02-05 7:29 rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io Paul Menzel
@ 2021-02-05 17:12 ` Paul E. McKenney
2021-02-08 14:07 ` Brian Foster
0 siblings, 1 reply; 10+ messages in thread
From: Paul E. McKenney @ 2021-02-05 17:12 UTC (permalink / raw)
To: Paul Menzel
Cc: Darrick J. Wong, linux-xfs, Josh Triplett, rcu, it+linux-rcu, LKML
On Fri, Feb 05, 2021 at 08:29:06AM +0100, Paul Menzel wrote:
> Dear Linux folks,
>
>
> On a Dell PowerEdge T630/0NT78X, BIOS 2.8.0 05/23/2018 with Linux 5.4.57, we
> twice saw a self-detected stall on a CPU (October 27th, 2020, January 18th,
> 2021).
>
> Both times, the workqueue is `xfs-conv/md0 xfs_end_io`.
>
> ```
> [ 0.000000] Linux version 5.4.57.mx64.340
> (root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC)) #1 SMP Tue Aug 11
> 13:20:33 CEST 2020
> […]
> [48962.981257] rcu: INFO: rcu_sched self-detected stall on CPU
> [48962.987511] rcu: 4-....: (20999 ticks this GP)
> idle=fe6/1/0x4000000000000002 softirq=3630188/3630188 fqs=4696
> [48962.998805] (t=21017 jiffies g=14529009 q=32263)
> [48963.004074] Task dump for CPU 4:
> [48963.007689] kworker/4:2 R running task 0 25587 2
> 0x80004008
> [48963.015591] Workqueue: xfs-conv/md0 xfs_end_io
> [48963.020570] Call Trace:
> [48963.023311] <IRQ>
> [48963.025560] sched_show_task+0x11e/0x150
> [48963.029957] rcu_dump_cpu_stacks+0x70/0xa0
> [48963.034545] rcu_sched_clock_irq+0x502/0x770
> [48963.039322] ? tick_sched_do_timer+0x60/0x60
> [48963.044106] update_process_times+0x24/0x60
> [48963.048791] tick_sched_timer+0x37/0x70
> [48963.053089] __hrtimer_run_queues+0x11f/0x2b0
> [48963.057960] ? recalibrate_cpu_khz+0x10/0x10
> [48963.062744] hrtimer_interrupt+0xe5/0x240
> [48963.067235] smp_apic_timer_interrupt+0x6f/0x130
> [48963.072407] apic_timer_interrupt+0xf/0x20
> [48963.076994] </IRQ>
> [48963.079347] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
> [48963.085491] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e
> ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3>
> 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
> [48963.106524] RSP: 0018:ffffc9000738fd40 EFLAGS: 00000202 ORIG_RAX:
> ffffffffffffff13
> [48963.115003] RAX: ffffffff82407588 RBX: ffffffff82407580 RCX:
> ffffffff82407588
> [48963.122994] RDX: ffffffff82407588 RSI: 0000000000000202 RDI:
> ffffffff82407580
> [48963.130989] RBP: 0000000000000202 R08: ffffffff8203ea00 R09:
> 0000000000000001
> [48963.138982] R10: ffffc9000738fbb8 R11: 0000000000000001 R12:
> ffffffff82407588
> [48963.146976] R13: ffffea005e7ae600 R14: ffff8897b7e5a040 R15:
> ffffea005e7ae600
> [48963.154971] wake_up_page_bit+0xe0/0x100
> [48963.159366] xfs_destroy_ioend+0xce/0x1c0
> [48963.163857] xfs_end_ioend+0xcf/0x1a0
> [48963.167958] xfs_end_io+0xa4/0xd0
> [48963.171672] process_one_work+0x1e5/0x410
> [48963.176163] worker_thread+0x2d/0x3c0
> [48963.180265] ? cancel_delayed_work+0x90/0x90
> [48963.185048] kthread+0x117/0x130
> [48963.188663] ? kthread_create_worker_on_cpu+0x70/0x70
> [48963.194321] ret_from_fork+0x35/0x40
> ```
>
> As it’s just log level INFO, is there anything what should be done, or was
> the system probably just “overloaded”?
I am assuming that you are building your kernel with CONFIG_PREEMPT_NONE=y
rather than CONFIG_PREEMPT_VOLUNTARY=y.
If so, and if the problem is that you are temporarily overdriving xfs I/O,
one approach would be as follows:
------------------------------------------------------------------------
diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index f16d5f1..06be426 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -390,6 +390,7 @@ xfs_end_io(
list_del_init(&ioend->io_list);
xfs_ioend_try_merge(ioend, &completion_list);
xfs_end_ioend(ioend);
+ cond_resched();
}
}
------------------------------------------------------------------------
If you have instead built with CONFIG_PREEMPT_VOLUNTARY=y, then your
problem is likely massive lock contention in wake_up_page_bit(), or
perhaps someone having failed to release that lock. The usual way to
work this out is by enabling lockdep (CONFIG_PROVE_LOCKING=y), but this
is often not what you want enabled in production.
Darrick, thoughts from an xfs perspective?
Thanx, Paul
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
2021-02-05 17:12 ` Paul E. McKenney
@ 2021-02-08 14:07 ` Brian Foster
2021-02-08 14:57 ` Paul E. McKenney
0 siblings, 1 reply; 10+ messages in thread
From: Brian Foster @ 2021-02-08 14:07 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Paul Menzel, Darrick J. Wong, linux-xfs, Josh Triplett, rcu,
it+linux-rcu, LKML
On Fri, Feb 05, 2021 at 09:12:40AM -0800, Paul E. McKenney wrote:
> On Fri, Feb 05, 2021 at 08:29:06AM +0100, Paul Menzel wrote:
> > Dear Linux folks,
> >
> >
> > On a Dell PowerEdge T630/0NT78X, BIOS 2.8.0 05/23/2018 with Linux 5.4.57, we
> > twice saw a self-detected stall on a CPU (October 27th, 2020, January 18th,
> > 2021).
> >
> > Both times, the workqueue is `xfs-conv/md0 xfs_end_io`.
> >
> > ```
> > [ 0.000000] Linux version 5.4.57.mx64.340
> > (root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC)) #1 SMP Tue Aug 11
> > 13:20:33 CEST 2020
> > […]
> > [48962.981257] rcu: INFO: rcu_sched self-detected stall on CPU
> > [48962.987511] rcu: 4-....: (20999 ticks this GP)
> > idle=fe6/1/0x4000000000000002 softirq=3630188/3630188 fqs=4696
> > [48962.998805] (t=21017 jiffies g=14529009 q=32263)
> > [48963.004074] Task dump for CPU 4:
> > [48963.007689] kworker/4:2 R running task 0 25587 2
> > 0x80004008
> > [48963.015591] Workqueue: xfs-conv/md0 xfs_end_io
> > [48963.020570] Call Trace:
> > [48963.023311] <IRQ>
> > [48963.025560] sched_show_task+0x11e/0x150
> > [48963.029957] rcu_dump_cpu_stacks+0x70/0xa0
> > [48963.034545] rcu_sched_clock_irq+0x502/0x770
> > [48963.039322] ? tick_sched_do_timer+0x60/0x60
> > [48963.044106] update_process_times+0x24/0x60
> > [48963.048791] tick_sched_timer+0x37/0x70
> > [48963.053089] __hrtimer_run_queues+0x11f/0x2b0
> > [48963.057960] ? recalibrate_cpu_khz+0x10/0x10
> > [48963.062744] hrtimer_interrupt+0xe5/0x240
> > [48963.067235] smp_apic_timer_interrupt+0x6f/0x130
> > [48963.072407] apic_timer_interrupt+0xf/0x20
> > [48963.076994] </IRQ>
> > [48963.079347] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
> > [48963.085491] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e
> > ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3>
> > 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
> > [48963.106524] RSP: 0018:ffffc9000738fd40 EFLAGS: 00000202 ORIG_RAX:
> > ffffffffffffff13
> > [48963.115003] RAX: ffffffff82407588 RBX: ffffffff82407580 RCX:
> > ffffffff82407588
> > [48963.122994] RDX: ffffffff82407588 RSI: 0000000000000202 RDI:
> > ffffffff82407580
> > [48963.130989] RBP: 0000000000000202 R08: ffffffff8203ea00 R09:
> > 0000000000000001
> > [48963.138982] R10: ffffc9000738fbb8 R11: 0000000000000001 R12:
> > ffffffff82407588
> > [48963.146976] R13: ffffea005e7ae600 R14: ffff8897b7e5a040 R15:
> > ffffea005e7ae600
> > [48963.154971] wake_up_page_bit+0xe0/0x100
> > [48963.159366] xfs_destroy_ioend+0xce/0x1c0
> > [48963.163857] xfs_end_ioend+0xcf/0x1a0
> > [48963.167958] xfs_end_io+0xa4/0xd0
> > [48963.171672] process_one_work+0x1e5/0x410
> > [48963.176163] worker_thread+0x2d/0x3c0
> > [48963.180265] ? cancel_delayed_work+0x90/0x90
> > [48963.185048] kthread+0x117/0x130
> > [48963.188663] ? kthread_create_worker_on_cpu+0x70/0x70
> > [48963.194321] ret_from_fork+0x35/0x40
> > ```
> >
> > As it’s just log level INFO, is there anything what should be done, or was
> > the system probably just “overloaded”?
>
> I am assuming that you are building your kernel with CONFIG_PREEMPT_NONE=y
> rather than CONFIG_PREEMPT_VOLUNTARY=y.
>
> If so, and if the problem is that you are temporarily overdriving xfs I/O,
> one approach would be as follows:
>
> ------------------------------------------------------------------------
>
> diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
> index f16d5f1..06be426 100644
> --- a/fs/xfs/xfs_aops.c
> +++ b/fs/xfs/xfs_aops.c
> @@ -390,6 +390,7 @@ xfs_end_io(
> list_del_init(&ioend->io_list);
> xfs_ioend_try_merge(ioend, &completion_list);
> xfs_end_ioend(ioend);
> + cond_resched();
> }
> }
>
> ------------------------------------------------------------------------
FWIW, this looks quite similar to the problem I attempted to fix with
these patches:
https://lore.kernel.org/linux-xfs/20201002153357.56409-1-bfoster@redhat.com/
Brian
>
> If you have instead built with CONFIG_PREEMPT_VOLUNTARY=y, then your
> problem is likely massive lock contention in wake_up_page_bit(), or
> perhaps someone having failed to release that lock. The usual way to
> work this out is by enabling lockdep (CONFIG_PROVE_LOCKING=y), but this
> is often not what you want enabled in production.
>
> Darrick, thoughts from an xfs perspective?
>
> Thanx, Paul
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
2021-02-08 14:07 ` Brian Foster
@ 2021-02-08 14:57 ` Paul E. McKenney
2021-02-08 15:44 ` Brian Foster
0 siblings, 1 reply; 10+ messages in thread
From: Paul E. McKenney @ 2021-02-08 14:57 UTC (permalink / raw)
To: Brian Foster
Cc: Paul Menzel, Darrick J. Wong, linux-xfs, Josh Triplett, rcu,
it+linux-rcu, LKML
On Mon, Feb 08, 2021 at 09:07:24AM -0500, Brian Foster wrote:
> On Fri, Feb 05, 2021 at 09:12:40AM -0800, Paul E. McKenney wrote:
> > On Fri, Feb 05, 2021 at 08:29:06AM +0100, Paul Menzel wrote:
> > > Dear Linux folks,
> > >
> > >
> > > On a Dell PowerEdge T630/0NT78X, BIOS 2.8.0 05/23/2018 with Linux 5.4.57, we
> > > twice saw a self-detected stall on a CPU (October 27th, 2020, January 18th,
> > > 2021).
> > >
> > > Both times, the workqueue is `xfs-conv/md0 xfs_end_io`.
> > >
> > > ```
> > > [ 0.000000] Linux version 5.4.57.mx64.340
> > > (root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC)) #1 SMP Tue Aug 11
> > > 13:20:33 CEST 2020
> > > […]
> > > [48962.981257] rcu: INFO: rcu_sched self-detected stall on CPU
> > > [48962.987511] rcu: 4-....: (20999 ticks this GP)
> > > idle=fe6/1/0x4000000000000002 softirq=3630188/3630188 fqs=4696
> > > [48962.998805] (t=21017 jiffies g=14529009 q=32263)
> > > [48963.004074] Task dump for CPU 4:
> > > [48963.007689] kworker/4:2 R running task 0 25587 2
> > > 0x80004008
> > > [48963.015591] Workqueue: xfs-conv/md0 xfs_end_io
> > > [48963.020570] Call Trace:
> > > [48963.023311] <IRQ>
> > > [48963.025560] sched_show_task+0x11e/0x150
> > > [48963.029957] rcu_dump_cpu_stacks+0x70/0xa0
> > > [48963.034545] rcu_sched_clock_irq+0x502/0x770
> > > [48963.039322] ? tick_sched_do_timer+0x60/0x60
> > > [48963.044106] update_process_times+0x24/0x60
> > > [48963.048791] tick_sched_timer+0x37/0x70
> > > [48963.053089] __hrtimer_run_queues+0x11f/0x2b0
> > > [48963.057960] ? recalibrate_cpu_khz+0x10/0x10
> > > [48963.062744] hrtimer_interrupt+0xe5/0x240
> > > [48963.067235] smp_apic_timer_interrupt+0x6f/0x130
> > > [48963.072407] apic_timer_interrupt+0xf/0x20
> > > [48963.076994] </IRQ>
> > > [48963.079347] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
> > > [48963.085491] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e
> > > ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3>
> > > 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
> > > [48963.106524] RSP: 0018:ffffc9000738fd40 EFLAGS: 00000202 ORIG_RAX:
> > > ffffffffffffff13
> > > [48963.115003] RAX: ffffffff82407588 RBX: ffffffff82407580 RCX:
> > > ffffffff82407588
> > > [48963.122994] RDX: ffffffff82407588 RSI: 0000000000000202 RDI:
> > > ffffffff82407580
> > > [48963.130989] RBP: 0000000000000202 R08: ffffffff8203ea00 R09:
> > > 0000000000000001
> > > [48963.138982] R10: ffffc9000738fbb8 R11: 0000000000000001 R12:
> > > ffffffff82407588
> > > [48963.146976] R13: ffffea005e7ae600 R14: ffff8897b7e5a040 R15:
> > > ffffea005e7ae600
> > > [48963.154971] wake_up_page_bit+0xe0/0x100
> > > [48963.159366] xfs_destroy_ioend+0xce/0x1c0
> > > [48963.163857] xfs_end_ioend+0xcf/0x1a0
> > > [48963.167958] xfs_end_io+0xa4/0xd0
> > > [48963.171672] process_one_work+0x1e5/0x410
> > > [48963.176163] worker_thread+0x2d/0x3c0
> > > [48963.180265] ? cancel_delayed_work+0x90/0x90
> > > [48963.185048] kthread+0x117/0x130
> > > [48963.188663] ? kthread_create_worker_on_cpu+0x70/0x70
> > > [48963.194321] ret_from_fork+0x35/0x40
> > > ```
> > >
> > > As it’s just log level INFO, is there anything what should be done, or was
> > > the system probably just “overloaded”?
> >
> > I am assuming that you are building your kernel with CONFIG_PREEMPT_NONE=y
> > rather than CONFIG_PREEMPT_VOLUNTARY=y.
> >
> > If so, and if the problem is that you are temporarily overdriving xfs I/O,
> > one approach would be as follows:
> >
> > ------------------------------------------------------------------------
> >
> > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
> > index f16d5f1..06be426 100644
> > --- a/fs/xfs/xfs_aops.c
> > +++ b/fs/xfs/xfs_aops.c
> > @@ -390,6 +390,7 @@ xfs_end_io(
> > list_del_init(&ioend->io_list);
> > xfs_ioend_try_merge(ioend, &completion_list);
> > xfs_end_ioend(ioend);
> > + cond_resched();
> > }
> > }
> >
> > ------------------------------------------------------------------------
>
> FWIW, this looks quite similar to the problem I attempted to fix with
> these patches:
>
> https://lore.kernel.org/linux-xfs/20201002153357.56409-1-bfoster@redhat.com/
Looks plausible to me! Do you plan to re-post taking the feedback
into account?
Thanx, Paul
> Brian
>
> >
> > If you have instead built with CONFIG_PREEMPT_VOLUNTARY=y, then your
> > problem is likely massive lock contention in wake_up_page_bit(), or
> > perhaps someone having failed to release that lock. The usual way to
> > work this out is by enabling lockdep (CONFIG_PROVE_LOCKING=y), but this
> > is often not what you want enabled in production.
> >
> > Darrick, thoughts from an xfs perspective?
> >
> > Thanx, Paul
> >
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
2021-02-08 14:57 ` Paul E. McKenney
@ 2021-02-08 15:44 ` Brian Foster
2021-02-08 17:11 ` Paul E. McKenney
0 siblings, 1 reply; 10+ messages in thread
From: Brian Foster @ 2021-02-08 15:44 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Paul Menzel, Darrick J. Wong, linux-xfs, Josh Triplett, rcu,
it+linux-rcu, LKML
On Mon, Feb 08, 2021 at 06:57:24AM -0800, Paul E. McKenney wrote:
> On Mon, Feb 08, 2021 at 09:07:24AM -0500, Brian Foster wrote:
> > On Fri, Feb 05, 2021 at 09:12:40AM -0800, Paul E. McKenney wrote:
> > > On Fri, Feb 05, 2021 at 08:29:06AM +0100, Paul Menzel wrote:
> > > > Dear Linux folks,
> > > >
> > > >
> > > > On a Dell PowerEdge T630/0NT78X, BIOS 2.8.0 05/23/2018 with Linux 5.4.57, we
> > > > twice saw a self-detected stall on a CPU (October 27th, 2020, January 18th,
> > > > 2021).
> > > >
> > > > Both times, the workqueue is `xfs-conv/md0 xfs_end_io`.
> > > >
> > > > ```
> > > > [ 0.000000] Linux version 5.4.57.mx64.340
> > > > (root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC)) #1 SMP Tue Aug 11
> > > > 13:20:33 CEST 2020
> > > > […]
> > > > [48962.981257] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > [48962.987511] rcu: 4-....: (20999 ticks this GP)
> > > > idle=fe6/1/0x4000000000000002 softirq=3630188/3630188 fqs=4696
> > > > [48962.998805] (t=21017 jiffies g=14529009 q=32263)
> > > > [48963.004074] Task dump for CPU 4:
> > > > [48963.007689] kworker/4:2 R running task 0 25587 2
> > > > 0x80004008
> > > > [48963.015591] Workqueue: xfs-conv/md0 xfs_end_io
> > > > [48963.020570] Call Trace:
> > > > [48963.023311] <IRQ>
> > > > [48963.025560] sched_show_task+0x11e/0x150
> > > > [48963.029957] rcu_dump_cpu_stacks+0x70/0xa0
> > > > [48963.034545] rcu_sched_clock_irq+0x502/0x770
> > > > [48963.039322] ? tick_sched_do_timer+0x60/0x60
> > > > [48963.044106] update_process_times+0x24/0x60
> > > > [48963.048791] tick_sched_timer+0x37/0x70
> > > > [48963.053089] __hrtimer_run_queues+0x11f/0x2b0
> > > > [48963.057960] ? recalibrate_cpu_khz+0x10/0x10
> > > > [48963.062744] hrtimer_interrupt+0xe5/0x240
> > > > [48963.067235] smp_apic_timer_interrupt+0x6f/0x130
> > > > [48963.072407] apic_timer_interrupt+0xf/0x20
> > > > [48963.076994] </IRQ>
> > > > [48963.079347] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
> > > > [48963.085491] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e
> > > > ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3>
> > > > 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
> > > > [48963.106524] RSP: 0018:ffffc9000738fd40 EFLAGS: 00000202 ORIG_RAX:
> > > > ffffffffffffff13
> > > > [48963.115003] RAX: ffffffff82407588 RBX: ffffffff82407580 RCX:
> > > > ffffffff82407588
> > > > [48963.122994] RDX: ffffffff82407588 RSI: 0000000000000202 RDI:
> > > > ffffffff82407580
> > > > [48963.130989] RBP: 0000000000000202 R08: ffffffff8203ea00 R09:
> > > > 0000000000000001
> > > > [48963.138982] R10: ffffc9000738fbb8 R11: 0000000000000001 R12:
> > > > ffffffff82407588
> > > > [48963.146976] R13: ffffea005e7ae600 R14: ffff8897b7e5a040 R15:
> > > > ffffea005e7ae600
> > > > [48963.154971] wake_up_page_bit+0xe0/0x100
> > > > [48963.159366] xfs_destroy_ioend+0xce/0x1c0
> > > > [48963.163857] xfs_end_ioend+0xcf/0x1a0
> > > > [48963.167958] xfs_end_io+0xa4/0xd0
> > > > [48963.171672] process_one_work+0x1e5/0x410
> > > > [48963.176163] worker_thread+0x2d/0x3c0
> > > > [48963.180265] ? cancel_delayed_work+0x90/0x90
> > > > [48963.185048] kthread+0x117/0x130
> > > > [48963.188663] ? kthread_create_worker_on_cpu+0x70/0x70
> > > > [48963.194321] ret_from_fork+0x35/0x40
> > > > ```
> > > >
> > > > As it’s just log level INFO, is there anything what should be done, or was
> > > > the system probably just “overloaded”?
> > >
> > > I am assuming that you are building your kernel with CONFIG_PREEMPT_NONE=y
> > > rather than CONFIG_PREEMPT_VOLUNTARY=y.
> > >
> > > If so, and if the problem is that you are temporarily overdriving xfs I/O,
> > > one approach would be as follows:
> > >
> > > ------------------------------------------------------------------------
> > >
> > > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
> > > index f16d5f1..06be426 100644
> > > --- a/fs/xfs/xfs_aops.c
> > > +++ b/fs/xfs/xfs_aops.c
> > > @@ -390,6 +390,7 @@ xfs_end_io(
> > > list_del_init(&ioend->io_list);
> > > xfs_ioend_try_merge(ioend, &completion_list);
> > > xfs_end_ioend(ioend);
> > > + cond_resched();
> > > }
> > > }
> > >
> > > ------------------------------------------------------------------------
> >
> > FWIW, this looks quite similar to the problem I attempted to fix with
> > these patches:
> >
> > https://lore.kernel.org/linux-xfs/20201002153357.56409-1-bfoster@redhat.com/
>
> Looks plausible to me! Do you plan to re-post taking the feedback
> into account?
>
There was a v2 inline that incorporated some directed feedback.
Otherwise there were questions and ideas about making the whole thing
faster, but I've no idea if that addresses the problem or not (if so,
that would be an entirely different set of patches). I'll wait and see
what Darrick thinks about this and rebase/repost if the approach is
agreeable..
Brian
> Thanx, Paul
>
> > Brian
> >
> > >
> > > If you have instead built with CONFIG_PREEMPT_VOLUNTARY=y, then your
> > > problem is likely massive lock contention in wake_up_page_bit(), or
> > > perhaps someone having failed to release that lock. The usual way to
> > > work this out is by enabling lockdep (CONFIG_PROVE_LOCKING=y), but this
> > > is often not what you want enabled in production.
> > >
> > > Darrick, thoughts from an xfs perspective?
> > >
> > > Thanx, Paul
> > >
> >
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
2021-02-08 15:44 ` Brian Foster
@ 2021-02-08 17:11 ` Paul E. McKenney
2021-02-08 17:28 ` Darrick J. Wong
0 siblings, 1 reply; 10+ messages in thread
From: Paul E. McKenney @ 2021-02-08 17:11 UTC (permalink / raw)
To: Brian Foster
Cc: Paul Menzel, Darrick J. Wong, linux-xfs, Josh Triplett, rcu,
it+linux-rcu, LKML
On Mon, Feb 08, 2021 at 10:44:58AM -0500, Brian Foster wrote:
> On Mon, Feb 08, 2021 at 06:57:24AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 08, 2021 at 09:07:24AM -0500, Brian Foster wrote:
> > > On Fri, Feb 05, 2021 at 09:12:40AM -0800, Paul E. McKenney wrote:
> > > > On Fri, Feb 05, 2021 at 08:29:06AM +0100, Paul Menzel wrote:
> > > > > Dear Linux folks,
> > > > >
> > > > >
> > > > > On a Dell PowerEdge T630/0NT78X, BIOS 2.8.0 05/23/2018 with Linux 5.4.57, we
> > > > > twice saw a self-detected stall on a CPU (October 27th, 2020, January 18th,
> > > > > 2021).
> > > > >
> > > > > Both times, the workqueue is `xfs-conv/md0 xfs_end_io`.
> > > > >
> > > > > ```
> > > > > [ 0.000000] Linux version 5.4.57.mx64.340
> > > > > (root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC)) #1 SMP Tue Aug 11
> > > > > 13:20:33 CEST 2020
> > > > > […]
> > > > > [48962.981257] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > [48962.987511] rcu: 4-....: (20999 ticks this GP)
> > > > > idle=fe6/1/0x4000000000000002 softirq=3630188/3630188 fqs=4696
> > > > > [48962.998805] (t=21017 jiffies g=14529009 q=32263)
> > > > > [48963.004074] Task dump for CPU 4:
> > > > > [48963.007689] kworker/4:2 R running task 0 25587 2
> > > > > 0x80004008
> > > > > [48963.015591] Workqueue: xfs-conv/md0 xfs_end_io
> > > > > [48963.020570] Call Trace:
> > > > > [48963.023311] <IRQ>
> > > > > [48963.025560] sched_show_task+0x11e/0x150
> > > > > [48963.029957] rcu_dump_cpu_stacks+0x70/0xa0
> > > > > [48963.034545] rcu_sched_clock_irq+0x502/0x770
> > > > > [48963.039322] ? tick_sched_do_timer+0x60/0x60
> > > > > [48963.044106] update_process_times+0x24/0x60
> > > > > [48963.048791] tick_sched_timer+0x37/0x70
> > > > > [48963.053089] __hrtimer_run_queues+0x11f/0x2b0
> > > > > [48963.057960] ? recalibrate_cpu_khz+0x10/0x10
> > > > > [48963.062744] hrtimer_interrupt+0xe5/0x240
> > > > > [48963.067235] smp_apic_timer_interrupt+0x6f/0x130
> > > > > [48963.072407] apic_timer_interrupt+0xf/0x20
> > > > > [48963.076994] </IRQ>
> > > > > [48963.079347] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
> > > > > [48963.085491] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e
> > > > > ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3>
> > > > > 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
> > > > > [48963.106524] RSP: 0018:ffffc9000738fd40 EFLAGS: 00000202 ORIG_RAX:
> > > > > ffffffffffffff13
> > > > > [48963.115003] RAX: ffffffff82407588 RBX: ffffffff82407580 RCX:
> > > > > ffffffff82407588
> > > > > [48963.122994] RDX: ffffffff82407588 RSI: 0000000000000202 RDI:
> > > > > ffffffff82407580
> > > > > [48963.130989] RBP: 0000000000000202 R08: ffffffff8203ea00 R09:
> > > > > 0000000000000001
> > > > > [48963.138982] R10: ffffc9000738fbb8 R11: 0000000000000001 R12:
> > > > > ffffffff82407588
> > > > > [48963.146976] R13: ffffea005e7ae600 R14: ffff8897b7e5a040 R15:
> > > > > ffffea005e7ae600
> > > > > [48963.154971] wake_up_page_bit+0xe0/0x100
> > > > > [48963.159366] xfs_destroy_ioend+0xce/0x1c0
> > > > > [48963.163857] xfs_end_ioend+0xcf/0x1a0
> > > > > [48963.167958] xfs_end_io+0xa4/0xd0
> > > > > [48963.171672] process_one_work+0x1e5/0x410
> > > > > [48963.176163] worker_thread+0x2d/0x3c0
> > > > > [48963.180265] ? cancel_delayed_work+0x90/0x90
> > > > > [48963.185048] kthread+0x117/0x130
> > > > > [48963.188663] ? kthread_create_worker_on_cpu+0x70/0x70
> > > > > [48963.194321] ret_from_fork+0x35/0x40
> > > > > ```
> > > > >
> > > > > As it’s just log level INFO, is there anything what should be done, or was
> > > > > the system probably just “overloaded”?
> > > >
> > > > I am assuming that you are building your kernel with CONFIG_PREEMPT_NONE=y
> > > > rather than CONFIG_PREEMPT_VOLUNTARY=y.
> > > >
> > > > If so, and if the problem is that you are temporarily overdriving xfs I/O,
> > > > one approach would be as follows:
> > > >
> > > > ------------------------------------------------------------------------
> > > >
> > > > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
> > > > index f16d5f1..06be426 100644
> > > > --- a/fs/xfs/xfs_aops.c
> > > > +++ b/fs/xfs/xfs_aops.c
> > > > @@ -390,6 +390,7 @@ xfs_end_io(
> > > > list_del_init(&ioend->io_list);
> > > > xfs_ioend_try_merge(ioend, &completion_list);
> > > > xfs_end_ioend(ioend);
> > > > + cond_resched();
> > > > }
> > > > }
> > > >
> > > > ------------------------------------------------------------------------
> > >
> > > FWIW, this looks quite similar to the problem I attempted to fix with
> > > these patches:
> > >
> > > https://lore.kernel.org/linux-xfs/20201002153357.56409-1-bfoster@redhat.com/
> >
> > Looks plausible to me! Do you plan to re-post taking the feedback
> > into account?
>
> There was a v2 inline that incorporated some directed feedback.
> Otherwise there were questions and ideas about making the whole thing
> faster, but I've no idea if that addresses the problem or not (if so,
> that would be an entirely different set of patches). I'll wait and see
> what Darrick thinks about this and rebase/repost if the approach is
> agreeable..
There is always the school of thought that says that the best way to
get people to focus on this is to rebase and repost. Otherwise, they
are all too likely to assume that you lost interest in this.
Thanx, Paul
> Brian
>
> > Thanx, Paul
> >
> > > Brian
> > >
> > > >
> > > > If you have instead built with CONFIG_PREEMPT_VOLUNTARY=y, then your
> > > > problem is likely massive lock contention in wake_up_page_bit(), or
> > > > perhaps someone having failed to release that lock. The usual way to
> > > > work this out is by enabling lockdep (CONFIG_PROVE_LOCKING=y), but this
> > > > is often not what you want enabled in production.
> > > >
> > > > Darrick, thoughts from an xfs perspective?
> > > >
> > > > Thanx, Paul
> > > >
> > >
> >
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
2021-02-08 17:11 ` Paul E. McKenney
@ 2021-02-08 17:28 ` Darrick J. Wong
2021-02-08 17:45 ` Brian Foster
2021-02-08 20:43 ` Dave Chinner
0 siblings, 2 replies; 10+ messages in thread
From: Darrick J. Wong @ 2021-02-08 17:28 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Brian Foster, Paul Menzel, Darrick J. Wong, linux-xfs,
Josh Triplett, rcu, it+linux-rcu, LKML
On Mon, Feb 09, 2021 at 09:11:40AM -0800, Paul E. McKenney wrote:
> On Mon, Feb 08, 2021 at 10:44:58AM -0500, Brian Foster wrote:
> > On Mon, Feb 08, 2021 at 06:57:24AM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 08, 2021 at 09:07:24AM -0500, Brian Foster wrote:
> > > > On Fri, Feb 05, 2021 at 09:12:40AM -0800, Paul E. McKenney wrote:
> > > > > On Fri, Feb 05, 2021 at 08:29:06AM +0100, Paul Menzel wrote:
> > > > > > Dear Linux folks,
> > > > > >
> > > > > >
> > > > > > On a Dell PowerEdge T630/0NT78X, BIOS 2.8.0 05/23/2018 with Linux 5.4.57, we
> > > > > > twice saw a self-detected stall on a CPU (October 27th, 2020, January 18th,
> > > > > > 2021).
> > > > > >
> > > > > > Both times, the workqueue is `xfs-conv/md0 xfs_end_io`.
> > > > > >
> > > > > > ```
> > > > > > [ 0.000000] Linux version 5.4.57.mx64.340
> > > > > > (root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC)) #1 SMP Tue Aug 11
> > > > > > 13:20:33 CEST 2020
> > > > > > […]
> > > > > > [48962.981257] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > > [48962.987511] rcu: 4-....: (20999 ticks this GP)
> > > > > > idle=fe6/1/0x4000000000000002 softirq=3630188/3630188 fqs=4696
> > > > > > [48962.998805] (t=21017 jiffies g=14529009 q=32263)
> > > > > > [48963.004074] Task dump for CPU 4:
> > > > > > [48963.007689] kworker/4:2 R running task 0 25587 2
> > > > > > 0x80004008
> > > > > > [48963.015591] Workqueue: xfs-conv/md0 xfs_end_io
> > > > > > [48963.020570] Call Trace:
> > > > > > [48963.023311] <IRQ>
> > > > > > [48963.025560] sched_show_task+0x11e/0x150
> > > > > > [48963.029957] rcu_dump_cpu_stacks+0x70/0xa0
> > > > > > [48963.034545] rcu_sched_clock_irq+0x502/0x770
> > > > > > [48963.039322] ? tick_sched_do_timer+0x60/0x60
> > > > > > [48963.044106] update_process_times+0x24/0x60
> > > > > > [48963.048791] tick_sched_timer+0x37/0x70
> > > > > > [48963.053089] __hrtimer_run_queues+0x11f/0x2b0
> > > > > > [48963.057960] ? recalibrate_cpu_khz+0x10/0x10
> > > > > > [48963.062744] hrtimer_interrupt+0xe5/0x240
> > > > > > [48963.067235] smp_apic_timer_interrupt+0x6f/0x130
> > > > > > [48963.072407] apic_timer_interrupt+0xf/0x20
> > > > > > [48963.076994] </IRQ>
> > > > > > [48963.079347] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
> > > > > > [48963.085491] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e
> > > > > > ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3>
> > > > > > 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
> > > > > > [48963.106524] RSP: 0018:ffffc9000738fd40 EFLAGS: 00000202 ORIG_RAX:
> > > > > > ffffffffffffff13
> > > > > > [48963.115003] RAX: ffffffff82407588 RBX: ffffffff82407580 RCX:
> > > > > > ffffffff82407588
> > > > > > [48963.122994] RDX: ffffffff82407588 RSI: 0000000000000202 RDI:
> > > > > > ffffffff82407580
> > > > > > [48963.130989] RBP: 0000000000000202 R08: ffffffff8203ea00 R09:
> > > > > > 0000000000000001
> > > > > > [48963.138982] R10: ffffc9000738fbb8 R11: 0000000000000001 R12:
> > > > > > ffffffff82407588
> > > > > > [48963.146976] R13: ffffea005e7ae600 R14: ffff8897b7e5a040 R15:
> > > > > > ffffea005e7ae600
> > > > > > [48963.154971] wake_up_page_bit+0xe0/0x100
> > > > > > [48963.159366] xfs_destroy_ioend+0xce/0x1c0
> > > > > > [48963.163857] xfs_end_ioend+0xcf/0x1a0
> > > > > > [48963.167958] xfs_end_io+0xa4/0xd0
> > > > > > [48963.171672] process_one_work+0x1e5/0x410
> > > > > > [48963.176163] worker_thread+0x2d/0x3c0
> > > > > > [48963.180265] ? cancel_delayed_work+0x90/0x90
> > > > > > [48963.185048] kthread+0x117/0x130
> > > > > > [48963.188663] ? kthread_create_worker_on_cpu+0x70/0x70
> > > > > > [48963.194321] ret_from_fork+0x35/0x40
> > > > > > ```
> > > > > >
> > > > > > As it’s just log level INFO, is there anything what should be done, or was
> > > > > > the system probably just “overloaded”?
> > > > >
> > > > > I am assuming that you are building your kernel with CONFIG_PREEMPT_NONE=y
> > > > > rather than CONFIG_PREEMPT_VOLUNTARY=y.
> > > > >
> > > > > If so, and if the problem is that you are temporarily overdriving xfs I/O,
> > > > > one approach would be as follows:
> > > > >
> > > > > ------------------------------------------------------------------------
> > > > >
> > > > > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
> > > > > index f16d5f1..06be426 100644
> > > > > --- a/fs/xfs/xfs_aops.c
> > > > > +++ b/fs/xfs/xfs_aops.c
> > > > > @@ -390,6 +390,7 @@ xfs_end_io(
> > > > > list_del_init(&ioend->io_list);
> > > > > xfs_ioend_try_merge(ioend, &completion_list);
> > > > > xfs_end_ioend(ioend);
> > > > > + cond_resched();
> > > > > }
> > > > > }
> > > > >
> > > > > ------------------------------------------------------------------------
> > > >
> > > > FWIW, this looks quite similar to the problem I attempted to fix with
> > > > these patches:
> > > >
> > > > https://lore.kernel.org/linux-xfs/20201002153357.56409-1-bfoster@redhat.com/
> > >
> > > Looks plausible to me! Do you plan to re-post taking the feedback
> > > into account?
> >
> > There was a v2 inline that incorporated some directed feedback.
> > Otherwise there were questions and ideas about making the whole thing
> > faster, but I've no idea if that addresses the problem or not (if so,
> > that would be an entirely different set of patches). I'll wait and see
> > what Darrick thinks about this and rebase/repost if the approach is
> > agreeable..
>
> There is always the school of thought that says that the best way to
> get people to focus on this is to rebase and repost. Otherwise, they
> are all too likely to assume that you lost interest in this.
I was hoping that a better solution would emerge for clearing
PageWriteback on hundreds of thousands of pages, but nothing easy popped
out.
The hardcoded threshold in "[PATCH v2 2/2] xfs: kick extra large ioends
to completion workqueue" gives me unease because who's to say if marking
262,144 pages on a particular CPU will actually stall it long enough to
trip the hangcheck? Is the number lower on (say) some pokey NAS box
with a lot of storage but a slow CPU?
That said, /some/ threshold is probably better than no threshold. Could
someone try to confirm if that series of Brian's fixes this problem too?
--D
> Thanx, Paul
>
> > Brian
> >
> > > Thanx, Paul
> > >
> > > > Brian
> > > >
> > > > >
> > > > > If you have instead built with CONFIG_PREEMPT_VOLUNTARY=y, then your
> > > > > problem is likely massive lock contention in wake_up_page_bit(), or
> > > > > perhaps someone having failed to release that lock. The usual way to
> > > > > work this out is by enabling lockdep (CONFIG_PROVE_LOCKING=y), but this
> > > > > is often not what you want enabled in production.
> > > > >
> > > > > Darrick, thoughts from an xfs perspective?
> > > > >
> > > > > Thanx, Paul
> > > > >
> > > >
> > >
> >
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
2021-02-08 17:28 ` Darrick J. Wong
@ 2021-02-08 17:45 ` Brian Foster
2021-02-08 20:43 ` Dave Chinner
1 sibling, 0 replies; 10+ messages in thread
From: Brian Foster @ 2021-02-08 17:45 UTC (permalink / raw)
To: Darrick J. Wong
Cc: Paul E. McKenney, Paul Menzel, Darrick J. Wong, linux-xfs,
Josh Triplett, rcu, it+linux-rcu, LKML
On Mon, Feb 08, 2021 at 09:28:24AM -0800, Darrick J. Wong wrote:
> On Mon, Feb 09, 2021 at 09:11:40AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 08, 2021 at 10:44:58AM -0500, Brian Foster wrote:
> > > On Mon, Feb 08, 2021 at 06:57:24AM -0800, Paul E. McKenney wrote:
> > > > On Mon, Feb 08, 2021 at 09:07:24AM -0500, Brian Foster wrote:
> > > > > On Fri, Feb 05, 2021 at 09:12:40AM -0800, Paul E. McKenney wrote:
> > > > > > On Fri, Feb 05, 2021 at 08:29:06AM +0100, Paul Menzel wrote:
> > > > > > > Dear Linux folks,
> > > > > > >
> > > > > > >
> > > > > > > On a Dell PowerEdge T630/0NT78X, BIOS 2.8.0 05/23/2018 with Linux 5.4.57, we
> > > > > > > twice saw a self-detected stall on a CPU (October 27th, 2020, January 18th,
> > > > > > > 2021).
> > > > > > >
> > > > > > > Both times, the workqueue is `xfs-conv/md0 xfs_end_io`.
> > > > > > >
> > > > > > > ```
> > > > > > > [ 0.000000] Linux version 5.4.57.mx64.340
> > > > > > > (root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC)) #1 SMP Tue Aug 11
> > > > > > > 13:20:33 CEST 2020
> > > > > > > […]
> > > > > > > [48962.981257] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > > > [48962.987511] rcu: 4-....: (20999 ticks this GP)
> > > > > > > idle=fe6/1/0x4000000000000002 softirq=3630188/3630188 fqs=4696
> > > > > > > [48962.998805] (t=21017 jiffies g=14529009 q=32263)
> > > > > > > [48963.004074] Task dump for CPU 4:
> > > > > > > [48963.007689] kworker/4:2 R running task 0 25587 2
> > > > > > > 0x80004008
> > > > > > > [48963.015591] Workqueue: xfs-conv/md0 xfs_end_io
> > > > > > > [48963.020570] Call Trace:
> > > > > > > [48963.023311] <IRQ>
> > > > > > > [48963.025560] sched_show_task+0x11e/0x150
> > > > > > > [48963.029957] rcu_dump_cpu_stacks+0x70/0xa0
> > > > > > > [48963.034545] rcu_sched_clock_irq+0x502/0x770
> > > > > > > [48963.039322] ? tick_sched_do_timer+0x60/0x60
> > > > > > > [48963.044106] update_process_times+0x24/0x60
> > > > > > > [48963.048791] tick_sched_timer+0x37/0x70
> > > > > > > [48963.053089] __hrtimer_run_queues+0x11f/0x2b0
> > > > > > > [48963.057960] ? recalibrate_cpu_khz+0x10/0x10
> > > > > > > [48963.062744] hrtimer_interrupt+0xe5/0x240
> > > > > > > [48963.067235] smp_apic_timer_interrupt+0x6f/0x130
> > > > > > > [48963.072407] apic_timer_interrupt+0xf/0x20
> > > > > > > [48963.076994] </IRQ>
> > > > > > > [48963.079347] RIP: 0010:_raw_spin_unlock_irqrestore+0xa/0x10
> > > > > > > [48963.085491] Code: f3 90 83 e8 01 75 e8 65 8b 3d 42 0f 56 7e e8 ed ea 5e
> > > > > > > ff 48 29 e8 4c 39 e8 76 cf 80 0b 08 eb 8c 0f 1f 44 00 00 c6 07 00 56 9d <c3>
> > > > > > > 0f 1f 44 00 00 0f 1f 44 00 00 b8 00 fe ff ff f0 0f c1 07 56 9d
> > > > > > > [48963.106524] RSP: 0018:ffffc9000738fd40 EFLAGS: 00000202 ORIG_RAX:
> > > > > > > ffffffffffffff13
> > > > > > > [48963.115003] RAX: ffffffff82407588 RBX: ffffffff82407580 RCX:
> > > > > > > ffffffff82407588
> > > > > > > [48963.122994] RDX: ffffffff82407588 RSI: 0000000000000202 RDI:
> > > > > > > ffffffff82407580
> > > > > > > [48963.130989] RBP: 0000000000000202 R08: ffffffff8203ea00 R09:
> > > > > > > 0000000000000001
> > > > > > > [48963.138982] R10: ffffc9000738fbb8 R11: 0000000000000001 R12:
> > > > > > > ffffffff82407588
> > > > > > > [48963.146976] R13: ffffea005e7ae600 R14: ffff8897b7e5a040 R15:
> > > > > > > ffffea005e7ae600
> > > > > > > [48963.154971] wake_up_page_bit+0xe0/0x100
> > > > > > > [48963.159366] xfs_destroy_ioend+0xce/0x1c0
> > > > > > > [48963.163857] xfs_end_ioend+0xcf/0x1a0
> > > > > > > [48963.167958] xfs_end_io+0xa4/0xd0
> > > > > > > [48963.171672] process_one_work+0x1e5/0x410
> > > > > > > [48963.176163] worker_thread+0x2d/0x3c0
> > > > > > > [48963.180265] ? cancel_delayed_work+0x90/0x90
> > > > > > > [48963.185048] kthread+0x117/0x130
> > > > > > > [48963.188663] ? kthread_create_worker_on_cpu+0x70/0x70
> > > > > > > [48963.194321] ret_from_fork+0x35/0x40
> > > > > > > ```
> > > > > > >
> > > > > > > As it’s just log level INFO, is there anything what should be done, or was
> > > > > > > the system probably just “overloaded”?
> > > > > >
> > > > > > I am assuming that you are building your kernel with CONFIG_PREEMPT_NONE=y
> > > > > > rather than CONFIG_PREEMPT_VOLUNTARY=y.
> > > > > >
> > > > > > If so, and if the problem is that you are temporarily overdriving xfs I/O,
> > > > > > one approach would be as follows:
> > > > > >
> > > > > > ------------------------------------------------------------------------
> > > > > >
> > > > > > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
> > > > > > index f16d5f1..06be426 100644
> > > > > > --- a/fs/xfs/xfs_aops.c
> > > > > > +++ b/fs/xfs/xfs_aops.c
> > > > > > @@ -390,6 +390,7 @@ xfs_end_io(
> > > > > > list_del_init(&ioend->io_list);
> > > > > > xfs_ioend_try_merge(ioend, &completion_list);
> > > > > > xfs_end_ioend(ioend);
> > > > > > + cond_resched();
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > ------------------------------------------------------------------------
> > > > >
> > > > > FWIW, this looks quite similar to the problem I attempted to fix with
> > > > > these patches:
> > > > >
> > > > > https://lore.kernel.org/linux-xfs/20201002153357.56409-1-bfoster@redhat.com/
> > > >
> > > > Looks plausible to me! Do you plan to re-post taking the feedback
> > > > into account?
> > >
> > > There was a v2 inline that incorporated some directed feedback.
> > > Otherwise there were questions and ideas about making the whole thing
> > > faster, but I've no idea if that addresses the problem or not (if so,
> > > that would be an entirely different set of patches). I'll wait and see
> > > what Darrick thinks about this and rebase/repost if the approach is
> > > agreeable..
> >
> > There is always the school of thought that says that the best way to
> > get people to focus on this is to rebase and repost. Otherwise, they
> > are all too likely to assume that you lost interest in this.
>
> I was hoping that a better solution would emerge for clearing
> PageWriteback on hundreds of thousands of pages, but nothing easy popped
> out.
>
> The hardcoded threshold in "[PATCH v2 2/2] xfs: kick extra large ioends
> to completion workqueue" gives me unease because who's to say if marking
> 262,144 pages on a particular CPU will actually stall it long enough to
> trip the hangcheck? Is the number lower on (say) some pokey NAS box
> with a lot of storage but a slow CPU?
>
> That said, /some/ threshold is probably better than no threshold. Could
> someone try to confirm if that series of Brian's fixes this problem too?
>
Note that this particular report looks like it's already in wq context,
so patch 1 by itself might be sufficient...
Brian
> --D
>
> > Thanx, Paul
> >
> > > Brian
> > >
> > > > Thanx, Paul
> > > >
> > > > > Brian
> > > > >
> > > > > >
> > > > > > If you have instead built with CONFIG_PREEMPT_VOLUNTARY=y, then your
> > > > > > problem is likely massive lock contention in wake_up_page_bit(), or
> > > > > > perhaps someone having failed to release that lock. The usual way to
> > > > > > work this out is by enabling lockdep (CONFIG_PROVE_LOCKING=y), but this
> > > > > > is often not what you want enabled in production.
> > > > > >
> > > > > > Darrick, thoughts from an xfs perspective?
> > > > > >
> > > > > > Thanx, Paul
> > > > > >
> > > > >
> > > >
> > >
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
2021-02-08 17:28 ` Darrick J. Wong
2021-02-08 17:45 ` Brian Foster
@ 2021-02-08 20:43 ` Dave Chinner
2021-02-08 21:24 ` Brian Foster
1 sibling, 1 reply; 10+ messages in thread
From: Dave Chinner @ 2021-02-08 20:43 UTC (permalink / raw)
To: Darrick J. Wong
Cc: Paul E. McKenney, Brian Foster, Paul Menzel, Darrick J. Wong,
linux-xfs, Josh Triplett, rcu, it+linux-rcu, LKML
On Mon, Feb 08, 2021 at 09:28:24AM -0800, Darrick J. Wong wrote:
> On Mon, Feb 09, 2021 at 09:11:40AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 08, 2021 at 10:44:58AM -0500, Brian Foster wrote:
> > > There was a v2 inline that incorporated some directed feedback.
> > > Otherwise there were questions and ideas about making the whole thing
> > > faster, but I've no idea if that addresses the problem or not (if so,
> > > that would be an entirely different set of patches). I'll wait and see
> > > what Darrick thinks about this and rebase/repost if the approach is
> > > agreeable..
> >
> > There is always the school of thought that says that the best way to
> > get people to focus on this is to rebase and repost. Otherwise, they
> > are all too likely to assume that you lost interest in this.
>
> I was hoping that a better solution would emerge for clearing
> PageWriteback on hundreds of thousands of pages, but nothing easy popped
> out.
>
> The hardcoded threshold in "[PATCH v2 2/2] xfs: kick extra large ioends
> to completion workqueue" gives me unease because who's to say if marking
> 262,144 pages on a particular CPU will actually stall it long enough to
> trip the hangcheck? Is the number lower on (say) some pokey NAS box
> with a lot of storage but a slow CPU?
It's also not the right thing to do given the IO completion
workqueue is a bound workqueue. Anything that is doing large amounts
of CPU intensive work should be on a unbound workqueue so that the
scheduler can bounce it around different CPUs as needed.
Quite frankly, the problem is a huge long ioend chain being built by
the submission code. We need to keep ioend completion overhead down.
It runs in either softirq or bound workqueue context and so
individual items of work that are performed in this context must not
be -unbounded- in size or time. Unbounded ioend chains are bad for
IO latency, they are bad for memory reclaim and they are bad for CPU
scheduling.
As I've said previously, we gain nothing by aggregating ioends past
a few tens of megabytes of submitted IO. The batching gains are
completely diminished once we've got enough IO in flight to keep the
submission queue full. We're talking here about gigabytes of
sequential IOs in a single ioend chain which are 2-3 orders of
magnitude larger than needed for optimal background IO submission
and completion efficiency and throughput. IOWs, we really should be
limiting the ioend chain length at submission time, not trying to
patch over bad completion behaviour that results from sub-optimal IO
submission behaviour...
> That said, /some/ threshold is probably better than no threshold. Could
> someone try to confirm if that series of Brian's fixes this problem too?
262144 pages is still too much work to be doing in a single softirq
IO completion callback. It's likely to be too much work for a bound
workqueue, too, especially when you consider that the workqueue
completion code will merge sequential ioends into one ioend, hence
making the IO completion loop counts bigger and latency problems worse
rather than better...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io
2021-02-08 20:43 ` Dave Chinner
@ 2021-02-08 21:24 ` Brian Foster
0 siblings, 0 replies; 10+ messages in thread
From: Brian Foster @ 2021-02-08 21:24 UTC (permalink / raw)
To: Dave Chinner
Cc: Darrick J. Wong, Paul E. McKenney, Paul Menzel, Darrick J. Wong,
linux-xfs, Josh Triplett, rcu, it+linux-rcu, LKML
On Tue, Feb 09, 2021 at 07:43:14AM +1100, Dave Chinner wrote:
> On Mon, Feb 08, 2021 at 09:28:24AM -0800, Darrick J. Wong wrote:
> > On Mon, Feb 09, 2021 at 09:11:40AM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 08, 2021 at 10:44:58AM -0500, Brian Foster wrote:
> > > > There was a v2 inline that incorporated some directed feedback.
> > > > Otherwise there were questions and ideas about making the whole thing
> > > > faster, but I've no idea if that addresses the problem or not (if so,
> > > > that would be an entirely different set of patches). I'll wait and see
> > > > what Darrick thinks about this and rebase/repost if the approach is
> > > > agreeable..
> > >
> > > There is always the school of thought that says that the best way to
> > > get people to focus on this is to rebase and repost. Otherwise, they
> > > are all too likely to assume that you lost interest in this.
> >
> > I was hoping that a better solution would emerge for clearing
> > PageWriteback on hundreds of thousands of pages, but nothing easy popped
> > out.
> >
> > The hardcoded threshold in "[PATCH v2 2/2] xfs: kick extra large ioends
> > to completion workqueue" gives me unease because who's to say if marking
> > 262,144 pages on a particular CPU will actually stall it long enough to
> > trip the hangcheck? Is the number lower on (say) some pokey NAS box
> > with a lot of storage but a slow CPU?
>
> It's also not the right thing to do given the IO completion
> workqueue is a bound workqueue. Anything that is doing large amounts
> of CPU intensive work should be on a unbound workqueue so that the
> scheduler can bounce it around different CPUs as needed.
>
> Quite frankly, the problem is a huge long ioend chain being built by
> the submission code. We need to keep ioend completion overhead down.
> It runs in either softirq or bound workqueue context and so
> individual items of work that are performed in this context must not
> be -unbounded- in size or time. Unbounded ioend chains are bad for
> IO latency, they are bad for memory reclaim and they are bad for CPU
> scheduling.
>
> As I've said previously, we gain nothing by aggregating ioends past
> a few tens of megabytes of submitted IO. The batching gains are
> completely diminished once we've got enough IO in flight to keep the
> submission queue full. We're talking here about gigabytes of
> sequential IOs in a single ioend chain which are 2-3 orders of
> magnitude larger than needed for optimal background IO submission
> and completion efficiency and throughput. IOWs, we really should be
> limiting the ioend chain length at submission time, not trying to
> patch over bad completion behaviour that results from sub-optimal IO
> submission behaviour...
>
That was the patch I posted prior to the aforementioned set. Granted, it
was an RFC, but for reference:
https://lore.kernel.org/linux-fsdevel/20200825144917.GA321765@bfoster/
(IIRC, you also had a variant that was essentially the same change.)
The discussion that followed in that thread was around the preference to
move completion of large chains into workqueue context instead of
breaking up the chains. The series referenced in my first reply fell out
of that as a targeted fix for the stall warning.
> > That said, /some/ threshold is probably better than no threshold. Could
> > someone try to confirm if that series of Brian's fixes this problem too?
>
> 262144 pages is still too much work to be doing in a single softirq
> IO completion callback. It's likely to be too much work for a bound
> workqueue, too, especially when you consider that the workqueue
> completion code will merge sequential ioends into one ioend, hence
> making the IO completion loop counts bigger and latency problems worse
> rather than better...
>
That was just a conservative number picked based on observation of the
original report (10+ GB ioends IIRC). I figured the review cycle would
involve narrowing it down to something more generically reasonable
(10s-100s of MB?) once we found an acceptable approach (and hopefully
received some testing feedback), but we've never really got to that
point..
Brian
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2021-02-08 21:47 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-05 7:29 rcu: INFO: rcu_sched self-detected stall on CPU: Workqueue: xfs-conv/md0 xfs_end_io Paul Menzel
2021-02-05 17:12 ` Paul E. McKenney
2021-02-08 14:07 ` Brian Foster
2021-02-08 14:57 ` Paul E. McKenney
2021-02-08 15:44 ` Brian Foster
2021-02-08 17:11 ` Paul E. McKenney
2021-02-08 17:28 ` Darrick J. Wong
2021-02-08 17:45 ` Brian Foster
2021-02-08 20:43 ` Dave Chinner
2021-02-08 21:24 ` Brian Foster
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).