All of lore.kernel.org
 help / color / mirror / Atom feed
* Soft lockup issue in Linux 4.1.9
@ 2015-09-30 21:59 Olivier Bonvalet
  2015-09-30 22:37 ` Holger Hoffstätte
       [not found] ` <560D1223.3070606@googlemail.com>
  0 siblings, 2 replies; 19+ messages in thread
From: Olivier Bonvalet @ 2015-09-30 21:59 UTC (permalink / raw)
  To: linux-kernel

Hi,

for information, I've just upgraded 6 servers from Linux 4.1.8 to Linux
4.1.9, and have some random soft lockup. If this can help :

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.1-dae-intel (root@yiu) (gcc version
5.2.1 20150911 (Debian 5.2.1-17) ) #1 SMP Wed Sep 30 09:30:00 CEST 2015
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.1-dae-intel
root=/dev/mapper/vg-root ro panic=60 quiet
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009abff]
usable
[    0.000000] BIOS-e820: [mem 0x000000000009ac00-0x000000000009ffff]
reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff]
reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007dec9fff]
usable
[    0.000000] BIOS-e820: [mem 0x000000007deca000-0x000000007df08fff]
reserved
[    0.000000] BIOS-e820: [mem 0x000000007df09000-0x000000007e011fff]
ACPI data
[    0.000000] BIOS-e820: [mem 0x000000007e012000-0x000000007e20ffff]
ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000007e210000-0x000000007f36bfff]
reserved
[    0.000000] BIOS-e820: [mem 0x000000007f36c000-0x000000007f7fffff]
ACPI NVS
[    0.000000] BIOS-e820: [mem 0x0000000080000000-0x000000008fffffff]
reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed3ffff]
reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff]
reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000087fffffff]
usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.7 present.
[    0.000000] DMI: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi
-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==>
reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x880000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 000000000000 mask 3FF800000000 write-back
[    0.000000]   1 base 000800000000 mask 3FFF80000000 write-back
[    0.000000]   2 base 000080000000 mask 3FFF80000000 uncachable
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000]   8 disabled
[    0.000000]   9 disabled
[    0.000000] PAT configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- UC 
[    0.000000] e820: update [mem 0x80000000-0xffffffff] usable ==>
reserved
[    0.000000] e820: last_pfn = 0x7deca max_arch_pfn = 0x400000000
[    0.000000] Base memory trampoline at [ffff880000094000] 94000 size
24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x017e0000, 0x017e0fff] PGTABLE
[    0.000000] BRK [0x017e1000, 0x017e1fff] PGTABLE
[    0.000000] BRK [0x017e2000, 0x017e2fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x87fe00000-0x87fffffff]
[    0.000000]  [mem 0x87fe00000-0x87fffffff] page 1G
[    0.000000] init_memory_mapping: [mem 0x860000000-0x87fdfffff]
[    0.000000]  [mem 0x860000000-0x87fdfffff] page 1G
[    0.000000] init_memory_mapping: [mem 0x00100000-0x7dec9fff]
[    0.000000]  [mem 0x00100000-0x001fffff] page 4k
[    0.000000]  [mem 0x00200000-0x7ddfffff] page 2M
[    0.000000]  [mem 0x7de00000-0x7dec9fff] page 4k
[    0.000000] init_memory_mapping: [mem 0x100000000-0x85fffffff]
[    0.000000]  [mem 0x100000000-0x85fffffff] page 1G
[    0.000000] RAMDISK: [mem 0x3690e000-0x3747efff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000F04A0 000024 (v02 SUPERM)
[    0.000000] ACPI: XSDT 0x000000007DF3E088 00008C (v01 SUPERM SMCI-
-MB 00000001 AMI  00010013)
[    0.000000] ACPI: FACP 0x000000007DF48DB0 0000F4 (v04 SUPERM SMCI-
-MB 00000001 AMI  00010013)
[    0.000000] ACPI: DSDT 0x000000007DF3E1A0 00AC09 (v02 SUPERM SMCI-
-MB 00000000 INTL 20091112)
[    0.000000] ACPI: FACS 0x000000007E207080 000040
[    0.000000] ACPI: APIC 0x000000007DF48EA8 0000C8 (v03               
  00000001 AMI  00010013)
[    0.000000] ACPI: FPDT 0x000000007DF48F70 000044 (v01               
  00000001 AMI  00010013)
[    0.000000] ACPI: HPET 0x000000007DF48FB8 000038 (v01 SUPERM SMCI-
-MB 00000001 AMI. 00000005)
[    0.000000] ACPI: PRAD 0x000000007DF48FF0 0000BE (v02 PRADID PRADTID
  00000001 MSFT 04000000)
[    0.000000] ACPI: SPMI 0x000000007DF490B0 000040 (v05 A M I  OEMSPMI
  00000000 AMI. 00000000)
[    0.000000] ACPI: SSDT 0x000000007DF490F0 0C7AE8 (v02 INTEL  CpuPm  
  00004000 INTL 20091112)
[    0.000000] ACPI: EINJ 0x000000007E010BD8 000130 (v01 AMI    AMI
EINJ 00000000      00000000)
[    0.000000] ACPI: ERST 0x000000007E010D08 000230 (v01 AMIER  AMI
ERST 00000000      00000000)
[    0.000000] ACPI: HEST 0x000000007E010F38 0000A8 (v01 AMI    AMI
HEST 00000000      00000000)
[    0.000000] ACPI: BERT 0x000000007E010FE0 000030 (v01 AMI    AMI
BERT 00000000      00000000)
[    0.000000] ACPI: DMAR 0x000000007E011010 0000BC (v01 A M I  OEMDMAR
  00000001 INTL 00000001)
[    0.000000] ACPI: MCFG 0x000000007E0110D0 00003C (v01 SUPERM SMCI-
-MB 00000001 MSFT 00000097)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000
-0x000000087fffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x87fffa000-0x87fffdfff]
[    0.000000]  [ffffea0000000000-ffffea001dbfffff] PMD ->
[ffff88085f600000-ffff88087b5fffff] on node 0
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000087fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x0000000000099fff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x000000007dec9fff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000087fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000
-0x000000087fffffff]
[    0.000000] On node 0 totalpages: 8380003
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3993 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 6996 pages used for memmap
[    0.000000]   DMA32 zone: 511690 pages, LIFO batch:31
[    0.000000]   Normal zone: 107520 pages used for memmap
[    0.000000]   Normal zone: 7864320 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x408
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000,
GSI 0-23
[    0.000000] IOAPIC[1]: apic_id 2, version 32, address 0xfec01000,
GSI 24-47
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high
level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000
-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009a000
-0x0009afff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009b000
-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000
-0x000dffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000e0000
-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x7deca000
-0x7df08fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7df09000
-0x7e011fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7e012000
-0x7e20ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x7e210000
-0x7f36bfff]
[    0.000000] PM: Registered nosave memory: [mem 0x7f36c000
-0x7f7fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x7f800000
-0x7fffffff]
[    0.000000] PM: Registered nosave memory: [mem 0x80000000
-0x8fffffff]
[    0.000000] PM: Registered nosave memory: [mem 0x90000000
-0xfed1bfff]
[    0.000000] PM: Registered nosave memory: [mem 0xfed1c000
-0xfed3ffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfed40000
-0xfeffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xff000000
-0xffffffff]
[    0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI
devices
[    0.000000] clocksource refined-jiffies: mask: 0xffffffff
max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:8
nr_node_ids:1
[    0.000000] PERCPU: Embedded 30 pages/cpu @ffff88087fc00000 s85400
r8192 d29288 u262144
[    0.000000] pcpu-alloc: s85400 r8192 d29288 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on. 
 Total pages: 8265410
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: rootdelay=15 BOOT_IMAGE=/vmlinuz
-4.1-dae-intel root=/dev/mapper/vg-root ro panic=60 quiet
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using
standard form
[    0.000000] Memory: 32974076K/33520012K available (3739K kernel
code, 552K rwdata, 1540K rodata, 800K init, 540K bss, 545936K reserved,
0K cma-reserved)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000]  RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=8.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16,
nr_cpu_ids=8
[    0.000000] NR_IRQS:4352 nr_irqs:896 16
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty0] enabled
[    0.000000] clocksource hpet: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 133484882848 ns
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 3700.251 MHz processor
[    0.000008] Calibrating delay loop (skipped), value calculated using
timer frequency.. 7400.50 BogoMIPS (lpj=14801004)
[    0.000010] pid_max: default: 32768 minimum: 301
[    0.000014] ACPI: Core revision 20150410
[    0.028328] ACPI: All ACPI Tables successfully acquired
[    0.029510] Dentry cache hash table entries: 4194304 (order: 13,
33554432 bytes)
[    0.034787] Inode-cache hash table entries: 2097152 (order: 12,
16777216 bytes)
[    0.037150] Mount-cache hash table entries: 65536 (order: 7, 524288
bytes)
[    0.037173] Mountpoint-cache hash table entries: 65536 (order: 7,
524288 bytes)
[    0.037388] Initializing cgroup subsys blkio
[    0.037390] Initializing cgroup subsys devices
[    0.037392] Initializing cgroup subsys freezer
[    0.037394] Initializing cgroup subsys net_cls
[    0.037395] Initializing cgroup subsys perf_event
[    0.037414] CPU: Physical Processor ID: 0
[    0.037414] CPU: Processor Core ID: 0
[    0.037418] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.037418] ENERGY_PERF_BIAS: View and update with
x86_energy_perf_policy(8)
[    0.038133] mce: CPU supports 21 MCE banks
[    0.038155] CPU0: Thermal monitoring enabled (TM1)
[    0.038172] process: using mwait in idle threads
[    0.038174] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 8
[    0.038175] Last level dTLB entries: 4KB 512, 2MB 0, 4MB 0, 1GB 4
[    0.038278] Freeing SMP alternatives memory: 12K (ffffffff81754000 -
ffffffff81757000)
[    0.038577] dmar: Host address width 46
[    0.038578] dmar: DRHD base: 0x000000fbffc000 flags: 0x1
[    0.038583] dmar: IOMMU 0: reg_base_addr fbffc000 ver 1:0 cap
d2078c106f0466 ecap f020df
[    0.038584] dmar: RMRR base: 0x0000007dedf000 end: 0x0000007deebfff
[    0.038585] dmar: ATSR flags: 0x0
[    0.038586] dmar: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.038588] IOAPIC id 0 under DRHD base  0xfbffc000 IOMMU 0
[    0.038589] IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 0
[    0.038590] HPET id 0 under DRHD base 0xfbffc000
[    0.038737] Queued invalidation will be enabled to support x2apic
and Intr-remapping.
[    0.038742] Enabled IRQ remapping in x2apic mode
[    0.038743] x2apic enabled
[    0.038747] Switched APIC routing to cluster x2apic.
[    0.039237] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.078960] TSC deadline timer enabled
[    0.078962] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz
(fam: 06, model: 3e, stepping: 04)
[    0.078973] Performance Events: PEBS fmt1+, 16-deep LBR, IvyBridge
events, full-width counters, Intel PMU driver.
[    0.078987] ... version:                3
[    0.078988] ... bit width:              48
[    0.078989] ... generic registers:      4
[    0.078989] ... value mask:             0000ffffffffffff
[    0.078990] ... max period:             0000ffffffffffff
[    0.078990] ... fixed-purpose events:   3
[    0.078991] ... event mask:             000000070000000f
[    0.079191] x86: Booting SMP configuration:
[    0.079192] .... node  #0, CPUs:      #1
[    0.092995] NMI watchdog: enabled on all CPUs, permanently consumes
one hw-PMU counter.
[    0.093052]  #2 #3 #4 #5 #6 #7
[    0.175851] x86: Booted up 1 node, 8 CPUs
[    0.175854] smpboot: Total of 8 processors activated (59204.01
BogoMIPS)
[    0.182067] devtmpfs: initialized
[    0.187606] PM: Registering ACPI NVS region [mem 0x7e012000
-0x7e20ffff] (2088960 bytes)
[    0.187642] PM: Registering ACPI NVS region [mem 0x7f36c000
-0x7f7fffff] (4800512 bytes)
[    0.187768] clocksource jiffies: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.189034] NET: Registered protocol family 16
[    0.197966] cpuidle: using governor ladder
[    0.209979] cpuidle: using governor menu
[    0.210012] ACPI FADT declares the system doesn't support PCIe ASPM,
so disable it
[    0.210013] ACPI: bus type PCI registered
[    0.210014] acpiphp: ACPI Hot Plug PCI Controller Driver version:
0.5
[    0.210075] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem
0x80000000-0x8fffffff] (base 0x80000000)
[    0.210076] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in
E820
[    0.210208] PCI: Using configuration type 1 for base access
[    0.210697] perf_event_intel: PMU erratum BJ122, BV98, HSD29 worked
around, HT is on
[    0.223155] ACPI: Added _OSI(Module Device)
[    0.223156] ACPI: Added _OSI(Processor Device)
[    0.223157] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.223158] ACPI: Added _OSI(Processor Aggregator Device)
[    0.231566] ACPI: Executed 1 blocks of module-level executable AML
code
[    0.469486] \_SB_:_OSC invalid UUID
[    0.469488] _OSC request data:1 1f 
[    0.470409] ACPI: Interpreter enabled
[    0.470415] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep
State [\_S2_] (20150410/hwxface-580)
[    0.470418] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep
State [\_S3_] (20150410/hwxface-580)
[    0.470424] ACPI: (supports S0 S1 S4 S5)
[    0.470425] ACPI: Using IOAPIC for interrupt routing
[    0.470473] HEST: Table parsing has been initialized.
[    0.470475] PCI: Using host bridge windows from ACPI; if necessary,
use "pci=nocrs" and report a bug
[    0.480735] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe])
[    0.480738] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM
ClockPM Segments MSI]
[    0.480830] acpi PNP0A08:00: _OSC: platform does not support [PME
AER]
[    0.480912] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug
PCIeCapability]
[    0.480913] acpi PNP0A08:00: FADT indicates ASPM is unsupported,
using BIOS configuration
[    0.481065] PCI host bridge to bus 0000:00
[    0.481067] pci_bus 0000:00: root bus resource [bus 00-fe]
[    0.481069] pci_bus 0000:00: root bus resource [io  0x0000-0x03af
window]
[    0.481070] pci_bus 0000:00: root bus resource [io  0x03e0-0x0cf7
window]
[    0.481071] pci_bus 0000:00: root bus resource [io  0x03b0-0x03df
window]
[    0.481072] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff
window]
[    0.481073] pci_bus 0000:00: root bus resource [mem 0x000a0000
-0x000bffff window]
[    0.481074] pci_bus 0000:00: root bus resource [mem 0x000c0000
-0x000dffff window]
[    0.481075] pci_bus 0000:00: root bus resource [mem 0xfed08000
-0xfed08fff window]
[    0.481076] pci_bus 0000:00: root bus resource [mem 0xfed0e000
-0xfed0ffff window]
[    0.481077] pci_bus 0000:00: root bus resource [mem 0x80000000
-0xfbffffff window]
[    0.481086] pci 0000:00:00.0: [8086:0e00] type 00 class 0x060000
[    0.481138] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    0.481199] pci 0000:00:01.0: [8086:0e02] type 01 class 0x060400
[    0.481256] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[    0.481291] pci 0000:00:01.0: System wakeup disabled by ACPI
[    0.481323] pci 0000:00:02.0: [8086:0e04] type 01 class 0x060400
[    0.481379] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
[    0.481416] pci 0000:00:02.0: System wakeup disabled by ACPI
[    0.481448] pci 0000:00:03.0: [8086:0e08] type 01 class 0x060400
[    0.481505] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[    0.481539] pci 0000:00:03.0: System wakeup disabled by ACPI
[    0.481568] pci 0000:00:03.2: [8086:0e0a] type 01 class 0x060400
[    0.481623] pci 0000:00:03.2: PME# supported from D0 D3hot D3cold
[    0.481657] pci 0000:00:03.2: System wakeup disabled by ACPI
[    0.481686] pci 0000:00:04.0: [8086:0e20] type 00 class 0x088000
[    0.481698] pci 0000:00:04.0: reg 0x10: [mem 0xfb81c000-0xfb81ffff
64bit]
[    0.481791] pci 0000:00:04.1: [8086:0e21] type 00 class 0x088000
[    0.481803] pci 0000:00:04.1: reg 0x10: [mem 0xfb818000-0xfb81bfff
64bit]
[    0.481895] pci 0000:00:04.2: [8086:0e22] type 00 class 0x088000
[    0.481907] pci 0000:00:04.2: reg 0x10: [mem 0xfb814000-0xfb817fff
64bit]
[    0.481999] pci 0000:00:04.3: [8086:0e23] type 00 class 0x088000
[    0.482011] pci 0000:00:04.3: reg 0x10: [mem 0xfb810000-0xfb813fff
64bit]
[    0.482102] pci 0000:00:04.4: [8086:0e24] type 00 class 0x088000
[    0.482114] pci 0000:00:04.4: reg 0x10: [mem 0xfb80c000-0xfb80ffff
64bit]
[    0.482206] pci 0000:00:04.5: [8086:0e25] type 00 class 0x088000
[    0.482218] pci 0000:00:04.5: reg 0x10: [mem 0xfb808000-0xfb80bfff
64bit]
[    0.482310] pci 0000:00:04.6: [8086:0e26] type 00 class 0x088000
[    0.482322] pci 0000:00:04.6: reg 0x10: [mem 0xfb804000-0xfb807fff
64bit]
[    0.482413] pci 0000:00:04.7: [8086:0e27] type 00 class 0x088000
[    0.482425] pci 0000:00:04.7: reg 0x10: [mem 0xfb800000-0xfb803fff
64bit]
[    0.482516] pci 0000:00:05.0: [8086:0e28] type 00 class 0x088000
[    0.482601] pci 0000:00:05.2: [8086:0e2a] type 00 class 0x088000
[    0.482686] pci 0000:00:05.4: [8086:0e2c] type 00 class 0x080020
[    0.482695] pci 0000:00:05.4: reg 0x10: [mem 0xfb825000-0xfb825fff]
[    0.482796] pci 0000:00:11.0: [8086:1d3e] type 01 class 0x060400
[    0.482872] pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
[    0.482939] pci 0000:00:16.0: [8086:1d3a] type 00 class 0x078000
[    0.482955] pci 0000:00:16.0: reg 0x10: [mem 0xfed0e000-0xfed0e00f
64bit]
[    0.483005] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
[    0.483058] pci 0000:00:16.1: [8086:1d3b] type 00 class 0x078000
[    0.483073] pci 0000:00:16.1: reg 0x10: [mem 0xfed0f000-0xfed0f00f
64bit]
[    0.483123] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
[    0.483186] pci 0000:00:1a.0: [8086:1d2d] type 00 class 0x0c0320
[    0.483201] pci 0000:00:1a.0: reg 0x10: [mem 0xfb823000-0xfb8233ff]
[    0.483269] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
[    0.483299] pci 0000:00:1a.0: System wakeup disabled by ACPI
[    0.483330] pci 0000:00:1d.0: [8086:1d26] type 00 class 0x0c0320
[    0.483345] pci 0000:00:1d.0: reg 0x10: [mem 0xfb822000-0xfb8223ff]
[    0.483413] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[    0.483442] pci 0000:00:1d.0: System wakeup disabled by ACPI
[    0.483469] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
[    0.483530] pci 0000:00:1e.0: System wakeup disabled by ACPI
[    0.483561] pci 0000:00:1f.0: [8086:1d41] type 00 class 0x060100
[    0.483698] pci 0000:00:1f.2: [8086:1d02] type 00 class 0x010601
[    0.483710] pci 0000:00:1f.2: reg 0x10: [io  0xf050-0xf057]
[    0.483716] pci 0000:00:1f.2: reg 0x14: [io  0xf040-0xf043]
[    0.483722] pci 0000:00:1f.2: reg 0x18: [io  0xf030-0xf037]
[    0.483728] pci 0000:00:1f.2: reg 0x1c: [io  0xf020-0xf023]
[    0.483734] pci 0000:00:1f.2: reg 0x20: [io  0xf000-0xf01f]
[    0.483740] pci 0000:00:1f.2: reg 0x24: [mem 0xfb821000-0xfb8217ff]
[    0.483769] pci 0000:00:1f.2: PME# supported from D3hot
[    0.483821] pci 0000:00:1f.3: [8086:1d22] type 00 class 0x0c0500
[    0.483833] pci 0000:00:1f.3: reg 0x10: [mem 0xfb820000-0xfb8200ff
64bit]
[    0.483850] pci 0000:00:1f.3: reg 0x20: [io  0x1180-0x119f]
[    0.483912] pci 0000:00:1f.6: [8086:1d24] type 00 class 0x118000
[    0.483929] pci 0000:00:1f.6: reg 0x10: [mem 0xfed08000-0xfed08fff
64bit]
[    0.484055] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.484093] pci 0000:00:02.0: PCI bridge to [bus 02]
[    0.484139] pci 0000:03:00.0: [8086:1521] type 00 class 0x020000
[    0.484148] pci 0000:03:00.0: reg 0x10: [mem 0xfb720000-0xfb73ffff]
[    0.484164] pci 0000:03:00.0: reg 0x1c: [mem 0xfb7c4000-0xfb7c7fff]
[    0.484216] pci 0000:03:00.0: PME# supported from D0 D3hot D3cold
[    0.484238] pci 0000:03:00.0: reg 0x184: [mem 0xfb7a0000-0xfb7a3fff]
[    0.484240] pci 0000:03:00.0: VF(n) BAR0 space: [mem 0xfb7a0000
-0xfb7bffff] (contains BAR0 for 8 VFs)
[    0.484253] pci 0000:03:00.0: reg 0x190: [mem 0xfb780000-0xfb783fff]
[    0.484255] pci 0000:03:00.0: VF(n) BAR3 space: [mem 0xfb780000
-0xfb79ffff] (contains BAR3 for 8 VFs)
[    0.484318] pci 0000:03:00.1: [8086:1521] type 00 class 0x020000
[    0.484328] pci 0000:03:00.1: reg 0x10: [mem 0xfb700000-0xfb71ffff]
[    0.484344] pci 0000:03:00.1: reg 0x1c: [mem 0xfb7c0000-0xfb7c3fff]
[    0.484393] pci 0000:03:00.1: PME# supported from D0 D3hot D3cold
[    0.484411] pci 0000:03:00.1: reg 0x184: [mem 0xfb760000-0xfb763fff]
[    0.484412] pci 0000:03:00.1: VF(n) BAR0 space: [mem 0xfb760000
-0xfb77ffff] (contains BAR0 for 8 VFs)
[    0.484426] pci 0000:03:00.1: reg 0x190: [mem 0xfb740000-0xfb743fff]
[    0.484428] pci 0000:03:00.1: VF(n) BAR3 space: [mem 0xfb740000
-0xfb75ffff] (contains BAR3 for 8 VFs)
[    0.489427] pci 0000:00:03.0: PCI bridge to [bus 03-04]
[    0.489431] pci 0000:00:03.0:   bridge window [mem 0xfb700000
-0xfb7fffff]
[    0.489471] pci 0000:05:00.0: [1000:0070] type 00 class 0x010700
[    0.489478] pci 0000:05:00.0: reg 0x10: [io  0xe000-0xe0ff]
[    0.489485] pci 0000:05:00.0: reg 0x14: [mem 0xfb600000-0xfb603fff
64bit]
[    0.489491] pci 0000:05:00.0: reg 0x1c: [mem 0xfb580000-0xfb5bffff
64bit]
[    0.489499] pci 0000:05:00.0: reg 0x30: [mem 0xfb100000-0xfb17ffff
pref]
[    0.489524] pci 0000:05:00.0: supports D1 D2
[    0.489540] pci 0000:05:00.0: reg 0x174: [mem 0xfb5c0000-0xfb5c3fff
64bit]
[    0.489541] pci 0000:05:00.0: VF(n) BAR0 space: [mem 0xfb5c0000
-0xfb5fffff 64bit] (contains BAR0 for 16 VFs)
[    0.489547] pci 0000:05:00.0: reg 0x17c: [mem 0xfb180000-0xfb1bffff
64bit]
[    0.489548] pci 0000:05:00.0: VF(n) BAR2 space: [mem 0xfb180000
-0xfb57ffff 64bit] (contains BAR2 for 16 VFs)
[    0.497432] pci 0000:00:03.2: PCI bridge to [bus 05]
[    0.497434] pci 0000:00:03.2:   bridge window [io  0xe000-0xefff]
[    0.497437] pci 0000:00:03.2:   bridge window [mem 0xfb100000
-0xfb6fffff]
[    0.497492] pci 0000:06:00.0: [8086:1d6b] type 00 class 0x010700
[    0.497510] pci 0000:06:00.0: reg 0x10: [mem 0xfa47c000-0xfa47ffff
64bit pref]
[    0.497523] pci 0000:06:00.0: reg 0x18: [mem 0xfa000000-0xfa3fffff
64bit pref]
[    0.497532] pci 0000:06:00.0: reg 0x20: [io  0xd000-0xd0ff]
[    0.497625] pci 0000:06:00.0: reg 0x164: [mem 0xfa400000-0xfa403fff
64bit pref]
[    0.497626] pci 0000:06:00.0: VF(n) BAR0 space: [mem 0xfa400000
-0xfa47bfff 64bit pref] (contains BAR0 for 31 VFs)
[    0.497785] pci 0000:00:11.0: PCI bridge to [bus 06]
[    0.497789] pci 0000:00:11.0:   bridge window [io  0xd000-0xdfff]
[    0.497796] pci 0000:00:11.0:   bridge window [mem 0xfa000000
-0xfa4fffff 64bit pref]
[    0.497830] pci 0000:07:04.0: [102b:0532] type 00 class 0x030000
[    0.497844] pci 0000:07:04.0: reg 0x10: [mem 0xf9000000-0xf9ffffff
pref]
[    0.497852] pci 0000:07:04.0: reg 0x14: [mem 0xfb000000-0xfb003fff]
[    0.497860] pci 0000:07:04.0: reg 0x18: [mem 0xfa800000-0xfaffffff]
[    0.497959] pci 0000:00:1e.0: PCI bridge to [bus 07] (subtractive
decode)
[    0.497963] pci 0000:00:1e.0:   bridge window [mem 0xfa800000
-0xfb0fffff]
[    0.497967] pci 0000:00:1e.0:   bridge window [mem 0xf9000000
-0xf9ffffff 64bit pref]
[    0.497968] pci 0000:00:1e.0:   bridge window [io  0x0000-0x03af
window] (subtractive decode)
[    0.497970] pci 0000:00:1e.0:   bridge window [io  0x03e0-0x0cf7
window] (subtractive decode)
[    0.497971] pci 0000:00:1e.0:   bridge window [io  0x03b0-0x03df
window] (subtractive decode)
[    0.497972] pci 0000:00:1e.0:   bridge window [io  0x0d00-0xffff
window] (subtractive decode)
[    0.497973] pci 0000:00:1e.0:   bridge window [mem 0x000a0000
-0x000bffff window] (subtractive decode)
[    0.497974] pci 0000:00:1e.0:   bridge window [mem 0x000c0000
-0x000dffff window] (subtractive decode)
[    0.497975] pci 0000:00:1e.0:   bridge window [mem 0xfed08000
-0xfed08fff window] (subtractive decode)
[    0.497976] pci 0000:00:1e.0:   bridge window [mem 0xfed0e000
-0xfed0ffff window] (subtractive decode)
[    0.497978] pci 0000:00:1e.0:   bridge window [mem 0x80000000
-0xfbffffff window] (subtractive decode)
[    0.498014] pci_bus 0000:00: on NUMA node 0
[    0.498369] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus ff])
[    0.498371] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM
ClockPM Segments MSI]
[    0.498385] acpi PNP0A03:00: _OSC: OS now controls [PCIeHotplug PME
AER PCIeCapability]
[    0.498386] acpi PNP0A03:00: FADT indicates ASPM is unsupported,
using BIOS configuration
[    0.498422] PCI host bridge to bus 0000:ff
[    0.498424] pci_bus 0000:ff: root bus resource [bus ff]
[    0.498429] pci 0000:ff:08.0: [8086:0e80] type 00 class 0x088000
[    0.498470] pci 0000:ff:09.0: [8086:0e90] type 00 class 0x088000
[    0.498509] pci 0000:ff:0a.0: [8086:0ec0] type 00 class 0x088000
[    0.498542] pci 0000:ff:0a.1: [8086:0ec1] type 00 class 0x088000
[    0.498575] pci 0000:ff:0a.2: [8086:0ec2] type 00 class 0x088000
[    0.498608] pci 0000:ff:0a.3: [8086:0ec3] type 00 class 0x088000
[    0.498642] pci 0000:ff:0b.0: [8086:0e1e] type 00 class 0x088000
[    0.498672] pci 0000:ff:0b.3: [8086:0e1f] type 00 class 0x088000
[    0.498704] pci 0000:ff:0c.0: [8086:0ee0] type 00 class 0x088000
[    0.498734] pci 0000:ff:0c.1: [8086:0ee2] type 00 class 0x088000
[    0.498766] pci 0000:ff:0d.0: [8086:0ee1] type 00 class 0x088000
[    0.498798] pci 0000:ff:0d.1: [8086:0ee3] type 00 class 0x088000
[    0.498830] pci 0000:ff:0e.0: [8086:0ea0] type 00 class 0x088000
[    0.498863] pci 0000:ff:0e.1: [8086:0e30] type 00 class 0x110100
[    0.498900] pci 0000:ff:0f.0: [8086:0ea8] type 00 class 0x088000
[    0.498940] pci 0000:ff:0f.1: [8086:0e71] type 00 class 0x088000
[    0.498980] pci 0000:ff:0f.2: [8086:0eaa] type 00 class 0x088000
[    0.499019] pci 0000:ff:0f.3: [8086:0eab] type 00 class 0x088000
[    0.499059] pci 0000:ff:0f.4: [8086:0eac] type 00 class 0x088000
[    0.499099] pci 0000:ff:0f.5: [8086:0ead] type 00 class 0x088000
[    0.499140] pci 0000:ff:10.0: [8086:0eb0] type 00 class 0x088000
[    0.499180] pci 0000:ff:10.1: [8086:0eb1] type 00 class 0x088000
[    0.499219] pci 0000:ff:10.2: [8086:0eb2] type 00 class 0x088000
[    0.499259] pci 0000:ff:10.3: [8086:0eb3] type 00 class 0x088000
[    0.499299] pci 0000:ff:10.4: [8086:0eb4] type 00 class 0x088000
[    0.499338] pci 0000:ff:10.5: [8086:0eb5] type 00 class 0x088000
[    0.499378] pci 0000:ff:10.6: [8086:0eb6] type 00 class 0x088000
[    0.499418] pci 0000:ff:10.7: [8086:0eb7] type 00 class 0x088000
[    0.499457] pci 0000:ff:13.0: [8086:0e1d] type 00 class 0x088000
[    0.499489] pci 0000:ff:13.1: [8086:0e34] type 00 class 0x110100
[    0.499521] pci 0000:ff:13.4: [8086:0e81] type 00 class 0x088000
[    0.499553] pci 0000:ff:13.5: [8086:0e36] type 00 class 0x110100
[    0.499585] pci 0000:ff:16.0: [8086:0ec8] type 00 class 0x088000
[    0.499616] pci 0000:ff:16.1: [8086:0ec9] type 00 class 0x088000
[    0.499647] pci 0000:ff:16.2: [8086:0eca] type 00 class 0x088000
[    0.499683] pci_bus 0000:ff: on NUMA node 0
[    0.499739] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 *11
12 14 15)
[    0.499770] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11
12 14 15)
[    0.499799] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 10 *11 12
14 15)
[    0.499827] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 10 11 12
14 15)
[    0.499855] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12
14 15) *0
[    0.499884] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12
14 15) *0
[    0.499913] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12
14 15) *0
[    0.499941] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *7 10 11
12 14 15)
[    0.500685] ACPI: Enabled 1 GPEs in block 00 to 3F
[    0.500754] vgaarb: setting as boot device: PCI:0000:07:04.0
[    0.500755] vgaarb: device added:
PCI:0000:07:04.0,decodes=io+mem,owns=io+mem,locks=none
[    0.500759] vgaarb: loaded
[    0.500760] vgaarb: bridge control possible 0000:07:04.0
[    0.500762] ACPI: bus type USB registered
[    0.500778] usbcore: registered new interface driver usbfs
[    0.500786] usbcore: registered new interface driver hub
[    0.500806] usbcore: registered new device driver usb
[    0.500860] PCI: Using ACPI for IRQ routing
[    0.506380] PCI: pci_cache_line_size set to 64 bytes
[    0.506459] e820: reserve RAM buffer [mem 0x0009ac00-0x0009ffff]
[    0.506460] e820: reserve RAM buffer [mem 0x7deca000-0x7fffffff]
[    0.506581] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    0.506584] hpet0: 8 comparators, 64-bit 14.318180 MHz counter
[    0.508605] Switched to clocksource hpet
[    0.508682] pnp: PnP ACPI init
[    0.508729] system 00:00: [mem 0xfc000000-0xfcffffff] has been
reserved
[    0.508731] system 00:00: [mem 0xfd000000-0xfdffffff] has been
reserved
[    0.508733] system 00:00: [mem 0xfe000000-0xfeafffff] has been
reserved
[    0.508734] system 00:00: [mem 0xfeb00000-0xfebfffff] has been
reserved
[    0.508736] system 00:00: Plug and Play ACPI device, IDs PNP0c01
(active)
[    0.508786] system 00:01: [mem 0xfbffc000-0xfbffdfff] could not be
reserved
[    0.508787] system 00:01: Plug and Play ACPI device, IDs PNP0c02
(active)
[    0.508811] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00
(active)
[    0.508854] system 00:03: [io  0x04d0-0x04d1] has been reserved
[    0.508856] system 00:03: [mem 0x00000400-0x000004ff] could not be
reserved
[    0.508857] system 00:03: Plug and Play ACPI device, IDs PNP0c02
(active)
[    0.508917] system 00:04: [io  0x0a30-0x0a3f] has been reserved
[    0.508919] system 00:04: Plug and Play ACPI device, IDs PNP0c02
(active)
[    0.508965] pnp 00:05: Plug and Play ACPI device, IDs PNP0f03
PNP0f13 (active)
[    0.509086] pnp 00:06: [dma 0 disabled]
[    0.509118] pnp 00:06: Plug and Play ACPI device, IDs PNP0501
(active)
[    0.509235] pnp 00:07: [dma 0 disabled]
[    0.509264] pnp 00:07: Plug and Play ACPI device, IDs PNP0501
(active)
[    0.509335] system 00:08: [io  0x0b00-0x0b7f] has been reserved
[    0.509336] system 00:08: Plug and Play ACPI device, IDs PNP0c02
(active)
[    0.509439] pnp 00:09: [dma 0 disabled]
[    0.509498] pnp 00:09: Plug and Play ACPI device, IDs PNP0501
(active)
[    0.509564] pnp 00:0a: Plug and Play ACPI device, IDs IPI0001
(active)
[    0.509662] system 00:0b: [io  0x0400-0x0453] could not be reserved
[    0.509663] system 00:0b: [io  0x0458-0x047f] has been reserved
[    0.509664] system 00:0b: [io  0x1180-0x119f] has been reserved
[    0.509666] system 00:0b: [io  0x0500-0x057f] has been reserved
[    0.509667] system 00:0b: [mem 0xfed1c000-0xfed1ffff] has been
reserved
[    0.509669] system 00:0b: [mem 0xfec00000-0xfecfffff] could not be
reserved
[    0.509670] system 00:0b: [mem 0xff000000-0xffffffff] has been
reserved
[    0.509671] system 00:0b: Plug and Play ACPI device, IDs PNP0c01
(active)
[    0.509715] system 00:0c: [io  0x0454-0x0457] has been reserved
[    0.509717] system 00:0c: Plug and Play ACPI device, IDs INT3f0d
PNP0c02 (active)
[    0.509855] system 00:0d: [mem 0x00000000-0x0009ffff] could not be
reserved
[    0.509857] system 00:0d: Plug and Play ACPI device, IDs PNP0c01
(active)
[    0.509958] pnp: PnP ACPI: found 14 devices
[    0.515837] clocksource acpi_pm: mask: 0xffffff max_cycles:
0xffffff, max_idle_ns: 2085701024 ns
[    0.515878] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.515885] pci 0000:00:02.0: PCI bridge to [bus 02]
[    0.515893] pci 0000:00:03.0: PCI bridge to [bus 03-04]
[    0.515896] pci 0000:00:03.0:   bridge window [mem 0xfb700000
-0xfb7fffff]
[    0.515901] pci 0000:00:03.2: PCI bridge to [bus 05]
[    0.515903] pci 0000:00:03.2:   bridge window [io  0xe000-0xefff]
[    0.515906] pci 0000:00:03.2:   bridge window [mem 0xfb100000
-0xfb6fffff]
[    0.515911] pci 0000:00:11.0: PCI bridge to [bus 06]
[    0.515913] pci 0000:00:11.0:   bridge window [io  0xd000-0xdfff]
[    0.515920] pci 0000:00:11.0:   bridge window [mem 0xfa000000
-0xfa4fffff 64bit pref]
[    0.515926] pci 0000:00:1e.0: PCI bridge to [bus 07]
[    0.515929] pci 0000:00:1e.0:   bridge window [mem 0xfa800000
-0xfb0fffff]
[    0.515932] pci 0000:00:1e.0:   bridge window [mem 0xf9000000
-0xf9ffffff 64bit pref]
[    0.515936] pci_bus 0000:00: resource 4 [io  0x0000-0x03af window]
[    0.515937] pci_bus 0000:00: resource 5 [io  0x03e0-0x0cf7 window]
[    0.515938] pci_bus 0000:00: resource 6 [io  0x03b0-0x03df window]
[    0.515939] pci_bus 0000:00: resource 7 [io  0x0d00-0xffff window]
[    0.515940] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff
window]
[    0.515942] pci_bus 0000:00: resource 9 [mem 0x000c0000-0x000dffff
window]
[    0.515943] pci_bus 0000:00: resource 10 [mem 0xfed08000-0xfed08fff
window]
[    0.515944] pci_bus 0000:00: resource 11 [mem 0xfed0e000-0xfed0ffff
window]
[    0.515945] pci_bus 0000:00: resource 12 [mem 0x80000000-0xfbffffff
window]
[    0.515946] pci_bus 0000:03: resource 1 [mem 0xfb700000-0xfb7fffff]
[    0.515947] pci_bus 0000:05: resource 0 [io  0xe000-0xefff]
[    0.515948] pci_bus 0000:05: resource 1 [mem 0xfb100000-0xfb6fffff]
[    0.515949] pci_bus 0000:06: resource 0 [io  0xd000-0xdfff]
[    0.515950] pci_bus 0000:06: resource 2 [mem 0xfa000000-0xfa4fffff
64bit pref]
[    0.515951] pci_bus 0000:07: resource 1 [mem 0xfa800000-0xfb0fffff]
[    0.515952] pci_bus 0000:07: resource 2 [mem 0xf9000000-0xf9ffffff
64bit pref]
[    0.515953] pci_bus 0000:07: resource 4 [io  0x0000-0x03af window]
[    0.515954] pci_bus 0000:07: resource 5 [io  0x03e0-0x0cf7 window]
[    0.515955] pci_bus 0000:07: resource 6 [io  0x03b0-0x03df window]
[    0.515957] pci_bus 0000:07: resource 7 [io  0x0d00-0xffff window]
[    0.515958] pci_bus 0000:07: resource 8 [mem 0x000a0000-0x000bffff
window]
[    0.515959] pci_bus 0000:07: resource 9 [mem 0x000c0000-0x000dffff
window]
[    0.515960] pci_bus 0000:07: resource 10 [mem 0xfed08000-0xfed08fff
window]
[    0.515961] pci_bus 0000:07: resource 11 [mem 0xfed0e000-0xfed0ffff
window]
[    0.515962] pci_bus 0000:07: resource 12 [mem 0x80000000-0xfbffffff
window]
[    0.516020] NET: Registered protocol family 2
[    0.516228] TCP established hash table entries: 262144 (order: 9,
2097152 bytes)
[    0.516493] TCP bind hash table entries: 65536 (order: 8, 1048576
bytes)
[    0.516586] TCP: Hash tables configured (established 262144 bind
65536)
[    0.516624] UDP hash table entries: 16384 (order: 7, 524288 bytes)
[    0.516697] UDP-Lite hash table entries: 16384 (order: 7, 524288
bytes)
[    0.516802] NET: Registered protocol family 1
[    0.556713] pci 0000:07:04.0: Video device with shadowed ROM
[    0.556741] PCI: CLS 64 bytes, default 64
[    0.556774] Unpacking initramfs...
[    0.689327] Freeing initrd memory: 11716K (ffff88003690e000 -
ffff88003747f000)
[    0.689486] IOMMU: dmar0 using Queued invalidation
[    0.689489] IOMMU: Setting RMRR:
[    0.689498] IOMMU: Setting identity map for device 0000:00:1a.0
[0x7dedf000 - 0x7deebfff]
[    0.689508] IOMMU: Setting identity map for device 0000:00:1d.0
[0x7dedf000 - 0x7deebfff]
[    0.689511] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.689516] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0
- 0xffffff]
[    0.689522] PCI-DMA: Intel(R) Virtualization Technology for Directed
I/O
[    0.691686] RAPL PMU detected, API unit is 2^-32 Joules, 3 fixed
counters 163840 ms ovfl timer
[    0.691687] hw unit of domain pp0-core 2^-16 Joules
[    0.691688] hw unit of domain package 2^-16 Joules
[    0.691689] hw unit of domain dram 2^-16 Joules
[    0.691898] alg: No test for crc32 (crc32-pclmul)
[    0.692130] futex hash table entries: 2048 (order: 5, 131072 bytes)
[    0.692147] Initialise system trusted keyring
[    0.692167] audit: initializing netlink subsys (disabled)
[    0.692179] audit: type=2000 audit(1443602514.524:1): initialized
[    0.693111] alg: No test for stdrng (krng)
[    0.693117] Key type asymmetric registered
[    0.693119] Asymmetric key parser 'x509' registered
[    0.693136] Block layer SCSI generic (bsg) driver version 0.4 loaded
(major 252)
[    0.693138] io scheduler noop registered
[    0.693140] io scheduler deadline registered
[    0.693151] io scheduler cfq registered (default)
[    0.693654] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.693665] pciehp: PCI Express Hot Plug Controller Driver version:
0.4
[    0.693686] intel_idle: MWAIT substates: 0x1120
[    0.693687] intel_idle: v0.4 model 0x3E
[    0.693687] intel_idle: lapic_timer_reliable_states 0xffffffff
[    0.699746] ERST: Error Record Serialization Table (ERST) support is
initialized.
[    0.699750] pstore: Registered erst as persistent store backend
[    0.699818] GHES: APEI firmware first mode is enabled by WHEA _OSC.
[    0.699858] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.720286] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200)
is a 16550A
[    0.740714] 00:07: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200)
is a 16550A
[    0.761168] 00:09: ttyS2 at I/O 0x3e8 (irq = 10, base_baud = 115200)
is a 16550A
[    0.761514] Linux agpgart interface v0.103
[    0.762002] mei_me 0000:00:16.0: Device doesn't have valid ME
Interface
[    0.762041] i8042: PNP: PS/2 Controller [PNP0f03:PS2M] at 0x60,0x64
irq 12
[    0.762042] i8042: PNP: PS/2 controller doesn't have KBD irq; using
default 1
[    0.764956] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.764961] serio: i8042 AUX port at 0x60,0x64 irq 12
[    0.765070] input: PC Speaker as
/devices/platform/pcspkr/input/input1
[    0.765088] rtc_cmos 00:02: RTC can wake from S4
[    0.765202] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
[    0.765226] rtc_cmos 00:02: alarms up to one month, y3k, 114 bytes
nvram, hpet irqs
[    0.765245] hidraw: raw HID events driver (C) Jiri Kosina
[    0.765261] usbcore: registered new interface driver usbhid
[    0.765262] usbhid: USB HID core driver
[    0.765386] NET: Registered protocol family 10
[    0.765544] mip6: Mobile IPv6
[    0.765546] NET: Registered protocol family 17
[    0.765549] 8021q: 802.1Q VLAN Support v1.8
[    0.765556] Key type dns_resolver registered
[    0.765907] Loading compiled-in X.509 certificates
[    0.766461] Loaded X.509 cert 'Build time autogenerated kernel key:
c8b9e4a03e5c0ac6af3f94589599aa244d72ed22'
[    0.766467] registered taskstats version 1
[    0.766861] rtc_cmos 00:02: setting system clock to 2015-09-30
08:41:55 UTC (1443602515)
[    0.767498] PM: Hibernation image not present or could not be
loaded.
[    0.767631] Freeing unused kernel memory: 800K (ffffffff8168c000 -
ffffffff81754000)
[    0.767632] Write protecting the kernel read-only data: 6144k
[    0.767779] Freeing unused kernel memory: 348K (ffff8800013a9000 -
ffff880001400000)
[    0.767883] Freeing unused kernel memory: 508K (ffff880001581000 -
ffff880001600000)
[    0.771936] systemd-udevd[96]: starting version 215
[    0.772109] random: systemd-udevd urandom read with 1 bits of
entropy available
[    0.788089] pps_core: LinuxPPS API ver. 1 registered
[    0.788092] pps_core: Software ver. 5.3.6 - Copyright 2005-2007
Rodolfo Giometti <giometti@linux.it>
[    0.788186] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI)
Driver
[    0.789343] SCSI subsystem initialized
[    0.789381] ehci-pci: EHCI PCI platform driver
[    0.789502] PTP clock support registered
[    0.789511] ehci-pci 0000:00:1a.0: EHCI Host Controller
[    0.789518] ehci-pci 0000:00:1a.0: new USB bus registered, assigned
bus number 1
[    0.789532] ehci-pci 0000:00:1a.0: debug port 2
[    0.790686] dca service started, version 1.12.1
[    0.791654] libata version 3.00 loaded.
[    0.793447] ehci-pci 0000:00:1a.0: cache line size of 64 is not
supported
[    0.793465] ehci-pci 0000:00:1a.0: irq 16, io mem 0xfb823000
[    0.794585] igb: Intel(R) Gigabit Ethernet Network Driver - version
5.2.15-k
[    0.794587] igb: Copyright (c) 2007-2014 Intel Corporation.
[    0.794886] mpt2sas version 20.100.00.00 loaded
[    0.795067] mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total
mem (33053412 kB)
[    0.797815] isci: Intel(R) C600 SAS Controller Driver - version
1.2.0
[    0.797829] isci 0000:06:00.0: driver configured for rev: 6 silicon
[    0.805636] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00
[    0.805828] usb usb1: New USB device found, idVendor=1d6b,
idProduct=0002
[    0.805830] usb usb1: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
[    0.805831] usb usb1: Product: EHCI Host Controller
[    0.805832] usb usb1: Manufacturer: Linux 4.1-dae-intel ehci_hcd
[    0.805833] usb usb1: SerialNumber: 0000:00:1a.0
[    0.807107] hub 1-0:1.0: USB hub found
[    0.807112] hub 1-0:1.0: 2 ports detected
[    0.809568] ehci-pci 0000:00:1d.0: EHCI Host Controller
[    0.809573] ehci-pci 0000:00:1d.0: new USB bus registered, assigned
bus number 2
[    0.809583] ehci-pci 0000:00:1d.0: debug port 2
[    0.811007] isci 0000:06:00.0: OEM parameter table found in OROM
[    0.811010] isci 0000:06:00.0: OEM SAS parameters (version: 1.0)
loaded (platform)
[    0.811136] isci 0000:06:00.0: SCU controller 0: phy 3-0 cables:
{short, short, short, short}
[    0.813432] scsi host1: isci
[    0.813485] ehci-pci 0000:00:1d.0: cache line size of 64 is not
supported
[    0.813499] ehci-pci 0000:00:1d.0: irq 23, io mem 0xfb822000
[    0.824900] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00
[    0.824920] usb usb2: New USB device found, idVendor=1d6b,
idProduct=0002
[    0.824921] usb usb2: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
[    0.824922] usb usb2: Product: EHCI Host Controller
[    0.824923] usb usb2: Manufacturer: Linux 4.1-dae-intel ehci_hcd
[    0.824924] usb usb2: SerialNumber: 0000:00:1d.0
[    0.825010] hub 2-0:1.0: USB hub found
[    0.825014] hub 2-0:1.0: 2 ports detected
[    0.825109] ahci 0000:00:1f.2: version 3.0
[    0.840973] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6
Gbps 0x3f impl SATA mode
[    0.840976] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio
slum part ems apst 
[    0.879306] igb 0000:03:00.0: added PHC on eth0
[    0.879308] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network
Connection
[    0.879309] igb 0000:03:00.0: eth0: (PCIe:5.0Gb/s:Width x4)
0c:c4:7a:44:c4:ae
[    0.879381] igb 0000:03:00.0: eth0: PBA No: FFFFFF-0FF
[    0.879382] igb 0000:03:00.0: Using MSI-X interrupts. 8 rx queue(s),
8 tx queue(s)
[    0.881398] scsi host2: ahci
[    0.881484] scsi host3: ahci
[    0.881566] scsi host4: ahci
[    0.881649] scsi host5: ahci
[    0.881732] scsi host6: ahci
[    0.881814] scsi host7: ahci
[    0.881850] ata1: SATA max UDMA/133 abar m2048@0xfb821000 port
0xfb821100 irq 40
[    0.881851] ata2: SATA max UDMA/133 abar m2048@0xfb821000 port
0xfb821180 irq 40
[    0.881853] ata3: SATA max UDMA/133 abar m2048@0xfb821000 port
0xfb821200 irq 40
[    0.881854] ata4: SATA max UDMA/133 abar m2048@0xfb821000 port
0xfb821280 irq 40
[    0.881855] ata5: SATA max UDMA/133 abar m2048@0xfb821000 port
0xfb821300 irq 40
[    0.881857] ata6: SATA max UDMA/133 abar m2048@0xfb821000 port
0xfb821380 irq 40
[    0.965380] igb 0000:03:00.1: added PHC on eth1
[    0.965381] igb 0000:03:00.1: Intel(R) Gigabit Ethernet Network
Connection
[    0.965383] igb 0000:03:00.1: eth1: (PCIe:5.0Gb/s:Width x4)
0c:c4:7a:44:c4:af
[    0.965455] igb 0000:03:00.1: eth1: PBA No: FFFFFF-0FF
[    0.965456] igb 0000:03:00.1: Using MSI-X interrupts. 8 rx queue(s),
8 tx queue(s)
[    1.065148] mpt2sas0: MSI-X vectors supported: 1, no of cores: 8,
max_msix_vectors: 8
[    1.065178] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 52
[    1.065179] mpt2sas0: iomem(0x00000000fb600000),
mapped(0xffffc90003650000), size(16384)
[    1.065180] mpt2sas0: ioport(0x000000000000e000), size(256)
[    1.117194] usb 1-1: new high-speed USB device number 2 using ehci
-pci
[    1.145215] usb 2-1: new high-speed USB device number 2 using ehci
-pci
[    1.201273] ata1: SATA link down (SStatus 0 SControl 300)
[    1.201292] ata6: SATA link down (SStatus 0 SControl 300)
[    1.201314] ata5: SATA link down (SStatus 0 SControl 300)
[    1.201339] ata2: SATA link down (SStatus 0 SControl 300)
[    1.201363] ata3: SATA link down (SStatus 0 SControl 300)
[    1.201387] ata4: SATA link down (SStatus 0 SControl 300)
[    1.249654] usb 1-1: New USB device found, idVendor=8087,
idProduct=0024
[    1.249657] usb 1-1: New USB device strings: Mfr=0, Product=0,
SerialNumber=0
[    1.249803] hub 1-1:1.0: USB hub found
[    1.249904] hub 1-1:1.0: 6 ports detected
[    1.277757] usb 2-1: New USB device found, idVendor=8087,
idProduct=0024
[    1.277759] usb 2-1: New USB device strings: Mfr=0, Product=0,
SerialNumber=0
[    1.278000] hub 2-1:1.0: USB hub found
[    1.278056] hub 2-1:1.0: 8 ports detected
[    1.353357] mpt2sas0: sending diag reset !!
[    1.521550] usb 1-1.6: new full-speed USB device number 3 using ehci
-pci
[    1.615351] usb 1-1.6: New USB device found, idVendor=0557,
idProduct=2221
[    1.615353] usb 1-1.6: New USB device strings: Mfr=1, Product=2,
SerialNumber=0
[    1.615354] usb 1-1.6: Product: Hermon USB hidmouse Device
[    1.615355] usb 1-1.6: Manufacturer: Winbond Electronics Corp
[    1.616293] input: Winbond Electronics Corp Hermon USB hidmouse
Device as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1
-1.6:1.0/0003:0557:2221.0001/input/input3
[    1.616351] hid-generic 0003:0557:2221.0001: input,hidraw0: USB HID
v1.00 Mouse [Winbond Electronics Corp Hermon USB hidmouse Device] on
usb-0000:00:1a.0-1.6/input0
[    1.617031] input: Winbond Electronics Corp Hermon USB hidmouse
Device as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1
-1.6:1.1/0003:0557:2221.0002/input/input4
[    1.669734] hid-generic 0003:0557:2221.0002: input,hidraw1: USB HID
v1.00 Keyboard [Winbond Electronics Corp Hermon USB hidmouse Device] on
usb-0000:00:1a.0-1.6/input1
[    1.689702] tsc: Refined TSC clocksource calibration: 3699.995 MHz
[    1.689704] clocksource tsc: mask: 0xffffffffffffffff max_cycles:
0x6aaaa371c79, max_idle_ns: 881590671907 ns
[    2.506421] mpt2sas0: diag reset: SUCCESS
[    2.652245] mpt2sas0: Allocated physical memory: size(3392 kB)
[    2.652246] mpt2sas0: Current Controller Queue Depth(1483), Max
Controller Queue Depth(1720)
[    2.652247] mpt2sas0: Scatter Gather Elements per IO(128)
[    2.690690] Switched to clocksource tsc
[    2.883151] mpt2sas0: LSISAS2004: FWVersion(19.00.00.00),
ChipRevision(0x03), BiosVersion(07.37.00.00)
[    2.883152] mpt2sas0: Protocol=(Initiator),
Capabilities=(Raid,TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set
Full,NCQ)
[    2.883200] scsi host0: Fusion MPT SAS Host
[    2.883324] mpt2sas0: sending port enable !!
[    4.422961] mpt2sas0: host_add: handle(0x0001),
sas_addr(0x500062b0002dc174), phys(8)
[   10.313273] mpt2sas0: port enable: SUCCESS
[   10.313941] scsi 0:0:0:0: Direct-Access     ATA      INTEL
SSDSC2BB30 0380 PQ: 0 ANSI: 6
[   10.313945] scsi 0:0:0:0: SATA: handle(0x000b),
sas_addr(0x4433221103000000), phy(3), device_name(0x55cd2e404b7cd3b0)
[   10.313947] scsi 0:0:0:0: SATA:
enclosure_logical_id(0x500062b0002dc174), slot(0)
[   10.314022] scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n),
smart(y), fua(y), sw_preserve(y)
[   10.314809] scsi 0:0:1:0: Direct-Access     ATA      INTEL
SSDSC2BB30 0380 PQ: 0 ANSI: 6
[   10.314811] scsi 0:0:1:0: SATA: handle(0x0009),
sas_addr(0x4433221101000000), phy(1), device_name(0x55cd2e404b7cd395)
[   10.314813] scsi 0:0:1:0: SATA:
enclosure_logical_id(0x500062b0002dc174), slot(2)
[   10.314881] scsi 0:0:1:0: atapi(n), ncq(y), asyn_notify(n),
smart(y), fua(y), sw_preserve(y)
[   10.315650] scsi 0:0:2:0: Direct-Access     ATA      INTEL
SSDSC2BB30 0380 PQ: 0 ANSI: 6
[   10.315652] scsi 0:0:2:0: SATA: handle(0x000a),
sas_addr(0x4433221102000000), phy(2), device_name(0x55cd2e404b7ab3f3)
[   10.315654] scsi 0:0:2:0: SATA:
enclosure_logical_id(0x500062b0002dc174), slot(1)
[   10.315722] scsi 0:0:2:0: atapi(n), ncq(y), asyn_notify(n),
smart(y), fua(y), sw_preserve(y)
[   10.318328] sd 0:0:0:0: [sda] 586072368 512-byte logical blocks:
(300 GB/279 GiB)
[   10.318330] sd 0:0:0:0: [sda] 4096-byte physical blocks
[   10.318350] sd 0:0:1:0: [sdb] 586072368 512-byte logical blocks:
(300 GB/279 GiB)
[   10.318352] sd 0:0:1:0: [sdb] 4096-byte physical blocks
[   10.318388] sd 0:0:2:0: [sdc] 586072368 512-byte logical blocks:
(300 GB/279 GiB)
[   10.318389] sd 0:0:2:0: [sdc] 4096-byte physical blocks
[   10.319520] sd 0:0:0:0: [sda] Write Protect is off
[   10.319522] sd 0:0:0:0: [sda] Mode Sense: 7f 00 10 08
[   10.319539] sd 0:0:1:0: [sdb] Write Protect is off
[   10.319541] sd 0:0:1:0: [sdb] Mode Sense: 7f 00 10 08
[   10.319579] sd 0:0:2:0: [sdc] Write Protect is off
[   10.319581] sd 0:0:2:0: [sdc] Mode Sense: 7f 00 10 08
[   10.319774] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[   10.319844] sd 0:0:1:0: [sdb] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[   10.319868] sd 0:0:2:0: [sdc] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[   10.320078] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   10.320118] sd 0:0:1:0: Attached scsi generic sg1 type 0
[   10.320148] sd 0:0:2:0: Attached scsi generic sg2 type 0
[   10.323514]  sdb: sdb1 sdb2 sdb3
[   10.323575]  sdc: sdc1 sdc2 sdc3
[   10.323757]  sda: sda1 sda2 sda3
[   10.325863] sd 0:0:0:0: [sda] Attached SCSI disk
[   10.326116] sd 0:0:1:0: [sdb] Attached SCSI disk
[   10.326155] sd 0:0:2:0: [sdc] Attached SCSI disk
[   10.336510] md: bind<sdb2>
[   10.337208] md: bind<sdb3>
[   10.338029] md: bind<sda2>
[   10.338949] md: bind<sdc3>
[   10.339917] md: bind<sdc2>
[   10.341103] md: bind<sda3>
[   10.341758] md: raid1 personality registered for level 1
[   10.341883] md/raid1:md2: not clean -- starting background
reconstruction
[   10.341885] md/raid1:md2: active with 3 out of 3 mirrors
[   10.341899] md2: detected capacity change from 0 to 209518592
[   10.409293] raid6: sse2x1   gen()  9678 MB/s
[   10.477352] raid6: sse2x1   xor()  7395 MB/s
[   10.545412] raid6: sse2x2   gen() 12444 MB/s
[   10.613472] raid6: sse2x2   xor()  8566 MB/s
[   10.681531] raid6: sse2x4   gen() 14425 MB/s
[   10.749591] raid6: sse2x4   xor() 10697 MB/s
[   10.749592] raid6: using algorithm sse2x4 gen() 14425 MB/s
[   10.749592] raid6: .... xor() 10697 MB/s, rmw enabled
[   10.749593] raid6: using ssse3x2 recovery algorithm
[   10.750330] xor: automatically using best checksumming function:
[   10.789623]    avx       : 27913.000 MB/sec
[   10.790340] async_tx: api initialized (async)
[   10.794153] md: raid6 personality registered for level 6
[   10.794154] md: raid5 personality registered for level 5
[   10.794155] md: raid4 personality registered for level 4
[   10.794285] md/raid:md3: device sda3 operational as raid disk 0
[   10.794286] md/raid:md3: device sdc3 operational as raid disk 2
[   10.794287] md/raid:md3: device sdb3 operational as raid disk 1
[   10.794454] md/raid:md3: allocated 3308kB
[   10.794468] md/raid:md3: raid level 5 active with 3 out of 3
devices, algorithm 2
[   10.794468] RAID conf printout:
[   10.794469]  --- level:5 rd:3 wd:3
[   10.794470]  disk 0, o:1, dev:sda3
[   10.794470]  disk 1, o:1, dev:sdb3
[   10.794471]  disk 2, o:1, dev:sdc3
[   10.794477] md/raid456: discard support disabled due to uncertainty.
[   10.794478] Set raid456.devices_handle_discard_safely=Y to override.
[   10.794494] md3: detected capacity change from 0 to 599446781952
[   25.828560] device-mapper: uevent: version 1.0.3
[   25.828607] device-mapper: ioctl: 4.31.0-ioctl (2015-3-12)
initialised: dm-devel@redhat.com
[   25.857869] Btrfs loaded
[   25.941519] EXT4-fs (dm-0): mounted filesystem with ordered data
mode. Opts: (null)
[   26.027346] systemd[1]: Cannot add dependency job for unit display
-manager.service, ignoring: Unit display-manager.service failed to
load: No such file or directory.
[   26.037103] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   26.045888] systemd-udevd[268]: starting version 215
[   26.070172] input: Power Button as
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input5
[   26.070176] ACPI: Power Button [PWRB]
[   26.070221] input: Power Button as
/devices/LNXSYSTM:00/LNXPWRBN:00/input/input6
[   26.070222] ACPI: Power Button [PWRF]
[   26.071829] wmi: Mapper loaded
[   26.072486] shpchp: Standard Hot Plug PCI Controller Driver version:
0.4
[   26.108361] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro
[   26.144799] ipmi message handler version 39.2
[   26.146778] IPMI System Interface driver.
[   26.146805] ipmi_si: probing via ACPI
[   26.146823] ipmi_si 00:0a: [io  0x0ca2] regsize 1 spacing 1 irq 0
[   26.146824] ipmi_si: Adding ACPI-specified kcs state machine
[   26.146835] ipmi_si: probing via SMBIOS
[   26.146836] ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
[   26.146837] ipmi_si: Adding SMBIOS-specified kcs state machine
duplicate interface
[   26.146839] ipmi_si: probing via SPMI
[   26.146841] ipmi_si: SPMI: io 0xca2 regsize 1 spacing 1 irq 0
[   26.146842] ipmi_si: Adding SPMI-specified kcs state machine
duplicate interface
[   26.146844] ipmi_si: Trying ACPI-specified kcs state machine at i/o
address 0xca2, slave address 0x0, irq 0
[   26.148680] ioatdma: Intel(R) QuickData Technology Driver 4.00
[   26.149934] igb 0000:03:00.0: DCA enabled
[   26.149958] igb 0000:03:00.1: DCA enabled
[   26.154693] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[   26.158863] EDAC MC: Ver: 3.0.0
[   26.175550] microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x415
[   26.175913] microcode: CPU1 sig=0x306e4, pf=0x1, revision=0x415
[   26.175929] microcode: CPU2 sig=0x306e4, pf=0x1, revision=0x415
[   26.175941] microcode: CPU3 sig=0x306e4, pf=0x1, revision=0x415
[   26.175952] microcode: CPU4 sig=0x306e4, pf=0x1, revision=0x415
[   26.175966] microcode: CPU5 sig=0x306e4, pf=0x1, revision=0x415
[   26.175977] microcode: CPU6 sig=0x306e4, pf=0x1, revision=0x415
[   26.175987] microcode: CPU7 sig=0x306e4, pf=0x1, revision=0x415
[   26.176034] microcode: Microcode Update Driver: v2.00 <
tigran@aivazian.fsnet.co.uk>, Peter Oruba
[   26.176779] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[   26.176787] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[   26.176790] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[   26.176794] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[   26.176796] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[   26.176799] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[   26.176801] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[   26.176804] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[   26.176806] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[   26.176810] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[   26.176811] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[   26.176815] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[   26.176816] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[   26.176820] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[   26.176821] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[   26.176825] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[   26.176826] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[   26.176830] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[   26.176831] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[   26.176836] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[   26.176837] EDAC sbridge: Seeking for: PCI ID 8086:0e60
[   26.176840] EDAC sbridge: Seeking for: PCI ID 8086:0e6a
[   26.176843] EDAC sbridge: Seeking for: PCI ID 8086:0e6b
[   26.176846] EDAC sbridge: Seeking for: PCI ID 8086:0eb8
[   26.176849] EDAC sbridge: Seeking for: PCI ID 8086:0ebc
[   26.177075] EDAC MC0: Giving out device to module sbridge_edac.c
controller Ivy Bridge Socket#0: DEV 0000:ff:0e.0 (POLLED)
[   26.177077] EDAC sbridge:  Ver: 1.1.0 
[   26.178626] iTCO_vendor_support: vendor-support=0
[   26.180109] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   26.180133] iTCO_wdt: unable to reset NO_REBOOT flag, device
disabled by hardware/BIOS
[   26.189491] AVX version of gcm_enc/dec engaged.
[   26.189493] AES CTR mode by8 optimization enabled
[   26.190578] alg: No test for __gcm-aes-aesni (__driver-gcm-aes
-aesni)
[   26.192646] md: resync of RAID array md2
[   26.192648] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[   26.192649] md: using maximum available idle IO bandwidth (but not
more than 200000 KB/sec) for resync.
[   26.192651] md: using 128k window, over a total of 204608k.
[   26.276645] EXT4-fs (md2): mounted filesystem with ordered data
mode. Opts: (null)
[   26.279275] ipmi_si: The BMC does not support clearing the recv irq
bit, compensating, but the BMC needs to be fixed.
[   26.315577] ipmi_si 00:0a: Found new BMC (man_id: 0x002a7c, prod_id:
0x062b, dev_id: 0x20)
[   26.315584] ipmi_si 00:0a: IPMI kcs interface initialized
[   26.535299] EXT4-fs (dm-1): mounted filesystem with ordered data
mode. Opts: (null)
[   26.558105] EXT4-fs (dm-2): mounted filesystem with ordered data
mode. Opts: (null)
[   26.575726] EXT4-fs (dm-3): mounted filesystem with ordered data
mode. Opts: (null)
[   26.579088] EXT4-fs (dm-7): mounted filesystem with ordered data
mode. Opts: (null)
[   26.579164] EXT4-fs (dm-5): mounted filesystem with ordered data
mode. Opts: (null)
[   26.586388] EXT4-fs (dm-4): mounted filesystem with ordered data
mode. Opts: (null)
[   26.592694] EXT4-fs (dm-6): mounted filesystem with ordered data
mode. Opts: (null)
[   26.744855] systemd-journald[260]: Received request to flush runtime
journal from PID 1
[   26.877690] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   26.877692] 8021q: adding VLAN 0 to HW filter on device eth0
[   26.995846] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   26.995849] 8021q: adding VLAN 0 to HW filter on device eth1
[   27.044814] ip_tables: (C) 2000-2006 Netfilter Core Team
[   27.169990] random: nonblocking pool is initialized
[   27.684004] md: md2: resync done.
[   27.693318] RAID1 conf printout:
[   27.693320]  --- wd:3 rd:3
[   27.693322]  disk 0, wo:0, o:1, dev:sda2
[   27.693323]  disk 1, wo:0, o:1, dev:sdb2
[   27.693323]  disk 2, wo:0, o:1, dev:sdc2
[   29.971764] igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000
Mbps Full Duplex, Flow Control: RX
[   29.971851] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   30.648355] igb 0000:03:00.0 eth0: igb: eth0 NIC Link is Up 1000
Mbps Full Duplex, Flow Control: RX
[   30.648419] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  176.453241] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s!
[swapper/4:0]
[  176.453300] Modules linked in: iptable_filter ip_tables x_tables
x86_pkg_temp_thermal coretemp ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper iTCO_wdt
iTCO_vendor_support cryptd sb_edac microcode serio_raw edac_core evdev
i2c_i801 lpc_ich mfd_core ioatdma ipmi_si ipmi_msghandler shpchp wmi
button nf_conntrack autofs4 btrfs dm_mod raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 md_mod sg
sd_mod isci crc32c_intel libsas mpt2sas igb ahci libahci raid_class
i2c_algo_bit libata scsi_transport_sas dca ptp ehci_pci scsi_mod
ehci_hcd pps_core i2c_core
[  176.453326] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.1-dae-intel
#1
[  176.453327] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi
-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[  176.453328] task: ffff88087b838920 ti: ffff88087b840000 task.ti:
ffff88087b840000
[  176.453329] RIP: 0010:[<ffffffff813a2e95>]  [<ffffffff813a2e95>]
_raw_spin_unlock_irqrestore+0x5/0x6
[  176.453334] RSP: 0018:ffff88087fd03e40  EFLAGS: 00000286
[  176.453335] RAX: 00000000ffffffff RBX: 000000000000001d RCX:
00000000000000ec
[  176.453336] RDX: 00000000000000eb RSI: 0000000000000286 RDI:
ffff88087fd0d600
[  176.453336] RBP: ffff88085dc50800 R08: ffff88087fd0d600 R09:
0000000000000f50
[  176.453337] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff88087fd03db8
[  176.453338] R13: ffffffff813a3e8b R14: ffff88085dc50800 R15:
ffff88085cf70ee0
[  176.453339] FS:  0000000000000000(0000) GS:ffff88087fd00000(0000)
knlGS:0000000000000000
[  176.453340] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  176.453341] CR2: 00007f8a42e21000 CR3: 000000000160b000 CR4:
00000000001406e0
[  176.453341] Stack:
[  176.453342]  ffffffff81076121 ffffffff810760de ffffffff00000010
0000000000000286
[  176.453343]  0000000000000286 ffff88085cf70ee0 ffffffff8107614b
ffff88085cf70e48
[  176.453345]  ffffffff8131c6d7 ffff88085cf70e48 ffff88085c3b0000
0000000000000001
[  176.453346] Call Trace:
[  176.453347]  <IRQ> 
[  176.453350]  [<ffffffff81076121>] ? try_to_del_timer_sync+0x43/0x4d
[  176.453352]  [<ffffffff810760de>] ? del_timer+0x4d/0x4d
[  176.453354]  [<ffffffff8107614b>] ? del_timer_sync+0x20/0x3d
[  176.453356]  [<ffffffff8131c6d7>] ?
inet_csk_reqsk_queue_drop+0x5f/0xa3
[  176.453357]  [<ffffffff8131c880>] ? reqsk_timer_handler+0x165/0x18e
[  176.453358]  [<ffffffff8131c71b>] ?
inet_csk_reqsk_queue_drop+0xa3/0xa3
[  176.453360]  [<ffffffff81075c0a>] ? call_timer_fn.isra.30+0x12/0x67
[  176.453361]  [<ffffffff81075dfe>] ? run_timer_softirq+0x17f/0x1a1
[  176.453365]  [<ffffffff8103d6e9>] ? __do_softirq+0xc5/0x1b4
[  176.453366]  [<ffffffff8103d910>] ? irq_exit+0x34/0x78
[  176.453369]  [<ffffffff81024e7f>] ?
smp_trace_apic_timer_interrupt+0x6f/0x7b
[  176.453371]  [<ffffffff813a3e8b>] ? apic_timer_interrupt+0x6b/0x70
[  176.453372]  <EOI> 
[  176.453375]  [<ffffffff812bc896>] ? cpuidle_enter_state+0xa7/0xff
[  176.453376]  [<ffffffff812bc83b>] ? cpuidle_enter_state+0x4c/0xff
[  176.453379]  [<ffffffff8106082a>] ? cpu_startup_entry+0x1ac/0x258
[  176.453380] Code: 78 88 ea ff 31 c0 eb 96 48 83 c4 48 89 d0 5b 5d 41
5c 41 5d c3 90 e8 14 ec cb ff 85 c0 75 01 c3 b8 01 00 00 00 c3 80 07 01
56 9d <c3> c6 07 00 56 9d c3 fa e9 3a ec cb ff e9 35 ec cb ff 8b 07 85 
[  204.478275] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s!
[swapper/4:0]
[  204.478334] Modules linked in: iptable_filter ip_tables x_tables
x86_pkg_temp_thermal coretemp ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper iTCO_wdt
iTCO_vendor_support cryptd sb_edac microcode serio_raw edac_core evdev
i2c_i801 lpc_ich mfd_core ioatdma ipmi_si ipmi_msghandler shpchp wmi
button nf_conntrack autofs4 btrfs dm_mod raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 md_mod sg
sd_mod isci crc32c_intel libsas mpt2sas igb ahci libahci raid_class
i2c_algo_bit libata scsi_transport_sas dca ptp ehci_pci scsi_mod
ehci_hcd pps_core i2c_core
[  204.478360] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G             L 
 4.1-dae-intel #1
[  204.478361] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi
-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
[  204.478362] task: ffff88087b838920 ti: ffff88087b840000 task.ti:
ffff88087b840000
[  204.478363] RIP: 0010:[<ffffffff813a2e95>]  [<ffffffff813a2e95>]
_raw_spin_unlock_irqrestore+0x5/0x6
[  204.478368] RSP: 0018:ffff88087fd03e40  EFLAGS: 00000286
[  204.478369] RAX: 00000000ffffffff RBX: ffff88085c3b0001 RCX:
000000000000002a
[  204.478370] RDX: 0000000000000029 RSI: 0000000000000286 RDI:
ffff88087fd0d600
[  204.478370] RBP: ffff88085dc50800 R08: ffff88087fd0d600 R09:
0000000000000f50
[  204.478371] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff88087fd03db8
[  204.478372] R13: ffffffff813a3e8b R14: ffff88085dc50800 R15:
ffff88085cf70ee0
[  204.478373] FS:  0000000000000000(0000) GS:ffff88087fd00000(0000)
knlGS:0000000000000000
[  204.478374] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  204.478375] CR2: 00007f8a42e21000 CR3: 000000000160b000 CR4:
00000000001406e0
[  204.478375] Stack:
[  204.478376]  ffffffff81076121 ffffffff810760de ffffffff00000010
0000000000000286
[  204.478377]  0000000000000286 ffff88085cf70ee0 ffffffff8107614b
ffff88085cf70e48
[  204.478379]  ffffffff8131c6d7 ffff88085cf70e48 ffff88085c3b0000
0000000000000001
[  204.478380] Call Trace:
[  204.478381]  <IRQ> 
[  204.478385]  [<ffffffff81076121>] ? try_to_del_timer_sync+0x43/0x4d
[  204.478386]  [<ffffffff810760de>] ? del_timer+0x4d/0x4d
[  204.478388]  [<ffffffff8107614b>] ? del_timer_sync+0x20/0x3d
[  204.478390]  [<ffffffff8131c6d7>] ?
inet_csk_reqsk_queue_drop+0x5f/0xa3
[  204.478391]  [<ffffffff8131c880>] ? reqsk_timer_handler+0x165/0x18e
[  204.478392]  [<ffffffff8131c71b>] ?
inet_csk_reqsk_queue_drop+0xa3/0xa3
[  204.478394]  [<ffffffff81075c0a>] ? call_timer_fn.isra.30+0x12/0x67
[  204.478395]  [<ffffffff81075dfe>] ? run_timer_softirq+0x17f/0x1a1
[  204.478399]  [<ffffffff8103d6e9>] ? __do_softirq+0xc5/0x1b4
[  204.478401]  [<ffffffff8103d910>] ? irq_exit+0x34/0x78
[  204.478403]  [<ffffffff81024e7f>] ?
smp_trace_apic_timer_interrupt+0x6f/0x7b
[  204.478405]  [<ffffffff813a3e8b>] ? apic_timer_interrupt+0x6b/0x70
[  204.478406]  <EOI> 
[  204.478409]  [<ffffffff812bc896>] ? cpuidle_enter_state+0xa7/0xff
[  204.478410]  [<ffffffff812bc83b>] ? cpuidle_enter_state+0x4c/0xff
[  204.478413]  [<ffffffff8106082a>] ? cpu_startup_entry+0x1ac/0x258
[  204.478414] Code: 78 88 ea ff 31 c0 eb 96 48 83 c4 48 89 d0 5b 5d 41
5c 41 5d c3 90 e8 14 ec cb ff 85 c0 75 01 c3 b8 01 00 00 00 c3 80 07 01
56 9d <c3> c6 07 00 56 9d c3 fa e9 3a ec cb ff e9 35 ec cb ff 8b 07 85 
[  209.995023] INFO: rcu_sched detected stalls on CPUs/tasks: { 4}
(detected by 3, t=15003 jiffies, g=2334, c=2333, q=11345)
[  209.995261] Task dump for CPU 4:
[  209.995262] swapper/4       R  running task        0     0      1
0x00000008
[  209.995265]  ffff88087b843eb8 0000000000000018 ffffffff812bc83b
0000000000000000
[  209.995267]  0000000400000000 0000000000000046 ffffe8ffffd0ad08
ffffffff81659de0
[  209.995269]  ffff88087b844000 ffff88087b840000 0000000000000001
ffff88087b844000
[  209.995271] Call Trace:
[  209.995275]  [<ffffffff812bc83b>] ? cpuidle_enter_state+0x4c/0xff
[  209.995278]  [<ffffffff8106082a>] ? cpu_startup_entry+0x1ac/0x258


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-09-30 21:59 Soft lockup issue in Linux 4.1.9 Olivier Bonvalet
@ 2015-09-30 22:37 ` Holger Hoffstätte
  2015-10-01  4:41   ` Andre Tomt
       [not found] ` <560D1223.3070606@googlemail.com>
  1 sibling, 1 reply; 19+ messages in thread
From: Holger Hoffstätte @ 2015-09-30 22:37 UTC (permalink / raw)
  To: linux-kernel

On Wed, 30 Sep 2015 23:59:43 +0200, Olivier Bonvalet wrote:

> for information, I've just upgraded 6 servers from Linux 4.1.8 to Linux
> 4.1.9, and have some random soft lockup. If this can help :

Congratulations! You're not the first one to get hit by this, but
you are probably the first one to get a meaningful stacktrace! \o/

> [  204.478380] Call Trace:
> [  204.478381]  <IRQ> 
> [  204.478385]  [<ffffffff81076121>] ? try_to_del_timer_sync+0x43/0x4d
> [  204.478386]  [<ffffffff810760de>] ? del_timer+0x4d/0x4d
> [  204.478388]  [<ffffffff8107614b>] ? del_timer_sync+0x20/0x3d

Can you try to revert

   [PATCH 4.1 157/159] inet: fix races with reqsk timers

and see how that works for you? I'll do the same on my end. So far the
only thing I ever could gleam was an rcu stall after cpuidle_enter(),
but never anything regarding the timer - though it was definitely
related to NIC activity after idle.

cheers
Holger


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-09-30 22:37 ` Holger Hoffstätte
@ 2015-10-01  4:41   ` Andre Tomt
  2015-10-01 10:51     ` Holger Hoffstätte
  0 siblings, 1 reply; 19+ messages in thread
From: Andre Tomt @ 2015-10-01  4:41 UTC (permalink / raw)
  To: Holger Hoffstätte, linux-kernel

On 01. okt. 2015 00:37, Holger Hoffstätte wrote:
> On Wed, 30 Sep 2015 23:59:43 +0200, Olivier Bonvalet wrote:
>
>> for information, I've just upgraded 6 servers from Linux 4.1.8 to Linux
>> 4.1.9, and have some random soft lockup. If this can help :
>
> Congratulations! You're not the first one to get hit by this, but
> you are probably the first one to get a meaningful stacktrace! \o/
>
>> [  204.478380] Call Trace:
>> [  204.478381]  <IRQ>
>> [  204.478385]  [<ffffffff81076121>] ? try_to_del_timer_sync+0x43/0x4d
>> [  204.478386]  [<ffffffff810760de>] ? del_timer+0x4d/0x4d
>> [  204.478388]  [<ffffffff8107614b>] ? del_timer_sync+0x20/0x3d
>
> Can you try to revert
>
>     [PATCH 4.1 157/159] inet: fix races with reqsk timers
>
> and see how that works for you? I'll do the same on my end. So far the
> only thing I ever could gleam was an rcu stall after cpuidle_enter(),
> but never anything regarding the timer - though it was definitely
> related to NIC activity after idle.

I'm running with this patch reverted now as well. 2 hours no issues so 
far, but I can't conclude anything yet as I've seen it take up to 6+ 
hours to explode here. As a result the bisect was going veeery slowly.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-01  4:41   ` Andre Tomt
@ 2015-10-01 10:51     ` Holger Hoffstätte
  0 siblings, 0 replies; 19+ messages in thread
From: Holger Hoffstätte @ 2015-10-01 10:51 UTC (permalink / raw)
  To: linux-kernel; +Cc: stable


On Thu, 01 Oct 2015 06:41:46 +0200, Andre Tomt wrote:

> On 01. okt. 2015 00:37, Holger Hoffstätte wrote:
>> On Wed, 30 Sep 2015 23:59:43 +0200, Olivier Bonvalet wrote:
>>
>>> for information, I've just upgraded 6 servers from Linux 4.1.8 to Linux
>>> 4.1.9, and have some random soft lockup. If this can help :
>>
>> Congratulations! You're not the first one to get hit by this, but
>> you are probably the first one to get a meaningful stacktrace! \o/
>>
>>> [  204.478380] Call Trace:
>>> [  204.478381]  <IRQ>
>>> [  204.478385]  [<ffffffff81076121>] ? try_to_del_timer_sync+0x43/0x4d
>>> [  204.478386]  [<ffffffff810760de>] ? del_timer+0x4d/0x4d
>>> [  204.478388]  [<ffffffff8107614b>] ? del_timer_sync+0x20/0x3d
>>
>> Can you try to revert
>>
>>     [PATCH 4.1 157/159] inet: fix races with reqsk timers
>>
>> and see how that works for you? I'll do the same on my end. So far the
>> only thing I ever could gleam was an rcu stall after cpuidle_enter(),
>> but never anything regarding the timer - though it was definitely
>> related to NIC activity after idle.
> 
> I'm running with this patch reverted now as well. 2 hours no issues so 
> far, but I can't conclude anything yet as I've seen it take up to 6+ 
> hours to explode here. As a result the bisect was going veeery slowly.

Now 12+ hours going without problems, never got this far with the patch
included, as it would usually freeze during idle periods.

As far as I'm concerned this is the culprit and should be reverted in
4.1.x, unless Eric can suggest how to fix this. (cc'ed).

cheers
Holger


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
       [not found]   ` <CANn89i+B5T4Rhs8HnrC0+f+GhLvBFfpr4BVDvhkVOveSfy9B8Q@mail.gmail.com>
@ 2015-10-01 11:43     ` Holger Hoffstätte
  2015-10-01 11:52       ` Eric Dumazet
  0 siblings, 1 reply; 19+ messages in thread
From: Holger Hoffstätte @ 2015-10-01 11:43 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: David S. Miller, Eric W. Biederman, Stephen Hemminger,
	Greg Kroah-Hartman, linux-kernel, stable, netdev

On 10/01/15 13:29, Eric Dumazet wrote:
> On Thu, Oct 1, 2015 at 3:59 AM, Holger Hoffstätte
> <holger.hoffstaette@googlemail.com> wrote:
>>
>> On Thu, 01 Oct 2015 06:41:46 +0200, Andre Tomt wrote:
>>
>>> On 01. okt. 2015 00:37, Holger Hoffstätte wrote:
>>>> On Wed, 30 Sep 2015 23:59:43 +0200, Olivier Bonvalet wrote:
>>>>
>>>>> for information, I've just upgraded 6 servers from Linux 4.1.8 to Linux
>>>>> 4.1.9, and have some random soft lockup. If this can help :
>>>>
>>>> Congratulations! You're not the first one to get hit by this, but
>>>> you are probably the first one to get a meaningful stacktrace! \o/
>>>>
>>>>> [  204.478380] Call Trace:
>>>>> [  204.478381]  <IRQ>
>>>>> [  204.478385]  [<ffffffff81076121>] ? try_to_del_timer_sync+0x43/0x4d
>>>>> [  204.478386]  [<ffffffff810760de>] ? del_timer+0x4d/0x4d
>>>>> [  204.478388]  [<ffffffff8107614b>] ? del_timer_sync+0x20/0x3d
>>>>
>>>> Can you try to revert
>>>>
>>>>     [PATCH 4.1 157/159] inet: fix races with reqsk timers
>>>>
>>>> and see how that works for you? I'll do the same on my end. So far the
>>>> only thing I ever could gleam was an rcu stall after cpuidle_enter(),
>>>> but never anything regarding the timer - though it was definitely
>>>> related to NIC activity after idle.
>>>
>>> I'm running with this patch reverted now as well. 2 hours no issues so
>>> far, but I can't conclude anything yet as I've seen it take up to 6+
>>> hours to explode here. As a result the bisect was going veeery slowly.
>>
>> Now 12+ hours going without problems, never got this far with the patch
>> included, as it would usually freeze during idle periods.
>>
>> As far as I'm concerned this is the culprit and should be reverted in
>> 4.1.x, unless Eric can suggest how to fix this. (cc'ed).
>>
> 
> Looks an old and known problem...
> 
> Following commit should be sent/added for 4.1 stable tree :
> 
> commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
> Author: Eric Dumazet <edumazet@google.com>
> Date:   Thu Aug 13 15:44:51 2015 -0700
> 
>     inet: fix potential deadlock in reqsk_queue_unlink()
> 
>     When replacing del_timer() with del_timer_sync(), I introduced
>     a deadlock condition :
> 
>     reqsk_queue_unlink() is called from inet_csk_reqsk_queue_drop()
> 
>     inet_csk_reqsk_queue_drop() can be called from many contexts,
>     one being the timer handler itself (reqsk_timer_handler()).
> 
>     In this case, del_timer_sync() loops forever.
> 
>     Simple fix is to test if timer is pending.
> 
>     Fixes: 2235f2ac75fd ("inet: fix races with reqsk timers")
>     Signed-off-by: Eric Dumazet <edumazet@google.com>
>     Signed-off-by: David S. Miller <davem@davemloft.net>

Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
we speak. Let's hope that this fixes the lockups.

Thanks for the quick reply!

Holger


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-01 11:43     ` Holger Hoffstätte
@ 2015-10-01 11:52       ` Eric Dumazet
  2015-10-02  6:52         ` Andre Tomt
                           ` (2 more replies)
  0 siblings, 3 replies; 19+ messages in thread
From: Eric Dumazet @ 2015-10-01 11:52 UTC (permalink / raw)
  To: Holger Hoffstätte
  Cc: David S. Miller, Eric W. Biederman, Stephen Hemminger,
	Greg Kroah-Hartman, LKML, stable, netdev

On Thu, Oct 1, 2015 at 4:43 AM, Holger Hoffstätte
<holger.hoffstaette@googlemail.com> wrote:
> On 10/01/15 13:29, Eric Dumazet wrote:

>> commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
>> Author: Eric Dumazet <edumazet@google.com>
>> Date:   Thu Aug 13 15:44:51 2015 -0700
>>
>>     inet: fix potential deadlock in reqsk_queue_unlink()
>>
>>     When replacing del_timer() with del_timer_sync(), I introduced
>>     a deadlock condition :
>>
>>     reqsk_queue_unlink() is called from inet_csk_reqsk_queue_drop()
>>
>>     inet_csk_reqsk_queue_drop() can be called from many contexts,
>>     one being the timer handler itself (reqsk_timer_handler()).
>>
>>     In this case, del_timer_sync() loops forever.
>>
>>     Simple fix is to test if timer is pending.
>>
>>     Fixes: 2235f2ac75fd ("inet: fix races with reqsk timers")
>>     Signed-off-by: Eric Dumazet <edumazet@google.com>
>>     Signed-off-by: David S. Miller <davem@davemloft.net>
>
> Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
> we speak. Let's hope that this fixes the lockups.
>

It definitely should help !

David, since patch is not yet seen on
http://patchwork.ozlabs.org/bundle/davem/stable/?state=*
could you please add it to your queue ?

Thanks.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-01 11:52       ` Eric Dumazet
@ 2015-10-02  6:52         ` Andre Tomt
  2015-10-02  7:17           ` Holger Hoffstätte
  2015-10-02 20:04         ` Thomas Gleixner
  2015-10-08 16:56         ` Christoph Biedl
  2 siblings, 1 reply; 19+ messages in thread
From: Andre Tomt @ 2015-10-02  6:52 UTC (permalink / raw)
  To: Eric Dumazet, Holger Hoffstätte
  Cc: David S. Miller, Eric W. Biederman, Stephen Hemminger,
	Greg Kroah-Hartman, LKML, stable, netdev

On 01. okt. 2015 13:52, Eric Dumazet wrote:
> On Thu, Oct 1, 2015 at 4:43 AM, Holger Hoffstätte
> <holger.hoffstaette@googlemail.com> wrote:
>> On 10/01/15 13:29, Eric Dumazet wrote:
>
>>> commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
>>> Author: Eric Dumazet <edumazet@google.com>
>>> Date:   Thu Aug 13 15:44:51 2015 -0700
>>>
>>>      inet: fix potential deadlock in reqsk_queue_unlink()
<snip>
>> Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
>> we speak. Let's hope that this fixes the lockups.
>>
>
> It definitely should help !
>
> David, since patch is not yet seen on
> http://patchwork.ozlabs.org/bundle/davem/stable/?state=*
> could you please add it to your queue ?

Seems to fix it for me as well. 3 systems have been running varying 
types of production-like loads with it for 14+ hours without hanging.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-02  6:52         ` Andre Tomt
@ 2015-10-02  7:17           ` Holger Hoffstätte
  2015-10-02 19:25               ` Wolfgang Walter
  2015-10-03 19:14             ` Thomas D.
  0 siblings, 2 replies; 19+ messages in thread
From: Holger Hoffstätte @ 2015-10-02  7:17 UTC (permalink / raw)
  To: Andre Tomt, Eric Dumazet
  Cc: David S. Miller, Eric W. Biederman, Stephen Hemminger,
	Greg Kroah-Hartman, LKML, stable, netdev

On 10/02/15 08:52, Andre Tomt wrote:
> On 01. okt. 2015 13:52, Eric Dumazet wrote:
>> On Thu, Oct 1, 2015 at 4:43 AM, Holger Hoffstätte
>> <holger.hoffstaette@googlemail.com> wrote:
>>> On 10/01/15 13:29, Eric Dumazet wrote:
>>
>>>> commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
>>>> Author: Eric Dumazet <edumazet@google.com>
>>>> Date:   Thu Aug 13 15:44:51 2015 -0700
>>>>
>>>>      inet: fix potential deadlock in reqsk_queue_unlink()
> <snip>
>>> Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
>>> we speak. Let's hope that this fixes the lockups.
>>>
>>
>> It definitely should help !
>>
>> David, since patch is not yet seen on
>> http://patchwork.ozlabs.org/bundle/davem/stable/?state=*
>> could you please add it to your queue ?
> 
> Seems to fix it for me as well. 3 systems have been running varying
> types of production-like loads with it for 14+ hours without hanging.

Just got up, and yes - my systems survived the night as well, no issues.

Greg, any chance you can drop this into the pending 4.1.10? Otherwise people
will get another broken release.

cheers
Holger


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-02  7:17           ` Holger Hoffstätte
@ 2015-10-02 19:25               ` Wolfgang Walter
  2015-10-03 19:14             ` Thomas D.
  1 sibling, 0 replies; 19+ messages in thread
From: Wolfgang Walter @ 2015-10-02 19:25 UTC (permalink / raw)
  To: Holger Hoffstätte
  Cc: Andre Tomt, Eric Dumazet, David S. Miller, Eric W. Biederman,
	Stephen Hemminger, Greg Kroah-Hartman, LKML, stable, netdev

Am Freitag, 2. Oktober 2015, 09:17:16 schrieb Holger Hoffstätte:
> On 10/02/15 08:52, Andre Tomt wrote:
> > On 01. okt. 2015 13:52, Eric Dumazet wrote:
> >> On Thu, Oct 1, 2015 at 4:43 AM, Holger Hoffstätte
> >> 
> >> <holger.hoffstaette@googlemail.com> wrote:
> >>> On 10/01/15 13:29, Eric Dumazet wrote:
> >>>> commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
> >>>> Author: Eric Dumazet <edumazet@google.com>
> >>>> Date:   Thu Aug 13 15:44:51 2015 -0700
> >>>> 
> >>>>      inet: fix potential deadlock in reqsk_queue_unlink()
> > 
> > <snip>
> > 
> >>> Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
> >>> we speak. Let's hope that this fixes the lockups.
> >> 
> >> It definitely should help !
> >> 
> >> David, since patch is not yet seen on
> >> http://patchwork.ozlabs.org/bundle/davem/stable/?state=*
> >> could you please add it to your queue ?
> > 
> > Seems to fix it for me as well. 3 systems have been running varying
> > types of production-like loads with it for 14+ hours without hanging.
> 
> Just got up, and yes - my systems survived the night as well, no issues.
> 
> Greg, any chance you can drop this into the pending 4.1.10? Otherwise people
> will get another broken release.
> 

Fixes the problem here, too.

Regards,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
@ 2015-10-02 19:25               ` Wolfgang Walter
  0 siblings, 0 replies; 19+ messages in thread
From: Wolfgang Walter @ 2015-10-02 19:25 UTC (permalink / raw)
  To: Holger Hoffstätte
  Cc: Andre Tomt, Eric Dumazet, David S. Miller, Eric W. Biederman,
	Stephen Hemminger, Greg Kroah-Hartman, LKML, stable, netdev

Am Freitag, 2. Oktober 2015, 09:17:16 schrieb Holger Hoffst�tte:
> On 10/02/15 08:52, Andre Tomt wrote:
> > On 01. okt. 2015 13:52, Eric Dumazet wrote:
> >> On Thu, Oct 1, 2015 at 4:43 AM, Holger Hoffst�tte
> >> 
> >> <holger.hoffstaette@googlemail.com> wrote:
> >>> On 10/01/15 13:29, Eric Dumazet wrote:
> >>>> commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
> >>>> Author: Eric Dumazet <edumazet@google.com>
> >>>> Date:   Thu Aug 13 15:44:51 2015 -0700
> >>>> 
> >>>>      inet: fix potential deadlock in reqsk_queue_unlink()
> > 
> > <snip>
> > 
> >>> Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
> >>> we speak. Let's hope that this fixes the lockups.
> >> 
> >> It definitely should help !
> >> 
> >> David, since patch is not yet seen on
> >> http://patchwork.ozlabs.org/bundle/davem/stable/?state=*
> >> could you please add it to your queue ?
> > 
> > Seems to fix it for me as well. 3 systems have been running varying
> > types of production-like loads with it for 14+ hours without hanging.
> 
> Just got up, and yes - my systems survived the night as well, no issues.
> 
> Greg, any chance you can drop this into the pending 4.1.10? Otherwise people
> will get another broken release.
> 

Fixes the problem here, too.

Regards,
-- 
Wolfgang Walter
Studentenwerk M�nchen
Anstalt des �ffentlichen Rechts

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-01 11:52       ` Eric Dumazet
  2015-10-02  6:52         ` Andre Tomt
@ 2015-10-02 20:04         ` Thomas Gleixner
  2015-10-02 20:59           ` Eric Dumazet
  2015-10-08 16:56         ` Christoph Biedl
  2 siblings, 1 reply; 19+ messages in thread
From: Thomas Gleixner @ 2015-10-02 20:04 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Holger Hoffstätte, David S. Miller, Eric W. Biederman,
	Stephen Hemminger, Greg Kroah-Hartman, LKML, stable, netdev

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1309 bytes --]

On Thu, 1 Oct 2015, Eric Dumazet wrote:
> On Thu, Oct 1, 2015 at 4:43 AM, Holger Hoffstätte
> <holger.hoffstaette@googlemail.com> wrote:
> > On 10/01/15 13:29, Eric Dumazet wrote:
> 
> >> commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
> >> Author: Eric Dumazet <edumazet@google.com>
> >> Date:   Thu Aug 13 15:44:51 2015 -0700
> >>
> >>     inet: fix potential deadlock in reqsk_queue_unlink()
> >>
> >>     When replacing del_timer() with del_timer_sync(), I introduced
> >>     a deadlock condition :
> >>
> >>     reqsk_queue_unlink() is called from inet_csk_reqsk_queue_drop()
> >>
> >>     inet_csk_reqsk_queue_drop() can be called from many contexts,
> >>     one being the timer handler itself (reqsk_timer_handler()).
> >>
> >>     In this case, del_timer_sync() loops forever.
> >>
> >>     Simple fix is to test if timer is pending.
> >>
> >>     Fixes: 2235f2ac75fd ("inet: fix races with reqsk timers")
> >>     Signed-off-by: Eric Dumazet <edumazet@google.com>
> >>     Signed-off-by: David S. Miller <davem@davemloft.net>
> >
> > Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
> > we speak. Let's hope that this fixes the lockups.
> >
> 
> It definitely should help !

What makes sure, that the timer cannot be readded while that timer
callback is running?

Thanks,

	tglx


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-02 20:04         ` Thomas Gleixner
@ 2015-10-02 20:59           ` Eric Dumazet
  2015-10-02 21:04             ` Thomas Gleixner
  0 siblings, 1 reply; 19+ messages in thread
From: Eric Dumazet @ 2015-10-02 20:59 UTC (permalink / raw)
  To: Thomas Gleixner
  Cc: Eric Dumazet, Holger Hoffstätte, David S. Miller,
	Eric W. Biederman, Stephen Hemminger, Greg Kroah-Hartman, LKML,
	stable, netdev

On Fri, 2015-10-02 at 22:04 +0200, Thomas Gleixner wrote:

> What makes sure, that the timer cannot be readded while that timer
> callback is running?

What is exactly your question ?




^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-02 20:59           ` Eric Dumazet
@ 2015-10-02 21:04             ` Thomas Gleixner
  2015-10-02 21:32               ` Eric Dumazet
  0 siblings, 1 reply; 19+ messages in thread
From: Thomas Gleixner @ 2015-10-02 21:04 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Eric Dumazet, Holger Hoffstätte, David S. Miller,
	Eric W. Biederman, Stephen Hemminger, Greg Kroah-Hartman, LKML,
	stable, netdev

On Fri, 2 Oct 2015, Eric Dumazet wrote:
> On Fri, 2015-10-02 at 22:04 +0200, Thomas Gleixner wrote:
> 
> > What makes sure, that the timer cannot be readded while that timer
> > callback is running?
> 
> What is exactly your question ?

CPU0   	  	       		CPU1

timer expires
  callback
				add timer
  timer_pending() == true
  ===> del_timer_sync()

I was just curious how this is prevented as I got lost in the
networking code as usual :)

Thanks,

	tglx


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-02 21:04             ` Thomas Gleixner
@ 2015-10-02 21:32               ` Eric Dumazet
  0 siblings, 0 replies; 19+ messages in thread
From: Eric Dumazet @ 2015-10-02 21:32 UTC (permalink / raw)
  To: Thomas Gleixner
  Cc: Eric Dumazet, Holger Hoffstätte, David S. Miller,
	Eric W. Biederman, Stephen Hemminger, Greg Kroah-Hartman, LKML,
	stable, netdev

On Fri, 2015-10-02 at 23:04 +0200, Thomas Gleixner wrote:
> On Fri, 2 Oct 2015, Eric Dumazet wrote:
> > On Fri, 2015-10-02 at 22:04 +0200, Thomas Gleixner wrote:
> > 
> > > What makes sure, that the timer cannot be readded while that timer
> > > callback is running?
> > 
> > What is exactly your question ?
> 
> CPU0   	  	       		CPU1
> 
> timer expires
>   callback
> 				add timer
>   timer_pending() == true
>   ===> del_timer_sync()
> 
> I was just curious how this is prevented as I got lost in the
> networking code as usual :)

Sure ;)

I believe this can not happen for following reasons :

mod_timer_pinned() is used only when req is created, while timer cannot
possibly be running on the same req. The _pinned part is critical
because we set the req->refcnt _after_ starting the timer,
to avoid being visible and caught from rcu lookups in hash tables.

Then, timer might be modified only by mod_timer_pending() from
tcp_check_req() : This should not re-start timer if another cpu is in
the timer callback.

Thanks




^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-02  7:17           ` Holger Hoffstätte
  2015-10-02 19:25               ` Wolfgang Walter
@ 2015-10-03 19:14             ` Thomas D.
  2015-10-17 23:41                 ` Greg Kroah-Hartman
  1 sibling, 1 reply; 19+ messages in thread
From: Thomas D. @ 2015-10-03 19:14 UTC (permalink / raw)
  To: Holger Hoffstätte, Andre Tomt, Eric Dumazet, stable
  Cc: David S. Miller, Eric W. Biederman, Stephen Hemminger,
	Greg Kroah-Hartman, LKML, netdev

Hi,

Holger Hoffstätte wrote:
> Greg, any chance you can drop this into the pending 4.1.10? Otherwise people
> will get another broken release.

For me it looks like the request was too late, the patch is not included
in 4.1.10. So don't forget to re-apply the patch when doing the upgrade.

Greg, do you need a dedicated inclusion request for
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
in 4.1.x or is it already on your list?


-Thomas




^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-01 11:52       ` Eric Dumazet
  2015-10-02  6:52         ` Andre Tomt
  2015-10-02 20:04         ` Thomas Gleixner
@ 2015-10-08 16:56         ` Christoph Biedl
  2015-10-08 19:27           ` Holger Hoffstätte
  2 siblings, 1 reply; 19+ messages in thread
From: Christoph Biedl @ 2015-10-08 16:56 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Holger Hoffstätte, Eric W. Biederman, LKML, stable

Eric Dumazet wrote...

[ commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af ]

> It definitely should help !

Yesterday, I've experienced issues somewhat similar to this, but I'm
not entirely sure:

Four of five systems running 4.1.9 stopped working. No reaction on
network, keyboard, serial console. In one case, the stack trace as
below made it to the loghost.

Two things are quite different. First, the systems had a reasonable
uptime, about a week.

And second, the scary part: All incidents happened within a rather
short time span of three minutes the most, beginning after 16:41:28 and
before 16:41:54 UTC. So I assumed a brownout first - until I realized
the systems faded away at slightly different times, and one is at a
different location. While other systems using different kernel versions
continued to operate on both sites.

So, I'd be glad for answers for

- Is this the same issue or should I be even more afraid?
- What might be the reason for this temporal coincidence? I have no
  plausible idea.

Confused,
    Christoph


 INFO: rcu_sched self-detected stall on CPU { 3}  (t=6000 jiffies g=8932806 c=8932805 q=58491)
 rcu_sched kthread starved for 5999 jiffies!
 Task dump for CPU 3:
 swapper/3       R  running task        0     0      1 0x00000008
  ffffffff81e396c0 ffff88042dcc3b20 ffffffff810807da 0000000000000003
  ffffffff81e396c0 ffff88042dcc3b40 ffffffff81083b78 ffff88042dcc3b80
  0000000000000003 ffff88042dcc3b70 ffffffff810a945c ffff88042dcd5740
 Call Trace:
  <IRQ>  [<ffffffff810807da>] sched_show_task+0xaa/0x110
  [<ffffffff81083b78>] dump_cpu_task+0x38/0x40
  [<ffffffff810a945c>] rcu_dump_cpu_stacks+0x8c/0xc0
  [<ffffffff810abf31>] rcu_check_callbacks+0x3b1/0x680
  [<ffffffff810e7bb7>] ? acct_account_cputime+0x17/0x20
  [<ffffffff8108484e>] ? account_system_time+0x8e/0x180
  [<ffffffff810ae4d3>] update_process_times+0x33/0x60
  [<ffffffff810bcae0>] tick_sched_handle.isra.14+0x30/0x40
  [<ffffffff810bcbd3>] tick_sched_timer+0x43/0x80
  [<ffffffff810aea2a>] __run_hrtimer.isra.32+0x4a/0xd0
  [<ffffffff810af225>] hrtimer_interrupt+0xd5/0x1f0
  [<ffffffff81034d84>] local_apic_timer_interrupt+0x34/0x60
 INFO: rcu_sched self-detected stall on CPU { 3}  (t=6000 jiffies g=8932806 c=8932805 q=58491)
 rcu_sched kthread starved for 5999 jiffies!
 Task dump for CPU 3:
 swapper/3       R  running task        0     0      1 0x00000008
  ffffffff81e396c0 ffff88042dcc3b20 ffffffff810807da 0000000000000003
  ffffffff81e396c0 ffff88042dcc3b40 ffffffff81083b78 ffff88042dcc3b80
  0000000000000003 ffff88042dcc3b70 ffffffff810a945c ffff88042dcd5740
 Call Trace:
  <IRQ>  [<ffffffff810807da>] sched_show_task+0xaa/0x110
  [<ffffffff81083b78>] dump_cpu_task+0x38/0x40
  [<ffffffff8103516c>] smp_apic_timer_interrupt+0x3c/0x60
  [<ffffffff8190db7b>] apic_timer_interrupt+0x6b/0x70
  [<ffffffff8190c8a9>] ? _raw_spin_unlock_irqrestore+0x9/0x10
  [<ffffffff810ade58>] try_to_del_timer_sync+0x48/0x60
  [<ffffffff810adeb2>] ? del_timer_sync+0x42/0x60
  [<ffffffff810adeba>] del_timer_sync+0x4a/0x60
  [<ffffffff8178b7da>] inet_csk_reqsk_queue_drop+0x7a/0x1f0
  [<ffffffff8178ba7f>] reqsk_timer_handler+0x12f/0x290
  [<ffffffff8178b950>] ? inet_csk_reqsk_queue_drop+0x1f0/0x1f0
  [<ffffffff810ad9e6>] call_timer_fn.isra.26+0x26/0x80
  [<ffffffff810a945c>] rcu_dump_cpu_stacks+0x8c/0xc0
  [<ffffffff810abf31>] rcu_check_callbacks+0x3b1/0x680
  [<ffffffff810e7bb7>] ? acct_account_cputime+0x17/0x20
  [<ffffffff8108484e>] ? account_system_time+0x8e/0x180
  [<ffffffff810ae4d3>] update_process_times+0x33/0x60
  [<ffffffff810bcae0>] tick_sched_handle.isra.14+0x30/0x40
  [<ffffffff810bcbd3>] tick_sched_timer+0x43/0x80
  [<ffffffff810aea2a>] __run_hrtimer.isra.32+0x4a/0xd0
  [<ffffffff810af225>] hrtimer_interrupt+0xd5/0x1f0
  [<ffffffff81034d84>] local_apic_timer_interrupt+0x34/0x60
  [<ffffffff810ae1ae>] run_timer_softirq+0x18e/0x220
  [<ffffffff81060b1a>] __do_softirq+0xda/0x1f0
  [<ffffffff81060e16>] irq_exit+0x76/0xa0
  [<ffffffff81035175>] smp_apic_timer_interrupt+0x45/0x60
  [<ffffffff8190db7b>] apic_timer_interrupt+0x6b/0x70
  <EOI>  [<ffffffff810844be>] ? sched_clock_cpu+0x9e/0xb0
  [<ffffffff8100bc15>] ? amd_e400_idle+0x35/0xd0
  [<ffffffff8100bc13>] ? amd_e400_idle+0x33/0xd0
  [<ffffffff8100c42a>] arch_cpu_idle+0xa/0x10
  [<ffffffff810929e3>] cpu_startup_entry+0x2c3/0x330
  [<ffffffff8103516c>] smp_apic_timer_interrupt+0x3c/0x60
  [<ffffffff8190db7b>] apic_timer_interrupt+0x6b/0x70
  [<ffffffff8190c8a9>] ? _raw_spin_unlock_irqrestore+0x9/0x10
  [<ffffffff810ade58>] try_to_del_timer_sync+0x48/0x60
  [<ffffffff810adeb2>] ? del_timer_sync+0x42/0x60
  [<ffffffff810adeba>] del_timer_sync+0x4a/0x60
  [<ffffffff8178b7da>] inet_csk_reqsk_queue_drop+0x7a/0x1f0
  [<ffffffff8178ba7f>] reqsk_timer_handler+0x12f/0x290
  [<ffffffff8178b950>] ? inet_csk_reqsk_queue_drop+0x1f0/0x1f0
  [<ffffffff810ad9e6>] call_timer_fn.isra.26+0x26/0x80
  [<ffffffff810332dc>] start_secondary+0x17c/0x1a0
  [<ffffffff810ae1ae>] run_timer_softirq+0x18e/0x220
  [<ffffffff81060b1a>] __do_softirq+0xda/0x1f0
  [<ffffffff81060e16>] irq_exit+0x76/0xa0
  [<ffffffff81035175>] smp_apic_timer_interrupt+0x45/0x60
  [<ffffffff8190db7b>] apic_timer_interrupt+0x6b/0x70
  <EOI>  [<ffffffff810844be>] ? sched_clock_cpu+0x9e/0xb0
  [<ffffffff8100bc15>] ? amd_e400_idle+0x35/0xd0
  [<ffffffff8100bc13>] ? amd_e400_idle+0x33/0xd0
  [<ffffffff8100c42a>] arch_cpu_idle+0xa/0x10
  [<ffffffff810929e3>] cpu_startup_entry+0x2c3/0x330
  [<ffffffff810332dc>] start_secondary+0x17c/0x1a0


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-08 16:56         ` Christoph Biedl
@ 2015-10-08 19:27           ` Holger Hoffstätte
  0 siblings, 0 replies; 19+ messages in thread
From: Holger Hoffstätte @ 2015-10-08 19:27 UTC (permalink / raw)
  To: Christoph Biedl, Eric Dumazet; +Cc: Eric W. Biederman, LKML, stable

On 10/08/15 18:56, Christoph Biedl wrote:
> Eric Dumazet wrote...
> 
> [ commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af ]
> 
>> It definitely should help !
> 
> Yesterday, I've experienced issues somewhat similar to this, but I'm
> not entirely sure:
> 
> Four of five systems running 4.1.9 stopped working. No reaction on
> network, keyboard, serial console. In one case, the stack trace as
> below made it to the loghost.
> 
> Two things are quite different. First, the systems had a reasonable
> uptime, about a week.
> 
> And second, the scary part: All incidents happened within a rather
> short time span of three minutes the most, beginning after 16:41:28 and
> before 16:41:54 UTC. So I assumed a brownout first - until I realized
> the systems faded away at slightly different times, and one is at a
> different location. While other systems using different kernel versions
> continued to operate on both sites.
> 
> So, I'd be glad for answers for
> 
> - Is this the same issue or should I be even more afraid?

There's always room for more. :-)

> - What might be the reason for this temporal coincidence? I have no
>   plausible idea.

More bugs?

> Confused,
>     Christoph
> 
> 
>  INFO: rcu_sched self-detected stall on CPU { 3}  (t=6000 jiffies g=8932806 c=8932805 q=58491)
>  rcu_sched kthread starved for 5999 jiffies!
>  Task dump for CPU 3:
>  swapper/3       R  running task        0     0      1 0x00000008
>   ffffffff81e396c0 ffff88042dcc3b20 ffffffff810807da 0000000000000003
>   ffffffff81e396c0 ffff88042dcc3b40 ffffffff81083b78 ffff88042dcc3b80
>   0000000000000003 ffff88042dcc3b70 ffffffff810a945c ffff88042dcd5740
>  Call Trace:
>   <IRQ>  [<ffffffff810807da>] sched_show_task+0xaa/0x110
>   [<ffffffff81083b78>] dump_cpu_task+0x38/0x40
>   [<ffffffff810a945c>] rcu_dump_cpu_stacks+0x8c/0xc0
>   [<ffffffff810abf31>] rcu_check_callbacks+0x3b1/0x680
>   [<ffffffff810e7bb7>] ? acct_account_cputime+0x17/0x20
>   [<ffffffff8108484e>] ? account_system_time+0x8e/0x180
>   [<ffffffff810ae4d3>] update_process_times+0x33/0x60
>   [<ffffffff810bcae0>] tick_sched_handle.isra.14+0x30/0x40
>   [<ffffffff810bcbd3>] tick_sched_timer+0x43/0x80
>   [<ffffffff810aea2a>] __run_hrtimer.isra.32+0x4a/0xd0
>   [<ffffffff810af225>] hrtimer_interrupt+0xd5/0x1f0
>   [<ffffffff81034d84>] local_apic_timer_interrupt+0x34/0x60
>  INFO: rcu_sched self-detected stall on CPU { 3}  (t=6000 jiffies g=8932806 c=8932805 q=58491)
>  rcu_sched kthread starved for 5999 jiffies!
>  Task dump for CPU 3:
>  swapper/3       R  running task        0     0      1 0x00000008
>   ffffffff81e396c0 ffff88042dcc3b20 ffffffff810807da 0000000000000003
>   ffffffff81e396c0 ffff88042dcc3b40 ffffffff81083b78 ffff88042dcc3b80
>   0000000000000003 ffff88042dcc3b70 ffffffff810a945c ffff88042dcd5740
>  Call Trace:
>   <IRQ>  [<ffffffff810807da>] sched_show_task+0xaa/0x110
>   [<ffffffff81083b78>] dump_cpu_task+0x38/0x40
>   [<ffffffff8103516c>] smp_apic_timer_interrupt+0x3c/0x60
>   [<ffffffff8190db7b>] apic_timer_interrupt+0x6b/0x70
>   [<ffffffff8190c8a9>] ? _raw_spin_unlock_irqrestore+0x9/0x10
>   [<ffffffff810ade58>] try_to_del_timer_sync+0x48/0x60
>   [<ffffffff810adeb2>] ? del_timer_sync+0x42/0x60
>   [<ffffffff810adeba>] del_timer_sync+0x4a/0x60
>   [<ffffffff8178b7da>] inet_csk_reqsk_queue_drop+0x7a/0x1f0
>   [<ffffffff8178ba7f>] reqsk_timer_handler+0x12f/0x290
>   [<ffffffff8178b950>] ? inet_csk_reqsk_queue_drop+0x1f0/0x1f0
>   [<ffffffff810ad9e6>] call_timer_fn.isra.26+0x26/0x80
>   [<ffffffff810a945c>] rcu_dump_cpu_stacks+0x8c/0xc0
>   [<ffffffff810abf31>] rcu_check_callbacks+0x3b1/0x680
>   [<ffffffff810e7bb7>] ? acct_account_cputime+0x17/0x20
>   [<ffffffff8108484e>] ? account_system_time+0x8e/0x180
>   [<ffffffff810ae4d3>] update_process_times+0x33/0x60
>   [<ffffffff810bcae0>] tick_sched_handle.isra.14+0x30/0x40
>   [<ffffffff810bcbd3>] tick_sched_timer+0x43/0x80
>   [<ffffffff810aea2a>] __run_hrtimer.isra.32+0x4a/0xd0
>   [<ffffffff810af225>] hrtimer_interrupt+0xd5/0x1f0
>   [<ffffffff81034d84>] local_apic_timer_interrupt+0x34/0x60
>   [<ffffffff810ae1ae>] run_timer_softirq+0x18e/0x220
>   [<ffffffff81060b1a>] __do_softirq+0xda/0x1f0
>   [<ffffffff81060e16>] irq_exit+0x76/0xa0
>   [<ffffffff81035175>] smp_apic_timer_interrupt+0x45/0x60
>   [<ffffffff8190db7b>] apic_timer_interrupt+0x6b/0x70
>   <EOI>  [<ffffffff810844be>] ? sched_clock_cpu+0x9e/0xb0
>   [<ffffffff8100bc15>] ? amd_e400_idle+0x35/0xd0
>   [<ffffffff8100bc13>] ? amd_e400_idle+0x33/0xd0
>   [<ffffffff8100c42a>] arch_cpu_idle+0xa/0x10
>   [<ffffffff810929e3>] cpu_startup_entry+0x2c3/0x330
>   [<ffffffff8103516c>] smp_apic_timer_interrupt+0x3c/0x60
>   [<ffffffff8190db7b>] apic_timer_interrupt+0x6b/0x70
>   [<ffffffff8190c8a9>] ? _raw_spin_unlock_irqrestore+0x9/0x10
>   [<ffffffff810ade58>] try_to_del_timer_sync+0x48/0x60
>   [<ffffffff810adeb2>] ? del_timer_sync+0x42/0x60
>   [<ffffffff810adeba>] del_timer_sync+0x4a/0x60
>   [<ffffffff8178b7da>] inet_csk_reqsk_queue_drop+0x7a/0x1f0
>   [<ffffffff8178ba7f>] reqsk_timer_handler+0x12f/0x290
>   [<ffffffff8178b950>] ? inet_csk_reqsk_queue_drop+0x1f0/0x1f0
>   [<ffffffff810ad9e6>] call_timer_fn.isra.26+0x26/0x80
>   [<ffffffff810332dc>] start_secondary+0x17c/0x1a0
>   [<ffffffff810ae1ae>] run_timer_softirq+0x18e/0x220
>   [<ffffffff81060b1a>] __do_softirq+0xda/0x1f0
>   [<ffffffff81060e16>] irq_exit+0x76/0xa0
>   [<ffffffff81035175>] smp_apic_timer_interrupt+0x45/0x60
>   [<ffffffff8190db7b>] apic_timer_interrupt+0x6b/0x70
>   <EOI>  [<ffffffff810844be>] ? sched_clock_cpu+0x9e/0xb0
>   [<ffffffff8100bc15>] ? amd_e400_idle+0x35/0xd0
>   [<ffffffff8100bc13>] ? amd_e400_idle+0x33/0xd0
>   [<ffffffff8100c42a>] arch_cpu_idle+0xa/0x10
>   [<ffffffff810929e3>] cpu_startup_entry+0x2c3/0x330
>   [<ffffffff810332dc>] start_secondary+0x17c/0x1a0
> 

The timer fixes were followups to a patch that went into 4.1 called
"tcp/dccp: get rid of central timewait timer", and it seems there were
a few more patches in that area very recently.

So after some git spelunking I am now running with the following patches
on top of 4.1.10 + 83fccfc3940.. (for the lockups), in the following
order:

fc01538f9fb75572c969ca9988176ffc2a8741d6 simplify timewait refcounting
dbe7faa4045ea83a37b691b12bb02a8f86c2d2e9 inet_twsk_deschedule factorization
29c6852602e259d2c1882f320b29d5c3fec0de04 fix races in reqsk_queue_hash_req()
ed2e923945892a8372ab70d2f61d364b0b6d9054 fix timewait races in timer handling

They may not all be required for the particular problem you just summoned,
but (from what I could tell) are required to apply everything properly.
They certainly can't make things worse. :-)

Oh and while you're at it you can apply these l33t cubic fixes :-)
30927520dbae297182990bb21d08762bcc35ce1d better follow cubic curve after idle period
c2e7204d180f8efc80f27959ca9cf16fa17f67db do not set epoch_start in the future
 
I've been running these on 3 machines for almost 10 minutes without issue,
so they are totally safe to go into production right away.

-h


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
  2015-10-03 19:14             ` Thomas D.
@ 2015-10-17 23:41                 ` Greg Kroah-Hartman
  0 siblings, 0 replies; 19+ messages in thread
From: Greg Kroah-Hartman @ 2015-10-17 23:41 UTC (permalink / raw)
  To: Thomas D.
  Cc: Holger Hoffstätte, Andre Tomt, Eric Dumazet, stable,
	David S. Miller, Eric W. Biederman, Stephen Hemminger, LKML,
	netdev

On Sat, Oct 03, 2015 at 09:14:16PM +0200, Thomas D. wrote:
> Hi,
> 
> Holger Hoffstätte wrote:
> > Greg, any chance you can drop this into the pending 4.1.10? Otherwise people
> > will get another broken release.
> 
> For me it looks like the request was too late, the patch is not included
> in 4.1.10. So don't forget to re-apply the patch when doing the upgrade.
> 
> Greg, do you need a dedicated inclusion request for
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
> in 4.1.x or is it already on your list?

Now applied, thanks.

greg k-h

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Soft lockup issue in Linux 4.1.9
@ 2015-10-17 23:41                 ` Greg Kroah-Hartman
  0 siblings, 0 replies; 19+ messages in thread
From: Greg Kroah-Hartman @ 2015-10-17 23:41 UTC (permalink / raw)
  To: Thomas D.
  Cc: Holger Hoffstätte, Andre Tomt, Eric Dumazet, stable,
	David S. Miller, Eric W. Biederman, Stephen Hemminger, LKML,
	netdev

On Sat, Oct 03, 2015 at 09:14:16PM +0200, Thomas D. wrote:
> Hi,
> 
> Holger Hoffst�tte wrote:
> > Greg, any chance you can drop this into the pending 4.1.10? Otherwise people
> > will get another broken release.
> 
> For me it looks like the request was too late, the patch is not included
> in 4.1.10. So don't forget to re-apply the patch when doing the upgrade.
> 
> Greg, do you need a dedicated inclusion request for
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
> in 4.1.x or is it already on your list?

Now applied, thanks.

greg k-h

^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2015-10-17 23:41 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-09-30 21:59 Soft lockup issue in Linux 4.1.9 Olivier Bonvalet
2015-09-30 22:37 ` Holger Hoffstätte
2015-10-01  4:41   ` Andre Tomt
2015-10-01 10:51     ` Holger Hoffstätte
     [not found] ` <560D1223.3070606@googlemail.com>
     [not found]   ` <CANn89i+B5T4Rhs8HnrC0+f+GhLvBFfpr4BVDvhkVOveSfy9B8Q@mail.gmail.com>
2015-10-01 11:43     ` Holger Hoffstätte
2015-10-01 11:52       ` Eric Dumazet
2015-10-02  6:52         ` Andre Tomt
2015-10-02  7:17           ` Holger Hoffstätte
2015-10-02 19:25             ` Wolfgang Walter
2015-10-02 19:25               ` Wolfgang Walter
2015-10-03 19:14             ` Thomas D.
2015-10-17 23:41               ` Greg Kroah-Hartman
2015-10-17 23:41                 ` Greg Kroah-Hartman
2015-10-02 20:04         ` Thomas Gleixner
2015-10-02 20:59           ` Eric Dumazet
2015-10-02 21:04             ` Thomas Gleixner
2015-10-02 21:32               ` Eric Dumazet
2015-10-08 16:56         ` Christoph Biedl
2015-10-08 19:27           ` Holger Hoffstätte

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.