All of lore.kernel.org
 help / color / mirror / Atom feed
* XFS hangs and freezes with LSI 9265-8i controller on high i/o
@ 2012-06-11 21:37 Matthew Whittaker-Williams
  2012-06-12  1:18 ` Dave Chinner
  0 siblings, 1 reply; 20+ messages in thread
From: Matthew Whittaker-Williams @ 2012-06-11 21:37 UTC (permalink / raw)
  To: xfs

[-- Attachment #1: Type: text/plain, Size: 7343 bytes --]

Dear Developers,

We are running into some problems with xfs and the LSI 9265-8i Controller.

http://www.lsi.com/products/storagecomponents/Pages/MegaRAIDSAS9265-8i.aspx

When running high i/o on raid 6 array with this controller xfs freezes 
up and we get the following errors:

Linux sd69 3.4.1-custom #4 SMP Mon Jun 11 09:35:31 CEST 2012 x86_64 
GNU/Linux

[   62.911481] XFS (sda): Mounting Filesystem
[   63.212456] XFS (sda): Starting recovery (logdev: internal)
[   64.016420] XFS (sda): Ending recovery (logdev: internal)
[   64.020549] XFS (sdb): Mounting Filesystem
[   64.371207] XFS (sdb): Starting recovery (logdev: internal)
[   65.265051] XFS (sdb): Ending recovery (logdev: internal)
[ 6110.298886] INFO: task kworker/0:0:11244 blocked for more than 120 
seconds.
[ 6110.298942] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[ 6110.299000] kworker/0:0     D ffff8805ecf52880     0 11244      2 
0x00000000
[ 6110.299044]  ffff8805ecf52880 0000000000000046 0000000000000000 
ffffffff81613020
[ 6110.299107]  00000000000132c0 ffff880582d65fd8 00000000000132c0 
ffff880582d65fd8
[ 6110.299170]  00000000000132c0 ffff8805ecf52880 00000000000132c0 
ffff880582d64010
[ 6110.299233] Call Trace:
[ 6110.299266]  [<ffffffff8134d55a>] ? schedule_timeout+0x2d/0xd7
[ 6110.299305]  [<ffffffff810f62f5>] ? kmem_cache_alloc+0x2a/0xee
[ 6110.299358]  [<ffffffffa02cbff4>] ? kmem_zone_alloc+0x58/0x9e [xfs]
[ 6110.299395]  [<ffffffff8134de6b>] ? __down_common+0x93/0xe4
[ 6110.299443]  [<ffffffffa03062b0>] ? xfs_getsb+0x2f/0x5c [xfs]
[ 6110.299480]  [<ffffffff81057994>] ? down+0x27/0x37
[ 6110.299520]  [<ffffffffa02b81e7>] ? xfs_buf_lock+0x65/0xb2 [xfs]
[ 6110.299568]  [<ffffffffa03062b0>] ? xfs_getsb+0x2f/0x5c [xfs]
[ 6110.299613]  [<ffffffffa0312e3b>] ? xfs_trans_getsb+0xa5/0xf5 [xfs]
[ 6110.299663]  [<ffffffffa0306c9a>] ? xfs_mod_sb+0x43/0x10f [xfs]
[ 6110.299710]  [<ffffffffa02c70f6>] ? xfs_flush_inodes+0x23/0x23 [xfs]
[ 6110.299755]  [<ffffffffa02bcd06>] ? xfs_fs_log_dummy+0x61/0x75 [xfs]
[ 6110.299802]  [<ffffffffa0311978>] ? xfs_ail_min_lsn+0xd/0x2e [xfs]
[ 6110.299849]  [<ffffffffa02c7133>] ? xfs_sync_worker+0x3d/0x60 [xfs]
[ 6110.299888]  [<ffffffff812703b6>] ? powersave_bias_target+0x14b/0x14b
[ 6110.299924]  [<ffffffff8104fa39>] ? process_one_work+0x1cd/0x2eb
[ 6110.299960]  [<ffffffff8104fc85>] ? worker_thread+0x12e/0x249
[ 6110.299993]  [<ffffffff8104fb57>] ? process_one_work+0x2eb/0x2eb
[ 6110.300029]  [<ffffffff8104fb57>] ? process_one_work+0x2eb/0x2eb
[ 6110.300064]  [<ffffffff8105356e>] ? kthread+0x81/0x89
[ 6110.300098]  [<ffffffff813569a4>] ? kernel_thread_helper+0x4/0x10
[ 6110.300133]  [<ffffffff810534ed>] ? 
kthread_freezable_should_stop+0x53/0x53
[ 6110.300171]  [<ffffffff813569a0>] ? gs_change+0x13/0x13
[ 7547.340316] INFO: task kworker/0:0:11244 blocked for more than 120 
seconds.
[ 7547.340359] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[ 7547.340414] kworker/0:0     D ffff8805ecf52880     0 11244      2 
0x00000000
[ 7547.340458]  ffff8805ecf52880 0000000000000046 0000000000000000 
ffffffff81613020
[ 7547.340522]  00000000000132c0 ffff880582d65fd8 00000000000132c0 
ffff880582d65fd8
[ 7547.340585]  00000000000132c0 ffff8805ecf52880 00000000000132c0 
ffff880582d64010
[ 7547.340648] Call Trace:
[ 7547.340680]  [<ffffffff8134d55a>] ? schedule_timeout+0x2d/0xd7
[ 7547.340719]  [<ffffffff810f62f5>] ? kmem_cache_alloc+0x2a/0xee
[ 7547.340772]  [<ffffffffa02cbff4>] ? kmem_zone_alloc+0x58/0x9e [xfs]
[ 7547.340809]  [<ffffffff8134de6b>] ? __down_common+0x93/0xe4
[ 7547.340858]  [<ffffffffa03062b0>] ? xfs_getsb+0x2f/0x5c [xfs]
[ 7547.340895]  [<ffffffff81057994>] ? down+0x27/0x37
[ 7547.340934]  [<ffffffffa02b81e7>] ? xfs_buf_lock+0x65/0xb2 [xfs]
[ 7547.340983]  [<ffffffffa03062b0>] ? xfs_getsb+0x2f/0x5c [xfs]
[ 7547.341028]  [<ffffffffa0312e3b>] ? xfs_trans_getsb+0xa5/0xf5 [xfs]
[ 7547.341078]  [<ffffffffa0306c9a>] ? xfs_mod_sb+0x43/0x10f [xfs]
[ 7547.341126]  [<ffffffffa02c70f6>] ? xfs_flush_inodes+0x23/0x23 [xfs]
[ 7547.341170]  [<ffffffffa02bcd06>] ? xfs_fs_log_dummy+0x61/0x75 [xfs]
[ 7547.341217]  [<ffffffffa0311978>] ? xfs_ail_min_lsn+0xd/0x2e [xfs]
[ 7547.346755]  [<ffffffffa02c7133>] ? xfs_sync_worker+0x3d/0x60 [xfs]
[ 7547.346794]  [<ffffffff812703b6>] ? powersave_bias_target+0x14b/0x14b
[ 7547.346832]  [<ffffffff8104fa39>] ? process_one_work+0x1cd/0x2eb
[ 7547.346870]  [<ffffffff8104fc85>] ? worker_thread+0x12e/0x249
[ 7547.346905]  [<ffffffff8104fb57>] ? process_one_work+0x2eb/0x2eb
[ 7547.346940]  [<ffffffff8104fb57>] ? process_one_work+0x2eb/0x2eb
[ 7547.346976]  [<ffffffff8105356e>] ? kthread+0x81/0x89
[ 7547.347012]  [<ffffffff813569a4>] ? kernel_thread_helper+0x4/0x10
[ 7547.347048]  [<ffffffff810534ed>] ? 
kthread_freezable_should_stop+0x53/0x53
[ 7547.347085]  [<ffffffff813569a0>] ? gs_change+0x13/0x13
[ 9463.398196] INFO: task kworker/0:0:11244 blocked for more than 120 
seconds.
[ 9463.398270] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[ 9463.398325] kworker/0:0     D ffff8805ecf52880     0 11244      2 
0x00000000
[ 9463.398369]  ffff8805ecf52880 0000000000000046 0000000000000000 
ffffffff81613020
[ 9463.398433]  00000000000132c0 ffff880582d65fd8 00000000000132c0 
ffff880582d65fd8
[ 9463.398496]  00000000000132c0 ffff8805ecf52880 00000000000132c0 
ffff880582d64010
[ 9463.398559] Call Trace:
[ 9463.398592]  [<ffffffff8134d55a>] ? schedule_timeout+0x2d/0xd7
[ 9463.398630]  [<ffffffff810f62f5>] ? kmem_cache_alloc+0x2a/0xee
[ 9463.398683]  [<ffffffffa02cbff4>] ? kmem_zone_alloc+0x58/0x9e [xfs]
[ 9463.398720]  [<ffffffff8134de6b>] ? __down_common+0x93/0xe4
[ 9463.398768]  [<ffffffffa03062b0>] ? xfs_getsb+0x2f/0x5c [xfs]
[ 9463.398804]  [<ffffffff81057994>] ? down+0x27/0x37
[ 9463.398843]  [<ffffffffa02b81e7>] ? xfs_buf_lock+0x65/0xb2 [xfs]
[ 9463.398892]  [<ffffffffa03062b0>] ? xfs_getsb+0x2f/0x5c [xfs]
[ 9463.398937]  [<ffffffffa0312e3b>] ? xfs_trans_getsb+0xa5/0xf5 [xfs]
[ 9463.398987]  [<ffffffffa0306c9a>] ? xfs_mod_sb+0x43/0x10f [xfs]
[ 9463.399034]  [<ffffffffa02c70f6>] ? xfs_flush_inodes+0x23/0x23 [xfs]
[ 9463.399079]  [<ffffffffa02bcd06>] ? xfs_fs_log_dummy+0x61/0x75 [xfs]
[ 9463.399126]  [<ffffffffa0311978>] ? xfs_ail_min_lsn+0xd/0x2e [xfs]
[ 9463.399174]  [<ffffffffa02c7133>] ? xfs_sync_worker+0x3d/0x60 [xfs]
[ 9463.399211]  [<ffffffff8104fa39>] ? process_one_work+0x1cd/0x2eb
[ 9463.399246]  [<ffffffff8104fc85>] ? worker_thread+0x12e/0x249
[ 9463.399281]  [<ffffffff8104fb57>] ? process_one_work+0x2eb/0x2eb
[ 9463.399316]  [<ffffffff8104fb57>] ? process_one_work+0x2eb/0x2eb
[ 9463.399351]  [<ffffffff8105356e>] ? kthread+0x81/0x89
[ 9463.399385]  [<ffffffff813569a4>] ? kernel_thread_helper+0x4/0x10
[ 9463.399422]  [<ffffffff810534ed>] ? 
kthread_freezable_should_stop+0x53/0x53
[ 9463.399459]  [<ffffffff813569a0>] ? gs_change+0x13/0x13


We tried the following linux kernels but same errors occurs.

3.0.33
3.2.14
3.2.18
3.3.7
3.4.1

We also tried the 2.6.35-13 kernel but this kernel is unable to load see 
the disk even when modules are loaded.

See attachments for system information.

Note we tried several of the same controller but errors persist.

Could you have a look into this issue?

If you need any more information I am happy to provide it.

Thanks

Kind regards,

Matthew Whittaker-Williams





[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 56991 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.4.1-custom (root@squeeze) (gcc version 4.4.5 (Debian 4.4.5-8) ) #4 SMP Mon Jun 11 09:35:31 CEST 2012
[    0.000000] Command line: vga=normal initrd=debian/squeeze/diskless/initrd.img-3.4.1-custom boot=live live-config noeject noswap noprompt nolocales edd=on nomodeset nosplash quickreboot scandelay=5 panic=60 debug=1 fetch=http://10.0.0.59/debian/squeeze/filesystem-3.4.1.squashfs BOOT_IMAGE=debian/squeeze/diskless/vmlinuz-3.4.1-custom auto
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 0000000000099800 (usable)
[    0.000000]  BIOS-e820: 0000000000099800 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000df780000 (usable)
[    0.000000]  BIOS-e820: 00000000df78e000 - 00000000df790000 type 9
[    0.000000]  BIOS-e820: 00000000df790000 - 00000000df79e000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000df79e000 - 00000000df7d0000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000df7d0000 - 00000000df7e0000 (reserved)
[    0.000000]  BIOS-e820: 00000000df7ec000 - 00000000f0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000620000000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI present.
[    0.000000] DMI: Supermicro X8ST3/X8ST3, BIOS 2.0        07/29/10  
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x620000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-CFFFF write-protect
[    0.000000]   D0000-DFFFF uncachable
[    0.000000]   E0000-E3FFF write-protect
[    0.000000]   E4000-EFFFF write-through
[    0.000000]   F0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 0000000000 mask FC00000000 write-back
[    0.000000]   1 base 0400000000 mask FE00000000 write-back
[    0.000000]   2 base 0600000000 mask FFE0000000 write-back
[    0.000000]   3 base 00E0000000 mask FFE0000000 uncachable
[    0.000000]   4 base 00DF800000 mask FFFF800000 uncachable
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] e820 update range: 00000000df800000 - 0000000100000000 (usable) ==> (reserved)
[    0.000000] last_pfn = 0xdf780 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [ffff8800000ff780] ff780
[    0.000000] initial memory mapped : 0 - 20000000
[    0.000000] Base memory trampoline at [ffff880000094000] 94000 size 20480
[    0.000000] init_memory_mapping: 0000000000000000-00000000df780000
[    0.000000]  0000000000 - 00df600000 page 2M
[    0.000000]  00df600000 - 00df780000 page 4k
[    0.000000] kernel direct mapping tables up to df780000 @ 1fffa000-20000000
[    0.000000] init_memory_mapping: 0000000100000000-0000000620000000
[    0.000000]  0100000000 - 0620000000 page 2M
[    0.000000] kernel direct mapping tables up to 620000000 @ df766000-df780000
[    0.000000] RAMDISK: 79808000 - 80000000
[    0.000000] ACPI: RSDP 00000000000faa70 00024 (v02 ACPIAM)
[    0.000000] ACPI: XSDT 00000000df790100 00074 (v01 SMCI            20100729 MSFT 00000097)
[    0.000000] ACPI: FACP 00000000df790290 000F4 (v03 072910 FACP2001 20100729 MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000df790630 06761 (v01  1F580 1F580000 00000000 INTL 20051117)
[    0.000000] ACPI: FACS 00000000df79e000 00040
[    0.000000] ACPI: APIC 00000000df790390 000D2 (v01 072910 APIC2001 20100729 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000df790470 0003C (v01 072910 OEMMCFG  20100729 MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000df79e040 0007B (v01 072910 OEMB2001 20100729 MSFT 00000097)
[    0.000000] ACPI: HPET 00000000df79a630 00038 (v01 072910 OEMHPET  20100729 MSFT 00000097)
[    0.000000] ACPI: SSDT 00000000df79ee90 00363 (v01 DpgPmm    CpuPm 00000012 INTL 20051117)
[    0.000000] ACPI: EINJ 00000000df79a670 00130 (v01  AMIER AMI_EINJ 20100729 MSFT 00000097)
[    0.000000] ACPI: BERT 00000000df79a800 00030 (v01  AMIER AMI_BERT 20100729 MSFT 00000097)
[    0.000000] ACPI: ERST 00000000df79a830 001B0 (v01  AMIER AMI_ERST 20100729 MSFT 00000097)
[    0.000000] ACPI: HEST 00000000df79a9e0 000A8 (v01  AMIER ABC_HEST 20100729 MSFT 00000097)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-0000000620000000
[    0.000000] Initmem setup node 0 0000000000000000-0000000620000000
[    0.000000]   NODE_DATA [000000061fffc000 - 000000061fffffff]
[    0.000000]  [ffffea0000000000-ffffea00187fffff] PMD -> [ffff880607600000-ffff88061f5fffff] on node 0
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00620000
[    0.000000] Movable zone start PFN for each node
[    0.000000] Early memory PFN ranges
[    0.000000]     0: 0x00000010 -> 0x00000099
[    0.000000]     0: 0x00000100 -> 0x000df780
[    0.000000]     0: 0x00100000 -> 0x00620000
[    0.000000] On node 0 totalpages: 6289161
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 5 pages reserved
[    0.000000]   DMA zone: 3908 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 16320 pages used for memmap
[    0.000000]   DMA32 zone: 894912 pages, LIFO batch:31
[    0.000000]   Normal zone: 83968 pages used for memmap
[    0.000000]   Normal zone: 5289984 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x86] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x87] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x88] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x89] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x8a] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x8b] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x8c] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x8d] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x8e] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x8f] disabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a301 base: 0xfed00000
[    0.000000] SMP: Allowing 16 CPUs, 12 hotplug CPUs
[    0.000000] nr_irqs_gsi: 40
[    0.000000] PM: Registered nosave memory: 0000000000099000 - 000000000009a000
[    0.000000] PM: Registered nosave memory: 000000000009a000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
[    0.000000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
[    0.000000] PM: Registered nosave memory: 00000000df780000 - 00000000df78e000
[    0.000000] PM: Registered nosave memory: 00000000df78e000 - 00000000df790000
[    0.000000] PM: Registered nosave memory: 00000000df790000 - 00000000df79e000
[    0.000000] PM: Registered nosave memory: 00000000df79e000 - 00000000df7d0000
[    0.000000] PM: Registered nosave memory: 00000000df7d0000 - 00000000df7e0000
[    0.000000] PM: Registered nosave memory: 00000000df7e0000 - 00000000df7ec000
[    0.000000] PM: Registered nosave memory: 00000000df7ec000 - 00000000f0000000
[    0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fee00000
[    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
[    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000ffc00000
[    0.000000] PM: Registered nosave memory: 00000000ffc00000 - 0000000100000000
[    0.000000] Allocating PCI resources starting at f0000000 (gap: f0000000:ee00000)
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:16 nr_node_ids:1
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88061fc00000 s81600 r8192 d20800 u131072
[    0.000000] pcpu-alloc: s81600 r8192 d20800 u131072 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 6188804
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: vga=normal initrd=debian/squeeze/diskless/initrd.img-3.4.1-custom boot=live live-config noeject noswap noprompt nolocales edd=on nomodeset nosplash quickreboot scandelay=5 panic=60 debug=1 fetch=http://10.0.0.59/debian/squeeze/filesystem-3.4.1.squashfs BOOT_IMAGE=debian/squeeze/diskless/vmlinuz-3.4.1-custom auto
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 24580644k/25690112k available (3428k kernel code, 533468k absent, 576000k reserved, 3203k data, 576k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=16, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS:33024 nr_irqs:808 16
[    0.000000] Extended CMOS year: 2000
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty0] enabled
[    0.000000] hpet clockevent registered
[    0.000000] Fast TSC calibration using PIT
[    0.004000] Detected 2133.449 MHz processor.
[    0.000003] Calibrating delay loop (skipped), value calculated using timer frequency.. 4266.89 BogoMIPS (lpj=8533796)
[    0.000132] pid_max: default: 32768 minimum: 301
[    0.000224] Security Framework initialized
[    0.000287] SELinux:  Disabled at boot.
[    0.002227] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
[    0.009748] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.012932] Mount-cache hash table entries: 256
[    0.013212] Initializing cgroup subsys cpuacct
[    0.013276] Initializing cgroup subsys devices
[    0.013338] Initializing cgroup subsys freezer
[    0.013400] Initializing cgroup subsys net_cls
[    0.013484] CPU: Physical Processor ID: 0
[    0.013545] CPU: Processor Core ID: 0
[    0.013609] mce: CPU supports 9 MCE banks
[    0.013680] CPU0: Thermal monitoring enabled (TM1)
[    0.013748] using mwait in idle threads.
[    0.014660] ACPI: Core revision 20120320
[    0.039679] Switched APIC routing to physical flat.
[    0.040138] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.079806] CPU0: Intel(R) Xeon(R) CPU           E5506  @ 2.13GHz stepping 05
[    0.184018] Performance Events: PEBS fmt1+, 16-deep LBR, Nehalem events, Intel PMU driver.
[    0.184252] CPU erratum AAJ80 worked around
[    0.184314] CPUID marked event: 'bus cycles' unavailable
[    0.184379] ... version:                3
[    0.184440] ... bit width:              48
[    0.184501] ... generic registers:      4
[    0.184562] ... value mask:             0000ffffffffffff
[    0.184626] ... max period:             000000007fffffff
[    0.184690] ... fixed-purpose events:   3
[    0.184750] ... event mask:             000000070000000f
[    0.184981] Booting Node   0, Processors  #1 #2 #3
[    0.224498] Brought up 4 CPUs
[    0.224612] Total of 4 processors activated (17067.59 BogoMIPS).
[    0.227001] devtmpfs: initialized
[    0.233189] PM: Registering ACPI NVS region [mem 0xdf79e000-0xdf7cffff] (204800 bytes)
[    0.233360] dummy: 
[    0.233465] NET: Registered protocol family 16
[    0.233744] ACPI: bus type pci registered
[    0.233864] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.233954] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[    0.256570] PCI: Using configuration type 1 for base access
[    0.257486] bio: create slab <bio-0> at 0
[    0.257614] ACPI: Added _OSI(Module Device)
[    0.257677] ACPI: Added _OSI(Processor Device)
[    0.257739] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.257803] ACPI: Added _OSI(Processor Aggregator Device)
[    0.258717] ACPI: EC: Look up EC in DSDT
[    0.259154] \_SB_:_OSC evaluation returned wrong type
[    0.259218] _OSC request data:1 7 
[    0.260127] ACPI: Executed 1 blocks of module-level executable AML code
[    0.263750] ACPI: SSDT 00000000df79e0c0 008F0 (v01 DpgPmm  P001Ist 00000011 INTL 20051117)
[    0.264166] ACPI: Dynamic OEM Table Load:
[    0.264301] ACPI: SSDT           (null) 008F0 (v01 DpgPmm  P001Ist 00000011 INTL 20051117)
[    0.264550] ACPI: SSDT 00000000df79e9b0 004D5 (v01  PmRef  P001Cst 00003001 INTL 20051117)
[    0.264909] ACPI: Dynamic OEM Table Load:
[    0.265043] ACPI: SSDT           (null) 004D5 (v01  PmRef  P001Cst 00003001 INTL 20051117)
[    0.265572] ACPI: Interpreter enabled
[    0.265635] ACPI: (supports S0 S1 S4 S5)
[    0.265858] ACPI: Using IOAPIC for interrupt routing
[    0.272421] ACPI: No dock devices found.
[    0.272485] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.272659] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.272866] pci_root PNP0A08:00: host bridge window [io  0x0000-0x0cf7]
[    0.272934] pci_root PNP0A08:00: host bridge window [io  0x0d00-0xffff]
[    0.273002] pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
[    0.273087] pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000dffff]
[    0.273172] pci_root PNP0A08:00: host bridge window [mem 0xf0000000-0xfed8ffff]
[    0.273283] PCI host bridge to bus 0000:00
[    0.273346] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
[    0.273413] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
[    0.273479] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
[    0.273547] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000dffff]
[    0.273615] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfed8ffff]
[    0.273695] pci 0000:00:00.0: [8086:3405] type 00 class 0x060000
[    0.273819] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    0.273909] pci 0000:00:01.0: [8086:3408] type 01 class 0x060400
[    0.274035] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[    0.274126] pci 0000:00:03.0: [8086:340a] type 01 class 0x060400
[    0.274252] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[    0.274342] pci 0000:00:05.0: [8086:340c] type 01 class 0x060400
[    0.274468] pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
[    0.274557] pci 0000:00:07.0: [8086:340e] type 01 class 0x060400
[    0.274683] pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
[    0.274773] pci 0000:00:09.0: [8086:3410] type 01 class 0x060400
[    0.274898] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
[    0.274991] pci 0000:00:14.0: [8086:342e] type 00 class 0x080000
[    0.275127] pci 0000:00:14.1: [8086:3422] type 00 class 0x080000
[    0.275265] pci 0000:00:14.2: [8086:3423] type 00 class 0x080000
[    0.275398] pci 0000:00:14.3: [8086:3438] type 00 class 0x080000
[    0.275526] pci 0000:00:16.0: [8086:3430] type 00 class 0x088000
[    0.275603] pci 0000:00:16.0: reg 10: [mem 0xfbef8000-0xfbefbfff 64bit]
[    0.275745] pci 0000:00:16.1: [8086:3431] type 00 class 0x088000
[    0.275826] pci 0000:00:16.1: reg 10: [mem 0xfbef4000-0xfbef7fff 64bit]
[    0.275969] pci 0000:00:16.2: [8086:3432] type 00 class 0x088000
[    0.276046] pci 0000:00:16.2: reg 10: [mem 0xfbef0000-0xfbef3fff 64bit]
[    0.276187] pci 0000:00:16.3: [8086:3433] type 00 class 0x088000
[    0.276265] pci 0000:00:16.3: reg 10: [mem 0xfbeec000-0xfbeeffff 64bit]
[    0.276406] pci 0000:00:16.4: [8086:3429] type 00 class 0x088000
[    0.276484] pci 0000:00:16.4: reg 10: [mem 0xfbee8000-0xfbeebfff 64bit]
[    0.276625] pci 0000:00:16.5: [8086:342a] type 00 class 0x088000
[    0.276702] pci 0000:00:16.5: reg 10: [mem 0xfbee4000-0xfbee7fff 64bit]
[    0.276845] pci 0000:00:16.6: [8086:342b] type 00 class 0x088000
[    0.276922] pci 0000:00:16.6: reg 10: [mem 0xfbee0000-0xfbee3fff 64bit]
[    0.277064] pci 0000:00:16.7: [8086:342c] type 00 class 0x088000
[    0.277141] pci 0000:00:16.7: reg 10: [mem 0xfbedc000-0xfbedffff 64bit]
[    0.277285] pci 0000:00:1a.0: [8086:3a37] type 00 class 0x0c0300
[    0.277393] pci 0000:00:1a.0: reg 20: [io  0xac00-0xac1f]
[    0.277509] pci 0000:00:1a.1: [8086:3a38] type 00 class 0x0c0300
[    0.277618] pci 0000:00:1a.1: reg 20: [io  0xa880-0xa89f]
[    0.277734] pci 0000:00:1a.2: [8086:3a39] type 00 class 0x0c0300
[    0.277842] pci 0000:00:1a.2: reg 20: [io  0xa800-0xa81f]
[    0.277968] pci 0000:00:1a.7: [8086:3a3c] type 00 class 0x0c0320
[    0.278054] pci 0000:00:1a.7: reg 10: [mem 0xfbeda000-0xfbeda3ff]
[    0.278211] pci 0000:00:1a.7: PME# supported from D0 D3hot D3cold
[    0.278300] pci 0000:00:1c.0: [8086:3a40] type 01 class 0x060400
[    0.278438] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    0.278527] pci 0000:00:1c.1: [8086:3a42] type 01 class 0x060400
[    0.278665] pci 0000:00:1c.1: PME# supported from D0 D3hot D3cold
[    0.278761] pci 0000:00:1d.0: [8086:3a34] type 00 class 0x0c0300
[    0.278870] pci 0000:00:1d.0: reg 20: [io  0xa480-0xa49f]
[    0.278986] pci 0000:00:1d.1: [8086:3a35] type 00 class 0x0c0300
[    0.279095] pci 0000:00:1d.1: reg 20: [io  0xa400-0xa41f]
[    0.279212] pci 0000:00:1d.2: [8086:3a36] type 00 class 0x0c0300
[    0.279320] pci 0000:00:1d.2: reg 20: [io  0xa080-0xa09f]
[    0.279447] pci 0000:00:1d.7: [8086:3a3a] type 00 class 0x0c0320
[    0.279532] pci 0000:00:1d.7: reg 10: [mem 0xfbed8000-0xfbed83ff]
[    0.279690] pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
[    0.279777] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
[    0.279910] pci 0000:00:1f.0: [8086:3a16] type 00 class 0x060100
[    0.280058] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 1 PIO at 4700 (mask 0097)
[    0.280144] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 2 PIO at 03e8 (mask 000f)
[    0.280230] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 3 PIO at 1640 (mask 000f)
[    0.280316] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 4 PIO at 0ca0 (mask 000f)
[    0.280448] pci 0000:00:1f.2: [8086:3a22] type 00 class 0x010601
[    0.280532] pci 0000:00:1f.2: reg 10: [io  0x9480-0x9487]
[    0.280603] pci 0000:00:1f.2: reg 14: [io  0xa000-0xa003]
[    0.280673] pci 0000:00:1f.2: reg 18: [io  0x9c00-0x9c07]
[    0.280744] pci 0000:00:1f.2: reg 1c: [io  0x9880-0x9883]
[    0.280815] pci 0000:00:1f.2: reg 20: [io  0x9800-0x981f]
[    0.280886] pci 0000:00:1f.2: reg 24: [mem 0xfbed6000-0xfbed67ff]
[    0.280999] pci 0000:00:1f.2: PME# supported from D3hot
[    0.281080] pci 0000:00:1f.3: [8086:3a30] type 00 class 0x0c0500
[    0.281160] pci 0000:00:1f.3: reg 10: [mem 0xfbed4000-0xfbed40ff 64bit]
[    0.281247] pci 0000:00:1f.3: reg 20: [io  0x0400-0x041f]
[    0.281375] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    0.281482] pci 0000:00:03.0: PCI bridge to [bus 02-02]
[    0.281589] pci 0000:00:05.0: PCI bridge to [bus 03-03]
[    0.281707] pci 0000:04:00.0: [1000:005b] type 00 class 0x010400
[    0.281782] pci 0000:04:00.0: reg 10: [io  0xb000-0xb0ff]
[    0.281857] pci 0000:04:00.0: reg 14: [mem 0xfb87c000-0xfb87ffff 64bit]
[    0.287396] pci 0000:04:00.0: reg 1c: [mem 0xfb8c0000-0xfb8fffff 64bit]
[    0.287475] pci 0000:04:00.0: reg 30: [mem 0xfb880000-0xfb8bffff pref]
[    0.287583] pci 0000:04:00.0: supports D1 D2
[    0.291803] pci 0000:00:07.0: PCI bridge to [bus 04-04]
[    0.291876] pci 0000:00:07.0:   bridge window [io  0xb000-0xbfff]
[    0.291944] pci 0000:00:07.0:   bridge window [mem 0xfb800000-0xfb8fffff]
[    0.292063] pci 0000:05:00.0: [1000:0058] type 00 class 0x010000
[    0.292139] pci 0000:05:00.0: reg 10: [io  0xc000-0xc0ff]
[    0.292214] pci 0000:05:00.0: reg 14: [mem 0xfb9ec000-0xfb9effff 64bit]
[    0.292291] pci 0000:05:00.0: reg 1c: [mem 0xfb9f0000-0xfb9fffff 64bit]
[    0.292372] pci 0000:05:00.0: reg 30: [mem 0xfba00000-0xfbbfffff pref]
[    0.292479] pci 0000:05:00.0: supports D1 D2
[    0.292553] pci 0000:05:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.292649] pci 0000:00:09.0: PCI bridge to [bus 05-05]
[    0.292715] pci 0000:00:09.0:   bridge window [io  0xc000-0xcfff]
[    0.292783] pci 0000:00:09.0:   bridge window [mem 0xfb900000-0xfbbfffff]
[    0.292921] pci 0000:06:00.0: [8086:10d3] type 00 class 0x020000
[    0.293009] pci 0000:06:00.0: reg 10: [mem 0xfbce0000-0xfbcfffff]
[    0.293107] pci 0000:06:00.0: reg 18: [io  0xdc00-0xdc1f]
[    0.293188] pci 0000:06:00.0: reg 1c: [mem 0xfbcdc000-0xfbcdffff]
[    0.293395] pci 0000:06:00.0: PME# supported from D0 D3hot D3cold
[    0.299794] pci 0000:00:1c.0: PCI bridge to [bus 06-06]
[    0.299867] pci 0000:00:1c.0:   bridge window [io  0xd000-0xdfff]
[    0.299936] pci 0000:00:1c.0:   bridge window [mem 0xfbc00000-0xfbcfffff]
[    0.300075] pci 0000:07:00.0: [8086:10d3] type 00 class 0x020000
[    0.300163] pci 0000:07:00.0: reg 10: [mem 0xfbde0000-0xfbdfffff]
[    0.300261] pci 0000:07:00.0: reg 18: [io  0xec00-0xec1f]
[    0.300342] pci 0000:07:00.0: reg 1c: [mem 0xfbddc000-0xfbddffff]
[    0.300549] pci 0000:07:00.0: PME# supported from D0 D3hot D3cold
[    0.307781] pci 0000:00:1c.1: PCI bridge to [bus 07-07]
[    0.307854] pci 0000:00:1c.1:   bridge window [io  0xe000-0xefff]
[    0.307923] pci 0000:00:1c.1:   bridge window [mem 0xfbd00000-0xfbdfffff]
[    0.308030] pci 0000:08:04.0: [102b:0532] type 00 class 0x030000
[    0.308111] pci 0000:08:04.0: reg 10: [mem 0xf9000000-0xf9ffffff pref]
[    0.308187] pci 0000:08:04.0: reg 14: [mem 0xfaffc000-0xfaffffff]
[    0.308261] pci 0000:08:04.0: reg 18: [mem 0xfb000000-0xfb7fffff]
[    0.308425] pci 0000:00:1e.0: PCI bridge to [bus 08-08] (subtractive decode)
[    0.308497] pci 0000:00:1e.0:   bridge window [mem 0xfaf00000-0xfb7fffff]
[    0.308569] pci 0000:00:1e.0:   bridge window [mem 0xf9000000-0xf9ffffff 64bit pref]
[    0.308654] pci 0000:00:1e.0:   bridge window [io  0x0000-0x0cf7] (subtractive decode)
[    0.308740] pci 0000:00:1e.0:   bridge window [io  0x0d00-0xffff] (subtractive decode)
[    0.308827] pci 0000:00:1e.0:   bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
[    0.308914] pci 0000:00:1e.0:   bridge window [mem 0x000d0000-0x000dffff] (subtractive decode)
[    0.309001] pci 0000:00:1e.0:   bridge window [mem 0xf0000000-0xfed8ffff] (subtractive decode)
[    0.309119] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    0.309447] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
[    0.309603] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P4._PRT]
[    0.309698] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P5._PRT]
[    0.309801] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE1._PRT]
[    0.309897] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE3._PRT]
[    0.309994] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE5._PRT]
[    0.310090] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE7._PRT]
[    0.310186] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE9._PRT]
[    0.310284]  pci0000:00: Requesting ACPI _OSC control (0x1d)
[    0.310351]  pci0000:00: ACPI _OSC request failed (AE_NOT_FOUND), returned control mask: 0x1d
[    0.310437] ACPI _OSC control for PCIe not granted, disabling ASPM
[    0.326052] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 6 7 10 *11 12 14 15)
[    0.326561] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 6 7 *10 11 12 14 15)
[    0.327067] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 6 7 10 11 12 14 *15)
[    0.327573] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 6 7 10 11 12 *14 15)
[    0.328084] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
[    0.328679] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 6 *7 10 11 12 14 15)
[    0.329185] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
[    0.329782] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 *6 7 10 11 12 14 15)
[    0.330333] vgaarb: device added: PCI:0000:08:04.0,decodes=io+mem,owns=io+mem,locks=none
[    0.330419] vgaarb: loaded
[    0.330477] vgaarb: bridge control possible 0000:08:04.0
[    0.330578] wmi: Mapper loaded
[    0.330637] PCI: Using ACPI for IRQ routing
[    0.337238] PCI: Discovered peer bus ff
[    0.337300] PCI: root bus ff: using default resources
[    0.337384] PCI host bridge to bus 0000:ff
[    0.337447] pci_bus 0000:ff: root bus resource [io  0x0000-0xffff]
[    0.337514] pci_bus 0000:ff: root bus resource [mem 0x00000000-0xffffffffff]
[    0.337586] pci 0000:ff:00.0: [8086:2c40] type 00 class 0x060000
[    0.337671] pci 0000:ff:00.1: [8086:2c01] type 00 class 0x060000
[    0.337757] pci 0000:ff:02.0: [8086:2c10] type 00 class 0x060000
[    0.337841] pci 0000:ff:02.1: [8086:2c11] type 00 class 0x060000
[    0.337925] pci 0000:ff:02.4: [8086:2c14] type 00 class 0x060000
[    0.338007] pci 0000:ff:02.5: [8086:2c15] type 00 class 0x060000
[    0.338091] pci 0000:ff:03.0: [8086:2c18] type 00 class 0x060000
[    0.338173] pci 0000:ff:03.1: [8086:2c19] type 00 class 0x060000
[    0.338256] pci 0000:ff:03.2: [8086:2c1a] type 00 class 0x060000
[    0.338339] pci 0000:ff:03.4: [8086:2c1c] type 00 class 0x060000
[    0.338422] pci 0000:ff:04.0: [8086:2c20] type 00 class 0x060000
[    0.338506] pci 0000:ff:04.1: [8086:2c21] type 00 class 0x060000
[    0.338589] pci 0000:ff:04.2: [8086:2c22] type 00 class 0x060000
[    0.338672] pci 0000:ff:04.3: [8086:2c23] type 00 class 0x060000
[    0.338756] pci 0000:ff:05.0: [8086:2c28] type 00 class 0x060000
[    0.338839] pci 0000:ff:05.1: [8086:2c29] type 00 class 0x060000
[    0.338921] pci 0000:ff:05.2: [8086:2c2a] type 00 class 0x060000
[    0.339004] pci 0000:ff:05.3: [8086:2c2b] type 00 class 0x060000
[    0.339089] pci 0000:ff:06.0: [8086:2c30] type 00 class 0x060000
[    0.339173] pci 0000:ff:06.1: [8086:2c31] type 00 class 0x060000
[    0.339255] pci 0000:ff:06.2: [8086:2c32] type 00 class 0x060000
[    0.339338] pci 0000:ff:06.3: [8086:2c33] type 00 class 0x060000
[    0.339826] PCI: pci_cache_line_size set to 64 bytes
[    0.340027] reserve RAM buffer: 0000000000099800 - 000000000009ffff 
[    0.340079] reserve RAM buffer: 00000000df780000 - 00000000dfffffff 
[    0.340344] HPET: 4 timers in total, 0 timers will be used for per-cpu timer
[    0.340468] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
[    0.340715] hpet0: 4 comparators, 64-bit 14.318180 MHz counter
[    0.348299] Switching to clocksource hpet
[    0.349851] pnp: PnP ACPI init
[    0.349921] ACPI: bus type pnp registered
[    0.350076] pnp 00:00: [bus 00-ff]
[    0.350137] pnp 00:00: [io  0x0cf8-0x0cff]
[    0.350199] pnp 00:00: [io  0x0000-0x0cf7 window]
[    0.350262] pnp 00:00: [io  0x0d00-0xffff window]
[    0.350326] pnp 00:00: [mem 0x000a0000-0x000bffff window]
[    0.350390] pnp 00:00: [mem 0x000d0000-0x000dffff window]
[    0.350456] pnp 00:00: [mem 0xe0000000-0xdfffffff window disabled]
[    0.350522] pnp 00:00: [mem 0xf0000000-0xfed8ffff window]
[    0.350626] pnp 00:00: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active)
[    0.350719] pnp 00:01: [mem 0xfbf00000-0xfbffffff]
[    0.350782] pnp 00:01: [mem 0xfc000000-0xfcffffff]
[    0.350846] pnp 00:01: [mem 0xfd000000-0xfdffffff]
[    0.350909] pnp 00:01: [mem 0xfe000000-0xfebfffff]
[    0.350972] pnp 00:01: [mem 0xfec8a000-0xfec8afff]
[    0.351036] pnp 00:01: [mem 0xfed10000-0xfed10fff]
[    0.351150] system 00:01: [mem 0xfbf00000-0xfbffffff] has been reserved
[    0.351219] system 00:01: [mem 0xfc000000-0xfcffffff] has been reserved
[    0.351287] system 00:01: [mem 0xfd000000-0xfdffffff] has been reserved
[    0.351355] system 00:01: [mem 0xfe000000-0xfebfffff] has been reserved
[    0.351423] system 00:01: [mem 0xfec8a000-0xfec8afff] has been reserved
[    0.351491] system 00:01: [mem 0xfed10000-0xfed10fff] has been reserved
[    0.351559] system 00:01: Plug and Play ACPI device, IDs PNP0c01 (active)
[    0.351655] pnp 00:02: [dma 4]
[    0.351724] pnp 00:02: [io  0x0000-0x000f]
[    0.351786] pnp 00:02: [io  0x0081-0x0083]
[    0.351848] pnp 00:02: [io  0x0087]
[    0.351908] pnp 00:02: [io  0x0089-0x008b]
[    0.351969] pnp 00:02: [io  0x008f]
[    0.352029] pnp 00:02: [io  0x00c0-0x00df]
[    0.352113] pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
[    0.352189] pnp 00:03: [io  0x0070-0x0071]
[    0.352258] pnp 00:03: [irq 8]
[    0.352336] pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.352416] pnp 00:04: [io  0x0061]
[    0.352495] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
[    0.352569] pnp 00:05: [io  0x00f0-0x00ff]
[    0.352636] pnp 00:05: [irq 13]
[    0.352716] pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active)
[    0.353025] pnp 00:06: [io  0x03f8-0x03ff]
[    0.353091] pnp 00:06: [irq 4]
[    0.353150] pnp 00:06: [dma 0 disabled]
[    0.353273] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.353574] pnp 00:07: [io  0x02f8-0x02ff]
[    0.353640] pnp 00:07: [irq 3]
[    0.353699] pnp 00:07: [dma 0 disabled]
[    0.353846] pnp 00:07: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.354321] pnp 00:08: [io  0x03e8-0x03ef]
[    0.354387] pnp 00:08: [irq 5]
[    0.354447] pnp 00:08: [dma 0 disabled]
[    0.354570] pnp 00:08: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.354689] pnp 00:09: [io  0x164e-0x164f]
[    0.354751] pnp 00:09: [io  0x0000-0xffffffffffffffff disabled]
[    0.354869] system 00:09: [io  0x164e-0x164f] has been reserved
[    0.354937] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.355121] pnp 00:0a: [io  0x0000-0xffffffffffffffff disabled]
[    0.355187] pnp 00:0a: [io  0x0a00-0x0a0f]
[    0.355249] pnp 00:0a: [io  0x0ca0-0x0caf]
[    0.355360] system 00:0a: [io  0x0a00-0x0a0f] has been reserved
[    0.355427] system 00:0a: [io  0x0ca0-0x0caf] has been reserved
[    0.355494] system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.355673] pnp 00:0b: [io  0x0010-0x001f]
[    0.355735] pnp 00:0b: [io  0x0022-0x003f]
[    0.355797] pnp 00:0b: [io  0x0044-0x005f]
[    0.361321] pnp 00:0b: [io  0x0062-0x0063]
[    0.361382] pnp 00:0b: [io  0x0065-0x006f]
[    0.361444] pnp 00:0b: [io  0x0072-0x007f]
[    0.361506] pnp 00:0b: [io  0x0080]
[    0.361566] pnp 00:0b: [io  0x0084-0x0086]
[    0.361628] pnp 00:0b: [io  0x0088]
[    0.361689] pnp 00:0b: [io  0x008c-0x008e]
[    0.361751] pnp 00:0b: [io  0x0090-0x009f]
[    0.361812] pnp 00:0b: [io  0x00a2-0x00bf]
[    0.361874] pnp 00:0b: [io  0x00e0-0x00ef]
[    0.361936] pnp 00:0b: [io  0x04d0-0x04d1]
[    0.361997] pnp 00:0b: [io  0x0800-0x087f]
[    0.362059] pnp 00:0b: [io  0x0000-0xffffffffffffffff disabled]
[    0.362125] pnp 00:0b: [io  0x0500-0x057f]
[    0.362186] pnp 00:0b: [mem 0xfed1c000-0xfed1ffff]
[    0.362250] pnp 00:0b: [mem 0xfed20000-0xfed3ffff]
[    0.362313] pnp 00:0b: [mem 0xfed40000-0xfed8ffff]
[    0.362448] system 00:0b: [io  0x04d0-0x04d1] has been reserved
[    0.362515] system 00:0b: [io  0x0800-0x087f] has been reserved
[    0.362581] system 00:0b: [io  0x0500-0x057f] has been reserved
[    0.362648] system 00:0b: [mem 0xfed1c000-0xfed1ffff] has been reserved
[    0.362716] system 00:0b: [mem 0xfed20000-0xfed3ffff] has been reserved
[    0.362785] system 00:0b: [mem 0xfed40000-0xfed8ffff] has been reserved
[    0.362853] system 00:0b: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.362975] pnp 00:0c: [mem 0xfed00000-0xfed003ff]
[    0.363064] pnp 00:0c: Plug and Play ACPI device, IDs PNP0103 (active)
[    0.363204] pnp 00:0d: [io  0x0060]
[    0.363265] pnp 00:0d: [io  0x0064]
[    0.363326] pnp 00:0d: [mem 0xfec00000-0xfec00fff]
[    0.363389] pnp 00:0d: [mem 0xfee00000-0xfee00fff]
[    0.363503] system 00:0d: [mem 0xfec00000-0xfec00fff] could not be reserved
[    0.363572] system 00:0d: [mem 0xfee00000-0xfee00fff] has been reserved
[    0.363641] system 00:0d: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.363755] pnp 00:0e: [mem 0xe0000000-0xefffffff]
[    0.363871] system 00:0e: [mem 0xe0000000-0xefffffff] has been reserved
[    0.363939] system 00:0e: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.364172] pnp 00:0f: [mem 0x00000000-0x0009ffff]
[    0.364236] pnp 00:0f: [mem 0x000c0000-0x000cffff]
[    0.364302] pnp 00:0f: [mem 0x000e0000-0x000fffff]
[    0.364373] pnp 00:0f: [mem 0x00100000-0xdfffffff]
[    0.364436] pnp 00:0f: [mem 0xfed90000-0xffffffff]
[    0.364562] system 00:0f: [mem 0x00000000-0x0009ffff] could not be reserved
[    0.364631] system 00:0f: [mem 0x000c0000-0x000cffff] could not be reserved
[    0.364700] system 00:0f: [mem 0x000e0000-0x000fffff] could not be reserved
[    0.364769] system 00:0f: [mem 0x00100000-0xdfffffff] could not be reserved
[    0.364838] system 00:0f: [mem 0xfed90000-0xffffffff] could not be reserved
[    0.364908] system 00:0f: Plug and Play ACPI device, IDs PNP0c01 (active)
[    0.365099] pnp: PnP ACPI: found 16 devices
[    0.365160] ACPI: ACPI bus type pnp unregistered
[    0.371331] pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 06-06] add_size 200000
[    0.371429] pci 0000:00:1c.1: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 07-07] add_size 200000
[    0.371535] pci 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[    0.371626] pci 0000:00:1c.1: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[    0.371717] pci 0000:00:1c.0: BAR 15: assigned [mem 0xf0000000-0xf01fffff 64bit pref]
[    0.371804] pci 0000:00:1c.1: BAR 15: assigned [mem 0xf0200000-0xf03fffff 64bit pref]
[    0.371891] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    0.371962] pci 0000:00:03.0: PCI bridge to [bus 02-02]
[    0.372033] pci 0000:00:05.0: PCI bridge to [bus 03-03]
[    0.372104] pci 0000:00:07.0: PCI bridge to [bus 04-04]
[    0.372169] pci 0000:00:07.0:   bridge window [io  0xb000-0xbfff]
[    0.372238] pci 0000:00:07.0:   bridge window [mem 0xfb800000-0xfb8fffff]
[    0.372316] pci 0000:00:09.0: PCI bridge to [bus 05-05]
[    0.372381] pci 0000:00:09.0:   bridge window [io  0xc000-0xcfff]
[    0.372450] pci 0000:00:09.0:   bridge window [mem 0xfb900000-0xfbbfffff]
[    0.372522] pci 0000:00:1c.0: PCI bridge to [bus 06-06]
[    0.372587] pci 0000:00:1c.0:   bridge window [io  0xd000-0xdfff]
[    0.372656] pci 0000:00:1c.0:   bridge window [mem 0xfbc00000-0xfbcfffff]
[    0.372726] pci 0000:00:1c.0:   bridge window [mem 0xf0000000-0xf01fffff 64bit pref]
[    0.372815] pci 0000:00:1c.1: PCI bridge to [bus 07-07]
[    0.372880] pci 0000:00:1c.1:   bridge window [io  0xe000-0xefff]
[    0.372949] pci 0000:00:1c.1:   bridge window [mem 0xfbd00000-0xfbdfffff]
[    0.373018] pci 0000:00:1c.1:   bridge window [mem 0xf0200000-0xf03fffff 64bit pref]
[    0.373107] pci 0000:00:1e.0: PCI bridge to [bus 08-08]
[    0.373174] pci 0000:00:1e.0:   bridge window [mem 0xfaf00000-0xfb7fffff]
[    0.373243] pci 0000:00:1e.0:   bridge window [mem 0xf9000000-0xf9ffffff 64bit pref]
[    0.373373] pci 0000:00:1e.0: setting latency timer to 64
[    0.373440] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
[    0.373505] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff]
[    0.373570] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
[    0.373637] pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000dffff]
[    0.373703] pci_bus 0000:00: resource 8 [mem 0xf0000000-0xfed8ffff]
[    0.373770] pci_bus 0000:04: resource 0 [io  0xb000-0xbfff]
[    0.373836] pci_bus 0000:04: resource 1 [mem 0xfb800000-0xfb8fffff]
[    0.373902] pci_bus 0000:05: resource 0 [io  0xc000-0xcfff]
[    0.373967] pci_bus 0000:05: resource 1 [mem 0xfb900000-0xfbbfffff]
[    0.374034] pci_bus 0000:06: resource 0 [io  0xd000-0xdfff]
[    0.374099] pci_bus 0000:06: resource 1 [mem 0xfbc00000-0xfbcfffff]
[    0.374166] pci_bus 0000:06: resource 2 [mem 0xf0000000-0xf01fffff 64bit pref]
[    0.374250] pci_bus 0000:07: resource 0 [io  0xe000-0xefff]
[    0.374315] pci_bus 0000:07: resource 1 [mem 0xfbd00000-0xfbdfffff]
[    0.374381] pci_bus 0000:07: resource 2 [mem 0xf0200000-0xf03fffff 64bit pref]
[    0.374465] pci_bus 0000:08: resource 1 [mem 0xfaf00000-0xfb7fffff]
[    0.374532] pci_bus 0000:08: resource 2 [mem 0xf9000000-0xf9ffffff 64bit pref]
[    0.374616] pci_bus 0000:08: resource 4 [io  0x0000-0x0cf7]
[    0.374681] pci_bus 0000:08: resource 5 [io  0x0d00-0xffff]
[    0.374746] pci_bus 0000:08: resource 6 [mem 0x000a0000-0x000bffff]
[    0.374812] pci_bus 0000:08: resource 7 [mem 0x000d0000-0x000dffff]
[    0.374879] pci_bus 0000:08: resource 8 [mem 0xf0000000-0xfed8ffff]
[    0.374946] pci_bus 0000:ff: resource 4 [io  0x0000-0xffff]
[    0.375011] pci_bus 0000:ff: resource 5 [mem 0x00000000-0xffffffffff]
[    0.375121] NET: Registered protocol family 2
[    0.375474] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.376721] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    0.378341] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.378588] TCP: Hash tables configured (established 524288 bind 65536)
[    0.378656] TCP: reno registered
[    0.378750] UDP hash table entries: 16384 (order: 7, 524288 bytes)
[    0.378951] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes)
[    0.379209] NET: Registered protocol family 1
[    0.379539] PCI: CLS mismatch (256 != 64), using 64 bytes
[    0.379611] pci 0000:08:04.0: Boot video device
[    0.379740] Unpacking initramfs...
[    2.839202] Freeing initrd memory: 106464k freed
[    2.858263] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    2.858335] Placing 64MB software IO TLB between ffff8800db766000 - ffff8800df766000
[    2.858420] software IO TLB at phys 0xdb766000 - 0xdf766000
[    2.858996] audit: initializing netlink socket (disabled)
[    2.859070] type=2000 audit(1339436723.740:1): initialized
[    2.873774] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    2.875309] VFS: Disk quotas dquot_6.5.2
[    2.875415] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.875569] msgmni has been set to 32768
[    2.875789] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    2.875874] io scheduler noop registered
[    2.875935] io scheduler deadline registered
[    2.876047] io scheduler cfq registered (default)
[    2.876426] pcieport 0000:00:1c.0: irq 40 for MSI/MSI-X
[    2.876584] pcieport 0000:00:1c.1: irq 41 for MSI/MSI-X
[    2.876811] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.897222] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.917642] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    2.938071] serial8250: ttyS2 at I/O 0x3e8 (irq = 4) is a 16550A
[    2.958615] 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.979064] 00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    2.999513] 00:08: ttyS2 at I/O 0x3e8 (irq = 5) is a 16550A
[    2.999843] Non-volatile memory driver v1.3
[    2.999906] Linux agpgart interface v0.103
[    3.000104] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    3.002606] serio: i8042 KBD port at 0x60,0x64 irq 1
[    3.002673] serio: i8042 AUX port at 0x60,0x64 irq 12
[    3.002825] mousedev: PS/2 mouse device common for all mice
[    3.002928] rtc_cmos 00:03: RTC can wake from S4
[    3.003078] rtc_cmos 00:03: rtc core: registered rtc_cmos as rtc0
[    3.003169] rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[    3.003244] cpuidle: using governor ladder
[    3.003306] cpuidle: using governor menu
[    3.003480] fujitsu_laptop: driver 0.6.0 successfully loaded
[    3.003633] TCP: cubic registered
[    3.003788] NET: Registered protocol family 10
[    3.004070] Mobile IPv6
[    3.004129] NET: Registered protocol family 17
[    3.004193] Registering the dns_resolver key type
[    3.004404] PM: Hibernation image not present or could not be loaded.
[    3.004483] registered taskstats version 1
[    3.005111] rtc_cmos 00:03: setting system clock to 2012-06-11 17:45:24 UTC (1339436724)
[    3.005234] Initializing network drop monitor service
[    3.119920] Freeing unused kernel memory: 576k freed
[    3.120086] Write protecting the kernel read-only data: 6144k
[    3.122503] Freeing unused kernel memory: 648k freed
[    3.125522] Freeing unused kernel memory: 808k freed
[    3.147921] udev[54]: starting version 164
[    3.169654] SCSI subsystem initialized
[    3.170326] e1000e: Intel(R) PRO/1000 Network Driver - 1.9.5-k
[    3.170394] e1000e: Copyright(c) 1999 - 2012 Intel Corporation.
[    3.194191] megasas: 00.00.06.14-rc1 Fri. Jan. 6 17:00:00 PDT 2012
[    3.194278] megasas: 0x1000:0x005b:0x1000:0x9265: bus 4:slot 0:func 0
[    3.194812] e1000e 0000:06:00.0: Disabling ASPM L0s 
[    3.194976] libata version 3.00 loaded.
[    3.195052] e1000e 0000:06:00.0: (unregistered net_device): Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    3.195237] e1000e 0000:06:00.0: irq 42 for MSI/MSI-X
[    3.195308] e1000e 0000:06:00.0: irq 43 for MSI/MSI-X
[    3.195377] e1000e 0000:06:00.0: irq 44 for MSI/MSI-X
[    3.195598] megasas: FW now in Ready state
[    3.195716] megaraid_sas 0000:04:00.0: irq 45 for MSI/MSI-X
[    3.195787] megaraid_sas 0000:04:00.0: irq 46 for MSI/MSI-X
[    3.195859] megaraid_sas 0000:04:00.0: irq 47 for MSI/MSI-X
[    3.195889] usbcore: registered new interface driver usbfs
[    3.195909] usbcore: registered new interface driver hub
[    3.195933] usbcore: registered new device driver usb
[    3.196111] ahci 0000:00:1f.2: version 3.0
[    3.196181] megaraid_sas 0000:04:00.0: irq 48 for MSI/MSI-X
[    3.196205] ahci 0000:00:1f.2: irq 49 for MSI/MSI-X
[    3.196243] ahci: SSS flag set, parallel bus scan disabled
[    3.196287] ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
[    3.196290] ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ccc ems sxs 
[    3.196294] ahci 0000:00:1f.2: setting latency timer to 64
[    3.196787] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.204138] Fusion MPT base driver 3.04.20
[    3.204204] Copyright (c) 1999-2008 LSI Corporation
[    3.205441] Fusion MPT SAS Host driver 3.04.20
[    3.205699] mptbase: ioc0: Initiating bringup
[    3.218406] megasas:IOC Init cmd success
[    3.234781] scsi1 : ahci
[    3.234907] scsi2 : ahci
[    3.235013] scsi3 : ahci
[    3.235120] scsi4 : ahci
[    3.235228] scsi5 : ahci
[    3.235334] scsi6 : ahci
[    3.235526] ata1: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6100 irq 49
[    3.235612] ata2: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6180 irq 49
[    3.235698] ata3: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6200 irq 49
[    3.235783] ata4: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6280 irq 49
[    3.235868] ata5: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6300 irq 49
[    3.235953] ata6: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6380 irq 49
[    3.236081] ehci_hcd 0000:00:1a.7: setting latency timer to 64
[    3.236150] ehci_hcd 0000:00:1a.7: EHCI Host Controller
[    3.236235] ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1
[    3.236345] ehci_hcd 0000:00:1a.7: debug port 1
[    3.240289] ehci_hcd 0000:00:1a.7: cache line size of 64 is not supported
[    3.240369] ehci_hcd 0000:00:1a.7: irq 18, io mem 0xfbeda000
[    3.242351] megasas: INIT adapter done
[    3.250318] ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00
[    3.250408] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    3.250476] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.250560] usb usb1: Product: EHCI Host Controller
[    3.250623] usb usb1: Manufacturer: Linux 3.4.1-custom ehci_hcd
[    3.250689] usb usb1: SerialNumber: 0000:00:1a.7
[    3.250843] hub 1-0:1.0: USB hub found
[    3.250906] hub 1-0:1.0: 6 ports detected
[    3.251069] ehci_hcd 0000:00:1d.7: setting latency timer to 64
[    3.251137] ehci_hcd 0000:00:1d.7: EHCI Host Controller
[    3.251206] ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2
[    3.251312] ehci_hcd 0000:00:1d.7: debug port 1
[    3.255242] ehci_hcd 0000:00:1d.7: cache line size of 64 is not supported
[    3.255321] ehci_hcd 0000:00:1d.7: irq 23, io mem 0xfbed8000
[    3.266288] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
[    3.266371] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[    3.266439] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.266524] usb usb2: Product: EHCI Host Controller
[    3.266587] usb usb2: Manufacturer: Linux 3.4.1-custom ehci_hcd
[    3.266653] usb usb2: SerialNumber: 0000:00:1d.7
[    3.266804] hub 2-0:1.0: USB hub found
[    3.266868] hub 2-0:1.0: 6 ports detected
[    3.279394] uhci_hcd: USB Universal Host Controller Interface driver
[    3.279484] uhci_hcd 0000:00:1a.0: setting latency timer to 64
[    3.279552] uhci_hcd 0000:00:1a.0: UHCI Host Controller
[    3.279622] uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
[    3.279737] uhci_hcd 0000:00:1a.0: irq 16, io base 0x0000ac00
[    3.279829] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[    3.279898] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.279982] usb usb3: Product: UHCI Host Controller
[    3.280045] usb usb3: Manufacturer: Linux 3.4.1-custom uhci_hcd
[    3.280111] usb usb3: SerialNumber: 0000:00:1a.0
[    3.280253] hub 3-0:1.0: USB hub found
[    3.280316] hub 3-0:1.0: 2 ports detected
[    3.280448] uhci_hcd 0000:00:1a.1: setting latency timer to 64
[    3.280515] uhci_hcd 0000:00:1a.1: UHCI Host Controller
[    3.280585] uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
[    3.280698] uhci_hcd 0000:00:1a.1: irq 21, io base 0x0000a880
[    3.280791] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[    3.280859] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.280943] usb usb4: Product: UHCI Host Controller
[    3.281006] usb usb4: Manufacturer: Linux 3.4.1-custom uhci_hcd
[    3.281072] usb usb4: SerialNumber: 0000:00:1a.1
[    3.281209] hub 4-0:1.0: USB hub found
[    3.281272] hub 4-0:1.0: 2 ports detected
[    3.281401] uhci_hcd 0000:00:1a.2: setting latency timer to 64
[    3.281468] uhci_hcd 0000:00:1a.2: UHCI Host Controller
[    3.281543] uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5
[    3.281657] uhci_hcd 0000:00:1a.2: irq 19, io base 0x0000a800
[    3.281747] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
[    3.281816] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.281900] usb usb5: Product: UHCI Host Controller
[    3.281963] usb usb5: Manufacturer: Linux 3.4.1-custom uhci_hcd
[    3.282029] usb usb5: SerialNumber: 0000:00:1a.2
[    3.282166] hub 5-0:1.0: USB hub found
[    3.282229] hub 5-0:1.0: 2 ports detected
[    3.282366] uhci_hcd 0000:00:1d.0: setting latency timer to 64
[    3.282433] uhci_hcd 0000:00:1d.0: UHCI Host Controller
[    3.282502] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6
[    3.282608] uhci_hcd 0000:00:1d.0: irq 23, io base 0x0000a480
[    3.282698] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
[    3.282766] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.282849] usb usb6: Product: UHCI Host Controller
[    3.282913] usb usb6: Manufacturer: Linux 3.4.1-custom uhci_hcd
[    3.282978] usb usb6: SerialNumber: 0000:00:1d.0
[    3.283115] hub 6-0:1.0: USB hub found
[    3.283178] hub 6-0:1.0: 2 ports detected
[    3.283309] uhci_hcd 0000:00:1d.1: setting latency timer to 64
[    3.283376] uhci_hcd 0000:00:1d.1: UHCI Host Controller
[    3.283445] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7
[    3.283551] uhci_hcd 0000:00:1d.1: irq 19, io base 0x0000a400
[    3.283642] usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
[    3.283711] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.283794] usb usb7: Product: UHCI Host Controller
[    3.283857] usb usb7: Manufacturer: Linux 3.4.1-custom uhci_hcd
[    3.283923] usb usb7: SerialNumber: 0000:00:1d.1
[    3.284061] hub 7-0:1.0: USB hub found
[    3.284124] hub 7-0:1.0: 2 ports detected
[    3.284252] uhci_hcd 0000:00:1d.2: setting latency timer to 64
[    3.284319] uhci_hcd 0000:00:1d.2: UHCI Host Controller
[    3.284388] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
[    3.284494] uhci_hcd 0000:00:1d.2: irq 18, io base 0x0000a080
[    3.284586] usb usb8: New USB device found, idVendor=1d6b, idProduct=0001
[    3.284654] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.284742] usb usb8: Product: UHCI Host Controller
[    3.284805] usb usb8: Manufacturer: Linux 3.4.1-custom uhci_hcd
[    3.284871] usb usb8: SerialNumber: 0000:00:1d.2
[    3.285009] hub 8-0:1.0: USB hub found
[    3.285072] hub 8-0:1.0: 2 ports detected
[    3.301389] e1000e 0000:06:00.0: eth0: (PCI Express:2.5GT/s:Width x1) 00:25:90:22:b5:de
[    3.301477] e1000e 0000:06:00.0: eth0: Intel(R) PRO/1000 Network Connection
[    3.301630] e1000e 0000:06:00.0: eth0: MAC: 3, PHY: 8, PBA No: 0101FF-0FF
[    3.301709] e1000e 0000:07:00.0: Disabling ASPM L0s 
[    3.301901] e1000e 0000:07:00.0: (unregistered net_device): Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    3.302080] e1000e 0000:07:00.0: irq 50 for MSI/MSI-X
[    3.302149] e1000e 0000:07:00.0: irq 51 for MSI/MSI-X
[    3.302224] e1000e 0000:07:00.0: irq 52 for MSI/MSI-X
[    3.314226] scsi0 : LSI SAS based MegaRAID driver
[    3.314488] scsi 0:0:0:0: Direct-Access     LSI      MR9265-8i        3.13 PQ: 0 ANSI: 5
[    3.314678] scsi 0:0:1:0: Direct-Access     LSI      MR9265-8i        3.13 PQ: 0 ANSI: 5
[    3.325627] scsi 0:2:0:0: Direct-Access     LSI      MR9265-8i        3.13 PQ: 0 ANSI: 5
[    3.325819] scsi 0:2:1:0: Direct-Access     LSI      MR9265-8i        3.13 PQ: 0 ANSI: 5
[    3.418877] e1000e 0000:07:00.0: eth1: (PCI Express:2.5GT/s:Width x1) 00:25:90:22:b5:df
[    3.418966] e1000e 0000:07:00.0: eth1: Intel(R) PRO/1000 Network Connection
[    3.419118] e1000e 0000:07:00.0: eth1: MAC: 3, PHY: 8, PBA No: 0101FF-0FF
[    3.557679] ata1: SATA link down (SStatus 0 SControl 300)
[    3.765237] ioc0: LSISAS1068E B3: Capabilities={Initiator}
[    3.853071] Refined TSC clocksource calibration: 2133.333 MHz.
[    3.853146] Switching to clocksource tsc
[    3.877005] ata2: SATA link down (SStatus 0 SControl 300)
[    3.972800] usb 5-2: new full-speed USB device number 2 using uhci_hcd
[    4.147943] usb 5-2: New USB device found, idVendor=046b, idProduct=ff10
[    4.148019] usb 5-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    4.148087] usb 5-2: Product: Virtual Keyboard and Mouse
[    4.148152] usb 5-2: Manufacturer: American Megatrends Inc.
[    4.148217] usb 5-2: SerialNumber: serial
[    4.196342] ata3: SATA link down (SStatus 0 SControl 300)
[    4.515674] ata4: SATA link down (SStatus 0 SControl 300)
[    4.835005] ata5: SATA link down (SStatus 0 SControl 300)
[    5.154339] ata6: SATA link down (SStatus 0 SControl 300)
[    5.157352] sd 0:2:0:0: [sda] 85930803200 512-byte logical blocks: (43.9 TB/40.0 TiB)
[    5.157451] sd 0:2:1:0: [sdb] 85930803200 512-byte logical blocks: (43.9 TB/40.0 TiB)
[    5.157505] sd 0:2:0:0: [sda] Write Protect is off
[    5.157508] sd 0:2:0:0: [sda] Mode Sense: 1f 00 00 08
[    5.157540] sd 0:2:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    5.157800] sd 0:2:1:0: [sdb] Write Protect is off
[    5.157866] sd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08
[    5.157979] sd 0:2:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    5.158200]  sda: unknown partition table
[    5.158440] sd 0:2:0:0: [sda] Attached SCSI disk
[    5.158651]  sdb: unknown partition table
[    5.158876] sd 0:2:1:0: [sdb] Attached SCSI disk
[    5.165993] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.2/usb5/5-2/5-2:1.0/input/input0
[    5.166135] generic-usb 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-2/input0
[    5.180920] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.2/usb5/5-2/5-2:1.1/input/input1
[    5.181070] generic-usb 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-2/input1
[    5.181197] usbcore: registered new interface driver usbhid
[    5.181260] usbhid: USB HID core driver
[   16.975522] scsi7 : ioc0: LSISAS1068E B3, FwRev=011a0000h, Ports=1, MaxQ=478, IRQ=16
[   17.215072] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   17.294840] ADDRCONF(NETDEV_UP): eth1: link is not ready
[   19.924507] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   19.926104] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   20.301021] Uniform Multi-Platform E-IDE driver
[   20.301521] ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
[   20.301618] Probing IDE interface ide0...
[   20.861594] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
[   20.861721] Probing IDE interface ide1...
[   21.420416] ide1 at 0x170-0x177,0x376 on irq 15
[   30.437650] eth0: no IPv6 routers present
[   56.451263] aufs 3.4-20120521
[   56.461766] loop: module loaded
[   56.470713] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[   57.358821] udev[994]: starting version 164
[   57.393390] ACPI: Requesting acpi_cpufreq
[   57.394174] Monitor-Mwait will be used to enter C-1 state
[   57.394336] Monitor-Mwait will be used to enter C-2 state
[   57.394618] Monitor-Mwait will be used to enter C-3 state
[   57.394699] ACPI: acpi_idle registered with cpuidle
[   57.396406] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input2
[   57.396498] ACPI: Power Button [PWRB]
[   57.400539] dca service started, version 1.12.1
[   57.400888] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
[   57.400977] ACPI: Power Button [PWRF]
[   57.422907] ioatdma: Intel(R) QuickData Technology Driver 4.00
[   57.423126] ioatdma 0000:00:16.0: irq 53 for MSI/MSI-X
[   57.423520] ioatdma 0000:00:16.1: irq 54 for MSI/MSI-X
[   57.423969] ioatdma 0000:00:16.2: irq 55 for MSI/MSI-X
[   57.424329] ioatdma 0000:00:16.3: irq 56 for MSI/MSI-X
[   57.424740] ioatdma 0000:00:16.4: irq 57 for MSI/MSI-X
[   57.425153] ioatdma 0000:00:16.5: irq 58 for MSI/MSI-X
[   57.425688] ioatdma 0000:00:16.6: irq 59 for MSI/MSI-X
[   57.426030] ioatdma 0000:00:16.7: irq 60 for MSI/MSI-X
[   57.471440] microcode: CPU0 sig=0x106a5, pf=0x1, revision=0x11
[   57.521149] microcode: CPU1 sig=0x106a5, pf=0x1, revision=0x11
[   57.524137] microcode: CPU2 sig=0x106a5, pf=0x1, revision=0x11
[   57.527186] microcode: CPU3 sig=0x106a5, pf=0x1, revision=0x11
[   57.530154] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[   58.757806] 8021q: 802.1Q VLAN Support v1.8
[   58.757886] 8021q: adding VLAN 0 to HW filter on device eth0
[   58.757966] 8021q: adding VLAN 0 to HW filter on device eth1
[   58.776430] myri10ge: Version 1.5.3-1.534
[   59.369974] fuse init (API version 7.18)

[-- Attachment #3: lsmod.txt --]
[-- Type: text/plain, Size: 2092 bytes --]

Module                  Size  Used by
xfs                   734716  2 
cpufreq_userspace       1920  0 
cpufreq_stats           2698  0 
cpufreq_powersave        926  0 
cpufreq_conservative     4507  0 
fuse                   55799  1 
myri10ge               34069  0 
inet_lro                4016  1 myri10ge
8021q                  15141  0 
garp                    4910  1 8021q
stp                     1448  1 garp
llc                     3295  2 garp,stp
coretemp                5185  0 
crc32c_intel            1752  0 
tpm_tis                 7917  0 
joydev                  9026  0 
microcode              21193  0 
evdev                   8600  0 
tpm                    11395  1 tpm_tis
tpm_bios                4447  1 tpm
serio_raw               4025  0 
i2c_i801                7430  0 
i2c_core               16654  1 i2c_i801
ioatdma                33323  24 
acpi_cpufreq            5165  1 
mperf                   1139  1 acpi_cpufreq
dca                     4392  2 myri10ge,ioatdma
button                  4207  0 
processor              25994  5 acpi_cpufreq
thermal_sys            12408  1 processor
squashfs               20995  1 
loop                   14297  2 
aufs                  141050  47 
exportfs                2983  2 xfs,aufs
ide_generic             1129  0 [permanent]
ide_core               73401  1 ide_generic
usbhid                 30465  0 
hid                    71737  1 usbhid
sd_mod                 30496  2 
crc_t10dif              1284  1 sd_mod
uhci_hcd               18857  0 
mptsas                 41169  0 
mptscsih               24130  1 mptsas
mptbase                71228  2 mptsas,mptscsih
scsi_transport_sas     20692  1 mptsas
ehci_hcd               32682  0 
ahci                   19429  0 
libahci                16642  1 ahci
megaraid_sas           59485  2 
libata                136413  2 ahci,libahci
usbcore               122255  4 usbhid,uhci_hcd,ehci_hcd
e1000e                121715  0 
scsi_mod              161423  6 sd_mod,mptsas,mptscsih,scsi_transport_sas,megaraid_sas,libata
usb_common               890  1 usbcore

[-- Attachment #4: lspci.txt --]
[-- Type: text/plain, Size: 73719 bytes --]

00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 22)
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit-
		Address: 00000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>

00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Super Micro Computer Inc Device f580
	Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit-
		Address: 00000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #4, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet+ LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Kernel driver in use: pcieport

00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Super Micro Computer Inc Device f580
	Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit-
		Address: 00000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #6, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet+ LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Kernel driver in use: pcieport

00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Super Micro Computer Inc Device f580
	Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit-
		Address: 00000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #5, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet+ LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Kernel driver in use: pcieport

00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 0000b000-0000bfff
	Memory behind bridge: fb800000-fb8fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Super Micro Computer Inc Device f580
	Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit-
		Address: 00000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #3, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd+
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Kernel driver in use: pcieport

00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: fb900000-fbbfffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Super Micro Computer Inc Device f580
	Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit-
		Address: 00000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #57, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Kernel driver in use: pcieport

00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 22) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB

00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB

00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB

00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 22) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at fbef8000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma

00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin B routed to IRQ 17
	Region 0: Memory at fbef4000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma

00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at fbef0000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma

00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin D routed to IRQ 19
	Region 0: Memory at fbeec000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma

00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at fbee8000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma

00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin B routed to IRQ 17
	Region 0: Memory at fbee4000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma

00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at fbee0000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma

00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin D routed to IRQ 19
	Region 0: Memory at fbedc000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma

00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 (prog-if 00 [UHCI])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 16
	Region 4: I/O ports at ac00 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 (prog-if 00 [UHCI])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 21
	Region 4: I/O ports at a880 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 (prog-if 00 [UHCI])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin D routed to IRQ 19
	Region 4: I/O ports at a800 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 (prog-if 20 [EHCI])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at fbeda000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ehci_hcd

00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: fbc00000-fbcfffff
	Prefetchable memory behind bridge: 00000000f0000000-00000000f01fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00000  Data: 4069
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device f580
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Root Complex Link
		Desc:	PortNumber=01 ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: pcieport

00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: fbd00000-fbdfffff
	Prefetchable memory behind bridge: 00000000f0200000-00000000f03fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00000  Data: 4071
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device f580
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Root Complex Link
		Desc:	PortNumber=02 ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: pcieport

00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 (prog-if 00 [UHCI])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 23
	Region 4: I/O ports at a480 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 (prog-if 00 [UHCI])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 19
	Region 4: I/O ports at a400 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 (prog-if 00 [UHCI])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin C routed to IRQ 18
	Region 4: I/O ports at a080 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 (prog-if 20 [EHCI])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 23
	Region 0: Memory at fbed8000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ehci_hcd

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Bus: primary=00, secondary=08, subordinate=08, sec-latency=32
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: faf00000-fb7fffff
	Prefetchable memory behind bridge: 00000000f9000000-00000000f9ffffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] Subsystem: Super Micro Computer Inc Device f580

00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller (prog-if 01 [AHCI 1.0])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 49
	Region 0: I/O ports at 9480 [size=8]
	Region 1: I/O ports at a000 [size=4]
	Region 2: I/O ports at 9c00 [size=8]
	Region 3: I/O ports at 9880 [size=4]
	Region 4: I/O ports at 9800 [size=32]
	Region 5: Memory at fbed6000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: [80] MSI: Enable+ Count=1/16 Maskable- 64bit-
		Address: fee00000  Data: 40b1
	Capabilities: [70] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
	Capabilities: [b0] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ahci

00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at fbed4000 (64-bit, non-prefetchable) [size=256]
	Region 4: I/O ports at 0400 [size=32]
	Kernel driver in use: i801_smbus

04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS TB (rev 01)
	Subsystem: LSI Logic / Symbios Logic Device 9265
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: I/O ports at b000 [size=256]
	Region 1: Memory at fb87c000 (64-bit, non-prefetchable) [size=16K]
	Region 3: Memory at fb8c0000 (64-bit, non-prefetchable) [size=256K]
	Expansion ROM at fb880000 [disabled] [size=256K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range BC, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [d0] Vital Product Data
		Unknown small resource type 00, will not decode more.
	Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] MSI-X: Enable+ Count=16 Masked-
		Vector table: BAR=1 offset=00002000
		PBA: BAR=1 offset=00003000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1e0 v1] #19
	Capabilities: [1c0 v1] Power Budgeting <?>
	Capabilities: [190 v1] #16
	Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Kernel driver in use: megaraid_sas

05:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
	Subsystem: Super Micro Computer Inc Device 0005
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: I/O ports at c000 [size=256]
	Region 1: Memory at fb9ec000 (64-bit, non-prefetchable) [size=16K]
	Region 3: Memory at fb9f0000 (64-bit, non-prefetchable) [size=64K]
	Expansion ROM at fba00000 [disabled] [size=2M]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x8, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [98] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=1 offset=00002000
		PBA: BAR=1 offset=00003000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: mptsas

06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Super Micro Computer Inc Device 10d3
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at fbce0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at dc00 [size=32]
	Region 3: Memory at fbcdc000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [e0] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: e1000e

07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Super Micro Computer Inc Device 10d3
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 17
	Region 0: Memory at fbde0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at ec00 [size=32]
	Region 3: Memory at fbddc000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [e0] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: e1000e

08:04.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW WPCM450 (rev 0a) (prog-if 00 [VGA controller])
	Subsystem: Super Micro Computer Inc Device f580
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64 (4000ns min, 8000ns max), Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 15
	Region 0: Memory at f9000000 (32-bit, prefetchable) [size=16M]
	Region 1: Memory at faffc000 (32-bit, non-prefetchable) [size=16K]
	Region 2: Memory at fb000000 (32-bit, non-prefetchable) [size=8M]
	Expansion ROM at <unassigned> [disabled]
	Capabilities: [dc] Power Management version 1
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

ff:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.4 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 1 (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.5 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 1 (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:03.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller RAS Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05)
	Subsystem: Intel Corporation Device 8086
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0


[-- Attachment #5: mount.txt --]
[-- Type: text/plain, Size: 828 bytes --]

aufs on / type aufs (rw)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
ram on /live/image type ramfs (rw,relatime)
tmpfs on /live/cow type tmpfs (rw,noatime,mode=755)
tmpfs on /live type tmpfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
/dev/sda on /news/spool/news/P.01 type xfs (rw,noatime,nodiratime,nobarrier,inode64,logbufs=8,logbsize=256k)
/dev/sdb on /news/spool/news/P.02 type xfs (rw,noatime,nodiratime,nobarrier,inode64,logbufs=8,logbsize=256k)
ramfs on /news/history type ramfs (rw,noatime)

[-- Attachment #6: sysctl.txt --]
[-- Type: text/plain, Size: 27406 bytes --]

abi.vsyscall32 = 1
debug.exception-trace = 1
debug.kprobes-optimization = 1
dev.hpet.max-user-freq = 64
dev.scsi.logging_level = 0
fs.aio-max-nr = 65536
fs.aio-nr = 0
fs.dentry-state = 14649	7194	45	0	0	0
fs.dir-notify-enable = 1
fs.epoll.max_user_watches = 5034106
fs.file-max = 2458026
fs.file-nr = 736	0	2458026
fs.inode-nr = 16836	4187
fs.inode-state = 16836	4187	0	0	0	0	0
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.lease-break-time = 45
fs.leases-enable = 1
fs.mqueue.msg_max = 10
fs.mqueue.msgsize_max = 8192
fs.mqueue.queues_max = 256
fs.nr_open = 1048576
fs.overflowgid = 65534
fs.overflowuid = 65534
fs.pipe-max-size = 1048576
fs.quota.allocated_dquots = 0
fs.quota.cache_hits = 0
fs.quota.drops = 0
fs.quota.free_dquots = 0
fs.quota.lookups = 0
fs.quota.reads = 0
fs.quota.syncs = 0
fs.quota.warnings = 1
fs.quota.writes = 0
fs.suid_dumpable = 0
fs.xfs.age_buffer_centisecs = 1500
fs.xfs.error_level = 3
fs.xfs.filestream_centisecs = 3000
fs.xfs.inherit_noatime = 1
fs.xfs.inherit_nodefrag = 1
fs.xfs.inherit_nodump = 1
fs.xfs.inherit_nosymlinks = 0
fs.xfs.inherit_sync = 1
fs.xfs.irix_sgid_inherit = 0
fs.xfs.irix_symlink_mode = 0
fs.xfs.panic_mask = 0
fs.xfs.rotorstep = 1
fs.xfs.stats_clear = 0
fs.xfs.xfsbufd_centisecs = 100
fs.xfs.xfssyncd_centisecs = 3000
kernel.acct = 4	2	30
kernel.acpi_video_flags = 0
kernel.auto_msgmni = 1
kernel.blk_iopoll = 1
kernel.bootloader_type = 50
kernel.bootloader_version = 2
kernel.cad_pid = 1
kernel.cap_last_cap = 35
kernel.compat-log = 1
kernel.core_pattern = core
kernel.core_pipe_limit = 0
kernel.core_uses_pid = 0
kernel.ctrl-alt-del = 0
kernel.dmesg_restrict = 0
kernel.domainname = (none)
kernel.ftrace_dump_on_oops = 0
kernel.hostname = sd69
kernel.hotplug = 
kernel.hung_task_check_count = 4194304
kernel.hung_task_panic = 0
kernel.hung_task_timeout_secs = 120
kernel.hung_task_warnings = 0
kernel.io_delay_type = 0
kernel.keys.gc_delay = 300
kernel.keys.maxbytes = 20000
kernel.keys.maxkeys = 200
kernel.keys.root_maxbytes = 20000
kernel.keys.root_maxkeys = 200
kernel.kptr_restrict = 0
kernel.kstack_depth_to_print = 12
kernel.max_lock_depth = 1024
kernel.modprobe = /sbin/modprobe
kernel.modules_disabled = 0
kernel.msgmax = 8192
kernel.msgmnb = 16384
kernel.msgmni = 32768
kernel.ngroups_max = 65536
kernel.ns_last_pid = 12222
kernel.osrelease = 3.4.1-custom
kernel.ostype = Linux
kernel.overflowgid = 65534
kernel.overflowuid = 65534
kernel.panic = 60
kernel.panic_on_io_nmi = 0
kernel.panic_on_oops = 0
kernel.panic_on_unrecovered_nmi = 0
kernel.perf_event_max_sample_rate = 100000
kernel.perf_event_mlock_kb = 516
kernel.perf_event_paranoid = 1
kernel.pid_max = 32768
kernel.poweroff_cmd = /sbin/poweroff
kernel.print-fatal-signals = 0
kernel.printk = 10	4	1	7
kernel.printk_delay = 0
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10
kernel.pty.max = 4096
kernel.pty.nr = 3
kernel.pty.reserve = 1024
kernel.random.boot_id = 193fa337-530b-4b7b-9f8f-152179da4b06
kernel.random.entropy_avail = 2852
kernel.random.poolsize = 4096
kernel.random.read_wakeup_threshold = 64
kernel.random.uuid = dc1547f7-77bc-491a-9b08-f9cdbd67d200
kernel.random.write_wakeup_threshold = 128
kernel.randomize_va_space = 2
kernel.real-root-dev = 0
kernel.sched_child_runs_first = 0
kernel.sched_domain.cpu0.domain0.busy_factor = 64
kernel.sched_domain.cpu0.domain0.busy_idx = 2
kernel.sched_domain.cpu0.domain0.cache_nice_tries = 1
kernel.sched_domain.cpu0.domain0.flags = 4655
kernel.sched_domain.cpu0.domain0.forkexec_idx = 0
kernel.sched_domain.cpu0.domain0.idle_idx = 0
kernel.sched_domain.cpu0.domain0.imbalance_pct = 125
kernel.sched_domain.cpu0.domain0.max_interval = 4
kernel.sched_domain.cpu0.domain0.min_interval = 1
kernel.sched_domain.cpu0.domain0.name = MC
kernel.sched_domain.cpu0.domain0.newidle_idx = 0
kernel.sched_domain.cpu0.domain0.wake_idx = 0
kernel.sched_domain.cpu1.domain0.busy_factor = 64
kernel.sched_domain.cpu1.domain0.busy_idx = 2
kernel.sched_domain.cpu1.domain0.cache_nice_tries = 1
kernel.sched_domain.cpu1.domain0.flags = 4655
kernel.sched_domain.cpu1.domain0.forkexec_idx = 0
kernel.sched_domain.cpu1.domain0.idle_idx = 0
kernel.sched_domain.cpu1.domain0.imbalance_pct = 125
kernel.sched_domain.cpu1.domain0.max_interval = 4
kernel.sched_domain.cpu1.domain0.min_interval = 1
kernel.sched_domain.cpu1.domain0.name = MC
kernel.sched_domain.cpu1.domain0.newidle_idx = 0
kernel.sched_domain.cpu1.domain0.wake_idx = 0
kernel.sched_domain.cpu2.domain0.busy_factor = 64
kernel.sched_domain.cpu2.domain0.busy_idx = 2
kernel.sched_domain.cpu2.domain0.cache_nice_tries = 1
kernel.sched_domain.cpu2.domain0.flags = 4655
kernel.sched_domain.cpu2.domain0.forkexec_idx = 0
kernel.sched_domain.cpu2.domain0.idle_idx = 0
kernel.sched_domain.cpu2.domain0.imbalance_pct = 125
kernel.sched_domain.cpu2.domain0.max_interval = 4
kernel.sched_domain.cpu2.domain0.min_interval = 1
kernel.sched_domain.cpu2.domain0.name = MC
kernel.sched_domain.cpu2.domain0.newidle_idx = 0
kernel.sched_domain.cpu2.domain0.wake_idx = 0
kernel.sched_domain.cpu3.domain0.busy_factor = 64
kernel.sched_domain.cpu3.domain0.busy_idx = 2
kernel.sched_domain.cpu3.domain0.cache_nice_tries = 1
kernel.sched_domain.cpu3.domain0.flags = 4655
kernel.sched_domain.cpu3.domain0.forkexec_idx = 0
kernel.sched_domain.cpu3.domain0.idle_idx = 0
kernel.sched_domain.cpu3.domain0.imbalance_pct = 125
kernel.sched_domain.cpu3.domain0.max_interval = 4
kernel.sched_domain.cpu3.domain0.min_interval = 1
kernel.sched_domain.cpu3.domain0.name = MC
kernel.sched_domain.cpu3.domain0.newidle_idx = 0
kernel.sched_domain.cpu3.domain0.wake_idx = 0
kernel.sched_latency_ns = 18000000
kernel.sched_migration_cost = 500000
kernel.sched_min_granularity_ns = 2250000
kernel.sched_nr_migrate = 32
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sched_shares_window = 10000000
kernel.sched_time_avg = 1000
kernel.sched_tunable_scaling = 1
kernel.sched_wakeup_granularity_ns = 3000000
kernel.sem = 250	32000	32	128
kernel.shm_rmid_forced = 0
kernel.shmall = 2097152
kernel.shmmax = 33554432
kernel.shmmni = 4096
kernel.sysrq = 1
kernel.tainted = 0
kernel.threads-max = 384072
kernel.timer_migration = 1
kernel.unknown_nmi_panic = 0
kernel.usermodehelper.bset = 4294967295	4294967295
kernel.usermodehelper.inheritable = 4294967295	4294967295
kernel.version = #4 SMP Mon Jun 11 09:35:31 CEST 2012
net.core.dev_weight = 64
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_budget = 300
net.core.netdev_max_backlog = 1000
net.core.netdev_tstamp_prequeue = 1
net.core.optmem_max = 20480
net.core.rmem_default = 212992
net.core.rmem_max = 131071
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 128
net.core.warnings = 1
net.core.wmem_default = 212992
net.core.wmem_max = 131071
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_etime = 10
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_larval_drop = 1
net.ipv4.conf.all.accept_local = 0
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_notify = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.promote_secondaries = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.proxy_arp_pvlan = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.src_valid_mark = 0
net.ipv4.conf.all.tag = 0
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.accept_source_route = 1
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_notify = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.proxy_arp_pvlan = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.src_valid_mark = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.eth0.accept_local = 0
net.ipv4.conf.eth0.accept_redirects = 1
net.ipv4.conf.eth0.accept_source_route = 1
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_notify = 0
net.ipv4.conf.eth0.bootp_relay = 0
net.ipv4.conf.eth0.disable_policy = 0
net.ipv4.conf.eth0.disable_xfrm = 0
net.ipv4.conf.eth0.force_igmp_version = 0
net.ipv4.conf.eth0.forwarding = 0
net.ipv4.conf.eth0.log_martians = 0
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth0.medium_id = 0
net.ipv4.conf.eth0.promote_secondaries = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.eth0.proxy_arp_pvlan = 0
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.conf.eth0.shared_media = 1
net.ipv4.conf.eth0.src_valid_mark = 0
net.ipv4.conf.eth0.tag = 0
net.ipv4.conf.eth1.accept_local = 0
net.ipv4.conf.eth1.accept_redirects = 1
net.ipv4.conf.eth1.accept_source_route = 1
net.ipv4.conf.eth1.arp_accept = 0
net.ipv4.conf.eth1.arp_announce = 0
net.ipv4.conf.eth1.arp_filter = 0
net.ipv4.conf.eth1.arp_ignore = 0
net.ipv4.conf.eth1.arp_notify = 0
net.ipv4.conf.eth1.bootp_relay = 0
net.ipv4.conf.eth1.disable_policy = 0
net.ipv4.conf.eth1.disable_xfrm = 0
net.ipv4.conf.eth1.force_igmp_version = 0
net.ipv4.conf.eth1.forwarding = 0
net.ipv4.conf.eth1.log_martians = 0
net.ipv4.conf.eth1.mc_forwarding = 0
net.ipv4.conf.eth1.medium_id = 0
net.ipv4.conf.eth1.promote_secondaries = 0
net.ipv4.conf.eth1.proxy_arp = 0
net.ipv4.conf.eth1.proxy_arp_pvlan = 0
net.ipv4.conf.eth1.rp_filter = 0
net.ipv4.conf.eth1.secure_redirects = 1
net.ipv4.conf.eth1.send_redirects = 1
net.ipv4.conf.eth1.shared_media = 1
net.ipv4.conf.eth1.src_valid_mark = 0
net.ipv4.conf.eth1.tag = 0
net.ipv4.conf.lo.accept_local = 0
net.ipv4.conf.lo.accept_redirects = 1
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_notify = 0
net.ipv4.conf.lo.bootp_relay = 0
net.ipv4.conf.lo.disable_policy = 1
net.ipv4.conf.lo.disable_xfrm = 1
net.ipv4.conf.lo.force_igmp_version = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.lo.medium_id = 0
net.ipv4.conf.lo.promote_secondaries = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.lo.proxy_arp_pvlan = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.lo.secure_redirects = 1
net.ipv4.conf.lo.send_redirects = 1
net.ipv4.conf.lo.shared_media = 1
net.ipv4.conf.lo.src_valid_mark = 0
net.ipv4.conf.lo.tag = 0
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 6168
net.ipv4.igmp_max_memberships = 20
net.ipv4.igmp_max_msf = 10
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_dynaddr = 0
net.ipv4.ip_forward = 0
net.ipv4.ip_local_port_range = 32768	61000
net.ipv4.ip_local_reserved_ports = 
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ipfrag_high_thresh = 262144
net.ipv4.ipfrag_low_thresh = 196608
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 600
net.ipv4.ipfrag_time = 30
net.ipv4.neigh.default.anycast_delay = 100
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.base_reachable_time = 30
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.neigh.default.locktime = 100
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.neigh.default.proxy_delay = 80
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.retrans_time = 100
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.unres_qlen = 32
net.ipv4.neigh.default.unres_qlen_bytes = 65536
net.ipv4.neigh.eth0.anycast_delay = 100
net.ipv4.neigh.eth0.app_solicit = 0
net.ipv4.neigh.eth0.base_reachable_time = 30
net.ipv4.neigh.eth0.base_reachable_time_ms = 30000
net.ipv4.neigh.eth0.delay_first_probe_time = 5
net.ipv4.neigh.eth0.gc_stale_time = 60
net.ipv4.neigh.eth0.locktime = 100
net.ipv4.neigh.eth0.mcast_solicit = 3
net.ipv4.neigh.eth0.proxy_delay = 80
net.ipv4.neigh.eth0.proxy_qlen = 64
net.ipv4.neigh.eth0.retrans_time = 100
net.ipv4.neigh.eth0.retrans_time_ms = 1000
net.ipv4.neigh.eth0.ucast_solicit = 3
net.ipv4.neigh.eth0.unres_qlen = 32
net.ipv4.neigh.eth0.unres_qlen_bytes = 65536
net.ipv4.neigh.eth1.anycast_delay = 100
net.ipv4.neigh.eth1.app_solicit = 0
net.ipv4.neigh.eth1.base_reachable_time = 30
net.ipv4.neigh.eth1.base_reachable_time_ms = 30000
net.ipv4.neigh.eth1.delay_first_probe_time = 5
net.ipv4.neigh.eth1.gc_stale_time = 60
net.ipv4.neigh.eth1.locktime = 100
net.ipv4.neigh.eth1.mcast_solicit = 3
net.ipv4.neigh.eth1.proxy_delay = 80
net.ipv4.neigh.eth1.proxy_qlen = 64
net.ipv4.neigh.eth1.retrans_time = 100
net.ipv4.neigh.eth1.retrans_time_ms = 1000
net.ipv4.neigh.eth1.ucast_solicit = 3
net.ipv4.neigh.eth1.unres_qlen = 32
net.ipv4.neigh.eth1.unres_qlen_bytes = 65536
net.ipv4.neigh.lo.anycast_delay = 100
net.ipv4.neigh.lo.app_solicit = 0
net.ipv4.neigh.lo.base_reachable_time = 30
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.locktime = 100
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.lo.proxy_delay = 80
net.ipv4.neigh.lo.proxy_qlen = 64
net.ipv4.neigh.lo.retrans_time = 100
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.neigh.lo.ucast_solicit = 3
net.ipv4.neigh.lo.unres_qlen = 32
net.ipv4.neigh.lo.unres_qlen_bytes = 65536
net.ipv4.ping_group_range = 1	0
net.ipv4.route.error_burst = 1250
net.ipv4.route.error_cost = 250
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_thresh = 524288
net.ipv4.route.gc_timeout = 300
net.ipv4.route.max_size = 8388608
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.redirect_load = 5
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_silence = 5120
net.ipv4.rt_cache_rebuild_count = 4
net.ipv4.tcp_abc = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_cookie_size = 0
net.ipv4.tcp_dma_copybreak = 262144
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_frto_response = 0
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_ssthresh = 0
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.tcp_mem = 579501	772668	1159002
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096	87380	6291456
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 5
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096	16384	4194304
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.udp_mem = 580200	773600	1160400
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv4.xfrm4_gc_thresh = 4194304
net.ipv6.bindv6only = 0
net.ipv6.conf.all.accept_dad = 1
net.ipv6.conf.all.accept_ra = 1
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.all.accept_ra_rtr_pref = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.all.dad_transmits = 1
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.force_mld_version = 0
net.ipv6.conf.all.force_tllao = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.hop_limit = 64
net.ipv6.conf.all.max_addresses = 16
net.ipv6.conf.all.max_desync_factor = 600
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.all.optimistic_dad = 0
net.ipv6.conf.all.proxy_ndp = 0
net.ipv6.conf.all.regen_max_retry = 3
net.ipv6.conf.all.router_probe_interval = 60
net.ipv6.conf.all.router_solicitation_delay = 1
net.ipv6.conf.all.router_solicitation_interval = 4
net.ipv6.conf.all.router_solicitations = 3
net.ipv6.conf.all.temp_prefered_lft = 86400
net.ipv6.conf.all.temp_valid_lft = 604800
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.accept_dad = 1
net.ipv6.conf.default.accept_ra = 1
net.ipv6.conf.default.accept_ra_defrtr = 1
net.ipv6.conf.default.accept_ra_pinfo = 1
net.ipv6.conf.default.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 1
net.ipv6.conf.default.accept_redirects = 1
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.default.dad_transmits = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.force_mld_version = 0
net.ipv6.conf.default.force_tllao = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.hop_limit = 64
net.ipv6.conf.default.max_addresses = 16
net.ipv6.conf.default.max_desync_factor = 600
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.default.optimistic_dad = 0
net.ipv6.conf.default.proxy_ndp = 0
net.ipv6.conf.default.regen_max_retry = 3
net.ipv6.conf.default.router_probe_interval = 60
net.ipv6.conf.default.router_solicitation_delay = 1
net.ipv6.conf.default.router_solicitation_interval = 4
net.ipv6.conf.default.router_solicitations = 3
net.ipv6.conf.default.temp_prefered_lft = 86400
net.ipv6.conf.default.temp_valid_lft = 604800
net.ipv6.conf.default.use_tempaddr = 0
net.ipv6.conf.eth0.accept_dad = 1
net.ipv6.conf.eth0.accept_ra = 1
net.ipv6.conf.eth0.accept_ra_defrtr = 1
net.ipv6.conf.eth0.accept_ra_pinfo = 1
net.ipv6.conf.eth0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.eth0.accept_ra_rtr_pref = 1
net.ipv6.conf.eth0.accept_redirects = 1
net.ipv6.conf.eth0.accept_source_route = 0
net.ipv6.conf.eth0.autoconf = 1
net.ipv6.conf.eth0.dad_transmits = 1
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.eth0.force_mld_version = 0
net.ipv6.conf.eth0.force_tllao = 0
net.ipv6.conf.eth0.forwarding = 0
net.ipv6.conf.eth0.hop_limit = 64
net.ipv6.conf.eth0.max_addresses = 16
net.ipv6.conf.eth0.max_desync_factor = 600
net.ipv6.conf.eth0.mc_forwarding = 0
net.ipv6.conf.eth0.mtu = 1500
net.ipv6.conf.eth0.optimistic_dad = 0
net.ipv6.conf.eth0.proxy_ndp = 0
net.ipv6.conf.eth0.regen_max_retry = 3
net.ipv6.conf.eth0.router_probe_interval = 60
net.ipv6.conf.eth0.router_solicitation_delay = 1
net.ipv6.conf.eth0.router_solicitation_interval = 4
net.ipv6.conf.eth0.router_solicitations = 3
net.ipv6.conf.eth0.temp_prefered_lft = 86400
net.ipv6.conf.eth0.temp_valid_lft = 604800
net.ipv6.conf.eth0.use_tempaddr = 0
net.ipv6.conf.eth1.accept_dad = 1
net.ipv6.conf.eth1.accept_ra = 1
net.ipv6.conf.eth1.accept_ra_defrtr = 1
net.ipv6.conf.eth1.accept_ra_pinfo = 1
net.ipv6.conf.eth1.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.eth1.accept_ra_rtr_pref = 1
net.ipv6.conf.eth1.accept_redirects = 1
net.ipv6.conf.eth1.accept_source_route = 0
net.ipv6.conf.eth1.autoconf = 1
net.ipv6.conf.eth1.dad_transmits = 1
net.ipv6.conf.eth1.disable_ipv6 = 0
net.ipv6.conf.eth1.force_mld_version = 0
net.ipv6.conf.eth1.force_tllao = 0
net.ipv6.conf.eth1.forwarding = 0
net.ipv6.conf.eth1.hop_limit = 64
net.ipv6.conf.eth1.max_addresses = 16
net.ipv6.conf.eth1.max_desync_factor = 600
net.ipv6.conf.eth1.mc_forwarding = 0
net.ipv6.conf.eth1.mtu = 1500
net.ipv6.conf.eth1.optimistic_dad = 0
net.ipv6.conf.eth1.proxy_ndp = 0
net.ipv6.conf.eth1.regen_max_retry = 3
net.ipv6.conf.eth1.router_probe_interval = 60
net.ipv6.conf.eth1.router_solicitation_delay = 1
net.ipv6.conf.eth1.router_solicitation_interval = 4
net.ipv6.conf.eth1.router_solicitations = 3
net.ipv6.conf.eth1.temp_prefered_lft = 86400
net.ipv6.conf.eth1.temp_valid_lft = 604800
net.ipv6.conf.eth1.use_tempaddr = 0
net.ipv6.conf.lo.accept_dad = -1
net.ipv6.conf.lo.accept_ra = 1
net.ipv6.conf.lo.accept_ra_defrtr = 1
net.ipv6.conf.lo.accept_ra_pinfo = 1
net.ipv6.conf.lo.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.lo.accept_ra_rtr_pref = 1
net.ipv6.conf.lo.accept_redirects = 1
net.ipv6.conf.lo.accept_source_route = 0
net.ipv6.conf.lo.autoconf = 1
net.ipv6.conf.lo.dad_transmits = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.force_mld_version = 0
net.ipv6.conf.lo.force_tllao = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.hop_limit = 64
net.ipv6.conf.lo.max_addresses = 16
net.ipv6.conf.lo.max_desync_factor = 600
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.lo.mtu = 16436
net.ipv6.conf.lo.optimistic_dad = 0
net.ipv6.conf.lo.proxy_ndp = 0
net.ipv6.conf.lo.regen_max_retry = 3
net.ipv6.conf.lo.router_probe_interval = 60
net.ipv6.conf.lo.router_solicitation_delay = 1
net.ipv6.conf.lo.router_solicitation_interval = 4
net.ipv6.conf.lo.router_solicitations = 3
net.ipv6.conf.lo.temp_prefered_lft = 86400
net.ipv6.conf.lo.temp_valid_lft = 604800
net.ipv6.conf.lo.use_tempaddr = -1
net.ipv6.icmp.ratelimit = 1000
net.ipv6.ip6frag_high_thresh = 262144
net.ipv6.ip6frag_low_thresh = 196608
net.ipv6.ip6frag_secret_interval = 600
net.ipv6.ip6frag_time = 60
net.ipv6.mld_max_msf = 64
net.ipv6.neigh.default.anycast_delay = 100
net.ipv6.neigh.default.app_solicit = 0
net.ipv6.neigh.default.base_reachable_time = 30
net.ipv6.neigh.default.base_reachable_time_ms = 30000
net.ipv6.neigh.default.delay_first_probe_time = 5
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 60
net.ipv6.neigh.default.gc_thresh1 = 128
net.ipv6.neigh.default.gc_thresh2 = 512
net.ipv6.neigh.default.gc_thresh3 = 1024
net.ipv6.neigh.default.locktime = 0
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.default.proxy_delay = 80
net.ipv6.neigh.default.proxy_qlen = 64
net.ipv6.neigh.default.retrans_time = 250
net.ipv6.neigh.default.retrans_time_ms = 1000
net.ipv6.neigh.default.ucast_solicit = 3
net.ipv6.neigh.default.unres_qlen = 32
net.ipv6.neigh.default.unres_qlen_bytes = 65536
net.ipv6.neigh.eth0.anycast_delay = 100
net.ipv6.neigh.eth0.app_solicit = 0
net.ipv6.neigh.eth0.base_reachable_time = 30
net.ipv6.neigh.eth0.base_reachable_time_ms = 30000
net.ipv6.neigh.eth0.delay_first_probe_time = 5
net.ipv6.neigh.eth0.gc_stale_time = 60
net.ipv6.neigh.eth0.locktime = 0
net.ipv6.neigh.eth0.mcast_solicit = 3
net.ipv6.neigh.eth0.proxy_delay = 80
net.ipv6.neigh.eth0.proxy_qlen = 64
net.ipv6.neigh.eth0.retrans_time = 250
net.ipv6.neigh.eth0.retrans_time_ms = 1000
net.ipv6.neigh.eth0.ucast_solicit = 3
net.ipv6.neigh.eth0.unres_qlen = 32
net.ipv6.neigh.eth0.unres_qlen_bytes = 65536
net.ipv6.neigh.eth1.anycast_delay = 100
net.ipv6.neigh.eth1.app_solicit = 0
net.ipv6.neigh.eth1.base_reachable_time = 30
net.ipv6.neigh.eth1.base_reachable_time_ms = 30000
net.ipv6.neigh.eth1.delay_first_probe_time = 5
net.ipv6.neigh.eth1.gc_stale_time = 60
net.ipv6.neigh.eth1.locktime = 0
net.ipv6.neigh.eth1.mcast_solicit = 3
net.ipv6.neigh.eth1.proxy_delay = 80
net.ipv6.neigh.eth1.proxy_qlen = 64
net.ipv6.neigh.eth1.retrans_time = 250
net.ipv6.neigh.eth1.retrans_time_ms = 1000
net.ipv6.neigh.eth1.ucast_solicit = 3
net.ipv6.neigh.eth1.unres_qlen = 32
net.ipv6.neigh.eth1.unres_qlen_bytes = 65536
net.ipv6.neigh.lo.anycast_delay = 100
net.ipv6.neigh.lo.app_solicit = 0
net.ipv6.neigh.lo.base_reachable_time = 30
net.ipv6.neigh.lo.base_reachable_time_ms = 30000
net.ipv6.neigh.lo.delay_first_probe_time = 5
net.ipv6.neigh.lo.gc_stale_time = 60
net.ipv6.neigh.lo.locktime = 0
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.lo.proxy_delay = 80
net.ipv6.neigh.lo.proxy_qlen = 64
net.ipv6.neigh.lo.retrans_time = 250
net.ipv6.neigh.lo.retrans_time_ms = 1000
net.ipv6.neigh.lo.ucast_solicit = 3
net.ipv6.neigh.lo.unres_qlen = 32
net.ipv6.neigh.lo.unres_qlen_bytes = 65536
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.min_adv_mss = 1220
net.ipv6.route.mtu_expires = 600
net.ipv6.xfrm6_gc_thresh = 2048
net.netfilter.nf_log.0 = NONE
net.netfilter.nf_log.1 = NONE
net.netfilter.nf_log.10 = NONE
net.netfilter.nf_log.11 = NONE
net.netfilter.nf_log.12 = NONE
net.netfilter.nf_log.2 = NONE
net.netfilter.nf_log.3 = NONE
net.netfilter.nf_log.4 = NONE
net.netfilter.nf_log.5 = NONE
net.netfilter.nf_log.6 = NONE
net.netfilter.nf_log.7 = NONE
net.netfilter.nf_log.8 = NONE
net.netfilter.nf_log.9 = NONE
net.unix.max_dgram_qlen = 10
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.drop_caches = 0
vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256	256	32
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 19901
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 65536
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.nr_pdflush_threads = 0
vm.numa_zonelist_order = default
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.scan_unevictable_pages = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.vfs_cache_pressure = 100
vm.zone_reclaim_mode = 0

[-- Attachment #7: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-11 21:37 XFS hangs and freezes with LSI 9265-8i controller on high i/o Matthew Whittaker-Williams
@ 2012-06-12  1:18 ` Dave Chinner
  2012-06-12 15:56   ` Matthew Whittaker-Williams
  0 siblings, 1 reply; 20+ messages in thread
From: Dave Chinner @ 2012-06-12  1:18 UTC (permalink / raw)
  To: Matthew Whittaker-Williams; +Cc: xfs

On Mon, Jun 11, 2012 at 11:37:23PM +0200, Matthew Whittaker-Williams wrote:
> Dear Developers,
> 
> We are running into some problems with xfs and the LSI 9265-8i Controller.
> 
> http://www.lsi.com/products/storagecomponents/Pages/MegaRAIDSAS9265-8i.aspx
> 
> When running high i/o on raid 6 array with this controller xfs
> freezes up and we get the following errors:
> 
> Linux sd69 3.4.1-custom #4 SMP Mon Jun 11 09:35:31 CEST 2012 x86_64
> GNU/Linux
> 
> [   62.911481] XFS (sda): Mounting Filesystem
> [   63.212456] XFS (sda): Starting recovery (logdev: internal)
> [   64.016420] XFS (sda): Ending recovery (logdev: internal)
> [   64.020549] XFS (sdb): Mounting Filesystem
> [   64.371207] XFS (sdb): Starting recovery (logdev: internal)
> [   65.265051] XFS (sdb): Ending recovery (logdev: internal)
> [ 6110.298886] INFO: task kworker/0:0:11244 blocked for more than
> 120 seconds.
> [ 6110.298942] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 6110.299000] kworker/0:0     D ffff8805ecf52880     0 11244      2
> 0x00000000
> [ 6110.299044]  ffff8805ecf52880 0000000000000046 0000000000000000
> ffffffff81613020
> [ 6110.299107]  00000000000132c0 ffff880582d65fd8 00000000000132c0
> ffff880582d65fd8
> [ 6110.299170]  00000000000132c0 ffff8805ecf52880 00000000000132c0
> ffff880582d64010
> [ 6110.299233] Call Trace:
> [ 6110.299266]  [<ffffffff8134d55a>] ? schedule_timeout+0x2d/0xd7
> [ 6110.299305]  [<ffffffff810f62f5>] ? kmem_cache_alloc+0x2a/0xee
> [ 6110.299358]  [<ffffffffa02cbff4>] ? kmem_zone_alloc+0x58/0x9e [xfs]
> [ 6110.299395]  [<ffffffff8134de6b>] ? __down_common+0x93/0xe4
> [ 6110.299443]  [<ffffffffa03062b0>] ? xfs_getsb+0x2f/0x5c [xfs]
> [ 6110.299480]  [<ffffffff81057994>] ? down+0x27/0x37
> [ 6110.299520]  [<ffffffffa02b81e7>] ? xfs_buf_lock+0x65/0xb2 [xfs]
> [ 6110.299568]  [<ffffffffa03062b0>] ? xfs_getsb+0x2f/0x5c [xfs]
> [ 6110.299613]  [<ffffffffa0312e3b>] ? xfs_trans_getsb+0xa5/0xf5 [xfs]
> [ 6110.299663]  [<ffffffffa0306c9a>] ? xfs_mod_sb+0x43/0x10f [xfs]
> [ 6110.299710]  [<ffffffffa02c70f6>] ? xfs_flush_inodes+0x23/0x23 [xfs]
> [ 6110.299755]  [<ffffffffa02bcd06>] ? xfs_fs_log_dummy+0x61/0x75 [xfs]
> [ 6110.299802]  [<ffffffffa0311978>] ? xfs_ail_min_lsn+0xd/0x2e [xfs]
> [ 6110.299849]  [<ffffffffa02c7133>] ? xfs_sync_worker+0x3d/0x60 [xfs]
> [ 6110.299888]  [<ffffffff812703b6>] ? powersave_bias_target+0x14b/0x14b
> [ 6110.299924]  [<ffffffff8104fa39>] ? process_one_work+0x1cd/0x2eb
> [ 6110.299960]  [<ffffffff8104fc85>] ? worker_thread+0x12e/0x249
> [ 6110.299993]  [<ffffffff8104fb57>] ? process_one_work+0x2eb/0x2eb
> [ 6110.300029]  [<ffffffff8104fb57>] ? process_one_work+0x2eb/0x2eb
> [ 6110.300064]  [<ffffffff8105356e>] ? kthread+0x81/0x89
> [ 6110.300098]  [<ffffffff813569a4>] ? kernel_thread_helper+0x4/0x10

That's pretty much a meaningless stack trace. Can you recompile your
kernel with frame pointers enabled so we can get a reliable stack
trace?

> Could you have a look into this issue?

We know there is a lurking problem that we've been trying to flush
out over the past couple of months. Do a search for hangs in
xlog_grant_log_space - we've found several problems in
the process, but there's still a remaining hang that is likely to be
the source of your problems.

> If you need any more information I am happy to provide it.

What workload are you running that triggers this?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-12  1:18 ` Dave Chinner
@ 2012-06-12 15:56   ` Matthew Whittaker-Williams
  2012-06-12 17:40     ` Matthew Whittaker-Williams
                       ` (2 more replies)
  0 siblings, 3 replies; 20+ messages in thread
From: Matthew Whittaker-Williams @ 2012-06-12 15:56 UTC (permalink / raw)
  To: xfs

[-- Attachment #1: Type: text/plain, Size: 4928 bytes --]

On 6/12/12 3:18 AM, Dave Chinner wrote:
> On Mon, Jun 11, 2012 at 11:37:23PM +0200, Matthew Whittaker-Williams wrote:
>> [ 6110.300098]  [<ffffffff813569a4>] ? kernel_thread_helper+0x4/0x10
> That's pretty much a meaningless stack trace. Can you recompile your
> kernel with frame pointers enabled so we can get a reliable stack
> trace?

See attached for new trace.
>> Could you have a look into this issue?
> We know there is a lurking problem that we've been trying to flush
> out over the past couple of months. Do a search for hangs in
> xlog_grant_log_space - we've found several problems in
> the process, but there's still a remaining hang that is likely to be
> the source of your problems.

I see, it indeed seems about the same issues we encounter.
>> If you need any more information I am happy to provide it.
> What workload are you running that triggers this?

About our workload, we are providing usenet with diablo.
We are currently triggering the issue with running several 
diloadfromspool commands.
This will scan the entire spool and extracts article location size and 
message-id and placement information.

/news/dbin/diloadfromspool -a -S01 &
/news/dbin/diloadfromspool -a -S02 &
/news/dbin/diloadfromspool -a -S03 &
/news/dbin/diloadfromspool -a -S04 &
/news/dbin/diloadfromspool -a -S05 &
/news/dbin/diloadfromspool -a -S06 &
/news/dbin/diloadfromspool -a -S07 &
/news/dbin/diloadfromspool -a -S08 &
/news/dbin/diloadfromspool -a -S09 &
/news/dbin/diloadfromspool -a -S10 &
/news/dbin/diloadfromspool -a -S11 &
/news/dbin/diloadfromspool -a -S12 &
/news/dbin/diloadfromspool -a -S13 &
/news/dbin/diloadfromspool -a -S14 &
/news/dbin/diloadfromspool -a -S15 &
/news/dbin/diloadfromspool -a -S16 &

But we have also been able to produce the same trigger with running 
multiple bonnie++ commands.

nohup bonnie++ -r 4096 -s 81920 -d /news/spool/news/P.01/bonnie1 -f -b 
-n 1 -u root &
nohup bonnie++ -r 4096 -s 81920 -d /news/spool/news/P.01/bonnie2 -f -b 
-n 1 -u root &
nohup bonnie++ -r 4096 -s 81920 -d /news/spool/news/P.01/bonnie2 -f -b 
-n 1 -u root &
nohup bonnie++ -r 4096 -s 81920 -d /news/spool/news/P.01/bonnie3 -f -b 
-n 1 -u root &
nohup bonnie++ -r 4096 -s 81920 -d /news/spool/news/P.01/bonnie4 -f -b 
-n 1 -u root &
nohup bonnie++ -r 4096 -s 81920 -d /news/spool/news/P.01/bonnie5 -f -b 
-n 1 -u root &
nohup bonnie++ -r 4096 -s 81920 -d /news/spool/news/P.01/bonnie6 -f -b 
-n 1 -u root &
nohup bonnie++ -r 4096 -s 81920 -d /news/spool/news/P.01/bonnie7 -f -b 
-n 1 -u root &
nohup bonnie++ -r 4096 -s 81920 -d /news/spool/news/P.01/bonnie8 -f -b 
-n 1 -u root &

It gets triggered when filesystem gets generally alot of reads.

Filesystem info:

meta-data=/dev/sda               isize=256    agcount=41, 
agsize=268435455 blks
          =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=10741350400, imaxpct=5
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

meta-data=/dev/sdb               isize=256    agcount=41, 
agsize=268435455 blks
          =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=10741350400, imaxpct=5
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
Size                : 40.014 TB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 24
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAhead, Cached, No Write Cache if 
Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Cached, No Write Cache if 
Bad BBU
Access Policy       : Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None


Virtual Drive: 1 (Target Id: 1)
Name                :
RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
Size                : 40.014 TB
State               : Optimal
Strip Size          : 1.0 MB
Number Of Drives    : 24
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAhead, Cached, No Write Cache if 
Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Cached, No Write Cache if 
Bad BBU
Access Policy       : Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None


See attachement for controller info if this might help any bit.


>
> Cheers,
>
> Dave.


[-- Attachment #2: kernel-trace.txt --]
[-- Type: text/plain, Size: 188381 bytes --]

Jun 12 17:02:45 sd70 kernel: imklog 4.6.4, log source = /proc/kmsg started.
Jun 12 17:06:27 sd70 kernel: [  323.966564] XFS (sdb): Mounting Filesystem
Jun 12 17:06:27 sd70 kernel: [  324.183373] XFS (sdb): Ending clean mount
Jun 12 17:09:23 sd70 kernel: Kernel logging (proc) stopped.
Jun 12 17:14:14 sd70 kernel: imklog 4.6.4, log source = /proc/kmsg started.
Jun 12 17:32:47 sd70 kernel: [ 1200.374551] INFO: task kworker/0:2:1115 blocked for more than 120 seconds.
Jun 12 17:32:47 sd70 kernel: [ 1200.374628] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 12 17:32:47 sd70 kernel: [ 1200.374722] kworker/0:2     D ffff8805ebdd3fa8     0  1115      2 0x00000000
Jun 12 17:32:47 sd70 kernel: [ 1200.374860]  ffff880602ae7bf0 0000000000000046 ffffffff814cecd0 ffff880603c56840
Jun 12 17:32:47 sd70 kernel: [ 1200.375098]  0000000000000400 ffff880602ae6010 0000000000013340 0000000000013340
Jun 12 17:32:47 sd70 kernel: [ 1200.375336]  ffff880602ae7fd8 0000000000013340 ffff880602ae7fd8 0000000000013340
Jun 12 17:32:47 sd70 kernel: [ 1200.375573] Call Trace:
Jun 12 17:32:47 sd70 kernel: [ 1200.375641]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:47 sd70 kernel: [ 1200.375709]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
Jun 12 17:32:47 sd70 kernel: [ 1200.375775]  [<ffffffff8137d912>] __down_common+0x96/0xe4
Jun 12 17:32:47 sd70 kernel: [ 1200.375858]  [<ffffffffa034338c>] ? xfs_getsb+0x32/0x60 [xfs]
Jun 12 17:32:47 sd70 kernel: [ 1200.375928]  [<ffffffff8137d9bf>] __down+0x18/0x1a
Jun 12 17:32:47 sd70 kernel: [ 1200.375996]  [<ffffffff81060428>] down+0x28/0x38
Jun 12 17:32:47 sd70 kernel: [ 1200.376071]  [<ffffffffa02f4786>] xfs_buf_lock+0x6f/0xc0 [xfs]
Jun 12 17:32:47 sd70 kernel: [ 1200.376155]  [<ffffffffa034338c>] xfs_getsb+0x32/0x60 [xfs]
Jun 12 17:32:47 sd70 kernel: [ 1200.376235]  [<ffffffffa035065c>] xfs_trans_getsb+0xb3/0x107 [xfs]
Jun 12 17:32:47 sd70 kernel: [ 1200.376319]  [<ffffffffa0343da5>] xfs_mod_sb+0x41/0x112 [xfs]
Jun 12 17:32:47 sd70 kernel: [ 1200.376401]  [<ffffffffa0303bac>] ? xfs_flush_inodes+0x2e/0x2e [xfs]
Jun 12 17:32:47 sd70 kernel: [ 1200.376482]  [<ffffffffa02f9434>] xfs_fs_log_dummy+0x61/0x76 [xfs]
Jun 12 17:32:47 sd70 kernel: [ 1200.382394]  [<ffffffffa034a656>] ? xfs_log_need_covered+0x5a/0xb4 [xfs]
Jun 12 17:32:47 sd70 kernel: [ 1200.382534]  [<ffffffffa0303bea>] xfs_sync_worker+0x3e/0x64 [xfs]
Jun 12 17:32:47 sd70 kernel: [ 1200.382607]  [<ffffffff810550d1>] process_one_work+0x1da/0x2e9
Jun 12 17:32:47 sd70 kernel: [ 1200.382741]  [<ffffffff8105531e>] worker_thread+0x13e/0x264
Jun 12 17:32:47 sd70 kernel: [ 1200.382884]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
Jun 12 17:32:47 sd70 kernel: [ 1200.383007]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:47 sd70 kernel: [ 1200.383119]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:47 sd70 kernel: [ 1200.383258]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:47 sd70 kernel: [ 1200.383331]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: 1386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.488066]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.488140]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.488252] kworker/2:1     S ffff8806039202e8     0    30      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.488398]  ffff88060390be70 0000000000000046 ffff88060390be20 ffffffff8129868a
Jun 12 17:32:48 sd70 kernel: [ 1201.488635]  ffff88060390be00 ffff88060390a010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.488873]  ffff88060390bfd8 0000000000013340 ffff88060390bfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.489109] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.489170]  [<ffffffff8129868a>] ? do_dbs_timer+0xe1/0xef
Jun 12 17:32:48 sd70 kernel: [ 1201.489240]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.489308]  [<ffffffff8105543f>] worker_thread+0x25f/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.489377]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
Jun 12 17:32:48 sd70 kernel: [ 1201.489447]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.489515]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.489585]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.489658]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.489726] kworker/3:1     S ffff880603e1a4a8     0    31      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.489865]  ffff880603945e70 0000000000000046 ffff880603945e20 ffffffff8129868a
Jun 12 17:32:48 sd70 kernel: [ 1201.490110]  ffff880603945e00 ffff880603944010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.490348]  ffff880603945fd8 0000000000013340 ffff880603945fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.490585] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.490646]  [<ffffffff8129868a>] ? do_dbs_timer+0xe1/0xef
Jun 12 17:32:48 sd70 kernel: [ 1201.490715]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.490783]  [<ffffffff8105543f>] worker_thread+0x25f/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.490852]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
Jun 12 17:32:48 sd70 kernel: [ 1201.490923]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.490991]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.491061]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.491135]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.491202] khungtaskd      S ffff880603e3b8e8     0    32      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.491349]  ffff8805ecd57e20 0000000000000046 ffff8805ecd57dd0 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.491585]  ffff8805ecd57dd0 ffff8805ecd56010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.491823]  ffff8805ecd57fd8 0000000000013340 ffff8805ecd57fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.492060] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.492122]  [<ffffffff810a3ed2>] ? check_hung_uninterruptible_tasks+0x1c8/0x1c8
Jun 12 17:32:48 sd70 kernel: [ 1201.492211]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.492278]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.492348]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.492418]  [<ffffffff8137d091>] schedule_timeout_interruptible+0x19/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.492491]  [<ffffffff810a3f0f>] watchdog+0x3d/0x4c
Jun 12 17:32:48 sd70 kernel: [ 1201.492558]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.492626]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.492697]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.492770]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.492838] kswapd0         S ffff880603e3bfa8     0    33      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.492984]  ffff880602fe3de0 0000000000000046 ffffffff8167c938 0000000000000f44
Jun 12 17:32:48 sd70 kernel: [ 1201.493221]  ffff88061fffc000 ffff880602fe2010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.493458]  ffff880602fe3fd8 0000000000013340 ffff880602fe3fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.493694] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.493756]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.493826]  [<ffffffff810dd7a9>] kswapd_try_to_sleep+0x125/0x16a
Jun 12 17:32:48 sd70 kernel: [ 1201.493896]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
Jun 12 17:32:48 sd70 kernel: [ 1201.493965]  [<ffffffff810df540>] kswapd+0xe7/0x19f
Jun 12 17:32:48 sd70 kernel: [ 1201.494033]  [<ffffffff810df459>] ? balance_pgdat+0x73b/0x73b
Jun 12 17:32:48 sd70 kernel: [ 1201.494103]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.494171]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.494241]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.494315]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.494382] ksmd            S ffff880603e3c668     0    34      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.494528]  ffff880602fe5e50 0000000000000046 ffff880603e3c380 ffff880603e3c380
Jun 12 17:32:48 sd70 kernel: [ 1201.494764]  ffff880602fe5df0 ffff880602fe4010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.495001]  ffff880602fe5fd8 0000000000013340 ffff880602fe5fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.495238] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.495299]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.495404]  [<ffffffff81105c33>] ksm_scan_thread+0x1aa/0x233
Jun 12 17:32:48 sd70 kernel: [ 1201.495474]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
Jun 12 17:32:48 sd70 kernel: [ 1201.495543]  [<ffffffff81105a89>] ? cmp_and_merge_page+0x2ee/0x2ee
Jun 12 17:32:48 sd70 kernel: [ 1201.495613]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.495747]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.495817]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.495890]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.501511] fsnotify_mark   S ffff880603e3cd28     0    35      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.501652]  ffff8805ecd8de40 0000000000000046 ffff8805ecd8de00 ffffffff81059450
Jun 12 17:32:48 sd70 kernel: [ 1201.501885]  0000000000000000 ffff8805ecd8c010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.502119]  ffff8805ecd8dfd8 0000000000013340 ffff8805ecd8dfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.502352] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.502412]  [<ffffffff81059450>] ? wait_rcu_gp+0x46/0x4f
Jun 12 17:32:48 sd70 kernel: [ 1201.502480]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.502547]  [<ffffffff811412ab>] fsnotify_mark_destroy+0x131/0x147
Jun 12 17:32:48 sd70 kernel: [ 1201.502616]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
Jun 12 17:32:48 sd70 kernel: [ 1201.502684]  [<ffffffff8114117a>] ? fsnotify_set_mark_ignored_mask_locked+0x1e/0x1e
Jun 12 17:32:48 sd70 kernel: [ 1201.502773]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.502839]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.502909]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.502981]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.503047] crypto          S ffff880603e3d3e8     0    36      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.503192]  ffff8805ecdb9e50 0000000000000046 ffff8805ecdb9df0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.503426]  ffff8805ecdb9df0 ffff8805ecdb8010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.503659]  ffff8805ecdb9fd8 0000000000013340 ffff8805ecdb9fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.503893] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.503953]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.504022]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.504089]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.504157]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.504227]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.504299]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.504367]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.504438]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.504511]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.504578] deferwq         S ffff880603e3e828     0    39      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.504724]  ffff8805ecdc3e50 0000000000000046 ffff8805ecdc3df0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.504961]  ffff8805ecdc3df0 ffff8805ecdc2010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.505198]  ffff8805ecdc3fd8 0000000000013340 ffff8805ecdc3fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.505434] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.505495]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.505565]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.505633]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.505703]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.505773]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.505842]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.505910]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.505981]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.506054]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.506121] scsi_eh_0       S ffff8805ece13800     0   232      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.506268]  ffff8805ebc29e90 0000000000000046 ffff8805ece13800 0000000000000000
Jun 12 17:32:48 sd70 kernel: [ 1201.506505]  ffff88061fc33340 ffff8805ebc28010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.506741]  ffff8805ebc29fd8 0000000000013340 ffff8805ebc29fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.506978] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.507039]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.507122]  [<ffffffffa0006a1c>] scsi_error_handler+0x80/0x1a3 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.507198]  [<ffffffffa000699c>] ? scsi_unjam_host+0x1a5/0x1a5 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.507272]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.507339]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.507411]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.507484]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.507551] khubd           S ffff880603901068     0   236      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.507698]  ffff8805ebc99e50 0000000000000046 ffff8805ebc99e60 ffffffffa009e6b2
Jun 12 17:32:48 sd70 kernel: [ 1201.507935]  0000000000000002 ffff8805ebc98010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.508231]  ffff8805ebc99fd8 0000000000013340 ffff8805ebc99fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.508469] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.508534]  [<ffffffffa009e6b2>] ? hub_events+0x1d/0x685 [usbcore]
Jun 12 17:32:48 sd70 kernel: [ 1201.508605]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.508675]  [<ffffffffa009ee06>] hub_thread+0xec/0x1a1 [usbcore]
Jun 12 17:32:48 sd70 kernel: [ 1201.508744]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
Jun 12 17:32:48 sd70 kernel: [ 1201.508816]  [<ffffffffa009ed1a>] ? hub_events+0x685/0x685 [usbcore]
Jun 12 17:32:48 sd70 kernel: [ 1201.508886]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.508952]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.509022]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.509094]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.509161] mpt_poll_0      S ffff880603901728     0   244      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.509306]  ffff8805ec071e50 0000000000000046 ffff8805ec071df0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.509539]  ffff8805ec071df0 ffff8805ec070010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.509772]  ffff8805ec071fd8 0000000000013340 ffff8805ec071fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.510005] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.510065]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.510135]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.510201]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.510270]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.510339]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.510407]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.510474]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.510543]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.510615]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.510681] mpt/0           S ffff880603903fa8     0   245      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.510826]  ffff8805ea4f3e50 0000000000000046 ffff8805ea4f3df0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.511059]  ffff8805ea4f3df0 ffff8805ea4f2010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.511293]  ffff8805ea4f3fd8 0000000000013340 ffff8805ea4f3fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.511525] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.511586]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.511655]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.511722]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.511790]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.511859]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.511927]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.511993]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.512063]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.512135]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.512205] ata_sff         S ffff8805eb5f6ee8     0   246      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.512352]  ffff8805ebb03e50 0000000000000046 ffff8805ebb03df0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.512589]  ffff8805ebb03df0 ffff8805ebb02010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.512826]  ffff8805ebb03fd8 0000000000013340 ffff8805ebb03fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.513063] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.513124]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.513195]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.513263]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.513333]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.513402]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.513471]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.513539]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.513610]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.513683]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.513750] scsi_eh_1       S ffff8805ec644d28     0   262      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.513897]  ffff8805ec62de90 0000000000000046 0000000000000000 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.514134]  ffff8805ec62de50 ffff8805ec62c010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.514371]  ffff8805ec62dfd8 0000000000013340 ffff8805ec62dfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.514607] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.514668]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.514740]  [<ffffffffa0006a1c>] scsi_error_handler+0x80/0x1a3 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.514817]  [<ffffffffa000699c>] ? scsi_unjam_host+0x1a5/0x1a5 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.514890]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.514954]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.515019]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.515092]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.515162] scsi_eh_2       S ffff880603905aa8     0   263      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.515312]  ffff8805ebe09e90 0000000000000046 0000000000000000 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.515548]  ffff8805ebe09e50 ffff8805ebe08010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.515785]  ffff8805ebe09fd8 0000000000013340 ffff8805ebe09fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.516084] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.516146]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.516217]  [<ffffffffa0006a1c>] scsi_error_handler+0x80/0x1a3 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.516294]  [<ffffffffa000699c>] ? scsi_unjam_host+0x1a5/0x1a5 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.516366]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.516434]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.516569]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.516643]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.516710] scsi_eh_3       S ffff8805eb5f75a8     0   264      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.516857]  ffff8805ea767e90 0000000000000046 0000000000000000 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.517093]  ffff8805ea767e50 ffff8805ea766010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.517330]  ffff8805ea767fd8 0000000000013340 ffff8805ea767fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.517566] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.517628]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.517699]  [<ffffffffa0006a1c>] scsi_error_handler+0x80/0x1a3 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.517775]  [<ffffffffa000699c>] ? scsi_unjam_host+0x1a5/0x1a5 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.517848]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.517916]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.517987]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.518059]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.523650] scsi_eh_4       S ffff8805ec6453e8     0   265      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.523797]  ffff8805eb6e7e90 0000000000000046 0000000000000000 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.524170]  ffff8805eb6e7e50 ffff8805eb6e6010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.524456]  ffff8805eb6e7fd8 0000000000013340 ffff8805eb6e7fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.524754] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.524815]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.524886]  [<ffffffffa0006a1c>] scsi_error_handler+0x80/0x1a3 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.524963]  [<ffffffffa000699c>] ? scsi_unjam_host+0x1a5/0x1a5 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.525035]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.525103]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.525174]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.525247]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.525314] scsi_eh_5       S ffff880603904d28     0   266      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.525461]  ffff8805ebee5e90 0000000000000046 0000000000000000 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.525693]  ffff8805ebee5e50 ffff8805ebee4010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.525931]  ffff8805ebee5fd8 0000000000013340 ffff8805ebee5fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.526169] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.526227]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.526294]  [<ffffffffa0006a1c>] scsi_error_handler+0x80/0x1a3 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.526369]  [<ffffffffa000699c>] ? scsi_unjam_host+0x1a5/0x1a5 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.526444]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.526515]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.526653]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.526726]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.526793] scsi_eh_6       S ffff8805eb5f2b68     0   267      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.526940]  ffff8805eb727e90 0000000000000046 0000000000000000 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.527176]  ffff8805eb727e50 ffff8805eb726010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.527413]  ffff8805eb727fd8 0000000000013340 ffff8805eb727fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.527650] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.527711]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.527783]  [<ffffffffa0006a1c>] scsi_error_handler+0x80/0x1a3 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.527859]  [<ffffffffa000699c>] ? scsi_unjam_host+0x1a5/0x1a5 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.527932]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.528060]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.528130]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.528203]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.528270] kworker/u:4     S ffff8805eabadb00     0   270      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.528417]  ffff8805eb5b9e70 0000000000000046 ffff8805ebd64180 ffff8805ebd64190
Jun 12 17:32:48 sd70 kernel: [ 1201.528654]  ffff8805eb5b9e20 ffff8805eb5b8010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.528890]  ffff8805eb5b9fd8 0000000000013340 ffff8805eb5b9fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.529127] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.529189]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.529256]  [<ffffffff8105543f>] worker_thread+0x25f/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.529326]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
Jun 12 17:32:48 sd70 kernel: [ 1201.529396]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.529464]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.529534]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.529607]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.529675] kworker/u:5     S ffff8805ec646168     0   271      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.529822]  ffff8805ec3e7e70 0000000000000046 ffff8805ec3e7df0 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.530059]  ffff8805ec3e7e20 ffff8805ec3e6010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.530296]  ffff8805ec3e7fd8 0000000000013340 ffff8805ec3e7fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.530532] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.530594]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.530661]  [<ffffffff8105543f>] worker_thread+0x25f/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.530731]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
Jun 12 17:32:48 sd70 kernel: [ 1201.530801]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.530869]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.530940]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.531013]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.531080] scsi_eh_7       S ffff8805ebdd75a8     0   295      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.531226]  ffff8805eb4f1e90 0000000000000046 ffff8805ece13480 0000000000000000
Jun 12 17:32:48 sd70 kernel: [ 1201.531464]  ffff88061fc73340 ffff8805eb4f0010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.531700]  ffff8805eb4f1fd8 0000000000013340 ffff8805eb4f1fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.531937] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.531998]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.532070]  [<ffffffffa0006a1c>] scsi_error_handler+0x80/0x1a3 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.532147]  [<ffffffffa000699c>] ? scsi_unjam_host+0x1a5/0x1a5 [scsi_mod]
Jun 12 17:32:48 sd70 kernel: [ 1201.532219]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.532287]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.532357]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.532431]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.532498] loop0           S ffff8805ec016168     0   620      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.532645]  ffff8805eaf29e40 0000000000000046 ffff8805eaf29dc0 ffffffff810cf725
Jun 12 17:32:48 sd70 kernel: [ 1201.532882]  ffff8805eaf29de0 ffff8805eaf28010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.533118]  ffff8805eaf29fd8 0000000000013340 ffff8805eaf29fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.533355] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.533417]  [<ffffffff810cf725>] ? mempool_free_slab+0x12/0x14
Jun 12 17:32:48 sd70 kernel: [ 1201.533487]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.533557]  [<ffffffffa01c969f>] loop_thread+0xbe/0x22e [loop]
Jun 12 17:32:48 sd70 kernel: [ 1201.533627]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
Jun 12 17:32:48 sd70 kernel: [ 1201.533698]  [<ffffffffa01c95e1>] ? do_bio_filebacked+0x2ae/0x2ae [loop]
Jun 12 17:32:48 sd70 kernel: [ 1201.533770]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.533837]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.533908]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.533981]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.534048] udevd           S ffff8805ec48e168     0   946      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.534195]  ffff8805ecee99e8 0000000000000082 0000000000000041 ffff8805ecee9e3c
Jun 12 17:32:48 sd70 kernel: [ 1201.534432]  ffff8805ecee9998 ffff8805ecee8010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.534668]  ffff8805ecee9fd8 0000000000013340 ffff8805ecee9fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.534905] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.534966]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.535034]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.535107]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.535177]  [<ffffffff8111f52c>] ? __pollwait+0xbe/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.535246]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.535318]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.535388]  [<ffffffff8111f83c>] do_poll+0x1da/0x1ff
Jun 12 17:32:48 sd70 kernel: [ 1201.535457]  [<ffffffff811204ad>] do_sys_poll+0x166/0x1d7
Jun 12 17:32:48 sd70 kernel: [ 1201.535526]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.535594]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.535663]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.535732]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.535801]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.535871]  [<ffffffff810aad21>] ? call_rcu_sched+0x12/0x14
Jun 12 17:32:48 sd70 kernel: [ 1201.535941]  [<ffffffff810413ab>] ? release_task+0xf5/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.536060]  [<ffffffff8104194c>] ? wait_task_zombie+0x590/0x5a7
Jun 12 17:32:48 sd70 kernel: [ 1201.536131]  [<ffffffff8105c268>] ? remove_wait_queue+0x4c/0x51
Jun 12 17:32:48 sd70 kernel: [ 1201.536201]  [<ffffffff8104250b>] ? do_wait+0x1d7/0x21a
Jun 12 17:32:48 sd70 kernel: [ 1201.536270]  [<ffffffff810425f6>] ? sys_wait4+0xa8/0xbc
Jun 12 17:32:48 sd70 kernel: [ 1201.536338]  [<ffffffff811206b0>] sys_poll+0x4e/0xb7
Jun 12 17:32:48 sd70 kernel: [ 1201.536406]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.536475] kworker/0:2     D ffff8805ebdd3fa8     0  1115      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.536622]  ffff880602ae7bf0 0000000000000046 ffffffff814cecd0 ffff880603c56840
Jun 12 17:32:48 sd70 kernel: [ 1201.536858]  0000000000000400 ffff880602ae6010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.537095]  ffff880602ae7fd8 0000000000013340 ffff880602ae7fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.537331] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.537392]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.537460]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.537530]  [<ffffffff8137d912>] __down_common+0x96/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.537613]  [<ffffffffa034338c>] ? xfs_getsb+0x32/0x60 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.537683]  [<ffffffff8137d9bf>] __down+0x18/0x1a
Jun 12 17:32:48 sd70 kernel: [ 1201.537750]  [<ffffffff81060428>] down+0x28/0x38
Jun 12 17:32:48 sd70 kernel: [ 1201.537824]  [<ffffffffa02f4786>] xfs_buf_lock+0x6f/0xc0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.537907]  [<ffffffffa034338c>] xfs_getsb+0x32/0x60 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.537987]  [<ffffffffa035065c>] xfs_trans_getsb+0xb3/0x107 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.538071]  [<ffffffffa0343da5>] xfs_mod_sb+0x41/0x112 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.538153]  [<ffffffffa0303bac>] ? xfs_flush_inodes+0x2e/0x2e [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.538233]  [<ffffffffa02f9434>] xfs_fs_log_dummy+0x61/0x76 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.538316]  [<ffffffffa034a656>] ? xfs_log_need_covered+0x5a/0xb4 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.538400]  [<ffffffffa0303bea>] xfs_sync_worker+0x3e/0x64 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.538471]  [<ffffffff810550d1>] process_one_work+0x1da/0x2e9
Jun 12 17:32:48 sd70 kernel: [ 1201.538541]  [<ffffffff8105531e>] worker_thread+0x13e/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.538610]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
Jun 12 17:32:48 sd70 kernel: [ 1201.538680]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.538748]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.538819]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.538892]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.538959] ifup            S ffff880603921068     0  1234      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.539106]  ffff8805bc5b7e48 0000000000000086 00007f71645664f8 ffff880500000065
Jun 12 17:32:48 sd70 kernel: [ 1201.539343]  ffffffff8106ec4e ffff8805bc5b6010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.539579]  ffff8805bc5b7fd8 0000000000013340 ffff8805bc5b7fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.539816] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.539877]  [<ffffffff8106ec4e>] ? enqueue_entity+0x121/0x1cd
Jun 12 17:32:48 sd70 kernel: [ 1201.539947]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.540015]  [<ffffffff810424da>] do_wait+0x1a6/0x21a
Jun 12 17:32:48 sd70 kernel: [ 1201.540150]  [<ffffffff8106a645>] ? wake_up_new_task+0xc2/0xce
Jun 12 17:32:48 sd70 kernel: [ 1201.540220]  [<ffffffff8103dc1c>] ? do_fork+0x193/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.540288]  [<ffffffff810425e7>] sys_wait4+0x99/0xbc
Jun 12 17:32:48 sd70 kernel: [ 1201.540356]  [<ffffffff81114852>] ? sys_newstat+0x2d/0x38
Jun 12 17:32:48 sd70 kernel: [ 1201.545900]  [<ffffffff810406fb>] ? wait_task_continued+0x14c/0x14c
Jun 12 17:32:48 sd70 kernel: [ 1201.545970]  [<ffffffff8137f0e5>] ? page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.546038]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.546107] dhclient        S ffff8805ec6b5aa8     0  1633   1234 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.546252]  ffff8806012d3918 0000000000000086 0000000000000000 0000000000000000
Jun 12 17:32:48 sd70 kernel: [ 1201.546485]  ffff8806012d38c8 ffff8806012d2010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.546719]  ffff8806012d3fd8 0000000000013340 ffff8806012d3fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.546952] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.547013]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.547079]  [<ffffffff8137d646>] schedule_hrtimeout_range_clock+0xe8/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.547152]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.547220]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.547290]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.547360]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.547430]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.547497]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.547565]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.547633]  [<ffffffff81308df6>] ? udp_recvmsg+0x1c7/0x23b
Jun 12 17:32:48 sd70 kernel: [ 1201.547701]  [<ffffffff8130f54e>] ? inet_recvmsg+0x66/0x77
Jun 12 17:32:48 sd70 kernel: [ 1201.547770]  [<ffffffff812aa5f7>] ? __sock_recvmsg_nosec+0x65/0x6e
Jun 12 17:32:48 sd70 kernel: [ 1201.547840]  [<ffffffff812ab41e>] ? __sock_recvmsg+0x49/0x54
Jun 12 17:32:48 sd70 kernel: [ 1201.547909]  [<ffffffff812abaa3>] ? sock_recvmsg+0xa6/0xbf
Jun 12 17:32:48 sd70 kernel: [ 1201.547978]  [<ffffffff810cdbc8>] ? unlock_page+0x25/0x29
Jun 12 17:32:48 sd70 kernel: [ 1201.548046]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.548115]  [<ffffffff812ab5a8>] ? move_addr_to_user+0x76/0xaa
Jun 12 17:32:48 sd70 kernel: [ 1201.548185]  [<ffffffff812abfd7>] ? sys_recvfrom+0x11a/0x147
Jun 12 17:32:48 sd70 kernel: [ 1201.548254]  [<ffffffff81014fc6>] ? read_tsc+0x9/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.548321]  [<ffffffff8107a59a>] ? timekeeping_get_ns+0x15/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.548390]  [<ffffffff8107b707>] ? ktime_get_ts+0x80/0x89
Jun 12 17:32:48 sd70 kernel: [ 1201.548458]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.548526]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.548594] uptimed         S ffff8805eb5f4668     0  1705      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.548739]  ffff8805ec687e48 0000000000000082 ffff8805ec687dc8 ffffffff8111b17f
Jun 12 17:32:48 sd70 kernel: [ 1201.548972]  ffff8805ec687f68 ffff8805ec686010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.549206]  ffff8805ec687fd8 0000000000013340 ffff8805ec687fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.549439] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.549499]  [<ffffffff8111b17f>] ? putname+0x2d/0x36
Jun 12 17:32:48 sd70 kernel: [ 1201.549566]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.549632]  [<ffffffff8137d71c>] do_nanosleep+0x70/0xb2
Jun 12 17:32:48 sd70 kernel: [ 1201.549700]  [<ffffffff8105f516>] hrtimer_nanosleep+0xba/0x13b
Jun 12 17:32:48 sd70 kernel: [ 1201.549769]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.549837]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.549908]  [<ffffffff8105f5f5>] sys_nanosleep+0x5e/0x72
Jun 12 17:32:48 sd70 kernel: [ 1201.549976]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.550044] atd             S ffff8805ec69a4a8     0  1708      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.550189]  ffff8806011a1e48 0000000000000082 ffff880606dd4a38 ffff880606dd4a38
Jun 12 17:32:48 sd70 kernel: [ 1201.550423]  ffff8806011a1dd8 ffff8806011a0010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.550656]  ffff8806011a1fd8 0000000000013340 ffff8806011a1fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.550890] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.550950]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.551017]  [<ffffffff8137d71c>] do_nanosleep+0x70/0xb2
Jun 12 17:32:48 sd70 kernel: [ 1201.551084]  [<ffffffff8105f516>] hrtimer_nanosleep+0xba/0x13b
Jun 12 17:32:48 sd70 kernel: [ 1201.551153]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.551222]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.551292]  [<ffffffff8105f5f5>] sys_nanosleep+0x5e/0x72
Jun 12 17:32:48 sd70 kernel: [ 1201.551360]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.551429] dbus-daemon     S ffff8805ec6b6828     0  1718      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.551573]  ffff88060102f9e8 0000000000000086 ffff8800dec4a230 ffff8805ba246000
Jun 12 17:32:48 sd70 kernel: [ 1201.551807]  00000000005ba246 ffff88060102e010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.552040]  ffff88060102ffd8 0000000000013340 ffff88060102ffd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.552273] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.552334]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.552400]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.552472]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.552540]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.552611]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.552681]  [<ffffffff8111f83c>] do_poll+0x1da/0x1ff
Jun 12 17:32:48 sd70 kernel: [ 1201.552749]  [<ffffffff811204ad>] do_sys_poll+0x166/0x1d7
Jun 12 17:32:48 sd70 kernel: [ 1201.552816]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.552884]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.552951]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.553019]  [<ffffffff810cdbc8>] ? unlock_page+0x25/0x29
Jun 12 17:32:48 sd70 kernel: [ 1201.553087]  [<ffffffff810ecc12>] ? __do_fault+0x3a7/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.553155]  [<ffffffff810edbe9>] ? handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.553225]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.553295]  [<ffffffff810ede10>] ? handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.553364]  [<ffffffff81382033>] ? do_page_fault+0x34a/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.553432]  [<ffffffff81111cd8>] ? alloc_file+0x19/0xab
Jun 12 17:32:48 sd70 kernel: [ 1201.553500]  [<ffffffff8104bdd7>] ? do_sigaction+0x80/0x187
Jun 12 17:32:48 sd70 kernel: [ 1201.553569]  [<ffffffff8111d6a3>] ? set_close_on_exec+0x2f/0x57
Jun 12 17:32:48 sd70 kernel: [ 1201.553638]  [<ffffffff811206b0>] sys_poll+0x4e/0xb7
Jun 12 17:32:48 sd70 kernel: [ 1201.553705]  [<ffffffff8137f0e5>] ? page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.553772]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.553841] dnsmasq         S ffff8805ec69f5a8     0  1728      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.553986]  ffff8805bc7db918 0000000000000082 ffff8805bc7db968 ffff8805ebd26400
Jun 12 17:32:48 sd70 kernel: [ 1201.554219]  ffff8805ebc72100 ffff8805bc7da010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.554453]  ffff8805bc7dbfd8 0000000000013340 ffff8805bc7dbfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.554687] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.554747]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.554814]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.554886]  [<ffffffff812eb50f>] ? ip_output+0x58/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.554954]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.555022]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.555090]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.555160]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.555230]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.555298]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.555365]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.555433]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.555501]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.555568]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.555636]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.555704]  [<ffffffff812ab41e>] ? __sock_recvmsg+0x49/0x54
Jun 12 17:32:48 sd70 kernel: [ 1201.555773]  [<ffffffff81108d59>] ? __slab_free+0xe1/0x22d
Jun 12 17:32:48 sd70 kernel: [ 1201.555842]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.555911]  [<ffffffff81109a27>] ? kmem_cache_free+0x6f/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.555980]  [<ffffffff81121ac3>] ? __d_free+0x51/0x56
Jun 12 17:32:48 sd70 kernel: [ 1201.556047]  [<ffffffff81121b0b>] ? d_free+0x43/0x56
Jun 12 17:32:48 sd70 kernel: [ 1201.556113]  [<ffffffff81122e0a>] ? dentry_kill+0x151/0x15d
Jun 12 17:32:48 sd70 kernel: [ 1201.556181]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.556249]  [<ffffffff8111222a>] ? fput+0x15/0x17
Jun 12 17:32:48 sd70 kernel: [ 1201.556315]  [<ffffffff8110f0c1>] ? filp_close+0x6d/0x78
Jun 12 17:32:48 sd70 kernel: [ 1201.556383]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.556451] irqbalance      S ffff8805ec69ab68     0  1745      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.556596]  ffff8805bc7a7e48 0000000000000086 ffff880602bc0958 ffff88020895e170
Jun 12 17:32:48 sd70 kernel: [ 1201.556830]  ffff8805bc7a7de8 ffff8805bc7a6010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.557063]  ffff8805bc7a7fd8 0000000000013340 ffff8805bc7a7fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.557297] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.557357]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.557423]  [<ffffffff8137d71c>] do_nanosleep+0x70/0xb2
Jun 12 17:32:48 sd70 kernel: [ 1201.557491]  [<ffffffff811227ad>] ? __d_drop+0x14/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.557558]  [<ffffffff8105f516>] hrtimer_nanosleep+0xba/0x13b
Jun 12 17:32:48 sd70 kernel: [ 1201.557627]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.557695]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.557766]  [<ffffffff8105f5f5>] sys_nanosleep+0x5e/0x72
Jun 12 17:32:48 sd70 kernel: [ 1201.557834]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.557902] cron            S ffff8805ec69e168     0  1862      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.558047]  ffff88060284de48 0000000000000082 ffff8805ec69de80 ffffffffff600400
Jun 12 17:32:48 sd70 kernel: [ 1201.558280]  0000000000000015 ffff88060284c010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.558514]  ffff88060284dfd8 0000000000013340 ffff88060284dfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.558747] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.558807]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.558873]  [<ffffffff8137d71c>] do_nanosleep+0x70/0xb2
Jun 12 17:32:48 sd70 kernel: [ 1201.558941]  [<ffffffff8105f516>] hrtimer_nanosleep+0xba/0x13b
Jun 12 17:32:48 sd70 kernel: [ 1201.559011]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.559079]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.559150]  [<ffffffff8105f5f5>] sys_nanosleep+0x5e/0x72
Jun 12 17:32:48 sd70 kernel: [ 1201.559217]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.559286] dhclient        S ffff8805ec6982e8     0  1886      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.559431]  ffff880602bd7918 0000000000000086 0000000000000000 0000000000000000
Jun 12 17:32:48 sd70 kernel: [ 1201.559664]  ffff880602bd78c8 ffff880602bd6010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.559898]  ffff880602bd7fd8 0000000000013340 ffff880602bd7fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.560203] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.560264]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.560332]  [<ffffffff8137d646>] schedule_hrtimeout_range_clock+0xe8/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.560405]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.560475]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.560546]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.560618]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.560689]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.560758]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.560827]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.560895]  [<ffffffff81308df6>] ? udp_recvmsg+0x1c7/0x23b
Jun 12 17:32:48 sd70 kernel: [ 1201.560965]  [<ffffffff8130f54e>] ? inet_recvmsg+0x66/0x77
Jun 12 17:32:48 sd70 kernel: [ 1201.561034]  [<ffffffff812aa5f7>] ? __sock_recvmsg_nosec+0x65/0x6e
Jun 12 17:32:48 sd70 kernel: [ 1201.561105]  [<ffffffff812ab41e>] ? __sock_recvmsg+0x49/0x54
Jun 12 17:32:48 sd70 kernel: [ 1201.566647]  [<ffffffff812abaa3>] ? sock_recvmsg+0xa6/0xbf
Jun 12 17:32:48 sd70 kernel: [ 1201.566716]  [<ffffffff812ab522>] ? sock_aio_read+0xf9/0x109
Jun 12 17:32:48 sd70 kernel: [ 1201.566785]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.566853]  [<ffffffff811108e9>] ? do_sync_read+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.566922]  [<ffffffff812ab5a8>] ? move_addr_to_user+0x76/0xaa
Jun 12 17:32:48 sd70 kernel: [ 1201.566991]  [<ffffffff812abfd7>] ? sys_recvfrom+0x11a/0x147
Jun 12 17:32:48 sd70 kernel: [ 1201.567059]  [<ffffffff81014fc6>] ? read_tsc+0x9/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.567126]  [<ffffffff8107a59a>] ? timekeeping_get_ns+0x15/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.567196]  [<ffffffff8107b707>] ? ktime_get_ts+0x80/0x89
Jun 12 17:32:48 sd70 kernel: [ 1201.567264]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.567331]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.567399] sshd            S ffff880603923fa8     0  1969      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.567544]  ffff8805fd725918 0000000000000086 ffffea00166d5840 0000000000000001
Jun 12 17:32:48 sd70 kernel: [ 1201.567777]  0000000000000202 ffff8805fd724010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.568011]  ffff8805fd725fd8 0000000000013340 ffff8805fd725fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.568251] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.568313]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.568380]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.568454]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.568523]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.568595]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.568665]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.568735]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.568803]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.568872]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.568941]  [<ffffffff810d4a84>] ? __alloc_pages_nodemask+0x12a/0x1ae
Jun 12 17:32:48 sd70 kernel: [ 1201.569014]  [<ffffffff811c22f9>] ? cpumask_any_but+0x29/0x36
Jun 12 17:32:48 sd70 kernel: [ 1201.569085]  [<ffffffff810d8569>] ? __lru_cache_add+0x2f/0x54
Jun 12 17:32:48 sd70 kernel: [ 1201.569155]  [<ffffffff810d85ca>] ? lru_cache_add_lru+0x3c/0x3e
Jun 12 17:32:48 sd70 kernel: [ 1201.569226]  [<ffffffff810f42fe>] ? page_add_new_anon_rmap+0x75/0x87
Jun 12 17:32:48 sd70 kernel: [ 1201.569297]  [<ffffffff810ed22c>] ? do_wp_page+0x5df/0x5f9
Jun 12 17:32:48 sd70 kernel: [ 1201.569367]  [<ffffffff8110b726>] ? kmem_cache_alloc+0x61/0x118
Jun 12 17:32:48 sd70 kernel: [ 1201.569437]  [<ffffffff812ab3c5>] ? sock_aio_write+0xf1/0x101
Jun 12 17:32:48 sd70 kernel: [ 1201.569507]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.569576]  [<ffffffff81124855>] ? destroy_inode+0x3b/0x54
Jun 12 17:32:48 sd70 kernel: [ 1201.569646]  [<ffffffff81107557>] ? virt_to_head_page+0x9/0x2c
Jun 12 17:32:48 sd70 kernel: [ 1201.569716]  [<ffffffff811099d3>] ? kmem_cache_free+0x1b/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.569786]  [<ffffffff81121ac3>] ? __d_free+0x51/0x56
Jun 12 17:32:48 sd70 kernel: [ 1201.569853]  [<ffffffff81121b0b>] ? d_free+0x43/0x56
Jun 12 17:32:48 sd70 kernel: [ 1201.569921]  [<ffffffff81122e0a>] ? dentry_kill+0x151/0x15d
Jun 12 17:32:48 sd70 kernel: [ 1201.569990]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.570058]  [<ffffffff8111222a>] ? fput+0x15/0x17
Jun 12 17:32:48 sd70 kernel: [ 1201.570125]  [<ffffffff8110f0c1>] ? filp_close+0x6d/0x78
Jun 12 17:32:48 sd70 kernel: [ 1201.570195]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.570264] udevd           S ffff880603923228     0  1972    946 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.570411]  ffff880602b63998 0000000000000086 0000005100010000 ffffea0016dbd140
Jun 12 17:32:48 sd70 kernel: [ 1201.570648]  ffff880602b63978 ffff880602b62010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.570885]  ffff880602b63fd8 0000000000013340 ffff880602b63fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.571122] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.571183]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.571250]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.571324]  [<ffffffff81109bd8>] ? __kmalloc_node_track_caller+0x10f/0x16f
Jun 12 17:32:48 sd70 kernel: [ 1201.571397]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.571466]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.571538]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.571608]  [<ffffffff8111f83c>] do_poll+0x1da/0x1ff
Jun 12 17:32:48 sd70 kernel: [ 1201.571677]  [<ffffffff811204ad>] do_sys_poll+0x166/0x1d7
Jun 12 17:32:48 sd70 kernel: [ 1201.571746]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.571815]  [<ffffffff812ab281>] ? __sock_sendmsg_nosec+0x5f/0x6a
Jun 12 17:32:48 sd70 kernel: [ 1201.571887]  [<ffffffff812ab2c9>] ? __sock_sendmsg+0x3d/0x48
Jun 12 17:32:48 sd70 kernel: [ 1201.572007]  [<ffffffff812abba0>] ? sock_sendmsg+0xa3/0xbc
Jun 12 17:32:48 sd70 kernel: [ 1201.572076]  [<ffffffff810cdbc8>] ? unlock_page+0x25/0x29
Jun 12 17:32:48 sd70 kernel: [ 1201.572148]  [<ffffffff810edbe9>] ? handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.572218]  [<ffffffff810ede10>] ? handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.572287]  [<ffffffff812abc21>] ? sockfd_lookup_light+0x1b/0x53
Jun 12 17:32:48 sd70 kernel: [ 1201.572357]  [<ffffffff812ac154>] ? sys_sendto+0x13f/0x16c
Jun 12 17:32:48 sd70 kernel: [ 1201.572425]  [<ffffffff8104c86f>] ? __set_task_blocked+0x60/0x69
Jun 12 17:32:48 sd70 kernel: [ 1201.572495]  [<ffffffff8104c8a9>] ? set_current_blocked+0x31/0x4c
Jun 12 17:32:48 sd70 kernel: [ 1201.572565]  [<ffffffff811205d9>] sys_ppoll+0xbb/0x144
Jun 12 17:32:48 sd70 kernel: [ 1201.572632]  [<ffffffff812ac616>] ? sys_sendmsg+0x4c/0x5b
Jun 12 17:32:48 sd70 kernel: [ 1201.572700]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.572768] xfsalloc        S ffff880603e3e168     0  1973      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.572910]  ffff8805f3039e50 0000000000000046 ffff8805f3039df0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.573142]  ffff8805f3039df0 ffff8805f3038010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.573374]  ffff8805f3039fd8 0000000000013340 ffff8805f3039fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.573605] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.573666]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.573735]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.573802]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.573870]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.573939]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.574007]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.574074]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.574143]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.574215]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.574282] xfs_mru_cache   S ffff8805bc4f75a8     0  1974      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.574426]  ffff880602b87e50 0000000000000046 ffff880602b87df0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.574660]  ffff880602b87df0 ffff880602b86010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.574893]  ffff880602b87fd8 0000000000013340 ffff880602b87fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.575126] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.575186]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.575255]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.575321]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.575390]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.575459]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.575527]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.575594]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.575664]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.575736]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.575839] xfslogd         S ffff8805ec699068     0  1975      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.575983]  ffff8805bc6f5e50 0000000000000046 ffff8805bc6f5df0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.576217]  ffff8805bc6f5df0 ffff8805bc6f4010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.576450]  ffff8805bc6f5fd8 0000000000013340 ffff8805bc6f5fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.576684] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.576744]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.576813]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.576880]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.576947]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.577013]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.577082]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.577149]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.577218]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.577290]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.577357] udevd           S ffff880603924d28     0  1976    946 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.577501]  ffff880601255998 0000000000000082 ffff880601255978 000000000001e24c
Jun 12 17:32:48 sd70 kernel: [ 1201.577734]  ffff880601255948 ffff880601254010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.577968]  ffff880601255fd8 0000000000013340 ffff880601255fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.578202] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.578262]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.578329]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.578402]  [<ffffffff81062b71>] ? check_preempt_curr+0x2d/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.578471]  [<ffffffff81069a0d>] ? ttwu_do_wakeup+0x54/0xcb
Jun 12 17:32:48 sd70 kernel: [ 1201.578540]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.578608]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.578679]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.578748]  [<ffffffff8111f83c>] do_poll+0x1da/0x1ff
Jun 12 17:32:48 sd70 kernel: [ 1201.578815]  [<ffffffff811204ad>] do_sys_poll+0x166/0x1d7
Jun 12 17:32:48 sd70 kernel: [ 1201.578884]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.578952]  [<ffffffff812ab281>] ? __sock_sendmsg_nosec+0x5f/0x6a
Jun 12 17:32:48 sd70 kernel: [ 1201.579022]  [<ffffffff811c22f9>] ? cpumask_any_but+0x29/0x36
Jun 12 17:32:48 sd70 kernel: [ 1201.579092]  [<ffffffff81037e67>] ? flush_tlb_page+0x5f/0x7f
Jun 12 17:32:48 sd70 kernel: [ 1201.579161]  [<ffffffff81036e1b>] ? ptep_set_access_flags+0x3f/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.579230]  [<ffffffff810eceee>] ? do_wp_page+0x2a1/0x5f9
Jun 12 17:32:48 sd70 kernel: [ 1201.579299]  [<ffffffff810edc8d>] ? handle_pte_fault+0x17e/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.579363]  [<ffffffff810ede10>] ? handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.579432]  [<ffffffff81382033>] ? do_page_fault+0x34a/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.579503]  [<ffffffff8100e71d>] ? __switch_to+0x227/0x380
Jun 12 17:32:48 sd70 kernel: [ 1201.579571]  [<ffffffff8104c86f>] ? __set_task_blocked+0x60/0x69
Jun 12 17:32:48 sd70 kernel: [ 1201.579641]  [<ffffffff8104c8a9>] ? set_current_blocked+0x31/0x4c
Jun 12 17:32:48 sd70 kernel: [ 1201.579710]  [<ffffffff811205d9>] sys_ppoll+0xbb/0x144
Jun 12 17:32:48 sd70 kernel: [ 1201.579777]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.579841] xfsbufd/sda     S ffff8805ec6989a8     0  1977      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.579986]  ffff8805bc65dda0 0000000000000046 ffff8805eaaed138 0000000000000000
Jun 12 17:32:48 sd70 kernel: [ 1201.580218]  ffff880591aff2e0 ffff8805bc65c010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.580449]  ffff8805bc65dfd8 0000000000013340 ffff8805bc65dfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.580680] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.580740]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.580807]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.580881]  [<ffffffffa02f4896>] ? xfs_buf_delwri_split+0x21/0x164 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.580950]  [<ffffffff8137d091>] schedule_timeout_interruptible+0x19/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.581028]  [<ffffffffa02f5e47>] xfsbufd+0xc4/0x14e [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.581102]  [<ffffffffa02f5d83>] ? xfs_free_buftarg+0x4c/0x4c [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.581168]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.581234]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.581304]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.581376]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.581442] xfs-data/sda    S ffff8805ec69d3e8     0  1978      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.581587]  ffff880602b5fe50 0000000000000046 ffff880602b5fdf0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.587326]  ffff880602b5fdf0 ffff880602b5e010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.587559]  ffff880602b5ffd8 0000000000013340 ffff880602b5ffd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.587793] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.587853]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.587922]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.587990]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.588058]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.588127]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.588195]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.588262]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.588331]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.588403]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.588469] xfs-conv/sda    S ffff8805ec69bfa8     0  1979      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.588614]  ffff8805bc9c5e50 0000000000000046 ffff8805bc9c5df0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.588848]  ffff8805bc9c5df0 ffff8805bc9c4010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.589082]  ffff8805bc9c5fd8 0000000000013340 ffff8805bc9c5fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.589316] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.589377]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.589446]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.589513]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.589581]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.589650]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.589718]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.589785]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.589855]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.589927]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.589993] xfsaild/sda     S ffff8805ec6b3228     0  1980      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.590138]  ffff8805fd599e10 0000000000000046 ffff8805bc8ac000 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.590371]  ffff8805fd599dc0 ffff8805fd598010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.590605]  ffff8805fd599fd8 0000000000013340 ffff8805fd599fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.590838] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.590898]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.590965]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.591034]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.591110]  [<ffffffffa034fa4b>] xfsaild+0x67/0xb0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.591185]  [<ffffffffa034f9e4>] ? xfsaild_push+0x540/0x540 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.591255]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.591321]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.591386]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.591458]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.591525] xfsbufd/sdb     S ffff8805ec6b6168     0  1983      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.591665]  ffff8805bc91fda0 0000000000000046 ffff8805eaaee138 0000000000000000
Jun 12 17:32:48 sd70 kernel: [ 1201.591898]  ffff8805eb592c00 ffff8805bc91e010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.592176]  ffff8805bc91ffd8 0000000000013340 ffff8805bc91ffd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.592409] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.592469]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.592536]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.592604]  [<ffffffff8137d091>] schedule_timeout_interruptible+0x19/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.592683]  [<ffffffffa02f5e47>] xfsbufd+0xc4/0x14e [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.592757]  [<ffffffffa02f5d83>] ? xfs_free_buftarg+0x4c/0x4c [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.592827]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.592894]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.592964]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.593036]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.593102] xfs-data/sdb    S ffff8805ec6b6ee8     0  1984      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.593247]  ffff8805bc9cfe50 0000000000000046 ffff8805bc9cfdf0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.593480]  ffff8805bc9cfdf0 ffff8805bc9ce010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.593714]  ffff8805bc9cffd8 0000000000013340 ffff8805bc9cffd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.593947] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.594008]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.594077]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.594144]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.594213]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.594282]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.594350]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.594416]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.594486]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.594558]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.594625] xfs-conv/sdb    S ffff8805ec6b3fa8     0  1985      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.594769]  ffff8805eabbde50 0000000000000046 ffff8805eabbddf0 ffffffff8106ed25
Jun 12 17:32:48 sd70 kernel: [ 1201.595002]  ffff8805eabbddf0 ffff8805eabbc010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.595235]  ffff8805eabbdfd8 0000000000013340 ffff8805eabbdfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.595469] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.595529]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.595598]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.595665]  [<ffffffff81055677>] rescuer_thread+0x233/0x238
Jun 12 17:32:48 sd70 kernel: [ 1201.595733]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.595802]  [<ffffffff81055444>] ? worker_thread+0x264/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.595871]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.595938]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.596007]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.596083]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.596151] xfsaild/sdb     S ffff8805ec6b02e8     0  1986      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.596297]  ffff8805bcbfbe10 0000000000000046 ffff8805bcbfbdb0 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.596534]  ffff8805bcbfbdc0 ffff8805bcbfa010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.596770]  ffff8805bcbfbfd8 0000000000013340 ffff8805bcbfbfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.597007] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.597068]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.597136]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.597205]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.597285]  [<ffffffffa034fa4b>] xfsaild+0x67/0xb0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.597364]  [<ffffffffa034f9e4>] ? xfsaild_push+0x540/0x540 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.597435]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.597503]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.597574]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.597647]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.597714] ntpd            S ffff8806039238e8     0  2012      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.597860]  ffff8805ec065918 0000000000000082 ffff8805ec0658d8 ffffffff812d56d6
Jun 12 17:32:48 sd70 kernel: [ 1201.598098]  0000000000000000 ffff8805ec064010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.598334]  ffff8805ec065fd8 0000000000013340 ffff8805ec065fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.598571] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.598633]  [<ffffffff812d56d6>] ? sch_direct_xmit+0x93/0x14a
Jun 12 17:32:48 sd70 kernel: [ 1201.598703]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.598771]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.598844]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.598914]  [<ffffffff8111f52c>] ? __pollwait+0xbe/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.598982]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.599054]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.599124]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.599194]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.599262]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.599331]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.599400]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.599468]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.599537]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.599606]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.599675]  [<ffffffff8105b030>] ? do_schedule_next_timer+0xaa/0xb5
Jun 12 17:32:48 sd70 kernel: [ 1201.599747]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.599817]  [<ffffffff8104caf8>] ? block_sigmask+0x41/0x43
Jun 12 17:32:48 sd70 kernel: [ 1201.599946]  [<ffffffff8100f455>] ? handle_signal+0x16a/0x191
Jun 12 17:32:48 sd70 kernel: [ 1201.600016]  [<ffffffff8100f4cc>] ? do_signal+0x50/0x103
Jun 12 17:32:48 sd70 kernel: [ 1201.600087]  [<ffffffff81016e76>] ? init_fpu+0x82/0x8f
Jun 12 17:32:48 sd70 kernel: [ 1201.600155]  [<ffffffff81017b28>] ? restore_i387_xstate+0x6d/0x17d
Jun 12 17:32:48 sd70 kernel: [ 1201.600224]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.600292]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.600360] nrpe            S ffff8805ec69b228     0  2123      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.600505]  ffff8805fd4b3918 0000000000000086 ffffea0006444300 0000000000000282
Jun 12 17:32:48 sd70 kernel: [ 1201.600738]  ffff8805fd4b38c8 ffff8805fd4b2010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.600972]  ffff8805fd4b3fd8 0000000000013340 ffff8805fd4b3fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.601206] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.601266]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.601332]  [<ffffffff8137d646>] schedule_hrtimeout_range_clock+0xe8/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.601404]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.601472]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.601543]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.601613]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.601682]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.601750]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.601820]  [<ffffffff811c87cb>] ? put_dec+0x20/0x3c
Jun 12 17:32:48 sd70 kernel: [ 1201.601886]  [<ffffffff811c89b7>] ? number+0x157/0x26b
Jun 12 17:32:48 sd70 kernel: [ 1201.601955]  [<ffffffff8102b884>] ? physflat_send_IPI_mask+0x9/0xb
Jun 12 17:32:48 sd70 kernel: [ 1201.602025]  [<ffffffff81037d4b>] ? flush_tlb_others_ipi+0xbe/0xf7
Jun 12 17:32:48 sd70 kernel: [ 1201.602095]  [<ffffffff81108d59>] ? __slab_free+0xe1/0x22d
Jun 12 17:32:48 sd70 kernel: [ 1201.602163]  [<ffffffff810aa4e4>] ? force_quiescent_state+0x176/0x18d
Jun 12 17:32:48 sd70 kernel: [ 1201.602234]  [<ffffffff81107557>] ? virt_to_head_page+0x9/0x2c
Jun 12 17:32:48 sd70 kernel: [ 1201.602303]  [<ffffffff811099d3>] ? kmem_cache_free+0x1b/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.602372]  [<ffffffff812aa469>] ? sock_destroy_inode+0x2e/0x32
Jun 12 17:32:48 sd70 kernel: [ 1201.602442]  [<ffffffff81124855>] ? destroy_inode+0x3b/0x54
Jun 12 17:32:48 sd70 kernel: [ 1201.602510]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.602579]  [<ffffffff8112914f>] ? mntput+0x25/0x27
Jun 12 17:32:48 sd70 kernel: [ 1201.602646]  [<ffffffff81112206>] ? __fput+0x1b6/0x1c5
Jun 12 17:32:48 sd70 kernel: [ 1201.602713]  [<ffffffff8110f02b>] ? put_unused_fd+0x26/0x4f
Jun 12 17:32:48 sd70 kernel: [ 1201.602782]  [<ffffffff81014fc6>] ? read_tsc+0x9/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.602849]  [<ffffffff8107a59a>] ? timekeeping_get_ns+0x15/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.608556]  [<ffffffff8107b707>] ? ktime_get_ts+0x80/0x89
Jun 12 17:32:48 sd70 kernel: [ 1201.608625]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.608693]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.608763] master          S ffff8805ec69eee8     0  2227      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.608910]  ffff8805bcbf5d78 0000000000000086 ffff8805bcbf5d28 ffffffff812ab41e
Jun 12 17:32:48 sd70 kernel: [ 1201.609146]  ffff8805bcbf5d28 ffff8805bcbf4010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.609383]  ffff8805bcbf5fd8 0000000000013340 ffff8805bcbf5fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.609619] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.609680]  [<ffffffff812ab41e>] ? __sock_recvmsg+0x49/0x54
Jun 12 17:32:48 sd70 kernel: [ 1201.609750]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.609818]  [<ffffffff8137d646>] schedule_hrtimeout_range_clock+0xe8/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.609891]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.609960]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.610031]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.610103]  [<ffffffff81143af1>] ep_poll+0x17a/0x2e0
Jun 12 17:32:48 sd70 kernel: [ 1201.610172]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
Jun 12 17:32:48 sd70 kernel: [ 1201.610242]  [<ffffffff81143ce5>] sys_epoll_wait+0x8e/0xa7
Jun 12 17:32:48 sd70 kernel: [ 1201.610311]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.610381] pickup          S ffff880603925aa8     0  2228   2227 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.610528]  ffff8805f3083d78 0000000000000086 ffff8805f3083d28 ffffffff812ab2c9
Jun 12 17:32:48 sd70 kernel: [ 1201.610764]  ffff8805f3083d18 ffff8805f3082010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.611001]  ffff8805f3083fd8 0000000000013340 ffff8805f3083fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.611238] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.611300]  [<ffffffff812ab2c9>] ? __sock_sendmsg+0x3d/0x48
Jun 12 17:32:48 sd70 kernel: [ 1201.611369]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.611437]  [<ffffffff8137d646>] schedule_hrtimeout_range_clock+0xe8/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.611509]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.611575]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.611642]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.611713]  [<ffffffff81143af1>] ep_poll+0x17a/0x2e0
Jun 12 17:32:48 sd70 kernel: [ 1201.611783]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
Jun 12 17:32:48 sd70 kernel: [ 1201.611853]  [<ffffffff81143ce5>] sys_epoll_wait+0x8e/0xa7
Jun 12 17:32:48 sd70 kernel: [ 1201.611925]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.611993] qmgr            S ffff8806039275a8     0  2229   2227 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.612143]  ffff8805ea7b9d78 0000000000000086 ffff8805ea7b9d18 ffffffff8110991e
Jun 12 17:32:48 sd70 kernel: [ 1201.612381]  ffff8805ea7b9d18 ffff8805ea7b8010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.612618]  ffff8805ea7b9fd8 0000000000013340 ffff8805ea7b9fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.612850] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.612909]  [<ffffffff8110991e>] ? kfree+0x52/0xc9
Jun 12 17:32:48 sd70 kernel: [ 1201.612977]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.613051]  [<ffffffff8137d646>] schedule_hrtimeout_range_clock+0xe8/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.613125]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.613202]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.613276]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.613348]  [<ffffffff81143af1>] ep_poll+0x17a/0x2e0
Jun 12 17:32:48 sd70 kernel: [ 1201.613412]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
Jun 12 17:32:48 sd70 kernel: [ 1201.613482]  [<ffffffff81143ce5>] sys_epoll_wait+0x8e/0xa7
Jun 12 17:32:48 sd70 kernel: [ 1201.613554]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.613628] snmpd           S ffff8805eb5f38e8     0  2256      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.613775]  ffff8805f30dd918 0000000000000086 0000000000167ae2 0000000000000282
Jun 12 17:32:48 sd70 kernel: [ 1201.614004]  ffff8805f30dd8c8 ffff8805f30dc010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.614246]  ffff8805f30ddfd8 0000000000013340 ffff8805f30ddfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.614482] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.614544]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.614611]  [<ffffffff8137d646>] schedule_hrtimeout_range_clock+0xe8/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.614685]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.614754]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.614826]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.614897]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.614968]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.615037]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.615106]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.615175]  [<ffffffff81120dfb>] ? __d_instantiate+0x91/0xe1
Jun 12 17:32:48 sd70 kernel: [ 1201.615245]  [<ffffffff8112aa16>] ? seq_open+0x3f/0x98
Jun 12 17:32:48 sd70 kernel: [ 1201.615314]  [<ffffffff81109e11>] ? kmem_cache_alloc_trace+0xd6/0xe8
Jun 12 17:32:48 sd70 kernel: [ 1201.615386]  [<ffffffff8112aa16>] ? seq_open+0x3f/0x98
Jun 12 17:32:48 sd70 kernel: [ 1201.615454]  [<ffffffff8112aa4e>] ? seq_open+0x77/0x98
Jun 12 17:32:48 sd70 kernel: [ 1201.615523]  [<ffffffff810e4c87>] ? vmstat_open+0x13/0x15
Jun 12 17:32:48 sd70 kernel: [ 1201.615592]  [<ffffffff811c87cb>] ? put_dec+0x20/0x3c
Jun 12 17:32:48 sd70 kernel: [ 1201.615660]  [<ffffffff811c89b7>] ? number+0x157/0x26b
Jun 12 17:32:48 sd70 kernel: [ 1201.615729]  [<ffffffff8110f50a>] ? __dentry_open+0x196/0x287
Jun 12 17:32:48 sd70 kernel: [ 1201.615799]  [<ffffffff8112914f>] ? mntput+0x25/0x27
Jun 12 17:32:48 sd70 kernel: [ 1201.615867]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.615994]  [<ffffffff81124855>] ? destroy_inode+0x3b/0x54
Jun 12 17:32:48 sd70 kernel: [ 1201.616081]  [<ffffffff810aad21>] ? call_rcu_sched+0x12/0x14
Jun 12 17:32:48 sd70 kernel: [ 1201.616150]  [<ffffffff81014fc6>] ? read_tsc+0x9/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.616218]  [<ffffffff8107a59a>] ? timekeeping_get_ns+0x15/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.616289]  [<ffffffff8107b707>] ? ktime_get_ts+0x80/0x89
Jun 12 17:32:48 sd70 kernel: [ 1201.616358]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.616426]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.616496] xinetd          S ffff8805bcaf02e8     0  4075      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.616643]  ffff8805ed20d918 0000000000000082 0000000000000002 ffff88061fffd900
Jun 12 17:32:48 sd70 kernel: [ 1201.616881]  ffff8805ed20d938 ffff8805ed20c010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.617119]  ffff8805ed20dfd8 0000000000013340 ffff8805ed20dfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.617358] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.617419]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.617487]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.617562]  [<ffffffff81101489>] ? alloc_pages_current+0xc5/0xe6
Jun 12 17:32:48 sd70 kernel: [ 1201.617633]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.617703]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.617774]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.617845]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.617914]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.617983]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.618052]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.618122]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.618190]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.618259]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.618327]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.618396]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.618465]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.618534]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.618605]  [<ffffffff81033e09>] ? __bad_area_nosemaphore+0x87/0x1e1
Jun 12 17:32:48 sd70 kernel: [ 1201.618676]  [<ffffffff812abc21>] ? sockfd_lookup_light+0x1b/0x53
Jun 12 17:32:48 sd70 kernel: [ 1201.618747]  [<ffffffff812ac154>] ? sys_sendto+0x13f/0x16c
Jun 12 17:32:48 sd70 kernel: [ 1201.618817]  [<ffffffff810440d7>] ? local_bh_enable+0xd/0xf
Jun 12 17:32:48 sd70 kernel: [ 1201.618886]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.618955]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.619025] rsyslogd        S ffff8805bcbbc280     0  4141      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.619172]  ffff8805bcbcf918 0000000000000082 ffff8805ed243d78 ffff88061fc4fea0
Jun 12 17:32:48 sd70 kernel: [ 1201.619410]  ffff8805bcbcf8c8 ffff8805bcbce010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.619648]  ffff8805bcbcffd8 0000000000013340 ffff8805bcbcffd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.619876] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.619941]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.620007]  [<ffffffff8137d646>] schedule_hrtimeout_range_clock+0xe8/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.620084]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.620153]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.620224]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.620293]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.620359]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.620424]  [<ffffffff81035301>] ? kernel_map_pages+0xb0/0xbd
Jun 12 17:32:48 sd70 kernel: [ 1201.620495]  [<ffffffff810d350e>] ? prep_new_page+0xf6/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.620566]  [<ffffffff8103495a>] ? static_protections+0x52/0xcd
Jun 12 17:32:48 sd70 kernel: [ 1201.620636]  [<ffffffff81034fec>] ? __change_page_attr+0x135/0x180
Jun 12 17:32:48 sd70 kernel: [ 1201.620706]  [<ffffffff8103506c>] ? __change_page_attr_set_clr+0x35/0x80
Jun 12 17:32:48 sd70 kernel: [ 1201.620777]  [<ffffffff81035301>] ? kernel_map_pages+0xb0/0xbd
Jun 12 17:32:48 sd70 kernel: [ 1201.620845]  [<ffffffff810d34bc>] ? prep_new_page+0xa4/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.620909]  [<ffffffff810d4a84>] ? __alloc_pages_nodemask+0x12a/0x1ae
Jun 12 17:32:48 sd70 kernel: [ 1201.620980]  [<ffffffff810d438c>] ? get_page_from_freelist+0x51d/0x5d9
Jun 12 17:32:48 sd70 kernel: [ 1201.621051]  [<ffffffff810cda2c>] ? find_get_page+0x43/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.621119]  [<ffffffff810cf4bd>] ? filemap_fault+0x199/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.621184]  [<ffffffff810cdbc8>] ? unlock_page+0x25/0x29
Jun 12 17:32:48 sd70 kernel: [ 1201.621252]  [<ffffffff810ecc12>] ? __do_fault+0x3a7/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.621321]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.621389]  [<ffffffff810ede10>] ? handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.621458]  [<ffffffff81382033>] ? do_page_fault+0x34a/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.621528]  [<ffffffff81062b71>] ? check_preempt_curr+0x2d/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.621598]  [<ffffffff81014fc6>] ? read_tsc+0x9/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.621665]  [<ffffffff8107a59a>] ? timekeeping_get_ns+0x15/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.621735]  [<ffffffff8107b707>] ? ktime_get_ts+0x80/0x89
Jun 12 17:32:48 sd70 kernel: [ 1201.621803]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.621870]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.621939] rsyslogd        S ffff8805ec641de8     0  4143      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.622084]  ffff8805bbc8b918 0000000000000082 00000000005ed338 80000005ed338160
Jun 12 17:32:48 sd70 kernel: [ 1201.622318]  ffff8805bbc8b8e8 ffff8805bbc8a010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.622552]  ffff8805bbc8bfd8 0000000000013340 ffff8805bbc8bfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.622786] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.622846]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.622913]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.622985]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.623054]  [<ffffffff8111f52c>] ? __pollwait+0xbe/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.623121]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.623191]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.623256]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.623324]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.623392]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.623460]  [<ffffffff8132c619>] ? unix_dgram_recvmsg+0x329/0x33b
Jun 12 17:32:48 sd70 kernel: [ 1201.623530]  [<ffffffff810d4a84>] ? __alloc_pages_nodemask+0x12a/0x1ae
Jun 12 17:32:48 sd70 kernel: [ 1201.623601]  [<ffffffff812aa5f7>] ? __sock_recvmsg_nosec+0x65/0x6e
Jun 12 17:32:48 sd70 kernel: [ 1201.629376]  [<ffffffff811c2332>] ? cpumask_next_and+0x2c/0x39
Jun 12 17:32:48 sd70 kernel: [ 1201.629447]  [<ffffffff8106ec4e>] ? enqueue_entity+0x121/0x1cd
Jun 12 17:32:48 sd70 kernel: [ 1201.629517]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.629594]  [<ffffffff810676d8>] ? enqueue_task+0x61/0x68
Jun 12 17:32:48 sd70 kernel: [ 1201.629662]  [<ffffffff81062b82>] ? check_preempt_curr+0x3e/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.629737]  [<ffffffff81069a0d>] ? ttwu_do_wakeup+0x54/0xcb
Jun 12 17:32:48 sd70 kernel: [ 1201.629805]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.629876]  [<ffffffff8108383f>] ? futex_wake_op+0x3c2/0x3e6
Jun 12 17:32:48 sd70 kernel: [ 1201.629943]  [<ffffffff8106ed25>] ? enqueue_task_fair+0x2b/0xcf
Jun 12 17:32:48 sd70 kernel: [ 1201.630013]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.630082]  [<ffffffff8137f520>] ? do_device_not_available+0x9/0xb
Jun 12 17:32:48 sd70 kernel: [ 1201.630153]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.630222] rsyslogd        S ffff8805ec644668     0  4144      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.630367]  ffff8805fd5c3e08 0000000000000082 ffff8805fd5c3de8 ffffffff00000001
Jun 12 17:32:48 sd70 kernel: [ 1201.630597]  0000000000000000 ffff8805fd5c2010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.630831]  ffff8805fd5c3fd8 0000000000013340 ffff8805fd5c3fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.631065] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.631125]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.631192]  [<ffffffff8103f7ee>] do_syslog+0x188/0x43f
Jun 12 17:32:48 sd70 kernel: [ 1201.631259]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
Jun 12 17:32:48 sd70 kernel: [ 1201.631327]  [<ffffffff811652b9>] kmsg_read+0x48/0x4f
Jun 12 17:32:48 sd70 kernel: [ 1201.631394]  [<ffffffff8115afde>] proc_reg_read+0x7b/0x94
Jun 12 17:32:48 sd70 kernel: [ 1201.631462]  [<ffffffff8111135f>] vfs_read+0xa6/0x102
Jun 12 17:32:48 sd70 kernel: [ 1201.631529]  [<ffffffff81111474>] sys_read+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.631596]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.631665] rs:main Q:Reg   S ffff8805bc4f38e8     0  7159      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.631811]  ffff880191041d08 0000000000000082 ffff880191041cb8 ffffffff812ab2c9
Jun 12 17:32:48 sd70 kernel: [ 1201.632045]  ffff880191041cf8 ffff880191040010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.632279]  ffff880191041fd8 0000000000013340 ffff880191041fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.632513] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.632573]  [<ffffffff812ab2c9>] ? __sock_sendmsg+0x3d/0x48
Jun 12 17:32:48 sd70 kernel: [ 1201.632641]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.632708]  [<ffffffff8108201b>] futex_wait_queue_me+0xbd/0xdc
Jun 12 17:32:48 sd70 kernel: [ 1201.632778]  [<ffffffff81083018>] futex_wait+0x117/0x25e
Jun 12 17:32:48 sd70 kernel: [ 1201.632846]  [<ffffffff8105eba4>] ? update_rmtp+0x65/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.632914]  [<ffffffff8105f45a>] ? hrtimer_start_range_ns+0xf/0x11
Jun 12 17:32:48 sd70 kernel: [ 1201.632980]  [<ffffffff81084790>] do_futex+0x84/0x117
Jun 12 17:32:48 sd70 kernel: [ 1201.633046]  [<ffffffff81116596>] ? set_task_comm+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.633110]  [<ffffffff8108494a>] sys_futex+0x127/0x141
Jun 12 17:32:48 sd70 kernel: [ 1201.633178]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.633249] rsync           S ffff8805bcaf1068     0  4155      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.633401]  ffff8805ece79918 0000000000000086 ffff8805fd5f7920 ffff8805ea4d2300
Jun 12 17:32:48 sd70 kernel: [ 1201.633638]  ffff8805ece798f8 ffff8805ece78010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.633873]  ffff8805ece79fd8 0000000000013340 ffff8805ece79fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.634108] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.634168]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.634231]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.634303]  [<ffffffff810440c8>] ? local_bh_enable_ip+0x9/0xb
Jun 12 17:32:48 sd70 kernel: [ 1201.634372]  [<ffffffff8105c2ef>] ? add_wait_queue+0x3f/0x44
Jun 12 17:32:48 sd70 kernel: [ 1201.634440]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.634511]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.634581]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.634647]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.634714]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.634783]  [<ffffffff8133e81a>] ? ip6_pol_route_output+0x11/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.634856]  [<ffffffff8135bfed>] ? fib6_rule_action+0x8d/0x156
Jun 12 17:32:48 sd70 kernel: [ 1201.634926]  [<ffffffff8133ae07>] ? __ipv6_addr_label+0x67/0xc5
Jun 12 17:32:48 sd70 kernel: [ 1201.634995]  [<ffffffff810440c8>] ? local_bh_enable_ip+0x9/0xb
Jun 12 17:32:48 sd70 kernel: [ 1201.635065]  [<ffffffff8137ebe3>] ? _raw_read_unlock_bh+0x10/0x12
Jun 12 17:32:48 sd70 kernel: [ 1201.635134]  [<ffffffff8133829d>] ? ipv6_dev_get_saddr+0x294/0x2a8
Jun 12 17:32:48 sd70 kernel: [ 1201.635205]  [<ffffffff810cda2c>] ? find_get_page+0x43/0x65
Jun 12 17:32:48 sd70 kernel: [ 1201.635273]  [<ffffffff810cf4bd>] ? filemap_fault+0x199/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.635342]  [<ffffffff810cdbc8>] ? unlock_page+0x25/0x29
Jun 12 17:32:48 sd70 kernel: [ 1201.635410]  [<ffffffff810ecc12>] ? __do_fault+0x3a7/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.635479]  [<ffffffff8111f2ff>] ? zero_fd_set+0x15/0x19
Jun 12 17:32:48 sd70 kernel: [ 1201.635548]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.635616]  [<ffffffff810ede10>] ? handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.635685]  [<ffffffff81382033>] ? do_page_fault+0x34a/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.635754]  [<ffffffff8135df5a>] ? __inet6_hash+0x153/0x165
Jun 12 17:32:48 sd70 kernel: [ 1201.635824]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.635891]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.635960] login           S ffff8805ec6e7700     0  5969      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.636099]  ffff8805bca03e48 0000000000000086 000000000060a270 ffff880500000065
Jun 12 17:32:48 sd70 kernel: [ 1201.636331]  ffffffff8106ec4e ffff8805bca02010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.636560]  ffff8805bca03fd8 0000000000013340 ffff8805bca03fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.636794] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.636855]  [<ffffffff8106ec4e>] ? enqueue_entity+0x121/0x1cd
Jun 12 17:32:48 sd70 kernel: [ 1201.636924]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.636988]  [<ffffffff810424da>] do_wait+0x1a6/0x21a
Jun 12 17:32:48 sd70 kernel: [ 1201.637053]  [<ffffffff8106a645>] ? wake_up_new_task+0xc2/0xce
Jun 12 17:32:48 sd70 kernel: [ 1201.637122]  [<ffffffff8103dc1c>] ? do_fork+0x193/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.637190]  [<ffffffff810425e7>] sys_wait4+0x99/0xbc
Jun 12 17:32:48 sd70 kernel: [ 1201.637256]  [<ffffffff810406fb>] ? wait_task_continued+0x14c/0x14c
Jun 12 17:32:48 sd70 kernel: [ 1201.637327]  [<ffffffff8137f0e5>] ? page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.637394]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.637463] login           S ffff8805bcaf1de8     0  5970      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.637607]  ffff8805ecfcde48 0000000000000082 000000000060a270 ffff880500000065
Jun 12 17:32:48 sd70 kernel: [ 1201.637841]  ffffffff8106ec4e ffff8805ecfcc010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.638074]  ffff8805ecfcdfd8 0000000000013340 ffff8805ecfcdfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.638308] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.638368]  [<ffffffff8106ec4e>] ? enqueue_entity+0x121/0x1cd
Jun 12 17:32:48 sd70 kernel: [ 1201.638437]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.638504]  [<ffffffff810424da>] do_wait+0x1a6/0x21a
Jun 12 17:32:48 sd70 kernel: [ 1201.638571]  [<ffffffff8106a645>] ? wake_up_new_task+0xc2/0xce
Jun 12 17:32:48 sd70 kernel: [ 1201.638639]  [<ffffffff8103dc1c>] ? do_fork+0x193/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.638707]  [<ffffffff810425e7>] sys_wait4+0x99/0xbc
Jun 12 17:32:48 sd70 kernel: [ 1201.638774]  [<ffffffff810406fb>] ? wait_task_continued+0x14c/0x14c
Jun 12 17:32:48 sd70 kernel: [ 1201.638844]  [<ffffffff8137f0e5>] ? page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.638911]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.638980] bash            S ffff8805bcaf38e8     0  5971   5969 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.639124]  ffff8805ed267e48 0000000000000086 0000000000ef4380 ffff880500000065
Jun 12 17:32:48 sd70 kernel: [ 1201.639357]  ffff8805ed267de8 ffff8805ed266010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.639591]  ffff8805ed267fd8 0000000000013340 ffff8805ed267fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.639825] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.639885]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.639952]  [<ffffffff810424da>] do_wait+0x1a6/0x21a
Jun 12 17:32:48 sd70 kernel: [ 1201.640019]  [<ffffffff8106a645>] ? wake_up_new_task+0xc2/0xce
Jun 12 17:32:48 sd70 kernel: [ 1201.640088]  [<ffffffff8103dc1c>] ? do_fork+0x193/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.640155]  [<ffffffff810425e7>] sys_wait4+0x99/0xbc
Jun 12 17:32:48 sd70 kernel: [ 1201.640223]  [<ffffffff8104c8a9>] ? set_current_blocked+0x31/0x4c
Jun 12 17:32:48 sd70 kernel: [ 1201.640292]  [<ffffffff810406fb>] ? wait_task_continued+0x14c/0x14c
Jun 12 17:32:48 sd70 kernel: [ 1201.640363]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.640431] bash            S ffff8805ec643228     0  5972   5970 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.640576]  ffff8805ece1bcf8 0000000000000086 ffff8805ecb8b378 000000000000009e
Jun 12 17:32:48 sd70 kernel: [ 1201.640809]  ffff8805ece1bce8 ffff8805ece1a010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.641043]  ffff8805ece1bfd8 0000000000013340 ffff8805ece1bfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.641276] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.641337]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.641403]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.641472]  [<ffffffff81053f2f>] ? flush_work+0x1a/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.641542]  [<ffffffff812458ca>] n_tty_read+0x49b/0x73f
Jun 12 17:32:48 sd70 kernel: [ 1201.641610]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
Jun 12 17:32:48 sd70 kernel: [ 1201.641679]  [<ffffffff812404e9>] tty_read+0x81/0xc2
Jun 12 17:32:48 sd70 kernel: [ 1201.641746]  [<ffffffff8111135f>] vfs_read+0xa6/0x102
Jun 12 17:32:48 sd70 kernel: [ 1201.641813]  [<ffffffff8104cab3>] ? sigprocmask+0x5e/0x62
Jun 12 17:32:48 sd70 kernel: [ 1201.641880]  [<ffffffff81111474>] sys_read+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.641947]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.642015] sshd            S ffff8805ebdd6828     0  6073   1969 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.642159]  ffff880601343918 0000000000000082 0000000000000092 ffff8805eb5dc000
Jun 12 17:32:48 sd70 kernel: [ 1201.642393]  ffff8806013438f8 ffff880601342010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.642627]  ffff880601343fd8 0000000000013340 ffff880601343fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.642860] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.642920]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.642987]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.643059]  [<ffffffff8106560e>] ? __wake_up+0x3f/0x48
Jun 12 17:32:48 sd70 kernel: [ 1201.643127]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.643197]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.643267]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.643335]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.643403]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.643472]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.643539]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.643607]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.643675]  [<ffffffff8131031d>] ? inet_sendmsg+0x8f/0x98
Jun 12 17:32:48 sd70 kernel: [ 1201.643743]  [<ffffffff812ab281>] ? __sock_sendmsg_nosec+0x5f/0x6a
Jun 12 17:32:48 sd70 kernel: [ 1201.643813]  [<ffffffff812ab2c9>] ? __sock_sendmsg+0x3d/0x48
Jun 12 17:32:48 sd70 kernel: [ 1201.643882]  [<ffffffff812ab3c5>] ? sock_aio_write+0xf1/0x101
Jun 12 17:32:48 sd70 kernel: [ 1201.643951]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.644020]  [<ffffffff811107e3>] ? do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.644089]  [<ffffffff811400a9>] ? fsnotify+0x255/0x285
Jun 12 17:32:48 sd70 kernel: [ 1201.644157]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.644224]  [<ffffffff81111292>] ? sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.644291]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.644360] bash            S ffff8805ebdd6168     0  6075   6073 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.644504]  ffff8805bc533cf8 0000000000000086 0000000000000001 ffff8805ed397950
Jun 12 17:32:48 sd70 kernel: [ 1201.650205]  0000000000000000 ffff8805bc532010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.650439]  ffff8805bc533fd8 0000000000013340 ffff8805bc533fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.650672] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.650732]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.650799]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.650867]  [<ffffffff81053f2f>] ? flush_work+0x1a/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.650935]  [<ffffffff81241913>] ? tty_wakeup+0x56/0x5b
Jun 12 17:32:48 sd70 kernel: [ 1201.651002]  [<ffffffff8124904f>] ? pty_write+0x4c/0x56
Jun 12 17:32:48 sd70 kernel: [ 1201.651070]  [<ffffffff812458ca>] n_tty_read+0x49b/0x73f
Jun 12 17:32:48 sd70 kernel: [ 1201.651138]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
Jun 12 17:32:48 sd70 kernel: [ 1201.651207]  [<ffffffff812404e9>] tty_read+0x81/0xc2
Jun 12 17:32:48 sd70 kernel: [ 1201.651273]  [<ffffffff8111135f>] vfs_read+0xa6/0x102
Jun 12 17:32:48 sd70 kernel: [ 1201.651340]  [<ffffffff8104cab3>] ? sigprocmask+0x5e/0x62
Jun 12 17:32:48 sd70 kernel: [ 1201.651407]  [<ffffffff81111474>] sys_read+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.651474]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.651542] diloadfromspool D ffff8805ebdd38e8     0  6083      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.651687]  ffff8805fd593b58 0000000000000082 ffff8805fd593b48 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.651920]  ffff8805ec918000 ffff8805fd592010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.652154]  ffff8805fd593fd8 0000000000013340 ffff8805fd593fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.652387] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.652448]  [<ffffffff810cdb2b>] ? sleep_on_page+0xd/0xd
Jun 12 17:32:48 sd70 kernel: [ 1201.652515]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.652582]  [<ffffffff8137e338>] io_schedule+0x87/0xca
Jun 12 17:32:48 sd70 kernel: [ 1201.652650]  [<ffffffff810cdb34>] sleep_on_page_killable+0x9/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.652719]  [<ffffffff8137d1d4>] __wait_on_bit+0x43/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.652787]  [<ffffffff810cdc84>] wait_on_page_bit_killable+0x6f/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.652858]  [<ffffffff8105c003>] ? autoremove_wake_function+0x34/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.652929]  [<ffffffff810cdd82>] __lock_page_or_retry+0x40/0x8e
Jun 12 17:32:48 sd70 kernel: [ 1201.652998]  [<ffffffff810cf4d9>] filemap_fault+0x1b5/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.653066]  [<ffffffff810ec910>] __do_fault+0xa5/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.653133]  [<ffffffff81107557>] ? virt_to_head_page+0x9/0x2c
Jun 12 17:32:48 sd70 kernel: [ 1201.653202]  [<ffffffff811099d3>] ? kmem_cache_free+0x1b/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.653271]  [<ffffffff810edbe9>] handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.653340]  [<ffffffff810ede10>] handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.653409]  [<ffffffff8138204d>] do_page_fault+0x364/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.653477]  [<ffffffff8114a984>] ? fcntl_setlk+0x178/0x18a
Jun 12 17:32:48 sd70 kernel: [ 1201.653546]  [<ffffffff8111dfc9>] ? do_fcntl+0x19b/0x312
Jun 12 17:32:48 sd70 kernel: [ 1201.653614]  [<ffffffff8137f0e5>] page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.653680] diloadfromspool D ffff8805ebdd4d28     0  6084      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.653825]  ffff8805c4d8db58 0000000000000086 ffff8805c4d8db48 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.654059]  ffff8805ec918000 ffff8805c4d8c010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.654293]  ffff8805c4d8dfd8 0000000000013340 ffff8805c4d8dfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.654527] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.654587]  [<ffffffff810cdb2b>] ? sleep_on_page+0xd/0xd
Jun 12 17:32:48 sd70 kernel: [ 1201.654655]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.654721]  [<ffffffff8137e338>] io_schedule+0x87/0xca
Jun 12 17:32:48 sd70 kernel: [ 1201.654789]  [<ffffffff810cdb34>] sleep_on_page_killable+0x9/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.654858]  [<ffffffff8137d1d4>] __wait_on_bit+0x43/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.654926]  [<ffffffff810cdc84>] wait_on_page_bit_killable+0x6f/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.654997]  [<ffffffff8105c003>] ? autoremove_wake_function+0x34/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.655068]  [<ffffffff810cdd82>] __lock_page_or_retry+0x40/0x8e
Jun 12 17:32:48 sd70 kernel: [ 1201.655137]  [<ffffffff810cf4d9>] filemap_fault+0x1b5/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.655206]  [<ffffffff810ec910>] __do_fault+0xa5/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.655273]  [<ffffffff81107557>] ? virt_to_head_page+0x9/0x2c
Jun 12 17:32:48 sd70 kernel: [ 1201.655338]  [<ffffffff811099d3>] ? kmem_cache_free+0x1b/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.655406]  [<ffffffff810edbe9>] handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.655475]  [<ffffffff810ede10>] handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.655543]  [<ffffffff8138204d>] do_page_fault+0x364/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.655611]  [<ffffffff8114a984>] ? fcntl_setlk+0x178/0x18a
Jun 12 17:32:48 sd70 kernel: [ 1201.655680]  [<ffffffff8111dfc9>] ? do_fcntl+0x19b/0x312
Jun 12 17:32:48 sd70 kernel: [ 1201.655747]  [<ffffffff8137f0e5>] page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.655813] diloadfromspool D ffff8805ebdd1728     0  6085      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.655959]  ffff8805fd5cdb58 0000000000000086 ffff8805fd5cdb48 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.656192]  ffff8805ec918000 ffff8805fd5cc010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.656425]  ffff8805fd5cdfd8 0000000000013340 ffff8805fd5cdfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.656658] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.656719]  [<ffffffff810cdb2b>] ? sleep_on_page+0xd/0xd
Jun 12 17:32:48 sd70 kernel: [ 1201.656786]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.656853]  [<ffffffff8137e338>] io_schedule+0x87/0xca
Jun 12 17:32:48 sd70 kernel: [ 1201.656921]  [<ffffffff810cdb34>] sleep_on_page_killable+0x9/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.656991]  [<ffffffff8137d1d4>] __wait_on_bit+0x43/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.657059]  [<ffffffff810cdc84>] wait_on_page_bit_killable+0x6f/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.657130]  [<ffffffff8105c003>] ? autoremove_wake_function+0x34/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.657202]  [<ffffffff810cdd82>] __lock_page_or_retry+0x40/0x8e
Jun 12 17:32:48 sd70 kernel: [ 1201.657270]  [<ffffffff810cf4d9>] filemap_fault+0x1b5/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.657339]  [<ffffffff810ec910>] __do_fault+0xa5/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.657407]  [<ffffffff81107557>] ? virt_to_head_page+0x9/0x2c
Jun 12 17:32:48 sd70 kernel: [ 1201.657476]  [<ffffffff811099d3>] ? kmem_cache_free+0x1b/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.657545]  [<ffffffff810edbe9>] handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.657614]  [<ffffffff810ede10>] handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.657682]  [<ffffffff8138204d>] do_page_fault+0x364/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.657750]  [<ffffffff8114a984>] ? fcntl_setlk+0x178/0x18a
Jun 12 17:32:48 sd70 kernel: [ 1201.657818]  [<ffffffff8111dfc9>] ? do_fcntl+0x19b/0x312
Jun 12 17:32:48 sd70 kernel: [ 1201.657886]  [<ffffffff8137f0e5>] page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.657952] diloadfromspool D ffff8805ebdd53e8     0  6086      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.658096]  ffff8806012b9b58 0000000000000082 ffff8806012b9b48 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.658330]  ffff8805ec918000 ffff8806012b8010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.658563]  ffff8806012b9fd8 0000000000013340 ffff8806012b9fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.658797] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.658857]  [<ffffffff810cdb2b>] ? sleep_on_page+0xd/0xd
Jun 12 17:32:48 sd70 kernel: [ 1201.658925]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.658992]  [<ffffffff8137e338>] io_schedule+0x87/0xca
Jun 12 17:32:48 sd70 kernel: [ 1201.659059]  [<ffffffff810cdb34>] sleep_on_page_killable+0x9/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.659130]  [<ffffffff8137d1d4>] __wait_on_bit+0x43/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.659198]  [<ffffffff810cdc84>] wait_on_page_bit_killable+0x6f/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.659268]  [<ffffffff8105c003>] ? autoremove_wake_function+0x34/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.659339]  [<ffffffff810cdd82>] __lock_page_or_retry+0x40/0x8e
Jun 12 17:32:48 sd70 kernel: [ 1201.659408]  [<ffffffff810cf4d9>] filemap_fault+0x1b5/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.659477]  [<ffffffff810ec910>] __do_fault+0xa5/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.659544]  [<ffffffff81107557>] ? virt_to_head_page+0x9/0x2c
Jun 12 17:32:48 sd70 kernel: [ 1201.659613]  [<ffffffff811099d3>] ? kmem_cache_free+0x1b/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.659682]  [<ffffffff810edbe9>] handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.659751]  [<ffffffff810ede10>] handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.659819]  [<ffffffff8138204d>] do_page_fault+0x364/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.659888]  [<ffffffff8114a984>] ? fcntl_setlk+0x178/0x18a
Jun 12 17:32:48 sd70 kernel: [ 1201.659957]  [<ffffffff8111dfc9>] ? do_fcntl+0x19b/0x312
Jun 12 17:32:48 sd70 kernel: [ 1201.660025]  [<ffffffff8137f0e5>] page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.660091] diloadfromspool D ffff8805ebdd1068     0  6087      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.660235]  ffff8805fd5efb58 0000000000000086 ffff8805fd5efb48 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.660469]  ffff8805ec918000 ffff8805fd5ee010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.660702]  ffff8805fd5effd8 0000000000013340 ffff8805fd5effd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.660935] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.660996]  [<ffffffff810cdb2b>] ? sleep_on_page+0xd/0xd
Jun 12 17:32:48 sd70 kernel: [ 1201.661064]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.661130]  [<ffffffff8137e338>] io_schedule+0x87/0xca
Jun 12 17:32:48 sd70 kernel: [ 1201.661198]  [<ffffffff810cdb34>] sleep_on_page_killable+0x9/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.661267]  [<ffffffff8137d1d4>] __wait_on_bit+0x43/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.661335]  [<ffffffff810cdc84>] wait_on_page_bit_killable+0x6f/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.661406]  [<ffffffff8105c003>] ? autoremove_wake_function+0x34/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.661477]  [<ffffffff810cdd82>] __lock_page_or_retry+0x40/0x8e
Jun 12 17:32:48 sd70 kernel: [ 1201.661547]  [<ffffffff810cf4d9>] filemap_fault+0x1b5/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.661615]  [<ffffffff810ec910>] __do_fault+0xa5/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.661682]  [<ffffffff81107557>] ? virt_to_head_page+0x9/0x2c
Jun 12 17:32:48 sd70 kernel: [ 1201.661751]  [<ffffffff811099d3>] ? kmem_cache_free+0x1b/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.661820]  [<ffffffff810edbe9>] handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.661889]  [<ffffffff810ede10>] handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.661958]  [<ffffffff8138204d>] do_page_fault+0x364/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.662026]  [<ffffffff8114a984>] ? fcntl_setlk+0x178/0x18a
Jun 12 17:32:48 sd70 kernel: [ 1201.662094]  [<ffffffff8111dfc9>] ? do_fcntl+0x19b/0x312
Jun 12 17:32:48 sd70 kernel: [ 1201.662162]  [<ffffffff8137f0e5>] page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.662228] diloadfromspool D ffff8805ebdd2b68     0  6088      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.662372]  ffff8805bc425b58 0000000000000082 ffff8805bc425b48 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.662606]  ffff8805ec918000 ffff8805bc424010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.662839]  ffff8805bc425fd8 0000000000013340 ffff8805bc425fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.663073] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.663134]  [<ffffffff810cdb2b>] ? sleep_on_page+0xd/0xd
Jun 12 17:32:48 sd70 kernel: [ 1201.663201]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.663268]  [<ffffffff8137e338>] io_schedule+0x87/0xca
Jun 12 17:32:48 sd70 kernel: [ 1201.663335]  [<ffffffff810cdb34>] sleep_on_page_killable+0x9/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.663405]  [<ffffffff8137d1d4>] __wait_on_bit+0x43/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.663473]  [<ffffffff810cdc84>] wait_on_page_bit_killable+0x6f/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.663544]  [<ffffffff8105c003>] ? autoremove_wake_function+0x34/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.663615]  [<ffffffff810cdd82>] __lock_page_or_retry+0x40/0x8e
Jun 12 17:32:48 sd70 kernel: [ 1201.663684]  [<ffffffff810cf4d9>] filemap_fault+0x1b5/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.663752]  [<ffffffff810ec910>] __do_fault+0xa5/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.663819]  [<ffffffff81107557>] ? virt_to_head_page+0x9/0x2c
Jun 12 17:32:48 sd70 kernel: [ 1201.663888]  [<ffffffff811099d3>] ? kmem_cache_free+0x1b/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.663957]  [<ffffffff810edbe9>] handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.664026]  [<ffffffff810ede10>] handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.664094]  [<ffffffff8138204d>] do_page_fault+0x364/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.664162]  [<ffffffff8114a984>] ? fcntl_setlk+0x178/0x18a
Jun 12 17:32:48 sd70 kernel: [ 1201.664230]  [<ffffffff8111dfc9>] ? do_fcntl+0x19b/0x312
Jun 12 17:32:48 sd70 kernel: [ 1201.664298]  [<ffffffff8137f0e5>] page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.664365] diloadfromspool D ffff8805ebdd02e8     0  6089      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.664509]  ffff8805e5f8fb58 0000000000000082 ffff8805e5f8fb48 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.664742]  ffff8805ec918000 ffff8805e5f8e010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.664976]  ffff8805e5f8ffd8 0000000000013340 ffff8805e5f8ffd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.670678] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.670742]  [<ffffffff810cdb2b>] ? sleep_on_page+0xd/0xd
Jun 12 17:32:48 sd70 kernel: [ 1201.670811]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.670875]  [<ffffffff8137e338>] io_schedule+0x87/0xca
Jun 12 17:32:48 sd70 kernel: [ 1201.670941]  [<ffffffff810cdb34>] sleep_on_page_killable+0x9/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.671010]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.671085]  [<ffffffff810cdc84>] ? wait_on_page_bit_killable+0x6f/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.671160]  [<ffffffff8105c003>] ? autoremove_wake_function+0x34/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.671231]  [<ffffffff810cdd82>] ? __lock_page_or_retry+0x40/0x8e
Jun 12 17:32:48 sd70 kernel: [ 1201.671302]  [<ffffffff810cf4d9>] ? filemap_fault+0x1b5/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.671373]  [<ffffffff810ec910>] ? __do_fault+0xa5/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.671441]  [<ffffffff81107557>] ? virt_to_head_page+0x9/0x2c
Jun 12 17:32:48 sd70 kernel: [ 1201.671511]  [<ffffffff811099d3>] ? kmem_cache_free+0x1b/0xac
Jun 12 17:32:48 sd70 kernel: [ 1201.671581]  [<ffffffff810edbe9>] ? handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.671652]  [<ffffffff810ede10>] ? handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.671722]  [<ffffffff8138204d>] ? do_page_fault+0x364/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.671792]  [<ffffffff8114a984>] ? fcntl_setlk+0x178/0x18a
Jun 12 17:32:48 sd70 kernel: [ 1201.671922]  [<ffffffff8111dfc9>] ? do_fcntl+0x19b/0x312
Jun 12 17:32:48 sd70 kernel: [ 1201.671991]  [<ffffffff8137f0e5>] ? page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.672058] diloadfromspool D ffff8805ebdd5aa8     0  6090      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.672205]  ffff88060290bb58 0000000000000086 ffff88060290bb48 0000000000000246
Jun 12 17:32:48 sd70 kernel: [ 1201.672441]  ffff8805ec918000 ffff88060290a010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.672678]  ffff88060290bfd8 0000000000013340 ffff88060290bfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.672915] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.672977]  [<ffffffff810cdb2b>] ? sleep_on_page+0xd/0xd
Jun 12 17:32:48 sd70 kernel: [ 1201.673044]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.673108]  [<ffffffff8137e338>] io_schedule+0x87/0xca
Jun 12 17:32:48 sd70 kernel: [ 1201.673173]  [<ffffffff810cdb34>] sleep_on_page_killable+0x9/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.673243]  [<ffffffff8137d1d4>] __wait_on_bit+0x43/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.673315]  [<ffffffff810cdc84>] wait_on_page_bit_killable+0x6f/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.673390]  [<ffffffff8105c003>] ? autoremove_wake_function+0x34/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.673463]  [<ffffffff810cdd82>] __lock_page_or_retry+0x40/0x8e
Jun 12 17:32:48 sd70 kernel: [ 1201.673533]  [<ffffffff810cf4d9>] filemap_fault+0x1b5/0x347
Jun 12 17:32:48 sd70 kernel: [ 1201.673603]  [<ffffffff810ec910>] __do_fault+0xa5/0x3e2
Jun 12 17:32:48 sd70 kernel: [ 1201.673671]  [<ffffffff810edbe9>] handle_pte_fault+0xda/0x1ad
Jun 12 17:32:48 sd70 kernel: [ 1201.673741]  [<ffffffff810370da>] ? pte_alloc_one+0x32/0x38
Jun 12 17:32:48 sd70 kernel: [ 1201.673810]  [<ffffffff810ea619>] ? __pte_alloc+0x7e/0xbc
Jun 12 17:32:48 sd70 kernel: [ 1201.673879]  [<ffffffff810ede10>] handle_mm_fault+0x154/0x168
Jun 12 17:32:48 sd70 kernel: [ 1201.673949]  [<ffffffff8138204d>] do_page_fault+0x364/0x389
Jun 12 17:32:48 sd70 kernel: [ 1201.674018]  [<ffffffff8114a984>] ? fcntl_setlk+0x178/0x18a
Jun 12 17:32:48 sd70 kernel: [ 1201.674087]  [<ffffffff8111dfc9>] ? do_fcntl+0x19b/0x312
Jun 12 17:32:48 sd70 kernel: [ 1201.674156]  [<ffffffff8137f0e5>] page_fault+0x25/0x30
Jun 12 17:32:48 sd70 kernel: [ 1201.674223] bonnie++        D ffff8805bc4f02e8     0  6337      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.674370]  ffff88059b5f59b8 0000000000000086 ffff880601a53bb8 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.674607]  ffff88059b5f5968 ffff88059b5f4010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.674844]  ffff88059b5f5fd8 0000000000013340 ffff88059b5f5fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.675081] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.675142]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.675212]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.675280]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.675350]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.675420]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.675491]  [<ffffffff810d5ecf>] ? bdi_dirty_limit+0x2c/0x8b
Jun 12 17:32:48 sd70 kernel: [ 1201.675561]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
Jun 12 17:32:48 sd70 kernel: [ 1201.675633]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
Jun 12 17:32:48 sd70 kernel: [ 1201.675723]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.675858]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
Jun 12 17:32:48 sd70 kernel: [ 1201.675939]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.676039]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.676117]  [<ffffffffa02f8002>] ? xfs_file_aio_write+0xb/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.676188]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.676257]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
Jun 12 17:32:48 sd70 kernel: [ 1201.676327]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.676394]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.676461]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.676530] bonnie++        D ffff8805bc4f1728     0  6338      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.676675]  ffff880602b659b8 0000000000000082 ffff880601a53838 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.676909]  ffff880602b65968 ffff880602b64010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.677138]  ffff880602b65fd8 0000000000013340 ffff880602b65fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.677371] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.677431]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.677501]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.677567]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.677636]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.677704]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.677774]  [<ffffffff810d5ecf>] ? bdi_dirty_limit+0x2c/0x8b
Jun 12 17:32:48 sd70 kernel: [ 1201.677843]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
Jun 12 17:32:48 sd70 kernel: [ 1201.677914]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
Jun 12 17:32:48 sd70 kernel: [ 1201.678002]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.678073]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
Jun 12 17:32:48 sd70 kernel: [ 1201.678151]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.678239]  [<ffffffff810d85ca>] ? lru_cache_add_lru+0x3c/0x3e
Jun 12 17:32:48 sd70 kernel: [ 1201.678316]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.678387]  [<ffffffff8106805e>] ? task_sched_runtime+0x5e/0x6e
Jun 12 17:32:48 sd70 kernel: [ 1201.678457]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.678525]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
Jun 12 17:32:48 sd70 kernel: [ 1201.678596]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.678661]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.678725]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.678793] bonnie++        D ffff8805bc4f1068     0  6339      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.678938]  ffff8805fd5039b8 0000000000000086 ffff88040c561fb8 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.679172]  ffff8805fd503968 ffff8805fd502010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.679405]  ffff8805fd503fd8 0000000000013340 ffff8805fd503fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.679636] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.679696]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.679765]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.679832]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.679901]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.679970]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.680034]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
Jun 12 17:32:48 sd70 kernel: [ 1201.680105]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
Jun 12 17:32:48 sd70 kernel: [ 1201.680193]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.680263]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
Jun 12 17:32:48 sd70 kernel: [ 1201.680341]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.680430]  [<ffffffff810d85ca>] ? lru_cache_add_lru+0x3c/0x3e
Jun 12 17:32:48 sd70 kernel: [ 1201.680501]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.680573]  [<ffffffff8111071a>] ? do_sync_readv_writev+0xfe/0xfe
Jun 12 17:32:48 sd70 kernel: [ 1201.680643]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.680711]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
Jun 12 17:32:48 sd70 kernel: [ 1201.680782]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.680849]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.680915]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.680979] bonnie++        D ffff8805bc4f2b68     0  6340      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.681123]  ffff880573db19b8 0000000000000086 ffff880601a70ab8 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.681357]  ffff880573db1968 ffff880573db0010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.681591]  ffff880573db1fd8 0000000000013340 ffff880573db1fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.681824] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.681884]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.681954]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.682020]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.682087]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.682152]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.682220]  [<ffffffff810d5ecf>] ? bdi_dirty_limit+0x2c/0x8b
Jun 12 17:32:48 sd70 kernel: [ 1201.682290]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
Jun 12 17:32:48 sd70 kernel: [ 1201.682360]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
Jun 12 17:32:48 sd70 kernel: [ 1201.682449]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.682519]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
Jun 12 17:32:48 sd70 kernel: [ 1201.682597]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.682685]  [<ffffffff810d85ca>] ? lru_cache_add_lru+0x3c/0x3e
Jun 12 17:32:48 sd70 kernel: [ 1201.682762]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.682833]  [<ffffffff8106805e>] ? task_sched_runtime+0x5e/0x6e
Jun 12 17:32:48 sd70 kernel: [ 1201.682902]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.682971]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
Jun 12 17:32:48 sd70 kernel: [ 1201.683042]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.683109]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.683176]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.683244] bonnie++        D ffff8805bc4f24a8     0  6341      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.683389]  ffff880591aa59b8 0000000000000082 ffff88040c561c38 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.683623]  ffff880591aa5968 ffff880591aa4010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.683857]  ffff880591aa5fd8 0000000000013340 ffff880591aa5fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.684090] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.684150]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.684220]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.684286]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.684354]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.684423]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.684492]  [<ffffffff810d5ecf>] ? bdi_dirty_limit+0x2c/0x8b
Jun 12 17:32:48 sd70 kernel: [ 1201.684561]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
Jun 12 17:32:48 sd70 kernel: [ 1201.684632]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
Jun 12 17:32:48 sd70 kernel: [ 1201.684720]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.684791]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
Jun 12 17:32:48 sd70 kernel: [ 1201.684869]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.690486]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.690558]  [<ffffffff8106805e>] ? task_sched_runtime+0x5e/0x6e
Jun 12 17:32:48 sd70 kernel: [ 1201.690628]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.690698]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
Jun 12 17:32:48 sd70 kernel: [ 1201.690769]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.690837]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.690905]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.690975] bonnie++        D ffff8805bc4f09a8     0  6342      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.691122]  ffff880573df59b8 0000000000000082 ffff880601a534b8 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.691357]  ffff880573df5968 ffff880573df4010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.691585]  ffff880573df5fd8 0000000000013340 ffff880573df5fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.691822] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.691883]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.691952]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.692019]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.692087]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.692152]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.692221]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
Jun 12 17:32:48 sd70 kernel: [ 1201.692292]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
Jun 12 17:32:48 sd70 kernel: [ 1201.692379]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.692450]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
Jun 12 17:32:48 sd70 kernel: [ 1201.692528]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.692615]  [<ffffffff810d85ca>] ? lru_cache_add_lru+0x3c/0x3e
Jun 12 17:32:48 sd70 kernel: [ 1201.692691]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.692763]  [<ffffffff8111071a>] ? do_sync_readv_writev+0xfe/0xfe
Jun 12 17:32:48 sd70 kernel: [ 1201.692832]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.692901]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
Jun 12 17:32:48 sd70 kernel: [ 1201.692972]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.693039]  [<ffffffff81111a55>] ? fget_light+0x35/0x94
Jun 12 17:32:48 sd70 kernel: [ 1201.693107]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.693173]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.693242] bonnie++        D ffff8805bc4f3fa8     0  6343      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.693386]  ffff880573de19b8 0000000000000086 ffff8806018e0ab8 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.693620]  ffff880573de1968 ffff880573de0010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.693854]  ffff880573de1fd8 0000000000013340 ffff880573de1fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.694086] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.694146]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.694216]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.694282]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.694351]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.694420]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.694489]  [<ffffffff810d5ecf>] ? bdi_dirty_limit+0x2c/0x8b
Jun 12 17:32:48 sd70 kernel: [ 1201.694557]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
Jun 12 17:32:48 sd70 kernel: [ 1201.694628]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
Jun 12 17:32:48 sd70 kernel: [ 1201.694716]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.694786]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
Jun 12 17:32:48 sd70 kernel: [ 1201.694865]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.694953]  [<ffffffff810d85ca>] ? lru_cache_add_lru+0x3c/0x3e
Jun 12 17:32:48 sd70 kernel: [ 1201.695029]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.695101]  [<ffffffff8111071a>] ? do_sync_readv_writev+0xfe/0xfe
Jun 12 17:32:48 sd70 kernel: [ 1201.695169]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.695234]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
Jun 12 17:32:48 sd70 kernel: [ 1201.695303]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.695368]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.695435]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.695504] bonnie++        D ffff8805bc4f6828     0  6344      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.695648]  ffff880573da99b8 0000000000000082 0000000000000286 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.695882]  ffff880573da9968 ffff880573da8010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.696113]  ffff880573da9fd8 0000000000013340 ffff880573da9fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.696346] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.696407]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.696477]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.696543]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.696612]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.696681]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.696750]  [<ffffffff810d5ecf>] ? bdi_dirty_limit+0x2c/0x8b
Jun 12 17:32:48 sd70 kernel: [ 1201.696819]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
Jun 12 17:32:48 sd70 kernel: [ 1201.696890]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
Jun 12 17:32:48 sd70 kernel: [ 1201.696978]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.697048]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
Jun 12 17:32:48 sd70 kernel: [ 1201.697126]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.697215]  [<ffffffff810d85ca>] ? lru_cache_add_lru+0x3c/0x3e
Jun 12 17:32:48 sd70 kernel: [ 1201.697291]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.697362]  [<ffffffff8106805e>] ? task_sched_runtime+0x5e/0x6e
Jun 12 17:32:48 sd70 kernel: [ 1201.697431]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.697500]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
Jun 12 17:32:48 sd70 kernel: [ 1201.697571]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.697638]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.697705]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.697773] bonnie++        D ffff8805bc4f3228     0  6345      1 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.697917]  ffff880573da59b8 0000000000000082 ffff880601a70738 0000000000000286
Jun 12 17:32:48 sd70 kernel: [ 1201.698151]  ffff880573da5968 ffff880573da4010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.698384]  ffff880573da5fd8 0000000000013340 ffff880573da5fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.698614] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.698672]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
Jun 12 17:32:48 sd70 kernel: [ 1201.698739]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.698807]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.698882]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
Jun 12 17:32:48 sd70 kernel: [ 1201.698948]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
Jun 12 17:32:48 sd70 kernel: [ 1201.699013]  [<ffffffff810d5ecf>] ? bdi_dirty_limit+0x2c/0x8b
Jun 12 17:32:48 sd70 kernel: [ 1201.699082]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
Jun 12 17:32:48 sd70 kernel: [ 1201.699157]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
Jun 12 17:32:48 sd70 kernel: [ 1201.699250]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
Jun 12 17:32:48 sd70 kernel: [ 1201.699322]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
Jun 12 17:32:48 sd70 kernel: [ 1201.699402]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.699492]  [<ffffffff810d85ca>] ? lru_cache_add_lru+0x3c/0x3e
Jun 12 17:32:48 sd70 kernel: [ 1201.699570]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.699642]  [<ffffffff8106805e>] ? task_sched_runtime+0x5e/0x6e
Jun 12 17:32:48 sd70 kernel: [ 1201.699713]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.699826]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
Jun 12 17:32:48 sd70 kernel: [ 1201.699898]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.699966]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.700034]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.700104] flush-8:0       D ffff8805bcaf5aa8     0  6346      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.700250]  ffff88059d78f710 0000000000000046 ffff88059d78f700 ffff88059d78f6e0
Jun 12 17:32:48 sd70 kernel: [ 1201.700487]  ffff8805eaef8000 ffff88059d78e010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.700724]  ffff88059d78ffd8 0000000000013340 ffff88059d78ffd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.700961] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.701022]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.701090]  [<ffffffff8137e338>] io_schedule+0x87/0xca
Jun 12 17:32:48 sd70 kernel: [ 1201.701159]  [<ffffffff811aca59>] get_request_wait+0x116/0x1b9
Jun 12 17:32:48 sd70 kernel: [ 1201.701229]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
Jun 12 17:32:48 sd70 kernel: [ 1201.701298]  [<ffffffff811a70d2>] ? elv_merge+0xae/0xbe
Jun 12 17:32:48 sd70 kernel: [ 1201.701367]  [<ffffffff811acf8f>] blk_queue_bio+0x1a8/0x30d
Jun 12 17:32:48 sd70 kernel: [ 1201.701436]  [<ffffffff811ac2e5>] generic_make_request+0x99/0xdc
Jun 12 17:32:48 sd70 kernel: [ 1201.701507]  [<ffffffff811ac407>] submit_bio+0xdf/0xfe
Jun 12 17:32:48 sd70 kernel: [ 1201.701580]  [<ffffffffa02f1c3d>] xfs_submit_ioend_bio+0x2e/0x30 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.701658]  [<ffffffffa02f236e>] xfs_submit_ioend+0xa1/0xea [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.701736]  [<ffffffffa02f353f>] xfs_vm_writepage+0x40f/0x484 [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.701808]  [<ffffffff811c761c>] ? rb_insert_color+0xbc/0xe5
Jun 12 17:32:48 sd70 kernel: [ 1201.701878]  [<ffffffff810d558a>] __writepage+0x12/0x2b
Jun 12 17:32:48 sd70 kernel: [ 1201.701946]  [<ffffffff810d6a0d>] write_cache_pages+0x25f/0x36a
Jun 12 17:32:48 sd70 kernel: [ 1201.702017]  [<ffffffff810d5578>] ? set_page_dirty+0x62/0x62
Jun 12 17:32:48 sd70 kernel: [ 1201.702087]  [<ffffffff810d6b58>] generic_writepages+0x40/0x57
Jun 12 17:32:48 sd70 kernel: [ 1201.702162]  [<ffffffffa02f1a30>] xfs_vm_writepages+0x48/0x4f [xfs]
Jun 12 17:32:48 sd70 kernel: [ 1201.702234]  [<ffffffff810d6b8b>] do_writepages+0x1c/0x25
Jun 12 17:32:48 sd70 kernel: [ 1201.702304]  [<ffffffff81130a04>] writeback_single_inode+0x183/0x370
Jun 12 17:32:48 sd70 kernel: [ 1201.702376]  [<ffffffff81130eaa>] writeback_sb_inodes+0x172/0x20b
Jun 12 17:32:48 sd70 kernel: [ 1201.702447]  [<ffffffff81130fb6>] __writeback_inodes_wb+0x73/0xb4
Jun 12 17:32:48 sd70 kernel: [ 1201.702518]  [<ffffffff8113162b>] wb_writeback+0x136/0x22c
Jun 12 17:32:48 sd70 kernel: [ 1201.702587]  [<ffffffff810d61c1>] ? global_dirty_limits+0x2a/0x10a
Jun 12 17:32:48 sd70 kernel: [ 1201.702659]  [<ffffffff811317b1>] wb_do_writeback+0x90/0x1de
Jun 12 17:32:48 sd70 kernel: [ 1201.702729]  [<ffffffff811319bf>] bdi_writeback_thread+0xc0/0x1e5
Jun 12 17:32:48 sd70 kernel: [ 1201.702800]  [<ffffffff811318ff>] ? wb_do_writeback+0x1de/0x1de
Jun 12 17:32:48 sd70 kernel: [ 1201.702871]  [<ffffffff811318ff>] ? wb_do_writeback+0x1de/0x1de
Jun 12 17:32:48 sd70 kernel: [ 1201.702941]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.703009]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.703079]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.703152]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.703219] kworker/2:2     S ffff8805ea4d4280     0  6347      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.703366]  ffff880573d53e70 0000000000000046 ffff880573d53e20 ffffffff8129868a
Jun 12 17:32:48 sd70 kernel: [ 1201.703603]  0000000000000002 ffff880573d52010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.703905]  ffff880573d53fd8 0000000000013340 ffff880573d53fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.704142] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.704202]  [<ffffffff8129868a>] ? do_dbs_timer+0xe1/0xef
Jun 12 17:32:48 sd70 kernel: [ 1201.704271]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.704338]  [<ffffffff8105543f>] worker_thread+0x25f/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.704406]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
Jun 12 17:32:48 sd70 kernel: [ 1201.704476]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.704542]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.704612]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.704684]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.704750] sshd            S ffff8805eb5f6828     0  6369   1969 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.704895]  ffff880573f13918 0000000000000086 0000000000000072 ffff8805eb5dc000
Jun 12 17:32:48 sd70 kernel: [ 1201.710638]  ffff880573f138f8 ffff880573f12010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.710875]  ffff880573f13fd8 0000000000013340 ffff880573f13fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.711111] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.711173]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.711240]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.711314]  [<ffffffff8106560e>] ? __wake_up+0x3f/0x48
Jun 12 17:32:48 sd70 kernel: [ 1201.711382]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.711455]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.711526]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.711595]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.711664]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.711797]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.711866]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.711935]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.712004]  [<ffffffff8131031d>] ? inet_sendmsg+0x8f/0x98
Jun 12 17:32:48 sd70 kernel: [ 1201.712073]  [<ffffffff8137e141>] ? wait_for_common+0xd7/0x14a
Jun 12 17:32:48 sd70 kernel: [ 1201.712144]  [<ffffffff812ab281>] ? __sock_sendmsg_nosec+0x5f/0x6a
Jun 12 17:32:48 sd70 kernel: [ 1201.712215]  [<ffffffff812ab2c9>] ? __sock_sendmsg+0x3d/0x48
Jun 12 17:32:48 sd70 kernel: [ 1201.712284]  [<ffffffff812ab3c5>] ? sock_aio_write+0xf1/0x101
Jun 12 17:32:48 sd70 kernel: [ 1201.712354]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.712424]  [<ffffffff811107e3>] ? do_sync_write+0xc9/0x106
Jun 12 17:32:48 sd70 kernel: [ 1201.712493]  [<ffffffff811400a9>] ? fsnotify+0x255/0x285
Jun 12 17:32:48 sd70 kernel: [ 1201.712562]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.712630]  [<ffffffff81111292>] ? sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.712699]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.712766] bash            S ffff8805eb5f6168     0  6371   6369 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.712905]  ffff880573f3fcf8 0000000000000082 0000000000000001 ffff880573e8c950
Jun 12 17:32:48 sd70 kernel: [ 1201.713135]  0000000000000000 ffff880573f3e010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.713378]  ffff880573f3ffd8 0000000000013340 ffff880573f3ffd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.713605] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.713667]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.713738]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
Jun 12 17:32:48 sd70 kernel: [ 1201.713803]  [<ffffffff81053f2f>] ? flush_work+0x1a/0x34
Jun 12 17:32:48 sd70 kernel: [ 1201.713868]  [<ffffffff81241913>] ? tty_wakeup+0x56/0x5b
Jun 12 17:32:48 sd70 kernel: [ 1201.713937]  [<ffffffff8124904f>] ? pty_write+0x4c/0x56
Jun 12 17:32:48 sd70 kernel: [ 1201.714011]  [<ffffffff812458ca>] n_tty_read+0x49b/0x73f
Jun 12 17:32:48 sd70 kernel: [ 1201.714076]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
Jun 12 17:32:48 sd70 kernel: [ 1201.714141]  [<ffffffff812404e9>] tty_read+0x81/0xc2
Jun 12 17:32:48 sd70 kernel: [ 1201.714208]  [<ffffffff8111135f>] vfs_read+0xa6/0x102
Jun 12 17:32:48 sd70 kernel: [ 1201.714279]  [<ffffffff8104cab3>] ? sigprocmask+0x5e/0x62
Jun 12 17:32:48 sd70 kernel: [ 1201.714350]  [<ffffffff81111474>] sys_read+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.714418]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.714488] kworker/0:3     S ffff8805bc4f6ee8     0  6387      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.714634]  ffff8806013b9e70 0000000000000046 ffffffff811be375 0000000000000000
Jun 12 17:32:48 sd70 kernel: [ 1201.714871]  ffff8806013b9e00 ffff8806013b8010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.715108]  ffff8806013b9fd8 0000000000013340 ffff8806013b9fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.715345] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.715405]  [<ffffffff811be375>] ? cfq_init_queue+0x280/0x280
Jun 12 17:32:48 sd70 kernel: [ 1201.715476]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.715543]  [<ffffffff8105543f>] worker_thread+0x25f/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.715613]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
Jun 12 17:32:48 sd70 kernel: [ 1201.715683]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.715789]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.715860]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.715933]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.716001] kworker/3:2     S ffff8805eb5f5aa8     0  6400      2 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.716146]  ffff88060299de70 0000000000000046 ffff88060299de20 ffffffff8129868a
Jun 12 17:32:48 sd70 kernel: [ 1201.716378]  ffff88060299de10 ffff88060299c010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.716618]  ffff88060299dfd8 0000000000013340 ffff88060299dfd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.716854] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.716916]  [<ffffffff8129868a>] ? do_dbs_timer+0xe1/0xef
Jun 12 17:32:48 sd70 kernel: [ 1201.716985]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.717053]  [<ffffffff8105543f>] worker_thread+0x25f/0x264
Jun 12 17:32:48 sd70 kernel: [ 1201.717122]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
Jun 12 17:32:48 sd70 kernel: [ 1201.717192]  [<ffffffff8105bb50>] kthread+0x84/0x8c
Jun 12 17:32:48 sd70 kernel: [ 1201.717260]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.717331]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
Jun 12 17:32:48 sd70 kernel: [ 1201.717404]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
Jun 12 17:32:48 sd70 kernel: [ 1201.717471] tail            S ffff8805ec5c6ee8     0  6979   5971 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.717614]  ffff8800df2c7e48 0000000000000082 ffff8800df2c7dd8 ffff8805ea47f970
Jun 12 17:32:48 sd70 kernel: [ 1201.717851]  ffff8800df2c7e18 ffff8800df2c6010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.718088]  ffff8800df2c7fd8 0000000000013340 ffff8800df2c7fd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.718325] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.718386]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.718455]  [<ffffffff81142e03>] inotify_read+0x196/0x1e7
Jun 12 17:32:48 sd70 kernel: [ 1201.718524]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
Jun 12 17:32:48 sd70 kernel: [ 1201.718592]  [<ffffffff8111135f>] vfs_read+0xa6/0x102
Jun 12 17:32:48 sd70 kernel: [ 1201.718660]  [<ffffffff81111474>] sys_read+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.718728]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.718798] sshd            S ffff8805bc7a82e8     0  7167   1969 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.718944]  ffff88019110f918 0000000000000086 0000000000000072 ffff8805eb5dc000
Jun 12 17:32:48 sd70 kernel: [ 1201.719181]  ffff88019110f8f8 ffff88019110e010 0000000000013340 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.719418]  ffff88019110ffd8 0000000000013340 ffff88019110ffd8 0000000000013340
Jun 12 17:32:48 sd70 kernel: [ 1201.719655] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.719782]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:32:48 sd70 kernel: [ 1201.719850]  [<ffffffff8137d5aa>] schedule_hrtimeout_range_clock+0x4c/0x12f
Jun 12 17:32:48 sd70 kernel: [ 1201.719923]  [<ffffffff8106560e>] ? __wake_up+0x3f/0x48
Jun 12 17:32:48 sd70 kernel: [ 1201.719992]  [<ffffffff8137d69b>] schedule_hrtimeout_range+0xe/0x10
Jun 12 17:32:48 sd70 kernel: [ 1201.720064]  [<ffffffff8111f3c2>] poll_schedule_timeout+0x43/0x5f
Jun 12 17:32:48 sd70 kernel: [ 1201.720134]  [<ffffffff8111fd59>] do_select+0x4f8/0x537
Jun 12 17:32:48 sd70 kernel: [ 1201.720204]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.720272]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.720341]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.720410]  [<ffffffff8111f535>] ? __pollwait+0xc7/0xc7
Jun 12 17:32:48 sd70 kernel: [ 1201.720479]  [<ffffffff81062b71>] ? check_preempt_curr+0x2d/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.720549]  [<ffffffff81069a0d>] ? ttwu_do_wakeup+0x54/0xcb
Jun 12 17:32:48 sd70 kernel: [ 1201.720619]  [<ffffffff8106a35a>] ? try_to_wake_up+0x1a2/0x1b4
Jun 12 17:32:48 sd70 kernel: [ 1201.720689]  [<ffffffff8106a379>] ? default_wake_function+0xd/0xf
Jun 12 17:32:48 sd70 kernel: [ 1201.720760]  [<ffffffff8106262b>] ? __wake_up_common+0x49/0x7f
Jun 12 17:32:48 sd70 kernel: [ 1201.720829]  [<ffffffff8137d53d>] ? mutex_lock+0x11/0x32
Jun 12 17:32:48 sd70 kernel: [ 1201.720898]  [<ffffffff812461c5>] ? copy_termios+0x35/0x3d
Jun 12 17:32:48 sd70 kernel: [ 1201.720967]  [<ffffffff81120043>] core_sys_select+0x16f/0x205
Jun 12 17:32:48 sd70 kernel: [ 1201.721038]  [<ffffffff812471ca>] ? put_ldisc+0xac/0xb1
Jun 12 17:32:48 sd70 kernel: [ 1201.721106]  [<ffffffff812471f9>] ? tty_ldisc_deref+0x9/0xb
Jun 12 17:32:48 sd70 kernel: [ 1201.721176]  [<ffffffff812429aa>] ? tty_ioctl+0x771/0x793
Jun 12 17:32:48 sd70 kernel: [ 1201.721245]  [<ffffffff8112031f>] sys_select+0x91/0xb9
Jun 12 17:32:48 sd70 kernel: [ 1201.721314]  [<ffffffff8111ed33>] ? sys_ioctl+0x51/0x76
Jun 12 17:32:48 sd70 kernel: [ 1201.721382]  [<ffffffff81111292>] ? sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.721450]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.721520] bash            R  running task        0  7169   7167 0x00000000
Jun 12 17:32:48 sd70 kernel: [ 1201.721666]  ffff880191145cb8 ffff880191145ca8 ffff880191145d18 0000000000000046
Jun 12 17:32:48 sd70 kernel: [ 1201.721904]  ffff880191145cb6 000000000000000f ffff880191145cb7 0000000000000046
Jun 12 17:32:48 sd70 kernel: [ 1201.722140]  ffffffff814c2aae 6300000000000004 372e31303231205b 00205d3034313232
Jun 12 17:32:48 sd70 kernel: [ 1201.722377] Call Trace:
Jun 12 17:32:48 sd70 kernel: [ 1201.722439]  [<ffffffff81010fb2>] ? dump_trace+0x266/0x275
Jun 12 17:32:48 sd70 kernel: [ 1201.722509]  [<ffffffff81011c60>] show_trace_log_lvl+0x4f/0x58
Jun 12 17:32:48 sd70 kernel: [ 1201.722579]  [<ffffffff810110f2>] show_stack_log_lvl+0x107/0x116
Jun 12 17:32:48 sd70 kernel: [ 1201.722650]  [<ffffffff81011c0f>] show_stack+0x17/0x19
Jun 12 17:32:48 sd70 kernel: [ 1201.722718]  [<ffffffff810671c3>] sched_show_task+0xac/0xb1
Jun 12 17:32:48 sd70 kernel: [ 1201.722788]  [<ffffffff81067213>] show_state_filter+0x4b/0x9e
Jun 12 17:32:48 sd70 kernel: [ 1201.722857]  [<ffffffff8124a0ba>] sysrq_handle_showstate+0xb/0xd
Jun 12 17:32:48 sd70 kernel: [ 1201.722928]  [<ffffffff8124a3fc>] __handle_sysrq+0xba/0x156
Jun 12 17:32:48 sd70 kernel: [ 1201.722997]  [<ffffffff8124a4d0>] write_sysrq_trigger+0x38/0x40
Jun 12 17:32:48 sd70 kernel: [ 1201.723069]  [<ffffffff8115af4a>] proc_reg_write+0x7b/0x94
Jun 12 17:32:48 sd70 kernel: [ 1201.723138]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
Jun 12 17:32:48 sd70 kernel: [ 1201.723205]  [<ffffffff81111292>] sys_write+0x45/0x6c
Jun 12 17:32:48 sd70 kernel: [ 1201.723273]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:32:48 sd70 kernel: [ 1201.723344] Sched Debug Version: v0.10, 3.4.1-custom #5
Jun 12 17:32:48 sd70 kernel: [ 1201.723411] ktime                                   : 1204099.656753
Jun 12 17:32:48 sd70 kernel: [ 1201.723481] sched_clk                               : 1201723.342856
Jun 12 17:32:48 sd70 kernel: [ 1201.723551] cpu_clk                                 : 1201723.342897
Jun 12 17:32:48 sd70 kernel: [ 1201.723621] jiffies                                 : 4295193320
Jun 12 17:32:48 sd70 kernel: [ 1201.723754] sched_clock_stable                      : 1
Jun 12 17:32:48 sd70 kernel: [ 1201.723820] 
Jun 12 17:32:48 sd70 kernel: [ 1201.723877] sysctl_sched
Jun 12 17:32:48 sd70 kernel: [ 1201.723937]   .sysctl_sched_latency                    : 18.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.724007]   .sysctl_sched_min_granularity            : 2.250000
Jun 12 17:32:48 sd70 kernel: [ 1201.724076]   .sysctl_sched_wakeup_granularity         : 3.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.724145]   .sysctl_sched_child_runs_first           : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.724212]   .sysctl_sched_features                   : 24119
Jun 12 17:32:48 sd70 kernel: [ 1201.724281]   .sysctl_sched_tunable_scaling            : 1 (logaritmic)
Jun 12 17:32:48 sd70 kernel: [ 1201.724353] 
Jun 12 17:32:48 sd70 kernel: [ 1201.724353] cpu#0, 2133.451 MHz
Jun 12 17:32:48 sd70 kernel: [ 1201.724470]   .nr_running                    : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.724535]   .load                          : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.724601]   .nr_switches                   : 1648682
Jun 12 17:32:48 sd70 kernel: [ 1201.724667]   .nr_load_updates               : 68666
Jun 12 17:32:48 sd70 kernel: [ 1201.724733]   .nr_uninterruptible            : 401
Jun 12 17:32:48 sd70 kernel: [ 1201.724799]   .next_balance                  : 4295.193319
Jun 12 17:32:48 sd70 kernel: [ 1201.724866]   .curr->pid                     : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.724932]   .clock                         : 1201717.804248
Jun 12 17:32:48 sd70 kernel: [ 1201.725000]   .cpu_load[0]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.725065]   .cpu_load[1]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.725130]   .cpu_load[2]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.725194]   .cpu_load[3]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.725259]   .cpu_load[4]                   : 7
Jun 12 17:32:48 sd70 kernel: [ 1201.725325] 
Jun 12 17:32:48 sd70 kernel: [ 1201.725326] cfs_rq[0]:/
Jun 12 17:32:48 sd70 kernel: [ 1201.725441]   .exec_clock                    : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.731112]   .MIN_vruntime                  : 0.000001
Jun 12 17:32:48 sd70 kernel: [ 1201.731179]   .min_vruntime                  : 38985.442621
Jun 12 17:32:48 sd70 kernel: [ 1201.731247]   .max_vruntime                  : 0.000001
Jun 12 17:32:48 sd70 kernel: [ 1201.731314]   .spread                        : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.731381]   .spread0                       : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.731448]   .nr_spread_over                : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.731513]   .nr_running                    : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.731578]   .load                          : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.731643]   .load_avg                      : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.731757]   .load_period                   : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.731823]   .load_contrib                  : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.731888]   .load_tg                       : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.731954] 
Jun 12 17:32:48 sd70 kernel: [ 1201.731954] rt_rq[0]:
Jun 12 17:32:48 sd70 kernel: [ 1201.732069]   .rt_nr_running                 : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.732133]   .rt_throttled                  : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.732199]   .rt_time                       : 0.007102
Jun 12 17:32:48 sd70 kernel: [ 1201.732266]   .rt_runtime                    : 950.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.732333] 
Jun 12 17:32:48 sd70 kernel: [ 1201.732334] runnable tasks:
Jun 12 17:32:48 sd70 kernel: [ 1201.732335]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
Jun 12 17:32:48 sd70 kernel: [ 1201.732336] ----------------------------------------------------------------------------------------------------------
Jun 12 17:32:48 sd70 kernel: [ 1201.732651] 
Jun 12 17:32:48 sd70 kernel: [ 1201.732652] cpu#1, 2133.451 MHz
Jun 12 17:32:48 sd70 kernel: [ 1201.732769]   .nr_running                    : 2
Jun 12 17:32:48 sd70 kernel: [ 1201.732834]   .load                          : 2048
Jun 12 17:32:48 sd70 kernel: [ 1201.732899]   .nr_switches                   : 157645
Jun 12 17:32:48 sd70 kernel: [ 1201.732965]   .nr_load_updates               : 56595
Jun 12 17:32:48 sd70 kernel: [ 1201.733027]   .nr_uninterruptible            : -9
Jun 12 17:32:48 sd70 kernel: [ 1201.733088]   .next_balance                  : 4295.193246
Jun 12 17:32:48 sd70 kernel: [ 1201.733155]   .curr->pid                     : 7169
Jun 12 17:32:48 sd70 kernel: [ 1201.733224]   .clock                         : 1201435.970296
Jun 12 17:32:48 sd70 kernel: [ 1201.733292]   .cpu_load[0]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.733353]   .cpu_load[1]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.733415]   .cpu_load[2]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.733480]   .cpu_load[3]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.733551]   .cpu_load[4]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.733616] 
Jun 12 17:32:48 sd70 kernel: [ 1201.733617] cfs_rq[1]:/
Jun 12 17:32:48 sd70 kernel: [ 1201.733732]   .exec_clock                    : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.733799]   .MIN_vruntime                  : 30918.883472
Jun 12 17:32:48 sd70 kernel: [ 1201.733867]   .min_vruntime                  : 30927.713140
Jun 12 17:32:48 sd70 kernel: [ 1201.733935]   .max_vruntime                  : 30918.883472
Jun 12 17:32:48 sd70 kernel: [ 1201.734001]   .spread                        : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.734064]   .spread0                       : -8057.729481
Jun 12 17:32:48 sd70 kernel: [ 1201.734128]   .nr_spread_over                : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.734194]   .nr_running                    : 2
Jun 12 17:32:48 sd70 kernel: [ 1201.734263]   .load                          : 2048
Jun 12 17:32:48 sd70 kernel: [ 1201.734325]   .load_avg                      : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.734387]   .load_period                   : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.734454]   .load_contrib                  : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.734522]   .load_tg                       : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.734586] 
Jun 12 17:32:48 sd70 kernel: [ 1201.734587] rt_rq[1]:
Jun 12 17:32:48 sd70 kernel: [ 1201.734700]   .rt_nr_running                 : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.734764]   .rt_throttled                  : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.734828]   .rt_time                       : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.734894]   .rt_runtime                    : 950.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.734961] 
Jun 12 17:32:48 sd70 kernel: [ 1201.734961] runnable tasks:
Jun 12 17:32:48 sd70 kernel: [ 1201.734962]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
Jun 12 17:32:48 sd70 kernel: [ 1201.734963] ----------------------------------------------------------------------------------------------------------
Jun 12 17:32:48 sd70 kernel: [ 1201.735266]      kworker/1:1    29     30918.883472     27764   120               0               0               0.000000               0.000000               0.000000 /
Jun 12 17:32:48 sd70 kernel: [ 1201.735534] R           bash  7169     30918.833402        58   120               0               0               0.000000               0.000000               0.000000 /
Jun 12 17:32:48 sd70 kernel: [ 1201.735804] 
Jun 12 17:32:48 sd70 kernel: [ 1201.735804] cpu#2, 2133.451 MHz
Jun 12 17:32:48 sd70 kernel: [ 1201.735920]   .nr_running                    : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.735984]   .load                          : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.736049]   .nr_switches                   : 123555
Jun 12 17:32:48 sd70 kernel: [ 1201.736114]   .nr_load_updates               : 40724
Jun 12 17:32:48 sd70 kernel: [ 1201.736179]   .nr_uninterruptible            : -72
Jun 12 17:32:48 sd70 kernel: [ 1201.736244]   .next_balance                  : 4295.193325
Jun 12 17:32:48 sd70 kernel: [ 1201.736311]   .curr->pid                     : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.736376]   .clock                         : 1201723.849747
Jun 12 17:32:48 sd70 kernel: [ 1201.736444]   .cpu_load[0]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.736508]   .cpu_load[1]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.736572]   .cpu_load[2]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.736637]   .cpu_load[3]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.736701]   .cpu_load[4]                   : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.736766] 
Jun 12 17:32:48 sd70 kernel: [ 1201.736766] cfs_rq[2]:/
Jun 12 17:32:48 sd70 kernel: [ 1201.736880]   .exec_clock                    : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.736946]   .MIN_vruntime                  : 0.000001
Jun 12 17:32:48 sd70 kernel: [ 1201.737012]   .min_vruntime                  : 29430.870364
Jun 12 17:32:48 sd70 kernel: [ 1201.737078]   .max_vruntime                  : 0.000001
Jun 12 17:32:48 sd70 kernel: [ 1201.737144]   .spread                        : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.737210]   .spread0                       : -9554.572257
Jun 12 17:32:48 sd70 kernel: [ 1201.737277]   .nr_spread_over                : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.737341]   .nr_running                    : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.737405]   .load                          : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.737469]   .load_avg                      : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.737535]   .load_period                   : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.737601]   .load_contrib                  : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.737665]   .load_tg                       : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.737729] 
Jun 12 17:32:48 sd70 kernel: [ 1201.737730] rt_rq[2]:
Jun 12 17:32:48 sd70 kernel: [ 1201.737843]   .rt_nr_running                 : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.737907]   .rt_throttled                  : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.737971]   .rt_time                       : 0.006317
Jun 12 17:32:48 sd70 kernel: [ 1201.738037]   .rt_runtime                    : 950.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.738104] 
Jun 12 17:32:48 sd70 kernel: [ 1201.738104] runnable tasks:
Jun 12 17:32:48 sd70 kernel: [ 1201.738105]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
Jun 12 17:32:48 sd70 kernel: [ 1201.738106] ----------------------------------------------------------------------------------------------------------
Jun 12 17:32:48 sd70 kernel: [ 1201.738412] 
Jun 12 17:32:48 sd70 kernel: [ 1201.738413] cpu#3, 2133.451 MHz
Jun 12 17:32:48 sd70 kernel: [ 1201.738527]   .nr_running                    : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.738591]   .load                          : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.738656]   .nr_switches                   : 122483
Jun 12 17:32:48 sd70 kernel: [ 1201.738721]   .nr_load_updates               : 45075
Jun 12 17:32:48 sd70 kernel: [ 1201.738786]   .nr_uninterruptible            : -301
Jun 12 17:32:48 sd70 kernel: [ 1201.738851]   .next_balance                  : 4295.193333
Jun 12 17:32:48 sd70 kernel: [ 1201.738917]   .curr->pid                     : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.738982]   .clock                         : 1201736.633267
Jun 12 17:32:48 sd70 kernel: [ 1201.739049]   .cpu_load[0]                   : 1024
Jun 12 17:32:48 sd70 kernel: [ 1201.739114]   .cpu_load[1]                   : 520
Jun 12 17:32:48 sd70 kernel: [ 1201.739178]   .cpu_load[2]                   : 301
Jun 12 17:32:48 sd70 kernel: [ 1201.739243]   .cpu_load[3]                   : 192
Jun 12 17:32:48 sd70 kernel: [ 1201.739307]   .cpu_load[4]                   : 127
Jun 12 17:32:48 sd70 kernel: [ 1201.739372] 
Jun 12 17:32:48 sd70 kernel: [ 1201.739372] cfs_rq[3]:/
Jun 12 17:32:48 sd70 kernel: [ 1201.739486]   .exec_clock                    : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.739552]   .MIN_vruntime                  : 0.000001
Jun 12 17:32:48 sd70 kernel: [ 1201.739618]   .min_vruntime                  : 32172.100305
Jun 12 17:32:48 sd70 kernel: [ 1201.739685]   .max_vruntime                  : 0.000001
Jun 12 17:32:48 sd70 kernel: [ 1201.739751]   .spread                        : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.739817]   .spread0                       : -6813.342316
Jun 12 17:32:48 sd70 kernel: [ 1201.739883]   .nr_spread_over                : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.739947]   .nr_running                    : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.740011]   .load                          : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.740076]   .load_avg                      : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.740141]   .load_period                   : 0.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.740207]   .load_contrib                  : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.740272]   .load_tg                       : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.740336] 
Jun 12 17:32:48 sd70 kernel: [ 1201.740336] rt_rq[3]:
Jun 12 17:32:48 sd70 kernel: [ 1201.740449]   .rt_nr_running                 : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.740513]   .rt_throttled                  : 0
Jun 12 17:32:48 sd70 kernel: [ 1201.740577]   .rt_time                       : 0.018701
Jun 12 17:32:48 sd70 kernel: [ 1201.740643]   .rt_runtime                    : 950.000000
Jun 12 17:32:48 sd70 kernel: [ 1201.740710] 
Jun 12 17:32:48 sd70 kernel: [ 1201.740710] runnable tasks:
Jun 12 17:32:48 sd70 kernel: [ 1201.740711]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
Jun 12 17:32:48 sd70 kernel: [ 1201.740712] ----------------------------------------------------------------------------------------------------------
Jun 12 17:32:48 sd70 kernel: [ 1201.741017] 
Jun 12 17:38:47 sd70 kernel: [ 1559.653101] INFO: task sync:7493 blocked for more than 120 seconds.
Jun 12 17:38:47 sd70 kernel: [ 1559.653179] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 12 17:38:47 sd70 kernel: [ 1559.653269] sync            D ffff8805ec48b8e8     0  7493   7169 0x00000000
Jun 12 17:38:47 sd70 kernel: [ 1559.653417]  ffff880208869cf8 0000000000000086 ffff8805bc02ae78 00000000000000c9
Jun 12 17:38:47 sd70 kernel: [ 1559.653655]  ffff880208869ce8 ffff880208868010 0000000000013340 0000000000013340
Jun 12 17:38:47 sd70 kernel: [ 1559.653892]  ffff880208869fd8 0000000000013340 ffff880208869fd8 0000000000013340
Jun 12 17:38:47 sd70 kernel: [ 1559.654129] Call Trace:
Jun 12 17:38:47 sd70 kernel: [ 1559.654197]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:38:47 sd70 kernel: [ 1559.654265]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
Jun 12 17:38:47 sd70 kernel: [ 1559.654337]  [<ffffffff8106b47b>] ? wake_affine+0x190/0x248
Jun 12 17:38:47 sd70 kernel: [ 1559.660092]  [<ffffffff8106ec4e>] ? enqueue_entity+0x121/0x1cd
Jun 12 17:38:47 sd70 kernel: [ 1559.660163]  [<ffffffff8137e136>] wait_for_common+0xcc/0x14a
Jun 12 17:38:47 sd70 kernel: [ 1559.660235]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
Jun 12 17:38:47 sd70 kernel: [ 1559.660307]  [<ffffffff810440c8>] ? local_bh_enable_ip+0x9/0xb
Jun 12 17:38:47 sd70 kernel: [ 1559.660382]  [<ffffffff811348f7>] ? __sync_filesystem+0x7a/0x7a
Jun 12 17:38:47 sd70 kernel: [ 1559.660452]  [<ffffffff8137e24e>] wait_for_completion+0x18/0x1a
Jun 12 17:38:47 sd70 kernel: [ 1559.660524]  [<ffffffff81131203>] writeback_inodes_sb_nr+0x78/0x7f
Jun 12 17:38:47 sd70 kernel: [ 1559.660596]  [<ffffffff811312b0>] writeback_inodes_sb+0x4e/0x59
Jun 12 17:38:47 sd70 kernel: [ 1559.660667]  [<ffffffff811348ce>] __sync_filesystem+0x51/0x7a
Jun 12 17:38:47 sd70 kernel: [ 1559.660738]  [<ffffffff81134908>] sync_one_sb+0x11/0x13
Jun 12 17:38:47 sd70 kernel: [ 1559.660827]  [<ffffffff81112aae>] iterate_supers+0x69/0xbb
Jun 12 17:38:47 sd70 kernel: [ 1559.660898]  [<ffffffff81134939>] sys_sync+0x2f/0x5c
Jun 12 17:38:47 sd70 kernel: [ 1559.661052]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
Jun 12 17:38:47 sd70 kernel: [ 1559.661125] INFO: task shutdown:7494 blocked for more than 120 seconds.
Jun 12 17:38:47 sd70 kernel: [ 1559.661197] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 12 17:38:47 sd70 kernel: [ 1559.661288] shutdown        D ffff8805bc7a89a8     0  7494      1 0x00000000
Jun 12 17:38:47 sd70 kernel: [ 1559.661446]  ffff8805ebd13cf8 0000000000000082 0000000000000000 ffff8805ec96ca80
Jun 12 17:38:47 sd70 kernel: [ 1559.661689]  ffff8806026f3a40 ffff8805ebd12010 0000000000013340 0000000000013340
Jun 12 17:38:47 sd70 kernel: [ 1559.661928]  ffff8805ebd13fd8 0000000000013340 ffff8805ebd13fd8 0000000000013340
Jun 12 17:38:47 sd70 kernel: [ 1559.662166] Call Trace:
Jun 12 17:38:47 sd70 kernel: [ 1559.662228]  [<ffffffff8137e2af>] schedule+0x5f/0x61
Jun 12 17:38:47 sd70 kernel: [ 1559.662295]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
Jun 12 17:38:47 sd70 kernel: [ 1559.662366]  [<ffffffff811c2332>] ? cpumask_next_and+0x2c/0x39
Jun 12 17:38:47 sd70 kernel: [ 1559.662437]  [<ffffffff8106ec4e>] ? enqueue_entity+0x121/0x1cd
Jun 12 17:38:47 sd70 kernel: [ 1559.662506]  [<ffffffff8137e136>] wait_for_common+0xcc/0x14a
Jun 12 17:38:47 sd70 kernel: [ 1559.662576]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
Jun 12 17:38:47 sd70 kernel: [ 1559.662646]  [<ffffffff810440c8>] ? local_bh_enable_ip+0x9/0xb
Jun 12 17:38:47 sd70 kernel: [ 1559.662716]  [<ffffffff811348f7>] ? __sync_filesystem+0x7a/0x7a
Jun 12 17:38:47 sd70 kernel: [ 1559.662786]  [<ffffffff8137e24e>] wait_for_completion+0x18/0x1a
Jun 12 17:38:47 sd70 kernel: [ 1559.662857]  [<ffffffff81131203>] writeback_inodes_sb_nr+0x78/0x7f
Jun 12 17:38:47 sd70 kernel: [ 1559.662928]  [<ffffffff811312b0>] writeback_inodes_sb+0x4e/0x59
Jun 12 17:38:47 sd70 kernel: [ 1559.662999]  [<ffffffff811348ce>] __sync_filesystem+0x51/0x7a
Jun 12 17:38:47 sd70 kernel: [ 1559.663069]  [<ffffffff81134908>] sync_one_sb+0x11/0x13
Jun 12 17:38:47 sd70 kernel: [ 1559.663137]  [<ffffffff81112aae>] iterate_supers+0x69/0xbb
Jun 12 17:38:47 sd70 kernel: [ 1559.663207]  [<ffffffff81134939>] sys_sync+0x2f/0x5c
Jun 12 17:38:47 sd70 kernel: [ 1559.663275]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b

[-- Attachment #3: controllerinfo.txt --]
[-- Type: text/plain, Size: 9567 bytes --]

\r                                     
Adapter #0

==============================================================================
                    Versions
                ================
Product Name    : LSI MegaRAID SAS 9265-8i
Serial No       : SV21635080
FW Package Build: 23.4.1-0028

                    Mfg. Data
                ================
Mfg. Date       : 04/21/12
Rework Date     : 00/00/00
Revision No     : 01C
Battery FRU     : N/A

                Image Versions in Flash:
                ================
BIOS Version       : 5.29.00_4.12.05.00_0x05090000
WebBIOS Version    : 6.1-31-e_31-Rel
Preboot CLI Version: 05.02-01:#%00001
FW Version         : 3.160.15-1559
NVDATA Version     : 2.1108.03-0078
Boot Block Version : 2.05.00.00-0004
BOOT Version       : 07.26.05.219

                Pending Images in Flash
                ================
None

                PCI Info
                ================
Vendor Id       : 1000
Device Id       : 005b
SubVendorId     : 1000
SubDeviceId     : 9265

Host Interface  : PCIE

ChipRevision    : B0

Number of Frontend Port: 0 
Device Interface  : PCIE

Number of Backend Port: 8 
Port  :  Address
0        50030480001591ff 
1        50030480001a34ff 
2        0000000000000000 
3        0000000000000000 
4        0000000000000000 
5        0000000000000000 
6        0000000000000000 
7        0000000000000000 

                HW Configuration
                ================
SAS Address      : 500605b0049af0f0
BBU              : Present
Alarm            : Present
NVRAM            : Present
Serial Debugger  : Present
Memory           : Present
Flash            : Present
Memory Size      : 1024MB
TPM              : Absent
On board Expander: Absent
Upgrade Key      : Absent
Temperature sensor for ROC    : Present
Temperature sensor for controller    : Absent

ROC temperature : 82  degree Celcius

                Settings
                ================
Current Time                     : 23:23:41 6/12, 2012
Predictive Fail Poll Interval    : 300sec
Interrupt Throttle Active Count  : 16
Interrupt Throttle Completion    : 50us
Rebuild Rate                     : 30%
PR Rate                          : 30%
BGI Rate                         : 30%
Check Consistency Rate           : 30%
Reconstruction Rate              : 30%
Cache Flush Interval             : 4s
Max Drives to Spinup at One Time : 2
Delay Among Spinup Groups        : 12s
Physical Drive Coercion Mode     : Disabled
Cluster Mode                     : Disabled
Alarm                            : Enabled
Auto Rebuild                     : Enabled
Battery Warning                  : Enabled
Ecc Bucket Size                  : 15
Ecc Bucket Leak Rate             : 1440 Minutes
Restore HotSpare on Insertion    : Disabled
Expose Enclosure Devices         : Enabled
Maintain PD Fail History         : Enabled
Host Request Reordering          : Enabled
Auto Detect BackPlane Enabled    : SGPIO/i2c SEP
Load Balance Mode                : Auto
Use FDE Only                     : No
Security Key Assigned            : No
Security Key Failed              : No
Security Key Not Backedup        : No
Default LD PowerSave Policy      : Controller Defined
Maximum number of direct attached drives to spin up in 1 min : 10 
Any Offline VD Cache Preserved   : No
Allow Boot with Preserved Cache  : No
Disable Online Controller Reset  : No
PFK in NVRAM                     : No
Use disk activity for locate     : No

                Capabilities
                ================
RAID Level Supported             : RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50, RAID60, PRL 11, PRL 11 with spanning, SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
Supported Drives                 : SAS, SATA

Allowed Mixing:

Mix in Enclosure Allowed
Mix of SAS/SATA of HDD type in VD Allowed

                Status
                ================
ECC Bucket Count                 : 0

                Limitations
                ================
Max Arms Per VD          : 32 
Max Spans Per VD         : 8 
Max Arrays               : 128 
Max Number of VDs        : 64 
Max Parallel Commands    : 1008 
Max SGE Count            : 60 
Max Data Transfer Size   : 8192 sectors 
Max Strips PerIO         : 42 
Min Strip Size           : 8 KB
Max Strip Size           : 1.0 MB
Max Configurable CacheCade Size: 0 GB
Current Size of CacheCade      : 0 GB
Current Size of FW Cache       : 923 MB

                Device Present
                ================
Virtual Drives    : 2 
  Degraded        : 0 
  Offline         : 0 
Physical Devices  : 49 
  Disks           : 48 
  Critical Disks  : 0 
  Failed Disks    : 0 

                Supported Adapter Operations
                ================
Rebuild Rate                    : Yes
CC Rate                         : Yes
BGI Rate                        : Yes
Reconstruct Rate                : Yes
Patrol Read Rate                : Yes
Alarm Control                   : Yes
Cluster Support                 : No
BBU                             : Yes
Spanning                        : Yes
Dedicated Hot Spare             : Yes
Revertible Hot Spares           : Yes
Foreign Config Import           : Yes
Self Diagnostic                 : Yes
Allow Mixed Redundancy on Array : No
Global Hot Spares               : Yes
Deny SCSI Passthrough           : No
Deny SMP Passthrough            : No
Deny STP Passthrough            : No
Support Security                : No
Snapshot Enabled                : No
Support the OCE without adding drives : Yes
Support PFK                     : Yes
PFK TrailTime Remaining         : 0 days 0 hours

                Supported VD Operations
                ================
Read Policy          : Yes
Write Policy         : Yes
IO Policy            : Yes
Access Policy        : Yes
Disk Cache Policy    : Yes
Reconstruction       : Yes
Deny Locate          : No
Deny CC              : No
Allow Ctrl Encryption: No
Enable LDBBM         : No
Support Breakmirror  : No
Power Savings        : No

                Supported PD Operations
                ================
Force Online                            : Yes
Force Offline                           : Yes
Force Rebuild                           : Yes
Deny Force Failed                       : No
Deny Force Good/Bad                     : No
Deny Missing Replace                    : No
Deny Clear                              : No
Deny Locate                             : No
Support Temperature                     : Yes
Disable Copyback                        : No
Enable JBOD                             : No
Enable Copyback on SMART                : No
Enable Copyback to SSD on SMART Error   : Yes
Enable SSD Patrol Read                  : No
PR Correct Unconfigured Areas           : Yes
Enable Spin Down of UnConfigured Drives : Yes
Disable Spin Down of hot spares         : No
Spin Down time                          : 30 
T10 Power State                         : No
                Error Counters
                ================
Memory Correctable Errors   : 0 
Memory Uncorrectable Errors : 0 

                Cluster Information
                ================
Cluster Permitted     : No
Cluster Active        : No

                Default Settings
                ================
Phy Polarity                     : 0 
Phy PolaritySplit                : 0 
Background Rate                  : 30 
Strip Size                       : 256kB
Flush Time                       : 4 seconds
Write Policy                     : WB
Read Policy                      : Adaptive
Cache When BBU Bad               : Disabled
Cached IO                        : No
SMART Mode                       : Mode 6
Alarm Disable                    : Yes
Coercion Mode                    : None
ZCR Config                       : Unknown
Dirty LED Shows Drive Activity   : No
BIOS Continue on Error           : No
Spin Down Mode                   : None
Allowed Device Type              : SAS/SATA Mix
Allow Mix in Enclosure           : Yes
Allow HDD SAS/SATA Mix in VD     : Yes
Allow SSD SAS/SATA Mix in VD     : No
Allow HDD/SSD Mix in VD          : No
Allow SATA in Cluster            : No
Max Chained Enclosures           : 16 
Disable Ctrl-R                   : Yes
Enable Web BIOS                  : Yes
Direct PD Mapping                : No
BIOS Enumerate VDs               : Yes
Restore Hot Spare on Insertion   : No
Expose Enclosure Devices         : Yes
Maintain PD Fail History         : Yes
Disable Puncturing               : No
Zero Based Enclosure Enumeration : No
PreBoot CLI Enabled              : Yes
LED Show Drive Activity          : Yes
Cluster Disable                  : Yes
SAS Disable                      : No
Auto Detect BackPlane Enable     : SGPIO/i2c SEP
Use FDE Only                     : No
Enable Led Header                : No
Delay during POST                : 0 
EnableCrashDump                  : No
Disable Online Controller Reset  : No
EnableLDBBM                      : No
Un-Certified Hard Disk Drives    : Allow
Treat Single span R1E as R10     : No
Max LD per array                 : 16
Power Saving option              : Don't Auto spin down Configured Drives
Max power savings option is  not allowed for LDs. Only T10 power conditions are to be used.
Default spin down time in minutes: 30 
Enable JBOD                      : No
TTY Log In Flash                 : No
Auto Enhanced Import             : No
BreakMirror RAID Support         : No
Disable Join Mirror              : No
Time taken to detect CME         : 60s

Exit Code: 0x00

[-- Attachment #4: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-12 15:56   ` Matthew Whittaker-Williams
@ 2012-06-12 17:40     ` Matthew Whittaker-Williams
  2012-06-13  0:12     ` Stan Hoeppner
  2012-06-13  1:19     ` Dave Chinner
  2 siblings, 0 replies; 20+ messages in thread
From: Matthew Whittaker-Williams @ 2012-06-12 17:40 UTC (permalink / raw)
  To: xfs

On 6/12/12 5:56 PM, Matthew Whittaker-Williams wrote:

Few more traces:

[ 2162.950080] INFO: task kworker/0:1:11 blocked for more than 120 seconds.
[ 2162.950172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[ 2162.950265] kworker/0:1     D ffff880603cac668     0    11      2 
0x00000000
[ 2162.950414]  ffff880603df9bf0 0000000000000046 0000000000000000 
ffff880603c56840
[ 2162.950652]  0000000000000400 ffff880603df8010 0000000000013340 
0000000000013340
[ 2162.950890]  ffff880603df9fd8 0000000000013340 ffff880603df9fd8 
0000000000013340
[ 2162.951128] Call Trace:
[ 2162.951195]  [<ffffffff8137e2af>] schedule+0x5f/0x61
[ 2162.951264]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
[ 2162.951334]  [<ffffffff8137d912>] __down_common+0x96/0xe4
[ 2162.951423]  [<ffffffffa037e38c>] ? xfs_getsb+0x32/0x60 [xfs]
[ 2162.951493]  [<ffffffff8137d9bf>] __down+0x18/0x1a
[ 2162.951562]  [<ffffffff81060428>] down+0x28/0x38
[ 2162.951637]  [<ffffffffa032f786>] xfs_buf_lock+0x6f/0xc0 [xfs]
[ 2162.951720]  [<ffffffffa037e38c>] xfs_getsb+0x32/0x60 [xfs]
[ 2162.951800]  [<ffffffffa038b65c>] xfs_trans_getsb+0xb3/0x107 [xfs]
[ 2162.951885]  [<ffffffffa037eda5>] xfs_mod_sb+0x41/0x112 [xfs]
[ 2162.951967]  [<ffffffffa033ebac>] ? xfs_flush_inodes+0x2e/0x2e [xfs]
[ 2162.952048]  [<ffffffffa0334434>] xfs_fs_log_dummy+0x61/0x76 [xfs]
[ 2162.952130]  [<ffffffffa0385656>] ? xfs_log_need_covered+0x5a/0xb4 [xfs]
[ 2162.952214]  [<ffffffffa033ebea>] xfs_sync_worker+0x3e/0x64 [xfs]
[ 2162.957959]  [<ffffffff810550d1>] process_one_work+0x1da/0x2e9
[ 2162.958030]  [<ffffffff8105531e>] worker_thread+0x13e/0x264
[ 2162.958099]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
[ 2162.958171]  [<ffffffff8105bb50>] kthread+0x84/0x8c
[ 2162.958242]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
[ 2162.958313]  [<ffffffff8105bacc>] ? 
kthread_freezable_should_stop+0x5d/0x5d
[ 2162.958387]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
[ 4083.044756] INFO: task kworker/0:2:538 blocked for more than 120 seconds.
[ 4083.044850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[ 4083.044944] kworker/0:2     D ffff8805ec091de8     0   538      2 
0x00000000
[ 4083.045092]  ffff8805ebb17bf0 0000000000000046 0000000000000000 
ffff880603c56840
[ 4083.045330]  0000000000000400 ffff8805ebb16010 0000000000013340 
0000000000013340
[ 4083.045567]  ffff8805ebb17fd8 0000000000013340 ffff8805ebb17fd8 
0000000000013340
[ 4083.045806] Call Trace:
[ 4083.045874]  [<ffffffff8137e2af>] schedule+0x5f/0x61
[ 4083.045943]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
[ 4083.046013]  [<ffffffff8137d912>] __down_common+0x96/0xe4
[ 4083.046101]  [<ffffffffa037e38c>] ? xfs_getsb+0x32/0x60 [xfs]
[ 4083.046171]  [<ffffffff8137d9bf>] __down+0x18/0x1a
[ 4083.046240]  [<ffffffff81060428>] down+0x28/0x38
[ 4083.046315]  [<ffffffffa032f786>] xfs_buf_lock+0x6f/0xc0 [xfs]
[ 4083.046399]  [<ffffffffa037e38c>] xfs_getsb+0x32/0x60 [xfs]
[ 4083.046479]  [<ffffffffa038b65c>] xfs_trans_getsb+0xb3/0x107 [xfs]
[ 4083.046564]  [<ffffffffa037eda5>] xfs_mod_sb+0x41/0x112 [xfs]
[ 4083.046646]  [<ffffffffa033ebac>] ? xfs_flush_inodes+0x2e/0x2e [xfs]
[ 4083.046726]  [<ffffffffa0334434>] xfs_fs_log_dummy+0x61/0x76 [xfs]
[ 4083.046810]  [<ffffffffa0385656>] ? xfs_log_need_covered+0x5a/0xb4 [xfs]
[ 4083.046895]  [<ffffffffa033ebea>] xfs_sync_worker+0x3e/0x64 [xfs]
[ 4083.046968]  [<ffffffff810550d1>] process_one_work+0x1da/0x2e9
[ 4083.047038]  [<ffffffff8105531e>] worker_thread+0x13e/0x264
[ 4083.047107]  [<ffffffff810551e0>] ? process_one_work+0x2e9/0x2e9
[ 4083.047178]  [<ffffffff8105bb50>] kthread+0x84/0x8c
[ 4083.047248]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
[ 4083.047319]  [<ffffffff8105bacc>] ? 
kthread_freezable_should_stop+0x5d/0x5d
[ 4083.047392]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13


Thanks

Kind regards

Matthew

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-12 15:56   ` Matthew Whittaker-Williams
  2012-06-12 17:40     ` Matthew Whittaker-Williams
@ 2012-06-13  0:12     ` Stan Hoeppner
  2012-06-13  1:19     ` Dave Chinner
  2 siblings, 0 replies; 20+ messages in thread
From: Stan Hoeppner @ 2012-06-13  0:12 UTC (permalink / raw)
  To: Matthew Whittaker-Williams; +Cc: xfs

On 6/12/2012 10:56 AM, Matthew Whittaker-Williams wrote:

>          =                       sunit=0      swidth=0 blks

> RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
> Size                : 40.014 TB
> Strip Size          : 64 KB
> Number Of Drives    : 24

This may or may not be directly related to your issue as you describe
the problem workload as primarily read oriented.  But I should point out
you've got a stripe size of 22*64K = 1408KB, yet you're doing 4KB
unaligned writeout.  An unaligned XFS typically leaves a lot of
performance on the table, and can have disastrous results with parity
RAID RMW cycles.  Though with 4KB writeout you're probably not going to
hit the same problem as with misalignment due to incorrect stripe
offset, such as specifying an effective stripe size of 1344 when the
hardware stripe is 1408 (off by 1 spindle).

Is there a particular reason you didn't create these two filesystems with:

~$ mkfs.xfs -d su=64k,sw=22 /dev/sdX

Surely I'm missing something.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-12 15:56   ` Matthew Whittaker-Williams
  2012-06-12 17:40     ` Matthew Whittaker-Williams
  2012-06-13  0:12     ` Stan Hoeppner
@ 2012-06-13  1:19     ` Dave Chinner
  2012-06-13  3:56       ` Stan Hoeppner
  2012-06-13  8:54       ` Matthew Whittaker-Williams
  2 siblings, 2 replies; 20+ messages in thread
From: Dave Chinner @ 2012-06-13  1:19 UTC (permalink / raw)
  To: Matthew Whittaker-Williams; +Cc: xfs

On Tue, Jun 12, 2012 at 05:56:23PM +0200, Matthew Whittaker-Williams wrote:
> On 6/12/12 3:18 AM, Dave Chinner wrote:
> >On Mon, Jun 11, 2012 at 11:37:23PM +0200, Matthew Whittaker-Williams wrote:
> >>[ 6110.300098]  [<ffffffff813569a4>] ? kernel_thread_helper+0x4/0x10
> >That's pretty much a meaningless stack trace. Can you recompile your
> >kernel with frame pointers enabled so we can get a reliable stack
> >trace?
> 
> See attached for new trace.

Thatnks, that's much more informative.

> >>Could you have a look into this issue?
> >We know there is a lurking problem that we've been trying to flush
> >out over the past couple of months. Do a search for hangs in
> >xlog_grant_log_space - we've found several problems in
> >the process, but there's still a remaining hang that is likely to be
> >the source of your problems.
> 
> I see, it indeed seems about the same issues we encounter.

With the valid stack traces, I see that it isn't related to the log,
though.

> >>If you need any more information I am happy to provide it.
> >What workload are you running that triggers this?
> 
> About our workload, we are providing usenet with diablo.
> We are currently triggering the issue with running several
> diloadfromspool commands.
> This will scan the entire spool and extracts article location size
> and message-id and placement information.
.....
> But we have also been able to produce the same trigger with running
> multiple bonnie++ commands.
.....
> It gets triggered when filesystem gets generally alot of reads.

And that is what makes it different to the other hangs we've been
seeing.

> Filesystem info:

Looks like a standard 41TB filesystem layout.

> RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
> Size                : 40.014 TB
> State               : Optimal
> Strip Size          : 64 KB
> Number Of Drives    : 24
.....
> Virtual Drive: 1 (Target Id: 1)
> Name                :
> RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
> Size                : 40.014 TB
> State               : Optimal
> Strip Size          : 1.0 MB
> Number Of Drives    : 24

OOC, any reason for the different stripe sizes on the two
RAID volumes?

It does, however, point to your problem - you have 24 disk RAID6
volumes and you are reading and writing lots of small files on them.
Every small random file write is going to cause a RMW cycle in the
RAID6 volume (i.e. across all 24 disks), so it is going to be *very*
slow when the BBWC can't hide that latency.

That, in turn, is going to make reads very slow as the are going to
get queued up behind those RMW cycles as the BBWC flushes all those
small writes it has buffered.

what does the output of a coupl eof minutes of 'iostat -d -x -m 5'
look like when this problem is occurring?

>  imklog 4.6.4, log source = /proc/kmsg started.
>  [  323.966564] XFS (sdb): Mounting Filesystem
>  [  324.183373] XFS (sdb): Ending clean mount
>  Kernel logging (proc) stopped.
>  imklog 4.6.4, log source = /proc/kmsg started.
>  [ 1200.374551] INFO: task kworker/0:2:1115 blocked for more than 120 seconds.
>  [ 1200.374628] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>  [ 1200.374722] kworker/0:2     D ffff8805ebdd3fa8     0  1115      2 0x00000000
>  [ 1200.374860]  ffff880602ae7bf0 0000000000000046 ffffffff814cecd0 ffff880603c56840
>  [ 1200.375098]  0000000000000400 ffff880602ae6010 0000000000013340 0000000000013340
>  [ 1200.375336]  ffff880602ae7fd8 0000000000013340 ffff880602ae7fd8 0000000000013340
>  [ 1200.375573] Call Trace:
>  [ 1200.375641]  [<ffffffff8137e2af>] schedule+0x5f/0x61
>  [ 1200.375709]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
>  [ 1200.375775]  [<ffffffff8137d912>] __down_common+0x96/0xe4
>  [ 1200.375858]  [<ffffffffa034338c>] ? xfs_getsb+0x32/0x60 [xfs]
>  [ 1200.375928]  [<ffffffff8137d9bf>] __down+0x18/0x1a
>  [ 1200.375996]  [<ffffffff81060428>] down+0x28/0x38
>  [ 1200.376071]  [<ffffffffa02f4786>] xfs_buf_lock+0x6f/0xc0 [xfs]
>  [ 1200.376155]  [<ffffffffa034338c>] xfs_getsb+0x32/0x60 [xfs]
>  [ 1200.376235]  [<ffffffffa035065c>] xfs_trans_getsb+0xb3/0x107 [xfs]
>  [ 1200.376319]  [<ffffffffa0343da5>] xfs_mod_sb+0x41/0x112 [xfs]
>  [ 1200.376401]  [<ffffffffa0303bac>] ? xfs_flush_inodes+0x2e/0x2e [xfs]
>  [ 1200.376482]  [<ffffffffa02f9434>] xfs_fs_log_dummy+0x61/0x76 [xfs]
>  [ 1200.382394]  [<ffffffffa034a656>] ? xfs_log_need_covered+0x5a/0xb4 [xfs]
>  [ 1200.382534]  [<ffffffffa0303bea>] xfs_sync_worker+0x3e/0x64 [xfs]
>  [ 1200.382607]  [<ffffffff810550d1>] process_one_work+0x1da/0x2e9
>  [ 1200.382741]  [<ffffffff8105531e>] worker_thread+0x13e/0x264

That is blocked trying to read the superblock, which means it is
probably locked for writeback. i.e. it's sitting in an IO queue
somewhere. The xfs_sync_worker is not blocked on log space which
means this is not the same as the problem I mentioned previously.


> [ 1201.674223] bonnie++        D ffff8805bc4f02e8     0  6337      1 0x00000000
> [ 1201.674370]  ffff88059b5f59b8 0000000000000086 ffff880601a53bb8 0000000000000286
> [ 1201.674607]  ffff88059b5f5968 ffff88059b5f4010 0000000000013340 0000000000013340
> [ 1201.674844]  ffff88059b5f5fd8 0000000000013340 ffff88059b5f5fd8 0000000000013340
> [ 1201.675081] Call Trace:
> [ 1201.675142]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
> [ 1201.675212]  [<ffffffff8137e2af>] schedule+0x5f/0x61
> [ 1201.675280]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
> [ 1201.675350]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
> [ 1201.675420]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
> [ 1201.675491]  [<ffffffff810d5ecf>] ? bdi_dirty_limit+0x2c/0x8b
> [ 1201.675561]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
> [ 1201.675633]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
> [ 1201.675723]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
> [ 1201.675858]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
> [ 1201.675939]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
> [ 1201.676039]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
> [ 1201.676117]  [<ffffffffa02f8002>] ? xfs_file_aio_write+0xb/0x1b0 [xfs]
> [ 1201.676188]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
> [ 1201.676257]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
> [ 1201.676327]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
> [ 1201.676394]  [<ffffffff81111292>] sys_write+0x45/0x6c
> [ 1201.676461]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b

All the bonnie++ processes are waiting for write IO to complete so
they can dirty more pages (i.e. write more in to the page cache).

>  [ 1201.700104] flush-8:0       D ffff8805bcaf5aa8     0  6346      2 0x00000000
>  [ 1201.700250]  ffff88059d78f710 0000000000000046 ffff88059d78f700 ffff88059d78f6e0
>  [ 1201.700487]  ffff8805eaef8000 ffff88059d78e010 0000000000013340 0000000000013340
>  [ 1201.700724]  ffff88059d78ffd8 0000000000013340 ffff88059d78ffd8 0000000000013340
>  [ 1201.700961] Call Trace:
>  [ 1201.701022]  [<ffffffff8137e2af>] schedule+0x5f/0x61
>  [ 1201.701090]  [<ffffffff8137e338>] io_schedule+0x87/0xca
>  [ 1201.701159]  [<ffffffff811aca59>] get_request_wait+0x116/0x1b9
>  [ 1201.701229]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
>  [ 1201.701298]  [<ffffffff811a70d2>] ? elv_merge+0xae/0xbe
>  [ 1201.701367]  [<ffffffff811acf8f>] blk_queue_bio+0x1a8/0x30d
>  [ 1201.701436]  [<ffffffff811ac2e5>] generic_make_request+0x99/0xdc
>  [ 1201.701507]  [<ffffffff811ac407>] submit_bio+0xdf/0xfe
>  [ 1201.701580]  [<ffffffffa02f1c3d>] xfs_submit_ioend_bio+0x2e/0x30 [xfs]
>  [ 1201.701658]  [<ffffffffa02f236e>] xfs_submit_ioend+0xa1/0xea [xfs]
>  [ 1201.701736]  [<ffffffffa02f353f>] xfs_vm_writepage+0x40f/0x484 [xfs]
>  [ 1201.701808]  [<ffffffff811c761c>] ? rb_insert_color+0xbc/0xe5
>  [ 1201.701878]  [<ffffffff810d558a>] __writepage+0x12/0x2b
>  [ 1201.701946]  [<ffffffff810d6a0d>] write_cache_pages+0x25f/0x36a
>  [ 1201.702017]  [<ffffffff810d5578>] ? set_page_dirty+0x62/0x62
>  [ 1201.702087]  [<ffffffff810d6b58>] generic_writepages+0x40/0x57
>  [ 1201.702162]  [<ffffffffa02f1a30>] xfs_vm_writepages+0x48/0x4f [xfs]
>  [ 1201.702234]  [<ffffffff810d6b8b>] do_writepages+0x1c/0x25
>  [ 1201.702304]  [<ffffffff81130a04>] writeback_single_inode+0x183/0x370
>  [ 1201.702376]  [<ffffffff81130eaa>] writeback_sb_inodes+0x172/0x20b
>  [ 1201.702447]  [<ffffffff81130fb6>] __writeback_inodes_wb+0x73/0xb4
>  [ 1201.702518]  [<ffffffff8113162b>] wb_writeback+0x136/0x22c
>  [ 1201.702587]  [<ffffffff810d61c1>] ? global_dirty_limits+0x2a/0x10a
>  [ 1201.702659]  [<ffffffff811317b1>] wb_do_writeback+0x90/0x1de
>  [ 1201.702729]  [<ffffffff811319bf>] bdi_writeback_thread+0xc0/0x1e5
>  [ 1201.702800]  [<ffffffff811318ff>] ? wb_do_writeback+0x1de/0x1de
>  [ 1201.702871]  [<ffffffff811318ff>] ? wb_do_writeback+0x1de/0x1de
>  [ 1201.702941]  [<ffffffff8105bb50>] kthread+0x84/0x8c
>  [ 1201.703009]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
>  [ 1201.703079]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
>  [ 1201.703152]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13

And writeback is stalled waiting for IO to complete as well (hence
the bonnie++ processes being blocked). IN this case, the IO request
queue is full, which indicates it is waiting on the disk subsystem
to complete IO requests....

>  [ 1559.653179] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>  [ 1559.653269] sync            D ffff8805ec48b8e8     0  7493   7169 0x00000000
>  [ 1559.653417]  ffff880208869cf8 0000000000000086 ffff8805bc02ae78 00000000000000c9
>  [ 1559.653655]  ffff880208869ce8 ffff880208868010 0000000000013340 0000000000013340
>  [ 1559.653892]  ffff880208869fd8 0000000000013340 ffff880208869fd8 0000000000013340
>  [ 1559.654129] Call Trace:
>  [ 1559.654197]  [<ffffffff8137e2af>] schedule+0x5f/0x61
>  [ 1559.654265]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
>  [ 1559.654337]  [<ffffffff8106b47b>] ? wake_affine+0x190/0x248
>  [ 1559.660092]  [<ffffffff8106ec4e>] ? enqueue_entity+0x121/0x1cd
>  [ 1559.660163]  [<ffffffff8137e136>] wait_for_common+0xcc/0x14a
>  [ 1559.660235]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
>  [ 1559.660307]  [<ffffffff810440c8>] ? local_bh_enable_ip+0x9/0xb
>  [ 1559.660382]  [<ffffffff811348f7>] ? __sync_filesystem+0x7a/0x7a
>  [ 1559.660452]  [<ffffffff8137e24e>] wait_for_completion+0x18/0x1a
>  [ 1559.660524]  [<ffffffff81131203>] writeback_inodes_sb_nr+0x78/0x7f
>  [ 1559.660596]  [<ffffffff811312b0>] writeback_inodes_sb+0x4e/0x59
>  [ 1559.660667]  [<ffffffff811348ce>] __sync_filesystem+0x51/0x7a
>  [ 1559.660738]  [<ffffffff81134908>] sync_one_sb+0x11/0x13
>  [ 1559.660827]  [<ffffffff81112aae>] iterate_supers+0x69/0xbb
>  [ 1559.660898]  [<ffffffff81134939>] sys_sync+0x2f/0x5c
>  [ 1559.661052]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b

And that is sync waiting for the flusher thread to complete
writeback of all the dirty inodes. The lack of other stall messages
at this time makes it pretty clear that the problem is not
filesystem related - the system is simply writeback IO bound.

The reason, I'd suggest, is that you've chosen the wrong RAID volume
type for your workload. Small random file read and write workloads
like news and mail spoolers are IOPS intensive workloads and do
not play well with RAID5/6. RAID5/6 really only work well for large
files with sequential access patterns - you need to use RAID1/10 for
IOPS intensive workloads because they don't suffer from the RMW
cycle problem that RAID5/6 has for small writes. The iostat output
will help clarify whether this is really the problem or not...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-13  1:19     ` Dave Chinner
@ 2012-06-13  3:56       ` Stan Hoeppner
  2012-06-13  8:54       ` Matthew Whittaker-Williams
  1 sibling, 0 replies; 20+ messages in thread
From: Stan Hoeppner @ 2012-06-13  3:56 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Matthew Whittaker-Williams, xfs

On 6/12/2012 8:19 PM, Dave Chinner wrote:
> On Tue, Jun 12, 2012 at 05:56:23PM +0200, Matthew Whittaker-Williams wrote:

>> RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
>> Size                : 40.014 TB
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 24
> .....
>> Virtual Drive: 1 (Target Id: 1)
>> Name                :
>> RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
>> Size                : 40.014 TB
>> State               : Optimal
>> Strip Size          : 1.0 MB
>> Number Of Drives    : 24

> The reason, I'd suggest, is that you've chosen the wrong RAID volume
> type for your workload. Small random file read and write workloads
> like news and mail spoolers are IOPS intensive workloads and do
> not play well with RAID5/6. RAID5/6 really only work well for large
> files with sequential access patterns - you need to use RAID1/10 for
> IOPS intensive workloads because they don't suffer from the RMW
> cycle problem that RAID5/6 has for small writes. The iostat output
> will help clarify whether this is really the problem or not...

If it is the problem, you'll want to consider something like the
following, assuming your working files are somewhat evenly spread over
24 or more directories and/or subdirectories.

1.  For each 24 drive JBOD, create 3x 8 drive RAID10s, strip 64KB.  That
yields a relatively small stripe of 256KB over 4 spindles.  Should be a
good fit for small file random IOPS.

2.  Make an md linear array of the 3 hardware RAID10 arrays, such as:

~$ mdadm -C /dev/md0 -n3 -l linear /dev/sd[abc]

3.  Create your stripe aligned XFS over the md linear array:

~$ mkfs.xfs -d su=64K,sw=4 agcount=24

This will yield excellent small file IOPS, ~1800 peak at the spindles,
while still giving a decent ~400MB/s kick if you have some streaming
workloads from time to time.  Absolute XFS small file high IOPS
performance is attained with a linear array of RAID1 pairs, but with 12
physical device names this tends to get unwieldy in mdadm.  And as I
mentioned this hybrid architecture still allows decent streaming
performance if you need it.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-13  1:19     ` Dave Chinner
  2012-06-13  3:56       ` Stan Hoeppner
@ 2012-06-13  8:54       ` Matthew Whittaker-Williams
  2012-06-13 11:59         ` Andre Noll
  2012-06-14  0:04         ` Dave Chinner
  1 sibling, 2 replies; 20+ messages in thread
From: Matthew Whittaker-Williams @ 2012-06-13  8:54 UTC (permalink / raw)
  To: xfs


[-- Attachment #1.1: Type: text/plain, Size: 13695 bytes --]

On 6/13/12 3:19 AM, Dave Chinner wrote:
>
> With the valid stack traces, I see that it isn't related to the log,
> though.

Ah ok, we are triggering a new issue?
>
>>>> If you need any more information I am happy to provide it.
>>> What workload are you running that triggers this?
>> About our workload, we are providing usenet with diablo.
>> We are currently triggering the issue with running several
>> diloadfromspool commands.
>> This will scan the entire spool and extracts article location size
>> and message-id and placement information.
> .....
>> But we have also been able to produce the same trigger with running
>> multiple bonnie++ commands.
> .....
>> It gets triggered when filesystem gets generally alot of reads.
> And that is what makes it different to the other hangs we've been
> seeing.
>
>> Filesystem info:
> Looks like a standard 41TB filesystem layout.

Correct
>
>> RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
>> Size                : 40.014 TB
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 24
> .....
>> Virtual Drive: 1 (Target Id: 1)
>> Name                :
>> RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
>> Size                : 40.014 TB
>> State               : Optimal
>> Strip Size          : 1.0 MB
>> Number Of Drives    : 24
> OOC, any reason for the different stripe sizes on the two
> RAID volumes?

This is a fluke, we are running several new systems and this is just one 
of the new servers.
Which indeed has a wrong stripe set, this should be 1MB.
We actually found stripe size set of 1MB to give better performance 
overall than 64/256/512
>
> It does, however, point to your problem - you have 24 disk RAID6
> volumes and you are reading and writing lots of small files on them.
> Every small random file write is going to cause a RMW cycle in the
> RAID6 volume (i.e. across all 24 disks), so it is going to be *very*
> slow when the BBWC can't hide that latency.
>
> That, in turn, is going to make reads very slow as the are going to
> get queued up behind those RMW cycles as the BBWC flushes all those
> small writes it has buffered.
>
> what does the output of a coupl eof minutes of 'iostat -d -x -m 5'
> look like when this problem is occurring?

I will get back to you on this.
>
>>   imklog 4.6.4, log source = /proc/kmsg started.
>>   [  323.966564] XFS (sdb): Mounting Filesystem
>>   [  324.183373] XFS (sdb): Ending clean mount
>>   Kernel logging (proc) stopped.
>>   imklog 4.6.4, log source = /proc/kmsg started.
>>   [ 1200.374551] INFO: task kworker/0:2:1115 blocked for more than 120 seconds.
>>   [ 1200.374628] "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>   [ 1200.374722] kworker/0:2     D ffff8805ebdd3fa8     0  1115      2 0x00000000
>>   [ 1200.374860]  ffff880602ae7bf0 0000000000000046 ffffffff814cecd0 ffff880603c56840
>>   [ 1200.375098]  0000000000000400 ffff880602ae6010 0000000000013340 0000000000013340
>>   [ 1200.375336]  ffff880602ae7fd8 0000000000013340 ffff880602ae7fd8 0000000000013340
>>   [ 1200.375573] Call Trace:
>>   [ 1200.375641]  [<ffffffff8137e2af>] schedule+0x5f/0x61
>>   [ 1200.375709]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
>>   [ 1200.375775]  [<ffffffff8137d912>] __down_common+0x96/0xe4
>>   [ 1200.375858]  [<ffffffffa034338c>] ? xfs_getsb+0x32/0x60 [xfs]
>>   [ 1200.375928]  [<ffffffff8137d9bf>] __down+0x18/0x1a
>>   [ 1200.375996]  [<ffffffff81060428>] down+0x28/0x38
>>   [ 1200.376071]  [<ffffffffa02f4786>] xfs_buf_lock+0x6f/0xc0 [xfs]
>>   [ 1200.376155]  [<ffffffffa034338c>] xfs_getsb+0x32/0x60 [xfs]
>>   [ 1200.376235]  [<ffffffffa035065c>] xfs_trans_getsb+0xb3/0x107 [xfs]
>>   [ 1200.376319]  [<ffffffffa0343da5>] xfs_mod_sb+0x41/0x112 [xfs]
>>   [ 1200.376401]  [<ffffffffa0303bac>] ? xfs_flush_inodes+0x2e/0x2e [xfs]
>>   [ 1200.376482]  [<ffffffffa02f9434>] xfs_fs_log_dummy+0x61/0x76 [xfs]
>>   [ 1200.382394]  [<ffffffffa034a656>] ? xfs_log_need_covered+0x5a/0xb4 [xfs]
>>   [ 1200.382534]  [<ffffffffa0303bea>] xfs_sync_worker+0x3e/0x64 [xfs]
>>   [ 1200.382607]  [<ffffffff810550d1>] process_one_work+0x1da/0x2e9
>>   [ 1200.382741]  [<ffffffff8105531e>] worker_thread+0x13e/0x264
> That is blocked trying to read the superblock, which means it is
> probably locked for writeback. i.e. it's sitting in an IO queue
> somewhere. The xfs_sync_worker is not blocked on log space which
> means this is not the same as the problem I mentioned previously.
>
>
>> [ 1201.674223] bonnie++        D ffff8805bc4f02e8     0  6337      1 0x00000000
>> [ 1201.674370]  ffff88059b5f59b8 0000000000000086 ffff880601a53bb8 0000000000000286
>> [ 1201.674607]  ffff88059b5f5968 ffff88059b5f4010 0000000000013340 0000000000013340
>> [ 1201.674844]  ffff88059b5f5fd8 0000000000013340 ffff88059b5f5fd8 0000000000013340
>> [ 1201.675081] Call Trace:
>> [ 1201.675142]  [<ffffffff810d0ef2>] ? __probe_kernel_read+0x36/0x55
>> [ 1201.675212]  [<ffffffff8137e2af>] schedule+0x5f/0x61
>> [ 1201.675280]  [<ffffffff8137d00a>] schedule_timeout+0xa6/0xde
>> [ 1201.675350]  [<ffffffff8104a2d7>] ? lock_timer_base+0x4d/0x4d
>> [ 1201.675420]  [<ffffffff8137da54>] io_schedule_timeout+0x93/0xe4
>> [ 1201.675491]  [<ffffffff810d5ecf>] ? bdi_dirty_limit+0x2c/0x8b
>> [ 1201.675561]  [<ffffffff810d70a5>] balance_dirty_pages+0x511/0x5ba
>> [ 1201.675633]  [<ffffffff810d7224>] balance_dirty_pages_ratelimited_nr+0xd6/0xd8
>> [ 1201.675723]  [<ffffffff810cd1fc>] generic_perform_write+0x192/0x1df
>> [ 1201.675858]  [<ffffffff810cd29c>] generic_file_buffered_write+0x53/0x7d
>> [ 1201.675939]  [<ffffffffa02f7d57>] xfs_file_buffered_aio_write+0xf2/0x156 [xfs]
>> [ 1201.676039]  [<ffffffffa02f8155>] xfs_file_aio_write+0x15e/0x1b0 [xfs]
>> [ 1201.676117]  [<ffffffffa02f8002>] ? xfs_file_aio_write+0xb/0x1b0 [xfs]
>> [ 1201.676188]  [<ffffffff811107e3>] do_sync_write+0xc9/0x106
>> [ 1201.676257]  [<ffffffff8117c919>] ? security_file_permission+0x29/0x2e
>> [ 1201.676327]  [<ffffffff8111117d>] vfs_write+0xa9/0x105
>> [ 1201.676394]  [<ffffffff81111292>] sys_write+0x45/0x6c
>> [ 1201.676461]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
> All the bonnie++ processes are waiting for write IO to complete so
> they can dirty more pages (i.e. write more in to the page cache).
>
>>   [ 1201.700104] flush-8:0       D ffff8805bcaf5aa8     0  6346      2 0x00000000
>>   [ 1201.700250]  ffff88059d78f710 0000000000000046 ffff88059d78f700 ffff88059d78f6e0
>>   [ 1201.700487]  ffff8805eaef8000 ffff88059d78e010 0000000000013340 0000000000013340
>>   [ 1201.700724]  ffff88059d78ffd8 0000000000013340 ffff88059d78ffd8 0000000000013340
>>   [ 1201.700961] Call Trace:
>>   [ 1201.701022]  [<ffffffff8137e2af>] schedule+0x5f/0x61
>>   [ 1201.701090]  [<ffffffff8137e338>] io_schedule+0x87/0xca
>>   [ 1201.701159]  [<ffffffff811aca59>] get_request_wait+0x116/0x1b9
>>   [ 1201.701229]  [<ffffffff8105bfcf>] ? wake_up_bit+0x25/0x25
>>   [ 1201.701298]  [<ffffffff811a70d2>] ? elv_merge+0xae/0xbe
>>   [ 1201.701367]  [<ffffffff811acf8f>] blk_queue_bio+0x1a8/0x30d
>>   [ 1201.701436]  [<ffffffff811ac2e5>] generic_make_request+0x99/0xdc
>>   [ 1201.701507]  [<ffffffff811ac407>] submit_bio+0xdf/0xfe
>>   [ 1201.701580]  [<ffffffffa02f1c3d>] xfs_submit_ioend_bio+0x2e/0x30 [xfs]
>>   [ 1201.701658]  [<ffffffffa02f236e>] xfs_submit_ioend+0xa1/0xea [xfs]
>>   [ 1201.701736]  [<ffffffffa02f353f>] xfs_vm_writepage+0x40f/0x484 [xfs]
>>   [ 1201.701808]  [<ffffffff811c761c>] ? rb_insert_color+0xbc/0xe5
>>   [ 1201.701878]  [<ffffffff810d558a>] __writepage+0x12/0x2b
>>   [ 1201.701946]  [<ffffffff810d6a0d>] write_cache_pages+0x25f/0x36a
>>   [ 1201.702017]  [<ffffffff810d5578>] ? set_page_dirty+0x62/0x62
>>   [ 1201.702087]  [<ffffffff810d6b58>] generic_writepages+0x40/0x57
>>   [ 1201.702162]  [<ffffffffa02f1a30>] xfs_vm_writepages+0x48/0x4f [xfs]
>>   [ 1201.702234]  [<ffffffff810d6b8b>] do_writepages+0x1c/0x25
>>   [ 1201.702304]  [<ffffffff81130a04>] writeback_single_inode+0x183/0x370
>>   [ 1201.702376]  [<ffffffff81130eaa>] writeback_sb_inodes+0x172/0x20b
>>   [ 1201.702447]  [<ffffffff81130fb6>] __writeback_inodes_wb+0x73/0xb4
>>   [ 1201.702518]  [<ffffffff8113162b>] wb_writeback+0x136/0x22c
>>   [ 1201.702587]  [<ffffffff810d61c1>] ? global_dirty_limits+0x2a/0x10a
>>   [ 1201.702659]  [<ffffffff811317b1>] wb_do_writeback+0x90/0x1de
>>   [ 1201.702729]  [<ffffffff811319bf>] bdi_writeback_thread+0xc0/0x1e5
>>   [ 1201.702800]  [<ffffffff811318ff>] ? wb_do_writeback+0x1de/0x1de
>>   [ 1201.702871]  [<ffffffff811318ff>] ? wb_do_writeback+0x1de/0x1de
>>   [ 1201.702941]  [<ffffffff8105bb50>] kthread+0x84/0x8c
>>   [ 1201.703009]  [<ffffffff81386ae4>] kernel_thread_helper+0x4/0x10
>>   [ 1201.703079]  [<ffffffff8105bacc>] ? kthread_freezable_should_stop+0x5d/0x5d
>>   [ 1201.703152]  [<ffffffff81386ae0>] ? gs_change+0x13/0x13
> And writeback is stalled waiting for IO to complete as well (hence
> the bonnie++ processes being blocked). IN this case, the IO request
> queue is full, which indicates it is waiting on the disk subsystem
> to complete IO requests....
>
>>   [ 1559.653179] "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>   [ 1559.653269] sync            D ffff8805ec48b8e8     0  7493   7169 0x00000000
>>   [ 1559.653417]  ffff880208869cf8 0000000000000086 ffff8805bc02ae78 00000000000000c9
>>   [ 1559.653655]  ffff880208869ce8 ffff880208868010 0000000000013340 0000000000013340
>>   [ 1559.653892]  ffff880208869fd8 0000000000013340 ffff880208869fd8 0000000000013340
>>   [ 1559.654129] Call Trace:
>>   [ 1559.654197]  [<ffffffff8137e2af>] schedule+0x5f/0x61
>>   [ 1559.654265]  [<ffffffff8137cf95>] schedule_timeout+0x31/0xde
>>   [ 1559.654337]  [<ffffffff8106b47b>] ? wake_affine+0x190/0x248
>>   [ 1559.660092]  [<ffffffff8106ec4e>] ? enqueue_entity+0x121/0x1cd
>>   [ 1559.660163]  [<ffffffff8137e136>] wait_for_common+0xcc/0x14a
>>   [ 1559.660235]  [<ffffffff8106a36c>] ? try_to_wake_up+0x1b4/0x1b4
>>   [ 1559.660307]  [<ffffffff810440c8>] ? local_bh_enable_ip+0x9/0xb
>>   [ 1559.660382]  [<ffffffff811348f7>] ? __sync_filesystem+0x7a/0x7a
>>   [ 1559.660452]  [<ffffffff8137e24e>] wait_for_completion+0x18/0x1a
>>   [ 1559.660524]  [<ffffffff81131203>] writeback_inodes_sb_nr+0x78/0x7f
>>   [ 1559.660596]  [<ffffffff811312b0>] writeback_inodes_sb+0x4e/0x59
>>   [ 1559.660667]  [<ffffffff811348ce>] __sync_filesystem+0x51/0x7a
>>   [ 1559.660738]  [<ffffffff81134908>] sync_one_sb+0x11/0x13
>>   [ 1559.660827]  [<ffffffff81112aae>] iterate_supers+0x69/0xbb
>>   [ 1559.660898]  [<ffffffff81134939>] sys_sync+0x2f/0x5c
>>   [ 1559.661052]  [<ffffffff813857f9>] system_call_fastpath+0x16/0x1b
> And that is sync waiting for the flusher thread to complete
> writeback of all the dirty inodes. The lack of other stall messages
> at this time makes it pretty clear that the problem is not
> filesystem related - the system is simply writeback IO bound.
>
> The reason, I'd suggest, is that you've chosen the wrong RAID volume
> type for your workload. Small random file read and write workloads
> like news and mail spoolers are IOPS intensive workloads and do
> not play well with RAID5/6. RAID5/6 really only work well for large
> files with sequential access patterns - you need to use RAID1/10 for
> IOPS intensive workloads because they don't suffer from the RMW
> cycle problem that RAID5/6 has for small writes. The iostat output
> will help clarify whether this is really the problem or not...

I understand that RAID 10 is better for performance for reads on small 
files sets.
But with raid 10 we of course loose a lot of disk space compared to RAID 6.
Side note to this we have been running RAID 6 for years now without any 
issues.

In the past we did tune our xfs filesystem with switches like sunit and 
swidth.
But back then we couldn't see much peformance difference between using:

mkfs.xfs -f -L P.01 -l lazy-count=1 -d su=1m,sw=22 /dev/sda

and

mkfs.xfs -f -L P.01 -l lazy-count=1 /dev/sda

xfs_info from a system that shows no problems with an H800 Controller 
from dell ( same chipset as the LSI controllers )

Product Name    : PERC H800 Adapter
Serial No       : 071002C
FW Package Build: 12.10.1-0001

sd60:~# xfs_info /dev/sda
meta-data=/dev/sda               isize=256    agcount=58, 
agsize=268435455 blks
          =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=15381037056, imaxpct=1
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Where we even have bigger spools:

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :p01
RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
Size                : 57.298 TB
State               : Optimal
Strip Size          : 1.0 MB
Number Of Drives    : 23
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAheadNone, Direct, Write Cache OK 
if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, Write Cache OK 
if Bad BBU
Access Policy       : Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: No



Aside from the wrong stripe set and write alignments, this still should 
not cause the kernel to crash like this.
We found that running with a newer driver of LSI it takes a bit longer 
for the kernel to crash but it still does.

>
> Cheers,
>
> Dave.

Thanks


[-- Attachment #1.2: Type: text/html, Size: 17793 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-13  8:54       ` Matthew Whittaker-Williams
@ 2012-06-13 11:59         ` Andre Noll
  2012-06-13 12:13           ` Michael Monnerie
  2012-06-14  0:04         ` Dave Chinner
  1 sibling, 1 reply; 20+ messages in thread
From: Andre Noll @ 2012-06-13 11:59 UTC (permalink / raw)
  To: Matthew Whittaker-Williams; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 986 bytes --]

On Wed, Jun 13, 10:54, Matthew Whittaker-Williams wrote:
> Current Cache Policy: WriteBack, ReadAheadNone, Direct, Write Cache OK if Bad BBU
> Disk Cache Policy   : Disk's Default

Check if these two policy settings are what you really want. AFAICT
they can make a huge difference, depending on the workload.

Here's what gives best performance. Of course these are also the most
risky settings, so use with care:

	Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU
	Disk Cache Policy: Enabled

To select these settings for all lds on adapter 0, use

	CLI=MegaCli; lsi_adapter_number=0
        $CLI -ldsetprop CachedBadBBU -LALL -a$lsi_adapter_number &&
        $CLI -ldsetprop -Cached -LAll -a$lsi_adapter_number &&
        $CLI -ldsetprop -EnDskCache -LAll -a$lsi_adapter_number &&
        $CLI -ldsetprop WB -LALL -a$lsi_adapter_number

Best
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-13 11:59         ` Andre Noll
@ 2012-06-13 12:13           ` Michael Monnerie
  2012-06-13 16:12             ` Stan Hoeppner
  0 siblings, 1 reply; 20+ messages in thread
From: Michael Monnerie @ 2012-06-13 12:13 UTC (permalink / raw)
  To: xfs


[-- Attachment #1.1: Type: text/plain, Size: 1207 bytes --]

Am Mittwoch, 13. Juni 2012, 13:59:34 schrieb Andre Noll:
> Here's what gives best performance. Of course these are also the most
> risky settings, so use with care:
> 
>         Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write
> Cache OK if Bad BBU, Disk Cache Policy: Enabled

I wanted to comment on the original posting, but it fits here too. Aside 
from this performance perspective, in order to have a system that 
survives crashes and powerouts, better use:
Disk (Write) Cache: disabled (is that Disk Cache Policy for writes, or 
read+writes?)
Write Cache DISABLED if Bad BBU 
While turning off disk write caches costs some performance always, 
disabling the controller write cache if the bbu is dead makes the server 
only slower during bbu outages, but at least it will survive a crash 
then.
It depends on your needs, but I don't know an admin with a +40TB storage 
that says "doesn't matter if it breaks, I'll restore from backup", as 
even with disk2disk backup that takes some time.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-13 12:13           ` Michael Monnerie
@ 2012-06-13 16:12             ` Stan Hoeppner
  2012-06-14  7:31               ` Michael Monnerie
  0 siblings, 1 reply; 20+ messages in thread
From: Stan Hoeppner @ 2012-06-13 16:12 UTC (permalink / raw)
  To: Michael Monnerie; +Cc: xfs

On 6/13/2012 7:13 AM, Michael Monnerie wrote:

> Write Cache DISABLED if Bad BBU

This is the LSI/3ware firmware default policy Michael, so every user is
safe out-of-the-box.  If the BBU/FBU is not present (never installed),
or the control logic determines it is not functioning properly, the
firmware disables writeback caching.

In absence of paying attention to logs/alerts, one will know pretty
quickly when the BBU has failed, as write performance with many/most
workloads will fall off a cliff.

-- 
Stan


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-13  8:54       ` Matthew Whittaker-Williams
  2012-06-13 11:59         ` Andre Noll
@ 2012-06-14  0:04         ` Dave Chinner
  2012-06-14 14:31           ` Matthew Whittaker-Williams
  1 sibling, 1 reply; 20+ messages in thread
From: Dave Chinner @ 2012-06-14  0:04 UTC (permalink / raw)
  To: Matthew Whittaker-Williams; +Cc: xfs

On Wed, Jun 13, 2012 at 10:54:04AM +0200, Matthew Whittaker-Williams wrote:
> On 6/13/12 3:19 AM, Dave Chinner wrote:
> >
> >With the valid stack traces, I see that it isn't related to the log,
> >though.
> 
> Ah ok, we are triggering a new issue?

No, your system appears to be stalling waiting for IO completion.

> >>RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
> >>Size                : 40.014 TB
> >>State               : Optimal
> >>Strip Size          : 64 KB
> >>Number Of Drives    : 24
> >.....
> >>Virtual Drive: 1 (Target Id: 1)
> >>Name                :
> >>RAID Level          : Primary-6, Secondary-0, RAID Level Qualifier-3
> >>Size                : 40.014 TB
> >>State               : Optimal
> >>Strip Size          : 1.0 MB
> >>Number Of Drives    : 24
> >OOC, any reason for the different stripe sizes on the two
> >RAID volumes?
> 
> This is a fluke, we are running several new systems and this is just
> one of the new servers.
> Which indeed has a wrong stripe set, this should be 1MB.
> We actually found stripe size set of 1MB to give better performance
> overall than 64/256/512

So if you fix that, does the problem go away?

> >And that is sync waiting for the flusher thread to complete
> >writeback of all the dirty inodes. The lack of other stall messages
> >at this time makes it pretty clear that the problem is not
> >filesystem related - the system is simply writeback IO bound.
> >
> >The reason, I'd suggest, is that you've chosen the wrong RAID volume
> >type for your workload. Small random file read and write workloads
> >like news and mail spoolers are IOPS intensive workloads and do
> >not play well with RAID5/6. RAID5/6 really only work well for large
> >files with sequential access patterns - you need to use RAID1/10 for
> >IOPS intensive workloads because they don't suffer from the RMW
> >cycle problem that RAID5/6 has for small writes. The iostat output
> >will help clarify whether this is really the problem or not...
> 

> I understand that RAID 10 is better for performance for reads on
> small files sets.  But with raid 10 we of course loose a lot of
> disk space compared to RAID 6.  Side note to this we have been
> running RAID 6 for years now without any issues.

but have you been running 24 disk RAID6 volumes? With RAID5/6, the
number of disks of the volume really matters - for small write IOs,
the more disks in the RAID6 volume, the slower it will be...

> In the past we did tune our xfs filesystem with switches like
> sunit and swidth.  But back then we couldn't see much peformance
> difference between using:
> 
> mkfs.xfs -f -L P.01 -l lazy-count=1 -d su=1m,sw=22 /dev/sda
> 
> and
> 
> mkfs.xfs -f -L P.01 -l lazy-count=1 /dev/sda

You won't see much difference with the BBWC enabled. It does affect
how files and inodes are allocated, though, so the aging
characteristics of the filesystem will be better for an aligned
filesystem. i.e. you might not notice the performance now, but after
a coupl eof years in production you probably will...

> xfs_info from a system that shows no problems with an H800
> Controller from dell ( same chipset as the LSI controllers )
> 
> Product Name    : PERC H800 Adapter
> Serial No       : 071002C
> FW Package Build: 12.10.1-0001
> 
> sd60:~# xfs_info /dev/sda
> meta-data=/dev/sda               isize=256    agcount=58,
> agsize=268435455 blks
>          =                       sectsz=512   attr=2
> data     =                       bsize=4096   blocks=15381037056, imaxpct=1
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal               bsize=4096   blocks=521728, version=2
>          =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> Where we even have bigger spools:

You have larger drives, not a wider RAID volume. That's a 23-disk
wide, 3TB drive RAID6 volume. And it's on a different controller
with different firmware, so there's lots different here...

> Aside from the wrong stripe set and write alignments, this still
> should not cause the kernel to crash like this.

The kernel is not crashing. It's emitting warnings that indicate the
IO subsystem is overloaded.

> We found that running with a newer driver of LSI it takes a bit
> longer for the kernel to crash but it still does.

Which indicates the problem is almost certainly related to the
storage configuration or drivers, not the filesystem....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-13 16:12             ` Stan Hoeppner
@ 2012-06-14  7:31               ` Michael Monnerie
  0 siblings, 0 replies; 20+ messages in thread
From: Michael Monnerie @ 2012-06-14  7:31 UTC (permalink / raw)
  To: xfs


[-- Attachment #1.1: Type: text/plain, Size: 1451 bytes --]

Am Mittwoch, 13. Juni 2012, 11:12:15 schrieb Stan Hoeppner:
> This is the LSI/3ware firmware default policy Michael, so every user
> is safe out-of-the-box.  If the BBU/FBU is not present (never
> installed), or the control logic determines it is not functioning
> properly, the firmware disables writeback caching.
> 
> In absence of paying attention to logs/alerts, one will know pretty
> quickly when the BBU has failed, as write performance with many/most
> workloads will fall off a cliff.

That's good, and I guess every serious raid with bbu behaves the same. I 
just wanted to make a statement because many people read "maximum 
performance" and want this then, without understanding the downsides aka 
"all data can be gone if a crash destroys the right metadata". 

I get calls from people then whining, and they expect me not yell at 
them but be friendly, which I don't like ;-)

I prefer to not speak about max perf if it means "probably eats your 
data". Maybe we should always write "set this for max performance" only 
with secure values, and then extra "and set this for turbo boost, but it 
eats your data on a crash". Hopefully this helps keeping people from 
just reading "max perf" and forget about the "eats your data" part.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-14  0:04         ` Dave Chinner
@ 2012-06-14 14:31           ` Matthew Whittaker-Williams
  2012-06-15  0:16             ` Dave Chinner
  0 siblings, 1 reply; 20+ messages in thread
From: Matthew Whittaker-Williams @ 2012-06-14 14:31 UTC (permalink / raw)
  To: xfs

[-- Attachment #1: Type: text/plain, Size: 11589 bytes --]

On 6/14/12 2:04 AM, Dave Chinner wrote:
> On Wed, Jun 13, 2012 at 10:54:04AM +0200, Matthew Whittaker-Williams wrote:
>> On 6/13/12 3:19 AM, Dave Chinner wrote:
>>> With the valid stack traces, I see that it isn't related to the log,
>>> though.
>> Ah ok, we are triggering a new issue?
> No, your system appears to be stalling waiting for IO completion.

Yes, this is indeed what we experience.
> This is a fluke, we are running several new systems and this is just
> one of the new servers.
> Which indeed has a wrong stripe set, this should be 1MB.
> We actually found stripe size set of 1MB to give better performance
> overall than 64/256/512
> So if you fix that, does the problem go away?

No , unfortunately not.

Currently with 1MB stripe set and:

root@sd70:~# xfs_info /dev/sda
meta-data=/dev/sda               isize=256    agcount=41, 
agsize=268435200 blks
          =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=10741350400, imaxpct=5
          =                       sunit=256    swidth=5632 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
          =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

It still stalled out with the same error.

>> I understand that RAID 10 is better for performance for reads on
>> small files sets.  But with raid 10 we of course loose a lot of
>> disk space compared to RAID 6.  Side note to this we have been
>> running RAID 6 for years now without any issues.
> but have you been running 24 disk RAID6 volumes? With RAID5/6, the
> number of disks of the volume really matters - for small write IOs,
> the more disks in the RAID6 volume, the slower it will be...
Yes we have, but haven't found large issues with performance in the past 
on a disk span of 24.
And most raid controllers don't support more than 32 disk in a raid 
array so we kept it at 24 disks per array with large arrays.

>
>> In the past we did tune our xfs filesystem with switches like
>> sunit and swidth.  But back then we couldn't see much peformance
>> difference between using:
>>
>> mkfs.xfs -f -L P.01 -l lazy-count=1 -d su=1m,sw=22 /dev/sda
>>
>> and
>>
>> mkfs.xfs -f -L P.01 -l lazy-count=1 /dev/sda
> You won't see much difference with the BBWC enabled. It does affect
> how files and inodes are allocated, though, so the aging
> characteristics of the filesystem will be better for an aligned
> filesystem. i.e. you might not notice the performance now, but after
> a coupl eof years in production you probably will...

We haven't seen this impact just yet and are doing about 120K number of 
reads of sector reads.
But this is probably not really high compared to our smaller arrays.

>
>> xfs_info from a system that shows no problems with an H800
>> Controller from dell ( same chipset as the LSI controllers )
>
I went and tried the H800 Controller with a single array of the new 
spool and unfortunately this also hung.

[ 6123.108138] INFO: task diablo:11963 blocked for more than 120 seconds.
[ 6123.108208] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[ 6123.108297] diablo          D ffff8805ec4e9068     0 11963  11210 
0x00000000
[ 6123.108444]  ffff88046fd315f8 0000000000000082 ffff8800dd1407a8 
ffff88025a8f5000
[ 6123.108681]  000000000025a8f5 ffff88046fd30010 0000000000013340 
0000000000013340
[ 6123.108935]  ffff88046fd31fd8 0000000000013340 ffff88046fd31fd8 
0000000000013340
[ 6123.109171] Call Trace:
[ 6123.109276]  [<ffffffff8137e3bf>] schedule+0x5f/0x61
[ 6123.109343]  [<ffffffff8137d0a5>] schedule_timeout+0x31/0xde
[ 6123.109415]  [<ffffffff810d0efe>] ? __probe_kernel_read+0x36/0x55
[ 6123.109487]  [<ffffffff8110b76d>] ? kmem_cache_alloc+0x61/0x118
[ 6123.109557]  [<ffffffff8137da22>] __down_common+0x96/0xe4
[ 6123.109634]  [<ffffffffa02e04b8>] ? _xfs_buf_find+0x1ea/0x299 [xfs]
[ 6123.109704]  [<ffffffff8137dacf>] __down+0x18/0x1a
[ 6123.109771]  [<ffffffff81060428>] down+0x28/0x38
[ 6123.109844]  [<ffffffffa02df786>] xfs_buf_lock+0x6f/0xc0 [xfs]
[ 6123.109922]  [<ffffffffa02e04b8>] _xfs_buf_find+0x1ea/0x299 [xfs]
[ 6123.110000]  [<ffffffffa02e0713>] xfs_buf_get+0x25/0x172 [xfs]
[ 6123.110090]  [<ffffffffa02e087a>] xfs_buf_read+0x1a/0xc5 [xfs]
[ 6123.110169]  [<ffffffffa033bc4c>] xfs_trans_read_buf+0x35d/0x54d [xfs]
[ 6123.110258]  [<ffffffffa0326507>] xfs_imap_to_bp+0x45/0x1fe [xfs]
[ 6123.110345]  [<ffffffffa032873e>] xfs_iread+0x5b/0x195 [xfs]
[ 6123.110423]  [<ffffffffa02e5e59>] xfs_iget_cache_miss+0x5e/0x1cf [xfs]
[ 6123.110507]  [<ffffffffa02e64e3>] xfs_iget+0xf7/0x184 [xfs]
[ 6123.110591]  [<ffffffffa0325b36>] xfs_ialloc+0xc1/0x5ef [xfs]
[ 6123.110673]  [<ffffffffa02f3b4d>] ? kmem_zone_zalloc+0x1f/0x30 [xfs]
[ 6123.110757]  [<ffffffffa0336007>] ? xlog_grant_head_check+0x8f/0x101 
[xfs]
[ 6123.110842]  [<ffffffffa02f00e1>] xfs_dir_ialloc+0x9d/0x284 [xfs]
[ 6123.110926]  [<ffffffffa02f382e>] xfs_create+0x2f5/0x547 [xfs]
[ 6123.111006]  [<ffffffffa02ea4a2>] xfs_vn_mknod+0xcc/0x160 [xfs]
[ 6123.111086]  [<ffffffffa02ea557>] xfs_vn_create+0xe/0x10 [xfs]
[ 6123.111156]  [<ffffffff8111aaba>] vfs_create+0x67/0x89
[ 6123.111224]  [<ffffffff8111b942>] do_last+0x236/0x565
[ 6123.111292]  [<ffffffff8111c23a>] path_openat+0xcb/0x30c
[ 6123.111360]  [<ffffffff8111c56a>] do_filp_open+0x38/0x84
[ 6123.111429]  [<ffffffff8111a32f>] ? getname_flags+0x15b/0x1e2
[ 6123.111499]  [<ffffffff8112694b>] ? alloc_fd+0x6c/0xfc
[ 6123.111566]  [<ffffffff8110f26c>] do_sys_open+0x6f/0x101
[ 6123.111634]  [<ffffffff8110f32b>] sys_open+0x1c/0x1e
[ 6123.111702]  [<ffffffff813858f9>] system_call_fastpath+0x16/0x1b


See attached dmesg.txt

iostat:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   81.80    1.40    10.22     0.18   
256.00   531.91 5349.11  12.02 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   83.40    1.20    10.37     0.15   
254.56   525.35 4350.67  11.82 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   79.20    0.80     9.90     0.10   
256.00   530.14 3153.38  12.50 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   72.80    2.20     9.09     0.13   
251.72   546.08 8709.54  13.33 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   79.80    1.40     9.95     0.12   
254.07   535.35 5172.22  12.32 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   99.60    1.20    12.41     0.08   
253.86   529.49 3560.89   9.92 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   60.80    1.40     7.59     0.11   
253.77   527.21 6545.50  16.08 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   79.00    1.80     9.84     0.08   
251.51   547.93 6400.42  12.38 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   82.20    2.20    10.25     0.01   
248.93   536.42 7415.77  11.85 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   89.40    2.20    11.17     0.01   
249.90   525.68 7232.96  10.92 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   82.00    1.20    10.22     0.08   
253.37   541.60 4170.95  12.02 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   62.80    2.60     7.85     0.14   
250.31   541.15 11260.81  15.29 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   85.00    1.80    10.61     0.21   
255.47   529.36 6514.85  11.52 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   75.20    1.40     9.38     0.11   
253.72   535.68 5416.70  13.05 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   66.80    1.20     8.33     0.11   
254.19   546.68 5459.11  14.71 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   81.40    0.80    10.15     0.10   
255.38   540.62 3171.57  12.17 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   72.20    1.20     9.02     0.15   
255.74   535.26 5345.51  13.62 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   91.00    1.00    11.35     0.12   
255.44   531.02 3637.72  10.87 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   81.00    1.60    10.12     0.20   
255.96   524.44 6513.22  12.11 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   72.80    2.40     9.04     0.26   
253.24   543.25 9071.66  13.30 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   73.80    1.20     9.18     0.15   
254.63   539.20 5087.91  13.33 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   79.20    1.40     9.90     0.18   
256.00   532.38 5592.38  12.41 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.20   79.40    1.00     9.90     0.12   
255.36   528.07 4091.22  12.44 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   88.40    1.20    11.05     0.15   
256.00   528.13 4349.35  11.16 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   69.60    2.40     8.65     0.23   
252.71   527.46 9334.37  13.89 100.00


> Which indicates the problem is almost certainly related to the
> storage configuration or drivers, not the filesystem....
We have seen issues with xfs in the past but ofcourse this might be 
related to the drivers.
The storage configuration shouldn't be the problem here, note also 
changed it to 1MB stripe size.
I'll continue to look if I am able to find the issue in the hardware.

>
> Cheers,
>
> Dave.

Thanks

Kind regards

Matthew

[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 59632 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.4.2-custom (root@squeeze) (gcc version 4.4.5 (Debian 4.4.5-8) ) #2 SMP Wed Jun 13 15:11:14 CEST 2012
[    0.000000] Command line: vga=normal initrd=debian/squeeze/diskless/initrd.img-3.4.2-custom boot=live live-config noeject noswap noprompt nolocales edd=on nomodeset nosplash quickreboot scandelay=5 panic=60 debug=1 fetch=http://10.0.0.59/debian/squeeze/filesystem-3.4.2.squashfs BOOT_IMAGE=debian/squeeze/diskless/vmlinuz-3.4.2-custom auto
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 0000000000099800 (usable)
[    0.000000]  BIOS-e820: 0000000000099800 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000df780000 (usable)
[    0.000000]  BIOS-e820: 00000000df78e000 - 00000000df790000 type 9
[    0.000000]  BIOS-e820: 00000000df790000 - 00000000df79e000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000df79e000 - 00000000df7d0000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000df7d0000 - 00000000df7e0000 (reserved)
[    0.000000]  BIOS-e820: 00000000df7ec000 - 00000000f0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000620000000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI present.
[    0.000000] DMI: Supermicro X8ST3/X8ST3, BIOS 2.0        07/29/10  
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x620000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-CFFFF write-protect
[    0.000000]   D0000-DFFFF uncachable
[    0.000000]   E0000-E3FFF write-protect
[    0.000000]   E4000-EFFFF write-through
[    0.000000]   F0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 0000000000 mask FC00000000 write-back
[    0.000000]   1 base 0400000000 mask FE00000000 write-back
[    0.000000]   2 base 0600000000 mask FFE0000000 write-back
[    0.000000]   3 base 00E0000000 mask FFE0000000 uncachable
[    0.000000]   4 base 00DF800000 mask FFFF800000 uncachable
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] e820 update range: 00000000df800000 - 0000000100000000 (usable) ==> (reserved)
[    0.000000] last_pfn = 0xdf780 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [ffff8800000ff780] ff780
[    0.000000] initial memory mapped : 0 - 20000000
[    0.000000] Base memory trampoline at [ffff880000094000] 94000 size 20480
[    0.000000] init_memory_mapping: 0000000000000000-00000000df780000
[    0.000000]  0000000000 - 00df780000 page 4k
[    0.000000] kernel direct mapping tables up to df780000 @ 1f8ff000-20000000
[    0.000000] init_memory_mapping: 0000000100000000-0000000620000000
[    0.000000]  0100000000 - 0620000000 page 4k
[    0.000000] kernel direct mapping tables up to 620000000 @ dc666000-df780000
[    0.000000] RAMDISK: 797e3000 - 80000000
[    0.000000] ACPI: RSDP 00000000000faa70 00024 (v02 ACPIAM)
[    0.000000] ACPI: XSDT 00000000df790100 00074 (v01 SMCI            20100729 MSFT 00000097)
[    0.000000] ACPI: FACP 00000000df790290 000F4 (v03 072910 FACP2001 20100729 MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000df790630 06761 (v01  1F580 1F580000 00000000 INTL 20051117)
[    0.000000] ACPI: FACS 00000000df79e000 00040
[    0.000000] ACPI: APIC 00000000df790390 000D2 (v01 072910 APIC2001 20100729 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000df790470 0003C (v01 072910 OEMMCFG  20100729 MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000df79e040 0007B (v01 072910 OEMB2001 20100729 MSFT 00000097)
[    0.000000] ACPI: HPET 00000000df79a630 00038 (v01 072910 OEMHPET  20100729 MSFT 00000097)
[    0.000000] ACPI: SSDT 00000000df79ee90 00363 (v01 DpgPmm    CpuPm 00000012 INTL 20051117)
[    0.000000] ACPI: EINJ 00000000df79a670 00130 (v01  AMIER AMI_EINJ 20100729 MSFT 00000097)
[    0.000000] ACPI: BERT 00000000df79a800 00030 (v01  AMIER AMI_BERT 20100729 MSFT 00000097)
[    0.000000] ACPI: ERST 00000000df79a830 001B0 (v01  AMIER AMI_ERST 20100729 MSFT 00000097)
[    0.000000] ACPI: HEST 00000000df79a9e0 000A8 (v01  AMIER ABC_HEST 20100729 MSFT 00000097)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-0000000620000000
[    0.000000] Initmem setup node 0 0000000000000000-0000000620000000
[    0.000000]   NODE_DATA [000000061fffc000 - 000000061fffffff]
[    0.000000]  [ffffea0000000000-ffffea00187fffff] PMD -> [ffff880607600000-ffff88061f5fffff] on node 0
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00620000
[    0.000000] Movable zone start PFN for each node
[    0.000000] Early memory PFN ranges
[    0.000000]     0: 0x00000010 -> 0x00000099
[    0.000000]     0: 0x00000100 -> 0x000df780
[    0.000000]     0: 0x00100000 -> 0x00620000
[    0.000000] On node 0 totalpages: 6289161
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 5 pages reserved
[    0.000000]   DMA zone: 3908 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 16320 pages used for memmap
[    0.000000]   DMA32 zone: 894912 pages, LIFO batch:31
[    0.000000]   Normal zone: 83968 pages used for memmap
[    0.000000]   Normal zone: 5289984 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x86] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x87] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x88] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x89] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x8a] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x8b] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x8c] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x8d] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x8e] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x8f] disabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a301 base: 0xfed00000
[    0.000000] SMP: Allowing 16 CPUs, 12 hotplug CPUs
[    0.000000] nr_irqs_gsi: 40
[    0.000000] PM: Registered nosave memory: 0000000000099000 - 000000000009a000
[    0.000000] PM: Registered nosave memory: 000000000009a000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
[    0.000000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
[    0.000000] PM: Registered nosave memory: 00000000df780000 - 00000000df78e000
[    0.000000] PM: Registered nosave memory: 00000000df78e000 - 00000000df790000
[    0.000000] PM: Registered nosave memory: 00000000df790000 - 00000000df79e000
[    0.000000] PM: Registered nosave memory: 00000000df79e000 - 00000000df7d0000
[    0.000000] PM: Registered nosave memory: 00000000df7d0000 - 00000000df7e0000
[    0.000000] PM: Registered nosave memory: 00000000df7e0000 - 00000000df7ec000
[    0.000000] PM: Registered nosave memory: 00000000df7ec000 - 00000000f0000000
[    0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fee00000
[    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
[    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000ffc00000
[    0.000000] PM: Registered nosave memory: 00000000ffc00000 - 0000000100000000
[    0.000000] Allocating PCI resources starting at f0000000 (gap: f0000000:ee00000)
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:16 nr_node_ids:1
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88061fc00000 s81728 r8192 d20672 u131072
[    0.000000] pcpu-alloc: s81728 r8192 d20672 u131072 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 6188804
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: vga=normal initrd=debian/squeeze/diskless/initrd.img-3.4.2-custom boot=live live-config noeject noswap noprompt nolocales edd=on nomodeset nosplash quickreboot scandelay=5 panic=60 debug=1 fetch=http://10.0.0.59/debian/squeeze/filesystem-3.4.2.squashfs BOOT_IMAGE=debian/squeeze/diskless/vmlinuz-3.4.2-custom auto
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 24531272k/25690112k available (3621k kernel code, 533468k absent, 625372k reserved, 3042k data, 588k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=16, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU debugfs-based tracing is enabled.
[    0.000000] NR_IRQS:33024 nr_irqs:808 16
[    0.000000] Extended CMOS year: 2000
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty0] enabled
[    0.000000] hpet clockevent registered
[    0.000000] Fast TSC calibration using PIT
[    0.004000] Detected 2133.265 MHz processor.
[    0.000004] Calibrating delay loop (skipped), value calculated using timer frequency.. 4266.53 BogoMIPS (lpj=8533060)
[    0.000136] pid_max: default: 32768 minimum: 301
[    0.000230] Security Framework initialized
[    0.000300] SELinux:  Disabled at boot.
[    0.006470] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
[    0.016036] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.019229] Mount-cache hash table entries: 256
[    0.019537] Initializing cgroup subsys cpuacct
[    0.019602] Initializing cgroup subsys devices
[    0.019665] Initializing cgroup subsys freezer
[    0.019729] Initializing cgroup subsys net_cls
[    0.019815] CPU: Physical Processor ID: 0
[    0.019877] CPU: Processor Core ID: 0
[    0.019942] mce: CPU supports 9 MCE banks
[    0.020014] CPU0: Thermal monitoring enabled (TM1)
[    0.020083] using mwait in idle threads.
[    0.021034] ACPI: Core revision 20120320
[    0.046538] Switched APIC routing to physical flat.
[    0.047001] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.086750] CPU0: Intel(R) Xeon(R) CPU           E5506  @ 2.13GHz stepping 05
[    0.192402] Performance Events: PEBS fmt1+, 16-deep LBR, Nehalem events, Intel PMU driver.
[    0.192639] CPU erratum AAJ80 worked around
[    0.192701] CPUID marked event: 'bus cycles' unavailable
[    0.192769] ... version:                3
[    0.192831] ... bit width:              48
[    0.192892] ... generic registers:      4
[    0.192955] ... value mask:             0000ffffffffffff
[    0.193020] ... max period:             000000007fffffff
[    0.193085] ... fixed-purpose events:   3
[    0.193147] ... event mask:             000000070000000f
[    0.193545] NMI watchdog: enabled, takes one hw-pmu counter.
[    0.193700] Booting Node   0, Processors  #1
[    0.206837] NMI watchdog: enabled, takes one hw-pmu counter.
[    0.207050]  #2
[    0.220240] NMI watchdog: enabled, takes one hw-pmu counter.
[    0.220460]  #3
[    0.233654] NMI watchdog: enabled, takes one hw-pmu counter.
[    0.233808] Brought up 4 CPUs
[    0.233869] Total of 4 processors activated (17066.12 BogoMIPS).
[    0.236419] devtmpfs: initialized
[    0.243047] PM: Registering ACPI NVS region [mem 0xdf79e000-0xdf7cffff] (204800 bytes)
[    0.243236] dummy: 
[    0.243351] NET: Registered protocol family 16
[    0.243700] ACPI: bus type pci registered
[    0.243827] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.243920] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[    0.267994] PCI: Using configuration type 1 for base access
[    0.269061] bio: create slab <bio-0> at 0
[    0.269196] ACPI: Added _OSI(Module Device)
[    0.269260] ACPI: Added _OSI(Processor Device)
[    0.269323] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.269388] ACPI: Added _OSI(Processor Aggregator Device)
[    0.270346] ACPI: EC: Look up EC in DSDT
[    0.270828] \_SB_:_OSC evaluation returned wrong type
[    0.270893] _OSC request data:1 7 
[    0.271838] ACPI: Executed 1 blocks of module-level executable AML code
[    0.276104] ACPI: SSDT 00000000df79e0c0 008F0 (v01 DpgPmm  P001Ist 00000011 INTL 20051117)
[    0.276546] ACPI: Dynamic OEM Table Load:
[    0.276682] ACPI: SSDT           (null) 008F0 (v01 DpgPmm  P001Ist 00000011 INTL 20051117)
[    0.276943] ACPI: SSDT 00000000df79e9b0 004D5 (v01  PmRef  P001Cst 00003001 INTL 20051117)
[    0.277319] ACPI: Dynamic OEM Table Load:
[    0.277455] ACPI: SSDT           (null) 004D5 (v01  PmRef  P001Cst 00003001 INTL 20051117)
[    0.278026] ACPI: Interpreter enabled
[    0.278089] ACPI: (supports S0 S1 S4 S5)
[    0.278314] ACPI: Using IOAPIC for interrupt routing
[    0.285806] ACPI: No dock devices found.
[    0.285871] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.286053] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.286277] pci_root PNP0A08:00: host bridge window [io  0x0000-0x0cf7]
[    0.286348] pci_root PNP0A08:00: host bridge window [io  0x0d00-0xffff]
[    0.286417] pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
[    0.286503] pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000dffff]
[    0.286589] pci_root PNP0A08:00: host bridge window [mem 0xf0000000-0xfed8ffff]
[    0.286702] PCI host bridge to bus 0000:00
[    0.286766] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
[    0.286835] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
[    0.286903] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
[    0.286974] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000dffff]
[    0.287044] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfed8ffff]
[    0.287126] pci 0000:00:00.0: [8086:3405] type 00 class 0x060000
[    0.287252] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    0.287345] pci 0000:00:01.0: [8086:3408] type 01 class 0x060400
[    0.287474] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[    0.287566] pci 0000:00:03.0: [8086:340a] type 01 class 0x060400
[    0.287693] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[    0.287786] pci 0000:00:05.0: [8086:340c] type 01 class 0x060400
[    0.287914] pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
[    0.288005] pci 0000:00:07.0: [8086:340e] type 01 class 0x060400
[    0.288133] pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
[    0.288225] pci 0000:00:09.0: [8086:3410] type 01 class 0x060400
[    0.288353] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
[    0.288452] pci 0000:00:14.0: [8086:342e] type 00 class 0x080000
[    0.288591] pci 0000:00:14.1: [8086:3422] type 00 class 0x080000
[    0.288733] pci 0000:00:14.2: [8086:3423] type 00 class 0x080000
[    0.288868] pci 0000:00:14.3: [8086:3438] type 00 class 0x080000
[    0.288999] pci 0000:00:16.0: [8086:3430] type 00 class 0x088000
[    0.289079] pci 0000:00:16.0: reg 10: [mem 0xfbef8000-0xfbefbfff 64bit]
[    0.289223] pci 0000:00:16.1: [8086:3431] type 00 class 0x088000
[    0.289302] pci 0000:00:16.1: reg 10: [mem 0xfbef4000-0xfbef7fff 64bit]
[    0.289446] pci 0000:00:16.2: [8086:3432] type 00 class 0x088000
[    0.289525] pci 0000:00:16.2: reg 10: [mem 0xfbef0000-0xfbef3fff 64bit]
[    0.289669] pci 0000:00:16.3: [8086:3433] type 00 class 0x088000
[    0.289748] pci 0000:00:16.3: reg 10: [mem 0xfbeec000-0xfbeeffff 64bit]
[    0.289893] pci 0000:00:16.4: [8086:3429] type 00 class 0x088000
[    0.289972] pci 0000:00:16.4: reg 10: [mem 0xfbee8000-0xfbeebfff 64bit]
[    0.290115] pci 0000:00:16.5: [8086:342a] type 00 class 0x088000
[    0.290194] pci 0000:00:16.5: reg 10: [mem 0xfbee4000-0xfbee7fff 64bit]
[    0.290341] pci 0000:00:16.6: [8086:342b] type 00 class 0x088000
[    0.290420] pci 0000:00:16.6: reg 10: [mem 0xfbee0000-0xfbee3fff 64bit]
[    0.290565] pci 0000:00:16.7: [8086:342c] type 00 class 0x088000
[    0.290644] pci 0000:00:16.7: reg 10: [mem 0xfbedc000-0xfbedffff 64bit]
[    0.290790] pci 0000:00:1a.0: [8086:3a37] type 00 class 0x0c0300
[    0.290900] pci 0000:00:1a.0: reg 20: [io  0xac00-0xac1f]
[    0.291019] pci 0000:00:1a.1: [8086:3a38] type 00 class 0x0c0300
[    0.291129] pci 0000:00:1a.1: reg 20: [io  0xa880-0xa89f]
[    0.291248] pci 0000:00:1a.2: [8086:3a39] type 00 class 0x0c0300
[    0.291358] pci 0000:00:1a.2: reg 20: [io  0xa800-0xa81f]
[    0.291486] pci 0000:00:1a.7: [8086:3a3c] type 00 class 0x0c0320
[    0.291574] pci 0000:00:1a.7: reg 10: [mem 0xfbeda000-0xfbeda3ff]
[    0.291735] pci 0000:00:1a.7: PME# supported from D0 D3hot D3cold
[    0.291826] pci 0000:00:1c.0: [8086:3a40] type 01 class 0x060400
[    0.291965] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    0.292056] pci 0000:00:1c.1: [8086:3a42] type 01 class 0x060400
[    0.292196] pci 0000:00:1c.1: PME# supported from D0 D3hot D3cold
[    0.292299] pci 0000:00:1d.0: [8086:3a34] type 00 class 0x0c0300
[    0.292412] pci 0000:00:1d.0: reg 20: [io  0xa480-0xa49f]
[    0.292531] pci 0000:00:1d.1: [8086:3a35] type 00 class 0x0c0300
[    0.292641] pci 0000:00:1d.1: reg 20: [io  0xa400-0xa41f]
[    0.292759] pci 0000:00:1d.2: [8086:3a36] type 00 class 0x0c0300
[    0.292872] pci 0000:00:1d.2: reg 20: [io  0xa080-0xa09f]
[    0.293000] pci 0000:00:1d.7: [8086:3a3a] type 00 class 0x0c0320
[    0.293088] pci 0000:00:1d.7: reg 10: [mem 0xfbed8000-0xfbed83ff]
[    0.293249] pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
[    0.293338] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
[    0.293470] pci 0000:00:1f.0: [8086:3a16] type 00 class 0x060100
[    0.293620] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 1 PIO at 4700 (mask 0097)
[    0.293708] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 2 PIO at 03e8 (mask 000f)
[    0.293796] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 3 PIO at 1640 (mask 000f)
[    0.293885] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 4 PIO at 0ca0 (mask 000f)
[    0.294020] pci 0000:00:1f.2: [8086:3a22] type 00 class 0x010601
[    0.294106] pci 0000:00:1f.2: reg 10: [io  0x9480-0x9487]
[    0.294178] pci 0000:00:1f.2: reg 14: [io  0xa000-0xa003]
[    0.294251] pci 0000:00:1f.2: reg 18: [io  0x9c00-0x9c07]
[    0.294323] pci 0000:00:1f.2: reg 1c: [io  0x9880-0x9883]
[    0.294395] pci 0000:00:1f.2: reg 20: [io  0x9800-0x981f]
[    0.294468] pci 0000:00:1f.2: reg 24: [mem 0xfbed6000-0xfbed67ff]
[    0.294583] pci 0000:00:1f.2: PME# supported from D3hot
[    0.294666] pci 0000:00:1f.3: [8086:3a30] type 00 class 0x0c0500
[    0.294748] pci 0000:00:1f.3: reg 10: [mem 0xfbed4000-0xfbed40ff 64bit]
[    0.294838] pci 0000:00:1f.3: reg 20: [io  0x0400-0x041f]
[    0.300448] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    0.300558] pci 0000:00:03.0: PCI bridge to [bus 02-02]
[    0.300668] pci 0000:00:05.0: PCI bridge to [bus 03-03]
[    0.300789] pci 0000:04:00.0: [1000:0079] type 00 class 0x010400
[    0.300865] pci 0000:04:00.0: reg 10: [io  0xb000-0xb0ff]
[    0.300940] pci 0000:04:00.0: reg 14: [mem 0xfb87c000-0xfb87ffff 64bit]
[    0.301019] pci 0000:04:00.0: reg 1c: [mem 0xfb8c0000-0xfb8fffff 64bit]
[    0.301101] pci 0000:04:00.0: reg 30: [mem 0xfb880000-0xfb8bffff pref]
[    0.301210] pci 0000:04:00.0: supports D1 D2
[    0.308409] pci 0000:00:07.0: PCI bridge to [bus 04-04]
[    0.308480] pci 0000:00:07.0:   bridge window [io  0xb000-0xbfff]
[    0.308552] pci 0000:00:07.0:   bridge window [mem 0xfb800000-0xfb8fffff]
[    0.308848] pci 0000:05:00.0: [1000:0058] type 00 class 0x010000
[    0.308925] pci 0000:05:00.0: reg 10: [io  0xc000-0xc0ff]
[    0.309002] pci 0000:05:00.0: reg 14: [mem 0xfb9ec000-0xfb9effff 64bit]
[    0.309081] pci 0000:05:00.0: reg 1c: [mem 0xfb9f0000-0xfb9fffff 64bit]
[    0.309164] pci 0000:05:00.0: reg 30: [mem 0xfba00000-0xfbbfffff pref]
[    0.309273] pci 0000:05:00.0: supports D1 D2
[    0.309349] pci 0000:05:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.309447] pci 0000:00:09.0: PCI bridge to [bus 05-05]
[    0.309515] pci 0000:00:09.0:   bridge window [io  0xc000-0xcfff]
[    0.309584] pci 0000:00:09.0:   bridge window [mem 0xfb900000-0xfbbfffff]
[    0.309725] pci 0000:06:00.0: [8086:10d3] type 00 class 0x020000
[    0.309816] pci 0000:06:00.0: reg 10: [mem 0xfbce0000-0xfbcfffff]
[    0.309916] pci 0000:06:00.0: reg 18: [io  0xdc00-0xdc1f]
[    0.309997] pci 0000:06:00.0: reg 1c: [mem 0xfbcdc000-0xfbcdffff]
[    0.310206] pci 0000:06:00.0: PME# supported from D0 D3hot D3cold
[    0.316417] pci 0000:00:1c.0: PCI bridge to [bus 06-06]
[    0.316489] pci 0000:00:1c.0:   bridge window [io  0xd000-0xdfff]
[    0.316559] pci 0000:00:1c.0:   bridge window [mem 0xfbc00000-0xfbcfffff]
[    0.316700] pci 0000:07:00.0: [8086:10d3] type 00 class 0x020000
[    0.316791] pci 0000:07:00.0: reg 10: [mem 0xfbde0000-0xfbdfffff]
[    0.316890] pci 0000:07:00.0: reg 18: [io  0xec00-0xec1f]
[    0.316972] pci 0000:07:00.0: reg 1c: [mem 0xfbddc000-0xfbddffff]
[    0.317181] pci 0000:07:00.0: PME# supported from D0 D3hot D3cold
[    0.324420] pci 0000:00:1c.1: PCI bridge to [bus 07-07]
[    0.324492] pci 0000:00:1c.1:   bridge window [io  0xe000-0xefff]
[    0.324562] pci 0000:00:1c.1:   bridge window [mem 0xfbd00000-0xfbdfffff]
[    0.324672] pci 0000:08:04.0: [102b:0532] type 00 class 0x030000
[    0.324756] pci 0000:08:04.0: reg 10: [mem 0xf9000000-0xf9ffffff pref]
[    0.324833] pci 0000:08:04.0: reg 14: [mem 0xfaffc000-0xfaffffff]
[    0.324909] pci 0000:08:04.0: reg 18: [mem 0xfb000000-0xfb7fffff]
[    0.325075] pci 0000:00:1e.0: PCI bridge to [bus 08-08] (subtractive decode)
[    0.325149] pci 0000:00:1e.0:   bridge window [mem 0xfaf00000-0xfb7fffff]
[    0.325222] pci 0000:00:1e.0:   bridge window [mem 0xf9000000-0xf9ffffff 64bit pref]
[    0.325310] pci 0000:00:1e.0:   bridge window [io  0x0000-0x0cf7] (subtractive decode)
[    0.325398] pci 0000:00:1e.0:   bridge window [io  0x0d00-0xffff] (subtractive decode)
[    0.325486] pci 0000:00:1e.0:   bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
[    0.325577] pci 0000:00:1e.0:   bridge window [mem 0x000d0000-0x000dffff] (subtractive decode)
[    0.325667] pci 0000:00:1e.0:   bridge window [mem 0xf0000000-0xfed8ffff] (subtractive decode)
[    0.325788] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    0.326171] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
[    0.326348] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P4._PRT]
[    0.326450] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P5._PRT]
[    0.326560] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE1._PRT]
[    0.326663] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE3._PRT]
[    0.326766] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE5._PRT]
[    0.326870] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE7._PRT]
[    0.326973] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE9._PRT]
[    0.327075]  pci0000:00: Requesting ACPI _OSC control (0x1d)
[    0.327144]  pci0000:00: ACPI _OSC request failed (AE_NOT_FOUND), returned control mask: 0x1d
[    0.327232] ACPI _OSC control for PCIe not granted, disabling ASPM
[    0.345539] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 6 7 10 *11 12 14 15)
[    0.346053] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 6 7 *10 11 12 14 15)
[    0.346565] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 6 7 10 11 12 14 *15)
[    0.347076] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 6 7 10 11 12 *14 15)
[    0.347587] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
[    0.348191] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 6 *7 10 11 12 14 15)
[    0.348708] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
[    0.349316] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 *6 7 10 11 12 14 15)
[    0.349874] vgaarb: device added: PCI:0000:08:04.0,decodes=io+mem,owns=io+mem,locks=none
[    0.349963] vgaarb: loaded
[    0.350022] vgaarb: bridge control possible 0000:08:04.0
[    0.350134] wmi: Mapper loaded
[    0.350194] PCI: Using ACPI for IRQ routing
[    0.356839] PCI: Discovered peer bus ff
[    0.356901] PCI: root bus ff: using default resources
[    0.356988] PCI host bridge to bus 0000:ff
[    0.357052] pci_bus 0000:ff: root bus resource [io  0x0000-0xffff]
[    0.357120] pci_bus 0000:ff: root bus resource [mem 0x00000000-0xffffffffff]
[    0.357195] pci 0000:ff:00.0: [8086:2c40] type 00 class 0x060000
[    0.357282] pci 0000:ff:00.1: [8086:2c01] type 00 class 0x060000
[    0.357370] pci 0000:ff:02.0: [8086:2c10] type 00 class 0x060000
[    0.357457] pci 0000:ff:02.1: [8086:2c11] type 00 class 0x060000
[    0.357543] pci 0000:ff:02.4: [8086:2c14] type 00 class 0x060000
[    0.357627] pci 0000:ff:02.5: [8086:2c15] type 00 class 0x060000
[    0.357713] pci 0000:ff:03.0: [8086:2c18] type 00 class 0x060000
[    0.357798] pci 0000:ff:03.1: [8086:2c19] type 00 class 0x060000
[    0.357882] pci 0000:ff:03.2: [8086:2c1a] type 00 class 0x060000
[    0.357967] pci 0000:ff:03.4: [8086:2c1c] type 00 class 0x060000
[    0.358052] pci 0000:ff:04.0: [8086:2c20] type 00 class 0x060000
[    0.358139] pci 0000:ff:04.1: [8086:2c21] type 00 class 0x060000
[    0.358223] pci 0000:ff:04.2: [8086:2c22] type 00 class 0x060000
[    0.358308] pci 0000:ff:04.3: [8086:2c23] type 00 class 0x060000
[    0.358394] pci 0000:ff:05.0: [8086:2c28] type 00 class 0x060000
[    0.358478] pci 0000:ff:05.1: [8086:2c29] type 00 class 0x060000
[    0.358563] pci 0000:ff:05.2: [8086:2c2a] type 00 class 0x060000
[    0.358647] pci 0000:ff:05.3: [8086:2c2b] type 00 class 0x060000
[    0.358733] pci 0000:ff:06.0: [8086:2c30] type 00 class 0x060000
[    0.358821] pci 0000:ff:06.1: [8086:2c31] type 00 class 0x060000
[    0.358906] pci 0000:ff:06.2: [8086:2c32] type 00 class 0x060000
[    0.358990] pci 0000:ff:06.3: [8086:2c33] type 00 class 0x060000
[    0.359546] PCI: pci_cache_line_size set to 64 bytes
[    0.359757] reserve RAM buffer: 0000000000099800 - 000000000009ffff 
[    0.359811] reserve RAM buffer: 00000000df780000 - 00000000dfffffff 
[    0.360088] HPET: 4 timers in total, 0 timers will be used for per-cpu timer
[    0.360214] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
[    0.360467] hpet0: 4 comparators, 64-bit 14.318180 MHz counter
[    0.369710] Switching to clocksource hpet
[    0.371688] pnp: PnP ACPI init
[    0.371760] ACPI: bus type pnp registered
[    0.371920] pnp 00:00: [bus 00-ff]
[    0.371982] pnp 00:00: [io  0x0cf8-0x0cff]
[    0.372045] pnp 00:00: [io  0x0000-0x0cf7 window]
[    0.372110] pnp 00:00: [io  0x0d00-0xffff window]
[    0.372175] pnp 00:00: [mem 0x000a0000-0x000bffff window]
[    0.372241] pnp 00:00: [mem 0x000d0000-0x000dffff window]
[    0.372308] pnp 00:00: [mem 0xe0000000-0xdfffffff window disabled]
[    0.372376] pnp 00:00: [mem 0xf0000000-0xfed8ffff window]
[    0.372493] pnp 00:00: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active)
[    0.372589] pnp 00:01: [mem 0xfbf00000-0xfbffffff]
[    0.372654] pnp 00:01: [mem 0xfc000000-0xfcffffff]
[    0.372719] pnp 00:01: [mem 0xfd000000-0xfdffffff]
[    0.372783] pnp 00:01: [mem 0xfe000000-0xfebfffff]
[    0.372848] pnp 00:01: [mem 0xfec8a000-0xfec8afff]
[    0.372912] pnp 00:01: [mem 0xfed10000-0xfed10fff]
[    0.373038] system 00:01: [mem 0xfbf00000-0xfbffffff] has been reserved
[    0.373109] system 00:01: [mem 0xfc000000-0xfcffffff] has been reserved
[    0.373179] system 00:01: [mem 0xfd000000-0xfdffffff] has been reserved
[    0.373249] system 00:01: [mem 0xfe000000-0xfebfffff] has been reserved
[    0.373319] system 00:01: [mem 0xfec8a000-0xfec8afff] has been reserved
[    0.373389] system 00:01: [mem 0xfed10000-0xfed10fff] has been reserved
[    0.373459] system 00:01: Plug and Play ACPI device, IDs PNP0c01 (active)
[    0.373560] pnp 00:02: [dma 4]
[    0.373620] pnp 00:02: [io  0x0000-0x000f]
[    0.373683] pnp 00:02: [io  0x0081-0x0083]
[    0.373745] pnp 00:02: [io  0x0087]
[    0.373815] pnp 00:02: [io  0x0089-0x008b]
[    0.373878] pnp 00:02: [io  0x008f]
[    0.373941] pnp 00:02: [io  0x00c0-0x00df]
[    0.374027] pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
[    0.374104] pnp 00:03: [io  0x0070-0x0071]
[    0.374175] pnp 00:03: [irq 8]
[    0.374254] pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.374329] pnp 00:04: [io  0x0061]
[    0.374413] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
[    0.374490] pnp 00:05: [io  0x00f0-0x00ff]
[    0.374557] pnp 00:05: [irq 13]
[    0.374638] pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active)
[    0.374982] pnp 00:06: [io  0x03f8-0x03ff]
[    0.375050] pnp 00:06: [irq 4]
[    0.375111] pnp 00:06: [dma 0 disabled]
[    0.375241] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.375575] pnp 00:07: [io  0x02f8-0x02ff]
[    0.375643] pnp 00:07: [irq 3]
[    0.375703] pnp 00:07: [dma 0 disabled]
[    0.375858] pnp 00:07: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.376401] pnp 00:08: [io  0x03e8-0x03ef]
[    0.376469] pnp 00:08: [irq 5]
[    0.376529] pnp 00:08: [dma 0 disabled]
[    0.376659] pnp 00:08: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.376785] pnp 00:09: [io  0x164e-0x164f]
[    0.376849] pnp 00:09: [io  0x0000-0xffffffffffffffff disabled]
[    0.376972] system 00:09: [io  0x164e-0x164f] has been reserved
[    0.377042] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.377241] pnp 00:0a: [io  0x0000-0xffffffffffffffff disabled]
[    0.377309] pnp 00:0a: [io  0x0a00-0x0a0f]
[    0.377372] pnp 00:0a: [io  0x0ca0-0x0caf]
[    0.382958] system 00:0a: [io  0x0a00-0x0a0f] has been reserved
[    0.383027] system 00:0a: [io  0x0ca0-0x0caf] has been reserved
[    0.383096] system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.383290] pnp 00:0b: [io  0x0010-0x001f]
[    0.383354] pnp 00:0b: [io  0x0022-0x003f]
[    0.383417] pnp 00:0b: [io  0x0044-0x005f]
[    0.383479] pnp 00:0b: [io  0x0062-0x0063]
[    0.383542] pnp 00:0b: [io  0x0065-0x006f]
[    0.383605] pnp 00:0b: [io  0x0072-0x007f]
[    0.383667] pnp 00:0b: [io  0x0080]
[    0.383732] pnp 00:0b: [io  0x0084-0x0086]
[    0.383795] pnp 00:0b: [io  0x0088]
[    0.383856] pnp 00:0b: [io  0x008c-0x008e]
[    0.383918] pnp 00:0b: [io  0x0090-0x009f]
[    0.383981] pnp 00:0b: [io  0x00a2-0x00bf]
[    0.384043] pnp 00:0b: [io  0x00e0-0x00ef]
[    0.384106] pnp 00:0b: [io  0x04d0-0x04d1]
[    0.384169] pnp 00:0b: [io  0x0800-0x087f]
[    0.384232] pnp 00:0b: [io  0x0000-0xffffffffffffffff disabled]
[    0.384299] pnp 00:0b: [io  0x0500-0x057f]
[    0.384362] pnp 00:0b: [mem 0xfed1c000-0xfed1ffff]
[    0.384426] pnp 00:0b: [mem 0xfed20000-0xfed3ffff]
[    0.384491] pnp 00:0b: [mem 0xfed40000-0xfed8ffff]
[    0.384637] system 00:0b: [io  0x04d0-0x04d1] has been reserved
[    0.384706] system 00:0b: [io  0x0800-0x087f] has been reserved
[    0.384774] system 00:0b: [io  0x0500-0x057f] has been reserved
[    0.384844] system 00:0b: [mem 0xfed1c000-0xfed1ffff] has been reserved
[    0.384914] system 00:0b: [mem 0xfed20000-0xfed3ffff] has been reserved
[    0.384984] system 00:0b: [mem 0xfed40000-0xfed8ffff] has been reserved
[    0.385055] system 00:0b: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.385184] pnp 00:0c: [mem 0xfed00000-0xfed003ff]
[    0.385274] pnp 00:0c: Plug and Play ACPI device, IDs PNP0103 (active)
[    0.385426] pnp 00:0d: [io  0x0060]
[    0.385487] pnp 00:0d: [io  0x0064]
[    0.385549] pnp 00:0d: [mem 0xfec00000-0xfec00fff]
[    0.385614] pnp 00:0d: [mem 0xfee00000-0xfee00fff]
[    0.385740] system 00:0d: [mem 0xfec00000-0xfec00fff] could not be reserved
[    0.385819] system 00:0d: [mem 0xfee00000-0xfee00fff] has been reserved
[    0.385890] system 00:0d: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.386008] pnp 00:0e: [mem 0xe0000000-0xefffffff]
[    0.386129] system 00:0e: [mem 0xe0000000-0xefffffff] has been reserved
[    0.386201] system 00:0e: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.386456] pnp 00:0f: [mem 0x00000000-0x0009ffff]
[    0.386521] pnp 00:0f: [mem 0x000c0000-0x000cffff]
[    0.386586] pnp 00:0f: [mem 0x000e0000-0x000fffff]
[    0.386651] pnp 00:0f: [mem 0x00100000-0xdfffffff]
[    0.386715] pnp 00:0f: [mem 0xfed90000-0xffffffff]
[    0.386855] system 00:0f: [mem 0x00000000-0x0009ffff] could not be reserved
[    0.386926] system 00:0f: [mem 0x000c0000-0x000cffff] could not be reserved
[    0.386998] system 00:0f: [mem 0x000e0000-0x000fffff] could not be reserved
[    0.387069] system 00:0f: [mem 0x00100000-0xdfffffff] could not be reserved
[    0.387139] system 00:0f: [mem 0xfed90000-0xffffffff] could not be reserved
[    0.387211] system 00:0f: Plug and Play ACPI device, IDs PNP0c01 (active)
[    0.387417] pnp: PnP ACPI: found 16 devices
[    0.387480] ACPI: ACPI bus type pnp unregistered
[    0.393885] pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 06-06] add_size 200000
[    0.393987] pci 0000:00:1c.1: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 07-07] add_size 200000
[    0.394097] pci 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[    0.394190] pci 0000:00:1c.1: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[    0.394286] pci 0000:00:1c.0: BAR 15: assigned [mem 0xf0000000-0xf01fffff 64bit pref]
[    0.394375] pci 0000:00:1c.1: BAR 15: assigned [mem 0xf0200000-0xf03fffff 64bit pref]
[    0.394463] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    0.394536] pci 0000:00:03.0: PCI bridge to [bus 02-02]
[    0.394609] pci 0000:00:05.0: PCI bridge to [bus 03-03]
[    0.394681] pci 0000:00:07.0: PCI bridge to [bus 04-04]
[    0.394748] pci 0000:00:07.0:   bridge window [io  0xb000-0xbfff]
[    0.394819] pci 0000:00:07.0:   bridge window [mem 0xfb800000-0xfb8fffff]
[    0.394893] pci 0000:00:09.0: PCI bridge to [bus 05-05]
[    0.394959] pci 0000:00:09.0:   bridge window [io  0xc000-0xcfff]
[    0.395030] pci 0000:00:09.0:   bridge window [mem 0xfb900000-0xfbbfffff]
[    0.395105] pci 0000:00:1c.0: PCI bridge to [bus 06-06]
[    0.395171] pci 0000:00:1c.0:   bridge window [io  0xd000-0xdfff]
[    0.395242] pci 0000:00:1c.0:   bridge window [mem 0xfbc00000-0xfbcfffff]
[    0.395314] pci 0000:00:1c.0:   bridge window [mem 0xf0000000-0xf01fffff 64bit pref]
[    0.395405] pci 0000:00:1c.1: PCI bridge to [bus 07-07]
[    0.395472] pci 0000:00:1c.1:   bridge window [io  0xe000-0xefff]
[    0.395542] pci 0000:00:1c.1:   bridge window [mem 0xfbd00000-0xfbdfffff]
[    0.395614] pci 0000:00:1c.1:   bridge window [mem 0xf0200000-0xf03fffff 64bit pref]
[    0.395705] pci 0000:00:1e.0: PCI bridge to [bus 08-08]
[    0.395774] pci 0000:00:1e.0:   bridge window [mem 0xfaf00000-0xfb7fffff]
[    0.395845] pci 0000:00:1e.0:   bridge window [mem 0xf9000000-0xf9ffffff 64bit pref]
[    0.395979] pci 0000:00:1e.0: setting latency timer to 64
[    0.396047] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
[    0.396114] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff]
[    0.396181] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
[    0.396250] pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000dffff]
[    0.396318] pci_bus 0000:00: resource 8 [mem 0xf0000000-0xfed8ffff]
[    0.396386] pci_bus 0000:04: resource 0 [io  0xb000-0xbfff]
[    0.396452] pci_bus 0000:04: resource 1 [mem 0xfb800000-0xfb8fffff]
[    0.396521] pci_bus 0000:05: resource 0 [io  0xc000-0xcfff]
[    0.396587] pci_bus 0000:05: resource 1 [mem 0xfb900000-0xfbbfffff]
[    0.396656] pci_bus 0000:06: resource 0 [io  0xd000-0xdfff]
[    0.396723] pci_bus 0000:06: resource 1 [mem 0xfbc00000-0xfbcfffff]
[    0.396791] pci_bus 0000:06: resource 2 [mem 0xf0000000-0xf01fffff 64bit pref]
[    0.396877] pci_bus 0000:07: resource 0 [io  0xe000-0xefff]
[    0.396943] pci_bus 0000:07: resource 1 [mem 0xfbd00000-0xfbdfffff]
[    0.397012] pci_bus 0000:07: resource 2 [mem 0xf0200000-0xf03fffff 64bit pref]
[    0.397098] pci_bus 0000:08: resource 1 [mem 0xfaf00000-0xfb7fffff]
[    0.397166] pci_bus 0000:08: resource 2 [mem 0xf9000000-0xf9ffffff 64bit pref]
[    0.397252] pci_bus 0000:08: resource 4 [io  0x0000-0x0cf7]
[    0.397319] pci_bus 0000:08: resource 5 [io  0x0d00-0xffff]
[    0.397386] pci_bus 0000:08: resource 6 [mem 0x000a0000-0x000bffff]
[    0.397454] pci_bus 0000:08: resource 7 [mem 0x000d0000-0x000dffff]
[    0.397521] pci_bus 0000:08: resource 8 [mem 0xf0000000-0xfed8ffff]
[    0.397590] pci_bus 0000:ff: resource 4 [io  0x0000-0xffff]
[    0.397657] pci_bus 0000:ff: resource 5 [mem 0x00000000-0xffffffffff]
[    0.397768] NET: Registered protocol family 2
[    0.398685] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.401045] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    0.402786] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.403019] TCP: Hash tables configured (established 524288 bind 65536)
[    0.403088] TCP: reno registered
[    0.403250] UDP hash table entries: 16384 (order: 7, 524288 bytes)
[    0.403523] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes)
[    0.403793] NET: Registered protocol family 1
[    0.404142] PCI: CLS mismatch (256 != 64), using 64 bytes
[    0.404216] pci 0000:08:04.0: Boot video device
[    0.404350] Unpacking initramfs...
[    2.883437] debug: unmapping init memory ffff8800797e3000..ffff880080000000
[    2.885419] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    2.885490] Placing 64MB software IO TLB between ffff8800d8666000 - ffff8800dc666000
[    2.885577] software IO TLB at phys 0xd8666000 - 0xdc666000
[    2.886217] audit: initializing netlink socket (disabled)
[    2.886294] type=2000 audit(1339704893.760:1): initialized
[    2.902216] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    2.903969] VFS: Disk quotas dquot_6.5.2
[    2.904081] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.904253] msgmni has been set to 32768
[    2.904491] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    2.904578] io scheduler noop registered
[    2.904640] io scheduler deadline registered
[    2.904759] io scheduler cfq registered (default)
[    2.905171] pcieport 0000:00:1c.0: irq 40 for MSI/MSI-X
[    2.905331] pcieport 0000:00:1c.1: irq 41 for MSI/MSI-X
[    2.905581] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.926053] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.946529] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    2.967002] serial8250: ttyS2 at I/O 0x3e8 (irq = 4) is a 16550A
[    2.987608] 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    3.008113] 00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    3.028615] 00:08: ttyS2 at I/O 0x3e8 (irq = 5) is a 16550A
[    3.028981] Non-volatile memory driver v1.3
[    3.029046] Linux agpgart interface v0.103
[    3.029252] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    3.031768] serio: i8042 KBD port at 0x60,0x64 irq 1
[    3.031837] serio: i8042 AUX port at 0x60,0x64 irq 12
[    3.032001] mousedev: PS/2 mouse device common for all mice
[    3.032108] rtc_cmos 00:03: RTC can wake from S4
[    3.032269] rtc_cmos 00:03: rtc core: registered rtc_cmos as rtc0
[    3.032362] rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[    3.032440] cpuidle: using governor ladder
[    3.032503] cpuidle: using governor menu
[    3.032685] fujitsu_laptop: driver 0.6.0 successfully loaded
[    3.032840] TCP: cubic registered
[    3.033012] NET: Registered protocol family 10
[    3.033317] Mobile IPv6
[    3.033377] NET: Registered protocol family 17
[    3.033443] Registering the dns_resolver key type
[    3.033678] PM: Hibernation image not present or could not be loaded.
[    3.033753] registered taskstats version 1
[    3.034424] rtc_cmos 00:03: setting system clock to 2012-06-14 20:14:54 UTC (1339704894)
[    3.034555] Initializing network drop monitor service
[    3.173806] debug: unmapping init memory ffffffff81684000..ffffffff81717000
[    3.179434] Write protecting the kernel read-only data: 6144k
[    3.179638] debug: unmapping init memory ffff88000138e000..ffff880001400000
[    3.179757] debug: unmapping init memory ffff880001548000..ffff880001600000
[    3.199066] udev[58]: starting version 164
[    3.220786] SCSI subsystem initialized
[    3.227955] Fusion MPT base driver 3.04.20
[    3.228022] Copyright (c) 1999-2008 LSI Corporation
[    3.229635] Fusion MPT SAS Host driver 3.04.20
[    3.255387] e1000e: Intel(R) PRO/1000 Network Driver - 1.9.5-k
[    3.255460] e1000e: Copyright(c) 1999 - 2012 Intel Corporation.
[    3.255553] e1000e 0000:06:00.0: Disabling ASPM L0s 
[    3.255767] e1000e 0000:06:00.0: (unregistered net_device): Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    3.255954] e1000e 0000:06:00.0: irq 42 for MSI/MSI-X
[    3.256007] usbcore: registered new interface driver usbfs
[    3.256034] usbcore: registered new interface driver hub
[    3.256156] e1000e 0000:06:00.0: irq 43 for MSI/MSI-X
[    3.256228] e1000e 0000:06:00.0: irq 44 for MSI/MSI-X
[    3.257978] usbcore: registered new device driver usb
[    3.258821] megasas: 00.00.06.15-rc1 Mon. Mar. 19 17:00:00 PDT 2012
[    3.258914] megasas: 0x1000:0x0079:0x1028:0x1f15: bus 4:slot 0:func 0
[    3.259044] mptbase: ioc0: Initiating bringup
[    3.259119] megasas: FW now in Ready state
[    3.259213] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.259299] megaraid_sas 0000:04:00.0: irq 45 for MSI/MSI-X
[    3.259321] ehci_hcd 0000:00:1a.7: setting latency timer to 64
[    3.259325] ehci_hcd 0000:00:1a.7: EHCI Host Controller
[    3.259350] ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1
[    3.259384] ehci_hcd 0000:00:1a.7: debug port 1
[    3.260733] libata version 3.00 loaded.
[    3.263280] ehci_hcd 0000:00:1a.7: cache line size of 64 is not supported
[    3.263370] ehci_hcd 0000:00:1a.7: irq 18, io mem 0xfbeda000
[    3.273890] ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00
[    3.273982] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    3.274053] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.274139] usb usb1: Product: EHCI Host Controller
[    3.274204] usb usb1: Manufacturer: Linux 3.4.2-custom ehci_hcd
[    3.274273] usb usb1: SerialNumber: 0000:00:1a.7
[    3.274455] hub 1-0:1.0: USB hub found
[    3.274521] hub 1-0:1.0: 6 ports detected
[    3.274735] ehci_hcd 0000:00:1d.7: setting latency timer to 64
[    3.274804] ehci_hcd 0000:00:1d.7: EHCI Host Controller
[    3.274878] ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2
[    3.274991] ehci_hcd 0000:00:1d.7: debug port 1
[    3.278934] ehci_hcd 0000:00:1d.7: cache line size of 64 is not supported
[    3.279020] ehci_hcd 0000:00:1d.7: irq 23, io mem 0xfbed8000
[    3.285809] megasas_init_mfi: fw_support_ieee=67108864
[    3.285869] megasas: INIT adapter done
[    3.289816] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
[    3.289903] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[    3.289973] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.290059] usb usb2: Product: EHCI Host Controller
[    3.290124] usb usb2: Manufacturer: Linux 3.4.2-custom ehci_hcd
[    3.290191] usb usb2: SerialNumber: 0000:00:1d.7
[    3.290349] hub 2-0:1.0: USB hub found
[    3.290414] hub 2-0:1.0: 6 ports detected
[    3.290564] ahci 0000:00:1f.2: version 3.0
[    3.290689] ahci 0000:00:1f.2: irq 46 for MSI/MSI-X
[    3.290794] ahci: SSS flag set, parallel bus scan disabled
[    3.290903] ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
[    3.290993] ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ccc ems sxs 
[    3.291086] ahci 0000:00:1f.2: setting latency timer to 64
[    3.291399] uhci_hcd: USB Universal Host Controller Interface driver
[    3.330299] scsi1 : ahci
[    3.330438] scsi2 : ahci
[    3.330560] scsi3 : ahci
[    3.330688] scsi4 : ahci
[    3.330811] scsi5 : ahci
[    3.330933] scsi6 : ahci
[    3.331148] ata1: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6100 irq 46
[    3.331237] ata2: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6180 irq 46
[    3.331324] ata3: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6200 irq 46
[    3.331411] ata4: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6280 irq 46
[    3.331499] ata5: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6300 irq 46
[    3.331586] ata6: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6380 irq 46
[    3.331709] uhci_hcd 0000:00:1a.0: setting latency timer to 64
[    3.331779] uhci_hcd 0000:00:1a.0: UHCI Host Controller
[    3.331854] uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
[    3.331978] uhci_hcd 0000:00:1a.0: irq 16, io base 0x0000ac00
[    3.332083] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[    3.332153] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.332239] usb usb3: Product: UHCI Host Controller
[    3.332304] usb usb3: Manufacturer: Linux 3.4.2-custom uhci_hcd
[    3.332371] usb usb3: SerialNumber: 0000:00:1a.0
[    3.332537] hub 3-0:1.0: USB hub found
[    3.332602] hub 3-0:1.0: 2 ports detected
[    3.332764] uhci_hcd 0000:00:1a.1: setting latency timer to 64
[    3.332833] uhci_hcd 0000:00:1a.1: UHCI Host Controller
[    3.332904] uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
[    3.333021] uhci_hcd 0000:00:1a.1: irq 21, io base 0x0000a880
[    3.333117] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[    3.333188] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.333274] usb usb4: Product: UHCI Host Controller
[    3.333339] usb usb4: Manufacturer: Linux 3.4.2-custom uhci_hcd
[    3.333406] usb usb4: SerialNumber: 0000:00:1a.1
[    3.333560] hub 4-0:1.0: USB hub found
[    3.333625] hub 4-0:1.0: 2 ports detected
[    3.333768] uhci_hcd 0000:00:1a.2: setting latency timer to 64
[    3.333847] uhci_hcd 0000:00:1a.2: UHCI Host Controller
[    3.333918] uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5
[    3.334036] uhci_hcd 0000:00:1a.2: irq 19, io base 0x0000a800
[    3.334133] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
[    3.334203] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.334289] usb usb5: Product: UHCI Host Controller
[    3.334353] usb usb5: Manufacturer: Linux 3.4.2-custom uhci_hcd
[    3.334420] usb usb5: SerialNumber: 0000:00:1a.2
[    3.334573] hub 5-0:1.0: USB hub found
[    3.334638] hub 5-0:1.0: 2 ports detected
[    3.334781] uhci_hcd 0000:00:1d.0: setting latency timer to 64
[    3.334850] uhci_hcd 0000:00:1d.0: UHCI Host Controller
[    3.334925] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6
[    3.335034] uhci_hcd 0000:00:1d.0: irq 23, io base 0x0000a480
[    3.335129] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
[    3.335200] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.335285] usb usb6: Product: UHCI Host Controller
[    3.335350] usb usb6: Manufacturer: Linux 3.4.2-custom uhci_hcd
[    3.335417] usb usb6: SerialNumber: 0000:00:1d.0
[    3.335568] hub 6-0:1.0: USB hub found
[    3.335633] hub 6-0:1.0: 2 ports detected
[    3.335775] uhci_hcd 0000:00:1d.1: setting latency timer to 64
[    3.335844] uhci_hcd 0000:00:1d.1: UHCI Host Controller
[    3.335915] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7
[    3.336023] uhci_hcd 0000:00:1d.1: irq 19, io base 0x0000a400
[    3.336121] usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
[    3.336191] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.336277] usb usb7: Product: UHCI Host Controller
[    3.336342] usb usb7: Manufacturer: Linux 3.4.2-custom uhci_hcd
[    3.336409] usb usb7: SerialNumber: 0000:00:1d.1
[    3.336560] hub 7-0:1.0: USB hub found
[    3.336625] hub 7-0:1.0: 2 ports detected
[    3.336766] uhci_hcd 0000:00:1d.2: setting latency timer to 64
[    3.336835] uhci_hcd 0000:00:1d.2: UHCI Host Controller
[    3.336906] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
[    3.337014] uhci_hcd 0000:00:1d.2: irq 18, io base 0x0000a080
[    3.337110] usb usb8: New USB device found, idVendor=1d6b, idProduct=0001
[    3.337180] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.337266] usb usb8: Product: UHCI Host Controller
[    3.337331] usb usb8: Manufacturer: Linux 3.4.2-custom uhci_hcd
[    3.342892] usb usb8: SerialNumber: 0000:00:1d.2
[    3.343044] hub 8-0:1.0: USB hub found
[    3.343108] hub 8-0:1.0: 2 ports detected
[    3.357808] scsi0 : LSI SAS based MegaRAID driver
[    3.365218] e1000e 0000:06:00.0: eth0: (PCI Express:2.5GT/s:Width x1) 00:25:90:22:b5:de
[    3.365315] e1000e 0000:06:00.0: eth0: Intel(R) PRO/1000 Network Connection
[    3.365481] e1000e 0000:06:00.0: eth0: MAC: 3, PHY: 8, PBA No: 0101FF-0FF
[    3.365569] e1000e 0000:07:00.0: Disabling ASPM L0s 
[    3.365811] e1000e 0000:07:00.0: (unregistered net_device): Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    3.366012] e1000e 0000:07:00.0: irq 47 for MSI/MSI-X
[    3.366088] e1000e 0000:07:00.0: irq 48 for MSI/MSI-X
[    3.366163] e1000e 0000:07:00.0: irq 49 for MSI/MSI-X
[    3.369434] scsi 0:1:44:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.372207] scsi 0:1:45:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.374954] scsi 0:1:46:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.377692] scsi 0:1:47:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.380459] scsi 0:1:48:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.384160] scsi 0:1:49:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.386904] scsi 0:1:50:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.389653] scsi 0:1:51:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.392433] scsi 0:1:52:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.395167] scsi 0:1:53:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.397894] scsi 0:1:54:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.400627] scsi 0:1:55:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.403362] scsi 0:1:56:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.406091] scsi 0:1:57:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.408805] scsi 0:1:58:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.411525] scsi 0:1:59:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.414256] scsi 0:1:60:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.416986] scsi 0:1:61:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.419715] scsi 0:1:62:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.422441] scsi 0:1:63:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.425162] scsi 0:1:64:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.427970] scsi 0:1:65:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.430696] scsi 0:1:66:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.433428] scsi 0:1:67:0: Direct-Access     ATA      Hitachi HDS5C302 A580 PQ: 0 ANSI: 5
[    3.436897] scsi 0:2:1:0: Direct-Access     DELL     PERC H800        2.10 PQ: 0 ANSI: 5
[    3.486731] e1000e 0000:07:00.0: eth1: (PCI Express:2.5GT/s:Width x1) 00:25:90:22:b5:df
[    3.486822] e1000e 0000:07:00.0: eth1: Intel(R) PRO/1000 Network Connection
[    3.486976] e1000e 0000:07:00.0: eth1: MAC: 3, PHY: 8, PBA No: 0101FF-0FF
[    3.649791] ata1: SATA link down (SStatus 0 SControl 300)
[    3.845780] ioc0: LSISAS1068E B3: Capabilities={Initiator}
[    3.881784] Refined TSC clocksource calibration: 2133.331 MHz.
[    3.881789] usb 3-1: new low-speed USB device number 2 using uhci_hcd
[    3.881932] Switching to clocksource tsc
[    3.969784] ata2: SATA link down (SStatus 0 SControl 300)
[    4.209083] usb 3-1: New USB device found, idVendor=046d, idProduct=c30e
[    4.209159] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    4.209230] usb 3-1: Product: HID compliant keyboard
[    4.209295] usb 3-1: Manufacturer: Logitech
[    4.289787] ata3: SATA link down (SStatus 0 SControl 300)
[    4.320296] input: Logitech HID compliant keyboard as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1:1.0/input/input0
[    4.320441] generic-usb 0003:046D:C30E.0001: input,hidraw0: USB HID v1.10 Keyboard [Logitech HID compliant keyboard] on usb-0000:00:1a.0-1/input0
[    4.449166] input: Logitech HID compliant keyboard as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1:1.1/input/input1
[    4.449300] generic-usb 0003:046D:C30E.0002: input,hidraw1: USB HID v1.10 Device [Logitech HID compliant keyboard] on usb-0000:00:1a.0-1/input1
[    4.449415] usbcore: registered new interface driver usbhid
[    4.449482] usbhid: USB HID core driver
[    4.457781] usb 5-2: new full-speed USB device number 2 using uhci_hcd
[    4.609789] ata4: SATA link down (SStatus 0 SControl 300)
[    4.633135] usb 5-2: New USB device found, idVendor=046b, idProduct=ff10
[    4.633211] usb 5-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    4.633282] usb 5-2: Product: Virtual Keyboard and Mouse
[    4.633347] usb 5-2: Manufacturer: American Megatrends Inc.
[    4.633414] usb 5-2: SerialNumber: serial
[    4.674343] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.2/usb5/5-2/5-2:1.0/input/input2
[    4.674481] generic-usb 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-2/input0
[    4.691306] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.2/usb5/5-2/5-2:1.1/input/input3
[    4.691460] generic-usb 0003:046B:FF10.0004: input,hidraw3: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.2-2/input1
[    4.929787] ata5: SATA link down (SStatus 0 SControl 300)
[    5.249784] ata6: SATA link down (SStatus 0 SControl 300)
[    5.254453] sd 0:2:1:0: [sda] 85930803200 512-byte logical blocks: (43.9 TB/40.0 TiB)
[    5.254599] sd 0:2:1:0: [sda] Write Protect is off
[    5.254672] sd 0:2:1:0: [sda] Mode Sense: 1f 00 00 08
[    5.254775] sd 0:2:1:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    5.267376]  sda: unknown partition table
[    5.267654] sd 0:2:1:0: [sda] Attached SCSI disk
[   17.075651] scsi7 : ioc0: LSISAS1068E B3, FwRev=011a0000h, Ports=1, MaxQ=478, IRQ=16
[   17.265656] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   17.347679] ADDRCONF(NETDEV_UP): eth1: link is not ready
[   20.014769] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   20.016371] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   20.363519] Uniform Multi-Platform E-IDE driver
[   20.364148] ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
[   20.364248] Probing IDE interface ide0...
[   20.925821] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
[   20.925940] Probing IDE interface ide1...
[   21.485817] ide1 at 0x170-0x177,0x376 on irq 15
[   30.089802] eth0: no IPv6 routers present
[   59.033035] aufs 3.4-20120521
[   59.044924] loop: module loaded
[   59.106801] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[   60.098888] udev[942]: starting version 164
[   60.140234] ACPI: Requesting acpi_cpufreq
[   60.143507] Monitor-Mwait will be used to enter C-1 state
[   60.143647] Monitor-Mwait will be used to enter C-2 state
[   60.144297] Monitor-Mwait will be used to enter C-3 state
[   60.144384] ACPI: acpi_idle registered with cpuidle
[   60.145441] dca service started, version 1.12.1
[   60.148621] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input4
[   60.148716] ACPI: Power Button [PWRB]
[   60.150451] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input5
[   60.150546] ACPI: Power Button [PWRF]
[   60.167133] ioatdma: Intel(R) QuickData Technology Driver 4.00
[   60.167364] ioatdma 0000:00:16.0: irq 50 for MSI/MSI-X
[   60.167797] ioatdma 0000:00:16.1: irq 51 for MSI/MSI-X
[   60.168227] ioatdma 0000:00:16.2: irq 52 for MSI/MSI-X
[   60.168609] ioatdma 0000:00:16.3: irq 53 for MSI/MSI-X
[   60.169000] ioatdma 0000:00:16.4: irq 54 for MSI/MSI-X
[   60.169478] ioatdma 0000:00:16.5: irq 55 for MSI/MSI-X
[   60.169895] ioatdma 0000:00:16.6: irq 56 for MSI/MSI-X
[   60.170396] ioatdma 0000:00:16.7: irq 57 for MSI/MSI-X
[   60.232097] microcode: CPU0 sig=0x106a5, pf=0x1, revision=0x11
[   60.286673] microcode: CPU1 sig=0x106a5, pf=0x1, revision=0x11
[   60.290266] microcode: CPU2 sig=0x106a5, pf=0x1, revision=0x11
[   60.293660] microcode: CPU3 sig=0x106a5, pf=0x1, revision=0x11
[   60.297192] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[   61.516563] 8021q: 802.1Q VLAN Support v1.8
[   61.516641] 8021q: adding VLAN 0 to HW filter on device eth0
[   61.516721] 8021q: adding VLAN 0 to HW filter on device eth1
[   61.536632] myri10ge: Version 1.5.3-1.534
[   62.196031] fuse init (API version 7.18)

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-14 14:31           ` Matthew Whittaker-Williams
@ 2012-06-15  0:16             ` Dave Chinner
  2012-06-15  9:52               ` Michael Monnerie
  2012-06-15 11:25               ` Bernd Schubert
  0 siblings, 2 replies; 20+ messages in thread
From: Dave Chinner @ 2012-06-15  0:16 UTC (permalink / raw)
  To: Matthew Whittaker-Williams; +Cc: xfs

On Thu, Jun 14, 2012 at 04:31:15PM +0200, Matthew Whittaker-Williams wrote:
> iostat:
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00     0.00   81.80    1.40    10.22     0.18 256.00   531.91 5349.11  12.02 100.00
> sda               0.00     0.00   83.40    1.20    10.37     0.15 254.56   525.35 4350.67  11.82 100.00
> sda               0.00     0.00   79.20    0.80     9.90     0.10 256.00   530.14 3153.38  12.50 100.00
> sda               0.00     0.00   72.80    2.20     9.09     0.13 251.72   546.08 8709.54  13.33 100.00
> sda               0.00     0.00   79.80    1.40     9.95     0.12 254.07   535.35 5172.22  12.32 100.00
> sda               0.00     0.00   99.60    1.20    12.41     0.08 253.86   529.49 3560.89   9.92 100.00
> sda               0.00     0.00   60.80    1.40     7.59     0.11 253.77   527.21 6545.50  16.08 100.00
> sda               0.00     0.00   79.00    1.80     9.84     0.08 251.51   547.93 6400.42  12.38 100.00
> sda               0.00     0.00   82.20    2.20    10.25     0.01 248.93   536.42 7415.77  11.85 100.00
> sda               0.00     0.00   89.40    2.20    11.17     0.01 249.90   525.68 7232.96  10.92 100.00
> sda               0.00     0.00   82.00    1.20    10.22     0.08 253.37   541.60 4170.95  12.02 100.00
> sda               0.00     0.00   62.80    2.60     7.85     0.14 250.31   541.15 11260.81  15.29 100.00
> sda               0.00     0.00   85.00    1.80    10.61     0.21 255.47   529.36 6514.85  11.52 100.00
> sda               0.00     0.00   75.20    1.40     9.38     0.11 253.72   535.68 5416.70  13.05 100.00
> sda               0.00     0.00   66.80    1.20     8.33     0.11 254.19   546.68 5459.11  14.71 100.00
> sda               0.00     0.00   81.40    0.80    10.15     0.10 255.38   540.62 3171.57  12.17 100.00
> sda               0.00     0.00   72.20    1.20     9.02     0.15 255.74   535.26 5345.51  13.62 100.00
> sda               0.00     0.00   91.00    1.00    11.35     0.12 255.44   531.02 3637.72  10.87 100.00
> sda               0.00     0.00   81.00    1.60    10.12     0.20 255.96   524.44 6513.22  12.11 100.00
> sda               0.00     0.00   72.80    2.40     9.04     0.26 253.24   543.25 9071.66  13.30 100.00
> sda               0.00     0.00   73.80    1.20     9.18     0.15 254.63   539.20 5087.91  13.33 100.00
> sda               0.00     0.00   79.20    1.40     9.90     0.18 256.00   532.38 5592.38  12.41 100.00
> sda               0.00     0.20   79.40    1.00     9.90     0.12 255.36   528.07 4091.22  12.44 100.00
> sda               0.00     0.00   88.40    1.20    11.05     0.15 256.00   528.13 4349.35  11.16 100.00
> sda               0.00     0.00   69.60    2.40     8.65     0.23 252.71   527.46 9334.37  13.89 100.00

So, the average service time for an IO is 10-16ms, which is a seek
per IO. You're doing primarily 128k read IOs, and maybe one or 2
writes a second. You have a very deep request queue: > 512 requests.
Have you tuned /sys/block/sda/queue/nr_requests up from the default
of 128? This is going to be one of the causes of your problems - you
have 511 oustanding write requests, and only one read at a time.
Reduce the ioscehduer queue depth, and potentially also the device
CTQ depth.

That tends to indicate that the write requests are causing RMW
cycles in the RAID when flushing the cache, otherwise they'd simply
hit the BBWC and return immediately. The other possibility is that
the BBWC is operating in write-through mode rather than write back,
but this is typical of a writeback cache filling up and then having
to flush and the flush being -extremely- slow due to RMW cycles....

Oh, I just noticed you are might be using CFQ (it's the default in
dmesg). Don't - CFQ is highly unsuited for hardware RAID - it's
hueristically tuned to work well on sngle SATA drives. Use deadline,
or preferably for hardware RAID, noop.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-15  0:16             ` Dave Chinner
@ 2012-06-15  9:52               ` Michael Monnerie
  2012-06-15 12:29                 ` Dave Chinner
  2012-06-15 11:25               ` Bernd Schubert
  1 sibling, 1 reply; 20+ messages in thread
From: Michael Monnerie @ 2012-06-15  9:52 UTC (permalink / raw)
  To: xfs


[-- Attachment #1.1: Type: text/plain, Size: 1709 bytes --]

Am Freitag, 15. Juni 2012, 10:16:02 schrieb Dave Chinner:
> So, the average service time for an IO is 10-16ms, which is a seek
> per IO. You're doing primarily 128k read IOs, and maybe one or 2
> writes a second. You have a very deep request queue: > 512 requests.
> Have you tuned /sys/block/sda/queue/nr_requests up from the default
> of 128? This is going to be one of the causes of your problems - you
> have 511 oustanding write requests, and only one read at a time.
> Reduce the ioscehduer queue depth, and potentially also the device
> CTQ depth.

Dave, I'm puzzled by this. I'd believe that a higher #req. would help 
the block layer to resort I/O in the elevator, and therefore help to 
gain throughput. Why would 128 be better than 512 here?

And maybe Matthew could profit from limiting the vm.dirty_bytes, I've 
seen when this value is too high the server stucks on lots of writes, 
for streaming it's better to have this smaller so the disk writes can 
keep up and delays are not too long.

> Oh, I just noticed you are might be using CFQ (it's the default in
> dmesg). Don't - CFQ is highly unsuited for hardware RAID - it's
> hueristically tuned to work well on sngle SATA drives. Use deadline,
> or preferably for hardware RAID, noop.

Wouldn't deadline be better with a higher rq_qu size? As I understand 
it, noop only groups adjacent I/Os together, while deadline does a bit 
more and should be able to get bigger adjacent I/O areas because it 
waits a bit longer before a flush.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-15  0:16             ` Dave Chinner
  2012-06-15  9:52               ` Michael Monnerie
@ 2012-06-15 11:25               ` Bernd Schubert
  2012-06-15 12:30                 ` Dave Chinner
  1 sibling, 1 reply; 20+ messages in thread
From: Bernd Schubert @ 2012-06-15 11:25 UTC (permalink / raw)
  To: linux-xfs; +Cc: Matthew Whittaker-Williams

On 06/15/2012 02:16 AM, Dave Chinner wrote:
> Oh, I just noticed you are might be using CFQ (it's the default in
> dmesg). Don't - CFQ is highly unsuited for hardware RAID - it's
> hueristically tuned to work well on sngle SATA drives. Use deadline,
> or preferably for hardware RAID, noop.

I'm not sure if noop is really a good recommendation even with hw raid, 
especially if the the request queue size is high. This week I did some 
benchmarks with a high rq write size (triggered with 
sync_file_range(..., SYNC_FILE_RANGE_WRITE) ) and with noop concuring 
reads then almost entirely got stalled.
With deadline read/write balance was much better, although writes still 
had been preferred (with sync_file_range() and without). I always 
thought deadline prefers reads and I hope I find some time later on to 
investigate further what was going on.
Test had been on a netapp E5400 hw raid, so rather high end hw raid.


Cheers,
Bernd



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-15  9:52               ` Michael Monnerie
@ 2012-06-15 12:29                 ` Dave Chinner
  0 siblings, 0 replies; 20+ messages in thread
From: Dave Chinner @ 2012-06-15 12:29 UTC (permalink / raw)
  To: Michael Monnerie; +Cc: xfs

On Fri, Jun 15, 2012 at 11:52:17AM +0200, Michael Monnerie wrote:
> Am Freitag, 15. Juni 2012, 10:16:02 schrieb Dave Chinner:
> > So, the average service time for an IO is 10-16ms, which is a seek
> > per IO. You're doing primarily 128k read IOs, and maybe one or 2
> > writes a second. You have a very deep request queue: > 512 requests.
> > Have you tuned /sys/block/sda/queue/nr_requests up from the default
> > of 128? This is going to be one of the causes of your problems - you
> > have 511 oustanding write requests, and only one read at a time.
> > Reduce the ioscehduer queue depth, and potentially also the device
> > CTQ depth.
> 
> Dave, I'm puzzled by this. I'd believe that a higher #req. would help 
> the block layer to resort I/O in the elevator, and therefore help to 
> gain throughput. Why would 128 be better than 512 here?

512 * 16ms per IO = 7-8s IO latency.

Fundamentally, deep queues are as harmful to latency as shallow
queues are to throughput. Everyone says "make the queues deeper" to
get the highest benchmark numbers, but in reality most benchmarks
measure throughput and aren't IO latency sensistive.

I did a bunch of measurement 7 or8 years ago on high end FC HW RAID,
and found that a CTQ depth per lun of 4 was all that was needed to
reach maximum write bandwidth under almost all circumstances. When
doing concurrent read and write with a CTQ depth of 4, the balance
was roughly 50/50 read/write. Al things the same except for a CTQ
depth of 6, and it was 30/70 read/write. And any CTQ depth deeper
than 8 is was roughly 10/90 read/write. That hardware supported a
CTQ depth of 240 IOs per lun....

So even high end hardware that can support a maximum CTQ depth of
256 IOs will see this problem - you'll get 255 writes and a single
read at a time, resulting in terrible read IO latency. There is
always another async write ready to be queued, but the application
doesn't queue another read until the first one completes. Hence
reads always are issued in small numbers and when any IO is
completed, there isn't another read queued ready for dispatch. Hence
all that happens is that async writes are sent to the drive.

And then when the BBWC fills up and has to flush all those writes,
everything slows right done because the cache effective becomes
a write-through cache - it can't take another read or write until
the flush completes another IO and space is freed in the BBWC for
the next IO.

> And maybe Matthew could profit from limiting the vm.dirty_bytes, I've 
> seen when this value is too high the server stucks on lots of writes, 
> for streaming it's better to have this smaller so the disk writes can 
> keep up and delays are not too long.

I pretty much never tune dirty limits anymore - most writeback
problems are storage stack related these days...

> > Oh, I just noticed you are might be using CFQ (it's the default in
> > dmesg). Don't - CFQ is highly unsuited for hardware RAID - it's
> > hueristically tuned to work well on sngle SATA drives. Use deadline,
> > or preferably for hardware RAID, noop.
> 
> Wouldn't deadline be better with a higher rq_qu size? As I understand 
> it, noop only groups adjacent I/Os together, while deadline does a bit 
> more and should be able to get bigger adjacent I/O areas because it 
> waits a bit longer before a flush.

The BBWC does a much better job of sorting and batching IOs than the
io scheduler can ever possibly hope to. Think about it - 512MB can
hold a 100,000 4k IOs and reorder and batch them far more
effectively than a io scheduler with even a 512 request deept
queue.

That's why making the IO scheduler queue deeper with HW RAID is
harmful - it's not needed to reach maximum performance for almost
all workloads, and all it does is add latency to the IO path...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-15 11:25               ` Bernd Schubert
@ 2012-06-15 12:30                 ` Dave Chinner
  2012-06-15 14:22                   ` Bernd Schubert
  0 siblings, 1 reply; 20+ messages in thread
From: Dave Chinner @ 2012-06-15 12:30 UTC (permalink / raw)
  To: Bernd Schubert; +Cc: Matthew Whittaker-Williams, xfs

On Fri, Jun 15, 2012 at 01:25:26PM +0200, Bernd Schubert wrote:
> On 06/15/2012 02:16 AM, Dave Chinner wrote:
> >Oh, I just noticed you are might be using CFQ (it's the default in
> >dmesg). Don't - CFQ is highly unsuited for hardware RAID - it's
> >hueristically tuned to work well on sngle SATA drives. Use deadline,
> >or preferably for hardware RAID, noop.
> 
> I'm not sure if noop is really a good recommendation even with hw
> raid, especially if the the request queue size is high. This week I
> did some benchmarks with a high rq write size (triggered with
> sync_file_range(..., SYNC_FILE_RANGE_WRITE) ) and with noop
> concuring reads then almost entirely got stalled.
> With deadline read/write balance was much better, although writes
> still had been preferred (with sync_file_range() and without). I
> always thought deadline prefers reads and I hope I find some time
> later on to investigate further what was going on.
> Test had been on a netapp E5400 hw raid, so rather high end hw raid.

Sounds like a case of the IO scheduler queue and/or CTQ being too
deep.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: XFS hangs and freezes with LSI 9265-8i controller on high i/o
  2012-06-15 12:30                 ` Dave Chinner
@ 2012-06-15 14:22                   ` Bernd Schubert
  0 siblings, 0 replies; 20+ messages in thread
From: Bernd Schubert @ 2012-06-15 14:22 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Matthew Whittaker-Williams, xfs

On 06/15/2012 02:30 PM, Dave Chinner wrote:
> On Fri, Jun 15, 2012 at 01:25:26PM +0200, Bernd Schubert wrote:
>> On 06/15/2012 02:16 AM, Dave Chinner wrote:
>>> Oh, I just noticed you are might be using CFQ (it's the default in
>>> dmesg). Don't - CFQ is highly unsuited for hardware RAID - it's
>>> hueristically tuned to work well on sngle SATA drives. Use deadline,
>>> or preferably for hardware RAID, noop.
>>
>> I'm not sure if noop is really a good recommendation even with hw
>> raid, especially if the the request queue size is high. This week I
>> did some benchmarks with a high rq write size (triggered with
>> sync_file_range(..., SYNC_FILE_RANGE_WRITE) ) and with noop
>> concuring reads then almost entirely got stalled.
>> With deadline read/write balance was much better, although writes
>> still had been preferred (with sync_file_range() and without). I
>> always thought deadline prefers reads and I hope I find some time
>> later on to investigate further what was going on.
>> Test had been on a netapp E5400 hw raid, so rather high end hw raid.
>
> Sounds like a case of the IO scheduler queue and/or CTQ being too
> deep.

Hmm yes probably. With a small request queue and the usage of 
sync_file_range(..., SYNC_FILE_RANGE_WRITE) we only have a small page 
cache buffer. And sync_file_range is required to get perfect IO sizes as 
given by max_sectors_kb.  Without sync_file_range IOs have more or less 
random size, but very rarely aligned to the raid-stripe-size (and yes, 
mkfs.xfs options are correctly set). That is another issue I need to 
find time to investigate.


Cheers,
Bernd

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2012-06-15 14:22 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-06-11 21:37 XFS hangs and freezes with LSI 9265-8i controller on high i/o Matthew Whittaker-Williams
2012-06-12  1:18 ` Dave Chinner
2012-06-12 15:56   ` Matthew Whittaker-Williams
2012-06-12 17:40     ` Matthew Whittaker-Williams
2012-06-13  0:12     ` Stan Hoeppner
2012-06-13  1:19     ` Dave Chinner
2012-06-13  3:56       ` Stan Hoeppner
2012-06-13  8:54       ` Matthew Whittaker-Williams
2012-06-13 11:59         ` Andre Noll
2012-06-13 12:13           ` Michael Monnerie
2012-06-13 16:12             ` Stan Hoeppner
2012-06-14  7:31               ` Michael Monnerie
2012-06-14  0:04         ` Dave Chinner
2012-06-14 14:31           ` Matthew Whittaker-Williams
2012-06-15  0:16             ` Dave Chinner
2012-06-15  9:52               ` Michael Monnerie
2012-06-15 12:29                 ` Dave Chinner
2012-06-15 11:25               ` Bernd Schubert
2012-06-15 12:30                 ` Dave Chinner
2012-06-15 14:22                   ` Bernd Schubert

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.