* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
@ 2011-03-25 10:16 Sedat Dilek
2011-03-25 13:05 ` Sedat Dilek
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-25 10:16 UTC (permalink / raw)
To: linux-next, LKML
Cc: Stephen Rothwell, Randy Dunlap, Theodore Ts'o, Jens Axboe,
Tejun Heo, Al Viro, Paul E. McKenney, Josh Triplett, Nick Piggin
[-- Attachment #1: Type: text/plain, Size: 478 bytes --]
Hi,
right after I have finished building a new linux-next kernel, booting
into desktop and archiving my build-tree (ext4) as tarball to an
external USB harddisk (partition there is ext3).
( Yesterday, I have seen similiar call-traces in my logs, but it was
hard to reproduce [1]. )
I am unsure from where the problem aroses, if you have a hint, let me know.
Regards,
- Sedat -
[1] http://lkml.org/lkml/2011/3/24/268
P.S.: Attached are the dmesg outputs and my kernel-config
[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 95110 bytes --]
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.38-next20110325-2-686-iniza (Debian 2.6.38-2~next20110325.dileks2) (sedat.dilek@gmail.com) (gcc version 4.6.0 20110322 (prerelease) (Debian 4.6.0~rc1-3) ) #1 SMP Fri Mar 25 10:37:32 CET 2011
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
[ 0.000000] BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000d2000 - 00000000000d4000 (reserved)
[ 0.000000] BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000003ff60000 (usable)
[ 0.000000] BIOS-e820: 000000003ff60000 - 000000003ff77000 (ACPI data)
[ 0.000000] BIOS-e820: 000000003ff77000 - 000000003ff79000 (ACPI NVS)
[ 0.000000] BIOS-e820: 000000003ff80000 - 0000000040000000 (reserved)
[ 0.000000] BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
[ 0.000000] Notice: NX (Execute Disable) protection missing in CPU!
[ 0.000000] DMI present.
[ 0.000000] DMI: IBM 2374SG6/2374SG6, BIOS 1RETDRWW (3.23 ) 06/18/2007
[ 0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
[ 0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
[ 0.000000] last_pfn = 0x3ff60 max_arch_pfn = 0x100000
[ 0.000000] MTRR default type: uncachable
[ 0.000000] MTRR fixed ranges enabled:
[ 0.000000] 00000-9FFFF write-back
[ 0.000000] A0000-BFFFF uncachable
[ 0.000000] C0000-CFFFF write-protect
[ 0.000000] D0000-DBFFF uncachable
[ 0.000000] DC000-DFFFF write-back
[ 0.000000] E0000-FFFFF write-protect
[ 0.000000] MTRR variable ranges enabled:
[ 0.000000] 0 base 000000000 mask FC0000000 write-back
[ 0.000000] 1 base 03FF80000 mask FFFF80000 uncachable
[ 0.000000] 2 disabled
[ 0.000000] 3 disabled
[ 0.000000] 4 disabled
[ 0.000000] 5 disabled
[ 0.000000] 6 disabled
[ 0.000000] 7 disabled
[ 0.000000] PAT not supported by CPU.
[ 0.000000] initial memory mapped : 0 - 01800000
[ 0.000000] Base memory trampoline at [c009b000] 9b000 size 16384
[ 0.000000] init_memory_mapping: 0000000000000000-00000000377fe000
[ 0.000000] 0000000000 - 0000400000 page 4k
[ 0.000000] 0000400000 - 0037400000 page 2M
[ 0.000000] 0037400000 - 00377fe000 page 4k
[ 0.000000] kernel direct mapping tables up to 377fe000 @ 17fb000-1800000
[ 0.000000] RAMDISK: 36aac000 - 3754e000
[ 0.000000] ACPI: RSDP 000f6d70 00024 (v02 IBM )
[ 0.000000] ACPI: XSDT 3ff6a672 0004C (v01 IBM TP-1R 00003230 LTP 00000000)
[ 0.000000] ACPI: FACP 3ff6a700 000F4 (v03 IBM TP-1R 00003230 IBM 00000001)
[ 0.000000] ACPI Warning: 32/64X length mismatch in Gpe1Block: 0/32 (20110316/tbfadt-529)
[ 0.000000] ACPI Warning: Optional field Gpe1Block has zero address or length: 0x000000000000102C/0x0 (20110316/tbfadt-560)
[ 0.000000] ACPI: DSDT 3ff6a8e7 0C530 (v01 IBM TP-1R 00003230 MSFT 0100000E)
[ 0.000000] ACPI: FACS 3ff78000 00040
[ 0.000000] ACPI: SSDT 3ff6a8b4 00033 (v01 IBM TP-1R 00003230 MSFT 0100000E)
[ 0.000000] ACPI: ECDT 3ff76e17 00052 (v01 IBM TP-1R 00003230 IBM 00000001)
[ 0.000000] ACPI: TCPA 3ff76e69 00032 (v01 IBM TP-1R 00003230 PTL 00000001)
[ 0.000000] ACPI: BOOT 3ff76fd8 00028 (v01 IBM TP-1R 00003230 LTP 00000001)
[ 0.000000] 135MB HIGHMEM available.
[ 0.000000] 887MB LOWMEM available.
[ 0.000000] mapped low ram: 0 - 377fe000
[ 0.000000] low ram: 0 - 377fe000
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000010 -> 0x00001000
[ 0.000000] Normal 0x00001000 -> 0x000377fe
[ 0.000000] HighMem 0x000377fe -> 0x0003ff60
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[2] active PFN ranges
[ 0.000000] 0: 0x00000010 -> 0x0000009f
[ 0.000000] 0: 0x00000100 -> 0x0003ff60
[ 0.000000] On node 0 totalpages: 261871
[ 0.000000] free_area_init_node: node 0, pgdat c1411100, node_mem_map f62ac200
[ 0.000000] DMA zone: 32 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 3951 pages, LIFO batch:0
[ 0.000000] Normal zone: 1744 pages used for memmap
[ 0.000000] Normal zone: 221486 pages, LIFO batch:31
[ 0.000000] HighMem zone: 271 pages used for memmap
[ 0.000000] HighMem zone: 34387 pages, LIFO batch:7
[ 0.000000] Using APIC driver default
[ 0.000000] ACPI: PM-Timer IO Port: 0x1008
[ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.000000] Local APIC disabled by BIOS -- reenabling.
[ 0.000000] Found and enabled local APIC!
[ 0.000000] nr_irqs_gsi: 16
[ 0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
[ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000d2000
[ 0.000000] PM: Registered nosave memory: 00000000000d2000 - 00000000000d4000
[ 0.000000] PM: Registered nosave memory: 00000000000d4000 - 00000000000dc000
[ 0.000000] PM: Registered nosave memory: 00000000000dc000 - 0000000000100000
[ 0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:bf800000)
[ 0.000000] Booting paravirtualized kernel on bare hardware
[ 0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 12 pages/cpu @f5c00000 s27328 r0 d21824 u4194304
[ 0.000000] pcpu-alloc: s27328 r0 d21824 u4194304 alloc=1*4194304
[ 0.000000] pcpu-alloc: [0] 0
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 259824
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-2.6.38-next20110325-2-686-iniza root=UUID=1ceb69a7-ecf4-47e9-a231-b74e0f0a9b62 ro init=/bin/systemd radeon.modeset=1 lapic 3
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Initializing CPU#0
[ 0.000000] Initializing HighMem for node 0 (000377fe:0003ff60)
[ 0.000000] Memory: 1022072k/1047936k available (2738k kernel code, 25412k reserved, 1451k data, 380k init, 138632k highmem)
[ 0.000000] virtual kernel memory layout:
[ 0.000000] fixmap : 0xffd36000 - 0xfffff000 (2852 kB)
[ 0.000000] pkmap : 0xff800000 - 0xffc00000 (4096 kB)
[ 0.000000] vmalloc : 0xf7ffe000 - 0xff7fe000 ( 120 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xf77fe000 ( 887 MB)
[ 0.000000] .init : 0xc1418000 - 0xc1477000 ( 380 kB)
[ 0.000000] .data : 0xc12aca6d - 0xc1417840 (1451 kB)
[ 0.000000] .text : 0xc1000000 - 0xc12aca6d (2738 kB)
[ 0.000000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 0.000000] NR_IRQS:1280
[ 0.000000] CPU 0 irqstacks, hard=f5818000 soft=f581a000
[ 0.000000] Extended CMOS year: 2000
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] Fast TSC calibration using PIT
[ 0.000000] Detected 1694.284 MHz processor.
[ 0.004005] Calibrating delay loop (skipped), value calculated using timer frequency.. 3388.56 BogoMIPS (lpj=6777136)
[ 0.004079] pid_max: default: 32768 minimum: 301
[ 0.004234] Security Framework initialized
[ 0.004276] SELinux: Disabled at boot.
[ 0.004419] Mount-cache hash table entries: 512
[ 0.004925] Initializing cgroup subsys debug
[ 0.004964] Initializing cgroup subsys ns
[ 0.005001] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[ 0.005050] Initializing cgroup subsys cpuacct
[ 0.005117] Initializing cgroup subsys devices
[ 0.005152] Initializing cgroup subsys freezer
[ 0.005188] Initializing cgroup subsys net_cls
[ 0.005224] Initializing cgroup subsys blkio
[ 0.005335] mce: CPU supports 5 MCE banks
[ 0.005380] CPU0: Thermal monitoring enabled (TM2)
[ 0.005591] SMP alternatives: switching to UP code
[ 0.008528] Freeing SMP alternatives: 8k freed
[ 0.008567] ACPI: Core revision 20110316
[ 0.015004] ACPI: setting ELCR to 0200 (from 0800)
[ 0.016093] weird, boot CPU (#0) not listed by the BIOS.
[ 0.016130] SMP motherboard not detected.
[ 0.016168] Enabling APIC mode: Flat. Using 0 I/O APICs
[ 0.020001] SMP disabled
[ 0.020001] Performance Events: p6 PMU driver.
[ 0.020001] ... version: 0
[ 0.020001] ... bit width: 32
[ 0.020001] ... generic registers: 2
[ 0.020001] ... value mask: 00000000ffffffff
[ 0.020001] ... max period: 000000007fffffff
[ 0.020001] ... fixed-purpose events: 0
[ 0.020001] ... event mask: 0000000000000003
[ 0.020001] NMI watchdog enabled, takes one hw-pmu counter.
[ 0.020001] Brought up 1 CPUs
[ 0.020001] Total of 1 processors activated (3388.56 BogoMIPS).
[ 0.020001] devtmpfs: initialized
[ 0.020001] PM: Registering ACPI NVS region at 3ff77000 (8192 bytes)
[ 0.020001] print_constraints: dummy:
[ 0.020001] NET: Registered protocol family 16
[ 0.020001] ACPI: bus type pci registered
[ 0.020001] PCI: PCI BIOS revision 2.10 entry at 0xfd8d6, last bus=8
[ 0.020001] PCI: Using configuration type 1 for base access
[ 0.020001] bio: create slab <bio-0> at 0
[ 0.021437] ACPI: EC: EC description table is found, configuring boot EC
[ 0.034306] ACPI: Interpreter enabled
[ 0.034345] ACPI: (supports S0 S3 S4 S5)
[ 0.034483] ACPI: Using PIC for interrupt routing
[ 0.036958] ACPI: Power Resource [PUBS] (on)
[ 0.040528] ACPI: EC: GPE = 0x1c, I/O: command/status = 0x66, data = 0x62
[ 0.041079] ACPI: ACPI Dock Station Driver: 3 docks/bays found
[ 0.041079] HEST: Table not found.
[ 0.044008] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
[ 0.044092] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.044206] pci_root PNP0A03:00: host bridge window [io 0x0000-0x0cf7] (ignored)
[ 0.044210] pci_root PNP0A03:00: host bridge window [io 0x0d00-0xffff] (ignored)
[ 0.044215] pci_root PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff] (ignored)
[ 0.044220] pci_root PNP0A03:00: host bridge window [mem 0x000d4000-0x000d7fff] (ignored)
[ 0.044224] pci_root PNP0A03:00: host bridge window [mem 0x000d8000-0x000dbfff] (ignored)
[ 0.044229] pci_root PNP0A03:00: host bridge window [mem 0x40000000-0xfebfffff] (ignored)
[ 0.044248] pci 0000:00:00.0: [8086:3340] type 0 class 0x000600
[ 0.044258] pci 0000:00:00.0: reg 10: [mem 0xd0000000-0xdfffffff pref]
[ 0.044300] pci 0000:00:01.0: [8086:3341] type 1 class 0x000604
[ 0.044360] pci 0000:00:1d.0: [8086:24c2] type 0 class 0x000c03
[ 0.044406] pci 0000:00:1d.0: reg 20: [io 0x1800-0x181f]
[ 0.044441] pci 0000:00:1d.1: [8086:24c4] type 0 class 0x000c03
[ 0.044488] pci 0000:00:1d.1: reg 20: [io 0x1820-0x183f]
[ 0.044529] pci 0000:00:1d.2: [8086:24c7] type 0 class 0x000c03
[ 0.044575] pci 0000:00:1d.2: reg 20: [io 0x1840-0x185f]
[ 0.044622] pci 0000:00:1d.7: [8086:24cd] type 0 class 0x000c03
[ 0.044646] pci 0000:00:1d.7: reg 10: [mem 0xc0000000-0xc00003ff]
[ 0.044727] pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
[ 0.044733] pci 0000:00:1d.7: PME# disabled
[ 0.044754] pci 0000:00:1e.0: [8086:2448] type 1 class 0x000604
[ 0.044799] pci 0000:00:1f.0: [8086:24cc] type 0 class 0x000601
[ 0.044868] pci 0000:00:1f.0: quirk: [io 0x1000-0x107f] claimed by ICH4 ACPI/GPIO/TCO
[ 0.044919] pci 0000:00:1f.0: quirk: [io 0x1180-0x11bf] claimed by ICH4 GPIO
[ 0.044971] pci 0000:00:1f.1: [8086:24ca] type 0 class 0x000101
[ 0.044987] pci 0000:00:1f.1: reg 10: [io 0x0000-0x0007]
[ 0.044998] pci 0000:00:1f.1: reg 14: [io 0x0000-0x0003]
[ 0.045009] pci 0000:00:1f.1: reg 18: [io 0x0000-0x0007]
[ 0.045021] pci 0000:00:1f.1: reg 1c: [io 0x0000-0x0003]
[ 0.045032] pci 0000:00:1f.1: reg 20: [io 0x1860-0x186f]
[ 0.045044] pci 0000:00:1f.1: reg 24: [mem 0x00000000-0x000003ff]
[ 0.045074] pci 0000:00:1f.3: [8086:24c3] type 0 class 0x000c05
[ 0.045120] pci 0000:00:1f.3: reg 20: [io 0x1880-0x189f]
[ 0.045159] pci 0000:00:1f.5: [8086:24c5] type 0 class 0x000401
[ 0.045176] pci 0000:00:1f.5: reg 10: [io 0x1c00-0x1cff]
[ 0.045187] pci 0000:00:1f.5: reg 14: [io 0x18c0-0x18ff]
[ 0.045197] pci 0000:00:1f.5: reg 18: [mem 0xc0000c00-0xc0000dff]
[ 0.045208] pci 0000:00:1f.5: reg 1c: [mem 0xc0000800-0xc00008ff]
[ 0.045248] pci 0000:00:1f.5: PME# supported from D0 D3hot D3cold
[ 0.045253] pci 0000:00:1f.5: PME# disabled
[ 0.045271] pci 0000:00:1f.6: [8086:24c6] type 0 class 0x000703
[ 0.045288] pci 0000:00:1f.6: reg 10: [io 0x2400-0x24ff]
[ 0.045298] pci 0000:00:1f.6: reg 14: [io 0x2000-0x207f]
[ 0.045351] pci 0000:00:1f.6: PME# supported from D0 D3hot D3cold
[ 0.045357] pci 0000:00:1f.6: PME# disabled
[ 0.045384] pci 0000:01:00.0: [1002:4c66] type 0 class 0x000300
[ 0.045400] pci 0000:01:00.0: reg 10: [mem 0xe0000000-0xe7ffffff pref]
[ 0.045410] pci 0000:01:00.0: reg 14: [io 0x3000-0x30ff]
[ 0.045419] pci 0000:01:00.0: reg 18: [mem 0xc0100000-0xc010ffff]
[ 0.045444] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
[ 0.045464] pci 0000:01:00.0: supports D1 D2
[ 0.045501] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[ 0.045539] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff]
[ 0.045544] pci 0000:00:01.0: bridge window [mem 0xc0100000-0xc01fffff]
[ 0.045549] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe7ffffff pref]
[ 0.045576] pci 0000:02:00.0: [104c:ac55] type 2 class 0x000607
[ 0.045594] pci 0000:02:00.0: reg 10: [mem 0xb0000000-0xb0000fff]
[ 0.045614] pci 0000:02:00.0: supports D1 D2
[ 0.045618] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 0.045624] pci 0000:02:00.0: PME# disabled
[ 0.045645] pci 0000:02:00.1: [104c:ac55] type 2 class 0x000607
[ 0.045663] pci 0000:02:00.1: reg 10: [mem 0xb1000000-0xb1000fff]
[ 0.045683] pci 0000:02:00.1: supports D1 D2
[ 0.045687] pci 0000:02:00.1: PME# supported from D0 D1 D2 D3hot D3cold
[ 0.045693] pci 0000:02:00.1: PME# disabled
[ 0.045721] pci 0000:02:01.0: [8086:101e] type 0 class 0x000200
[ 0.045741] pci 0000:02:01.0: reg 10: [mem 0xc0220000-0xc023ffff]
[ 0.045753] pci 0000:02:01.0: reg 14: [mem 0xc0200000-0xc020ffff]
[ 0.045764] pci 0000:02:01.0: reg 18: [io 0x8000-0x803f]
[ 0.045797] pci 0000:02:01.0: reg 30: [mem 0x00000000-0x0000ffff pref]
[ 0.045822] pci 0000:02:01.0: PME# supported from D0 D3hot D3cold
[ 0.045828] pci 0000:02:01.0: PME# disabled
[ 0.045849] pci 0000:02:02.0: [168c:1014] type 0 class 0x000200
[ 0.045867] pci 0000:02:02.0: reg 10: [mem 0xc0210000-0xc021ffff]
[ 0.045969] pci 0000:00:1e.0: PCI bridge to [bus 02-08] (subtractive decode)
[ 0.046010] pci 0000:00:1e.0: bridge window [io 0x4000-0x8fff]
[ 0.046016] pci 0000:00:1e.0: bridge window [mem 0xc0200000-0xcfffffff]
[ 0.046022] pci 0000:00:1e.0: bridge window [mem 0xe8000000-0xefffffff pref]
[ 0.046026] pci 0000:00:1e.0: bridge window [io 0x0000-0xffff] (subtractive decode)
[ 0.046031] pci 0000:00:1e.0: bridge window [mem 0x00000000-0xffffffff] (subtractive decode)
[ 0.046109] pci_bus 0000:00: on NUMA node 0
[ 0.046114] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[ 0.046164] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGP_._PRT]
[ 0.046191] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
[ 0.049472] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.049812] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.050148] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.050484] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.050800] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
[ 0.051166] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
[ 0.051533] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
[ 0.051920] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.052283] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.052332] vgaarb: loaded
[ 0.052413] PCI: Using ACPI for IRQ routing
[ 0.052583] PCI: pci_cache_line_size set to 64 bytes
[ 0.052652] reserve RAM buffer: 000000000009f000 - 000000000009ffff
[ 0.052656] reserve RAM buffer: 000000003ff60000 - 000000003fffffff
[ 0.054780] pnp: PnP ACPI init
[ 0.054838] ACPI: bus type pnp registered
[ 0.055489] pnp 00:00: [mem 0x00000000-0x0009ffff]
[ 0.055493] pnp 00:00: [mem 0x000c0000-0x000c3fff]
[ 0.055497] pnp 00:00: [mem 0x000c4000-0x000c7fff]
[ 0.055501] pnp 00:00: [mem 0x000c8000-0x000cbfff]
[ 0.055504] pnp 00:00: [mem 0x000cc000-0x000cffff]
[ 0.055508] pnp 00:00: [mem 0x000d0000-0x000d3fff]
[ 0.055511] pnp 00:00: [mem 0x000d4000-0x000d3fff disabled]
[ 0.055515] pnp 00:00: [mem 0x000d8000-0x000d7fff disabled]
[ 0.055519] pnp 00:00: [mem 0x000dc000-0x000dffff]
[ 0.055522] pnp 00:00: [mem 0x000e0000-0x000e3fff]
[ 0.055526] pnp 00:00: [mem 0x000e4000-0x000e7fff]
[ 0.055529] pnp 00:00: [mem 0x000e8000-0x000ebfff]
[ 0.055533] pnp 00:00: [mem 0x000ec000-0x000effff]
[ 0.055536] pnp 00:00: [mem 0x000f0000-0x000fffff]
[ 0.055540] pnp 00:00: [mem 0x00100000-0x3fffffff]
[ 0.055543] pnp 00:00: [mem 0xfec00000-0xffffffff]
[ 0.055633] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
[ 0.055674] system 00:00: [mem 0x000c0000-0x000c3fff] could not be reserved
[ 0.055714] system 00:00: [mem 0x000c4000-0x000c7fff] could not be reserved
[ 0.055753] system 00:00: [mem 0x000c8000-0x000cbfff] could not be reserved
[ 0.055793] system 00:00: [mem 0x000cc000-0x000cffff] could not be reserved
[ 0.055832] system 00:00: [mem 0x000d0000-0x000d3fff] could not be reserved
[ 0.055872] system 00:00: [mem 0x000dc000-0x000dffff] could not be reserved
[ 0.055911] system 00:00: [mem 0x000e0000-0x000e3fff] could not be reserved
[ 0.055950] system 00:00: [mem 0x000e4000-0x000e7fff] could not be reserved
[ 0.056031] system 00:00: [mem 0x000e8000-0x000ebfff] could not be reserved
[ 0.056071] system 00:00: [mem 0x000ec000-0x000effff] could not be reserved
[ 0.056111] system 00:00: [mem 0x000f0000-0x000fffff] could not be reserved
[ 0.056150] system 00:00: [mem 0x00100000-0x3fffffff] could not be reserved
[ 0.056190] system 00:00: [mem 0xfec00000-0xffffffff] could not be reserved
[ 0.056231] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active)
[ 0.056266] pnp 00:01: [bus 00-ff]
[ 0.056270] pnp 00:01: [io 0x0cf8-0x0cff]
[ 0.056274] pnp 00:01: [io 0x0000-0x0cf7 window]
[ 0.056277] pnp 00:01: [io 0x0d00-0xffff window]
[ 0.056289] pnp 00:01: [mem 0x000a0000-0x000bffff window]
[ 0.056292] pnp 00:01: [mem 0x000c0000-0x000c3fff window]
[ 0.056296] pnp 00:01: [mem 0x000c4000-0x000c7fff window]
[ 0.056300] pnp 00:01: [mem 0x000c8000-0x000cbfff window]
[ 0.056304] pnp 00:01: [mem 0x000cc000-0x000cffff window]
[ 0.056308] pnp 00:01: [mem 0x000d0000-0x000d3fff window]
[ 0.056311] pnp 00:01: [mem 0x000d4000-0x000d7fff window]
[ 0.056315] pnp 00:01: [mem 0x000d8000-0x000dbfff window]
[ 0.056319] pnp 00:01: [mem 0x000dc000-0x000dffff window]
[ 0.056323] pnp 00:01: [mem 0x000e0000-0x000e3fff window]
[ 0.056326] pnp 00:01: [mem 0x000e4000-0x000e7fff window]
[ 0.056330] pnp 00:01: [mem 0x000e8000-0x000ebfff window]
[ 0.056334] pnp 00:01: [mem 0x000ec000-0x000effff window]
[ 0.056338] pnp 00:01: [mem 0x40000000-0xfebfffff window]
[ 0.056406] pnp 00:01: Plug and Play ACPI device, IDs PNP0a03 (active)
[ 0.056515] pnp 00:02: [io 0x0010-0x001f]
[ 0.056519] pnp 00:02: [io 0x0090-0x009f]
[ 0.056522] pnp 00:02: [io 0x0024-0x0025]
[ 0.056525] pnp 00:02: [io 0x0028-0x0029]
[ 0.056529] pnp 00:02: [io 0x002c-0x002d]
[ 0.056532] pnp 00:02: [io 0x0030-0x0031]
[ 0.056535] pnp 00:02: [io 0x0034-0x0035]
[ 0.056538] pnp 00:02: [io 0x0038-0x0039]
[ 0.056542] pnp 00:02: [io 0x003c-0x003d]
[ 0.056545] pnp 00:02: [io 0x00a4-0x00a5]
[ 0.056548] pnp 00:02: [io 0x00a8-0x00a9]
[ 0.056551] pnp 00:02: [io 0x00ac-0x00ad]
[ 0.056555] pnp 00:02: [io 0x00b0-0x00b5]
[ 0.056558] pnp 00:02: [io 0x00b8-0x00b9]
[ 0.056561] pnp 00:02: [io 0x00bc-0x00bd]
[ 0.056564] pnp 00:02: [io 0x0050-0x0053]
[ 0.056568] pnp 00:02: [io 0x0072-0x0077]
[ 0.056571] pnp 00:02: [io 0x002e-0x002f]
[ 0.056574] pnp 00:02: [io 0x1000-0x107f]
[ 0.056578] pnp 00:02: [io 0x1180-0x11bf]
[ 0.056581] pnp 00:02: [io 0x15e0-0x15ef]
[ 0.056584] pnp 00:02: [io 0x1600-0x162f]
[ 0.056587] pnp 00:02: [io 0x1632-0x167f]
[ 0.056591] pnp 00:02: [io 0x004e-0x004f]
[ 0.056594] pnp 00:02: [io 0x1630-0x1631]
[ 0.056697] system 00:02: [io 0x1000-0x107f] has been reserved
[ 0.056736] system 00:02: [io 0x1180-0x11bf] has been reserved
[ 0.056774] system 00:02: [io 0x15e0-0x15ef] has been reserved
[ 0.056812] system 00:02: [io 0x1600-0x162f] has been reserved
[ 0.056850] system 00:02: [io 0x1632-0x167f] has been reserved
[ 0.056889] system 00:02: [io 0x1630-0x1631] has been reserved
[ 0.056927] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 0.056947] pnp 00:03: [io 0x0000-0x000f]
[ 0.056951] pnp 00:03: [io 0x0080-0x008f]
[ 0.056954] pnp 00:03: [io 0x00c0-0x00df]
[ 0.056958] pnp 00:03: [dma 4]
[ 0.056994] pnp 00:03: Plug and Play ACPI device, IDs PNP0200 (active)
[ 0.057007] pnp 00:04: [io 0x0061]
[ 0.057042] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
[ 0.057055] pnp 00:05: [io 0x00f0]
[ 0.057060] pnp 00:05: [irq 13]
[ 0.057096] pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active)
[ 0.057109] pnp 00:06: [io 0x0070-0x0071]
[ 0.057113] pnp 00:06: [irq 8]
[ 0.057153] pnp 00:06: Plug and Play ACPI device, IDs PNP0b00 (active)
[ 0.057167] pnp 00:07: [io 0x0060]
[ 0.057170] pnp 00:07: [io 0x0064]
[ 0.057173] pnp 00:07: [irq 1]
[ 0.057208] pnp 00:07: Plug and Play ACPI device, IDs PNP0303 (active)
[ 0.057221] pnp 00:08: [irq 12]
[ 0.057258] pnp 00:08: Plug and Play ACPI device, IDs IBM0057 PNP0f13 (active)
[ 0.057296] pnp 00:09: [io 0x03f0-0x03f5]
[ 0.057299] pnp 00:09: [io 0x03f7]
[ 0.057303] pnp 00:09: [irq 6]
[ 0.057306] pnp 00:09: [dma 2]
[ 0.057358] pnp 00:09: Plug and Play ACPI device, IDs PNP0700 (active)
[ 0.057460] pnp 00:0a: [io 0x03f8-0x03ff]
[ 0.057464] pnp 00:0a: [irq 4]
[ 0.057565] pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (active)
[ 0.057682] pnp 00:0b: [io 0x03bc-0x03be]
[ 0.057686] pnp 00:0b: [irq 7]
[ 0.057779] pnp 00:0b: Plug and Play ACPI device, IDs PNP0400 (active)
[ 0.057939] pnp 00:0c: Plug and Play ACPI device, IDs IBM0071 PNP0511 (disabled)
[ 0.057999] pnp: PnP ACPI: found 13 devices
[ 0.057999] ACPI: ACPI bus type pnp unregistered
[ 0.057999] PnPBIOS: Disabled by ACPI PNP
[ 0.098313] Switching to clocksource acpi_pm
[ 0.098400] pci 0000:00:1f.1: BAR 5: assigned [mem 0x40000000-0x400003ff]
[ 0.098444] pci 0000:00:1f.1: BAR 5: set to [mem 0x40000000-0x400003ff] (PCI address [0x40000000-0x400003ff])
[ 0.098497] pci 0000:01:00.0: BAR 6: assigned [mem 0xc0120000-0xc013ffff pref]
[ 0.098544] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[ 0.098581] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff]
[ 0.098621] pci 0000:00:01.0: bridge window [mem 0xc0100000-0xc01fffff]
[ 0.098661] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe7ffffff pref]
[ 0.098721] pci 0000:02:00.0: BAR 15: assigned [mem 0xe8000000-0xebffffff pref]
[ 0.098769] pci 0000:02:00.0: BAR 16: assigned [mem 0xc4000000-0xc7ffffff]
[ 0.098808] pci 0000:02:00.1: BAR 15: assigned [mem 0xec000000-0xefffffff pref]
[ 0.098856] pci 0000:02:00.1: BAR 16: assigned [mem 0xc8000000-0xcbffffff]
[ 0.098896] pci 0000:02:01.0: BAR 6: assigned [mem 0xc0240000-0xc024ffff pref]
[ 0.098943] pci 0000:02:00.0: BAR 13: assigned [io 0x4000-0x40ff]
[ 0.098981] pci 0000:02:00.0: BAR 14: assigned [io 0x4400-0x44ff]
[ 0.099019] pci 0000:02:00.1: BAR 13: assigned [io 0x4800-0x48ff]
[ 0.099058] pci 0000:02:00.1: BAR 14: assigned [io 0x4c00-0x4cff]
[ 0.099096] pci 0000:02:00.0: CardBus bridge to [bus 03-06]
[ 0.099133] pci 0000:02:00.0: bridge window [io 0x4000-0x40ff]
[ 0.099173] pci 0000:02:00.0: bridge window [io 0x4400-0x44ff]
[ 0.099212] pci 0000:02:00.0: bridge window [mem 0xe8000000-0xebffffff pref]
[ 0.099261] pci 0000:02:00.0: bridge window [mem 0xc4000000-0xc7ffffff]
[ 0.099302] pci 0000:02:00.1: CardBus bridge to [bus 07-07]
[ 0.099339] pci 0000:02:00.1: bridge window [io 0x4800-0x48ff]
[ 0.099378] pci 0000:02:00.1: bridge window [io 0x4c00-0x4cff]
[ 0.099418] pci 0000:02:00.1: bridge window [mem 0xec000000-0xefffffff pref]
[ 0.099467] pci 0000:02:00.1: bridge window [mem 0xc8000000-0xcbffffff]
[ 0.099508] pci 0000:00:1e.0: PCI bridge to [bus 02-08]
[ 0.099545] pci 0000:00:1e.0: bridge window [io 0x4000-0x8fff]
[ 0.099586] pci 0000:00:1e.0: bridge window [mem 0xc0200000-0xcfffffff]
[ 0.099627] pci 0000:00:1e.0: bridge window [mem 0xe8000000-0xefffffff pref]
[ 0.099690] pci 0000:00:1e.0: setting latency timer to 64
[ 0.099902] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
[ 0.099940] PCI: setting IRQ 11 as level-triggered
[ 0.099947] pci 0000:02:00.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[ 0.100005] Switched to NOHz mode on CPU #0
[ 0.100005] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[ 0.100005] pci 0000:02:00.1: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 0.100005] pci_bus 0000:00: resource 0 [io 0x0000-0xffff]
[ 0.100005] pci_bus 0000:00: resource 1 [mem 0x00000000-0xffffffff]
[ 0.100005] pci_bus 0000:01: resource 0 [io 0x3000-0x3fff]
[ 0.100005] pci_bus 0000:01: resource 1 [mem 0xc0100000-0xc01fffff]
[ 0.100005] pci_bus 0000:01: resource 2 [mem 0xe0000000-0xe7ffffff pref]
[ 0.100005] pci_bus 0000:02: resource 0 [io 0x4000-0x8fff]
[ 0.100005] pci_bus 0000:02: resource 1 [mem 0xc0200000-0xcfffffff]
[ 0.100005] pci_bus 0000:02: resource 2 [mem 0xe8000000-0xefffffff pref]
[ 0.100005] pci_bus 0000:02: resource 4 [io 0x0000-0xffff]
[ 0.100005] pci_bus 0000:02: resource 5 [mem 0x00000000-0xffffffff]
[ 0.100005] pci_bus 0000:03: resource 0 [io 0x4000-0x40ff]
[ 0.100005] pci_bus 0000:03: resource 1 [io 0x4400-0x44ff]
[ 0.100005] pci_bus 0000:03: resource 2 [mem 0xe8000000-0xebffffff pref]
[ 0.100005] pci_bus 0000:03: resource 3 [mem 0xc4000000-0xc7ffffff]
[ 0.100005] pci_bus 0000:07: resource 0 [io 0x4800-0x48ff]
[ 0.100005] pci_bus 0000:07: resource 1 [io 0x4c00-0x4cff]
[ 0.100005] pci_bus 0000:07: resource 2 [mem 0xec000000-0xefffffff pref]
[ 0.100005] pci_bus 0000:07: resource 3 [mem 0xc8000000-0xcbffffff]
[ 0.100005] NET: Registered protocol family 2
[ 0.100005] IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.100005] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.100758] TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.101914] TCP: Hash tables configured (established 131072 bind 65536)
[ 0.101956] TCP reno registered
[ 0.101996] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 0.102070] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 0.102492] NET: Registered protocol family 1
[ 0.102671] pci 0000:01:00.0: Boot video device
[ 0.102689] PCI: CLS 32 bytes, default 64
[ 0.102788] Unpacking initramfs...
[ 0.551184] Freeing initrd memory: 10888k freed
[ 0.569522] Simple Boot Flag at 0x35 set to 0x1
[ 0.570126] audit: initializing netlink socket (disabled)
[ 0.570186] type=2000 audit(1301050466.567:1): initialized
[ 0.592240] highmem bounce pool size: 64 pages
[ 0.592283] HugeTLB registered 4 MB page size, pre-allocated 0 pages
[ 0.595202] VFS: Disk quotas dquot_6.5.2
[ 0.595417] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 0.595662] msgmni has been set to 1746
[ 0.595992] alg: No test for stdrng (krng)
[ 0.596142] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 0.596192] io scheduler noop registered
[ 0.596226] io scheduler deadline registered
[ 0.596283] io scheduler cfq registered (default)
[ 0.596595] ERST: Table is not found!
[ 0.596643] isapnp: Scanning for PnP cards...
[ 0.949555] isapnp: No Plug & Play device found
[ 0.949707] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.949834] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
[ 0.950357] 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
[ 0.950534] serial 0000:00:1f.6: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 0.950588] serial 0000:00:1f.6: PCI INT B disabled
[ 0.950771] Linux agpgart interface v0.103
[ 0.950959] agpgart-intel 0000:00:00.0: Intel 855PM Chipset
[ 0.964577] agpgart-intel 0000:00:00.0: AGP aperture is 256M @ 0xd0000000
[ 0.964850] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[ 0.971122] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 0.971167] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 0.971364] mousedev: PS/2 mouse device common for all mice
[ 0.971461] rtc_cmos 00:06: RTC can wake from S4
[ 0.971604] rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
[ 0.971658] rtc0: alarms up to one month, y3k, 114 bytes nvram
[ 0.971706] cpuidle: using governor ladder
[ 0.971741] cpuidle: using governor menu
[ 0.972191] TCP cubic registered
[ 0.972479] NET: Registered protocol family 10
[ 0.973515] Mobile IPv6
[ 0.973550] NET: Registered protocol family 17
[ 0.973588] Registering the dns_resolver key type
[ 0.973647] Using IPI No-Shortcut mode
[ 0.973791] PM: Hibernation image not present or could not be loaded.
[ 0.973806] registered taskstats version 1
[ 0.974173] rtc_cmos 00:06: setting system clock to 2011-03-25 10:54:28 UTC (1301050468)
[ 0.974258] Initializing network drop monitor service
[ 0.974363] Freeing unused kernel memory: 380k freed
[ 0.975520] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[ 1.000330] udev[43]: starting version 166
[ 1.267001] Floppy drive(s): fd0 is 1.44M
[ 1.287183] FDC 0 is a National Semiconductor PC87306
[ 1.316852] thermal LNXTHERM:00: registered as thermal_zone0
[ 1.316897] ACPI: Thermal Zone [THM0] (57 C)
[ 1.320038] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[ 1.320080] e1000: Copyright (c) 1999-2006 Intel Corporation.
[ 1.320167] e1000 0000:02:01.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[ 1.601733] Refined TSC clocksource calibration: 1694.501 MHz.
[ 1.601779] Switching to clocksource tsc
[ 1.629395] e1000 0000:02:01.0: eth0: (PCI:33MHz:32-bit) 00:0d:60:b0:62:87
[ 1.629440] e1000 0000:02:01.0: eth0: Intel(R) PRO/1000 Network Connection
[ 1.636981] usbcore: registered new interface driver usbfs
[ 1.637223] usbcore: registered new interface driver hub
[ 1.641277] usbcore: registered new device driver usb
[ 1.648897] SCSI subsystem initialized
[ 1.672083] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1.672175] ehci_hcd 0000:00:1d.7: power state changed by ACPI to D0
[ 1.672216] ehci_hcd 0000:00:1d.7: power state changed by ACPI to D0
[ 1.672495] ACPI: PCI Interrupt Link [LNKH] enabled at IRQ 11
[ 1.672536] ehci_hcd 0000:00:1d.7: PCI INT D -> Link[LNKH] -> GSI 11 (level, low) -> IRQ 11
[ 1.672607] ehci_hcd 0000:00:1d.7: setting latency timer to 64
[ 1.672612] ehci_hcd 0000:00:1d.7: EHCI Host Controller
[ 1.672706] ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
[ 1.672795] ehci_hcd 0000:00:1d.7: debug port 1
[ 1.676703] ehci_hcd 0000:00:1d.7: cache line size of 32 is not supported
[ 1.688082] ehci_hcd 0000:00:1d.7: irq 11, io mem 0xc0000000
[ 1.707435] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
[ 1.707537] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 1.707576] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.707622] usb usb1: Product: EHCI Host Controller
[ 1.707658] usb usb1: Manufacturer: Linux 2.6.38-next20110325-2-686-iniza ehci_hcd
[ 1.707706] usb usb1: SerialNumber: 0000:00:1d.7
[ 1.712191] hub 1-0:1.0: USB hub found
[ 1.712236] hub 1-0:1.0: 6 ports detected
[ 1.760551] libata version 3.00 loaded.
[ 1.765769] uhci_hcd: USB Universal Host Controller Interface driver
[ 1.765905] uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
[ 1.765945] uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
[ 1.765993] uhci_hcd 0000:00:1d.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[ 1.766054] uhci_hcd 0000:00:1d.0: setting latency timer to 64
[ 1.766059] uhci_hcd 0000:00:1d.0: UHCI Host Controller
[ 1.766112] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
[ 1.766192] uhci_hcd 0000:00:1d.0: irq 11, io base 0x00001800
[ 1.766286] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[ 1.766325] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.766372] usb usb2: Product: UHCI Host Controller
[ 1.766407] usb usb2: Manufacturer: Linux 2.6.38-next20110325-2-686-iniza uhci_hcd
[ 1.766454] usb usb2: SerialNumber: 0000:00:1d.0
[ 1.766779] hub 2-0:1.0: USB hub found
[ 1.766818] hub 2-0:1.0: 2 ports detected
[ 1.766941] uhci_hcd 0000:00:1d.1: power state changed by ACPI to D0
[ 1.766980] uhci_hcd 0000:00:1d.1: power state changed by ACPI to D0
[ 1.767249] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
[ 1.767288] uhci_hcd 0000:00:1d.1: PCI INT B -> Link[LNKD] -> GSI 11 (level, low) -> IRQ 11
[ 1.767341] uhci_hcd 0000:00:1d.1: setting latency timer to 64
[ 1.767345] uhci_hcd 0000:00:1d.1: UHCI Host Controller
[ 1.767390] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
[ 1.767457] uhci_hcd 0000:00:1d.1: irq 11, io base 0x00001820
[ 1.767533] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[ 1.767572] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.767619] usb usb3: Product: UHCI Host Controller
[ 1.767655] usb usb3: Manufacturer: Linux 2.6.38-next20110325-2-686-iniza uhci_hcd
[ 1.770417] usb usb3: SerialNumber: 0000:00:1d.1
[ 1.776199] hub 3-0:1.0: USB hub found
[ 1.776241] hub 3-0:1.0: 2 ports detected
[ 1.776576] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[ 1.776616] uhci_hcd 0000:00:1d.2: PCI INT C -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
[ 1.776672] uhci_hcd 0000:00:1d.2: setting latency timer to 64
[ 1.776677] uhci_hcd 0000:00:1d.2: UHCI Host Controller
[ 1.776722] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
[ 1.776795] uhci_hcd 0000:00:1d.2: irq 11, io base 0x00001840
[ 1.776873] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[ 1.776913] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.776959] usb usb4: Product: UHCI Host Controller
[ 1.776995] usb usb4: Manufacturer: Linux 2.6.38-next20110325-2-686-iniza uhci_hcd
[ 1.777041] usb usb4: SerialNumber: 0000:00:1d.2
[ 1.779579] hub 4-0:1.0: USB hub found
[ 1.779624] hub 4-0:1.0: 2 ports detected
[ 1.784905] ata_piix 0000:00:1f.1: version 2.13
[ 1.784925] ata_piix 0000:00:1f.1: enabling device (0005 -> 0007)
[ 1.784976] ata_piix 0000:00:1f.1: PCI INT A -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
[ 1.785084] ata_piix 0000:00:1f.1: setting latency timer to 64
[ 1.791225] scsi0 : ata_piix
[ 1.793618] scsi1 : ata_piix
[ 1.794402] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x1860 irq 14
[ 1.794442] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x1868 irq 15
[ 1.948392] ata2.01: NODEV after polling detection
[ 1.956946] ata1.00: HPA detected: current 110257519, native 117210240
[ 1.956990] ata1.00: ATA-6: HTS726060M9AT00, MH4OA6BA, max UDMA/100
[ 1.957028] ata1.00: 110257519 sectors, multi 16: LBA
[ 1.957155] ata2.00: ATAPI: UJDA755yDVD/CDRW, 1.70, max UDMA/33
[ 1.972655] ata2.00: configured for UDMA/33
[ 1.972745] ata1.00: configured for UDMA/100
[ 1.972970] scsi 0:0:0:0: Direct-Access ATA HTS726060M9AT00 MH4O PQ: 0 ANSI: 5
[ 1.976178] scsi 1:0:0:0: CD-ROM MATSHITA UJDA755yDVD/CDRW 1.70 PQ: 0 ANSI: 5
[ 2.010777] sd 0:0:0:0: [sda] 110257519 512-byte logical blocks: (56.4 GB/52.5 GiB)
[ 2.010903] sd 0:0:0:0: [sda] Write Protect is off
[ 2.010941] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 2.010972] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.014238] sr0: scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray
[ 2.014281] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 2.014721] sr 1:0:0:0: Attached scsi CD-ROM sr0
[ 2.070762] sda: sda1 sda2 sda3 sda4 < sda5 sda6 >
[ 2.071622] sd 0:0:0:0: [sda] Attached SCSI disk
[ 2.080224] usb 1-4: new high speed USB device number 3 using ehci_hcd
[ 2.097356] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 2.098014] sr 1:0:0:0: Attached scsi generic sg1 type 5
[ 2.212951] usb 1-4: New USB device found, idVendor=152d, idProduct=2329
[ 2.212999] usb 1-4: New USB device strings: Mfr=10, Product=11, SerialNumber=3
[ 2.213048] usb 1-4: Product: Storagebird 35EV821
[ 2.213084] usb 1-4: Manufacturer: 0123456
[ 2.213120] usb 1-4: SerialNumber: 000000000340
[ 2.223102] usbcore: registered new interface driver uas
[ 2.235997] Initializing USB Mass Storage driver...
[ 2.236178] scsi2 : usb-storage 1-4:1.0
[ 2.236388] usbcore: registered new interface driver usb-storage
[ 2.236426] USB Mass Storage support registered.
[ 2.452023] usb 3-1: new low speed USB device number 2 using uhci_hcd
[ 2.628399] usb 3-1: New USB device found, idVendor=046d, idProduct=c00e
[ 2.628445] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 2.628484] usb 3-1: Product: USB-PS/2 Optical Mouse
[ 2.628520] usb 3-1: Manufacturer: Logitech
[ 2.670901] input: Logitech USB-PS/2 Optical Mouse as /devices/pci0000:00/0000:00:1d.1/usb3/3-1/3-1:1.0/input/input1
[ 2.671162] generic-usb 0003:046D:C00E.0001: input,hidraw0: USB HID v1.10 Mouse [Logitech USB-PS/2 Optical Mouse] on usb-0000:00:1d.1-1/input0
[ 2.671424] usbcore: registered new interface driver usbhid
[ 2.671461] usbhid: USB HID core driver
[ 2.736230] device-mapper: uevent: version 1.0.3
[ 2.736953] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02) initialised: dm-devel@redhat.com
[ 2.936539] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
[ 3.278186] scsi 2:0:0:0: Direct-Access WDC WD10 EAVS-00D7B0 PQ: 0 ANSI: 2 CCS
[ 3.278833] sd 2:0:0:0: Attached scsi generic sg2 type 0
[ 3.279279] sd 2:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[ 3.280151] sd 2:0:0:0: [sdb] Write Protect is off
[ 3.280190] sd 2:0:0:0: [sdb] Mode Sense: 34 00 00 00
[ 3.280195] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[ 3.281648] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[ 3.372139] sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 sdb6 sdb7 sdb8 >
[ 3.374008] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[ 3.374047] sd 2:0:0:0: [sdb] Attached SCSI disk
[ 4.058593] systemd[1]: systemd 17 running in system mode. (+PAM -LIBWRAP +AUDIT +SELINUX +SYSVINIT +LIBCRYPTSETUP; debian)
[ 4.171770] systemd[1]: Set hostname to <tbox>.
[ 4.171977] IPv4 FIB: Using LC-trie version 0.409
[ 6.224954] cfg80211: Calling CRDA to update world regulatory domain
[ 6.911340] ath5k 0000:02:02.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
[ 6.911468] ath5k 0000:02:02.0: registered as 'phy0'
[ 7.203696] ath: EEPROM regdomain: 0x61
[ 7.203701] ath: EEPROM indicates we should expect a direct regpair map
[ 7.203707] ath: Country alpha2 being used: 00
[ 7.203709] ath: Regpair used: 0x61
[ 8.184121] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[ 8.185015] Registered led device: ath5k-phy0::rx
[ 8.185042] Registered led device: ath5k-phy0::tx
[ 8.185061] ath5k phy0: Atheros AR5212 chip found (MAC: 0x56, PHY: 0x41)
[ 8.185106] ath5k phy0: RF5111 5GHz radio found (0x17)
[ 8.185142] ath5k phy0: RF2111 2GHz radio found (0x23)
[ 8.345266] udev[269]: starting version 166
[ 8.978809] input: PC Speaker as /devices/platform/pcspkr/input/input2
[ 9.216870] Non-volatile memory driver v1.3
[ 9.223768] input: Lid Switch as /devices/LNXSYSTM:00/device:00/PNP0C0D:00/input/input3
[ 9.224566] ACPI: Lid Switch [LID]
[ 9.225313] input: Sleep Button as /devices/LNXSYSTM:00/device:00/PNP0C0E:00/input/input4
[ 9.225505] ACPI: Sleep Button [SLPB]
[ 9.225701] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input5
[ 9.225827] ACPI: Power Button [PWRF]
[ 9.254737] input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A03:00/device:03/LNXVIDEO:00/input/input6
[ 9.254932] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
[ 9.263455] ACPI: AC Adapter [AC] (on-line)
[ 9.278867] ACPI: acpi_idle registered with cpuidle
[ 9.279454] Marking TSC unstable due to TSC halts in idle
[ 9.281736] Switching to clocksource acpi_pm
[ 9.400778] NET: Registered protocol family 23
[ 9.426149] ACPI: Battery Slot [BAT0] (battery present)
[ 9.456563] parport_pc 00:0b: reported by Plug and Play ACPI
[ 9.456650] parport0: PC-style at 0x3bc, irq 7 [PCSPP,TRISTATE]
[ 9.511306] intel_rng: FWH not detected
[ 9.530499] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 9.660599] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 9.662015] nsc-ircc 00:0c: [io 0x02f8-0x02ff]
[ 9.662076] nsc-ircc 00:0c: [irq 3]
[ 9.662081] nsc-ircc 00:0c: [dma 1]
[ 9.662527] nsc-ircc 00:0c: activated
[ 9.662734] nsc-ircc, chip->init
[ 9.662774] nsc-ircc, Found chip at base=0x02e
[ 9.662831] nsc-ircc, driver loaded (Dag Brattli)
[ 9.693300] IrDA: Registered device irda0
[ 9.693349] nsc-ircc, Using dongle: IBM31T1100 or Temic TFDS6000/TFDS6500
[ 10.038113] i801_smbus 0000:00:1f.3: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 10.143228] Synaptics Touchpad, model: 1, fw: 5.9, id: 0x2c6ab1, caps: 0x884793/0x0/0x0
[ 10.143289] serio: Synaptics pass-through port at isa0060/serio1/input0
[ 10.187331] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input7
[ 10.547908] yenta_cardbus 0000:02:00.0: CardBus bridge found [1014:0512]
[ 10.547970] yenta_cardbus 0000:02:00.0: Using INTVAL to route CSC interrupts to PCI
[ 10.548040] yenta_cardbus 0000:02:00.0: Routing CardBus interrupts to PCI
[ 10.548082] yenta_cardbus 0000:02:00.0: TI: mfunc 0x01d21022, devctl 0x64
[ 10.661703] [drm] Initialized drm 1.1.0 20060810
[ 10.788969] yenta_cardbus 0000:02:00.0: ISA IRQ mask 0x0438, PCI irq 11
[ 10.789019] yenta_cardbus 0000:02:00.0: Socket status: 30000006
[ 10.789077] yenta_cardbus 0000:02:00.0: pcmcia: parent PCI bridge window: [io 0x4000-0x8fff]
[ 10.789129] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x4000-0x8fff: excluding 0x4000-0x40ff 0x4400-0x44ff 0x4800-0x48ff 0x4c00-0x4cff 0x8000-0x803f
[ 10.810092] yenta_cardbus 0000:02:00.0: pcmcia: parent PCI bridge window: [mem 0xc0200000-0xcfffffff]
[ 10.810145] pcmcia_socket pcmcia_socket0: cs: memory probe 0xc0200000-0xcfffffff: excluding 0xc0200000-0xc09fffff 0xc3a00000-0xcc1fffff 0xcfa00000-0xd01fffff
[ 10.810456] yenta_cardbus 0000:02:00.0: pcmcia: parent PCI bridge window: [mem 0xe8000000-0xefffffff pref]
[ 10.810509] pcmcia_socket pcmcia_socket0: cs: memory probe 0xe8000000-0xefffffff: excluding 0xe8000000-0xefffffff
[ 10.814307] yenta_cardbus 0000:02:00.1: CardBus bridge found [1014:0512]
[ 10.814363] yenta_cardbus 0000:02:00.1: Using INTVAL to route CSC interrupts to PCI
[ 10.814412] yenta_cardbus 0000:02:00.1: Routing CardBus interrupts to PCI
[ 10.814456] yenta_cardbus 0000:02:00.1: TI: mfunc 0x01d21022, devctl 0x64
[ 10.997001] cfg80211: World regulatory domain updated:
[ 10.997049] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
[ 10.997101] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[ 10.997151] cfg80211: (2457000 KHz - 2482000 KHz @ 20000 KHz), (300 mBi, 2000 mBm)
[ 10.997199] cfg80211: (2474000 KHz - 2494000 KHz @ 20000 KHz), (300 mBi, 2000 mBm)
[ 10.997247] cfg80211: (5170000 KHz - 5250000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[ 10.997297] cfg80211: (5735000 KHz - 5835000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[ 11.045093] yenta_cardbus 0000:02:00.1: ISA IRQ mask 0x0438, PCI irq 11
[ 11.045142] yenta_cardbus 0000:02:00.1: Socket status: 30000006
[ 11.045192] yenta_cardbus 0000:02:00.1: pcmcia: parent PCI bridge window: [io 0x4000-0x8fff]
[ 11.045244] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x4000-0x8fff: excluding 0x4000-0x40ff 0x4400-0x44ff 0x4800-0x48ff 0x4c00-0x4cff 0x8000-0x803f
[ 11.104971] yenta_cardbus 0000:02:00.1: pcmcia: parent PCI bridge window: [mem 0xc0200000-0xcfffffff]
[ 11.105023] pcmcia_socket pcmcia_socket1: cs: memory probe 0xc0200000-0xcfffffff: excluding 0xc0200000-0xc09fffff 0xc3a00000-0xcc1fffff 0xcfa00000-0xd01fffff
[ 11.105214] yenta_cardbus 0000:02:00.1: pcmcia: parent PCI bridge window: [mem 0xe8000000-0xefffffff pref]
[ 11.105265] pcmcia_socket pcmcia_socket1: cs: memory probe 0xe8000000-0xefffffff: excluding 0xe8000000-0xefffffff
[ 11.129465] thinkpad_acpi: ThinkPad ACPI Extras v0.24
[ 11.129511] thinkpad_acpi: http://ibm-acpi.sf.net/
[ 11.129548] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
[ 11.129587] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
[ 11.131459] thinkpad_acpi: detected a 8-level brightness capable ThinkPad
[ 11.145906] thinkpad_acpi: rfkill switch tpacpi_bluetooth_sw: radio is blocked
[ 11.146959] Registered led device: tpacpi::thinklight
[ 11.147287] Registered led device: tpacpi::power
[ 11.147594] Registered led device: tpacpi::standby
[ 11.152758] thinkpad_acpi: Console audio control enabled, mode: monitor (read only)
[ 11.156432] input: ThinkPad Extra Buttons as /devices/platform/thinkpad_acpi/input/input8
[ 11.882880] [drm] radeon kernel modesetting enabled.
[ 11.883063] radeon 0000:01:00.0: power state changed by ACPI to D0
[ 11.883104] radeon 0000:01:00.0: power state changed by ACPI to D0
[ 11.883151] radeon 0000:01:00.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[ 11.884326] [drm] initializing kernel modesetting (RV250 0x1002:0x4C66).
[ 11.884390] [drm] register mmio base: 0xC0100000
[ 11.884425] [drm] register mmio size: 65536
[ 11.884777] agpgart-intel 0000:00:00.0: AGP 2.0 bridge
[ 11.884827] agpgart-intel 0000:00:00.0: putting AGP V2 device into 4x mode
[ 11.884901] radeon 0000:01:00.0: putting AGP V2 device into 4x mode
[ 11.884964] radeon 0000:01:00.0: GTT: 256M 0xD0000000 - 0xDFFFFFFF
[ 11.885007] radeon 0000:01:00.0: VRAM: 128M 0x00000000E0000000 - 0x00000000E7FFFFFF (64M used)
[ 11.885062] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[ 11.885099] [drm] Driver supports precise vblank timestamp query.
[ 11.885148] [drm] radeon: irq initialized.
[ 11.885380] [drm] Detected VRAM RAM=128M, BAR=128M
[ 11.885422] [drm] RAM width 128bits DDR
[ 11.904735] [TTM] Zone kernel: Available graphics memory: 447358 kiB.
[ 11.904779] [TTM] Zone highmem: Available graphics memory: 516674 kiB.
[ 11.904816] [TTM] Initializing pool allocator.
[ 11.904884] [drm] radeon: 64M of VRAM memory ready
[ 11.904920] [drm] radeon: 256M of GTT memory ready.
[ 11.910053] radeon 0000:01:00.0: WB enabled
[ 11.910862] [drm] Loading R200 Microcode
[ 11.997425] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x100-0x3af: excluding 0x170-0x177 0x1f0-0x1f7 0x2f8-0x2ff 0x370-0x377
[ 11.998646] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x3e0-0x4ff: excluding 0x3f0-0x3ff
[ 11.999165] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x100-0x3af: excluding 0x170-0x177 0x1f0-0x1f7 0x2f8-0x2ff 0x370-0x377
[ 12.000434] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x3e0-0x4ff: excluding 0x3f0-0x3ff 0x4d0-0x4d7
[ 12.001023] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x820-0x8ff: clean.
[ 12.001483] pcmcia_socket pcmcia_socket0: cs: IO port probe 0xc00-0xcf7: clean.
[ 12.001977] pcmcia_socket pcmcia_socket0: cs: memory probe 0x0c0000-0x0fffff: excluding 0xc0000-0xd3fff 0xdc000-0xfffff
[ 12.002186] pcmcia_socket pcmcia_socket0: cs: memory probe 0xa0000000-0xa0ffffff: clean.
[ 12.002323] pcmcia_socket pcmcia_socket0: cs: memory probe 0x60000000-0x60ffffff: clean.
[ 12.002464] pcmcia_socket pcmcia_socket0: cs: IO port probe 0xa00-0xaff: clean.
[ 12.003085] 0x4d0-0x4d7
[ 12.003225] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x820-0x8ff: clean.
[ 12.003692] pcmcia_socket pcmcia_socket1: cs: IO port probe 0xc00-0xcf7: clean.
[ 12.004275] pcmcia_socket pcmcia_socket1: cs: memory probe 0x0c0000-0x0fffff: excluding 0xc0000-0xd3fff 0xdc000-0xfffff
[ 12.004472] pcmcia_socket pcmcia_socket1: cs: memory probe 0xa0000000-0xa0ffffff: clean.
[ 12.004608] pcmcia_socket pcmcia_socket1: cs: memory probe 0x60000000-0x60ffffff: clean.
[ 12.004750] pcmcia_socket pcmcia_socket1: cs: IO port probe 0xa00-0xaff: clean.
[ 12.186040] EXT4-fs (sda5): re-mounted. Opts: (null)
[ 12.275947] [drm] radeon: ring at 0x00000000D0001000
[ 12.276576] [drm] ring test succeeded in 1 usecs
[ 12.276904] [drm] radeon: ib pool ready.
[ 12.277066] [drm] ib test succeeded in 0 usecs
[ 12.278842] [drm] Panel ID String: SXGA+ Single (85MHz)
[ 12.278880] [drm] Panel Size 1400x1050
[ 12.289992] [drm] radeon legacy LVDS backlight initialized
[ 12.294552] [drm] No TV DAC info found in BIOS
[ 12.294765] [drm] Radeon Display Connectors
[ 12.294801] [drm] Connector 0:
[ 12.294833] [drm] VGA
[ 12.294867] [drm] DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60
[ 12.294903] [drm] Encoders:
[ 12.294936] [drm] CRT1: INTERNAL_DAC1
[ 12.294970] [drm] Connector 1:
[ 12.295003] [drm] DVI-D
[ 12.295035] [drm] HPD1
[ 12.295068] [drm] DDC: 0x64 0x64 0x64 0x64 0x64 0x64 0x64 0x64
[ 12.295104] [drm] Encoders:
[ 12.295136] [drm] DFP1: INTERNAL_TMDS1
[ 12.295170] [drm] Connector 2:
[ 12.295202] [drm] LVDS
[ 12.295234] [drm] Encoders:
[ 12.295266] [drm] LCD1: INTERNAL_LVDS
[ 12.295300] [drm] Connector 3:
[ 12.295332] [drm] S-video
[ 12.295364] [drm] Encoders:
[ 12.295396] [drm] TV1: INTERNAL_DAC2
[ 12.321728] Intel ICH 0000:00:1f.5: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 12.321815] Intel ICH 0000:00:1f.5: setting latency timer to 64
[ 12.335053] Adding 1052244k swap on /dev/sda2. Priority:0 extents:1 across:1052244k
[ 12.341750] [drm] radeon: power management initialized
[ 12.407449] [drm] fb mappable at 0xE0040000
[ 12.407489] [drm] vram apper at 0xE0000000
[ 12.407523] [drm] size 5914624
[ 12.407556] [drm] fb depth is 24
[ 12.407589] [drm] pitch is 5632
[ 12.477692] Console: switching to colour frame buffer device 175x65
[ 12.512192] fb0: radeondrmfb frame buffer device
[ 12.512358] drm: registered panic notifier
[ 12.512801] [drm] Initialized radeon 2.9.0 20080528 for 0000:01:00.0 on minor 0
[ 12.694651] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: mblk_io_submit
[ 13.055685] fuse init (API version 7.16)
[ 13.248040] intel8x0_measure_ac97_clock: measured 55076 usecs (2653 samples)
[ 13.248307] intel8x0: clocking to 48000
[ 13.251142] Intel ICH Modem 0000:00:1f.6: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 13.251492] Intel ICH Modem 0000:00:1f.6: setting latency timer to 64
[ 14.311743] ADDRCONF(NETDEV_UP): wlan0: link is not ready
[ 16.166964] IBM TrackPoint firmware: 0x0e, buttons: 3/3
[ 16.398732] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/serio2/input/input9
[ 16.894566] P-state transition latency capped at 20 uS
[ 16.913012] lp0: using parport0 (interrupt-driven).
[ 16.947001] ppdev: user-space parallel port driver
[ 18.646512] wlan0: authenticate with 00:04:0e:e4:00:3d (try 1)
[ 18.647976] wlan0: authenticated
[ 18.648696] wlan0: associate with 00:04:0e:e4:00:3d (try 1)
[ 18.652233] wlan0: RX AssocResp from 00:04:0e:e4:00:3d (capab=0x411 status=0 aid=1)
[ 18.652245] wlan0: associated
[ 18.654324] ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[ 19.804601] Bluetooth: Core ver 2.16
[ 19.820127] NET: Registered protocol family 31
[ 19.831781] Bluetooth: HCI device and connection manager initialized
[ 19.843547] Bluetooth: HCI socket layer initialized
[ 19.855154] Bluetooth: L2CAP socket layer initialized
[ 19.868218] Bluetooth: SCO socket layer initialized
[ 19.955741] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 19.967720] Bluetooth: BNEP filters: protocol multicast
[ 19.981038] padlock_aes: VIA PadLock not detected.
[ 20.003452] Bluetooth: RFCOMM TTY layer initialized
[ 20.015568] Bluetooth: RFCOMM socket layer initialized
[ 20.027151] Bluetooth: RFCOMM ver 1.11
[ 29.176042] wlan0: no IPv6 routers present
[ 38.017869] start_kdeinit (1362): /proc/1364/oom_adj is deprecated, please use /proc/1364/oom_score_adj instead.
[ 45.673651] EXT4-fs (sda5): re-mounted. Opts: commit=0
[ 45.867088] EXT4-fs (sda3): re-mounted. Opts: mblk_io_submit,commit=0
[ 85.734460] EXT3-fs: barriers not enabled
[ 85.746025] kjournald starting. Commit interval 5 seconds
[ 85.746388] EXT3-fs (sdb5): warning: maximal mount count reached, running e2fsck is recommended
[ 85.747182] EXT3-fs (sdb5): using internal journal
[ 85.747196] EXT3-fs (sdb5): mounted filesystem with ordered data mode
[ 157.340021] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
[ 157.340035] sending NMI to all CPUs:
[ 157.340049] NMI backtrace for cpu 0
[ 157.340056] Modules linked in: ext3 jbd rfcomm bnep aes_i586 bluetooth aes_generic binfmt_misc ppdev lp acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss radeon ttm snd_mixer_oss snd_pcm pcmcia drm_kms_helper thinkpad_acpi snd_seq_midi snd_rawmidi drm snd_seq_midi_event yenta_socket pcmcia_rsrc pcmcia_core joydev snd_seq i2c_i801 snd_timer i2c_algo_bit snd_seq_device snd shpchp nsc_ircc snd_page_alloc pci_hotplug soundcore psmouse parport_pc i2c_core rng_core battery irda processor ac tpm_tis video parport crc_ccitt nvram power_supply serio_raw button tpm tpm_bios pcspkr evdev arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore scsi_mod e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 157.340343]
[ 157.340353] Pid: 1849, comm: tar Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 157.340372] EIP: 0060:[<c101652d>] EFLAGS: 00200002 CPU: 0
[ 157.340393] EIP is at arch_trigger_all_cpu_backtrace+0x4b/0x62
[ 157.340404] EAX: 00418958 EBX: 00002710 ECX: c133faf5 EDX: 00000c00
[ 157.340414] ESI: c13ee580 EDI: c13ee600 EBP: da793a58 ESP: da793a54
[ 157.340425] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 157.340437] Process tar (pid: 1849, ti=da792000 task=dccd5c70 task.ti=da792000)
[ 157.340445] Stack:
[ 157.340451] c13ee580 da793a74 c1078dc1 c134e61e c134e6c2 00000000 00003a98 f5c03488
[ 157.340476] da793a88 c1078e36 00000000 00000000 dccd5c70 da793a94 c10791df 00000000
[ 157.340499] da793aa8 c1039c6c da793b34 a231f39c 00000024 da793ac0 c10509e9 f5c032c0
[ 157.340523] Call Trace:
[ 157.340543] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 157.340558] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 157.340572] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 157.340587] [<c1039c6c>] update_process_times+0x2d/0x58
[ 157.340602] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 157.340618] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 157.340630] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 157.340644] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 157.340659] [<c10158fa>] smp_apic_timer_interrupt+0x5b/0x6e
[ 157.340675] [<c12a6e31>] apic_timer_interrupt+0x31/0x38
[ 157.340690] [<c109007b>] ? perf_callchain+0xe9/0xfd
[ 157.340703] [<c10900e0>] ? perf_event_task_enable+0x51/0x6b
[ 157.340717] [<c100695e>] ? paravirt_read_tsc+0x6/0xd
[ 157.340730] [<c10070d0>] native_sched_clock+0x41/0x91
[ 157.340744] [<c1097424>] ? mempool_alloc+0x3a/0xd5
[ 157.340756] [<c1007129>] sched_clock+0x9/0xd
[ 157.340770] [<c113aaa9>] blk_rq_init+0x80/0x92
[ 157.340783] [<c113cc10>] get_request+0x185/0x240
[ 157.340796] [<c113ccec>] get_request_wait+0x21/0x122
[ 157.340810] [<c1147a56>] ? cfq_merge+0x54/0x76
[ 157.340821] [<c1147a02>] ? cfq_merge+0x0/0x76
[ 157.340837] [<c113868d>] ? elv_merge+0x8f/0x96
[ 157.340851] [<c113d6f2>] __make_request+0x104/0x1e6
[ 157.340865] [<c113c869>] generic_make_request.part.50+0x25a/0x2c5
[ 157.340881] [<c113c90c>] generic_make_request+0x38/0x3a
[ 157.340894] [<c113c9d1>] submit_bio+0xc3/0xdc
[ 157.340910] [<c109c49f>] ? ____pagevec_lru_add+0xf/0x11
[ 157.340926] [<c10e6612>] ? mpage_readpages+0x9d/0xdb
[ 157.340939] [<c10e663e>] mpage_readpages+0xc9/0xdb
[ 157.340954] [<c109a94a>] ? __alloc_pages_nodemask+0x5a6/0x5ef
[ 157.341015] [<f8647a89>] ? ext4_readpages+0x0/0x16 [ext4]
[ 157.341064] [<f8647a9d>] ext4_readpages+0x14/0x16 [ext4]
[ 157.341114] [<f864af41>] ? ext4_get_block+0x0/0x1b [ext4]
[ 157.341128] [<c109b932>] read_pages+0x2e/0xa7
[ 157.341143] [<c109ba8f>] __do_page_cache_readahead+0xe4/0xfb
[ 157.341158] [<c109bc92>] ra_submit+0x17/0x1c
[ 157.341172] [<c109be25>] ondemand_readahead+0x18e/0x196
[ 157.341187] [<c109be81>] page_cache_async_readahead+0x54/0x5f
[ 157.341202] [<c1096292>] do_generic_file_read.constprop.36+0xf0/0x370
[ 157.341218] [<c1096dad>] generic_file_aio_read+0x189/0x1bb
[ 157.341231] [<c113b212>] ? blk_finish_plug+0x17/0x2d
[ 157.341259] [<c10c26e6>] do_sync_read+0x8a/0xc5
[ 157.341277] [<c1116859>] ? security_file_permission+0x22/0x26
[ 157.341291] [<c10c29d8>] ? rw_verify_area+0xc3/0xe6
[ 157.341305] [<c10c2d2b>] vfs_read+0x82/0xd5
[ 157.341318] [<c10c265c>] ? do_sync_read+0x0/0xc5
[ 157.341331] [<c10c2db9>] sys_read+0x3b/0x5d
[ 157.341344] [<c12abb5f>] sysenter_do_call+0x12/0x28
[ 157.341353] Code: f5 fa 33 c1 a3 d0 49 41 c1 e8 da 5a 28 00 8b 15 10 87 3e c1 b8 02 00 00 00 ff 52 7c 59 83 3d d0 49 41 c1 00 74 0d b8 58 89 41 00 <e8> 6f af 13 00 4b 75 ea 3e 80 25 d8 31 48 c1 fe 8b 5d fc c9 c3
[ 157.341477] Call Trace:
[ 157.341490] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 157.341504] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 157.341519] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 157.341532] [<c1039c6c>] update_process_times+0x2d/0x58
[ 157.341544] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 157.341558] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 157.341570] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 157.341584] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 157.341599] [<c10158fa>] smp_apic_timer_interrupt+0x5b/0x6e
[ 157.341612] [<c12a6e31>] apic_timer_interrupt+0x31/0x38
[ 157.341626] [<c109007b>] ? perf_callchain+0xe9/0xfd
[ 157.341638] [<c10900e0>] ? perf_event_task_enable+0x51/0x6b
[ 157.341651] [<c100695e>] ? paravirt_read_tsc+0x6/0xd
[ 157.341663] [<c10070d0>] native_sched_clock+0x41/0x91
[ 157.341676] [<c1097424>] ? mempool_alloc+0x3a/0xd5
[ 157.341688] [<c1007129>] sched_clock+0x9/0xd
[ 157.341699] [<c113aaa9>] blk_rq_init+0x80/0x92
[ 157.341712] [<c113cc10>] get_request+0x185/0x240
[ 157.341725] [<c113ccec>] get_request_wait+0x21/0x122
[ 157.341737] [<c1147a56>] ? cfq_merge+0x54/0x76
[ 157.341749] [<c1147a02>] ? cfq_merge+0x0/0x76
[ 157.341762] [<c113868d>] ? elv_merge+0x8f/0x96
[ 157.341775] [<c113d6f2>] __make_request+0x104/0x1e6
[ 157.341789] [<c113c869>] generic_make_request.part.50+0x25a/0x2c5
[ 157.341805] [<c113c90c>] generic_make_request+0x38/0x3a
[ 157.341818] [<c113c9d1>] submit_bio+0xc3/0xdc
[ 157.341831] [<c109c49f>] ? ____pagevec_lru_add+0xf/0x11
[ 157.341845] [<c10e6612>] ? mpage_readpages+0x9d/0xdb
[ 157.341859] [<c10e663e>] mpage_readpages+0xc9/0xdb
[ 157.341873] [<c109a94a>] ? __alloc_pages_nodemask+0x5a6/0x5ef
[ 157.341927] [<f8647a89>] ? ext4_readpages+0x0/0x16 [ext4]
[ 157.341975] [<f8647a9d>] ext4_readpages+0x14/0x16 [ext4]
[ 157.342025] [<f864af41>] ? ext4_get_block+0x0/0x1b [ext4]
[ 157.342039] [<c109b932>] read_pages+0x2e/0xa7
[ 157.342054] [<c109ba8f>] __do_page_cache_readahead+0xe4/0xfb
[ 157.342068] [<c109bc92>] ra_submit+0x17/0x1c
[ 157.342082] [<c109be25>] ondemand_readahead+0x18e/0x196
[ 157.342097] [<c109be81>] page_cache_async_readahead+0x54/0x5f
[ 157.342111] [<c1096292>] do_generic_file_read.constprop.36+0xf0/0x370
[ 157.342127] [<c1096dad>] generic_file_aio_read+0x189/0x1bb
[ 157.342140] [<c113b212>] ? blk_finish_plug+0x17/0x2d
[ 157.342164] [<c10c26e6>] do_sync_read+0x8a/0xc5
[ 157.342180] [<c1116859>] ? security_file_permission+0x22/0x26
[ 157.342194] [<c10c29d8>] ? rw_verify_area+0xc3/0xe6
[ 157.342207] [<c10c2d2b>] vfs_read+0x82/0xd5
[ 157.342220] [<c10c265c>] ? do_sync_read+0x0/0xc5
[ 157.342233] [<c10c2db9>] sys_read+0x3b/0x5d
[ 157.342245] [<c12abb5f>] sysenter_do_call+0x12/0x28
[ 337.460033] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
[ 337.460046] sending NMI to all CPUs:
[ 337.460059] NMI backtrace for cpu 0
[ 337.460066] Modules linked in: ext3 jbd rfcomm bnep aes_i586 bluetooth aes_generic binfmt_misc ppdev lp acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss radeon ttm snd_mixer_oss snd_pcm pcmcia drm_kms_helper thinkpad_acpi snd_seq_midi snd_rawmidi drm snd_seq_midi_event yenta_socket pcmcia_rsrc pcmcia_core joydev snd_seq i2c_i801 snd_timer i2c_algo_bit snd_seq_device snd shpchp nsc_ircc snd_page_alloc pci_hotplug soundcore psmouse parport_pc i2c_core rng_core battery irda processor ac tpm_tis video parport crc_ccitt nvram power_supply serio_raw button tpm tpm_bios pcspkr evdev arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore scsi_mod e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 337.460350]
[ 337.460361] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 337.460380] EIP: 0060:[<c1151499>] EFLAGS: 00000807 CPU: 0
[ 337.460396] EIP is at __delay+0x3/0xb
[ 337.460406] EAX: 00090d34 EBX: 00002710 ECX: c133faf5 EDX: 00090d33
[ 337.460416] ESI: c13ee580 EDI: c13ee600 EBP: f5819ea4 ESP: f5819ea4
[ 337.460427] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 337.460439] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
[ 337.460447] Stack:
[ 337.460453] f5819eac c11514bb f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e
[ 337.460477] c134e6c2 00000000 0000ea7e f5c03488 f5819ee8 c1078e36 00000000 00000000
[ 337.460501] c13e3fa0 f5819ef4 c10791df 00000000 f5819f08 c1039c6c c13b7ef8 922f41d8
[ 337.460524] Call Trace:
[ 337.460539] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 337.460558] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 337.460575] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 337.460589] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 337.460604] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 337.460619] [<c1039c6c>] update_process_times+0x2d/0x58
[ 337.460634] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 337.460649] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 337.460661] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 337.460676] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 337.460694] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 337.460708] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 337.460724] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 337.460739] [<c1004215>] timer_interrupt+0x15/0x1c
[ 337.460752] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 337.460766] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 337.460778] [<c10754b9>] handle_irq_event+0x36/0x51
[ 337.460791] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 337.460804] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 337.460812] <IRQ>
[ 337.460824] [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 337.460839] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 337.460854] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 337.460886] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 337.460910] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 337.460935] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 337.460951] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 337.460965] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 337.460981] [<c128c26c>] ? rest_init+0x58/0x5a
[ 337.460994] [<c1418722>] ? start_kernel+0x315/0x31a
[ 337.461008] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 337.461015] Code: 89 e5 85 c0 74 1f eb 0a 8d 76 00 8d bc 27 00 00 00 00 eb 0e 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 48 75 fd 48 5d c3 55 89 e5 <ff> 15 f8 f2 3f c1 5d c3 55 89 e5 64 8b 15 dc 62 47 c1 c1 e0 02
[ 337.461140] Call Trace:
[ 337.461151] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 337.461166] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 337.461180] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 337.461194] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 337.461208] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 337.461221] [<c1039c6c>] update_process_times+0x2d/0x58
[ 337.461234] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 337.461248] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 337.461260] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 337.461273] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 337.461289] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 337.461304] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 337.461318] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 337.461332] [<c1004215>] timer_interrupt+0x15/0x1c
[ 337.461344] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 337.461358] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 337.461370] [<c10754b9>] handle_irq_event+0x36/0x51
[ 337.461383] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 337.461396] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 337.461404] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 337.461425] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 337.461439] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 337.461465] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 337.461488] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 337.461512] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 337.461527] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 337.461540] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 337.461554] [<c128c26c>] ? rest_init+0x58/0x5a
[ 337.461566] [<c1418722>] ? start_kernel+0x315/0x31a
[ 337.461579] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 517.580032] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
[ 517.580046] sending NMI to all CPUs:
[ 517.580059] NMI backtrace for cpu 0
[ 517.580066] Modules linked in: ext3 jbd rfcomm bnep aes_i586 bluetooth aes_generic binfmt_misc ppdev lp acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss radeon ttm snd_mixer_oss snd_pcm pcmcia drm_kms_helper thinkpad_acpi snd_seq_midi snd_rawmidi drm snd_seq_midi_event yenta_socket pcmcia_rsrc pcmcia_core joydev snd_seq i2c_i801 snd_timer i2c_algo_bit snd_seq_device snd shpchp nsc_ircc snd_page_alloc pci_hotplug soundcore psmouse parport_pc i2c_core rng_core battery irda processor ac tpm_tis video parport crc_ccitt nvram power_supply serio_raw button tpm tpm_bios pcspkr evdev arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore scsi_mod e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 517.580351]
[ 517.580361] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 517.580380] EIP: 0060:[<c11514b3>] EFLAGS: 00000807 CPU: 0
[ 517.580397] EIP is at __const_udelay+0x12/0x1c
[ 517.580408] EAX: 282b1080 EBX: 00002710 ECX: c133faf5 EDX: 00090d33
[ 517.580418] ESI: c13ee580 EDI: c13ee600 EBP: f5819eac ESP: f5819eac
[ 517.580429] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 517.580441] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
[ 517.580450] Stack:
[ 517.580456] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
[ 517.580480] 00019a64 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
[ 517.580503] c10791df 00000000 f5819f08 c1039c6c c13b7ef8 822c563c 00000078 f5819f20
[ 517.580527] Call Trace:
[ 517.580547] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 517.580565] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 517.580579] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 517.580593] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 517.580608] [<c1039c6c>] update_process_times+0x2d/0x58
[ 517.580622] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 517.580638] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 517.580650] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 517.580664] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 517.580682] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 517.580697] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 517.580712] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 517.580727] [<c1004215>] timer_interrupt+0x15/0x1c
[ 517.580740] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 517.580754] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 517.580766] [<c10754b9>] handle_irq_event+0x36/0x51
[ 517.580779] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 517.580792] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 517.580800] <IRQ>
[ 517.580812] [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 517.580826] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 517.580841] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 517.580873] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 517.580898] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 517.580922] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 517.580939] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 517.580953] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 517.580969] [<c128c26c>] ? rest_init+0x58/0x5a
[ 517.580982] [<c1418722>] ? start_kernel+0x315/0x31a
[ 517.580995] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 517.581002] Code: 00 8d bc 27 00 00 00 00 48 75 fd 48 5d c3 55 89 e5 ff 15 f8 f2 3f c1 5d c3 55 89 e5 64 8b 15 dc 62 47 c1 c1 e0 02 6b d2 3e f7 e2 <8d> 42 01 e8 db ff ff ff 5d c3 69 c0 c7 10 00 00 55 89 e5 e8 d6
[ 517.581127] Call Trace:
[ 517.581140] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 517.581155] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 517.581169] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 517.581183] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 517.581196] [<c1039c6c>] update_process_times+0x2d/0x58
[ 517.581209] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 517.581223] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 517.581235] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 517.581249] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 517.581265] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 517.581280] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 517.581294] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 517.581307] [<c1004215>] timer_interrupt+0x15/0x1c
[ 517.581320] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 517.581333] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 517.581346] [<c10754b9>] handle_irq_event+0x36/0x51
[ 517.581359] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 517.581371] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 517.581379] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 517.581401] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 517.581415] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 517.581440] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 517.581463] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 517.581487] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 517.581501] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 517.581515] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 517.581528] [<c128c26c>] ? rest_init+0x58/0x5a
[ 517.581541] [<c1418722>] ? start_kernel+0x315/0x31a
[ 517.581554] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 697.700033] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
[ 697.700046] sending NMI to all CPUs:
[ 697.700059] NMI backtrace for cpu 0
[ 697.700066] Modules linked in: ext3 jbd rfcomm bnep aes_i586 bluetooth aes_generic binfmt_misc ppdev lp acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss radeon ttm snd_mixer_oss snd_pcm pcmcia drm_kms_helper thinkpad_acpi snd_seq_midi snd_rawmidi drm snd_seq_midi_event yenta_socket pcmcia_rsrc pcmcia_core joydev snd_seq i2c_i801 snd_timer i2c_algo_bit snd_seq_device snd shpchp nsc_ircc snd_page_alloc pci_hotplug soundcore psmouse parport_pc i2c_core rng_core battery irda processor ac tpm_tis video parport crc_ccitt nvram power_supply serio_raw button tpm tpm_bios pcspkr evdev arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore scsi_mod e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 697.700351]
[ 697.700361] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 697.700380] EIP: 0060:[<c11514db>] EFLAGS: 00000807 CPU: 0
[ 697.700397] EIP is at delay_tsc+0x1/0x5e
[ 697.700407] EAX: 00090d34 EBX: 00002710 ECX: c133faf5 EDX: 00090d33
[ 697.700418] ESI: c13ee580 EDI: c13ee600 EBP: f5819ea4 ESP: f5819e9c
[ 697.700428] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 697.700440] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
[ 697.700449] Stack:
[ 697.700455] f5819ea4 c115149f f5819eac c11514bb f5819eb8 c1016532 c13ee580 f5819ed4
[ 697.700479] c1078dc1 c134e61e c134e6c2 00000000 00024a4a f5c03488 f5819ee8 c1078e36
[ 697.700502] 00000000 00000000 c13e3fa0 f5819ef4 c10791df 00000000 f5819f08 c1039c6c
[ 697.700526] Call Trace:
[ 697.700540] [<c115149f>] ? __delay+0x9/0xb
[ 697.700552] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 697.700570] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 697.700588] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 697.700602] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 697.700616] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 697.700631] [<c1039c6c>] update_process_times+0x2d/0x58
[ 697.700646] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 697.700661] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 697.700673] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 697.700687] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 697.700705] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 697.700720] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 697.700735] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 697.700750] [<c1004215>] timer_interrupt+0x15/0x1c
[ 697.700763] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 697.700777] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 697.700789] [<c10754b9>] handle_irq_event+0x36/0x51
[ 697.700802] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 697.700815] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 697.700823] <IRQ>
[ 697.700835] [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 697.700849] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 697.700864] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 697.700896] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 697.700920] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 697.700944] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 697.700961] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 697.700975] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 697.700991] [<c128c26c>] ? rest_init+0x58/0x5a
[ 697.701004] [<c1418722>] ? start_kernel+0x315/0x31a
[ 697.701017] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 697.701024] Code: 3e f7 e2 8d 42 01 e8 db ff ff ff 5d c3 69 c0 c7 10 00 00 55 89 e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55
[ 697.701109] e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8
[ 697.701153] Call Trace:
[ 697.701164] [<c115149f>] ? __delay+0x9/0xb
[ 697.701177] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 697.701191] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 697.701205] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 697.701219] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 697.701233] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 697.701246] [<c1039c6c>] update_process_times+0x2d/0x58
[ 697.701259] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 697.701272] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 697.701284] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 697.701298] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 697.701314] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 697.701329] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 697.701343] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 697.701356] [<c1004215>] timer_interrupt+0x15/0x1c
[ 697.701369] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 697.701382] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 697.701395] [<c10754b9>] handle_irq_event+0x36/0x51
[ 697.701407] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 697.701420] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 697.701428] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 697.701449] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 697.701463] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 697.701489] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 697.701512] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 697.701536] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 697.701551] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 697.701564] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 697.701578] [<c128c26c>] ? rest_init+0x58/0x5a
[ 697.701590] [<c1418722>] ? start_kernel+0x315/0x31a
[ 697.701603] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 877.820034] INFO: rcu_sched_state detected stall on CPU 0 (t=195120 jiffies)
[ 877.820046] sending NMI to all CPUs:
[ 877.820059] NMI backtrace for cpu 0
[ 877.820066] Modules linked in: ext3 jbd rfcomm bnep aes_i586 bluetooth aes_generic binfmt_misc ppdev lp acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss radeon ttm snd_mixer_oss snd_pcm pcmcia drm_kms_helper thinkpad_acpi snd_seq_midi snd_rawmidi drm snd_seq_midi_event yenta_socket pcmcia_rsrc pcmcia_core joydev snd_seq i2c_i801 snd_timer i2c_algo_bit snd_seq_device snd shpchp nsc_ircc snd_page_alloc pci_hotplug soundcore psmouse parport_pc i2c_core rng_core battery irda processor ac tpm_tis video parport crc_ccitt nvram power_supply serio_raw button tpm tpm_bios pcspkr evdev arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore scsi_mod e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 877.820350]
[ 877.820360] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 877.820379] EIP: 0060:[<c11514ab>] EFLAGS: 00000002 CPU: 0
[ 877.820396] EIP is at __const_udelay+0xa/0x1c
[ 877.820407] EAX: 00418958 EBX: 00002710 ECX: c133faf5 EDX: 00247f7a
[ 877.820417] ESI: c13ee580 EDI: c13ee600 EBP: f5819eac ESP: f5819eac
[ 877.820428] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 877.820440] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
[ 877.820448] Stack:
[ 877.820454] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
[ 877.820479] 0002fa30 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
[ 877.820502] c10791df 00000000 f5819f08 c1039c6c c13b7ef8 6226858f 000000cc f5819f20
[ 877.820526] Call Trace:
[ 877.820546] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 877.820563] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 877.820577] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 877.820592] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 877.820607] [<c1039c6c>] update_process_times+0x2d/0x58
[ 877.820621] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 877.820637] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 877.820649] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 877.820663] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 877.820681] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 877.820695] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 877.820711] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 877.820726] [<c1004215>] timer_interrupt+0x15/0x1c
[ 877.820738] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 877.820753] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 877.820765] [<c10754b9>] handle_irq_event+0x36/0x51
[ 877.820778] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 877.820791] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 877.820799] <IRQ>
[ 877.820811] [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 877.820825] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 877.820840] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 877.820872] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 877.820896] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 877.820921] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 877.820938] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 877.820952] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 877.820968] [<c128c26c>] ? rest_init+0x58/0x5a
[ 877.820981] [<c1418722>] ? start_kernel+0x315/0x31a
[ 877.820994] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 877.821002] Code: eb 0e 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 48 75 fd 48 5d c3 55 89 e5 ff 15 f8 f2 3f c1 5d c3 55 89 e5 64 8b 15 dc 62 47 c1 <c1> e0 02 6b d2 3e f7 e2 8d 42 01 e8 db ff ff ff 5d c3 69 c0 c7
[ 877.821126] Call Trace:
[ 877.821139] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 877.821154] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 877.821168] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 877.821182] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 877.821195] [<c1039c6c>] update_process_times+0x2d/0x58
[ 877.821208] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 877.821222] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 877.821233] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 877.821247] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 877.821263] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 877.821278] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 877.821292] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 877.821305] [<c1004215>] timer_interrupt+0x15/0x1c
[ 877.821318] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 877.821332] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 877.821344] [<c10754b9>] handle_irq_event+0x36/0x51
[ 877.821357] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 877.821370] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 877.821378] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 877.821399] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 877.821413] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 877.821438] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 877.821461] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 877.821485] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 877.821500] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 877.821513] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 877.821526] [<c128c26c>] ? rest_init+0x58/0x5a
[ 877.821539] [<c1418722>] ? start_kernel+0x315/0x31a
[ 877.821552] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 1057.940032] INFO: rcu_sched_state detected stall on CPU 0 (t=240150 jiffies)
[ 1057.940044] sending NMI to all CPUs:
[ 1057.940057] NMI backtrace for cpu 0
[ 1057.940064] Modules linked in: ext3 jbd rfcomm bnep aes_i586 bluetooth aes_generic binfmt_misc ppdev lp acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss radeon ttm snd_mixer_oss snd_pcm pcmcia drm_kms_helper thinkpad_acpi snd_seq_midi snd_rawmidi drm snd_seq_midi_event yenta_socket pcmcia_rsrc pcmcia_core joydev snd_seq i2c_i801 snd_timer i2c_algo_bit snd_seq_device snd shpchp nsc_ircc snd_page_alloc pci_hotplug soundcore psmouse parport_pc i2c_core rng_core battery irda processor ac tpm_tis video parport crc_ccitt nvram power_supply serio_raw button tpm tpm_bios pcspkr evdev arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore scsi_mod e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 1057.940348]
[ 1057.940358] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 1057.940377] EIP: 0060:[<c11514da>] EFLAGS: 00000807 CPU: 0
[ 1057.940394] EIP is at delay_tsc+0x0/0x5e
[ 1057.940404] EAX: 00090d34 EBX: 00002710 ECX: c133faf5 EDX: 00090d33
[ 1057.940415] ESI: c13ee580 EDI: c13ee600 EBP: f5819ea4 ESP: f5819ea0
[ 1057.940425] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 1057.940438] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
[ 1057.940446] Stack:
[ 1057.940452] c115149f f5819eac c11514bb f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1
[ 1057.940476] c134e61e c134e6c2 00000000 0003aa16 f5c03488 f5819ee8 c1078e36 00000000
[ 1057.940500] 00000000 c13e3fa0 f5819ef4 c10791df 00000000 f5819f08 c1039c6c c13b7ef8
[ 1057.940523] Call Trace:
[ 1057.940537] [<c115149f>] ? __delay+0x9/0xb
[ 1057.940550] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 1057.940568] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 1057.940586] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 1057.940600] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 1057.940614] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 1057.940630] [<c1039c6c>] update_process_times+0x2d/0x58
[ 1057.940644] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 1057.940660] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 1057.940671] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 1057.940686] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 1057.940703] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 1057.940718] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 1057.940733] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 1057.940749] [<c1004215>] timer_interrupt+0x15/0x1c
[ 1057.940761] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 1057.940775] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 1057.940788] [<c10754b9>] handle_irq_event+0x36/0x51
[ 1057.940801] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 1057.940813] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 1057.940822] <IRQ>
[ 1057.940834] [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 1057.940848] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 1057.940863] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 1057.940894] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 1057.940919] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 1057.940943] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 1057.940960] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 1057.940974] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 1057.940990] [<c128c26c>] ? rest_init+0x58/0x5a
[ 1057.941003] [<c1418722>] ? start_kernel+0x315/0x31a
[ 1057.941016] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 1057.941024] Code: d2 3e f7 e2 8d 42 01 e8 db ff ff ff 5d c3 69 c0 c7 10 00 00 55 89 e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3
[ 1057.941109] 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae
[ 1057.941149] Call Trace:
[ 1057.941160] [<c115149f>] ? __delay+0x9/0xb
[ 1057.941172] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 1057.941186] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 1057.941201] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 1057.941215] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 1057.941228] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 1057.941241] [<c1039c6c>] update_process_times+0x2d/0x58
[ 1057.941254] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 1057.941268] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 1057.941279] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 1057.941293] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 1057.941309] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 1057.941324] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 1057.941338] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 1057.941351] [<c1004215>] timer_interrupt+0x15/0x1c
[ 1057.941364] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 1057.941377] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 1057.941389] [<c10754b9>] handle_irq_event+0x36/0x51
[ 1057.941402] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 1057.941415] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 1057.941423] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 1057.941444] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 1057.941458] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 1057.941483] [<f88fe30f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 1057.941506] [<f88feef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 1057.941530] [<f88fec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 1057.941545] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 1057.941559] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 1057.941572] [<c128c26c>] ? rest_init+0x58/0x5a
[ 1057.941585] [<c1418722>] ? start_kernel+0x315/0x31a
[ 1057.941598] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 1238.060029] INFO: rcu_sched_state detected stall on CPU 0 (t=285180 jiffies)
[ 1238.060029] sending NMI to all CPUs:
[ 1238.060029] NMI backtrace for cpu 0
[ 1238.060029] Modules linked in: ext3 jbd rfcomm bnep aes_i586 bluetooth aes_generic binfmt_misc ppdev lp acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss radeon ttm snd_mixer_oss snd_pcm pcmcia drm_kms_helper thinkpad_acpi snd_seq_midi snd_rawmidi drm snd_seq_midi_event yenta_socket pcmcia_rsrc pcmcia_core joydev snd_seq i2c_i801 snd_timer i2c_algo_bit snd_seq_device snd shpchp nsc_ircc snd_page_alloc pci_hotplug soundcore psmouse parport_pc i2c_core rng_core battery irda processor ac tpm_tis video parport crc_ccitt nvram power_supply serio_raw button tpm tpm_bios pcspkr evdev arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore scsi_mod e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 1238.060029]
[ 1238.060029] Pid: 1787, comm: firefox-bin Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 1238.060029] EIP: 0060:[<c101652d>] EFLAGS: 00200002 CPU: 0
[ 1238.060029] EIP is at arch_trigger_all_cpu_backtrace+0x4b/0x62
[ 1238.060029] EAX: 00418958 EBX: 00002710 ECX: c133faf5 EDX: 00000c00
[ 1238.060029] ESI: c13ee580 EDI: c13ee600 EBP: eda3fed8 ESP: eda3fed4
[ 1238.060029] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 1238.060029] Process firefox-bin (pid: 1787, ti=eda3e000 task=eda81450 task.ti=eda3e000)
[ 1238.060029] Stack:
[ 1238.060029] c13ee580 eda3fef4 c1078dc1 c134e61e c134e6c2 00000000 000459fc f5c03488
[ 1238.060029] eda3ff08 c1078e36 00000000 00000000 eda81450 eda3ff14 c10791df 00000001
[ 1238.060029] eda3ff28 c1039c6c eda3ffb4 42208b6d 00000120 eda3ff40 c10509e9 f5c032c0
[ 1238.060029] Call Trace:
[ 1238.060029] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 1238.060029] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 1238.060029] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 1238.060029] [<c1039c6c>] update_process_times+0x2d/0x58
[ 1238.060029] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 1238.060029] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 1238.060029] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 1238.060029] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 1238.060029] [<c10158fa>] smp_apic_timer_interrupt+0x5b/0x6e
[ 1238.060029] [<c12a6e31>] apic_timer_interrupt+0x31/0x38
[ 1238.060029] [<c12a0000>] ? load_elf_interp+0x36/0x2db
[ 1238.060029] Code: f5 fa 33 c1 a3 d0 49 41 c1 e8 da 5a 28 00 8b 15 10 87 3e c1 b8 02 00 00 00 ff 52 7c 59 83 3d d0 49 41 c1 00 74 0d b8 58 89 41 00 <e8> 6f af 13 00 4b 75 ea 3e 80 25 d8 31 48 c1 fe 8b 5d fc c9 c3
[ 1238.060029] Call Trace:
[ 1238.060029] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 1238.060029] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 1238.060029] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 1238.060029] [<c1039c6c>] update_process_times+0x2d/0x58
[ 1238.060029] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 1238.060029] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 1238.060029] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 1238.060029] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 1238.060029] [<c10158fa>] smp_apic_timer_interrupt+0x5b/0x6e
[ 1238.060029] [<c12a6e31>] apic_timer_interrupt+0x31/0x38
[ 1238.060029] [<c12a0000>] ? load_elf_interp+0x36/0x2db
[-- Attachment #3: config-2.6.38-next20110325-2-686-iniza --]
[-- Type: application/octet-stream, Size: 128351 bytes --]
#
# Automatically generated make config: don't edit
# Linux/x86 2.6.38 Kernel Configuration
# Fri Mar 25 08:54:44 2011
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf32-i386"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_GPIO=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-ecx -fcall-saved-edx"
CONFIG_KTIME_SCALAR=y
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
# CONFIG_FHANDLE is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_HAVE_GENERIC_HARDIRQS=y
#
# IRQ subsystem
#
CONFIG_GENERIC_HARDIRQS=y
CONFIG_HAVE_SPARSE_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y
# CONFIG_SPARSE_IRQ is not set
#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_DEBUG=y
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
# CONFIG_RESOURCE_COUNTERS is not set
# CONFIG_CGROUP_PERF is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_BLK_CGROUP=y
CONFIG_DEBUG_BLK_CGROUP=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EXPERT is not set
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y
#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_PERF_COUNTERS is not set
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
# CONFIG_OPROFILE_EVENT_MULTIPLEX is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
# CONFIG_JUMP_LABEL is not set
CONFIG_OPTPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_LBDAF=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_INTEGRITY=y
# CONFIG_BLK_DEV_THROTTLING is not set
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
# CONFIG_INLINE_SPIN_TRYLOCK is not set
# CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK is not set
# CONFIG_INLINE_SPIN_LOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK_IRQ is not set
# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
CONFIG_INLINE_SPIN_UNLOCK=y
# CONFIG_INLINE_SPIN_UNLOCK_BH is not set
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_READ_TRYLOCK is not set
# CONFIG_INLINE_READ_LOCK is not set
# CONFIG_INLINE_READ_LOCK_BH is not set
# CONFIG_INLINE_READ_LOCK_IRQ is not set
# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
CONFIG_INLINE_READ_UNLOCK=y
# CONFIG_INLINE_READ_UNLOCK_BH is not set
CONFIG_INLINE_READ_UNLOCK_IRQ=y
# CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_WRITE_TRYLOCK is not set
# CONFIG_INLINE_WRITE_LOCK is not set
# CONFIG_INLINE_WRITE_LOCK_BH is not set
# CONFIG_INLINE_WRITE_LOCK_IRQ is not set
# CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
CONFIG_INLINE_WRITE_UNLOCK=y
# CONFIG_INLINE_WRITE_UNLOCK_BH is not set
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
# CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_FREEZER=y
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_BIGSMP=y
# CONFIG_X86_EXTENDED_PLATFORM is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_X86_32_IRIS is not set
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_PARAVIRT_GUEST=y
# CONFIG_XEN_PRIVILEGED_GUEST is not set
CONFIG_KVM_CLOCK=y
CONFIG_KVM_GUEST=y
CONFIG_LGUEST_GUEST=y
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_SPINLOCKS is not set
CONFIG_PARAVIRT_CLOCK=y
# CONFIG_PARAVIRT_DEBUG is not set
CONFIG_NO_BOOTMEM=y
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
CONFIG_M686=y
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_X86_GENERIC=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_CMPXCHG_LOCAL=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
# CONFIG_X86_PPRO_FENCE is not set
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=5
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_CYRIX_32=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_CPU_SUP_TRANSMETA_32=y
CONFIG_CPU_SUP_UMC_32=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
CONFIG_IOMMU_API=y
CONFIG_NR_CPUS=32
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
# CONFIG_IRQ_TIME_ACCOUNTING is not set
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
# CONFIG_X86_ANCIENT_MCE is not set
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_MCE_INJECT=m
CONFIG_X86_THERMAL_VECTOR=y
CONFIG_VM86=y
CONFIG_TOSHIBA=m
CONFIG_I8K=m
CONFIG_X86_REBOOTFIXUPS=y
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
# CONFIG_ARCH_PHYS_ADDR_T_64BIT is not set
# CONFIG_ARCH_DMA_ADDR_T_64BIT is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
# CONFIG_PHYS_ADDR_T_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
# CONFIG_TRANSPARENT_HUGEPAGE is not set
CONFIG_CLEANCACHE=y
CONFIG_HIGHPTE=y
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW=64
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_EFI=y
CONFIG_SECCOMP=y
CONFIG_CC_STACKPROTECTOR=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
# CONFIG_CRASH_DUMP is not set
# CONFIG_KEXEC_JUMP is not set
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_X86_NEED_RELOCS=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
#
# Power management and ACPI options
#
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
# CONFIG_PM_VERBOSE is not set
CONFIG_PM_ADVANCED_DEBUG=y
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_CAN_PM_TRACE=y
# CONFIG_PM_TRACE_RTC is not set
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_POWER_METER=m
# CONFIG_ACPI_EC_DEBUGFS is not set
# CONFIG_ACPI_PROC_EVENT is not set
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=m
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=m
# CONFIG_ACPI_IPMI is not set
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=m
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_PCI_SLOT=m
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=m
CONFIG_ACPI_SBS=m
CONFIG_ACPI_HED=m
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=m
# CONFIG_ACPI_APEI_PCIEAER is not set
# CONFIG_ACPI_APEI_EINJ is not set
# CONFIG_ACPI_APEI_ERST_DEBUG is not set
CONFIG_SFI=y
CONFIG_X86_APM_BOOT=y
CONFIG_APM=m
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
# CONFIG_APM_DO_ENABLE is not set
# CONFIG_APM_CPU_IDLE is not set
# CONFIG_APM_DISPLAY_BLANK is not set
# CONFIG_APM_ALLOW_INTS is not set
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=m
# CONFIG_CPU_FREQ_STAT_DETAILS is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m
#
# CPUFreq processor drivers
#
CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K6=m
CONFIG_X86_POWERNOW_K7=m
CONFIG_X86_POWERNOW_K7_ACPI=y
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_GX_SUSPMOD=m
CONFIG_X86_SPEEDSTEP_CENTRINO=m
CONFIG_X86_SPEEDSTEP_CENTRINO_TABLE=y
CONFIG_X86_SPEEDSTEP_ICH=m
CONFIG_X86_SPEEDSTEP_SMI=m
CONFIG_X86_P4_CLOCKMOD=m
CONFIG_X86_CPUFREQ_NFORCE2=m
CONFIG_X86_LONGRUN=m
CONFIG_X86_LONGHAUL=m
# CONFIG_X86_E_POWERSAVER is not set
#
# shared options
#
CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_X86_SPEEDSTEP_RELAXED_CAP_CHECK=y
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_INTEL_IDLE is not set
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
CONFIG_DMAR=y
# CONFIG_DMAR_DEFAULT_ON is not set
CONFIG_DMAR_FLOPPY_WA=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=m
CONFIG_PCIEAER=y
# CONFIG_PCIE_ECRC is not set
CONFIG_PCIEAER_INJECT=m
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_PCIE_PME=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_DEBUG is not set
CONFIG_PCI_STUB=m
CONFIG_HT_IRQ=y
CONFIG_PCI_IOV=y
CONFIG_PCI_IOAPIC=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
CONFIG_SCx200=m
CONFIG_SCx200HR_TIMER=m
# CONFIG_OLPC is not set
CONFIG_AMD_NB=y
CONFIG_PCCARD=m
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y
#
# PC-card bridges
#
CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_I82365=m
CONFIG_TCIC=m
CONFIG_PCMCIA_PROBE=y
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_COMPAQ=m
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not set
CONFIG_HOTPLUG_PCI_IBM=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_SHPC=m
# CONFIG_RAPIDIO is not set
# CONFIG_VBUS_PROXY is not set
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_HAVE_AOUT=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m
CONFIG_HAVE_ATOMIC_IOMAP=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
# CONFIG_IP_FIB_TRIE_STATS is not set
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_CLASSID=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=m
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
CONFIG_IPV6_MIP6=y
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_SIT_6RD=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_MULTIPLE_TABLES=y
CONFIG_IPV6_SUBTREES=y
CONFIG_IPV6_MROUTE=y
CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y
CONFIG_IPV6_PIMSM_V2=y
# CONFIG_NETLABEL is not set
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CONNTRACK_EVENTS=y
# CONFIG_NF_CONNTRACK_TIMESTAMP is not set
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CT_PROTO_UDPLITE=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
# CONFIG_NF_CONNTRACK_SNMP is not set
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_TPROXY=m
CONFIG_NETFILTER_XTABLES=m
#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
#
# Xtables targets
#
# CONFIG_NETFILTER_XT_TARGET_AUDIT is not set
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
#
# Xtables matches
#
# CONFIG_NETFILTER_XT_MATCH_ADDRTYPE is not set
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
# CONFIG_NETFILTER_XT_MATCH_DEVGROUP is not set
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
# CONFIG_IP_SET is not set
CONFIG_IP_VS=m
CONFIG_IP_VS_IPV6=y
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12
#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_PROTO_SCTP=y
#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m
#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
#
# IPv6: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
#
# DECnet: Netfilter Configuration
#
CONFIG_DECNET_NF_GRABULATOR=m
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m
#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_TFRC_LIB=y
#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
CONFIG_NET_DCCPPROBE=m
CONFIG_IP_SCTP=m
CONFIG_NET_SCTPPROBE=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
CONFIG_RDS=m
CONFIG_RDS_RDMA=m
CONFIG_RDS_TCP=m
# CONFIG_RDS_DEBUG is not set
CONFIG_TIPC=m
CONFIG_TIPC_ADVANCED=y
CONFIG_TIPC_PORTS=8191
CONFIG_TIPC_LOG=0
# CONFIG_TIPC_DEBUG is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
CONFIG_ATM_MPOA=m
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_L2TP=m
CONFIG_L2TP_DEBUGFS=m
CONFIG_L2TP_V3=y
CONFIG_L2TP_IP=m
CONFIG_L2TP_ETH=m
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y
CONFIG_DECNET=m
# CONFIG_DECNET_ROUTER is not set
CONFIG_LLC=y
CONFIG_LLC2=m
CONFIG_IPX=m
# CONFIG_IPX_INTERN is not set
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=m
CONFIG_LTPC=m
# CONFIG_COPS is not set
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
# CONFIG_X25 is not set
CONFIG_LAPB=m
# CONFIG_ECONET is not set
CONFIG_WAN_ROUTER=m
CONFIG_PHONET=m
CONFIG_IEEE802154=m
CONFIG_NET_SCHED=y
#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
# CONFIG_NET_SCH_MQPRIO is not set
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_INGRESS=m
#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=y
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
CONFIG_DCB=y
CONFIG_DNS_RESOLVER=y
# CONFIG_BATMAN_ADV is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_TCPPROBE is not set
CONFIG_NET_DROP_MONITOR=y
CONFIG_HAMRADIO=y
#
# Packet Radio protocols
#
CONFIG_AX25=m
# CONFIG_AX25_DAMA_SLAVE is not set
CONFIG_NETROM=m
CONFIG_ROSE=m
#
# AX.25 network device drivers
#
CONFIG_MKISS=m
CONFIG_6PACK=m
CONFIG_BPQETHER=m
CONFIG_SCC=m
# CONFIG_SCC_DELAY is not set
# CONFIG_SCC_TRXECHO is not set
CONFIG_BAYCOM_SER_FDX=m
CONFIG_BAYCOM_SER_HDX=m
CONFIG_BAYCOM_PAR=m
CONFIG_BAYCOM_EPP=m
CONFIG_YAM=m
CONFIG_CAN=m
CONFIG_CAN_RAW=m
CONFIG_CAN_BCM=m
#
# CAN Device Drivers
#
CONFIG_CAN_VCAN=m
# CONFIG_CAN_SLCAN is not set
CONFIG_CAN_DEV=m
CONFIG_CAN_CALC_BITTIMING=y
CONFIG_CAN_MCP251X=m
# CONFIG_PCH_CAN is not set
CONFIG_CAN_SJA1000=m
CONFIG_CAN_SJA1000_ISA=m
# CONFIG_CAN_SJA1000_PLATFORM is not set
CONFIG_CAN_EMS_PCI=m
CONFIG_CAN_KVASER_PCI=m
CONFIG_CAN_PLX_PCI=m
CONFIG_CAN_TSCAN1=m
# CONFIG_CAN_C_CAN is not set
#
# CAN USB interfaces
#
CONFIG_CAN_EMS_USB=m
CONFIG_CAN_ESD_USB2=m
# CONFIG_CAN_SOFTING is not set
# CONFIG_CAN_DEBUG_DEVICES is not set
CONFIG_IRDA=m
#
# IrDA protocols
#
CONFIG_IRLAN=m
CONFIG_IRNET=m
CONFIG_IRCOMM=m
# CONFIG_IRDA_ULTRA is not set
#
# IrDA options
#
CONFIG_IRDA_CACHE_LAST_LSAP=y
CONFIG_IRDA_FAST_RR=y
# CONFIG_IRDA_DEBUG is not set
#
# Infrared-port device drivers
#
#
# SIR device drivers
#
CONFIG_IRTTY_SIR=m
#
# Dongle support
#
CONFIG_DONGLE=y
CONFIG_ESI_DONGLE=m
CONFIG_ACTISYS_DONGLE=m
CONFIG_TEKRAM_DONGLE=m
CONFIG_TOIM3232_DONGLE=m
CONFIG_LITELINK_DONGLE=m
CONFIG_MA600_DONGLE=m
CONFIG_GIRBIL_DONGLE=m
CONFIG_MCP2120_DONGLE=m
CONFIG_OLD_BELKIN_DONGLE=m
CONFIG_ACT200L_DONGLE=m
CONFIG_KINGSUN_DONGLE=m
CONFIG_KSDAZZLE_DONGLE=m
CONFIG_KS959_DONGLE=m
#
# FIR device drivers
#
CONFIG_USB_IRDA=m
CONFIG_SIGMATEL_FIR=m
CONFIG_NSC_FIR=m
CONFIG_WINBOND_FIR=m
CONFIG_TOSHIBA_FIR=m
CONFIG_SMC_IRCC_FIR=m
CONFIG_ALI_FIR=m
CONFIG_VLSI_FIR=m
CONFIG_VIA_FIR=m
CONFIG_MCS_FIR=m
CONFIG_BT=m
CONFIG_BT_L2CAP=y
CONFIG_BT_SCO=y
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_CMTP=m
CONFIG_BT_HIDP=m
#
# Bluetooth device drivers
#
CONFIG_BT_HCIBTUSB=m
CONFIG_BT_HCIBTSDIO=m
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_ATH3K=y
CONFIG_BT_HCIUART_LL=y
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIDTL1=m
CONFIG_BT_HCIBT3C=m
CONFIG_BT_HCIBLUECARD=m
CONFIG_BT_HCIBTUART=m
CONFIG_BT_HCIVHCI=m
CONFIG_BT_MRVL=m
CONFIG_BT_MRVL_SDIO=m
CONFIG_BT_ATH3K=m
CONFIG_AF_RXRPC=m
# CONFIG_AF_RXRPC_DEBUG is not set
CONFIG_RXKAD=m
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
CONFIG_WIRELESS_EXT=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_WEXT_SPY=y
CONFIG_WEXT_PRIV=y
CONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
# CONFIG_CFG80211_REG_DEBUG is not set
CONFIG_CFG80211_DEFAULT_PS=y
# CONFIG_CFG80211_DEBUGFS is not set
# CONFIG_CFG80211_INTERNAL_REGDB is not set
CONFIG_CFG80211_WEXT=y
# CONFIG_WIRELESS_EXT_SYSFS is not set
CONFIG_LIB80211=m
CONFIG_LIB80211_CRYPT_WEP=m
CONFIG_LIB80211_CRYPT_CCMP=m
CONFIG_LIB80211_CRYPT_TKIP=m
# CONFIG_LIB80211_DEBUG is not set
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_MINSTREL_HT=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_LEDS=y
# CONFIG_MAC80211_DEBUGFS is not set
# CONFIG_MAC80211_DEBUG_MENU is not set
CONFIG_WIMAX=m
CONFIG_WIMAX_DEBUG_LEVEL=8
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
CONFIG_NET_9P=m
CONFIG_NET_9P_VIRTIO=m
CONFIG_NET_9P_RDMA=m
# CONFIG_NET_9P_DEBUG is not set
# CONFIG_CAIF is not set
CONFIG_CEPH_LIB=m
# CONFIG_CEPH_LIB_PRETTYDEBUG is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
# CONFIG_DEVTMPFS_MOUNT is not set
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_FIRMWARE_IN_KERNEL is not set
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_ARCH_NO_SYSDEV_OPS=y
CONFIG_CONNECTOR=m
CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not set
# CONFIG_MTD_TESTS is not set
CONFIG_MTD_PARTITIONS=y
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1
# CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set
# CONFIG_MTD_REDBOOT_PARTS_READONLY is not set
CONFIG_MTD_AR7_PARTS=m
#
# User Modules And Translation Layers
#
CONFIG_MTD_CHAR=m
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
CONFIG_MTD_BLOCK_RO=m
CONFIG_FTL=m
CONFIG_NFTL=m
CONFIG_NFTL_RW=y
CONFIG_INFTL=m
CONFIG_RFD_FTL=m
CONFIG_SSFDC=m
# CONFIG_SM_FTL is not set
CONFIG_MTD_OOPS=m
#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=m
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
CONFIG_MTD_CFI_INTELEXT=m
CONFIG_MTD_CFI_AMDSTD=m
CONFIG_MTD_CFI_STAA=m
CONFIG_MTD_CFI_UTIL=m
CONFIG_MTD_RAM=m
CONFIG_MTD_ROM=m
CONFIG_MTD_ABSENT=m
#
# Mapping drivers for chip access
#
CONFIG_MTD_COMPLEX_MAPPINGS=y
CONFIG_MTD_PHYSMAP=m
# CONFIG_MTD_PHYSMAP_COMPAT is not set
CONFIG_MTD_SC520CDP=m
CONFIG_MTD_NETSC520=m
CONFIG_MTD_TS5500=m
CONFIG_MTD_SBC_GXX=m
CONFIG_MTD_SCx200_DOCFLASH=m
# CONFIG_MTD_AMD76XROM is not set
# CONFIG_MTD_ICHXROM is not set
# CONFIG_MTD_ESB2ROM is not set
# CONFIG_MTD_CK804XROM is not set
# CONFIG_MTD_SCB2_FLASH is not set
CONFIG_MTD_NETtel=m
# CONFIG_MTD_L440GX is not set
CONFIG_MTD_PCI=m
CONFIG_MTD_PCMCIA=m
# CONFIG_MTD_PCMCIA_ANONYMOUS is not set
# CONFIG_MTD_GPIO_ADDR is not set
CONFIG_MTD_INTEL_VR_NOR=m
CONFIG_MTD_PLATRAM=m
# CONFIG_MTD_LATCH_ADDR is not set
#
# Self-contained MTD device drivers
#
CONFIG_MTD_PMC551=m
# CONFIG_MTD_PMC551_BUGFIX is not set
# CONFIG_MTD_PMC551_DEBUG is not set
CONFIG_MTD_DATAFLASH=m
# CONFIG_MTD_DATAFLASH_WRITE_VERIFY is not set
# CONFIG_MTD_DATAFLASH_OTP is not set
CONFIG_MTD_M25P80=m
CONFIG_M25PXX_USE_FAST_READ=y
CONFIG_MTD_SST25L=m
CONFIG_MTD_SLRAM=m
CONFIG_MTD_PHRAM=m
CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=m
#
# Disk-On-Chip Device Drivers
#
CONFIG_MTD_DOC2000=m
CONFIG_MTD_DOC2001=m
CONFIG_MTD_DOC2001PLUS=m
CONFIG_MTD_DOCPROBE=m
CONFIG_MTD_DOCECC=m
# CONFIG_MTD_DOCPROBE_ADVANCED is not set
CONFIG_MTD_DOCPROBE_ADDRESS=0
CONFIG_MTD_NAND_ECC=m
# CONFIG_MTD_NAND_ECC_SMC is not set
CONFIG_MTD_NAND=m
# CONFIG_MTD_NAND_VERIFY_WRITE is not set
# CONFIG_MTD_NAND_ECC_BCH is not set
CONFIG_MTD_SM_COMMON=m
# CONFIG_MTD_NAND_MUSEUM_IDS is not set
# CONFIG_MTD_NAND_DENALI is not set
CONFIG_MTD_NAND_IDS=m
CONFIG_MTD_NAND_RICOH=m
CONFIG_MTD_NAND_DISKONCHIP=m
# CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADVANCED is not set
CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADDRESS=0
# CONFIG_MTD_NAND_DISKONCHIP_BBTWRITE is not set
CONFIG_MTD_NAND_CAFE=m
CONFIG_MTD_NAND_CS553X=m
CONFIG_MTD_NAND_NANDSIM=m
CONFIG_MTD_NAND_PLATFORM=m
CONFIG_MTD_ALAUDA=m
CONFIG_MTD_ONENAND=m
CONFIG_MTD_ONENAND_VERIFY_WRITE=y
CONFIG_MTD_ONENAND_GENERIC=m
# CONFIG_MTD_ONENAND_OTP is not set
CONFIG_MTD_ONENAND_2X_PROGRAM=y
CONFIG_MTD_ONENAND_SIM=m
#
# LPDDR flash memory drivers
#
CONFIG_MTD_LPDDR=m
CONFIG_MTD_QINFO_PROBE=m
CONFIG_MTD_UBI=m
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_RESERVE=1
# CONFIG_MTD_UBI_GLUEBI is not set
# CONFIG_MTD_UBI_DEBUG is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
CONFIG_PARPORT_PC_PCMCIA=m
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_AX88796=m
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set
#
# Protocols
#
CONFIG_ISAPNP=y
CONFIG_PNPBIOS=y
CONFIG_PNPBIOS_PROC_FS=y
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_XD=m
CONFIG_PARIDE=m
#
# Parallel IDE high-level drivers
#
CONFIG_PARIDE_PD=m
CONFIG_PARIDE_PCD=m
CONFIG_PARIDE_PF=m
CONFIG_PARIDE_PT=m
CONFIG_PARIDE_PG=m
#
# Parallel IDE protocol modules
#
CONFIG_PARIDE_ATEN=m
CONFIG_PARIDE_BPCK=m
CONFIG_PARIDE_BPCK6=m
CONFIG_PARIDE_COMM=m
CONFIG_PARIDE_DSTR=m
CONFIG_PARIDE_FIT2=m
CONFIG_PARIDE_FIT3=m
CONFIG_PARIDE_EPAT=m
# CONFIG_PARIDE_EPATC8 is not set
CONFIG_PARIDE_EPIA=m
CONFIG_PARIDE_FRIQ=m
CONFIG_PARIDE_FRPW=m
CONFIG_PARIDE_KBIC=m
CONFIG_PARIDE_KTTI=m
CONFIG_PARIDE_ON20=m
CONFIG_PARIDE_ON26=m
CONFIG_BLK_CPQ_DA=m
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_UMEM=m
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
CONFIG_BLK_DEV_DRBD=m
# CONFIG_DRBD_FAULT_INJECTION is not set
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_OSD=m
CONFIG_BLK_DEV_SX8=m
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=8192
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=m
CONFIG_VIRTIO_BLK=m
# CONFIG_BLK_DEV_HD is not set
CONFIG_BLK_DEV_RBD=m
CONFIG_SENSORS_LIS3LV02D=m
CONFIG_MISC_DEVICES=y
CONFIG_AD525X_DPOT=m
CONFIG_AD525X_DPOT_I2C=m
CONFIG_AD525X_DPOT_SPI=m
CONFIG_IBM_ASM=m
CONFIG_PHANTOM=m
CONFIG_SGI_IOC4=m
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
CONFIG_ICS932S401=m
CONFIG_ENCLOSURE_SERVICES=m
CONFIG_CS5535_MFGPT=m
CONFIG_CS5535_MFGPT_DEFAULT_IRQ=7
CONFIG_CS5535_CLOCK_EVENT_SRC=m
CONFIG_HP_ILO=m
CONFIG_APDS9802ALS=m
CONFIG_ISL29003=m
CONFIG_ISL29020=m
CONFIG_SENSORS_TSL2550=m
CONFIG_SENSORS_BH1780=m
CONFIG_SENSORS_BH1770=m
CONFIG_SENSORS_APDS990X=m
CONFIG_HMC6352=m
CONFIG_DS1682=m
CONFIG_TI_DAC7512=m
CONFIG_VMWARE_BALLOON=m
CONFIG_BMP085=m
# CONFIG_PCH_PHUB is not set
CONFIG_C2PORT=m
CONFIG_C2PORT_DURAMAR_2150=m
#
# EEPROM support
#
CONFIG_EEPROM_AT24=m
CONFIG_EEPROM_AT25=m
CONFIG_EEPROM_LEGACY=m
CONFIG_EEPROM_MAX6875=m
CONFIG_EEPROM_93CX6=m
CONFIG_CB710_CORE=m
# CONFIG_CB710_DEBUG is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y
CONFIG_IWMC3200TOP=m
# CONFIG_IWMC3200TOP_DEBUG is not set
# CONFIG_IWMC3200TOP_DEBUGFS is not set
#
# Texas Instruments shared transport line discipline
#
# CONFIG_TI_ST is not set
CONFIG_SENSORS_LIS3_I2C=m
CONFIG_HAVE_IDE=y
CONFIG_IDE=m
#
# Please see Documentation/ide/ide.txt for help/info on IDE drives
#
CONFIG_IDE_XFER_MODE=y
CONFIG_IDE_ATAPI=y
# CONFIG_BLK_DEV_IDE_SATA is not set
CONFIG_IDE_GD=m
CONFIG_IDE_GD_ATA=y
CONFIG_IDE_GD_ATAPI=y
# CONFIG_BLK_DEV_IDECS is not set
CONFIG_BLK_DEV_DELKIN=m
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
CONFIG_BLK_DEV_IDETAPE=m
# CONFIG_BLK_DEV_IDEACPI is not set
# CONFIG_IDE_TASK_IOCTL is not set
CONFIG_IDE_PROC_FS=y
#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=m
# CONFIG_BLK_DEV_PLATFORM is not set
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPNP=m
CONFIG_BLK_DEV_IDEDMA_SFF=y
#
# PCI IDE chipsets support
#
CONFIG_BLK_DEV_IDEPCI=y
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_GENERIC is not set
CONFIG_BLK_DEV_OPTI621=m
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
CONFIG_BLK_DEV_CS5535=m
# CONFIG_BLK_DEV_CS5536 is not set
CONFIG_BLK_DEV_HPT366=m
# CONFIG_BLK_DEV_JMICRON is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
CONFIG_BLK_DEV_IT8172=m
CONFIG_BLK_DEV_IT8213=m
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
CONFIG_BLK_DEV_TRM290=m
# CONFIG_BLK_DEV_VIA82CXXX is not set
CONFIG_BLK_DEV_TC86C001=m
#
# Other IDE chipsets support
#
#
# Note: most of these also require special kernel boot parameters
#
# CONFIG_BLK_DEV_4DRIVES is not set
# CONFIG_BLK_DEV_ALI14XX is not set
# CONFIG_BLK_DEV_DTC2278 is not set
# CONFIG_BLK_DEV_HT6560B is not set
# CONFIG_BLK_DEV_QD65XX is not set
# CONFIG_BLK_DEV_UMC8672 is not set
CONFIG_BLK_DEV_IDEDMA=y
#
# SCSI device support
#
CONFIG_SCSI_MOD=m
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=m
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=m
CONFIG_SCSI_NETLINK=y
# CONFIG_SCSI_PROC_FS is not set
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_ENCLOSURE=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_FC_TGT_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_SRP_ATTRS=m
CONFIG_SCSI_SRP_TGT_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_CXGB4_ISCSI=m
CONFIG_SCSI_BNX2_ISCSI=m
# CONFIG_SCSI_BNX2X_FCOE is not set
CONFIG_BE2ISCSI=m
CONFIG_BLK_DEV_3W_XXXX_RAID=m
CONFIG_SCSI_HPSA=m
CONFIG_SCSI_3W_9XXX=m
CONFIG_SCSI_3W_SAS=m
CONFIG_SCSI_7000FASST=m
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AHA152X=m
CONFIG_SCSI_AHA1542=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=8
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=15000
CONFIG_AIC79XX_DEBUG_ENABLE=y
CONFIG_AIC79XX_DEBUG_MASK=0
CONFIG_AIC79XX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC94XX=m
# CONFIG_AIC94XX_DEBUG is not set
CONFIG_SCSI_MVSAS=m
# CONFIG_SCSI_MVSAS_DEBUG is not set
CONFIG_SCSI_DPT_I2O=m
CONFIG_SCSI_ADVANSYS=m
CONFIG_SCSI_IN2000=m
CONFIG_SCSI_ARCMSR=m
# CONFIG_SCSI_ARCMSR_AER is not set
CONFIG_MEGARAID_NEWGEN=y
CONFIG_MEGARAID_MM=m
CONFIG_MEGARAID_MAILBOX=m
CONFIG_MEGARAID_LEGACY=m
CONFIG_MEGARAID_SAS=m
CONFIG_SCSI_MPT2SAS=m
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
# CONFIG_SCSI_MPT2SAS_LOGGING is not set
CONFIG_SCSI_HPTIOP=m
CONFIG_SCSI_BUSLOGIC=m
# CONFIG_SCSI_FLASHPOINT is not set
CONFIG_VMWARE_PVSCSI=m
CONFIG_LIBFC=m
CONFIG_LIBFCOE=m
CONFIG_FCOE=m
CONFIG_FCOE_FNIC=m
CONFIG_SCSI_DMX3191D=m
CONFIG_SCSI_DTC3280=m
CONFIG_SCSI_EATA=m
CONFIG_SCSI_EATA_TAGGED_QUEUE=y
CONFIG_SCSI_EATA_LINKED_COMMANDS=y
CONFIG_SCSI_EATA_MAX_TAGS=16
CONFIG_SCSI_FUTURE_DOMAIN=m
CONFIG_SCSI_GDTH=m
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
CONFIG_SCSI_IPS=m
CONFIG_SCSI_INITIO=m
# CONFIG_SCSI_INIA100 is not set
CONFIG_SCSI_PPA=m
CONFIG_SCSI_IMM=m
# CONFIG_SCSI_IZIP_EPP16 is not set
# CONFIG_SCSI_IZIP_SLOW_CTR is not set
CONFIG_SCSI_NCR53C406A=m
CONFIG_SCSI_STEX=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
CONFIG_SCSI_IPR=m
# CONFIG_SCSI_IPR_TRACE is not set
# CONFIG_SCSI_IPR_DUMP is not set
CONFIG_SCSI_PAS16=m
CONFIG_SCSI_QLOGIC_FAS=m
CONFIG_SCSI_QLOGIC_1280=m
CONFIG_SCSI_QLA_FC=m
CONFIG_SCSI_QLA_ISCSI=m
CONFIG_SCSI_LPFC=m
# CONFIG_SCSI_LPFC_DEBUG_FS is not set
CONFIG_SCSI_SYM53C416=m
CONFIG_SCSI_DC395x=m
CONFIG_SCSI_DC390T=m
CONFIG_SCSI_T128=m
CONFIG_SCSI_U14_34F=m
CONFIG_SCSI_U14_34F_TAGGED_QUEUE=y
CONFIG_SCSI_U14_34F_LINKED_COMMANDS=y
CONFIG_SCSI_U14_34F_MAX_TAGS=8
CONFIG_SCSI_ULTRASTOR=m
CONFIG_SCSI_NSP32=m
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_PMCRAID=m
CONFIG_SCSI_PM8001=m
CONFIG_SCSI_SRP=m
CONFIG_SCSI_BFA_FC=m
CONFIG_SCSI_LOWLEVEL_PCMCIA=y
CONFIG_PCMCIA_AHA152X=m
CONFIG_PCMCIA_FDOMAIN=m
CONFIG_PCMCIA_NINJA_SCSI=m
CONFIG_PCMCIA_QLOGIC=m
CONFIG_PCMCIA_SYM53C500=m
CONFIG_SCSI_DH=m
CONFIG_SCSI_DH_RDAC=m
CONFIG_SCSI_DH_HP_SW=m
CONFIG_SCSI_DH_EMC=m
CONFIG_SCSI_DH_ALUA=m
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_SCSI_OSD_DPRINT_SENSE=1
# CONFIG_SCSI_OSD_DEBUG is not set
CONFIG_ATA=m
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y
#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=m
# CONFIG_SATA_AHCI_PLATFORM is not set
CONFIG_SATA_INIC162X=m
# CONFIG_SATA_ACARD_AHCI is not set
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y
#
# SFF controllers with custom DMA interface
#
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_SX4=m
CONFIG_ATA_BMDMA=y
#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=m
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_SVW=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m
#
# PATA SFF controllers with BMDMA
#
CONFIG_PATA_ALI=m
CONFIG_PATA_AMD=m
# CONFIG_PATA_ARASAN_CF is not set
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
CONFIG_PATA_ATP867X=m
CONFIG_PATA_CMD64X=m
CONFIG_PATA_CS5520=m
CONFIG_PATA_CS5530=m
# CONFIG_PATA_CS5535 is not set
CONFIG_PATA_CS5536=m
# CONFIG_PATA_CYPRESS is not set
CONFIG_PATA_EFAR=m
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_IT821X=m
CONFIG_PATA_JMICRON=m
CONFIG_PATA_MARVELL=m
CONFIG_PATA_NETCELL=m
# CONFIG_PATA_NINJA32 is not set
CONFIG_PATA_NS87415=m
CONFIG_PATA_OLDPIIX=m
# CONFIG_PATA_OPTIDMA is not set
CONFIG_PATA_PDC2027X=m
CONFIG_PATA_PDC_OLD=m
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RDC=m
CONFIG_PATA_SC1200=m
CONFIG_PATA_SCH=m
CONFIG_PATA_SERVERWORKS=m
CONFIG_PATA_SIL680=m
CONFIG_PATA_SIS=m
CONFIG_PATA_TOSHIBA=m
CONFIG_PATA_TRIFLEX=m
CONFIG_PATA_VIA=m
# CONFIG_PATA_WINBOND is not set
#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_ISAPNP is not set
CONFIG_PATA_MPIIX=m
CONFIG_PATA_NS87410=m
# CONFIG_PATA_OPTI is not set
CONFIG_PATA_PCMCIA=m
# CONFIG_PATA_QDI is not set
CONFIG_PATA_RZ1000=m
# CONFIG_PATA_WINBOND_VLB is not set
#
# Generic fallback / legacy drivers
#
# CONFIG_PATA_ACPI is not set
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
# CONFIG_MULTICORE_RAID456 is not set
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
# CONFIG_DM_RAID is not set
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
CONFIG_DM_DELAY=m
CONFIG_DM_UEVENT=y
# CONFIG_DM_FLAKEY is not set
# CONFIG_TARGET_CORE is not set
CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
CONFIG_FUSION_FC=m
CONFIG_FUSION_SAS=m
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LAN=m
# CONFIG_FUSION_LOGGING is not set
#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_OHCI_DEBUG=y
CONFIG_FIREWIRE_SBP2=m
CONFIG_FIREWIRE_NET=m
CONFIG_FIREWIRE_NOSY=m
CONFIG_I2O=m
CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_CONFIG=m
CONFIG_I2O_CONFIG_OLD_IOCTL=y
CONFIG_I2O_BUS=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_IFB=m
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
CONFIG_EQUALIZER=m
CONFIG_TUN=m
CONFIG_VETH=m
CONFIG_NET_SB1000=m
CONFIG_ARCNET=m
CONFIG_ARCNET_1201=m
CONFIG_ARCNET_1051=m
CONFIG_ARCNET_RAW=m
CONFIG_ARCNET_CAP=m
CONFIG_ARCNET_COM90xx=m
CONFIG_ARCNET_COM90xxIO=m
CONFIG_ARCNET_RIM_I=m
CONFIG_ARCNET_COM20020=m
CONFIG_ARCNET_COM20020_ISA=m
CONFIG_ARCNET_COM20020_PCI=m
CONFIG_MII=m
CONFIG_PHYLIB=m
#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=m
CONFIG_DAVICOM_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_LXT_PHY=m
CONFIG_CICADA_PHY=m
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=m
CONFIG_BROADCOM_PHY=m
# CONFIG_BCM63XX_PHY is not set
CONFIG_ICPLUS_PHY=m
CONFIG_REALTEK_PHY=m
CONFIG_NATIONAL_PHY=m
CONFIG_STE10XP=m
CONFIG_LSI_ET1011C_PHY=m
CONFIG_MICREL_PHY=m
CONFIG_MDIO_BITBANG=m
# CONFIG_MDIO_GPIO is not set
CONFIG_NET_ETHERNET=y
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_CASSINI=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_EL1=m
CONFIG_EL2=m
CONFIG_ELPLUS=m
CONFIG_EL16=m
CONFIG_EL3=m
CONFIG_3C515=m
CONFIG_VORTEX=m
CONFIG_TYPHOON=m
CONFIG_LANCE=m
CONFIG_NET_VENDOR_SMC=y
CONFIG_WD80x3=m
CONFIG_ULTRA=m
CONFIG_SMC9194=m
CONFIG_ENC28J60=m
# CONFIG_ENC28J60_WRITEVERIFY is not set
CONFIG_ETHOC=m
CONFIG_NET_VENDOR_RACAL=y
CONFIG_NI52=m
CONFIG_NI65=m
CONFIG_DNET=m
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
# CONFIG_TULIP_MMIO is not set
CONFIG_TULIP_NAPI=y
CONFIG_TULIP_NAPI_HW_MITIGATION=y
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
CONFIG_AT1700=m
CONFIG_DEPCA=m
CONFIG_HP100=m
CONFIG_NET_ISA=y
CONFIG_E2100=m
CONFIG_EWRK3=m
CONFIG_EEXPRESS=m
CONFIG_EEXPRESS_PRO=m
CONFIG_HPLAN_PLUS=m
CONFIG_HPLAN=m
CONFIG_LP486E=m
CONFIG_ETH16I=m
CONFIG_NE2000=m
CONFIG_ZNET=m
CONFIG_SEEQ8005=m
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
CONFIG_ADAPTEC_STARFIRE=m
CONFIG_AC3200=m
CONFIG_KSZ884X_PCI=m
CONFIG_APRICOT=m
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_FORCEDETH=m
CONFIG_CS89x0=m
CONFIG_E100=m
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=m
CONFIG_8139TOO_PIO=y
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_R6040=m
CONFIG_SIS900=m
CONFIG_EPIC100=m
CONFIG_SMSC9420=m
CONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not set
CONFIG_TLAN=m
CONFIG_KS8842=m
CONFIG_KS8851=m
CONFIG_KS8851_MLL=m
CONFIG_VIA_RHINE=m
# CONFIG_VIA_RHINE_MMIO is not set
CONFIG_SC92031=m
# CONFIG_NET_POCKET is not set
CONFIG_ATL2=m
CONFIG_NETDEV_1000=y
CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not set
CONFIG_DL2K=m
CONFIG_E1000=m
CONFIG_E1000E=m
CONFIG_IP1000=m
CONFIG_IGB=m
CONFIG_IGB_DCA=y
CONFIG_IGBVF=m
CONFIG_NS83820=m
CONFIG_HAMACHI=m
CONFIG_YELLOWFIN=m
CONFIG_R8169=m
CONFIG_SIS190=m
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
CONFIG_VIA_VELOCITY=m
CONFIG_TIGON3=m
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_QLA3XXX=m
CONFIG_ATL1=m
CONFIG_ATL1E=m
CONFIG_ATL1C=m
CONFIG_JME=m
# CONFIG_STMMAC_ETH is not set
# CONFIG_PCH_GBE is not set
CONFIG_NETDEV_10000=y
CONFIG_MDIO=m
CONFIG_CHELSIO_T1=m
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4=m
CONFIG_CHELSIO_T4VF=m
CONFIG_ENIC=m
CONFIG_IXGBE=m
CONFIG_IXGBE_DCA=y
CONFIG_IXGBE_DCB=y
CONFIG_IXGBEVF=m
CONFIG_IXGB=m
CONFIG_S2IO=m
CONFIG_VXGE=m
# CONFIG_VXGE_DEBUG_TRACE_ALL is not set
CONFIG_MYRI10GE=m
CONFIG_MYRI10GE_DCA=y
CONFIG_NETXEN_NIC=m
CONFIG_NIU=m
CONFIG_MLX4_EN=m
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_TEHUTI=m
CONFIG_BNX2X=m
CONFIG_QLCNIC=m
CONFIG_QLGE=m
CONFIG_BNA=m
CONFIG_SFC=m
CONFIG_SFC_MTD=y
CONFIG_BE2NET=m
CONFIG_TR=y
CONFIG_IBMTR=m
CONFIG_IBMOL=m
CONFIG_IBMLS=m
CONFIG_3C359=m
CONFIG_TMS380TR=m
CONFIG_TMSPCI=m
CONFIG_SKISA=m
CONFIG_PROTEON=m
CONFIG_ABYSS=m
CONFIG_SMCTR=m
CONFIG_WLAN=y
CONFIG_PCMCIA_RAYCS=m
CONFIG_LIBERTAS_THINFIRM=m
# CONFIG_LIBERTAS_THINFIRM_DEBUG is not set
CONFIG_LIBERTAS_THINFIRM_USB=m
CONFIG_AIRO=m
CONFIG_ATMEL=m
CONFIG_PCI_ATMEL=m
CONFIG_PCMCIA_ATMEL=m
CONFIG_AT76C50X_USB=m
CONFIG_AIRO_CS=m
CONFIG_PCMCIA_WL3501=m
# CONFIG_PRISM54 is not set
CONFIG_USB_ZD1201=m
CONFIG_USB_NET_RNDIS_WLAN=m
CONFIG_RTL8180=m
CONFIG_RTL8187=m
CONFIG_RTL8187_LEDS=y
CONFIG_ADM8211=m
CONFIG_MAC80211_HWSIM=m
CONFIG_MWL8K=m
CONFIG_ATH_COMMON=m
CONFIG_ATH_DEBUG=y
CONFIG_ATH5K=m
CONFIG_ATH5K_DEBUG=y
CONFIG_ATH5K_TRACER=y
CONFIG_ATH5K_PCI=y
CONFIG_ATH9K_HW=m
CONFIG_ATH9K_COMMON=m
CONFIG_ATH9K=m
# CONFIG_ATH9K_DEBUGFS is not set
CONFIG_ATH9K_RATE_CONTROL=y
CONFIG_ATH9K_HTC=m
# CONFIG_ATH9K_HTC_DEBUGFS is not set
CONFIG_AR9170_USB=m
CONFIG_AR9170_LEDS=y
# CONFIG_CARL9170 is not set
CONFIG_B43=m
CONFIG_B43_PCI_AUTOSELECT=y
CONFIG_B43_PCICORE_AUTOSELECT=y
CONFIG_B43_PCMCIA=y
CONFIG_B43_SDIO=y
CONFIG_B43_PIO=y
# CONFIG_B43_PHY_N is not set
CONFIG_B43_PHY_LP=y
CONFIG_B43_LEDS=y
CONFIG_B43_HWRNG=y
# CONFIG_B43_DEBUG is not set
CONFIG_B43LEGACY=m
CONFIG_B43LEGACY_PCI_AUTOSELECT=y
CONFIG_B43LEGACY_PCICORE_AUTOSELECT=y
CONFIG_B43LEGACY_LEDS=y
CONFIG_B43LEGACY_HWRNG=y
CONFIG_B43LEGACY_DEBUG=y
CONFIG_B43LEGACY_DMA=y
CONFIG_B43LEGACY_PIO=y
CONFIG_B43LEGACY_DMA_AND_PIO_MODE=y
# CONFIG_B43LEGACY_DMA_MODE is not set
# CONFIG_B43LEGACY_PIO_MODE is not set
CONFIG_HOSTAP=m
CONFIG_HOSTAP_FIRMWARE=y
# CONFIG_HOSTAP_FIRMWARE_NVRAM is not set
CONFIG_HOSTAP_PLX=m
CONFIG_HOSTAP_PCI=m
CONFIG_HOSTAP_CS=m
CONFIG_IPW2100=m
CONFIG_IPW2100_MONITOR=y
# CONFIG_IPW2100_DEBUG is not set
CONFIG_IPW2200=m
CONFIG_IPW2200_MONITOR=y
CONFIG_IPW2200_RADIOTAP=y
CONFIG_IPW2200_PROMISCUOUS=y
CONFIG_IPW2200_QOS=y
# CONFIG_IPW2200_DEBUG is not set
CONFIG_LIBIPW=m
# CONFIG_LIBIPW_DEBUG is not set
CONFIG_IWLAGN=m
#
# Debugging Options
#
# CONFIG_IWLWIFI_DEBUG is not set
# CONFIG_IWLWIFI_DEVICE_TRACING is not set
# CONFIG_IWL_P2P is not set
CONFIG_IWLWIFI_LEGACY=m
#
# Debugging Options
#
# CONFIG_IWLWIFI_LEGACY_DEBUG is not set
# CONFIG_IWLWIFI_LEGACY_DEVICE_TRACING is not set
CONFIG_IWL4965=m
CONFIG_IWL3945=m
CONFIG_IWM=m
# CONFIG_IWM_DEBUG is not set
# CONFIG_IWM_TRACING is not set
CONFIG_LIBERTAS=m
CONFIG_LIBERTAS_USB=m
CONFIG_LIBERTAS_CS=m
CONFIG_LIBERTAS_SDIO=m
CONFIG_LIBERTAS_SPI=m
# CONFIG_LIBERTAS_DEBUG is not set
CONFIG_LIBERTAS_MESH=y
CONFIG_HERMES=m
# CONFIG_HERMES_PRISM is not set
CONFIG_HERMES_CACHE_FW_ON_INIT=y
CONFIG_PLX_HERMES=m
CONFIG_TMD_HERMES=m
CONFIG_NORTEL_HERMES=m
CONFIG_PCMCIA_HERMES=m
CONFIG_PCMCIA_SPECTRUM=m
CONFIG_ORINOCO_USB=m
CONFIG_P54_COMMON=m
CONFIG_P54_USB=m
CONFIG_P54_PCI=m
CONFIG_P54_SPI=m
# CONFIG_P54_SPI_DEFAULT_EEPROM is not set
CONFIG_P54_LEDS=y
CONFIG_RT2X00=m
CONFIG_RT2400PCI=m
CONFIG_RT2500PCI=m
CONFIG_RT61PCI=m
CONFIG_RT2800PCI=m
# CONFIG_RT2800PCI_RT33XX is not set
# CONFIG_RT2800PCI_RT35XX is not set
# CONFIG_RT2800PCI_RT53XX is not set
CONFIG_RT2500USB=m
CONFIG_RT73USB=m
CONFIG_RT2800USB=m
# CONFIG_RT2800USB_RT33XX is not set
# CONFIG_RT2800USB_RT35XX is not set
# CONFIG_RT2800USB_UNKNOWN is not set
CONFIG_RT2800_LIB=m
CONFIG_RT2X00_LIB_PCI=m
CONFIG_RT2X00_LIB_USB=m
CONFIG_RT2X00_LIB=m
CONFIG_RT2X00_LIB_HT=y
CONFIG_RT2X00_LIB_FIRMWARE=y
CONFIG_RT2X00_LIB_CRYPTO=y
CONFIG_RT2X00_LIB_LEDS=y
# CONFIG_RT2X00_DEBUG is not set
# CONFIG_RTL8192CE is not set
# CONFIG_RTL8192CU is not set
CONFIG_WL1251=m
CONFIG_WL1251_SPI=m
CONFIG_WL1251_SDIO=m
# CONFIG_WL12XX_MENU is not set
CONFIG_WL12XX_PLATFORM_DATA=y
CONFIG_ZD1211RW=m
# CONFIG_ZD1211RW_DEBUG is not set
#
# WiMAX Wireless Broadband devices
#
CONFIG_WIMAX_I2400M=m
CONFIG_WIMAX_I2400M_USB=m
CONFIG_WIMAX_I2400M_SDIO=m
CONFIG_WIMAX_IWMC3200_SDIO=y
CONFIG_WIMAX_I2400M_DEBUG_LEVEL=8
#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_CDCETHER=m
CONFIG_USB_NET_CDC_EEM=m
CONFIG_USB_NET_CDC_NCM=m
CONFIG_USB_NET_DM9601=m
CONFIG_USB_NET_SMSC75XX=m
CONFIG_USB_NET_SMSC95XX=m
CONFIG_USB_NET_GL620A=m
CONFIG_USB_NET_NET1080=m
CONFIG_USB_NET_PLUSB=m
CONFIG_USB_NET_MCS7830=m
CONFIG_USB_NET_RNDIS_HOST=m
CONFIG_USB_NET_CDC_SUBSET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=m
CONFIG_USB_NET_CX82310_ETH=m
CONFIG_USB_HSO=m
CONFIG_USB_NET_INT51X1=m
CONFIG_USB_CDC_PHONET=m
CONFIG_USB_IPHETH=m
CONFIG_USB_SIERRA_NET=m
CONFIG_NET_PCMCIA=y
CONFIG_PCMCIA_3C589=m
CONFIG_PCMCIA_3C574=m
CONFIG_PCMCIA_FMVJ18X=m
CONFIG_PCMCIA_PCNET=m
CONFIG_PCMCIA_NMCLAN=m
CONFIG_PCMCIA_SMC91C92=m
CONFIG_PCMCIA_XIRC2PS=m
CONFIG_PCMCIA_AXNET=m
CONFIG_ARCNET_COM20020_CS=m
CONFIG_PCMCIA_IBMTR=m
CONFIG_WAN=y
CONFIG_HOSTESS_SV11=m
CONFIG_COSA=m
CONFIG_LANMEDIA=m
CONFIG_SEALEVEL_4021=m
CONFIG_HDLC=m
CONFIG_HDLC_RAW=m
CONFIG_HDLC_RAW_ETH=m
CONFIG_HDLC_CISCO=m
CONFIG_HDLC_FR=m
CONFIG_HDLC_PPP=m
# CONFIG_HDLC_X25 is not set
CONFIG_PCI200SYN=m
CONFIG_WANXL=m
# CONFIG_PC300TOO is not set
CONFIG_N2=m
CONFIG_C101=m
CONFIG_FARSYNC=m
CONFIG_DSCC4=m
CONFIG_DSCC4_PCISYNC=y
CONFIG_DSCC4_PCI_RST=y
CONFIG_DLCI=m
CONFIG_DLCI_MAX=8
CONFIG_SDLA=m
CONFIG_WAN_ROUTER_DRIVERS=m
CONFIG_CYCLADES_SYNC=m
# CONFIG_CYCLOMX_X25 is not set
CONFIG_SBNI=m
# CONFIG_SBNI_MULTILINE is not set
CONFIG_ATM_DRIVERS=y
CONFIG_ATM_DUMMY=m
CONFIG_ATM_TCP=m
CONFIG_ATM_LANAI=m
CONFIG_ATM_ENI=m
# CONFIG_ATM_ENI_DEBUG is not set
# CONFIG_ATM_ENI_TUNE_BURST is not set
CONFIG_ATM_FIRESTREAM=m
CONFIG_ATM_ZATM=m
# CONFIG_ATM_ZATM_DEBUG is not set
CONFIG_ATM_NICSTAR=m
# CONFIG_ATM_NICSTAR_USE_SUNI is not set
# CONFIG_ATM_NICSTAR_USE_IDT77105 is not set
CONFIG_ATM_IDT77252=m
# CONFIG_ATM_IDT77252_DEBUG is not set
# CONFIG_ATM_IDT77252_RCV_ALL is not set
CONFIG_ATM_IDT77252_USE_SUNI=y
CONFIG_ATM_AMBASSADOR=m
# CONFIG_ATM_AMBASSADOR_DEBUG is not set
CONFIG_ATM_HORIZON=m
# CONFIG_ATM_HORIZON_DEBUG is not set
CONFIG_ATM_IA=m
# CONFIG_ATM_IA_DEBUG is not set
CONFIG_ATM_FORE200E=m
# CONFIG_ATM_FORE200E_USE_TASKLET is not set
CONFIG_ATM_FORE200E_TX_RETRY=16
CONFIG_ATM_FORE200E_DEBUG=0
CONFIG_ATM_HE=m
CONFIG_ATM_HE_USE_SUNI=y
CONFIG_ATM_SOLOS=m
CONFIG_IEEE802154_DRIVERS=m
CONFIG_IEEE802154_FAKEHARD=m
#
# CAIF transport drivers
#
CONFIG_FDDI=y
CONFIG_DEFXX=m
# CONFIG_DEFXX_MMIO is not set
CONFIG_SKFP=m
CONFIG_HIPPI=y
CONFIG_ROADRUNNER=m
# CONFIG_ROADRUNNER_LARGE_RINGS is not set
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_PPTP=m
CONFIG_PPPOATM=m
CONFIG_PPPOL2TP=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=m
CONFIG_SLIP_SMART=y
CONFIG_SLIP_MODE_SLIP6=y
CONFIG_NET_FC=y
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_VIRTIO_NET=m
CONFIG_VMXNET3=m
CONFIG_ISDN=y
# CONFIG_ISDN_I4L is not set
CONFIG_ISDN_CAPI=m
CONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=y
CONFIG_CAPI_TRACE=y
CONFIG_ISDN_CAPI_MIDDLEWARE=y
CONFIG_ISDN_CAPI_CAPI20=m
CONFIG_ISDN_CAPI_CAPIFS_BOOL=y
CONFIG_ISDN_CAPI_CAPIFS=m
#
# CAPI hardware drivers
#
CONFIG_CAPI_AVM=y
CONFIG_ISDN_DRV_AVMB1_B1ISA=m
CONFIG_ISDN_DRV_AVMB1_B1PCI=m
CONFIG_ISDN_DRV_AVMB1_B1PCIV4=y
CONFIG_ISDN_DRV_AVMB1_T1ISA=m
CONFIG_ISDN_DRV_AVMB1_B1PCMCIA=m
CONFIG_ISDN_DRV_AVMB1_AVM_CS=m
CONFIG_ISDN_DRV_AVMB1_T1PCI=m
CONFIG_ISDN_DRV_AVMB1_C4=m
CONFIG_CAPI_EICON=y
CONFIG_ISDN_DIVAS=m
CONFIG_ISDN_DIVAS_BRIPCI=y
CONFIG_ISDN_DIVAS_PRIPCI=y
CONFIG_ISDN_DIVAS_DIVACAPI=m
CONFIG_ISDN_DIVAS_USERIDI=m
CONFIG_ISDN_DIVAS_MAINT=m
CONFIG_ISDN_DRV_GIGASET=m
CONFIG_GIGASET_CAPI=y
# CONFIG_GIGASET_DUMMYLL is not set
CONFIG_GIGASET_BASE=m
CONFIG_GIGASET_M105=m
CONFIG_GIGASET_M101=m
# CONFIG_GIGASET_DEBUG is not set
CONFIG_HYSDN=m
CONFIG_HYSDN_CAPI=y
CONFIG_MISDN=m
CONFIG_MISDN_DSP=m
CONFIG_MISDN_L1OIP=m
#
# mISDN hardware drivers
#
CONFIG_MISDN_HFCPCI=m
CONFIG_MISDN_HFCMULTI=m
CONFIG_MISDN_HFCUSB=m
CONFIG_MISDN_AVMFRITZ=m
CONFIG_MISDN_SPEEDFAX=m
CONFIG_MISDN_INFINEON=m
CONFIG_MISDN_W6692=m
# CONFIG_MISDN_NETJET is not set
CONFIG_MISDN_IPAC=m
CONFIG_MISDN_ISAR=m
CONFIG_PHONE=m
CONFIG_PHONE_IXJ=m
CONFIG_PHONE_IXJ_PCMCIA=m
#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=m
CONFIG_INPUT_POLLDEV=m
CONFIG_INPUT_SPARSEKMAP=m
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ADP5588=m
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1070 is not set
CONFIG_KEYBOARD_QT2160=m
CONFIG_KEYBOARD_LKKBD=m
# CONFIG_KEYBOARD_GPIO is not set
# CONFIG_KEYBOARD_GPIO_POLLED is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_MATRIX is not set
CONFIG_KEYBOARD_LM8323=m
CONFIG_KEYBOARD_MAX7359=m
# CONFIG_KEYBOARD_MCS is not set
CONFIG_KEYBOARD_NEWTON=m
CONFIG_KEYBOARD_OPENCORES=m
CONFIG_KEYBOARD_STOWAWAY=m
CONFIG_KEYBOARD_SUNKBD=m
CONFIG_KEYBOARD_XTKBD=m
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_PS2_ELANTECH=y
CONFIG_MOUSE_PS2_SENTELIC=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_APPLETOUCH=m
CONFIG_MOUSE_BCM5974=m
CONFIG_MOUSE_INPORT=m
# CONFIG_MOUSE_ATIXL is not set
CONFIG_MOUSE_LOGIBM=m
CONFIG_MOUSE_PC110PAD=m
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_GPIO is not set
CONFIG_MOUSE_SYNAPTICS_I2C=m
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDJOY=m
CONFIG_JOYSTICK_ZHENHUA=m
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
# CONFIG_JOYSTICK_AS5011 is not set
CONFIG_JOYSTICK_JOYDUMP=m
CONFIG_JOYSTICK_XPAD=m
CONFIG_JOYSTICK_XPAD_FF=y
CONFIG_JOYSTICK_XPAD_LEDS=y
CONFIG_JOYSTICK_WALKERA0701=m
CONFIG_INPUT_TABLET=y
CONFIG_TABLET_USB_ACECAD=m
CONFIG_TABLET_USB_AIPTEK=m
CONFIG_TABLET_USB_GTCO=m
# CONFIG_TABLET_USB_HANWANG is not set
CONFIG_TABLET_USB_KBTAB=m
CONFIG_TABLET_USB_WACOM=m
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_ADS7846=m
CONFIG_TOUCHSCREEN_AD7877=m
CONFIG_TOUCHSCREEN_AD7879=m
CONFIG_TOUCHSCREEN_AD7879_I2C=m
# CONFIG_TOUCHSCREEN_AD7879_SPI is not set
# CONFIG_TOUCHSCREEN_ATMEL_MXT is not set
# CONFIG_TOUCHSCREEN_BU21013 is not set
# CONFIG_TOUCHSCREEN_CY8CTMG110 is not set
CONFIG_TOUCHSCREEN_DYNAPRO=m
CONFIG_TOUCHSCREEN_HAMPSHIRE=m
CONFIG_TOUCHSCREEN_EETI=m
CONFIG_TOUCHSCREEN_FUJITSU=m
CONFIG_TOUCHSCREEN_GUNZE=m
CONFIG_TOUCHSCREEN_ELO=m
CONFIG_TOUCHSCREEN_WACOM_W8001=m
CONFIG_TOUCHSCREEN_MCS5000=m
CONFIG_TOUCHSCREEN_MTOUCH=m
CONFIG_TOUCHSCREEN_INEXIO=m
CONFIG_TOUCHSCREEN_MK712=m
CONFIG_TOUCHSCREEN_HTCPEN=m
CONFIG_TOUCHSCREEN_PENMOUNT=m
CONFIG_TOUCHSCREEN_TOUCHRIGHT=m
CONFIG_TOUCHSCREEN_TOUCHWIN=m
CONFIG_TOUCHSCREEN_WM97XX=m
CONFIG_TOUCHSCREEN_WM9705=y
CONFIG_TOUCHSCREEN_WM9712=y
CONFIG_TOUCHSCREEN_WM9713=y
CONFIG_TOUCHSCREEN_USB_COMPOSITE=m
CONFIG_TOUCHSCREEN_USB_EGALAX=y
CONFIG_TOUCHSCREEN_USB_PANJIT=y
CONFIG_TOUCHSCREEN_USB_3M=y
CONFIG_TOUCHSCREEN_USB_ITM=y
CONFIG_TOUCHSCREEN_USB_ETURBO=y
CONFIG_TOUCHSCREEN_USB_GUNZE=y
CONFIG_TOUCHSCREEN_USB_DMC_TSC10=y
CONFIG_TOUCHSCREEN_USB_IRTOUCH=y
CONFIG_TOUCHSCREEN_USB_IDEALTEK=y
CONFIG_TOUCHSCREEN_USB_GENERAL_TOUCH=y
CONFIG_TOUCHSCREEN_USB_GOTOP=y
CONFIG_TOUCHSCREEN_USB_JASTEC=y
CONFIG_TOUCHSCREEN_USB_E2I=y
CONFIG_TOUCHSCREEN_USB_ZYTRONIC=y
CONFIG_TOUCHSCREEN_USB_ETT_TC45USB=y
CONFIG_TOUCHSCREEN_USB_NEXIO=y
CONFIG_TOUCHSCREEN_TOUCHIT213=m
# CONFIG_TOUCHSCREEN_TSC2005 is not set
CONFIG_TOUCHSCREEN_TSC2007=m
# CONFIG_TOUCHSCREEN_ST1232 is not set
CONFIG_TOUCHSCREEN_TPS6507X=m
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
CONFIG_INPUT_PCSPKR=m
CONFIG_INPUT_APANEL=m
CONFIG_INPUT_WISTRON_BTNS=m
CONFIG_INPUT_ATLAS_BTNS=m
CONFIG_INPUT_ATI_REMOTE=m
CONFIG_INPUT_ATI_REMOTE2=m
CONFIG_INPUT_KEYSPAN_REMOTE=m
CONFIG_INPUT_POWERMATE=m
CONFIG_INPUT_YEALINK=m
CONFIG_INPUT_CM109=m
CONFIG_INPUT_UINPUT=m
CONFIG_INPUT_PCF50633_PMU=m
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_GPIO_ROTARY_ENCODER is not set
# CONFIG_INPUT_ADXL34X is not set
# CONFIG_INPUT_CMA3000 is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=m
CONFIG_SERIO_PARKBD=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_SERIO_ALTERA_PS2=m
# CONFIG_SERIO_PS2MULT is not set
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m
#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_ROCKETPORT=m
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
# CONFIG_MOXA_INTELLIO is not set
CONFIG_MOXA_SMARTIO=m
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_NOZOMI=m
# CONFIG_ISI is not set
CONFIG_N_HDLC=m
CONFIG_N_GSM=m
# CONFIG_DEVKMEM is not set
CONFIG_STALDRV=y
#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_FOURPORT=m
CONFIG_SERIAL_8250_ACCENT=m
CONFIG_SERIAL_8250_BOCA=m
CONFIG_SERIAL_8250_EXAR_ST16C554=m
CONFIG_SERIAL_8250_HUB6=m
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
CONFIG_SERIAL_8250_RSA=y
#
# Non-8250 serial port support
#
CONFIG_SERIAL_MAX3100=m
# CONFIG_SERIAL_MAX3107 is not set
CONFIG_SERIAL_MFD_HSU=m
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_CONSOLE_POLL=y
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_IFX6X60 is not set
# CONFIG_SERIAL_PCH_UART is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
CONFIG_HVC_DRIVER=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=m
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_GEODE=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_NVRAM=m
CONFIG_DTLK=m
CONFIG_R3964=m
CONFIG_APPLICOM=m
CONFIG_SONYPI=m
#
# PCMCIA character devices
#
CONFIG_SYNCLINK_CS=m
CONFIG_CARDMAN_4000=m
CONFIG_CARDMAN_4040=m
CONFIG_IPWIRELESS=m
CONFIG_MWAVE=m
CONFIG_SCx200_GPIO=m
CONFIG_PC8736x_GPIO=m
CONFIG_NSC_GPIO=m
CONFIG_RAW_DRIVER=m
CONFIG_MAX_RAW_DEVS=256
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=m
CONFIG_TCG_TIS=m
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
CONFIG_TELCLOCK=m
CONFIG_DEVPORT=y
CONFIG_RAMOOPS=m
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=m
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCA=m
#
# I2C Hardware Bus support
#
#
# PC SMBus host controller drivers
#
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_NFORCE2_S4985=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
#
# ACPI drivers
#
CONFIG_I2C_SCMI=m
#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_DESIGNWARE_PCI is not set
# CONFIG_I2C_GPIO is not set
CONFIG_I2C_OCORES=m
CONFIG_I2C_PCA_PLATFORM=m
# CONFIG_I2C_PXA_PCI is not set
CONFIG_I2C_SIMTEC=m
# CONFIG_I2C_XILINX is not set
# CONFIG_I2C_EG20T is not set
#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_DIOLAN_U2C is not set
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_TAOS_EVM=m
CONFIG_I2C_TINY_USB=m
#
# Other I2C/SMBus bus drivers
#
CONFIG_I2C_PCA_ISA=m
CONFIG_I2C_STUB=m
CONFIG_SCx200_I2C=m
CONFIG_SCx200_I2C_SCL=12
CONFIG_SCx200_I2C_SDA=13
CONFIG_SCx200_ACB=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
CONFIG_SPI=y
# CONFIG_SPI_DEBUG is not set
CONFIG_SPI_MASTER=y
#
# SPI Master Controller Drivers
#
# CONFIG_SPI_ALTERA is not set
CONFIG_SPI_BITBANG=m
CONFIG_SPI_BUTTERFLY=m
# CONFIG_SPI_GPIO is not set
CONFIG_SPI_LM70_LLP=m
# CONFIG_SPI_OC_TINY is not set
# CONFIG_SPI_PXA2XX is not set
# CONFIG_SPI_PXA2XX_PCI is not set
# CONFIG_SPI_TOPCLIFF_PCH is not set
# CONFIG_SPI_XILINX is not set
# CONFIG_SPI_DESIGNWARE is not set
#
# SPI Protocol Masters
#
# CONFIG_SPI_SPIDEV is not set
CONFIG_SPI_TLE62X0=m
#
# PPS support
#
CONFIG_PPS=m
# CONFIG_PPS_DEBUG is not set
#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=m
# CONFIG_PPS_CLIENT_PARPORT is not set
#
# PPS generators support
#
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_GPIOLIB=y
# CONFIG_DEBUG_GPIO is not set
# CONFIG_GPIO_SYSFS is not set
#
# Memory mapped GPIO expanders:
#
# CONFIG_GPIO_BASIC_MMIO is not set
# CONFIG_GPIO_IT8761E is not set
# CONFIG_GPIO_SCH is not set
# CONFIG_GPIO_VX855 is not set
#
# I2C GPIO expanders:
#
# CONFIG_GPIO_MAX7300 is not set
# CONFIG_GPIO_MAX732X is not set
# CONFIG_GPIO_PCA953X is not set
# CONFIG_GPIO_PCF857X is not set
# CONFIG_GPIO_ADP5588 is not set
#
# PCI GPIO expanders:
#
# CONFIG_GPIO_CS5535 is not set
# CONFIG_GPIO_LANGWELL is not set
# CONFIG_GPIO_PCH is not set
# CONFIG_GPIO_ML_IOH is not set
# CONFIG_GPIO_RDC321X is not set
#
# SPI GPIO expanders:
#
# CONFIG_GPIO_MAX7301 is not set
# CONFIG_GPIO_MCP23S08 is not set
# CONFIG_GPIO_MC33880 is not set
# CONFIG_GPIO_74X164 is not set
#
# AC97 GPIO expanders:
#
#
# MODULbus GPIO expanders:
#
CONFIG_W1=m
CONFIG_W1_CON=y
#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=m
CONFIG_W1_MASTER_DS2490=m
CONFIG_W1_MASTER_DS2482=m
# CONFIG_W1_MASTER_GPIO is not set
#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=m
CONFIG_W1_SLAVE_SMEM=m
# CONFIG_W1_SLAVE_DS2423 is not set
CONFIG_W1_SLAVE_DS2431=m
CONFIG_W1_SLAVE_DS2433=m
# CONFIG_W1_SLAVE_DS2433_CRC is not set
CONFIG_W1_SLAVE_DS2760=m
CONFIG_W1_SLAVE_BQ27000=m
CONFIG_POWER_SUPPLY=m
# CONFIG_POWER_SUPPLY_DEBUG is not set
CONFIG_PDA_POWER=m
# CONFIG_TEST_POWER is not set
CONFIG_BATTERY_DS2760=m
CONFIG_BATTERY_DS2782=m
# CONFIG_BATTERY_BQ20Z75 is not set
CONFIG_BATTERY_BQ27x00=m
CONFIG_BATTERY_BQ27X00_I2C=y
CONFIG_BATTERY_BQ27X00_PLATFORM=y
CONFIG_BATTERY_MAX17040=m
# CONFIG_BATTERY_MAX17042 is not set
CONFIG_CHARGER_PCF50633=m
# CONFIG_CHARGER_ISP1704 is not set
# CONFIG_CHARGER_GPIO is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set
#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ABITUGURU3=m
CONFIG_SENSORS_AD7414=m
CONFIG_SENSORS_AD7418=m
CONFIG_SENSORS_ADCXX=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7411=m
CONFIG_SENSORS_ADT7462=m
CONFIG_SENSORS_ADT7470=m
CONFIG_SENSORS_ADT7475=m
CONFIG_SENSORS_ASC7621=m
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
# CONFIG_SENSORS_DS620 is not set
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHMD=m
CONFIG_SENSORS_G760A=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
# CONFIG_SENSORS_GPIO_FAN is not set
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_PKGTEMP=m
CONFIG_SENSORS_IBMAEM=m
CONFIG_SENSORS_IBMPEX=m
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_JC42 is not set
# CONFIG_SENSORS_LINEAGE is not set
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM70=m
CONFIG_SENSORS_LM73=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
# CONFIG_SENSORS_LTC4151 is not set
CONFIG_SENSORS_LTC4215=m
CONFIG_SENSORS_LTC4245=m
# CONFIG_SENSORS_LTC4261 is not set
CONFIG_SENSORS_LM95241=m
CONFIG_SENSORS_MAX1111=m
CONFIG_SENSORS_MAX1619=m
# CONFIG_SENSORS_MAX6639 is not set
CONFIG_SENSORS_MAX6650=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_PCF8591=m
# CONFIG_PMBUS is not set
CONFIG_SENSORS_SHT15=m
# CONFIG_SENSORS_SHT21 is not set
CONFIG_SENSORS_SIS5595=m
# CONFIG_SENSORS_SMM665 is not set
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_EMC1403=m
# CONFIG_SENSORS_EMC2103 is not set
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
# CONFIG_SENSORS_SCH5627 is not set
# CONFIG_SENSORS_ADS1015 is not set
CONFIG_SENSORS_ADS7828=m
CONFIG_SENSORS_ADS7871=m
CONFIG_SENSORS_AMC6821=m
CONFIG_SENSORS_THMC50=m
CONFIG_SENSORS_TMP102=m
CONFIG_SENSORS_TMP401=m
CONFIG_SENSORS_TMP421=m
CONFIG_SENSORS_VIA_CPUTEMP=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
# CONFIG_SENSORS_W83795 is not set
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_APPLESMC=m
#
# ACPI drivers
#
CONFIG_SENSORS_ATK0110=m
CONFIG_THERMAL=m
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_ACQUIRE_WDT=m
CONFIG_ADVANTECH_WDT=m
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_F71808E_WDT=m
# CONFIG_SP5100_TCO is not set
CONFIG_GEODE_WDT=m
CONFIG_SC520_WDT=m
CONFIG_SBC_FITPC2_WATCHDOG=m
CONFIG_EUROTECH_WDT=m
CONFIG_IB700_WDT=m
CONFIG_IBMASR=m
CONFIG_WAFER_WDT=m
CONFIG_I6300ESB_WDT=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y
CONFIG_IT8712F_WDT=m
CONFIG_IT87_WDT=m
CONFIG_HP_WATCHDOG=m
CONFIG_HPWDT_NMI_DECODING=y
CONFIG_SC1200_WDT=m
CONFIG_SCx200_WDT=m
CONFIG_PC87413_WDT=m
# CONFIG_NV_TCO is not set
CONFIG_60XX_WDT=m
CONFIG_SBC8360_WDT=m
CONFIG_SBC7240_WDT=m
CONFIG_CPU5_WDT=m
CONFIG_SMSC_SCH311X_WDT=m
CONFIG_SMSC37B787_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83697HF_WDT=m
CONFIG_W83697UG_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
CONFIG_SBC_EPX_C3_WATCHDOG=m
#
# ISA-based Watchdog Cards
#
CONFIG_PCWATCHDOG=m
CONFIG_MIXCOMWD=m
CONFIG_WDT=m
#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m
#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_SSB_POSSIBLE=y
#
# Sonics Silicon Backplane
#
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_BLOCKIO=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
CONFIG_SSB_B43_PCI_BRIDGE=y
CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
CONFIG_SSB_PCMCIAHOST=y
CONFIG_SSB_SDIOHOST_POSSIBLE=y
CONFIG_SSB_SDIOHOST=y
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_MFD_SUPPORT=y
CONFIG_MFD_CORE=m
CONFIG_MFD_SM501=m
# CONFIG_MFD_SM501_GPIO is not set
CONFIG_HTC_PASIC3=m
# CONFIG_UCB1400_CORE is not set
# CONFIG_TPS6105X is not set
CONFIG_TPS65010=m
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TMIO is not set
CONFIG_MFD_WM8400=m
# CONFIG_MFD_WM831X_SPI is not set
CONFIG_MFD_PCF50633=m
# CONFIG_MFD_MC13XXX is not set
CONFIG_PCF50633_ADC=m
CONFIG_PCF50633_GPIO=m
# CONFIG_ABX500_CORE is not set
# CONFIG_EZX_PCAP is not set
# CONFIG_MFD_CS5535 is not set
# CONFIG_MFD_TIMBERDALE is not set
CONFIG_LPC_SCH=m
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_VX855 is not set
# CONFIG_MFD_WL1273_CORE is not set
CONFIG_REGULATOR=y
# CONFIG_REGULATOR_DEBUG is not set
# CONFIG_REGULATOR_DUMMY is not set
CONFIG_REGULATOR_FIXED_VOLTAGE=m
# CONFIG_REGULATOR_VIRTUAL_CONSUMER is not set
CONFIG_REGULATOR_USERSPACE_CONSUMER=m
CONFIG_REGULATOR_BQ24022=m
CONFIG_REGULATOR_MAX1586=m
CONFIG_REGULATOR_MAX8649=m
CONFIG_REGULATOR_MAX8660=m
# CONFIG_REGULATOR_MAX8952 is not set
CONFIG_REGULATOR_WM8400=m
CONFIG_REGULATOR_PCF50633=m
CONFIG_REGULATOR_LP3971=m
# CONFIG_REGULATOR_LP3972 is not set
CONFIG_REGULATOR_TPS65023=m
CONFIG_REGULATOR_TPS6507X=m
# CONFIG_REGULATOR_ISL6271A is not set
# CONFIG_REGULATOR_AD5398 is not set
# CONFIG_REGULATOR_TPS6524X is not set
CONFIG_MEDIA_SUPPORT=m
#
# Multimedia core support
#
# CONFIG_MEDIA_CONTROLLER is not set
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_DVB_CORE=m
CONFIG_VIDEO_MEDIA=m
#
# Multimedia drivers
#
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_RC_CORE=m
CONFIG_LIRC=m
CONFIG_RC_MAP=m
CONFIG_IR_NEC_DECODER=m
CONFIG_IR_RC5_DECODER=m
CONFIG_IR_RC6_DECODER=m
CONFIG_IR_JVC_DECODER=m
CONFIG_IR_SONY_DECODER=m
CONFIG_IR_RC5_SZ_DECODER=m
CONFIG_IR_LIRC_CODEC=m
CONFIG_IR_ENE=m
CONFIG_IR_IMON=m
CONFIG_IR_MCEUSB=m
# CONFIG_IR_ITE_CIR is not set
CONFIG_IR_NUVOTON=m
CONFIG_IR_STREAMZAP=m
# CONFIG_IR_WINBOND_CIR is not set
# CONFIG_RC_LOOPBACK is not set
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
# CONFIG_MEDIA_TUNER_CUSTOMISE is not set
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA827X=m
CONFIG_MEDIA_TUNER_TDA18271=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_MT2131=m
CONFIG_MEDIA_TUNER_QT1010=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_MEDIA_TUNER_MXL5005S=m
CONFIG_MEDIA_TUNER_MXL5007T=m
CONFIG_MEDIA_TUNER_MC44S803=m
CONFIG_MEDIA_TUNER_MAX2165=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_VMALLOC=m
CONFIG_VIDEOBUF_DVB=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_VMALLOC=m
CONFIG_VIDEO_CAPTURE_DRIVERS=y
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_IR_I2C=m
#
# Audio decoders
#
CONFIG_VIDEO_TVAUDIO=m
CONFIG_VIDEO_TDA7432=m
CONFIG_VIDEO_TDA9840=m
CONFIG_VIDEO_TEA6415C=m
CONFIG_VIDEO_TEA6420=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS5345=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_M52790=m
CONFIG_VIDEO_WM8775=m
CONFIG_VIDEO_WM8739=m
CONFIG_VIDEO_VP27SMPX=m
#
# RDS decoders
#
CONFIG_VIDEO_SAA6588=m
#
# Video decoders
#
CONFIG_VIDEO_BT819=m
CONFIG_VIDEO_BT856=m
CONFIG_VIDEO_BT866=m
CONFIG_VIDEO_KS0127=m
CONFIG_VIDEO_OV7670=m
CONFIG_VIDEO_MT9V011=m
CONFIG_VIDEO_SAA7110=m
CONFIG_VIDEO_SAA711X=m
CONFIG_VIDEO_SAA717X=m
CONFIG_VIDEO_TVP5150=m
CONFIG_VIDEO_VPX3220=m
#
# Video and audio decoders
#
CONFIG_VIDEO_CX25840=m
#
# MPEG video encoders
#
CONFIG_VIDEO_CX2341X=m
#
# Video encoders
#
CONFIG_VIDEO_SAA7127=m
CONFIG_VIDEO_SAA7185=m
CONFIG_VIDEO_ADV7170=m
CONFIG_VIDEO_ADV7175=m
#
# Video improvement chips
#
CONFIG_VIDEO_UPD64031A=m
CONFIG_VIDEO_UPD64083=m
CONFIG_VIDEO_VIVI=m
CONFIG_VIDEO_BT848=m
CONFIG_VIDEO_BT848_DVB=y
CONFIG_VIDEO_PMS=m
CONFIG_VIDEO_BWQCAM=m
CONFIG_VIDEO_CQCAM=m
CONFIG_VIDEO_W9966=m
CONFIG_VIDEO_CPIA2=m
CONFIG_VIDEO_ZORAN=m
CONFIG_VIDEO_ZORAN_DC30=m
CONFIG_VIDEO_ZORAN_ZR36060=m
CONFIG_VIDEO_ZORAN_BUZ=m
CONFIG_VIDEO_ZORAN_DC10=m
CONFIG_VIDEO_ZORAN_LML33=m
CONFIG_VIDEO_ZORAN_LML33R10=m
CONFIG_VIDEO_ZORAN_AVS6EYES=m
CONFIG_VIDEO_MEYE=m
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_VIDEO_SAA7134_DVB=m
CONFIG_VIDEO_MXB=m
CONFIG_VIDEO_HEXIUM_ORION=m
CONFIG_VIDEO_HEXIUM_GEMINI=m
# CONFIG_VIDEO_TIMBERDALE is not set
CONFIG_VIDEO_CX88=m
CONFIG_VIDEO_CX88_ALSA=m
CONFIG_VIDEO_CX88_BLACKBIRD=m
CONFIG_VIDEO_CX88_DVB=m
CONFIG_VIDEO_CX88_MPEG=m
CONFIG_VIDEO_CX88_VP3054=m
CONFIG_VIDEO_CX23885=m
# CONFIG_MEDIA_ALTERA_CI is not set
CONFIG_VIDEO_AU0828=m
CONFIG_VIDEO_IVTV=m
CONFIG_VIDEO_FB_IVTV=m
CONFIG_VIDEO_CX18=m
CONFIG_VIDEO_CX18_ALSA=m
CONFIG_VIDEO_SAA7164=m
CONFIG_VIDEO_CAFE_CCIC=m
# CONFIG_VIDEO_SR030PC30 is not set
# CONFIG_VIDEO_VIA_CAMERA is not set
# CONFIG_VIDEO_NOON010PC30 is not set
CONFIG_SOC_CAMERA=m
# CONFIG_SOC_CAMERA_IMX074 is not set
CONFIG_SOC_CAMERA_MT9M001=m
CONFIG_SOC_CAMERA_MT9M111=m
CONFIG_SOC_CAMERA_MT9T031=m
CONFIG_SOC_CAMERA_MT9T112=m
CONFIG_SOC_CAMERA_MT9V022=m
CONFIG_SOC_CAMERA_RJ54N1=m
CONFIG_SOC_CAMERA_TW9910=m
CONFIG_SOC_CAMERA_PLATFORM=m
# CONFIG_SOC_CAMERA_OV2640 is not set
# CONFIG_SOC_CAMERA_OV6650 is not set
CONFIG_SOC_CAMERA_OV772X=m
CONFIG_SOC_CAMERA_OV9640=m
# CONFIG_SOC_CAMERA_OV9740 is not set
CONFIG_V4L_USB_DRIVERS=y
CONFIG_USB_VIDEO_CLASS=m
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_USB_GSPCA=m
CONFIG_USB_M5602=m
CONFIG_USB_STV06XX=m
CONFIG_USB_GL860=m
CONFIG_USB_GSPCA_BENQ=m
CONFIG_USB_GSPCA_CONEX=m
CONFIG_USB_GSPCA_CPIA1=m
CONFIG_USB_GSPCA_ETOMS=m
CONFIG_USB_GSPCA_FINEPIX=m
CONFIG_USB_GSPCA_JEILINJ=m
CONFIG_USB_GSPCA_KONICA=m
CONFIG_USB_GSPCA_MARS=m
CONFIG_USB_GSPCA_MR97310A=m
# CONFIG_USB_GSPCA_NW80X is not set
CONFIG_USB_GSPCA_OV519=m
CONFIG_USB_GSPCA_OV534=m
CONFIG_USB_GSPCA_OV534_9=m
CONFIG_USB_GSPCA_PAC207=m
CONFIG_USB_GSPCA_PAC7302=m
CONFIG_USB_GSPCA_PAC7311=m
CONFIG_USB_GSPCA_SN9C2028=m
CONFIG_USB_GSPCA_SN9C20X=m
CONFIG_USB_GSPCA_SONIXB=m
CONFIG_USB_GSPCA_SONIXJ=m
CONFIG_USB_GSPCA_SPCA500=m
CONFIG_USB_GSPCA_SPCA501=m
CONFIG_USB_GSPCA_SPCA505=m
CONFIG_USB_GSPCA_SPCA506=m
CONFIG_USB_GSPCA_SPCA508=m
CONFIG_USB_GSPCA_SPCA561=m
CONFIG_USB_GSPCA_SPCA1528=m
CONFIG_USB_GSPCA_SQ905=m
CONFIG_USB_GSPCA_SQ905C=m
CONFIG_USB_GSPCA_SQ930X=m
CONFIG_USB_GSPCA_STK014=m
CONFIG_USB_GSPCA_STV0680=m
CONFIG_USB_GSPCA_SUNPLUS=m
CONFIG_USB_GSPCA_T613=m
CONFIG_USB_GSPCA_TV8532=m
CONFIG_USB_GSPCA_VC032X=m
# CONFIG_USB_GSPCA_VICAM is not set
CONFIG_USB_GSPCA_XIRLINK_CIT=m
CONFIG_USB_GSPCA_ZC3XX=m
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
# CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set
CONFIG_VIDEO_HDPVR=m
CONFIG_VIDEO_EM28XX=m
CONFIG_VIDEO_EM28XX_ALSA=m
CONFIG_VIDEO_EM28XX_DVB=m
CONFIG_VIDEO_TLG2300=m
CONFIG_VIDEO_CX231XX=m
CONFIG_VIDEO_CX231XX_RC=y
CONFIG_VIDEO_CX231XX_ALSA=m
CONFIG_VIDEO_CX231XX_DVB=m
CONFIG_VIDEO_USBVISION=m
CONFIG_USB_ET61X251=m
CONFIG_USB_SN9C102=m
CONFIG_USB_PWC=m
# CONFIG_USB_PWC_DEBUG is not set
CONFIG_USB_PWC_INPUT_EVDEV=y
CONFIG_USB_ZR364XX=m
CONFIG_USB_STKWEBCAM=m
CONFIG_USB_S2255=m
CONFIG_V4L_MEM2MEM_DRIVERS=y
# CONFIG_VIDEO_MEM2MEM_TESTDEV is not set
CONFIG_RADIO_ADAPTERS=y
CONFIG_RADIO_CADET=m
CONFIG_RADIO_RTRACK=m
CONFIG_RADIO_RTRACK2=m
CONFIG_RADIO_AZTECH=m
CONFIG_RADIO_GEMTEK=m
CONFIG_RADIO_MAXIRADIO=m
CONFIG_RADIO_MAESTRO=m
# CONFIG_RADIO_MIROPCM20 is not set
CONFIG_RADIO_SF16FMI=m
CONFIG_RADIO_SF16FMR2=m
CONFIG_RADIO_TERRATEC=m
CONFIG_RADIO_TRUST=m
CONFIG_RADIO_TYPHOON=m
CONFIG_RADIO_ZOLTRIX=m
CONFIG_I2C_SI4713=m
CONFIG_RADIO_SI4713=m
CONFIG_USB_DSBR=m
CONFIG_RADIO_SI470X=y
CONFIG_USB_SI470X=m
CONFIG_I2C_SI470X=m
CONFIG_USB_MR800=m
CONFIG_RADIO_TEA5764=m
CONFIG_RADIO_SAA7706H=m
CONFIG_RADIO_TEF6862=m
# CONFIG_RADIO_WL1273 is not set
#
# Texas Instruments WL128x FM driver (ST based)
#
# CONFIG_RADIO_WL128X is not set
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
CONFIG_DVB_CAPTURE_DRIVERS=y
#
# Supported SAA7146 based PCI Adapters
#
CONFIG_TTPCI_EEPROM=m
CONFIG_DVB_AV7110=m
CONFIG_DVB_AV7110_OSD=y
CONFIG_DVB_BUDGET_CORE=m
CONFIG_DVB_BUDGET=m
CONFIG_DVB_BUDGET_CI=m
CONFIG_DVB_BUDGET_AV=m
CONFIG_DVB_BUDGET_PATCH=m
#
# Supported USB Adapters
#
CONFIG_DVB_USB=m
# CONFIG_DVB_USB_DEBUG is not set
CONFIG_DVB_USB_A800=m
CONFIG_DVB_USB_DIBUSB_MB=m
CONFIG_DVB_USB_DIBUSB_MB_FAULTY=y
CONFIG_DVB_USB_DIBUSB_MC=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_UMT_010=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_M920X=m
CONFIG_DVB_USB_GL861=m
CONFIG_DVB_USB_AU6610=m
CONFIG_DVB_USB_DIGITV=m
CONFIG_DVB_USB_VP7045=m
CONFIG_DVB_USB_VP702X=m
CONFIG_DVB_USB_GP8PSK=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_TTUSB2=m
CONFIG_DVB_USB_DTT200U=m
CONFIG_DVB_USB_OPERA1=m
CONFIG_DVB_USB_AF9005=m
CONFIG_DVB_USB_AF9005_REMOTE=m
CONFIG_DVB_USB_DW2102=m
CONFIG_DVB_USB_CINERGY_T2=m
CONFIG_DVB_USB_ANYSEE=m
CONFIG_DVB_USB_DTV5100=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_USB_CE6230=m
CONFIG_DVB_USB_FRIIO=m
CONFIG_DVB_USB_EC168=m
CONFIG_DVB_USB_AZ6027=m
CONFIG_DVB_USB_LME2510=m
# CONFIG_DVB_USB_TECHNISAT_USB2 is not set
CONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_TTUSB_DEC=m
CONFIG_SMS_SIANO_MDTV=m
#
# Siano module components
#
CONFIG_SMS_USB_DRV=m
CONFIG_SMS_SDIO_DRV=m
#
# Supported FlexCopII (B2C2) Adapters
#
CONFIG_DVB_B2C2_FLEXCOP=m
CONFIG_DVB_B2C2_FLEXCOP_PCI=m
CONFIG_DVB_B2C2_FLEXCOP_USB=m
# CONFIG_DVB_B2C2_FLEXCOP_DEBUG is not set
#
# Supported BT878 Adapters
#
CONFIG_DVB_BT8XX=m
#
# Supported Pluto2 Adapters
#
CONFIG_DVB_PLUTO2=m
#
# Supported SDMC DM1105 Adapters
#
CONFIG_DVB_DM1105=m
#
# Supported FireWire (IEEE 1394) Adapters
#
CONFIG_DVB_FIREDTV=m
CONFIG_DVB_FIREDTV_INPUT=y
#
# Supported Earthsoft PT1 Adapters
#
CONFIG_DVB_PT1=m
#
# Supported Mantis Adapters
#
CONFIG_MANTIS_CORE=m
CONFIG_DVB_MANTIS=m
CONFIG_DVB_HOPPER=m
#
# Supported nGene Adapters
#
CONFIG_DVB_NGENE=m
#
# Supported DVB Frontends
#
# CONFIG_DVB_FE_CUSTOMISE is not set
#
# Multistandard (satellite) frontends
#
CONFIG_DVB_STB0899=m
CONFIG_DVB_STB6100=m
CONFIG_DVB_STV090x=m
CONFIG_DVB_STV6110x=m
#
# DVB-S (satellite) frontends
#
CONFIG_DVB_CX24110=m
CONFIG_DVB_CX24123=m
CONFIG_DVB_MT312=m
CONFIG_DVB_ZL10036=m
CONFIG_DVB_ZL10039=m
CONFIG_DVB_S5H1420=m
CONFIG_DVB_STV0288=m
CONFIG_DVB_STB6000=m
CONFIG_DVB_STV0299=m
CONFIG_DVB_STV6110=m
CONFIG_DVB_STV0900=m
CONFIG_DVB_TDA8083=m
CONFIG_DVB_TDA10086=m
CONFIG_DVB_TDA8261=m
CONFIG_DVB_VES1X93=m
CONFIG_DVB_TUNER_ITD1000=m
CONFIG_DVB_TUNER_CX24113=m
CONFIG_DVB_TDA826X=m
CONFIG_DVB_TUA6100=m
CONFIG_DVB_CX24116=m
CONFIG_DVB_SI21XX=m
CONFIG_DVB_DS3000=m
CONFIG_DVB_MB86A16=m
#
# DVB-T (terrestrial) frontends
#
CONFIG_DVB_SP8870=m
CONFIG_DVB_SP887X=m
CONFIG_DVB_CX22700=m
CONFIG_DVB_CX22702=m
CONFIG_DVB_L64781=m
CONFIG_DVB_TDA1004X=m
CONFIG_DVB_NXT6000=m
CONFIG_DVB_MT352=m
CONFIG_DVB_ZL10353=m
CONFIG_DVB_DIB3000MB=m
CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_TDA10048=m
CONFIG_DVB_AF9013=m
CONFIG_DVB_EC100=m
#
# DVB-C (cable) frontends
#
CONFIG_DVB_VES1820=m
CONFIG_DVB_TDA10021=m
CONFIG_DVB_TDA10023=m
CONFIG_DVB_STV0297=m
#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
CONFIG_DVB_NXT200X=m
CONFIG_DVB_OR51211=m
CONFIG_DVB_OR51132=m
CONFIG_DVB_BCM3510=m
CONFIG_DVB_LGDT330X=m
CONFIG_DVB_LGDT3305=m
CONFIG_DVB_S5H1409=m
CONFIG_DVB_AU8522=m
CONFIG_DVB_S5H1411=m
#
# ISDB-T (terrestrial) frontends
#
CONFIG_DVB_S921=m
CONFIG_DVB_DIB8000=m
CONFIG_DVB_MB86A20S=m
#
# Digital terrestrial only tuners/PLL
#
CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m
CONFIG_DVB_TUNER_DIB0090=m
#
# SEC control devices for DVB-S
#
CONFIG_DVB_LNBP21=m
CONFIG_DVB_ISL6405=m
CONFIG_DVB_ISL6421=m
CONFIG_DVB_ISL6423=m
CONFIG_DVB_LGS8GXX=m
CONFIG_DVB_ATBM8830=m
CONFIG_DVB_TDA665x=m
CONFIG_DVB_IX2505V=m
#
# Tools to develop new frontends
#
# CONFIG_DVB_DUMMY_FE is not set
#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_ALI=y
CONFIG_AGP_ATI=y
CONFIG_AGP_AMD=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_NVIDIA=y
CONFIG_AGP_SIS=y
CONFIG_AGP_SWORKS=y
CONFIG_AGP_VIA=y
CONFIG_AGP_EFFICEON=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_VGA_SWITCHEROO=y
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m
CONFIG_DRM_TDFX=m
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_RADEON_KMS=y
CONFIG_DRM_I810=m
CONFIG_DRM_I915=m
# CONFIG_DRM_I915_KMS is not set
CONFIG_DRM_MGA=m
CONFIG_DRM_SIS=m
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
# CONFIG_STUB_POULSBO is not set
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_DDC=m
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=m
# CONFIG_FB_WMT_GE_ROPS is not set
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_HECUBA=m
CONFIG_FB_SVGALIB=m
# CONFIG_FB_MACMODES is not set
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
#
# Frame buffer hardware drivers
#
CONFIG_FB_CIRRUS=m
CONFIG_FB_PM2=m
CONFIG_FB_PM2_FIFO_DISCONNECT=y
CONFIG_FB_CYBER2000=m
CONFIG_FB_CYBER2000_DDC=y
CONFIG_FB_ARC=m
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
CONFIG_FB_UVESA=m
CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_FB_N411=m
CONFIG_FB_HGA=m
CONFIG_FB_S1D13XXX=m
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
CONFIG_FB_NVIDIA_BACKLIGHT=y
# CONFIG_FB_RIVA is not set
CONFIG_FB_I810=m
# CONFIG_FB_I810_GTF is not set
CONFIG_FB_LE80578=m
CONFIG_FB_CARILLO_RANCH=m
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
CONFIG_FB_MATROX_MYSTIQUE=y
CONFIG_FB_MATROX_G=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
CONFIG_FB_RADEON_BACKLIGHT=y
# CONFIG_FB_RADEON_DEBUG is not set
CONFIG_FB_ATY128=m
CONFIG_FB_ATY128_BACKLIGHT=y
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
CONFIG_FB_ATY_GENERIC_LCD=y
CONFIG_FB_ATY_GX=y
CONFIG_FB_ATY_BACKLIGHT=y
CONFIG_FB_S3=m
CONFIG_FB_SAVAGE=m
CONFIG_FB_SAVAGE_I2C=y
# CONFIG_FB_SAVAGE_ACCEL is not set
CONFIG_FB_SIS=m
CONFIG_FB_SIS_300=y
CONFIG_FB_SIS_315=y
CONFIG_FB_VIA=m
# CONFIG_FB_VIA_DIRECT_PROCFS is not set
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=m
CONFIG_FB_3DFX=m
# CONFIG_FB_3DFX_ACCEL is not set
CONFIG_FB_3DFX_I2C=y
CONFIG_FB_VOODOO1=m
CONFIG_FB_VT8623=m
CONFIG_FB_TRIDENT=m
CONFIG_FB_ARK=m
CONFIG_FB_PM3=m
# CONFIG_FB_CARMINE is not set
CONFIG_FB_GEODE=y
CONFIG_FB_GEODE_LX=m
CONFIG_FB_GEODE_GX=m
CONFIG_FB_GEODE_GX1=m
# CONFIG_FB_TMIO is not set
CONFIG_FB_SM501=m
# CONFIG_FB_UDL is not set
CONFIG_FB_VIRTUAL=m
CONFIG_FB_METRONOME=m
CONFIG_FB_MB862XX=m
CONFIG_FB_MB862XX_PCI_GDC=y
# CONFIG_FB_BROADSHEET is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
# CONFIG_LCD_CLASS_DEVICE is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
# CONFIG_BACKLIGHT_GENERIC is not set
CONFIG_BACKLIGHT_PROGEAR=m
# CONFIG_BACKLIGHT_APPLE is not set
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
# CONFIG_BACKLIGHT_PCF50633 is not set
#
# Display device support
#
CONFIG_DISPLAY_SUPPORT=m
#
# Display hardware drivers
#
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
CONFIG_MDA_CONSOLE=m
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_LOGO is not set
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
# CONFIG_SOUND_OSS_CORE_PRECLAIM is not set
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
# CONFIG_SND_SEQUENCER_OSS is not set
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_OPL3_LIB_SEQ=m
CONFIG_SND_OPL4_LIB_SEQ=m
CONFIG_SND_SBAWE_SEQ=m
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_OPL4_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_PCSP=m
CONFIG_SND_DUMMY=m
# CONFIG_SND_ALOOP is not set
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
CONFIG_SND_MTS64=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_PORTMAN2X4=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_WSS_LIB=m
CONFIG_SND_SB_COMMON=m
CONFIG_SND_SB8_DSP=m
CONFIG_SND_SB16_DSP=m
CONFIG_SND_ISA=y
CONFIG_SND_ADLIB=m
CONFIG_SND_AD1816A=m
CONFIG_SND_AD1848=m
CONFIG_SND_ALS100=m
CONFIG_SND_AZT1605=m
CONFIG_SND_AZT2316=m
CONFIG_SND_AZT2320=m
CONFIG_SND_CMI8330=m
CONFIG_SND_CS4231=m
CONFIG_SND_CS4236=m
CONFIG_SND_ES1688=m
CONFIG_SND_ES18XX=m
CONFIG_SND_SC6000=m
CONFIG_SND_GUSCLASSIC=m
CONFIG_SND_GUSEXTREME=m
CONFIG_SND_GUSMAX=m
CONFIG_SND_INTERWAVE=m
CONFIG_SND_INTERWAVE_STB=m
CONFIG_SND_JAZZ16=m
CONFIG_SND_OPL3SA2=m
CONFIG_SND_OPTI92X_AD1848=m
CONFIG_SND_OPTI92X_CS4231=m
CONFIG_SND_OPTI93X=m
CONFIG_SND_MIRO=m
CONFIG_SND_SB8=m
CONFIG_SND_SB16=m
CONFIG_SND_SBAWE=m
CONFIG_SND_SB16_CSP=y
CONFIG_SND_SSCAPE=m
CONFIG_SND_WAVEFRONT=m
CONFIG_SND_MSND_PINNACLE=m
CONFIG_SND_MSND_CLASSIC=m
CONFIG_SND_PCI=y
CONFIG_SND_AD1889=m
CONFIG_SND_ALS300=m
CONFIG_SND_ALS4000=m
CONFIG_SND_ALI5451=m
CONFIG_SND_ASIHPI=m
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
# CONFIG_SND_AW2 is not set
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_OXYGEN_LIB=m
CONFIG_SND_OXYGEN=m
CONFIG_SND_CS4281=m
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CS5530=m
CONFIG_SND_CS5535AUDIO=m
CONFIG_SND_CTXFI=m
CONFIG_SND_DARLA20=m
CONFIG_SND_GINA20=m
CONFIG_SND_LAYLA20=m
CONFIG_SND_DARLA24=m
CONFIG_SND_GINA24=m
CONFIG_SND_LAYLA24=m
CONFIG_SND_MONA=m
CONFIG_SND_MIA=m
CONFIG_SND_ECHO3G=m
CONFIG_SND_INDIGO=m
CONFIG_SND_INDIGOIO=m
CONFIG_SND_INDIGODJ=m
CONFIG_SND_INDIGOIOX=m
CONFIG_SND_INDIGODJX=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
CONFIG_SND_ES1968_INPUT=y
CONFIG_SND_FM801=m
CONFIG_SND_FM801_TEA575X_BOOL=y
CONFIG_SND_FM801_TEA575X=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_HWDEP=y
CONFIG_SND_HDA_RECONFIG=y
CONFIG_SND_HDA_INPUT_BEEP=y
CONFIG_SND_HDA_INPUT_BEEP_MODE=1
CONFIG_SND_HDA_INPUT_JACK=y
CONFIG_SND_HDA_PATCH_LOADER=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CA0110=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
CONFIG_SND_HDSP=m
CONFIG_SND_HDSPM=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_KORG1212=m
CONFIG_SND_LX6464ES=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_MAESTRO3_INPUT=y
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_PCXHR=m
CONFIG_SND_RIPTIDE=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_SIS7019=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_TRIDENT=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
CONFIG_SND_VIRTUOSO=m
CONFIG_SND_VX222=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_SPI=y
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=m
CONFIG_SND_USB_UA101=m
CONFIG_SND_USB_USX2Y=m
CONFIG_SND_USB_CAIAQ=m
CONFIG_SND_USB_CAIAQ_INPUT=y
CONFIG_SND_USB_US122L=m
# CONFIG_SND_USB_6FIRE is not set
CONFIG_SND_FIREWIRE=y
# CONFIG_SND_FIREWIRE_SPEAKERS is not set
CONFIG_SND_PCMCIA=y
CONFIG_SND_VXPOCKET=m
CONFIG_SND_PDAUDIOCF=m
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
CONFIG_HID_SUPPORT=y
CONFIG_HID=m
CONFIG_HIDRAW=y
#
# USB Input Devices
#
CONFIG_USB_HID=m
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y
#
# Special HID drivers
#
CONFIG_HID_3M_PCT=m
CONFIG_HID_A4TECH=m
# CONFIG_HID_ACRUX is not set
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CANDO=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_PRODIKEYS=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_DRAGONRISE=m
CONFIG_DRAGONRISE_FF=y
# CONFIG_HID_EMS_FF is not set
CONFIG_HID_ELECOM=m
CONFIG_HID_EZKEY=m
# CONFIG_HID_KEYTOUCH is not set
CONFIG_HID_KYE=m
CONFIG_HID_UCLOGIC=m
CONFIG_HID_WALTOP=m
CONFIG_HID_GYRATION=m
CONFIG_HID_TWINHAN=m
CONFIG_HID_KENSINGTON=m
# CONFIG_HID_LCPOWER is not set
CONFIG_HID_LOGITECH=m
CONFIG_LOGITECH_FF=y
CONFIG_LOGIRUMBLEPAD2_FF=y
CONFIG_LOGIG940_FF=y
CONFIG_LOGIWII_FF=y
CONFIG_HID_MAGICMOUSE=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MOSART=m
CONFIG_HID_MONTEREY=m
# CONFIG_HID_MULTITOUCH is not set
CONFIG_HID_NTRIG=m
CONFIG_HID_ORTEK=m
CONFIG_HID_PANTHERLORD=m
CONFIG_PANTHERLORD_FF=y
CONFIG_HID_PETALYNX=m
CONFIG_HID_PICOLCD=m
CONFIG_HID_PICOLCD_FB=y
CONFIG_HID_PICOLCD_BACKLIGHT=y
CONFIG_HID_PICOLCD_LEDS=y
CONFIG_HID_QUANTA=m
CONFIG_HID_ROCCAT=m
CONFIG_HID_ROCCAT_COMMON=m
# CONFIG_HID_ROCCAT_ARVO is not set
CONFIG_HID_ROCCAT_KONE=m
# CONFIG_HID_ROCCAT_KONEPLUS is not set
# CONFIG_HID_ROCCAT_KOVAPLUS is not set
CONFIG_HID_ROCCAT_PYRA=m
CONFIG_HID_SAMSUNG=m
CONFIG_HID_SONY=m
CONFIG_HID_STANTUM=m
CONFIG_HID_SUNPLUS=m
CONFIG_HID_GREENASIA=m
CONFIG_GREENASIA_FF=y
CONFIG_HID_SMARTJOYPLUS=m
CONFIG_SMARTJOYPLUS_FF=y
CONFIG_HID_TOPSEED=m
CONFIG_HID_THRUSTMASTER=m
CONFIG_THRUSTMASTER_FF=y
CONFIG_HID_WACOM=m
CONFIG_HID_WACOM_POWER_SUPPLY=y
CONFIG_HID_ZEROPLUS=m
CONFIG_ZEROPLUS_FF=y
CONFIG_HID_ZYDACRON=m
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_DEVICE_CLASS is not set
CONFIG_USB_DYNAMIC_MINORS=y
CONFIG_USB_SUSPEND=y
# CONFIG_USB_OTG is not set
CONFIG_USB_MON=m
CONFIG_USB_WUSB=m
CONFIG_USB_WUSB_CBAF=m
# CONFIG_USB_WUSB_CBAF_DEBUG is not set
#
# USB Host Controller Drivers
#
CONFIG_USB_C67X00_HCD=m
CONFIG_USB_XHCI_HCD=m
# CONFIG_USB_XHCI_HCD_DEBUGGING is not set
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_OXU210HP_HCD is not set
CONFIG_USB_ISP116X_HCD=m
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_HCD_SSB is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_U132_HCD=m
CONFIG_USB_SL811_HCD=m
CONFIG_USB_SL811_CS=m
CONFIG_USB_R8A66597_HCD=m
CONFIG_USB_WHCI_HCD=m
CONFIG_USB_HWA_HCD=m
#
# Enable Host or Gadget support to see Inventra options
#
#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m
#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#
#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_REALTEK is not set
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
CONFIG_USB_STORAGE_ONETOUCH=m
CONFIG_USB_STORAGE_KARMA=m
CONFIG_USB_STORAGE_CYPRESS_ATACB=m
# CONFIG_USB_STORAGE_ENE_UB6250 is not set
CONFIG_USB_UAS=m
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m
#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_CH341=m
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CP210X=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
CONFIG_USB_SERIAL_IUU=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_MOS7720=m
CONFIG_USB_SERIAL_MOS7715_PARPORT=y
CONFIG_USB_SERIAL_MOS7840=m
CONFIG_USB_SERIAL_MOTOROLA=m
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_OTI6858=m
CONFIG_USB_SERIAL_QCAUX=m
CONFIG_USB_SERIAL_QUALCOMM=m
CONFIG_USB_SERIAL_SPCP8X5=m
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
# CONFIG_USB_SERIAL_SAFE_PADDED is not set
CONFIG_USB_SERIAL_SAMBA=m
CONFIG_USB_SERIAL_SIEMENS_MPI=m
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
CONFIG_USB_SERIAL_SYMBOL=m
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_SERIAL_OPTICON=m
CONFIG_USB_SERIAL_VIVOPAY_SERIAL=m
CONFIG_USB_SERIAL_ZIO=m
CONFIG_USB_SERIAL_SSU100=m
CONFIG_USB_SERIAL_DEBUG=m
#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_ADUTUX=m
CONFIG_USB_SEVSEG=m
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
CONFIG_USB_CYPRESS_CY7C63=m
CONFIG_USB_CYTHERM=m
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
CONFIG_USB_TRANCEVIBRATOR=m
CONFIG_USB_IOWARRIOR=m
CONFIG_USB_TEST=m
CONFIG_USB_ISIGHTFW=m
# CONFIG_USB_YUREX is not set
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m
# CONFIG_USB_GADGET is not set
#
# OTG and related infrastructure
#
CONFIG_USB_OTG_UTILS=y
# CONFIG_USB_GPIO_VBUS is not set
CONFIG_NOP_USB_XCEIV=m
CONFIG_UWB=m
CONFIG_UWB_HWA=m
CONFIG_UWB_WHCI=m
CONFIG_UWB_I1480U=m
CONFIG_MMC=m
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set
# CONFIG_MMC_CLKGATE is not set
#
# MMC/SD/SDIO Card Drivers
#
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y
CONFIG_SDIO_UART=m
# CONFIG_MMC_TEST is not set
#
# MMC/SD/SDIO Host Controller Drivers
#
CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_RICOH_MMC=y
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_MMC_WBSD=m
CONFIG_MMC_TIFM_SD=m
CONFIG_MMC_SDRICOH_CS=m
CONFIG_MMC_CB710=m
CONFIG_MMC_VIA_SDMMC=m
# CONFIG_MMC_USHC is not set
CONFIG_MEMSTICK=m
# CONFIG_MEMSTICK_DEBUG is not set
#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
CONFIG_MSPRO_BLOCK=m
#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=m
CONFIG_MEMSTICK_JMICRON_38X=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
#
# LED drivers
#
# CONFIG_LEDS_LM3530 is not set
CONFIG_LEDS_NET48XX=m
CONFIG_LEDS_WRAP=m
CONFIG_LEDS_ALIX2=m
CONFIG_LEDS_PCA9532=m
# CONFIG_LEDS_GPIO is not set
CONFIG_LEDS_LP3944=m
# CONFIG_LEDS_LP5521 is not set
# CONFIG_LEDS_LP5523 is not set
CONFIG_LEDS_CLEVO_MAIL=m
CONFIG_LEDS_PCA955X=m
CONFIG_LEDS_DAC124S085=m
CONFIG_LEDS_REGULATOR=m
CONFIG_LEDS_BD2802=m
CONFIG_LEDS_INTEL_SS4200=m
CONFIG_LEDS_LT3593=m
CONFIG_LEDS_DELL_NETBOOKS=m
CONFIG_LEDS_TRIGGERS=y
#
# LED Triggers
#
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_IDE_DISK=y
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
CONFIG_LEDS_TRIGGER_BACKLIGHT=m
# CONFIG_LEDS_TRIGGER_GPIO is not set
CONFIG_LEDS_TRIGGER_DEFAULT_ON=m
#
# iptables trigger is under Netfilter config (LED target)
#
# CONFIG_NFC_DEVICES is not set
CONFIG_ACCESSIBILITY=y
CONFIG_A11Y_BRAILLE_CONSOLE=y
CONFIG_INFINIBAND=m
CONFIG_INFINIBAND_USER_MAD=m
CONFIG_INFINIBAND_USER_ACCESS=m
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_MTHCA=m
CONFIG_INFINIBAND_MTHCA_DEBUG=y
CONFIG_INFINIBAND_AMSO1100=m
# CONFIG_INFINIBAND_AMSO1100_DEBUG is not set
CONFIG_INFINIBAND_CXGB3=m
# CONFIG_INFINIBAND_CXGB3_DEBUG is not set
CONFIG_INFINIBAND_CXGB4=m
CONFIG_MLX4_INFINIBAND=m
CONFIG_INFINIBAND_NES=m
# CONFIG_INFINIBAND_NES_DEBUG is not set
CONFIG_INFINIBAND_IPOIB=m
CONFIG_INFINIBAND_IPOIB_CM=y
CONFIG_INFINIBAND_IPOIB_DEBUG=y
# CONFIG_INFINIBAND_IPOIB_DEBUG_DATA is not set
CONFIG_INFINIBAND_SRP=m
CONFIG_INFINIBAND_ISER=m
CONFIG_EDAC=y
#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=m
# CONFIG_EDAC_MCE_INJ is not set
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_MCE=y
CONFIG_EDAC_AMD76X=m
CONFIG_EDAC_E7XXX=m
CONFIG_EDAC_E752X=m
CONFIG_EDAC_I82875P=m
CONFIG_EDAC_I82975X=m
CONFIG_EDAC_I3000=m
CONFIG_EDAC_I3200=m
CONFIG_EDAC_X38=m
CONFIG_EDAC_I5400=m
CONFIG_EDAC_I7CORE=m
CONFIG_EDAC_I82860=m
CONFIG_EDAC_R82600=m
CONFIG_EDAC_I5000=m
CONFIG_EDAC_I5100=m
# CONFIG_EDAC_I7300 is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
CONFIG_RTC_DRV_DS1374=m
CONFIG_RTC_DRV_DS1672=m
# CONFIG_RTC_DRV_DS3232 is not set
CONFIG_RTC_DRV_MAX6900=m
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
# CONFIG_RTC_DRV_ISL12022 is not set
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
CONFIG_RTC_DRV_M41T80=m
# CONFIG_RTC_DRV_M41T80_WDT is not set
CONFIG_RTC_DRV_BQ32K=m
CONFIG_RTC_DRV_S35390A=m
CONFIG_RTC_DRV_FM3130=m
CONFIG_RTC_DRV_RX8581=m
CONFIG_RTC_DRV_RX8025=m
#
# SPI RTC drivers
#
CONFIG_RTC_DRV_M41T94=m
CONFIG_RTC_DRV_DS1305=m
CONFIG_RTC_DRV_DS1390=m
CONFIG_RTC_DRV_MAX6902=m
CONFIG_RTC_DRV_R9701=m
CONFIG_RTC_DRV_RS5C348=m
CONFIG_RTC_DRV_DS3234=m
CONFIG_RTC_DRV_PCF2123=m
#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
CONFIG_RTC_DRV_DS1286=m
CONFIG_RTC_DRV_DS1511=m
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1742=m
CONFIG_RTC_DRV_STK17TA8=m
CONFIG_RTC_DRV_M48T86=m
CONFIG_RTC_DRV_M48T35=m
CONFIG_RTC_DRV_M48T59=m
CONFIG_RTC_DRV_MSM6242=m
CONFIG_RTC_DRV_BQ4802=m
CONFIG_RTC_DRV_RP5C01=m
CONFIG_RTC_DRV_V3020=m
CONFIG_RTC_DRV_PCF50633=m
#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y
# CONFIG_DMADEVICES_DEBUG is not set
#
# DMA Devices
#
# CONFIG_INTEL_MID_DMAC is not set
CONFIG_INTEL_IOATDMA=m
# CONFIG_TIMB_DMA is not set
# CONFIG_PCH_DMA is not set
CONFIG_DMA_ENGINE=y
#
# DMA Clients
#
CONFIG_NET_DMA=y
CONFIG_ASYNC_TX_DMA=y
# CONFIG_DMATEST is not set
CONFIG_DCA=m
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=m
CONFIG_UIO_CIF=m
CONFIG_UIO_PDRV=m
CONFIG_UIO_PDRV_GENIRQ=m
CONFIG_UIO_AEC=m
CONFIG_UIO_SERCOS3=m
CONFIG_UIO_PCI_GENERIC=m
CONFIG_UIO_NETX=m
CONFIG_STAGING=y
# CONFIG_STAGING_EXCLUDE_BUILD is not set
# CONFIG_STALLION is not set
# CONFIG_ISTALLION is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_RISCOM8 is not set
# CONFIG_SPECIALIX is not set
CONFIG_COMPUTONE=m
CONFIG_ET131X=m
# CONFIG_ET131X_DEBUG is not set
# CONFIG_SLICOSS is not set
# CONFIG_VIDEO_GO7007 is not set
# CONFIG_VIDEO_CX25821 is not set
CONFIG_VIDEO_TM6000=m
CONFIG_VIDEO_TM6000_ALSA=m
CONFIG_VIDEO_TM6000_DVB=m
# CONFIG_DVB_CXD2099 is not set
CONFIG_USB_IP_COMMON=m
CONFIG_USB_IP_VHCI_HCD=m
CONFIG_USB_IP_HOST=m
# CONFIG_USB_IP_DEBUG_ENABLE is not set
# CONFIG_W35UND is not set
CONFIG_PRISM2_USB=m
# CONFIG_ECHO is not set
CONFIG_BRCM80211=m
# CONFIG_BRCMSMAC is not set
# CONFIG_BRCMFMAC is not set
# CONFIG_BRCMDBG is not set
# CONFIG_RT2860 is not set
# CONFIG_RT2870 is not set
CONFIG_COMEDI=m
# CONFIG_COMEDI_DEBUG is not set
# CONFIG_COMEDI_MISC_DRIVERS is not set
# CONFIG_COMEDI_ISA_DRIVERS is not set
CONFIG_COMEDI_PCI_DRIVERS=m
# CONFIG_COMEDI_ADDI_APCI_035 is not set
# CONFIG_COMEDI_ADDI_APCI_1032 is not set
# CONFIG_COMEDI_ADDI_APCI_1500 is not set
# CONFIG_COMEDI_ADDI_APCI_1516 is not set
# CONFIG_COMEDI_ADDI_APCI_1564 is not set
# CONFIG_COMEDI_ADDI_APCI_16XX is not set
# CONFIG_COMEDI_ADDI_APCI_2016 is not set
# CONFIG_COMEDI_ADDI_APCI_2032 is not set
# CONFIG_COMEDI_ADDI_APCI_2200 is not set
# CONFIG_COMEDI_ADDI_APCI_3001 is not set
# CONFIG_COMEDI_ADDI_APCI_3120 is not set
# CONFIG_COMEDI_ADDI_APCI_3501 is not set
# CONFIG_COMEDI_ADDI_APCI_3XXX is not set
# CONFIG_COMEDI_ADL_PCI6208 is not set
# CONFIG_COMEDI_ADL_PCI7230 is not set
# CONFIG_COMEDI_ADL_PCI7296 is not set
# CONFIG_COMEDI_ADL_PCI7432 is not set
# CONFIG_COMEDI_ADL_PCI8164 is not set
# CONFIG_COMEDI_ADL_PCI9111 is not set
# CONFIG_COMEDI_ADL_PCI9118 is not set
# CONFIG_COMEDI_ADV_PCI1710 is not set
# CONFIG_COMEDI_ADV_PCI1723 is not set
# CONFIG_COMEDI_ADV_PCI_DIO is not set
# CONFIG_COMEDI_AMPLC_DIO200 is not set
# CONFIG_COMEDI_AMPLC_PC236 is not set
# CONFIG_COMEDI_AMPLC_PC263 is not set
# CONFIG_COMEDI_AMPLC_PCI224 is not set
# CONFIG_COMEDI_AMPLC_PCI230 is not set
# CONFIG_COMEDI_CONTEC_PCI_DIO is not set
# CONFIG_COMEDI_DT3000 is not set
# CONFIG_COMEDI_UNIOXX5 is not set
# CONFIG_COMEDI_GSC_HPDI is not set
# CONFIG_COMEDI_ICP_MULTI is not set
# CONFIG_COMEDI_II_PCI20KC is not set
# CONFIG_COMEDI_DAQBOARD2000 is not set
# CONFIG_COMEDI_JR3_PCI is not set
# CONFIG_COMEDI_KE_COUNTER is not set
# CONFIG_COMEDI_CB_PCIDAS64 is not set
# CONFIG_COMEDI_CB_PCIDAS is not set
# CONFIG_COMEDI_CB_PCIDDA is not set
# CONFIG_COMEDI_CB_PCIDIO is not set
# CONFIG_COMEDI_CB_PCIMDAS is not set
# CONFIG_COMEDI_CB_PCIMDDA is not set
# CONFIG_COMEDI_ME4000 is not set
# CONFIG_COMEDI_ME_DAQ is not set
# CONFIG_COMEDI_RTD520 is not set
# CONFIG_COMEDI_S526 is not set
# CONFIG_COMEDI_S626 is not set
# CONFIG_COMEDI_SSV_DNP is not set
CONFIG_COMEDI_PCMCIA_DRIVERS=m
# CONFIG_COMEDI_CB_DAS16_CS is not set
# CONFIG_COMEDI_DAS08_CS is not set
# CONFIG_COMEDI_QUATECH_DAQP_CS is not set
CONFIG_COMEDI_USB_DRIVERS=m
# CONFIG_COMEDI_DT9812 is not set
# CONFIG_COMEDI_USBDUX is not set
# CONFIG_COMEDI_USBDUXFAST is not set
# CONFIG_COMEDI_VMK80XX is not set
# CONFIG_COMEDI_NI_COMMON is not set
# CONFIG_COMEDI_8255 is not set
# CONFIG_COMEDI_DAS08 is not set
# CONFIG_COMEDI_FC is not set
# CONFIG_ASUS_OLED is not set
# CONFIG_PANEL is not set
CONFIG_R8187SE=m
CONFIG_RTL8192U=m
CONFIG_RTL8192E=m
CONFIG_R8712U=m
# CONFIG_R8712_AP is not set
# CONFIG_RTS_PSTOR is not set
# CONFIG_TRANZPORT is not set
# CONFIG_POHMELFS is not set
# CONFIG_IDE_PHISON is not set
# CONFIG_LINE6_USB is not set
CONFIG_DRM_VMWGFX=m
CONFIG_DRM_NOUVEAU=m
CONFIG_DRM_NOUVEAU_BACKLIGHT=y
# CONFIG_DRM_NOUVEAU_DEBUG is not set
#
# I2C encoder or helper chips
#
CONFIG_DRM_I2C_CH7006=m
CONFIG_DRM_I2C_SIL164=m
# CONFIG_USB_SERIAL_QUATECH2 is not set
# CONFIG_USB_SERIAL_QUATECH_USB2 is not set
# CONFIG_VT6655 is not set
# CONFIG_VT6656 is not set
# CONFIG_HYPERV is not set
# CONFIG_VME_BUS is not set
# CONFIG_DX_SEP is not set
# CONFIG_IIO is not set
CONFIG_CS5535_GPIO=m
CONFIG_XVMALLOC=y
CONFIG_ZRAM=m
# CONFIG_ZRAM_DEBUG is not set
# CONFIG_ZCACHE is not set
# CONFIG_WLAGS49_H2 is not set
# CONFIG_WLAGS49_H25 is not set
CONFIG_SAMSUNG_LAPTOP=m
# CONFIG_FB_SM7XX is not set
# CONFIG_VIDEO_DT3155 is not set
# CONFIG_CRYSTALHD is not set
# CONFIG_CXT1E1 is not set
# CONFIG_FB_XGI is not set
CONFIG_LIRC_STAGING=y
CONFIG_LIRC_BT829=m
CONFIG_LIRC_IGORPLUGUSB=m
CONFIG_LIRC_IMON=m
# CONFIG_LIRC_PARALLEL is not set
CONFIG_LIRC_SASEM=m
CONFIG_LIRC_SERIAL=m
CONFIG_LIRC_SERIAL_TRANSMITTER=y
CONFIG_LIRC_SIR=m
CONFIG_LIRC_TTUSBIR=m
CONFIG_LIRC_ZILOG=m
# CONFIG_EASYCAP is not set
# CONFIG_SOLO6X10 is not set
# CONFIG_ACPI_QUICKSTART is not set
CONFIG_MACH_NO_WESTBRIDGE=y
# CONFIG_SBE_2T3E3 is not set
# CONFIG_ATH6K_LEGACY is not set
# CONFIG_USB_ENESTORAGE is not set
# CONFIG_BCM_WIMAX is not set
# CONFIG_FT1000 is not set
#
# Speakup console speech
#
CONFIG_SPEAKUP=m
CONFIG_SPEAKUP_SYNTH_ACNTSA=m
CONFIG_SPEAKUP_SYNTH_ACNTPC=m
CONFIG_SPEAKUP_SYNTH_APOLLO=m
CONFIG_SPEAKUP_SYNTH_AUDPTR=m
CONFIG_SPEAKUP_SYNTH_BNS=m
CONFIG_SPEAKUP_SYNTH_DECTLK=m
CONFIG_SPEAKUP_SYNTH_DECEXT=m
# CONFIG_SPEAKUP_SYNTH_DECPC is not set
CONFIG_SPEAKUP_SYNTH_DTLK=m
CONFIG_SPEAKUP_SYNTH_KEYPC=m
CONFIG_SPEAKUP_SYNTH_LTLK=m
CONFIG_SPEAKUP_SYNTH_SOFT=m
CONFIG_SPEAKUP_SYNTH_SPKOUT=m
CONFIG_SPEAKUP_SYNTH_TXPRT=m
CONFIG_SPEAKUP_SYNTH_DUMMY=m
# CONFIG_TOUCHSCREEN_CLEARPAD_TM1217 is not set
# CONFIG_TOUCHSCREEN_SYNAPTICS_I2C_RMI4 is not set
# CONFIG_DRM_PSB is not set
#
# Altera FPGA firmware download module
#
# CONFIG_ALTERA_STAPL is not set
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_ACER_WMI=m
CONFIG_ACERHDF=m
CONFIG_ASUS_LAPTOP=m
CONFIG_DELL_LAPTOP=m
CONFIG_DELL_WMI=m
# CONFIG_DELL_WMI_AIO is not set
CONFIG_FUJITSU_LAPTOP=m
# CONFIG_FUJITSU_LAPTOP_DEBUG is not set
CONFIG_TC1100_WMI=m
CONFIG_HP_ACCEL=m
CONFIG_HP_WMI=m
CONFIG_MSI_LAPTOP=m
CONFIG_PANASONIC_LAPTOP=m
CONFIG_COMPAL_LAPTOP=m
CONFIG_SONY_LAPTOP=m
CONFIG_SONYPI_COMPAT=y
CONFIG_IDEAPAD_LAPTOP=m
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
# CONFIG_THINKPAD_ACPI_DEBUGFACILITIES is not set
# CONFIG_THINKPAD_ACPI_DEBUG is not set
# CONFIG_THINKPAD_ACPI_UNSAFE_LEDS is not set
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_THINKPAD_ACPI_HOTKEY_POLL=y
CONFIG_SENSORS_HDAPS=m
# CONFIG_INTEL_MENLOW is not set
CONFIG_EEEPC_LAPTOP=m
# CONFIG_ASUS_WMI is not set
CONFIG_ACPI_WMI=m
CONFIG_MSI_WMI=m
CONFIG_ACPI_ASUS=m
CONFIG_TOPSTAR_LAPTOP=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_TOSHIBA_BT_RFKILL=m
CONFIG_ACPI_CMPC=m
# CONFIG_INTEL_IPS is not set
# CONFIG_IBM_RTL is not set
# CONFIG_XO15_EBOOK is not set
#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_EFI_VARS=m
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
CONFIG_DMIID=y
# CONFIG_DMI_SYSFS is not set
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
# CONFIG_SIGMA is not set
#
# File systems
#
CONFIG_EXT2_FS=m
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=m
CONFIG_EXT3_DEFAULTS_TO_ORDERED=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=m
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD=m
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=m
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
# CONFIG_XFS_DEBUG is not set
CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_DLM=y
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
CONFIG_OCFS2_FS_STATS=y
CONFIG_OCFS2_DEBUG_MASKLOG=y
# CONFIG_OCFS2_DEBUG_FS is not set
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_NILFS2_FS=m
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
# CONFIG_FANOTIFY_ACCESS_PERMISSIONS is not set
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=m
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_OVERLAYFS_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y
#
# Caches
#
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
CONFIG_CACHEFILES=m
# CONFIG_CACHEFILES_DEBUG is not set
# CONFIG_CACHEFILES_HISTOGRAM is not set
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="utf8"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
CONFIG_NTFS_RW=y
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
CONFIG_ADFS_FS=m
# CONFIG_ADFS_FS_RW is not set
CONFIG_AFFS_FS=m
CONFIG_ECRYPT_FS=m
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_BEFS_FS=m
# CONFIG_BEFS_DEBUG is not set
CONFIG_BFS_FS=m
CONFIG_EFS_FS=m
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_JFFS2_FS_WRITEBUFFER=y
# CONFIG_JFFS2_FS_WBUF_VERIFY is not set
CONFIG_JFFS2_SUMMARY=y
CONFIG_JFFS2_FS_XATTR=y
CONFIG_JFFS2_FS_POSIX_ACL=y
CONFIG_JFFS2_FS_SECURITY=y
CONFIG_JFFS2_COMPRESSION_OPTIONS=y
CONFIG_JFFS2_ZLIB=y
CONFIG_JFFS2_LZO=y
CONFIG_JFFS2_RTIME=y
# CONFIG_JFFS2_RUBIN is not set
# CONFIG_JFFS2_CMODE_NONE is not set
CONFIG_JFFS2_CMODE_PRIORITY=y
# CONFIG_JFFS2_CMODE_SIZE is not set
# CONFIG_JFFS2_CMODE_FAVOURLZO is not set
CONFIG_UBIFS_FS=m
CONFIG_UBIFS_FS_XATTR=y
CONFIG_UBIFS_FS_ADVANCED_COMPR=y
CONFIG_UBIFS_FS_LZO=y
CONFIG_UBIFS_FS_ZLIB=y
# CONFIG_UBIFS_FS_DEBUG is not set
CONFIG_LOGFS=m
CONFIG_CRAMFS=m
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_XATTR=y
# CONFIG_SQUASHFS_LZO is not set
CONFIG_SQUASHFS_XZ=y
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
CONFIG_VXFS_FS=m
CONFIG_MINIX_FS=m
CONFIG_OMFS_FS=m
CONFIG_HPFS_FS=m
CONFIG_QNX4FS_FS=m
CONFIG_ROMFS_FS=m
# CONFIG_ROMFS_BACKED_BY_BLOCK is not set
# CONFIG_ROMFS_BACKED_BY_MTD is not set
CONFIG_ROMFS_BACKED_BY_BOTH=y
CONFIG_ROMFS_ON_BLOCK=y
CONFIG_ROMFS_ON_MTD=y
CONFIG_PSTORE=y
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_UFS_DEBUG is not set
CONFIG_EXOFS_FS=m
# CONFIG_EXOFS_DEBUG is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_V4_1 is not set
CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
# CONFIG_NFS_USE_NEW_IDMAPPER is not set
CONFIG_NFSD=m
CONFIG_NFSD_DEPRECATED=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_XPRT_RDMA=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_CEPH_FS=m
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
CONFIG_CIFS_DFS_UPCALL=y
CONFIG_CIFS_FSCACHE=y
CONFIG_CIFS_ACL=y
# CONFIG_CIFS_NFSD_EXPORT is not set
CONFIG_NCP_FS=m
CONFIG_NCPFS_PACKET_SIGNING=y
CONFIG_NCPFS_IOCTL_LOCKING=y
CONFIG_NCPFS_STRONG=y
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
# CONFIG_NCPFS_SMALLDOS is not set
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
CONFIG_AFS_FS=m
# CONFIG_AFS_DEBUG is not set
CONFIG_AFS_FSCACHE=y
CONFIG_9P_FS=m
CONFIG_9P_FSCACHE=y
CONFIG_9P_FS_POSIX_ACL=y
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
CONFIG_ACORN_PARTITION=y
# CONFIG_ACORN_PARTITION_CUMANA is not set
# CONFIG_ACORN_PARTITION_EESOX is not set
CONFIG_ACORN_PARTITION_ICS=y
# CONFIG_ACORN_PARTITION_ADFS is not set
# CONFIG_ACORN_PARTITION_POWERTEC is not set
CONFIG_ACORN_PARTITION_RISCIX=y
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not set
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_MASK=0x01b6
CONFIG_STRIP_ASM_SYMS=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_HARDLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHEDSTATS is not set
CONFIG_TIMER_STATS=y
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_LKDTM is not set
# CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_SYSCTL_SYSCALL_CHECK=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_EVENT_POWER_TRACING_DEPRECATED=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
# CONFIG_FUNCTION_TRACER is not set
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
# CONFIG_FTRACE_SYSCALLS is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_STACK_TRACER is not set
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_ASYNC_RAID6_TEST is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
# CONFIG_KGDB_LOW_LEVEL_TRAP is not set
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_TEST_KSTRTOX is not set
CONFIG_STRICT_DEVMEM=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_DEBUG_RODATA is not set
# CONFIG_DEBUG_SET_MODULE_RONX is not set
# CONFIG_DEBUG_NX_TEST is not set
CONFIG_DOUBLEFAULT=y
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
# CONFIG_X86_DECODER_SELFTEST is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
CONFIG_OPTIMIZE_INLINING=y
# CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set
#
# Security options
#
CONFIG_KEYS=y
# CONFIG_TRUSTED_KEYS is not set
CONFIG_KEYS_DEBUG_PROC_KEYS=y
# CONFIG_SECURITY_DMESG_RESTRICT is not set
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_SECURITY_PATH=y
# CONFIG_INTEL_TXT is not set
CONFIG_LSM_MMAP_MIN_ADDR=0
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=0
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
CONFIG_SECURITY_TOMOYO=y
# CONFIG_SECURITY_APPARMOR is not set
# CONFIG_IMA is not set
CONFIG_DEFAULT_SECURITY_SELINUX=y
# CONFIG_DEFAULT_SECURITY_TOMOYO is not set
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=m
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_MANAGER_DISABLE_TESTS is not set
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m
#
# Block modes
#
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m
#
# Digest
#
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_CRC32C_INTEL=m
CONFIG_CRYPTO_GHASH=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_586=m
# CONFIG_CRYPTO_AES_NI_INTEL is not set
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_586=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_586=m
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_ZLIB=m
CONFIG_CRYPTO_LZO=m
#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
CONFIG_CRYPTO_DEV_GEODE=m
CONFIG_CRYPTO_DEV_HIFN_795X=m
CONFIG_CRYPTO_DEV_HIFN_795X_RNG=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
# CONFIG_KVM_MMU_AUDIT is not set
CONFIG_VHOST_NET=m
CONFIG_LGUEST=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_BINARY_PRINTF=y
#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_T10DIF=m
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC7=m
CONFIG_LIBCRC32C=m
CONFIG_AUDIT_GENERIC=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_REED_SOLOMON=m
CONFIG_REED_SOLOMON_DEC16=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_BTREE=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_NLATTR=y
CONFIG_LRU_CACHE=m
CONFIG_AVERAGE=y
# CONFIG_SHM_SIGNAL is not set
# CONFIG_IOQ is not set
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 10:16 linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?) Sedat Dilek
@ 2011-03-25 13:05 ` Sedat Dilek
2011-03-25 15:55 ` Josh Triplett
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-25 13:05 UTC (permalink / raw)
To: linux-next, LKML
Cc: Stephen Rothwell, Randy Dunlap, Theodore Ts'o, Jens Axboe,
Tejun Heo, Al Viro, Paul E. McKenney, Josh Triplett, Nick Piggin
[-- Attachment #1: Type: text/plain, Size: 792 bytes --]
On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
<sedat.dilek@googlemail.com> wrote:
> Hi,
>
> right after I have finished building a new linux-next kernel, booting
> into desktop and archiving my build-tree (ext4) as tarball to an
> external USB harddisk (partition there is ext3).
> ( Yesterday, I have seen similiar call-traces in my logs, but it was
> hard to reproduce [1]. )
> I am unsure from where the problem aroses, if you have a hint, let me know.
>
> Regards,
> - Sedat -
>
> [1] http://lkml.org/lkml/2011/3/24/268
>
> P.S.: Attached are the dmesg outputs and my kernel-config
>
I turned off the notebook for about 2hrs to avoid thermal problems and
hoax reports.
Jumped into desktop and started an archive job as 1st job while doing daily job.
Yeah, it is reproducible.
- Sedat -
[-- Attachment #2: dmesg_2.txt --]
[-- Type: text/plain, Size: 77933 bytes --]
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.38-next20110325-2-686-iniza (Debian 2.6.38-2~next20110325.dileks2) (sedat.dilek@gmail.com) (gcc version 4.6.0 20110322 (prerelease) (Debian 4.6.0~rc1-3) ) #1 SMP Fri Mar 25 10:37:32 CET 2011
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
[ 0.000000] BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000d2000 - 00000000000d4000 (reserved)
[ 0.000000] BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000003ff60000 (usable)
[ 0.000000] BIOS-e820: 000000003ff60000 - 000000003ff77000 (ACPI data)
[ 0.000000] BIOS-e820: 000000003ff77000 - 000000003ff79000 (ACPI NVS)
[ 0.000000] BIOS-e820: 000000003ff80000 - 0000000040000000 (reserved)
[ 0.000000] BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
[ 0.000000] Notice: NX (Execute Disable) protection missing in CPU!
[ 0.000000] DMI present.
[ 0.000000] DMI: IBM 2374SG6/2374SG6, BIOS 1RETDRWW (3.23 ) 06/18/2007
[ 0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
[ 0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
[ 0.000000] last_pfn = 0x3ff60 max_arch_pfn = 0x100000
[ 0.000000] MTRR default type: uncachable
[ 0.000000] MTRR fixed ranges enabled:
[ 0.000000] 00000-9FFFF write-back
[ 0.000000] A0000-BFFFF uncachable
[ 0.000000] C0000-CFFFF write-protect
[ 0.000000] D0000-DBFFF uncachable
[ 0.000000] DC000-DFFFF write-back
[ 0.000000] E0000-FFFFF write-protect
[ 0.000000] MTRR variable ranges enabled:
[ 0.000000] 0 base 000000000 mask FC0000000 write-back
[ 0.000000] 1 base 03FF80000 mask FFFF80000 uncachable
[ 0.000000] 2 disabled
[ 0.000000] 3 disabled
[ 0.000000] 4 disabled
[ 0.000000] 5 disabled
[ 0.000000] 6 disabled
[ 0.000000] 7 disabled
[ 0.000000] PAT not supported by CPU.
[ 0.000000] initial memory mapped : 0 - 01800000
[ 0.000000] Base memory trampoline at [c009b000] 9b000 size 16384
[ 0.000000] init_memory_mapping: 0000000000000000-00000000377fe000
[ 0.000000] 0000000000 - 0000400000 page 4k
[ 0.000000] 0000400000 - 0037400000 page 2M
[ 0.000000] 0037400000 - 00377fe000 page 4k
[ 0.000000] kernel direct mapping tables up to 377fe000 @ 17fb000-1800000
[ 0.000000] RAMDISK: 36aac000 - 3754e000
[ 0.000000] ACPI: RSDP 000f6d70 00024 (v02 IBM )
[ 0.000000] ACPI: XSDT 3ff6a672 0004C (v01 IBM TP-1R 00003230 LTP 00000000)
[ 0.000000] ACPI: FACP 3ff6a700 000F4 (v03 IBM TP-1R 00003230 IBM 00000001)
[ 0.000000] ACPI Warning: 32/64X length mismatch in Gpe1Block: 0/32 (20110316/tbfadt-529)
[ 0.000000] ACPI Warning: Optional field Gpe1Block has zero address or length: 0x000000000000102C/0x0 (20110316/tbfadt-560)
[ 0.000000] ACPI: DSDT 3ff6a8e7 0C530 (v01 IBM TP-1R 00003230 MSFT 0100000E)
[ 0.000000] ACPI: FACS 3ff78000 00040
[ 0.000000] ACPI: SSDT 3ff6a8b4 00033 (v01 IBM TP-1R 00003230 MSFT 0100000E)
[ 0.000000] ACPI: ECDT 3ff76e17 00052 (v01 IBM TP-1R 00003230 IBM 00000001)
[ 0.000000] ACPI: TCPA 3ff76e69 00032 (v01 IBM TP-1R 00003230 PTL 00000001)
[ 0.000000] ACPI: BOOT 3ff76fd8 00028 (v01 IBM TP-1R 00003230 LTP 00000001)
[ 0.000000] 135MB HIGHMEM available.
[ 0.000000] 887MB LOWMEM available.
[ 0.000000] mapped low ram: 0 - 377fe000
[ 0.000000] low ram: 0 - 377fe000
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000010 -> 0x00001000
[ 0.000000] Normal 0x00001000 -> 0x000377fe
[ 0.000000] HighMem 0x000377fe -> 0x0003ff60
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[2] active PFN ranges
[ 0.000000] 0: 0x00000010 -> 0x0000009f
[ 0.000000] 0: 0x00000100 -> 0x0003ff60
[ 0.000000] On node 0 totalpages: 261871
[ 0.000000] free_area_init_node: node 0, pgdat c1411100, node_mem_map f62ac200
[ 0.000000] DMA zone: 32 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 3951 pages, LIFO batch:0
[ 0.000000] Normal zone: 1744 pages used for memmap
[ 0.000000] Normal zone: 221486 pages, LIFO batch:31
[ 0.000000] HighMem zone: 271 pages used for memmap
[ 0.000000] HighMem zone: 34387 pages, LIFO batch:7
[ 0.000000] Using APIC driver default
[ 0.000000] ACPI: PM-Timer IO Port: 0x1008
[ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.000000] Local APIC disabled by BIOS -- reenabling.
[ 0.000000] Found and enabled local APIC!
[ 0.000000] nr_irqs_gsi: 16
[ 0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
[ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000d2000
[ 0.000000] PM: Registered nosave memory: 00000000000d2000 - 00000000000d4000
[ 0.000000] PM: Registered nosave memory: 00000000000d4000 - 00000000000dc000
[ 0.000000] PM: Registered nosave memory: 00000000000dc000 - 0000000000100000
[ 0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:bf800000)
[ 0.000000] Booting paravirtualized kernel on bare hardware
[ 0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 12 pages/cpu @f5c00000 s27328 r0 d21824 u4194304
[ 0.000000] pcpu-alloc: s27328 r0 d21824 u4194304 alloc=1*4194304
[ 0.000000] pcpu-alloc: [0] 0
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 259824
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-2.6.38-next20110325-2-686-iniza root=UUID=1ceb69a7-ecf4-47e9-a231-b74e0f0a9b62 ro init=/bin/systemd radeon.modeset=1 lapic 3
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Initializing CPU#0
[ 0.000000] Initializing HighMem for node 0 (000377fe:0003ff60)
[ 0.000000] Memory: 1022072k/1047936k available (2738k kernel code, 25412k reserved, 1451k data, 380k init, 138632k highmem)
[ 0.000000] virtual kernel memory layout:
[ 0.000000] fixmap : 0xffd36000 - 0xfffff000 (2852 kB)
[ 0.000000] pkmap : 0xff800000 - 0xffc00000 (4096 kB)
[ 0.000000] vmalloc : 0xf7ffe000 - 0xff7fe000 ( 120 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xf77fe000 ( 887 MB)
[ 0.000000] .init : 0xc1418000 - 0xc1477000 ( 380 kB)
[ 0.000000] .data : 0xc12aca6d - 0xc1417840 (1451 kB)
[ 0.000000] .text : 0xc1000000 - 0xc12aca6d (2738 kB)
[ 0.000000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 0.000000] NR_IRQS:1280
[ 0.000000] CPU 0 irqstacks, hard=f5818000 soft=f581a000
[ 0.000000] Extended CMOS year: 2000
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] Fast TSC calibration using PIT
[ 0.000000] Detected 1694.325 MHz processor.
[ 0.004005] Calibrating delay loop (skipped), value calculated using timer frequency.. 3388.65 BogoMIPS (lpj=6777300)
[ 0.004079] pid_max: default: 32768 minimum: 301
[ 0.004233] Security Framework initialized
[ 0.004275] SELinux: Disabled at boot.
[ 0.004420] Mount-cache hash table entries: 512
[ 0.004929] Initializing cgroup subsys debug
[ 0.004968] Initializing cgroup subsys ns
[ 0.005005] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[ 0.005054] Initializing cgroup subsys cpuacct
[ 0.005121] Initializing cgroup subsys devices
[ 0.005156] Initializing cgroup subsys freezer
[ 0.005192] Initializing cgroup subsys net_cls
[ 0.005228] Initializing cgroup subsys blkio
[ 0.005340] mce: CPU supports 5 MCE banks
[ 0.005386] CPU0: Thermal monitoring enabled (TM2)
[ 0.005597] SMP alternatives: switching to UP code
[ 0.008536] Freeing SMP alternatives: 8k freed
[ 0.008575] ACPI: Core revision 20110316
[ 0.015007] ACPI: setting ELCR to 0200 (from 0800)
[ 0.020009] weird, boot CPU (#0) not listed by the BIOS.
[ 0.020046] SMP motherboard not detected.
[ 0.020084] Enabling APIC mode: Flat. Using 0 I/O APICs
[ 0.024001] SMP disabled
[ 0.024001] Performance Events: p6 PMU driver.
[ 0.024001] ... version: 0
[ 0.024001] ... bit width: 32
[ 0.024001] ... generic registers: 2
[ 0.024001] ... value mask: 00000000ffffffff
[ 0.024001] ... max period: 000000007fffffff
[ 0.024001] ... fixed-purpose events: 0
[ 0.024001] ... event mask: 0000000000000003
[ 0.024001] NMI watchdog enabled, takes one hw-pmu counter.
[ 0.024001] Brought up 1 CPUs
[ 0.024001] Total of 1 processors activated (3388.65 BogoMIPS).
[ 0.024001] devtmpfs: initialized
[ 0.024001] PM: Registering ACPI NVS region at 3ff77000 (8192 bytes)
[ 0.024001] print_constraints: dummy:
[ 0.024001] NET: Registered protocol family 16
[ 0.024001] ACPI: bus type pci registered
[ 0.024001] PCI: PCI BIOS revision 2.10 entry at 0xfd8d6, last bus=8
[ 0.024001] PCI: Using configuration type 1 for base access
[ 0.024001] bio: create slab <bio-0> at 0
[ 0.025428] ACPI: EC: EC description table is found, configuring boot EC
[ 0.037936] ACPI: Interpreter enabled
[ 0.037975] ACPI: (supports S0 S3 S4 S5)
[ 0.038114] ACPI: Using PIC for interrupt routing
[ 0.040959] ACPI: Power Resource [PUBS] (on)
[ 0.044528] ACPI: EC: GPE = 0x1c, I/O: command/status = 0x66, data = 0x62
[ 0.045078] ACPI: ACPI Dock Station Driver: 3 docks/bays found
[ 0.045078] HEST: Table not found.
[ 0.045078] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
[ 0.045078] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.045078] pci_root PNP0A03:00: host bridge window [io 0x0000-0x0cf7] (ignored)
[ 0.045078] pci_root PNP0A03:00: host bridge window [io 0x0d00-0xffff] (ignored)
[ 0.045078] pci_root PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff] (ignored)
[ 0.045078] pci_root PNP0A03:00: host bridge window [mem 0x000d4000-0x000d7fff] (ignored)
[ 0.045078] pci_root PNP0A03:00: host bridge window [mem 0x000d8000-0x000dbfff] (ignored)
[ 0.045078] pci_root PNP0A03:00: host bridge window [mem 0x40000000-0xfebfffff] (ignored)
[ 0.045078] pci 0000:00:00.0: [8086:3340] type 0 class 0x000600
[ 0.045078] pci 0000:00:00.0: reg 10: [mem 0xd0000000-0xdfffffff pref]
[ 0.048041] pci 0000:00:01.0: [8086:3341] type 1 class 0x000604
[ 0.048100] pci 0000:00:1d.0: [8086:24c2] type 0 class 0x000c03
[ 0.048147] pci 0000:00:1d.0: reg 20: [io 0x1800-0x181f]
[ 0.048182] pci 0000:00:1d.1: [8086:24c4] type 0 class 0x000c03
[ 0.048228] pci 0000:00:1d.1: reg 20: [io 0x1820-0x183f]
[ 0.048270] pci 0000:00:1d.2: [8086:24c7] type 0 class 0x000c03
[ 0.048316] pci 0000:00:1d.2: reg 20: [io 0x1840-0x185f]
[ 0.048363] pci 0000:00:1d.7: [8086:24cd] type 0 class 0x000c03
[ 0.048386] pci 0000:00:1d.7: reg 10: [mem 0xc0000000-0xc00003ff]
[ 0.048467] pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
[ 0.048474] pci 0000:00:1d.7: PME# disabled
[ 0.048495] pci 0000:00:1e.0: [8086:2448] type 1 class 0x000604
[ 0.048540] pci 0000:00:1f.0: [8086:24cc] type 0 class 0x000601
[ 0.048608] pci 0000:00:1f.0: quirk: [io 0x1000-0x107f] claimed by ICH4 ACPI/GPIO/TCO
[ 0.048659] pci 0000:00:1f.0: quirk: [io 0x1180-0x11bf] claimed by ICH4 GPIO
[ 0.048711] pci 0000:00:1f.1: [8086:24ca] type 0 class 0x000101
[ 0.048727] pci 0000:00:1f.1: reg 10: [io 0x0000-0x0007]
[ 0.048738] pci 0000:00:1f.1: reg 14: [io 0x0000-0x0003]
[ 0.048749] pci 0000:00:1f.1: reg 18: [io 0x0000-0x0007]
[ 0.048761] pci 0000:00:1f.1: reg 1c: [io 0x0000-0x0003]
[ 0.048772] pci 0000:00:1f.1: reg 20: [io 0x1860-0x186f]
[ 0.048783] pci 0000:00:1f.1: reg 24: [mem 0x00000000-0x000003ff]
[ 0.048814] pci 0000:00:1f.3: [8086:24c3] type 0 class 0x000c05
[ 0.048860] pci 0000:00:1f.3: reg 20: [io 0x1880-0x189f]
[ 0.048899] pci 0000:00:1f.5: [8086:24c5] type 0 class 0x000401
[ 0.048916] pci 0000:00:1f.5: reg 10: [io 0x1c00-0x1cff]
[ 0.048926] pci 0000:00:1f.5: reg 14: [io 0x18c0-0x18ff]
[ 0.048937] pci 0000:00:1f.5: reg 18: [mem 0xc0000c00-0xc0000dff]
[ 0.048948] pci 0000:00:1f.5: reg 1c: [mem 0xc0000800-0xc00008ff]
[ 0.048987] pci 0000:00:1f.5: PME# supported from D0 D3hot D3cold
[ 0.048992] pci 0000:00:1f.5: PME# disabled
[ 0.049010] pci 0000:00:1f.6: [8086:24c6] type 0 class 0x000703
[ 0.049027] pci 0000:00:1f.6: reg 10: [io 0x2400-0x24ff]
[ 0.049038] pci 0000:00:1f.6: reg 14: [io 0x2000-0x207f]
[ 0.049091] pci 0000:00:1f.6: PME# supported from D0 D3hot D3cold
[ 0.049096] pci 0000:00:1f.6: PME# disabled
[ 0.049123] pci 0000:01:00.0: [1002:4c66] type 0 class 0x000300
[ 0.049140] pci 0000:01:00.0: reg 10: [mem 0xe0000000-0xe7ffffff pref]
[ 0.049149] pci 0000:01:00.0: reg 14: [io 0x3000-0x30ff]
[ 0.049158] pci 0000:01:00.0: reg 18: [mem 0xc0100000-0xc010ffff]
[ 0.049183] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
[ 0.049204] pci 0000:01:00.0: supports D1 D2
[ 0.049240] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[ 0.049278] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff]
[ 0.049283] pci 0000:00:01.0: bridge window [mem 0xc0100000-0xc01fffff]
[ 0.049288] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe7ffffff pref]
[ 0.049315] pci 0000:02:00.0: [104c:ac55] type 2 class 0x000607
[ 0.049333] pci 0000:02:00.0: reg 10: [mem 0xb0000000-0xb0000fff]
[ 0.049354] pci 0000:02:00.0: supports D1 D2
[ 0.049357] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 0.049363] pci 0000:02:00.0: PME# disabled
[ 0.049384] pci 0000:02:00.1: [104c:ac55] type 2 class 0x000607
[ 0.049403] pci 0000:02:00.1: reg 10: [mem 0xb1000000-0xb1000fff]
[ 0.049423] pci 0000:02:00.1: supports D1 D2
[ 0.049426] pci 0000:02:00.1: PME# supported from D0 D1 D2 D3hot D3cold
[ 0.049432] pci 0000:02:00.1: PME# disabled
[ 0.049461] pci 0000:02:01.0: [8086:101e] type 0 class 0x000200
[ 0.049481] pci 0000:02:01.0: reg 10: [mem 0xc0220000-0xc023ffff]
[ 0.049492] pci 0000:02:01.0: reg 14: [mem 0xc0200000-0xc020ffff]
[ 0.049503] pci 0000:02:01.0: reg 18: [io 0x8000-0x803f]
[ 0.049536] pci 0000:02:01.0: reg 30: [mem 0x00000000-0x0000ffff pref]
[ 0.049561] pci 0000:02:01.0: PME# supported from D0 D3hot D3cold
[ 0.049567] pci 0000:02:01.0: PME# disabled
[ 0.049589] pci 0000:02:02.0: [168c:1014] type 0 class 0x000200
[ 0.049607] pci 0000:02:02.0: reg 10: [mem 0xc0210000-0xc021ffff]
[ 0.049708] pci 0000:00:1e.0: PCI bridge to [bus 02-08] (subtractive decode)
[ 0.049749] pci 0000:00:1e.0: bridge window [io 0x4000-0x8fff]
[ 0.049755] pci 0000:00:1e.0: bridge window [mem 0xc0200000-0xcfffffff]
[ 0.049762] pci 0000:00:1e.0: bridge window [mem 0xe8000000-0xefffffff pref]
[ 0.049766] pci 0000:00:1e.0: bridge window [io 0x0000-0xffff] (subtractive decode)
[ 0.049770] pci 0000:00:1e.0: bridge window [mem 0x00000000-0xffffffff] (subtractive decode)
[ 0.049849] pci_bus 0000:00: on NUMA node 0
[ 0.049853] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[ 0.049904] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGP_._PRT]
[ 0.049931] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
[ 0.053184] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.053524] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.053859] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.054195] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.054511] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
[ 0.054877] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
[ 0.055243] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11) *0, disabled.
[ 0.055629] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11)
[ 0.055992] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.056012] vgaarb: loaded
[ 0.056094] PCI: Using ACPI for IRQ routing
[ 0.056264] PCI: pci_cache_line_size set to 64 bytes
[ 0.056332] reserve RAM buffer: 000000000009f000 - 000000000009ffff
[ 0.056337] reserve RAM buffer: 000000003ff60000 - 000000003fffffff
[ 0.058468] pnp: PnP ACPI init
[ 0.058527] ACPI: bus type pnp registered
[ 0.059178] pnp 00:00: [mem 0x00000000-0x0009ffff]
[ 0.059183] pnp 00:00: [mem 0x000c0000-0x000c3fff]
[ 0.059186] pnp 00:00: [mem 0x000c4000-0x000c7fff]
[ 0.059190] pnp 00:00: [mem 0x000c8000-0x000cbfff]
[ 0.059193] pnp 00:00: [mem 0x000cc000-0x000cffff]
[ 0.059197] pnp 00:00: [mem 0x000d0000-0x000d3fff]
[ 0.059201] pnp 00:00: [mem 0x000d4000-0x000d3fff disabled]
[ 0.059204] pnp 00:00: [mem 0x000d8000-0x000d7fff disabled]
[ 0.059208] pnp 00:00: [mem 0x000dc000-0x000dffff]
[ 0.059212] pnp 00:00: [mem 0x000e0000-0x000e3fff]
[ 0.059215] pnp 00:00: [mem 0x000e4000-0x000e7fff]
[ 0.059219] pnp 00:00: [mem 0x000e8000-0x000ebfff]
[ 0.059222] pnp 00:00: [mem 0x000ec000-0x000effff]
[ 0.059226] pnp 00:00: [mem 0x000f0000-0x000fffff]
[ 0.059229] pnp 00:00: [mem 0x00100000-0x3fffffff]
[ 0.059233] pnp 00:00: [mem 0xfec00000-0xffffffff]
[ 0.059322] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
[ 0.059363] system 00:00: [mem 0x000c0000-0x000c3fff] could not be reserved
[ 0.059402] system 00:00: [mem 0x000c4000-0x000c7fff] could not be reserved
[ 0.059442] system 00:00: [mem 0x000c8000-0x000cbfff] could not be reserved
[ 0.059481] system 00:00: [mem 0x000cc000-0x000cffff] could not be reserved
[ 0.059521] system 00:00: [mem 0x000d0000-0x000d3fff] could not be reserved
[ 0.059560] system 00:00: [mem 0x000dc000-0x000dffff] could not be reserved
[ 0.059600] system 00:00: [mem 0x000e0000-0x000e3fff] could not be reserved
[ 0.059639] system 00:00: [mem 0x000e4000-0x000e7fff] could not be reserved
[ 0.059679] system 00:00: [mem 0x000e8000-0x000ebfff] could not be reserved
[ 0.059718] system 00:00: [mem 0x000ec000-0x000effff] could not be reserved
[ 0.059757] system 00:00: [mem 0x000f0000-0x000fffff] could not be reserved
[ 0.059797] system 00:00: [mem 0x00100000-0x3fffffff] could not be reserved
[ 0.059837] system 00:00: [mem 0xfec00000-0xffffffff] could not be reserved
[ 0.059877] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active)
[ 0.059912] pnp 00:01: [bus 00-ff]
[ 0.059916] pnp 00:01: [io 0x0cf8-0x0cff]
[ 0.059919] pnp 00:01: [io 0x0000-0x0cf7 window]
[ 0.059923] pnp 00:01: [io 0x0d00-0xffff window]
[ 0.059935] pnp 00:01: [mem 0x000a0000-0x000bffff window]
[ 0.059939] pnp 00:01: [mem 0x000c0000-0x000c3fff window]
[ 0.059943] pnp 00:01: [mem 0x000c4000-0x000c7fff window]
[ 0.059947] pnp 00:01: [mem 0x000c8000-0x000cbfff window]
[ 0.059951] pnp 00:01: [mem 0x000cc000-0x000cffff window]
[ 0.059955] pnp 00:01: [mem 0x000d0000-0x000d3fff window]
[ 0.060027] pnp 00:01: [mem 0x000d4000-0x000d7fff window]
[ 0.060031] pnp 00:01: [mem 0x000d8000-0x000dbfff window]
[ 0.060035] pnp 00:01: [mem 0x000dc000-0x000dffff window]
[ 0.060039] pnp 00:01: [mem 0x000e0000-0x000e3fff window]
[ 0.060043] pnp 00:01: [mem 0x000e4000-0x000e7fff window]
[ 0.060046] pnp 00:01: [mem 0x000e8000-0x000ebfff window]
[ 0.060050] pnp 00:01: [mem 0x000ec000-0x000effff window]
[ 0.060054] pnp 00:01: [mem 0x40000000-0xfebfffff window]
[ 0.060123] pnp 00:01: Plug and Play ACPI device, IDs PNP0a03 (active)
[ 0.060232] pnp 00:02: [io 0x0010-0x001f]
[ 0.060236] pnp 00:02: [io 0x0090-0x009f]
[ 0.060239] pnp 00:02: [io 0x0024-0x0025]
[ 0.060243] pnp 00:02: [io 0x0028-0x0029]
[ 0.060246] pnp 00:02: [io 0x002c-0x002d]
[ 0.060249] pnp 00:02: [io 0x0030-0x0031]
[ 0.060252] pnp 00:02: [io 0x0034-0x0035]
[ 0.060256] pnp 00:02: [io 0x0038-0x0039]
[ 0.060259] pnp 00:02: [io 0x003c-0x003d]
[ 0.060262] pnp 00:02: [io 0x00a4-0x00a5]
[ 0.060266] pnp 00:02: [io 0x00a8-0x00a9]
[ 0.060269] pnp 00:02: [io 0x00ac-0x00ad]
[ 0.060272] pnp 00:02: [io 0x00b0-0x00b5]
[ 0.060275] pnp 00:02: [io 0x00b8-0x00b9]
[ 0.060279] pnp 00:02: [io 0x00bc-0x00bd]
[ 0.060282] pnp 00:02: [io 0x0050-0x0053]
[ 0.060285] pnp 00:02: [io 0x0072-0x0077]
[ 0.060289] pnp 00:02: [io 0x002e-0x002f]
[ 0.060292] pnp 00:02: [io 0x1000-0x107f]
[ 0.060295] pnp 00:02: [io 0x1180-0x11bf]
[ 0.060298] pnp 00:02: [io 0x15e0-0x15ef]
[ 0.060302] pnp 00:02: [io 0x1600-0x162f]
[ 0.060305] pnp 00:02: [io 0x1632-0x167f]
[ 0.060308] pnp 00:02: [io 0x004e-0x004f]
[ 0.060312] pnp 00:02: [io 0x1630-0x1631]
[ 0.060414] system 00:02: [io 0x1000-0x107f] has been reserved
[ 0.060453] system 00:02: [io 0x1180-0x11bf] has been reserved
[ 0.060492] system 00:02: [io 0x15e0-0x15ef] has been reserved
[ 0.060530] system 00:02: [io 0x1600-0x162f] has been reserved
[ 0.060568] system 00:02: [io 0x1632-0x167f] has been reserved
[ 0.060606] system 00:02: [io 0x1630-0x1631] has been reserved
[ 0.060645] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 0.060665] pnp 00:03: [io 0x0000-0x000f]
[ 0.060668] pnp 00:03: [io 0x0080-0x008f]
[ 0.060672] pnp 00:03: [io 0x00c0-0x00df]
[ 0.060675] pnp 00:03: [dma 4]
[ 0.060712] pnp 00:03: Plug and Play ACPI device, IDs PNP0200 (active)
[ 0.060725] pnp 00:04: [io 0x0061]
[ 0.060759] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
[ 0.060773] pnp 00:05: [io 0x00f0]
[ 0.060778] pnp 00:05: [irq 13]
[ 0.060814] pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active)
[ 0.060827] pnp 00:06: [io 0x0070-0x0071]
[ 0.060830] pnp 00:06: [irq 8]
[ 0.060870] pnp 00:06: Plug and Play ACPI device, IDs PNP0b00 (active)
[ 0.060884] pnp 00:07: [io 0x0060]
[ 0.060887] pnp 00:07: [io 0x0064]
[ 0.060890] pnp 00:07: [irq 1]
[ 0.060926] pnp 00:07: Plug and Play ACPI device, IDs PNP0303 (active)
[ 0.060939] pnp 00:08: [irq 12]
[ 0.060975] pnp 00:08: Plug and Play ACPI device, IDs IBM0057 PNP0f13 (active)
[ 0.061012] pnp 00:09: [io 0x03f0-0x03f5]
[ 0.061016] pnp 00:09: [io 0x03f7]
[ 0.061019] pnp 00:09: [irq 6]
[ 0.061022] pnp 00:09: [dma 2]
[ 0.061075] pnp 00:09: Plug and Play ACPI device, IDs PNP0700 (active)
[ 0.061178] pnp 00:0a: [io 0x03f8-0x03ff]
[ 0.061182] pnp 00:0a: [irq 4]
[ 0.061283] pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (active)
[ 0.061399] pnp 00:0b: [io 0x03bc-0x03be]
[ 0.061403] pnp 00:0b: [irq 7]
[ 0.061497] pnp 00:0b: Plug and Play ACPI device, IDs PNP0400 (active)
[ 0.061657] pnp 00:0c: Plug and Play ACPI device, IDs IBM0071 PNP0511 (disabled)
[ 0.061715] pnp: PnP ACPI: found 13 devices
[ 0.061715] ACPI: ACPI bus type pnp unregistered
[ 0.061715] PnPBIOS: Disabled by ACPI PNP
[ 0.102048] Switching to clocksource acpi_pm
[ 0.102136] pci 0000:00:1f.1: BAR 5: assigned [mem 0x40000000-0x400003ff]
[ 0.102179] pci 0000:00:1f.1: BAR 5: set to [mem 0x40000000-0x400003ff] (PCI address [0x40000000-0x400003ff])
[ 0.102232] pci 0000:01:00.0: BAR 6: assigned [mem 0xc0120000-0xc013ffff pref]
[ 0.102280] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[ 0.102317] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff]
[ 0.102357] pci 0000:00:01.0: bridge window [mem 0xc0100000-0xc01fffff]
[ 0.102397] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe7ffffff pref]
[ 0.102456] pci 0000:02:00.0: BAR 15: assigned [mem 0xe8000000-0xebffffff pref]
[ 0.102504] pci 0000:02:00.0: BAR 16: assigned [mem 0xc4000000-0xc7ffffff]
[ 0.102543] pci 0000:02:00.1: BAR 15: assigned [mem 0xec000000-0xefffffff pref]
[ 0.102591] pci 0000:02:00.1: BAR 16: assigned [mem 0xc8000000-0xcbffffff]
[ 0.102631] pci 0000:02:01.0: BAR 6: assigned [mem 0xc0240000-0xc024ffff pref]
[ 0.102679] pci 0000:02:00.0: BAR 13: assigned [io 0x4000-0x40ff]
[ 0.102717] pci 0000:02:00.0: BAR 14: assigned [io 0x4400-0x44ff]
[ 0.102755] pci 0000:02:00.1: BAR 13: assigned [io 0x4800-0x48ff]
[ 0.102794] pci 0000:02:00.1: BAR 14: assigned [io 0x4c00-0x4cff]
[ 0.102833] pci 0000:02:00.0: CardBus bridge to [bus 03-06]
[ 0.102869] pci 0000:02:00.0: bridge window [io 0x4000-0x40ff]
[ 0.102909] pci 0000:02:00.0: bridge window [io 0x4400-0x44ff]
[ 0.102949] pci 0000:02:00.0: bridge window [mem 0xe8000000-0xebffffff pref]
[ 0.102998] pci 0000:02:00.0: bridge window [mem 0xc4000000-0xc7ffffff]
[ 0.103038] pci 0000:02:00.1: CardBus bridge to [bus 07-07]
[ 0.103075] pci 0000:02:00.1: bridge window [io 0x4800-0x48ff]
[ 0.103115] pci 0000:02:00.1: bridge window [io 0x4c00-0x4cff]
[ 0.103155] pci 0000:02:00.1: bridge window [mem 0xec000000-0xefffffff pref]
[ 0.103204] pci 0000:02:00.1: bridge window [mem 0xc8000000-0xcbffffff]
[ 0.103244] pci 0000:00:1e.0: PCI bridge to [bus 02-08]
[ 0.103282] pci 0000:00:1e.0: bridge window [io 0x4000-0x8fff]
[ 0.103323] pci 0000:00:1e.0: bridge window [mem 0xc0200000-0xcfffffff]
[ 0.103363] pci 0000:00:1e.0: bridge window [mem 0xe8000000-0xefffffff pref]
[ 0.103427] pci 0000:00:1e.0: setting latency timer to 64
[ 0.103636] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
[ 0.103674] PCI: setting IRQ 11 as level-triggered
[ 0.103681] pci 0000:02:00.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[ 0.103900] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[ 0.103939] pci 0000:02:00.1: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 0.103991] pci_bus 0000:00: resource 0 [io 0x0000-0xffff]
[ 0.103995] pci_bus 0000:00: resource 1 [mem 0x00000000-0xffffffff]
[ 0.103999] pci_bus 0000:01: resource 0 [io 0x3000-0x3fff]
[ 0.104003] pci_bus 0000:01: resource 1 [mem 0xc0100000-0xc01fffff]
[ 0.104005] Switched to NOHz mode on CPU #0
[ 0.104005] pci_bus 0000:01: resource 2 [mem 0xe0000000-0xe7ffffff pref]
[ 0.104005] pci_bus 0000:02: resource 0 [io 0x4000-0x8fff]
[ 0.104005] pci_bus 0000:02: resource 1 [mem 0xc0200000-0xcfffffff]
[ 0.104005] pci_bus 0000:02: resource 2 [mem 0xe8000000-0xefffffff pref]
[ 0.104005] pci_bus 0000:02: resource 4 [io 0x0000-0xffff]
[ 0.104005] pci_bus 0000:02: resource 5 [mem 0x00000000-0xffffffff]
[ 0.104005] pci_bus 0000:03: resource 0 [io 0x4000-0x40ff]
[ 0.104005] pci_bus 0000:03: resource 1 [io 0x4400-0x44ff]
[ 0.104005] pci_bus 0000:03: resource 2 [mem 0xe8000000-0xebffffff pref]
[ 0.104005] pci_bus 0000:03: resource 3 [mem 0xc4000000-0xc7ffffff]
[ 0.104005] pci_bus 0000:07: resource 0 [io 0x4800-0x48ff]
[ 0.104005] pci_bus 0000:07: resource 1 [io 0x4c00-0x4cff]
[ 0.104005] pci_bus 0000:07: resource 2 [mem 0xec000000-0xefffffff pref]
[ 0.104005] pci_bus 0000:07: resource 3 [mem 0xc8000000-0xcbffffff]
[ 0.104005] NET: Registered protocol family 2
[ 0.104005] IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.104005] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.104753] TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.105912] TCP: Hash tables configured (established 131072 bind 65536)
[ 0.105953] TCP reno registered
[ 0.105995] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 0.106069] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 0.106488] NET: Registered protocol family 1
[ 0.106669] pci 0000:01:00.0: Boot video device
[ 0.106688] PCI: CLS 32 bytes, default 64
[ 0.106785] Unpacking initramfs...
[ 0.555165] Freeing initrd memory: 10888k freed
[ 0.573482] Simple Boot Flag at 0x35 set to 0x1
[ 0.574084] audit: initializing netlink socket (disabled)
[ 0.574143] type=2000 audit(1301061022.571:1): initialized
[ 0.596195] highmem bounce pool size: 64 pages
[ 0.596238] HugeTLB registered 4 MB page size, pre-allocated 0 pages
[ 0.599161] VFS: Disk quotas dquot_6.5.2
[ 0.599377] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 0.599624] msgmni has been set to 1746
[ 0.599953] alg: No test for stdrng (krng)
[ 0.600098] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 0.600148] io scheduler noop registered
[ 0.600183] io scheduler deadline registered
[ 0.600239] io scheduler cfq registered (default)
[ 0.600552] ERST: Table is not found!
[ 0.600601] isapnp: Scanning for PnP cards...
[ 0.953520] isapnp: No Plug & Play device found
[ 0.953674] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.953800] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
[ 0.954325] 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
[ 0.954501] serial 0000:00:1f.6: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 0.954555] serial 0000:00:1f.6: PCI INT B disabled
[ 0.954737] Linux agpgart interface v0.103
[ 0.954925] agpgart-intel 0000:00:00.0: Intel 855PM Chipset
[ 0.968545] agpgart-intel 0000:00:00.0: AGP aperture is 256M @ 0xd0000000
[ 0.968818] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[ 0.974582] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 0.974626] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 0.974822] mousedev: PS/2 mouse device common for all mice
[ 0.974919] rtc_cmos 00:06: RTC can wake from S4
[ 0.975062] rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
[ 0.975117] rtc0: alarms up to one month, y3k, 114 bytes nvram
[ 0.975165] cpuidle: using governor ladder
[ 0.975200] cpuidle: using governor menu
[ 0.975633] TCP cubic registered
[ 0.975920] NET: Registered protocol family 10
[ 0.976987] Mobile IPv6
[ 0.977022] NET: Registered protocol family 17
[ 0.977060] Registering the dns_resolver key type
[ 0.977119] Using IPI No-Shortcut mode
[ 0.977264] PM: Hibernation image not present or could not be loaded.
[ 0.977279] registered taskstats version 1
[ 0.977647] rtc_cmos 00:06: setting system clock to 2011-03-25 13:50:23 UTC (1301061023)
[ 0.977747] Initializing network drop monitor service
[ 0.977851] Freeing unused kernel memory: 380k freed
[ 0.978251] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[ 1.004313] udev[43]: starting version 166
[ 1.279231] Floppy drive(s): fd0 is 1.44M
[ 1.295210] FDC 0 is a National Semiconductor PC87306
[ 1.324419] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[ 1.324490] e1000: Copyright (c) 1999-2006 Intel Corporation.
[ 1.324594] e1000 0000:02:01.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[ 1.329124] thermal LNXTHERM:00: registered as thermal_zone0
[ 1.329164] ACPI: Thermal Zone [THM0] (33 C)
[ 1.605734] Refined TSC clocksource calibration: 1694.501 MHz.
[ 1.605781] Switching to clocksource tsc
[ 1.633373] e1000 0000:02:01.0: eth0: (PCI:33MHz:32-bit) 00:0d:60:b0:62:87
[ 1.633418] e1000 0000:02:01.0: eth0: Intel(R) PRO/1000 Network Connection
[ 1.644967] usbcore: registered new interface driver usbfs
[ 1.645207] usbcore: registered new interface driver hub
[ 1.648045] usbcore: registered new device driver usb
[ 1.657584] SCSI subsystem initialized
[ 1.676144] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1.676237] ehci_hcd 0000:00:1d.7: power state changed by ACPI to D0
[ 1.676278] ehci_hcd 0000:00:1d.7: power state changed by ACPI to D0
[ 1.676560] ACPI: PCI Interrupt Link [LNKH] enabled at IRQ 11
[ 1.676601] ehci_hcd 0000:00:1d.7: PCI INT D -> Link[LNKH] -> GSI 11 (level, low) -> IRQ 11
[ 1.676669] ehci_hcd 0000:00:1d.7: setting latency timer to 64
[ 1.676674] ehci_hcd 0000:00:1d.7: EHCI Host Controller
[ 1.676765] ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
[ 1.676853] ehci_hcd 0000:00:1d.7: debug port 1
[ 1.680758] ehci_hcd 0000:00:1d.7: cache line size of 32 is not supported
[ 1.700133] ehci_hcd 0000:00:1d.7: irq 11, io mem 0xc0000000
[ 1.704042] uhci_hcd: USB Universal Host Controller Interface driver
[ 1.750635] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
[ 1.750747] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 1.750786] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.750833] usb usb1: Product: EHCI Host Controller
[ 1.750869] usb usb1: Manufacturer: Linux 2.6.38-next20110325-2-686-iniza ehci_hcd
[ 1.750917] usb usb1: SerialNumber: 0000:00:1d.7
[ 1.751316] hub 1-0:1.0: USB hub found
[ 1.751356] hub 1-0:1.0: 6 ports detected
[ 1.751507] uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
[ 1.751547] uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
[ 1.751595] uhci_hcd 0000:00:1d.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[ 1.751657] uhci_hcd 0000:00:1d.0: setting latency timer to 64
[ 1.751662] uhci_hcd 0000:00:1d.0: UHCI Host Controller
[ 1.751714] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
[ 1.751788] uhci_hcd 0000:00:1d.0: irq 11, io base 0x00001800
[ 1.751871] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[ 1.751910] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.751956] usb usb2: Product: UHCI Host Controller
[ 1.751992] usb usb2: Manufacturer: Linux 2.6.38-next20110325-2-686-iniza uhci_hcd
[ 1.752056] usb usb2: SerialNumber: 0000:00:1d.0
[ 1.752316] hub 2-0:1.0: USB hub found
[ 1.752355] hub 2-0:1.0: 2 ports detected
[ 1.752463] uhci_hcd 0000:00:1d.1: power state changed by ACPI to D0
[ 1.752503] uhci_hcd 0000:00:1d.1: power state changed by ACPI to D0
[ 1.752779] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
[ 1.752819] uhci_hcd 0000:00:1d.1: PCI INT B -> Link[LNKD] -> GSI 11 (level, low) -> IRQ 11
[ 1.752872] uhci_hcd 0000:00:1d.1: setting latency timer to 64
[ 1.752876] uhci_hcd 0000:00:1d.1: UHCI Host Controller
[ 1.752920] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
[ 1.752988] uhci_hcd 0000:00:1d.1: irq 11, io base 0x00001820
[ 1.753060] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[ 1.753100] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.753146] usb usb3: Product: UHCI Host Controller
[ 1.753182] usb usb3: Manufacturer: Linux 2.6.38-next20110325-2-686-iniza uhci_hcd
[ 1.755923] usb usb3: SerialNumber: 0000:00:1d.1
[ 1.759733] libata version 3.00 loaded.
[ 1.764367] hub 3-0:1.0: USB hub found
[ 1.764413] hub 3-0:1.0: 2 ports detected
[ 1.764790] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[ 1.764831] uhci_hcd 0000:00:1d.2: PCI INT C -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
[ 1.764890] uhci_hcd 0000:00:1d.2: setting latency timer to 64
[ 1.764895] uhci_hcd 0000:00:1d.2: UHCI Host Controller
[ 1.764948] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
[ 1.765025] uhci_hcd 0000:00:1d.2: irq 11, io base 0x00001840
[ 1.765108] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[ 1.765147] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.765194] usb usb4: Product: UHCI Host Controller
[ 1.765230] usb usb4: Manufacturer: Linux 2.6.38-next20110325-2-686-iniza uhci_hcd
[ 1.765276] usb usb4: SerialNumber: 0000:00:1d.2
[ 1.766310] hub 4-0:1.0: USB hub found
[ 1.766351] hub 4-0:1.0: 2 ports detected
[ 1.776084] ata_piix 0000:00:1f.1: version 2.13
[ 1.776105] ata_piix 0000:00:1f.1: enabling device (0005 -> 0007)
[ 1.776156] ata_piix 0000:00:1f.1: PCI INT A -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
[ 1.776262] ata_piix 0000:00:1f.1: setting latency timer to 64
[ 1.777699] scsi0 : ata_piix
[ 1.777953] scsi1 : ata_piix
[ 1.778640] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x1860 irq 14
[ 1.778680] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x1868 irq 15
[ 1.932433] ata2.01: NODEV after polling detection
[ 1.940951] ata1.00: HPA detected: current 110257519, native 117210240
[ 1.940998] ata1.00: ATA-6: HTS726060M9AT00, MH4OA6BA, max UDMA/100
[ 1.941036] ata1.00: 110257519 sectors, multi 16: LBA
[ 1.941166] ata2.00: ATAPI: UJDA755yDVD/CDRW, 1.70, max UDMA/33
[ 1.956656] ata2.00: configured for UDMA/33
[ 1.956745] ata1.00: configured for UDMA/100
[ 1.957000] scsi 0:0:0:0: Direct-Access ATA HTS726060M9AT00 MH4O PQ: 0 ANSI: 5
[ 1.959931] scsi 1:0:0:0: CD-ROM MATSHITA UJDA755yDVD/CDRW 1.70 PQ: 0 ANSI: 5
[ 1.994704] sd 0:0:0:0: [sda] 110257519 512-byte logical blocks: (56.4 GB/52.5 GiB)
[ 1.994830] sd 0:0:0:0: [sda] Write Protect is off
[ 1.994867] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 1.994899] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.995564] sr0: scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray
[ 1.995604] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 1.996051] sr 1:0:0:0: Attached scsi CD-ROM sr0
[ 2.058861] sda: sda1 sda2 sda3 sda4 < sda5 sda6 >
[ 2.059713] sd 0:0:0:0: [sda] Attached SCSI disk
[ 2.080515] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 2.081139] sr 1:0:0:0: Attached scsi generic sg1 type 5
[ 2.116032] usb 1-4: new high speed USB device number 3 using ehci_hcd
[ 2.264991] usb 1-4: New USB device found, idVendor=152d, idProduct=2329
[ 2.265039] usb 1-4: New USB device strings: Mfr=10, Product=11, SerialNumber=3
[ 2.265086] usb 1-4: Product: Storagebird 35EV821
[ 2.265123] usb 1-4: Manufacturer: 0123456
[ 2.265159] usb 1-4: SerialNumber: 000000000340
[ 2.275310] usbcore: registered new interface driver uas
[ 2.288044] Initializing USB Mass Storage driver...
[ 2.288211] scsi2 : usb-storage 1-4:1.0
[ 2.288419] usbcore: registered new interface driver usb-storage
[ 2.288456] USB Mass Storage support registered.
[ 2.504027] usb 3-1: new low speed USB device number 2 using uhci_hcd
[ 2.680912] usb 3-1: New USB device found, idVendor=046d, idProduct=c00e
[ 2.680959] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 2.680998] usb 3-1: Product: USB-PS/2 Optical Mouse
[ 2.681034] usb 3-1: Manufacturer: Logitech
[ 2.721407] input: Logitech USB-PS/2 Optical Mouse as /devices/pci0000:00/0000:00:1d.1/usb3/3-1/3-1:1.0/input/input1
[ 2.721671] generic-usb 0003:046D:C00E.0001: input,hidraw0: USB HID v1.10 Mouse [Logitech USB-PS/2 Optical Mouse] on usb-0000:00:1d.1-1/input0
[ 2.721924] usbcore: registered new interface driver usbhid
[ 2.721961] usbhid: USB HID core driver
[ 2.786647] device-mapper: uevent: version 1.0.3
[ 2.787358] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02) initialised: dm-devel@redhat.com
[ 2.960900] EXT4-fs (sda5): INFO: recovery required on readonly filesystem
[ 2.960946] EXT4-fs (sda5): write access will be enabled during recovery
[ 3.330600] scsi 2:0:0:0: Direct-Access WDC WD10 EAVS-00D7B0 PQ: 0 ANSI: 2 CCS
[ 3.332261] sd 2:0:0:0: Attached scsi generic sg2 type 0
[ 3.332578] sd 2:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[ 3.333722] sd 2:0:0:0: [sdb] Write Protect is off
[ 3.333765] sd 2:0:0:0: [sdb] Mode Sense: 34 00 00 00
[ 3.333770] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[ 3.335697] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[ 3.428555] sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 sdb6 sdb7 sdb8 >
[ 3.431561] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[ 3.431609] sd 2:0:0:0: [sdb] Attached SCSI disk
[ 4.678166] EXT4-fs (sda5): recovery complete
[ 4.679423] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
[ 5.926488] systemd[1]: systemd 17 running in system mode. (+PAM -LIBWRAP +AUDIT +SELINUX +SYSVINIT +LIBCRYPTSETUP; debian)
[ 6.039692] systemd[1]: Set hostname to <tbox>.
[ 6.039899] IPv4 FIB: Using LC-trie version 0.409
[ 8.685005] cfg80211: Calling CRDA to update world regulatory domain
[ 9.855281] udev[292]: starting version 166
[ 9.919375] ath5k 0000:02:02.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
[ 9.919497] ath5k 0000:02:02.0: registered as 'phy0'
[ 10.210315] ath: EEPROM regdomain: 0x61
[ 10.210320] ath: EEPROM indicates we should expect a direct regpair map
[ 10.210326] ath: Country alpha2 being used: 00
[ 10.210329] ath: Regpair used: 0x61
[ 10.560990] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[ 10.561891] Registered led device: ath5k-phy0::rx
[ 10.561923] Registered led device: ath5k-phy0::tx
[ 10.561937] ath5k phy0: Atheros AR5212 chip found (MAC: 0x56, PHY: 0x41)
[ 10.561982] ath5k phy0: RF5111 5GHz radio found (0x17)
[ 10.562018] ath5k phy0: RF2111 2GHz radio found (0x23)
[ 10.640729] input: Lid Switch as /devices/LNXSYSTM:00/device:00/PNP0C0D:00/input/input2
[ 10.641011] ACPI: Lid Switch [LID]
[ 10.641734] input: Sleep Button as /devices/LNXSYSTM:00/device:00/PNP0C0E:00/input/input3
[ 10.641789] ACPI: Sleep Button [SLPB]
[ 10.642494] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input4
[ 10.642546] ACPI: Power Button [PWRF]
[ 10.663258] ACPI: acpi_idle registered with cpuidle
[ 10.663809] Marking TSC unstable due to TSC halts in idle
[ 10.666064] Switching to clocksource acpi_pm
[ 10.740620] ACPI: AC Adapter [AC] (on-line)
[ 10.748395] input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A03:00/device:03/LNXVIDEO:00/input/input5
[ 10.748459] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
[ 10.770950] ACPI: Battery Slot [BAT0] (battery present)
[ 10.898007] input: PC Speaker as /devices/platform/pcspkr/input/input6
[ 10.916250] parport_pc 00:0b: reported by Plug and Play ACPI
[ 10.916334] parport0: PC-style at 0x3bc, irq 7 [PCSPP,TRISTATE]
[ 10.923828] Non-volatile memory driver v1.3
[ 11.291894] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 11.360064] NET: Registered protocol family 23
[ 11.437611] nsc-ircc 00:0c: [io 0x02f8-0x02ff]
[ 11.437679] nsc-ircc 00:0c: [irq 3]
[ 11.437684] nsc-ircc 00:0c: [dma 1]
[ 11.438138] nsc-ircc 00:0c: activated
[ 11.438350] nsc-ircc, chip->init
[ 11.438393] nsc-ircc, Found chip at base=0x02e
[ 11.438453] nsc-ircc, driver loaded (Dag Brattli)
[ 11.440699] IrDA: Registered device irda0
[ 11.440738] nsc-ircc, Using dongle: IBM31T1100 or Temic TFDS6000/TFDS6500
[ 11.540896] intel_rng: FWH not detected
[ 11.587877] i801_smbus 0000:00:1f.3: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 11.589658] Synaptics Touchpad, model: 1, fw: 5.9, id: 0x2c6ab1, caps: 0x884793/0x0/0x0
[ 11.589718] serio: Synaptics pass-through port at isa0060/serio1/input0
[ 11.626562] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input7
[ 11.645104] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 12.098553] [drm] Initialized drm 1.1.0 20060810
[ 12.294407] yenta_cardbus 0000:02:00.0: CardBus bridge found [1014:0512]
[ 12.294472] yenta_cardbus 0000:02:00.0: Using INTVAL to route CSC interrupts to PCI
[ 12.294521] yenta_cardbus 0000:02:00.0: Routing CardBus interrupts to PCI
[ 12.294565] yenta_cardbus 0000:02:00.0: TI: mfunc 0x01d21022, devctl 0x64
[ 12.524939] yenta_cardbus 0000:02:00.0: ISA IRQ mask 0x0438, PCI irq 11
[ 12.524986] yenta_cardbus 0000:02:00.0: Socket status: 30000006
[ 12.525032] yenta_cardbus 0000:02:00.0: pcmcia: parent PCI bridge window: [io 0x4000-0x8fff]
[ 12.525082] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x4000-0x8fff: excluding 0x4000-0x40ff 0x4400-0x44ff 0x4800-0x48ff 0x4c00-0x4cff 0x8000-0x803f
[ 12.543165] yenta_cardbus 0000:02:00.0: pcmcia: parent PCI bridge window: [mem 0xc0200000-0xcfffffff]
[ 12.543345] pcmcia_socket pcmcia_socket0: cs: memory probe 0xc0200000-0xcfffffff: excluding 0xc0200000-0xc09fffff 0xc3a00000-0xcc1fffff 0xcfa00000-0xd01fffff
[ 12.546522] yenta_cardbus 0000:02:00.0: pcmcia: parent PCI bridge window: [mem 0xe8000000-0xefffffff pref]
[ 12.546572] pcmcia_socket pcmcia_socket0: cs: memory probe 0xe8000000-0xefffffff: excluding 0xe8000000-0xefffffff
[ 12.547167] yenta_cardbus 0000:02:00.1: CardBus bridge found [1014:0512]
[ 12.547221] yenta_cardbus 0000:02:00.1: Using INTVAL to route CSC interrupts to PCI
[ 12.547268] yenta_cardbus 0000:02:00.1: Routing CardBus interrupts to PCI
[ 12.547308] yenta_cardbus 0000:02:00.1: TI: mfunc 0x01d21022, devctl 0x64
[ 12.776998] yenta_cardbus 0000:02:00.1: ISA IRQ mask 0x0438, PCI irq 11
[ 12.777043] yenta_cardbus 0000:02:00.1: Socket status: 30000006
[ 12.780264] yenta_cardbus 0000:02:00.1: pcmcia: parent PCI bridge window: [io 0x4000-0x8fff]
[ 12.780319] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x4000-0x8fff: excluding 0x4000-0x40ff 0x4400-0x44ff 0x4800-0x48ff 0x4c00-0x4cff 0x8000-0x803f
[ 12.828214] yenta_cardbus 0000:02:00.1: pcmcia: parent PCI bridge window: [mem 0xc0200000-0xcfffffff]
[ 12.828266] pcmcia_socket pcmcia_socket1: cs: memory probe 0xc0200000-0xcfffffff: excluding 0xc0200000-0xc09fffff 0xc3a00000-0xcc1fffff 0xcfa00000-0xd01fffff
[ 12.828445] yenta_cardbus 0000:02:00.1: pcmcia: parent PCI bridge window: [mem 0xe8000000-0xefffffff pref]
[ 12.828495] pcmcia_socket pcmcia_socket1: cs: memory probe 0xe8000000-0xefffffff: excluding 0xe8000000-0xefffffff
[ 12.852248] thinkpad_acpi: ThinkPad ACPI Extras v0.24
[ 12.852293] thinkpad_acpi: http://ibm-acpi.sf.net/
[ 12.852330] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
[ 12.852369] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
[ 12.856433] thinkpad_acpi: detected a 8-level brightness capable ThinkPad
[ 12.864342] thinkpad_acpi: rfkill switch tpacpi_bluetooth_sw: radio is blocked
[ 12.864784] Registered led device: tpacpi::thinklight
[ 12.864990] Registered led device: tpacpi::power
[ 12.865047] Registered led device: tpacpi::standby
[ 12.873188] thinkpad_acpi: Console audio control enabled, mode: monitor (read only)
[ 12.876884] input: ThinkPad Extra Buttons as /devices/platform/thinkpad_acpi/input/input8
[ 13.121231] [drm] radeon kernel modesetting enabled.
[ 13.121408] radeon 0000:01:00.0: power state changed by ACPI to D0
[ 13.121448] radeon 0000:01:00.0: power state changed by ACPI to D0
[ 13.121495] radeon 0000:01:00.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
[ 13.122461] [drm] initializing kernel modesetting (RV250 0x1002:0x4C66).
[ 13.122529] [drm] register mmio base: 0xC0100000
[ 13.122564] [drm] register mmio size: 65536
[ 13.122918] agpgart-intel 0000:00:00.0: AGP 2.0 bridge
[ 13.122969] agpgart-intel 0000:00:00.0: putting AGP V2 device into 4x mode
[ 13.123046] radeon 0000:01:00.0: putting AGP V2 device into 4x mode
[ 13.123112] radeon 0000:01:00.0: GTT: 256M 0xD0000000 - 0xDFFFFFFF
[ 13.123157] radeon 0000:01:00.0: VRAM: 128M 0x00000000E0000000 - 0x00000000E7FFFFFF (64M used)
[ 13.123292] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[ 13.123330] [drm] Driver supports precise vblank timestamp query.
[ 13.123382] [drm] radeon: irq initialized.
[ 13.123619] [drm] Detected VRAM RAM=128M, BAR=128M
[ 13.123662] [drm] RAM width 128bits DDR
[ 13.123798] [TTM] Zone kernel: Available graphics memory: 447358 kiB.
[ 13.123837] [TTM] Zone highmem: Available graphics memory: 516674 kiB.
[ 13.123874] [TTM] Initializing pool allocator.
[ 13.123954] [drm] radeon: 64M of VRAM memory ready
[ 13.123991] [drm] radeon: 256M of GTT memory ready.
[ 13.125430] radeon 0000:01:00.0: WB enabled
[ 13.126343] [drm] Loading R200 Microcode
[ 13.222974] cfg80211: World regulatory domain updated:
[ 13.223018] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
[ 13.223067] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[ 13.223114] cfg80211: (2457000 KHz - 2482000 KHz @ 20000 KHz), (300 mBi, 2000 mBm)
[ 13.223161] cfg80211: (2474000 KHz - 2494000 KHz @ 20000 KHz), (300 mBi, 2000 mBm)
[ 13.223209] cfg80211: (5170000 KHz - 5250000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[ 13.223256] cfg80211: (5735000 KHz - 5835000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
[ 13.684465] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x100-0x3af: excluding 0x170-0x177 0x1f0-0x1f7 0x2f8-0x2ff 0x370-0x377
[ 13.685695] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x3e0-0x4ff: excluding 0x3f0-0x3ff 0x4d0-0x4d7
[ 13.686285] pcmcia_socket pcmcia_socket1: cs: IO port probe 0x820-0x8ff: clean.
[ 13.686746] pcmcia_socket pcmcia_socket1: cs: IO port probe 0xc00-0xcf7: clean.
[ 13.687250] pcmcia_socket pcmcia_socket1: cs: memory probe 0x0c0000-0x0fffff: excluding 0xc0000-0xd3fff 0xdc000-0xfffff
[ 13.687448] pcmcia_socket pcmcia_socket1: cs: memory probe 0xa0000000-0xa0ffffff: clean.
[ 13.687585] pcmcia_socket pcmcia_socket1: cs: memory probe 0x60000000-0x60ffffff: clean.
[ 13.687727] pcmcia_socket pcmcia_socket1: cs: IO port probe 0xa00-0xaff:
[ 13.688208] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x100-0x3af: excluding 0x170-0x177 0x1f0-0x1f7 0x2f8-0x2ff 0x370-0x377
[ 13.689450] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x3e0-0x4ff: excluding 0x3f0-0x3ff 0x4d0-0x4d7
[ 13.690039] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x820-0x8ff: clean.
[ 13.690500] pcmcia_socket pcmcia_socket0: cs: IO port probe 0xc00-0xcf7: clean.
[ 13.691002] pcmcia_socket pcmcia_socket0: cs: memory probe 0x0c0000-0x0fffff: excluding 0xc0000-0xd3fff 0xdc000-0xfffff
[ 13.691199] pcmcia_socket pcmcia_socket0: cs: memory probe 0xa0000000-0xa0ffffff: clean.
[ 13.691336] pcmcia_socket pcmcia_socket0: cs: memory probe 0x60000000-0x60ffffff: clean.
[ 13.691478] pcmcia_socket pcmcia_socket0: cs: IO port probe 0xa00-0xaff: clean.
[ 13.692719] clean.
[ 13.930042] [drm] radeon: ring at 0x00000000D0001000
[ 13.930108] [drm] ring test succeeded in 1 usecs
[ 13.930433] [drm] radeon: ib pool ready.
[ 13.930576] [drm] ib test succeeded in 0 usecs
[ 13.930890] [drm] Panel ID String: SXGA+ Single (85MHz)
[ 13.930928] [drm] Panel Size 1400x1050
[ 13.941961] [drm] radeon legacy LVDS backlight initialized
[ 13.946441] [drm] No TV DAC info found in BIOS
[ 13.946704] [drm] Radeon Display Connectors
[ 13.946741] [drm] Connector 0:
[ 13.946797] [drm] VGA
[ 13.946843] [drm] DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60
[ 13.946880] [drm] Encoders:
[ 13.946914] [drm] CRT1: INTERNAL_DAC1
[ 13.946949] [drm] Connector 1:
[ 13.946983] [drm] DVI-D
[ 13.947016] [drm] HPD1
[ 13.947050] [drm] DDC: 0x64 0x64 0x64 0x64 0x64 0x64 0x64 0x64
[ 13.947087] [drm] Encoders:
[ 13.947120] [drm] DFP1: INTERNAL_TMDS1
[ 13.947154] [drm] Connector 2:
[ 13.947188] [drm] LVDS
[ 13.947221] [drm] Encoders:
[ 13.947255] [drm] LCD1: INTERNAL_LVDS
[ 13.947290] [drm] Connector 3:
[ 13.947323] [drm] S-video
[ 13.947355] [drm] Encoders:
[ 13.947388] [drm] TV1: INTERNAL_DAC2
[ 14.005990] [drm] radeon: power management initialized
[ 14.081380] Adding 1052244k swap on /dev/sda2. Priority:0 extents:1 across:1052244k
[ 14.099615] [drm] fb mappable at 0xE0040000
[ 14.099660] [drm] vram apper at 0xE0000000
[ 14.099696] [drm] size 5914624
[ 14.099730] [drm] fb depth is 24
[ 14.099765] [drm] pitch is 5632
[ 14.212552] Console: switching to colour frame buffer device 175x65
[ 14.247024] fb0: radeondrmfb frame buffer device
[ 14.247026] drm: registered panic notifier
[ 14.252380] Intel ICH Modem 0000:00:1f.6: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 14.252737] Intel ICH Modem 0000:00:1f.6: setting latency timer to 64
[ 14.255283] [drm] Initialized radeon 2.9.0 20080528 for 0000:01:00.0 on minor 0
[ 14.269634] Intel ICH 0000:00:1f.5: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
[ 14.269986] Intel ICH 0000:00:1f.5: setting latency timer to 64
[ 14.388103] EXT4-fs (sda5): re-mounted. Opts: (null)
[ 15.196068] intel8x0_measure_ac97_clock: measured 55383 usecs (2668 samples)
[ 15.196338] intel8x0: clocking to 48000
[ 16.868046] IBM TrackPoint firmware: 0x0e, buttons: 3/3
[ 17.065516] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/serio2/input/input9
[ 17.416996] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: mblk_io_submit
[ 17.790626] fuse init (API version 7.16)
[ 18.743370] ADDRCONF(NETDEV_UP): wlan0: link is not ready
[ 20.527004] lp0: using parport0 (interrupt-driven).
[ 20.780830] P-state transition latency capped at 20 uS
[ 20.814690] ppdev: user-space parallel port driver
[ 23.086395] wlan0: authenticate with 00:04:0e:e4:00:3d (try 1)
[ 23.088259] wlan0: authenticated
[ 23.088706] wlan0: associate with 00:04:0e:e4:00:3d (try 1)
[ 23.093086] wlan0: RX AssocResp from 00:04:0e:e4:00:3d (capab=0x411 status=0 aid=1)
[ 23.093099] wlan0: associated
[ 23.095288] ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[ 24.360260] padlock_aes: VIA PadLock not detected.
[ 24.526828] Bluetooth: Core ver 2.16
[ 24.538846] NET: Registered protocol family 31
[ 24.550747] Bluetooth: HCI device and connection manager initialized
[ 24.562689] Bluetooth: HCI socket layer initialized
[ 24.574538] Bluetooth: L2CAP socket layer initialized
[ 24.586937] Bluetooth: SCO socket layer initialized
[ 24.718105] Bluetooth: RFCOMM TTY layer initialized
[ 24.731045] Bluetooth: RFCOMM socket layer initialized
[ 24.742729] Bluetooth: RFCOMM ver 1.11
[ 24.761148] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 24.772823] Bluetooth: BNEP filters: protocol multicast
[ 33.944038] wlan0: no IPv6 routers present
[ 166.888664] start_kdeinit (1379): /proc/1381/oom_adj is deprecated, please use /proc/1381/oom_score_adj instead.
[ 173.833472] EXT4-fs (sda5): re-mounted. Opts: commit=0
[ 174.007845] EXT4-fs (sda3): re-mounted. Opts: mblk_io_submit,commit=0
[ 210.798119] EXT3-fs: barriers not enabled
[ 212.451523] kjournald starting. Commit interval 5 seconds
[ 212.451904] EXT3-fs (sdb5): warning: maximal mount count reached, running e2fsck is recommended
[ 212.453206] EXT3-fs (sdb5): using internal journal
[ 212.453218] EXT3-fs (sdb5): recovery complete
[ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
[ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
[ 273.224059] sending NMI to all CPUs:
[ 273.224074] NMI backtrace for cpu 0
[ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 273.224367]
[ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
[ 273.224414] EIP is at delay_tsc+0x16/0x5e
[ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
[ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
[ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
[ 273.224466] Stack:
[ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
[ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
[ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
[ 273.224544] Call Trace:
[ 273.224559] [<c115149f>] __delay+0x9/0xb
[ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
[ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
[ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
[ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 273.224845] <IRQ>
[ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
[ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
[ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
[ 273.225154] Call Trace:
[ 273.225166] [<c115149f>] __delay+0x9/0xb
[ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
[ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
[ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
[ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
[ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
[ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 453.344036] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
[ 453.344050] sending NMI to all CPUs:
[ 453.344064] NMI backtrace for cpu 0
[ 453.344071] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 453.344356]
[ 453.344366] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 453.344385] EIP: 0060:[<c11514b3>] EFLAGS: 00000807 CPU: 0
[ 453.344402] EIP is at __const_udelay+0x12/0x1c
[ 453.344412] EAX: 8a801100 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
[ 453.344423] ESI: c13ee580 EDI: c13ee600 EBP: f5819eac ESP: f5819eac
[ 453.344433] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 453.344445] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
[ 453.344454] Stack:
[ 453.344460] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
[ 453.344484] 0000ea7e f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
[ 453.344508] c10791df 00000000 f5819f08 c1039c6c c13b7f28 8d68cac9 00000069 f5819f20
[ 453.344532] Call Trace:
[ 453.344551] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 453.344570] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 453.344584] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 453.344598] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 453.344613] [<c1039c6c>] update_process_times+0x2d/0x58
[ 453.344628] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 453.344643] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 453.344655] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 453.344669] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 453.344687] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 453.344702] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 453.344717] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 453.344732] [<c1004215>] timer_interrupt+0x15/0x1c
[ 453.344745] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 453.344759] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 453.344771] [<c10754b9>] handle_irq_event+0x36/0x51
[ 453.344784] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 453.344797] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 453.344805] <IRQ>
[ 453.344817] [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 453.344832] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 453.344846] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 453.344881] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 453.344906] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 453.344923] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 453.344936] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 453.344953] [<c128c26c>] ? rest_init+0x58/0x5a
[ 453.344966] [<c1418722>] ? start_kernel+0x315/0x31a
[ 453.344980] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 453.344987] Code: 00 8d bc 27 00 00 00 00 48 75 fd 48 5d c3 55 89 e5 ff 15 f8 f2 3f c1 5d c3 55 89 e5 64 8b 15 dc 62 47 c1 c1 e0 02 6b d2 3e f7 e2 <8d> 42 01 e8 db ff ff ff 5d c3 69 c0 c7 10 00 00 55 89 e5 e8 d6
[ 453.345112] Call Trace:
[ 453.345125] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 453.345140] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 453.345154] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 453.345168] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 453.345181] [<c1039c6c>] update_process_times+0x2d/0x58
[ 453.345194] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 453.345208] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 453.345220] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 453.345234] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 453.345250] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 453.345265] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 453.345279] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 453.345292] [<c1004215>] timer_interrupt+0x15/0x1c
[ 453.345305] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 453.345318] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 453.345331] [<c10754b9>] handle_irq_event+0x36/0x51
[ 453.345344] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 453.345356] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 453.345364] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 453.345385] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 453.345399] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 453.345428] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 453.345452] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 453.345467] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 453.345481] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 453.345494] [<c128c26c>] ? rest_init+0x58/0x5a
[ 453.345507] [<c1418722>] ? start_kernel+0x315/0x31a
[ 453.345520] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 633.464043] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
[ 633.464056] sending NMI to all CPUs:
[ 633.464068] NMI backtrace for cpu 0
[ 633.464075] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 633.464360]
[ 633.464370] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 633.464389] EIP: 0060:[<c11514db>] EFLAGS: 00000807 CPU: 0
[ 633.464406] EIP is at delay_tsc+0x1/0x5e
[ 633.464416] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
[ 633.464426] ESI: c13ee580 EDI: c13ee600 EBP: f5819ea4 ESP: f5819e9c
[ 633.464437] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 633.464449] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
[ 633.464457] Stack:
[ 633.464463] f5819ea4 c115149f f5819eac c11514bb f5819eb8 c1016532 c13ee580 f5819ed4
[ 633.464487] c1078dc1 c134e61e c134e6c2 00000000 00019a64 f5c03488 f5819ee8 c1078e36
[ 633.464511] 00000000 00000000 c13e3fa0 f5819ef4 c10791df 00000000 f5819f08 c1039c6c
[ 633.464534] Call Trace:
[ 633.464548] [<c115149f>] ? __delay+0x9/0xb
[ 633.464561] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 633.464579] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 633.464597] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 633.464611] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 633.464625] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 633.464641] [<c1039c6c>] update_process_times+0x2d/0x58
[ 633.464655] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 633.464670] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 633.464682] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 633.464696] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 633.464714] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 633.464728] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 633.464743] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 633.464758] [<c1004215>] timer_interrupt+0x15/0x1c
[ 633.464771] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 633.464785] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 633.464797] [<c10754b9>] handle_irq_event+0x36/0x51
[ 633.464810] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 633.464823] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 633.464831] <IRQ>
[ 633.464844] [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 633.464858] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 633.464873] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 633.464907] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 633.464932] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 633.464949] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 633.464963] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 633.464980] [<c128c26c>] ? rest_init+0x58/0x5a
[ 633.464993] [<c1418722>] ? start_kernel+0x315/0x31a
[ 633.465006] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 633.465014] Code: 3e f7 e2 8d 42 01 e8 db ff ff ff 5d c3 69 c0 c7 10 00 00 55 89 e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55
[ 633.465099] e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8
[ 633.465142] Call Trace:
[ 633.465154] [<c115149f>] ? __delay+0x9/0xb
[ 633.465166] [<c11514bb>] __const_udelay+0x1a/0x1c
[ 633.465180] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
[ 633.465195] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 633.465209] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 633.465223] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 633.465236] [<c1039c6c>] update_process_times+0x2d/0x58
[ 633.465248] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 633.465262] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 633.465274] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 633.465288] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 633.465304] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 633.465318] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 633.465333] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 633.465346] [<c1004215>] timer_interrupt+0x15/0x1c
[ 633.465358] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 633.465372] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 633.465384] [<c10754b9>] handle_irq_event+0x36/0x51
[ 633.465397] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 633.465410] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 633.465418] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 633.465439] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 633.465453] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 633.465482] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 633.465506] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 633.465521] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 633.465534] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 633.465548] [<c128c26c>] ? rest_init+0x58/0x5a
[ 633.465561] [<c1418722>] ? start_kernel+0x315/0x31a
[ 633.465574] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 813.584031] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
[ 813.584043] sending NMI to all CPUs:
[ 813.584056] NMI backtrace for cpu 0
[ 813.584063] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
[ 813.584348]
[ 813.584358] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
[ 813.584377] EIP: 0060:[<c11514a1>] EFLAGS: 00000002 CPU: 0
[ 813.584395] EIP is at __const_udelay+0x0/0x1c
[ 813.584405] EAX: 00418958 EBX: 00002710 ECX: c133faf5 EDX: 00000c00
[ 813.584415] ESI: c13ee580 EDI: c13ee600 EBP: f5819eb8 ESP: f5819eb0
[ 813.584426] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 813.584438] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
[ 813.584446] Stack:
[ 813.584452] c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000 00024a4a
[ 813.584477] f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4 c10791df
[ 813.584500] 00000000 f5819f08 c1039c6c c13b7ef8 6d62ef35 000000bd f5819f20 c10509e9
[ 813.584524] Call Trace:
[ 813.584544] [<c1016532>] ? arch_trigger_all_cpu_backtrace+0x50/0x62
[ 813.584562] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 813.584576] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 813.584591] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 813.584606] [<c1039c6c>] update_process_times+0x2d/0x58
[ 813.584621] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 813.584636] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 813.584648] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 813.584662] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 813.584680] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 813.584694] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 813.584710] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 813.584725] [<c1004215>] timer_interrupt+0x15/0x1c
[ 813.584738] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 813.584751] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 813.584764] [<c10754b9>] handle_irq_event+0x36/0x51
[ 813.584777] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 813.584790] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 813.584798] <IRQ>
[ 813.584810] [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 813.584824] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 813.584839] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 813.584872] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 813.584898] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 813.584923] [<f87b7c6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 813.584939] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 813.584953] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 813.584969] [<c128c26c>] ? rest_init+0x58/0x5a
[ 813.584982] [<c1418722>] ? start_kernel+0x315/0x31a
[ 813.584995] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
[ 813.585003] Code: 8d 76 00 8d bc 27 00 00 00 00 eb 0e 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 48 75 fd 48 5d c3 55 89 e5 ff 15 f8 f2 3f c1 5d c3
[ 813.585088] 89 e5 64 8b 15 dc 62 47 c1 c1 e0 02 6b d2 3e f7 e2 8d 42 01
[ 813.585128] Call Trace:
[ 813.585141] [<c1016532>] ? arch_trigger_all_cpu_backtrace+0x50/0x62
[ 813.585156] [<c1078dc1>] check_cpu_stall+0x58/0xb8
[ 813.585170] [<c1078e36>] __rcu_pending+0x15/0xc4
[ 813.585184] [<c10791df>] rcu_check_callbacks+0x6d/0x93
[ 813.585197] [<c1039c6c>] update_process_times+0x2d/0x58
[ 813.585210] [<c10509e9>] tick_sched_timer+0x6b/0x9a
[ 813.585224] [<c1047196>] __run_hrtimer+0x9c/0x111
[ 813.585236] [<c105097e>] ? tick_sched_timer+0x0/0x9a
[ 813.585249] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
[ 813.585265] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
[ 813.585280] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
[ 813.585295] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 813.585308] [<c1004215>] timer_interrupt+0x15/0x1c
[ 813.585320] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
[ 813.585334] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 813.585346] [<c10754b9>] handle_irq_event+0x36/0x51
[ 813.585359] [<c1076cc2>] ? handle_level_irq+0x0/0x63
[ 813.585372] [<c1076d0f>] handle_level_irq+0x4d/0x63
[ 813.585380] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
[ 813.585401] [<c12ac0f0>] ? common_interrupt+0x30/0x38
[ 813.585415] [<c10400d8>] ? destroy_worker+0x52/0x6c
[ 813.585443] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
[ 813.585467] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
[ 813.585492] [<f87b7c6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
[ 813.585506] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
[ 813.585520] [<c1001da3>] ? cpu_idle+0x89/0xa3
[ 813.585533] [<c128c26c>] ? rest_init+0x58/0x5a
[ 813.585546] [<c1418722>] ? start_kernel+0x315/0x31a
[ 813.585559] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 13:05 ` Sedat Dilek
@ 2011-03-25 15:55 ` Josh Triplett
0 siblings, 0 replies; 36+ messages in thread
From: Josh Triplett @ 2011-03-25 15:55 UTC (permalink / raw)
To: sedat.dilek
Cc: linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro,
Paul E. McKenney, Nick Piggin
On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
> On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
> <sedat.dilek@googlemail.com> wrote:
> > right after I have finished building a new linux-next kernel, booting
> > into desktop and archiving my build-tree (ext4) as tarball to an
> > external USB harddisk (partition there is ext3).
> > ( Yesterday, I have seen similiar call-traces in my logs, but it was
> > hard to reproduce [1]. )
> > I am unsure from where the problem aroses, if you have a hint, let me know.
> >
> > Regards,
> > - Sedat -
> >
> > [1] http://lkml.org/lkml/2011/3/24/268
> >
> > P.S.: Attached are the dmesg outputs and my kernel-config
> >
>
> I turned off the notebook for about 2hrs to avoid thermal problems and
> hoax reports.
> Jumped into desktop and started an archive job as 1st job while doing daily job.
> Yeah, it is reproducible.
[...]
> [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
> [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
15000 jiffies matches this 60-second gap, assuming you use HZ=250.
> [ 273.224059] sending NMI to all CPUs:
> [ 273.224074] NMI backtrace for cpu 0
> [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
> [ 273.224367]
> [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
> [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
> [ 273.224414] EIP is at delay_tsc+0x16/0x5e
> [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
> [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
> [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
> [ 273.224466] Stack:
> [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
> [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
> [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
> [ 273.224544] Call Trace:
> [ 273.224559] [<c115149f>] __delay+0x9/0xb
> [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
> [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
> [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
> [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
> [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
> [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
> [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
> [ 273.224845] <IRQ>
> [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
> [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
> [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
> [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
> [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
> [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
> [ 273.225154] Call Trace:
> [ 273.225166] [<c115149f>] __delay+0x9/0xb
> [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
> [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
> [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
> [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
> [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
> [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
> [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
> [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
> [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
> [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
> [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
> [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
> [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
Interesting. Looks like RCU detected a stall while the CPU sits in
cpu_idle. That *shouldn't* happen...
- Josh Triplett
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
@ 2011-03-25 15:55 ` Josh Triplett
0 siblings, 0 replies; 36+ messages in thread
From: Josh Triplett @ 2011-03-25 15:55 UTC (permalink / raw)
To: sedat.dilek
Cc: linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro,
Paul E. McKenney, Nick Piggin
On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
> On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
> <sedat.dilek@googlemail.com> wrote:
> > right after I have finished building a new linux-next kernel, booting
> > into desktop and archiving my build-tree (ext4) as tarball to an
> > external USB harddisk (partition there is ext3).
> > ( Yesterday, I have seen similiar call-traces in my logs, but it was
> > hard to reproduce [1]. )
> > I am unsure from where the problem aroses, if you have a hint, let me know.
> >
> > Regards,
> > - Sedat -
> >
> > [1] http://lkml.org/lkml/2011/3/24/268
> >
> > P.S.: Attached are the dmesg outputs and my kernel-config
> >
>
> I turned off the notebook for about 2hrs to avoid thermal problems and
> hoax reports.
> Jumped into desktop and started an archive job as 1st job while doing daily job.
> Yeah, it is reproducible.
[...]
> [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
> [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
15000 jiffies matches this 60-second gap, assuming you use HZ=250.
> [ 273.224059] sending NMI to all CPUs:
> [ 273.224074] NMI backtrace for cpu 0
> [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_stora
ge uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
> [ 273.224367]
> [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
> [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
> [ 273.224414] EIP is at delay_tsc+0x16/0x5e
> [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
> [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
> [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
> [ 273.224466] Stack:
> [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
> [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
> [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
> [ 273.224544] Call Trace:
> [ 273.224559] [<c115149f>] __delay+0x9/0xb
> [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
> [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
> [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
> [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
> [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
> [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
> [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
> [ 273.224845] <IRQ>
> [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
> [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
> [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
> [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
> [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
> [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
> [ 273.225154] Call Trace:
> [ 273.225166] [<c115149f>] __delay+0x9/0xb
> [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
> [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
> [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
> [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
> [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
> [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
> [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
> [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
> [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
> [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
> [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
> [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
> [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
Interesting. Looks like RCU detected a stall while the CPU sits in
cpu_idle. That *shouldn't* happen...
- Josh Triplett
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 15:55 ` Josh Triplett
(?)
@ 2011-03-25 16:25 ` Sedat Dilek
-1 siblings, 0 replies; 36+ messages in thread
From: Sedat Dilek @ 2011-03-25 16:25 UTC (permalink / raw)
To: Josh Triplett
Cc: linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro,
Paul E. McKenney, Nick Piggin
On Fri, Mar 25, 2011 at 4:55 PM, Josh Triplett <josh@joshtriplett.org> wrote:
> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
>> On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
>> <sedat.dilek@googlemail.com> wrote:
>> > right after I have finished building a new linux-next kernel, booting
>> > into desktop and archiving my build-tree (ext4) as tarball to an
>> > external USB harddisk (partition there is ext3).
>> > ( Yesterday, I have seen similiar call-traces in my logs, but it was
>> > hard to reproduce [1]. )
>> > I am unsure from where the problem aroses, if you have a hint, let me know.
>> >
>> > Regards,
>> > - Sedat -
>> >
>> > [1] http://lkml.org/lkml/2011/3/24/268
>> >
>> > P.S.: Attached are the dmesg outputs and my kernel-config
>> >
>>
>> I turned off the notebook for about 2hrs to avoid thermal problems and
>> hoax reports.
>> Jumped into desktop and started an archive job as 1st job while doing daily job.
>> Yeah, it is reproducible.
> [...]
>> [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
>> [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
>
> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
>
Yeah, CONFIG_HZ_250=y.
Change HZ value?
Might be worth to change (new) default CONFIG_RCU_CPU_STALL_TIMEOUT=60 ?
Increase/Decrease?
Can I change the value from user-space? If yes, how?
Any other hints?
$ egrep -i '_HZ|_RCU' /boot/config-2.6.38-next20110325-2-686-iniza
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_NO_HZ=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> [ 273.224059] sending NMI to all CPUs:
>> [ 273.224074] NMI backtrace for cpu 0
>> [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
>> [ 273.224367]
>> [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
>> [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
>> [ 273.224414] EIP is at delay_tsc+0x16/0x5e
>> [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
>> [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
>> [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
>> [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
>> [ 273.224466] Stack:
>> [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
>> [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
>> [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
>> [ 273.224544] Call Trace:
>> [ 273.224559] [<c115149f>] __delay+0x9/0xb
>> [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
>> [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
>> [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
>> [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
>> [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
>> [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
>> [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> [ 273.224845] <IRQ>
>> [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
>> [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
>> [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
>> [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>> [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
>> [ 273.225154] Call Trace:
>> [ 273.225166] [<c115149f>] __delay+0x9/0xb
>> [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
>> [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
>> [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
>> [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
>> [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
>> [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
>> [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
>> [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
>> [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
>> [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>
> Interesting. Looks like RCU detected a stall while the CPU sits in
> cpu_idle. That *shouldn't* happen...
>
The 2nd dmesg showed me (as there are no more VFS/EXT4 call traces)
the problem could occur from RCU stuff.
> - Josh Triplett
>
Regards,
- Sedat -
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 15:55 ` Josh Triplett
@ 2011-03-25 16:42 ` Paul E. McKenney
-1 siblings, 0 replies; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-25 16:42 UTC (permalink / raw)
To: Josh Triplett
Cc: sedat.dilek, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
> > <sedat.dilek@googlemail.com> wrote:
> > > right after I have finished building a new linux-next kernel, booting
> > > into desktop and archiving my build-tree (ext4) as tarball to an
> > > external USB harddisk (partition there is ext3).
> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
> > > hard to reproduce [1]. )
> > > I am unsure from where the problem aroses, if you have a hint, let me know.
> > >
> > > Regards,
> > > - Sedat -
> > >
> > > [1] http://lkml.org/lkml/2011/3/24/268
> > >
> > > P.S.: Attached are the dmesg outputs and my kernel-config
> > >
> >
> > I turned off the notebook for about 2hrs to avoid thermal problems and
> > hoax reports.
> > Jumped into desktop and started an archive job as 1st job while doing daily job.
> > Yeah, it is reproducible.
> [...]
> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
>
> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
>
> > [ 273.224059] sending NMI to all CPUs:
> > [ 273.224074] NMI backtrace for cpu 0
> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
> > [ 273.224367]
> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
> > [ 273.224466] Stack:
> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
> > [ 273.224544] Call Trace:
> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
> > [ 273.224845] <IRQ>
> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
> > [ 273.225154] Call Trace:
> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>
> Interesting. Looks like RCU detected a stall while the CPU sits in
> cpu_idle. That *shouldn't* happen...
There have been a few of these things recently that turned out to
be BIOS misconfigurations, though that would not be the first thing
I would suspect if the system had run other versions successfully.
Another possibility is that the CPU spent the full time in interrupt.
Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
get an RCU CPU stall warning.
Or I could have somehow inserted a bug in RCU. But I am not seeing
this in my testing.
Thanx, Paul
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
@ 2011-03-25 16:42 ` Paul E. McKenney
0 siblings, 0 replies; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-25 16:42 UTC (permalink / raw)
To: Josh Triplett
Cc: sedat.dilek, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
> > <sedat.dilek@googlemail.com> wrote:
> > > right after I have finished building a new linux-next kernel, booting
> > > into desktop and archiving my build-tree (ext4) as tarball to an
> > > external USB harddisk (partition there is ext3).
> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
> > > hard to reproduce [1]. )
> > > I am unsure from where the problem aroses, if you have a hint, let me know.
> > >
> > > Regards,
> > > - Sedat -
> > >
> > > [1] http://lkml.org/lkml/2011/3/24/268
> > >
> > > P.S.: Attached are the dmesg outputs and my kernel-config
> > >
> >
> > I turned off the notebook for about 2hrs to avoid thermal problems and
> > hoax reports.
> > Jumped into desktop and started an archive job as 1st job while doing daily job.
> > Yeah, it is reproducible.
> [...]
> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
>
> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
>
> > [ 273.224059] sending NMI to all CPUs:
> > [ 273.224074] NMI backtrace for cpu 0
> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_sto
rage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
> > [ 273.224367]
> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
> > [ 273.224466] Stack:
> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
> > [ 273.224544] Call Trace:
> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
> > [ 273.224845] <IRQ>
> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
> > [ 273.225154] Call Trace:
> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>
> Interesting. Looks like RCU detected a stall while the CPU sits in
> cpu_idle. That *shouldn't* happen...
There have been a few of these things recently that turned out to
be BIOS misconfigurations, though that would not be the first thing
I would suspect if the system had run other versions successfully.
Another possibility is that the CPU spent the full time in interrupt.
Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
get an RCU CPU stall warning.
Or I could have somehow inserted a bug in RCU. But I am not seeing
this in my testing.
Thanx, Paul
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 16:42 ` Paul E. McKenney
(?)
@ 2011-03-25 16:51 ` Sedat Dilek
2011-03-25 17:40 ` Sedat Dilek
2011-03-25 17:44 ` Paul E. McKenney
-1 siblings, 2 replies; 36+ messages in thread
From: Sedat Dilek @ 2011-03-25 16:51 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
>> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
>> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
>> > <sedat.dilek@googlemail.com> wrote:
>> > > right after I have finished building a new linux-next kernel, booting
>> > > into desktop and archiving my build-tree (ext4) as tarball to an
>> > > external USB harddisk (partition there is ext3).
>> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
>> > > hard to reproduce [1]. )
>> > > I am unsure from where the problem aroses, if you have a hint, let me know.
>> > >
>> > > Regards,
>> > > - Sedat -
>> > >
>> > > [1] http://lkml.org/lkml/2011/3/24/268
>> > >
>> > > P.S.: Attached are the dmesg outputs and my kernel-config
>> > >
>> >
>> > I turned off the notebook for about 2hrs to avoid thermal problems and
>> > hoax reports.
>> > Jumped into desktop and started an archive job as 1st job while doing daily job.
>> > Yeah, it is reproducible.
>> [...]
>> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
>> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
>>
>> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
>>
>> > [ 273.224059] sending NMI to all CPUs:
>> > [ 273.224074] NMI backtrace for cpu 0
>> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
>> > [ 273.224367]
>> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
>> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
>> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
>> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
>> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
>> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
>> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
>> > [ 273.224466] Stack:
>> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
>> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
>> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
>> > [ 273.224544] Call Trace:
>> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
>> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
>> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
>> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
>> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
>> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
>> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
>> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> > [ 273.224845] <IRQ>
>> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
>> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
>> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
>> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
>> > [ 273.225154] Call Trace:
>> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
>> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
>> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
>> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
>> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
>> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
>> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
>> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
>> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
>> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
>> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>>
>> Interesting. Looks like RCU detected a stall while the CPU sits in
>> cpu_idle. That *shouldn't* happen...
>
> There have been a few of these things recently that turned out to
> be BIOS misconfigurations, though that would not be the first thing
> I would suspect if the system had run other versions successfully.
> Another possibility is that the CPU spent the full time in interrupt.
> Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
> get an RCU CPU stall warning.
>
> Or I could have somehow inserted a bug in RCU. But I am not seeing
> this in my testing.
>
> Thanx, Paul
>
The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
in my configs.
This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
IIRC I have flashed the latest BIOS available for this notebook.
[ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
[ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
As I am still sitting in the dark, it would be very helpful to know if
I can play with HZ or RCU kernel-config parameters.
Can I change RCU behaviour from user-space?
- Sedat -
P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
possible values in lib/Kconfig.debug
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 16:51 ` Sedat Dilek
@ 2011-03-25 17:40 ` Sedat Dilek
2011-03-25 17:48 ` Paul E. McKenney
2011-03-25 17:44 ` Paul E. McKenney
1 sibling, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-25 17:40 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Fri, Mar 25, 2011 at 5:51 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
> On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
>> On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
>>> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
>>> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
>>> > <sedat.dilek@googlemail.com> wrote:
>>> > > right after I have finished building a new linux-next kernel, booting
>>> > > into desktop and archiving my build-tree (ext4) as tarball to an
>>> > > external USB harddisk (partition there is ext3).
>>> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
>>> > > hard to reproduce [1]. )
>>> > > I am unsure from where the problem aroses, if you have a hint, let me know.
>>> > >
>>> > > Regards,
>>> > > - Sedat -
>>> > >
>>> > > [1] http://lkml.org/lkml/2011/3/24/268
>>> > >
>>> > > P.S.: Attached are the dmesg outputs and my kernel-config
>>> > >
>>> >
>>> > I turned off the notebook for about 2hrs to avoid thermal problems and
>>> > hoax reports.
>>> > Jumped into desktop and started an archive job as 1st job while doing daily job.
>>> > Yeah, it is reproducible.
>>> [...]
>>> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
>>> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
>>>
>>> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
>>>
>>> > [ 273.224059] sending NMI to all CPUs:
>>> > [ 273.224074] NMI backtrace for cpu 0
>>> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
>>> > [ 273.224367]
>>> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
>>> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
>>> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
>>> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
>>> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
>>> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
>>> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
>>> > [ 273.224466] Stack:
>>> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
>>> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
>>> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
>>> > [ 273.224544] Call Trace:
>>> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
>>> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
>>> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>>> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>>> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
>>> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>>> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
>>> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>>> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
>>> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>>> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>>> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>>> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>>> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>>> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
>>> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>>> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>>> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
>>> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>>> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
>>> > [ 273.224845] <IRQ>
>>> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
>>> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>>> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
>>> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>>> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>>> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>>> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
>>> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
>>> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
>>> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>>> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
>>> > [ 273.225154] Call Trace:
>>> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
>>> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
>>> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>>> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>>> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
>>> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>>> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
>>> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>>> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
>>> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>>> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>>> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>>> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>>> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>>> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
>>> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>>> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>>> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
>>> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>>> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
>>> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
>>> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>>> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
>>> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>>> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>>> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>>> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
>>> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
>>> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
>>> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>>>
>>> Interesting. Looks like RCU detected a stall while the CPU sits in
>>> cpu_idle. That *shouldn't* happen...
>>
>> There have been a few of these things recently that turned out to
>> be BIOS misconfigurations, though that would not be the first thing
>> I would suspect if the system had run other versions successfully.
>> Another possibility is that the CPU spent the full time in interrupt.
>> Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
>> get an RCU CPU stall warning.
>>
>> Or I could have somehow inserted a bug in RCU. But I am not seeing
>> this in my testing.
>>
>> Thanx, Paul
>>
>
> The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
> in my configs.
>
> This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
> IIRC I have flashed the latest BIOS available for this notebook.
>
> [ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
> [ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
>
> As I am still sitting in the dark, it would be very helpful to know if
> I can play with HZ or RCU kernel-config parameters.
> Can I change RCU behaviour from user-space?
>
> - Sedat -
>
> P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
> possible values in lib/Kconfig.debug
>
OK, I had a deeper look at the RCU (STALL) kernel-configs.
$ grep RCU /boot/config-2.6.38-next20110323-3-686-iniza | grep STALL
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
$ grep RCU /boot/config-2.6.38-next20110324-2-686-iniza | grep STALL
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
$ grep RCU /boot/config-2.6.38-next20110325-2-686-iniza | grep STALL
CONFIG_RCU_CPU_STALL_TIMEOUT=60
With my todays (next-20110325) linux-next kernel I cannot work!
The yesterday call-traces could be indeed a different issue (I am
currently testing with the 2 patches from block-tree [1]).
Now, I am building a new linux-next kernel with CONFIG_TREE_RCU=y as
recommended in Documentation/RCU/stallwarn.txt file.
- Sedat -
[1] http://lkml.org/lkml/2011/3/25/326
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 16:51 ` Sedat Dilek
2011-03-25 17:40 ` Sedat Dilek
@ 2011-03-25 17:44 ` Paul E. McKenney
1 sibling, 0 replies; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-25 17:44 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Fri, Mar 25, 2011 at 05:51:09PM +0100, Sedat Dilek wrote:
> On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
> >> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
> >> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
> >> > <sedat.dilek@googlemail.com> wrote:
> >> > > right after I have finished building a new linux-next kernel, booting
> >> > > into desktop and archiving my build-tree (ext4) as tarball to an
> >> > > external USB harddisk (partition there is ext3).
> >> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
> >> > > hard to reproduce [1]. )
> >> > > I am unsure from where the problem aroses, if you have a hint, let me know.
> >> > >
> >> > > Regards,
> >> > > - Sedat -
> >> > >
> >> > > [1] http://lkml.org/lkml/2011/3/24/268
> >> > >
> >> > > P.S.: Attached are the dmesg outputs and my kernel-config
> >> > >
> >> >
> >> > I turned off the notebook for about 2hrs to avoid thermal problems and
> >> > hoax reports.
> >> > Jumped into desktop and started an archive job as 1st job while doing daily job.
> >> > Yeah, it is reproducible.
> >> [...]
> >> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
> >> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
> >>
> >> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
> >>
> >> > [ 273.224059] sending NMI to all CPUs:
> >> > [ 273.224074] NMI backtrace for cpu 0
> >> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
> >> > [ 273.224367]
> >> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
> >> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
> >> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
> >> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
> >> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
> >> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> >> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
> >> > [ 273.224466] Stack:
> >> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
> >> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
> >> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
> >> > [ 273.224544] Call Trace:
> >> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
> >> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
> >> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
> >> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
> >> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
> >> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
> >> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
> >> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >> > [ 273.224845] <IRQ>
> >> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
> >> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
> >> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
> >> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
> >> > [ 273.225154] Call Trace:
> >> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
> >> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
> >> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
> >> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
> >> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
> >> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
> >> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
> >> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
> >> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
> >> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
> >> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >>
> >> Interesting. Looks like RCU detected a stall while the CPU sits in
> >> cpu_idle. That *shouldn't* happen...
> >
> > There have been a few of these things recently that turned out to
> > be BIOS misconfigurations, though that would not be the first thing
> > I would suspect if the system had run other versions successfully.
> > Another possibility is that the CPU spent the full time in interrupt.
> > Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
> > get an RCU CPU stall warning.
> >
> > Or I could have somehow inserted a bug in RCU. But I am not seeing
> > this in my testing.
> >
> > Thanx, Paul
> >
>
> The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
> in my configs.
Stall warnings used to be disabled by default, so perhaps you started
seeing this when that changed.
> This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
> IIRC I have flashed the latest BIOS available for this notebook.
>
> [ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
> [ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
>
> As I am still sitting in the dark, it would be very helpful to know if
> I can play with HZ or RCU kernel-config parameters.
> Can I change RCU behaviour from user-space?
There is always the rcu_cpu_stall_suppress module parameter, which can
be set (to 1) from the boot command line or from sysfs. This will
stop RCU from complaining about any stalls, though it won't do anything
about the stall itself.
> - Sedat -
>
> P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
> possible values in lib/Kconfig.debug
Yep, the rcu_cpu_stall_suppress module parameter is the first thing
mentioned in Documentation/RCU/stallwarn.txt. ;-)
Thanx, Paul
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 17:40 ` Sedat Dilek
@ 2011-03-25 17:48 ` Paul E. McKenney
2011-03-25 19:42 ` Sedat Dilek
0 siblings, 1 reply; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-25 17:48 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Fri, Mar 25, 2011 at 06:40:38PM +0100, Sedat Dilek wrote:
> On Fri, Mar 25, 2011 at 5:51 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
> > On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
> > <paulmck@linux.vnet.ibm.com> wrote:
> >> On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
> >>> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
> >>> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
> >>> > <sedat.dilek@googlemail.com> wrote:
> >>> > > right after I have finished building a new linux-next kernel, booting
> >>> > > into desktop and archiving my build-tree (ext4) as tarball to an
> >>> > > external USB harddisk (partition there is ext3).
> >>> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
> >>> > > hard to reproduce [1]. )
> >>> > > I am unsure from where the problem aroses, if you have a hint, let me know.
> >>> > >
> >>> > > Regards,
> >>> > > - Sedat -
> >>> > >
> >>> > > [1] http://lkml.org/lkml/2011/3/24/268
> >>> > >
> >>> > > P.S.: Attached are the dmesg outputs and my kernel-config
> >>> > >
> >>> >
> >>> > I turned off the notebook for about 2hrs to avoid thermal problems and
> >>> > hoax reports.
> >>> > Jumped into desktop and started an archive job as 1st job while doing daily job.
> >>> > Yeah, it is reproducible.
> >>> [...]
> >>> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
> >>> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
> >>>
> >>> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
> >>>
> >>> > [ 273.224059] sending NMI to all CPUs:
> >>> > [ 273.224074] NMI backtrace for cpu 0
> >>> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
> >>> > [ 273.224367]
> >>> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
> >>> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
> >>> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
> >>> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
> >>> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
> >>> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> >>> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
> >>> > [ 273.224466] Stack:
> >>> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
> >>> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
> >>> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
> >>> > [ 273.224544] Call Trace:
> >>> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
> >>> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
> >>> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >>> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >>> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
> >>> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >>> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
> >>> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >>> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
> >>> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >>> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >>> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >>> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >>> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >>> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
> >>> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >>> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >>> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
> >>> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >>> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >>> > [ 273.224845] <IRQ>
> >>> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
> >>> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >>> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >>> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >>> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >>> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >>> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >>> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
> >>> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
> >>> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >>> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
> >>> > [ 273.225154] Call Trace:
> >>> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
> >>> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
> >>> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >>> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >>> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
> >>> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >>> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
> >>> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >>> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
> >>> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >>> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >>> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >>> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >>> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >>> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
> >>> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >>> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >>> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
> >>> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >>> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >>> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
> >>> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >>> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >>> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >>> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >>> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >>> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >>> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
> >>> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
> >>> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >>>
> >>> Interesting. Looks like RCU detected a stall while the CPU sits in
> >>> cpu_idle. That *shouldn't* happen...
> >>
> >> There have been a few of these things recently that turned out to
> >> be BIOS misconfigurations, though that would not be the first thing
> >> I would suspect if the system had run other versions successfully.
> >> Another possibility is that the CPU spent the full time in interrupt.
> >> Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
> >> get an RCU CPU stall warning.
> >>
> >> Or I could have somehow inserted a bug in RCU. But I am not seeing
> >> this in my testing.
> >>
> >> Thanx, Paul
> >>
> >
> > The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
> > in my configs.
> >
> > This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
> > IIRC I have flashed the latest BIOS available for this notebook.
> >
> > [ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
> > [ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
> >
> > As I am still sitting in the dark, it would be very helpful to know if
> > I can play with HZ or RCU kernel-config parameters.
> > Can I change RCU behaviour from user-space?
> >
> > - Sedat -
> >
> > P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
> > possible values in lib/Kconfig.debug
> >
>
> OK, I had a deeper look at the RCU (STALL) kernel-configs.
>
> $ grep RCU /boot/config-2.6.38-next20110323-3-686-iniza | grep STALL
> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>
> $ grep RCU /boot/config-2.6.38-next20110324-2-686-iniza | grep STALL
> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>
> $ grep RCU /boot/config-2.6.38-next20110325-2-686-iniza | grep STALL
> CONFIG_RCU_CPU_STALL_TIMEOUT=60
Yep, you moved from a kernel version that had the stall detected disabled
by default to one that enables it by default.
But -next has had stall detection enabled by default for a good
long time now.
> With my todays (next-20110325) linux-next kernel I cannot work!
> The yesterday call-traces could be indeed a different issue (I am
> currently testing with the 2 patches from block-tree [1]).
>
> Now, I am building a new linux-next kernel with CONFIG_TREE_RCU=y as
> recommended in Documentation/RCU/stallwarn.txt file.
You had CONFIG_TREE_PREEMPT_RCU=y earlier? Tiny RCU does not have
a stall detector.
Thanx, Paul
> - Sedat -
>
> [1] http://lkml.org/lkml/2011/3/25/326
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 17:48 ` Paul E. McKenney
@ 2011-03-25 19:42 ` Sedat Dilek
2011-03-26 3:42 ` Paul E. McKenney
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-25 19:42 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Fri, Mar 25, 2011 at 6:48 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Fri, Mar 25, 2011 at 06:40:38PM +0100, Sedat Dilek wrote:
>> On Fri, Mar 25, 2011 at 5:51 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
>> > On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
>> > <paulmck@linux.vnet.ibm.com> wrote:
>> >> On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
>> >>> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
>> >>> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
>> >>> > <sedat.dilek@googlemail.com> wrote:
>> >>> > > right after I have finished building a new linux-next kernel, booting
>> >>> > > into desktop and archiving my build-tree (ext4) as tarball to an
>> >>> > > external USB harddisk (partition there is ext3).
>> >>> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
>> >>> > > hard to reproduce [1]. )
>> >>> > > I am unsure from where the problem aroses, if you have a hint, let me know.
>> >>> > >
>> >>> > > Regards,
>> >>> > > - Sedat -
>> >>> > >
>> >>> > > [1] http://lkml.org/lkml/2011/3/24/268
>> >>> > >
>> >>> > > P.S.: Attached are the dmesg outputs and my kernel-config
>> >>> > >
>> >>> >
>> >>> > I turned off the notebook for about 2hrs to avoid thermal problems and
>> >>> > hoax reports.
>> >>> > Jumped into desktop and started an archive job as 1st job while doing daily job.
>> >>> > Yeah, it is reproducible.
>> >>> [...]
>> >>> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
>> >>> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
>> >>>
>> >>> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
>> >>>
>> >>> > [ 273.224059] sending NMI to all CPUs:
>> >>> > [ 273.224074] NMI backtrace for cpu 0
>> >>> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
>> >>> > [ 273.224367]
>> >>> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
>> >>> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
>> >>> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
>> >>> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
>> >>> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
>> >>> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
>> >>> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
>> >>> > [ 273.224466] Stack:
>> >>> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
>> >>> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
>> >>> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
>> >>> > [ 273.224544] Call Trace:
>> >>> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
>> >>> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
>> >>> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> >>> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> >>> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
>> >>> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> >>> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
>> >>> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> >>> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
>> >>> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> >>> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> >>> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> >>> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> >>> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >>> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
>> >>> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> >>> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >>> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
>> >>> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >>> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> >>> > [ 273.224845] <IRQ>
>> >>> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
>> >>> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> >>> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> >>> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> >>> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> >>> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> >>> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> >>> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
>> >>> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
>> >>> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>> >>> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
>> >>> > [ 273.225154] Call Trace:
>> >>> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
>> >>> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
>> >>> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> >>> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> >>> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
>> >>> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> >>> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
>> >>> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> >>> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
>> >>> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> >>> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> >>> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> >>> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> >>> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >>> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
>> >>> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> >>> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >>> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
>> >>> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >>> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> >>> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
>> >>> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> >>> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> >>> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> >>> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> >>> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> >>> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> >>> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
>> >>> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
>> >>> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>> >>>
>> >>> Interesting. Looks like RCU detected a stall while the CPU sits in
>> >>> cpu_idle. That *shouldn't* happen...
>> >>
>> >> There have been a few of these things recently that turned out to
>> >> be BIOS misconfigurations, though that would not be the first thing
>> >> I would suspect if the system had run other versions successfully.
>> >> Another possibility is that the CPU spent the full time in interrupt.
>> >> Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
>> >> get an RCU CPU stall warning.
>> >>
>> >> Or I could have somehow inserted a bug in RCU. But I am not seeing
>> >> this in my testing.
>> >>
>> >> Thanx, Paul
>> >>
>> >
>> > The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> > in my configs.
>> >
>> > This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
>> > IIRC I have flashed the latest BIOS available for this notebook.
>> >
>> > [ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
>> > [ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
>> >
>> > As I am still sitting in the dark, it would be very helpful to know if
>> > I can play with HZ or RCU kernel-config parameters.
>> > Can I change RCU behaviour from user-space?
>> >
>> > - Sedat -
>> >
>> > P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
>> > possible values in lib/Kconfig.debug
>> >
>>
>> OK, I had a deeper look at the RCU (STALL) kernel-configs.
>>
>> $ grep RCU /boot/config-2.6.38-next20110323-3-686-iniza | grep STALL
>> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>>
>> $ grep RCU /boot/config-2.6.38-next20110324-2-686-iniza | grep STALL
>> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>>
>> $ grep RCU /boot/config-2.6.38-next20110325-2-686-iniza | grep STALL
>> CONFIG_RCU_CPU_STALL_TIMEOUT=60
>
> Yep, you moved from a kernel version that had the stall detected disabled
> by default to one that enables it by default.
>
> But -next has had stall detection enabled by default for a good
> long time now.
>
>> With my todays (next-20110325) linux-next kernel I cannot work!
>> The yesterday call-traces could be indeed a different issue (I am
>> currently testing with the 2 patches from block-tree [1]).
>>
>> Now, I am building a new linux-next kernel with CONFIG_TREE_RCU=y as
>> recommended in Documentation/RCU/stallwarn.txt file.
>
> You had CONFIG_TREE_PREEMPT_RCU=y earlier? Tiny RCU does not have
> a stall detector.
>
> Thanx, Paul
>
>> - Sedat -
>>
>> [1] http://lkml.org/lkml/2011/3/25/326
>
No, and I have here SMP configured.
Yesterday's RCU and SMP kernel config settings:
# egrep '_RCU|RCU_|_SMP' /boot/config-2.6.38-next20110324-2-686-iniza
CONFIG_X86_32_SMP=y
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_SMP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_SCSI_SAS_HOST_SMP=y
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
IIRC Tiny RCU and SMP bite each other?
So, what do you recommend for an UP processor machine?
- Sedat -
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-25 19:42 ` Sedat Dilek
@ 2011-03-26 3:42 ` Paul E. McKenney
2011-03-26 8:11 ` Sedat Dilek
2011-03-26 12:34 ` Sedat Dilek
0 siblings, 2 replies; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-26 3:42 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Fri, Mar 25, 2011 at 08:42:14PM +0100, Sedat Dilek wrote:
> On Fri, Mar 25, 2011 at 6:48 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Fri, Mar 25, 2011 at 06:40:38PM +0100, Sedat Dilek wrote:
> >> On Fri, Mar 25, 2011 at 5:51 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
> >> > On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
> >> > <paulmck@linux.vnet.ibm.com> wrote:
> >> >> On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
> >> >>> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
> >> >>> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
> >> >>> > <sedat.dilek@googlemail.com> wrote:
> >> >>> > > right after I have finished building a new linux-next kernel, booting
> >> >>> > > into desktop and archiving my build-tree (ext4) as tarball to an
> >> >>> > > external USB harddisk (partition there is ext3).
> >> >>> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
> >> >>> > > hard to reproduce [1]. )
> >> >>> > > I am unsure from where the problem aroses, if you have a hint, let me know.
> >> >>> > >
> >> >>> > > Regards,
> >> >>> > > - Sedat -
> >> >>> > >
> >> >>> > > [1] http://lkml.org/lkml/2011/3/24/268
> >> >>> > >
> >> >>> > > P.S.: Attached are the dmesg outputs and my kernel-config
> >> >>> > >
> >> >>> >
> >> >>> > I turned off the notebook for about 2hrs to avoid thermal problems and
> >> >>> > hoax reports.
> >> >>> > Jumped into desktop and started an archive job as 1st job while doing daily job.
> >> >>> > Yeah, it is reproducible.
> >> >>> [...]
> >> >>> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
> >> >>> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
> >> >>>
> >> >>> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
> >> >>>
> >> >>> > [ 273.224059] sending NMI to all CPUs:
> >> >>> > [ 273.224074] NMI backtrace for cpu 0
> >> >>> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
> >> >>> > [ 273.224367]
> >> >>> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
> >> >>> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
> >> >>> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
> >> >>> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
> >> >>> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
> >> >>> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> >> >>> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
> >> >>> > [ 273.224466] Stack:
> >> >>> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
> >> >>> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
> >> >>> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
> >> >>> > [ 273.224544] Call Trace:
> >> >>> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
> >> >>> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
> >> >>> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >> >>> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >> >>> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
> >> >>> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >> >>> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
> >> >>> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >> >>> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
> >> >>> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >> >>> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >> >>> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >> >>> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >> >>> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >>> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
> >> >>> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >> >>> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >>> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
> >> >>> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >>> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >> >>> > [ 273.224845] <IRQ>
> >> >>> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
> >> >>> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >> >>> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >> >>> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >> >>> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >> >>> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >> >>> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >> >>> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
> >> >>> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
> >> >>> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >> >>> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
> >> >>> > [ 273.225154] Call Trace:
> >> >>> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
> >> >>> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
> >> >>> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >> >>> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >> >>> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
> >> >>> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >> >>> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
> >> >>> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >> >>> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
> >> >>> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >> >>> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >> >>> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >> >>> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >> >>> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >>> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
> >> >>> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >> >>> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >>> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
> >> >>> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >>> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >> >>> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
> >> >>> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >> >>> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >> >>> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >> >>> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >> >>> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >> >>> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >> >>> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
> >> >>> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
> >> >>> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >> >>>
> >> >>> Interesting. Looks like RCU detected a stall while the CPU sits in
> >> >>> cpu_idle. That *shouldn't* happen...
> >> >>
> >> >> There have been a few of these things recently that turned out to
> >> >> be BIOS misconfigurations, though that would not be the first thing
> >> >> I would suspect if the system had run other versions successfully.
> >> >> Another possibility is that the CPU spent the full time in interrupt.
> >> >> Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
> >> >> get an RCU CPU stall warning.
> >> >>
> >> >> Or I could have somehow inserted a bug in RCU. But I am not seeing
> >> >> this in my testing.
> >> >>
> >> >> Thanx, Paul
> >> >>
> >> >
> >> > The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
> >> > in my configs.
> >> >
> >> > This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
> >> > IIRC I have flashed the latest BIOS available for this notebook.
> >> >
> >> > [ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
> >> > [ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
> >> >
> >> > As I am still sitting in the dark, it would be very helpful to know if
> >> > I can play with HZ or RCU kernel-config parameters.
> >> > Can I change RCU behaviour from user-space?
> >> >
> >> > - Sedat -
> >> >
> >> > P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
> >> > possible values in lib/Kconfig.debug
> >> >
> >>
> >> OK, I had a deeper look at the RCU (STALL) kernel-configs.
> >>
> >> $ grep RCU /boot/config-2.6.38-next20110323-3-686-iniza | grep STALL
> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >>
> >> $ grep RCU /boot/config-2.6.38-next20110324-2-686-iniza | grep STALL
> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >>
> >> $ grep RCU /boot/config-2.6.38-next20110325-2-686-iniza | grep STALL
> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
> >
> > Yep, you moved from a kernel version that had the stall detected disabled
> > by default to one that enables it by default.
> >
> > But -next has had stall detection enabled by default for a good
> > long time now.
> >
> >> With my todays (next-20110325) linux-next kernel I cannot work!
> >> The yesterday call-traces could be indeed a different issue (I am
> >> currently testing with the 2 patches from block-tree [1]).
> >>
> >> Now, I am building a new linux-next kernel with CONFIG_TREE_RCU=y as
> >> recommended in Documentation/RCU/stallwarn.txt file.
> >
> > You had CONFIG_TREE_PREEMPT_RCU=y earlier? Tiny RCU does not have
> > a stall detector.
> >
> > Thanx, Paul
> >
> >> - Sedat -
> >>
> >> [1] http://lkml.org/lkml/2011/3/25/326
> >
>
> No, and I have here SMP configured.
> Yesterday's RCU and SMP kernel config settings:
>
> # egrep '_RCU|RCU_|_SMP' /boot/config-2.6.38-next20110324-2-686-iniza
> CONFIG_X86_32_SMP=y
> CONFIG_TREE_RCU=y
> # CONFIG_PREEMPT_RCU is not set
> # CONFIG_RCU_TRACE is not set
> CONFIG_RCU_FANOUT=32
> # CONFIG_RCU_FANOUT_EXACT is not set
> CONFIG_RCU_FAST_NO_HZ=y
> # CONFIG_TREE_RCU_TRACE is not set
> CONFIG_USE_GENERIC_SMP_HELPERS=y
> CONFIG_SMP=y
> CONFIG_PM_SLEEP_SMP=y
> CONFIG_HAVE_TEXT_POKE_SMP=y
> CONFIG_SCSI_SAS_HOST_SMP=y
> # CONFIG_SPARSE_RCU_POINTER is not set
> # CONFIG_RCU_TORTURE_TEST is not set
> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>
> IIRC Tiny RCU and SMP bite each other?
> So, what do you recommend for an UP processor machine?
If you want RCU stall warnings, or if you are building an SMP kernel, it
has to be either TREE_RCU or TREE_PREEMPT_RCU. If you are on UP and don't
care about RCU stall warnings, then either TINY_RCU or TINY_PREEMPT_RCU
will work fine.
I just saw your "Now, I am building a new linux-next kernel with
CONFIG_TREE_RCU=y" and thought that you were hinting that you had
been running with something other than TREE_RCU.
> - Sedat -
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-26 3:42 ` Paul E. McKenney
@ 2011-03-26 8:11 ` Sedat Dilek
2011-03-26 15:53 ` Paul E. McKenney
2011-03-26 12:34 ` Sedat Dilek
1 sibling, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-26 8:11 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
[-- Attachment #1: Type: text/plain, Size: 14653 bytes --]
On Sat, Mar 26, 2011 at 4:42 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Fri, Mar 25, 2011 at 08:42:14PM +0100, Sedat Dilek wrote:
>> On Fri, Mar 25, 2011 at 6:48 PM, Paul E. McKenney
>> <paulmck@linux.vnet.ibm.com> wrote:
>> > On Fri, Mar 25, 2011 at 06:40:38PM +0100, Sedat Dilek wrote:
>> >> On Fri, Mar 25, 2011 at 5:51 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
>> >> > On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
>> >> > <paulmck@linux.vnet.ibm.com> wrote:
>> >> >> On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
>> >> >>> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
>> >> >>> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
>> >> >>> > <sedat.dilek@googlemail.com> wrote:
>> >> >>> > > right after I have finished building a new linux-next kernel, booting
>> >> >>> > > into desktop and archiving my build-tree (ext4) as tarball to an
>> >> >>> > > external USB harddisk (partition there is ext3).
>> >> >>> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
>> >> >>> > > hard to reproduce [1]. )
>> >> >>> > > I am unsure from where the problem aroses, if you have a hint, let me know.
>> >> >>> > >
>> >> >>> > > Regards,
>> >> >>> > > - Sedat -
>> >> >>> > >
>> >> >>> > > [1] http://lkml.org/lkml/2011/3/24/268
>> >> >>> > >
>> >> >>> > > P.S.: Attached are the dmesg outputs and my kernel-config
>> >> >>> > >
>> >> >>> >
>> >> >>> > I turned off the notebook for about 2hrs to avoid thermal problems and
>> >> >>> > hoax reports.
>> >> >>> > Jumped into desktop and started an archive job as 1st job while doing daily job.
>> >> >>> > Yeah, it is reproducible.
>> >> >>> [...]
>> >> >>> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
>> >> >>> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
>> >> >>>
>> >> >>> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
>> >> >>>
>> >> >>> > [ 273.224059] sending NMI to all CPUs:
>> >> >>> > [ 273.224074] NMI backtrace for cpu 0
>> >> >>> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
>> >> >>> > [ 273.224367]
>> >> >>> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
>> >> >>> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
>> >> >>> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
>> >> >>> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
>> >> >>> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
>> >> >>> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
>> >> >>> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
>> >> >>> > [ 273.224466] Stack:
>> >> >>> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
>> >> >>> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
>> >> >>> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
>> >> >>> > [ 273.224544] Call Trace:
>> >> >>> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
>> >> >>> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
>> >> >>> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> >> >>> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> >> >>> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
>> >> >>> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> >> >>> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
>> >> >>> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> >> >>> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
>> >> >>> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> >> >>> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> >> >>> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> >> >>> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> >> >>> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
>> >> >>> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> >> >>> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
>> >> >>> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> >> >>> > [ 273.224845] <IRQ>
>> >> >>> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
>> >> >>> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> >> >>> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> >> >>> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> >> >>> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> >> >>> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> >> >>> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> >> >>> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
>> >> >>> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
>> >> >>> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>> >> >>> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
>> >> >>> > [ 273.225154] Call Trace:
>> >> >>> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
>> >> >>> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
>> >> >>> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> >> >>> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> >> >>> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
>> >> >>> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> >> >>> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
>> >> >>> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> >> >>> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
>> >> >>> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> >> >>> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> >> >>> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> >> >>> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> >> >>> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
>> >> >>> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> >> >>> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
>> >> >>> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> >> >>> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
>> >> >>> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> >> >>> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> >> >>> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> >> >>> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> >> >>> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> >> >>> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> >> >>> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
>> >> >>> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
>> >> >>> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>> >> >>>
>> >> >>> Interesting. Looks like RCU detected a stall while the CPU sits in
>> >> >>> cpu_idle. That *shouldn't* happen...
>> >> >>
>> >> >> There have been a few of these things recently that turned out to
>> >> >> be BIOS misconfigurations, though that would not be the first thing
>> >> >> I would suspect if the system had run other versions successfully.
>> >> >> Another possibility is that the CPU spent the full time in interrupt.
>> >> >> Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
>> >> >> get an RCU CPU stall warning.
>> >> >>
>> >> >> Or I could have somehow inserted a bug in RCU. But I am not seeing
>> >> >> this in my testing.
>> >> >>
>> >> >> Thanx, Paul
>> >> >>
>> >> >
>> >> > The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> >> > in my configs.
>> >> >
>> >> > This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
>> >> > IIRC I have flashed the latest BIOS available for this notebook.
>> >> >
>> >> > [ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
>> >> > [ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
>> >> >
>> >> > As I am still sitting in the dark, it would be very helpful to know if
>> >> > I can play with HZ or RCU kernel-config parameters.
>> >> > Can I change RCU behaviour from user-space?
>> >> >
>> >> > - Sedat -
>> >> >
>> >> > P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
>> >> > possible values in lib/Kconfig.debug
>> >> >
>> >>
>> >> OK, I had a deeper look at the RCU (STALL) kernel-configs.
>> >>
>> >> $ grep RCU /boot/config-2.6.38-next20110323-3-686-iniza | grep STALL
>> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>> >>
>> >> $ grep RCU /boot/config-2.6.38-next20110324-2-686-iniza | grep STALL
>> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>> >>
>> >> $ grep RCU /boot/config-2.6.38-next20110325-2-686-iniza | grep STALL
>> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> >
>> > Yep, you moved from a kernel version that had the stall detected disabled
>> > by default to one that enables it by default.
>> >
>> > But -next has had stall detection enabled by default for a good
>> > long time now.
>> >
>> >> With my todays (next-20110325) linux-next kernel I cannot work!
>> >> The yesterday call-traces could be indeed a different issue (I am
>> >> currently testing with the 2 patches from block-tree [1]).
>> >>
>> >> Now, I am building a new linux-next kernel with CONFIG_TREE_RCU=y as
>> >> recommended in Documentation/RCU/stallwarn.txt file.
>> >
>> > You had CONFIG_TREE_PREEMPT_RCU=y earlier? Tiny RCU does not have
>> > a stall detector.
>> >
>> > Thanx, Paul
>> >
>> >> - Sedat -
>> >>
>> >> [1] http://lkml.org/lkml/2011/3/25/326
>> >
>>
>> No, and I have here SMP configured.
>> Yesterday's RCU and SMP kernel config settings:
>>
>> # egrep '_RCU|RCU_|_SMP' /boot/config-2.6.38-next20110324-2-686-iniza
>> CONFIG_X86_32_SMP=y
>> CONFIG_TREE_RCU=y
>> # CONFIG_PREEMPT_RCU is not set
>> # CONFIG_RCU_TRACE is not set
>> CONFIG_RCU_FANOUT=32
>> # CONFIG_RCU_FANOUT_EXACT is not set
>> CONFIG_RCU_FAST_NO_HZ=y
>> # CONFIG_TREE_RCU_TRACE is not set
>> CONFIG_USE_GENERIC_SMP_HELPERS=y
>> CONFIG_SMP=y
>> CONFIG_PM_SLEEP_SMP=y
>> CONFIG_HAVE_TEXT_POKE_SMP=y
>> CONFIG_SCSI_SAS_HOST_SMP=y
>> # CONFIG_SPARSE_RCU_POINTER is not set
>> # CONFIG_RCU_TORTURE_TEST is not set
>> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>>
>> IIRC Tiny RCU and SMP bite each other?
>> So, what do you recommend for an UP processor machine?
>
> If you want RCU stall warnings, or if you are building an SMP kernel, it
> has to be either TREE_RCU or TREE_PREEMPT_RCU. If you are on UP and don't
> care about RCU stall warnings, then either TINY_RCU or TINY_PREEMPT_RCU
> will work fine.
>
> I just saw your "Now, I am building a new linux-next kernel with
> CONFIG_TREE_RCU=y" and thought that you were hinting that you had
> been running with something other than TREE_RCU.
>
I could suppress RCU STALL warning via sysfs, but the complete system
is just freaky.
With doing a simple tar (without compression!) job, I can't use my webbrowser.
Selecting TINY_RCU means no CONFIG_SMP!
(OK, I can hack the Kconfig dependency to change this.)
I am thinking of distributions, shall they offer another linux-kernel
binary especially for UP machines?
Somehow ridiculuous.
I still haven't get some hints to play with the timeouts, so I tried
CONFIG_RCU_CPU_STALL_TIMEOUT=120 (default according to Kconfig entry
is 60).
Can this really help here. Why?
I had a closer look into my logs, there I saw a max jiffies of
t=780510 (six digits!)
/var/log/syslog.1:Mar 25 18:09:15 tbox kernel: [ 3221.112034] INFO:
rcu_sched_state detected stall on CPU 0 (t=780510 jiffies)
And I asked for the common play between HZ and RCU values in
kernel-config parameters.
Currently, I have CONFIG_HZ_250=y.
Can you enlighten me. please.
I have the impression that even I could disable "old"
CONFIG_RCU_CPU_STALL_DETECTOR value, this stalling here has other
reasons.
BROKEN ACPI? Might be... But what about supporting such systems (with
quirks or whatever)?
- Sedat -
P.S.: I have attached all jiffies values from my today's and
yesterday's syslogs and my last used kernel-config.
>> - Sedat -
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at http://www.tux.org/lkml/
>
[-- Attachment #2: RCU_jiffies.txt --]
[-- Type: text/plain, Size: 7599 bytes --]
/var/log/syslog:Mar 26 00:36:55 tbox kernel: [ 589.064031] INFO: rcu_sched_state detected stall on CPU 0 (t=120030 jiffies)
/var/log/syslog:Mar 26 00:42:55 tbox kernel: [ 949.184032] INFO: rcu_sched_state detected stall on CPU 0 (t=210060 jiffies)
/var/log/syslog:Mar 26 00:48:55 tbox kernel: [ 1309.304033] INFO: rcu_sched_state detected stall on CPU 0 (t=300090 jiffies)
/var/log/syslog:Mar 26 01:11:09 tbox kernel: [ 208.416034] INFO: rcu_sched_state detected stall on CPU 0 (t=30000 jiffies)
/var/log/syslog:Mar 26 01:16:30 tbox kernel: [ 154.532013] INFO: rcu_sched_state detected stall on CPU 0 (t=30000 jiffies)
/var/log/syslog.1:Mar 25 10:57:04 tbox kernel: [ 157.340021] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
/var/log/syslog.1:Mar 25 11:00:04 tbox kernel: [ 337.460033] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
/var/log/syslog.1:Mar 25 11:03:05 tbox kernel: [ 517.580032] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
/var/log/syslog.1:Mar 25 11:06:05 tbox kernel: [ 697.700033] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
/var/log/syslog.1:Mar 25 11:09:05 tbox kernel: [ 877.820034] INFO: rcu_sched_state detected stall on CPU 0 (t=195120 jiffies)
/var/log/syslog.1:Mar 25 11:12:05 tbox kernel: [ 1057.940032] INFO: rcu_sched_state detected stall on CPU 0 (t=240150 jiffies)
/var/log/syslog.1:Mar 25 11:15:05 tbox kernel: [ 1238.060029] INFO: rcu_sched_state detected stall on CPU 0 (t=285180 jiffies)
/var/log/syslog.1:Mar 25 11:18:05 tbox kernel: [ 1418.180033] INFO: rcu_sched_state detected stall on CPU 0 (t=330210 jiffies)
/var/log/syslog.1:Mar 25 11:21:05 tbox kernel: [ 1598.300034] INFO: rcu_sched_state detected stall on CPU 0 (t=375240 jiffies)
/var/log/syslog.1:Mar 25 11:24:05 tbox kernel: [ 1778.420018] INFO: rcu_sched_state detected stall on CPU 0 (t=420270 jiffies)
/var/log/syslog.1:Mar 25 11:27:06 tbox kernel: [ 1958.540024] INFO: rcu_sched_state detected stall on CPU 0 (t=465300 jiffies)
/var/log/syslog.1:Mar 25 11:30:06 tbox kernel: [ 2138.660008] INFO: rcu_sched_state detected stall on CPU 0 (t=510330 jiffies)
/var/log/syslog.1:Mar 25 11:33:06 tbox kernel: [ 2318.780033] INFO: rcu_sched_state detected stall on CPU 0 (t=555360 jiffies)
/var/log/syslog.1:Mar 25 13:54:55 tbox kernel: [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
/var/log/syslog.1:Mar 25 13:57:56 tbox kernel: [ 453.344036] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
/var/log/syslog.1:Mar 25 14:00:56 tbox kernel: [ 633.464043] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
/var/log/syslog.1:Mar 25 14:03:56 tbox kernel: [ 813.584031] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
/var/log/syslog.1:Mar 25 14:06:56 tbox kernel: [ 993.704031] INFO: rcu_sched_state detected stall on CPU 0 (t=195120 jiffies)
/var/log/syslog.1:Mar 25 14:09:56 tbox kernel: [ 1173.824042] INFO: rcu_sched_state detected stall on CPU 0 (t=240150 jiffies)
/var/log/syslog.1:Mar 25 14:12:56 tbox kernel: [ 1353.944020] INFO: rcu_sched_state detected stall on CPU 0 (t=285180 jiffies)
/var/log/syslog.1:Mar 25 14:15:56 tbox kernel: [ 1534.064033] INFO: rcu_sched_state detected stall on CPU 0 (t=330210 jiffies)
/var/log/syslog.1:Mar 25 14:18:56 tbox kernel: [ 1714.184033] INFO: rcu_sched_state detected stall on CPU 0 (t=375240 jiffies)
/var/log/syslog.1:Mar 25 14:21:56 tbox kernel: [ 1894.304038] INFO: rcu_sched_state detected stall on CPU 0 (t=420270 jiffies)
/var/log/syslog.1:Mar 25 14:24:57 tbox kernel: [ 2074.424033] INFO: rcu_sched_state detected stall on CPU 0 (t=465300 jiffies)
/var/log/syslog.1:Mar 25 17:18:13 tbox kernel: [ 159.072032] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
/var/log/syslog.1:Mar 25 17:21:13 tbox kernel: [ 339.192034] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
/var/log/syslog.1:Mar 25 17:24:13 tbox kernel: [ 519.312032] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
/var/log/syslog.1:Mar 25 17:27:13 tbox kernel: [ 699.432033] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
/var/log/syslog.1:Mar 25 17:30:14 tbox kernel: [ 879.552031] INFO: rcu_sched_state detected stall on CPU 0 (t=195120 jiffies)
/var/log/syslog.1:Mar 25 17:33:14 tbox kernel: [ 1059.672035] INFO: rcu_sched_state detected stall on CPU 0 (t=240150 jiffies)
/var/log/syslog.1:Mar 25 17:36:14 tbox kernel: [ 1239.792026] INFO: rcu_sched_state detected stall on CPU 0 (t=285180 jiffies)
/var/log/syslog.1:Mar 25 17:39:14 tbox kernel: [ 1419.912036] INFO: rcu_sched_state detected stall on CPU 0 (t=330210 jiffies)
/var/log/syslog.1:Mar 25 17:42:14 tbox kernel: [ 1600.032035] INFO: rcu_sched_state detected stall on CPU 0 (t=375240 jiffies)
/var/log/syslog.1:Mar 25 17:45:14 tbox kernel: [ 1780.152033] INFO: rcu_sched_state detected stall on CPU 0 (t=420270 jiffies)
/var/log/syslog.1:Mar 25 17:48:14 tbox kernel: [ 1960.272034] INFO: rcu_sched_state detected stall on CPU 0 (t=465300 jiffies)
/var/log/syslog.1:Mar 25 17:51:14 tbox kernel: [ 2140.392012] INFO: rcu_sched_state detected stall on CPU 0 (t=510330 jiffies)
/var/log/syslog.1:Mar 25 17:54:15 tbox kernel: [ 2320.512031] INFO: rcu_sched_state detected stall on CPU 0 (t=555360 jiffies)
/var/log/syslog.1:Mar 25 17:57:15 tbox kernel: [ 2500.632036] INFO: rcu_sched_state detected stall on CPU 0 (t=600390 jiffies)
/var/log/syslog.1:Mar 25 18:00:15 tbox kernel: [ 2680.751704] INFO: rcu_sched_state detected stall on CPU 0 (t=645420 jiffies)
/var/log/syslog.1:Mar 25 18:03:15 tbox kernel: [ 2860.872032] INFO: rcu_sched_state detected stall on CPU 0 (t=690450 jiffies)
/var/log/syslog.1:Mar 25 18:06:15 tbox kernel: [ 3040.992034] INFO: rcu_sched_state detected stall on CPU 0 (t=735480 jiffies)
/var/log/syslog.1:Mar 25 18:09:15 tbox kernel: [ 3221.112034] INFO: rcu_sched_state detected stall on CPU 0 (t=780510 jiffies)
/var/log/syslog.1:Mar 25 20:37:17 tbox kernel: [ 162.424039] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
/var/log/syslog.1:Mar 25 20:40:17 tbox kernel: [ 342.544034] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
/var/log/syslog.1:Mar 25 20:43:17 tbox kernel: [ 522.664036] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
/var/log/syslog.1:Mar 25 20:46:17 tbox kernel: [ 702.784032] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
/var/log/syslog.1:Mar 25 20:49:17 tbox kernel: [ 882.904036] INFO: rcu_sched_state detected stall on CPU 0 (t=195120 jiffies)
/var/log/syslog.1:Mar 25 20:52:17 tbox kernel: [ 1063.024052] INFO: rcu_sched_state detected stall on CPU 0 (t=240150 jiffies)
/var/log/syslog.1:Mar 25 20:55:17 tbox kernel: [ 1243.144034] INFO: rcu_sched_state detected stall on CPU 0 (t=285180 jiffies)
/var/log/syslog.1:Mar 25 20:58:17 tbox kernel: [ 1423.264034] INFO: rcu_sched_state detected stall on CPU 0 (t=330210 jiffies)
/var/log/syslog.1:Mar 25 21:01:18 tbox kernel: [ 1603.384032] INFO: rcu_sched_state detected stall on CPU 0 (t=375240 jiffies)
/var/log/syslog.1:Mar 25 21:04:18 tbox kernel: [ 1783.504018] INFO: rcu_sched_state detected stall on CPU 0 (t=420270 jiffies)
/var/log/syslog.1:Mar 25 21:07:18 tbox kernel: [ 1963.624034] INFO: rcu_sched_state detected stall on CPU 0 (t=465300 jiffies)
/var/log/syslog.1:Mar 25 21:10:19 tbox kernel: [ 2143.744028] INFO: rcu_sched_state detected stall on CPU 0 (t=510330 jiffies)
/var/log/syslog.1:Mar 26 00:30:55 tbox kernel: [ 228.944029] INFO: rcu_sched_state detected stall on CPU 0 (t=30000 jiffies)
[-- Attachment #3: config-2.6.38-next20110325-5-686-iniza --]
[-- Type: application/octet-stream, Size: 128330 bytes --]
#
# Automatically generated make config: don't edit
# Linux/x86 2.6.38 Kernel Configuration
# Fri Mar 25 22:28:35 2011
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf32-i386"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_GPIO=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-ecx -fcall-saved-edx"
CONFIG_KTIME_SCALAR=y
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
# CONFIG_FHANDLE is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_HAVE_GENERIC_HARDIRQS=y
#
# IRQ subsystem
#
CONFIG_GENERIC_HARDIRQS=y
CONFIG_HAVE_SPARSE_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y
# CONFIG_SPARSE_IRQ is not set
#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
CONFIG_RCU_TRACE=y
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
CONFIG_TREE_RCU_TRACE=y
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_DEBUG=y
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
# CONFIG_RESOURCE_COUNTERS is not set
# CONFIG_CGROUP_PERF is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_BLK_CGROUP=y
CONFIG_DEBUG_BLK_CGROUP=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EXPERT is not set
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y
#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_PERF_COUNTERS is not set
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
# CONFIG_OPROFILE_EVENT_MULTIPLEX is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
# CONFIG_JUMP_LABEL is not set
CONFIG_OPTPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_LBDAF=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_INTEGRITY=y
# CONFIG_BLK_DEV_THROTTLING is not set
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
# CONFIG_INLINE_SPIN_TRYLOCK is not set
# CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK is not set
# CONFIG_INLINE_SPIN_LOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK_IRQ is not set
# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
CONFIG_INLINE_SPIN_UNLOCK=y
# CONFIG_INLINE_SPIN_UNLOCK_BH is not set
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_READ_TRYLOCK is not set
# CONFIG_INLINE_READ_LOCK is not set
# CONFIG_INLINE_READ_LOCK_BH is not set
# CONFIG_INLINE_READ_LOCK_IRQ is not set
# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
CONFIG_INLINE_READ_UNLOCK=y
# CONFIG_INLINE_READ_UNLOCK_BH is not set
CONFIG_INLINE_READ_UNLOCK_IRQ=y
# CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_WRITE_TRYLOCK is not set
# CONFIG_INLINE_WRITE_LOCK is not set
# CONFIG_INLINE_WRITE_LOCK_BH is not set
# CONFIG_INLINE_WRITE_LOCK_IRQ is not set
# CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
CONFIG_INLINE_WRITE_UNLOCK=y
# CONFIG_INLINE_WRITE_UNLOCK_BH is not set
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
# CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_FREEZER=y
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_BIGSMP=y
# CONFIG_X86_EXTENDED_PLATFORM is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_X86_32_IRIS is not set
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_PARAVIRT_GUEST=y
# CONFIG_XEN_PRIVILEGED_GUEST is not set
CONFIG_KVM_CLOCK=y
CONFIG_KVM_GUEST=y
CONFIG_LGUEST_GUEST=y
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_SPINLOCKS is not set
CONFIG_PARAVIRT_CLOCK=y
# CONFIG_PARAVIRT_DEBUG is not set
CONFIG_NO_BOOTMEM=y
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
CONFIG_M686=y
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_X86_GENERIC=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_CMPXCHG_LOCAL=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
# CONFIG_X86_PPRO_FENCE is not set
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=5
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_CYRIX_32=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_CPU_SUP_TRANSMETA_32=y
CONFIG_CPU_SUP_UMC_32=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
CONFIG_IOMMU_API=y
CONFIG_NR_CPUS=32
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
# CONFIG_IRQ_TIME_ACCOUNTING is not set
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
# CONFIG_X86_ANCIENT_MCE is not set
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_MCE_INJECT=m
CONFIG_X86_THERMAL_VECTOR=y
CONFIG_VM86=y
CONFIG_TOSHIBA=m
CONFIG_I8K=m
CONFIG_X86_REBOOTFIXUPS=y
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
# CONFIG_ARCH_PHYS_ADDR_T_64BIT is not set
# CONFIG_ARCH_DMA_ADDR_T_64BIT is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
# CONFIG_PHYS_ADDR_T_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
# CONFIG_TRANSPARENT_HUGEPAGE is not set
CONFIG_CLEANCACHE=y
CONFIG_HIGHPTE=y
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW=64
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_EFI=y
CONFIG_SECCOMP=y
CONFIG_CC_STACKPROTECTOR=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
# CONFIG_CRASH_DUMP is not set
# CONFIG_KEXEC_JUMP is not set
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_X86_NEED_RELOCS=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
#
# Power management and ACPI options
#
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
# CONFIG_PM_VERBOSE is not set
CONFIG_PM_ADVANCED_DEBUG=y
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_CAN_PM_TRACE=y
# CONFIG_PM_TRACE_RTC is not set
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_POWER_METER=m
# CONFIG_ACPI_EC_DEBUGFS is not set
# CONFIG_ACPI_PROC_EVENT is not set
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=m
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=m
# CONFIG_ACPI_IPMI is not set
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=m
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_PCI_SLOT=m
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=m
CONFIG_ACPI_SBS=m
CONFIG_ACPI_HED=m
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=m
# CONFIG_ACPI_APEI_PCIEAER is not set
# CONFIG_ACPI_APEI_EINJ is not set
# CONFIG_ACPI_APEI_ERST_DEBUG is not set
CONFIG_SFI=y
CONFIG_X86_APM_BOOT=y
CONFIG_APM=m
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
# CONFIG_APM_DO_ENABLE is not set
# CONFIG_APM_CPU_IDLE is not set
# CONFIG_APM_DISPLAY_BLANK is not set
# CONFIG_APM_ALLOW_INTS is not set
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=m
# CONFIG_CPU_FREQ_STAT_DETAILS is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m
#
# CPUFreq processor drivers
#
CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K6=m
CONFIG_X86_POWERNOW_K7=m
CONFIG_X86_POWERNOW_K7_ACPI=y
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_GX_SUSPMOD=m
CONFIG_X86_SPEEDSTEP_CENTRINO=m
CONFIG_X86_SPEEDSTEP_CENTRINO_TABLE=y
CONFIG_X86_SPEEDSTEP_ICH=m
CONFIG_X86_SPEEDSTEP_SMI=m
CONFIG_X86_P4_CLOCKMOD=m
CONFIG_X86_CPUFREQ_NFORCE2=m
CONFIG_X86_LONGRUN=m
CONFIG_X86_LONGHAUL=m
# CONFIG_X86_E_POWERSAVER is not set
#
# shared options
#
CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_X86_SPEEDSTEP_RELAXED_CAP_CHECK=y
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_INTEL_IDLE is not set
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
CONFIG_DMAR=y
# CONFIG_DMAR_DEFAULT_ON is not set
CONFIG_DMAR_FLOPPY_WA=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=m
CONFIG_PCIEAER=y
# CONFIG_PCIE_ECRC is not set
CONFIG_PCIEAER_INJECT=m
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_PCIE_PME=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_DEBUG is not set
CONFIG_PCI_STUB=m
CONFIG_HT_IRQ=y
CONFIG_PCI_IOV=y
CONFIG_PCI_IOAPIC=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
CONFIG_SCx200=m
CONFIG_SCx200HR_TIMER=m
# CONFIG_OLPC is not set
CONFIG_AMD_NB=y
CONFIG_PCCARD=m
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y
#
# PC-card bridges
#
CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_I82365=m
CONFIG_TCIC=m
CONFIG_PCMCIA_PROBE=y
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_COMPAQ=m
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not set
CONFIG_HOTPLUG_PCI_IBM=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_SHPC=m
# CONFIG_RAPIDIO is not set
# CONFIG_VBUS_PROXY is not set
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_HAVE_AOUT=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m
CONFIG_HAVE_ATOMIC_IOMAP=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
# CONFIG_IP_FIB_TRIE_STATS is not set
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_CLASSID=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=m
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
CONFIG_IPV6_MIP6=y
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_SIT_6RD=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_MULTIPLE_TABLES=y
CONFIG_IPV6_SUBTREES=y
CONFIG_IPV6_MROUTE=y
CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y
CONFIG_IPV6_PIMSM_V2=y
# CONFIG_NETLABEL is not set
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CONNTRACK_EVENTS=y
# CONFIG_NF_CONNTRACK_TIMESTAMP is not set
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CT_PROTO_UDPLITE=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
# CONFIG_NF_CONNTRACK_SNMP is not set
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_TPROXY=m
CONFIG_NETFILTER_XTABLES=m
#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
#
# Xtables targets
#
# CONFIG_NETFILTER_XT_TARGET_AUDIT is not set
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
#
# Xtables matches
#
# CONFIG_NETFILTER_XT_MATCH_ADDRTYPE is not set
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
# CONFIG_NETFILTER_XT_MATCH_DEVGROUP is not set
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
# CONFIG_IP_SET is not set
CONFIG_IP_VS=m
CONFIG_IP_VS_IPV6=y
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12
#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_PROTO_SCTP=y
#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m
#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
#
# IPv6: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
#
# DECnet: Netfilter Configuration
#
CONFIG_DECNET_NF_GRABULATOR=m
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m
#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_TFRC_LIB=y
#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
CONFIG_NET_DCCPPROBE=m
CONFIG_IP_SCTP=m
CONFIG_NET_SCTPPROBE=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
CONFIG_RDS=m
CONFIG_RDS_RDMA=m
CONFIG_RDS_TCP=m
# CONFIG_RDS_DEBUG is not set
CONFIG_TIPC=m
CONFIG_TIPC_ADVANCED=y
CONFIG_TIPC_PORTS=8191
CONFIG_TIPC_LOG=0
# CONFIG_TIPC_DEBUG is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
CONFIG_ATM_MPOA=m
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_L2TP=m
CONFIG_L2TP_DEBUGFS=m
CONFIG_L2TP_V3=y
CONFIG_L2TP_IP=m
CONFIG_L2TP_ETH=m
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y
CONFIG_DECNET=m
# CONFIG_DECNET_ROUTER is not set
CONFIG_LLC=y
CONFIG_LLC2=m
CONFIG_IPX=m
# CONFIG_IPX_INTERN is not set
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=m
CONFIG_LTPC=m
# CONFIG_COPS is not set
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
# CONFIG_X25 is not set
CONFIG_LAPB=m
# CONFIG_ECONET is not set
CONFIG_WAN_ROUTER=m
CONFIG_PHONET=m
CONFIG_IEEE802154=m
CONFIG_NET_SCHED=y
#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
# CONFIG_NET_SCH_MQPRIO is not set
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_INGRESS=m
#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=y
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
CONFIG_DCB=y
CONFIG_DNS_RESOLVER=y
# CONFIG_BATMAN_ADV is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_TCPPROBE is not set
CONFIG_NET_DROP_MONITOR=y
CONFIG_HAMRADIO=y
#
# Packet Radio protocols
#
CONFIG_AX25=m
# CONFIG_AX25_DAMA_SLAVE is not set
CONFIG_NETROM=m
CONFIG_ROSE=m
#
# AX.25 network device drivers
#
CONFIG_MKISS=m
CONFIG_6PACK=m
CONFIG_BPQETHER=m
CONFIG_SCC=m
# CONFIG_SCC_DELAY is not set
# CONFIG_SCC_TRXECHO is not set
CONFIG_BAYCOM_SER_FDX=m
CONFIG_BAYCOM_SER_HDX=m
CONFIG_BAYCOM_PAR=m
CONFIG_BAYCOM_EPP=m
CONFIG_YAM=m
CONFIG_CAN=m
CONFIG_CAN_RAW=m
CONFIG_CAN_BCM=m
#
# CAN Device Drivers
#
CONFIG_CAN_VCAN=m
# CONFIG_CAN_SLCAN is not set
CONFIG_CAN_DEV=m
CONFIG_CAN_CALC_BITTIMING=y
CONFIG_CAN_MCP251X=m
# CONFIG_PCH_CAN is not set
CONFIG_CAN_SJA1000=m
CONFIG_CAN_SJA1000_ISA=m
# CONFIG_CAN_SJA1000_PLATFORM is not set
CONFIG_CAN_EMS_PCI=m
CONFIG_CAN_KVASER_PCI=m
CONFIG_CAN_PLX_PCI=m
CONFIG_CAN_TSCAN1=m
# CONFIG_CAN_C_CAN is not set
#
# CAN USB interfaces
#
CONFIG_CAN_EMS_USB=m
CONFIG_CAN_ESD_USB2=m
# CONFIG_CAN_SOFTING is not set
# CONFIG_CAN_DEBUG_DEVICES is not set
CONFIG_IRDA=m
#
# IrDA protocols
#
CONFIG_IRLAN=m
CONFIG_IRNET=m
CONFIG_IRCOMM=m
# CONFIG_IRDA_ULTRA is not set
#
# IrDA options
#
CONFIG_IRDA_CACHE_LAST_LSAP=y
CONFIG_IRDA_FAST_RR=y
# CONFIG_IRDA_DEBUG is not set
#
# Infrared-port device drivers
#
#
# SIR device drivers
#
CONFIG_IRTTY_SIR=m
#
# Dongle support
#
CONFIG_DONGLE=y
CONFIG_ESI_DONGLE=m
CONFIG_ACTISYS_DONGLE=m
CONFIG_TEKRAM_DONGLE=m
CONFIG_TOIM3232_DONGLE=m
CONFIG_LITELINK_DONGLE=m
CONFIG_MA600_DONGLE=m
CONFIG_GIRBIL_DONGLE=m
CONFIG_MCP2120_DONGLE=m
CONFIG_OLD_BELKIN_DONGLE=m
CONFIG_ACT200L_DONGLE=m
CONFIG_KINGSUN_DONGLE=m
CONFIG_KSDAZZLE_DONGLE=m
CONFIG_KS959_DONGLE=m
#
# FIR device drivers
#
CONFIG_USB_IRDA=m
CONFIG_SIGMATEL_FIR=m
CONFIG_NSC_FIR=m
CONFIG_WINBOND_FIR=m
CONFIG_TOSHIBA_FIR=m
CONFIG_SMC_IRCC_FIR=m
CONFIG_ALI_FIR=m
CONFIG_VLSI_FIR=m
CONFIG_VIA_FIR=m
CONFIG_MCS_FIR=m
CONFIG_BT=m
CONFIG_BT_L2CAP=y
CONFIG_BT_SCO=y
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_CMTP=m
CONFIG_BT_HIDP=m
#
# Bluetooth device drivers
#
CONFIG_BT_HCIBTUSB=m
CONFIG_BT_HCIBTSDIO=m
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_ATH3K=y
CONFIG_BT_HCIUART_LL=y
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIDTL1=m
CONFIG_BT_HCIBT3C=m
CONFIG_BT_HCIBLUECARD=m
CONFIG_BT_HCIBTUART=m
CONFIG_BT_HCIVHCI=m
CONFIG_BT_MRVL=m
CONFIG_BT_MRVL_SDIO=m
CONFIG_BT_ATH3K=m
CONFIG_AF_RXRPC=m
# CONFIG_AF_RXRPC_DEBUG is not set
CONFIG_RXKAD=m
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
CONFIG_WIRELESS_EXT=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_WEXT_SPY=y
CONFIG_WEXT_PRIV=y
CONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
# CONFIG_CFG80211_REG_DEBUG is not set
CONFIG_CFG80211_DEFAULT_PS=y
# CONFIG_CFG80211_DEBUGFS is not set
# CONFIG_CFG80211_INTERNAL_REGDB is not set
CONFIG_CFG80211_WEXT=y
# CONFIG_WIRELESS_EXT_SYSFS is not set
CONFIG_LIB80211=m
CONFIG_LIB80211_CRYPT_WEP=m
CONFIG_LIB80211_CRYPT_CCMP=m
CONFIG_LIB80211_CRYPT_TKIP=m
# CONFIG_LIB80211_DEBUG is not set
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_MINSTREL_HT=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_LEDS=y
# CONFIG_MAC80211_DEBUGFS is not set
# CONFIG_MAC80211_DEBUG_MENU is not set
CONFIG_WIMAX=m
CONFIG_WIMAX_DEBUG_LEVEL=8
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
CONFIG_NET_9P=m
CONFIG_NET_9P_VIRTIO=m
CONFIG_NET_9P_RDMA=m
# CONFIG_NET_9P_DEBUG is not set
# CONFIG_CAIF is not set
CONFIG_CEPH_LIB=m
# CONFIG_CEPH_LIB_PRETTYDEBUG is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
# CONFIG_DEVTMPFS_MOUNT is not set
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_FIRMWARE_IN_KERNEL is not set
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_ARCH_NO_SYSDEV_OPS=y
CONFIG_CONNECTOR=m
CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not set
# CONFIG_MTD_TESTS is not set
CONFIG_MTD_PARTITIONS=y
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1
# CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set
# CONFIG_MTD_REDBOOT_PARTS_READONLY is not set
CONFIG_MTD_AR7_PARTS=m
#
# User Modules And Translation Layers
#
CONFIG_MTD_CHAR=m
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
CONFIG_MTD_BLOCK_RO=m
CONFIG_FTL=m
CONFIG_NFTL=m
CONFIG_NFTL_RW=y
CONFIG_INFTL=m
CONFIG_RFD_FTL=m
CONFIG_SSFDC=m
# CONFIG_SM_FTL is not set
CONFIG_MTD_OOPS=m
#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=m
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
CONFIG_MTD_CFI_INTELEXT=m
CONFIG_MTD_CFI_AMDSTD=m
CONFIG_MTD_CFI_STAA=m
CONFIG_MTD_CFI_UTIL=m
CONFIG_MTD_RAM=m
CONFIG_MTD_ROM=m
CONFIG_MTD_ABSENT=m
#
# Mapping drivers for chip access
#
CONFIG_MTD_COMPLEX_MAPPINGS=y
CONFIG_MTD_PHYSMAP=m
# CONFIG_MTD_PHYSMAP_COMPAT is not set
CONFIG_MTD_SC520CDP=m
CONFIG_MTD_NETSC520=m
CONFIG_MTD_TS5500=m
CONFIG_MTD_SBC_GXX=m
CONFIG_MTD_SCx200_DOCFLASH=m
# CONFIG_MTD_AMD76XROM is not set
# CONFIG_MTD_ICHXROM is not set
# CONFIG_MTD_ESB2ROM is not set
# CONFIG_MTD_CK804XROM is not set
# CONFIG_MTD_SCB2_FLASH is not set
CONFIG_MTD_NETtel=m
# CONFIG_MTD_L440GX is not set
CONFIG_MTD_PCI=m
CONFIG_MTD_PCMCIA=m
# CONFIG_MTD_PCMCIA_ANONYMOUS is not set
# CONFIG_MTD_GPIO_ADDR is not set
CONFIG_MTD_INTEL_VR_NOR=m
CONFIG_MTD_PLATRAM=m
# CONFIG_MTD_LATCH_ADDR is not set
#
# Self-contained MTD device drivers
#
CONFIG_MTD_PMC551=m
# CONFIG_MTD_PMC551_BUGFIX is not set
# CONFIG_MTD_PMC551_DEBUG is not set
CONFIG_MTD_DATAFLASH=m
# CONFIG_MTD_DATAFLASH_WRITE_VERIFY is not set
# CONFIG_MTD_DATAFLASH_OTP is not set
CONFIG_MTD_M25P80=m
CONFIG_M25PXX_USE_FAST_READ=y
CONFIG_MTD_SST25L=m
CONFIG_MTD_SLRAM=m
CONFIG_MTD_PHRAM=m
CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=m
#
# Disk-On-Chip Device Drivers
#
CONFIG_MTD_DOC2000=m
CONFIG_MTD_DOC2001=m
CONFIG_MTD_DOC2001PLUS=m
CONFIG_MTD_DOCPROBE=m
CONFIG_MTD_DOCECC=m
# CONFIG_MTD_DOCPROBE_ADVANCED is not set
CONFIG_MTD_DOCPROBE_ADDRESS=0
CONFIG_MTD_NAND_ECC=m
# CONFIG_MTD_NAND_ECC_SMC is not set
CONFIG_MTD_NAND=m
# CONFIG_MTD_NAND_VERIFY_WRITE is not set
# CONFIG_MTD_NAND_ECC_BCH is not set
CONFIG_MTD_SM_COMMON=m
# CONFIG_MTD_NAND_MUSEUM_IDS is not set
# CONFIG_MTD_NAND_DENALI is not set
CONFIG_MTD_NAND_IDS=m
CONFIG_MTD_NAND_RICOH=m
CONFIG_MTD_NAND_DISKONCHIP=m
# CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADVANCED is not set
CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADDRESS=0
# CONFIG_MTD_NAND_DISKONCHIP_BBTWRITE is not set
CONFIG_MTD_NAND_CAFE=m
CONFIG_MTD_NAND_CS553X=m
CONFIG_MTD_NAND_NANDSIM=m
CONFIG_MTD_NAND_PLATFORM=m
CONFIG_MTD_ALAUDA=m
CONFIG_MTD_ONENAND=m
CONFIG_MTD_ONENAND_VERIFY_WRITE=y
CONFIG_MTD_ONENAND_GENERIC=m
# CONFIG_MTD_ONENAND_OTP is not set
CONFIG_MTD_ONENAND_2X_PROGRAM=y
CONFIG_MTD_ONENAND_SIM=m
#
# LPDDR flash memory drivers
#
CONFIG_MTD_LPDDR=m
CONFIG_MTD_QINFO_PROBE=m
CONFIG_MTD_UBI=m
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_RESERVE=1
# CONFIG_MTD_UBI_GLUEBI is not set
# CONFIG_MTD_UBI_DEBUG is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
CONFIG_PARPORT_PC_PCMCIA=m
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_AX88796=m
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set
#
# Protocols
#
CONFIG_ISAPNP=y
CONFIG_PNPBIOS=y
CONFIG_PNPBIOS_PROC_FS=y
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_XD=m
CONFIG_PARIDE=m
#
# Parallel IDE high-level drivers
#
CONFIG_PARIDE_PD=m
CONFIG_PARIDE_PCD=m
CONFIG_PARIDE_PF=m
CONFIG_PARIDE_PT=m
CONFIG_PARIDE_PG=m
#
# Parallel IDE protocol modules
#
CONFIG_PARIDE_ATEN=m
CONFIG_PARIDE_BPCK=m
CONFIG_PARIDE_BPCK6=m
CONFIG_PARIDE_COMM=m
CONFIG_PARIDE_DSTR=m
CONFIG_PARIDE_FIT2=m
CONFIG_PARIDE_FIT3=m
CONFIG_PARIDE_EPAT=m
# CONFIG_PARIDE_EPATC8 is not set
CONFIG_PARIDE_EPIA=m
CONFIG_PARIDE_FRIQ=m
CONFIG_PARIDE_FRPW=m
CONFIG_PARIDE_KBIC=m
CONFIG_PARIDE_KTTI=m
CONFIG_PARIDE_ON20=m
CONFIG_PARIDE_ON26=m
CONFIG_BLK_CPQ_DA=m
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_UMEM=m
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
CONFIG_BLK_DEV_DRBD=m
# CONFIG_DRBD_FAULT_INJECTION is not set
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_OSD=m
CONFIG_BLK_DEV_SX8=m
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=8192
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=m
CONFIG_VIRTIO_BLK=m
# CONFIG_BLK_DEV_HD is not set
CONFIG_BLK_DEV_RBD=m
CONFIG_SENSORS_LIS3LV02D=m
CONFIG_MISC_DEVICES=y
CONFIG_AD525X_DPOT=m
CONFIG_AD525X_DPOT_I2C=m
CONFIG_AD525X_DPOT_SPI=m
CONFIG_IBM_ASM=m
CONFIG_PHANTOM=m
CONFIG_SGI_IOC4=m
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
CONFIG_ICS932S401=m
CONFIG_ENCLOSURE_SERVICES=m
CONFIG_CS5535_MFGPT=m
CONFIG_CS5535_MFGPT_DEFAULT_IRQ=7
CONFIG_CS5535_CLOCK_EVENT_SRC=m
CONFIG_HP_ILO=m
CONFIG_APDS9802ALS=m
CONFIG_ISL29003=m
CONFIG_ISL29020=m
CONFIG_SENSORS_TSL2550=m
CONFIG_SENSORS_BH1780=m
CONFIG_SENSORS_BH1770=m
CONFIG_SENSORS_APDS990X=m
CONFIG_HMC6352=m
CONFIG_DS1682=m
CONFIG_TI_DAC7512=m
CONFIG_VMWARE_BALLOON=m
CONFIG_BMP085=m
# CONFIG_PCH_PHUB is not set
CONFIG_C2PORT=m
CONFIG_C2PORT_DURAMAR_2150=m
#
# EEPROM support
#
CONFIG_EEPROM_AT24=m
CONFIG_EEPROM_AT25=m
CONFIG_EEPROM_LEGACY=m
CONFIG_EEPROM_MAX6875=m
CONFIG_EEPROM_93CX6=m
CONFIG_CB710_CORE=m
# CONFIG_CB710_DEBUG is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y
CONFIG_IWMC3200TOP=m
# CONFIG_IWMC3200TOP_DEBUG is not set
# CONFIG_IWMC3200TOP_DEBUGFS is not set
#
# Texas Instruments shared transport line discipline
#
# CONFIG_TI_ST is not set
CONFIG_SENSORS_LIS3_I2C=m
CONFIG_HAVE_IDE=y
CONFIG_IDE=m
#
# Please see Documentation/ide/ide.txt for help/info on IDE drives
#
CONFIG_IDE_XFER_MODE=y
CONFIG_IDE_ATAPI=y
# CONFIG_BLK_DEV_IDE_SATA is not set
CONFIG_IDE_GD=m
CONFIG_IDE_GD_ATA=y
CONFIG_IDE_GD_ATAPI=y
# CONFIG_BLK_DEV_IDECS is not set
CONFIG_BLK_DEV_DELKIN=m
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
CONFIG_BLK_DEV_IDETAPE=m
# CONFIG_BLK_DEV_IDEACPI is not set
# CONFIG_IDE_TASK_IOCTL is not set
CONFIG_IDE_PROC_FS=y
#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=m
# CONFIG_BLK_DEV_PLATFORM is not set
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPNP=m
CONFIG_BLK_DEV_IDEDMA_SFF=y
#
# PCI IDE chipsets support
#
CONFIG_BLK_DEV_IDEPCI=y
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_GENERIC is not set
CONFIG_BLK_DEV_OPTI621=m
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
CONFIG_BLK_DEV_CS5535=m
# CONFIG_BLK_DEV_CS5536 is not set
CONFIG_BLK_DEV_HPT366=m
# CONFIG_BLK_DEV_JMICRON is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
CONFIG_BLK_DEV_IT8172=m
CONFIG_BLK_DEV_IT8213=m
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
CONFIG_BLK_DEV_TRM290=m
# CONFIG_BLK_DEV_VIA82CXXX is not set
CONFIG_BLK_DEV_TC86C001=m
#
# Other IDE chipsets support
#
#
# Note: most of these also require special kernel boot parameters
#
# CONFIG_BLK_DEV_4DRIVES is not set
# CONFIG_BLK_DEV_ALI14XX is not set
# CONFIG_BLK_DEV_DTC2278 is not set
# CONFIG_BLK_DEV_HT6560B is not set
# CONFIG_BLK_DEV_QD65XX is not set
# CONFIG_BLK_DEV_UMC8672 is not set
CONFIG_BLK_DEV_IDEDMA=y
#
# SCSI device support
#
CONFIG_SCSI_MOD=m
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=m
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=m
CONFIG_SCSI_NETLINK=y
# CONFIG_SCSI_PROC_FS is not set
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_ENCLOSURE=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_FC_TGT_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_SRP_ATTRS=m
CONFIG_SCSI_SRP_TGT_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_CXGB4_ISCSI=m
CONFIG_SCSI_BNX2_ISCSI=m
# CONFIG_SCSI_BNX2X_FCOE is not set
CONFIG_BE2ISCSI=m
CONFIG_BLK_DEV_3W_XXXX_RAID=m
CONFIG_SCSI_HPSA=m
CONFIG_SCSI_3W_9XXX=m
CONFIG_SCSI_3W_SAS=m
CONFIG_SCSI_7000FASST=m
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AHA152X=m
CONFIG_SCSI_AHA1542=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=8
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=15000
CONFIG_AIC79XX_DEBUG_ENABLE=y
CONFIG_AIC79XX_DEBUG_MASK=0
CONFIG_AIC79XX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC94XX=m
# CONFIG_AIC94XX_DEBUG is not set
CONFIG_SCSI_MVSAS=m
# CONFIG_SCSI_MVSAS_DEBUG is not set
CONFIG_SCSI_DPT_I2O=m
CONFIG_SCSI_ADVANSYS=m
CONFIG_SCSI_IN2000=m
CONFIG_SCSI_ARCMSR=m
# CONFIG_SCSI_ARCMSR_AER is not set
CONFIG_MEGARAID_NEWGEN=y
CONFIG_MEGARAID_MM=m
CONFIG_MEGARAID_MAILBOX=m
CONFIG_MEGARAID_LEGACY=m
CONFIG_MEGARAID_SAS=m
CONFIG_SCSI_MPT2SAS=m
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
# CONFIG_SCSI_MPT2SAS_LOGGING is not set
CONFIG_SCSI_HPTIOP=m
CONFIG_SCSI_BUSLOGIC=m
# CONFIG_SCSI_FLASHPOINT is not set
CONFIG_VMWARE_PVSCSI=m
CONFIG_LIBFC=m
CONFIG_LIBFCOE=m
CONFIG_FCOE=m
CONFIG_FCOE_FNIC=m
CONFIG_SCSI_DMX3191D=m
CONFIG_SCSI_DTC3280=m
CONFIG_SCSI_EATA=m
CONFIG_SCSI_EATA_TAGGED_QUEUE=y
CONFIG_SCSI_EATA_LINKED_COMMANDS=y
CONFIG_SCSI_EATA_MAX_TAGS=16
CONFIG_SCSI_FUTURE_DOMAIN=m
CONFIG_SCSI_GDTH=m
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
CONFIG_SCSI_IPS=m
CONFIG_SCSI_INITIO=m
# CONFIG_SCSI_INIA100 is not set
CONFIG_SCSI_PPA=m
CONFIG_SCSI_IMM=m
# CONFIG_SCSI_IZIP_EPP16 is not set
# CONFIG_SCSI_IZIP_SLOW_CTR is not set
CONFIG_SCSI_NCR53C406A=m
CONFIG_SCSI_STEX=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
CONFIG_SCSI_IPR=m
# CONFIG_SCSI_IPR_TRACE is not set
# CONFIG_SCSI_IPR_DUMP is not set
CONFIG_SCSI_PAS16=m
CONFIG_SCSI_QLOGIC_FAS=m
CONFIG_SCSI_QLOGIC_1280=m
CONFIG_SCSI_QLA_FC=m
CONFIG_SCSI_QLA_ISCSI=m
CONFIG_SCSI_LPFC=m
# CONFIG_SCSI_LPFC_DEBUG_FS is not set
CONFIG_SCSI_SYM53C416=m
CONFIG_SCSI_DC395x=m
CONFIG_SCSI_DC390T=m
CONFIG_SCSI_T128=m
CONFIG_SCSI_U14_34F=m
CONFIG_SCSI_U14_34F_TAGGED_QUEUE=y
CONFIG_SCSI_U14_34F_LINKED_COMMANDS=y
CONFIG_SCSI_U14_34F_MAX_TAGS=8
CONFIG_SCSI_ULTRASTOR=m
CONFIG_SCSI_NSP32=m
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_PMCRAID=m
CONFIG_SCSI_PM8001=m
CONFIG_SCSI_SRP=m
CONFIG_SCSI_BFA_FC=m
CONFIG_SCSI_LOWLEVEL_PCMCIA=y
CONFIG_PCMCIA_AHA152X=m
CONFIG_PCMCIA_FDOMAIN=m
CONFIG_PCMCIA_NINJA_SCSI=m
CONFIG_PCMCIA_QLOGIC=m
CONFIG_PCMCIA_SYM53C500=m
CONFIG_SCSI_DH=m
CONFIG_SCSI_DH_RDAC=m
CONFIG_SCSI_DH_HP_SW=m
CONFIG_SCSI_DH_EMC=m
CONFIG_SCSI_DH_ALUA=m
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_SCSI_OSD_DPRINT_SENSE=1
# CONFIG_SCSI_OSD_DEBUG is not set
CONFIG_ATA=m
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y
#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=m
# CONFIG_SATA_AHCI_PLATFORM is not set
CONFIG_SATA_INIC162X=m
# CONFIG_SATA_ACARD_AHCI is not set
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y
#
# SFF controllers with custom DMA interface
#
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_SX4=m
CONFIG_ATA_BMDMA=y
#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=m
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_SVW=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m
#
# PATA SFF controllers with BMDMA
#
CONFIG_PATA_ALI=m
CONFIG_PATA_AMD=m
# CONFIG_PATA_ARASAN_CF is not set
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
CONFIG_PATA_ATP867X=m
CONFIG_PATA_CMD64X=m
CONFIG_PATA_CS5520=m
CONFIG_PATA_CS5530=m
# CONFIG_PATA_CS5535 is not set
CONFIG_PATA_CS5536=m
# CONFIG_PATA_CYPRESS is not set
CONFIG_PATA_EFAR=m
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_IT821X=m
CONFIG_PATA_JMICRON=m
CONFIG_PATA_MARVELL=m
CONFIG_PATA_NETCELL=m
# CONFIG_PATA_NINJA32 is not set
CONFIG_PATA_NS87415=m
CONFIG_PATA_OLDPIIX=m
# CONFIG_PATA_OPTIDMA is not set
CONFIG_PATA_PDC2027X=m
CONFIG_PATA_PDC_OLD=m
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RDC=m
CONFIG_PATA_SC1200=m
CONFIG_PATA_SCH=m
CONFIG_PATA_SERVERWORKS=m
CONFIG_PATA_SIL680=m
CONFIG_PATA_SIS=m
CONFIG_PATA_TOSHIBA=m
CONFIG_PATA_TRIFLEX=m
CONFIG_PATA_VIA=m
# CONFIG_PATA_WINBOND is not set
#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_ISAPNP is not set
CONFIG_PATA_MPIIX=m
CONFIG_PATA_NS87410=m
# CONFIG_PATA_OPTI is not set
CONFIG_PATA_PCMCIA=m
# CONFIG_PATA_QDI is not set
CONFIG_PATA_RZ1000=m
# CONFIG_PATA_WINBOND_VLB is not set
#
# Generic fallback / legacy drivers
#
# CONFIG_PATA_ACPI is not set
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
# CONFIG_MULTICORE_RAID456 is not set
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
# CONFIG_DM_RAID is not set
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
CONFIG_DM_DELAY=m
CONFIG_DM_UEVENT=y
# CONFIG_DM_FLAKEY is not set
# CONFIG_TARGET_CORE is not set
CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
CONFIG_FUSION_FC=m
CONFIG_FUSION_SAS=m
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LAN=m
# CONFIG_FUSION_LOGGING is not set
#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_OHCI_DEBUG=y
CONFIG_FIREWIRE_SBP2=m
CONFIG_FIREWIRE_NET=m
CONFIG_FIREWIRE_NOSY=m
CONFIG_I2O=m
CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_CONFIG=m
CONFIG_I2O_CONFIG_OLD_IOCTL=y
CONFIG_I2O_BUS=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_IFB=m
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
CONFIG_EQUALIZER=m
CONFIG_TUN=m
CONFIG_VETH=m
CONFIG_NET_SB1000=m
CONFIG_ARCNET=m
CONFIG_ARCNET_1201=m
CONFIG_ARCNET_1051=m
CONFIG_ARCNET_RAW=m
CONFIG_ARCNET_CAP=m
CONFIG_ARCNET_COM90xx=m
CONFIG_ARCNET_COM90xxIO=m
CONFIG_ARCNET_RIM_I=m
CONFIG_ARCNET_COM20020=m
CONFIG_ARCNET_COM20020_ISA=m
CONFIG_ARCNET_COM20020_PCI=m
CONFIG_MII=m
CONFIG_PHYLIB=m
#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=m
CONFIG_DAVICOM_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_LXT_PHY=m
CONFIG_CICADA_PHY=m
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=m
CONFIG_BROADCOM_PHY=m
# CONFIG_BCM63XX_PHY is not set
CONFIG_ICPLUS_PHY=m
CONFIG_REALTEK_PHY=m
CONFIG_NATIONAL_PHY=m
CONFIG_STE10XP=m
CONFIG_LSI_ET1011C_PHY=m
CONFIG_MICREL_PHY=m
CONFIG_MDIO_BITBANG=m
# CONFIG_MDIO_GPIO is not set
CONFIG_NET_ETHERNET=y
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_CASSINI=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_EL1=m
CONFIG_EL2=m
CONFIG_ELPLUS=m
CONFIG_EL16=m
CONFIG_EL3=m
CONFIG_3C515=m
CONFIG_VORTEX=m
CONFIG_TYPHOON=m
CONFIG_LANCE=m
CONFIG_NET_VENDOR_SMC=y
CONFIG_WD80x3=m
CONFIG_ULTRA=m
CONFIG_SMC9194=m
CONFIG_ENC28J60=m
# CONFIG_ENC28J60_WRITEVERIFY is not set
CONFIG_ETHOC=m
CONFIG_NET_VENDOR_RACAL=y
CONFIG_NI52=m
CONFIG_NI65=m
CONFIG_DNET=m
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
# CONFIG_TULIP_MMIO is not set
CONFIG_TULIP_NAPI=y
CONFIG_TULIP_NAPI_HW_MITIGATION=y
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
CONFIG_AT1700=m
CONFIG_DEPCA=m
CONFIG_HP100=m
CONFIG_NET_ISA=y
CONFIG_E2100=m
CONFIG_EWRK3=m
CONFIG_EEXPRESS=m
CONFIG_EEXPRESS_PRO=m
CONFIG_HPLAN_PLUS=m
CONFIG_HPLAN=m
CONFIG_LP486E=m
CONFIG_ETH16I=m
CONFIG_NE2000=m
CONFIG_ZNET=m
CONFIG_SEEQ8005=m
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
CONFIG_ADAPTEC_STARFIRE=m
CONFIG_AC3200=m
CONFIG_KSZ884X_PCI=m
CONFIG_APRICOT=m
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_FORCEDETH=m
CONFIG_CS89x0=m
CONFIG_E100=m
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=m
CONFIG_8139TOO_PIO=y
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_R6040=m
CONFIG_SIS900=m
CONFIG_EPIC100=m
CONFIG_SMSC9420=m
CONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not set
CONFIG_TLAN=m
CONFIG_KS8842=m
CONFIG_KS8851=m
CONFIG_KS8851_MLL=m
CONFIG_VIA_RHINE=m
# CONFIG_VIA_RHINE_MMIO is not set
CONFIG_SC92031=m
# CONFIG_NET_POCKET is not set
CONFIG_ATL2=m
CONFIG_NETDEV_1000=y
CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not set
CONFIG_DL2K=m
CONFIG_E1000=m
CONFIG_E1000E=m
CONFIG_IP1000=m
CONFIG_IGB=m
CONFIG_IGB_DCA=y
CONFIG_IGBVF=m
CONFIG_NS83820=m
CONFIG_HAMACHI=m
CONFIG_YELLOWFIN=m
CONFIG_R8169=m
CONFIG_SIS190=m
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
CONFIG_VIA_VELOCITY=m
CONFIG_TIGON3=m
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_QLA3XXX=m
CONFIG_ATL1=m
CONFIG_ATL1E=m
CONFIG_ATL1C=m
CONFIG_JME=m
# CONFIG_STMMAC_ETH is not set
# CONFIG_PCH_GBE is not set
CONFIG_NETDEV_10000=y
CONFIG_MDIO=m
CONFIG_CHELSIO_T1=m
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4=m
CONFIG_CHELSIO_T4VF=m
CONFIG_ENIC=m
CONFIG_IXGBE=m
CONFIG_IXGBE_DCA=y
CONFIG_IXGBE_DCB=y
CONFIG_IXGBEVF=m
CONFIG_IXGB=m
CONFIG_S2IO=m
CONFIG_VXGE=m
# CONFIG_VXGE_DEBUG_TRACE_ALL is not set
CONFIG_MYRI10GE=m
CONFIG_MYRI10GE_DCA=y
CONFIG_NETXEN_NIC=m
CONFIG_NIU=m
CONFIG_MLX4_EN=m
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_TEHUTI=m
CONFIG_BNX2X=m
CONFIG_QLCNIC=m
CONFIG_QLGE=m
CONFIG_BNA=m
CONFIG_SFC=m
CONFIG_SFC_MTD=y
CONFIG_BE2NET=m
CONFIG_TR=y
CONFIG_IBMTR=m
CONFIG_IBMOL=m
CONFIG_IBMLS=m
CONFIG_3C359=m
CONFIG_TMS380TR=m
CONFIG_TMSPCI=m
CONFIG_SKISA=m
CONFIG_PROTEON=m
CONFIG_ABYSS=m
CONFIG_SMCTR=m
CONFIG_WLAN=y
CONFIG_PCMCIA_RAYCS=m
CONFIG_LIBERTAS_THINFIRM=m
# CONFIG_LIBERTAS_THINFIRM_DEBUG is not set
CONFIG_LIBERTAS_THINFIRM_USB=m
CONFIG_AIRO=m
CONFIG_ATMEL=m
CONFIG_PCI_ATMEL=m
CONFIG_PCMCIA_ATMEL=m
CONFIG_AT76C50X_USB=m
CONFIG_AIRO_CS=m
CONFIG_PCMCIA_WL3501=m
# CONFIG_PRISM54 is not set
CONFIG_USB_ZD1201=m
CONFIG_USB_NET_RNDIS_WLAN=m
CONFIG_RTL8180=m
CONFIG_RTL8187=m
CONFIG_RTL8187_LEDS=y
CONFIG_ADM8211=m
CONFIG_MAC80211_HWSIM=m
CONFIG_MWL8K=m
CONFIG_ATH_COMMON=m
CONFIG_ATH_DEBUG=y
CONFIG_ATH5K=m
CONFIG_ATH5K_DEBUG=y
CONFIG_ATH5K_TRACER=y
CONFIG_ATH5K_PCI=y
CONFIG_ATH9K_HW=m
CONFIG_ATH9K_COMMON=m
CONFIG_ATH9K=m
# CONFIG_ATH9K_DEBUGFS is not set
CONFIG_ATH9K_RATE_CONTROL=y
CONFIG_ATH9K_HTC=m
# CONFIG_ATH9K_HTC_DEBUGFS is not set
CONFIG_AR9170_USB=m
CONFIG_AR9170_LEDS=y
# CONFIG_CARL9170 is not set
CONFIG_B43=m
CONFIG_B43_PCI_AUTOSELECT=y
CONFIG_B43_PCICORE_AUTOSELECT=y
CONFIG_B43_PCMCIA=y
CONFIG_B43_SDIO=y
CONFIG_B43_PIO=y
# CONFIG_B43_PHY_N is not set
CONFIG_B43_PHY_LP=y
CONFIG_B43_LEDS=y
CONFIG_B43_HWRNG=y
# CONFIG_B43_DEBUG is not set
CONFIG_B43LEGACY=m
CONFIG_B43LEGACY_PCI_AUTOSELECT=y
CONFIG_B43LEGACY_PCICORE_AUTOSELECT=y
CONFIG_B43LEGACY_LEDS=y
CONFIG_B43LEGACY_HWRNG=y
CONFIG_B43LEGACY_DEBUG=y
CONFIG_B43LEGACY_DMA=y
CONFIG_B43LEGACY_PIO=y
CONFIG_B43LEGACY_DMA_AND_PIO_MODE=y
# CONFIG_B43LEGACY_DMA_MODE is not set
# CONFIG_B43LEGACY_PIO_MODE is not set
CONFIG_HOSTAP=m
CONFIG_HOSTAP_FIRMWARE=y
# CONFIG_HOSTAP_FIRMWARE_NVRAM is not set
CONFIG_HOSTAP_PLX=m
CONFIG_HOSTAP_PCI=m
CONFIG_HOSTAP_CS=m
CONFIG_IPW2100=m
CONFIG_IPW2100_MONITOR=y
# CONFIG_IPW2100_DEBUG is not set
CONFIG_IPW2200=m
CONFIG_IPW2200_MONITOR=y
CONFIG_IPW2200_RADIOTAP=y
CONFIG_IPW2200_PROMISCUOUS=y
CONFIG_IPW2200_QOS=y
# CONFIG_IPW2200_DEBUG is not set
CONFIG_LIBIPW=m
# CONFIG_LIBIPW_DEBUG is not set
CONFIG_IWLAGN=m
#
# Debugging Options
#
# CONFIG_IWLWIFI_DEBUG is not set
# CONFIG_IWLWIFI_DEVICE_TRACING is not set
# CONFIG_IWL_P2P is not set
CONFIG_IWLWIFI_LEGACY=m
#
# Debugging Options
#
# CONFIG_IWLWIFI_LEGACY_DEBUG is not set
# CONFIG_IWLWIFI_LEGACY_DEVICE_TRACING is not set
CONFIG_IWL4965=m
CONFIG_IWL3945=m
CONFIG_IWM=m
# CONFIG_IWM_DEBUG is not set
# CONFIG_IWM_TRACING is not set
CONFIG_LIBERTAS=m
CONFIG_LIBERTAS_USB=m
CONFIG_LIBERTAS_CS=m
CONFIG_LIBERTAS_SDIO=m
CONFIG_LIBERTAS_SPI=m
# CONFIG_LIBERTAS_DEBUG is not set
CONFIG_LIBERTAS_MESH=y
CONFIG_HERMES=m
# CONFIG_HERMES_PRISM is not set
CONFIG_HERMES_CACHE_FW_ON_INIT=y
CONFIG_PLX_HERMES=m
CONFIG_TMD_HERMES=m
CONFIG_NORTEL_HERMES=m
CONFIG_PCMCIA_HERMES=m
CONFIG_PCMCIA_SPECTRUM=m
CONFIG_ORINOCO_USB=m
CONFIG_P54_COMMON=m
CONFIG_P54_USB=m
CONFIG_P54_PCI=m
CONFIG_P54_SPI=m
# CONFIG_P54_SPI_DEFAULT_EEPROM is not set
CONFIG_P54_LEDS=y
CONFIG_RT2X00=m
CONFIG_RT2400PCI=m
CONFIG_RT2500PCI=m
CONFIG_RT61PCI=m
CONFIG_RT2800PCI=m
# CONFIG_RT2800PCI_RT33XX is not set
# CONFIG_RT2800PCI_RT35XX is not set
# CONFIG_RT2800PCI_RT53XX is not set
CONFIG_RT2500USB=m
CONFIG_RT73USB=m
CONFIG_RT2800USB=m
# CONFIG_RT2800USB_RT33XX is not set
# CONFIG_RT2800USB_RT35XX is not set
# CONFIG_RT2800USB_UNKNOWN is not set
CONFIG_RT2800_LIB=m
CONFIG_RT2X00_LIB_PCI=m
CONFIG_RT2X00_LIB_USB=m
CONFIG_RT2X00_LIB=m
CONFIG_RT2X00_LIB_HT=y
CONFIG_RT2X00_LIB_FIRMWARE=y
CONFIG_RT2X00_LIB_CRYPTO=y
CONFIG_RT2X00_LIB_LEDS=y
# CONFIG_RT2X00_DEBUG is not set
# CONFIG_RTL8192CE is not set
# CONFIG_RTL8192CU is not set
CONFIG_WL1251=m
CONFIG_WL1251_SPI=m
CONFIG_WL1251_SDIO=m
# CONFIG_WL12XX_MENU is not set
CONFIG_WL12XX_PLATFORM_DATA=y
CONFIG_ZD1211RW=m
# CONFIG_ZD1211RW_DEBUG is not set
#
# WiMAX Wireless Broadband devices
#
CONFIG_WIMAX_I2400M=m
CONFIG_WIMAX_I2400M_USB=m
CONFIG_WIMAX_I2400M_SDIO=m
CONFIG_WIMAX_IWMC3200_SDIO=y
CONFIG_WIMAX_I2400M_DEBUG_LEVEL=8
#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_CDCETHER=m
CONFIG_USB_NET_CDC_EEM=m
CONFIG_USB_NET_CDC_NCM=m
CONFIG_USB_NET_DM9601=m
CONFIG_USB_NET_SMSC75XX=m
CONFIG_USB_NET_SMSC95XX=m
CONFIG_USB_NET_GL620A=m
CONFIG_USB_NET_NET1080=m
CONFIG_USB_NET_PLUSB=m
CONFIG_USB_NET_MCS7830=m
CONFIG_USB_NET_RNDIS_HOST=m
CONFIG_USB_NET_CDC_SUBSET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=m
CONFIG_USB_NET_CX82310_ETH=m
CONFIG_USB_HSO=m
CONFIG_USB_NET_INT51X1=m
CONFIG_USB_CDC_PHONET=m
CONFIG_USB_IPHETH=m
CONFIG_USB_SIERRA_NET=m
CONFIG_NET_PCMCIA=y
CONFIG_PCMCIA_3C589=m
CONFIG_PCMCIA_3C574=m
CONFIG_PCMCIA_FMVJ18X=m
CONFIG_PCMCIA_PCNET=m
CONFIG_PCMCIA_NMCLAN=m
CONFIG_PCMCIA_SMC91C92=m
CONFIG_PCMCIA_XIRC2PS=m
CONFIG_PCMCIA_AXNET=m
CONFIG_ARCNET_COM20020_CS=m
CONFIG_PCMCIA_IBMTR=m
CONFIG_WAN=y
CONFIG_HOSTESS_SV11=m
CONFIG_COSA=m
CONFIG_LANMEDIA=m
CONFIG_SEALEVEL_4021=m
CONFIG_HDLC=m
CONFIG_HDLC_RAW=m
CONFIG_HDLC_RAW_ETH=m
CONFIG_HDLC_CISCO=m
CONFIG_HDLC_FR=m
CONFIG_HDLC_PPP=m
# CONFIG_HDLC_X25 is not set
CONFIG_PCI200SYN=m
CONFIG_WANXL=m
# CONFIG_PC300TOO is not set
CONFIG_N2=m
CONFIG_C101=m
CONFIG_FARSYNC=m
CONFIG_DSCC4=m
CONFIG_DSCC4_PCISYNC=y
CONFIG_DSCC4_PCI_RST=y
CONFIG_DLCI=m
CONFIG_DLCI_MAX=8
CONFIG_SDLA=m
CONFIG_WAN_ROUTER_DRIVERS=m
CONFIG_CYCLADES_SYNC=m
# CONFIG_CYCLOMX_X25 is not set
CONFIG_SBNI=m
# CONFIG_SBNI_MULTILINE is not set
CONFIG_ATM_DRIVERS=y
CONFIG_ATM_DUMMY=m
CONFIG_ATM_TCP=m
CONFIG_ATM_LANAI=m
CONFIG_ATM_ENI=m
# CONFIG_ATM_ENI_DEBUG is not set
# CONFIG_ATM_ENI_TUNE_BURST is not set
CONFIG_ATM_FIRESTREAM=m
CONFIG_ATM_ZATM=m
# CONFIG_ATM_ZATM_DEBUG is not set
CONFIG_ATM_NICSTAR=m
# CONFIG_ATM_NICSTAR_USE_SUNI is not set
# CONFIG_ATM_NICSTAR_USE_IDT77105 is not set
CONFIG_ATM_IDT77252=m
# CONFIG_ATM_IDT77252_DEBUG is not set
# CONFIG_ATM_IDT77252_RCV_ALL is not set
CONFIG_ATM_IDT77252_USE_SUNI=y
CONFIG_ATM_AMBASSADOR=m
# CONFIG_ATM_AMBASSADOR_DEBUG is not set
CONFIG_ATM_HORIZON=m
# CONFIG_ATM_HORIZON_DEBUG is not set
CONFIG_ATM_IA=m
# CONFIG_ATM_IA_DEBUG is not set
CONFIG_ATM_FORE200E=m
# CONFIG_ATM_FORE200E_USE_TASKLET is not set
CONFIG_ATM_FORE200E_TX_RETRY=16
CONFIG_ATM_FORE200E_DEBUG=0
CONFIG_ATM_HE=m
CONFIG_ATM_HE_USE_SUNI=y
CONFIG_ATM_SOLOS=m
CONFIG_IEEE802154_DRIVERS=m
CONFIG_IEEE802154_FAKEHARD=m
#
# CAIF transport drivers
#
CONFIG_FDDI=y
CONFIG_DEFXX=m
# CONFIG_DEFXX_MMIO is not set
CONFIG_SKFP=m
CONFIG_HIPPI=y
CONFIG_ROADRUNNER=m
# CONFIG_ROADRUNNER_LARGE_RINGS is not set
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_PPTP=m
CONFIG_PPPOATM=m
CONFIG_PPPOL2TP=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=m
CONFIG_SLIP_SMART=y
CONFIG_SLIP_MODE_SLIP6=y
CONFIG_NET_FC=y
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_VIRTIO_NET=m
CONFIG_VMXNET3=m
CONFIG_ISDN=y
# CONFIG_ISDN_I4L is not set
CONFIG_ISDN_CAPI=m
CONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=y
CONFIG_CAPI_TRACE=y
CONFIG_ISDN_CAPI_MIDDLEWARE=y
CONFIG_ISDN_CAPI_CAPI20=m
CONFIG_ISDN_CAPI_CAPIFS_BOOL=y
CONFIG_ISDN_CAPI_CAPIFS=m
#
# CAPI hardware drivers
#
CONFIG_CAPI_AVM=y
CONFIG_ISDN_DRV_AVMB1_B1ISA=m
CONFIG_ISDN_DRV_AVMB1_B1PCI=m
CONFIG_ISDN_DRV_AVMB1_B1PCIV4=y
CONFIG_ISDN_DRV_AVMB1_T1ISA=m
CONFIG_ISDN_DRV_AVMB1_B1PCMCIA=m
CONFIG_ISDN_DRV_AVMB1_AVM_CS=m
CONFIG_ISDN_DRV_AVMB1_T1PCI=m
CONFIG_ISDN_DRV_AVMB1_C4=m
CONFIG_CAPI_EICON=y
CONFIG_ISDN_DIVAS=m
CONFIG_ISDN_DIVAS_BRIPCI=y
CONFIG_ISDN_DIVAS_PRIPCI=y
CONFIG_ISDN_DIVAS_DIVACAPI=m
CONFIG_ISDN_DIVAS_USERIDI=m
CONFIG_ISDN_DIVAS_MAINT=m
CONFIG_ISDN_DRV_GIGASET=m
CONFIG_GIGASET_CAPI=y
# CONFIG_GIGASET_DUMMYLL is not set
CONFIG_GIGASET_BASE=m
CONFIG_GIGASET_M105=m
CONFIG_GIGASET_M101=m
# CONFIG_GIGASET_DEBUG is not set
CONFIG_HYSDN=m
CONFIG_HYSDN_CAPI=y
CONFIG_MISDN=m
CONFIG_MISDN_DSP=m
CONFIG_MISDN_L1OIP=m
#
# mISDN hardware drivers
#
CONFIG_MISDN_HFCPCI=m
CONFIG_MISDN_HFCMULTI=m
CONFIG_MISDN_HFCUSB=m
CONFIG_MISDN_AVMFRITZ=m
CONFIG_MISDN_SPEEDFAX=m
CONFIG_MISDN_INFINEON=m
CONFIG_MISDN_W6692=m
# CONFIG_MISDN_NETJET is not set
CONFIG_MISDN_IPAC=m
CONFIG_MISDN_ISAR=m
CONFIG_PHONE=m
CONFIG_PHONE_IXJ=m
CONFIG_PHONE_IXJ_PCMCIA=m
#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=m
CONFIG_INPUT_POLLDEV=m
CONFIG_INPUT_SPARSEKMAP=m
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ADP5588=m
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1070 is not set
CONFIG_KEYBOARD_QT2160=m
CONFIG_KEYBOARD_LKKBD=m
# CONFIG_KEYBOARD_GPIO is not set
# CONFIG_KEYBOARD_GPIO_POLLED is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_MATRIX is not set
CONFIG_KEYBOARD_LM8323=m
CONFIG_KEYBOARD_MAX7359=m
# CONFIG_KEYBOARD_MCS is not set
CONFIG_KEYBOARD_NEWTON=m
CONFIG_KEYBOARD_OPENCORES=m
CONFIG_KEYBOARD_STOWAWAY=m
CONFIG_KEYBOARD_SUNKBD=m
CONFIG_KEYBOARD_XTKBD=m
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_PS2_ELANTECH=y
CONFIG_MOUSE_PS2_SENTELIC=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_APPLETOUCH=m
CONFIG_MOUSE_BCM5974=m
CONFIG_MOUSE_INPORT=m
# CONFIG_MOUSE_ATIXL is not set
CONFIG_MOUSE_LOGIBM=m
CONFIG_MOUSE_PC110PAD=m
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_GPIO is not set
CONFIG_MOUSE_SYNAPTICS_I2C=m
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDJOY=m
CONFIG_JOYSTICK_ZHENHUA=m
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
# CONFIG_JOYSTICK_AS5011 is not set
CONFIG_JOYSTICK_JOYDUMP=m
CONFIG_JOYSTICK_XPAD=m
CONFIG_JOYSTICK_XPAD_FF=y
CONFIG_JOYSTICK_XPAD_LEDS=y
CONFIG_JOYSTICK_WALKERA0701=m
CONFIG_INPUT_TABLET=y
CONFIG_TABLET_USB_ACECAD=m
CONFIG_TABLET_USB_AIPTEK=m
CONFIG_TABLET_USB_GTCO=m
# CONFIG_TABLET_USB_HANWANG is not set
CONFIG_TABLET_USB_KBTAB=m
CONFIG_TABLET_USB_WACOM=m
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_ADS7846=m
CONFIG_TOUCHSCREEN_AD7877=m
CONFIG_TOUCHSCREEN_AD7879=m
CONFIG_TOUCHSCREEN_AD7879_I2C=m
# CONFIG_TOUCHSCREEN_AD7879_SPI is not set
# CONFIG_TOUCHSCREEN_ATMEL_MXT is not set
# CONFIG_TOUCHSCREEN_BU21013 is not set
# CONFIG_TOUCHSCREEN_CY8CTMG110 is not set
CONFIG_TOUCHSCREEN_DYNAPRO=m
CONFIG_TOUCHSCREEN_HAMPSHIRE=m
CONFIG_TOUCHSCREEN_EETI=m
CONFIG_TOUCHSCREEN_FUJITSU=m
CONFIG_TOUCHSCREEN_GUNZE=m
CONFIG_TOUCHSCREEN_ELO=m
CONFIG_TOUCHSCREEN_WACOM_W8001=m
CONFIG_TOUCHSCREEN_MCS5000=m
CONFIG_TOUCHSCREEN_MTOUCH=m
CONFIG_TOUCHSCREEN_INEXIO=m
CONFIG_TOUCHSCREEN_MK712=m
CONFIG_TOUCHSCREEN_HTCPEN=m
CONFIG_TOUCHSCREEN_PENMOUNT=m
CONFIG_TOUCHSCREEN_TOUCHRIGHT=m
CONFIG_TOUCHSCREEN_TOUCHWIN=m
CONFIG_TOUCHSCREEN_WM97XX=m
CONFIG_TOUCHSCREEN_WM9705=y
CONFIG_TOUCHSCREEN_WM9712=y
CONFIG_TOUCHSCREEN_WM9713=y
CONFIG_TOUCHSCREEN_USB_COMPOSITE=m
CONFIG_TOUCHSCREEN_USB_EGALAX=y
CONFIG_TOUCHSCREEN_USB_PANJIT=y
CONFIG_TOUCHSCREEN_USB_3M=y
CONFIG_TOUCHSCREEN_USB_ITM=y
CONFIG_TOUCHSCREEN_USB_ETURBO=y
CONFIG_TOUCHSCREEN_USB_GUNZE=y
CONFIG_TOUCHSCREEN_USB_DMC_TSC10=y
CONFIG_TOUCHSCREEN_USB_IRTOUCH=y
CONFIG_TOUCHSCREEN_USB_IDEALTEK=y
CONFIG_TOUCHSCREEN_USB_GENERAL_TOUCH=y
CONFIG_TOUCHSCREEN_USB_GOTOP=y
CONFIG_TOUCHSCREEN_USB_JASTEC=y
CONFIG_TOUCHSCREEN_USB_E2I=y
CONFIG_TOUCHSCREEN_USB_ZYTRONIC=y
CONFIG_TOUCHSCREEN_USB_ETT_TC45USB=y
CONFIG_TOUCHSCREEN_USB_NEXIO=y
CONFIG_TOUCHSCREEN_TOUCHIT213=m
# CONFIG_TOUCHSCREEN_TSC2005 is not set
CONFIG_TOUCHSCREEN_TSC2007=m
# CONFIG_TOUCHSCREEN_ST1232 is not set
CONFIG_TOUCHSCREEN_TPS6507X=m
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
CONFIG_INPUT_PCSPKR=m
CONFIG_INPUT_APANEL=m
CONFIG_INPUT_WISTRON_BTNS=m
CONFIG_INPUT_ATLAS_BTNS=m
CONFIG_INPUT_ATI_REMOTE=m
CONFIG_INPUT_ATI_REMOTE2=m
CONFIG_INPUT_KEYSPAN_REMOTE=m
CONFIG_INPUT_POWERMATE=m
CONFIG_INPUT_YEALINK=m
CONFIG_INPUT_CM109=m
CONFIG_INPUT_UINPUT=m
CONFIG_INPUT_PCF50633_PMU=m
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_GPIO_ROTARY_ENCODER is not set
# CONFIG_INPUT_ADXL34X is not set
# CONFIG_INPUT_CMA3000 is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=m
CONFIG_SERIO_PARKBD=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_SERIO_ALTERA_PS2=m
# CONFIG_SERIO_PS2MULT is not set
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m
#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_ROCKETPORT=m
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
# CONFIG_MOXA_INTELLIO is not set
CONFIG_MOXA_SMARTIO=m
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_NOZOMI=m
# CONFIG_ISI is not set
CONFIG_N_HDLC=m
CONFIG_N_GSM=m
# CONFIG_DEVKMEM is not set
CONFIG_STALDRV=y
#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_FOURPORT=m
CONFIG_SERIAL_8250_ACCENT=m
CONFIG_SERIAL_8250_BOCA=m
CONFIG_SERIAL_8250_EXAR_ST16C554=m
CONFIG_SERIAL_8250_HUB6=m
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
CONFIG_SERIAL_8250_RSA=y
#
# Non-8250 serial port support
#
CONFIG_SERIAL_MAX3100=m
# CONFIG_SERIAL_MAX3107 is not set
CONFIG_SERIAL_MFD_HSU=m
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_CONSOLE_POLL=y
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_IFX6X60 is not set
# CONFIG_SERIAL_PCH_UART is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
CONFIG_HVC_DRIVER=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=m
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_GEODE=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_NVRAM=m
CONFIG_DTLK=m
CONFIG_R3964=m
CONFIG_APPLICOM=m
CONFIG_SONYPI=m
#
# PCMCIA character devices
#
CONFIG_SYNCLINK_CS=m
CONFIG_CARDMAN_4000=m
CONFIG_CARDMAN_4040=m
CONFIG_IPWIRELESS=m
CONFIG_MWAVE=m
CONFIG_SCx200_GPIO=m
CONFIG_PC8736x_GPIO=m
CONFIG_NSC_GPIO=m
CONFIG_RAW_DRIVER=m
CONFIG_MAX_RAW_DEVS=256
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=m
CONFIG_TCG_TIS=m
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
CONFIG_TELCLOCK=m
CONFIG_DEVPORT=y
CONFIG_RAMOOPS=m
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=m
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCA=m
#
# I2C Hardware Bus support
#
#
# PC SMBus host controller drivers
#
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_NFORCE2_S4985=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
#
# ACPI drivers
#
CONFIG_I2C_SCMI=m
#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_DESIGNWARE_PCI is not set
# CONFIG_I2C_GPIO is not set
CONFIG_I2C_OCORES=m
CONFIG_I2C_PCA_PLATFORM=m
# CONFIG_I2C_PXA_PCI is not set
CONFIG_I2C_SIMTEC=m
# CONFIG_I2C_XILINX is not set
# CONFIG_I2C_EG20T is not set
#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_DIOLAN_U2C is not set
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_TAOS_EVM=m
CONFIG_I2C_TINY_USB=m
#
# Other I2C/SMBus bus drivers
#
CONFIG_I2C_PCA_ISA=m
CONFIG_I2C_STUB=m
CONFIG_SCx200_I2C=m
CONFIG_SCx200_I2C_SCL=12
CONFIG_SCx200_I2C_SDA=13
CONFIG_SCx200_ACB=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
CONFIG_SPI=y
# CONFIG_SPI_DEBUG is not set
CONFIG_SPI_MASTER=y
#
# SPI Master Controller Drivers
#
# CONFIG_SPI_ALTERA is not set
CONFIG_SPI_BITBANG=m
CONFIG_SPI_BUTTERFLY=m
# CONFIG_SPI_GPIO is not set
CONFIG_SPI_LM70_LLP=m
# CONFIG_SPI_OC_TINY is not set
# CONFIG_SPI_PXA2XX is not set
# CONFIG_SPI_PXA2XX_PCI is not set
# CONFIG_SPI_TOPCLIFF_PCH is not set
# CONFIG_SPI_XILINX is not set
# CONFIG_SPI_DESIGNWARE is not set
#
# SPI Protocol Masters
#
# CONFIG_SPI_SPIDEV is not set
CONFIG_SPI_TLE62X0=m
#
# PPS support
#
CONFIG_PPS=m
# CONFIG_PPS_DEBUG is not set
#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=m
# CONFIG_PPS_CLIENT_PARPORT is not set
#
# PPS generators support
#
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_GPIOLIB=y
# CONFIG_DEBUG_GPIO is not set
# CONFIG_GPIO_SYSFS is not set
#
# Memory mapped GPIO expanders:
#
# CONFIG_GPIO_BASIC_MMIO is not set
# CONFIG_GPIO_IT8761E is not set
# CONFIG_GPIO_SCH is not set
# CONFIG_GPIO_VX855 is not set
#
# I2C GPIO expanders:
#
# CONFIG_GPIO_MAX7300 is not set
# CONFIG_GPIO_MAX732X is not set
# CONFIG_GPIO_PCA953X is not set
# CONFIG_GPIO_PCF857X is not set
# CONFIG_GPIO_ADP5588 is not set
#
# PCI GPIO expanders:
#
# CONFIG_GPIO_CS5535 is not set
# CONFIG_GPIO_LANGWELL is not set
# CONFIG_GPIO_PCH is not set
# CONFIG_GPIO_ML_IOH is not set
# CONFIG_GPIO_RDC321X is not set
#
# SPI GPIO expanders:
#
# CONFIG_GPIO_MAX7301 is not set
# CONFIG_GPIO_MCP23S08 is not set
# CONFIG_GPIO_MC33880 is not set
# CONFIG_GPIO_74X164 is not set
#
# AC97 GPIO expanders:
#
#
# MODULbus GPIO expanders:
#
CONFIG_W1=m
CONFIG_W1_CON=y
#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=m
CONFIG_W1_MASTER_DS2490=m
CONFIG_W1_MASTER_DS2482=m
# CONFIG_W1_MASTER_GPIO is not set
#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=m
CONFIG_W1_SLAVE_SMEM=m
# CONFIG_W1_SLAVE_DS2423 is not set
CONFIG_W1_SLAVE_DS2431=m
CONFIG_W1_SLAVE_DS2433=m
# CONFIG_W1_SLAVE_DS2433_CRC is not set
CONFIG_W1_SLAVE_DS2760=m
CONFIG_W1_SLAVE_BQ27000=m
CONFIG_POWER_SUPPLY=m
# CONFIG_POWER_SUPPLY_DEBUG is not set
CONFIG_PDA_POWER=m
# CONFIG_TEST_POWER is not set
CONFIG_BATTERY_DS2760=m
CONFIG_BATTERY_DS2782=m
# CONFIG_BATTERY_BQ20Z75 is not set
CONFIG_BATTERY_BQ27x00=m
CONFIG_BATTERY_BQ27X00_I2C=y
CONFIG_BATTERY_BQ27X00_PLATFORM=y
CONFIG_BATTERY_MAX17040=m
# CONFIG_BATTERY_MAX17042 is not set
CONFIG_CHARGER_PCF50633=m
# CONFIG_CHARGER_ISP1704 is not set
# CONFIG_CHARGER_GPIO is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set
#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ABITUGURU3=m
CONFIG_SENSORS_AD7414=m
CONFIG_SENSORS_AD7418=m
CONFIG_SENSORS_ADCXX=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7411=m
CONFIG_SENSORS_ADT7462=m
CONFIG_SENSORS_ADT7470=m
CONFIG_SENSORS_ADT7475=m
CONFIG_SENSORS_ASC7621=m
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
# CONFIG_SENSORS_DS620 is not set
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHMD=m
CONFIG_SENSORS_G760A=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
# CONFIG_SENSORS_GPIO_FAN is not set
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_PKGTEMP=m
CONFIG_SENSORS_IBMAEM=m
CONFIG_SENSORS_IBMPEX=m
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_JC42 is not set
# CONFIG_SENSORS_LINEAGE is not set
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM70=m
CONFIG_SENSORS_LM73=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
# CONFIG_SENSORS_LTC4151 is not set
CONFIG_SENSORS_LTC4215=m
CONFIG_SENSORS_LTC4245=m
# CONFIG_SENSORS_LTC4261 is not set
CONFIG_SENSORS_LM95241=m
CONFIG_SENSORS_MAX1111=m
CONFIG_SENSORS_MAX1619=m
# CONFIG_SENSORS_MAX6639 is not set
CONFIG_SENSORS_MAX6650=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_PCF8591=m
# CONFIG_PMBUS is not set
CONFIG_SENSORS_SHT15=m
# CONFIG_SENSORS_SHT21 is not set
CONFIG_SENSORS_SIS5595=m
# CONFIG_SENSORS_SMM665 is not set
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_EMC1403=m
# CONFIG_SENSORS_EMC2103 is not set
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
# CONFIG_SENSORS_SCH5627 is not set
# CONFIG_SENSORS_ADS1015 is not set
CONFIG_SENSORS_ADS7828=m
CONFIG_SENSORS_ADS7871=m
CONFIG_SENSORS_AMC6821=m
CONFIG_SENSORS_THMC50=m
CONFIG_SENSORS_TMP102=m
CONFIG_SENSORS_TMP401=m
CONFIG_SENSORS_TMP421=m
CONFIG_SENSORS_VIA_CPUTEMP=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
# CONFIG_SENSORS_W83795 is not set
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_APPLESMC=m
#
# ACPI drivers
#
CONFIG_SENSORS_ATK0110=m
CONFIG_THERMAL=m
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_ACQUIRE_WDT=m
CONFIG_ADVANTECH_WDT=m
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_F71808E_WDT=m
# CONFIG_SP5100_TCO is not set
CONFIG_GEODE_WDT=m
CONFIG_SC520_WDT=m
CONFIG_SBC_FITPC2_WATCHDOG=m
CONFIG_EUROTECH_WDT=m
CONFIG_IB700_WDT=m
CONFIG_IBMASR=m
CONFIG_WAFER_WDT=m
CONFIG_I6300ESB_WDT=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y
CONFIG_IT8712F_WDT=m
CONFIG_IT87_WDT=m
CONFIG_HP_WATCHDOG=m
CONFIG_HPWDT_NMI_DECODING=y
CONFIG_SC1200_WDT=m
CONFIG_SCx200_WDT=m
CONFIG_PC87413_WDT=m
# CONFIG_NV_TCO is not set
CONFIG_60XX_WDT=m
CONFIG_SBC8360_WDT=m
CONFIG_SBC7240_WDT=m
CONFIG_CPU5_WDT=m
CONFIG_SMSC_SCH311X_WDT=m
CONFIG_SMSC37B787_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83697HF_WDT=m
CONFIG_W83697UG_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
CONFIG_SBC_EPX_C3_WATCHDOG=m
#
# ISA-based Watchdog Cards
#
CONFIG_PCWATCHDOG=m
CONFIG_MIXCOMWD=m
CONFIG_WDT=m
#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m
#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_SSB_POSSIBLE=y
#
# Sonics Silicon Backplane
#
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_BLOCKIO=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
CONFIG_SSB_B43_PCI_BRIDGE=y
CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
CONFIG_SSB_PCMCIAHOST=y
CONFIG_SSB_SDIOHOST_POSSIBLE=y
CONFIG_SSB_SDIOHOST=y
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_MFD_SUPPORT=y
CONFIG_MFD_CORE=m
CONFIG_MFD_SM501=m
# CONFIG_MFD_SM501_GPIO is not set
CONFIG_HTC_PASIC3=m
# CONFIG_UCB1400_CORE is not set
# CONFIG_TPS6105X is not set
CONFIG_TPS65010=m
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TMIO is not set
CONFIG_MFD_WM8400=m
# CONFIG_MFD_WM831X_SPI is not set
CONFIG_MFD_PCF50633=m
# CONFIG_MFD_MC13XXX is not set
CONFIG_PCF50633_ADC=m
CONFIG_PCF50633_GPIO=m
# CONFIG_ABX500_CORE is not set
# CONFIG_EZX_PCAP is not set
# CONFIG_MFD_CS5535 is not set
# CONFIG_MFD_TIMBERDALE is not set
CONFIG_LPC_SCH=m
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_VX855 is not set
# CONFIG_MFD_WL1273_CORE is not set
CONFIG_REGULATOR=y
# CONFIG_REGULATOR_DEBUG is not set
# CONFIG_REGULATOR_DUMMY is not set
CONFIG_REGULATOR_FIXED_VOLTAGE=m
# CONFIG_REGULATOR_VIRTUAL_CONSUMER is not set
CONFIG_REGULATOR_USERSPACE_CONSUMER=m
CONFIG_REGULATOR_BQ24022=m
CONFIG_REGULATOR_MAX1586=m
CONFIG_REGULATOR_MAX8649=m
CONFIG_REGULATOR_MAX8660=m
# CONFIG_REGULATOR_MAX8952 is not set
CONFIG_REGULATOR_WM8400=m
CONFIG_REGULATOR_PCF50633=m
CONFIG_REGULATOR_LP3971=m
# CONFIG_REGULATOR_LP3972 is not set
CONFIG_REGULATOR_TPS65023=m
CONFIG_REGULATOR_TPS6507X=m
# CONFIG_REGULATOR_ISL6271A is not set
# CONFIG_REGULATOR_AD5398 is not set
# CONFIG_REGULATOR_TPS6524X is not set
CONFIG_MEDIA_SUPPORT=m
#
# Multimedia core support
#
# CONFIG_MEDIA_CONTROLLER is not set
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_DVB_CORE=m
CONFIG_VIDEO_MEDIA=m
#
# Multimedia drivers
#
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_RC_CORE=m
CONFIG_LIRC=m
CONFIG_RC_MAP=m
CONFIG_IR_NEC_DECODER=m
CONFIG_IR_RC5_DECODER=m
CONFIG_IR_RC6_DECODER=m
CONFIG_IR_JVC_DECODER=m
CONFIG_IR_SONY_DECODER=m
CONFIG_IR_RC5_SZ_DECODER=m
CONFIG_IR_LIRC_CODEC=m
CONFIG_IR_ENE=m
CONFIG_IR_IMON=m
CONFIG_IR_MCEUSB=m
# CONFIG_IR_ITE_CIR is not set
CONFIG_IR_NUVOTON=m
CONFIG_IR_STREAMZAP=m
# CONFIG_IR_WINBOND_CIR is not set
# CONFIG_RC_LOOPBACK is not set
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
# CONFIG_MEDIA_TUNER_CUSTOMISE is not set
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA827X=m
CONFIG_MEDIA_TUNER_TDA18271=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_MT2131=m
CONFIG_MEDIA_TUNER_QT1010=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_MEDIA_TUNER_MXL5005S=m
CONFIG_MEDIA_TUNER_MXL5007T=m
CONFIG_MEDIA_TUNER_MC44S803=m
CONFIG_MEDIA_TUNER_MAX2165=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_VMALLOC=m
CONFIG_VIDEOBUF_DVB=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_VMALLOC=m
CONFIG_VIDEO_CAPTURE_DRIVERS=y
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_IR_I2C=m
#
# Audio decoders
#
CONFIG_VIDEO_TVAUDIO=m
CONFIG_VIDEO_TDA7432=m
CONFIG_VIDEO_TDA9840=m
CONFIG_VIDEO_TEA6415C=m
CONFIG_VIDEO_TEA6420=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS5345=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_M52790=m
CONFIG_VIDEO_WM8775=m
CONFIG_VIDEO_WM8739=m
CONFIG_VIDEO_VP27SMPX=m
#
# RDS decoders
#
CONFIG_VIDEO_SAA6588=m
#
# Video decoders
#
CONFIG_VIDEO_BT819=m
CONFIG_VIDEO_BT856=m
CONFIG_VIDEO_BT866=m
CONFIG_VIDEO_KS0127=m
CONFIG_VIDEO_OV7670=m
CONFIG_VIDEO_MT9V011=m
CONFIG_VIDEO_SAA7110=m
CONFIG_VIDEO_SAA711X=m
CONFIG_VIDEO_SAA717X=m
CONFIG_VIDEO_TVP5150=m
CONFIG_VIDEO_VPX3220=m
#
# Video and audio decoders
#
CONFIG_VIDEO_CX25840=m
#
# MPEG video encoders
#
CONFIG_VIDEO_CX2341X=m
#
# Video encoders
#
CONFIG_VIDEO_SAA7127=m
CONFIG_VIDEO_SAA7185=m
CONFIG_VIDEO_ADV7170=m
CONFIG_VIDEO_ADV7175=m
#
# Video improvement chips
#
CONFIG_VIDEO_UPD64031A=m
CONFIG_VIDEO_UPD64083=m
CONFIG_VIDEO_VIVI=m
CONFIG_VIDEO_BT848=m
CONFIG_VIDEO_BT848_DVB=y
CONFIG_VIDEO_PMS=m
CONFIG_VIDEO_BWQCAM=m
CONFIG_VIDEO_CQCAM=m
CONFIG_VIDEO_W9966=m
CONFIG_VIDEO_CPIA2=m
CONFIG_VIDEO_ZORAN=m
CONFIG_VIDEO_ZORAN_DC30=m
CONFIG_VIDEO_ZORAN_ZR36060=m
CONFIG_VIDEO_ZORAN_BUZ=m
CONFIG_VIDEO_ZORAN_DC10=m
CONFIG_VIDEO_ZORAN_LML33=m
CONFIG_VIDEO_ZORAN_LML33R10=m
CONFIG_VIDEO_ZORAN_AVS6EYES=m
CONFIG_VIDEO_MEYE=m
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_VIDEO_SAA7134_DVB=m
CONFIG_VIDEO_MXB=m
CONFIG_VIDEO_HEXIUM_ORION=m
CONFIG_VIDEO_HEXIUM_GEMINI=m
# CONFIG_VIDEO_TIMBERDALE is not set
CONFIG_VIDEO_CX88=m
CONFIG_VIDEO_CX88_ALSA=m
CONFIG_VIDEO_CX88_BLACKBIRD=m
CONFIG_VIDEO_CX88_DVB=m
CONFIG_VIDEO_CX88_MPEG=m
CONFIG_VIDEO_CX88_VP3054=m
CONFIG_VIDEO_CX23885=m
# CONFIG_MEDIA_ALTERA_CI is not set
CONFIG_VIDEO_AU0828=m
CONFIG_VIDEO_IVTV=m
CONFIG_VIDEO_FB_IVTV=m
CONFIG_VIDEO_CX18=m
CONFIG_VIDEO_CX18_ALSA=m
CONFIG_VIDEO_SAA7164=m
CONFIG_VIDEO_CAFE_CCIC=m
# CONFIG_VIDEO_SR030PC30 is not set
# CONFIG_VIDEO_VIA_CAMERA is not set
# CONFIG_VIDEO_NOON010PC30 is not set
CONFIG_SOC_CAMERA=m
# CONFIG_SOC_CAMERA_IMX074 is not set
CONFIG_SOC_CAMERA_MT9M001=m
CONFIG_SOC_CAMERA_MT9M111=m
CONFIG_SOC_CAMERA_MT9T031=m
CONFIG_SOC_CAMERA_MT9T112=m
CONFIG_SOC_CAMERA_MT9V022=m
CONFIG_SOC_CAMERA_RJ54N1=m
CONFIG_SOC_CAMERA_TW9910=m
CONFIG_SOC_CAMERA_PLATFORM=m
# CONFIG_SOC_CAMERA_OV2640 is not set
# CONFIG_SOC_CAMERA_OV6650 is not set
CONFIG_SOC_CAMERA_OV772X=m
CONFIG_SOC_CAMERA_OV9640=m
# CONFIG_SOC_CAMERA_OV9740 is not set
CONFIG_V4L_USB_DRIVERS=y
CONFIG_USB_VIDEO_CLASS=m
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_USB_GSPCA=m
CONFIG_USB_M5602=m
CONFIG_USB_STV06XX=m
CONFIG_USB_GL860=m
CONFIG_USB_GSPCA_BENQ=m
CONFIG_USB_GSPCA_CONEX=m
CONFIG_USB_GSPCA_CPIA1=m
CONFIG_USB_GSPCA_ETOMS=m
CONFIG_USB_GSPCA_FINEPIX=m
CONFIG_USB_GSPCA_JEILINJ=m
CONFIG_USB_GSPCA_KONICA=m
CONFIG_USB_GSPCA_MARS=m
CONFIG_USB_GSPCA_MR97310A=m
# CONFIG_USB_GSPCA_NW80X is not set
CONFIG_USB_GSPCA_OV519=m
CONFIG_USB_GSPCA_OV534=m
CONFIG_USB_GSPCA_OV534_9=m
CONFIG_USB_GSPCA_PAC207=m
CONFIG_USB_GSPCA_PAC7302=m
CONFIG_USB_GSPCA_PAC7311=m
CONFIG_USB_GSPCA_SN9C2028=m
CONFIG_USB_GSPCA_SN9C20X=m
CONFIG_USB_GSPCA_SONIXB=m
CONFIG_USB_GSPCA_SONIXJ=m
CONFIG_USB_GSPCA_SPCA500=m
CONFIG_USB_GSPCA_SPCA501=m
CONFIG_USB_GSPCA_SPCA505=m
CONFIG_USB_GSPCA_SPCA506=m
CONFIG_USB_GSPCA_SPCA508=m
CONFIG_USB_GSPCA_SPCA561=m
CONFIG_USB_GSPCA_SPCA1528=m
CONFIG_USB_GSPCA_SQ905=m
CONFIG_USB_GSPCA_SQ905C=m
CONFIG_USB_GSPCA_SQ930X=m
CONFIG_USB_GSPCA_STK014=m
CONFIG_USB_GSPCA_STV0680=m
CONFIG_USB_GSPCA_SUNPLUS=m
CONFIG_USB_GSPCA_T613=m
CONFIG_USB_GSPCA_TV8532=m
CONFIG_USB_GSPCA_VC032X=m
# CONFIG_USB_GSPCA_VICAM is not set
CONFIG_USB_GSPCA_XIRLINK_CIT=m
CONFIG_USB_GSPCA_ZC3XX=m
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
# CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set
CONFIG_VIDEO_HDPVR=m
CONFIG_VIDEO_EM28XX=m
CONFIG_VIDEO_EM28XX_ALSA=m
CONFIG_VIDEO_EM28XX_DVB=m
CONFIG_VIDEO_TLG2300=m
CONFIG_VIDEO_CX231XX=m
CONFIG_VIDEO_CX231XX_RC=y
CONFIG_VIDEO_CX231XX_ALSA=m
CONFIG_VIDEO_CX231XX_DVB=m
CONFIG_VIDEO_USBVISION=m
CONFIG_USB_ET61X251=m
CONFIG_USB_SN9C102=m
CONFIG_USB_PWC=m
# CONFIG_USB_PWC_DEBUG is not set
CONFIG_USB_PWC_INPUT_EVDEV=y
CONFIG_USB_ZR364XX=m
CONFIG_USB_STKWEBCAM=m
CONFIG_USB_S2255=m
CONFIG_V4L_MEM2MEM_DRIVERS=y
# CONFIG_VIDEO_MEM2MEM_TESTDEV is not set
CONFIG_RADIO_ADAPTERS=y
CONFIG_RADIO_CADET=m
CONFIG_RADIO_RTRACK=m
CONFIG_RADIO_RTRACK2=m
CONFIG_RADIO_AZTECH=m
CONFIG_RADIO_GEMTEK=m
CONFIG_RADIO_MAXIRADIO=m
CONFIG_RADIO_MAESTRO=m
# CONFIG_RADIO_MIROPCM20 is not set
CONFIG_RADIO_SF16FMI=m
CONFIG_RADIO_SF16FMR2=m
CONFIG_RADIO_TERRATEC=m
CONFIG_RADIO_TRUST=m
CONFIG_RADIO_TYPHOON=m
CONFIG_RADIO_ZOLTRIX=m
CONFIG_I2C_SI4713=m
CONFIG_RADIO_SI4713=m
CONFIG_USB_DSBR=m
CONFIG_RADIO_SI470X=y
CONFIG_USB_SI470X=m
CONFIG_I2C_SI470X=m
CONFIG_USB_MR800=m
CONFIG_RADIO_TEA5764=m
CONFIG_RADIO_SAA7706H=m
CONFIG_RADIO_TEF6862=m
# CONFIG_RADIO_WL1273 is not set
#
# Texas Instruments WL128x FM driver (ST based)
#
# CONFIG_RADIO_WL128X is not set
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
CONFIG_DVB_CAPTURE_DRIVERS=y
#
# Supported SAA7146 based PCI Adapters
#
CONFIG_TTPCI_EEPROM=m
CONFIG_DVB_AV7110=m
CONFIG_DVB_AV7110_OSD=y
CONFIG_DVB_BUDGET_CORE=m
CONFIG_DVB_BUDGET=m
CONFIG_DVB_BUDGET_CI=m
CONFIG_DVB_BUDGET_AV=m
CONFIG_DVB_BUDGET_PATCH=m
#
# Supported USB Adapters
#
CONFIG_DVB_USB=m
# CONFIG_DVB_USB_DEBUG is not set
CONFIG_DVB_USB_A800=m
CONFIG_DVB_USB_DIBUSB_MB=m
CONFIG_DVB_USB_DIBUSB_MB_FAULTY=y
CONFIG_DVB_USB_DIBUSB_MC=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_UMT_010=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_M920X=m
CONFIG_DVB_USB_GL861=m
CONFIG_DVB_USB_AU6610=m
CONFIG_DVB_USB_DIGITV=m
CONFIG_DVB_USB_VP7045=m
CONFIG_DVB_USB_VP702X=m
CONFIG_DVB_USB_GP8PSK=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_TTUSB2=m
CONFIG_DVB_USB_DTT200U=m
CONFIG_DVB_USB_OPERA1=m
CONFIG_DVB_USB_AF9005=m
CONFIG_DVB_USB_AF9005_REMOTE=m
CONFIG_DVB_USB_DW2102=m
CONFIG_DVB_USB_CINERGY_T2=m
CONFIG_DVB_USB_ANYSEE=m
CONFIG_DVB_USB_DTV5100=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_USB_CE6230=m
CONFIG_DVB_USB_FRIIO=m
CONFIG_DVB_USB_EC168=m
CONFIG_DVB_USB_AZ6027=m
CONFIG_DVB_USB_LME2510=m
# CONFIG_DVB_USB_TECHNISAT_USB2 is not set
CONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_TTUSB_DEC=m
CONFIG_SMS_SIANO_MDTV=m
#
# Siano module components
#
CONFIG_SMS_USB_DRV=m
CONFIG_SMS_SDIO_DRV=m
#
# Supported FlexCopII (B2C2) Adapters
#
CONFIG_DVB_B2C2_FLEXCOP=m
CONFIG_DVB_B2C2_FLEXCOP_PCI=m
CONFIG_DVB_B2C2_FLEXCOP_USB=m
# CONFIG_DVB_B2C2_FLEXCOP_DEBUG is not set
#
# Supported BT878 Adapters
#
CONFIG_DVB_BT8XX=m
#
# Supported Pluto2 Adapters
#
CONFIG_DVB_PLUTO2=m
#
# Supported SDMC DM1105 Adapters
#
CONFIG_DVB_DM1105=m
#
# Supported FireWire (IEEE 1394) Adapters
#
CONFIG_DVB_FIREDTV=m
CONFIG_DVB_FIREDTV_INPUT=y
#
# Supported Earthsoft PT1 Adapters
#
CONFIG_DVB_PT1=m
#
# Supported Mantis Adapters
#
CONFIG_MANTIS_CORE=m
CONFIG_DVB_MANTIS=m
CONFIG_DVB_HOPPER=m
#
# Supported nGene Adapters
#
CONFIG_DVB_NGENE=m
#
# Supported DVB Frontends
#
# CONFIG_DVB_FE_CUSTOMISE is not set
#
# Multistandard (satellite) frontends
#
CONFIG_DVB_STB0899=m
CONFIG_DVB_STB6100=m
CONFIG_DVB_STV090x=m
CONFIG_DVB_STV6110x=m
#
# DVB-S (satellite) frontends
#
CONFIG_DVB_CX24110=m
CONFIG_DVB_CX24123=m
CONFIG_DVB_MT312=m
CONFIG_DVB_ZL10036=m
CONFIG_DVB_ZL10039=m
CONFIG_DVB_S5H1420=m
CONFIG_DVB_STV0288=m
CONFIG_DVB_STB6000=m
CONFIG_DVB_STV0299=m
CONFIG_DVB_STV6110=m
CONFIG_DVB_STV0900=m
CONFIG_DVB_TDA8083=m
CONFIG_DVB_TDA10086=m
CONFIG_DVB_TDA8261=m
CONFIG_DVB_VES1X93=m
CONFIG_DVB_TUNER_ITD1000=m
CONFIG_DVB_TUNER_CX24113=m
CONFIG_DVB_TDA826X=m
CONFIG_DVB_TUA6100=m
CONFIG_DVB_CX24116=m
CONFIG_DVB_SI21XX=m
CONFIG_DVB_DS3000=m
CONFIG_DVB_MB86A16=m
#
# DVB-T (terrestrial) frontends
#
CONFIG_DVB_SP8870=m
CONFIG_DVB_SP887X=m
CONFIG_DVB_CX22700=m
CONFIG_DVB_CX22702=m
CONFIG_DVB_L64781=m
CONFIG_DVB_TDA1004X=m
CONFIG_DVB_NXT6000=m
CONFIG_DVB_MT352=m
CONFIG_DVB_ZL10353=m
CONFIG_DVB_DIB3000MB=m
CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_TDA10048=m
CONFIG_DVB_AF9013=m
CONFIG_DVB_EC100=m
#
# DVB-C (cable) frontends
#
CONFIG_DVB_VES1820=m
CONFIG_DVB_TDA10021=m
CONFIG_DVB_TDA10023=m
CONFIG_DVB_STV0297=m
#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
CONFIG_DVB_NXT200X=m
CONFIG_DVB_OR51211=m
CONFIG_DVB_OR51132=m
CONFIG_DVB_BCM3510=m
CONFIG_DVB_LGDT330X=m
CONFIG_DVB_LGDT3305=m
CONFIG_DVB_S5H1409=m
CONFIG_DVB_AU8522=m
CONFIG_DVB_S5H1411=m
#
# ISDB-T (terrestrial) frontends
#
CONFIG_DVB_S921=m
CONFIG_DVB_DIB8000=m
CONFIG_DVB_MB86A20S=m
#
# Digital terrestrial only tuners/PLL
#
CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m
CONFIG_DVB_TUNER_DIB0090=m
#
# SEC control devices for DVB-S
#
CONFIG_DVB_LNBP21=m
CONFIG_DVB_ISL6405=m
CONFIG_DVB_ISL6421=m
CONFIG_DVB_ISL6423=m
CONFIG_DVB_LGS8GXX=m
CONFIG_DVB_ATBM8830=m
CONFIG_DVB_TDA665x=m
CONFIG_DVB_IX2505V=m
#
# Tools to develop new frontends
#
# CONFIG_DVB_DUMMY_FE is not set
#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_ALI=y
CONFIG_AGP_ATI=y
CONFIG_AGP_AMD=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_NVIDIA=y
CONFIG_AGP_SIS=y
CONFIG_AGP_SWORKS=y
CONFIG_AGP_VIA=y
CONFIG_AGP_EFFICEON=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_VGA_SWITCHEROO=y
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m
CONFIG_DRM_TDFX=m
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_RADEON_KMS=y
CONFIG_DRM_I810=m
CONFIG_DRM_I915=m
# CONFIG_DRM_I915_KMS is not set
CONFIG_DRM_MGA=m
CONFIG_DRM_SIS=m
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
# CONFIG_STUB_POULSBO is not set
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_DDC=m
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=m
# CONFIG_FB_WMT_GE_ROPS is not set
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_HECUBA=m
CONFIG_FB_SVGALIB=m
# CONFIG_FB_MACMODES is not set
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
#
# Frame buffer hardware drivers
#
CONFIG_FB_CIRRUS=m
CONFIG_FB_PM2=m
CONFIG_FB_PM2_FIFO_DISCONNECT=y
CONFIG_FB_CYBER2000=m
CONFIG_FB_CYBER2000_DDC=y
CONFIG_FB_ARC=m
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
CONFIG_FB_UVESA=m
CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_FB_N411=m
CONFIG_FB_HGA=m
CONFIG_FB_S1D13XXX=m
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
CONFIG_FB_NVIDIA_BACKLIGHT=y
# CONFIG_FB_RIVA is not set
CONFIG_FB_I810=m
# CONFIG_FB_I810_GTF is not set
CONFIG_FB_LE80578=m
CONFIG_FB_CARILLO_RANCH=m
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
CONFIG_FB_MATROX_MYSTIQUE=y
CONFIG_FB_MATROX_G=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
CONFIG_FB_RADEON_BACKLIGHT=y
# CONFIG_FB_RADEON_DEBUG is not set
CONFIG_FB_ATY128=m
CONFIG_FB_ATY128_BACKLIGHT=y
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
CONFIG_FB_ATY_GENERIC_LCD=y
CONFIG_FB_ATY_GX=y
CONFIG_FB_ATY_BACKLIGHT=y
CONFIG_FB_S3=m
CONFIG_FB_SAVAGE=m
CONFIG_FB_SAVAGE_I2C=y
# CONFIG_FB_SAVAGE_ACCEL is not set
CONFIG_FB_SIS=m
CONFIG_FB_SIS_300=y
CONFIG_FB_SIS_315=y
CONFIG_FB_VIA=m
# CONFIG_FB_VIA_DIRECT_PROCFS is not set
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=m
CONFIG_FB_3DFX=m
# CONFIG_FB_3DFX_ACCEL is not set
CONFIG_FB_3DFX_I2C=y
CONFIG_FB_VOODOO1=m
CONFIG_FB_VT8623=m
CONFIG_FB_TRIDENT=m
CONFIG_FB_ARK=m
CONFIG_FB_PM3=m
# CONFIG_FB_CARMINE is not set
CONFIG_FB_GEODE=y
CONFIG_FB_GEODE_LX=m
CONFIG_FB_GEODE_GX=m
CONFIG_FB_GEODE_GX1=m
# CONFIG_FB_TMIO is not set
CONFIG_FB_SM501=m
# CONFIG_FB_UDL is not set
CONFIG_FB_VIRTUAL=m
CONFIG_FB_METRONOME=m
CONFIG_FB_MB862XX=m
CONFIG_FB_MB862XX_PCI_GDC=y
# CONFIG_FB_BROADSHEET is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
# CONFIG_LCD_CLASS_DEVICE is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
# CONFIG_BACKLIGHT_GENERIC is not set
CONFIG_BACKLIGHT_PROGEAR=m
# CONFIG_BACKLIGHT_APPLE is not set
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
# CONFIG_BACKLIGHT_PCF50633 is not set
#
# Display device support
#
CONFIG_DISPLAY_SUPPORT=m
#
# Display hardware drivers
#
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
CONFIG_MDA_CONSOLE=m
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_LOGO is not set
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
# CONFIG_SOUND_OSS_CORE_PRECLAIM is not set
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
# CONFIG_SND_SEQUENCER_OSS is not set
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_OPL3_LIB_SEQ=m
CONFIG_SND_OPL4_LIB_SEQ=m
CONFIG_SND_SBAWE_SEQ=m
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_OPL4_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_PCSP=m
CONFIG_SND_DUMMY=m
# CONFIG_SND_ALOOP is not set
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
CONFIG_SND_MTS64=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_PORTMAN2X4=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_WSS_LIB=m
CONFIG_SND_SB_COMMON=m
CONFIG_SND_SB8_DSP=m
CONFIG_SND_SB16_DSP=m
CONFIG_SND_ISA=y
CONFIG_SND_ADLIB=m
CONFIG_SND_AD1816A=m
CONFIG_SND_AD1848=m
CONFIG_SND_ALS100=m
CONFIG_SND_AZT1605=m
CONFIG_SND_AZT2316=m
CONFIG_SND_AZT2320=m
CONFIG_SND_CMI8330=m
CONFIG_SND_CS4231=m
CONFIG_SND_CS4236=m
CONFIG_SND_ES1688=m
CONFIG_SND_ES18XX=m
CONFIG_SND_SC6000=m
CONFIG_SND_GUSCLASSIC=m
CONFIG_SND_GUSEXTREME=m
CONFIG_SND_GUSMAX=m
CONFIG_SND_INTERWAVE=m
CONFIG_SND_INTERWAVE_STB=m
CONFIG_SND_JAZZ16=m
CONFIG_SND_OPL3SA2=m
CONFIG_SND_OPTI92X_AD1848=m
CONFIG_SND_OPTI92X_CS4231=m
CONFIG_SND_OPTI93X=m
CONFIG_SND_MIRO=m
CONFIG_SND_SB8=m
CONFIG_SND_SB16=m
CONFIG_SND_SBAWE=m
CONFIG_SND_SB16_CSP=y
CONFIG_SND_SSCAPE=m
CONFIG_SND_WAVEFRONT=m
CONFIG_SND_MSND_PINNACLE=m
CONFIG_SND_MSND_CLASSIC=m
CONFIG_SND_PCI=y
CONFIG_SND_AD1889=m
CONFIG_SND_ALS300=m
CONFIG_SND_ALS4000=m
CONFIG_SND_ALI5451=m
CONFIG_SND_ASIHPI=m
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
# CONFIG_SND_AW2 is not set
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_OXYGEN_LIB=m
CONFIG_SND_OXYGEN=m
CONFIG_SND_CS4281=m
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CS5530=m
CONFIG_SND_CS5535AUDIO=m
CONFIG_SND_CTXFI=m
CONFIG_SND_DARLA20=m
CONFIG_SND_GINA20=m
CONFIG_SND_LAYLA20=m
CONFIG_SND_DARLA24=m
CONFIG_SND_GINA24=m
CONFIG_SND_LAYLA24=m
CONFIG_SND_MONA=m
CONFIG_SND_MIA=m
CONFIG_SND_ECHO3G=m
CONFIG_SND_INDIGO=m
CONFIG_SND_INDIGOIO=m
CONFIG_SND_INDIGODJ=m
CONFIG_SND_INDIGOIOX=m
CONFIG_SND_INDIGODJX=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
CONFIG_SND_ES1968_INPUT=y
CONFIG_SND_FM801=m
CONFIG_SND_FM801_TEA575X_BOOL=y
CONFIG_SND_FM801_TEA575X=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_HWDEP=y
CONFIG_SND_HDA_RECONFIG=y
CONFIG_SND_HDA_INPUT_BEEP=y
CONFIG_SND_HDA_INPUT_BEEP_MODE=1
CONFIG_SND_HDA_INPUT_JACK=y
CONFIG_SND_HDA_PATCH_LOADER=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CA0110=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
CONFIG_SND_HDSP=m
CONFIG_SND_HDSPM=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_KORG1212=m
CONFIG_SND_LX6464ES=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_MAESTRO3_INPUT=y
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_PCXHR=m
CONFIG_SND_RIPTIDE=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_SIS7019=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_TRIDENT=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
CONFIG_SND_VIRTUOSO=m
CONFIG_SND_VX222=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_SPI=y
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=m
CONFIG_SND_USB_UA101=m
CONFIG_SND_USB_USX2Y=m
CONFIG_SND_USB_CAIAQ=m
CONFIG_SND_USB_CAIAQ_INPUT=y
CONFIG_SND_USB_US122L=m
# CONFIG_SND_USB_6FIRE is not set
CONFIG_SND_FIREWIRE=y
# CONFIG_SND_FIREWIRE_SPEAKERS is not set
CONFIG_SND_PCMCIA=y
CONFIG_SND_VXPOCKET=m
CONFIG_SND_PDAUDIOCF=m
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
CONFIG_HID_SUPPORT=y
CONFIG_HID=m
CONFIG_HIDRAW=y
#
# USB Input Devices
#
CONFIG_USB_HID=m
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y
#
# Special HID drivers
#
CONFIG_HID_3M_PCT=m
CONFIG_HID_A4TECH=m
# CONFIG_HID_ACRUX is not set
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CANDO=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_PRODIKEYS=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_DRAGONRISE=m
CONFIG_DRAGONRISE_FF=y
# CONFIG_HID_EMS_FF is not set
CONFIG_HID_ELECOM=m
CONFIG_HID_EZKEY=m
# CONFIG_HID_KEYTOUCH is not set
CONFIG_HID_KYE=m
CONFIG_HID_UCLOGIC=m
CONFIG_HID_WALTOP=m
CONFIG_HID_GYRATION=m
CONFIG_HID_TWINHAN=m
CONFIG_HID_KENSINGTON=m
# CONFIG_HID_LCPOWER is not set
CONFIG_HID_LOGITECH=m
CONFIG_LOGITECH_FF=y
CONFIG_LOGIRUMBLEPAD2_FF=y
CONFIG_LOGIG940_FF=y
CONFIG_LOGIWII_FF=y
CONFIG_HID_MAGICMOUSE=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MOSART=m
CONFIG_HID_MONTEREY=m
# CONFIG_HID_MULTITOUCH is not set
CONFIG_HID_NTRIG=m
CONFIG_HID_ORTEK=m
CONFIG_HID_PANTHERLORD=m
CONFIG_PANTHERLORD_FF=y
CONFIG_HID_PETALYNX=m
CONFIG_HID_PICOLCD=m
CONFIG_HID_PICOLCD_FB=y
CONFIG_HID_PICOLCD_BACKLIGHT=y
CONFIG_HID_PICOLCD_LEDS=y
CONFIG_HID_QUANTA=m
CONFIG_HID_ROCCAT=m
CONFIG_HID_ROCCAT_COMMON=m
# CONFIG_HID_ROCCAT_ARVO is not set
CONFIG_HID_ROCCAT_KONE=m
# CONFIG_HID_ROCCAT_KONEPLUS is not set
# CONFIG_HID_ROCCAT_KOVAPLUS is not set
CONFIG_HID_ROCCAT_PYRA=m
CONFIG_HID_SAMSUNG=m
CONFIG_HID_SONY=m
CONFIG_HID_STANTUM=m
CONFIG_HID_SUNPLUS=m
CONFIG_HID_GREENASIA=m
CONFIG_GREENASIA_FF=y
CONFIG_HID_SMARTJOYPLUS=m
CONFIG_SMARTJOYPLUS_FF=y
CONFIG_HID_TOPSEED=m
CONFIG_HID_THRUSTMASTER=m
CONFIG_THRUSTMASTER_FF=y
CONFIG_HID_WACOM=m
CONFIG_HID_WACOM_POWER_SUPPLY=y
CONFIG_HID_ZEROPLUS=m
CONFIG_ZEROPLUS_FF=y
CONFIG_HID_ZYDACRON=m
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_DEVICE_CLASS is not set
CONFIG_USB_DYNAMIC_MINORS=y
CONFIG_USB_SUSPEND=y
# CONFIG_USB_OTG is not set
CONFIG_USB_MON=m
CONFIG_USB_WUSB=m
CONFIG_USB_WUSB_CBAF=m
# CONFIG_USB_WUSB_CBAF_DEBUG is not set
#
# USB Host Controller Drivers
#
CONFIG_USB_C67X00_HCD=m
CONFIG_USB_XHCI_HCD=m
# CONFIG_USB_XHCI_HCD_DEBUGGING is not set
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_OXU210HP_HCD is not set
CONFIG_USB_ISP116X_HCD=m
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_HCD_SSB is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_U132_HCD=m
CONFIG_USB_SL811_HCD=m
CONFIG_USB_SL811_CS=m
CONFIG_USB_R8A66597_HCD=m
CONFIG_USB_WHCI_HCD=m
CONFIG_USB_HWA_HCD=m
#
# Enable Host or Gadget support to see Inventra options
#
#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m
#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#
#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_REALTEK is not set
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
CONFIG_USB_STORAGE_ONETOUCH=m
CONFIG_USB_STORAGE_KARMA=m
CONFIG_USB_STORAGE_CYPRESS_ATACB=m
# CONFIG_USB_STORAGE_ENE_UB6250 is not set
CONFIG_USB_UAS=m
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m
#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_CH341=m
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CP210X=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
CONFIG_USB_SERIAL_IUU=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_MOS7720=m
CONFIG_USB_SERIAL_MOS7715_PARPORT=y
CONFIG_USB_SERIAL_MOS7840=m
CONFIG_USB_SERIAL_MOTOROLA=m
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_OTI6858=m
CONFIG_USB_SERIAL_QCAUX=m
CONFIG_USB_SERIAL_QUALCOMM=m
CONFIG_USB_SERIAL_SPCP8X5=m
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
# CONFIG_USB_SERIAL_SAFE_PADDED is not set
CONFIG_USB_SERIAL_SAMBA=m
CONFIG_USB_SERIAL_SIEMENS_MPI=m
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
CONFIG_USB_SERIAL_SYMBOL=m
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_SERIAL_OPTICON=m
CONFIG_USB_SERIAL_VIVOPAY_SERIAL=m
CONFIG_USB_SERIAL_ZIO=m
CONFIG_USB_SERIAL_SSU100=m
CONFIG_USB_SERIAL_DEBUG=m
#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_ADUTUX=m
CONFIG_USB_SEVSEG=m
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
CONFIG_USB_CYPRESS_CY7C63=m
CONFIG_USB_CYTHERM=m
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
CONFIG_USB_TRANCEVIBRATOR=m
CONFIG_USB_IOWARRIOR=m
CONFIG_USB_TEST=m
CONFIG_USB_ISIGHTFW=m
# CONFIG_USB_YUREX is not set
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m
# CONFIG_USB_GADGET is not set
#
# OTG and related infrastructure
#
CONFIG_USB_OTG_UTILS=y
# CONFIG_USB_GPIO_VBUS is not set
CONFIG_NOP_USB_XCEIV=m
CONFIG_UWB=m
CONFIG_UWB_HWA=m
CONFIG_UWB_WHCI=m
CONFIG_UWB_I1480U=m
CONFIG_MMC=m
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set
# CONFIG_MMC_CLKGATE is not set
#
# MMC/SD/SDIO Card Drivers
#
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y
CONFIG_SDIO_UART=m
# CONFIG_MMC_TEST is not set
#
# MMC/SD/SDIO Host Controller Drivers
#
CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_RICOH_MMC=y
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_MMC_WBSD=m
CONFIG_MMC_TIFM_SD=m
CONFIG_MMC_SDRICOH_CS=m
CONFIG_MMC_CB710=m
CONFIG_MMC_VIA_SDMMC=m
# CONFIG_MMC_USHC is not set
CONFIG_MEMSTICK=m
# CONFIG_MEMSTICK_DEBUG is not set
#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
CONFIG_MSPRO_BLOCK=m
#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=m
CONFIG_MEMSTICK_JMICRON_38X=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
#
# LED drivers
#
# CONFIG_LEDS_LM3530 is not set
CONFIG_LEDS_NET48XX=m
CONFIG_LEDS_WRAP=m
CONFIG_LEDS_ALIX2=m
CONFIG_LEDS_PCA9532=m
# CONFIG_LEDS_GPIO is not set
CONFIG_LEDS_LP3944=m
# CONFIG_LEDS_LP5521 is not set
# CONFIG_LEDS_LP5523 is not set
CONFIG_LEDS_CLEVO_MAIL=m
CONFIG_LEDS_PCA955X=m
CONFIG_LEDS_DAC124S085=m
CONFIG_LEDS_REGULATOR=m
CONFIG_LEDS_BD2802=m
CONFIG_LEDS_INTEL_SS4200=m
CONFIG_LEDS_LT3593=m
CONFIG_LEDS_DELL_NETBOOKS=m
CONFIG_LEDS_TRIGGERS=y
#
# LED Triggers
#
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_IDE_DISK=y
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
CONFIG_LEDS_TRIGGER_BACKLIGHT=m
# CONFIG_LEDS_TRIGGER_GPIO is not set
CONFIG_LEDS_TRIGGER_DEFAULT_ON=m
#
# iptables trigger is under Netfilter config (LED target)
#
# CONFIG_NFC_DEVICES is not set
CONFIG_ACCESSIBILITY=y
CONFIG_A11Y_BRAILLE_CONSOLE=y
CONFIG_INFINIBAND=m
CONFIG_INFINIBAND_USER_MAD=m
CONFIG_INFINIBAND_USER_ACCESS=m
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_MTHCA=m
CONFIG_INFINIBAND_MTHCA_DEBUG=y
CONFIG_INFINIBAND_AMSO1100=m
# CONFIG_INFINIBAND_AMSO1100_DEBUG is not set
CONFIG_INFINIBAND_CXGB3=m
# CONFIG_INFINIBAND_CXGB3_DEBUG is not set
CONFIG_INFINIBAND_CXGB4=m
CONFIG_MLX4_INFINIBAND=m
CONFIG_INFINIBAND_NES=m
# CONFIG_INFINIBAND_NES_DEBUG is not set
CONFIG_INFINIBAND_IPOIB=m
CONFIG_INFINIBAND_IPOIB_CM=y
CONFIG_INFINIBAND_IPOIB_DEBUG=y
# CONFIG_INFINIBAND_IPOIB_DEBUG_DATA is not set
CONFIG_INFINIBAND_SRP=m
CONFIG_INFINIBAND_ISER=m
CONFIG_EDAC=y
#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=m
# CONFIG_EDAC_MCE_INJ is not set
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_MCE=y
CONFIG_EDAC_AMD76X=m
CONFIG_EDAC_E7XXX=m
CONFIG_EDAC_E752X=m
CONFIG_EDAC_I82875P=m
CONFIG_EDAC_I82975X=m
CONFIG_EDAC_I3000=m
CONFIG_EDAC_I3200=m
CONFIG_EDAC_X38=m
CONFIG_EDAC_I5400=m
CONFIG_EDAC_I7CORE=m
CONFIG_EDAC_I82860=m
CONFIG_EDAC_R82600=m
CONFIG_EDAC_I5000=m
CONFIG_EDAC_I5100=m
# CONFIG_EDAC_I7300 is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
CONFIG_RTC_DRV_DS1374=m
CONFIG_RTC_DRV_DS1672=m
# CONFIG_RTC_DRV_DS3232 is not set
CONFIG_RTC_DRV_MAX6900=m
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
# CONFIG_RTC_DRV_ISL12022 is not set
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
CONFIG_RTC_DRV_M41T80=m
# CONFIG_RTC_DRV_M41T80_WDT is not set
CONFIG_RTC_DRV_BQ32K=m
CONFIG_RTC_DRV_S35390A=m
CONFIG_RTC_DRV_FM3130=m
CONFIG_RTC_DRV_RX8581=m
CONFIG_RTC_DRV_RX8025=m
#
# SPI RTC drivers
#
CONFIG_RTC_DRV_M41T94=m
CONFIG_RTC_DRV_DS1305=m
CONFIG_RTC_DRV_DS1390=m
CONFIG_RTC_DRV_MAX6902=m
CONFIG_RTC_DRV_R9701=m
CONFIG_RTC_DRV_RS5C348=m
CONFIG_RTC_DRV_DS3234=m
CONFIG_RTC_DRV_PCF2123=m
#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
CONFIG_RTC_DRV_DS1286=m
CONFIG_RTC_DRV_DS1511=m
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1742=m
CONFIG_RTC_DRV_STK17TA8=m
CONFIG_RTC_DRV_M48T86=m
CONFIG_RTC_DRV_M48T35=m
CONFIG_RTC_DRV_M48T59=m
CONFIG_RTC_DRV_MSM6242=m
CONFIG_RTC_DRV_BQ4802=m
CONFIG_RTC_DRV_RP5C01=m
CONFIG_RTC_DRV_V3020=m
CONFIG_RTC_DRV_PCF50633=m
#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y
# CONFIG_DMADEVICES_DEBUG is not set
#
# DMA Devices
#
# CONFIG_INTEL_MID_DMAC is not set
CONFIG_INTEL_IOATDMA=m
# CONFIG_TIMB_DMA is not set
# CONFIG_PCH_DMA is not set
CONFIG_DMA_ENGINE=y
#
# DMA Clients
#
CONFIG_NET_DMA=y
CONFIG_ASYNC_TX_DMA=y
# CONFIG_DMATEST is not set
CONFIG_DCA=m
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=m
CONFIG_UIO_CIF=m
CONFIG_UIO_PDRV=m
CONFIG_UIO_PDRV_GENIRQ=m
CONFIG_UIO_AEC=m
CONFIG_UIO_SERCOS3=m
CONFIG_UIO_PCI_GENERIC=m
CONFIG_UIO_NETX=m
CONFIG_STAGING=y
# CONFIG_STAGING_EXCLUDE_BUILD is not set
# CONFIG_STALLION is not set
# CONFIG_ISTALLION is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_RISCOM8 is not set
# CONFIG_SPECIALIX is not set
CONFIG_COMPUTONE=m
CONFIG_ET131X=m
# CONFIG_ET131X_DEBUG is not set
# CONFIG_SLICOSS is not set
# CONFIG_VIDEO_GO7007 is not set
# CONFIG_VIDEO_CX25821 is not set
CONFIG_VIDEO_TM6000=m
CONFIG_VIDEO_TM6000_ALSA=m
CONFIG_VIDEO_TM6000_DVB=m
# CONFIG_DVB_CXD2099 is not set
CONFIG_USB_IP_COMMON=m
CONFIG_USB_IP_VHCI_HCD=m
CONFIG_USB_IP_HOST=m
# CONFIG_USB_IP_DEBUG_ENABLE is not set
# CONFIG_W35UND is not set
CONFIG_PRISM2_USB=m
# CONFIG_ECHO is not set
CONFIG_BRCM80211=m
# CONFIG_BRCMSMAC is not set
# CONFIG_BRCMFMAC is not set
# CONFIG_BRCMDBG is not set
# CONFIG_RT2860 is not set
# CONFIG_RT2870 is not set
CONFIG_COMEDI=m
# CONFIG_COMEDI_DEBUG is not set
# CONFIG_COMEDI_MISC_DRIVERS is not set
# CONFIG_COMEDI_ISA_DRIVERS is not set
CONFIG_COMEDI_PCI_DRIVERS=m
# CONFIG_COMEDI_ADDI_APCI_035 is not set
# CONFIG_COMEDI_ADDI_APCI_1032 is not set
# CONFIG_COMEDI_ADDI_APCI_1500 is not set
# CONFIG_COMEDI_ADDI_APCI_1516 is not set
# CONFIG_COMEDI_ADDI_APCI_1564 is not set
# CONFIG_COMEDI_ADDI_APCI_16XX is not set
# CONFIG_COMEDI_ADDI_APCI_2016 is not set
# CONFIG_COMEDI_ADDI_APCI_2032 is not set
# CONFIG_COMEDI_ADDI_APCI_2200 is not set
# CONFIG_COMEDI_ADDI_APCI_3001 is not set
# CONFIG_COMEDI_ADDI_APCI_3120 is not set
# CONFIG_COMEDI_ADDI_APCI_3501 is not set
# CONFIG_COMEDI_ADDI_APCI_3XXX is not set
# CONFIG_COMEDI_ADL_PCI6208 is not set
# CONFIG_COMEDI_ADL_PCI7230 is not set
# CONFIG_COMEDI_ADL_PCI7296 is not set
# CONFIG_COMEDI_ADL_PCI7432 is not set
# CONFIG_COMEDI_ADL_PCI8164 is not set
# CONFIG_COMEDI_ADL_PCI9111 is not set
# CONFIG_COMEDI_ADL_PCI9118 is not set
# CONFIG_COMEDI_ADV_PCI1710 is not set
# CONFIG_COMEDI_ADV_PCI1723 is not set
# CONFIG_COMEDI_ADV_PCI_DIO is not set
# CONFIG_COMEDI_AMPLC_DIO200 is not set
# CONFIG_COMEDI_AMPLC_PC236 is not set
# CONFIG_COMEDI_AMPLC_PC263 is not set
# CONFIG_COMEDI_AMPLC_PCI224 is not set
# CONFIG_COMEDI_AMPLC_PCI230 is not set
# CONFIG_COMEDI_CONTEC_PCI_DIO is not set
# CONFIG_COMEDI_DT3000 is not set
# CONFIG_COMEDI_UNIOXX5 is not set
# CONFIG_COMEDI_GSC_HPDI is not set
# CONFIG_COMEDI_ICP_MULTI is not set
# CONFIG_COMEDI_II_PCI20KC is not set
# CONFIG_COMEDI_DAQBOARD2000 is not set
# CONFIG_COMEDI_JR3_PCI is not set
# CONFIG_COMEDI_KE_COUNTER is not set
# CONFIG_COMEDI_CB_PCIDAS64 is not set
# CONFIG_COMEDI_CB_PCIDAS is not set
# CONFIG_COMEDI_CB_PCIDDA is not set
# CONFIG_COMEDI_CB_PCIDIO is not set
# CONFIG_COMEDI_CB_PCIMDAS is not set
# CONFIG_COMEDI_CB_PCIMDDA is not set
# CONFIG_COMEDI_ME4000 is not set
# CONFIG_COMEDI_ME_DAQ is not set
# CONFIG_COMEDI_RTD520 is not set
# CONFIG_COMEDI_S526 is not set
# CONFIG_COMEDI_S626 is not set
# CONFIG_COMEDI_SSV_DNP is not set
CONFIG_COMEDI_PCMCIA_DRIVERS=m
# CONFIG_COMEDI_CB_DAS16_CS is not set
# CONFIG_COMEDI_DAS08_CS is not set
# CONFIG_COMEDI_QUATECH_DAQP_CS is not set
CONFIG_COMEDI_USB_DRIVERS=m
# CONFIG_COMEDI_DT9812 is not set
# CONFIG_COMEDI_USBDUX is not set
# CONFIG_COMEDI_USBDUXFAST is not set
# CONFIG_COMEDI_VMK80XX is not set
# CONFIG_COMEDI_NI_COMMON is not set
# CONFIG_COMEDI_8255 is not set
# CONFIG_COMEDI_DAS08 is not set
# CONFIG_COMEDI_FC is not set
# CONFIG_ASUS_OLED is not set
# CONFIG_PANEL is not set
CONFIG_R8187SE=m
CONFIG_RTL8192U=m
CONFIG_RTL8192E=m
CONFIG_R8712U=m
# CONFIG_R8712_AP is not set
# CONFIG_RTS_PSTOR is not set
# CONFIG_TRANZPORT is not set
# CONFIG_POHMELFS is not set
# CONFIG_IDE_PHISON is not set
# CONFIG_LINE6_USB is not set
CONFIG_DRM_VMWGFX=m
CONFIG_DRM_NOUVEAU=m
CONFIG_DRM_NOUVEAU_BACKLIGHT=y
# CONFIG_DRM_NOUVEAU_DEBUG is not set
#
# I2C encoder or helper chips
#
CONFIG_DRM_I2C_CH7006=m
CONFIG_DRM_I2C_SIL164=m
# CONFIG_USB_SERIAL_QUATECH2 is not set
# CONFIG_USB_SERIAL_QUATECH_USB2 is not set
# CONFIG_VT6655 is not set
# CONFIG_VT6656 is not set
# CONFIG_HYPERV is not set
# CONFIG_VME_BUS is not set
# CONFIG_DX_SEP is not set
# CONFIG_IIO is not set
CONFIG_CS5535_GPIO=m
CONFIG_XVMALLOC=y
CONFIG_ZRAM=m
# CONFIG_ZRAM_DEBUG is not set
# CONFIG_ZCACHE is not set
# CONFIG_WLAGS49_H2 is not set
# CONFIG_WLAGS49_H25 is not set
CONFIG_SAMSUNG_LAPTOP=m
# CONFIG_FB_SM7XX is not set
# CONFIG_VIDEO_DT3155 is not set
# CONFIG_CRYSTALHD is not set
# CONFIG_CXT1E1 is not set
# CONFIG_FB_XGI is not set
CONFIG_LIRC_STAGING=y
CONFIG_LIRC_BT829=m
CONFIG_LIRC_IGORPLUGUSB=m
CONFIG_LIRC_IMON=m
# CONFIG_LIRC_PARALLEL is not set
CONFIG_LIRC_SASEM=m
CONFIG_LIRC_SERIAL=m
CONFIG_LIRC_SERIAL_TRANSMITTER=y
CONFIG_LIRC_SIR=m
CONFIG_LIRC_TTUSBIR=m
CONFIG_LIRC_ZILOG=m
# CONFIG_EASYCAP is not set
# CONFIG_SOLO6X10 is not set
# CONFIG_ACPI_QUICKSTART is not set
CONFIG_MACH_NO_WESTBRIDGE=y
# CONFIG_SBE_2T3E3 is not set
# CONFIG_ATH6K_LEGACY is not set
# CONFIG_USB_ENESTORAGE is not set
# CONFIG_BCM_WIMAX is not set
# CONFIG_FT1000 is not set
#
# Speakup console speech
#
CONFIG_SPEAKUP=m
CONFIG_SPEAKUP_SYNTH_ACNTSA=m
CONFIG_SPEAKUP_SYNTH_ACNTPC=m
CONFIG_SPEAKUP_SYNTH_APOLLO=m
CONFIG_SPEAKUP_SYNTH_AUDPTR=m
CONFIG_SPEAKUP_SYNTH_BNS=m
CONFIG_SPEAKUP_SYNTH_DECTLK=m
CONFIG_SPEAKUP_SYNTH_DECEXT=m
# CONFIG_SPEAKUP_SYNTH_DECPC is not set
CONFIG_SPEAKUP_SYNTH_DTLK=m
CONFIG_SPEAKUP_SYNTH_KEYPC=m
CONFIG_SPEAKUP_SYNTH_LTLK=m
CONFIG_SPEAKUP_SYNTH_SOFT=m
CONFIG_SPEAKUP_SYNTH_SPKOUT=m
CONFIG_SPEAKUP_SYNTH_TXPRT=m
CONFIG_SPEAKUP_SYNTH_DUMMY=m
# CONFIG_TOUCHSCREEN_CLEARPAD_TM1217 is not set
# CONFIG_TOUCHSCREEN_SYNAPTICS_I2C_RMI4 is not set
# CONFIG_DRM_PSB is not set
#
# Altera FPGA firmware download module
#
# CONFIG_ALTERA_STAPL is not set
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_ACER_WMI=m
CONFIG_ACERHDF=m
CONFIG_ASUS_LAPTOP=m
CONFIG_DELL_LAPTOP=m
CONFIG_DELL_WMI=m
# CONFIG_DELL_WMI_AIO is not set
CONFIG_FUJITSU_LAPTOP=m
# CONFIG_FUJITSU_LAPTOP_DEBUG is not set
CONFIG_TC1100_WMI=m
CONFIG_HP_ACCEL=m
CONFIG_HP_WMI=m
CONFIG_MSI_LAPTOP=m
CONFIG_PANASONIC_LAPTOP=m
CONFIG_COMPAL_LAPTOP=m
CONFIG_SONY_LAPTOP=m
CONFIG_SONYPI_COMPAT=y
CONFIG_IDEAPAD_LAPTOP=m
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
# CONFIG_THINKPAD_ACPI_DEBUGFACILITIES is not set
# CONFIG_THINKPAD_ACPI_DEBUG is not set
# CONFIG_THINKPAD_ACPI_UNSAFE_LEDS is not set
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_THINKPAD_ACPI_HOTKEY_POLL=y
CONFIG_SENSORS_HDAPS=m
# CONFIG_INTEL_MENLOW is not set
CONFIG_EEEPC_LAPTOP=m
# CONFIG_ASUS_WMI is not set
CONFIG_ACPI_WMI=m
CONFIG_MSI_WMI=m
CONFIG_ACPI_ASUS=m
CONFIG_TOPSTAR_LAPTOP=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_TOSHIBA_BT_RFKILL=m
CONFIG_ACPI_CMPC=m
# CONFIG_INTEL_IPS is not set
# CONFIG_IBM_RTL is not set
# CONFIG_XO15_EBOOK is not set
#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_EFI_VARS=m
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
CONFIG_DMIID=y
# CONFIG_DMI_SYSFS is not set
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
# CONFIG_SIGMA is not set
#
# File systems
#
CONFIG_EXT2_FS=m
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=m
CONFIG_EXT3_DEFAULTS_TO_ORDERED=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=m
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD=m
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=m
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
# CONFIG_XFS_DEBUG is not set
CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_DLM=y
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
CONFIG_OCFS2_FS_STATS=y
CONFIG_OCFS2_DEBUG_MASKLOG=y
# CONFIG_OCFS2_DEBUG_FS is not set
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_NILFS2_FS=m
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
# CONFIG_FANOTIFY_ACCESS_PERMISSIONS is not set
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=m
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_OVERLAYFS_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y
#
# Caches
#
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
CONFIG_CACHEFILES=m
# CONFIG_CACHEFILES_DEBUG is not set
# CONFIG_CACHEFILES_HISTOGRAM is not set
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="utf8"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
CONFIG_NTFS_RW=y
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
CONFIG_ADFS_FS=m
# CONFIG_ADFS_FS_RW is not set
CONFIG_AFFS_FS=m
CONFIG_ECRYPT_FS=m
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_BEFS_FS=m
# CONFIG_BEFS_DEBUG is not set
CONFIG_BFS_FS=m
CONFIG_EFS_FS=m
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_JFFS2_FS_WRITEBUFFER=y
# CONFIG_JFFS2_FS_WBUF_VERIFY is not set
CONFIG_JFFS2_SUMMARY=y
CONFIG_JFFS2_FS_XATTR=y
CONFIG_JFFS2_FS_POSIX_ACL=y
CONFIG_JFFS2_FS_SECURITY=y
CONFIG_JFFS2_COMPRESSION_OPTIONS=y
CONFIG_JFFS2_ZLIB=y
CONFIG_JFFS2_LZO=y
CONFIG_JFFS2_RTIME=y
# CONFIG_JFFS2_RUBIN is not set
# CONFIG_JFFS2_CMODE_NONE is not set
CONFIG_JFFS2_CMODE_PRIORITY=y
# CONFIG_JFFS2_CMODE_SIZE is not set
# CONFIG_JFFS2_CMODE_FAVOURLZO is not set
CONFIG_UBIFS_FS=m
CONFIG_UBIFS_FS_XATTR=y
CONFIG_UBIFS_FS_ADVANCED_COMPR=y
CONFIG_UBIFS_FS_LZO=y
CONFIG_UBIFS_FS_ZLIB=y
# CONFIG_UBIFS_FS_DEBUG is not set
CONFIG_LOGFS=m
CONFIG_CRAMFS=m
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_XATTR=y
# CONFIG_SQUASHFS_LZO is not set
CONFIG_SQUASHFS_XZ=y
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
CONFIG_VXFS_FS=m
CONFIG_MINIX_FS=m
CONFIG_OMFS_FS=m
CONFIG_HPFS_FS=m
CONFIG_QNX4FS_FS=m
CONFIG_ROMFS_FS=m
# CONFIG_ROMFS_BACKED_BY_BLOCK is not set
# CONFIG_ROMFS_BACKED_BY_MTD is not set
CONFIG_ROMFS_BACKED_BY_BOTH=y
CONFIG_ROMFS_ON_BLOCK=y
CONFIG_ROMFS_ON_MTD=y
CONFIG_PSTORE=y
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_UFS_DEBUG is not set
CONFIG_EXOFS_FS=m
# CONFIG_EXOFS_DEBUG is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_V4_1 is not set
CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
# CONFIG_NFS_USE_NEW_IDMAPPER is not set
CONFIG_NFSD=m
CONFIG_NFSD_DEPRECATED=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_XPRT_RDMA=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_CEPH_FS=m
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
CONFIG_CIFS_DFS_UPCALL=y
CONFIG_CIFS_FSCACHE=y
CONFIG_CIFS_ACL=y
# CONFIG_CIFS_NFSD_EXPORT is not set
CONFIG_NCP_FS=m
CONFIG_NCPFS_PACKET_SIGNING=y
CONFIG_NCPFS_IOCTL_LOCKING=y
CONFIG_NCPFS_STRONG=y
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
# CONFIG_NCPFS_SMALLDOS is not set
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
CONFIG_AFS_FS=m
# CONFIG_AFS_DEBUG is not set
CONFIG_AFS_FSCACHE=y
CONFIG_9P_FS=m
CONFIG_9P_FSCACHE=y
CONFIG_9P_FS_POSIX_ACL=y
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
CONFIG_ACORN_PARTITION=y
# CONFIG_ACORN_PARTITION_CUMANA is not set
# CONFIG_ACORN_PARTITION_EESOX is not set
CONFIG_ACORN_PARTITION_ICS=y
# CONFIG_ACORN_PARTITION_ADFS is not set
# CONFIG_ACORN_PARTITION_POWERTEC is not set
CONFIG_ACORN_PARTITION_RISCIX=y
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not set
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_MASK=0x01b6
CONFIG_STRIP_ASM_SYMS=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_HARDLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHEDSTATS is not set
CONFIG_TIMER_STATS=y
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=120
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_LKDTM is not set
# CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_SYSCTL_SYSCALL_CHECK=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_EVENT_POWER_TRACING_DEPRECATED=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
# CONFIG_FUNCTION_TRACER is not set
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
# CONFIG_FTRACE_SYSCALLS is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_STACK_TRACER is not set
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_ASYNC_RAID6_TEST is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
# CONFIG_KGDB_LOW_LEVEL_TRAP is not set
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_TEST_KSTRTOX is not set
CONFIG_STRICT_DEVMEM=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_DEBUG_RODATA is not set
# CONFIG_DEBUG_SET_MODULE_RONX is not set
# CONFIG_DEBUG_NX_TEST is not set
CONFIG_DOUBLEFAULT=y
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
# CONFIG_X86_DECODER_SELFTEST is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
CONFIG_OPTIMIZE_INLINING=y
# CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set
#
# Security options
#
CONFIG_KEYS=y
# CONFIG_TRUSTED_KEYS is not set
CONFIG_KEYS_DEBUG_PROC_KEYS=y
# CONFIG_SECURITY_DMESG_RESTRICT is not set
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_SECURITY_PATH=y
# CONFIG_INTEL_TXT is not set
CONFIG_LSM_MMAP_MIN_ADDR=0
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=0
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
CONFIG_SECURITY_TOMOYO=y
# CONFIG_SECURITY_APPARMOR is not set
# CONFIG_IMA is not set
CONFIG_DEFAULT_SECURITY_SELINUX=y
# CONFIG_DEFAULT_SECURITY_TOMOYO is not set
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=m
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_MANAGER_DISABLE_TESTS is not set
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m
#
# Block modes
#
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m
#
# Digest
#
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_CRC32C_INTEL=m
CONFIG_CRYPTO_GHASH=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_586=m
# CONFIG_CRYPTO_AES_NI_INTEL is not set
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_586=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_586=m
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_ZLIB=m
CONFIG_CRYPTO_LZO=m
#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
CONFIG_CRYPTO_DEV_GEODE=m
CONFIG_CRYPTO_DEV_HIFN_795X=m
CONFIG_CRYPTO_DEV_HIFN_795X_RNG=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
# CONFIG_KVM_MMU_AUDIT is not set
CONFIG_VHOST_NET=m
CONFIG_LGUEST=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_BINARY_PRINTF=y
#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_T10DIF=m
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC7=m
CONFIG_LIBCRC32C=m
CONFIG_AUDIT_GENERIC=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_REED_SOLOMON=m
CONFIG_REED_SOLOMON_DEC16=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_BTREE=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_NLATTR=y
CONFIG_LRU_CACHE=m
CONFIG_AVERAGE=y
# CONFIG_SHM_SIGNAL is not set
# CONFIG_IOQ is not set
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-26 3:42 ` Paul E. McKenney
2011-03-26 8:11 ` Sedat Dilek
@ 2011-03-26 12:34 ` Sedat Dilek
2011-03-26 16:02 ` Paul E. McKenney
1 sibling, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-26 12:34 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sat, Mar 26, 2011 at 4:42 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Fri, Mar 25, 2011 at 08:42:14PM +0100, Sedat Dilek wrote:
>> On Fri, Mar 25, 2011 at 6:48 PM, Paul E. McKenney
>> <paulmck@linux.vnet.ibm.com> wrote:
>> > On Fri, Mar 25, 2011 at 06:40:38PM +0100, Sedat Dilek wrote:
>> >> On Fri, Mar 25, 2011 at 5:51 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
>> >> > On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
>> >> > <paulmck@linux.vnet.ibm.com> wrote:
>> >> >> On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
>> >> >>> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
>> >> >>> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
>> >> >>> > <sedat.dilek@googlemail.com> wrote:
>> >> >>> > > right after I have finished building a new linux-next kernel, booting
>> >> >>> > > into desktop and archiving my build-tree (ext4) as tarball to an
>> >> >>> > > external USB harddisk (partition there is ext3).
>> >> >>> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
>> >> >>> > > hard to reproduce [1]. )
>> >> >>> > > I am unsure from where the problem aroses, if you have a hint, let me know.
>> >> >>> > >
>> >> >>> > > Regards,
>> >> >>> > > - Sedat -
>> >> >>> > >
>> >> >>> > > [1] http://lkml.org/lkml/2011/3/24/268
>> >> >>> > >
>> >> >>> > > P.S.: Attached are the dmesg outputs and my kernel-config
>> >> >>> > >
>> >> >>> >
>> >> >>> > I turned off the notebook for about 2hrs to avoid thermal problems and
>> >> >>> > hoax reports.
>> >> >>> > Jumped into desktop and started an archive job as 1st job while doing daily job.
>> >> >>> > Yeah, it is reproducible.
>> >> >>> [...]
>> >> >>> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
>> >> >>> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
>> >> >>>
>> >> >>> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
>> >> >>>
>> >> >>> > [ 273.224059] sending NMI to all CPUs:
>> >> >>> > [ 273.224074] NMI backtrace for cpu 0
>> >> >>> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
>> >> >>> > [ 273.224367]
>> >> >>> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
>> >> >>> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
>> >> >>> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
>> >> >>> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
>> >> >>> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
>> >> >>> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
>> >> >>> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
>> >> >>> > [ 273.224466] Stack:
>> >> >>> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
>> >> >>> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
>> >> >>> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
>> >> >>> > [ 273.224544] Call Trace:
>> >> >>> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
>> >> >>> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
>> >> >>> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> >> >>> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> >> >>> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
>> >> >>> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> >> >>> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
>> >> >>> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> >> >>> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
>> >> >>> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> >> >>> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> >> >>> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> >> >>> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> >> >>> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
>> >> >>> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> >> >>> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
>> >> >>> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> >> >>> > [ 273.224845] <IRQ>
>> >> >>> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
>> >> >>> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> >> >>> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> >> >>> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> >> >>> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> >> >>> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> >> >>> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> >> >>> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
>> >> >>> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
>> >> >>> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>> >> >>> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
>> >> >>> > [ 273.225154] Call Trace:
>> >> >>> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
>> >> >>> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
>> >> >>> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
>> >> >>> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
>> >> >>> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
>> >> >>> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
>> >> >>> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
>> >> >>> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
>> >> >>> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
>> >> >>> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
>> >> >>> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
>> >> >>> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
>> >> >>> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
>> >> >>> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
>> >> >>> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
>> >> >>> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
>> >> >>> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
>> >> >>> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
>> >> >>> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
>> >> >>> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
>> >> >>> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
>> >> >>> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
>> >> >>> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
>> >> >>> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
>> >> >>> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
>> >> >>> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
>> >> >>> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
>> >> >>> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
>> >> >>>
>> >> >>> Interesting. Looks like RCU detected a stall while the CPU sits in
>> >> >>> cpu_idle. That *shouldn't* happen...
>> >> >>
>> >> >> There have been a few of these things recently that turned out to
>> >> >> be BIOS misconfigurations, though that would not be the first thing
>> >> >> I would suspect if the system had run other versions successfully.
>> >> >> Another possibility is that the CPU spent the full time in interrupt.
>> >> >> Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
>> >> >> get an RCU CPU stall warning.
>> >> >>
>> >> >> Or I could have somehow inserted a bug in RCU. But I am not seeing
>> >> >> this in my testing.
>> >> >>
>> >> >> Thanx, Paul
>> >> >>
>> >> >
>> >> > The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> >> > in my configs.
>> >> >
>> >> > This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
>> >> > IIRC I have flashed the latest BIOS available for this notebook.
>> >> >
>> >> > [ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
>> >> > [ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
>> >> >
>> >> > As I am still sitting in the dark, it would be very helpful to know if
>> >> > I can play with HZ or RCU kernel-config parameters.
>> >> > Can I change RCU behaviour from user-space?
>> >> >
>> >> > - Sedat -
>> >> >
>> >> > P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
>> >> > possible values in lib/Kconfig.debug
>> >> >
>> >>
>> >> OK, I had a deeper look at the RCU (STALL) kernel-configs.
>> >>
>> >> $ grep RCU /boot/config-2.6.38-next20110323-3-686-iniza | grep STALL
>> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>> >>
>> >> $ grep RCU /boot/config-2.6.38-next20110324-2-686-iniza | grep STALL
>> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>> >>
>> >> $ grep RCU /boot/config-2.6.38-next20110325-2-686-iniza | grep STALL
>> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> >
>> > Yep, you moved from a kernel version that had the stall detected disabled
>> > by default to one that enables it by default.
>> >
>> > But -next has had stall detection enabled by default for a good
>> > long time now.
>> >
>> >> With my todays (next-20110325) linux-next kernel I cannot work!
>> >> The yesterday call-traces could be indeed a different issue (I am
>> >> currently testing with the 2 patches from block-tree [1]).
>> >>
>> >> Now, I am building a new linux-next kernel with CONFIG_TREE_RCU=y as
>> >> recommended in Documentation/RCU/stallwarn.txt file.
>> >
>> > You had CONFIG_TREE_PREEMPT_RCU=y earlier? Tiny RCU does not have
>> > a stall detector.
>> >
>> > Thanx, Paul
>> >
>> >> - Sedat -
>> >>
>> >> [1] http://lkml.org/lkml/2011/3/25/326
>> >
>>
>> No, and I have here SMP configured.
>> Yesterday's RCU and SMP kernel config settings:
>>
>> # egrep '_RCU|RCU_|_SMP' /boot/config-2.6.38-next20110324-2-686-iniza
>> CONFIG_X86_32_SMP=y
>> CONFIG_TREE_RCU=y
>> # CONFIG_PREEMPT_RCU is not set
>> # CONFIG_RCU_TRACE is not set
>> CONFIG_RCU_FANOUT=32
>> # CONFIG_RCU_FANOUT_EXACT is not set
>> CONFIG_RCU_FAST_NO_HZ=y
>> # CONFIG_TREE_RCU_TRACE is not set
>> CONFIG_USE_GENERIC_SMP_HELPERS=y
>> CONFIG_SMP=y
>> CONFIG_PM_SLEEP_SMP=y
>> CONFIG_HAVE_TEXT_POKE_SMP=y
>> CONFIG_SCSI_SAS_HOST_SMP=y
>> # CONFIG_SPARSE_RCU_POINTER is not set
>> # CONFIG_RCU_TORTURE_TEST is not set
>> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>>
>> IIRC Tiny RCU and SMP bite each other?
>> So, what do you recommend for an UP processor machine?
>
> If you want RCU stall warnings, or if you are building an SMP kernel, it
> has to be either TREE_RCU or TREE_PREEMPT_RCU. If you are on UP and don't
> care about RCU stall warnings, then either TINY_RCU or TINY_PREEMPT_RCU
> will work fine.
>
> I just saw your "Now, I am building a new linux-next kernel with
> CONFIG_TREE_RCU=y" and thought that you were hinting that you had
> been running with something other than TREE_RCU.
>
>> - Sedat -
>>
Just FYI: Changed to the following settings:
- Enable Preemptible Kernel (Low-Latency Desktop)
- Enable Preemptible tree-based hierarchical RCU
- Enable RCU priority boosting
- Reset RCU CPU stall timeout to default (60 seconds)
So far I see no RCU stalls in the logs and my system runs as expected.
( I have noticed here some "stalling" in the webbrowser, but I can do
my daily business. )
I am not sure what the change to PREEMPT exactly mean in the end.
( Let's work with this new kernel and carefully check for possible
side-effects. )
For example CONFIG_RCU_FAST_NO_HZ=y is now dropped, where the Kconfig
descriptive text says some words on better energy saving. For a
notebook this is no good.
I have also questions to some Kconfig dependencies, for example why I
can't select TREE_PREEMPT_RCU if CONFIG_PREEMPT_VOLUNTARY=y, etc.
Intended?
Maybe I collect all my askings in a separate email to RCU folks and ML
and do not disturb further people from other sub-trees.
I enjoyed to read the numerous docs in Documentation/RCU/ (and noticed
some typos as well).
The RCU folk gave the word "FAQ" a new meaning: Frequenty Asked
Questions & Q*uiz* :-).
Thanks for the helpful hints and explanations from the RCU folks!
- Sedat -
P.S.: Current RCU and HZ kernel-config settings
# grep RCU /boot/config-$(uname -r)
# RCU Subsystem
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_TRACE=y
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_TREE_RCU_TRACE=y
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=1
CONFIG_RCU_BOOST_DELAY=500
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_VERBOSE=y
# grep _HZ /boot/config-$(uname -r)
CONFIG_NO_HZ=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
- Sedat -
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-26 8:11 ` Sedat Dilek
@ 2011-03-26 15:53 ` Paul E. McKenney
0 siblings, 0 replies; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-26 15:53 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sat, Mar 26, 2011 at 09:11:29AM +0100, Sedat Dilek wrote:
> On Sat, Mar 26, 2011 at 4:42 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Fri, Mar 25, 2011 at 08:42:14PM +0100, Sedat Dilek wrote:
> >> On Fri, Mar 25, 2011 at 6:48 PM, Paul E. McKenney
> >> <paulmck@linux.vnet.ibm.com> wrote:
> >> > On Fri, Mar 25, 2011 at 06:40:38PM +0100, Sedat Dilek wrote:
> >> >> On Fri, Mar 25, 2011 at 5:51 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
> >> >> > On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
> >> >> > <paulmck@linux.vnet.ibm.com> wrote:
> >> >> >> On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
> >> >> >>> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
> >> >> >>> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
> >> >> >>> > <sedat.dilek@googlemail.com> wrote:
> >> >> >>> > > right after I have finished building a new linux-next kernel, booting
> >> >> >>> > > into desktop and archiving my build-tree (ext4) as tarball to an
> >> >> >>> > > external USB harddisk (partition there is ext3).
> >> >> >>> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
> >> >> >>> > > hard to reproduce [1]. )
> >> >> >>> > > I am unsure from where the problem aroses, if you have a hint, let me know.
> >> >> >>> > >
> >> >> >>> > > Regards,
> >> >> >>> > > - Sedat -
> >> >> >>> > >
> >> >> >>> > > [1] http://lkml.org/lkml/2011/3/24/268
> >> >> >>> > >
> >> >> >>> > > P.S.: Attached are the dmesg outputs and my kernel-config
> >> >> >>> > >
> >> >> >>> >
> >> >> >>> > I turned off the notebook for about 2hrs to avoid thermal problems and
> >> >> >>> > hoax reports.
> >> >> >>> > Jumped into desktop and started an archive job as 1st job while doing daily job.
> >> >> >>> > Yeah, it is reproducible.
> >> >> >>> [...]
> >> >> >>> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
> >> >> >>> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
> >> >> >>>
> >> >> >>> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
> >> >> >>>
> >> >> >>> > [ 273.224059] sending NMI to all CPUs:
> >> >> >>> > [ 273.224074] NMI backtrace for cpu 0
> >> >> >>> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
> >> >> >>> > [ 273.224367]
> >> >> >>> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
> >> >> >>> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
> >> >> >>> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
> >> >> >>> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
> >> >> >>> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
> >> >> >>> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> >> >> >>> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
> >> >> >>> > [ 273.224466] Stack:
> >> >> >>> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
> >> >> >>> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
> >> >> >>> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
> >> >> >>> > [ 273.224544] Call Trace:
> >> >> >>> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
> >> >> >>> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
> >> >> >>> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >> >> >>> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >> >> >>> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
> >> >> >>> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >> >> >>> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
> >> >> >>> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >> >> >>> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
> >> >> >>> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >> >> >>> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >> >> >>> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >> >> >>> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >> >> >>> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
> >> >> >>> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >> >> >>> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
> >> >> >>> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >> >> >>> > [ 273.224845] <IRQ>
> >> >> >>> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
> >> >> >>> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >> >> >>> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >> >> >>> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >> >> >>> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >> >> >>> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >> >> >>> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >> >> >>> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
> >> >> >>> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
> >> >> >>> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >> >> >>> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
> >> >> >>> > [ 273.225154] Call Trace:
> >> >> >>> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
> >> >> >>> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
> >> >> >>> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >> >> >>> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >> >> >>> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
> >> >> >>> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >> >> >>> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
> >> >> >>> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >> >> >>> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
> >> >> >>> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >> >> >>> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >> >> >>> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >> >> >>> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >> >> >>> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
> >> >> >>> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >> >> >>> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
> >> >> >>> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >> >> >>> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
> >> >> >>> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >> >> >>> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >> >> >>> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >> >> >>> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >> >> >>> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >> >> >>> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >> >> >>> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
> >> >> >>> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
> >> >> >>> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >> >> >>>
> >> >> >>> Interesting. Looks like RCU detected a stall while the CPU sits in
> >> >> >>> cpu_idle. That *shouldn't* happen...
> >> >> >>
> >> >> >> There have been a few of these things recently that turned out to
> >> >> >> be BIOS misconfigurations, though that would not be the first thing
> >> >> >> I would suspect if the system had run other versions successfully.
> >> >> >> Another possibility is that the CPU spent the full time in interrupt.
> >> >> >> Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
> >> >> >> get an RCU CPU stall warning.
> >> >> >>
> >> >> >> Or I could have somehow inserted a bug in RCU. But I am not seeing
> >> >> >> this in my testing.
> >> >> >>
> >> >> >> Thanx, Paul
> >> >> >>
> >> >> >
> >> >> > The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
> >> >> > in my configs.
> >> >> >
> >> >> > This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
> >> >> > IIRC I have flashed the latest BIOS available for this notebook.
> >> >> >
> >> >> > [ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
> >> >> > [ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
> >> >> >
> >> >> > As I am still sitting in the dark, it would be very helpful to know if
> >> >> > I can play with HZ or RCU kernel-config parameters.
> >> >> > Can I change RCU behaviour from user-space?
> >> >> >
> >> >> > - Sedat -
> >> >> >
> >> >> > P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
> >> >> > possible values in lib/Kconfig.debug
> >> >> >
> >> >>
> >> >> OK, I had a deeper look at the RCU (STALL) kernel-configs.
> >> >>
> >> >> $ grep RCU /boot/config-2.6.38-next20110323-3-686-iniza | grep STALL
> >> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >> >>
> >> >> $ grep RCU /boot/config-2.6.38-next20110324-2-686-iniza | grep STALL
> >> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >> >>
> >> >> $ grep RCU /boot/config-2.6.38-next20110325-2-686-iniza | grep STALL
> >> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
> >> >
> >> > Yep, you moved from a kernel version that had the stall detected disabled
> >> > by default to one that enables it by default.
> >> >
> >> > But -next has had stall detection enabled by default for a good
> >> > long time now.
> >> >
> >> >> With my todays (next-20110325) linux-next kernel I cannot work!
> >> >> The yesterday call-traces could be indeed a different issue (I am
> >> >> currently testing with the 2 patches from block-tree [1]).
> >> >>
> >> >> Now, I am building a new linux-next kernel with CONFIG_TREE_RCU=y as
> >> >> recommended in Documentation/RCU/stallwarn.txt file.
> >> >
> >> > You had CONFIG_TREE_PREEMPT_RCU=y earlier? Tiny RCU does not have
> >> > a stall detector.
> >> >
> >> > Thanx, Paul
> >> >
> >> >> - Sedat -
> >> >>
> >> >> [1] http://lkml.org/lkml/2011/3/25/326
> >> >
> >>
> >> No, and I have here SMP configured.
> >> Yesterday's RCU and SMP kernel config settings:
> >>
> >> # egrep '_RCU|RCU_|_SMP' /boot/config-2.6.38-next20110324-2-686-iniza
> >> CONFIG_X86_32_SMP=y
> >> CONFIG_TREE_RCU=y
> >> # CONFIG_PREEMPT_RCU is not set
> >> # CONFIG_RCU_TRACE is not set
> >> CONFIG_RCU_FANOUT=32
> >> # CONFIG_RCU_FANOUT_EXACT is not set
> >> CONFIG_RCU_FAST_NO_HZ=y
> >> # CONFIG_TREE_RCU_TRACE is not set
> >> CONFIG_USE_GENERIC_SMP_HELPERS=y
> >> CONFIG_SMP=y
> >> CONFIG_PM_SLEEP_SMP=y
> >> CONFIG_HAVE_TEXT_POKE_SMP=y
> >> CONFIG_SCSI_SAS_HOST_SMP=y
> >> # CONFIG_SPARSE_RCU_POINTER is not set
> >> # CONFIG_RCU_TORTURE_TEST is not set
> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >>
> >> IIRC Tiny RCU and SMP bite each other?
> >> So, what do you recommend for an UP processor machine?
> >
> > If you want RCU stall warnings, or if you are building an SMP kernel, it
> > has to be either TREE_RCU or TREE_PREEMPT_RCU. If you are on UP and don't
> > care about RCU stall warnings, then either TINY_RCU or TINY_PREEMPT_RCU
> > will work fine.
> >
> > I just saw your "Now, I am building a new linux-next kernel with
> > CONFIG_TREE_RCU=y" and thought that you were hinting that you had
> > been running with something other than TREE_RCU.
> >
>
> I could suppress RCU STALL warning via sysfs, but the complete system
> is just freaky.
> With doing a simple tar (without compression!) job, I can't use my webbrowser.
Then it does sound like the RCU stall warnings really are locating a
legitimate problem, despite the strange stack trace.
> Selecting TINY_RCU means no CONFIG_SMP!
> (OK, I can hack the Kconfig dependency to change this.)
Yes, TINY_RCU is designed to be UP-only. Please do not try it in an
SMP kernel. It will break badly.
> I am thinking of distributions, shall they offer another linux-kernel
> binary especially for UP machines?
> Somehow ridiculuous.
No, they should continue to do what they are currently doing. TINY_RCU
is primarily for embedded systems.
> I still haven't get some hints to play with the timeouts, so I tried
> CONFIG_RCU_CPU_STALL_TIMEOUT=120 (default according to Kconfig entry
> is 60).
> Can this really help here. Why?
> I had a closer look into my logs, there I saw a max jiffies of
> t=780510 (six digits!)
Yow! That is a very long stall.
> /var/log/syslog.1:Mar 25 18:09:15 tbox kernel: [ 3221.112034] INFO:
> rcu_sched_state detected stall on CPU 0 (t=780510 jiffies)
> And I asked for the common play between HZ and RCU values in
> kernel-config parameters.
> Currently, I have CONFIG_HZ_250=y.
> Can you enlighten me. please.
CONFIG_HZ_250 is entirely reasonable.
> I have the impression that even I could disable "old"
> CONFIG_RCU_CPU_STALL_DETECTOR value, this stalling here has other
> reasons.
The stall detector used to be disabled by default.
You were running something that worked -- what exactly was that?
> BROKEN ACPI? Might be... But what about supporting such systems (with
> quirks or whatever)?
Or a messed up commit.
Thanx, Paul
> - Sedat -
>
> P.S.: I have attached all jiffies values from my today's and
> yesterday's syslogs and my last used kernel-config.
>
> >> - Sedat -
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >> Please read the FAQ at http://www.tux.org/lkml/
> >
> /var/log/syslog:Mar 26 00:36:55 tbox kernel: [ 589.064031] INFO: rcu_sched_state detected stall on CPU 0 (t=120030 jiffies)
> /var/log/syslog:Mar 26 00:42:55 tbox kernel: [ 949.184032] INFO: rcu_sched_state detected stall on CPU 0 (t=210060 jiffies)
> /var/log/syslog:Mar 26 00:48:55 tbox kernel: [ 1309.304033] INFO: rcu_sched_state detected stall on CPU 0 (t=300090 jiffies)
> /var/log/syslog:Mar 26 01:11:09 tbox kernel: [ 208.416034] INFO: rcu_sched_state detected stall on CPU 0 (t=30000 jiffies)
> /var/log/syslog:Mar 26 01:16:30 tbox kernel: [ 154.532013] INFO: rcu_sched_state detected stall on CPU 0 (t=30000 jiffies)
> /var/log/syslog.1:Mar 25 10:57:04 tbox kernel: [ 157.340021] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
> /var/log/syslog.1:Mar 25 11:00:04 tbox kernel: [ 337.460033] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
> /var/log/syslog.1:Mar 25 11:03:05 tbox kernel: [ 517.580032] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
> /var/log/syslog.1:Mar 25 11:06:05 tbox kernel: [ 697.700033] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
> /var/log/syslog.1:Mar 25 11:09:05 tbox kernel: [ 877.820034] INFO: rcu_sched_state detected stall on CPU 0 (t=195120 jiffies)
> /var/log/syslog.1:Mar 25 11:12:05 tbox kernel: [ 1057.940032] INFO: rcu_sched_state detected stall on CPU 0 (t=240150 jiffies)
> /var/log/syslog.1:Mar 25 11:15:05 tbox kernel: [ 1238.060029] INFO: rcu_sched_state detected stall on CPU 0 (t=285180 jiffies)
> /var/log/syslog.1:Mar 25 11:18:05 tbox kernel: [ 1418.180033] INFO: rcu_sched_state detected stall on CPU 0 (t=330210 jiffies)
> /var/log/syslog.1:Mar 25 11:21:05 tbox kernel: [ 1598.300034] INFO: rcu_sched_state detected stall on CPU 0 (t=375240 jiffies)
> /var/log/syslog.1:Mar 25 11:24:05 tbox kernel: [ 1778.420018] INFO: rcu_sched_state detected stall on CPU 0 (t=420270 jiffies)
> /var/log/syslog.1:Mar 25 11:27:06 tbox kernel: [ 1958.540024] INFO: rcu_sched_state detected stall on CPU 0 (t=465300 jiffies)
> /var/log/syslog.1:Mar 25 11:30:06 tbox kernel: [ 2138.660008] INFO: rcu_sched_state detected stall on CPU 0 (t=510330 jiffies)
> /var/log/syslog.1:Mar 25 11:33:06 tbox kernel: [ 2318.780033] INFO: rcu_sched_state detected stall on CPU 0 (t=555360 jiffies)
> /var/log/syslog.1:Mar 25 13:54:55 tbox kernel: [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
> /var/log/syslog.1:Mar 25 13:57:56 tbox kernel: [ 453.344036] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
> /var/log/syslog.1:Mar 25 14:00:56 tbox kernel: [ 633.464043] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
> /var/log/syslog.1:Mar 25 14:03:56 tbox kernel: [ 813.584031] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
> /var/log/syslog.1:Mar 25 14:06:56 tbox kernel: [ 993.704031] INFO: rcu_sched_state detected stall on CPU 0 (t=195120 jiffies)
> /var/log/syslog.1:Mar 25 14:09:56 tbox kernel: [ 1173.824042] INFO: rcu_sched_state detected stall on CPU 0 (t=240150 jiffies)
> /var/log/syslog.1:Mar 25 14:12:56 tbox kernel: [ 1353.944020] INFO: rcu_sched_state detected stall on CPU 0 (t=285180 jiffies)
> /var/log/syslog.1:Mar 25 14:15:56 tbox kernel: [ 1534.064033] INFO: rcu_sched_state detected stall on CPU 0 (t=330210 jiffies)
> /var/log/syslog.1:Mar 25 14:18:56 tbox kernel: [ 1714.184033] INFO: rcu_sched_state detected stall on CPU 0 (t=375240 jiffies)
> /var/log/syslog.1:Mar 25 14:21:56 tbox kernel: [ 1894.304038] INFO: rcu_sched_state detected stall on CPU 0 (t=420270 jiffies)
> /var/log/syslog.1:Mar 25 14:24:57 tbox kernel: [ 2074.424033] INFO: rcu_sched_state detected stall on CPU 0 (t=465300 jiffies)
> /var/log/syslog.1:Mar 25 17:18:13 tbox kernel: [ 159.072032] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
> /var/log/syslog.1:Mar 25 17:21:13 tbox kernel: [ 339.192034] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
> /var/log/syslog.1:Mar 25 17:24:13 tbox kernel: [ 519.312032] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
> /var/log/syslog.1:Mar 25 17:27:13 tbox kernel: [ 699.432033] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
> /var/log/syslog.1:Mar 25 17:30:14 tbox kernel: [ 879.552031] INFO: rcu_sched_state detected stall on CPU 0 (t=195120 jiffies)
> /var/log/syslog.1:Mar 25 17:33:14 tbox kernel: [ 1059.672035] INFO: rcu_sched_state detected stall on CPU 0 (t=240150 jiffies)
> /var/log/syslog.1:Mar 25 17:36:14 tbox kernel: [ 1239.792026] INFO: rcu_sched_state detected stall on CPU 0 (t=285180 jiffies)
> /var/log/syslog.1:Mar 25 17:39:14 tbox kernel: [ 1419.912036] INFO: rcu_sched_state detected stall on CPU 0 (t=330210 jiffies)
> /var/log/syslog.1:Mar 25 17:42:14 tbox kernel: [ 1600.032035] INFO: rcu_sched_state detected stall on CPU 0 (t=375240 jiffies)
> /var/log/syslog.1:Mar 25 17:45:14 tbox kernel: [ 1780.152033] INFO: rcu_sched_state detected stall on CPU 0 (t=420270 jiffies)
> /var/log/syslog.1:Mar 25 17:48:14 tbox kernel: [ 1960.272034] INFO: rcu_sched_state detected stall on CPU 0 (t=465300 jiffies)
> /var/log/syslog.1:Mar 25 17:51:14 tbox kernel: [ 2140.392012] INFO: rcu_sched_state detected stall on CPU 0 (t=510330 jiffies)
> /var/log/syslog.1:Mar 25 17:54:15 tbox kernel: [ 2320.512031] INFO: rcu_sched_state detected stall on CPU 0 (t=555360 jiffies)
> /var/log/syslog.1:Mar 25 17:57:15 tbox kernel: [ 2500.632036] INFO: rcu_sched_state detected stall on CPU 0 (t=600390 jiffies)
> /var/log/syslog.1:Mar 25 18:00:15 tbox kernel: [ 2680.751704] INFO: rcu_sched_state detected stall on CPU 0 (t=645420 jiffies)
> /var/log/syslog.1:Mar 25 18:03:15 tbox kernel: [ 2860.872032] INFO: rcu_sched_state detected stall on CPU 0 (t=690450 jiffies)
> /var/log/syslog.1:Mar 25 18:06:15 tbox kernel: [ 3040.992034] INFO: rcu_sched_state detected stall on CPU 0 (t=735480 jiffies)
> /var/log/syslog.1:Mar 25 18:09:15 tbox kernel: [ 3221.112034] INFO: rcu_sched_state detected stall on CPU 0 (t=780510 jiffies)
> /var/log/syslog.1:Mar 25 20:37:17 tbox kernel: [ 162.424039] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
> /var/log/syslog.1:Mar 25 20:40:17 tbox kernel: [ 342.544034] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
> /var/log/syslog.1:Mar 25 20:43:17 tbox kernel: [ 522.664036] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
> /var/log/syslog.1:Mar 25 20:46:17 tbox kernel: [ 702.784032] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
> /var/log/syslog.1:Mar 25 20:49:17 tbox kernel: [ 882.904036] INFO: rcu_sched_state detected stall on CPU 0 (t=195120 jiffies)
> /var/log/syslog.1:Mar 25 20:52:17 tbox kernel: [ 1063.024052] INFO: rcu_sched_state detected stall on CPU 0 (t=240150 jiffies)
> /var/log/syslog.1:Mar 25 20:55:17 tbox kernel: [ 1243.144034] INFO: rcu_sched_state detected stall on CPU 0 (t=285180 jiffies)
> /var/log/syslog.1:Mar 25 20:58:17 tbox kernel: [ 1423.264034] INFO: rcu_sched_state detected stall on CPU 0 (t=330210 jiffies)
> /var/log/syslog.1:Mar 25 21:01:18 tbox kernel: [ 1603.384032] INFO: rcu_sched_state detected stall on CPU 0 (t=375240 jiffies)
> /var/log/syslog.1:Mar 25 21:04:18 tbox kernel: [ 1783.504018] INFO: rcu_sched_state detected stall on CPU 0 (t=420270 jiffies)
> /var/log/syslog.1:Mar 25 21:07:18 tbox kernel: [ 1963.624034] INFO: rcu_sched_state detected stall on CPU 0 (t=465300 jiffies)
> /var/log/syslog.1:Mar 25 21:10:19 tbox kernel: [ 2143.744028] INFO: rcu_sched_state detected stall on CPU 0 (t=510330 jiffies)
> /var/log/syslog.1:Mar 26 00:30:55 tbox kernel: [ 228.944029] INFO: rcu_sched_state detected stall on CPU 0 (t=30000 jiffies)
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-26 12:34 ` Sedat Dilek
@ 2011-03-26 16:02 ` Paul E. McKenney
2011-03-26 22:15 ` Sedat Dilek
0 siblings, 1 reply; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-26 16:02 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sat, Mar 26, 2011 at 01:34:53PM +0100, Sedat Dilek wrote:
> On Sat, Mar 26, 2011 at 4:42 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Fri, Mar 25, 2011 at 08:42:14PM +0100, Sedat Dilek wrote:
> >> On Fri, Mar 25, 2011 at 6:48 PM, Paul E. McKenney
> >> <paulmck@linux.vnet.ibm.com> wrote:
> >> > On Fri, Mar 25, 2011 at 06:40:38PM +0100, Sedat Dilek wrote:
> >> >> On Fri, Mar 25, 2011 at 5:51 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
> >> >> > On Fri, Mar 25, 2011 at 5:42 PM, Paul E. McKenney
> >> >> > <paulmck@linux.vnet.ibm.com> wrote:
> >> >> >> On Fri, Mar 25, 2011 at 08:55:16AM -0700, Josh Triplett wrote:
> >> >> >>> On Fri, Mar 25, 2011 at 02:05:33PM +0100, Sedat Dilek wrote:
> >> >> >>> > On Fri, Mar 25, 2011 at 11:16 AM, Sedat Dilek
> >> >> >>> > <sedat.dilek@googlemail.com> wrote:
> >> >> >>> > > right after I have finished building a new linux-next kernel, booting
> >> >> >>> > > into desktop and archiving my build-tree (ext4) as tarball to an
> >> >> >>> > > external USB harddisk (partition there is ext3).
> >> >> >>> > > ( Yesterday, I have seen similiar call-traces in my logs, but it was
> >> >> >>> > > hard to reproduce [1]. )
> >> >> >>> > > I am unsure from where the problem aroses, if you have a hint, let me know.
> >> >> >>> > >
> >> >> >>> > > Regards,
> >> >> >>> > > - Sedat -
> >> >> >>> > >
> >> >> >>> > > [1] http://lkml.org/lkml/2011/3/24/268
> >> >> >>> > >
> >> >> >>> > > P.S.: Attached are the dmesg outputs and my kernel-config
> >> >> >>> > >
> >> >> >>> >
> >> >> >>> > I turned off the notebook for about 2hrs to avoid thermal problems and
> >> >> >>> > hoax reports.
> >> >> >>> > Jumped into desktop and started an archive job as 1st job while doing daily job.
> >> >> >>> > Yeah, it is reproducible.
> >> >> >>> [...]
> >> >> >>> > [ 212.453822] EXT3-fs (sdb5): mounted filesystem with ordered data mode
> >> >> >>> > [ 273.224044] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
> >> >> >>>
> >> >> >>> 15000 jiffies matches this 60-second gap, assuming you use HZ=250.
> >> >> >>>
> >> >> >>> > [ 273.224059] sending NMI to all CPUs:
> >> >> >>> > [ 273.224074] NMI backtrace for cpu 0
> >> >> >>> > [ 273.224081] Modules linked in: ext3 jbd bnep rfcomm bluetooth aes_i586 aes_generic binfmt_misc ppdev acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace lp cpufreq_stats cpufreq_conservative fuse snd_intel8x0 snd_intel8x0m snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm radeon thinkpad_acpi snd_seq_midi pcmcia ttm snd_rawmidi snd_seq_midi_event drm_kms_helper yenta_socket snd_seq pcmcia_rsrc drm pcmcia_core joydev snd_timer snd_seq_device snd i2c_algo_bit tpm_tis shpchp i2c_i801 tpm nsc_ircc irda snd_page_alloc soundcore pci_hotplug rng_core i2c_core tpm_bios psmouse crc_ccitt nvram parport_pc pcspkr parport evdev battery video ac processor power_supply serio_raw button arc4 ecb ath5k ath mac80211 cfg80211 rfkill autofs4 ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sd_mod sr_mod crc_t10dif cdrom ata_generic ata_piix libata uhci_hcd ehci_hcd usbcore scsi_mod thermal e1000 thermal_sys floppy [last unloaded: scsi_wait_scan]
> >> >> >>> > [ 273.224367]
> >> >> >>> > [ 273.224377] Pid: 0, comm: swapper Not tainted 2.6.38-next20110325-2-686-iniza #1 IBM 2374SG6/2374SG6
> >> >> >>> > [ 273.224397] EIP: 0060:[<c11514f0>] EFLAGS: 00000807 CPU: 0
> >> >> >>> > [ 273.224414] EIP is at delay_tsc+0x16/0x5e
> >> >> >>> > [ 273.224424] EAX: 00090d42 EBX: 00002710 ECX: c133faf5 EDX: 00090d41
> >> >> >>> > [ 273.224435] ESI: 00000000 EDI: 00090d42 EBP: f5819e9c ESP: f5819e8c
> >> >> >>> > [ 273.224445] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> >> >> >>> > [ 273.224458] Process swapper (pid: 0, ti=f5818000 task=c13e3fa0 task.ti=c13b6000)
> >> >> >>> > [ 273.224466] Stack:
> >> >> >>> > [ 273.224472] 00090d41 00002710 c13ee580 c13ee600 f5819ea4 c115149f f5819eac c11514bb
> >> >> >>> > [ 273.224497] f5819eb8 c1016532 c13ee580 f5819ed4 c1078dc1 c134e61e c134e6c2 00000000
> >> >> >>> > [ 273.224520] 00003a98 f5c03488 f5819ee8 c1078e36 00000000 00000000 c13e3fa0 f5819ef4
> >> >> >>> > [ 273.224544] Call Trace:
> >> >> >>> > [ 273.224559] [<c115149f>] __delay+0x9/0xb
> >> >> >>> > [ 273.224571] [<c11514bb>] __const_udelay+0x1a/0x1c
> >> >> >>> > [ 273.224590] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >> >> >>> > [ 273.224608] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >> >> >>> > [ 273.224622] [<c1078e36>] __rcu_pending+0x15/0xc4
> >> >> >>> > [ 273.224637] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >> >> >>> > [ 273.224652] [<c1039c6c>] update_process_times+0x2d/0x58
> >> >> >>> > [ 273.224666] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >> >> >>> > [ 273.224682] [<c1047196>] __run_hrtimer+0x9c/0x111
> >> >> >>> > [ 273.224694] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >> >> >>> > [ 273.224708] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >> >> >>> > [ 273.224727] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >> >> >>> > [ 273.224741] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >> >> >>> > [ 273.224757] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.224772] [<c1004215>] timer_interrupt+0x15/0x1c
> >> >> >>> > [ 273.224785] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >> >> >>> > [ 273.224799] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.224811] [<c10754b9>] handle_irq_event+0x36/0x51
> >> >> >>> > [ 273.224824] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.224837] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >> >> >>> > [ 273.224845] <IRQ>
> >> >> >>> > [ 273.224857] [<c1003b8d>] ? do_IRQ+0x35/0x80
> >> >> >>> > [ 273.224871] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >> >> >>> > [ 273.224886] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >> >> >>> > [ 273.224922] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >> >> >>> > [ 273.224947] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >> >> >>> > [ 273.224964] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >> >> >>> > [ 273.224978] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >> >> >>> > [ 273.224995] [<c128c26c>] ? rest_init+0x58/0x5a
> >> >> >>> > [ 273.225008] [<c1418722>] ? start_kernel+0x315/0x31a
> >> >> >>> > [ 273.225022] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >> >> >>> > [ 273.225029] Code: e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3 55 89 e5 57 89 c7 56 53 52 64 8b 35 04 20 47 c1 8d 76 00 0f ae e8 <e8> 6b ff ff ff 89 c3 8d 76 00 0f ae e8 e8 5e ff ff ff 89 c2 29
> >> >> >>> > [ 273.225154] Call Trace:
> >> >> >>> > [ 273.225166] [<c115149f>] __delay+0x9/0xb
> >> >> >>> > [ 273.225178] [<c11514bb>] __const_udelay+0x1a/0x1c
> >> >> >>> > [ 273.225192] [<c1016532>] arch_trigger_all_cpu_backtrace+0x50/0x62
> >> >> >>> > [ 273.225207] [<c1078dc1>] check_cpu_stall+0x58/0xb8
> >> >> >>> > [ 273.225220] [<c1078e36>] __rcu_pending+0x15/0xc4
> >> >> >>> > [ 273.225234] [<c10791df>] rcu_check_callbacks+0x6d/0x93
> >> >> >>> > [ 273.225247] [<c1039c6c>] update_process_times+0x2d/0x58
> >> >> >>> > [ 273.225260] [<c10509e9>] tick_sched_timer+0x6b/0x9a
> >> >> >>> > [ 273.225274] [<c1047196>] __run_hrtimer+0x9c/0x111
> >> >> >>> > [ 273.225286] [<c105097e>] ? tick_sched_timer+0x0/0x9a
> >> >> >>> > [ 273.225300] [<c1047b38>] hrtimer_interrupt+0xd6/0x1bb
> >> >> >>> > [ 273.225316] [<c104fca1>] tick_do_broadcast.constprop.4+0x38/0x6a
> >> >> >>> > [ 273.225330] [<c104fd80>] tick_handle_oneshot_broadcast+0xad/0xe1
> >> >> >>> > [ 273.225345] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.225358] [<c1004215>] timer_interrupt+0x15/0x1c
> >> >> >>> > [ 273.225370] [<c107536d>] handle_irq_event_percpu+0x4e/0x164
> >> >> >>> > [ 273.225384] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.225396] [<c10754b9>] handle_irq_event+0x36/0x51
> >> >> >>> > [ 273.225409] [<c1076cc2>] ? handle_level_irq+0x0/0x63
> >> >> >>> > [ 273.225421] [<c1076d0f>] handle_level_irq+0x4d/0x63
> >> >> >>> > [ 273.225429] <IRQ> [<c1003b8d>] ? do_IRQ+0x35/0x80
> >> >> >>> > [ 273.225450] [<c12ac0f0>] ? common_interrupt+0x30/0x38
> >> >> >>> > [ 273.225464] [<c10400d8>] ? destroy_worker+0x52/0x6c
> >> >> >>> > [ 273.225493] [<f87b730f>] ? arch_local_irq_enable+0x5/0xb [processor]
> >> >> >>> > [ 273.225517] [<f87b7ef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
> >> >> >>> > [ 273.225532] [<c11ebd92>] ? cpuidle_idle_call+0xc2/0x137
> >> >> >>> > [ 273.225545] [<c1001da3>] ? cpu_idle+0x89/0xa3
> >> >> >>> > [ 273.225559] [<c128c26c>] ? rest_init+0x58/0x5a
> >> >> >>> > [ 273.225571] [<c1418722>] ? start_kernel+0x315/0x31a
> >> >> >>> > [ 273.225584] [<c14180a2>] ? i386_start_kernel+0xa2/0xaa
> >> >> >>>
> >> >> >>> Interesting. Looks like RCU detected a stall while the CPU sits in
> >> >> >>> cpu_idle. That *shouldn't* happen...
> >> >> >>
> >> >> >> There have been a few of these things recently that turned out to
> >> >> >> be BIOS misconfigurations, though that would not be the first thing
> >> >> >> I would suspect if the system had run other versions successfully.
> >> >> >> Another possibility is that the CPU spent the full time in interrupt.
> >> >> >> Get an interrupt from the idle loop, stay in interrupt for 60 seconds,
> >> >> >> get an RCU CPU stall warning.
> >> >> >>
> >> >> >> Or I could have somehow inserted a bug in RCU. But I am not seeing
> >> >> >> this in my testing.
> >> >> >>
> >> >> >> Thanx, Paul
> >> >> >>
> >> >> >
> >> >> > The problems started when I first saw CONFIG_RCU_CPU_STALL_TIMEOUT=60
> >> >> > in my configs.
> >> >> >
> >> >> > This an old IBM T40p notebook with Pentium-M (Banias) UP processor.
> >> >> > IIRC I have flashed the latest BIOS available for this notebook.
> >> >> >
> >> >> > [ 11.786073] thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04
> >> >> > [ 11.786111] thinkpad_acpi: IBM ThinkPad T40p, model 2374SG6
> >> >> >
> >> >> > As I am still sitting in the dark, it would be very helpful to know if
> >> >> > I can play with HZ or RCU kernel-config parameters.
> >> >> > Can I change RCU behaviour from user-space?
> >> >> >
> >> >> > - Sedat -
> >> >> >
> >> >> > P.S.: Note to myself: Read Documentation/RCU/stallwarn.txt & check
> >> >> > possible values in lib/Kconfig.debug
> >> >> >
> >> >>
> >> >> OK, I had a deeper look at the RCU (STALL) kernel-configs.
> >> >>
> >> >> $ grep RCU /boot/config-2.6.38-next20110323-3-686-iniza | grep STALL
> >> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >> >>
> >> >> $ grep RCU /boot/config-2.6.38-next20110324-2-686-iniza | grep STALL
> >> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >> >>
> >> >> $ grep RCU /boot/config-2.6.38-next20110325-2-686-iniza | grep STALL
> >> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
> >> >
> >> > Yep, you moved from a kernel version that had the stall detected disabled
> >> > by default to one that enables it by default.
> >> >
> >> > But -next has had stall detection enabled by default for a good
> >> > long time now.
> >> >
> >> >> With my todays (next-20110325) linux-next kernel I cannot work!
> >> >> The yesterday call-traces could be indeed a different issue (I am
> >> >> currently testing with the 2 patches from block-tree [1]).
> >> >>
> >> >> Now, I am building a new linux-next kernel with CONFIG_TREE_RCU=y as
> >> >> recommended in Documentation/RCU/stallwarn.txt file.
> >> >
> >> > You had CONFIG_TREE_PREEMPT_RCU=y earlier? Tiny RCU does not have
> >> > a stall detector.
> >> >
> >> > Thanx, Paul
> >> >
> >> >> - Sedat -
> >> >>
> >> >> [1] http://lkml.org/lkml/2011/3/25/326
> >> >
> >>
> >> No, and I have here SMP configured.
> >> Yesterday's RCU and SMP kernel config settings:
> >>
> >> # egrep '_RCU|RCU_|_SMP' /boot/config-2.6.38-next20110324-2-686-iniza
> >> CONFIG_X86_32_SMP=y
> >> CONFIG_TREE_RCU=y
> >> # CONFIG_PREEMPT_RCU is not set
> >> # CONFIG_RCU_TRACE is not set
> >> CONFIG_RCU_FANOUT=32
> >> # CONFIG_RCU_FANOUT_EXACT is not set
> >> CONFIG_RCU_FAST_NO_HZ=y
> >> # CONFIG_TREE_RCU_TRACE is not set
> >> CONFIG_USE_GENERIC_SMP_HELPERS=y
> >> CONFIG_SMP=y
> >> CONFIG_PM_SLEEP_SMP=y
> >> CONFIG_HAVE_TEXT_POKE_SMP=y
> >> CONFIG_SCSI_SAS_HOST_SMP=y
> >> # CONFIG_SPARSE_RCU_POINTER is not set
> >> # CONFIG_RCU_TORTURE_TEST is not set
> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >>
> >> IIRC Tiny RCU and SMP bite each other?
> >> So, what do you recommend for an UP processor machine?
> >
> > If you want RCU stall warnings, or if you are building an SMP kernel, it
> > has to be either TREE_RCU or TREE_PREEMPT_RCU. If you are on UP and don't
> > care about RCU stall warnings, then either TINY_RCU or TINY_PREEMPT_RCU
> > will work fine.
> >
> > I just saw your "Now, I am building a new linux-next kernel with
> > CONFIG_TREE_RCU=y" and thought that you were hinting that you had
> > been running with something other than TREE_RCU.
> >
> >> - Sedat -
> >>
>
> Just FYI: Changed to the following settings:
>
> - Enable Preemptible Kernel (Low-Latency Desktop)
> - Enable Preemptible tree-based hierarchical RCU
> - Enable RCU priority boosting
> - Reset RCU CPU stall timeout to default (60 seconds)
>
> So far I see no RCU stalls in the logs and my system runs as expected.
> ( I have noticed here some "stalling" in the webbrowser, but I can do
> my daily business. )
OK, good to see some progress!
Is there a runaway process consuming CPU? The reason that I ask is that
an infinite loop in the kernel can result in a stall when PREEMPT=n
but is less likely to if PREEMPT=y. Could you please check with "top",
"ps", or whatever?
> I am not sure what the change to PREEMPT exactly mean in the end.
> ( Let's work with this new kernel and carefully check for possible
> side-effects. )
> For example CONFIG_RCU_FAST_NO_HZ=y is now dropped, where the Kconfig
> descriptive text says some words on better energy saving. For a
> notebook this is no good.
CONFIG_RCU_FAST_NO_HZ is of no use on a uniprocessor system, so OK
to disable it. But are you saying that CONFIG_RCU_FAST_NO_HZ=y
results in problems that are removed by CONFIG_RCU_FAST_NO_HZ=n?
That would be a surprise, and I need to know if this is the case.
> I have also questions to some Kconfig dependencies, for example why I
> can't select TREE_PREEMPT_RCU if CONFIG_PREEMPT_VOLUNTARY=y, etc.
> Intended?
Yes. There is no point in TREE_PREEMPT_RCU unless PREEMPT=y.
> Maybe I collect all my askings in a separate email to RCU folks and ML
> and do not disturb further people from other sub-trees.
>
> I enjoyed to read the numerous docs in Documentation/RCU/ (and noticed
> some typos as well).
> The RCU folk gave the word "FAQ" a new meaning: Frequenty Asked
> Questions & Q*uiz* :-).
>
> Thanks for the helpful hints and explanations from the RCU folks!
Glad you liked them! ;-)
> - Sedat -
>
> P.S.: Current RCU and HZ kernel-config settings
>
> # grep RCU /boot/config-$(uname -r)
> # RCU Subsystem
> CONFIG_TREE_PREEMPT_RCU=y
> CONFIG_PREEMPT_RCU=y
> CONFIG_RCU_TRACE=y
> CONFIG_RCU_FANOUT=32
> # CONFIG_RCU_FANOUT_EXACT is not set
> CONFIG_TREE_RCU_TRACE=y
> CONFIG_RCU_BOOST=y
> CONFIG_RCU_BOOST_PRIO=1
> CONFIG_RCU_BOOST_DELAY=500
> # CONFIG_SPARSE_RCU_POINTER is not set
> # CONFIG_RCU_TORTURE_TEST is not set
> CONFIG_RCU_CPU_STALL_TIMEOUT=60
> CONFIG_RCU_CPU_STALL_VERBOSE=y
>
> # grep _HZ /boot/config-$(uname -r)
> CONFIG_NO_HZ=y
> # CONFIG_HZ_100 is not set
> CONFIG_HZ_250=y
> # CONFIG_HZ_300 is not set
> # CONFIG_HZ_1000 is not set
> CONFIG_HZ=250
OK, thank you for the info!
Thanx, Paul
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-26 16:02 ` Paul E. McKenney
@ 2011-03-26 22:15 ` Sedat Dilek
2011-03-27 0:09 ` Paul E. McKenney
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-26 22:15 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
[-- Attachment #1: Type: text/plain, Size: 4608 bytes --]
On Sat, Mar 26, 2011 at 5:02 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Sat, Mar 26, 2011 at 01:34:53PM +0100, Sedat Dilek wrote:
[...]
>> Just FYI: Changed to the following settings:
>>
>> - Enable Preemptible Kernel (Low-Latency Desktop)
>> - Enable Preemptible tree-based hierarchical RCU
>> - Enable RCU priority boosting
>> - Reset RCU CPU stall timeout to default (60 seconds)
>>
>> So far I see no RCU stalls in the logs and my system runs as expected.
>> ( I have noticed here some "stalling" in the webbrowser, but I can do
>> my daily business. )
>
> OK, good to see some progress!
>
On 1st impression thing went fine, but after a while jobs like opening
several tabs in firefox or doing a simple df command stalled the
machine. No, my system even got frozen and required a brutal reset.
>From the logs (more see file-attachment):
Mar 26 19:58:40 tbox kernel: [ 1440.640060] INFO: task systemd:1
blocked for more than 120 seconds.
Mar 26 19:58:40 tbox kernel: [ 1440.640074] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Following it -> NOPE
$ echo 0 > /proc/sys/kernel/hung_task_timeout_secs
> Is there a runaway process consuming CPU? The reason that I ask is that
> an infinite loop in the kernel can result in a stall when PREEMPT=n
> but is less likely to if PREEMPT=y. Could you please check with "top",
> "ps", or whatever?
>
Unsure what you mean by this, as you can see from the logs, it's not
only one special task "stalling".
BTW, I have systemd here running.
>> I am not sure what the change to PREEMPT exactly mean in the end.
>> ( Let's work with this new kernel and carefully check for possible
>> side-effects. )
>> For example CONFIG_RCU_FAST_NO_HZ=y is now dropped, where the Kconfig
>> descriptive text says some words on better energy saving. For a
>> notebook this is no good.
>
> CONFIG_RCU_FAST_NO_HZ is of no use on a uniprocessor system, so OK
> to disable it. But are you saying that CONFIG_RCU_FAST_NO_HZ=y
> results in problems that are removed by CONFIG_RCU_FAST_NO_HZ=n?
> That would be a surprise, and I need to know if this is the case.
>
In my current setup (PREEMPT and TREE_PREEMPT_RCU)
CONFIG_RCU_FAST_NO_HZ is not considered and set via Kconfig-system
(see excerpt below).
But when you say for UP it is of no use, I need no more info.
Might be good to add some recommended (and/or useless) kernel-config
settings to RCU/UP.txt?
[ init/Kconfig ]
config RCU_FAST_NO_HZ
bool "Accelerate last non-dyntick-idle CPU's grace periods"
depends on TREE_RCU && NO_HZ && SMP
default n
>> I have also questions to some Kconfig dependencies, for example why I
>> can't select TREE_PREEMPT_RCU if CONFIG_PREEMPT_VOLUNTARY=y, etc.
>> Intended?
>
> Yes. There is no point in TREE_PREEMPT_RCU unless PREEMPT=y.
>
OK.
>> Maybe I collect all my askings in a separate email to RCU folks and ML
>> and do not disturb further people from other sub-trees.
>>
>> I enjoyed to read the numerous docs in Documentation/RCU/ (and noticed
>> some typos as well).
>> The RCU folk gave the word "FAQ" a new meaning: Frequenty Asked
>> Questions & Q*uiz* :-).
>>
>> Thanks for the helpful hints and explanations from the RCU folks!
>
> Glad you liked them! ;-)
>
Other sub-trees lack of no good or up2date docs.
>> - Sedat -
>>
>> P.S.: Current RCU and HZ kernel-config settings
>>
>> # grep RCU /boot/config-$(uname -r)
>> # RCU Subsystem
>> CONFIG_TREE_PREEMPT_RCU=y
>> CONFIG_PREEMPT_RCU=y
>> CONFIG_RCU_TRACE=y
>> CONFIG_RCU_FANOUT=32
>> # CONFIG_RCU_FANOUT_EXACT is not set
>> CONFIG_TREE_RCU_TRACE=y
>> CONFIG_RCU_BOOST=y
>> CONFIG_RCU_BOOST_PRIO=1
>> CONFIG_RCU_BOOST_DELAY=500
>> # CONFIG_SPARSE_RCU_POINTER is not set
>> # CONFIG_RCU_TORTURE_TEST is not set
>> CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> CONFIG_RCU_CPU_STALL_VERBOSE=y
>>
>> # grep _HZ /boot/config-$(uname -r)
>> CONFIG_NO_HZ=y
>> # CONFIG_HZ_100 is not set
>> CONFIG_HZ_250=y
>> # CONFIG_HZ_300 is not set
>> # CONFIG_HZ_1000 is not set
>> CONFIG_HZ=250
>
> OK, thank you for the info!
>
N.P.
> Thanx, Paul
>
I guees I will revert step-by-step the RCU commits in linux-next and report.
This weekend I wanted to enhance Debian's live-cd framework with
overlayfs support and a customized kernel.
But then came RCU :-(.
Can you say some words to kfree_rcu.2011.03.25b (rcu/kfree_rcu) GIT branch(es)?
- Sedat -
[-- Attachment #2: syslog-hung_task_timeout_secs.txt --]
[-- Type: text/plain, Size: 10949 bytes --]
Mar 26 00:45:07 tbox kernel: [ 1080.588102] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 00:45:07 tbox kernel: [ 1080.588114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 00:47:07 tbox kernel: [ 1200.588103] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 00:47:07 tbox kernel: [ 1200.588116] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 00:49:07 tbox kernel: [ 1320.588102] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 00:49:07 tbox kernel: [ 1320.588115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 00:51:07 tbox kernel: [ 1440.588100] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 00:51:07 tbox kernel: [ 1440.588114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 00:53:07 tbox kernel: [ 1560.588101] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 00:53:07 tbox kernel: [ 1560.588114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 00:55:07 tbox kernel: [ 1680.588148] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 00:55:07 tbox kernel: [ 1680.588161] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 00:57:07 tbox kernel: [ 1800.588101] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 00:57:07 tbox kernel: [ 1800.588114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 00:59:08 tbox kernel: [ 1920.588108] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 00:59:08 tbox kernel: [ 1920.588122] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 01:01:07 tbox kernel: [ 2040.588102] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 01:01:07 tbox kernel: [ 2040.588115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 01:03:07 tbox kernel: [ 2160.588102] INFO: task umount:2091 blocked for more than 120 seconds.
Mar 26 01:03:07 tbox kernel: [ 2160.588115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:20:00 tbox kernel: [ 1080.640170] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 13:20:00 tbox kernel: [ 1080.640184] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:20:00 tbox kernel: [ 1080.640708] INFO: task kded4:1367 blocked for more than 120 seconds.
Mar 26 13:20:00 tbox kernel: [ 1080.640717] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:20:00 tbox kernel: [ 1080.641116] INFO: task sd:exec:1904 blocked for more than 120 seconds.
Mar 26 13:20:00 tbox kernel: [ 1080.641125] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:20:00 tbox kernel: [ 1080.641488] INFO: task cron:1905 blocked for more than 120 seconds.
Mar 26 13:20:00 tbox kernel: [ 1080.641497] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:22:00 tbox kernel: [ 1200.640058] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 13:22:00 tbox kernel: [ 1200.640071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:22:00 tbox kernel: [ 1200.640592] INFO: task kded4:1367 blocked for more than 120 seconds.
Mar 26 13:22:00 tbox kernel: [ 1200.640602] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:22:00 tbox kernel: [ 1200.641000] INFO: task sd:exec:1904 blocked for more than 120 seconds.
Mar 26 13:22:00 tbox kernel: [ 1200.641009] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:22:00 tbox kernel: [ 1200.641372] INFO: task cron:1905 blocked for more than 120 seconds.
Mar 26 13:22:00 tbox kernel: [ 1200.641381] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:24:00 tbox kernel: [ 1320.640055] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 13:24:00 tbox kernel: [ 1320.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 13:24:00 tbox kernel: [ 1320.640602] INFO: task kded4:1367 blocked for more than 120 seconds.
Mar 26 13:24:00 tbox kernel: [ 1320.640611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 14:09:31 tbox kernel: [ 480.640055] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 14:09:31 tbox kernel: [ 480.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 14:11:31 tbox kernel: [ 600.640055] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 14:11:31 tbox kernel: [ 600.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 14:13:31 tbox kernel: [ 720.640054] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 14:13:31 tbox kernel: [ 720.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 14:15:31 tbox kernel: [ 840.640067] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 14:15:31 tbox kernel: [ 840.640080] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 14:30:12 tbox kernel: [ 840.644071] INFO: task kworker/u:1:173 blocked for more than 120 seconds.
Mar 26 14:30:12 tbox kernel: [ 840.644085] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 14:30:12 tbox kernel: [ 840.645204] INFO: task firefox-bin:1940 blocked for more than 120 seconds.
Mar 26 14:30:12 tbox kernel: [ 840.645213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 14:30:12 tbox kernel: [ 840.645829] INFO: task firefox-bin:1947 blocked for more than 120 seconds.
Mar 26 14:30:12 tbox kernel: [ 840.645839] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 14:30:12 tbox kernel: [ 840.646441] INFO: task firefox-bin:1949 blocked for more than 120 seconds.
Mar 26 14:30:12 tbox kernel: [ 840.646450] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 14:30:12 tbox kernel: [ 840.647053] INFO: task kworker/0:1:1926 blocked for more than 120 seconds.
Mar 26 14:30:12 tbox kernel: [ 840.647062] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:31:45 tbox kernel: [ 360.640052] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
Mar 26 19:31:45 tbox kernel: [ 360.640059] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:31:45 tbox kernel: [ 360.640265] INFO: task kworker/u:1:190 blocked for more than 120 seconds.
Mar 26 19:31:45 tbox kernel: [ 360.640268] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:31:45 tbox kernel: [ 360.640721] INFO: task firefox-bin:1994 blocked for more than 120 seconds.
Mar 26 19:31:45 tbox kernel: [ 360.640725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:31:45 tbox kernel: [ 360.640945] INFO: task firefox-bin:2009 blocked for more than 120 seconds.
Mar 26 19:31:45 tbox kernel: [ 360.640948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:31:45 tbox kernel: [ 360.641143] INFO: task firefox-bin:2010 blocked for more than 120 seconds.
Mar 26 19:31:45 tbox kernel: [ 360.641146] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:31:45 tbox kernel: [ 360.641352] INFO: task firefox-bin:2011 blocked for more than 120 seconds.
Mar 26 19:31:45 tbox kernel: [ 360.641355] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:31:45 tbox kernel: [ 360.641576] INFO: task su:2015 blocked for more than 120 seconds.
Mar 26 19:31:45 tbox kernel: [ 360.641579] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:33:45 tbox kernel: [ 480.640053] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 19:33:45 tbox kernel: [ 480.640066] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:33:45 tbox kernel: [ 480.640514] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
Mar 26 19:33:45 tbox kernel: [ 480.640523] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:33:45 tbox kernel: [ 480.641014] INFO: task kworker/u:1:190 blocked for more than 120 seconds.
Mar 26 19:33:45 tbox kernel: [ 480.641024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:52:40 tbox kernel: [ 1080.640062] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 19:52:40 tbox kernel: [ 1080.640075] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:52:40 tbox kernel: [ 1080.640612] INFO: task kded4:1514 blocked for more than 120 seconds.
Mar 26 19:52:40 tbox kernel: [ 1080.640621] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:52:40 tbox kernel: [ 1080.641033] INFO: task sd:exec:1978 blocked for more than 120 seconds.
Mar 26 19:52:40 tbox kernel: [ 1080.641042] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:54:40 tbox kernel: [ 1200.640055] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 19:54:40 tbox kernel: [ 1200.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:54:40 tbox kernel: [ 1200.640604] INFO: task kded4:1514 blocked for more than 120 seconds.
Mar 26 19:54:40 tbox kernel: [ 1200.640613] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:54:40 tbox kernel: [ 1200.641026] INFO: task sd:exec:1978 blocked for more than 120 seconds.
Mar 26 19:54:40 tbox kernel: [ 1200.641035] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:56:40 tbox kernel: [ 1320.640059] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 19:56:40 tbox kernel: [ 1320.640072] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:56:40 tbox kernel: [ 1320.640607] INFO: task kded4:1514 blocked for more than 120 seconds.
Mar 26 19:56:40 tbox kernel: [ 1320.640616] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:56:40 tbox kernel: [ 1320.641027] INFO: task sd:exec:1978 blocked for more than 120 seconds.
Mar 26 19:56:40 tbox kernel: [ 1320.641036] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
--
Mar 26 19:58:40 tbox kernel: [ 1440.640060] INFO: task systemd:1 blocked for more than 120 seconds.
Mar 26 19:58:40 tbox kernel: [ 1440.640074] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-26 22:15 ` Sedat Dilek
@ 2011-03-27 0:09 ` Paul E. McKenney
2011-03-27 1:30 ` Sedat Dilek
0 siblings, 1 reply; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-27 0:09 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
> On Sat, Mar 26, 2011 at 5:02 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Sat, Mar 26, 2011 at 01:34:53PM +0100, Sedat Dilek wrote:
> [...]
> >> Just FYI: Changed to the following settings:
> >>
> >> - Enable Preemptible Kernel (Low-Latency Desktop)
> >> - Enable Preemptible tree-based hierarchical RCU
> >> - Enable RCU priority boosting
> >> - Reset RCU CPU stall timeout to default (60 seconds)
> >>
> >> So far I see no RCU stalls in the logs and my system runs as expected.
> >> ( I have noticed here some "stalling" in the webbrowser, but I can do
> >> my daily business. )
> >
> > OK, good to see some progress!
> >
>
> On 1st impression thing went fine, but after a while jobs like opening
> several tabs in firefox or doing a simple df command stalled the
> machine. No, my system even got frozen and required a brutal reset.
>
> >From the logs (more see file-attachment):
>
> Mar 26 19:58:40 tbox kernel: [ 1440.640060] INFO: task systemd:1
> blocked for more than 120 seconds.
> Mar 26 19:58:40 tbox kernel: [ 1440.640074] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>
> Following it -> NOPE
> $ echo 0 > /proc/sys/kernel/hung_task_timeout_secs
These are tasks that are blocked, not tasks that are consuming CPU.
> > Is there a runaway process consuming CPU? The reason that I ask is that
> > an infinite loop in the kernel can result in a stall when PREEMPT=n
> > but is less likely to if PREEMPT=y. Could you please check with "top",
> > "ps", or whatever?
>
> Unsure what you mean by this, as you can see from the logs, it's not
> only one special task "stalling".
> BTW, I have systemd here running.
Right. But I need to know if there are tasks consuming lots of CPU,
which is different than tasks that are stalled for a long time. Look
at the message: it says "blocked for more than 120 seconds", not
"running for more than 120 seconds".
For example, if one of the RCU kthreads is consuming lots of CPU, that
would tell me that I should look for an RCU bug (and yank the patches
from -next in the meantime). On the other hand, if some other task is
consuming lots of CPU, then that would be a hint as to where to find
the bug.
> >> I am not sure what the change to PREEMPT exactly mean in the end.
> >> ( Let's work with this new kernel and carefully check for possible
> >> side-effects. )
> >> For example CONFIG_RCU_FAST_NO_HZ=y is now dropped, where the Kconfig
> >> descriptive text says some words on better energy saving. For a
> >> notebook this is no good.
> >
> > CONFIG_RCU_FAST_NO_HZ is of no use on a uniprocessor system, so OK
> > to disable it. But are you saying that CONFIG_RCU_FAST_NO_HZ=y
> > results in problems that are removed by CONFIG_RCU_FAST_NO_HZ=n?
> > That would be a surprise, and I need to know if this is the case.
>
> In my current setup (PREEMPT and TREE_PREEMPT_RCU)
> CONFIG_RCU_FAST_NO_HZ is not considered and set via Kconfig-system
> (see excerpt below).
> But when you say for UP it is of no use, I need no more info.
OK, "of no use" is overstating things a bit. But CONFIG_RCU_FAST_NO_HZ's
main purpose is to get the last CPU into dyntick-idle state
(CONFIG_NO_HZ), which is most useful if the system has several CPUs.
> Might be good to add some recommended (and/or useless) kernel-config
> settings to RCU/UP.txt?
>
> [ init/Kconfig ]
> config RCU_FAST_NO_HZ
> bool "Accelerate last non-dyntick-idle CPU's grace periods"
> depends on TREE_RCU && NO_HZ && SMP
> default n
The "depends on TREE_RCU && NO_HZ && SMP" already excludes it from
UP kernel builds, so no need to document.
> >> I have also questions to some Kconfig dependencies, for example why I
> >> can't select TREE_PREEMPT_RCU if CONFIG_PREEMPT_VOLUNTARY=y, etc.
> >> Intended?
> >
> > Yes. There is no point in TREE_PREEMPT_RCU unless PREEMPT=y.
> >
>
> OK.
>
> >> Maybe I collect all my askings in a separate email to RCU folks and ML
> >> and do not disturb further people from other sub-trees.
> >>
> >> I enjoyed to read the numerous docs in Documentation/RCU/ (and noticed
> >> some typos as well).
> >> The RCU folk gave the word "FAQ" a new meaning: Frequenty Asked
> >> Questions & Q*uiz* :-).
> >>
> >> Thanks for the helpful hints and explanations from the RCU folks!
> >
> > Glad you liked them! ;-)
> >
>
> Other sub-trees lack of no good or up2date docs.
>
> >> - Sedat -
> >>
> >> P.S.: Current RCU and HZ kernel-config settings
> >>
> >> # grep RCU /boot/config-$(uname -r)
> >> # RCU Subsystem
> >> CONFIG_TREE_PREEMPT_RCU=y
> >> CONFIG_PREEMPT_RCU=y
> >> CONFIG_RCU_TRACE=y
> >> CONFIG_RCU_FANOUT=32
> >> # CONFIG_RCU_FANOUT_EXACT is not set
> >> CONFIG_TREE_RCU_TRACE=y
> >> CONFIG_RCU_BOOST=y
> >> CONFIG_RCU_BOOST_PRIO=1
> >> CONFIG_RCU_BOOST_DELAY=500
> >> # CONFIG_SPARSE_RCU_POINTER is not set
> >> # CONFIG_RCU_TORTURE_TEST is not set
> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
> >> CONFIG_RCU_CPU_STALL_VERBOSE=y
> >>
> >> # grep _HZ /boot/config-$(uname -r)
> >> CONFIG_NO_HZ=y
> >> # CONFIG_HZ_100 is not set
> >> CONFIG_HZ_250=y
> >> # CONFIG_HZ_300 is not set
> >> # CONFIG_HZ_1000 is not set
> >> CONFIG_HZ=250
> >
> > OK, thank you for the info!
> >
>
> N.P.
>
> > Thanx, Paul
> >
>
> I guees I will revert step-by-step the RCU commits in linux-next and report.
> This weekend I wanted to enhance Debian's live-cd framework with
> overlayfs support and a customized kernel.
> But then came RCU :-(.
Well, if it turns out to be a problem in RCU I will certainly apologize.
> Can you say some words to kfree_rcu.2011.03.25b (rcu/kfree_rcu) GIT branch(es)?
These are just applying Lia Jiangshan's kfree_rcu() to a number of places
in the kernel. You can safely ignore them.
Thanx, Paul
> - Sedat -
> Mar 26 00:45:07 tbox kernel: [ 1080.588102] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 00:45:07 tbox kernel: [ 1080.588114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 00:47:07 tbox kernel: [ 1200.588103] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 00:47:07 tbox kernel: [ 1200.588116] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 00:49:07 tbox kernel: [ 1320.588102] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 00:49:07 tbox kernel: [ 1320.588115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 00:51:07 tbox kernel: [ 1440.588100] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 00:51:07 tbox kernel: [ 1440.588114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 00:53:07 tbox kernel: [ 1560.588101] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 00:53:07 tbox kernel: [ 1560.588114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 00:55:07 tbox kernel: [ 1680.588148] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 00:55:07 tbox kernel: [ 1680.588161] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 00:57:07 tbox kernel: [ 1800.588101] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 00:57:07 tbox kernel: [ 1800.588114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 00:59:08 tbox kernel: [ 1920.588108] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 00:59:08 tbox kernel: [ 1920.588122] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 01:01:07 tbox kernel: [ 2040.588102] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 01:01:07 tbox kernel: [ 2040.588115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 01:03:07 tbox kernel: [ 2160.588102] INFO: task umount:2091 blocked for more than 120 seconds.
> Mar 26 01:03:07 tbox kernel: [ 2160.588115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:20:00 tbox kernel: [ 1080.640170] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 13:20:00 tbox kernel: [ 1080.640184] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:20:00 tbox kernel: [ 1080.640708] INFO: task kded4:1367 blocked for more than 120 seconds.
> Mar 26 13:20:00 tbox kernel: [ 1080.640717] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:20:00 tbox kernel: [ 1080.641116] INFO: task sd:exec:1904 blocked for more than 120 seconds.
> Mar 26 13:20:00 tbox kernel: [ 1080.641125] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:20:00 tbox kernel: [ 1080.641488] INFO: task cron:1905 blocked for more than 120 seconds.
> Mar 26 13:20:00 tbox kernel: [ 1080.641497] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:22:00 tbox kernel: [ 1200.640058] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 13:22:00 tbox kernel: [ 1200.640071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:22:00 tbox kernel: [ 1200.640592] INFO: task kded4:1367 blocked for more than 120 seconds.
> Mar 26 13:22:00 tbox kernel: [ 1200.640602] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:22:00 tbox kernel: [ 1200.641000] INFO: task sd:exec:1904 blocked for more than 120 seconds.
> Mar 26 13:22:00 tbox kernel: [ 1200.641009] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:22:00 tbox kernel: [ 1200.641372] INFO: task cron:1905 blocked for more than 120 seconds.
> Mar 26 13:22:00 tbox kernel: [ 1200.641381] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:24:00 tbox kernel: [ 1320.640055] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 13:24:00 tbox kernel: [ 1320.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 13:24:00 tbox kernel: [ 1320.640602] INFO: task kded4:1367 blocked for more than 120 seconds.
> Mar 26 13:24:00 tbox kernel: [ 1320.640611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 14:09:31 tbox kernel: [ 480.640055] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 14:09:31 tbox kernel: [ 480.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 14:11:31 tbox kernel: [ 600.640055] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 14:11:31 tbox kernel: [ 600.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 14:13:31 tbox kernel: [ 720.640054] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 14:13:31 tbox kernel: [ 720.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 14:15:31 tbox kernel: [ 840.640067] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 14:15:31 tbox kernel: [ 840.640080] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 14:30:12 tbox kernel: [ 840.644071] INFO: task kworker/u:1:173 blocked for more than 120 seconds.
> Mar 26 14:30:12 tbox kernel: [ 840.644085] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 14:30:12 tbox kernel: [ 840.645204] INFO: task firefox-bin:1940 blocked for more than 120 seconds.
> Mar 26 14:30:12 tbox kernel: [ 840.645213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 14:30:12 tbox kernel: [ 840.645829] INFO: task firefox-bin:1947 blocked for more than 120 seconds.
> Mar 26 14:30:12 tbox kernel: [ 840.645839] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 14:30:12 tbox kernel: [ 840.646441] INFO: task firefox-bin:1949 blocked for more than 120 seconds.
> Mar 26 14:30:12 tbox kernel: [ 840.646450] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 14:30:12 tbox kernel: [ 840.647053] INFO: task kworker/0:1:1926 blocked for more than 120 seconds.
> Mar 26 14:30:12 tbox kernel: [ 840.647062] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:31:45 tbox kernel: [ 360.640052] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
> Mar 26 19:31:45 tbox kernel: [ 360.640059] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:31:45 tbox kernel: [ 360.640265] INFO: task kworker/u:1:190 blocked for more than 120 seconds.
> Mar 26 19:31:45 tbox kernel: [ 360.640268] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:31:45 tbox kernel: [ 360.640721] INFO: task firefox-bin:1994 blocked for more than 120 seconds.
> Mar 26 19:31:45 tbox kernel: [ 360.640725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:31:45 tbox kernel: [ 360.640945] INFO: task firefox-bin:2009 blocked for more than 120 seconds.
> Mar 26 19:31:45 tbox kernel: [ 360.640948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:31:45 tbox kernel: [ 360.641143] INFO: task firefox-bin:2010 blocked for more than 120 seconds.
> Mar 26 19:31:45 tbox kernel: [ 360.641146] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:31:45 tbox kernel: [ 360.641352] INFO: task firefox-bin:2011 blocked for more than 120 seconds.
> Mar 26 19:31:45 tbox kernel: [ 360.641355] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:31:45 tbox kernel: [ 360.641576] INFO: task su:2015 blocked for more than 120 seconds.
> Mar 26 19:31:45 tbox kernel: [ 360.641579] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:33:45 tbox kernel: [ 480.640053] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 19:33:45 tbox kernel: [ 480.640066] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:33:45 tbox kernel: [ 480.640514] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
> Mar 26 19:33:45 tbox kernel: [ 480.640523] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:33:45 tbox kernel: [ 480.641014] INFO: task kworker/u:1:190 blocked for more than 120 seconds.
> Mar 26 19:33:45 tbox kernel: [ 480.641024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:52:40 tbox kernel: [ 1080.640062] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 19:52:40 tbox kernel: [ 1080.640075] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:52:40 tbox kernel: [ 1080.640612] INFO: task kded4:1514 blocked for more than 120 seconds.
> Mar 26 19:52:40 tbox kernel: [ 1080.640621] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:52:40 tbox kernel: [ 1080.641033] INFO: task sd:exec:1978 blocked for more than 120 seconds.
> Mar 26 19:52:40 tbox kernel: [ 1080.641042] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:54:40 tbox kernel: [ 1200.640055] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 19:54:40 tbox kernel: [ 1200.640068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:54:40 tbox kernel: [ 1200.640604] INFO: task kded4:1514 blocked for more than 120 seconds.
> Mar 26 19:54:40 tbox kernel: [ 1200.640613] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:54:40 tbox kernel: [ 1200.641026] INFO: task sd:exec:1978 blocked for more than 120 seconds.
> Mar 26 19:54:40 tbox kernel: [ 1200.641035] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:56:40 tbox kernel: [ 1320.640059] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 19:56:40 tbox kernel: [ 1320.640072] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:56:40 tbox kernel: [ 1320.640607] INFO: task kded4:1514 blocked for more than 120 seconds.
> Mar 26 19:56:40 tbox kernel: [ 1320.640616] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:56:40 tbox kernel: [ 1320.641027] INFO: task sd:exec:1978 blocked for more than 120 seconds.
> Mar 26 19:56:40 tbox kernel: [ 1320.641036] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> --
> Mar 26 19:58:40 tbox kernel: [ 1440.640060] INFO: task systemd:1 blocked for more than 120 seconds.
> Mar 26 19:58:40 tbox kernel: [ 1440.640074] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-27 0:09 ` Paul E. McKenney
@ 2011-03-27 1:30 ` Sedat Dilek
2011-03-27 3:25 ` Paul E. McKenney
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-27 1:30 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
>> On Sat, Mar 26, 2011 at 5:02 PM, Paul E. McKenney
>> <paulmck@linux.vnet.ibm.com> wrote:
>> > On Sat, Mar 26, 2011 at 01:34:53PM +0100, Sedat Dilek wrote:
>> [...]
>> >> Just FYI: Changed to the following settings:
>> >>
>> >> - Enable Preemptible Kernel (Low-Latency Desktop)
>> >> - Enable Preemptible tree-based hierarchical RCU
>> >> - Enable RCU priority boosting
>> >> - Reset RCU CPU stall timeout to default (60 seconds)
>> >>
>> >> So far I see no RCU stalls in the logs and my system runs as expected.
>> >> ( I have noticed here some "stalling" in the webbrowser, but I can do
>> >> my daily business. )
>> >
>> > OK, good to see some progress!
>> >
>>
>> On 1st impression thing went fine, but after a while jobs like opening
>> several tabs in firefox or doing a simple df command stalled the
>> machine. No, my system even got frozen and required a brutal reset.
>>
>> >From the logs (more see file-attachment):
>>
>> Mar 26 19:58:40 tbox kernel: [ 1440.640060] INFO: task systemd:1
>> blocked for more than 120 seconds.
>> Mar 26 19:58:40 tbox kernel: [ 1440.640074] "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>
>> Following it -> NOPE
>> $ echo 0 > /proc/sys/kernel/hung_task_timeout_secs
>
> These are tasks that are blocked, not tasks that are consuming CPU.
>
>> > Is there a runaway process consuming CPU? The reason that I ask is that
>> > an infinite loop in the kernel can result in a stall when PREEMPT=n
>> > but is less likely to if PREEMPT=y. Could you please check with "top",
>> > "ps", or whatever?
>>
>> Unsure what you mean by this, as you can see from the logs, it's not
>> only one special task "stalling".
>> BTW, I have systemd here running.
>
> Right. But I need to know if there are tasks consuming lots of CPU,
> which is different than tasks that are stalled for a long time. Look
> at the message: it says "blocked for more than 120 seconds", not
> "running for more than 120 seconds".
>
> For example, if one of the RCU kthreads is consuming lots of CPU, that
> would tell me that I should look for an RCU bug (and yank the patches
> from -next in the meantime). On the other hand, if some other task is
> consuming lots of CPU, then that would be a hint as to where to find
> the bug.
>
>> >> I am not sure what the change to PREEMPT exactly mean in the end.
>> >> ( Let's work with this new kernel and carefully check for possible
>> >> side-effects. )
>> >> For example CONFIG_RCU_FAST_NO_HZ=y is now dropped, where the Kconfig
>> >> descriptive text says some words on better energy saving. For a
>> >> notebook this is no good.
>> >
>> > CONFIG_RCU_FAST_NO_HZ is of no use on a uniprocessor system, so OK
>> > to disable it. But are you saying that CONFIG_RCU_FAST_NO_HZ=y
>> > results in problems that are removed by CONFIG_RCU_FAST_NO_HZ=n?
>> > That would be a surprise, and I need to know if this is the case.
>>
>> In my current setup (PREEMPT and TREE_PREEMPT_RCU)
>> CONFIG_RCU_FAST_NO_HZ is not considered and set via Kconfig-system
>> (see excerpt below).
>> But when you say for UP it is of no use, I need no more info.
>
> OK, "of no use" is overstating things a bit. But CONFIG_RCU_FAST_NO_HZ's
> main purpose is to get the last CPU into dyntick-idle state
> (CONFIG_NO_HZ), which is most useful if the system has several CPUs.
>
>> Might be good to add some recommended (and/or useless) kernel-config
>> settings to RCU/UP.txt?
>>
>> [ init/Kconfig ]
>> config RCU_FAST_NO_HZ
>> bool "Accelerate last non-dyntick-idle CPU's grace periods"
>> depends on TREE_RCU && NO_HZ && SMP
>> default n
>
> The "depends on TREE_RCU && NO_HZ && SMP" already excludes it from
> UP kernel builds, so no need to document.
>
>> >> I have also questions to some Kconfig dependencies, for example why I
>> >> can't select TREE_PREEMPT_RCU if CONFIG_PREEMPT_VOLUNTARY=y, etc.
>> >> Intended?
>> >
>> > Yes. There is no point in TREE_PREEMPT_RCU unless PREEMPT=y.
>> >
>>
>> OK.
>>
>> >> Maybe I collect all my askings in a separate email to RCU folks and ML
>> >> and do not disturb further people from other sub-trees.
>> >>
>> >> I enjoyed to read the numerous docs in Documentation/RCU/ (and noticed
>> >> some typos as well).
>> >> The RCU folk gave the word "FAQ" a new meaning: Frequenty Asked
>> >> Questions & Q*uiz* :-).
>> >>
>> >> Thanks for the helpful hints and explanations from the RCU folks!
>> >
>> > Glad you liked them! ;-)
>> >
>>
>> Other sub-trees lack of no good or up2date docs.
>>
>> >> - Sedat -
>> >>
>> >> P.S.: Current RCU and HZ kernel-config settings
>> >>
>> >> # grep RCU /boot/config-$(uname -r)
>> >> # RCU Subsystem
>> >> CONFIG_TREE_PREEMPT_RCU=y
>> >> CONFIG_PREEMPT_RCU=y
>> >> CONFIG_RCU_TRACE=y
>> >> CONFIG_RCU_FANOUT=32
>> >> # CONFIG_RCU_FANOUT_EXACT is not set
>> >> CONFIG_TREE_RCU_TRACE=y
>> >> CONFIG_RCU_BOOST=y
>> >> CONFIG_RCU_BOOST_PRIO=1
>> >> CONFIG_RCU_BOOST_DELAY=500
>> >> # CONFIG_SPARSE_RCU_POINTER is not set
>> >> # CONFIG_RCU_TORTURE_TEST is not set
>> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> >> CONFIG_RCU_CPU_STALL_VERBOSE=y
>> >>
>> >> # grep _HZ /boot/config-$(uname -r)
>> >> CONFIG_NO_HZ=y
>> >> # CONFIG_HZ_100 is not set
>> >> CONFIG_HZ_250=y
>> >> # CONFIG_HZ_300 is not set
>> >> # CONFIG_HZ_1000 is not set
>> >> CONFIG_HZ=250
>> >
>> > OK, thank you for the info!
>> >
>>
>> N.P.
>>
>> > Thanx, Paul
>> >
>>
>> I guees I will revert step-by-step the RCU commits in linux-next and report.
>> This weekend I wanted to enhance Debian's live-cd framework with
>> overlayfs support and a customized kernel.
>> But then came RCU :-(.
>
> Well, if it turns out to be a problem in RCU I will certainly apologize.
>
No, that's not so dramatic.
Dealing with this RCU issue has nice side-effects: I remembered (and
finally did) to use a reduced kernel-config set.
The base for it I created with 'make localmodconfig' and did some
manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
etc.).
Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
So, I started building with
"revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
I will let you know.
- Sedat -
>> Can you say some words to kfree_rcu.2011.03.25b (rcu/kfree_rcu) GIT branch(es)?
>
> These are just applying Lia Jiangshan's kfree_rcu() to a number of places
> in the kernel. You can safely ignore them.
>
> Thanx, Paul
>
>> - Sedat -
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-27 1:30 ` Sedat Dilek
@ 2011-03-27 3:25 ` Paul E. McKenney
2011-03-27 5:07 ` Paul E. McKenney
0 siblings, 1 reply; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-27 3:25 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
> On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
> >> On Sat, Mar 26, 2011 at 5:02 PM, Paul E. McKenney
> >> <paulmck@linux.vnet.ibm.com> wrote:
> >> > On Sat, Mar 26, 2011 at 01:34:53PM +0100, Sedat Dilek wrote:
> >> [...]
> >> >> Just FYI: Changed to the following settings:
> >> >>
> >> >> - Enable Preemptible Kernel (Low-Latency Desktop)
> >> >> - Enable Preemptible tree-based hierarchical RCU
> >> >> - Enable RCU priority boosting
> >> >> - Reset RCU CPU stall timeout to default (60 seconds)
> >> >>
> >> >> So far I see no RCU stalls in the logs and my system runs as expected.
> >> >> ( I have noticed here some "stalling" in the webbrowser, but I can do
> >> >> my daily business. )
> >> >
> >> > OK, good to see some progress!
> >> >
> >>
> >> On 1st impression thing went fine, but after a while jobs like opening
> >> several tabs in firefox or doing a simple df command stalled the
> >> machine. No, my system even got frozen and required a brutal reset.
> >>
> >> >From the logs (more see file-attachment):
> >>
> >> Mar 26 19:58:40 tbox kernel: [ 1440.640060] INFO: task systemd:1
> >> blocked for more than 120 seconds.
> >> Mar 26 19:58:40 tbox kernel: [ 1440.640074] "echo 0 >
> >> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> >>
> >> Following it -> NOPE
> >> $ echo 0 > /proc/sys/kernel/hung_task_timeout_secs
> >
> > These are tasks that are blocked, not tasks that are consuming CPU.
> >
> >> > Is there a runaway process consuming CPU? The reason that I ask is that
> >> > an infinite loop in the kernel can result in a stall when PREEMPT=n
> >> > but is less likely to if PREEMPT=y. Could you please check with "top",
> >> > "ps", or whatever?
> >>
> >> Unsure what you mean by this, as you can see from the logs, it's not
> >> only one special task "stalling".
> >> BTW, I have systemd here running.
> >
> > Right. But I need to know if there are tasks consuming lots of CPU,
> > which is different than tasks that are stalled for a long time. Look
> > at the message: it says "blocked for more than 120 seconds", not
> > "running for more than 120 seconds".
> >
> > For example, if one of the RCU kthreads is consuming lots of CPU, that
> > would tell me that I should look for an RCU bug (and yank the patches
> > from -next in the meantime). On the other hand, if some other task is
> > consuming lots of CPU, then that would be a hint as to where to find
> > the bug.
> >
> >> >> I am not sure what the change to PREEMPT exactly mean in the end.
> >> >> ( Let's work with this new kernel and carefully check for possible
> >> >> side-effects. )
> >> >> For example CONFIG_RCU_FAST_NO_HZ=y is now dropped, where the Kconfig
> >> >> descriptive text says some words on better energy saving. For a
> >> >> notebook this is no good.
> >> >
> >> > CONFIG_RCU_FAST_NO_HZ is of no use on a uniprocessor system, so OK
> >> > to disable it. But are you saying that CONFIG_RCU_FAST_NO_HZ=y
> >> > results in problems that are removed by CONFIG_RCU_FAST_NO_HZ=n?
> >> > That would be a surprise, and I need to know if this is the case.
> >>
> >> In my current setup (PREEMPT and TREE_PREEMPT_RCU)
> >> CONFIG_RCU_FAST_NO_HZ is not considered and set via Kconfig-system
> >> (see excerpt below).
> >> But when you say for UP it is of no use, I need no more info.
> >
> > OK, "of no use" is overstating things a bit. But CONFIG_RCU_FAST_NO_HZ's
> > main purpose is to get the last CPU into dyntick-idle state
> > (CONFIG_NO_HZ), which is most useful if the system has several CPUs.
> >
> >> Might be good to add some recommended (and/or useless) kernel-config
> >> settings to RCU/UP.txt?
> >>
> >> [ init/Kconfig ]
> >> config RCU_FAST_NO_HZ
> >> bool "Accelerate last non-dyntick-idle CPU's grace periods"
> >> depends on TREE_RCU && NO_HZ && SMP
> >> default n
> >
> > The "depends on TREE_RCU && NO_HZ && SMP" already excludes it from
> > UP kernel builds, so no need to document.
> >
> >> >> I have also questions to some Kconfig dependencies, for example why I
> >> >> can't select TREE_PREEMPT_RCU if CONFIG_PREEMPT_VOLUNTARY=y, etc.
> >> >> Intended?
> >> >
> >> > Yes. There is no point in TREE_PREEMPT_RCU unless PREEMPT=y.
> >> >
> >>
> >> OK.
> >>
> >> >> Maybe I collect all my askings in a separate email to RCU folks and ML
> >> >> and do not disturb further people from other sub-trees.
> >> >>
> >> >> I enjoyed to read the numerous docs in Documentation/RCU/ (and noticed
> >> >> some typos as well).
> >> >> The RCU folk gave the word "FAQ" a new meaning: Frequenty Asked
> >> >> Questions & Q*uiz* :-).
> >> >>
> >> >> Thanks for the helpful hints and explanations from the RCU folks!
> >> >
> >> > Glad you liked them! ;-)
> >> >
> >>
> >> Other sub-trees lack of no good or up2date docs.
> >>
> >> >> - Sedat -
> >> >>
> >> >> P.S.: Current RCU and HZ kernel-config settings
> >> >>
> >> >> # grep RCU /boot/config-$(uname -r)
> >> >> # RCU Subsystem
> >> >> CONFIG_TREE_PREEMPT_RCU=y
> >> >> CONFIG_PREEMPT_RCU=y
> >> >> CONFIG_RCU_TRACE=y
> >> >> CONFIG_RCU_FANOUT=32
> >> >> # CONFIG_RCU_FANOUT_EXACT is not set
> >> >> CONFIG_TREE_RCU_TRACE=y
> >> >> CONFIG_RCU_BOOST=y
> >> >> CONFIG_RCU_BOOST_PRIO=1
> >> >> CONFIG_RCU_BOOST_DELAY=500
> >> >> # CONFIG_SPARSE_RCU_POINTER is not set
> >> >> # CONFIG_RCU_TORTURE_TEST is not set
> >> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
> >> >> CONFIG_RCU_CPU_STALL_VERBOSE=y
> >> >>
> >> >> # grep _HZ /boot/config-$(uname -r)
> >> >> CONFIG_NO_HZ=y
> >> >> # CONFIG_HZ_100 is not set
> >> >> CONFIG_HZ_250=y
> >> >> # CONFIG_HZ_300 is not set
> >> >> # CONFIG_HZ_1000 is not set
> >> >> CONFIG_HZ=250
> >> >
> >> > OK, thank you for the info!
> >> >
> >>
> >> N.P.
> >>
> >> > Thanx, Paul
> >> >
> >>
> >> I guees I will revert step-by-step the RCU commits in linux-next and report.
> >> This weekend I wanted to enhance Debian's live-cd framework with
> >> overlayfs support and a customized kernel.
> >> But then came RCU :-(.
> >
> > Well, if it turns out to be a problem in RCU I will certainly apologize.
> >
>
> No, that's not so dramatic.
> Dealing with this RCU issue has nice side-effects: I remembered (and
> finally did) to use a reduced kernel-config set.
> The base for it I created with 'make localmodconfig' and did some
> manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
> etc.).
> Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
>
> So, I started building with
> "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
> I will let you know.
And please also check for tasks consuming all available CPU.
Thanx, Paul
> - Sedat -
>
> >> Can you say some words to kfree_rcu.2011.03.25b (rcu/kfree_rcu) GIT branch(es)?
> >
> > These are just applying Lia Jiangshan's kfree_rcu() to a number of places
> > in the kernel. You can safely ignore them.
> >
> > Thanx, Paul
> >
> >> - Sedat -
> >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-27 3:25 ` Paul E. McKenney
@ 2011-03-27 5:07 ` Paul E. McKenney
2011-03-27 12:26 ` Sedat Dilek
0 siblings, 1 reply; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-27 5:07 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
> > <paulmck@linux.vnet.ibm.com> wrote:
> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
[ . . . ]
> > >> But then came RCU :-(.
> > >
> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
> > >
> >
> > No, that's not so dramatic.
> > Dealing with this RCU issue has nice side-effects: I remembered (and
> > finally did) to use a reduced kernel-config set.
> > The base for it I created with 'make localmodconfig' and did some
> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
> > etc.).
> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
> >
> > So, I started building with
> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
> > I will let you know.
>
> And please also check for tasks consuming all available CPU.
And I still cannot reproduce with the full RCU stack (but based off of
2.6.38 rather than -next). Nevertheless, if you would like to try a
speculative patch, here you go.
Thanx, Paul
------------------------------------------------------------------------
rcu: further lower priority in rcu_yield()
Although rcu_yield() dropped from real-time to normal priority, there
is always the possibility that the competing tasks have been niced.
So nice to 19 in rcu_yield() to help ensure that other tasks have a
better chance of running.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 759f54b..5477764 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
mod_timer(&yield_timer, jiffies + 2);
sp.sched_priority = 0;
sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
+ set_user_nice(current, 19);
schedule();
sp.sched_priority = RCU_KTHREAD_PRIO;
sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
^ permalink raw reply related [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-27 5:07 ` Paul E. McKenney
@ 2011-03-27 12:26 ` Sedat Dilek
2011-03-27 21:32 ` Paul E. McKenney
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-27 12:26 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sun, Mar 27, 2011 at 7:07 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
>> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
>> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
>> > <paulmck@linux.vnet.ibm.com> wrote:
>> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
>
> [ . . . ]
>
>> > >> But then came RCU :-(.
>> > >
>> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
>> > >
>> >
>> > No, that's not so dramatic.
>> > Dealing with this RCU issue has nice side-effects: I remembered (and
>> > finally did) to use a reduced kernel-config set.
>> > The base for it I created with 'make localmodconfig' and did some
>> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
>> > etc.).
>> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
>> >
>> > So, I started building with
>> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
>> > I will let you know.
>>
>> And please also check for tasks consuming all available CPU.
>
> And I still cannot reproduce with the full RCU stack (but based off of
> 2.6.38 rather than -next). Nevertheless, if you would like to try a
> speculative patch, here you go.
>
You are right and my strategy on handling the (possible RCU?) issue is wrong.
Surely, you tested your RCU stuff in your own repo and everything
might be OK on top of stable 2.6.38.
Linux-next gets daily updates from a lot of different trees, so there
might be interferences with other stuff.
Please, understand I am interested in finding out what is the cause
for my issues, my aim is not to blame you.
As I was wrong and want to be 99.9% sure it is RCU stuff, I reverted
all (18) RCU patches from linux-next (next-20110325) by keeping the
RCU|PREEMPT|HZ settings from last working next-20110323.
$ egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110325-7-686-iniza
# RCU Subsystem
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_NO_HZ=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
I will work and stress this kernel before doing any step-by-step
revert of RCU stuff.
Thanks for your patch, I applied it on top of "naked" next-20110325,
but I still see call-traces.
- Sedat -
> Thanx, Paul
>
> ------------------------------------------------------------------------
>
> rcu: further lower priority in rcu_yield()
>
> Although rcu_yield() dropped from real-time to normal priority, there
> is always the possibility that the competing tasks have been niced.
> So nice to 19 in rcu_yield() to help ensure that other tasks have a
> better chance of running.
>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 759f54b..5477764 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
> mod_timer(&yield_timer, jiffies + 2);
> sp.sched_priority = 0;
> sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
> + set_user_nice(current, 19);
> schedule();
> sp.sched_priority = RCU_KTHREAD_PRIO;
> sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-27 12:26 ` Sedat Dilek
@ 2011-03-27 21:32 ` Paul E. McKenney
2011-03-27 21:48 ` Sedat Dilek
0 siblings, 1 reply; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-27 21:32 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sun, Mar 27, 2011 at 02:26:15PM +0200, Sedat Dilek wrote:
> On Sun, Mar 27, 2011 at 7:07 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
> >> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
> >> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
> >> > <paulmck@linux.vnet.ibm.com> wrote:
> >> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
> >
> > [ . . . ]
> >
> >> > >> But then came RCU :-(.
> >> > >
> >> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
> >> > >
> >> >
> >> > No, that's not so dramatic.
> >> > Dealing with this RCU issue has nice side-effects: I remembered (and
> >> > finally did) to use a reduced kernel-config set.
> >> > The base for it I created with 'make localmodconfig' and did some
> >> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
> >> > etc.).
> >> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
> >> >
> >> > So, I started building with
> >> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
> >> > I will let you know.
> >>
> >> And please also check for tasks consuming all available CPU.
> >
> > And I still cannot reproduce with the full RCU stack (but based off of
> > 2.6.38 rather than -next). Nevertheless, if you would like to try a
> > speculative patch, here you go.
>
> You are right and my strategy on handling the (possible RCU?) issue is wrong.
> Surely, you tested your RCU stuff in your own repo and everything
> might be OK on top of stable 2.6.38.
> Linux-next gets daily updates from a lot of different trees, so there
> might be interferences with other stuff.
> Please, understand I am interested in finding out what is the cause
> for my issues, my aim is not to blame you.
I am not worried about blame, but rather getting the bug fixed. The
bug might be in RCU, it might be elsewhere, or it might be a combination
of problems in RCU and elsewhere.
So the first priority is locating the bug.
And that is why I have been asking you over and over to PLEASE take
a look at what tasks are consuming CPU while the problem is occuring.
The reason that I have been asking over and over is that the symptoms
you describe are likely caused by a loop in some kernel code. Yes,
there might be other causes, but this is the most likely. Given that
TREE_PREEMPT_RCU behaves better than TREE_RCU, it is likely that this
loop is in preemptible code with irqs enabled. Therefore, the process
accounting code is likely to be able to see the CPU consumption, and
you should be able to see it via the "top" or "ps" commands -- or via
any number of other tools.
For example, if the problem is confined to RCU, you would likely see
the "rcuc0" or "rcun0" tasks consuming lots of CPU. This would narrow
the problem down to a few tens of lines of code. If the problem was
in some other kthread, then identifying the kthread would very likely
narrow things down as well.
So, please do take a look to see what taks consuming CPU.
> As I was wrong and want to be 99.9% sure it is RCU stuff, I reverted
> all (18) RCU patches from linux-next (next-20110325) by keeping the
> RCU|PREEMPT|HZ settings from last working next-20110323.
Makes sense.
> $ egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110325-7-686-iniza
> # RCU Subsystem
> CONFIG_TREE_RCU=y
> # CONFIG_PREEMPT_RCU is not set
> # CONFIG_RCU_TRACE is not set
> CONFIG_RCU_FANOUT=32
> # CONFIG_RCU_FANOUT_EXACT is not set
> CONFIG_RCU_FAST_NO_HZ=y
> # CONFIG_TREE_RCU_TRACE is not set
> CONFIG_PREEMPT_NOTIFIERS=y
> CONFIG_NO_HZ=y
> # CONFIG_PREEMPT_NONE is not set
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set
> # CONFIG_HZ_100 is not set
> CONFIG_HZ_250=y
> # CONFIG_HZ_300 is not set
> # CONFIG_HZ_1000 is not set
> CONFIG_HZ=250
> # CONFIG_SPARSE_RCU_POINTER is not set
> # CONFIG_RCU_TORTURE_TEST is not set
> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>
> I will work and stress this kernel before doing any step-by-step
> revert of RCU stuff.
>
> Thanks for your patch, I applied it on top of "naked" next-20110325,
> but I still see call-traces.
Thank you very much for testing it!
I intend to keep that patch, as it should increase robustness in other
situations.
Thanx, Paul
> - Sedat -
>
>
>
> > Thanx, Paul
> >
> > ------------------------------------------------------------------------
> >
> > rcu: further lower priority in rcu_yield()
> >
> > Although rcu_yield() dropped from real-time to normal priority, there
> > is always the possibility that the competing tasks have been niced.
> > So nice to 19 in rcu_yield() to help ensure that other tasks have a
> > better chance of running.
> >
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> >
> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> > index 759f54b..5477764 100644
> > --- a/kernel/rcutree.c
> > +++ b/kernel/rcutree.c
> > @@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
> > mod_timer(&yield_timer, jiffies + 2);
> > sp.sched_priority = 0;
> > sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
> > + set_user_nice(current, 19);
> > schedule();
> > sp.sched_priority = RCU_KTHREAD_PRIO;
> > sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
> >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-27 21:32 ` Paul E. McKenney
@ 2011-03-27 21:48 ` Sedat Dilek
2011-03-28 4:08 ` Paul E. McKenney
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-27 21:48 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sun, Mar 27, 2011 at 11:32 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Sun, Mar 27, 2011 at 02:26:15PM +0200, Sedat Dilek wrote:
>> On Sun, Mar 27, 2011 at 7:07 AM, Paul E. McKenney
>> <paulmck@linux.vnet.ibm.com> wrote:
>> > On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
>> >> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
>> >> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
>> >> > <paulmck@linux.vnet.ibm.com> wrote:
>> >> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
>> >
>> > [ . . . ]
>> >
>> >> > >> But then came RCU :-(.
>> >> > >
>> >> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
>> >> > >
>> >> >
>> >> > No, that's not so dramatic.
>> >> > Dealing with this RCU issue has nice side-effects: I remembered (and
>> >> > finally did) to use a reduced kernel-config set.
>> >> > The base for it I created with 'make localmodconfig' and did some
>> >> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
>> >> > etc.).
>> >> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
>> >> >
>> >> > So, I started building with
>> >> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
>> >> > I will let you know.
>> >>
>> >> And please also check for tasks consuming all available CPU.
>> >
>> > And I still cannot reproduce with the full RCU stack (but based off of
>> > 2.6.38 rather than -next). Nevertheless, if you would like to try a
>> > speculative patch, here you go.
>>
>> You are right and my strategy on handling the (possible RCU?) issue is wrong.
>> Surely, you tested your RCU stuff in your own repo and everything
>> might be OK on top of stable 2.6.38.
>> Linux-next gets daily updates from a lot of different trees, so there
>> might be interferences with other stuff.
>> Please, understand I am interested in finding out what is the cause
>> for my issues, my aim is not to blame you.
>
> I am not worried about blame, but rather getting the bug fixed. The
> bug might be in RCU, it might be elsewhere, or it might be a combination
> of problems in RCU and elsewhere.
>
> So the first priority is locating the bug.
>
> And that is why I have been asking you over and over to PLEASE take
> a look at what tasks are consuming CPU while the problem is occuring.
> The reason that I have been asking over and over is that the symptoms
> you describe are likely caused by a loop in some kernel code. Yes,
> there might be other causes, but this is the most likely. Given that
> TREE_PREEMPT_RCU behaves better than TREE_RCU, it is likely that this
> loop is in preemptible code with irqs enabled. Therefore, the process
> accounting code is likely to be able to see the CPU consumption, and
> you should be able to see it via the "top" or "ps" commands -- or via
> any number of other tools.
>
> For example, if the problem is confined to RCU, you would likely see
> the "rcuc0" or "rcun0" tasks consuming lots of CPU. This would narrow
> the problem down to a few tens of lines of code. If the problem was
> in some other kthread, then identifying the kthread would very likely
> narrow things down as well.
>
> So, please do take a look to see what taks consuming CPU.
>
>> As I was wrong and want to be 99.9% sure it is RCU stuff, I reverted
>> all (18) RCU patches from linux-next (next-20110325) by keeping the
>> RCU|PREEMPT|HZ settings from last working next-20110323.
>
> Makes sense.
>
>> $ egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110325-7-686-iniza
>> # RCU Subsystem
>> CONFIG_TREE_RCU=y
>> # CONFIG_PREEMPT_RCU is not set
>> # CONFIG_RCU_TRACE is not set
>> CONFIG_RCU_FANOUT=32
>> # CONFIG_RCU_FANOUT_EXACT is not set
>> CONFIG_RCU_FAST_NO_HZ=y
>> # CONFIG_TREE_RCU_TRACE is not set
>> CONFIG_PREEMPT_NOTIFIERS=y
>> CONFIG_NO_HZ=y
>> # CONFIG_PREEMPT_NONE is not set
>> CONFIG_PREEMPT_VOLUNTARY=y
>> # CONFIG_PREEMPT is not set
>> # CONFIG_HZ_100 is not set
>> CONFIG_HZ_250=y
>> # CONFIG_HZ_300 is not set
>> # CONFIG_HZ_1000 is not set
>> CONFIG_HZ=250
>> # CONFIG_SPARSE_RCU_POINTER is not set
>> # CONFIG_RCU_TORTURE_TEST is not set
>> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>>
>> I will work and stress this kernel before doing any step-by-step
>> revert of RCU stuff.
>>
>> Thanks for your patch, I applied it on top of "naked" next-20110325,
>> but I still see call-traces.
>
> Thank you very much for testing it!
>
> I intend to keep that patch, as it should increase robustness in other
> situations.
>
> Thanx, Paul
>
>> - Sedat -
>>
>>
>>
>> > Thanx, Paul
>> >
>> > ------------------------------------------------------------------------
>> >
>> > rcu: further lower priority in rcu_yield()
>> >
>> > Although rcu_yield() dropped from real-time to normal priority, there
>> > is always the possibility that the competing tasks have been niced.
>> > So nice to 19 in rcu_yield() to help ensure that other tasks have a
>> > better chance of running.
>> >
>> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>> >
>> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
>> > index 759f54b..5477764 100644
>> > --- a/kernel/rcutree.c
>> > +++ b/kernel/rcutree.c
>> > @@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
>> > mod_timer(&yield_timer, jiffies + 2);
>> > sp.sched_priority = 0;
>> > sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
>> > + set_user_nice(current, 19);
>> > schedule();
>> > sp.sched_priority = RCU_KTHREAD_PRIO;
>> > sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
>> >
>
Sorry, my attempt was to identify and isolate the culprit commit.
Reverting all RCU patches resulted in a stable system, the following 8
kernels with reduced k-config setup where all built using this kernel.
All kernels used TREE_RCU (see above), I did not change it (no
mixing/switching to PREEMPT and TREE_PREEMPT_RCU).
( I doubt that TREE_PREEMPT_RCU was some kind of more stable here. )
The culprit commit is bc56163ebd4580199ac7e63f5e160bf139ba0dd6 (from
rcu/next GIT tree):
"rcu: move TREE_RCU from softirq to kthread"
I can do parallelly a tar job, open 20 tabs in firefox and run a flash
video in one of them (I did this several times).
[ setup.log ]
...
(+) OK revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch
(+) OK revert-rcu-patches/0002-Revert-rcu-fix-spelling.patch
(+) OK revert-rcu-patches/0003-Revert-rcu-fix-rcu_cpu_kthread_task-synchronization.patch
(+) OK revert-rcu-patches/0004-Revert-rcu-call-__rcu_read_unlock-in-exit_rcu-for-tr.patch
(+) OK revert-rcu-patches/0005-Revert-rcu-Converge-TINY_RCU-expedited-and-normal-bo.patch
(+) OK revert-rcu-patches/0006-Revert-rcu-remove-useless-boosted_this_gp-field.patch
(+) OK revert-rcu-patches/0007-Revert-rcu-code-cleanups-in-TINY_RCU-priority-boosti.patch
(+) OK revert-rcu-patches/0008-Revert-rcu-Switch-to-this_cpu-primitives.patch
(+) OK revert-rcu-patches/0009-Revert-rcu-Use-WARN_ON_ONCE-for-DEBUG_OBJECTS_RCU_HE.patch
(+) OK revert-rcu-patches/0010-Revert-rcu-Enable-DEBUG_OBJECTS_RCU_HEAD-from-PREEMP.patch
(+) OK revert-rcu-patches/0011-Revert-rcu-Add-boosting-to-TREE_PREEMPT_RCU-tracing.patch
(+) OK revert-rcu-patches/0012-Revert-rcu-eliminate-unused-boosting-statistics.patch
(+) OK revert-rcu-patches/0013-Revert-rcu-priority-boosting-for-TREE_PREEMPT_RCU.patch
(+) OK revert-rcu-patches/0014-Revert-rcu-move-TREE_RCU-from-softirq-to-kthread.patch
...
Hope this helps to narrow down the problem.
As I kept all kernels I can have a look at the tasks consuming high
CPU usage tomorrow.
- Sedat -
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-27 21:48 ` Sedat Dilek
@ 2011-03-28 4:08 ` Paul E. McKenney
2011-03-28 12:33 ` Sedat Dilek
0 siblings, 1 reply; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-28 4:08 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Sun, Mar 27, 2011 at 11:48:30PM +0200, Sedat Dilek wrote:
> On Sun, Mar 27, 2011 at 11:32 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Sun, Mar 27, 2011 at 02:26:15PM +0200, Sedat Dilek wrote:
> >> On Sun, Mar 27, 2011 at 7:07 AM, Paul E. McKenney
> >> <paulmck@linux.vnet.ibm.com> wrote:
> >> > On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
> >> >> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
> >> >> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
> >> >> > <paulmck@linux.vnet.ibm.com> wrote:
> >> >> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
> >> >
> >> > [ . . . ]
> >> >
> >> >> > >> But then came RCU :-(.
> >> >> > >
> >> >> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
> >> >> > >
> >> >> >
> >> >> > No, that's not so dramatic.
> >> >> > Dealing with this RCU issue has nice side-effects: I remembered (and
> >> >> > finally did) to use a reduced kernel-config set.
> >> >> > The base for it I created with 'make localmodconfig' and did some
> >> >> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
> >> >> > etc.).
> >> >> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
> >> >> >
> >> >> > So, I started building with
> >> >> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
> >> >> > I will let you know.
> >> >>
> >> >> And please also check for tasks consuming all available CPU.
> >> >
> >> > And I still cannot reproduce with the full RCU stack (but based off of
> >> > 2.6.38 rather than -next). Nevertheless, if you would like to try a
> >> > speculative patch, here you go.
> >>
> >> You are right and my strategy on handling the (possible RCU?) issue is wrong.
> >> Surely, you tested your RCU stuff in your own repo and everything
> >> might be OK on top of stable 2.6.38.
> >> Linux-next gets daily updates from a lot of different trees, so there
> >> might be interferences with other stuff.
> >> Please, understand I am interested in finding out what is the cause
> >> for my issues, my aim is not to blame you.
> >
> > I am not worried about blame, but rather getting the bug fixed. The
> > bug might be in RCU, it might be elsewhere, or it might be a combination
> > of problems in RCU and elsewhere.
> >
> > So the first priority is locating the bug.
> >
> > And that is why I have been asking you over and over to PLEASE take
> > a look at what tasks are consuming CPU while the problem is occuring.
> > The reason that I have been asking over and over is that the symptoms
> > you describe are likely caused by a loop in some kernel code. Yes,
> > there might be other causes, but this is the most likely. Given that
> > TREE_PREEMPT_RCU behaves better than TREE_RCU, it is likely that this
> > loop is in preemptible code with irqs enabled. Therefore, the process
> > accounting code is likely to be able to see the CPU consumption, and
> > you should be able to see it via the "top" or "ps" commands -- or via
> > any number of other tools.
> >
> > For example, if the problem is confined to RCU, you would likely see
> > the "rcuc0" or "rcun0" tasks consuming lots of CPU. This would narrow
> > the problem down to a few tens of lines of code. If the problem was
> > in some other kthread, then identifying the kthread would very likely
> > narrow things down as well.
> >
> > So, please do take a look to see what taks consuming CPU.
> >
> >> As I was wrong and want to be 99.9% sure it is RCU stuff, I reverted
> >> all (18) RCU patches from linux-next (next-20110325) by keeping the
> >> RCU|PREEMPT|HZ settings from last working next-20110323.
> >
> > Makes sense.
> >
> >> $ egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110325-7-686-iniza
> >> # RCU Subsystem
> >> CONFIG_TREE_RCU=y
> >> # CONFIG_PREEMPT_RCU is not set
> >> # CONFIG_RCU_TRACE is not set
> >> CONFIG_RCU_FANOUT=32
> >> # CONFIG_RCU_FANOUT_EXACT is not set
> >> CONFIG_RCU_FAST_NO_HZ=y
> >> # CONFIG_TREE_RCU_TRACE is not set
> >> CONFIG_PREEMPT_NOTIFIERS=y
> >> CONFIG_NO_HZ=y
> >> # CONFIG_PREEMPT_NONE is not set
> >> CONFIG_PREEMPT_VOLUNTARY=y
> >> # CONFIG_PREEMPT is not set
> >> # CONFIG_HZ_100 is not set
> >> CONFIG_HZ_250=y
> >> # CONFIG_HZ_300 is not set
> >> # CONFIG_HZ_1000 is not set
> >> CONFIG_HZ=250
> >> # CONFIG_SPARSE_RCU_POINTER is not set
> >> # CONFIG_RCU_TORTURE_TEST is not set
> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >>
> >> I will work and stress this kernel before doing any step-by-step
> >> revert of RCU stuff.
> >>
> >> Thanks for your patch, I applied it on top of "naked" next-20110325,
> >> but I still see call-traces.
> >
> > Thank you very much for testing it!
> >
> > I intend to keep that patch, as it should increase robustness in other
> > situations.
> >
> > Thanx, Paul
> >
> >> - Sedat -
> >>
> >>
> >>
> >> > Thanx, Paul
> >> >
> >> > ------------------------------------------------------------------------
> >> >
> >> > rcu: further lower priority in rcu_yield()
> >> >
> >> > Although rcu_yield() dropped from real-time to normal priority, there
> >> > is always the possibility that the competing tasks have been niced.
> >> > So nice to 19 in rcu_yield() to help ensure that other tasks have a
> >> > better chance of running.
> >> >
> >> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> >> >
> >> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> >> > index 759f54b..5477764 100644
> >> > --- a/kernel/rcutree.c
> >> > +++ b/kernel/rcutree.c
> >> > @@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
> >> > mod_timer(&yield_timer, jiffies + 2);
> >> > sp.sched_priority = 0;
> >> > sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
> >> > + set_user_nice(current, 19);
> >> > schedule();
> >> > sp.sched_priority = RCU_KTHREAD_PRIO;
> >> > sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
>
> Sorry, my attempt was to identify and isolate the culprit commit.
>
> Reverting all RCU patches resulted in a stable system, the following 8
> kernels with reduced k-config setup where all built using this kernel.
>
> All kernels used TREE_RCU (see above), I did not change it (no
> mixing/switching to PREEMPT and TREE_PREEMPT_RCU).
> ( I doubt that TREE_PREEMPT_RCU was some kind of more stable here. )
>
> The culprit commit is bc56163ebd4580199ac7e63f5e160bf139ba0dd6 (from
> rcu/next GIT tree):
> "rcu: move TREE_RCU from softirq to kthread"
OK, please accept my apologies for your lost weekend. And thank you for
testing this.
> I can do parallelly a tar job, open 20 tabs in firefox and run a flash
> video in one of them (I did this several times).
How many files in the tar job? Is this creating a tar archive, expanding
it, or both?
Do you have a script for this? Are all of these running at normal
priority, or are some of them running at real-time priority?
> [ setup.log ]
> ...
> (+) OK revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch
> (+) OK revert-rcu-patches/0002-Revert-rcu-fix-spelling.patch
> (+) OK revert-rcu-patches/0003-Revert-rcu-fix-rcu_cpu_kthread_task-synchronization.patch
> (+) OK revert-rcu-patches/0004-Revert-rcu-call-__rcu_read_unlock-in-exit_rcu-for-tr.patch
> (+) OK revert-rcu-patches/0005-Revert-rcu-Converge-TINY_RCU-expedited-and-normal-bo.patch
> (+) OK revert-rcu-patches/0006-Revert-rcu-remove-useless-boosted_this_gp-field.patch
> (+) OK revert-rcu-patches/0007-Revert-rcu-code-cleanups-in-TINY_RCU-priority-boosti.patch
> (+) OK revert-rcu-patches/0008-Revert-rcu-Switch-to-this_cpu-primitives.patch
> (+) OK revert-rcu-patches/0009-Revert-rcu-Use-WARN_ON_ONCE-for-DEBUG_OBJECTS_RCU_HE.patch
> (+) OK revert-rcu-patches/0010-Revert-rcu-Enable-DEBUG_OBJECTS_RCU_HEAD-from-PREEMP.patch
> (+) OK revert-rcu-patches/0011-Revert-rcu-Add-boosting-to-TREE_PREEMPT_RCU-tracing.patch
> (+) OK revert-rcu-patches/0012-Revert-rcu-eliminate-unused-boosting-statistics.patch
> (+) OK revert-rcu-patches/0013-Revert-rcu-priority-boosting-for-TREE_PREEMPT_RCU.patch
> (+) OK revert-rcu-patches/0014-Revert-rcu-move-TREE_RCU-from-softirq-to-kthread.patch
> ...
>
> Hope this helps to narrow down the problem.
>
> As I kept all kernels I can have a look at the tasks consuming high
> CPU usage tomorrow.
Could you please?
Also, could you please mount debugfs and list out the files in the
"rcu" directory? The "ql=" value from the "rcu/rcudata" file is of
particular interest.
Thanx, Paul
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-28 4:08 ` Paul E. McKenney
@ 2011-03-28 12:33 ` Sedat Dilek
2011-03-28 13:24 ` Paul E. McKenney
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-28 12:33 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Mon, Mar 28, 2011 at 6:08 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Sun, Mar 27, 2011 at 11:48:30PM +0200, Sedat Dilek wrote:
>> On Sun, Mar 27, 2011 at 11:32 PM, Paul E. McKenney
>> <paulmck@linux.vnet.ibm.com> wrote:
>> > On Sun, Mar 27, 2011 at 02:26:15PM +0200, Sedat Dilek wrote:
>> >> On Sun, Mar 27, 2011 at 7:07 AM, Paul E. McKenney
>> >> <paulmck@linux.vnet.ibm.com> wrote:
>> >> > On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
>> >> >> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
>> >> >> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
>> >> >> > <paulmck@linux.vnet.ibm.com> wrote:
>> >> >> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
>> >> >
>> >> > [ . . . ]
>> >> >
>> >> >> > >> But then came RCU :-(.
>> >> >> > >
>> >> >> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
>> >> >> > >
>> >> >> >
>> >> >> > No, that's not so dramatic.
>> >> >> > Dealing with this RCU issue has nice side-effects: I remembered (and
>> >> >> > finally did) to use a reduced kernel-config set.
>> >> >> > The base for it I created with 'make localmodconfig' and did some
>> >> >> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
>> >> >> > etc.).
>> >> >> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
>> >> >> >
>> >> >> > So, I started building with
>> >> >> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
>> >> >> > I will let you know.
>> >> >>
>> >> >> And please also check for tasks consuming all available CPU.
>> >> >
>> >> > And I still cannot reproduce with the full RCU stack (but based off of
>> >> > 2.6.38 rather than -next). Nevertheless, if you would like to try a
>> >> > speculative patch, here you go.
>> >>
>> >> You are right and my strategy on handling the (possible RCU?) issue is wrong.
>> >> Surely, you tested your RCU stuff in your own repo and everything
>> >> might be OK on top of stable 2.6.38.
>> >> Linux-next gets daily updates from a lot of different trees, so there
>> >> might be interferences with other stuff.
>> >> Please, understand I am interested in finding out what is the cause
>> >> for my issues, my aim is not to blame you.
>> >
>> > I am not worried about blame, but rather getting the bug fixed. The
>> > bug might be in RCU, it might be elsewhere, or it might be a combination
>> > of problems in RCU and elsewhere.
>> >
>> > So the first priority is locating the bug.
>> >
>> > And that is why I have been asking you over and over to PLEASE take
>> > a look at what tasks are consuming CPU while the problem is occuring.
>> > The reason that I have been asking over and over is that the symptoms
>> > you describe are likely caused by a loop in some kernel code. Yes,
>> > there might be other causes, but this is the most likely. Given that
>> > TREE_PREEMPT_RCU behaves better than TREE_RCU, it is likely that this
>> > loop is in preemptible code with irqs enabled. Therefore, the process
>> > accounting code is likely to be able to see the CPU consumption, and
>> > you should be able to see it via the "top" or "ps" commands -- or via
>> > any number of other tools.
>> >
>> > For example, if the problem is confined to RCU, you would likely see
>> > the "rcuc0" or "rcun0" tasks consuming lots of CPU. This would narrow
>> > the problem down to a few tens of lines of code. If the problem was
>> > in some other kthread, then identifying the kthread would very likely
>> > narrow things down as well.
>> >
>> > So, please do take a look to see what taks consuming CPU.
>> >
>> >> As I was wrong and want to be 99.9% sure it is RCU stuff, I reverted
>> >> all (18) RCU patches from linux-next (next-20110325) by keeping the
>> >> RCU|PREEMPT|HZ settings from last working next-20110323.
>> >
>> > Makes sense.
>> >
>> >> $ egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110325-7-686-iniza
>> >> # RCU Subsystem
>> >> CONFIG_TREE_RCU=y
>> >> # CONFIG_PREEMPT_RCU is not set
>> >> # CONFIG_RCU_TRACE is not set
>> >> CONFIG_RCU_FANOUT=32
>> >> # CONFIG_RCU_FANOUT_EXACT is not set
>> >> CONFIG_RCU_FAST_NO_HZ=y
>> >> # CONFIG_TREE_RCU_TRACE is not set
>> >> CONFIG_PREEMPT_NOTIFIERS=y
>> >> CONFIG_NO_HZ=y
>> >> # CONFIG_PREEMPT_NONE is not set
>> >> CONFIG_PREEMPT_VOLUNTARY=y
>> >> # CONFIG_PREEMPT is not set
>> >> # CONFIG_HZ_100 is not set
>> >> CONFIG_HZ_250=y
>> >> # CONFIG_HZ_300 is not set
>> >> # CONFIG_HZ_1000 is not set
>> >> CONFIG_HZ=250
>> >> # CONFIG_SPARSE_RCU_POINTER is not set
>> >> # CONFIG_RCU_TORTURE_TEST is not set
>> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>> >>
>> >> I will work and stress this kernel before doing any step-by-step
>> >> revert of RCU stuff.
>> >>
>> >> Thanks for your patch, I applied it on top of "naked" next-20110325,
>> >> but I still see call-traces.
>> >
>> > Thank you very much for testing it!
>> >
>> > I intend to keep that patch, as it should increase robustness in other
>> > situations.
>> >
>> > Thanx, Paul
>> >
>> >> - Sedat -
>> >>
>> >>
>> >>
>> >> > Thanx, Paul
>> >> >
>> >> > ------------------------------------------------------------------------
>> >> >
>> >> > rcu: further lower priority in rcu_yield()
>> >> >
>> >> > Although rcu_yield() dropped from real-time to normal priority, there
>> >> > is always the possibility that the competing tasks have been niced.
>> >> > So nice to 19 in rcu_yield() to help ensure that other tasks have a
>> >> > better chance of running.
>> >> >
>> >> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>> >> >
>> >> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
>> >> > index 759f54b..5477764 100644
>> >> > --- a/kernel/rcutree.c
>> >> > +++ b/kernel/rcutree.c
>> >> > @@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
>> >> > mod_timer(&yield_timer, jiffies + 2);
>> >> > sp.sched_priority = 0;
>> >> > sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
>> >> > + set_user_nice(current, 19);
>> >> > schedule();
>> >> > sp.sched_priority = RCU_KTHREAD_PRIO;
>> >> > sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
>>
>> Sorry, my attempt was to identify and isolate the culprit commit.
>>
>> Reverting all RCU patches resulted in a stable system, the following 8
>> kernels with reduced k-config setup where all built using this kernel.
>>
>> All kernels used TREE_RCU (see above), I did not change it (no
>> mixing/switching to PREEMPT and TREE_PREEMPT_RCU).
>> ( I doubt that TREE_PREEMPT_RCU was some kind of more stable here. )
>>
>> The culprit commit is bc56163ebd4580199ac7e63f5e160bf139ba0dd6 (from
>> rcu/next GIT tree):
>> "rcu: move TREE_RCU from softirq to kthread"
>
Hi Paul,
> OK, please accept my apologies for your lost weekend. And thank you for
> testing this.
>
No worries, it was mostly a rainy day.
The only thing I did @ 16:30 was to go to regional election (the new
(regional) prime minister will be the 1st from The German Green
party).
But back to RCU :-):
The reduced kernel-config setup decreased the build-time from approx.
2hrs (full, generic build) down to approx. 35mins.
>> I can do parallelly a tar job, open 20 tabs in firefox and run a flash
>> video in one of them (I did this several times).
>
> How many files in the tar job? Is this creating a tar archive, expanding
> it, or both?
>
I am doing a simple tar (filesize: 1.6G for full and 1.0G for reduced build):
$ tar -cf $archivedir-on-external-usbhdd/$tarfile $kernel-build-dir
...plus parallelly opening 20 tabs in firefox.
That's normally enough to get my system freaky and see RCU related
messages in the logs.
> Do you have a script for this? Are all of these running at normal
> priority, or are some of them running at real-time priority?
>
Nothing special.
>> [ setup.log ]
>> ...
>> (+) OK revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch
>> (+) OK revert-rcu-patches/0002-Revert-rcu-fix-spelling.patch
>> (+) OK revert-rcu-patches/0003-Revert-rcu-fix-rcu_cpu_kthread_task-synchronization.patch
>> (+) OK revert-rcu-patches/0004-Revert-rcu-call-__rcu_read_unlock-in-exit_rcu-for-tr.patch
>> (+) OK revert-rcu-patches/0005-Revert-rcu-Converge-TINY_RCU-expedited-and-normal-bo.patch
>> (+) OK revert-rcu-patches/0006-Revert-rcu-remove-useless-boosted_this_gp-field.patch
>> (+) OK revert-rcu-patches/0007-Revert-rcu-code-cleanups-in-TINY_RCU-priority-boosti.patch
>> (+) OK revert-rcu-patches/0008-Revert-rcu-Switch-to-this_cpu-primitives.patch
>> (+) OK revert-rcu-patches/0009-Revert-rcu-Use-WARN_ON_ONCE-for-DEBUG_OBJECTS_RCU_HE.patch
>> (+) OK revert-rcu-patches/0010-Revert-rcu-Enable-DEBUG_OBJECTS_RCU_HEAD-from-PREEMP.patch
>> (+) OK revert-rcu-patches/0011-Revert-rcu-Add-boosting-to-TREE_PREEMPT_RCU-tracing.patch
>> (+) OK revert-rcu-patches/0012-Revert-rcu-eliminate-unused-boosting-statistics.patch
>> (+) OK revert-rcu-patches/0013-Revert-rcu-priority-boosting-for-TREE_PREEMPT_RCU.patch
>> (+) OK revert-rcu-patches/0014-Revert-rcu-move-TREE_RCU-from-softirq-to-kthread.patch
>> ...
>>
>> Hope this helps to narrow down the problem.
>>
>> As I kept all kernels I can have a look at the tasks consuming high
>> CPU usage tomorrow.
>
> Could you please?
>
I recalled (as you say I requested over and over again from you :-)) I
looked with top, htop and 'ps axu', but there was nothing special.
Sometimes the system got frozen - at this point (or short before) I
did not see anything suspicious with top.
> Also, could you please mount debugfs and list out the files in the
> "rcu" directory? The "ql=" value from the "rcu/rcudata" file is of
> particular interest.
>
Ah, before I forget...
I used TREE_RCU (was the default before noticing RCU issue) for
finding the culprit commit.
If it is from your POV more helpful to switch to PREEMPT + PREEMPT_RCU
+ RCU_BOOST, please let me *now* know.
( Both RCU setups freaks up the system. )
I think top & Co. are not enough to track the problem down.
I have seen tracing and debugging facililities for RCU.
Some questions to debug and trace setup:
Case #1: TREE_RCU
CONFIG_RCU_TRACE=y
CONFIG_TREE_RCU_TRACE=y
Case #2: PREEMPT + PREEMPT_RCU + RCU_BOOST
CONFIG_RCU_TRACE=y
CONFIG_TREE_RCU_TRACE=y
CONFIG_DEBUG_PREEMPT=y <--- Helpful?
CONFIG_PREEMPT_TRACER=y <--- Helpful?
Any other recommends for useful/helpful trace and/or debug options?
Any other intructions for debugging/tracing?
BTW, today's linux-next (next-20110328) is still freaky, I applied the
revert-rcu-patches patchset and all is fine.
- Sedat -
P.S.: Note to myself
# mount -t debugfs none /sys/kernel/debug/
# ln -s /sys/kernel/debug /debug
# find /debug -name rcu
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-28 12:33 ` Sedat Dilek
@ 2011-03-28 13:24 ` Paul E. McKenney
2011-03-28 15:11 ` Paul E. McKenney
2011-03-29 4:39 ` Sedat Dilek
0 siblings, 2 replies; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-28 13:24 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Mon, Mar 28, 2011 at 02:33:36PM +0200, Sedat Dilek wrote:
> On Mon, Mar 28, 2011 at 6:08 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Sun, Mar 27, 2011 at 11:48:30PM +0200, Sedat Dilek wrote:
> >> On Sun, Mar 27, 2011 at 11:32 PM, Paul E. McKenney
> >> <paulmck@linux.vnet.ibm.com> wrote:
> >> > On Sun, Mar 27, 2011 at 02:26:15PM +0200, Sedat Dilek wrote:
> >> >> On Sun, Mar 27, 2011 at 7:07 AM, Paul E. McKenney
> >> >> <paulmck@linux.vnet.ibm.com> wrote:
> >> >> > On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
> >> >> >> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
> >> >> >> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
> >> >> >> > <paulmck@linux.vnet.ibm.com> wrote:
> >> >> >> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
> >> >> >
> >> >> > [ . . . ]
> >> >> >
> >> >> >> > >> But then came RCU :-(.
> >> >> >> > >
> >> >> >> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
> >> >> >> > >
> >> >> >> >
> >> >> >> > No, that's not so dramatic.
> >> >> >> > Dealing with this RCU issue has nice side-effects: I remembered (and
> >> >> >> > finally did) to use a reduced kernel-config set.
> >> >> >> > The base for it I created with 'make localmodconfig' and did some
> >> >> >> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
> >> >> >> > etc.).
> >> >> >> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
> >> >> >> >
> >> >> >> > So, I started building with
> >> >> >> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
> >> >> >> > I will let you know.
> >> >> >>
> >> >> >> And please also check for tasks consuming all available CPU.
> >> >> >
> >> >> > And I still cannot reproduce with the full RCU stack (but based off of
> >> >> > 2.6.38 rather than -next). Nevertheless, if you would like to try a
> >> >> > speculative patch, here you go.
> >> >>
> >> >> You are right and my strategy on handling the (possible RCU?) issue is wrong.
> >> >> Surely, you tested your RCU stuff in your own repo and everything
> >> >> might be OK on top of stable 2.6.38.
> >> >> Linux-next gets daily updates from a lot of different trees, so there
> >> >> might be interferences with other stuff.
> >> >> Please, understand I am interested in finding out what is the cause
> >> >> for my issues, my aim is not to blame you.
> >> >
> >> > I am not worried about blame, but rather getting the bug fixed. The
> >> > bug might be in RCU, it might be elsewhere, or it might be a combination
> >> > of problems in RCU and elsewhere.
> >> >
> >> > So the first priority is locating the bug.
> >> >
> >> > And that is why I have been asking you over and over to PLEASE take
> >> > a look at what tasks are consuming CPU while the problem is occuring.
> >> > The reason that I have been asking over and over is that the symptoms
> >> > you describe are likely caused by a loop in some kernel code. Yes,
> >> > there might be other causes, but this is the most likely. Given that
> >> > TREE_PREEMPT_RCU behaves better than TREE_RCU, it is likely that this
> >> > loop is in preemptible code with irqs enabled. Therefore, the process
> >> > accounting code is likely to be able to see the CPU consumption, and
> >> > you should be able to see it via the "top" or "ps" commands -- or via
> >> > any number of other tools.
> >> >
> >> > For example, if the problem is confined to RCU, you would likely see
> >> > the "rcuc0" or "rcun0" tasks consuming lots of CPU. This would narrow
> >> > the problem down to a few tens of lines of code. If the problem was
> >> > in some other kthread, then identifying the kthread would very likely
> >> > narrow things down as well.
> >> >
> >> > So, please do take a look to see what taks consuming CPU.
> >> >
> >> >> As I was wrong and want to be 99.9% sure it is RCU stuff, I reverted
> >> >> all (18) RCU patches from linux-next (next-20110325) by keeping the
> >> >> RCU|PREEMPT|HZ settings from last working next-20110323.
> >> >
> >> > Makes sense.
> >> >
> >> >> $ egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110325-7-686-iniza
> >> >> # RCU Subsystem
> >> >> CONFIG_TREE_RCU=y
> >> >> # CONFIG_PREEMPT_RCU is not set
> >> >> # CONFIG_RCU_TRACE is not set
> >> >> CONFIG_RCU_FANOUT=32
> >> >> # CONFIG_RCU_FANOUT_EXACT is not set
> >> >> CONFIG_RCU_FAST_NO_HZ=y
> >> >> # CONFIG_TREE_RCU_TRACE is not set
> >> >> CONFIG_PREEMPT_NOTIFIERS=y
> >> >> CONFIG_NO_HZ=y
> >> >> # CONFIG_PREEMPT_NONE is not set
> >> >> CONFIG_PREEMPT_VOLUNTARY=y
> >> >> # CONFIG_PREEMPT is not set
> >> >> # CONFIG_HZ_100 is not set
> >> >> CONFIG_HZ_250=y
> >> >> # CONFIG_HZ_300 is not set
> >> >> # CONFIG_HZ_1000 is not set
> >> >> CONFIG_HZ=250
> >> >> # CONFIG_SPARSE_RCU_POINTER is not set
> >> >> # CONFIG_RCU_TORTURE_TEST is not set
> >> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> >> >>
> >> >> I will work and stress this kernel before doing any step-by-step
> >> >> revert of RCU stuff.
> >> >>
> >> >> Thanks for your patch, I applied it on top of "naked" next-20110325,
> >> >> but I still see call-traces.
> >> >
> >> > Thank you very much for testing it!
> >> >
> >> > I intend to keep that patch, as it should increase robustness in other
> >> > situations.
> >> >
> >> > Thanx, Paul
> >> >
> >> >> - Sedat -
> >> >>
> >> >>
> >> >>
> >> >> > Thanx, Paul
> >> >> >
> >> >> > ------------------------------------------------------------------------
> >> >> >
> >> >> > rcu: further lower priority in rcu_yield()
> >> >> >
> >> >> > Although rcu_yield() dropped from real-time to normal priority, there
> >> >> > is always the possibility that the competing tasks have been niced.
> >> >> > So nice to 19 in rcu_yield() to help ensure that other tasks have a
> >> >> > better chance of running.
> >> >> >
> >> >> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> >> >> >
> >> >> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> >> >> > index 759f54b..5477764 100644
> >> >> > --- a/kernel/rcutree.c
> >> >> > +++ b/kernel/rcutree.c
> >> >> > @@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
> >> >> > mod_timer(&yield_timer, jiffies + 2);
> >> >> > sp.sched_priority = 0;
> >> >> > sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
> >> >> > + set_user_nice(current, 19);
> >> >> > schedule();
> >> >> > sp.sched_priority = RCU_KTHREAD_PRIO;
> >> >> > sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
> >>
> >> Sorry, my attempt was to identify and isolate the culprit commit.
> >>
> >> Reverting all RCU patches resulted in a stable system, the following 8
> >> kernels with reduced k-config setup where all built using this kernel.
> >>
> >> All kernels used TREE_RCU (see above), I did not change it (no
> >> mixing/switching to PREEMPT and TREE_PREEMPT_RCU).
> >> ( I doubt that TREE_PREEMPT_RCU was some kind of more stable here. )
> >>
> >> The culprit commit is bc56163ebd4580199ac7e63f5e160bf139ba0dd6 (from
> >> rcu/next GIT tree):
> >> "rcu: move TREE_RCU from softirq to kthread"
> >
>
> Hi Paul,
>
> > OK, please accept my apologies for your lost weekend. And thank you for
> > testing this.
> >
>
> No worries, it was mostly a rainy day.
> The only thing I did @ 16:30 was to go to regional election (the new
> (regional) prime minister will be the 1st from The German Green
> party).
;-)
> But back to RCU :-):
> The reduced kernel-config setup decreased the build-time from approx.
> 2hrs (full, generic build) down to approx. 35mins.
Very good!
> >> I can do parallelly a tar job, open 20 tabs in firefox and run a flash
> >> video in one of them (I did this several times).
> >
> > How many files in the tar job? Is this creating a tar archive, expanding
> > it, or both?
>
> I am doing a simple tar (filesize: 1.6G for full and 1.0G for reduced build):
>
> $ tar -cf $archivedir-on-external-usbhdd/$tarfile $kernel-build-dir
OK, I was extracting a tarball and then removing the resulting source
tree. I will try this. Though it does seem strange -- I can understand
how removing a file tree would stress RCU, but not creating a tarball.
Ah, well, if I fully understood it, there would not be a bug.
> ...plus parallelly opening 20 tabs in firefox.
> That's normally enough to get my system freaky and see RCU related
> messages in the logs.
Hmmm... My normal test systems don't have X -- I will need to set
this up.
> > Do you have a script for this? Are all of these running at normal
> > priority, or are some of them running at real-time priority?
> >
>
> Nothing special.
OK.
> >> [ setup.log ]
> >> ...
> >> (+) OK revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch
> >> (+) OK revert-rcu-patches/0002-Revert-rcu-fix-spelling.patch
> >> (+) OK revert-rcu-patches/0003-Revert-rcu-fix-rcu_cpu_kthread_task-synchronization.patch
> >> (+) OK revert-rcu-patches/0004-Revert-rcu-call-__rcu_read_unlock-in-exit_rcu-for-tr.patch
> >> (+) OK revert-rcu-patches/0005-Revert-rcu-Converge-TINY_RCU-expedited-and-normal-bo.patch
> >> (+) OK revert-rcu-patches/0006-Revert-rcu-remove-useless-boosted_this_gp-field.patch
> >> (+) OK revert-rcu-patches/0007-Revert-rcu-code-cleanups-in-TINY_RCU-priority-boosti.patch
> >> (+) OK revert-rcu-patches/0008-Revert-rcu-Switch-to-this_cpu-primitives.patch
> >> (+) OK revert-rcu-patches/0009-Revert-rcu-Use-WARN_ON_ONCE-for-DEBUG_OBJECTS_RCU_HE.patch
> >> (+) OK revert-rcu-patches/0010-Revert-rcu-Enable-DEBUG_OBJECTS_RCU_HEAD-from-PREEMP.patch
> >> (+) OK revert-rcu-patches/0011-Revert-rcu-Add-boosting-to-TREE_PREEMPT_RCU-tracing.patch
> >> (+) OK revert-rcu-patches/0012-Revert-rcu-eliminate-unused-boosting-statistics.patch
> >> (+) OK revert-rcu-patches/0013-Revert-rcu-priority-boosting-for-TREE_PREEMPT_RCU.patch
> >> (+) OK revert-rcu-patches/0014-Revert-rcu-move-TREE_RCU-from-softirq-to-kthread.patch
> >> ...
> >>
> >> Hope this helps to narrow down the problem.
> >>
> >> As I kept all kernels I can have a look at the tasks consuming high
> >> CPU usage tomorrow.
> >
> > Could you please?
>
> I recalled (as you say I requested over and over again from you :-)) I
> looked with top, htop and 'ps axu', but there was nothing special.
> Sometimes the system got frozen - at this point (or short before) I
> did not see anything suspicious with top.
OK, thank you for the info.
> > Also, could you please mount debugfs and list out the files in the
> > "rcu" directory? The "ql=" value from the "rcu/rcudata" file is of
> > particular interest.
> >
>
> Ah, before I forget...
>
> I used TREE_RCU (was the default before noticing RCU issue) for
> finding the culprit commit.
> If it is from your POV more helpful to switch to PREEMPT + PREEMPT_RCU
> + RCU_BOOST, please let me *now* know.
> ( Both RCU setups freaks up the system. )
If TREE_RCU hits problems faster, it is probably best to stay with
TREE_RCU.
> I think top & Co. are not enough to track the problem down.
> I have seen tracing and debugging facililities for RCU.
>
> Some questions to debug and trace setup:
>
> Case #1: TREE_RCU
>
> CONFIG_RCU_TRACE=y
> CONFIG_TREE_RCU_TRACE=y
Yep.
> Case #2: PREEMPT + PREEMPT_RCU + RCU_BOOST
>
> CONFIG_RCU_TRACE=y
> CONFIG_TREE_RCU_TRACE=y
> CONFIG_DEBUG_PREEMPT=y <--- Helpful?
> CONFIG_PREEMPT_TRACER=y <--- Helpful?
>
> Any other recommends for useful/helpful trace and/or debug options?
>
> Any other intructions for debugging/tracing?
Not at the moment. I will be looking at diagnostics will going
through the code, so might have something later.
> BTW, today's linux-next (next-20110328) is still freaky, I applied the
> revert-rcu-patches patchset and all is fine.
I reverted back to the commit preceding the one you pointed out last night
my time, so the upcoming -next should be less freaky.
> - Sedat -
>
> P.S.: Note to myself
>
> # mount -t debugfs none /sys/kernel/debug/
> # ln -s /sys/kernel/debug /debug
>
> # find /debug -name rcu
Or:
# cd /debug/rcu
then dump out everything except for the .csv file (which is the same
as the non-.csv equivalent, but in spreadsheet format -- intended
for systems with 100s or 1000s of CPUs).
Thanx, Paul
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-28 13:24 ` Paul E. McKenney
@ 2011-03-28 15:11 ` Paul E. McKenney
2011-03-28 16:38 ` Sedat Dilek
2011-03-29 4:39 ` Sedat Dilek
1 sibling, 1 reply; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-28 15:11 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Mon, Mar 28, 2011 at 06:24:36AM -0700, Paul E. McKenney wrote:
> On Mon, Mar 28, 2011 at 02:33:36PM +0200, Sedat Dilek wrote:
> > On Mon, Mar 28, 2011 at 6:08 AM, Paul E. McKenney
> > <paulmck@linux.vnet.ibm.com> wrote:
> > > On Sun, Mar 27, 2011 at 11:48:30PM +0200, Sedat Dilek wrote:
> > >> On Sun, Mar 27, 2011 at 11:32 PM, Paul E. McKenney
> > >> <paulmck@linux.vnet.ibm.com> wrote:
> > >> > On Sun, Mar 27, 2011 at 02:26:15PM +0200, Sedat Dilek wrote:
> > >> >> On Sun, Mar 27, 2011 at 7:07 AM, Paul E. McKenney
> > >> >> <paulmck@linux.vnet.ibm.com> wrote:
> > >> >> > On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
> > >> >> >> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
> > >> >> >> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
> > >> >> >> > <paulmck@linux.vnet.ibm.com> wrote:
> > >> >> >> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
> > >> >> >
> > >> >> > [ . . . ]
> > >> >> >
> > >> >> >> > >> But then came RCU :-(.
> > >> >> >> > >
> > >> >> >> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
> > >> >> >> > >
> > >> >> >> >
> > >> >> >> > No, that's not so dramatic.
> > >> >> >> > Dealing with this RCU issue has nice side-effects: I remembered (and
> > >> >> >> > finally did) to use a reduced kernel-config set.
> > >> >> >> > The base for it I created with 'make localmodconfig' and did some
> > >> >> >> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
> > >> >> >> > etc.).
> > >> >> >> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
> > >> >> >> >
> > >> >> >> > So, I started building with
> > >> >> >> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
> > >> >> >> > I will let you know.
> > >> >> >>
> > >> >> >> And please also check for tasks consuming all available CPU.
> > >> >> >
> > >> >> > And I still cannot reproduce with the full RCU stack (but based off of
> > >> >> > 2.6.38 rather than -next). Nevertheless, if you would like to try a
> > >> >> > speculative patch, here you go.
> > >> >>
> > >> >> You are right and my strategy on handling the (possible RCU?) issue is wrong.
> > >> >> Surely, you tested your RCU stuff in your own repo and everything
> > >> >> might be OK on top of stable 2.6.38.
> > >> >> Linux-next gets daily updates from a lot of different trees, so there
> > >> >> might be interferences with other stuff.
> > >> >> Please, understand I am interested in finding out what is the cause
> > >> >> for my issues, my aim is not to blame you.
> > >> >
> > >> > I am not worried about blame, but rather getting the bug fixed. The
> > >> > bug might be in RCU, it might be elsewhere, or it might be a combination
> > >> > of problems in RCU and elsewhere.
> > >> >
> > >> > So the first priority is locating the bug.
> > >> >
> > >> > And that is why I have been asking you over and over to PLEASE take
> > >> > a look at what tasks are consuming CPU while the problem is occuring.
> > >> > The reason that I have been asking over and over is that the symptoms
> > >> > you describe are likely caused by a loop in some kernel code. Yes,
> > >> > there might be other causes, but this is the most likely. Given that
> > >> > TREE_PREEMPT_RCU behaves better than TREE_RCU, it is likely that this
> > >> > loop is in preemptible code with irqs enabled. Therefore, the process
> > >> > accounting code is likely to be able to see the CPU consumption, and
> > >> > you should be able to see it via the "top" or "ps" commands -- or via
> > >> > any number of other tools.
> > >> >
> > >> > For example, if the problem is confined to RCU, you would likely see
> > >> > the "rcuc0" or "rcun0" tasks consuming lots of CPU. This would narrow
> > >> > the problem down to a few tens of lines of code. If the problem was
> > >> > in some other kthread, then identifying the kthread would very likely
> > >> > narrow things down as well.
> > >> >
> > >> > So, please do take a look to see what taks consuming CPU.
> > >> >
> > >> >> As I was wrong and want to be 99.9% sure it is RCU stuff, I reverted
> > >> >> all (18) RCU patches from linux-next (next-20110325) by keeping the
> > >> >> RCU|PREEMPT|HZ settings from last working next-20110323.
> > >> >
> > >> > Makes sense.
> > >> >
> > >> >> $ egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110325-7-686-iniza
> > >> >> # RCU Subsystem
> > >> >> CONFIG_TREE_RCU=y
> > >> >> # CONFIG_PREEMPT_RCU is not set
> > >> >> # CONFIG_RCU_TRACE is not set
> > >> >> CONFIG_RCU_FANOUT=32
> > >> >> # CONFIG_RCU_FANOUT_EXACT is not set
> > >> >> CONFIG_RCU_FAST_NO_HZ=y
> > >> >> # CONFIG_TREE_RCU_TRACE is not set
> > >> >> CONFIG_PREEMPT_NOTIFIERS=y
> > >> >> CONFIG_NO_HZ=y
> > >> >> # CONFIG_PREEMPT_NONE is not set
> > >> >> CONFIG_PREEMPT_VOLUNTARY=y
> > >> >> # CONFIG_PREEMPT is not set
> > >> >> # CONFIG_HZ_100 is not set
> > >> >> CONFIG_HZ_250=y
> > >> >> # CONFIG_HZ_300 is not set
> > >> >> # CONFIG_HZ_1000 is not set
> > >> >> CONFIG_HZ=250
> > >> >> # CONFIG_SPARSE_RCU_POINTER is not set
> > >> >> # CONFIG_RCU_TORTURE_TEST is not set
> > >> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> > >> >>
> > >> >> I will work and stress this kernel before doing any step-by-step
> > >> >> revert of RCU stuff.
> > >> >>
> > >> >> Thanks for your patch, I applied it on top of "naked" next-20110325,
> > >> >> but I still see call-traces.
> > >> >
> > >> > Thank you very much for testing it!
> > >> >
> > >> > I intend to keep that patch, as it should increase robustness in other
> > >> > situations.
> > >> >
> > >> > Thanx, Paul
> > >> >
> > >> >> - Sedat -
> > >> >>
> > >> >>
> > >> >>
> > >> >> > Thanx, Paul
> > >> >> >
> > >> >> > ------------------------------------------------------------------------
> > >> >> >
> > >> >> > rcu: further lower priority in rcu_yield()
> > >> >> >
> > >> >> > Although rcu_yield() dropped from real-time to normal priority, there
> > >> >> > is always the possibility that the competing tasks have been niced.
> > >> >> > So nice to 19 in rcu_yield() to help ensure that other tasks have a
> > >> >> > better chance of running.
> > >> >> >
> > >> >> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > >> >> >
> > >> >> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> > >> >> > index 759f54b..5477764 100644
> > >> >> > --- a/kernel/rcutree.c
> > >> >> > +++ b/kernel/rcutree.c
> > >> >> > @@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
> > >> >> > mod_timer(&yield_timer, jiffies + 2);
> > >> >> > sp.sched_priority = 0;
> > >> >> > sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
> > >> >> > + set_user_nice(current, 19);
> > >> >> > schedule();
> > >> >> > sp.sched_priority = RCU_KTHREAD_PRIO;
> > >> >> > sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
> > >>
> > >> Sorry, my attempt was to identify and isolate the culprit commit.
> > >>
> > >> Reverting all RCU patches resulted in a stable system, the following 8
> > >> kernels with reduced k-config setup where all built using this kernel.
> > >>
> > >> All kernels used TREE_RCU (see above), I did not change it (no
> > >> mixing/switching to PREEMPT and TREE_PREEMPT_RCU).
> > >> ( I doubt that TREE_PREEMPT_RCU was some kind of more stable here. )
> > >>
> > >> The culprit commit is bc56163ebd4580199ac7e63f5e160bf139ba0dd6 (from
> > >> rcu/next GIT tree):
> > >> "rcu: move TREE_RCU from softirq to kthread"
> > >
> >
> > Hi Paul,
> >
> > > OK, please accept my apologies for your lost weekend. And thank you for
> > > testing this.
> > >
> >
> > No worries, it was mostly a rainy day.
> > The only thing I did @ 16:30 was to go to regional election (the new
> > (regional) prime minister will be the 1st from The German Green
> > party).
>
> ;-)
>
> > But back to RCU :-):
> > The reduced kernel-config setup decreased the build-time from approx.
> > 2hrs (full, generic build) down to approx. 35mins.
>
> Very good!
>
> > >> I can do parallelly a tar job, open 20 tabs in firefox and run a flash
> > >> video in one of them (I did this several times).
> > >
> > > How many files in the tar job? Is this creating a tar archive, expanding
> > > it, or both?
> >
> > I am doing a simple tar (filesize: 1.6G for full and 1.0G for reduced build):
> >
> > $ tar -cf $archivedir-on-external-usbhdd/$tarfile $kernel-build-dir
>
> OK, I was extracting a tarball and then removing the resulting source
> tree. I will try this. Though it does seem strange -- I can understand
> how removing a file tree would stress RCU, but not creating a tarball.
> Ah, well, if I fully understood it, there would not be a bug.
>
> > ...plus parallelly opening 20 tabs in firefox.
> > That's normally enough to get my system freaky and see RCU related
> > messages in the logs.
>
> Hmmm... My normal test systems don't have X -- I will need to set
> this up.
>
> > > Do you have a script for this? Are all of these running at normal
> > > priority, or are some of them running at real-time priority?
> > >
> >
> > Nothing special.
>
> OK.
>
> > >> [ setup.log ]
> > >> ...
> > >> (+) OK revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch
> > >> (+) OK revert-rcu-patches/0002-Revert-rcu-fix-spelling.patch
> > >> (+) OK revert-rcu-patches/0003-Revert-rcu-fix-rcu_cpu_kthread_task-synchronization.patch
> > >> (+) OK revert-rcu-patches/0004-Revert-rcu-call-__rcu_read_unlock-in-exit_rcu-for-tr.patch
> > >> (+) OK revert-rcu-patches/0005-Revert-rcu-Converge-TINY_RCU-expedited-and-normal-bo.patch
> > >> (+) OK revert-rcu-patches/0006-Revert-rcu-remove-useless-boosted_this_gp-field.patch
> > >> (+) OK revert-rcu-patches/0007-Revert-rcu-code-cleanups-in-TINY_RCU-priority-boosti.patch
> > >> (+) OK revert-rcu-patches/0008-Revert-rcu-Switch-to-this_cpu-primitives.patch
> > >> (+) OK revert-rcu-patches/0009-Revert-rcu-Use-WARN_ON_ONCE-for-DEBUG_OBJECTS_RCU_HE.patch
> > >> (+) OK revert-rcu-patches/0010-Revert-rcu-Enable-DEBUG_OBJECTS_RCU_HEAD-from-PREEMP.patch
> > >> (+) OK revert-rcu-patches/0011-Revert-rcu-Add-boosting-to-TREE_PREEMPT_RCU-tracing.patch
> > >> (+) OK revert-rcu-patches/0012-Revert-rcu-eliminate-unused-boosting-statistics.patch
> > >> (+) OK revert-rcu-patches/0013-Revert-rcu-priority-boosting-for-TREE_PREEMPT_RCU.patch
> > >> (+) OK revert-rcu-patches/0014-Revert-rcu-move-TREE_RCU-from-softirq-to-kthread.patch
> > >> ...
> > >>
> > >> Hope this helps to narrow down the problem.
> > >>
> > >> As I kept all kernels I can have a look at the tasks consuming high
> > >> CPU usage tomorrow.
> > >
> > > Could you please?
> >
> > I recalled (as you say I requested over and over again from you :-)) I
> > looked with top, htop and 'ps axu', but there was nothing special.
> > Sometimes the system got frozen - at this point (or short before) I
> > did not see anything suspicious with top.
>
> OK, thank you for the info.
>
> > > Also, could you please mount debugfs and list out the files in the
> > > "rcu" directory? The "ql=" value from the "rcu/rcudata" file is of
> > > particular interest.
> > >
> >
> > Ah, before I forget...
> >
> > I used TREE_RCU (was the default before noticing RCU issue) for
> > finding the culprit commit.
> > If it is from your POV more helpful to switch to PREEMPT + PREEMPT_RCU
> > + RCU_BOOST, please let me *now* know.
> > ( Both RCU setups freaks up the system. )
>
> If TREE_RCU hits problems faster, it is probably best to stay with
> TREE_RCU.
And of course, one exception to this advice is if TREE_RCU hangs so hard
and fast that you don't have time to get any diagnostics. If this is the
case, then TREE_PREEMPT_RCU might be more productive.
Thanx, Paul
> > I think top & Co. are not enough to track the problem down.
> > I have seen tracing and debugging facililities for RCU.
> >
> > Some questions to debug and trace setup:
> >
> > Case #1: TREE_RCU
> >
> > CONFIG_RCU_TRACE=y
> > CONFIG_TREE_RCU_TRACE=y
>
> Yep.
>
> > Case #2: PREEMPT + PREEMPT_RCU + RCU_BOOST
> >
> > CONFIG_RCU_TRACE=y
> > CONFIG_TREE_RCU_TRACE=y
> > CONFIG_DEBUG_PREEMPT=y <--- Helpful?
> > CONFIG_PREEMPT_TRACER=y <--- Helpful?
> >
> > Any other recommends for useful/helpful trace and/or debug options?
> >
> > Any other intructions for debugging/tracing?
>
> Not at the moment. I will be looking at diagnostics will going
> through the code, so might have something later.
>
> > BTW, today's linux-next (next-20110328) is still freaky, I applied the
> > revert-rcu-patches patchset and all is fine.
>
> I reverted back to the commit preceding the one you pointed out last night
> my time, so the upcoming -next should be less freaky.
>
> > - Sedat -
> >
> > P.S.: Note to myself
> >
> > # mount -t debugfs none /sys/kernel/debug/
> > # ln -s /sys/kernel/debug /debug
> >
> > # find /debug -name rcu
>
> Or:
>
> # cd /debug/rcu
>
> then dump out everything except for the .csv file (which is the same
> as the non-.csv equivalent, but in spreadsheet format -- intended
> for systems with 100s or 1000s of CPUs).
>
> Thanx, Paul
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-28 15:11 ` Paul E. McKenney
@ 2011-03-28 16:38 ` Sedat Dilek
2011-03-28 16:46 ` Sedat Dilek
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-28 16:38 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
[-- Attachment #1: Type: text/plain, Size: 14994 bytes --]
On Mon, Mar 28, 2011 at 5:11 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Mon, Mar 28, 2011 at 06:24:36AM -0700, Paul E. McKenney wrote:
>> On Mon, Mar 28, 2011 at 02:33:36PM +0200, Sedat Dilek wrote:
>> > On Mon, Mar 28, 2011 at 6:08 AM, Paul E. McKenney
>> > <paulmck@linux.vnet.ibm.com> wrote:
>> > > On Sun, Mar 27, 2011 at 11:48:30PM +0200, Sedat Dilek wrote:
>> > >> On Sun, Mar 27, 2011 at 11:32 PM, Paul E. McKenney
>> > >> <paulmck@linux.vnet.ibm.com> wrote:
>> > >> > On Sun, Mar 27, 2011 at 02:26:15PM +0200, Sedat Dilek wrote:
>> > >> >> On Sun, Mar 27, 2011 at 7:07 AM, Paul E. McKenney
>> > >> >> <paulmck@linux.vnet.ibm.com> wrote:
>> > >> >> > On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
>> > >> >> >> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
>> > >> >> >> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
>> > >> >> >> > <paulmck@linux.vnet.ibm.com> wrote:
>> > >> >> >> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
>> > >> >> >
>> > >> >> > [ . . . ]
>> > >> >> >
>> > >> >> >> > >> But then came RCU :-(.
>> > >> >> >> > >
>> > >> >> >> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
>> > >> >> >> > >
>> > >> >> >> >
>> > >> >> >> > No, that's not so dramatic.
>> > >> >> >> > Dealing with this RCU issue has nice side-effects: I remembered (and
>> > >> >> >> > finally did) to use a reduced kernel-config set.
>> > >> >> >> > The base for it I created with 'make localmodconfig' and did some
>> > >> >> >> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
>> > >> >> >> > etc.).
>> > >> >> >> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
>> > >> >> >> >
>> > >> >> >> > So, I started building with
>> > >> >> >> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
>> > >> >> >> > I will let you know.
>> > >> >> >>
>> > >> >> >> And please also check for tasks consuming all available CPU.
>> > >> >> >
>> > >> >> > And I still cannot reproduce with the full RCU stack (but based off of
>> > >> >> > 2.6.38 rather than -next). Nevertheless, if you would like to try a
>> > >> >> > speculative patch, here you go.
>> > >> >>
>> > >> >> You are right and my strategy on handling the (possible RCU?) issue is wrong.
>> > >> >> Surely, you tested your RCU stuff in your own repo and everything
>> > >> >> might be OK on top of stable 2.6.38.
>> > >> >> Linux-next gets daily updates from a lot of different trees, so there
>> > >> >> might be interferences with other stuff.
>> > >> >> Please, understand I am interested in finding out what is the cause
>> > >> >> for my issues, my aim is not to blame you.
>> > >> >
>> > >> > I am not worried about blame, but rather getting the bug fixed. The
>> > >> > bug might be in RCU, it might be elsewhere, or it might be a combination
>> > >> > of problems in RCU and elsewhere.
>> > >> >
>> > >> > So the first priority is locating the bug.
>> > >> >
>> > >> > And that is why I have been asking you over and over to PLEASE take
>> > >> > a look at what tasks are consuming CPU while the problem is occuring.
>> > >> > The reason that I have been asking over and over is that the symptoms
>> > >> > you describe are likely caused by a loop in some kernel code. Yes,
>> > >> > there might be other causes, but this is the most likely. Given that
>> > >> > TREE_PREEMPT_RCU behaves better than TREE_RCU, it is likely that this
>> > >> > loop is in preemptible code with irqs enabled. Therefore, the process
>> > >> > accounting code is likely to be able to see the CPU consumption, and
>> > >> > you should be able to see it via the "top" or "ps" commands -- or via
>> > >> > any number of other tools.
>> > >> >
>> > >> > For example, if the problem is confined to RCU, you would likely see
>> > >> > the "rcuc0" or "rcun0" tasks consuming lots of CPU. This would narrow
>> > >> > the problem down to a few tens of lines of code. If the problem was
>> > >> > in some other kthread, then identifying the kthread would very likely
>> > >> > narrow things down as well.
>> > >> >
>> > >> > So, please do take a look to see what taks consuming CPU.
>> > >> >
>> > >> >> As I was wrong and want to be 99.9% sure it is RCU stuff, I reverted
>> > >> >> all (18) RCU patches from linux-next (next-20110325) by keeping the
>> > >> >> RCU|PREEMPT|HZ settings from last working next-20110323.
>> > >> >
>> > >> > Makes sense.
>> > >> >
>> > >> >> $ egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110325-7-686-iniza
>> > >> >> # RCU Subsystem
>> > >> >> CONFIG_TREE_RCU=y
>> > >> >> # CONFIG_PREEMPT_RCU is not set
>> > >> >> # CONFIG_RCU_TRACE is not set
>> > >> >> CONFIG_RCU_FANOUT=32
>> > >> >> # CONFIG_RCU_FANOUT_EXACT is not set
>> > >> >> CONFIG_RCU_FAST_NO_HZ=y
>> > >> >> # CONFIG_TREE_RCU_TRACE is not set
>> > >> >> CONFIG_PREEMPT_NOTIFIERS=y
>> > >> >> CONFIG_NO_HZ=y
>> > >> >> # CONFIG_PREEMPT_NONE is not set
>> > >> >> CONFIG_PREEMPT_VOLUNTARY=y
>> > >> >> # CONFIG_PREEMPT is not set
>> > >> >> # CONFIG_HZ_100 is not set
>> > >> >> CONFIG_HZ_250=y
>> > >> >> # CONFIG_HZ_300 is not set
>> > >> >> # CONFIG_HZ_1000 is not set
>> > >> >> CONFIG_HZ=250
>> > >> >> # CONFIG_SPARSE_RCU_POINTER is not set
>> > >> >> # CONFIG_RCU_TORTURE_TEST is not set
>> > >> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>> > >> >>
>> > >> >> I will work and stress this kernel before doing any step-by-step
>> > >> >> revert of RCU stuff.
>> > >> >>
>> > >> >> Thanks for your patch, I applied it on top of "naked" next-20110325,
>> > >> >> but I still see call-traces.
>> > >> >
>> > >> > Thank you very much for testing it!
>> > >> >
>> > >> > I intend to keep that patch, as it should increase robustness in other
>> > >> > situations.
>> > >> >
>> > >> > Thanx, Paul
>> > >> >
>> > >> >> - Sedat -
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> > Thanx, Paul
>> > >> >> >
>> > >> >> > ------------------------------------------------------------------------
>> > >> >> >
>> > >> >> > rcu: further lower priority in rcu_yield()
>> > >> >> >
>> > >> >> > Although rcu_yield() dropped from real-time to normal priority, there
>> > >> >> > is always the possibility that the competing tasks have been niced.
>> > >> >> > So nice to 19 in rcu_yield() to help ensure that other tasks have a
>> > >> >> > better chance of running.
>> > >> >> >
>> > >> >> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>> > >> >> >
>> > >> >> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
>> > >> >> > index 759f54b..5477764 100644
>> > >> >> > --- a/kernel/rcutree.c
>> > >> >> > +++ b/kernel/rcutree.c
>> > >> >> > @@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
>> > >> >> > mod_timer(&yield_timer, jiffies + 2);
>> > >> >> > sp.sched_priority = 0;
>> > >> >> > sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
>> > >> >> > + set_user_nice(current, 19);
>> > >> >> > schedule();
>> > >> >> > sp.sched_priority = RCU_KTHREAD_PRIO;
>> > >> >> > sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
>> > >>
>> > >> Sorry, my attempt was to identify and isolate the culprit commit.
>> > >>
>> > >> Reverting all RCU patches resulted in a stable system, the following 8
>> > >> kernels with reduced k-config setup where all built using this kernel.
>> > >>
>> > >> All kernels used TREE_RCU (see above), I did not change it (no
>> > >> mixing/switching to PREEMPT and TREE_PREEMPT_RCU).
>> > >> ( I doubt that TREE_PREEMPT_RCU was some kind of more stable here. )
>> > >>
>> > >> The culprit commit is bc56163ebd4580199ac7e63f5e160bf139ba0dd6 (from
>> > >> rcu/next GIT tree):
>> > >> "rcu: move TREE_RCU from softirq to kthread"
>> > >
>> >
>> > Hi Paul,
>> >
>> > > OK, please accept my apologies for your lost weekend. And thank you for
>> > > testing this.
>> > >
>> >
>> > No worries, it was mostly a rainy day.
>> > The only thing I did @ 16:30 was to go to regional election (the new
>> > (regional) prime minister will be the 1st from The German Green
>> > party).
>>
>> ;-)
>>
>> > But back to RCU :-):
>> > The reduced kernel-config setup decreased the build-time from approx.
>> > 2hrs (full, generic build) down to approx. 35mins.
>>
>> Very good!
>>
>> > >> I can do parallelly a tar job, open 20 tabs in firefox and run a flash
>> > >> video in one of them (I did this several times).
>> > >
>> > > How many files in the tar job? Is this creating a tar archive, expanding
>> > > it, or both?
>> >
>> > I am doing a simple tar (filesize: 1.6G for full and 1.0G for reduced build):
>> >
>> > $ tar -cf $archivedir-on-external-usbhdd/$tarfile $kernel-build-dir
>>
>> OK, I was extracting a tarball and then removing the resulting source
>> tree. I will try this. Though it does seem strange -- I can understand
>> how removing a file tree would stress RCU, but not creating a tarball.
>> Ah, well, if I fully understood it, there would not be a bug.
>>
>> > ...plus parallelly opening 20 tabs in firefox.
>> > That's normally enough to get my system freaky and see RCU related
>> > messages in the logs.
>>
>> Hmmm... My normal test systems don't have X -- I will need to set
>> this up.
>>
>> > > Do you have a script for this? Are all of these running at normal
>> > > priority, or are some of them running at real-time priority?
>> > >
>> >
>> > Nothing special.
>>
>> OK.
>>
>> > >> [ setup.log ]
>> > >> ...
>> > >> (+) OK revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch
>> > >> (+) OK revert-rcu-patches/0002-Revert-rcu-fix-spelling.patch
>> > >> (+) OK revert-rcu-patches/0003-Revert-rcu-fix-rcu_cpu_kthread_task-synchronization.patch
>> > >> (+) OK revert-rcu-patches/0004-Revert-rcu-call-__rcu_read_unlock-in-exit_rcu-for-tr.patch
>> > >> (+) OK revert-rcu-patches/0005-Revert-rcu-Converge-TINY_RCU-expedited-and-normal-bo.patch
>> > >> (+) OK revert-rcu-patches/0006-Revert-rcu-remove-useless-boosted_this_gp-field.patch
>> > >> (+) OK revert-rcu-patches/0007-Revert-rcu-code-cleanups-in-TINY_RCU-priority-boosti.patch
>> > >> (+) OK revert-rcu-patches/0008-Revert-rcu-Switch-to-this_cpu-primitives.patch
>> > >> (+) OK revert-rcu-patches/0009-Revert-rcu-Use-WARN_ON_ONCE-for-DEBUG_OBJECTS_RCU_HE.patch
>> > >> (+) OK revert-rcu-patches/0010-Revert-rcu-Enable-DEBUG_OBJECTS_RCU_HEAD-from-PREEMP.patch
>> > >> (+) OK revert-rcu-patches/0011-Revert-rcu-Add-boosting-to-TREE_PREEMPT_RCU-tracing.patch
>> > >> (+) OK revert-rcu-patches/0012-Revert-rcu-eliminate-unused-boosting-statistics.patch
>> > >> (+) OK revert-rcu-patches/0013-Revert-rcu-priority-boosting-for-TREE_PREEMPT_RCU.patch
>> > >> (+) OK revert-rcu-patches/0014-Revert-rcu-move-TREE_RCU-from-softirq-to-kthread.patch
>> > >> ...
>> > >>
>> > >> Hope this helps to narrow down the problem.
>> > >>
>> > >> As I kept all kernels I can have a look at the tasks consuming high
>> > >> CPU usage tomorrow.
>> > >
>> > > Could you please?
>> >
>> > I recalled (as you say I requested over and over again from you :-)) I
>> > looked with top, htop and 'ps axu', but there was nothing special.
>> > Sometimes the system got frozen - at this point (or short before) I
>> > did not see anything suspicious with top.
>>
>> OK, thank you for the info.
>>
>> > > Also, could you please mount debugfs and list out the files in the
>> > > "rcu" directory? The "ql=" value from the "rcu/rcudata" file is of
>> > > particular interest.
>> > >
>> >
>> > Ah, before I forget...
>> >
>> > I used TREE_RCU (was the default before noticing RCU issue) for
>> > finding the culprit commit.
>> > If it is from your POV more helpful to switch to PREEMPT + PREEMPT_RCU
>> > + RCU_BOOST, please let me *now* know.
>> > ( Both RCU setups freaks up the system. )
>>
>> If TREE_RCU hits problems faster, it is probably best to stay with
>> TREE_RCU.
>
> And of course, one exception to this advice is if TREE_RCU hangs so hard
> and fast that you don't have time to get any diagnostics. If this is the
> case, then TREE_PREEMPT_RCU might be more productive.
>
OK, that would somehow explain why I could not really get some debug
infos when doing "my stress-test" and checking via:
$ LC_ALL=C tail -f /sys/kernel/debug/rcu/rcudata
Then I remembered I saw a snippet for a RCU torture script mentionned
in the kernel-docs (see Documentation/RCU/torture.txt).
189 The following script may be used to torture RCU:
190
191 #!/bin/sh
192
193 modprobe rcutorture
194 sleep 100
195 rmmod rcutorture
196 dmesg | grep torture:
So, I recompiled a new TREE_RC-based kernel and build with
CONFIG_RCU_TORTURE_TEST=m.
Unfortunately, the rmmod (I prefer modprobe -r -v) hangs... the
messages in the logs look promising.
- Sedat -
> Thanx, Paul
>
>> > I think top & Co. are not enough to track the problem down.
>> > I have seen tracing and debugging facililities for RCU.
>> >
>> > Some questions to debug and trace setup:
>> >
>> > Case #1: TREE_RCU
>> >
>> > CONFIG_RCU_TRACE=y
>> > CONFIG_TREE_RCU_TRACE=y
>>
>> Yep.
>>
>> > Case #2: PREEMPT + PREEMPT_RCU + RCU_BOOST
>> >
>> > CONFIG_RCU_TRACE=y
>> > CONFIG_TREE_RCU_TRACE=y
>> > CONFIG_DEBUG_PREEMPT=y <--- Helpful?
>> > CONFIG_PREEMPT_TRACER=y <--- Helpful?
>> >
>> > Any other recommends for useful/helpful trace and/or debug options?
>> >
>> > Any other intructions for debugging/tracing?
>>
>> Not at the moment. I will be looking at diagnostics will going
>> through the code, so might have something later.
>>
>> > BTW, today's linux-next (next-20110328) is still freaky, I applied the
>> > revert-rcu-patches patchset and all is fine.
>>
>> I reverted back to the commit preceding the one you pointed out last night
>> my time, so the upcoming -next should be less freaky.
>>
>> > - Sedat -
>> >
>> > P.S.: Note to myself
>> >
>> > # mount -t debugfs none /sys/kernel/debug/
>> > # ln -s /sys/kernel/debug /debug
>> >
>> > # find /debug -name rcu
>>
>> Or:
>>
>> # cd /debug/rcu
>>
>> then dump out everything except for the .csv file (which is the same
>> as the non-.csv equivalent, but in spreadsheet format -- intended
>> for systems with 100s or 1000s of CPUs).
>>
>> Thanx, Paul
>
[-- Attachment #2: msg_rcu-torture.txt --]
[-- Type: text/plain, Size: 32581 bytes --]
Mar 28 18:12:03 tbox kernel: [ 325.716030] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
Mar 28 18:12:03 tbox kernel: [ 325.720011] sending NMI to all CPUs:
Mar 28 18:12:03 tbox kernel: [ 325.720011] NMI backtrace for cpu 0
Mar 28 18:12:03 tbox kernel: [ 325.720011] Modules linked in: rcutorture(-) acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative ppdev lp aes_i586 aes_generic rfcomm bnep bluetooth binfmt_misc fuse snd_intel8x0m snd_intel8x0 snd_ac97_codec radeon ac97_bus snd_pcm_oss arc4 snd_mixer_oss ecb ath5k joydev snd_pcm thinkpad_acpi pcmcia snd_seq_midi ath ttm mac80211 snd_rawmidi drm_kms_helper drm cfg80211 yenta_socket snd_seq_midi_event pcmcia_rsrc snd_seq pcmcia_core snd_timer i2c_algo_bit rfkill snd_seq_device psmouse snd nsc_ircc tpm_tis parport_pc battery ac pcspkr tpm irda i2c_i801 shpchp serio_raw tpm_bios soundcore processor i2c_core evdev pci_hotplug snd_page_alloc crc_ccitt parport nvram power_supply video button ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd scsi_mod usbcore e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
Mar 28 18:12:03 tbox kernel: [ 325.720011]
Mar 28 18:12:03 tbox kernel: [ 325.720011] Pid: 0, comm: swapper Not tainted 2.6.38-next20110328-3-686-iniza #1 IBM 2374SG6/2374SG6
Mar 28 18:12:03 tbox kernel: [ 325.720011] EIP: 0060:[<c114b94a>] EFLAGS: 00000807 CPU: 0
Mar 28 18:12:03 tbox kernel: [ 325.720011] EIP is at delay_tsc+0x0/0x5e
Mar 28 18:12:03 tbox kernel: [ 325.720011] EAX: 00090d40 EBX: 00002710 ECX: c132e541 EDX: 00090d3f
Mar 28 18:12:03 tbox kernel: [ 325.720011] ESI: c13d7500 EDI: c13d7580 EBP: f6019ea4 ESP: f6019ea0
Mar 28 18:12:03 tbox kernel: [ 325.720011] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Mar 28 18:12:03 tbox kernel: [ 325.720011] Process swapper (pid: 0, ti=f6018000 task=c13ccfa0 task.ti=c13a0000)
Mar 28 18:12:03 tbox kernel: [ 325.720011] Stack:
Mar 28 18:12:03 tbox kernel: [ 325.720011] c114b90f f6019eac c114b92b f6019eb8 c10164ca c13d7500 f6019ed4 c1074e8d
Mar 28 18:12:03 tbox kernel: [ 325.720011] c133c4ef c133c5be 00000000 0000ea7e f6403488 f6019ee8 c1074f02 00000000
Mar 28 18:12:03 tbox kernel: [ 325.720011] 00000000 c13ccfa0 f6019ef4 c10752ab 00000000 f6019f08 c1039c2c c13a1ef8
Mar 28 18:12:03 tbox kernel: [ 325.720011] Call Trace:
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c114b90f>] ? __delay+0x9/0xb
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c114b92b>] __const_udelay+0x1a/0x1c
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:12:03 tbox kernel: [ 325.720011] <IRQ>
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:12:03 tbox kernel: [ 325.720011] Code: d2 3e f7 e2 8d 42 01 e8 db ff ff ff 5d c3 69 c0 c7 10 00 00 55 89 e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3
Mar 28 18:12:03 tbox kernel: [ 325.720011] Call Trace:
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c114b90f>] ? __delay+0x9/0xb
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c114b92b>] __const_udelay+0x1a/0x1c
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:12:03 tbox kernel: [ 325.720011] <IRQ> [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:12:37 tbox kernel: [ 360.304066] INFO: task modprobe:1662 blocked for more than 120 seconds.
Mar 28 18:12:37 tbox kernel: [ 360.314213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 28 18:12:37 tbox kernel: [ 360.324527] modprobe D f5127ea4 0 1662 1651 0x00000000
Mar 28 18:12:37 tbox kernel: [ 360.335003] f5127e98 00000082 f594ac00 f5127ea4 c145d380 ed715c1f 0000002a c145d380
Mar 28 18:12:37 tbox kernel: [ 360.345745] 00000000 00000000 f594ac00 f637dc00 0000002a f594ac00 ffffffff ffffffff
Mar 28 18:12:37 tbox kernel: [ 360.356521] f5127ed8 ed715020 f64063c4 ed715c1f 00000000 3d090000 00000000 f594ac2c
Mar 28 18:12:37 tbox kernel: [ 360.367389] Call Trace:
Mar 28 18:12:37 tbox kernel: [ 360.378083] [<c10215b4>] ? wakeup_preempt_entity+0x36/0x53
Mar 28 18:12:37 tbox kernel: [ 360.388915] [<c1295ed0>] schedule_timeout+0x21/0xaa
Mar 28 18:12:37 tbox kernel: [ 360.399708] [<c1295ed0>] ? schedule_timeout+0x21/0xaa
Mar 28 18:12:37 tbox kernel: [ 360.410473] [<c114507f>] ? cpumask_next_and+0x23/0x2f
Mar 28 18:12:37 tbox kernel: [ 360.421240] [<c1295d41>] wait_for_common+0x77/0xcd
Mar 28 18:12:37 tbox kernel: [ 360.432032] [<c102a570>] ? try_to_wake_up+0x142/0x142
Mar 28 18:12:37 tbox kernel: [ 360.442808] [<c107499c>] ? call_rcu_bh+0xf/0xf
Mar 28 18:12:37 tbox kernel: [ 360.453598] [<c1295e18>] wait_for_completion+0x12/0x14
Mar 28 18:12:37 tbox kernel: [ 360.464436] [<c1074df6>] _rcu_barrier.isra.41+0x78/0x85
Mar 28 18:12:37 tbox kernel: [ 360.475315] [<c1074e10>] rcu_barrier_sched+0xd/0xf
Mar 28 18:12:37 tbox kernel: [ 360.485900] [<c1074e1a>] rcu_barrier+0x8/0xa
Mar 28 18:12:37 tbox kernel: [ 360.496150] [<f82e35df>] rcu_torture_cleanup+0x2cd/0x317 [rcutorture]
Mar 28 18:12:37 tbox kernel: [ 360.506544] [<c1057d05>] sys_delete_module+0x198/0x1f5
Mar 28 18:12:37 tbox kernel: [ 360.516992] [<c1075093>] ? rcu_enter_nohz+0x28/0x9c
Mar 28 18:12:37 tbox kernel: [ 360.527410] [<c1075219>] ? rcu_irq_exit+0x8/0xa
Mar 28 18:12:37 tbox kernel: [ 360.537813] [<c129c4df>] sysenter_do_call+0x12/0x28
Mar 28 18:14:37 tbox kernel: [ 480.548068] INFO: task modprobe:1662 blocked for more than 120 seconds.
Mar 28 18:14:37 tbox kernel: [ 480.558653] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 28 18:14:37 tbox kernel: [ 480.569378] modprobe D f5127ea4 0 1662 1651 0x00000000
Mar 28 18:14:37 tbox kernel: [ 480.580205] f5127e98 00000082 f594ac00 f5127ea4 c145d380 ed715c1f 0000002a c145d380
Mar 28 18:14:37 tbox kernel: [ 480.591303] 00000000 00000000 f594ac00 f637dc00 0000002a f594ac00 ffffffff ffffffff
Mar 28 18:14:37 tbox kernel: [ 480.602479] f5127ed8 ed715020 f64063c4 ed715c1f 00000000 3d090000 00000000 f594ac2c
Mar 28 18:14:37 tbox kernel: [ 480.613746] Call Trace:
Mar 28 18:14:37 tbox kernel: [ 480.624810] [<c10215b4>] ? wakeup_preempt_entity+0x36/0x53
Mar 28 18:14:37 tbox kernel: [ 480.636044] [<c1295ed0>] schedule_timeout+0x21/0xaa
Mar 28 18:14:37 tbox kernel: [ 480.647248] [<c1295ed0>] ? schedule_timeout+0x21/0xaa
Mar 28 18:14:37 tbox kernel: [ 480.658528] [<c114507f>] ? cpumask_next_and+0x23/0x2f
Mar 28 18:14:37 tbox kernel: [ 480.669762] [<c1295d41>] wait_for_common+0x77/0xcd
Mar 28 18:14:37 tbox kernel: [ 480.680957] [<c102a570>] ? try_to_wake_up+0x142/0x142
Mar 28 18:14:37 tbox kernel: [ 480.692187] [<c107499c>] ? call_rcu_bh+0xf/0xf
Mar 28 18:14:37 tbox kernel: [ 480.703404] [<c1295e18>] wait_for_completion+0x12/0x14
Mar 28 18:14:37 tbox kernel: [ 480.714691] [<c1074df6>] _rcu_barrier.isra.41+0x78/0x85
Mar 28 18:14:37 tbox kernel: [ 480.725994] [<c1074e10>] rcu_barrier_sched+0xd/0xf
Mar 28 18:14:37 tbox kernel: [ 480.737266] [<c1074e1a>] rcu_barrier+0x8/0xa
Mar 28 18:14:37 tbox kernel: [ 480.748460] [<f82e35df>] rcu_torture_cleanup+0x2cd/0x317 [rcutorture]
Mar 28 18:14:37 tbox kernel: [ 480.759798] [<c1057d05>] sys_delete_module+0x198/0x1f5
Mar 28 18:14:37 tbox kernel: [ 480.771072] [<c1075093>] ? rcu_enter_nohz+0x28/0x9c
Mar 28 18:14:37 tbox kernel: [ 480.782354] [<c1075219>] ? rcu_irq_exit+0x8/0xa
Mar 28 18:14:37 tbox kernel: [ 480.793576] [<c129c4df>] sysenter_do_call+0x12/0x28
Mar 28 18:15:03 tbox kernel: [ 505.836029] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
Mar 28 18:15:03 tbox kernel: [ 505.840011] sending NMI to all CPUs:
Mar 28 18:15:03 tbox kernel: [ 505.840011] NMI backtrace for cpu 0
Mar 28 18:15:03 tbox kernel: [ 505.840011] Modules linked in: rcutorture(-) acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative ppdev lp aes_i586 aes_generic rfcomm bnep bluetooth binfmt_misc fuse snd_intel8x0m snd_intel8x0 snd_ac97_codec radeon ac97_bus snd_pcm_oss arc4 snd_mixer_oss ecb ath5k joydev snd_pcm thinkpad_acpi pcmcia snd_seq_midi ath ttm mac80211 snd_rawmidi drm_kms_helper drm cfg80211 yenta_socket snd_seq_midi_event pcmcia_rsrc snd_seq pcmcia_core snd_timer i2c_algo_bit rfkill snd_seq_device psmouse snd nsc_ircc tpm_tis parport_pc battery ac pcspkr tpm irda i2c_i801 shpchp serio_raw tpm_bios soundcore processor i2c_core evdev pci_hotplug snd_page_alloc crc_ccitt parport nvram power_supply video button ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd scsi_mod usbcore e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
Mar 28 18:15:03 tbox kernel: [ 505.840011]
Mar 28 18:15:03 tbox kernel: [ 505.840011] Pid: 0, comm: swapper Not tainted 2.6.38-next20110328-3-686-iniza #1 IBM 2374SG6/2374SG6
Mar 28 18:15:03 tbox kernel: [ 505.840011] EIP: 0060:[<c114b91b>] EFLAGS: 00000002 CPU: 0
Mar 28 18:15:03 tbox kernel: [ 505.840011] EIP is at __const_udelay+0xa/0x1c
Mar 28 18:15:03 tbox kernel: [ 505.840011] EAX: 00418958 EBX: 00002710 ECX: c132e541 EDX: 00247fab
Mar 28 18:15:03 tbox kernel: [ 505.840011] ESI: c13d7500 EDI: c13d7580 EBP: f6019eac ESP: f6019eac
Mar 28 18:15:03 tbox kernel: [ 505.840011] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Mar 28 18:15:03 tbox kernel: [ 505.840011] Process swapper (pid: 0, ti=f6018000 task=c13ccfa0 task.ti=c13a0000)
Mar 28 18:15:03 tbox kernel: [ 505.840011] Stack:
Mar 28 18:15:03 tbox kernel: [ 505.840011] f6019eb8 c10164ca c13d7500 f6019ed4 c1074e8d c133c4ef c133c5be 00000000
Mar 28 18:15:03 tbox kernel: [ 505.840011] 00019a64 f6403488 f6019ee8 c1074f02 00000000 00000000 c13ccfa0 f6019ef4
Mar 28 18:15:03 tbox kernel: [ 505.840011] c10752ab 00000000 f6019f08 c1039c2c c13a1ef8 c62d1b06 00000075 f6019f20
Mar 28 18:15:03 tbox kernel: [ 505.840011] Call Trace:
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:15:03 tbox kernel: [ 505.840011] <IRQ>
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:15:03 tbox kernel: [ 505.840011] Code: eb 0e 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 48 75 fd 48 5d c3 55 89 e5 ff 15 a8 81 3e c1 5d c3 55 89 e5 64 8b 15 dc d2 45 c1 <c1> e0 02 6b d2 3e f7 e2 8d 42 01 e8 db ff ff ff 5d c3 69 c0 c7
Mar 28 18:15:03 tbox kernel: [ 505.840011] Call Trace:
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:15:03 tbox kernel: [ 505.840011] <IRQ> [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:16:38 tbox kernel: [ 600.804066] INFO: task modprobe:1662 blocked for more than 120 seconds.
Mar 28 18:16:38 tbox kernel: [ 600.814684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 28 18:16:38 tbox kernel: [ 600.825484] modprobe D f5127ea4 0 1662 1651 0x00000000
Mar 28 18:16:38 tbox kernel: [ 600.836381] f5127e98 00000082 f594ac00 f5127ea4 c145d380 ed715c1f 0000002a c145d380
Mar 28 18:16:38 tbox kernel: [ 600.847562] 00000000 00000000 f594ac00 f637dc00 0000002a f594ac00 ffffffff ffffffff
Mar 28 18:16:38 tbox kernel: [ 600.858807] f5127ed8 ed715020 f64063c4 ed715c1f 00000000 3d090000 00000000 f594ac2c
Mar 28 18:16:38 tbox kernel: [ 600.870196] Call Trace:
Mar 28 18:16:38 tbox kernel: [ 600.881367] [<c10215b4>] ? wakeup_preempt_entity+0x36/0x53
Mar 28 18:16:38 tbox kernel: [ 600.892671] [<c1295ed0>] schedule_timeout+0x21/0xaa
Mar 28 18:16:38 tbox kernel: [ 600.903940] [<c1295ed0>] ? schedule_timeout+0x21/0xaa
Mar 28 18:16:38 tbox kernel: [ 600.915184] [<c114507f>] ? cpumask_next_and+0x23/0x2f
Mar 28 18:16:38 tbox kernel: [ 600.926423] [<c1295d41>] wait_for_common+0x77/0xcd
Mar 28 18:16:38 tbox kernel: [ 600.937628] [<c102a570>] ? try_to_wake_up+0x142/0x142
Mar 28 18:16:38 tbox kernel: [ 600.948873] [<c107499c>] ? call_rcu_bh+0xf/0xf
Mar 28 18:16:38 tbox kernel: [ 600.960134] [<c1295e18>] wait_for_completion+0x12/0x14
Mar 28 18:16:38 tbox kernel: [ 600.971503] [<c1074df6>] _rcu_barrier.isra.41+0x78/0x85
Mar 28 18:16:38 tbox kernel: [ 600.982899] [<c1074e10>] rcu_barrier_sched+0xd/0xf
Mar 28 18:16:38 tbox kernel: [ 600.993893] [<c1074e1a>] rcu_barrier+0x8/0xa
Mar 28 18:16:38 tbox kernel: [ 601.004417] [<f82e35df>] rcu_torture_cleanup+0x2cd/0x317 [rcutorture]
Mar 28 18:16:38 tbox kernel: [ 601.015076] [<c1057d05>] sys_delete_module+0x198/0x1f5
Mar 28 18:16:38 tbox kernel: [ 601.025690] [<c1075093>] ? rcu_enter_nohz+0x28/0x9c
Mar 28 18:16:38 tbox kernel: [ 601.036275] [<c1075219>] ? rcu_irq_exit+0x8/0xa
Mar 28 18:16:38 tbox kernel: [ 601.046855] [<c129c4df>] sysenter_do_call+0x12/0x28
Mar 28 18:17:01 tbox /USR/SBIN/CRON[1665]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Mar 28 18:18:04 tbox kernel: [ 685.956029] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
Mar 28 18:18:04 tbox kernel: [ 685.960012] sending NMI to all CPUs:
Mar 28 18:18:04 tbox kernel: [ 685.960012] NMI backtrace for cpu 0
Mar 28 18:18:04 tbox kernel: [ 685.960012] Modules linked in: rcutorture(-) acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative ppdev lp aes_i586 aes_generic rfcomm bnep bluetooth binfmt_misc fuse snd_intel8x0m snd_intel8x0 snd_ac97_codec radeon ac97_bus snd_pcm_oss arc4 snd_mixer_oss ecb ath5k joydev snd_pcm thinkpad_acpi pcmcia snd_seq_midi ath ttm mac80211 snd_rawmidi drm_kms_helper drm cfg80211 yenta_socket snd_seq_midi_event pcmcia_rsrc snd_seq pcmcia_core snd_timer i2c_algo_bit rfkill snd_seq_device psmouse snd nsc_ircc tpm_tis parport_pc battery ac pcspkr tpm irda i2c_i801 shpchp serio_raw tpm_bios soundcore processor i2c_core evdev pci_hotplug snd_page_alloc crc_ccitt parport nvram power_supply video button ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd scsi_mod usbcore e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
Mar 28 18:18:04 tbox kernel: [ 685.960012]
Mar 28 18:18:04 tbox kernel: [ 685.960012] Pid: 0, comm: swapper Not tainted 2.6.38-next20110328-3-686-iniza #1 IBM 2374SG6/2374SG6
Mar 28 18:18:04 tbox kernel: [ 685.960012] EIP: 0060:[<c114b923>] EFLAGS: 00000807 CPU: 0
Mar 28 18:18:04 tbox kernel: [ 685.960012] EIP is at __const_udelay+0x12/0x1c
Mar 28 18:18:04 tbox kernel: [ 685.960012] EAX: 4f1a99c0 EBX: 00002710 ECX: c132e541 EDX: 00090d3f
Mar 28 18:18:04 tbox kernel: [ 685.960012] ESI: c13d7500 EDI: c13d7580 EBP: f6019eac ESP: f6019eac
Mar 28 18:18:04 tbox kernel: [ 685.960012] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Mar 28 18:18:04 tbox kernel: [ 685.960012] Process swapper (pid: 0, ti=f6018000 task=c13ccfa0 task.ti=c13a0000)
Mar 28 18:18:04 tbox kernel: [ 685.960012] Stack:
Mar 28 18:18:04 tbox kernel: [ 685.960012] f6019eb8 c10164ca c13d7500 f6019ed4 c1074e8d c133c4ef c133c5be 00000000
Mar 28 18:18:04 tbox kernel: [ 685.960012] 00024a4a f6403488 f6019ee8 c1074f02 00000000 00000000 c13ccfa0 f6019ef4
Mar 28 18:18:04 tbox kernel: [ 685.960012] c10752ab 00000000 f6019f08 c1039c2c c13a1ef8 b62a34e0 0000009f f6019f20
Mar 28 18:18:04 tbox kernel: [ 685.960012] Call Trace:
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:18:04 tbox kernel: [ 685.960012] <IRQ>
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:18:04 tbox kernel: [ 685.960012] Code: 00 8d bc 27 00 00 00 00 48 75 fd 48 5d c3 55 89 e5 ff 15 a8 81 3e c1 5d c3 55 89 e5 64 8b 15 dc d2 45 c1 c1 e0 02 6b d2 3e f7 e2 <8d> 42 01 e8 db ff ff ff 5d c3 69 c0 c7 10 00 00 55 89 e5 e8 d6
Mar 28 18:18:04 tbox kernel: [ 685.960012] Call Trace:
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:18:04 tbox kernel: [ 685.960012] <IRQ> [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:18:38 tbox kernel: [ 721.056065] INFO: task modprobe:1662 blocked for more than 120 seconds.
Mar 28 18:18:38 tbox kernel: [ 721.066683] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 28 18:18:38 tbox kernel: [ 721.077467] modprobe D f5127ea4 0 1662 1651 0x00000000
Mar 28 18:18:38 tbox kernel: [ 721.088422] f5127e98 00000082 f594ac00 f5127ea4 c145d380 ed715c1f 0000002a c145d380
Mar 28 18:18:38 tbox kernel: [ 721.099608] 00000000 00000000 f594ac00 f637dc00 0000002a f594ac00 ffffffff ffffffff
Mar 28 18:18:38 tbox kernel: [ 721.110877] f5127ed8 ed715020 f64063c4 ed715c1f 00000000 3d090000 00000000 f594ac2c
Mar 28 18:18:38 tbox kernel: [ 721.122224] Call Trace:
Mar 28 18:18:38 tbox kernel: [ 721.133395] [<c10215b4>] ? wakeup_preempt_entity+0x36/0x53
Mar 28 18:18:38 tbox kernel: [ 721.144707] [<c1295ed0>] schedule_timeout+0x21/0xaa
Mar 28 18:18:38 tbox kernel: [ 721.155977] [<c1295ed0>] ? schedule_timeout+0x21/0xaa
Mar 28 18:18:38 tbox kernel: [ 721.167211] [<c114507f>] ? cpumask_next_and+0x23/0x2f
Mar 28 18:18:38 tbox kernel: [ 721.178445] [<c1295d41>] wait_for_common+0x77/0xcd
Mar 28 18:18:38 tbox kernel: [ 721.189693] [<c102a570>] ? try_to_wake_up+0x142/0x142
Mar 28 18:18:38 tbox kernel: [ 721.200939] [<c107499c>] ? call_rcu_bh+0xf/0xf
Mar 28 18:18:38 tbox kernel: [ 721.212205] [<c1295e18>] wait_for_completion+0x12/0x14
Mar 28 18:18:38 tbox kernel: [ 721.223518] [<c1074df6>] _rcu_barrier.isra.41+0x78/0x85
Mar 28 18:18:38 tbox kernel: [ 721.234876] [<c1074e10>] rcu_barrier_sched+0xd/0xf
Mar 28 18:18:38 tbox kernel: [ 721.245861] [<c1074e1a>] rcu_barrier+0x8/0xa
Mar 28 18:18:38 tbox kernel: [ 721.256390] [<f82e35df>] rcu_torture_cleanup+0x2cd/0x317 [rcutorture]
Mar 28 18:18:38 tbox kernel: [ 721.267042] [<c1057d05>] sys_delete_module+0x198/0x1f5
Mar 28 18:18:38 tbox kernel: [ 721.277652] [<c1075093>] ? rcu_enter_nohz+0x28/0x9c
Mar 28 18:18:38 tbox kernel: [ 721.288273] [<c1075219>] ? rcu_irq_exit+0x8/0xa
Mar 28 18:18:38 tbox kernel: [ 721.298872] [<c129c4df>] sysenter_do_call+0x12/0x28
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-28 16:38 ` Sedat Dilek
@ 2011-03-28 16:46 ` Sedat Dilek
2011-03-29 0:10 ` Paul E. McKenney
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-28 16:46 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
[-- Attachment #1: Type: text/plain, Size: 15517 bytes --]
On Mon, Mar 28, 2011 at 6:38 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
> On Mon, Mar 28, 2011 at 5:11 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
>> On Mon, Mar 28, 2011 at 06:24:36AM -0700, Paul E. McKenney wrote:
>>> On Mon, Mar 28, 2011 at 02:33:36PM +0200, Sedat Dilek wrote:
>>> > On Mon, Mar 28, 2011 at 6:08 AM, Paul E. McKenney
>>> > <paulmck@linux.vnet.ibm.com> wrote:
>>> > > On Sun, Mar 27, 2011 at 11:48:30PM +0200, Sedat Dilek wrote:
>>> > >> On Sun, Mar 27, 2011 at 11:32 PM, Paul E. McKenney
>>> > >> <paulmck@linux.vnet.ibm.com> wrote:
>>> > >> > On Sun, Mar 27, 2011 at 02:26:15PM +0200, Sedat Dilek wrote:
>>> > >> >> On Sun, Mar 27, 2011 at 7:07 AM, Paul E. McKenney
>>> > >> >> <paulmck@linux.vnet.ibm.com> wrote:
>>> > >> >> > On Sat, Mar 26, 2011 at 08:25:29PM -0700, Paul E. McKenney wrote:
>>> > >> >> >> On Sun, Mar 27, 2011 at 03:30:34AM +0200, Sedat Dilek wrote:
>>> > >> >> >> > On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
>>> > >> >> >> > <paulmck@linux.vnet.ibm.com> wrote:
>>> > >> >> >> > > On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
>>> > >> >> >
>>> > >> >> > [ . . . ]
>>> > >> >> >
>>> > >> >> >> > >> But then came RCU :-(.
>>> > >> >> >> > >
>>> > >> >> >> > > Well, if it turns out to be a problem in RCU I will certainly apologize.
>>> > >> >> >> > >
>>> > >> >> >> >
>>> > >> >> >> > No, that's not so dramatic.
>>> > >> >> >> > Dealing with this RCU issue has nice side-effects: I remembered (and
>>> > >> >> >> > finally did) to use a reduced kernel-config set.
>>> > >> >> >> > The base for it I created with 'make localmodconfig' and did some
>>> > >> >> >> > manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
>>> > >> >> >> > etc.).
>>> > >> >> >> > Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.
>>> > >> >> >> >
>>> > >> >> >> > So, I started building with
>>> > >> >> >> > "revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
>>> > >> >> >> > I will let you know.
>>> > >> >> >>
>>> > >> >> >> And please also check for tasks consuming all available CPU.
>>> > >> >> >
>>> > >> >> > And I still cannot reproduce with the full RCU stack (but based off of
>>> > >> >> > 2.6.38 rather than -next). Nevertheless, if you would like to try a
>>> > >> >> > speculative patch, here you go.
>>> > >> >>
>>> > >> >> You are right and my strategy on handling the (possible RCU?) issue is wrong.
>>> > >> >> Surely, you tested your RCU stuff in your own repo and everything
>>> > >> >> might be OK on top of stable 2.6.38.
>>> > >> >> Linux-next gets daily updates from a lot of different trees, so there
>>> > >> >> might be interferences with other stuff.
>>> > >> >> Please, understand I am interested in finding out what is the cause
>>> > >> >> for my issues, my aim is not to blame you.
>>> > >> >
>>> > >> > I am not worried about blame, but rather getting the bug fixed. The
>>> > >> > bug might be in RCU, it might be elsewhere, or it might be a combination
>>> > >> > of problems in RCU and elsewhere.
>>> > >> >
>>> > >> > So the first priority is locating the bug.
>>> > >> >
>>> > >> > And that is why I have been asking you over and over to PLEASE take
>>> > >> > a look at what tasks are consuming CPU while the problem is occuring.
>>> > >> > The reason that I have been asking over and over is that the symptoms
>>> > >> > you describe are likely caused by a loop in some kernel code. Yes,
>>> > >> > there might be other causes, but this is the most likely. Given that
>>> > >> > TREE_PREEMPT_RCU behaves better than TREE_RCU, it is likely that this
>>> > >> > loop is in preemptible code with irqs enabled. Therefore, the process
>>> > >> > accounting code is likely to be able to see the CPU consumption, and
>>> > >> > you should be able to see it via the "top" or "ps" commands -- or via
>>> > >> > any number of other tools.
>>> > >> >
>>> > >> > For example, if the problem is confined to RCU, you would likely see
>>> > >> > the "rcuc0" or "rcun0" tasks consuming lots of CPU. This would narrow
>>> > >> > the problem down to a few tens of lines of code. If the problem was
>>> > >> > in some other kthread, then identifying the kthread would very likely
>>> > >> > narrow things down as well.
>>> > >> >
>>> > >> > So, please do take a look to see what taks consuming CPU.
>>> > >> >
>>> > >> >> As I was wrong and want to be 99.9% sure it is RCU stuff, I reverted
>>> > >> >> all (18) RCU patches from linux-next (next-20110325) by keeping the
>>> > >> >> RCU|PREEMPT|HZ settings from last working next-20110323.
>>> > >> >
>>> > >> > Makes sense.
>>> > >> >
>>> > >> >> $ egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110325-7-686-iniza
>>> > >> >> # RCU Subsystem
>>> > >> >> CONFIG_TREE_RCU=y
>>> > >> >> # CONFIG_PREEMPT_RCU is not set
>>> > >> >> # CONFIG_RCU_TRACE is not set
>>> > >> >> CONFIG_RCU_FANOUT=32
>>> > >> >> # CONFIG_RCU_FANOUT_EXACT is not set
>>> > >> >> CONFIG_RCU_FAST_NO_HZ=y
>>> > >> >> # CONFIG_TREE_RCU_TRACE is not set
>>> > >> >> CONFIG_PREEMPT_NOTIFIERS=y
>>> > >> >> CONFIG_NO_HZ=y
>>> > >> >> # CONFIG_PREEMPT_NONE is not set
>>> > >> >> CONFIG_PREEMPT_VOLUNTARY=y
>>> > >> >> # CONFIG_PREEMPT is not set
>>> > >> >> # CONFIG_HZ_100 is not set
>>> > >> >> CONFIG_HZ_250=y
>>> > >> >> # CONFIG_HZ_300 is not set
>>> > >> >> # CONFIG_HZ_1000 is not set
>>> > >> >> CONFIG_HZ=250
>>> > >> >> # CONFIG_SPARSE_RCU_POINTER is not set
>>> > >> >> # CONFIG_RCU_TORTURE_TEST is not set
>>> > >> >> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
>>> > >> >>
>>> > >> >> I will work and stress this kernel before doing any step-by-step
>>> > >> >> revert of RCU stuff.
>>> > >> >>
>>> > >> >> Thanks for your patch, I applied it on top of "naked" next-20110325,
>>> > >> >> but I still see call-traces.
>>> > >> >
>>> > >> > Thank you very much for testing it!
>>> > >> >
>>> > >> > I intend to keep that patch, as it should increase robustness in other
>>> > >> > situations.
>>> > >> >
>>> > >> > Thanx, Paul
>>> > >> >
>>> > >> >> - Sedat -
>>> > >> >>
>>> > >> >>
>>> > >> >>
>>> > >> >> > Thanx, Paul
>>> > >> >> >
>>> > >> >> > ------------------------------------------------------------------------
>>> > >> >> >
>>> > >> >> > rcu: further lower priority in rcu_yield()
>>> > >> >> >
>>> > >> >> > Although rcu_yield() dropped from real-time to normal priority, there
>>> > >> >> > is always the possibility that the competing tasks have been niced.
>>> > >> >> > So nice to 19 in rcu_yield() to help ensure that other tasks have a
>>> > >> >> > better chance of running.
>>> > >> >> >
>>> > >> >> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>>> > >> >> >
>>> > >> >> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
>>> > >> >> > index 759f54b..5477764 100644
>>> > >> >> > --- a/kernel/rcutree.c
>>> > >> >> > +++ b/kernel/rcutree.c
>>> > >> >> > @@ -1492,6 +1492,7 @@ static void rcu_yield(void (*f)(unsigned long), unsigned long arg)
>>> > >> >> > mod_timer(&yield_timer, jiffies + 2);
>>> > >> >> > sp.sched_priority = 0;
>>> > >> >> > sched_setscheduler_nocheck(current, SCHED_NORMAL, &sp);
>>> > >> >> > + set_user_nice(current, 19);
>>> > >> >> > schedule();
>>> > >> >> > sp.sched_priority = RCU_KTHREAD_PRIO;
>>> > >> >> > sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
>>> > >>
>>> > >> Sorry, my attempt was to identify and isolate the culprit commit.
>>> > >>
>>> > >> Reverting all RCU patches resulted in a stable system, the following 8
>>> > >> kernels with reduced k-config setup where all built using this kernel.
>>> > >>
>>> > >> All kernels used TREE_RCU (see above), I did not change it (no
>>> > >> mixing/switching to PREEMPT and TREE_PREEMPT_RCU).
>>> > >> ( I doubt that TREE_PREEMPT_RCU was some kind of more stable here. )
>>> > >>
>>> > >> The culprit commit is bc56163ebd4580199ac7e63f5e160bf139ba0dd6 (from
>>> > >> rcu/next GIT tree):
>>> > >> "rcu: move TREE_RCU from softirq to kthread"
>>> > >
>>> >
>>> > Hi Paul,
>>> >
>>> > > OK, please accept my apologies for your lost weekend. And thank you for
>>> > > testing this.
>>> > >
>>> >
>>> > No worries, it was mostly a rainy day.
>>> > The only thing I did @ 16:30 was to go to regional election (the new
>>> > (regional) prime minister will be the 1st from The German Green
>>> > party).
>>>
>>> ;-)
>>>
>>> > But back to RCU :-):
>>> > The reduced kernel-config setup decreased the build-time from approx.
>>> > 2hrs (full, generic build) down to approx. 35mins.
>>>
>>> Very good!
>>>
>>> > >> I can do parallelly a tar job, open 20 tabs in firefox and run a flash
>>> > >> video in one of them (I did this several times).
>>> > >
>>> > > How many files in the tar job? Is this creating a tar archive, expanding
>>> > > it, or both?
>>> >
>>> > I am doing a simple tar (filesize: 1.6G for full and 1.0G for reduced build):
>>> >
>>> > $ tar -cf $archivedir-on-external-usbhdd/$tarfile $kernel-build-dir
>>>
>>> OK, I was extracting a tarball and then removing the resulting source
>>> tree. I will try this. Though it does seem strange -- I can understand
>>> how removing a file tree would stress RCU, but not creating a tarball.
>>> Ah, well, if I fully understood it, there would not be a bug.
>>>
>>> > ...plus parallelly opening 20 tabs in firefox.
>>> > That's normally enough to get my system freaky and see RCU related
>>> > messages in the logs.
>>>
>>> Hmmm... My normal test systems don't have X -- I will need to set
>>> this up.
>>>
>>> > > Do you have a script for this? Are all of these running at normal
>>> > > priority, or are some of them running at real-time priority?
>>> > >
>>> >
>>> > Nothing special.
>>>
>>> OK.
>>>
>>> > >> [ setup.log ]
>>> > >> ...
>>> > >> (+) OK revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch
>>> > >> (+) OK revert-rcu-patches/0002-Revert-rcu-fix-spelling.patch
>>> > >> (+) OK revert-rcu-patches/0003-Revert-rcu-fix-rcu_cpu_kthread_task-synchronization.patch
>>> > >> (+) OK revert-rcu-patches/0004-Revert-rcu-call-__rcu_read_unlock-in-exit_rcu-for-tr.patch
>>> > >> (+) OK revert-rcu-patches/0005-Revert-rcu-Converge-TINY_RCU-expedited-and-normal-bo.patch
>>> > >> (+) OK revert-rcu-patches/0006-Revert-rcu-remove-useless-boosted_this_gp-field.patch
>>> > >> (+) OK revert-rcu-patches/0007-Revert-rcu-code-cleanups-in-TINY_RCU-priority-boosti.patch
>>> > >> (+) OK revert-rcu-patches/0008-Revert-rcu-Switch-to-this_cpu-primitives.patch
>>> > >> (+) OK revert-rcu-patches/0009-Revert-rcu-Use-WARN_ON_ONCE-for-DEBUG_OBJECTS_RCU_HE.patch
>>> > >> (+) OK revert-rcu-patches/0010-Revert-rcu-Enable-DEBUG_OBJECTS_RCU_HEAD-from-PREEMP.patch
>>> > >> (+) OK revert-rcu-patches/0011-Revert-rcu-Add-boosting-to-TREE_PREEMPT_RCU-tracing.patch
>>> > >> (+) OK revert-rcu-patches/0012-Revert-rcu-eliminate-unused-boosting-statistics.patch
>>> > >> (+) OK revert-rcu-patches/0013-Revert-rcu-priority-boosting-for-TREE_PREEMPT_RCU.patch
>>> > >> (+) OK revert-rcu-patches/0014-Revert-rcu-move-TREE_RCU-from-softirq-to-kthread.patch
>>> > >> ...
>>> > >>
>>> > >> Hope this helps to narrow down the problem.
>>> > >>
>>> > >> As I kept all kernels I can have a look at the tasks consuming high
>>> > >> CPU usage tomorrow.
>>> > >
>>> > > Could you please?
>>> >
>>> > I recalled (as you say I requested over and over again from you :-)) I
>>> > looked with top, htop and 'ps axu', but there was nothing special.
>>> > Sometimes the system got frozen - at this point (or short before) I
>>> > did not see anything suspicious with top.
>>>
>>> OK, thank you for the info.
>>>
>>> > > Also, could you please mount debugfs and list out the files in the
>>> > > "rcu" directory? The "ql=" value from the "rcu/rcudata" file is of
>>> > > particular interest.
>>> > >
>>> >
>>> > Ah, before I forget...
>>> >
>>> > I used TREE_RCU (was the default before noticing RCU issue) for
>>> > finding the culprit commit.
>>> > If it is from your POV more helpful to switch to PREEMPT + PREEMPT_RCU
>>> > + RCU_BOOST, please let me *now* know.
>>> > ( Both RCU setups freaks up the system. )
>>>
>>> If TREE_RCU hits problems faster, it is probably best to stay with
>>> TREE_RCU.
>>
>> And of course, one exception to this advice is if TREE_RCU hangs so hard
>> and fast that you don't have time to get any diagnostics. If this is the
>> case, then TREE_PREEMPT_RCU might be more productive.
>>
>
> OK, that would somehow explain why I could not really get some debug
> infos when doing "my stress-test" and checking via:
>
> $ LC_ALL=C tail -f /sys/kernel/debug/rcu/rcudata
>
> Then I remembered I saw a snippet for a RCU torture script mentionned
> in the kernel-docs (see Documentation/RCU/torture.txt).
>
> 189 The following script may be used to torture RCU:
> 190
> 191 #!/bin/sh
> 192
> 193 modprobe rcutorture
> 194 sleep 100
> 195 rmmod rcutorture
> 196 dmesg | grep torture:
>
> So, I recompiled a new TREE_RC-based kernel and build with
> CONFIG_RCU_TORTURE_TEST=m.
>
> Unfortunately, the rmmod (I prefer modprobe -r -v) hangs... the
> messages in the logs look promising.
>
> - Sedat -
>
Wrong attachment, correct attached.
- Sedat -
>
>
>> Thanx, Paul
>>
>>> > I think top & Co. are not enough to track the problem down.
>>> > I have seen tracing and debugging facililities for RCU.
>>> >
>>> > Some questions to debug and trace setup:
>>> >
>>> > Case #1: TREE_RCU
>>> >
>>> > CONFIG_RCU_TRACE=y
>>> > CONFIG_TREE_RCU_TRACE=y
>>>
>>> Yep.
>>>
>>> > Case #2: PREEMPT + PREEMPT_RCU + RCU_BOOST
>>> >
>>> > CONFIG_RCU_TRACE=y
>>> > CONFIG_TREE_RCU_TRACE=y
>>> > CONFIG_DEBUG_PREEMPT=y <--- Helpful?
>>> > CONFIG_PREEMPT_TRACER=y <--- Helpful?
>>> >
>>> > Any other recommends for useful/helpful trace and/or debug options?
>>> >
>>> > Any other intructions for debugging/tracing?
>>>
>>> Not at the moment. I will be looking at diagnostics will going
>>> through the code, so might have something later.
>>>
>>> > BTW, today's linux-next (next-20110328) is still freaky, I applied the
>>> > revert-rcu-patches patchset and all is fine.
>>>
>>> I reverted back to the commit preceding the one you pointed out last night
>>> my time, so the upcoming -next should be less freaky.
>>>
>>> > - Sedat -
>>> >
>>> > P.S.: Note to myself
>>> >
>>> > # mount -t debugfs none /sys/kernel/debug/
>>> > # ln -s /sys/kernel/debug /debug
>>> >
>>> > # find /debug -name rcu
>>>
>>> Or:
>>>
>>> # cd /debug/rcu
>>>
>>> then dump out everything except for the .csv file (which is the same
>>> as the non-.csv equivalent, but in spreadsheet format -- intended
>>> for systems with 100s or 1000s of CPUs).
>>>
>>> Thanx, Paul
>>
>
[-- Attachment #2: msg_rcu-torture_correct.txt --]
[-- Type: text/plain, Size: 39880 bytes --]
Mar 28 18:09:03 tbox kernel: [ 145.596004] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
Mar 28 18:09:03 tbox kernel: [ 145.596004] sending NMI to all CPUs:
Mar 28 18:09:03 tbox kernel: [ 145.596004] NMI backtrace for cpu 0
Mar 28 18:09:03 tbox kernel: [ 145.596004] Modules linked in: rcutorture acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative ppdev lp aes_i586 aes_generic rfcomm bnep bluetooth binfmt_misc fuse snd_intel8x0m snd_intel8x0 snd_ac97_codec radeon ac97_bus snd_pcm_oss arc4 snd_mixer_oss ecb ath5k joydev snd_pcm thinkpad_acpi pcmcia snd_seq_midi ath ttm mac80211 snd_rawmidi drm_kms_helper drm cfg80211 yenta_socket snd_seq_midi_event pcmcia_rsrc snd_seq pcmcia_core snd_timer i2c_algo_bit rfkill snd_seq_device psmouse snd nsc_ircc tpm_tis parport_pc battery ac pcspkr tpm irda i2c_i801 shpchp serio_raw tpm_bios soundcore processor i2c_core evdev pci_hotplug snd_page_alloc crc_ccitt parport nvram power_supply video button ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd scsi_mod usbcore e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
Mar 28 18:09:03 tbox kernel: [ 145.596004]
Mar 28 18:09:03 tbox kernel: [ 145.596004] Pid: 1659, comm: rcu_torture_rea Not tainted 2.6.38-next20110328-3-686-iniza #1 IBM 2374SG6/2374SG6
Mar 28 18:09:03 tbox kernel: [ 145.596004] EIP: 0060:[<c114b8d6>] EFLAGS: 00000046 CPU: 0
Mar 28 18:09:03 tbox kernel: [ 145.596004] EIP is at paravirt_read_tsc+0x6/0xd
Mar 28 18:09:03 tbox kernel: [ 145.596004] EAX: 99652d8e EBX: 99652d00 ECX: 00000000 EDX: 0000002c
Mar 28 18:09:03 tbox kernel: [ 145.596004] ESI: 00000000 EDI: 0019a58a EBP: f5075ddc ESP: f5075dd8
Mar 28 18:09:03 tbox kernel: [ 145.596004] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Mar 28 18:09:03 tbox kernel: [ 145.596004] Process rcu_torture_rea (pid: 1659, ti=f5074000 task=f637d400 task.ti=f5074000)
Mar 28 18:09:03 tbox kernel: [ 145.596004] Stack:
Mar 28 18:09:03 tbox kernel: [ 145.596004] 0019a58a f5075df4 c114b972 0019a589 00002710 c13d7500 c13d7580 f5075dfc
Mar 28 18:09:03 tbox kernel: [ 145.596004] c114b90f f5075e04 c114b92b f5075e10 c10164ca c13d7500 f5075e2c c1074e8d
Mar 28 18:09:03 tbox kernel: [ 145.596004] c133c4ef c133c5be 00000000 00003a98 f6403488 f5075e40 c1074f02 00000000
Mar 28 18:09:03 tbox kernel: [ 145.596004] Call Trace:
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b972>] delay_tsc+0x28/0x5e
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b90f>] __delay+0x9/0xb
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b92b>] __const_udelay+0x1a/0x1c
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1015892>] smp_apic_timer_interrupt+0x5b/0x6e
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1297789>] apic_timer_interrupt+0x31/0x38
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c10300e0>] ? resume_console+0x3/0x27
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b8d6>] ? paravirt_read_tsc+0x6/0xd
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b972>] delay_tsc+0x28/0x5e
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b90f>] __delay+0x9/0xb
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b92b>] __const_udelay+0x1a/0x1c
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<f82e23fa>] rcu_read_delay+0x2f/0x59 [rcutorture]
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<f82e2c77>] rcu_torture_reader+0xf5/0x1c4 [rcutorture]
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<f82e250a>] ? rcu_torture_shuffle_tasks+0xe6/0xe6 [rcutorture]
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<f82e2b82>] ? rcu_torture_writer+0x171/0x171 [rcutorture]
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c10444f6>] kthread+0x62/0x67
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1044494>] ? kthread_worker_fn+0x111/0x111
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c129ca7e>] kernel_thread_helper+0x6/0xd
Mar 28 18:09:03 tbox kernel: [ 145.596004] Code: 02 72 11 66 8b 16 8d 76 02 66 89 17 8d 7f 02 74 07 c1 e2 10 8a 16 88 17 01 d0 83 d0 00 5e 5f 5b c3 90 90 90 90 55 89 e5 57 0f 31 <8d> 74 26 00 5f 5d c3 55 89 e5 85 c0 74 1f eb 0a 8d 76 00 8d bc
Mar 28 18:09:03 tbox kernel: [ 145.596004] Call Trace:
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b972>] delay_tsc+0x28/0x5e
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b90f>] __delay+0x9/0xb
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b92b>] __const_udelay+0x1a/0x1c
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1015892>] smp_apic_timer_interrupt+0x5b/0x6e
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1297789>] apic_timer_interrupt+0x31/0x38
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c10300e0>] ? resume_console+0x3/0x27
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b8d6>] ? paravirt_read_tsc+0x6/0xd
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b972>] delay_tsc+0x28/0x5e
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b90f>] __delay+0x9/0xb
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c114b92b>] __const_udelay+0x1a/0x1c
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<f82e23fa>] rcu_read_delay+0x2f/0x59 [rcutorture]
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<f82e2c77>] rcu_torture_reader+0xf5/0x1c4 [rcutorture]
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<f82e250a>] ? rcu_torture_shuffle_tasks+0xe6/0xe6 [rcutorture]
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<f82e2b82>] ? rcu_torture_writer+0x171/0x171 [rcutorture]
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c10444f6>] kthread+0x62/0x67
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c1044494>] ? kthread_worker_fn+0x111/0x111
Mar 28 18:09:03 tbox kernel: [ 145.596004] [<c129ca7e>] kernel_thread_helper+0x6/0xd
Mar 28 18:12:03 tbox kernel: [ 325.716030] INFO: rcu_sched_state detected stall on CPU 0 (t=60030 jiffies)
Mar 28 18:12:03 tbox kernel: [ 325.720011] sending NMI to all CPUs:
Mar 28 18:12:03 tbox kernel: [ 325.720011] NMI backtrace for cpu 0
Mar 28 18:12:03 tbox kernel: [ 325.720011] Modules linked in: rcutorture(-) acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative ppdev lp aes_i586 aes_generic rfcomm bnep bluetooth binfmt_misc fuse snd_intel8x0m snd_intel8x0 snd_ac97_codec radeon ac97_bus snd_pcm_oss arc4 snd_mixer_oss ecb ath5k joydev snd_pcm thinkpad_acpi pcmcia snd_seq_midi ath ttm mac80211 snd_rawmidi drm_kms_helper drm cfg80211 yenta_socket snd_seq_midi_event pcmcia_rsrc snd_seq pcmcia_core snd_timer i2c_algo_bit rfkill snd_seq_device psmouse snd nsc_ircc tpm_tis parport_pc battery ac pcspkr tpm irda i2c_i801 shpchp serio_raw tpm_bios soundcore processor i2c_core evdev pci_hotplug snd_page_alloc crc_ccitt parport nvram power_supply video button ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd scsi_mod usbcore e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
Mar 28 18:12:03 tbox kernel: [ 325.720011]
Mar 28 18:12:03 tbox kernel: [ 325.720011] Pid: 0, comm: swapper Not tainted 2.6.38-next20110328-3-686-iniza #1 IBM 2374SG6/2374SG6
Mar 28 18:12:03 tbox kernel: [ 325.720011] EIP: 0060:[<c114b94a>] EFLAGS: 00000807 CPU: 0
Mar 28 18:12:03 tbox kernel: [ 325.720011] EIP is at delay_tsc+0x0/0x5e
Mar 28 18:12:03 tbox kernel: [ 325.720011] EAX: 00090d40 EBX: 00002710 ECX: c132e541 EDX: 00090d3f
Mar 28 18:12:03 tbox kernel: [ 325.720011] ESI: c13d7500 EDI: c13d7580 EBP: f6019ea4 ESP: f6019ea0
Mar 28 18:12:03 tbox kernel: [ 325.720011] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Mar 28 18:12:03 tbox kernel: [ 325.720011] Process swapper (pid: 0, ti=f6018000 task=c13ccfa0 task.ti=c13a0000)
Mar 28 18:12:03 tbox kernel: [ 325.720011] Stack:
Mar 28 18:12:03 tbox kernel: [ 325.720011] c114b90f f6019eac c114b92b f6019eb8 c10164ca c13d7500 f6019ed4 c1074e8d
Mar 28 18:12:03 tbox kernel: [ 325.720011] c133c4ef c133c5be 00000000 0000ea7e f6403488 f6019ee8 c1074f02 00000000
Mar 28 18:12:03 tbox kernel: [ 325.720011] 00000000 c13ccfa0 f6019ef4 c10752ab 00000000 f6019f08 c1039c2c c13a1ef8
Mar 28 18:12:03 tbox kernel: [ 325.720011] Call Trace:
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c114b90f>] ? __delay+0x9/0xb
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c114b92b>] __const_udelay+0x1a/0x1c
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:12:03 tbox kernel: [ 325.720011] <IRQ>
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:12:03 tbox kernel: [ 325.720011] Code: d2 3e f7 e2 8d 42 01 e8 db ff ff ff 5d c3 69 c0 c7 10 00 00 55 89 e5 e8 d6 ff ff ff 5d c3 55 89 e5 8d 04 80 e8 c9 ff ff ff 5d c3
Mar 28 18:12:03 tbox kernel: [ 325.720011] Call Trace:
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c114b90f>] ? __delay+0x9/0xb
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c114b92b>] __const_udelay+0x1a/0x1c
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:12:03 tbox kernel: [ 325.720011] <IRQ> [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:12:03 tbox kernel: [ 325.720011] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:12:37 tbox kernel: [ 360.304066] INFO: task modprobe:1662 blocked for more than 120 seconds.
Mar 28 18:12:37 tbox kernel: [ 360.314213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 28 18:12:37 tbox kernel: [ 360.324527] modprobe D f5127ea4 0 1662 1651 0x00000000
Mar 28 18:12:37 tbox kernel: [ 360.335003] f5127e98 00000082 f594ac00 f5127ea4 c145d380 ed715c1f 0000002a c145d380
Mar 28 18:12:37 tbox kernel: [ 360.345745] 00000000 00000000 f594ac00 f637dc00 0000002a f594ac00 ffffffff ffffffff
Mar 28 18:12:37 tbox kernel: [ 360.356521] f5127ed8 ed715020 f64063c4 ed715c1f 00000000 3d090000 00000000 f594ac2c
Mar 28 18:12:37 tbox kernel: [ 360.367389] Call Trace:
Mar 28 18:12:37 tbox kernel: [ 360.378083] [<c10215b4>] ? wakeup_preempt_entity+0x36/0x53
Mar 28 18:12:37 tbox kernel: [ 360.388915] [<c1295ed0>] schedule_timeout+0x21/0xaa
Mar 28 18:12:37 tbox kernel: [ 360.399708] [<c1295ed0>] ? schedule_timeout+0x21/0xaa
Mar 28 18:12:37 tbox kernel: [ 360.410473] [<c114507f>] ? cpumask_next_and+0x23/0x2f
Mar 28 18:12:37 tbox kernel: [ 360.421240] [<c1295d41>] wait_for_common+0x77/0xcd
Mar 28 18:12:37 tbox kernel: [ 360.432032] [<c102a570>] ? try_to_wake_up+0x142/0x142
Mar 28 18:12:37 tbox kernel: [ 360.442808] [<c107499c>] ? call_rcu_bh+0xf/0xf
Mar 28 18:12:37 tbox kernel: [ 360.453598] [<c1295e18>] wait_for_completion+0x12/0x14
Mar 28 18:12:37 tbox kernel: [ 360.464436] [<c1074df6>] _rcu_barrier.isra.41+0x78/0x85
Mar 28 18:12:37 tbox kernel: [ 360.475315] [<c1074e10>] rcu_barrier_sched+0xd/0xf
Mar 28 18:12:37 tbox kernel: [ 360.485900] [<c1074e1a>] rcu_barrier+0x8/0xa
Mar 28 18:12:37 tbox kernel: [ 360.496150] [<f82e35df>] rcu_torture_cleanup+0x2cd/0x317 [rcutorture]
Mar 28 18:12:37 tbox kernel: [ 360.506544] [<c1057d05>] sys_delete_module+0x198/0x1f5
Mar 28 18:12:37 tbox kernel: [ 360.516992] [<c1075093>] ? rcu_enter_nohz+0x28/0x9c
Mar 28 18:12:37 tbox kernel: [ 360.527410] [<c1075219>] ? rcu_irq_exit+0x8/0xa
Mar 28 18:12:37 tbox kernel: [ 360.537813] [<c129c4df>] sysenter_do_call+0x12/0x28
Mar 28 18:14:37 tbox kernel: [ 480.548068] INFO: task modprobe:1662 blocked for more than 120 seconds.
Mar 28 18:14:37 tbox kernel: [ 480.558653] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 28 18:14:37 tbox kernel: [ 480.569378] modprobe D f5127ea4 0 1662 1651 0x00000000
Mar 28 18:14:37 tbox kernel: [ 480.580205] f5127e98 00000082 f594ac00 f5127ea4 c145d380 ed715c1f 0000002a c145d380
Mar 28 18:14:37 tbox kernel: [ 480.591303] 00000000 00000000 f594ac00 f637dc00 0000002a f594ac00 ffffffff ffffffff
Mar 28 18:14:37 tbox kernel: [ 480.602479] f5127ed8 ed715020 f64063c4 ed715c1f 00000000 3d090000 00000000 f594ac2c
Mar 28 18:14:37 tbox kernel: [ 480.613746] Call Trace:
Mar 28 18:14:37 tbox kernel: [ 480.624810] [<c10215b4>] ? wakeup_preempt_entity+0x36/0x53
Mar 28 18:14:37 tbox kernel: [ 480.636044] [<c1295ed0>] schedule_timeout+0x21/0xaa
Mar 28 18:14:37 tbox kernel: [ 480.647248] [<c1295ed0>] ? schedule_timeout+0x21/0xaa
Mar 28 18:14:37 tbox kernel: [ 480.658528] [<c114507f>] ? cpumask_next_and+0x23/0x2f
Mar 28 18:14:37 tbox kernel: [ 480.669762] [<c1295d41>] wait_for_common+0x77/0xcd
Mar 28 18:14:37 tbox kernel: [ 480.680957] [<c102a570>] ? try_to_wake_up+0x142/0x142
Mar 28 18:14:37 tbox kernel: [ 480.692187] [<c107499c>] ? call_rcu_bh+0xf/0xf
Mar 28 18:14:37 tbox kernel: [ 480.703404] [<c1295e18>] wait_for_completion+0x12/0x14
Mar 28 18:14:37 tbox kernel: [ 480.714691] [<c1074df6>] _rcu_barrier.isra.41+0x78/0x85
Mar 28 18:14:37 tbox kernel: [ 480.725994] [<c1074e10>] rcu_barrier_sched+0xd/0xf
Mar 28 18:14:37 tbox kernel: [ 480.737266] [<c1074e1a>] rcu_barrier+0x8/0xa
Mar 28 18:14:37 tbox kernel: [ 480.748460] [<f82e35df>] rcu_torture_cleanup+0x2cd/0x317 [rcutorture]
Mar 28 18:14:37 tbox kernel: [ 480.759798] [<c1057d05>] sys_delete_module+0x198/0x1f5
Mar 28 18:14:37 tbox kernel: [ 480.771072] [<c1075093>] ? rcu_enter_nohz+0x28/0x9c
Mar 28 18:14:37 tbox kernel: [ 480.782354] [<c1075219>] ? rcu_irq_exit+0x8/0xa
Mar 28 18:14:37 tbox kernel: [ 480.793576] [<c129c4df>] sysenter_do_call+0x12/0x28
Mar 28 18:15:03 tbox kernel: [ 505.836029] INFO: rcu_sched_state detected stall on CPU 0 (t=105060 jiffies)
Mar 28 18:15:03 tbox kernel: [ 505.840011] sending NMI to all CPUs:
Mar 28 18:15:03 tbox kernel: [ 505.840011] NMI backtrace for cpu 0
Mar 28 18:15:03 tbox kernel: [ 505.840011] Modules linked in: rcutorture(-) acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative ppdev lp aes_i586 aes_generic rfcomm bnep bluetooth binfmt_misc fuse snd_intel8x0m snd_intel8x0 snd_ac97_codec radeon ac97_bus snd_pcm_oss arc4 snd_mixer_oss ecb ath5k joydev snd_pcm thinkpad_acpi pcmcia snd_seq_midi ath ttm mac80211 snd_rawmidi drm_kms_helper drm cfg80211 yenta_socket snd_seq_midi_event pcmcia_rsrc snd_seq pcmcia_core snd_timer i2c_algo_bit rfkill snd_seq_device psmouse snd nsc_ircc tpm_tis parport_pc battery ac pcspkr tpm irda i2c_i801 shpchp serio_raw tpm_bios soundcore processor i2c_core evdev pci_hotplug snd_page_alloc crc_ccitt parport nvram power_supply video button ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd scsi_mod usbcore e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
Mar 28 18:15:03 tbox kernel: [ 505.840011]
Mar 28 18:15:03 tbox kernel: [ 505.840011] Pid: 0, comm: swapper Not tainted 2.6.38-next20110328-3-686-iniza #1 IBM 2374SG6/2374SG6
Mar 28 18:15:03 tbox kernel: [ 505.840011] EIP: 0060:[<c114b91b>] EFLAGS: 00000002 CPU: 0
Mar 28 18:15:03 tbox kernel: [ 505.840011] EIP is at __const_udelay+0xa/0x1c
Mar 28 18:15:03 tbox kernel: [ 505.840011] EAX: 00418958 EBX: 00002710 ECX: c132e541 EDX: 00247fab
Mar 28 18:15:03 tbox kernel: [ 505.840011] ESI: c13d7500 EDI: c13d7580 EBP: f6019eac ESP: f6019eac
Mar 28 18:15:03 tbox kernel: [ 505.840011] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Mar 28 18:15:03 tbox kernel: [ 505.840011] Process swapper (pid: 0, ti=f6018000 task=c13ccfa0 task.ti=c13a0000)
Mar 28 18:15:03 tbox kernel: [ 505.840011] Stack:
Mar 28 18:15:03 tbox kernel: [ 505.840011] f6019eb8 c10164ca c13d7500 f6019ed4 c1074e8d c133c4ef c133c5be 00000000
Mar 28 18:15:03 tbox kernel: [ 505.840011] 00019a64 f6403488 f6019ee8 c1074f02 00000000 00000000 c13ccfa0 f6019ef4
Mar 28 18:15:03 tbox kernel: [ 505.840011] c10752ab 00000000 f6019f08 c1039c2c c13a1ef8 c62d1b06 00000075 f6019f20
Mar 28 18:15:03 tbox kernel: [ 505.840011] Call Trace:
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:15:03 tbox kernel: [ 505.840011] <IRQ>
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:15:03 tbox kernel: [ 505.840011] Code: eb 0e 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 48 75 fd 48 5d c3 55 89 e5 ff 15 a8 81 3e c1 5d c3 55 89 e5 64 8b 15 dc d2 45 c1 <c1> e0 02 6b d2 3e f7 e2 8d 42 01 e8 db ff ff ff 5d c3 69 c0 c7
Mar 28 18:15:03 tbox kernel: [ 505.840011] Call Trace:
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:15:03 tbox kernel: [ 505.840011] <IRQ> [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:15:03 tbox kernel: [ 505.840011] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:16:38 tbox kernel: [ 600.804066] INFO: task modprobe:1662 blocked for more than 120 seconds.
Mar 28 18:16:38 tbox kernel: [ 600.814684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 28 18:16:38 tbox kernel: [ 600.825484] modprobe D f5127ea4 0 1662 1651 0x00000000
Mar 28 18:16:38 tbox kernel: [ 600.836381] f5127e98 00000082 f594ac00 f5127ea4 c145d380 ed715c1f 0000002a c145d380
Mar 28 18:16:38 tbox kernel: [ 600.847562] 00000000 00000000 f594ac00 f637dc00 0000002a f594ac00 ffffffff ffffffff
Mar 28 18:16:38 tbox kernel: [ 600.858807] f5127ed8 ed715020 f64063c4 ed715c1f 00000000 3d090000 00000000 f594ac2c
Mar 28 18:16:38 tbox kernel: [ 600.870196] Call Trace:
Mar 28 18:16:38 tbox kernel: [ 600.881367] [<c10215b4>] ? wakeup_preempt_entity+0x36/0x53
Mar 28 18:16:38 tbox kernel: [ 600.892671] [<c1295ed0>] schedule_timeout+0x21/0xaa
Mar 28 18:16:38 tbox kernel: [ 600.903940] [<c1295ed0>] ? schedule_timeout+0x21/0xaa
Mar 28 18:16:38 tbox kernel: [ 600.915184] [<c114507f>] ? cpumask_next_and+0x23/0x2f
Mar 28 18:16:38 tbox kernel: [ 600.926423] [<c1295d41>] wait_for_common+0x77/0xcd
Mar 28 18:16:38 tbox kernel: [ 600.937628] [<c102a570>] ? try_to_wake_up+0x142/0x142
Mar 28 18:16:38 tbox kernel: [ 600.948873] [<c107499c>] ? call_rcu_bh+0xf/0xf
Mar 28 18:16:38 tbox kernel: [ 600.960134] [<c1295e18>] wait_for_completion+0x12/0x14
Mar 28 18:16:38 tbox kernel: [ 600.971503] [<c1074df6>] _rcu_barrier.isra.41+0x78/0x85
Mar 28 18:16:38 tbox kernel: [ 600.982899] [<c1074e10>] rcu_barrier_sched+0xd/0xf
Mar 28 18:16:38 tbox kernel: [ 600.993893] [<c1074e1a>] rcu_barrier+0x8/0xa
Mar 28 18:16:38 tbox kernel: [ 601.004417] [<f82e35df>] rcu_torture_cleanup+0x2cd/0x317 [rcutorture]
Mar 28 18:16:38 tbox kernel: [ 601.015076] [<c1057d05>] sys_delete_module+0x198/0x1f5
Mar 28 18:16:38 tbox kernel: [ 601.025690] [<c1075093>] ? rcu_enter_nohz+0x28/0x9c
Mar 28 18:16:38 tbox kernel: [ 601.036275] [<c1075219>] ? rcu_irq_exit+0x8/0xa
Mar 28 18:16:38 tbox kernel: [ 601.046855] [<c129c4df>] sysenter_do_call+0x12/0x28
Mar 28 18:17:01 tbox /USR/SBIN/CRON[1665]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Mar 28 18:18:04 tbox kernel: [ 685.956029] INFO: rcu_sched_state detected stall on CPU 0 (t=150090 jiffies)
Mar 28 18:18:04 tbox kernel: [ 685.960012] sending NMI to all CPUs:
Mar 28 18:18:04 tbox kernel: [ 685.960012] NMI backtrace for cpu 0
Mar 28 18:18:04 tbox kernel: [ 685.960012] Modules linked in: rcutorture(-) acpi_cpufreq mperf cpufreq_powersave cpufreq_userspace cpufreq_stats cpufreq_conservative ppdev lp aes_i586 aes_generic rfcomm bnep bluetooth binfmt_misc fuse snd_intel8x0m snd_intel8x0 snd_ac97_codec radeon ac97_bus snd_pcm_oss arc4 snd_mixer_oss ecb ath5k joydev snd_pcm thinkpad_acpi pcmcia snd_seq_midi ath ttm mac80211 snd_rawmidi drm_kms_helper drm cfg80211 yenta_socket snd_seq_midi_event pcmcia_rsrc snd_seq pcmcia_core snd_timer i2c_algo_bit rfkill snd_seq_device psmouse snd nsc_ircc tpm_tis parport_pc battery ac pcspkr tpm irda i2c_i801 shpchp serio_raw tpm_bios soundcore processor i2c_core evdev pci_hotplug snd_page_alloc crc_ccitt parport nvram power_supply video button ext4 mbcache jbd2 crc16 dm_mod usbhid hid usb_storage uas sg sr_mod sd_mod cdrom crc_t10dif ata_generic ata_piix uhci_hcd libata ehci_hcd scsi_mod usbcore e1000 thermal thermal_sys floppy [last unloaded: scsi_wait_scan]
Mar 28 18:18:04 tbox kernel: [ 685.960012]
Mar 28 18:18:04 tbox kernel: [ 685.960012] Pid: 0, comm: swapper Not tainted 2.6.38-next20110328-3-686-iniza #1 IBM 2374SG6/2374SG6
Mar 28 18:18:04 tbox kernel: [ 685.960012] EIP: 0060:[<c114b923>] EFLAGS: 00000807 CPU: 0
Mar 28 18:18:04 tbox kernel: [ 685.960012] EIP is at __const_udelay+0x12/0x1c
Mar 28 18:18:04 tbox kernel: [ 685.960012] EAX: 4f1a99c0 EBX: 00002710 ECX: c132e541 EDX: 00090d3f
Mar 28 18:18:04 tbox kernel: [ 685.960012] ESI: c13d7500 EDI: c13d7580 EBP: f6019eac ESP: f6019eac
Mar 28 18:18:04 tbox kernel: [ 685.960012] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Mar 28 18:18:04 tbox kernel: [ 685.960012] Process swapper (pid: 0, ti=f6018000 task=c13ccfa0 task.ti=c13a0000)
Mar 28 18:18:04 tbox kernel: [ 685.960012] Stack:
Mar 28 18:18:04 tbox kernel: [ 685.960012] f6019eb8 c10164ca c13d7500 f6019ed4 c1074e8d c133c4ef c133c5be 00000000
Mar 28 18:18:04 tbox kernel: [ 685.960012] 00024a4a f6403488 f6019ee8 c1074f02 00000000 00000000 c13ccfa0 f6019ef4
Mar 28 18:18:04 tbox kernel: [ 685.960012] c10752ab 00000000 f6019f08 c1039c2c c13a1ef8 b62a34e0 0000009f f6019f20
Mar 28 18:18:04 tbox kernel: [ 685.960012] Call Trace:
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:18:04 tbox kernel: [ 685.960012] <IRQ>
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:18:04 tbox kernel: [ 685.960012] Code: 00 8d bc 27 00 00 00 00 48 75 fd 48 5d c3 55 89 e5 ff 15 a8 81 3e c1 5d c3 55 89 e5 64 8b 15 dc d2 45 c1 c1 e0 02 6b d2 3e f7 e2 <8d> 42 01 e8 db ff ff ff 5d c3 69 c0 c7 10 00 00 55 89 e5 e8 d6
Mar 28 18:18:04 tbox kernel: [ 685.960012] Call Trace:
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10164ca>] arch_trigger_all_cpu_backtrace+0x50/0x62
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1074e8d>] check_cpu_stall+0x58/0xb8
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1074f02>] __rcu_pending+0x15/0xc4
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10752ab>] rcu_check_callbacks+0x6d/0x93
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1039c2c>] update_process_times+0x2d/0x58
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1050991>] tick_sched_timer+0x6b/0x9a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104713e>] __run_hrtimer+0x9c/0x111
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1050926>] ? tick_nohz_handler+0xdf/0xdf
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1047ae0>] hrtimer_interrupt+0xd6/0x1bb
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104fc49>] tick_do_broadcast.constprop.4+0x38/0x6a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c104fd28>] tick_handle_oneshot_broadcast+0xad/0xe1
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1004209>] timer_interrupt+0x15/0x1c
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c107140d>] handle_irq_event_percpu+0x4e/0x164
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1071559>] handle_irq_event+0x36/0x51
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072d62>] ? unmask_irq+0x25/0x25
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1072daf>] handle_level_irq+0x4d/0x63
Mar 28 18:18:04 tbox kernel: [ 685.960012] <IRQ> [<c1003b81>] ? do_IRQ+0x35/0x80
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c129ca70>] ? common_interrupt+0x30/0x38
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c10400d8>] ? worker_maybe_bind_and_lock+0x3e/0x83
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868e30f>] ? arch_local_irq_enable+0x5/0xb [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868eef5>] ? acpi_idle_enter_simple+0x100/0x138 [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<f868ec6a>] ? acpi_idle_enter_bm+0xd0/0x25b [processor]
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c11e1ea2>] ? cpuidle_idle_call+0xc2/0x137
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1001da3>] ? cpu_idle+0x89/0xa3
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c127d300>] ? rest_init+0x58/0x5a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c1401722>] ? start_kernel+0x315/0x31a
Mar 28 18:18:04 tbox kernel: [ 685.960012] [<c14010a2>] ? i386_start_kernel+0xa2/0xaa
Mar 28 18:18:38 tbox kernel: [ 721.056065] INFO: task modprobe:1662 blocked for more than 120 seconds.
Mar 28 18:18:38 tbox kernel: [ 721.066683] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 28 18:18:38 tbox kernel: [ 721.077467] modprobe D f5127ea4 0 1662 1651 0x00000000
Mar 28 18:18:38 tbox kernel: [ 721.088422] f5127e98 00000082 f594ac00 f5127ea4 c145d380 ed715c1f 0000002a c145d380
Mar 28 18:18:38 tbox kernel: [ 721.099608] 00000000 00000000 f594ac00 f637dc00 0000002a f594ac00 ffffffff ffffffff
Mar 28 18:18:38 tbox kernel: [ 721.110877] f5127ed8 ed715020 f64063c4 ed715c1f 00000000 3d090000 00000000 f594ac2c
Mar 28 18:18:38 tbox kernel: [ 721.122224] Call Trace:
Mar 28 18:18:38 tbox kernel: [ 721.133395] [<c10215b4>] ? wakeup_preempt_entity+0x36/0x53
Mar 28 18:18:38 tbox kernel: [ 721.144707] [<c1295ed0>] schedule_timeout+0x21/0xaa
Mar 28 18:18:38 tbox kernel: [ 721.155977] [<c1295ed0>] ? schedule_timeout+0x21/0xaa
Mar 28 18:18:38 tbox kernel: [ 721.167211] [<c114507f>] ? cpumask_next_and+0x23/0x2f
Mar 28 18:18:38 tbox kernel: [ 721.178445] [<c1295d41>] wait_for_common+0x77/0xcd
Mar 28 18:18:38 tbox kernel: [ 721.189693] [<c102a570>] ? try_to_wake_up+0x142/0x142
Mar 28 18:18:38 tbox kernel: [ 721.200939] [<c107499c>] ? call_rcu_bh+0xf/0xf
Mar 28 18:18:38 tbox kernel: [ 721.212205] [<c1295e18>] wait_for_completion+0x12/0x14
Mar 28 18:18:38 tbox kernel: [ 721.223518] [<c1074df6>] _rcu_barrier.isra.41+0x78/0x85
Mar 28 18:18:38 tbox kernel: [ 721.234876] [<c1074e10>] rcu_barrier_sched+0xd/0xf
Mar 28 18:18:38 tbox kernel: [ 721.245861] [<c1074e1a>] rcu_barrier+0x8/0xa
Mar 28 18:18:38 tbox kernel: [ 721.256390] [<f82e35df>] rcu_torture_cleanup+0x2cd/0x317 [rcutorture]
Mar 28 18:18:38 tbox kernel: [ 721.267042] [<c1057d05>] sys_delete_module+0x198/0x1f5
Mar 28 18:18:38 tbox kernel: [ 721.277652] [<c1075093>] ? rcu_enter_nohz+0x28/0x9c
Mar 28 18:18:38 tbox kernel: [ 721.288273] [<c1075219>] ? rcu_irq_exit+0x8/0xa
Mar 28 18:18:38 tbox kernel: [ 721.298872] [<c129c4df>] sysenter_do_call+0x12/0x28
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-28 16:46 ` Sedat Dilek
@ 2011-03-29 0:10 ` Paul E. McKenney
2011-03-29 2:42 ` Sedat Dilek
0 siblings, 1 reply; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-29 0:10 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Mon, Mar 28, 2011 at 06:46:48PM +0200, Sedat Dilek wrote:
> On Mon, Mar 28, 2011 at 6:38 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
> > On Mon, Mar 28, 2011 at 5:11 PM, Paul E. McKenney
> > <paulmck@linux.vnet.ibm.com> wrote:
> >> On Mon, Mar 28, 2011 at 06:24:36AM -0700, Paul E. McKenney wrote:
> >>> On Mon, Mar 28, 2011 at 02:33:36PM +0200, Sedat Dilek wrote:
[ . . . ]
> >>> > Ah, before I forget...
> >>> >
> >>> > I used TREE_RCU (was the default before noticing RCU issue) for
> >>> > finding the culprit commit.
> >>> > If it is from your POV more helpful to switch to PREEMPT + PREEMPT_RCU
> >>> > + RCU_BOOST, please let me *now* know.
> >>> > ( Both RCU setups freaks up the system. )
> >>>
> >>> If TREE_RCU hits problems faster, it is probably best to stay with
> >>> TREE_RCU.
> >>
> >> And of course, one exception to this advice is if TREE_RCU hangs so hard
> >> and fast that you don't have time to get any diagnostics. If this is the
> >> case, then TREE_PREEMPT_RCU might be more productive.
> >>
> >
> > OK, that would somehow explain why I could not really get some debug
> > infos when doing "my stress-test" and checking via:
> >
> > $ LC_ALL=C tail -f /sys/kernel/debug/rcu/rcudata
> >
> > Then I remembered I saw a snippet for a RCU torture script mentionned
> > in the kernel-docs (see Documentation/RCU/torture.txt).
> >
> > 189 The following script may be used to torture RCU:
> > 190
> > 191 #!/bin/sh
> > 192
> > 193 modprobe rcutorture
> > 194 sleep 100
> > 195 rmmod rcutorture
> > 196 dmesg | grep torture:
> >
> > So, I recompiled a new TREE_RC-based kernel and build with
> > CONFIG_RCU_TORTURE_TEST=m.
> >
> > Unfortunately, the rmmod (I prefer modprobe -r -v) hangs... the
> > messages in the logs look promising.
> >
> > - Sedat -
> >
>
> Wrong attachment, correct attached.
And one stupid problem located thus far. I can make a (tortured) case
for it resulting in the symptoms you see, but it does seem unlikely to
happen repeatedly, as it would require a burst of CPU just at the wrong
time. But who knows?
In any case, I am still looking.
Thanx, Paul
------------------------------------------------------------------------
Fix stupid typo.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 5477764..f311228 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1618,7 +1618,7 @@ static int rcu_node_kthread(void *arg)
rnp->wakemask = 0;
raw_spin_unlock_irqrestore(&rnp->lock, flags);
rcu_initiate_boost(rnp);
- for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask <<= 1) {
+ for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask >>= 1) {
if ((mask & 0x1) == 0)
continue;
preempt_disable();
^ permalink raw reply related [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-29 0:10 ` Paul E. McKenney
@ 2011-03-29 2:42 ` Sedat Dilek
2011-03-29 4:17 ` Paul E. McKenney
0 siblings, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-29 2:42 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
[-- Attachment #1: Type: text/plain, Size: 4537 bytes --]
On Tue, Mar 29, 2011 at 2:10 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Mon, Mar 28, 2011 at 06:46:48PM +0200, Sedat Dilek wrote:
>> On Mon, Mar 28, 2011 at 6:38 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
>> > On Mon, Mar 28, 2011 at 5:11 PM, Paul E. McKenney
>> > <paulmck@linux.vnet.ibm.com> wrote:
>> >> On Mon, Mar 28, 2011 at 06:24:36AM -0700, Paul E. McKenney wrote:
>> >>> On Mon, Mar 28, 2011 at 02:33:36PM +0200, Sedat Dilek wrote:
>
> [ . . . ]
>
>> >>> > Ah, before I forget...
>> >>> >
>> >>> > I used TREE_RCU (was the default before noticing RCU issue) for
>> >>> > finding the culprit commit.
>> >>> > If it is from your POV more helpful to switch to PREEMPT + PREEMPT_RCU
>> >>> > + RCU_BOOST, please let me *now* know.
>> >>> > ( Both RCU setups freaks up the system. )
>> >>>
>> >>> If TREE_RCU hits problems faster, it is probably best to stay with
>> >>> TREE_RCU.
>> >>
>> >> And of course, one exception to this advice is if TREE_RCU hangs so hard
>> >> and fast that you don't have time to get any diagnostics. If this is the
>> >> case, then TREE_PREEMPT_RCU might be more productive.
>> >>
>> >
>> > OK, that would somehow explain why I could not really get some debug
>> > infos when doing "my stress-test" and checking via:
>> >
>> > $ LC_ALL=C tail -f /sys/kernel/debug/rcu/rcudata
>> >
>> > Then I remembered I saw a snippet for a RCU torture script mentionned
>> > in the kernel-docs (see Documentation/RCU/torture.txt).
>> >
>> > 189 The following script may be used to torture RCU:
>> > 190
>> > 191 #!/bin/sh
>> > 192
>> > 193 modprobe rcutorture
>> > 194 sleep 100
>> > 195 rmmod rcutorture
>> > 196 dmesg | grep torture:
>> >
>> > So, I recompiled a new TREE_RC-based kernel and build with
>> > CONFIG_RCU_TORTURE_TEST=m.
>> >
>> > Unfortunately, the rmmod (I prefer modprobe -r -v) hangs... the
>> > messages in the logs look promising.
>> >
>> > - Sedat -
>> >
>>
>> Wrong attachment, correct attached.
>
> And one stupid problem located thus far. I can make a (tortured) case
> for it resulting in the symptoms you see, but it does seem unlikely to
> happen repeatedly, as it would require a burst of CPU just at the wrong
> time. But who knows?
>
> In any case, I am still looking.
>
> Thanx, Paul
>
> ------------------------------------------------------------------------
>
> Fix stupid typo.
>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 5477764..f311228 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1618,7 +1618,7 @@ static int rcu_node_kthread(void *arg)
> rnp->wakemask = 0;
> raw_spin_unlock_irqrestore(&rnp->lock, flags);
> rcu_initiate_boost(rnp);
> - for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask <<= 1) {
> + for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask >>= 1) {
> if ((mask & 0x1) == 0)
> continue;
> preempt_disable();
>
I have tested this patch and the previous one you send:
(+) OK rcu-fix/rcu-further-lower-priority-in-rcu_yield.patch
(+) OK rcu-fix/Fix-stupid-typo.patch
As you suggested I switched to PREEMPT and RCU with rcu-boost:
# egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110328-5-686-iniza
# RCU Subsystem
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_TRACE=y
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_TREE_RCU_TRACE=y
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=1
CONFIG_RCU_BOOST_DELAY=500
CONFIG_NO_HZ=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_DEBUG_PREEMPT=y
# CONFIG_SPARSE_RCU_POINTER is not set
CONFIG_RCU_TORTURE_TEST=m
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_VERBOSE=y
CONFIG_PREEMPT_TRACER=y
Unfortunately, the rcu-torture script hangs again on unloading the
rcu-torture-test module.
Attached are RCU-related messages in my logs.
( I tailed for rcudata changes - no logs. )
- Sedat -
[-- Attachment #2: msg_rcu-torture_20110329.txt --]
[-- Type: text/plain, Size: 26844 bytes --]
Mar 29 04:21:20 tbox kernel: [ 132.160108] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/1 test_boost_interval=7 test_boost_duration=4
Mar 29 04:25:08 tbox kernel: [ 360.352072] INFO: task rcu_torture_fak:1750 blocked for more than 120 seconds.
Mar 29 04:25:08 tbox kernel: [ 360.364505] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:25:08 tbox kernel: [ 360.377380] rcu_torture_fak D 00000036 0 1750 2 0x00000000
Mar 29 04:25:08 tbox kernel: [ 360.389888] f5169f08 00000046 10c2d7ee 00000036 c1467400 000036da 00000000 c1467400
Mar 29 04:25:08 tbox kernel: [ 360.405450] 00000000 00000036 f5971ce0 c108395c c129f22b 00000001 f5169ed0 c12a313a
Mar 29 04:25:08 tbox kernel: [ 360.417251] f6406400 f5169ed8 c1025e91 f5169f54 c129f230 1a975683 00000021 c1467400
Mar 29 04:25:08 tbox kernel: [ 360.429130] Call Trace:
Mar 29 04:25:08 tbox kernel: [ 360.440754] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:25:08 tbox kernel: [ 360.452377] [<c129f22b>] ? schedule+0x3e5/0x3fa
Mar 29 04:25:08 tbox kernel: [ 360.463904] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:08 tbox kernel: [ 360.475636] [<c1025e91>] ? need_resched+0x14/0x1e
Mar 29 04:25:08 tbox kernel: [ 360.487202] [<c129f230>] ? schedule+0x3ea/0x3fa
Mar 29 04:25:08 tbox kernel: [ 360.498801] [<c108397d>] ? trace_preempt_off+0xf/0x22
Mar 29 04:25:08 tbox kernel: [ 360.510305] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
Mar 29 04:25:08 tbox kernel: [ 360.521903] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:25:08 tbox kernel: [ 360.533342] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:08 tbox kernel: [ 360.544920] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:25:08 tbox kernel: [ 360.556383] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:25:08 tbox kernel: [ 360.567690] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:08 tbox kernel: [ 360.578785] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:25:08 tbox kernel: [ 360.589832] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:25:08 tbox kernel: [ 360.600831] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:25:08 tbox kernel: [ 360.611948] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
Mar 29 04:25:08 tbox kernel: [ 360.623083] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:25:08 tbox kernel: [ 360.634254] [<c1077004>] synchronize_rcu+0x38/0x3a
Mar 29 04:25:08 tbox kernel: [ 360.645301] [<c1043868>] ? find_ge_pid+0x2f/0x2f
Mar 29 04:25:08 tbox kernel: [ 360.656373] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
Mar 29 04:25:08 tbox kernel: [ 360.667452] [<c1045511>] kthread+0x62/0x67
Mar 29 04:25:08 tbox kernel: [ 360.678588] [<c10454af>] ? kthread_worker_fn+0x111/0x111
Mar 29 04:25:08 tbox kernel: [ 360.689635] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
Mar 29 04:25:08 tbox kernel: [ 360.700677] INFO: task rcu_torture_fak:1751 blocked for more than 120 seconds.
Mar 29 04:25:08 tbox kernel: [ 360.711751] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:25:08 tbox kernel: [ 360.723000] rcu_torture_fak D 00000021 0 1751 2 0x00000000
Mar 29 04:25:08 tbox kernel: [ 360.734284] f5807f08 00000046 1b8afabc 00000021 c1467400 21ab5ecd 00000000 c1467400
Mar 29 04:25:08 tbox kernel: [ 360.745904] 00000000 00000021 f5970840 c108395c c129f22b 00000001 f5807ed0 c12a313a
Mar 29 04:25:08 tbox kernel: [ 360.757524] f6406400 f5807ed8 c1025e91 f5807f54 c129f230 1a975e7e 00000021 c1467400
Mar 29 04:25:08 tbox kernel: [ 360.769310] Call Trace:
Mar 29 04:25:08 tbox kernel: [ 360.780844] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:25:08 tbox kernel: [ 360.792587] [<c129f22b>] ? schedule+0x3e5/0x3fa
Mar 29 04:25:08 tbox kernel: [ 360.804259] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:08 tbox kernel: [ 360.816103] [<c1025e91>] ? need_resched+0x14/0x1e
Mar 29 04:25:08 tbox kernel: [ 360.827845] [<c129f230>] ? schedule+0x3ea/0x3fa
Mar 29 04:25:08 tbox kernel: [ 360.839621] [<c108397d>] ? trace_preempt_off+0xf/0x22
Mar 29 04:25:08 tbox kernel: [ 360.851319] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
Mar 29 04:25:08 tbox kernel: [ 360.863052] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:25:08 tbox kernel: [ 360.874618] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:08 tbox kernel: [ 360.886321] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:25:08 tbox kernel: [ 360.897896] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:25:08 tbox kernel: [ 360.909449] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:08 tbox kernel: [ 360.920917] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:25:08 tbox kernel: [ 360.932305] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:25:08 tbox kernel: [ 360.943525] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:25:08 tbox kernel: [ 360.954839] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
Mar 29 04:25:09 tbox kernel: [ 360.966148] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:25:09 tbox kernel: [ 360.977617] [<c1077004>] synchronize_rcu+0x38/0x3a
Mar 29 04:25:09 tbox kernel: [ 360.989076] [<c1043868>] ? find_ge_pid+0x2f/0x2f
Mar 29 04:25:09 tbox kernel: [ 361.000603] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
Mar 29 04:25:09 tbox kernel: [ 361.012020] [<c1045511>] kthread+0x62/0x67
Mar 29 04:25:09 tbox kernel: [ 361.023329] [<c10454af>] ? kthread_worker_fn+0x111/0x111
Mar 29 04:25:09 tbox kernel: [ 361.034680] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
Mar 29 04:25:09 tbox kernel: [ 361.046110] INFO: task rcu_torture_fak:1752 blocked for more than 120 seconds.
Mar 29 04:25:09 tbox kernel: [ 361.057508] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:25:09 tbox kernel: [ 361.068706] rcu_torture_fak D 00000021 0 1752 2 0x00000000
Mar 29 04:25:09 tbox kernel: [ 361.079547] f59aff08 00000046 1c05091a 00000021 c1467400 2144d81f 00000000 c1467400
Mar 29 04:25:09 tbox kernel: [ 361.090601] 00000000 00000021 f5b2cc60 c108395c c129f22b 00000001 f59afed0 c12a313a
Mar 29 04:25:09 tbox kernel: [ 361.101522] f6406400 f59afed8 c1025e91 f59aff54 c129f230 1a973f63 00000021 c1467400
Mar 29 04:25:09 tbox kernel: [ 361.112488] Call Trace:
Mar 29 04:25:09 tbox kernel: [ 361.123082] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:25:09 tbox kernel: [ 361.133760] [<c129f22b>] ? schedule+0x3e5/0x3fa
Mar 29 04:25:09 tbox kernel: [ 361.144275] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:09 tbox kernel: [ 361.154858] [<c1025e91>] ? need_resched+0x14/0x1e
Mar 29 04:25:09 tbox kernel: [ 361.165231] [<c129f230>] ? schedule+0x3ea/0x3fa
Mar 29 04:25:09 tbox kernel: [ 361.175636] [<c108397d>] ? trace_preempt_off+0xf/0x22
Mar 29 04:25:09 tbox kernel: [ 361.185990] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
Mar 29 04:25:09 tbox kernel: [ 361.196475] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:25:09 tbox kernel: [ 361.206819] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:09 tbox kernel: [ 361.217327] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:25:09 tbox kernel: [ 361.227750] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:25:09 tbox kernel: [ 361.238216] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:09 tbox kernel: [ 361.248644] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:25:09 tbox kernel: [ 361.259051] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:25:09 tbox kernel: [ 361.269341] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:25:09 tbox kernel: [ 361.279737] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
Mar 29 04:25:09 tbox kernel: [ 361.290095] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:25:09 tbox kernel: [ 361.300570] [<c1077004>] synchronize_rcu+0x38/0x3a
Mar 29 04:25:09 tbox kernel: [ 361.310963] [<c1043868>] ? find_ge_pid+0x2f/0x2f
Mar 29 04:25:09 tbox kernel: [ 361.321416] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
Mar 29 04:25:09 tbox kernel: [ 361.331890] [<c1045511>] kthread+0x62/0x67
Mar 29 04:25:09 tbox kernel: [ 361.342365] [<c10454af>] ? kthread_worker_fn+0x111/0x111
Mar 29 04:25:09 tbox kernel: [ 361.352806] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
Mar 29 04:25:09 tbox kernel: [ 361.363332] INFO: task rcu_torture_fak:1753 blocked for more than 120 seconds.
Mar 29 04:25:09 tbox kernel: [ 361.373899] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:25:09 tbox kernel: [ 361.384605] rcu_torture_fak D 00000021 0 1753 2 0x00000000
Mar 29 04:25:09 tbox kernel: [ 361.395327] f591bf08 00000046 3d298aff 00000021 c1467400 002590a2 00000000 c1467400
Mar 29 04:25:09 tbox kernel: [ 361.406361] 00000000 00000021 f5844840 c108395c c129f22b 00000001 f591bed0 c12a313a
Mar 29 04:25:09 tbox kernel: [ 361.417290] f6406400 f591bed8 c1025e91 f591bf54 c129f230 1a974f36 00000021 c1467400
Mar 29 04:25:09 tbox kernel: [ 361.428261] Call Trace:
Mar 29 04:25:09 tbox kernel: [ 361.438867] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:25:09 tbox kernel: [ 361.449542] [<c129f22b>] ? schedule+0x3e5/0x3fa
Mar 29 04:25:09 tbox kernel: [ 361.460061] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:09 tbox kernel: [ 361.470675] [<c1025e91>] ? need_resched+0x14/0x1e
Mar 29 04:25:09 tbox kernel: [ 361.481046] [<c129f230>] ? schedule+0x3ea/0x3fa
Mar 29 04:25:09 tbox kernel: [ 361.491448] [<c108397d>] ? trace_preempt_off+0xf/0x22
Mar 29 04:25:09 tbox kernel: [ 361.501801] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
Mar 29 04:25:09 tbox kernel: [ 361.512251] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:25:09 tbox kernel: [ 361.522588] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:09 tbox kernel: [ 361.533084] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:25:09 tbox kernel: [ 361.543495] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:25:09 tbox kernel: [ 361.553946] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:09 tbox kernel: [ 361.564368] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:25:09 tbox kernel: [ 361.574778] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:25:09 tbox kernel: [ 361.585055] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:25:09 tbox kernel: [ 361.595446] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
Mar 29 04:25:09 tbox kernel: [ 361.605792] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:25:09 tbox kernel: [ 361.616270] [<c1077004>] synchronize_rcu+0x38/0x3a
Mar 29 04:25:09 tbox kernel: [ 361.626652] [<c1043868>] ? find_ge_pid+0x2f/0x2f
Mar 29 04:25:09 tbox kernel: [ 361.637107] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
Mar 29 04:25:09 tbox kernel: [ 361.647577] [<c1045511>] kthread+0x62/0x67
Mar 29 04:25:09 tbox kernel: [ 361.658037] [<c10454af>] ? kthread_worker_fn+0x111/0x111
Mar 29 04:25:09 tbox kernel: [ 361.668459] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
Mar 29 04:25:09 tbox kernel: [ 361.678968] INFO: task modprobe:1759 blocked for more than 120 seconds.
Mar 29 04:25:09 tbox kernel: [ 361.689455] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:25:09 tbox kernel: [ 361.700107] modprobe D 00000036 0 1759 1747 0x00000000
Mar 29 04:25:09 tbox kernel: [ 361.710781] f5995ea4 00000082 10c2f37d 00000036 c1467400 00000c21 00000000 c1467400
Mar 29 04:25:09 tbox kernel: [ 361.721757] 00000000 00000036 f5844420 ffffffff ffffffff c1027d65 10c2d7ee f6406444
Mar 29 04:25:09 tbox kernel: [ 361.732657] 002dab31 f5995e84 3d090000 00000000 f584444c 00001b8f 00000000 f5995e94
Mar 29 04:25:09 tbox kernel: [ 361.743604] Call Trace:
Mar 29 04:25:09 tbox kernel: [ 361.754185] [<c1027d65>] ? finish_task_switch+0x67/0x6d
Mar 29 04:25:09 tbox kernel: [ 361.764863] [<c10219ba>] ? wakeup_preempt_entity+0x36/0x53
Mar 29 04:25:09 tbox kernel: [ 361.775397] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:25:09 tbox kernel: [ 361.785917] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:25:09 tbox kernel: [ 361.796240] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:25:09 tbox kernel: [ 361.806599] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:25:09 tbox kernel: [ 361.816960] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:25:09 tbox kernel: [ 361.827308] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:25:09 tbox kernel: [ 361.837555] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:25:09 tbox kernel: [ 361.847902] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:25:09 tbox kernel: [ 361.858181] [<c1045576>] kthread_stop+0x60/0xaf
Mar 29 04:25:09 tbox kernel: [ 361.868503] [<f8760592>] rcu_torture_cleanup+0x1d5/0x318 [rcutorture]
Mar 29 04:25:09 tbox kernel: [ 361.878857] [<c10592bc>] sys_delete_module+0x198/0x1f5
Mar 29 04:25:09 tbox kernel: [ 361.889253] [<c10c1612>] ? vfs_write+0xa4/0xd7
Mar 29 04:25:09 tbox kernel: [ 361.899525] [<c11aade0>] ? tty_write_lock+0x3d/0x3d
Mar 29 04:25:09 tbox kernel: [ 361.909876] [<c10c17ca>] ? sys_write+0x53/0x5d
Mar 29 04:25:09 tbox kernel: [ 361.920083] [<c12a575f>] sysenter_do_call+0x12/0x28
Mar 29 04:27:09 tbox kernel: [ 481.928074] INFO: task rcu_torture_fak:1750 blocked for more than 120 seconds.
Mar 29 04:27:09 tbox kernel: [ 481.940037] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:27:09 tbox kernel: [ 481.953070] rcu_torture_fak D 00000036 0 1750 2 0x00000000
Mar 29 04:27:10 tbox kernel: [ 481.972056] f5169f08 00000046 10c2d7ee 00000036 c1467400 000036da 00000000 c1467400
Mar 29 04:27:10 tbox kernel: [ 481.984708] 00000000 00000036 f5971ce0 c108395c c129f22b 00000001 f5169ed0 c12a313a
Mar 29 04:27:10 tbox kernel: [ 481.996536] f6406400 f5169ed8 c1025e91 f5169f54 c129f230 1a975683 00000021 c1467400
Mar 29 04:27:10 tbox kernel: [ 482.008580] Call Trace:
Mar 29 04:27:10 tbox kernel: [ 482.020181] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:27:10 tbox kernel: [ 482.031882] [<c129f22b>] ? schedule+0x3e5/0x3fa
Mar 29 04:27:10 tbox kernel: [ 482.043546] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:10 tbox kernel: [ 482.055367] [<c1025e91>] ? need_resched+0x14/0x1e
Mar 29 04:27:10 tbox kernel: [ 482.067076] [<c129f230>] ? schedule+0x3ea/0x3fa
Mar 29 04:27:10 tbox kernel: [ 482.078839] [<c108397d>] ? trace_preempt_off+0xf/0x22
Mar 29 04:27:10 tbox kernel: [ 482.090503] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
Mar 29 04:27:10 tbox kernel: [ 482.102239] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:27:10 tbox kernel: [ 482.113842] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:10 tbox kernel: [ 482.125558] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:27:10 tbox kernel: [ 482.137128] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:27:10 tbox kernel: [ 482.148707] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:10 tbox kernel: [ 482.160239] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:27:10 tbox kernel: [ 482.171766] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:27:10 tbox kernel: [ 482.183180] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:27:10 tbox kernel: [ 482.194676] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
Mar 29 04:27:10 tbox kernel: [ 482.206123] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:27:10 tbox kernel: [ 482.217623] [<c1077004>] synchronize_rcu+0x38/0x3a
Mar 29 04:27:10 tbox kernel: [ 482.228876] [<c1043868>] ? find_ge_pid+0x2f/0x2f
Mar 29 04:27:10 tbox kernel: [ 482.240065] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
Mar 29 04:27:10 tbox kernel: [ 482.251134] [<c1045511>] kthread+0x62/0x67
Mar 29 04:27:10 tbox kernel: [ 482.262103] [<c10454af>] ? kthread_worker_fn+0x111/0x111
Mar 29 04:27:10 tbox kernel: [ 482.272901] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
Mar 29 04:27:10 tbox kernel: [ 482.283635] INFO: task rcu_torture_fak:1751 blocked for more than 120 seconds.
Mar 29 04:27:10 tbox kernel: [ 482.294244] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:27:10 tbox kernel: [ 482.304837] rcu_torture_fak D 00000021 0 1751 2 0x00000000
Mar 29 04:27:10 tbox kernel: [ 482.315409] f5807f08 00000046 1b8afabc 00000021 c1467400 21ab5ecd 00000000 c1467400
Mar 29 04:27:10 tbox kernel: [ 482.326258] 00000000 00000021 f5970840 c108395c c129f22b 00000001 f5807ed0 c12a313a
Mar 29 04:27:10 tbox kernel: [ 482.337071] f6406400 f5807ed8 c1025e91 f5807f54 c129f230 1a975e7e 00000021 c1467400
Mar 29 04:27:10 tbox kernel: [ 482.348016] Call Trace:
Mar 29 04:27:10 tbox kernel: [ 482.358653] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:27:10 tbox kernel: [ 482.369453] [<c129f22b>] ? schedule+0x3e5/0x3fa
Mar 29 04:27:10 tbox kernel: [ 482.380116] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:10 tbox kernel: [ 482.390934] [<c1025e91>] ? need_resched+0x14/0x1e
Mar 29 04:27:10 tbox kernel: [ 482.401650] [<c129f230>] ? schedule+0x3ea/0x3fa
Mar 29 04:27:10 tbox kernel: [ 482.412396] [<c108397d>] ? trace_preempt_off+0xf/0x22
Mar 29 04:27:10 tbox kernel: [ 482.423046] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
Mar 29 04:27:10 tbox kernel: [ 482.433759] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:27:10 tbox kernel: [ 482.444321] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:10 tbox kernel: [ 482.455086] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:27:10 tbox kernel: [ 482.465806] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:27:10 tbox kernel: [ 482.476454] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:10 tbox kernel: [ 482.486994] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:27:10 tbox kernel: [ 482.497590] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:27:10 tbox kernel: [ 482.508130] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:27:10 tbox kernel: [ 482.518661] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
Mar 29 04:27:10 tbox kernel: [ 482.529095] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:27:10 tbox kernel: [ 482.539659] [<c1077004>] synchronize_rcu+0x38/0x3a
Mar 29 04:27:10 tbox kernel: [ 482.550091] [<c1043868>] ? find_ge_pid+0x2f/0x2f
Mar 29 04:27:10 tbox kernel: [ 482.560540] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
Mar 29 04:27:10 tbox kernel: [ 482.571002] [<c1045511>] kthread+0x62/0x67
Mar 29 04:27:10 tbox kernel: [ 482.581466] [<c10454af>] ? kthread_worker_fn+0x111/0x111
Mar 29 04:27:10 tbox kernel: [ 482.591845] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
Mar 29 04:27:10 tbox kernel: [ 482.602246] INFO: task rcu_torture_fak:1752 blocked for more than 120 seconds.
Mar 29 04:27:10 tbox kernel: [ 482.612710] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:27:10 tbox kernel: [ 482.623331] rcu_torture_fak D 00000021 0 1752 2 0x00000000
Mar 29 04:27:10 tbox kernel: [ 482.634036] f59aff08 00000046 1c05091a 00000021 c1467400 2144d81f 00000000 c1467400
Mar 29 04:27:10 tbox kernel: [ 482.645198] 00000000 00000021 f5b2cc60 c108395c c129f22b 00000001 f59afed0 c12a313a
Mar 29 04:27:10 tbox kernel: [ 482.656175] f6406400 f59afed8 c1025e91 f59aff54 c129f230 1a973f63 00000021 c1467400
Mar 29 04:27:10 tbox kernel: [ 482.667124] Call Trace:
Mar 29 04:27:10 tbox kernel: [ 482.677703] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:27:10 tbox kernel: [ 482.688365] [<c129f22b>] ? schedule+0x3e5/0x3fa
Mar 29 04:27:10 tbox kernel: [ 482.698868] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:10 tbox kernel: [ 482.709440] [<c1025e91>] ? need_resched+0x14/0x1e
Mar 29 04:27:10 tbox kernel: [ 482.719793] [<c129f230>] ? schedule+0x3ea/0x3fa
Mar 29 04:27:10 tbox kernel: [ 482.730182] [<c108397d>] ? trace_preempt_off+0xf/0x22
Mar 29 04:27:10 tbox kernel: [ 482.740519] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
Mar 29 04:27:10 tbox kernel: [ 482.750947] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:27:10 tbox kernel: [ 482.761271] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:10 tbox kernel: [ 482.771756] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:27:10 tbox kernel: [ 482.782160] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:27:10 tbox kernel: [ 482.792602] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:10 tbox kernel: [ 482.803006] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:27:10 tbox kernel: [ 482.813393] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:27:10 tbox kernel: [ 482.823666] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:27:10 tbox kernel: [ 482.834058] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
Mar 29 04:27:10 tbox kernel: [ 482.844417] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:27:10 tbox kernel: [ 482.854887] [<c1077004>] synchronize_rcu+0x38/0x3a
Mar 29 04:27:10 tbox kernel: [ 482.865264] [<c1043868>] ? find_ge_pid+0x2f/0x2f
Mar 29 04:27:10 tbox kernel: [ 482.875699] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
Mar 29 04:27:10 tbox kernel: [ 482.886157] [<c1045511>] kthread+0x62/0x67
Mar 29 04:27:10 tbox kernel: [ 482.896608] [<c10454af>] ? kthread_worker_fn+0x111/0x111
Mar 29 04:27:10 tbox kernel: [ 482.907033] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
Mar 29 04:27:10 tbox kernel: [ 482.917579] INFO: task rcu_torture_fak:1753 blocked for more than 120 seconds.
Mar 29 04:27:10 tbox kernel: [ 482.928135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:27:10 tbox kernel: [ 482.938831] rcu_torture_fak D 00000021 0 1753 2 0x00000000
Mar 29 04:27:10 tbox kernel: [ 482.949547] f591bf08 00000046 3d298aff 00000021 c1467400 002590a2 00000000 c1467400
Mar 29 04:27:11 tbox kernel: [ 482.960568] 00000000 00000021 f5844840 c108395c c129f22b 00000001 f591bed0 c12a313a
Mar 29 04:27:11 tbox kernel: [ 482.971488] f6406400 f591bed8 c1025e91 f591bf54 c129f230 1a974f36 00000021 c1467400
Mar 29 04:27:11 tbox kernel: [ 482.982445] Call Trace:
Mar 29 04:27:11 tbox kernel: [ 482.993023] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:27:11 tbox kernel: [ 483.003687] [<c129f22b>] ? schedule+0x3e5/0x3fa
Mar 29 04:27:11 tbox kernel: [ 483.014180] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:11 tbox kernel: [ 483.024748] [<c1025e91>] ? need_resched+0x14/0x1e
Mar 29 04:27:11 tbox kernel: [ 483.035095] [<c129f230>] ? schedule+0x3ea/0x3fa
Mar 29 04:27:11 tbox kernel: [ 483.045485] [<c108397d>] ? trace_preempt_off+0xf/0x22
Mar 29 04:27:11 tbox kernel: [ 483.055817] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
Mar 29 04:27:11 tbox kernel: [ 483.066241] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:27:11 tbox kernel: [ 483.076562] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:11 tbox kernel: [ 483.087036] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:27:11 tbox kernel: [ 483.097426] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:27:11 tbox kernel: [ 483.107856] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:11 tbox kernel: [ 483.118256] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:27:11 tbox kernel: [ 483.128627] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:27:11 tbox kernel: [ 483.138884] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:27:11 tbox kernel: [ 483.149275] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
Mar 29 04:27:11 tbox kernel: [ 483.159611] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:27:11 tbox kernel: [ 483.170072] [<c1077004>] synchronize_rcu+0x38/0x3a
Mar 29 04:27:11 tbox kernel: [ 483.180444] [<c1043868>] ? find_ge_pid+0x2f/0x2f
Mar 29 04:27:11 tbox kernel: [ 483.190875] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
Mar 29 04:27:11 tbox kernel: [ 483.201327] [<c1045511>] kthread+0x62/0x67
Mar 29 04:27:11 tbox kernel: [ 483.211784] [<c10454af>] ? kthread_worker_fn+0x111/0x111
Mar 29 04:27:11 tbox kernel: [ 483.222207] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
Mar 29 04:27:11 tbox kernel: [ 483.232717] INFO: task modprobe:1759 blocked for more than 120 seconds.
Mar 29 04:27:11 tbox kernel: [ 483.243213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:27:11 tbox kernel: [ 483.253882] modprobe D 00000036 0 1759 1747 0x00000000
Mar 29 04:27:11 tbox kernel: [ 483.264562] f5995ea4 00000082 10c2f37d 00000036 c1467400 00000c21 00000000 c1467400
Mar 29 04:27:11 tbox kernel: [ 483.275540] 00000000 00000036 f5844420 ffffffff ffffffff c1027d65 10c2d7ee f6406444
Mar 29 04:27:11 tbox kernel: [ 483.286424] 002dab31 f5995e84 3d090000 00000000 f584444c 00001b8f 00000000 f5995e94
Mar 29 04:27:11 tbox kernel: [ 483.297363] Call Trace:
Mar 29 04:27:11 tbox kernel: [ 483.307928] [<c1027d65>] ? finish_task_switch+0x67/0x6d
Mar 29 04:27:11 tbox kernel: [ 483.318648] [<c10219ba>] ? wakeup_preempt_entity+0x36/0x53
Mar 29 04:27:11 tbox kernel: [ 483.329172] [<c129f514>] schedule_timeout+0x21/0xaa
Mar 29 04:27:11 tbox kernel: [ 483.339689] [<c108395c>] ? trace_preempt_on+0xf/0x21
Mar 29 04:27:11 tbox kernel: [ 483.350012] [<c129ed68>] ? wait_for_common+0x6f/0xca
Mar 29 04:27:11 tbox kernel: [ 483.360371] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
Mar 29 04:27:11 tbox kernel: [ 483.370729] [<c12a3188>] ? sub_preempt_count+0x41/0x43
Mar 29 04:27:11 tbox kernel: [ 483.381108] [<c129ed6f>] wait_for_common+0x76/0xca
Mar 29 04:27:11 tbox kernel: [ 483.391338] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
Mar 29 04:27:11 tbox kernel: [ 483.401670] [<c129ee44>] wait_for_completion+0x12/0x14
Mar 29 04:27:11 tbox kernel: [ 483.411936] [<c1045576>] kthread_stop+0x60/0xaf
Mar 29 04:27:11 tbox kernel: [ 483.422248] [<f8760592>] rcu_torture_cleanup+0x1d5/0x318 [rcutorture]
Mar 29 04:27:11 tbox kernel: [ 483.432594] [<c10592bc>] sys_delete_module+0x198/0x1f5
Mar 29 04:27:11 tbox kernel: [ 483.442984] [<c10c1612>] ? vfs_write+0xa4/0xd7
Mar 29 04:27:11 tbox kernel: [ 483.453255] [<c11aade0>] ? tty_write_lock+0x3d/0x3d
Mar 29 04:27:11 tbox kernel: [ 483.463598] [<c10c17ca>] ? sys_write+0x53/0x5d
Mar 29 04:27:11 tbox kernel: [ 483.473787] [<c12a575f>] sysenter_do_call+0x12/0x28
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-29 2:42 ` Sedat Dilek
@ 2011-03-29 4:17 ` Paul E. McKenney
0 siblings, 0 replies; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-29 4:17 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Tue, Mar 29, 2011 at 04:42:31AM +0200, Sedat Dilek wrote:
> On Tue, Mar 29, 2011 at 2:10 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Mon, Mar 28, 2011 at 06:46:48PM +0200, Sedat Dilek wrote:
> >> On Mon, Mar 28, 2011 at 6:38 PM, Sedat Dilek <sedat.dilek@googlemail.com> wrote:
> >> > On Mon, Mar 28, 2011 at 5:11 PM, Paul E. McKenney
> >> > <paulmck@linux.vnet.ibm.com> wrote:
> >> >> On Mon, Mar 28, 2011 at 06:24:36AM -0700, Paul E. McKenney wrote:
> >> >>> On Mon, Mar 28, 2011 at 02:33:36PM +0200, Sedat Dilek wrote:
> >
> > [ . . . ]
> >
> >> >>> > Ah, before I forget...
> >> >>> >
> >> >>> > I used TREE_RCU (was the default before noticing RCU issue) for
> >> >>> > finding the culprit commit.
> >> >>> > If it is from your POV more helpful to switch to PREEMPT + PREEMPT_RCU
> >> >>> > + RCU_BOOST, please let me *now* know.
> >> >>> > ( Both RCU setups freaks up the system. )
> >> >>>
> >> >>> If TREE_RCU hits problems faster, it is probably best to stay with
> >> >>> TREE_RCU.
> >> >>
> >> >> And of course, one exception to this advice is if TREE_RCU hangs so hard
> >> >> and fast that you don't have time to get any diagnostics. If this is the
> >> >> case, then TREE_PREEMPT_RCU might be more productive.
> >> >>
> >> >
> >> > OK, that would somehow explain why I could not really get some debug
> >> > infos when doing "my stress-test" and checking via:
> >> >
> >> > $ LC_ALL=C tail -f /sys/kernel/debug/rcu/rcudata
> >> >
> >> > Then I remembered I saw a snippet for a RCU torture script mentionned
> >> > in the kernel-docs (see Documentation/RCU/torture.txt).
> >> >
> >> > 189 The following script may be used to torture RCU:
> >> > 190
> >> > 191 #!/bin/sh
> >> > 192
> >> > 193 modprobe rcutorture
> >> > 194 sleep 100
> >> > 195 rmmod rcutorture
> >> > 196 dmesg | grep torture:
> >> >
> >> > So, I recompiled a new TREE_RC-based kernel and build with
> >> > CONFIG_RCU_TORTURE_TEST=m.
> >> >
> >> > Unfortunately, the rmmod (I prefer modprobe -r -v) hangs... the
> >> > messages in the logs look promising.
> >> >
> >> > - Sedat -
> >> >
> >>
> >> Wrong attachment, correct attached.
> >
> > And one stupid problem located thus far. I can make a (tortured) case
> > for it resulting in the symptoms you see, but it does seem unlikely to
> > happen repeatedly, as it would require a burst of CPU just at the wrong
> > time. But who knows?
> >
> > In any case, I am still looking.
> >
> > Thanx, Paul
> >
> > ------------------------------------------------------------------------
> >
> > Fix stupid typo.
> >
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> >
> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> > index 5477764..f311228 100644
> > --- a/kernel/rcutree.c
> > +++ b/kernel/rcutree.c
> > @@ -1618,7 +1618,7 @@ static int rcu_node_kthread(void *arg)
> > rnp->wakemask = 0;
> > raw_spin_unlock_irqrestore(&rnp->lock, flags);
> > rcu_initiate_boost(rnp);
> > - for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask <<= 1) {
> > + for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask >>= 1) {
> > if ((mask & 0x1) == 0)
> > continue;
> > preempt_disable();
> >
>
> I have tested this patch and the previous one you send:
>
> (+) OK rcu-fix/rcu-further-lower-priority-in-rcu_yield.patch
> (+) OK rcu-fix/Fix-stupid-typo.patch
>
> As you suggested I switched to PREEMPT and RCU with rcu-boost:
>
> # egrep 'RCU|PREEMPT|_HZ' /boot/config-2.6.38-next20110328-5-686-iniza
> # RCU Subsystem
> CONFIG_TREE_PREEMPT_RCU=y
> CONFIG_PREEMPT_RCU=y
> CONFIG_RCU_TRACE=y
> CONFIG_RCU_FANOUT=32
> # CONFIG_RCU_FANOUT_EXACT is not set
> CONFIG_TREE_RCU_TRACE=y
> CONFIG_RCU_BOOST=y
> CONFIG_RCU_BOOST_PRIO=1
> CONFIG_RCU_BOOST_DELAY=500
> CONFIG_NO_HZ=y
> # CONFIG_PREEMPT_NONE is not set
> # CONFIG_PREEMPT_VOLUNTARY is not set
> CONFIG_PREEMPT=y
> # CONFIG_HZ_100 is not set
> CONFIG_HZ_250=y
> # CONFIG_HZ_300 is not set
> # CONFIG_HZ_1000 is not set
> CONFIG_HZ=250
> CONFIG_DEBUG_PREEMPT=y
> # CONFIG_SPARSE_RCU_POINTER is not set
> CONFIG_RCU_TORTURE_TEST=m
> CONFIG_RCU_CPU_STALL_TIMEOUT=60
> CONFIG_RCU_CPU_STALL_VERBOSE=y
> CONFIG_PREEMPT_TRACER=y
>
> Unfortunately, the rcu-torture script hangs again on unloading the
> rcu-torture-test module.
> Attached are RCU-related messages in my logs.
> ( I tailed for rcudata changes - no logs. )
OK, still looks like grace periods are being stalled, which would
explain the hang at module-unload time.
On /debug/rcu/rcudata, you need to "cat" it each time. It just
dumps some internal RCU state once, it is not a running log.
So could you please "cat" it out once the problem has occured?
Thanx, Paul
> - Sedat -
> Mar 29 04:21:20 tbox kernel: [ 132.160108] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/1 test_boost_interval=7 test_boost_duration=4
> Mar 29 04:25:08 tbox kernel: [ 360.352072] INFO: task rcu_torture_fak:1750 blocked for more than 120 seconds.
> Mar 29 04:25:08 tbox kernel: [ 360.364505] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:25:08 tbox kernel: [ 360.377380] rcu_torture_fak D 00000036 0 1750 2 0x00000000
> Mar 29 04:25:08 tbox kernel: [ 360.389888] f5169f08 00000046 10c2d7ee 00000036 c1467400 000036da 00000000 c1467400
> Mar 29 04:25:08 tbox kernel: [ 360.405450] 00000000 00000036 f5971ce0 c108395c c129f22b 00000001 f5169ed0 c12a313a
> Mar 29 04:25:08 tbox kernel: [ 360.417251] f6406400 f5169ed8 c1025e91 f5169f54 c129f230 1a975683 00000021 c1467400
> Mar 29 04:25:08 tbox kernel: [ 360.429130] Call Trace:
> Mar 29 04:25:08 tbox kernel: [ 360.440754] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:25:08 tbox kernel: [ 360.452377] [<c129f22b>] ? schedule+0x3e5/0x3fa
> Mar 29 04:25:08 tbox kernel: [ 360.463904] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:08 tbox kernel: [ 360.475636] [<c1025e91>] ? need_resched+0x14/0x1e
> Mar 29 04:25:08 tbox kernel: [ 360.487202] [<c129f230>] ? schedule+0x3ea/0x3fa
> Mar 29 04:25:08 tbox kernel: [ 360.498801] [<c108397d>] ? trace_preempt_off+0xf/0x22
> Mar 29 04:25:08 tbox kernel: [ 360.510305] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
> Mar 29 04:25:08 tbox kernel: [ 360.521903] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:25:08 tbox kernel: [ 360.533342] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:08 tbox kernel: [ 360.544920] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:25:08 tbox kernel: [ 360.556383] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:25:08 tbox kernel: [ 360.567690] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:08 tbox kernel: [ 360.578785] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:25:08 tbox kernel: [ 360.589832] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:25:08 tbox kernel: [ 360.600831] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:25:08 tbox kernel: [ 360.611948] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
> Mar 29 04:25:08 tbox kernel: [ 360.623083] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:25:08 tbox kernel: [ 360.634254] [<c1077004>] synchronize_rcu+0x38/0x3a
> Mar 29 04:25:08 tbox kernel: [ 360.645301] [<c1043868>] ? find_ge_pid+0x2f/0x2f
> Mar 29 04:25:08 tbox kernel: [ 360.656373] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
> Mar 29 04:25:08 tbox kernel: [ 360.667452] [<c1045511>] kthread+0x62/0x67
> Mar 29 04:25:08 tbox kernel: [ 360.678588] [<c10454af>] ? kthread_worker_fn+0x111/0x111
> Mar 29 04:25:08 tbox kernel: [ 360.689635] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
> Mar 29 04:25:08 tbox kernel: [ 360.700677] INFO: task rcu_torture_fak:1751 blocked for more than 120 seconds.
> Mar 29 04:25:08 tbox kernel: [ 360.711751] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:25:08 tbox kernel: [ 360.723000] rcu_torture_fak D 00000021 0 1751 2 0x00000000
> Mar 29 04:25:08 tbox kernel: [ 360.734284] f5807f08 00000046 1b8afabc 00000021 c1467400 21ab5ecd 00000000 c1467400
> Mar 29 04:25:08 tbox kernel: [ 360.745904] 00000000 00000021 f5970840 c108395c c129f22b 00000001 f5807ed0 c12a313a
> Mar 29 04:25:08 tbox kernel: [ 360.757524] f6406400 f5807ed8 c1025e91 f5807f54 c129f230 1a975e7e 00000021 c1467400
> Mar 29 04:25:08 tbox kernel: [ 360.769310] Call Trace:
> Mar 29 04:25:08 tbox kernel: [ 360.780844] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:25:08 tbox kernel: [ 360.792587] [<c129f22b>] ? schedule+0x3e5/0x3fa
> Mar 29 04:25:08 tbox kernel: [ 360.804259] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:08 tbox kernel: [ 360.816103] [<c1025e91>] ? need_resched+0x14/0x1e
> Mar 29 04:25:08 tbox kernel: [ 360.827845] [<c129f230>] ? schedule+0x3ea/0x3fa
> Mar 29 04:25:08 tbox kernel: [ 360.839621] [<c108397d>] ? trace_preempt_off+0xf/0x22
> Mar 29 04:25:08 tbox kernel: [ 360.851319] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
> Mar 29 04:25:08 tbox kernel: [ 360.863052] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:25:08 tbox kernel: [ 360.874618] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:08 tbox kernel: [ 360.886321] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:25:08 tbox kernel: [ 360.897896] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:25:08 tbox kernel: [ 360.909449] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:08 tbox kernel: [ 360.920917] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:25:08 tbox kernel: [ 360.932305] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:25:08 tbox kernel: [ 360.943525] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:25:08 tbox kernel: [ 360.954839] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
> Mar 29 04:25:09 tbox kernel: [ 360.966148] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:25:09 tbox kernel: [ 360.977617] [<c1077004>] synchronize_rcu+0x38/0x3a
> Mar 29 04:25:09 tbox kernel: [ 360.989076] [<c1043868>] ? find_ge_pid+0x2f/0x2f
> Mar 29 04:25:09 tbox kernel: [ 361.000603] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
> Mar 29 04:25:09 tbox kernel: [ 361.012020] [<c1045511>] kthread+0x62/0x67
> Mar 29 04:25:09 tbox kernel: [ 361.023329] [<c10454af>] ? kthread_worker_fn+0x111/0x111
> Mar 29 04:25:09 tbox kernel: [ 361.034680] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
> Mar 29 04:25:09 tbox kernel: [ 361.046110] INFO: task rcu_torture_fak:1752 blocked for more than 120 seconds.
> Mar 29 04:25:09 tbox kernel: [ 361.057508] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:25:09 tbox kernel: [ 361.068706] rcu_torture_fak D 00000021 0 1752 2 0x00000000
> Mar 29 04:25:09 tbox kernel: [ 361.079547] f59aff08 00000046 1c05091a 00000021 c1467400 2144d81f 00000000 c1467400
> Mar 29 04:25:09 tbox kernel: [ 361.090601] 00000000 00000021 f5b2cc60 c108395c c129f22b 00000001 f59afed0 c12a313a
> Mar 29 04:25:09 tbox kernel: [ 361.101522] f6406400 f59afed8 c1025e91 f59aff54 c129f230 1a973f63 00000021 c1467400
> Mar 29 04:25:09 tbox kernel: [ 361.112488] Call Trace:
> Mar 29 04:25:09 tbox kernel: [ 361.123082] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:25:09 tbox kernel: [ 361.133760] [<c129f22b>] ? schedule+0x3e5/0x3fa
> Mar 29 04:25:09 tbox kernel: [ 361.144275] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:09 tbox kernel: [ 361.154858] [<c1025e91>] ? need_resched+0x14/0x1e
> Mar 29 04:25:09 tbox kernel: [ 361.165231] [<c129f230>] ? schedule+0x3ea/0x3fa
> Mar 29 04:25:09 tbox kernel: [ 361.175636] [<c108397d>] ? trace_preempt_off+0xf/0x22
> Mar 29 04:25:09 tbox kernel: [ 361.185990] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
> Mar 29 04:25:09 tbox kernel: [ 361.196475] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:25:09 tbox kernel: [ 361.206819] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:09 tbox kernel: [ 361.217327] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:25:09 tbox kernel: [ 361.227750] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:25:09 tbox kernel: [ 361.238216] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:09 tbox kernel: [ 361.248644] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:25:09 tbox kernel: [ 361.259051] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:25:09 tbox kernel: [ 361.269341] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:25:09 tbox kernel: [ 361.279737] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
> Mar 29 04:25:09 tbox kernel: [ 361.290095] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:25:09 tbox kernel: [ 361.300570] [<c1077004>] synchronize_rcu+0x38/0x3a
> Mar 29 04:25:09 tbox kernel: [ 361.310963] [<c1043868>] ? find_ge_pid+0x2f/0x2f
> Mar 29 04:25:09 tbox kernel: [ 361.321416] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
> Mar 29 04:25:09 tbox kernel: [ 361.331890] [<c1045511>] kthread+0x62/0x67
> Mar 29 04:25:09 tbox kernel: [ 361.342365] [<c10454af>] ? kthread_worker_fn+0x111/0x111
> Mar 29 04:25:09 tbox kernel: [ 361.352806] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
> Mar 29 04:25:09 tbox kernel: [ 361.363332] INFO: task rcu_torture_fak:1753 blocked for more than 120 seconds.
> Mar 29 04:25:09 tbox kernel: [ 361.373899] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:25:09 tbox kernel: [ 361.384605] rcu_torture_fak D 00000021 0 1753 2 0x00000000
> Mar 29 04:25:09 tbox kernel: [ 361.395327] f591bf08 00000046 3d298aff 00000021 c1467400 002590a2 00000000 c1467400
> Mar 29 04:25:09 tbox kernel: [ 361.406361] 00000000 00000021 f5844840 c108395c c129f22b 00000001 f591bed0 c12a313a
> Mar 29 04:25:09 tbox kernel: [ 361.417290] f6406400 f591bed8 c1025e91 f591bf54 c129f230 1a974f36 00000021 c1467400
> Mar 29 04:25:09 tbox kernel: [ 361.428261] Call Trace:
> Mar 29 04:25:09 tbox kernel: [ 361.438867] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:25:09 tbox kernel: [ 361.449542] [<c129f22b>] ? schedule+0x3e5/0x3fa
> Mar 29 04:25:09 tbox kernel: [ 361.460061] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:09 tbox kernel: [ 361.470675] [<c1025e91>] ? need_resched+0x14/0x1e
> Mar 29 04:25:09 tbox kernel: [ 361.481046] [<c129f230>] ? schedule+0x3ea/0x3fa
> Mar 29 04:25:09 tbox kernel: [ 361.491448] [<c108397d>] ? trace_preempt_off+0xf/0x22
> Mar 29 04:25:09 tbox kernel: [ 361.501801] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
> Mar 29 04:25:09 tbox kernel: [ 361.512251] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:25:09 tbox kernel: [ 361.522588] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:09 tbox kernel: [ 361.533084] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:25:09 tbox kernel: [ 361.543495] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:25:09 tbox kernel: [ 361.553946] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:09 tbox kernel: [ 361.564368] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:25:09 tbox kernel: [ 361.574778] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:25:09 tbox kernel: [ 361.585055] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:25:09 tbox kernel: [ 361.595446] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
> Mar 29 04:25:09 tbox kernel: [ 361.605792] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:25:09 tbox kernel: [ 361.616270] [<c1077004>] synchronize_rcu+0x38/0x3a
> Mar 29 04:25:09 tbox kernel: [ 361.626652] [<c1043868>] ? find_ge_pid+0x2f/0x2f
> Mar 29 04:25:09 tbox kernel: [ 361.637107] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
> Mar 29 04:25:09 tbox kernel: [ 361.647577] [<c1045511>] kthread+0x62/0x67
> Mar 29 04:25:09 tbox kernel: [ 361.658037] [<c10454af>] ? kthread_worker_fn+0x111/0x111
> Mar 29 04:25:09 tbox kernel: [ 361.668459] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
> Mar 29 04:25:09 tbox kernel: [ 361.678968] INFO: task modprobe:1759 blocked for more than 120 seconds.
> Mar 29 04:25:09 tbox kernel: [ 361.689455] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:25:09 tbox kernel: [ 361.700107] modprobe D 00000036 0 1759 1747 0x00000000
> Mar 29 04:25:09 tbox kernel: [ 361.710781] f5995ea4 00000082 10c2f37d 00000036 c1467400 00000c21 00000000 c1467400
> Mar 29 04:25:09 tbox kernel: [ 361.721757] 00000000 00000036 f5844420 ffffffff ffffffff c1027d65 10c2d7ee f6406444
> Mar 29 04:25:09 tbox kernel: [ 361.732657] 002dab31 f5995e84 3d090000 00000000 f584444c 00001b8f 00000000 f5995e94
> Mar 29 04:25:09 tbox kernel: [ 361.743604] Call Trace:
> Mar 29 04:25:09 tbox kernel: [ 361.754185] [<c1027d65>] ? finish_task_switch+0x67/0x6d
> Mar 29 04:25:09 tbox kernel: [ 361.764863] [<c10219ba>] ? wakeup_preempt_entity+0x36/0x53
> Mar 29 04:25:09 tbox kernel: [ 361.775397] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:25:09 tbox kernel: [ 361.785917] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:25:09 tbox kernel: [ 361.796240] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:25:09 tbox kernel: [ 361.806599] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:25:09 tbox kernel: [ 361.816960] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:25:09 tbox kernel: [ 361.827308] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:25:09 tbox kernel: [ 361.837555] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:25:09 tbox kernel: [ 361.847902] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:25:09 tbox kernel: [ 361.858181] [<c1045576>] kthread_stop+0x60/0xaf
> Mar 29 04:25:09 tbox kernel: [ 361.868503] [<f8760592>] rcu_torture_cleanup+0x1d5/0x318 [rcutorture]
> Mar 29 04:25:09 tbox kernel: [ 361.878857] [<c10592bc>] sys_delete_module+0x198/0x1f5
> Mar 29 04:25:09 tbox kernel: [ 361.889253] [<c10c1612>] ? vfs_write+0xa4/0xd7
> Mar 29 04:25:09 tbox kernel: [ 361.899525] [<c11aade0>] ? tty_write_lock+0x3d/0x3d
> Mar 29 04:25:09 tbox kernel: [ 361.909876] [<c10c17ca>] ? sys_write+0x53/0x5d
> Mar 29 04:25:09 tbox kernel: [ 361.920083] [<c12a575f>] sysenter_do_call+0x12/0x28
> Mar 29 04:27:09 tbox kernel: [ 481.928074] INFO: task rcu_torture_fak:1750 blocked for more than 120 seconds.
> Mar 29 04:27:09 tbox kernel: [ 481.940037] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:27:09 tbox kernel: [ 481.953070] rcu_torture_fak D 00000036 0 1750 2 0x00000000
> Mar 29 04:27:10 tbox kernel: [ 481.972056] f5169f08 00000046 10c2d7ee 00000036 c1467400 000036da 00000000 c1467400
> Mar 29 04:27:10 tbox kernel: [ 481.984708] 00000000 00000036 f5971ce0 c108395c c129f22b 00000001 f5169ed0 c12a313a
> Mar 29 04:27:10 tbox kernel: [ 481.996536] f6406400 f5169ed8 c1025e91 f5169f54 c129f230 1a975683 00000021 c1467400
> Mar 29 04:27:10 tbox kernel: [ 482.008580] Call Trace:
> Mar 29 04:27:10 tbox kernel: [ 482.020181] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:27:10 tbox kernel: [ 482.031882] [<c129f22b>] ? schedule+0x3e5/0x3fa
> Mar 29 04:27:10 tbox kernel: [ 482.043546] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:10 tbox kernel: [ 482.055367] [<c1025e91>] ? need_resched+0x14/0x1e
> Mar 29 04:27:10 tbox kernel: [ 482.067076] [<c129f230>] ? schedule+0x3ea/0x3fa
> Mar 29 04:27:10 tbox kernel: [ 482.078839] [<c108397d>] ? trace_preempt_off+0xf/0x22
> Mar 29 04:27:10 tbox kernel: [ 482.090503] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
> Mar 29 04:27:10 tbox kernel: [ 482.102239] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:27:10 tbox kernel: [ 482.113842] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:10 tbox kernel: [ 482.125558] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:27:10 tbox kernel: [ 482.137128] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:27:10 tbox kernel: [ 482.148707] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:10 tbox kernel: [ 482.160239] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:27:10 tbox kernel: [ 482.171766] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:27:10 tbox kernel: [ 482.183180] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:27:10 tbox kernel: [ 482.194676] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
> Mar 29 04:27:10 tbox kernel: [ 482.206123] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:27:10 tbox kernel: [ 482.217623] [<c1077004>] synchronize_rcu+0x38/0x3a
> Mar 29 04:27:10 tbox kernel: [ 482.228876] [<c1043868>] ? find_ge_pid+0x2f/0x2f
> Mar 29 04:27:10 tbox kernel: [ 482.240065] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
> Mar 29 04:27:10 tbox kernel: [ 482.251134] [<c1045511>] kthread+0x62/0x67
> Mar 29 04:27:10 tbox kernel: [ 482.262103] [<c10454af>] ? kthread_worker_fn+0x111/0x111
> Mar 29 04:27:10 tbox kernel: [ 482.272901] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
> Mar 29 04:27:10 tbox kernel: [ 482.283635] INFO: task rcu_torture_fak:1751 blocked for more than 120 seconds.
> Mar 29 04:27:10 tbox kernel: [ 482.294244] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:27:10 tbox kernel: [ 482.304837] rcu_torture_fak D 00000021 0 1751 2 0x00000000
> Mar 29 04:27:10 tbox kernel: [ 482.315409] f5807f08 00000046 1b8afabc 00000021 c1467400 21ab5ecd 00000000 c1467400
> Mar 29 04:27:10 tbox kernel: [ 482.326258] 00000000 00000021 f5970840 c108395c c129f22b 00000001 f5807ed0 c12a313a
> Mar 29 04:27:10 tbox kernel: [ 482.337071] f6406400 f5807ed8 c1025e91 f5807f54 c129f230 1a975e7e 00000021 c1467400
> Mar 29 04:27:10 tbox kernel: [ 482.348016] Call Trace:
> Mar 29 04:27:10 tbox kernel: [ 482.358653] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:27:10 tbox kernel: [ 482.369453] [<c129f22b>] ? schedule+0x3e5/0x3fa
> Mar 29 04:27:10 tbox kernel: [ 482.380116] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:10 tbox kernel: [ 482.390934] [<c1025e91>] ? need_resched+0x14/0x1e
> Mar 29 04:27:10 tbox kernel: [ 482.401650] [<c129f230>] ? schedule+0x3ea/0x3fa
> Mar 29 04:27:10 tbox kernel: [ 482.412396] [<c108397d>] ? trace_preempt_off+0xf/0x22
> Mar 29 04:27:10 tbox kernel: [ 482.423046] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
> Mar 29 04:27:10 tbox kernel: [ 482.433759] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:27:10 tbox kernel: [ 482.444321] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:10 tbox kernel: [ 482.455086] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:27:10 tbox kernel: [ 482.465806] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:27:10 tbox kernel: [ 482.476454] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:10 tbox kernel: [ 482.486994] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:27:10 tbox kernel: [ 482.497590] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:27:10 tbox kernel: [ 482.508130] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:27:10 tbox kernel: [ 482.518661] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
> Mar 29 04:27:10 tbox kernel: [ 482.529095] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:27:10 tbox kernel: [ 482.539659] [<c1077004>] synchronize_rcu+0x38/0x3a
> Mar 29 04:27:10 tbox kernel: [ 482.550091] [<c1043868>] ? find_ge_pid+0x2f/0x2f
> Mar 29 04:27:10 tbox kernel: [ 482.560540] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
> Mar 29 04:27:10 tbox kernel: [ 482.571002] [<c1045511>] kthread+0x62/0x67
> Mar 29 04:27:10 tbox kernel: [ 482.581466] [<c10454af>] ? kthread_worker_fn+0x111/0x111
> Mar 29 04:27:10 tbox kernel: [ 482.591845] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
> Mar 29 04:27:10 tbox kernel: [ 482.602246] INFO: task rcu_torture_fak:1752 blocked for more than 120 seconds.
> Mar 29 04:27:10 tbox kernel: [ 482.612710] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:27:10 tbox kernel: [ 482.623331] rcu_torture_fak D 00000021 0 1752 2 0x00000000
> Mar 29 04:27:10 tbox kernel: [ 482.634036] f59aff08 00000046 1c05091a 00000021 c1467400 2144d81f 00000000 c1467400
> Mar 29 04:27:10 tbox kernel: [ 482.645198] 00000000 00000021 f5b2cc60 c108395c c129f22b 00000001 f59afed0 c12a313a
> Mar 29 04:27:10 tbox kernel: [ 482.656175] f6406400 f59afed8 c1025e91 f59aff54 c129f230 1a973f63 00000021 c1467400
> Mar 29 04:27:10 tbox kernel: [ 482.667124] Call Trace:
> Mar 29 04:27:10 tbox kernel: [ 482.677703] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:27:10 tbox kernel: [ 482.688365] [<c129f22b>] ? schedule+0x3e5/0x3fa
> Mar 29 04:27:10 tbox kernel: [ 482.698868] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:10 tbox kernel: [ 482.709440] [<c1025e91>] ? need_resched+0x14/0x1e
> Mar 29 04:27:10 tbox kernel: [ 482.719793] [<c129f230>] ? schedule+0x3ea/0x3fa
> Mar 29 04:27:10 tbox kernel: [ 482.730182] [<c108397d>] ? trace_preempt_off+0xf/0x22
> Mar 29 04:27:10 tbox kernel: [ 482.740519] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
> Mar 29 04:27:10 tbox kernel: [ 482.750947] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:27:10 tbox kernel: [ 482.761271] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:10 tbox kernel: [ 482.771756] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:27:10 tbox kernel: [ 482.782160] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:27:10 tbox kernel: [ 482.792602] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:10 tbox kernel: [ 482.803006] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:27:10 tbox kernel: [ 482.813393] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:27:10 tbox kernel: [ 482.823666] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:27:10 tbox kernel: [ 482.834058] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
> Mar 29 04:27:10 tbox kernel: [ 482.844417] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:27:10 tbox kernel: [ 482.854887] [<c1077004>] synchronize_rcu+0x38/0x3a
> Mar 29 04:27:10 tbox kernel: [ 482.865264] [<c1043868>] ? find_ge_pid+0x2f/0x2f
> Mar 29 04:27:10 tbox kernel: [ 482.875699] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
> Mar 29 04:27:10 tbox kernel: [ 482.886157] [<c1045511>] kthread+0x62/0x67
> Mar 29 04:27:10 tbox kernel: [ 482.896608] [<c10454af>] ? kthread_worker_fn+0x111/0x111
> Mar 29 04:27:10 tbox kernel: [ 482.907033] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
> Mar 29 04:27:10 tbox kernel: [ 482.917579] INFO: task rcu_torture_fak:1753 blocked for more than 120 seconds.
> Mar 29 04:27:10 tbox kernel: [ 482.928135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:27:10 tbox kernel: [ 482.938831] rcu_torture_fak D 00000021 0 1753 2 0x00000000
> Mar 29 04:27:10 tbox kernel: [ 482.949547] f591bf08 00000046 3d298aff 00000021 c1467400 002590a2 00000000 c1467400
> Mar 29 04:27:11 tbox kernel: [ 482.960568] 00000000 00000021 f5844840 c108395c c129f22b 00000001 f591bed0 c12a313a
> Mar 29 04:27:11 tbox kernel: [ 482.971488] f6406400 f591bed8 c1025e91 f591bf54 c129f230 1a974f36 00000021 c1467400
> Mar 29 04:27:11 tbox kernel: [ 482.982445] Call Trace:
> Mar 29 04:27:11 tbox kernel: [ 482.993023] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:27:11 tbox kernel: [ 483.003687] [<c129f22b>] ? schedule+0x3e5/0x3fa
> Mar 29 04:27:11 tbox kernel: [ 483.014180] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:11 tbox kernel: [ 483.024748] [<c1025e91>] ? need_resched+0x14/0x1e
> Mar 29 04:27:11 tbox kernel: [ 483.035095] [<c129f230>] ? schedule+0x3ea/0x3fa
> Mar 29 04:27:11 tbox kernel: [ 483.045485] [<c108397d>] ? trace_preempt_off+0xf/0x22
> Mar 29 04:27:11 tbox kernel: [ 483.055817] [<c103a53b>] ? lock_timer_base.isra.29+0x1e/0x3c
> Mar 29 04:27:11 tbox kernel: [ 483.066241] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:27:11 tbox kernel: [ 483.076562] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:11 tbox kernel: [ 483.087036] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:27:11 tbox kernel: [ 483.097426] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:27:11 tbox kernel: [ 483.107856] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:11 tbox kernel: [ 483.118256] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:27:11 tbox kernel: [ 483.128627] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:27:11 tbox kernel: [ 483.138884] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:27:11 tbox kernel: [ 483.149275] [<f875f93e>] ? rcu_torture_fqs+0xc6/0xc6 [rcutorture]
> Mar 29 04:27:11 tbox kernel: [ 483.159611] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:27:11 tbox kernel: [ 483.170072] [<c1077004>] synchronize_rcu+0x38/0x3a
> Mar 29 04:27:11 tbox kernel: [ 483.180444] [<c1043868>] ? find_ge_pid+0x2f/0x2f
> Mar 29 04:27:11 tbox kernel: [ 483.190875] [<f875f9b1>] rcu_torture_fakewriter+0x73/0xd0 [rcutorture]
> Mar 29 04:27:11 tbox kernel: [ 483.201327] [<c1045511>] kthread+0x62/0x67
> Mar 29 04:27:11 tbox kernel: [ 483.211784] [<c10454af>] ? kthread_worker_fn+0x111/0x111
> Mar 29 04:27:11 tbox kernel: [ 483.222207] [<c12a5cfe>] kernel_thread_helper+0x6/0xd
> Mar 29 04:27:11 tbox kernel: [ 483.232717] INFO: task modprobe:1759 blocked for more than 120 seconds.
> Mar 29 04:27:11 tbox kernel: [ 483.243213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar 29 04:27:11 tbox kernel: [ 483.253882] modprobe D 00000036 0 1759 1747 0x00000000
> Mar 29 04:27:11 tbox kernel: [ 483.264562] f5995ea4 00000082 10c2f37d 00000036 c1467400 00000c21 00000000 c1467400
> Mar 29 04:27:11 tbox kernel: [ 483.275540] 00000000 00000036 f5844420 ffffffff ffffffff c1027d65 10c2d7ee f6406444
> Mar 29 04:27:11 tbox kernel: [ 483.286424] 002dab31 f5995e84 3d090000 00000000 f584444c 00001b8f 00000000 f5995e94
> Mar 29 04:27:11 tbox kernel: [ 483.297363] Call Trace:
> Mar 29 04:27:11 tbox kernel: [ 483.307928] [<c1027d65>] ? finish_task_switch+0x67/0x6d
> Mar 29 04:27:11 tbox kernel: [ 483.318648] [<c10219ba>] ? wakeup_preempt_entity+0x36/0x53
> Mar 29 04:27:11 tbox kernel: [ 483.329172] [<c129f514>] schedule_timeout+0x21/0xaa
> Mar 29 04:27:11 tbox kernel: [ 483.339689] [<c108395c>] ? trace_preempt_on+0xf/0x21
> Mar 29 04:27:11 tbox kernel: [ 483.350012] [<c129ed68>] ? wait_for_common+0x6f/0xca
> Mar 29 04:27:11 tbox kernel: [ 483.360371] [<c12a313a>] ? sub_preempt_count.part.167+0x67/0x74
> Mar 29 04:27:11 tbox kernel: [ 483.370729] [<c12a3188>] ? sub_preempt_count+0x41/0x43
> Mar 29 04:27:11 tbox kernel: [ 483.381108] [<c129ed6f>] wait_for_common+0x76/0xca
> Mar 29 04:27:11 tbox kernel: [ 483.391338] [<c102ca4a>] ? try_to_wake_up+0x181/0x181
> Mar 29 04:27:11 tbox kernel: [ 483.401670] [<c129ee44>] wait_for_completion+0x12/0x14
> Mar 29 04:27:11 tbox kernel: [ 483.411936] [<c1045576>] kthread_stop+0x60/0xaf
> Mar 29 04:27:11 tbox kernel: [ 483.422248] [<f8760592>] rcu_torture_cleanup+0x1d5/0x318 [rcutorture]
> Mar 29 04:27:11 tbox kernel: [ 483.432594] [<c10592bc>] sys_delete_module+0x198/0x1f5
> Mar 29 04:27:11 tbox kernel: [ 483.442984] [<c10c1612>] ? vfs_write+0xa4/0xd7
> Mar 29 04:27:11 tbox kernel: [ 483.453255] [<c11aade0>] ? tty_write_lock+0x3d/0x3d
> Mar 29 04:27:11 tbox kernel: [ 483.463598] [<c10c17ca>] ? sys_write+0x53/0x5d
> Mar 29 04:27:11 tbox kernel: [ 483.473787] [<c12a575f>] sysenter_do_call+0x12/0x28
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-28 13:24 ` Paul E. McKenney
2011-03-28 15:11 ` Paul E. McKenney
@ 2011-03-29 4:39 ` Sedat Dilek
2011-03-29 5:48 ` Paul E. McKenney
1 sibling, 1 reply; 36+ messages in thread
From: Sedat Dilek @ 2011-03-29 4:39 UTC (permalink / raw)
To: paulmck
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
[-- Attachment #1: Type: text/plain, Size: 644 bytes --]
[...]
>> BTW, today's linux-next (next-20110328) is still freaky, I applied the
>> revert-rcu-patches patchset and all is fine.
>
> I reverted back to the commit preceding the one you pointed out last night
> my time, so the upcoming -next should be less freaky.
>
Makes sense.
Looks like the "old" commits are now in boost.2011.03.25b GIT branch [1]?
I have attached the result from rcu-torture for linux-next (20110329)
as I wanted to see what is logged (loaded rcutorture module 3x normal,
1x with verbose=1).
- Sedat -
[1] http://git.us.kernel.org/?p=linux/kernel/git/paulmck/linux-2.6-rcu.git;a=shortlog;h=refs/heads/boost.2011.03.25b
[-- Attachment #2: msg_rcu-torture_next-20110329.txt --]
[-- Type: text/plain, Size: 5719 bytes --]
[ 688.623078] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
[ 788.672108] rcu-torture: rtc: (null) ver: 12030 tfle: 0 rta: 12030 rtaf: 0 rtf: 12025 rtmbe: 0 rtbke: 0 rtbre: 0 rtbae: 0 rtbafe: 0 rtbf: 0 rtb: 0 nt: 20281
[ 788.672118] rcu-torture: Reader Pipe: 30198896 782 0 0 0 0 0 0 0 0 0
[ 788.672123] rcu-torture: Reader Batch: 30196680 2998 0 0 0 0 0 0 0 0 0
[ 788.672128] rcu-torture: Free-Block Circulation: 12029 12027 12025 12025 12025 12025 12025 12025 12025 12025 0
[ 788.672158] rcu-torture:--- End of test: SUCCESS: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
[ 923.777108] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
[ 1023.820120] rcu-torture: rtc: (null) ver: 12206 tfle: 0 rta: 12206 rtaf: 0 rtf: 12203 rtmbe: 0 rtbke: 0 rtbre: 0 rtbae: 0 rtbafe: 0 rtbf: 0 rtb: 0 nt: 20509
[ 1023.820130] rcu-torture: Reader Pipe: 31024643 786 0 0 0 0 0 0 0 0 0
[ 1023.820135] rcu-torture: Reader Batch: 31022307 3122 0 0 0 0 0 0 0 0 0
[ 1023.820140] rcu-torture: Free-Block Circulation: 12205 12203 12203 12203 12203 12203 12203 12203 12203 12203 0
[ 1023.820171] rcu-torture:--- End of test: SUCCESS: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
[ 1292.474530] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
[ 1392.504192] rcu-torture: rtc: (null) ver: 12248 tfle: 0 rta: 12248 rtaf: 0 rtf: 12245 rtmbe: 0 rtbke: 0 rtbre: 0 rtbae: 0 rtbafe: 0 rtbf: 0 rtb: 0 nt: 21087
[ 1392.504201] rcu-torture: Reader Pipe: 31674422 790 0 0 0 0 0 0 0 0 0
[ 1392.504207] rcu-torture: Reader Batch: 31672026 3186 0 0 0 0 0 0 0 0 0
[ 1392.504212] rcu-torture: Free-Block Circulation: 12247 12246 12245 12245 12245 12245 12245 12245 12245 12245 0
[ 1392.504242] rcu-torture:--- End of test: SUCCESS: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
[ 1601.082101] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=1 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
[ 1601.082114] rcu-torture:Creating rcu_torture_writer task
[ 1601.082239] rcu-torture:Creating rcu_torture_fakewriter task
[ 1601.082261] rcu-torture:rcu_torture_writer task started
[ 1601.087257] rcu-torture:Creating rcu_torture_fakewriter task
[ 1601.087300] rcu-torture:rcu_torture_fakewriter task started
[ 1601.090700] rcu-torture:Creating rcu_torture_fakewriter task
[ 1601.090742] rcu-torture:rcu_torture_fakewriter task started
[ 1601.090754] rcu-torture:Creating rcu_torture_fakewriter task
[ 1601.090773] rcu-torture:rcu_torture_fakewriter task started
[ 1601.090784] rcu-torture:Creating rcu_torture_reader task
[ 1601.090802] rcu-torture:rcu_torture_fakewriter task started
[ 1601.090812] rcu-torture:Creating rcu_torture_reader task
[ 1601.090830] rcu-torture:rcu_torture_reader task started
[ 1601.090883] rcu-torture:rcu_torture_reader task started
[ 1601.091303] rcu-torture:rcu_torture_stutter task started
[ 1701.099323] rcu-torture:Stopping rcu_torture_stutter task
[ 1701.099347] rcu-torture:rcu_torture_stutter task stopping
[ 1701.099395] rcu-torture:Stopping rcu_torture_writer task
[ 1701.099409] rcu-torture:rcu_torture_reader task stopping
[ 1701.099422] rcu-torture:rcu_torture_reader task stopping
[ 1701.099437] rcu-torture:rcu_torture_writer task stopping
[ 1701.099456] rcu-torture:Stopping rcu_torture_reader task
[ 1701.099497] rcu-torture:Stopping rcu_torture_reader task
[ 1701.099535] rcu-torture:Stopping rcu_torture_fakewriter task
[ 1701.100726] rcu-torture:rcu_torture_fakewriter task stopping
[ 1701.104291] rcu-torture:rcu_torture_fakewriter task stopping
[ 1701.104315] rcu-torture:Stopping rcu_torture_fakewriter task
[ 1701.104359] rcu-torture:Stopping rcu_torture_fakewriter task
[ 1701.131318] rcu-torture:rcu_torture_fakewriter task stopping
[ 1701.139632] rcu-torture:rcu_torture_fakewriter task stopping
[ 1701.139668] rcu-torture:Stopping rcu_torture_fakewriter task
[ 1701.144211] rcu-torture: rtc: (null) ver: 12179 tfle: 0 rta: 12179 rtaf: 0 rtf: 12177 rtmbe: 0 rtbke: 0 rtbre: 0 rtbae: 0 rtbafe: 0 rtbf: 0 rtb: 0 nt: 20951
[ 1701.144220] rcu-torture: Reader Pipe: 31547701 757 0 0 0 0 0 0 0 0 0
[ 1701.144226] rcu-torture: Reader Batch: 31545363 3095 0 0 0 0 0 0 0 0 0
[ 1701.144231] rcu-torture: Free-Block Circulation: 12178 12177 12177 12177 12177 12177 12177 12177 12177 12177 0
[ 1701.144259] rcu-torture:--- End of test: SUCCESS: nreaders=2 nfakewriters=4 stat_interval=0 verbose=1 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?)
2011-03-29 4:39 ` Sedat Dilek
@ 2011-03-29 5:48 ` Paul E. McKenney
0 siblings, 0 replies; 36+ messages in thread
From: Paul E. McKenney @ 2011-03-29 5:48 UTC (permalink / raw)
To: sedat.dilek
Cc: Josh Triplett, linux-next, LKML, Stephen Rothwell, Randy Dunlap,
Theodore Ts'o, Jens Axboe, Tejun Heo, Al Viro, Nick Piggin
On Tue, Mar 29, 2011 at 06:39:45AM +0200, Sedat Dilek wrote:
> [...]
> >> BTW, today's linux-next (next-20110328) is still freaky, I applied the
> >> revert-rcu-patches patchset and all is fine.
> >
> > I reverted back to the commit preceding the one you pointed out last night
> > my time, so the upcoming -next should be less freaky.
> >
>
> Makes sense.
> Looks like the "old" commits are now in boost.2011.03.25b GIT branch [1]?
Yep. I keep them always, digital packrat that I am.
I will be posting updates.
> I have attached the result from rcu-torture for linux-next (20110329)
> as I wanted to see what is logged (loaded rcutorture module 3x normal,
> 1x with verbose=1).
Yep, that is expected output. Now I just need to work out what else
is messed up.
Thanx, Paul
> - Sedat -
>
> [1] http://git.us.kernel.org/?p=linux/kernel/git/paulmck/linux-2.6-rcu.git;a=shortlog;h=refs/heads/boost.2011.03.25b
> [ 688.623078] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
> [ 788.672108] rcu-torture: rtc: (null) ver: 12030 tfle: 0 rta: 12030 rtaf: 0 rtf: 12025 rtmbe: 0 rtbke: 0 rtbre: 0 rtbae: 0 rtbafe: 0 rtbf: 0 rtb: 0 nt: 20281
> [ 788.672118] rcu-torture: Reader Pipe: 30198896 782 0 0 0 0 0 0 0 0 0
> [ 788.672123] rcu-torture: Reader Batch: 30196680 2998 0 0 0 0 0 0 0 0 0
> [ 788.672128] rcu-torture: Free-Block Circulation: 12029 12027 12025 12025 12025 12025 12025 12025 12025 12025 0
> [ 788.672158] rcu-torture:--- End of test: SUCCESS: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
> [ 923.777108] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
> [ 1023.820120] rcu-torture: rtc: (null) ver: 12206 tfle: 0 rta: 12206 rtaf: 0 rtf: 12203 rtmbe: 0 rtbke: 0 rtbre: 0 rtbae: 0 rtbafe: 0 rtbf: 0 rtb: 0 nt: 20509
> [ 1023.820130] rcu-torture: Reader Pipe: 31024643 786 0 0 0 0 0 0 0 0 0
> [ 1023.820135] rcu-torture: Reader Batch: 31022307 3122 0 0 0 0 0 0 0 0 0
> [ 1023.820140] rcu-torture: Free-Block Circulation: 12205 12203 12203 12203 12203 12203 12203 12203 12203 12203 0
> [ 1023.820171] rcu-torture:--- End of test: SUCCESS: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
> [ 1292.474530] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
> [ 1392.504192] rcu-torture: rtc: (null) ver: 12248 tfle: 0 rta: 12248 rtaf: 0 rtf: 12245 rtmbe: 0 rtbke: 0 rtbre: 0 rtbae: 0 rtbafe: 0 rtbf: 0 rtb: 0 nt: 21087
> [ 1392.504201] rcu-torture: Reader Pipe: 31674422 790 0 0 0 0 0 0 0 0 0
> [ 1392.504207] rcu-torture: Reader Batch: 31672026 3186 0 0 0 0 0 0 0 0 0
> [ 1392.504212] rcu-torture: Free-Block Circulation: 12247 12246 12245 12245 12245 12245 12245 12245 12245 12245 0
> [ 1392.504242] rcu-torture:--- End of test: SUCCESS: nreaders=2 nfakewriters=4 stat_interval=0 verbose=0 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
> [ 1601.082101] rcu-torture:--- Start of test: nreaders=2 nfakewriters=4 stat_interval=0 verbose=1 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
> [ 1601.082114] rcu-torture:Creating rcu_torture_writer task
> [ 1601.082239] rcu-torture:Creating rcu_torture_fakewriter task
> [ 1601.082261] rcu-torture:rcu_torture_writer task started
> [ 1601.087257] rcu-torture:Creating rcu_torture_fakewriter task
> [ 1601.087300] rcu-torture:rcu_torture_fakewriter task started
> [ 1601.090700] rcu-torture:Creating rcu_torture_fakewriter task
> [ 1601.090742] rcu-torture:rcu_torture_fakewriter task started
> [ 1601.090754] rcu-torture:Creating rcu_torture_fakewriter task
> [ 1601.090773] rcu-torture:rcu_torture_fakewriter task started
> [ 1601.090784] rcu-torture:Creating rcu_torture_reader task
> [ 1601.090802] rcu-torture:rcu_torture_fakewriter task started
> [ 1601.090812] rcu-torture:Creating rcu_torture_reader task
> [ 1601.090830] rcu-torture:rcu_torture_reader task started
> [ 1601.090883] rcu-torture:rcu_torture_reader task started
> [ 1601.091303] rcu-torture:rcu_torture_stutter task started
> [ 1701.099323] rcu-torture:Stopping rcu_torture_stutter task
> [ 1701.099347] rcu-torture:rcu_torture_stutter task stopping
> [ 1701.099395] rcu-torture:Stopping rcu_torture_writer task
> [ 1701.099409] rcu-torture:rcu_torture_reader task stopping
> [ 1701.099422] rcu-torture:rcu_torture_reader task stopping
> [ 1701.099437] rcu-torture:rcu_torture_writer task stopping
> [ 1701.099456] rcu-torture:Stopping rcu_torture_reader task
> [ 1701.099497] rcu-torture:Stopping rcu_torture_reader task
> [ 1701.099535] rcu-torture:Stopping rcu_torture_fakewriter task
> [ 1701.100726] rcu-torture:rcu_torture_fakewriter task stopping
> [ 1701.104291] rcu-torture:rcu_torture_fakewriter task stopping
> [ 1701.104315] rcu-torture:Stopping rcu_torture_fakewriter task
> [ 1701.104359] rcu-torture:Stopping rcu_torture_fakewriter task
> [ 1701.131318] rcu-torture:rcu_torture_fakewriter task stopping
> [ 1701.139632] rcu-torture:rcu_torture_fakewriter task stopping
> [ 1701.139668] rcu-torture:Stopping rcu_torture_fakewriter task
> [ 1701.144211] rcu-torture: rtc: (null) ver: 12179 tfle: 0 rta: 12179 rtaf: 0 rtf: 12177 rtmbe: 0 rtbke: 0 rtbre: 0 rtbae: 0 rtbafe: 0 rtbf: 0 rtb: 0 nt: 20951
> [ 1701.144220] rcu-torture: Reader Pipe: 31547701 757 0 0 0 0 0 0 0 0 0
> [ 1701.144226] rcu-torture: Reader Batch: 31545363 3095 0 0 0 0 0 0 0 0 0
> [ 1701.144231] rcu-torture: Free-Block Circulation: 12178 12177 12177 12177 12177 12177 12177 12177 12177 12177 0
> [ 1701.144259] rcu-torture:--- End of test: SUCCESS: nreaders=2 nfakewriters=4 stat_interval=0 verbose=1 test_no_idle_hz=0 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
^ permalink raw reply [flat|nested] 36+ messages in thread
end of thread, other threads:[~2011-03-29 5:48 UTC | newest]
Thread overview: 36+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-03-25 10:16 linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4 related?) Sedat Dilek
2011-03-25 13:05 ` Sedat Dilek
2011-03-25 15:55 ` Josh Triplett
2011-03-25 15:55 ` Josh Triplett
2011-03-25 16:25 ` Sedat Dilek
2011-03-25 16:42 ` Paul E. McKenney
2011-03-25 16:42 ` Paul E. McKenney
2011-03-25 16:51 ` Sedat Dilek
2011-03-25 17:40 ` Sedat Dilek
2011-03-25 17:48 ` Paul E. McKenney
2011-03-25 19:42 ` Sedat Dilek
2011-03-26 3:42 ` Paul E. McKenney
2011-03-26 8:11 ` Sedat Dilek
2011-03-26 15:53 ` Paul E. McKenney
2011-03-26 12:34 ` Sedat Dilek
2011-03-26 16:02 ` Paul E. McKenney
2011-03-26 22:15 ` Sedat Dilek
2011-03-27 0:09 ` Paul E. McKenney
2011-03-27 1:30 ` Sedat Dilek
2011-03-27 3:25 ` Paul E. McKenney
2011-03-27 5:07 ` Paul E. McKenney
2011-03-27 12:26 ` Sedat Dilek
2011-03-27 21:32 ` Paul E. McKenney
2011-03-27 21:48 ` Sedat Dilek
2011-03-28 4:08 ` Paul E. McKenney
2011-03-28 12:33 ` Sedat Dilek
2011-03-28 13:24 ` Paul E. McKenney
2011-03-28 15:11 ` Paul E. McKenney
2011-03-28 16:38 ` Sedat Dilek
2011-03-28 16:46 ` Sedat Dilek
2011-03-29 0:10 ` Paul E. McKenney
2011-03-29 2:42 ` Sedat Dilek
2011-03-29 4:17 ` Paul E. McKenney
2011-03-29 4:39 ` Sedat Dilek
2011-03-29 5:48 ` Paul E. McKenney
2011-03-25 17:44 ` Paul E. McKenney
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.