All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ross Green <rgkernel@gmail.com>
To: Paul McKenney <paulmck@linux.vnet.ibm.com>
Cc: "Peter Zijlstra" <peterz@infradead.org>,
	linux-kernel@vger.kernel.org, mingo@kernel.org,
	jiangshanlai@gmail.com, dipankar@in.ibm.com,
	akpm@linux-foundation.org,
	"Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>,
	josh@joshtriplett.org, tglx@linutronix.de, rostedt@goodmis.org,
	dhowells@redhat.com, "Eric Dumazet" <edumazet@google.com>,
	dvhart@linux.intel.com,
	"Frédéric Weisbecker" <fweisbec@gmail.com>,
	oleg@redhat.com, "pranith kumar" <bobby.prani@gmail.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17
Date: Thu, 18 Feb 2016 22:51:29 +1100	[thread overview]
Message-ID: <CANfgCY0sy7Fib6yihxFrj+iZx7Y8JkBKMcw2Fjc7JX_58MuFiQ@mail.gmail.com> (raw)
In-Reply-To: <20160217231945.GA21140@linux.vnet.ibm.com>

[-- Attachment #1: Type: text/plain, Size: 3355 bytes --]

On Thu, Feb 18, 2016 at 10:19 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Wed, Feb 17, 2016 at 12:28:29PM -0800, Paul E. McKenney wrote:
>> On Wed, Feb 17, 2016 at 08:45:54PM +0100, Peter Zijlstra wrote:
>> > On Wed, Feb 17, 2016 at 11:28:17AM -0800, Paul E. McKenney wrote:
>> > > On Tue, Feb 16, 2016 at 09:45:49PM -0800, Paul E. McKenney wrote:
>> > > > On Tue, Feb 09, 2016 at 09:11:55PM +1100, Ross Green wrote:
>> > > > > Continued testing with the latest linux-4.5-rc3 release.
>> > > > >
>> > > > > Please find attached a copy of traces from dmesg:
>> > > > >
>> > > > > There is a lot more debug and trace data so hopefully this will shed
>> > > > > some light on what might be happening here.
>> > > > >
>> > > > > My testing remains run a series of simple benchmarks, let that run to
>> > > > > completion and then leave the system idle away with just a few daemons
>> > > > > running.
>> > > > >
>> > > > > the self detected stalls in this instance turned up after a days run time.
>> > > > > There were  NO heavy artificial computational loads on the machine.
>> > > >
>> > > > It does indeed look quiet on that dmesg for a good long time.
>> > > >
>> > > > The following insanely crude not-for-mainline hack -might- be producing
>> > > > good results in my testing.  It will take some time before I can claim
>> > > > statistically different results.  But please feel free to give it a go
>> > > > in the meantime.  (Thanks to Al Viro for pointing me in this direction.)
>> >
>> > Your case was special in that is was hotplug triggering it, right?
>>
>> Yes, it has thus far only shown up with CPU hotplug enabled.
>>
>> > I was auditing the hotplug paths involved when I fell ill two weeks ago,
>> > and have not really made any progress on that because of that :/
>>
>> I have always said that being sick is bad for one's health, but I didn't
>> realize that it could be bad for the kernel's health as well.  ;-)
>>
>> > I'll go have another look, I had a vague feeling for a race back then,
>> > lets see if I can still remember how..
>>
>> I believe that I can -finally- get an ftrace_dump() to happen within
>> 10-20 milliseconds of the problem, which just might be soon enough
>> after the problem to gather some useful information.  I am currently
>> testing this theory with "ftrace trace_event=sched_waking,sched_wakeup"
>> boot arguments on a two-hour run.
>
> And apparently another way to greatly reduce the probability of this
> bug occurring is to enable ftrace.  :-/
>
> Will try longer runs.
>
>                                                         Thanx, Paul
>
>> If this works out, what would be a useful set of trace events for me
>> to capture?
>>
>>                                                       Thanx, Paul
>

Well managed to catch this one on linux-4.5-rc4.

Took over 3 days and 7 hours to hit.

Same test as before, boot, run a series of simple benchmarks and then
let the machine just idle away.

As before, the reported stall, AND everything keeps on running as if
nothing had happened.

I notice in the task dump for both the cpus, the swapper is running on
both cpus.

does that make any sense?
There is around 3% of memory actually used.

Anyway, please find attached a copy of the dmesg output.

Hope this helps a few people fill in the missing pieces here.

Regards,

Ross Green

[-- Attachment #2: dmesg-4.5-rc4 --]
[-- Type: application/octet-stream, Size: 30353 bytes --]

$ uptime
 22:14:35 up 3 days,  7:45,  load average: 0.02, 0.06, 0.11
$ dmesg
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.5.0-rc4-panda (rfg@red.dwarf) (gcc version 5.3.0 (Buildroot 2016.02-git-00183-g1682aee) ) #1 SMP PREEMPT Mon Feb 15 12:35:52 AEDT 2016
[    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine model: TI OMAP4 PandaBoard-ES
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] OMAP4: Map 0xbfe00000 to fe600000 for dram barrier
[    0.000000] On node 0 totalpages: 261632
[    0.000000] free_area_init_node: node 0, pgdat c09662c0, node_mem_map ef7f9000
[    0.000000]   Normal zone: 1536 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 196608 pages, LIFO batch:31
[    0.000000]   HighMem zone: 65024 pages, LIFO batch:15
[    0.000000] OMAP4460 ES1.1
[    0.000000] PERCPU: Embedded 12 pages/cpu @ef7ab000 s17984 r8192 d22976 u49152
[    0.000000] pcpu-alloc: s17984 r8192 d22976 u49152 alloc=12*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 260096
[    0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Memory: 1027232K/1046528K available (6879K kernel code, 252K rwdata, 2156K rodata, 304K init, 297K bss, 19296K reserved, 0K cma-reserved, 260096K highmem)
[    0.000000] Virtual kernel memory layout:
                   vector  : 0xffff0000 - 0xffff1000   (   4 kB)
                   fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
                   vmalloc : 0xf0800000 - 0xff800000   ( 240 MB)
                   lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
                   pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
                   modules : 0xbf000000 - 0xbfe00000   (  14 MB)
                     .text : 0xc0008000 - 0xc08db0d4   (9037 kB)
                     .init : 0xc08dc000 - 0xc0928000   ( 304 kB)
                     .data : 0xc0928000 - 0xc09672c0   ( 253 kB)
                      .bss : 0xc096a000 - 0xc09b47b8   ( 298 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 32.
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[    0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] OMAP L2C310: ROM does not support power control setting
[    0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[    0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[    0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[    0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[    0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[    0.000000] OMAP clockevent source: timer1 at 32768 Hz
[    0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[    0.000000] OMAP clocksource: 32k_counter at 32768 Hz
[    0.000854] Console: colour dummy device 80x30
[    0.000885] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[    0.089172] pid_max: default: 32768 minimum: 301
[    0.089233] Security Framework initialized
[    0.089294] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.089294] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.089904] CPU: Testing write buffer coherency: ok
[    0.090148] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.090209] Setting up static identity map for 0x80008280 - 0x800082d8
[    0.209167] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.209228] Brought up 2 CPUs
[    0.209259] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[    0.209259] CPU: All CPU(s) started in SVC mode.
[    0.210052] devtmpfs: initialized
[    0.227111] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.227600] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.227752] xor: measuring software checksum speed
[    0.319152]    arm4regs  :  1006.000 MB/sec
[    0.419128]    8regs     :   803.600 MB/sec
[    0.519134]    32regs    :   760.800 MB/sec
[    0.519165] xor: using function: arm4regs (1006.000 MB/sec)
[    0.519409] pinctrl core: initialized pinctrl subsystem
[    0.521301] NET: Registered protocol family 16
[    0.522216] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.523925] omap_hwmod: l3_main_3 using broken dt data from ocp
[    0.525451] omap_hwmod: l3_main_2 using broken dt data from ocp
[    0.630523] OMAP GPIO hardware version 0.1
[    0.635803] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[    0.635803] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[    0.636352] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[    0.636718] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[    0.637084] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[    0.648925] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[    0.649261] platform 480a5000.des: Cannot lookup hwmod 'des'
[    0.655273] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[    0.655303] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.655944] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[    0.656158] OMAP DMA hardware revision 0.0
[    0.879425] raid6: int32x1  gen()   103 MB/s
[    1.049407] raid6: int32x1  xor()    83 MB/s
[    1.219207] raid6: int32x2  gen()   134 MB/s
[    1.389343] raid6: int32x2  xor()    99 MB/s
[    1.559448] raid6: int32x4  gen()   145 MB/s
[    1.729278] raid6: int32x4  xor()    97 MB/s
[    1.899230] raid6: int32x8  gen()   160 MB/s
[    2.069366] raid6: int32x8  xor()    94 MB/s
[    2.069396] raid6: using algorithm int32x8 gen() 160 MB/s
[    2.069396] raid6: .... xor() 94 MB/s, rmw enabled
[    2.069396] raid6: using intx1 recovery algorithm
[    2.075439] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[    2.149291] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[    2.153533] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[    2.153656] omap-iommu 55082000.mmu: 55082000.mmu registered
[    2.155761] SCSI subsystem initialized
[    2.156036] usbcore: registered new interface driver usbfs
[    2.156188] usbcore: registered new interface driver hub
[    2.156311] usbcore: registered new device driver usb
[    2.157287] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[    2.157318] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[    2.157379] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[    2.157409] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[    2.157684] Linux video capture interface: v2.00
[    2.158416] Advanced Linux Sound Architecture Driver Initialized.
[    2.159881] clocksource: Switched to clocksource 32k_counter
[    2.174499] NET: Registered protocol family 2
[    2.175109] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[    2.175201] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[    2.175323] TCP: Hash tables configured (established 8192 bind 8192)
[    2.175415] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    2.175476] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    2.175659] NET: Registered protocol family 1
[    2.175994] RPC: Registered named UNIX socket transport module.
[    2.176025] RPC: Registered udp transport module.
[    2.176025] RPC: Registered tcp transport module.
[    2.176025] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    2.178314] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[    2.181396] futex hash table entries: 512 (order: 3, 32768 bytes)
[    2.204132] NFS: Registering the id_resolver key type
[    2.204162] Key type id_resolver registered
[    2.204162] Key type id_legacy registered
[    2.204193] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    2.204223] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    2.205780] ntfs: driver 2.1.32 [Flags: R/O].
[    2.206176] fuse init (API version 7.24)
[    2.209625] jitterentropy: Initialization failed with host not compliant with requirements: 2
[    2.209716] bounce: pool size: 64 pages
[    2.209716] io scheduler noop registered
[    2.209747] io scheduler deadline registered
[    2.209808] io scheduler cfq registered (default)
[    2.210723] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[    2.210937] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[    2.214263] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.216156] omap_uart 4806a000.serial: no wakeirq for uart0
[    2.216278] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[    2.216857] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[    2.217407] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[    3.114410] console [ttyO2] enabled
[    3.118560] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[    3.129302] [drm] Initialized drm 1.1.0 20060810
[    3.135803] 58000000.dss supply vdda_video not found, using dummy regulator
[    3.143188] OMAP DSS rev 4.0
[    3.146362] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[    3.154693] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[    3.163330] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[    3.215209] loop: module loaded
[    3.221374] usbcore: registered new interface driver smsc95xx
[    3.227447] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.234344] ehci-omap: OMAP-EHCI Host Controller driver
[    3.259948] ehci-omap 4a064c00.ehci: EHCI Host Controller
[    3.265624] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[    3.273712] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[    3.299926] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[    3.306304] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    3.313415] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.321014] usb usb1: Product: EHCI Host Controller
[    3.326110] usb usb1: Manufacturer: Linux 4.5.0-rc4-panda ehci_hcd
[    3.332580] usb usb1: SerialNumber: 4a064c00.ehci
[    3.338165] hub 1-0:1.0: USB hub found
[    3.342163] hub 1-0:1.0: 3 ports detected
[    3.347045] usbcore: registered new interface driver cdc_wdm
[    3.353149] usbcore: registered new interface driver usb-storage
[    3.359832] i2c /dev entries driver
[    3.363647] usbcore: registered new interface driver uvcvideo
[    3.369659] USB Video Class driver (1.1.1)
[    3.375152] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[    3.382720] device-mapper: uevent: version 1.0.3
[    3.387878] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[    3.460937] hidraw: raw HID events driver (C) Jiri Kosina
[    3.466796] usbcore: registered new interface driver usbhid
[    3.472656] usbhid: USB HID core driver
[    3.477142] emif 4c000000.emif: emif_probe: device configured with addr = f08a8000 and IRQ247
[    3.486450] emif 4d000000.emif: emif_probe: device configured with addr = f08aa000 and IRQ248
[    3.497375] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[    3.505340] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[    3.512817] ip_tables: (C) 2000-2006 Netfilter Core Team
[    3.518493] Initializing XFRM netlink socket
[    3.523040] NET: Registered protocol family 17
[    3.527984] Key type dns_resolver registered
[    3.532653] twl: not initialized
[    3.536071] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.544677] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.553192] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.561676] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.570159] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.578643] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.587127] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[    3.595642] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[    3.603210] omap2_set_init_voltage: unable to set vdd_mpu
[    3.608947] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[    3.616333] omap2_set_init_voltage: unable to set vdd_core
[    3.622161] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[    3.629425] omap2_set_init_voltage: unable to set vdd_iva
[    3.636047] Power Management for TI OMAP4+ devices.
[    3.641235] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[    3.648437] ThumbEE CPU extension supported.
[    3.653076] Registering SWP/SWPB emulation handler
[    3.653076] SmartReflex Class3 initialized
[    3.662567] usb 1-1: new high-speed USB device number 2 using ehci-omap
[    3.684265] Btrfs loaded
[    3.689392] dmm 4e000000.dmm: initialized all PAT entries
[    3.710876] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[    3.730407] twl 0-0048: PIH (irq 340) nested IRQs
[    3.736816] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[    3.745361] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[    3.767181] twl6040 0-004b: clk32k is not handled
[    3.775054] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[    3.781982] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[    3.789031] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[    3.795806] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[    3.801757] omap_hsmmc 4809c000.mmc: omap_device: omap_device_enable() called from invalid state 1
[    3.827178] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[    3.834411] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.842651] hub 1-1:1.0: USB hub found
[    3.846832] hub 1-1:1.0: 5 ports detected
[    3.862304] omap_hsmmc 480d5000.mmc: omap_device: omap_device_enable() called from invalid state 1
[    3.994659] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[    4.010894] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[    4.022277] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2016-02-15 03:29:23 UTC (1455506963)
[    4.032470] sr_init: No PMIC hook to init smartreflex
[    4.038116] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[    4.046844] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[    4.055389] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[    4.065307] mmc1: new SDIO card at address 0001
[    4.072692] vwl1271: disabling
[    4.076934] mmc0: host does not support reading read-only switch, assuming write-enable
[    4.078979] VDAC: disabling
[    4.079437] VANA: disabling
[    4.079864] VUSB: disabling
[    4.080230] ALSA device list:
[    4.080230]   #0: PandaBoardES
[    4.101013] Waiting for root device /dev/mmcblk0p2...
[    4.106475] mmc0: new high speed SDHC card at address e624
[    4.112945] mmcblk0: mmc0:e624 SL32G 28.8 GiB
[    4.147369]  mmcblk0: p1 p2 p3 p4
[    4.152496] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[    4.220825] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[    4.230010] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[    4.260223] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[    4.270416] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[    4.286865] devtmpfs: mounted
[    4.290008] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[    4.297332] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.297332] Freeing unused kernel memory: 304K (c08dc000 - c0928000)
[    4.314666] smsc95xx v1.0.4
[    4.409088] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, 96:f9:8a:33:0d:a7
[    4.640808] EXT4-fs (mmcblk0p2): warning: mounting unchecked fs, running e2fsck is recommended
[    4.694946] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[    4.840026] random: dd urandom read with 29 bits of entropy available
[   10.340759] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[   20.158081] Bluetooth: Core ver 2.21
[   20.161895] NET: Registered protocol family 31
[   20.167266] Bluetooth: HCI device and connection manager initialized
[   20.174987] Bluetooth: HCI socket layer initialized
[   20.180236] Bluetooth: L2CAP socket layer initialized
[   20.185699] Bluetooth: SCO socket layer initialized
[   27.729614] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[   27.740997] NFSD: starting 90-second grace period (net c09607c0)
[   45.050445] random: nonblocking pool is initialized
[   54.592010] wlcore: loaded
[   68.823089] wlcore: firmware booted (Rev 6.3.10.0.133)
[   72.377471] wlan0: authenticate with 90:f6:52:81:ee:70
[   72.392028] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[   72.399688] wlan0: send auth to 90:f6:52:81:ee:70 (try 2/3)
[   72.406982] wlan0: send auth to 90:f6:52:81:ee:70 (try 3/3)
[   72.414276] wlan0: authentication with 90:f6:52:81:ee:70 timed out
[   75.921600] wlan0: authenticate with 90:f6:52:81:ee:70
[   75.950622] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[   75.958770] wlan0: authenticated
[   75.976959] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[   75.987701] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=1)
[   76.002685] wlan0: associated
[   76.066345] wlcore: Association completed.
[  148.034057] NFSD: Unable to end grace period: -110
[  215.277313] Adding 2097148k swap on /dev/mmcblk0p3.  Priority:-1 extents:1 across:2097148k SS
[280121.367279] INFO: rcu_preempt self-detected stall on CPU
[280121.367279] INFO: rcu_preempt self-detected stall on CPU
[280121.367309]         0-...: (1 GPs behind) idle=917/1/0 softirq=1840575/1840576 fqs=0
[280121.367309]          (t=18827 jiffies g=867073 c=867072 q=2)
[280121.367309] rcu_preempt kthread starved for 18827 jiffies! g867073 c867072 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0
[280121.367340] rcu_preempt     R running      0     7      2 0x00000000
[280121.367340] Backtrace:
[280121.367370] [<c06b9d94>] (__schedule) from [<c06ba2f0>] (schedule+0x54/0xc4)
[280121.367401]  r10:c092a518 r9:00000001 r8:01aaaf55 r7:c092a100 r6:ef097ed4 r5:ef7b73c0
[280121.367401]  r4:ef096000
[280121.367401] [<c06ba29c>] (schedule) from [<c06bd0f0>] (schedule_timeout+0x118/0x1ac)
[280121.367431]  r5:ef7b73c0 r4:ef7b73c0
[280121.367431] [<c06bcfd8>] (schedule_timeout) from [<c00860ec>] (rcu_gp_kthread+0x64c/0x99c)
[280121.367431]  r8:c092a100 r7:c093a210 r6:c093a21c r5:c093a080 r4:00000000
[280121.367492] [<c0085aa0>] (rcu_gp_kthread) from [<c0050534>] (kthread+0xe4/0xfc)
[280121.367492]  r7:c0085aa0
[280121.367523] [<c0050450>] (kthread) from [<c000fb98>] (ret_from_fork+0x14/0x3c)
[280121.367523]  r7:00000000 r6:00000000 r5:c0050450 r4:ef0407c0
[280121.367523] Task dump for CPU 0:
[280121.367553] swapper/0       R running      0     0      0 0x00000000
[280121.367553] Backtrace:
[280121.367584] [<c0013f80>] (dump_backtrace) from [<c001417c>] (show_stack+0x18/0x1c)
[280121.367584]  r7:c092a518 r6:800c0193 r5:00000000 r4:c092d180
[280121.367614] [<c0014164>] (show_stack) from [<c005ac68>] (sched_show_task+0xbc/0x110)
[280121.367614] [<c005abac>] (sched_show_task) from [<c005d0f8>] (dump_cpu_task+0x40/0x48)
[280121.367645]  r5:c093a080 r4:00000000
[280121.367645] [<c005d0b8>] (dump_cpu_task) from [<c00828c4>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[280121.367645]  r5:c093a080 r4:00000001
[280121.367675] [<c0082828>] (rcu_dump_cpu_stacks) from [<c0086d28>] (rcu_check_callbacks+0x500/0x8e4)
[280121.367706]  r9:c093a080 r8:c092a628 r7:2ee89000 r6:c0926340 r5:c093a080 r4:ef7af340
[280121.367706] [<c0086828>] (rcu_check_callbacks) from [<c0089f58>] (update_process_times+0x40/0x6c)
[280121.367736]  r10:c009b20c r9:00000001 r8:ef7abc4c r7:0000fec4 r6:a1351daf r5:00000000
[280121.367736]  r4:c092d180
[280121.367767] [<c0089f18>] (update_process_times) from [<c009b208>] (tick_sched_handle+0x50/0x54)
[280121.367767]  r5:c0929f10 r4:ef7abe40
[280121.367767] [<c009b1b8>] (tick_sched_handle) from [<c009b25c>] (tick_sched_timer+0x50/0x94)
[280121.367797] [<c009b20c>] (tick_sched_timer) from [<c008a99c>] (__hrtimer_run_queues+0x110/0x1a4)
[280121.367797]  r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[280121.367828] [<c008a88c>] (__hrtimer_run_queues) from [<c008b030>] (hrtimer_interrupt+0xac/0x1fc)
[280121.367828]  r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8 r6:ffffffff r5:00000003
[280121.367858]  r4:ef7abc00
[280121.367858] [<c008af84>] (hrtimer_interrupt) from [<c001720c>] (twd_handler+0x38/0x48)
[280121.367889]  r10:c092a4ec r9:fa241100 r8:00000011 r7:ef0287c0 r6:c092a7a8 r5:ef005cc0
[280121.367889]  r4:00000001
[280121.367919] [<c00171d4>] (twd_handler) from [<c007d02c>] (handle_percpu_devid_irq+0x74/0x8c)
[280121.367919]  r5:ef005cc0 r4:ef7b1780
[280121.367950] [<c007cfb8>] (handle_percpu_devid_irq) from [<c0078ab4>] (generic_handle_irq+0x2c/0x3c)
[280121.367950]  r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c092341c
[280121.367980] [<c0078a88>] (generic_handle_irq) from [<c0078dac>] (__handle_domain_irq+0x64/0xbc)
[280121.367980] [<c0078d48>] (__handle_domain_irq) from [<c00094d4>] (gic_handle_irq+0x50/0x90)
[280121.368011]  r9:fa241100 r8:fa240100 r7:c0929f10 r6:fa24010c r5:c092a7a8 r4:c0947f48
[280121.368011] [<c0009484>] (gic_handle_irq) from [<c0014cd4>] (__irq_svc+0x54/0x90)
[280121.368011] Exception stack(0xc0929f10 to 0xc0929f58)
[280121.368041] 9f00:                                     00000000 ef7ab390 fe600000 00000000
[280121.368041] 9f20: c0928000 c092a49c c092236c c0929f80 c0967065 c0967065 c092a4ec c0929f6c
[280121.368072] 9f40: c0929f4c c0929f60 c0028a30 c0010ca0 600c0113 ffffffff
[280121.368072]  r9:c0967065 r8:c0967065 r7:c0929f44 r6:ffffffff r5:600c0113 r4:c0010ca0
[280121.368103] [<c0010c78>] (arch_cpu_idle) from [<c006f720>] (default_idle_call+0x28/0x34)
[280121.368103] [<c006f6f8>] (default_idle_call) from [<c006f844>] (cpu_startup_entry+0x118/0x188)
[280121.368133] [<c006f72c>] (cpu_startup_entry) from [<c06b96c8>] (rest_init+0x90/0x94)
[280121.368133]  r7:ffffffff r4:00000002
[280121.368133] [<c06b9638>] (rest_init) from [<c08dcd0c>] (start_kernel+0x390/0x39c)
[280121.368133]  r5:c096a000 r4:00000000
[280121.368194] [<c08dc97c>] (start_kernel) from [<8000807c>] (0x8000807c)
[280121.368194] Task dump for CPU 1:
[280121.368225] swapper/1       R running      0     0      1 0x00000002
[280121.368225] Backtrace:
[280121.368225] Backtrace aborted due to bad frame pointer <ef09ff9c>

[280121.833038]         1-...: (2 ticks this GP) idle=ded/2/0 softirq=3193168/3193168 fqs=0
[280121.833038]          (t=18874 jiffies g=867073 c=867072 q=5)
[280121.845672] rcu_preempt kthread starved for 18875 jiffies! g867073 c867072 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0
[280121.845672] rcu_preempt     R running      0     7      2 0x00000000
[280121.864105] Backtrace:
[280121.864105] [<c06b9d94>] (__schedule) from [<c06ba2f0>] (schedule+0x54/0xc4)
[280121.874359]  r10:c092a518 r9:00000001 r8:01aaaf55 r7:c092a100 r6:ef097ed4 r5:ef7b73c0
[280121.874359]  r4:ef096000
[280121.883331] [<c06ba29c>] (schedule) from [<c06bd0f0>] (schedule_timeout+0x118/0x1ac)
[280121.887329]  r5:ef7b73c0 r4:ef7b73c0
[280121.887329] [<c06bcfd8>] (schedule_timeout) from [<c00860ec>] (rcu_gp_kthread+0x64c/0x99c)
[280121.897735]  r8:c092a100 r7:c093a210 r6:c093a21c r5:c093a080 r4:00000000
[280121.907257] [<c0085aa0>] (rcu_gp_kthread) from [<c0050534>] (kthread+0xe4/0xfc)
[280121.921600]  r7:c0085aa0
[280121.921600] [<c0050450>] (kthread) from [<c000fb98>] (ret_from_fork+0x14/0x3c)
[280121.921600]  r7:00000000 r6:00000000 r5:c0050450 r4:ef0407c0
[280121.932891] Task dump for CPU 1:
[280121.941711] swapper/1       R running      0     0      1 0x00000002
[280121.947265] Backtrace:
[280121.947265] [<c0013f80>] (dump_backtrace) from [<c001417c>] (show_stack+0x18/0x1c)
[280121.947265]  r7:c092a518 r6:80080193 r5:00000001 r4:ef077480
[280121.965454] [<c0014164>] (show_stack) from [<c005ac68>] (sched_show_task+0xbc/0x110)
[280121.969024] [<c005abac>] (sched_show_task) from [<c005d0f8>] (dump_cpu_task+0x40/0x48)
[280121.982208]  r5:c093a080 r4:00000001
[280121.982208] [<c005d0b8>] (dump_cpu_task) from [<c00828c4>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[280121.990051]  r5:c093a080 r4:00000002
[280121.990051] [<c0082828>] (rcu_dump_cpu_stacks) from [<c0086d28>] (rcu_check_callbacks+0x500/0x8e4)
[280122.008392]  r9:c093a080 r8:c092a628 r7:2ee95000 r6:c0926340 r5:c093a080 r4:ef7bb340
[280122.010040] [<c0086828>] (rcu_check_callbacks) from [<c0089f58>] (update_process_times+0x40/0x6c)
[280122.017761]  r10:c009b20c r9:00000001 r8:ef7b7c4c r7:0000fec4 r6:a1b4b4e7 r5:00000000
[280122.027770]  r4:ef077480
[280122.037261] [<c0089f18>] (update_process_times) from [<c009b208>] (tick_sched_handle+0x50/0x54)
[280122.040313]  r5:ef09fdf0 r4:ef7b7e40
[280122.050567] [<c009b1b8>] (tick_sched_handle) from [<c009b25c>] (tick_sched_timer+0x50/0x94)
[280122.050567] [<c009b20c>] (tick_sched_timer) from [<c008a99c>] (__hrtimer_run_queues+0x110/0x1a4)
[280122.063934]  r7:00000000 r6:ef7b7c40 r5:ef7b7e40 r4:ef7b7c00
[280122.070587] [<c008a88c>] (__hrtimer_run_queues) from [<c008b030>] (hrtimer_interrupt+0xac/0x1fc)
[280122.077270]  r10:ef7b7c78 r9:ef7b7c98 r8:ef7b7c14 r7:ef7b7cb8 r6:ffffffff r5:00000003
[280122.077270]  r4:ef7b7c00
[280122.077270] [<c008af84>] (hrtimer_interrupt) from [<c001720c>] (twd_handler+0x38/0x48)
[280122.097229]  r10:c05fdb80 r9:fa241100 r8:00000011 r7:ef0287c0 r6:c092a7a8 r5:ef005cc0
[280122.107330]  r4:00000001
[280122.112884] [<c00171d4>] (twd_handler) from [<c007d02c>] (handle_percpu_devid_irq+0x74/0x8c)
[280122.118530]  r5:ef005cc0 r4:ef7bd780
[280122.118530] [<c007cfb8>] (handle_percpu_devid_irq) from [<c0078ab4>] (generic_handle_irq+0x2c/0x3c)
[280122.137786]  r9:fa241100 r8:ef008000 r7:00000001 r6:ef09ff58 r5:00000000 r4:c092341c
[280122.140899] [<c0078a88>] (generic_handle_irq) from [<c0078dac>] (__handle_domain_irq+0x64/0xbc)
[280122.153594] [<c0078d48>] (__handle_domain_irq) from [<c00094d4>] (gic_handle_irq+0x50/0x90)
[280122.163665]  r9:fa241100 r8:fa240100 r7:ef09fdf0 r6:fa24010c r5:c092a7a8 r4:c0947f48
[280122.167266] [<c0009484>] (gic_handle_irq) from [<c0014cd4>] (__irq_svc+0x54/0x90)
[280122.180664] Exception stack(0xef09fdf0 to 0xef09fe38)
[280122.180664] fde0:                                     ef7b73c0 00000005 ef09fe5c 000053b3
[280122.194854] fe00: ef3ff230 ef7b73c0 00000000 00000200 ef7b73e0 ef3ff200 c05fdb80 ef09fe4c
[280122.199645] fe20: ef09fe50 ef09fe40 c0089290 c06bdf78 60080113 ffffffff
[280122.210693]  r9:ef3ff200 r8:ef7b73e0 r7:ef09fe24 r6:ffffffff r5:60080113 r4:c06bdf78
[280122.216888] [<c06bdf50>] (_raw_spin_unlock_irq) from [<c0089290>] (run_timer_softirq+0x18c/0x214)
[280122.227264] [<c0089104>] (run_timer_softirq) from [<c00387ac>] (__do_softirq+0x124/0x238)
[280122.227264]  r10:40000001 r9:c092a080 r8:00000001 r7:00000101 r6:ef09e000 r5:c092a084
[280122.245666]  r4:000000a0
[280122.247253] [<c0038688>] (__do_softirq) from [<c0038bb0>] (irq_exit+0xb8/0x120)
[280122.247253]  r10:c092a4ec r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000
[280122.264678]  r4:c092341c
[280122.264678] [<c0038af8>] (irq_exit) from [<c0078db0>] (__handle_domain_irq+0x68/0xbc)
[280122.268859]  r5:00000000 r4:c092341c
[280122.277282] [<c0078d48>] (__handle_domain_irq) from [<c00094d4>] (gic_handle_irq+0x50/0x90)
[280122.287261]  r9:fa241100 r8:fa240100 r7:ef09ff58 r6:fa24010c r5:c092a7a8 r4:c0947f48
[280122.296966] [<c0009484>] (gic_handle_irq) from [<c0014cd4>] (__irq_svc+0x54/0x90)
[280122.304962] Exception stack(0xef09ff58 to 0xef09ffa0)
[280122.307250] ff40:                                                       00000000 ef7b7390
[280122.319152] ff60: fe600000 00000000 ef09e000 c092a49c c092236c ef09ffc8 c0967065 c0967065
[280122.327301] ff80: c092a4ec ef09ffb4 ef09ff94 ef09ffa8 c0028a30 c0010ca0 60080113 ffffffff
[280122.327911]  r9:c0967065 r8:c0967065 r7:ef09ff8c r6:ffffffff r5:60080113 r4:c0010ca0
[280122.344970] [<c0010c78>] (arch_cpu_idle) from [<c006f720>] (default_idle_call+0x28/0x34)
[280122.347259] [<c006f6f8>] (default_idle_call) from [<c006f844>] (cpu_startup_entry+0x118/0x188)
[280122.353637] [<c006f72c>] (cpu_startup_entry) from [<c001627c>] (secondary_start_kernel+0x178/0x184)
[280122.372497]  r7:c096a288 r4:c093d140
[280122.376068] [<c0016104>] (secondary_start_kernel) from [<800095ac>] (0x800095ac)
[280122.384307]  r5:00000051 r4:af08406a
$


  reply	other threads:[~2016-02-18 11:51 UTC|newest]

Thread overview: 79+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-02-09 10:11 rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17 Ross Green
2016-02-17  5:45 ` Paul E. McKenney
2016-02-17 19:28   ` Paul E. McKenney
2016-02-17 19:45     ` Peter Zijlstra
2016-02-17 20:28       ` Paul E. McKenney
2016-02-17 23:19         ` Paul E. McKenney
2016-02-18 11:51           ` Ross Green [this message]
2016-02-18 23:14             ` Mathieu Desnoyers
2016-02-19  3:56               ` Ross Green
2016-02-19  4:13                 ` John Stultz
2016-02-19 17:33                   ` Paul E. McKenney
2016-02-20  4:34                     ` Ross Green
2016-02-20  6:32                       ` Paul E. McKenney
2016-02-21  5:04                         ` Ross Green
2016-02-21 18:15                           ` Ross Green
2016-02-23 20:34                             ` Mathieu Desnoyers
2016-02-23 20:55                               ` Paul E. McKenney
2016-02-23 21:28                                 ` Ross Green
2016-02-25  5:13                                   ` Ross Green
2016-02-26  0:56                                     ` Paul E. McKenney
2016-02-26  1:35                                       ` Paul E. McKenney
2016-03-04  5:30                                         ` Ross Green
2016-03-04 15:18                                           ` Paul E. McKenney
2016-03-18 21:00                                       ` Josh Triplett
2016-03-18 23:56                                         ` Paul E. McKenney
2016-03-21 16:22                                           ` Jacob Pan
2016-03-21 17:26                                             ` Paul E. McKenney
2016-03-22 16:35                                               ` Chatre, Reinette
2016-03-22 17:40                                                 ` Paul E. McKenney
2016-03-22 21:04                                                   ` Chatre, Reinette
2016-03-22 21:19                                                     ` Paul E. McKenney
2016-03-23 17:15                                                       ` Chatre, Reinette
2016-03-23 18:20                                                         ` Paul E. McKenney
2016-03-23 18:25                                                           ` Chatre, Reinette
2016-03-23 19:50                                                             ` Paul E. McKenney
2016-03-25 21:24                                                           ` Chatre, Reinette
2016-03-25 21:46                                                             ` Paul E. McKenney
2016-03-26 12:29                                                               ` Mathieu Desnoyers
2016-03-26 15:28                                                                 ` Paul E. McKenney
2016-03-26 18:49                                                                   ` Paul E. McKenney
2016-03-26 22:22                                                                     ` Mathieu Desnoyers
2016-03-27  1:34                                                                       ` Paul E. McKenney
2016-03-27 13:48                                                                         ` Mathieu Desnoyers
2016-03-27 15:40                                                                           ` Paul E. McKenney
2016-03-27 20:00                                                                             ` Paul E. McKenney
2016-03-27 20:45                                                                             ` Peter Zijlstra
2016-03-27 21:06                                                                               ` Paul E. McKenney
2016-03-28  6:25                                                                                 ` Peter Zijlstra
2016-03-28 13:08                                                                                   ` Paul E. McKenney
2016-03-29  0:25                                                                                     ` Paul E. McKenney
2016-03-29  0:28                                                                                       ` Paul E. McKenney
2016-03-29 13:49                                                                                         ` Paul E. McKenney
2016-03-30 14:55                                                                                           ` Paul E. McKenney
2016-03-31 15:42                                                                                             ` Paul E. McKenney
2016-04-03  8:18                                                                                               ` Paul E. McKenney
2016-05-06  6:25                                                                                                 ` Ross Green
2016-05-07 15:25                                                                                                   ` Paul E. McKenney
2016-05-10  2:36                                                                                                     ` Ross Green
2016-06-30 17:52                                                                                                     ` Paul E. McKenney
2016-03-28  1:44                                                                               ` Mathieu Desnoyers
2016-03-28  2:23                                                                                 ` Mathieu Desnoyers
2016-03-28  6:13                                                                                   ` Peter Zijlstra
2016-03-28 13:50                                                                                     ` Paul E. McKenney
2016-03-28 14:15                                                                                     ` Mathieu Desnoyers
2016-03-27 20:53                                                                             ` Peter Zijlstra
2016-03-27 21:07                                                                               ` Paul E. McKenney
2016-03-27 20:54                                             ` Peter Zijlstra
2016-03-27 21:09                                               ` Paul E. McKenney
2016-03-28  6:28                                                 ` Peter Zijlstra
2016-03-28 13:29                                                   ` Paul E. McKenney
2016-03-28 15:07                                                     ` Mathieu Desnoyers
2016-03-28 15:56                                                       ` Paul E. McKenney
2016-03-28 16:12                                                         ` Mathieu Desnoyers
2016-03-28 16:29                                                           ` Paul E. McKenney
2016-03-30 12:58                                                     ` Boqun Feng
2016-03-30 13:30                                                       ` Paul E. McKenney
2016-03-30 14:15                                                         ` Boqun Feng
2016-02-19  4:22               ` Paul E. McKenney
2016-02-19  5:59                 ` Ross Green

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CANfgCY0sy7Fib6yihxFrj+iZx7Y8JkBKMcw2Fjc7JX_58MuFiQ@mail.gmail.com \
    --to=rgkernel@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=bobby.prani@gmail.com \
    --cc=dhowells@redhat.com \
    --cc=dipankar@in.ibm.com \
    --cc=dvhart@linux.intel.com \
    --cc=edumazet@google.com \
    --cc=fweisbec@gmail.com \
    --cc=jiangshanlai@gmail.com \
    --cc=josh@joshtriplett.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=mingo@kernel.org \
    --cc=oleg@redhat.com \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.