From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752453Ab2GTPBo (ORCPT ); Fri, 20 Jul 2012 11:01:44 -0400 Received: from e28smtp04.in.ibm.com ([122.248.162.4]:35724 "EHLO e28smtp04.in.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751297Ab2GTPBn (ORCPT ); Fri, 20 Jul 2012 11:01:43 -0400 Message-ID: <5009728E.7080902@linux.vnet.ibm.com> Date: Fri, 20 Jul 2012 20:30:30 +0530 From: "Srivatsa S. Bhat" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120605 Thunderbird/13.0 MIME-Version: 1.0 To: paulmck@linux.vnet.ibm.com CC: Thomas Gleixner , LKML , Ingo Molnar , Peter Zijlstra , Rusty Russell , Namhyung Kim Subject: Re: [Patch 0/7] Per cpu thread hotplug infrastructure - V3 References: <20120716103749.122800930@linutronix.de> <20120716152224.GF2403@linux.vnet.ibm.com> <5006F434.9010707@linux.vnet.ibm.com> <20120718235402.GP2435@linux.vnet.ibm.com> <50095A6A.4080404@linux.vnet.ibm.com> <20120720143559.GB2721@linux.vnet.ibm.com> In-Reply-To: <20120720143559.GB2721@linux.vnet.ibm.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit x-cbid: 12072015-5564-0000-0000-000003AEDE0E Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07/20/2012 08:05 PM, Paul E. McKenney wrote: > On Fri, Jul 20, 2012 at 06:47:30PM +0530, Srivatsa S. Bhat wrote: >> On 07/19/2012 05:24 AM, Paul E. McKenney wrote: >>> On Wed, Jul 18, 2012 at 11:06:52PM +0530, Srivatsa S. Bhat wrote: >>>> On 07/16/2012 08:52 PM, Paul E. McKenney wrote: >>>>> On Mon, Jul 16, 2012 at 10:42:34AM -0000, Thomas Gleixner wrote: >>>>>> The following series implements the infrastructure for parking and >>>>>> unparking kernel threads to avoid the full teardown and fork on cpu >>>>>> hotplug operations along with management infrastructure for hotplug >>>>>> and users. >>>>>> >>>>>> Changes vs. V2: >>>>>> >>>>>> Use callbacks for all functionality. Thanks to Rusty for pointing >>>>>> that out. It makes the use sites nice and simple and keeps all the >>>>>> code which would be duplicated otherwise on the core. >>>>> >>>>> Hello, Thomas, >>>>> >>>>> What version should I apply this patchset to? I tried v3.5-rc7, but >>>>> got lots of warnings (one shown below) and the watchdog patch did not >>>>> apply. >>>>> >>>> >>>> Hi Paul, >>>> >>>> This patchset applies cleanly on Thomas' smp/hotplug branch in the -tip >>>> tree. >>> >>> Thank you, Srivatsa, works much better. Still get "scheduling while >>> atomic", looking into that. >>> >> >> Got a chance to run this patchset now.. Even I am getting "scheduling while >> atomic" messages like shown below.. Hmmm... > > Here is what little I have done so far (lots of completing demands on time > this week, but I should have a goodly block of time to focus on this today): > > 1. The failure is from the softirq modifications. Reverting that > commit gets rid of the failures. > > 2. As one would expect, CONFIG_PREEMPT=n kernels do not have the > problem, which of course indicates a preempt_disable() imbalance. > Right.. > 3. I was unable to spot the problem by inspection, but this is not > too surprising given the high level of distraction this week. > > 4. Instrumentation shows that preempt_count() grows slowly with > time, but with the upper bits zero. This confirms the > preempt_disable imbalance. > > 5. I am currently placing WARN_ONCE() calls in the code to track > this down. When I do find it, I fully expect to feel very stupid > about my efforts on #3 above. ;-) > Hehe :-) I'll also see if I can dig out the problem.. Regards, Srivatsa S. Bhat >> ----------------- >> >> BUG: sleeping function called from invalid context at kernel/softirq.c:748 >> in_atomic(): 1, irqs_disabled(): 0, pid: 3, name: ksoftirqd/0 >> INFO: lockdep is turned off. >> Pid: 3, comm: ksoftirqd/0 Tainted: G W 3.5.0-rc1-threadhotplugv3-debug #2 >> Call Trace: >> [] __might_sleep+0x18c/0x250 >> [] run_ksoftirqd+0x4f/0x70 >> [] smpboot_thread_fn+0x157/0x1f0 >> [] ? smpboot_create_threads+0x80/0x80 >> [] kthread+0xee/0x100 >> [] kernel_thread_helper+0x4/0x10 >> [] ? retint_restore_args+0x13/0x13 >> [] ? __init_kthread_worker+0x70/0x70 >> [] ? gs_change+0x13/0x13 >> BUG: scheduling while atomic: ksoftirqd/0/3/0x1000000b >> INFO: lockdep is turned off. >> Modules linked in: ext3 jbd mbcache sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt mptsas mptscsih mptbase scsi_transport_sas [last unloaded: scsi_wait_scan] >> Pid: 3, comm: ksoftirqd/0 Tainted: G W 3.5.0-rc1-threadhotplugv3-debug #2 >> Call Trace: >> [] __schedule_bug+0x6c/0x90 >> [] __schedule+0x7c8/0x880 >> [] __cond_resched+0x2a/0x40 >> [] _cond_resched+0x30/0x40 >> SELinux: 2048 avtab hash slots, 250818 rules. >> [] run_ksoftirqd+0x54/0x70 >> [] smpboot_thread_fn+0x157/0x1f0 >> [] ? smpboot_create_threads+0x80/0x80 >> [] kthread+0xee/0x100 >> [] kernel_thread_helper+0x4/0x10 >> [] ? retint_restore_args+0x13/0x13 >> [] ? __init_kthread_worker+0x70/0x70 >> [] ? gs_change+0x13/0x13 >> >> [...] >> >> BUG: sleeping function called from invalid context at kernel/softirq.c:748 >> in_atomic(): 1, irqs_disabled(): 0, pid: 37, name: ksoftirqd/8 >> INFO: lockdep is turned off. >> Pid: 37, comm: ksoftirqd/8 Tainted: G W 3.5.0-rc1-threadhotplugv3-debug #2 >> Call Trace: >> [] __might_sleep+0x18c/0x250 >> [] run_ksoftirqd+0x4f/0x70 >> [] smpboot_thread_fn+0x157/0x1f0 >> Welcome to [] ? smpboot_create_threads+0x80/0x80 >> [] kthread+0xee/0x100 >> Red Hat [] kernel_thread_helper+0x4/0x10 >> [] ? retint_restore_args+0x13/0x13 >> [] ? __init_kthread_worker+0x70/0x70 >> [] ? gs_change+0x13/0x13 >> BUG: scheduling while atomic: ksoftirqd/8/37/0x10000004 >> INFO: lockdep is turned off. >> Modules linked in: ext3 jbd mbcache sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt mptsas mptscsih mptbase scsi_transport_sas [last unloaded: scsi_wait_scan] >> Pid: 37, comm: ksoftirqd/8 Tainted: G W 3.5.0-rc1-threadhotplugv3-debug #2 >> Call Trace: >> Enterprise Linu [] __schedule_bug+0x6c/0x90 >> x Server >> [] __schedule+0x7c8/0x880 >> [] __cond_resched+0x2a/0x40 >> [] _cond_resched+0x30/0x40 >> [] run_ksoftirqd+0x54/0x70 >> [] smpboot_thread_fn+0x157/0x1f0 >> [] ? smpboot_create_threads+0x80/0x80 >> [] kthread+0xee/0x100 >> [] kernel_thread_helper+0x4/0x10 >> [] ? retint_restore_args+0x13/0x13 >> [] ? __init_kthread_worker+0x70/0x70 >> [] ? gs_change+0x13/0x13 >> >> [...] >> >> BUG: scheduling while atomic: watchdog/15/64/0x0000009b >> INFO: lockdep is turned off. >> Modules linked in: ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 cpufreq_powersave sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log dm_round_robin scsi_dh_rdac dm_multipath vhost_net macvtap macvlan tun uinput cdc_ether usbnet mii serio_raw kvm_intel kvm microcode i2c_i801 i2c_core lpc_ich mfd_core shpchp ioatdma dca i7core_edac edac_core bnx2 dm_mod sg ext3 jbd mbcache sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt mptsas mptscsih mptbase scsi_transport_sas [last unloaded: scsi_wait_scan] >> Pid: 64, comm: watchdog/15 Tainted: G W 3.5.0-rc1-threadhotplugv3-debug #2 >> Call Trace: >> [] __schedule_bug+0x6c/0x90 >> [] __schedule+0x7c8/0x880 >> [] schedule+0x29/0x70 >> [] schedule_preempt_disabled+0x27/0x40 >> [] smpboot_thread_fn+0xf3/0x1f0 >> [] ? smpboot_create_threads+0x80/0x80 >> [] kthread+0xee/0x100 >> [] kernel_thread_helper+0x4/0x10 >> [] ? retint_restore_args+0x13/0x13 >> [] ? __init_kthread_worker+0x70/0x70 >> [] ? gs_change+0x13/0x13 >>