From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932297AbbFQVEY (ORCPT ); Wed, 17 Jun 2015 17:04:24 -0400 Received: from mail.skyhub.de ([78.46.96.112]:54283 "EHLO mail.skyhub.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753711AbbFQVEP (ORCPT ); Wed, 17 Jun 2015 17:04:15 -0400 Date: Wed, 17 Jun 2015 23:04:11 +0200 From: Borislav Petkov To: "H. Peter Anvin" Cc: Gu Zheng , Ingo Molnar , Andy Lutomirski , Andy Lutomirski , Thomas Gleixner , "linux-kernel@vger.kernel.org" , X86 ML Subject: Re: [PATCH V1] x86, espfix: postpone the initialization of espfix stack for AP Message-ID: <20150617210411.GD16999@pd.tnic> References: <20150515065417.GB29973@gmail.com> <55559FDA.3010205@zytor.com> <555A40C9.6010605@kernel.org> <555B5105.4040808@zytor.com> <555F0139.9040404@cn.fujitsu.com> <55666D4A.5040006@cn.fujitsu.com> <556D7687.70402@cn.fujitsu.com> <55701E54.6090802@cn.fujitsu.com> <55812149.1040804@zytor.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <55812149.1040804@zytor.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jun 17, 2015 at 12:27:05AM -0700, H. Peter Anvin wrote: > On 06/04/2015 02:45 AM, Gu Zheng wrote: > > The following lockdep warning occurrs when running with latest kernel: > > [ 3.178000] ------------[ cut here ]------------ > > [ 3.183000] WARNING: CPU: 128 PID: 0 at kernel/locking/lockdep.c:2755 lockdep_trace_alloc+0xdd/0xe0() > > [ 3.193000] DEBUG_LOCKS_WARN_ON(irqs_disabled_flags(flags)) > > [ 3.199000] Modules linked in: > > > > [ 3.203000] CPU: 128 PID: 0 Comm: swapper/128 Not tainted 4.1.0-rc3 #70 > > [ 3.221000] 0000000000000000 2d6601fb3e6d4e4c ffff88086fd5fc38 ffffffff81773f0a > > [ 3.230000] 0000000000000000 ffff88086fd5fc90 ffff88086fd5fc78 ffffffff8108c85a > > [ 3.238000] ffff88086fd60000 0000000000000092 ffff88086fd60000 00000000000000d0 > > [ 3.246000] Call Trace: > > [ 3.249000] [] dump_stack+0x4c/0x65 > > [ 3.255000] [] warn_slowpath_common+0x8a/0xc0 > > [ 3.261000] [] warn_slowpath_fmt+0x55/0x70 > > [ 3.268000] [] lockdep_trace_alloc+0xdd/0xe0 > > [ 3.274000] [] __alloc_pages_nodemask+0xad/0xca0 > > [ 3.281000] [] ? __lock_acquire+0xf6d/0x1560 > > [ 3.288000] [] alloc_page_interleave+0x3a/0x90 > > [ 3.295000] [] alloc_pages_current+0x17d/0x1a0 > > [ 3.301000] [] ? __get_free_pages+0xe/0x50 > > [ 3.308000] [] __get_free_pages+0xe/0x50 > > [ 3.314000] [] init_espfix_ap+0x17b/0x320 > > [ 3.320000] [] start_secondary+0xf1/0x1f0 > > [ 3.327000] ---[ end trace 1b3327d9d6a1d62c ]--- > > > > As we alloc pages with GFP_KERNEL in init_espfix_ap() which is called > > before enabled local irq, and the lockdep sub-system considers this > > behaviour as allocating memory with GFP_FS with local irq disabled, > > then trigger the warning as mentioned about. > > > > Though we could allocate them on the boot CPU side and hand them over to > > the secondary CPU, but it seemes a bit waste if some of cpus are offline. > > As thers is no need to these pages(espfix stack) until we try to run user > > code, so we postpone the initialization of espfix stack, and let the boot > > up routine init the espfix stack for the target cpu after it booted to > > avoid the noise. > > > > It isn't *at all* obvious to me at least that if the GFP_KERNEL > allocation fails we may not get rescheduled on another CPU and/or get stuck. > > I'm starting to think that the right thing to do is to allocate these on > the CPU that is bringing up the other CPU, at the same time we allocate > the percpu area. This won't affect offline CPUs. Btw, as part of experimenting for something else, I was able to trigger this even on a guest here. It is an insane guest though: 16 NUMA nodes, with 8 cores each: [ 0.032000] ------------[ cut here ]------------ [ 0.032000] WARNING: CPU: 64 PID: 0 at kernel/locking/lockdep.c:2755 lockdep_trace_alloc+0x10c/0x120() [ 0.032000] DEBUG_LOCKS_WARN_ON(irqs_disabled_flags(flags)) [ 0.032000] Modules linked in: [ 0.032000] CPU: 64 PID: 0 Comm: swapper/64 Not tainted 4.1.0-rc3+ #4 [ 0.032000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 [ 0.032000] ffffffff818dd1a1 ffff880006a1fca8 ffffffff816a9685 0000000000000000 [ 0.032000] ffff880006a1fcf8 ffff880006a1fce8 ffffffff81058585 00000000001d74c0 [ 0.032000] 0000000000000080 0000000000000046 ffff880047ffcd00 ffff88003e804058 [ 0.032000] Call Trace: [ 0.032000] [] dump_stack+0x4f/0x7b [ 0.032000] [] warn_slowpath_common+0x95/0xe0 [ 0.032000] [] warn_slowpath_fmt+0x46/0x50 [ 0.032000] [] lockdep_trace_alloc+0x10c/0x120 [ 0.032000] [] __alloc_pages_nodemask+0xad/0xab0 [ 0.032000] [] ? debug_smp_processor_id+0x17/0x20 [ 0.032000] [] ? put_lock_stats.isra.19+0xe/0x30 [ 0.032000] [] ? mutex_lock_nested+0x2e8/0x420 [ 0.032000] [] alloc_page_interleave+0x3c/0x90 [ 0.032000] [] alloc_pages_current+0xc5/0xd0 [ 0.032000] [] __get_free_pages+0x14/0x50 [ 0.032000] [] init_espfix_ap.part.5+0x164/0x270 [ 0.032000] [] init_espfix_ap+0x21/0x30 [ 0.032000] [] start_secondary+0xe8/0x180 [ 0.032000] ---[ end trace 6a7abdb28fbb7667 ]--- Now I can test the future fix too. :) -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply. --