From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Rafael J. Wysocki" Date: Wed, 15 Jan 2020 09:49:17 +0000 Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot Message-Id: List-Id: References: <20200115063410.131692-1-hsinyi@chromium.org> In-Reply-To: <20200115063410.131692-1-hsinyi@chromium.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Hsin-Yi Wang Cc: Mark Rutland , linux-ia64@vger.kernel.org, Linux-sh list , Peter Zijlstra , Heiko Carstens , Linux Kernel Mailing List , sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Ingo Molnar , linux-s390@vger.kernel.org, linux-csky@vger.kernel.org, Aaro Koskinen , Fenghua Yu , Linux PM , linux-xtensa@linux-xtensa.org, Stephen Boyd , Josh Poimboeuf , Thomas Gleixner , Pavankumar Kondeti , Linux ARM , linux-parisc@vger.kernel.org, Greg Kroah-Hartman , linux-mips@vger.kernel.org, James Morse , Jiri Kosina , Vitaly Kuznetsov , linuxppc-dev On Wed, Jan 15, 2020 at 7:35 AM Hsin-Yi Wang wrote: > > Currently system reboots uses architecture specific codes (smp_send_stop) > to offline non reboot CPUs. Most architecture's implementation is looping > through all non reboot online CPUs and call ipi function to each of them. Some > architecture like arm64, arm, and x86... would set offline masks to cpu without > really offline them. This causes some race condition and kernel warning comes > out sometimes when system reboots. > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > checking online cpus would be an empty loop. If architecture don't enable this > config, or some cpus somehow fails to offline, it would fallback to ipi > function. > > Opt in this config for architectures that support CONFIG_HOTPLUG_CPU. > > Signed-off-by: Hsin-Yi Wang > --- > Change from v4: > * fix a few nits: naming, comments, remove Kconfig text... > > Change from v3: > * Opt in config for architectures that support CONFIG_HOTPLUG_CPU > * Merge function offline_secondary_cpus() and freeze_secondary_cpus() > with an additional flag. This does not seem to be a very good idea, since freeze_secondary_cpus() does much more than you need for reboot. For reboot, you basically only need to do something like this AFAICS: cpu_maps_update_begin(); for_each_online_cpu(i) { if (i != cpu) _cpu_down(i, 1, CPUHP_OFFLINE); } cpu_hotplug_disabled++; cpu_maps_update_done(); And you may put this into a function defined outside of CONFIG_PM_SLEEP. > > Change from v2: > * Add another config instead of configed by CONFIG_HOTPLUG_CPU So why exactly is this new Kconfig option needed? Everybody supporting CPU hotplug seems to opt in anyway. [cut] > > -int freeze_secondary_cpus(int primary) > +int freeze_secondary_cpus(int primary, bool reboot) > { > int cpu, error = 0; > > @@ -1237,11 +1237,13 @@ int freeze_secondary_cpus(int primary) > if (cpu = primary) > continue; > > - if (pm_wakeup_pending()) { > +#ifdef CONFIG_PM_SLEEP > + if (!reboot && pm_wakeup_pending()) { > pr_info("Wakeup pending. Abort CPU freeze\n"); > error = -EBUSY; > break; > } > +#endif Please avoid using #ifdefs in function bodies. This makes the code hard to maintain in the long term. > > trace_suspend_resume(TPS("CPU_OFF"), cpu, true); > error = _cpu_down(cpu, 1, CPUHP_OFFLINE); > @@ -1250,7 +1252,9 @@ int freeze_secondary_cpus(int primary) > cpumask_set_cpu(cpu, frozen_cpus); > else { > pr_err("Error taking CPU%d down: %d\n", cpu, error); > - break; > + /* When rebooting, offline as many CPUs as possible. */ > + if (!reboot) > + break; > } > } > > diff --git a/kernel/reboot.c b/kernel/reboot.c > index c4d472b7f1b4..12f643b66e57 100644 > --- a/kernel/reboot.c > +++ b/kernel/reboot.c > @@ -7,6 +7,7 @@ > > #define pr_fmt(fmt) "reboot: " fmt > > +#include > #include > #include > #include > @@ -220,7 +221,9 @@ void migrate_to_reboot_cpu(void) > /* The boot cpu is always logical cpu 0 */ > int cpu = reboot_cpu; > > +#if !IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT) > cpu_hotplug_disable(); > +#endif You can write this as if (!IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT)) cpu_hotplug_disable(); That's what IS_ENABLED() is there for. > > /* Make certain the cpu I'm about to reboot on is online */ > if (!cpu_online(cpu)) > @@ -231,6 +234,11 @@ void migrate_to_reboot_cpu(void) > > /* Make certain I only run on the appropriate processor */ > set_cpus_allowed_ptr(current, cpumask_of(cpu)); > + > +#if IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT) > + /* Offline other cpus if possible */ > + freeze_secondary_cpus(cpu, true); > +#endif The above comment applies here too. > } > > /** > -- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB61DC33CB3 for ; Wed, 15 Jan 2020 09:49:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A377B24671 for ; Wed, 15 Jan 2020 09:49:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1579081774; bh=s7oQ95LUMoNiII/w7YJlT+AHtSuE4O6j/QDVgfxLDyM=; h=References:In-Reply-To:From:Date:Subject:To:Cc:List-ID:From; b=ja+FVUwNipDFD8CCIPQgOTW+ApeaKjVT8DGsPC8MCZBStU0nvj4jSmeJIEle3Px9D usbj8bKoamty3ujHBnKEey/kAcCYgrU/lBhCs7G9gWS3fCSwCL+65GrM7C70apkauu 9yreLk0tJjTCYTvc+8SgSw8W3Kx76McoItUiqQos= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729568AbgAOJta (ORCPT ); Wed, 15 Jan 2020 04:49:30 -0500 Received: from mail-oi1-f193.google.com ([209.85.167.193]:35278 "EHLO mail-oi1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729559AbgAOJta (ORCPT ); Wed, 15 Jan 2020 04:49:30 -0500 Received: by mail-oi1-f193.google.com with SMTP id k4so14830271oik.2; Wed, 15 Jan 2020 01:49:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xlblpk3Q7AFTy8Enno95ZSx0IUH147BofZyxZsTxPpQ=; b=TucKJXUCA3iBOurtQTxVBa/z8C9REkYA7BJC4bLHsU20pWHX8G0CPEON/zgGfgROz8 jwyhbLr5j2JjwsgdL2GIT3EzLxo7Di8fOQrERkdEJ7U7/QiPIC90oZ50fmH92oivcX4w 4nk8FEs52YgRW4roJaXdtPnaDSiUZuO+jsaSeqw5obwUuylTkiKm+kAy0NqZaqaEbJGp HyvO4SXZNtNmMLSezXxOt1GFZUvU8xOr9u0riNPtbG6A2o43vIEf5c94v4ZdcUQ/+rNA 09NbTz5Mgy/U+8mMr6AAl1/vUPkueJt0UHwsbWG3+q7l+/SeT1Za/tB2usl63ImC6Iqf Y4Og== X-Gm-Message-State: APjAAAVX1XL4PXVWExXRv2a3FQ2mHbtqRNbZ2YgU8z0/8nYn4u+xpCf+ 7f32MQtaREYqi5KzILq3sFk0XhhtpRHD3q7EPg8= X-Google-Smtp-Source: APXvYqzvsE9JWkTyA/qGCPe6FC5SfX/eWtD++abCAXNNUKmVhLTpM4TrHUCIWNo6kY2HmIQTdm2QXbcLDAmQBj9bi24= X-Received: by 2002:aca:d6c8:: with SMTP id n191mr21412346oig.103.1579081768509; Wed, 15 Jan 2020 01:49:28 -0800 (PST) MIME-Version: 1.0 References: <20200115063410.131692-1-hsinyi@chromium.org> In-Reply-To: <20200115063410.131692-1-hsinyi@chromium.org> From: "Rafael J. Wysocki" Date: Wed, 15 Jan 2020 10:49:17 +0100 Message-ID: Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot To: Hsin-Yi Wang Cc: Thomas Gleixner , Josh Poimboeuf , Ingo Molnar , Peter Zijlstra , Jiri Kosina , Pavankumar Kondeti , Vitaly Kuznetsov , Aaro Koskinen , Greg Kroah-Hartman , Will Deacon , Fenghua Yu , James Morse , Mark Rutland , Heiko Carstens , Guenter Roeck , Stephen Boyd , Linux Kernel Mailing List , Linux ARM , linux-csky@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev , linux-s390@vger.kernel.org, Linux-sh list , sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, Linux PM Content-Type: text/plain; charset="UTF-8" Sender: linux-parisc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-parisc@vger.kernel.org On Wed, Jan 15, 2020 at 7:35 AM Hsin-Yi Wang wrote: > > Currently system reboots uses architecture specific codes (smp_send_stop) > to offline non reboot CPUs. Most architecture's implementation is looping > through all non reboot online CPUs and call ipi function to each of them. Some > architecture like arm64, arm, and x86... would set offline masks to cpu without > really offline them. This causes some race condition and kernel warning comes > out sometimes when system reboots. > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > checking online cpus would be an empty loop. If architecture don't enable this > config, or some cpus somehow fails to offline, it would fallback to ipi > function. > > Opt in this config for architectures that support CONFIG_HOTPLUG_CPU. > > Signed-off-by: Hsin-Yi Wang > --- > Change from v4: > * fix a few nits: naming, comments, remove Kconfig text... > > Change from v3: > * Opt in config for architectures that support CONFIG_HOTPLUG_CPU > * Merge function offline_secondary_cpus() and freeze_secondary_cpus() > with an additional flag. This does not seem to be a very good idea, since freeze_secondary_cpus() does much more than you need for reboot. For reboot, you basically only need to do something like this AFAICS: cpu_maps_update_begin(); for_each_online_cpu(i) { if (i != cpu) _cpu_down(i, 1, CPUHP_OFFLINE); } cpu_hotplug_disabled++; cpu_maps_update_done(); And you may put this into a function defined outside of CONFIG_PM_SLEEP. > > Change from v2: > * Add another config instead of configed by CONFIG_HOTPLUG_CPU So why exactly is this new Kconfig option needed? Everybody supporting CPU hotplug seems to opt in anyway. [cut] > > -int freeze_secondary_cpus(int primary) > +int freeze_secondary_cpus(int primary, bool reboot) > { > int cpu, error = 0; > > @@ -1237,11 +1237,13 @@ int freeze_secondary_cpus(int primary) > if (cpu == primary) > continue; > > - if (pm_wakeup_pending()) { > +#ifdef CONFIG_PM_SLEEP > + if (!reboot && pm_wakeup_pending()) { > pr_info("Wakeup pending. Abort CPU freeze\n"); > error = -EBUSY; > break; > } > +#endif Please avoid using #ifdefs in function bodies. This makes the code hard to maintain in the long term. > > trace_suspend_resume(TPS("CPU_OFF"), cpu, true); > error = _cpu_down(cpu, 1, CPUHP_OFFLINE); > @@ -1250,7 +1252,9 @@ int freeze_secondary_cpus(int primary) > cpumask_set_cpu(cpu, frozen_cpus); > else { > pr_err("Error taking CPU%d down: %d\n", cpu, error); > - break; > + /* When rebooting, offline as many CPUs as possible. */ > + if (!reboot) > + break; > } > } > > diff --git a/kernel/reboot.c b/kernel/reboot.c > index c4d472b7f1b4..12f643b66e57 100644 > --- a/kernel/reboot.c > +++ b/kernel/reboot.c > @@ -7,6 +7,7 @@ > > #define pr_fmt(fmt) "reboot: " fmt > > +#include > #include > #include > #include > @@ -220,7 +221,9 @@ void migrate_to_reboot_cpu(void) > /* The boot cpu is always logical cpu 0 */ > int cpu = reboot_cpu; > > +#if !IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT) > cpu_hotplug_disable(); > +#endif You can write this as if (!IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT)) cpu_hotplug_disable(); That's what IS_ENABLED() is there for. > > /* Make certain the cpu I'm about to reboot on is online */ > if (!cpu_online(cpu)) > @@ -231,6 +234,11 @@ void migrate_to_reboot_cpu(void) > > /* Make certain I only run on the appropriate processor */ > set_cpus_allowed_ptr(current, cpumask_of(cpu)); > + > +#if IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT) > + /* Offline other cpus if possible */ > + freeze_secondary_cpus(cpu, true); > +#endif The above comment applies here too. > } > > /** > -- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90F67C33CB1 for ; Wed, 15 Jan 2020 09:51:44 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0C2FD207FF for ; Wed, 15 Jan 2020 09:51:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C2FD207FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 47yMzd257nzDqSC for ; Wed, 15 Jan 2020 20:51:41 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=209.85.167.196; helo=mail-oi1-f196.google.com; envelope-from=rjwysocki@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=fail (p=none dis=none) header.from=kernel.org Received: from mail-oi1-f196.google.com (mail-oi1-f196.google.com [209.85.167.196]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 47yMxB5zvmzDqQT for ; Wed, 15 Jan 2020 20:49:31 +1100 (AEDT) Received: by mail-oi1-f196.google.com with SMTP id i1so14786187oie.8 for ; Wed, 15 Jan 2020 01:49:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xlblpk3Q7AFTy8Enno95ZSx0IUH147BofZyxZsTxPpQ=; b=Qjsz7S8TC0kT17pOyv0/vU9dix0kBMQuABaRNs0c0UxGBkCa/BaAwqlV84c4Yl798H K2r2nC0MiiNjzZRFG6wcXOe/kT9JAQ/jWt6CQZMG5rdlIV98G9YnEI1srBKSusPArXHl xKeFZ19wSDjpD/g/iL2nDH+QaLNnF/fI/9a3ClIkhtYc/Mw4+UJ2sjSHFvI0tOokaoQX llZRxJFYqyj1YYEemSVVr9NgqAIpXG+Z6u35gqZK2WwIZgMPfhHsRAxe0GzQnZPIg8dl V/5/ndcK2DngYmJOAgudqLljzoy2K4otb9H3CnXbzeCTuaiTkImkOdPfw+b66RBDSkjl 2IxQ== X-Gm-Message-State: APjAAAWLwPbla1KNcugYaRW2r9f7a521kxPmdytyEgyoI0nPgsFwLrpv 1wOkkv4an8Yx+3zgKgy8vFDwI3vNQRe+Fw5zr84= X-Google-Smtp-Source: APXvYqzvsE9JWkTyA/qGCPe6FC5SfX/eWtD++abCAXNNUKmVhLTpM4TrHUCIWNo6kY2HmIQTdm2QXbcLDAmQBj9bi24= X-Received: by 2002:aca:d6c8:: with SMTP id n191mr21412346oig.103.1579081768509; Wed, 15 Jan 2020 01:49:28 -0800 (PST) MIME-Version: 1.0 References: <20200115063410.131692-1-hsinyi@chromium.org> In-Reply-To: <20200115063410.131692-1-hsinyi@chromium.org> From: "Rafael J. Wysocki" Date: Wed, 15 Jan 2020 10:49:17 +0100 Message-ID: Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot To: Hsin-Yi Wang Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , linux-ia64@vger.kernel.org, Linux-sh list , Peter Zijlstra , Heiko Carstens , Linux Kernel Mailing List , sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Ingo Molnar , linux-s390@vger.kernel.org, linux-csky@vger.kernel.org, Aaro Koskinen , Fenghua Yu , Linux PM , linux-xtensa@linux-xtensa.org, Stephen Boyd , Josh Poimboeuf , Thomas Gleixner , Pavankumar Kondeti , Linux ARM , linux-parisc@vger.kernel.org, Greg Kroah-Hartman , linux-mips@vger.kernel.org, James Morse , Jiri Kosina , Vitaly Kuznetsov , linuxppc-dev Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Wed, Jan 15, 2020 at 7:35 AM Hsin-Yi Wang wrote: > > Currently system reboots uses architecture specific codes (smp_send_stop) > to offline non reboot CPUs. Most architecture's implementation is looping > through all non reboot online CPUs and call ipi function to each of them. Some > architecture like arm64, arm, and x86... would set offline masks to cpu without > really offline them. This causes some race condition and kernel warning comes > out sometimes when system reboots. > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > checking online cpus would be an empty loop. If architecture don't enable this > config, or some cpus somehow fails to offline, it would fallback to ipi > function. > > Opt in this config for architectures that support CONFIG_HOTPLUG_CPU. > > Signed-off-by: Hsin-Yi Wang > --- > Change from v4: > * fix a few nits: naming, comments, remove Kconfig text... > > Change from v3: > * Opt in config for architectures that support CONFIG_HOTPLUG_CPU > * Merge function offline_secondary_cpus() and freeze_secondary_cpus() > with an additional flag. This does not seem to be a very good idea, since freeze_secondary_cpus() does much more than you need for reboot. For reboot, you basically only need to do something like this AFAICS: cpu_maps_update_begin(); for_each_online_cpu(i) { if (i != cpu) _cpu_down(i, 1, CPUHP_OFFLINE); } cpu_hotplug_disabled++; cpu_maps_update_done(); And you may put this into a function defined outside of CONFIG_PM_SLEEP. > > Change from v2: > * Add another config instead of configed by CONFIG_HOTPLUG_CPU So why exactly is this new Kconfig option needed? Everybody supporting CPU hotplug seems to opt in anyway. [cut] > > -int freeze_secondary_cpus(int primary) > +int freeze_secondary_cpus(int primary, bool reboot) > { > int cpu, error = 0; > > @@ -1237,11 +1237,13 @@ int freeze_secondary_cpus(int primary) > if (cpu == primary) > continue; > > - if (pm_wakeup_pending()) { > +#ifdef CONFIG_PM_SLEEP > + if (!reboot && pm_wakeup_pending()) { > pr_info("Wakeup pending. Abort CPU freeze\n"); > error = -EBUSY; > break; > } > +#endif Please avoid using #ifdefs in function bodies. This makes the code hard to maintain in the long term. > > trace_suspend_resume(TPS("CPU_OFF"), cpu, true); > error = _cpu_down(cpu, 1, CPUHP_OFFLINE); > @@ -1250,7 +1252,9 @@ int freeze_secondary_cpus(int primary) > cpumask_set_cpu(cpu, frozen_cpus); > else { > pr_err("Error taking CPU%d down: %d\n", cpu, error); > - break; > + /* When rebooting, offline as many CPUs as possible. */ > + if (!reboot) > + break; > } > } > > diff --git a/kernel/reboot.c b/kernel/reboot.c > index c4d472b7f1b4..12f643b66e57 100644 > --- a/kernel/reboot.c > +++ b/kernel/reboot.c > @@ -7,6 +7,7 @@ > > #define pr_fmt(fmt) "reboot: " fmt > > +#include > #include > #include > #include > @@ -220,7 +221,9 @@ void migrate_to_reboot_cpu(void) > /* The boot cpu is always logical cpu 0 */ > int cpu = reboot_cpu; > > +#if !IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT) > cpu_hotplug_disable(); > +#endif You can write this as if (!IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT)) cpu_hotplug_disable(); That's what IS_ENABLED() is there for. > > /* Make certain the cpu I'm about to reboot on is online */ > if (!cpu_online(cpu)) > @@ -231,6 +234,11 @@ void migrate_to_reboot_cpu(void) > > /* Make certain I only run on the appropriate processor */ > set_cpus_allowed_ptr(current, cpumask_of(cpu)); > + > +#if IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT) > + /* Offline other cpus if possible */ > + freeze_secondary_cpus(cpu, true); > +#endif The above comment applies here too. > } > > /** > -- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63D09C33CB1 for ; Wed, 15 Jan 2020 09:49:41 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 36F55222C3 for ; Wed, 15 Jan 2020 09:49:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="awsXsgkU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 36F55222C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=T1me4JrR7ZupS6Wzxsxe3G33Fg7WOYsvX6zYx1rFJ3w=; b=awsXsgkU7B5Spx TqNB7wSqH7DJJeegDMsJBkOzCs+a5pG8H3EAxgcFDMO78ljkVWKcKoKpuqu3z0zUltXBZlXqFNK8e 8F9+5BM1f1ePQ2nPNNNL92PxBlsMiBRMqY160ZT8e6ITxqSzcqGb94jMovF1H/XtUzouuSVqldzol OdZypUklDDPXQO5fdoSdYKIuof1U9qAYN1urdZ1nCzbyyUSvrTlFfY3F+Tfa395MHZ1P749koCQrr CNzNd+23uxNbGZVC+QnB8hfFTpbu1U0z0v3VTvvpuEVZmU6bICMv4t5hGL3VIpmJoR5Ft2fiwQ9RC sqPUgjhC+URtrOE/QHsw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1irfIy-0006ky-OK; Wed, 15 Jan 2020 09:49:40 +0000 Received: from mail-oi1-f193.google.com ([209.85.167.193]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1irfIn-0006aa-St for linux-arm-kernel@lists.infradead.org; Wed, 15 Jan 2020 09:49:31 +0000 Received: by mail-oi1-f193.google.com with SMTP id l136so14824896oig.1 for ; Wed, 15 Jan 2020 01:49:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xlblpk3Q7AFTy8Enno95ZSx0IUH147BofZyxZsTxPpQ=; b=E6jTvjLossMzpYec3Hgv2n7FqkQKKrvWeI5cMbI22dJX6l3MA2hJsh5D+ZWgNFckyV 5DWJufqNRWeyeYirpEyREfylIKaR6Vf3aHu0vQTnKthSjCq+Gm1qKlUw00CtFs/PrQ4D 4p56zlgs9KtXCIInJ0/+2/+2CgEjqAjRMD+meYQCybm7VrFi9165/hmdzy5N2tO8T0SU sUPpEuid8YsT098/Uvpapm4nybkrhf8IJmO/Wqp7pChBIeIWLDduYwr8v5F7Yx46kvxi aFunADRmJJDpDDQh81DXwmkQ7ZaOctX8EP3mzdSo/Dkn/3RMRm7ZHTGkX+JgXs0blwua IYbw== X-Gm-Message-State: APjAAAVClmmM0BHkG3QdY5pRskthYemR2t4WlSnKw5QiqibweJ3SD47L uVjhJmwDiuakbqdcwmz98ut5+cg1RzJXf51Rp2josVfD X-Google-Smtp-Source: APXvYqzvsE9JWkTyA/qGCPe6FC5SfX/eWtD++abCAXNNUKmVhLTpM4TrHUCIWNo6kY2HmIQTdm2QXbcLDAmQBj9bi24= X-Received: by 2002:aca:d6c8:: with SMTP id n191mr21412346oig.103.1579081768509; Wed, 15 Jan 2020 01:49:28 -0800 (PST) MIME-Version: 1.0 References: <20200115063410.131692-1-hsinyi@chromium.org> In-Reply-To: <20200115063410.131692-1-hsinyi@chromium.org> From: "Rafael J. Wysocki" Date: Wed, 15 Jan 2020 10:49:17 +0100 Message-ID: Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot To: Hsin-Yi Wang X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200115_014929_952462_C426235C X-CRM114-Status: GOOD ( 23.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , linux-ia64@vger.kernel.org, Linux-sh list , Peter Zijlstra , Heiko Carstens , Linux Kernel Mailing List , sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Ingo Molnar , linux-s390@vger.kernel.org, linux-csky@vger.kernel.org, Aaro Koskinen , Fenghua Yu , Linux PM , linux-xtensa@linux-xtensa.org, Stephen Boyd , Josh Poimboeuf , Thomas Gleixner , Pavankumar Kondeti , Linux ARM , linux-parisc@vger.kernel.org, Greg Kroah-Hartman , linux-mips@vger.kernel.org, James Morse , Jiri Kosina , Vitaly Kuznetsov , linuxppc-dev Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jan 15, 2020 at 7:35 AM Hsin-Yi Wang wrote: > > Currently system reboots uses architecture specific codes (smp_send_stop) > to offline non reboot CPUs. Most architecture's implementation is looping > through all non reboot online CPUs and call ipi function to each of them. Some > architecture like arm64, arm, and x86... would set offline masks to cpu without > really offline them. This causes some race condition and kernel warning comes > out sometimes when system reboots. > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > checking online cpus would be an empty loop. If architecture don't enable this > config, or some cpus somehow fails to offline, it would fallback to ipi > function. > > Opt in this config for architectures that support CONFIG_HOTPLUG_CPU. > > Signed-off-by: Hsin-Yi Wang > --- > Change from v4: > * fix a few nits: naming, comments, remove Kconfig text... > > Change from v3: > * Opt in config for architectures that support CONFIG_HOTPLUG_CPU > * Merge function offline_secondary_cpus() and freeze_secondary_cpus() > with an additional flag. This does not seem to be a very good idea, since freeze_secondary_cpus() does much more than you need for reboot. For reboot, you basically only need to do something like this AFAICS: cpu_maps_update_begin(); for_each_online_cpu(i) { if (i != cpu) _cpu_down(i, 1, CPUHP_OFFLINE); } cpu_hotplug_disabled++; cpu_maps_update_done(); And you may put this into a function defined outside of CONFIG_PM_SLEEP. > > Change from v2: > * Add another config instead of configed by CONFIG_HOTPLUG_CPU So why exactly is this new Kconfig option needed? Everybody supporting CPU hotplug seems to opt in anyway. [cut] > > -int freeze_secondary_cpus(int primary) > +int freeze_secondary_cpus(int primary, bool reboot) > { > int cpu, error = 0; > > @@ -1237,11 +1237,13 @@ int freeze_secondary_cpus(int primary) > if (cpu == primary) > continue; > > - if (pm_wakeup_pending()) { > +#ifdef CONFIG_PM_SLEEP > + if (!reboot && pm_wakeup_pending()) { > pr_info("Wakeup pending. Abort CPU freeze\n"); > error = -EBUSY; > break; > } > +#endif Please avoid using #ifdefs in function bodies. This makes the code hard to maintain in the long term. > > trace_suspend_resume(TPS("CPU_OFF"), cpu, true); > error = _cpu_down(cpu, 1, CPUHP_OFFLINE); > @@ -1250,7 +1252,9 @@ int freeze_secondary_cpus(int primary) > cpumask_set_cpu(cpu, frozen_cpus); > else { > pr_err("Error taking CPU%d down: %d\n", cpu, error); > - break; > + /* When rebooting, offline as many CPUs as possible. */ > + if (!reboot) > + break; > } > } > > diff --git a/kernel/reboot.c b/kernel/reboot.c > index c4d472b7f1b4..12f643b66e57 100644 > --- a/kernel/reboot.c > +++ b/kernel/reboot.c > @@ -7,6 +7,7 @@ > > #define pr_fmt(fmt) "reboot: " fmt > > +#include > #include > #include > #include > @@ -220,7 +221,9 @@ void migrate_to_reboot_cpu(void) > /* The boot cpu is always logical cpu 0 */ > int cpu = reboot_cpu; > > +#if !IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT) > cpu_hotplug_disable(); > +#endif You can write this as if (!IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT)) cpu_hotplug_disable(); That's what IS_ENABLED() is there for. > > /* Make certain the cpu I'm about to reboot on is online */ > if (!cpu_online(cpu)) > @@ -231,6 +234,11 @@ void migrate_to_reboot_cpu(void) > > /* Make certain I only run on the appropriate processor */ > set_cpus_allowed_ptr(current, cpumask_of(cpu)); > + > +#if IS_ENABLED(CONFIG_ARCH_OFFLINE_CPUS_ON_REBOOT) > + /* Offline other cpus if possible */ > + freeze_secondary_cpus(cpu, true); > +#endif The above comment applies here too. > } > > /** > -- _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel