From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sudeep Holla Date: Wed, 15 Jan 2020 11:41:12 +0000 Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot Message-Id: <20200115114112.GA3663@bogus> List-Id: References: <20200115063410.131692-1-hsinyi@chromium.org> In-Reply-To: <20200115063410.131692-1-hsinyi@chromium.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Hsin-Yi Wang Cc: Mark Rutland , linux-ia64@vger.kernel.org, linux-sh@vger.kernel.org, Peter Zijlstra , Heiko Carstens , linux-kernel@vger.kernel.org, sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Ingo Molnar , linux-s390@vger.kernel.org, linux-csky@vger.kernel.org, Aaro Koskinen , Fenghua Yu , linux-pm@vger.kernel.org, linux-xtensa@linux-xtensa.org, Sudeep Holla , Stephen Boyd , Josh Poimboeuf , Thomas Gleixner , Pavankumar Kondeti , linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, Greg Kroah-Hartman , linux-mips@vger.kernel.org, James Morse , Jiri Kosina , Vitaly Kuznetsov , linuxppc-dev@lists.ozlabs.org On Wed, Jan 15, 2020 at 02:34:10PM +0800, Hsin-Yi Wang wrote: > Currently system reboots uses architecture specific codes (smp_send_stop) > to offline non reboot CPUs. Most architecture's implementation is looping > through all non reboot online CPUs and call ipi function to each of them. Some > architecture like arm64, arm, and x86... would set offline masks to cpu without > really offline them. This causes some race condition and kernel warning comes > out sometimes when system reboots. > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > checking online cpus would be an empty loop. If architecture don't enable this > config, or some cpus somehow fails to offline, it would fallback to ipi > function. > What's the timing impact on systems with large number of CPUs(say 256 or more) ? I remember we added some change to reduce the wait times for offlining CPUs in system suspend path on arm64, still not negligible. -- Regards, Sudeep From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4023FC33CB2 for ; Wed, 15 Jan 2020 11:41:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 15ED9222C3 for ; Wed, 15 Jan 2020 11:41:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729903AbgAOLlX (ORCPT ); Wed, 15 Jan 2020 06:41:23 -0500 Received: from foss.arm.com ([217.140.110.172]:35530 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729900AbgAOLlW (ORCPT ); Wed, 15 Jan 2020 06:41:22 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B1E1931B; Wed, 15 Jan 2020 03:41:21 -0800 (PST) Received: from bogus (e103737-lin.cambridge.arm.com [10.1.197.49]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2ABE23F6C4; Wed, 15 Jan 2020 03:41:18 -0800 (PST) Date: Wed, 15 Jan 2020 11:41:12 +0000 From: Sudeep Holla To: Hsin-Yi Wang Cc: Thomas Gleixner , Josh Poimboeuf , Ingo Molnar , Peter Zijlstra , Jiri Kosina , Pavankumar Kondeti , Vitaly Kuznetsov , Aaro Koskinen , Greg Kroah-Hartman , Will Deacon , Fenghua Yu , James Morse , Mark Rutland , Heiko Carstens , Guenter Roeck , Stephen Boyd , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, Sudeep Holla , linux-pm@vger.kernel.org Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot Message-ID: <20200115114112.GA3663@bogus> References: <20200115063410.131692-1-hsinyi@chromium.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200115063410.131692-1-hsinyi@chromium.org> User-Agent: Mutt/1.9.4 (2018-02-28) Sender: linux-parisc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-parisc@vger.kernel.org On Wed, Jan 15, 2020 at 02:34:10PM +0800, Hsin-Yi Wang wrote: > Currently system reboots uses architecture specific codes (smp_send_stop) > to offline non reboot CPUs. Most architecture's implementation is looping > through all non reboot online CPUs and call ipi function to each of them. Some > architecture like arm64, arm, and x86... would set offline masks to cpu without > really offline them. This causes some race condition and kernel warning comes > out sometimes when system reboots. > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > checking online cpus would be an empty loop. If architecture don't enable this > config, or some cpus somehow fails to offline, it would fallback to ipi > function. > What's the timing impact on systems with large number of CPUs(say 256 or more) ? I remember we added some change to reduce the wait times for offlining CPUs in system suspend path on arm64, still not negligible. -- Regards, Sudeep From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60FD6C33CB1 for ; Wed, 15 Jan 2020 11:41:31 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 333FC24679 for ; Wed, 15 Jan 2020 11:41:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="gntwOnEt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 333FC24679 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mE7yHAkodIXaseibqxTZT6wb0wLpkxqmNXT7wqaeg7o=; b=gntwOnEtlhKnnF c6JarkglIFey6CEKFDqXAA+9zpnvWgd/JepuMAomdGj2/xpKHqBK9dORKn4U4kux8+68wkgjKL8Ti FENMZYtLWTXgqCtGt0DnrVmYIacUzUBksvA/TXFjIt+YBDRsOC+nxr9TdFBMAW1ipMfeP8mGxn6g/ U0hsR+GhmbJriZQb2MaTYLay2q/I5iop8D3y89kdEMMfucz6Dw4np8JNpXhxkFwQzUsqM0yksNthA s6hDaCN8vWsju5xmPMINNHmRUU5PFe3q6HyU5IirjIC1sqit+j0abViczT/feMBFHq8hhau5rjVcb 3gIEsJn39/VxF8O4hxSg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1irh3C-0007wN-DI; Wed, 15 Jan 2020 11:41:30 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1irh34-0007vB-KX for linux-arm-kernel@lists.infradead.org; Wed, 15 Jan 2020 11:41:28 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B1E1931B; Wed, 15 Jan 2020 03:41:21 -0800 (PST) Received: from bogus (e103737-lin.cambridge.arm.com [10.1.197.49]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2ABE23F6C4; Wed, 15 Jan 2020 03:41:18 -0800 (PST) Date: Wed, 15 Jan 2020 11:41:12 +0000 From: Sudeep Holla To: Hsin-Yi Wang Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot Message-ID: <20200115114112.GA3663@bogus> References: <20200115063410.131692-1-hsinyi@chromium.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200115063410.131692-1-hsinyi@chromium.org> User-Agent: Mutt/1.9.4 (2018-02-28) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200115_034122_719506_7DE561E6 X-CRM114-Status: GOOD ( 13.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , linux-ia64@vger.kernel.org, linux-sh@vger.kernel.org, Peter Zijlstra , Heiko Carstens , linux-kernel@vger.kernel.org, sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Ingo Molnar , linux-s390@vger.kernel.org, linux-csky@vger.kernel.org, Aaro Koskinen , Fenghua Yu , linux-pm@vger.kernel.org, linux-xtensa@linux-xtensa.org, Sudeep Holla , Stephen Boyd , Josh Poimboeuf , Thomas Gleixner , Pavankumar Kondeti , linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, Greg Kroah-Hartman , linux-mips@vger.kernel.org, James Morse , Jiri Kosina , Vitaly Kuznetsov , linuxppc-dev@lists.ozlabs.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jan 15, 2020 at 02:34:10PM +0800, Hsin-Yi Wang wrote: > Currently system reboots uses architecture specific codes (smp_send_stop) > to offline non reboot CPUs. Most architecture's implementation is looping > through all non reboot online CPUs and call ipi function to each of them. Some > architecture like arm64, arm, and x86... would set offline masks to cpu without > really offline them. This causes some race condition and kernel warning comes > out sometimes when system reboots. > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > checking online cpus would be an empty loop. If architecture don't enable this > config, or some cpus somehow fails to offline, it would fallback to ipi > function. > What's the timing impact on systems with large number of CPUs(say 256 or more) ? I remember we added some change to reduce the wait times for offlining CPUs in system suspend path on arm64, still not negligible. -- Regards, Sudeep _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel