From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hsin-Yi Wang Date: Thu, 16 Jan 2020 09:25:43 +0000 Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot Message-Id: List-Id: References: <20200115063410.131692-1-hsinyi@chromium.org> <20200115114112.GA3663@bogus> In-Reply-To: <20200115114112.GA3663@bogus> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Sudeep Holla Cc: Mark Rutland , linux-ia64@vger.kernel.org, Linux-sh list , Peter Zijlstra , Heiko Carstens , lkml , sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Ingo Molnar , linux-s390@vger.kernel.org, linux-csky@vger.kernel.org, Aaro Koskinen , Fenghua Yu , Linux PM , linux-xtensa@linux-xtensa.org, Stephen Boyd , Josh Poimboeuf , Thomas Gleixner , Pavankumar Kondeti , "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , linux-parisc@vger.kernel.org, Greg Kroah-Hartman , linux-mips@vger.kernel.org, James Morse , Jiri Kosina , Vitaly Kuznetsov , linuxppc-dev On Wed, Jan 15, 2020 at 7:41 PM Sudeep Holla wrote: > > On Wed, Jan 15, 2020 at 02:34:10PM +0800, Hsin-Yi Wang wrote: > > Currently system reboots uses architecture specific codes (smp_send_stop) > > to offline non reboot CPUs. Most architecture's implementation is looping > > through all non reboot online CPUs and call ipi function to each of them. Some > > architecture like arm64, arm, and x86... would set offline masks to cpu without > > really offline them. This causes some race condition and kernel warning comes > > out sometimes when system reboots. > > > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > > checking online cpus would be an empty loop. If architecture don't enable this > > config, or some cpus somehow fails to offline, it would fallback to ipi > > function. > > > > What's the timing impact on systems with large number of CPUs(say 256 or > more) ? I remember we added some change to reduce the wait times for > offlining CPUs in system suspend path on arm64, still not negligible. > This is not the final solution, but I would still provided some data points here: Tested on my arm64 with 4 cpu: 2 a53 and 2 a72. Offlining 3 cpu takes about 60~65 ms Offlining 2 cpu(a53+a72 or a72+a72) takes about 42~47 ms Offlining 1 cpu(a53 or a72) takes about 23~25 ms. It would take longer time for systems with large number of CPUs. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0269BC33CB6 for ; Thu, 16 Jan 2020 09:26:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C8A882075B for ; Thu, 16 Jan 2020 09:26:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="YNsZvqMC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730663AbgAPJ0K (ORCPT ); Thu, 16 Jan 2020 04:26:10 -0500 Received: from mail-il1-f194.google.com ([209.85.166.194]:33262 "EHLO mail-il1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730597AbgAPJ0K (ORCPT ); Thu, 16 Jan 2020 04:26:10 -0500 Received: by mail-il1-f194.google.com with SMTP id v15so17570058iln.0 for ; Thu, 16 Jan 2020 01:26:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5+T2+f3YUqy/mz/g0zqdHuyixNv3uIRSRLqHpYCvIA8=; b=YNsZvqMC3qBwhA390kR+0O1+Z775NXpgorZ1Uj0LTNfhunlCVnboeL+hdbdTD7zPyj cO4YdefQbXLyr/t0viYTQNIs2/R9znJRs+JPdwF+zlywPmjg5MJOT97iGih9buKS3tvr detFCcSpM8To0oU1Jp0ON3jtZma7mcyCUwQck= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5+T2+f3YUqy/mz/g0zqdHuyixNv3uIRSRLqHpYCvIA8=; b=ix6KNVxYweSexurBvSjfppPThl0KjOnPTDeYF7CPNgiDUPpSs5b5uUuxj6dR2UIucV bwHBxA/1sS1/nVJ+VtKvweoiJ8hpPSmZa36XcOcSTFFM1HCmkGcq66bYTgbghJhz099/ sHTW9s7rJ5AdaehLs84bh+gpxX0G70J9JfpR+aTYNVwwtTK6CnDfq3ceQ0ZjQtNlQk9d IagtTkH4Tdzu2rzatRPGq3d2XNGsQolwxHE2aBWd5PljwVZwGFMbTP8/1wHDWXmPg1qw /Ycbafii5CfEE5BI3EdMwTdBaOL1O5ExjjDAuFA+cOwrzkQY/GVEOhitqhlWxXrpBiiQ XC1g== X-Gm-Message-State: APjAAAVhlMcjs6g35T/w6XkI/Mmo0xqwkmwPSe1Vwlq9oMS6JBn/Ildw Az1g0XfCfdDMZ4/xo0e62kKCOr2wDSUBH/UMk/kRTQ== X-Google-Smtp-Source: APXvYqzcJuvwLp0x5+fMsP13ADUUEVQCs/aLETOB1GPnAQGr8kZxelM+YUGAkJZnZhbBbnQ8HdB5j4UwUbclM7dRrHw= X-Received: by 2002:a92:8851:: with SMTP id h78mr2599028ild.308.1579166769584; Thu, 16 Jan 2020 01:26:09 -0800 (PST) MIME-Version: 1.0 References: <20200115063410.131692-1-hsinyi@chromium.org> <20200115114112.GA3663@bogus> In-Reply-To: <20200115114112.GA3663@bogus> From: Hsin-Yi Wang Date: Thu, 16 Jan 2020 17:25:43 +0800 Message-ID: Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot To: Sudeep Holla Cc: Thomas Gleixner , Josh Poimboeuf , Ingo Molnar , Peter Zijlstra , Jiri Kosina , Pavankumar Kondeti , Vitaly Kuznetsov , Aaro Koskinen , Greg Kroah-Hartman , Will Deacon , Fenghua Yu , James Morse , Mark Rutland , Heiko Carstens , Guenter Roeck , Stephen Boyd , lkml , "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , linux-csky@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev , linux-s390@vger.kernel.org, Linux-sh list , sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, Linux PM Content-Type: text/plain; charset="UTF-8" Sender: linux-parisc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-parisc@vger.kernel.org On Wed, Jan 15, 2020 at 7:41 PM Sudeep Holla wrote: > > On Wed, Jan 15, 2020 at 02:34:10PM +0800, Hsin-Yi Wang wrote: > > Currently system reboots uses architecture specific codes (smp_send_stop) > > to offline non reboot CPUs. Most architecture's implementation is looping > > through all non reboot online CPUs and call ipi function to each of them. Some > > architecture like arm64, arm, and x86... would set offline masks to cpu without > > really offline them. This causes some race condition and kernel warning comes > > out sometimes when system reboots. > > > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > > checking online cpus would be an empty loop. If architecture don't enable this > > config, or some cpus somehow fails to offline, it would fallback to ipi > > function. > > > > What's the timing impact on systems with large number of CPUs(say 256 or > more) ? I remember we added some change to reduce the wait times for > offlining CPUs in system suspend path on arm64, still not negligible. > This is not the final solution, but I would still provided some data points here: Tested on my arm64 with 4 cpu: 2 a53 and 2 a72. Offlining 3 cpu takes about 60~65 ms Offlining 2 cpu(a53+a72 or a72+a72) takes about 42~47 ms Offlining 1 cpu(a53 or a72) takes about 23~25 ms. It would take longer time for systems with large number of CPUs. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-il1-f196.google.com ([209.85.166.196]:35945 "EHLO mail-il1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729619AbgAPJ0K (ORCPT ); Thu, 16 Jan 2020 04:26:10 -0500 Received: by mail-il1-f196.google.com with SMTP id b15so17562192iln.3 for ; Thu, 16 Jan 2020 01:26:09 -0800 (PST) MIME-Version: 1.0 References: <20200115063410.131692-1-hsinyi@chromium.org> <20200115114112.GA3663@bogus> In-Reply-To: <20200115114112.GA3663@bogus> From: Hsin-Yi Wang Date: Thu, 16 Jan 2020 17:25:43 +0800 Message-ID: Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot Content-Type: text/plain; charset="UTF-8" Sender: linux-s390-owner@vger.kernel.org List-ID: To: Sudeep Holla Cc: Thomas Gleixner , Josh Poimboeuf , Ingo Molnar , Peter Zijlstra , Jiri Kosina , Pavankumar Kondeti , Vitaly Kuznetsov , Aaro Koskinen , Greg Kroah-Hartman , Will Deacon , Fenghua Yu , James Morse , Mark Rutland , Heiko Carstens , Guenter Roeck , Stephen Boyd , lkml , "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , linux-csky@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev , linux-s390@vger.kernel.org, Linux-sh list , sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, Linux PM On Wed, Jan 15, 2020 at 7:41 PM Sudeep Holla wrote: > > On Wed, Jan 15, 2020 at 02:34:10PM +0800, Hsin-Yi Wang wrote: > > Currently system reboots uses architecture specific codes (smp_send_stop) > > to offline non reboot CPUs. Most architecture's implementation is looping > > through all non reboot online CPUs and call ipi function to each of them. Some > > architecture like arm64, arm, and x86... would set offline masks to cpu without > > really offline them. This causes some race condition and kernel warning comes > > out sometimes when system reboots. > > > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > > checking online cpus would be an empty loop. If architecture don't enable this > > config, or some cpus somehow fails to offline, it would fallback to ipi > > function. > > > > What's the timing impact on systems with large number of CPUs(say 256 or > more) ? I remember we added some change to reduce the wait times for > offlining CPUs in system suspend path on arm64, still not negligible. > This is not the final solution, but I would still provided some data points here: Tested on my arm64 with 4 cpu: 2 a53 and 2 a72. Offlining 3 cpu takes about 60~65 ms Offlining 2 cpu(a53+a72 or a72+a72) takes about 42~47 ms Offlining 1 cpu(a53 or a72) takes about 23~25 ms. It would take longer time for systems with large number of CPUs. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BF54C33CB1 for ; Thu, 16 Jan 2020 09:29:31 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 191E9207FF for ; Thu, 16 Jan 2020 09:29:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="YNsZvqMC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 191E9207FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 47yzRY07gxzDq6Q for ; Thu, 16 Jan 2020 20:29:29 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=chromium.org (client-ip=2607:f8b0:4864:20::144; helo=mail-il1-x144.google.com; envelope-from=hsinyi@chromium.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=YNsZvqMC; dkim-atps=neutral Received: from mail-il1-x144.google.com (mail-il1-x144.google.com [IPv6:2607:f8b0:4864:20::144]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 47yzMn3LwszDqZy for ; Thu, 16 Jan 2020 20:26:13 +1100 (AEDT) Received: by mail-il1-x144.google.com with SMTP id t17so17515590ilm.13 for ; Thu, 16 Jan 2020 01:26:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5+T2+f3YUqy/mz/g0zqdHuyixNv3uIRSRLqHpYCvIA8=; b=YNsZvqMC3qBwhA390kR+0O1+Z775NXpgorZ1Uj0LTNfhunlCVnboeL+hdbdTD7zPyj cO4YdefQbXLyr/t0viYTQNIs2/R9znJRs+JPdwF+zlywPmjg5MJOT97iGih9buKS3tvr detFCcSpM8To0oU1Jp0ON3jtZma7mcyCUwQck= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5+T2+f3YUqy/mz/g0zqdHuyixNv3uIRSRLqHpYCvIA8=; b=WmCKdn2IEJTLKwxPhefh5KeXldnj9G/KyirbrooySVjdJrnR7HjiaAv27q1zQP4npz ccTlvQfLuIiaM70E7tRiEab5grIOIjVm7D0rf2c7oy7V+dMCI+6HIiyx+m0/BxEaX2wG tcFVYFzRDJyO8Medzxyx8gFM3m3qEnPovOmdXTTF+hapyU07RAvt8SVUisciTAydSRik F1yCLkXO7rkHi8Dwq50NzF4tfMiI/htCN1i11VCcYSWKdODefq8FyYxYyAfuSelGaB4g 1hvH18O5DIJtXxom+aTn5iw46JVY31A411Zb2vKs0IPMDxpEw0W0RmwjsZGf2J7smC2m jsRg== X-Gm-Message-State: APjAAAU4bqXyLliRmwiSaVR8T5M/6I4LR23GiNdms+8eQ0a5UbESq5u3 +HTZs7TUNmlBvPzgXPvmv4fuEB6AJ8WefBXYEPQ9Lg== X-Google-Smtp-Source: APXvYqzcJuvwLp0x5+fMsP13ADUUEVQCs/aLETOB1GPnAQGr8kZxelM+YUGAkJZnZhbBbnQ8HdB5j4UwUbclM7dRrHw= X-Received: by 2002:a92:8851:: with SMTP id h78mr2599028ild.308.1579166769584; Thu, 16 Jan 2020 01:26:09 -0800 (PST) MIME-Version: 1.0 References: <20200115063410.131692-1-hsinyi@chromium.org> <20200115114112.GA3663@bogus> In-Reply-To: <20200115114112.GA3663@bogus> From: Hsin-Yi Wang Date: Thu, 16 Jan 2020 17:25:43 +0800 Message-ID: Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot To: Sudeep Holla Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , linux-ia64@vger.kernel.org, Linux-sh list , Peter Zijlstra , Heiko Carstens , lkml , sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Ingo Molnar , linux-s390@vger.kernel.org, linux-csky@vger.kernel.org, Aaro Koskinen , Fenghua Yu , Linux PM , linux-xtensa@linux-xtensa.org, Stephen Boyd , Josh Poimboeuf , Thomas Gleixner , Pavankumar Kondeti , "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , linux-parisc@vger.kernel.org, Greg Kroah-Hartman , linux-mips@vger.kernel.org, James Morse , Jiri Kosina , Vitaly Kuznetsov , linuxppc-dev Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Wed, Jan 15, 2020 at 7:41 PM Sudeep Holla wrote: > > On Wed, Jan 15, 2020 at 02:34:10PM +0800, Hsin-Yi Wang wrote: > > Currently system reboots uses architecture specific codes (smp_send_stop) > > to offline non reboot CPUs. Most architecture's implementation is looping > > through all non reboot online CPUs and call ipi function to each of them. Some > > architecture like arm64, arm, and x86... would set offline masks to cpu without > > really offline them. This causes some race condition and kernel warning comes > > out sometimes when system reboots. > > > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > > checking online cpus would be an empty loop. If architecture don't enable this > > config, or some cpus somehow fails to offline, it would fallback to ipi > > function. > > > > What's the timing impact on systems with large number of CPUs(say 256 or > more) ? I remember we added some change to reduce the wait times for > offlining CPUs in system suspend path on arm64, still not negligible. > This is not the final solution, but I would still provided some data points here: Tested on my arm64 with 4 cpu: 2 a53 and 2 a72. Offlining 3 cpu takes about 60~65 ms Offlining 2 cpu(a53+a72 or a72+a72) takes about 42~47 ms Offlining 1 cpu(a53 or a72) takes about 23~25 ms. It would take longer time for systems with large number of CPUs. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C561C33CB1 for ; Thu, 16 Jan 2020 09:26:20 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6F8502075B for ; Thu, 16 Jan 2020 09:26:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="evU3paJ6"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="YNsZvqMC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6F8502075B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8UiW4QeqISnOOcaClZIRMmfdUaKR72nXRz21OcSSHUc=; b=evU3paJ62a5ZQK JcQHB9l5/FJg9xtp8PQN2kEprQI1sCWJwW5yckAgDpiij0txwZCQvXw083aGwrqKkYROGzDVF9tsf jtphSMBunGTcwG5q9tY3kB96p6Wy+iQG89sVlpeySiNOmZt82GfJBIdfrtlvTWYAFlpR4Smqu7eIn lPzPMytTE3vkNyZyEyspR127yXG36It9qmMl8zgjAuLIJf27z56gHp5z/yeX0l8XRyKAyGcbsfbQl vaVpquJQPDq6zdmvQp8lrH5/1YPu8JAOZUe02YPr6r3hXmmbBU1+e8Pj+pCI8d5x0yG12Kcjz4UH3 TvuAL/tqn6tAw8LI+XWw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1is1Ps-0007pQ-Qe; Thu, 16 Jan 2020 09:26:16 +0000 Received: from mail-il1-x143.google.com ([2607:f8b0:4864:20::143]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1is1Pm-0007oO-Fn for linux-arm-kernel@lists.infradead.org; Thu, 16 Jan 2020 09:26:14 +0000 Received: by mail-il1-x143.google.com with SMTP id c4so17549902ilo.7 for ; Thu, 16 Jan 2020 01:26:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5+T2+f3YUqy/mz/g0zqdHuyixNv3uIRSRLqHpYCvIA8=; b=YNsZvqMC3qBwhA390kR+0O1+Z775NXpgorZ1Uj0LTNfhunlCVnboeL+hdbdTD7zPyj cO4YdefQbXLyr/t0viYTQNIs2/R9znJRs+JPdwF+zlywPmjg5MJOT97iGih9buKS3tvr detFCcSpM8To0oU1Jp0ON3jtZma7mcyCUwQck= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5+T2+f3YUqy/mz/g0zqdHuyixNv3uIRSRLqHpYCvIA8=; b=pPPsAGK4cKKXMinIWNFFxswRyTv3VyflcYkcGA0yr90cdf2Ac1XFsLHrWe38GTiHn4 sUGBkiNxBYcqGp85R3tpqclZjgrpwyra8kMR0ASH3X05atD9DoZ+6EvvUjAsCDdoes4o 1rBpQWkU5xtXQAYDpFGzTjmulD4kmGa87bzSy+4oFTb6PbPpjh0ntAt2cngaSILcRQOD 35lAuJt9Z4v+QfTECc39wYBKQZUz/kQedeppG8SniApv4ZYl4cWCnD5mk0VI4pI3P4nn xRGRdenCPllFD4X14YC+iocL0E1nP08x57k7LZuZP0KQYr7z5jFB4wDDwNkyZhV+R9ET WVvQ== X-Gm-Message-State: APjAAAUMyP7XOrubkK9iIHaI73OBk5GF33XoJSITxROmpV1hsolp4RqL 2QEaxqrhT1IcHTYO48soIlp+VGgHh5ukZPWCv7LtGg== X-Google-Smtp-Source: APXvYqzcJuvwLp0x5+fMsP13ADUUEVQCs/aLETOB1GPnAQGr8kZxelM+YUGAkJZnZhbBbnQ8HdB5j4UwUbclM7dRrHw= X-Received: by 2002:a92:8851:: with SMTP id h78mr2599028ild.308.1579166769584; Thu, 16 Jan 2020 01:26:09 -0800 (PST) MIME-Version: 1.0 References: <20200115063410.131692-1-hsinyi@chromium.org> <20200115114112.GA3663@bogus> In-Reply-To: <20200115114112.GA3663@bogus> From: Hsin-Yi Wang Date: Thu, 16 Jan 2020 17:25:43 +0800 Message-ID: Subject: Re: [PATCH v5] reboot: support offline CPUs before reboot To: Sudeep Holla X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200116_012610_551715_CC3A0166 X-CRM114-Status: GOOD ( 15.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , linux-ia64@vger.kernel.org, Linux-sh list , Peter Zijlstra , Heiko Carstens , lkml , sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Ingo Molnar , linux-s390@vger.kernel.org, linux-csky@vger.kernel.org, Aaro Koskinen , Fenghua Yu , Linux PM , linux-xtensa@linux-xtensa.org, Stephen Boyd , Josh Poimboeuf , Thomas Gleixner , Pavankumar Kondeti , "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , linux-parisc@vger.kernel.org, Greg Kroah-Hartman , linux-mips@vger.kernel.org, James Morse , Jiri Kosina , Vitaly Kuznetsov , linuxppc-dev Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jan 15, 2020 at 7:41 PM Sudeep Holla wrote: > > On Wed, Jan 15, 2020 at 02:34:10PM +0800, Hsin-Yi Wang wrote: > > Currently system reboots uses architecture specific codes (smp_send_stop) > > to offline non reboot CPUs. Most architecture's implementation is looping > > through all non reboot online CPUs and call ipi function to each of them. Some > > architecture like arm64, arm, and x86... would set offline masks to cpu without > > really offline them. This causes some race condition and kernel warning comes > > out sometimes when system reboots. > > > > This patch adds a config ARCH_OFFLINE_CPUS_ON_REBOOT, which would offline cpus in > > migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for > > checking online cpus would be an empty loop. If architecture don't enable this > > config, or some cpus somehow fails to offline, it would fallback to ipi > > function. > > > > What's the timing impact on systems with large number of CPUs(say 256 or > more) ? I remember we added some change to reduce the wait times for > offlining CPUs in system suspend path on arm64, still not negligible. > This is not the final solution, but I would still provided some data points here: Tested on my arm64 with 4 cpu: 2 a53 and 2 a72. Offlining 3 cpu takes about 60~65 ms Offlining 2 cpu(a53+a72 or a72+a72) takes about 42~47 ms Offlining 1 cpu(a53 or a72) takes about 23~25 ms. It would take longer time for systems with large number of CPUs. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel