From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1A59C004D4 for ; Thu, 19 Jan 2023 18:42:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230286AbjASSmm (ORCPT ); Thu, 19 Jan 2023 13:42:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229997AbjASSmi (ORCPT ); Thu, 19 Jan 2023 13:42:38 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B9CC2E0D9 for ; Thu, 19 Jan 2023 10:42:35 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id z5so2754964wrt.6 for ; Thu, 19 Jan 2023 10:42:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=yl655zxTlgKELJ7swQ9SsdqOuQT3cGOOSK8xkhPG2HU=; b=xBUP8DNf2arziCDahO9pk12ouk+PsskloAOQBC50IXTwelJgqpyn3nieGTuaG8+Ibe VGeIMw04x2WMAIh09ApFbjnoGrdSwJ/nrQIncdDHA2gkOjTyamuGDWhmP9RhmrS3PaUL FPuzu0/9zO2xF/7q3fELemdpSWVWc07FUjAMHQncnlJFB8IAUW2KIl20F4PQLUYUu0ZP 6gXoZZsfybADlm+5xZjK5RL/s1jHRkfac6j6LUycFIfz/G7VQI4mOkJv7LeqTEllgLWo 4P1qWB+7FfJu5oAqjnEduXhMHFmlLmCwcL/lUyZmd7XtyKZsS1RdJBHBBu4mCLHyGRqE F0QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yl655zxTlgKELJ7swQ9SsdqOuQT3cGOOSK8xkhPG2HU=; b=LhESeuO7dyenGNXu9/BkRGN7liqAPih9gl0YXqqWlRNcOpE1icHSqFGs3s8Qa0cmtx MYxDiVoeW3UuC+pt8jUqeBUaDh/EYNeO917/5eSvcvABK0xoT3ekQLQbPSzSHp6IdkJX CQMK+5hmyEnJJKJzkZzOu5U9JjI/j+9G27H61lU+Huh+YbAjCZo62W4vQCIX6F0J7h1V 65tt/+5LJgdic5rJ3aiVXM8CIK2Add5ZbdqPpU55vwIUlx1mjs4IgPeUCbR5L2uUjT+2 nBmQYa/qq6KgW+x4cGVakL94b37JoLEKSu8N7qn8YaZeuzf6cNhLR0oMwJTvGqOe3Jb+ hthQ== X-Gm-Message-State: AFqh2krLUqZVuj6Mri1jqX1UqbBPaeY+C/7SILgqSyphmMFODNhZfhEi LBh8B5RwFz5X8THKv9aP+F2mfQ== X-Google-Smtp-Source: AMrXdXsNVX1Zn89uw1K4g/OFWItkRWvguxRz92nMDKqyqmJCDwfcv/k8QdIe+EOUorPW7tOKQuQgNw== X-Received: by 2002:adf:a318:0:b0:2bd:dc0c:ffd1 with SMTP id c24-20020adfa318000000b002bddc0cffd1mr10185811wrb.13.1674153754017; Thu, 19 Jan 2023 10:42:34 -0800 (PST) Received: from krzk-bin.. ([178.197.216.144]) by smtp.gmail.com with ESMTPSA id q18-20020adfdfd2000000b002bdc129c8f6sm22865410wrn.43.2023.01.19.10.42.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Jan 2023 10:42:33 -0800 (PST) From: Krzysztof Kozlowski To: "Rafael J. Wysocki" , Daniel Lezcano , Lorenzo Pieralisi , Sudeep Holla , linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Sebastian Andrzej Siewior , Ulf Hansson , Krzysztof Kozlowski , Adrien Thierry , Brian Masney , linux-rt-users@vger.kernel.org Subject: [PATCH v3] cpuidle: psci: Do not suspend topology CPUs on PREEMPT_RT Date: Thu, 19 Jan 2023 19:42:28 +0100 Message-Id: <20230119184228.683892-1-krzysztof.kozlowski@linaro.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The runtime Power Management of CPU topology is not compatible with PREEMPT_RT: 1. Core cpuidle path disables IRQs. 2. Core cpuidle calls cpuidle-psci. 3. cpuidle-psci in __psci_enter_domain_idle_state() calls pm_runtime_put_sync_suspend() and pm_runtime_get_sync() which use spinlocks (which are sleeping on PREEMPT_RT). Deep sleep modes are not a priority of Realtime kernels because the latencies might become unpredictable. On the other hand the PSCI CPU idle power domain is a parent of other devices and power domain controllers, thus it cannot be simply skipped (e.g. on Qualcomm SM8250). Disable the runtime PM calls from cpuidle-psci, which effectively stops suspending the cpuidle PSCI domain. This is a trade-off between making PREEMPT_RT working and still having a proper power domain hierarchy in the system. Cc: Adrien Thierry Cc: Brian Masney Cc: linux-rt-users@vger.kernel.org Signed-off-by: Krzysztof Kozlowski --- Changes since v1: 1. Re-work commit msg. 2. Add note to Kconfig. Several other patches were dropped, as this is the only one actually needed. It effectively stops PSCI cpuidle power domains from suspending thus solving all other issues I experienced. --- drivers/cpuidle/Kconfig.arm | 3 +++ drivers/cpuidle/cpuidle-psci.c | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/drivers/cpuidle/Kconfig.arm b/drivers/cpuidle/Kconfig.arm index 747aa537389b..24429b5bfd1c 100644 --- a/drivers/cpuidle/Kconfig.arm +++ b/drivers/cpuidle/Kconfig.arm @@ -24,6 +24,9 @@ config ARM_PSCI_CPUIDLE It provides an idle driver that is capable of detecting and managing idle states through the PSCI firmware interface. + The driver is not yet compatible with PREEMPT_RT: no idle states will + be entered by CPUs on such kernel. + config ARM_PSCI_CPUIDLE_DOMAIN bool "PSCI CPU idle Domain" depends on ARM_PSCI_CPUIDLE diff --git a/drivers/cpuidle/cpuidle-psci.c b/drivers/cpuidle/cpuidle-psci.c index 312a34ef28dc..c25592718984 100644 --- a/drivers/cpuidle/cpuidle-psci.c +++ b/drivers/cpuidle/cpuidle-psci.c @@ -66,7 +66,7 @@ static __cpuidle int __psci_enter_domain_idle_state(struct cpuidle_device *dev, /* Do runtime PM to manage a hierarchical CPU toplogy. */ if (s2idle) dev_pm_genpd_suspend(pd_dev); - else + else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) pm_runtime_put_sync_suspend(pd_dev); state = psci_get_domain_state(); @@ -77,7 +77,7 @@ static __cpuidle int __psci_enter_domain_idle_state(struct cpuidle_device *dev, if (s2idle) dev_pm_genpd_resume(pd_dev); - else + else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) pm_runtime_get_sync(pd_dev); cpu_pm_exit(); -- 2.34.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 206EAC46467 for ; Thu, 19 Jan 2023 18:43:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=iLWLfhrj6+13qu2S9wj64cHfX+uQMMY4ExcHT27XmVI=; b=lDGiSAGq54c0kj MPYrsGwQFI9ApWlvfbCHrGPyO6ypARckxuqpi+rboxXpjTFzUo50mtHyshlOiZU1tiZ0aEJGnuXuf J5aZnvurMUN+updXc7y4joMb0zOj02AYBQdv6p/U82YS25hA5/sY1thfj/q46oaXkCvUqk4VX+NPE 6nGlwiVONhLiEvBtw1lOV4/IzSKchDgmgDdnDY8CqZuYaVbBa14467zkeho4JUQRQumB/rP+1XbWg uqzNwZZKGSPEIvHfNzKatLNjMFwWn0Di8JLXg6VG7EikB/9uA87p0JheiRtZxSMk4lJk9VntjCN4W 12rRegGoNOyFAVRKT17w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pIZs1-006bt9-UR; Thu, 19 Jan 2023 18:42:42 +0000 Received: from mail-wr1-x42e.google.com ([2a00:1450:4864:20::42e]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pIZrz-006bs7-3I for linux-arm-kernel@lists.infradead.org; Thu, 19 Jan 2023 18:42:40 +0000 Received: by mail-wr1-x42e.google.com with SMTP id h16so2727524wrz.12 for ; Thu, 19 Jan 2023 10:42:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=yl655zxTlgKELJ7swQ9SsdqOuQT3cGOOSK8xkhPG2HU=; b=xBUP8DNf2arziCDahO9pk12ouk+PsskloAOQBC50IXTwelJgqpyn3nieGTuaG8+Ibe VGeIMw04x2WMAIh09ApFbjnoGrdSwJ/nrQIncdDHA2gkOjTyamuGDWhmP9RhmrS3PaUL FPuzu0/9zO2xF/7q3fELemdpSWVWc07FUjAMHQncnlJFB8IAUW2KIl20F4PQLUYUu0ZP 6gXoZZsfybADlm+5xZjK5RL/s1jHRkfac6j6LUycFIfz/G7VQI4mOkJv7LeqTEllgLWo 4P1qWB+7FfJu5oAqjnEduXhMHFmlLmCwcL/lUyZmd7XtyKZsS1RdJBHBBu4mCLHyGRqE F0QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yl655zxTlgKELJ7swQ9SsdqOuQT3cGOOSK8xkhPG2HU=; b=mChgNSg+n/AZS200T8xPNuO2TUPGXYTZxpeeTU4yNuvoKid/NJePBVcS4FJxPh1vOE uLq6uXxvLbxDP1HHl6OcsFhpeBSfl9/GD3q1W5PNEOKEDIillMKR4+jzjonmNbMz2kXl zMmy82rFu+fL85giBpWhSY2RVxc4eMPgkY6BW8UUu59bpjraO+4WtpjHyW3uRhuCuvE1 t90j8xsD3IOW78TN/X6QEJ0cjV8Jgcq1Ia8qNdhORP7cFd4fqFuk0hXZQcbE0zmaJlS8 GtHMTDhxpzX9RHXJZdNVsiTTQ36mvF9ESRIEM8Ri6ufTqeJh9+9R5Wy/mldwP8NNCOFt E3Cg== X-Gm-Message-State: AFqh2kpAu/dW0HNX2H9JtW6h8cT/+myQfz9NaGlN9NbqLg8k36DKYwDr gd6waokVXk7Lz+mPiTmckQ4N0Q== X-Google-Smtp-Source: AMrXdXsNVX1Zn89uw1K4g/OFWItkRWvguxRz92nMDKqyqmJCDwfcv/k8QdIe+EOUorPW7tOKQuQgNw== X-Received: by 2002:adf:a318:0:b0:2bd:dc0c:ffd1 with SMTP id c24-20020adfa318000000b002bddc0cffd1mr10185811wrb.13.1674153754017; Thu, 19 Jan 2023 10:42:34 -0800 (PST) Received: from krzk-bin.. ([178.197.216.144]) by smtp.gmail.com with ESMTPSA id q18-20020adfdfd2000000b002bdc129c8f6sm22865410wrn.43.2023.01.19.10.42.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Jan 2023 10:42:33 -0800 (PST) From: Krzysztof Kozlowski To: "Rafael J. Wysocki" , Daniel Lezcano , Lorenzo Pieralisi , Sudeep Holla , linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Sebastian Andrzej Siewior , Ulf Hansson , Krzysztof Kozlowski , Adrien Thierry , Brian Masney , linux-rt-users@vger.kernel.org Subject: [PATCH v3] cpuidle: psci: Do not suspend topology CPUs on PREEMPT_RT Date: Thu, 19 Jan 2023 19:42:28 +0100 Message-Id: <20230119184228.683892-1-krzysztof.kozlowski@linaro.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230119_104239_186757_8FECEB87 X-CRM114-Status: GOOD ( 14.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The runtime Power Management of CPU topology is not compatible with PREEMPT_RT: 1. Core cpuidle path disables IRQs. 2. Core cpuidle calls cpuidle-psci. 3. cpuidle-psci in __psci_enter_domain_idle_state() calls pm_runtime_put_sync_suspend() and pm_runtime_get_sync() which use spinlocks (which are sleeping on PREEMPT_RT). Deep sleep modes are not a priority of Realtime kernels because the latencies might become unpredictable. On the other hand the PSCI CPU idle power domain is a parent of other devices and power domain controllers, thus it cannot be simply skipped (e.g. on Qualcomm SM8250). Disable the runtime PM calls from cpuidle-psci, which effectively stops suspending the cpuidle PSCI domain. This is a trade-off between making PREEMPT_RT working and still having a proper power domain hierarchy in the system. Cc: Adrien Thierry Cc: Brian Masney Cc: linux-rt-users@vger.kernel.org Signed-off-by: Krzysztof Kozlowski --- Changes since v1: 1. Re-work commit msg. 2. Add note to Kconfig. Several other patches were dropped, as this is the only one actually needed. It effectively stops PSCI cpuidle power domains from suspending thus solving all other issues I experienced. --- drivers/cpuidle/Kconfig.arm | 3 +++ drivers/cpuidle/cpuidle-psci.c | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/drivers/cpuidle/Kconfig.arm b/drivers/cpuidle/Kconfig.arm index 747aa537389b..24429b5bfd1c 100644 --- a/drivers/cpuidle/Kconfig.arm +++ b/drivers/cpuidle/Kconfig.arm @@ -24,6 +24,9 @@ config ARM_PSCI_CPUIDLE It provides an idle driver that is capable of detecting and managing idle states through the PSCI firmware interface. + The driver is not yet compatible with PREEMPT_RT: no idle states will + be entered by CPUs on such kernel. + config ARM_PSCI_CPUIDLE_DOMAIN bool "PSCI CPU idle Domain" depends on ARM_PSCI_CPUIDLE diff --git a/drivers/cpuidle/cpuidle-psci.c b/drivers/cpuidle/cpuidle-psci.c index 312a34ef28dc..c25592718984 100644 --- a/drivers/cpuidle/cpuidle-psci.c +++ b/drivers/cpuidle/cpuidle-psci.c @@ -66,7 +66,7 @@ static __cpuidle int __psci_enter_domain_idle_state(struct cpuidle_device *dev, /* Do runtime PM to manage a hierarchical CPU toplogy. */ if (s2idle) dev_pm_genpd_suspend(pd_dev); - else + else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) pm_runtime_put_sync_suspend(pd_dev); state = psci_get_domain_state(); @@ -77,7 +77,7 @@ static __cpuidle int __psci_enter_domain_idle_state(struct cpuidle_device *dev, if (s2idle) dev_pm_genpd_resume(pd_dev); - else + else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) pm_runtime_get_sync(pd_dev); cpu_pm_exit(); -- 2.34.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel