From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FFFCC433DF for ; Fri, 3 Jul 2020 14:42:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 60D9620899 for ; Fri, 3 Jul 2020 14:42:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726363AbgGCOmg (ORCPT ); Fri, 3 Jul 2020 10:42:36 -0400 Received: from foss.arm.com ([217.140.110.172]:40346 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726053AbgGCOmf (ORCPT ); Fri, 3 Jul 2020 10:42:35 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2925931B; Fri, 3 Jul 2020 07:42:34 -0700 (PDT) Received: from [10.57.21.32] (unknown [10.57.21.32]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DE9E03F73C; Fri, 3 Jul 2020 07:42:32 -0700 (PDT) Subject: Re: [RFC PATCH] perf/smmuv3: Fix shared interrupt handling To: Will Deacon Cc: mark.rutland@arm.com, tuanphan@os.amperecomputing.com, john.garry@huawei.com, linux-kernel@vger.kernel.org, shameerali.kolothum.thodi@huawei.com, harb@amperecomputing.com, linux-arm-kernel@lists.infradead.org References: <20200624125045.GC6270@willie-the-truck> <20200703134213.GE18953@willie-the-truck> From: Robin Murphy Message-ID: <6099af78-0fd8-77de-fe50-be40b239f06e@arm.com> Date: Fri, 3 Jul 2020 15:42:31 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20200703134213.GE18953@willie-the-truck> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2020-07-03 14:42, Will Deacon wrote: > On Wed, Jun 24, 2020 at 02:08:30PM +0100, Robin Murphy wrote: >> On 2020-06-24 13:50, Will Deacon wrote: >>> On Wed, Jun 24, 2020 at 12:48:14PM +0100, Robin Murphy wrote: >>>> On 2020-04-08 17:49, Robin Murphy wrote: >>>>> IRQF_SHARED is dangerous, since it allows other agents to retarget the >>>>> IRQ's affinity without migrating PMU contexts to match, breaking the way >>>>> in which perf manages mutual exclusion for accessing events. Although >>>>> this means it's not realistically possible to support PMU IRQs being >>>>> shared with other drivers, we *can* handle sharing between multiple PMU >>>>> instances with some explicit affinity bookkeeping and manual interrupt >>>>> multiplexing. >>>>> >>>>> RCU helps us handle interrupts efficiently without having to worry about >>>>> fine-grained locking for relatively-theoretical race conditions with the >>>>> probe/remove/CPU hotplug slow paths. The resulting machinery ends up >>>>> looking largely generic, so it should be feasible to factor out with a >>>>> "system PMU" base class for similar multi-instance drivers. >>>>> >>>>> Signed-off-by: Robin Murphy >>>>> --- >>>>> >>>>> RFC because I don't have the means to test it, and if the general >>>>> approach passes muster then I'd want to tackle the aforementioned >>>>> factoring-out before merging anything anyway. >>>> >>>> Any comments on whether it's worth pursuing this? >>> >>> Sorry, I don't really get the problem that it's solving. Is there a crash >>> log somewhere I can look at? If all the users of the IRQ are managed by >>> this driver, why is IRQF_SHARED dangerous? >> >> Because as-is, multiple PMU instances may make different choices about which >> CPU they associate with, change the shared IRQ affinity behind each others' >> backs, and break the "IRQ handler runs on event->cpu" assumption that perf >> core relies on for correctness. I'm not sure how likely it would be to >> actually crash rather than just lead to subtle nastiness, but wither way >> it's not good, and since people seem to be tempted to wire up system PMU >> instances this way we could do with a general approach for dealing with it. > > Ok, thanks for the explanation. If we're just talking about multiple > instances of the same driver, why is it not sufficient to have a static > atomic_t initialised to -1 which tracks the current affinity and then just > CAS that during probe()? Hotplug notifiers can just check whether or not > it points to an online CPU Yeah, forcing *all* PMUs owned by a driver to be affine to the same CPU is another way to go about it, however it slightly penalises systems that are wired up sensibly and *would* otherwise be able to distribute non-shared affinities around in a balanced manner (optimising the initial pmu->cpu selection in the face of NUMA is an exercise still on the table in some cases). And we'd still have to have all the "has another instance already requested this IRQ or not yet?" logic (the general condition is "1 <= number of IRQs <= number of PMUs"), plus some way for the global affinity to migrate all the PMU contexts and IRQs at once in a controlled and race-free manner, so things wouldn't be *massively* simpler even then. Robin. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDAE2C433E0 for ; Fri, 3 Jul 2020 14:44:07 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A1F452070B for ; Fri, 3 Jul 2020 14:44:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="DSU/zo1y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A1F452070B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=DGCZNOsgMwvCQTYDZdUYcEQAOEP3z0DZCyg5OR0ua4I=; b=DSU/zo1ypSRGmEUaZIAnrVt1T r6ccWJoM0gDGtljDihcykqcA+31okdUj1Pc3A2xtPdurTIf8k81bFZu3HD21NewdOjtuNCN18sBhT 3YhdoxnEXIhHH9JlfMtZcacpQTBytwzsIYYf0dTcUXkBry4TPoaKuIhthBmf+8KVuP6tH6D45Q71M sUzKLhBhSR+a5Wlcad6yl2P5IMBgP3Mm96HtwArELGtXzkCfEjt9ZkbNR9IIGD6Xfa5DdY6cS1q8F s2I4URc1IuYmE0fOEyH33O/2VBiX5MBiBxI81EWB9um/IkqiTgIuGUJmUr8/Ae3p3jGFm4vU+qsZ+ wL1F13Lbw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jrMtk-0002KS-J3; Fri, 03 Jul 2020 14:42:40 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jrMth-0002Jg-Se for linux-arm-kernel@lists.infradead.org; Fri, 03 Jul 2020 14:42:38 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2925931B; Fri, 3 Jul 2020 07:42:34 -0700 (PDT) Received: from [10.57.21.32] (unknown [10.57.21.32]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DE9E03F73C; Fri, 3 Jul 2020 07:42:32 -0700 (PDT) Subject: Re: [RFC PATCH] perf/smmuv3: Fix shared interrupt handling To: Will Deacon References: <20200624125045.GC6270@willie-the-truck> <20200703134213.GE18953@willie-the-truck> From: Robin Murphy Message-ID: <6099af78-0fd8-77de-fe50-be40b239f06e@arm.com> Date: Fri, 3 Jul 2020 15:42:31 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20200703134213.GE18953@willie-the-truck> Content-Language: en-GB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200703_104238_038674_BB4D2041 X-CRM114-Status: GOOD ( 25.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, tuanphan@os.amperecomputing.com, john.garry@huawei.com, linux-kernel@vger.kernel.org, shameerali.kolothum.thodi@huawei.com, harb@amperecomputing.com, linux-arm-kernel@lists.infradead.org Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2020-07-03 14:42, Will Deacon wrote: > On Wed, Jun 24, 2020 at 02:08:30PM +0100, Robin Murphy wrote: >> On 2020-06-24 13:50, Will Deacon wrote: >>> On Wed, Jun 24, 2020 at 12:48:14PM +0100, Robin Murphy wrote: >>>> On 2020-04-08 17:49, Robin Murphy wrote: >>>>> IRQF_SHARED is dangerous, since it allows other agents to retarget the >>>>> IRQ's affinity without migrating PMU contexts to match, breaking the way >>>>> in which perf manages mutual exclusion for accessing events. Although >>>>> this means it's not realistically possible to support PMU IRQs being >>>>> shared with other drivers, we *can* handle sharing between multiple PMU >>>>> instances with some explicit affinity bookkeeping and manual interrupt >>>>> multiplexing. >>>>> >>>>> RCU helps us handle interrupts efficiently without having to worry about >>>>> fine-grained locking for relatively-theoretical race conditions with the >>>>> probe/remove/CPU hotplug slow paths. The resulting machinery ends up >>>>> looking largely generic, so it should be feasible to factor out with a >>>>> "system PMU" base class for similar multi-instance drivers. >>>>> >>>>> Signed-off-by: Robin Murphy >>>>> --- >>>>> >>>>> RFC because I don't have the means to test it, and if the general >>>>> approach passes muster then I'd want to tackle the aforementioned >>>>> factoring-out before merging anything anyway. >>>> >>>> Any comments on whether it's worth pursuing this? >>> >>> Sorry, I don't really get the problem that it's solving. Is there a crash >>> log somewhere I can look at? If all the users of the IRQ are managed by >>> this driver, why is IRQF_SHARED dangerous? >> >> Because as-is, multiple PMU instances may make different choices about which >> CPU they associate with, change the shared IRQ affinity behind each others' >> backs, and break the "IRQ handler runs on event->cpu" assumption that perf >> core relies on for correctness. I'm not sure how likely it would be to >> actually crash rather than just lead to subtle nastiness, but wither way >> it's not good, and since people seem to be tempted to wire up system PMU >> instances this way we could do with a general approach for dealing with it. > > Ok, thanks for the explanation. If we're just talking about multiple > instances of the same driver, why is it not sufficient to have a static > atomic_t initialised to -1 which tracks the current affinity and then just > CAS that during probe()? Hotplug notifiers can just check whether or not > it points to an online CPU Yeah, forcing *all* PMUs owned by a driver to be affine to the same CPU is another way to go about it, however it slightly penalises systems that are wired up sensibly and *would* otherwise be able to distribute non-shared affinities around in a balanced manner (optimising the initial pmu->cpu selection in the face of NUMA is an exercise still on the table in some cases). And we'd still have to have all the "has another instance already requested this IRQ or not yet?" logic (the general condition is "1 <= number of IRQs <= number of PMUs"), plus some way for the global affinity to migrate all the PMU contexts and IRQs at once in a controlled and race-free manner, so things wouldn't be *massively* simpler even then. Robin. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel