From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD6A7C49ED7 for ; Tue, 10 Sep 2019 15:45:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8FCAF2089F for ; Tue, 10 Sep 2019 15:45:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FUDx1Vss" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436560AbfIJPpl (ORCPT ); Tue, 10 Sep 2019 11:45:41 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:41223 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726664AbfIJPpk (ORCPT ); Tue, 10 Sep 2019 11:45:40 -0400 Received: by mail-ed1-f67.google.com with SMTP id z9so17554148edq.8; Tue, 10 Sep 2019 08:45:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ab44L4UsH9lWBk/ePs+t6LQUuKmRHFW0SwCWgf68+rc=; b=FUDx1VssMY5+grYTJ9mTO+OtexdcbSTMSsn3//VPNV0zPdSp95pA3zNOdpx4boIzLL +8wfwTnUnSWaD0DyKayLxvtLd1DNtf9KpsXdscr1lunnkyj9RgO10ZaIY4qDgscT06xv gsu7UNxm0Xryuv9rrw5a4GpIEVPRKWhPVPWKzBpbTQx6lkHw+mIFYb+srXGudTMQJCt5 gonRN5rE57R/vyk+7sS8MiPSdeYzF3ZPo7wNMrmBj3lVkr6zixiEP6LKYNeZGu9gq5mv nhhDZVq5ROoxeNpfmQu0BO+jdHGVOWyuw0NkkZqryUCoeygfaKH+6X5QUf5q5jC92gdh i8nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ab44L4UsH9lWBk/ePs+t6LQUuKmRHFW0SwCWgf68+rc=; b=Es2qF/PBt86WViNHx5n1wadDtRjmQoaA+GpjzHtD0+mZGww26rr0GqG6hVJLPgpNGY igaZU0Z3wdB+SqSQ+/ESQb/8FoN9shxzLkKTthuTGCosuMJr04PbGINwu12HnMJHUI4W YkDXFwLz/whOmCtVIe23FU0wnWCYhHSnSHFJRgYq60SZTclvyel632ufHsdg8Iswz1Cw tejkec4dNsbjSFt5TlYar5gfoJQ1hsyPUaps6qlVJz4F/RjnfdnAZAwX7nUqx0kO+RpN CTFrjAutoFQhPb9CiF7GcJkIdLl6eWvPYa0zmOr9s/XXoDlQPFWYOUsvqmdr9klOGLvE ZyYg== X-Gm-Message-State: APjAAAVTvQKHFbwU2miQMHe4YwJlOj1xL0mvMkBZYcxZ/abR6CES7J3e 0LNxNEHvVvEMXuqL5HWzwCD8d0rAlhBb7pD+HOibdQfY X-Google-Smtp-Source: APXvYqzoKSvtwc7h3YF+c3ClC8RRDwvQEOuO9vmMklmmZ6/J8nMaSUMmvks3o2VNkPh4WYQkviIc241SjZs7jn1Ep9c= X-Received: by 2002:a17:906:2451:: with SMTP id a17mr16927007ejb.164.1568130338310; Tue, 10 Sep 2019 08:45:38 -0700 (PDT) MIME-Version: 1.0 References: <20190907175013.24246-1-robdclark@gmail.com> <418d8426-f299-1269-2b2e-f86677cf22c2@arm.com> In-Reply-To: <418d8426-f299-1269-2b2e-f86677cf22c2@arm.com> From: Rob Clark Date: Tue, 10 Sep 2019 08:45:27 -0700 Message-ID: Subject: Re: [PATCH] iommu/arm-smmu: fix "hang" when games exit To: Robin Murphy Cc: "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , linux-arm-msm , freedreno , Rob Clark , Will Deacon , Joerg Roedel , "moderated list:ARM SMMU DRIVERS" , open list , Jordan Crouse Content-Type: text/plain; charset="UTF-8" Sender: linux-arm-msm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org On Tue, Sep 10, 2019 at 8:01 AM Robin Murphy wrote: > > On 07/09/2019 18:50, Rob Clark wrote: > > From: Rob Clark > > > > When games, browser, or anything using a lot of GPU buffers exits, there > > can be many hundreds or thousands of buffers to unmap and free. If the > > GPU is otherwise suspended, this can cause arm-smmu to resume/suspend > > for each buffer, resulting 5-10 seconds worth of reprogramming the > > context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). > > To the user it would appear that the system is locked up. > > > > A simple solution is to use pm_runtime_put_autosuspend() instead, so we > > don't immediately suspend the SMMU device. > > > > Signed-off-by: Rob Clark > > --- > > Note: I've tied the autosuspend enable/delay to the consumer device, > > based on the reasoning that if the consumer device benefits from using > > an autosuspend delay, then it's corresponding SMMU probably does too. > > Maybe that is overkill and we should just unconditionally enable > > autosuspend. > > I'm not sure there's really any reason to expect that a supplier's usage > model when doing things for itself bears any relation to that of its > consumer(s), so I'd certainly lean towards the "unconditional" argument > myself. Sounds good, I'll respin w/ unconditional autosuspend > Of course ideally we'd skip resuming altogether in the map/unmap paths > (since resume implies a full TLB reset anyway), but IIRC that approach > started to get messy in the context of the initial RPM patchset. I'm > planning to fiddle around a bit more to clean up the implementation of > the new iommu_flush_ops stuff, so I've made a note to myself to revisit > RPM to see if there's a sufficiently clean way to do better. In the > meantime, though, I don't have any real objection to using some > reasonable autosuspend delay on the principle that if we've been woken > up to map/unmap one page, there's a high likelihood that more will > follow in short order (and in the configuration slow-paths it won't have > much impact either way). It does sort of remind me about something I was chatting with Jordan the other day.. about how we could possibly skip the TLB inv for unmaps from non-current pagetables once we have per-context pagetables. The challenge is, since the GPU's command parser is the one switching pagetables, we don't have any race-free way to know which pagetables are current. But we do know which contexts have work queued up for the GPU, so we can know either that a given context definitely isn't current, or that it might be current. And in the "definitely not current" case we could skip TLB inv. BR, -R > > Robin. > > > drivers/iommu/arm-smmu.c | 11 ++++++++++- > > 1 file changed, 10 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > index c2733b447d9c..73a0dd53c8a3 100644 > > --- a/drivers/iommu/arm-smmu.c > > +++ b/drivers/iommu/arm-smmu.c > > @@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu) > > static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu) > > { > > if (pm_runtime_enabled(smmu->dev)) > > - pm_runtime_put(smmu->dev); > > + pm_runtime_put_autosuspend(smmu->dev); > > } > > > > static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) > > @@ -1445,6 +1445,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > > /* Looks ok, so add the device to the domain */ > > ret = arm_smmu_domain_add_master(smmu_domain, fwspec); > > > > +#ifdef CONFIG_PM > > + /* TODO maybe device_link_add() should do this for us? */ > > + if (dev->power.use_autosuspend) { > > + pm_runtime_set_autosuspend_delay(smmu->dev, > > + dev->power.autosuspend_delay); > > + pm_runtime_use_autosuspend(smmu->dev); > > + } > > +#endif > > + > > rpm_put: > > arm_smmu_rpm_put(smmu); > > return ret; > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41F07C5AE59 for ; Tue, 10 Sep 2019 15:45:42 +0000 (UTC) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0C3712089F for ; Tue, 10 Sep 2019 15:45:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FUDx1Vss" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C3712089F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id CC1D3D83; Tue, 10 Sep 2019 15:45:41 +0000 (UTC) Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 44A6ED7E for ; Tue, 10 Sep 2019 15:45:41 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f66.google.com (mail-ed1-f66.google.com [209.85.208.66]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id D2C0281A for ; Tue, 10 Sep 2019 15:45:39 +0000 (UTC) Received: by mail-ed1-f66.google.com with SMTP id y91so17555642ede.9 for ; Tue, 10 Sep 2019 08:45:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ab44L4UsH9lWBk/ePs+t6LQUuKmRHFW0SwCWgf68+rc=; b=FUDx1VssMY5+grYTJ9mTO+OtexdcbSTMSsn3//VPNV0zPdSp95pA3zNOdpx4boIzLL +8wfwTnUnSWaD0DyKayLxvtLd1DNtf9KpsXdscr1lunnkyj9RgO10ZaIY4qDgscT06xv gsu7UNxm0Xryuv9rrw5a4GpIEVPRKWhPVPWKzBpbTQx6lkHw+mIFYb+srXGudTMQJCt5 gonRN5rE57R/vyk+7sS8MiPSdeYzF3ZPo7wNMrmBj3lVkr6zixiEP6LKYNeZGu9gq5mv nhhDZVq5ROoxeNpfmQu0BO+jdHGVOWyuw0NkkZqryUCoeygfaKH+6X5QUf5q5jC92gdh i8nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ab44L4UsH9lWBk/ePs+t6LQUuKmRHFW0SwCWgf68+rc=; b=bHKbQw5MbSu8p6jVdhsEfFc6kV5n07BK4IZx7d4gHnsWsfogpau9EPUIESPkmaS2wC zlKBkvySb1YslJVG6kfERHk+bjr6DUAHs1CK4m2gs9FzVXd++XkkcldAeg77LgaUUNCe kNFv8e2mkzucDqh+aD9EgGiC2KsCO3EbXDTlgGc7FiXSxrlOizMbQAc/RR82DiuqraqB c+dTUeN7EVvFf8Qep8IcW8ijfkFoLSmy1V5zUXK9e5p7yk0M4oRwlO+fRr2zUs9FsZEq bTs/Nx+KIIZ/Q9/o8zYIKs5YYOgrXmI+NfIX4NVZJH5ghwTqBxmcvijTNL9Cn45cxwn2 3g5Q== X-Gm-Message-State: APjAAAUuVe5RU+8rKr/IZNgy+fptoFi9v3oauAxbPSWcRlLmT/aoyFrI BXnMgfoxPkYIgFY2CdUaCtLxjBpx8z/GO7VyPEI= X-Google-Smtp-Source: APXvYqzoKSvtwc7h3YF+c3ClC8RRDwvQEOuO9vmMklmmZ6/J8nMaSUMmvks3o2VNkPh4WYQkviIc241SjZs7jn1Ep9c= X-Received: by 2002:a17:906:2451:: with SMTP id a17mr16927007ejb.164.1568130338310; Tue, 10 Sep 2019 08:45:38 -0700 (PDT) MIME-Version: 1.0 References: <20190907175013.24246-1-robdclark@gmail.com> <418d8426-f299-1269-2b2e-f86677cf22c2@arm.com> In-Reply-To: <418d8426-f299-1269-2b2e-f86677cf22c2@arm.com> From: Rob Clark Date: Tue, 10 Sep 2019 08:45:27 -0700 Message-ID: Subject: Re: [PATCH] iommu/arm-smmu: fix "hang" when games exit To: Robin Murphy Cc: Rob Clark , Will Deacon , linux-arm-msm , open list , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , freedreno , "moderated list:ARM SMMU DRIVERS" X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: iommu-bounces@lists.linux-foundation.org Errors-To: iommu-bounces@lists.linux-foundation.org On Tue, Sep 10, 2019 at 8:01 AM Robin Murphy wrote: > > On 07/09/2019 18:50, Rob Clark wrote: > > From: Rob Clark > > > > When games, browser, or anything using a lot of GPU buffers exits, there > > can be many hundreds or thousands of buffers to unmap and free. If the > > GPU is otherwise suspended, this can cause arm-smmu to resume/suspend > > for each buffer, resulting 5-10 seconds worth of reprogramming the > > context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). > > To the user it would appear that the system is locked up. > > > > A simple solution is to use pm_runtime_put_autosuspend() instead, so we > > don't immediately suspend the SMMU device. > > > > Signed-off-by: Rob Clark > > --- > > Note: I've tied the autosuspend enable/delay to the consumer device, > > based on the reasoning that if the consumer device benefits from using > > an autosuspend delay, then it's corresponding SMMU probably does too. > > Maybe that is overkill and we should just unconditionally enable > > autosuspend. > > I'm not sure there's really any reason to expect that a supplier's usage > model when doing things for itself bears any relation to that of its > consumer(s), so I'd certainly lean towards the "unconditional" argument > myself. Sounds good, I'll respin w/ unconditional autosuspend > Of course ideally we'd skip resuming altogether in the map/unmap paths > (since resume implies a full TLB reset anyway), but IIRC that approach > started to get messy in the context of the initial RPM patchset. I'm > planning to fiddle around a bit more to clean up the implementation of > the new iommu_flush_ops stuff, so I've made a note to myself to revisit > RPM to see if there's a sufficiently clean way to do better. In the > meantime, though, I don't have any real objection to using some > reasonable autosuspend delay on the principle that if we've been woken > up to map/unmap one page, there's a high likelihood that more will > follow in short order (and in the configuration slow-paths it won't have > much impact either way). It does sort of remind me about something I was chatting with Jordan the other day.. about how we could possibly skip the TLB inv for unmaps from non-current pagetables once we have per-context pagetables. The challenge is, since the GPU's command parser is the one switching pagetables, we don't have any race-free way to know which pagetables are current. But we do know which contexts have work queued up for the GPU, so we can know either that a given context definitely isn't current, or that it might be current. And in the "definitely not current" case we could skip TLB inv. BR, -R > > Robin. > > > drivers/iommu/arm-smmu.c | 11 ++++++++++- > > 1 file changed, 10 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > index c2733b447d9c..73a0dd53c8a3 100644 > > --- a/drivers/iommu/arm-smmu.c > > +++ b/drivers/iommu/arm-smmu.c > > @@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu) > > static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu) > > { > > if (pm_runtime_enabled(smmu->dev)) > > - pm_runtime_put(smmu->dev); > > + pm_runtime_put_autosuspend(smmu->dev); > > } > > > > static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) > > @@ -1445,6 +1445,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > > /* Looks ok, so add the device to the domain */ > > ret = arm_smmu_domain_add_master(smmu_domain, fwspec); > > > > +#ifdef CONFIG_PM > > + /* TODO maybe device_link_add() should do this for us? */ > > + if (dev->power.use_autosuspend) { > > + pm_runtime_set_autosuspend_delay(smmu->dev, > > + dev->power.autosuspend_delay); > > + pm_runtime_use_autosuspend(smmu->dev); > > + } > > +#endif > > + > > rpm_put: > > arm_smmu_rpm_put(smmu); > > return ret; > > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59595C49ED7 for ; Tue, 10 Sep 2019 15:45:51 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2C29521670 for ; Tue, 10 Sep 2019 15:45:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="mqKubZ4H"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FUDx1Vss" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2C29521670 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4XJcQ2gqDQsjXiLUTIfXjJIvHKwOsNAeEjt3NxEhkDw=; b=mqKubZ4H+nzVbp mKa9a/79qHi37xwekN4kv9beo9kfufTEPzbyDxdG6ad6ZKrPlYBK6/IP4KCCDkB27IFllRbhUDeZ7 O9eTXe6dNpUOgM8chMqYGu/KmcMLxvGovh4yB7GP0vWnMrR3zVgXtQ8U0zj5jzf9felwQH2E5/Oi1 pIpEaBuRrQ3U+IIwi3AzKpIDlDEELrFo/GvHe+g9meY5s1QkIBMoJgB1WGYkgngYQc3BSIBJ/K2dH BAWFM12n5TC/cHOxU6K+NuqSfjJ4DXqbBU/SsEH+JWm4KZSraMGPkXTMaY3gVXRQ/4hTF2M/2IZip FbHbC+r86nTeNQRGsZwQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1i7iKu-0005Af-I3; Tue, 10 Sep 2019 15:45:44 +0000 Received: from mail-ed1-x544.google.com ([2a00:1450:4864:20::544]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1i7iKq-00059Z-FL for linux-arm-kernel@lists.infradead.org; Tue, 10 Sep 2019 15:45:43 +0000 Received: by mail-ed1-x544.google.com with SMTP id c20so8556272eds.1 for ; Tue, 10 Sep 2019 08:45:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ab44L4UsH9lWBk/ePs+t6LQUuKmRHFW0SwCWgf68+rc=; b=FUDx1VssMY5+grYTJ9mTO+OtexdcbSTMSsn3//VPNV0zPdSp95pA3zNOdpx4boIzLL +8wfwTnUnSWaD0DyKayLxvtLd1DNtf9KpsXdscr1lunnkyj9RgO10ZaIY4qDgscT06xv gsu7UNxm0Xryuv9rrw5a4GpIEVPRKWhPVPWKzBpbTQx6lkHw+mIFYb+srXGudTMQJCt5 gonRN5rE57R/vyk+7sS8MiPSdeYzF3ZPo7wNMrmBj3lVkr6zixiEP6LKYNeZGu9gq5mv nhhDZVq5ROoxeNpfmQu0BO+jdHGVOWyuw0NkkZqryUCoeygfaKH+6X5QUf5q5jC92gdh i8nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ab44L4UsH9lWBk/ePs+t6LQUuKmRHFW0SwCWgf68+rc=; b=DOSE/yNd3XQlr2necz2/ZptWFBfLLcb9yfokvHGCs8p0NynWfq+2AuqUaqCBYKsIoU urn91lnfPe83pqalyPNfVctyUx60zJBezXbqPIVoOgDB4MFj5qumBE7GxGRa9+FwByx4 cMV6tnDJZNLkltNa9QhnqphNW5TATM4x7kMz+SGxmjry0y/uoG7oq+Pozdt3W4mRhbsK 3aJ+Pe24grWs8cexqZA8Mod+bfvvcRP8zEhnMmNPO57OwccEnVQs6Ba4S1v5CVNIHlqa NAlgWW+wzxoH28TSOeWVfnPAM23HpWBde6PKBQNIx88NpQBdiwHLK/wEbxifo63on2f5 lpOA== X-Gm-Message-State: APjAAAWxHMUEQdPJdD5NXUBSFobtdeH1hSy0Bc7PGLXsBFOgbTSzSEYz Ib/XrbTuW55/8F9/LHWU5wBmxVdYJFy/68Su1QY= X-Google-Smtp-Source: APXvYqzoKSvtwc7h3YF+c3ClC8RRDwvQEOuO9vmMklmmZ6/J8nMaSUMmvks3o2VNkPh4WYQkviIc241SjZs7jn1Ep9c= X-Received: by 2002:a17:906:2451:: with SMTP id a17mr16927007ejb.164.1568130338310; Tue, 10 Sep 2019 08:45:38 -0700 (PDT) MIME-Version: 1.0 References: <20190907175013.24246-1-robdclark@gmail.com> <418d8426-f299-1269-2b2e-f86677cf22c2@arm.com> In-Reply-To: <418d8426-f299-1269-2b2e-f86677cf22c2@arm.com> From: Rob Clark Date: Tue, 10 Sep 2019 08:45:27 -0700 Message-ID: Subject: Re: [PATCH] iommu/arm-smmu: fix "hang" when games exit To: Robin Murphy X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190910_084540_521747_0B33C6C8 X-CRM114-Status: GOOD ( 30.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Clark , Jordan Crouse , Will Deacon , linux-arm-msm , Joerg Roedel , open list , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , freedreno , "moderated list:ARM SMMU DRIVERS" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Sep 10, 2019 at 8:01 AM Robin Murphy wrote: > > On 07/09/2019 18:50, Rob Clark wrote: > > From: Rob Clark > > > > When games, browser, or anything using a lot of GPU buffers exits, there > > can be many hundreds or thousands of buffers to unmap and free. If the > > GPU is otherwise suspended, this can cause arm-smmu to resume/suspend > > for each buffer, resulting 5-10 seconds worth of reprogramming the > > context bank (arm_smmu_write_context_bank()/arm_smmu_write_s2cr()/etc). > > To the user it would appear that the system is locked up. > > > > A simple solution is to use pm_runtime_put_autosuspend() instead, so we > > don't immediately suspend the SMMU device. > > > > Signed-off-by: Rob Clark > > --- > > Note: I've tied the autosuspend enable/delay to the consumer device, > > based on the reasoning that if the consumer device benefits from using > > an autosuspend delay, then it's corresponding SMMU probably does too. > > Maybe that is overkill and we should just unconditionally enable > > autosuspend. > > I'm not sure there's really any reason to expect that a supplier's usage > model when doing things for itself bears any relation to that of its > consumer(s), so I'd certainly lean towards the "unconditional" argument > myself. Sounds good, I'll respin w/ unconditional autosuspend > Of course ideally we'd skip resuming altogether in the map/unmap paths > (since resume implies a full TLB reset anyway), but IIRC that approach > started to get messy in the context of the initial RPM patchset. I'm > planning to fiddle around a bit more to clean up the implementation of > the new iommu_flush_ops stuff, so I've made a note to myself to revisit > RPM to see if there's a sufficiently clean way to do better. In the > meantime, though, I don't have any real objection to using some > reasonable autosuspend delay on the principle that if we've been woken > up to map/unmap one page, there's a high likelihood that more will > follow in short order (and in the configuration slow-paths it won't have > much impact either way). It does sort of remind me about something I was chatting with Jordan the other day.. about how we could possibly skip the TLB inv for unmaps from non-current pagetables once we have per-context pagetables. The challenge is, since the GPU's command parser is the one switching pagetables, we don't have any race-free way to know which pagetables are current. But we do know which contexts have work queued up for the GPU, so we can know either that a given context definitely isn't current, or that it might be current. And in the "definitely not current" case we could skip TLB inv. BR, -R > > Robin. > > > drivers/iommu/arm-smmu.c | 11 ++++++++++- > > 1 file changed, 10 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > > index c2733b447d9c..73a0dd53c8a3 100644 > > --- a/drivers/iommu/arm-smmu.c > > +++ b/drivers/iommu/arm-smmu.c > > @@ -289,7 +289,7 @@ static inline int arm_smmu_rpm_get(struct arm_smmu_device *smmu) > > static inline void arm_smmu_rpm_put(struct arm_smmu_device *smmu) > > { > > if (pm_runtime_enabled(smmu->dev)) > > - pm_runtime_put(smmu->dev); > > + pm_runtime_put_autosuspend(smmu->dev); > > } > > > > static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) > > @@ -1445,6 +1445,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > > /* Looks ok, so add the device to the domain */ > > ret = arm_smmu_domain_add_master(smmu_domain, fwspec); > > > > +#ifdef CONFIG_PM > > + /* TODO maybe device_link_add() should do this for us? */ > > + if (dev->power.use_autosuspend) { > > + pm_runtime_set_autosuspend_delay(smmu->dev, > > + dev->power.autosuspend_delay); > > + pm_runtime_use_autosuspend(smmu->dev); > > + } > > +#endif > > + > > rpm_put: > > arm_smmu_rpm_put(smmu); > > return ret; > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel