From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.1 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44774C433E1 for ; Wed, 22 Jul 2020 23:37:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1129820825 for ; Wed, 22 Jul 2020 23:37:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="A1cijpiU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732981AbgGVXhp (ORCPT ); Wed, 22 Jul 2020 19:37:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726685AbgGVXho (ORCPT ); Wed, 22 Jul 2020 19:37:44 -0400 Received: from mail-qk1-x742.google.com (mail-qk1-x742.google.com [IPv6:2607:f8b0:4864:20::742]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2490EC0619DC for ; Wed, 22 Jul 2020 16:37:44 -0700 (PDT) Received: by mail-qk1-x742.google.com with SMTP id e13so3769506qkg.5 for ; Wed, 22 Jul 2020 16:37:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:subject:from:to:cc:date:in-reply-to:references :organization:user-agent:mime-version:content-transfer-encoding; bh=H8BbmRX160xyQeVFFIdlPVklsyPOWgbirdL1MMhs52A=; b=A1cijpiUtfu4FxFuW41K7F27e5icdy4NAmOps6rIUBiNTdyAQ/pQhZjIiCkL8iJNXx J1rdf5hKePDxGJjUR6xQSsJ7sp36kRtPuxWXMNykZmOkBkjyw2fbJZU1Jj9ObC4VFfB9 rZJRTbAw2TPSSkgLCzqohsZ2QG9TWwJN+soaTTpJ5kiZ5BEIbhCCXdo5PhkVOhSTHlH6 x9/fLgicxrPczXatLBCAu/sfeC8sqvKv2y/OefXQeGQWsaf/IYt4vB3kzIOf8gwqRTII xwpHG9uIqhkr5jRWh6u0tj02tUqRsEnrKe23/Eyxrzk9QoArDOjs+T5VlaYua7a8Hpk7 l1Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:organization:user-agent:mime-version :content-transfer-encoding; bh=H8BbmRX160xyQeVFFIdlPVklsyPOWgbirdL1MMhs52A=; b=NUCW6rr2jRkbFcfsKhQn9Kj7cYyBOjs3ikijKmOsyQxFEv2/ogNhmR/kMmtRenFkL7 t+LXMp5MJ9cy8fl4bTYuGIkXFL3DTs8kI4ZWAPtOxVYHwujU1ugtwkgtvOEyCA7Yu8RP dcIrp+297gfN9pYGZW7P/EGfmJFkxRJ9016/GQFXqONQt0kVCBJZLdpoaNQLwGR23UBu YHasVoq6k3SBm0osBrKCsruC3IgOnVgr3Z7BHXd9e4tURyy2tvRaOIiSK4OPqOqpCXRE uIF+hk3e/UmH30ol+XLbbSy+zJlb3fi2G2IwbEhE7oShVHJ6BBBKFxKAYy5mN//cCLWq NGlw== X-Gm-Message-State: AOAM5330np89IRMy/vBqRxAQQiA50GNvnV+OKMiQd5zvZ7PjWtN3Yx/I 3+qdqUJ+anIIQ/QODfTf5nfk+6ow X-Google-Smtp-Source: ABdhPJxUn3wK41CBa+sKu5oWixw9j3f8UMBn8EM27juDma3YS+Ar9XmjwiziY9s5nsJoVr1BNeg/Uw== X-Received: by 2002:a37:4048:: with SMTP id n69mr2386577qka.421.1595461063336; Wed, 22 Jul 2020 16:37:43 -0700 (PDT) Received: from LeoBras (179-125-153-225.dynamic.desktop.com.br. [179.125.153.225]) by smtp.gmail.com with ESMTPSA id c70sm1126936qke.109.2020.07.22.16.37.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2020 16:37:42 -0700 (PDT) Message-ID: <396d343962bb4e3b845bab4f46721a5ebed39cf6.camel@gmail.com> Subject: Re: [PATCH v4 5/7] powerpc/iommu: Move iommu_table cleaning routine to iommu_table_clean From: Leonardo Bras To: Alexey Kardashevskiy , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Joel Stanley , Christophe Leroy , Thiago Jung Bauermann , Ram Pai , Brian King Cc: linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org Date: Wed, 22 Jul 2020 20:37:36 -0300 In-Reply-To: References: <20200716071658.467820-1-leobras.c@gmail.com> <20200716071658.467820-6-leobras.c@gmail.com> <51235292-a571-8792-c693-d0dc6faeb21c@ozlabs.ru> <0f4c2d84d0958e98e7ada53c25750fe548cadf0b.camel@gmail.com> Organization: IBM Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.4 (3.34.4-1.fc31) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 2020-07-22 at 11:28 +1000, Alexey Kardashevskiy wrote: > > On 22/07/2020 08:13, Leonardo Bras wrote: > > On Tue, 2020-07-21 at 14:59 +1000, Alexey Kardashevskiy wrote: > > > On 16/07/2020 17:16, Leonardo Bras wrote: > > > > Move the part of iommu_table_free() that does struct iommu_table cleaning > > > > into iommu_table_clean, so we can invoke it separately. > > > > > > > > This new function is useful for cleaning struct iommu_table before > > > > initializing it again with a new DMA window, without having it freed and > > > > allocated again. > > > > > > > > Signed-off-by: Leonardo Bras > > > > --- > > > > arch/powerpc/kernel/iommu.c | 30 ++++++++++++++++++------------ > > > > 1 file changed, 18 insertions(+), 12 deletions(-) > > > > > > > > diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c > > > > index 9704f3f76e63..c3242253a4e7 100644 > > > > --- a/arch/powerpc/kernel/iommu.c > > > > +++ b/arch/powerpc/kernel/iommu.c > > > > @@ -735,21 +735,10 @@ struct iommu_table *iommu_init_table(struct iommu_table *tbl, int nid, > > > > return tbl; > > > > } > > > > > > > > -static void iommu_table_free(struct kref *kref) > > > > +static void iommu_table_clean(struct iommu_table *tbl) > > > > > > iommu_table_free() + iommu_init_table() + set_iommu_table_base() should > > > work too, why new helper? > > > > iommu_table_free() also frees the tbl, which would need allocate it > > again (new address) and to fill it up again, unnecessarily. > > It is a new table in fact, everything is new there. You are only saving > kfree+kzalloc which does not seem a huge win. > > Also, iommu_table_update() simply assumes 64bit window by passing > res_start=res_end=0 to iommu_init_table() which is not horribly robust > either. Yeah, I know, iommu_init_table() is always called with zeroes in > pseries but this is somewhat ok as those tables are from the device tree > and those windows don't overlap with 32bit MMIO but under KVM they will > (well, if we hack QEMU to advertise a single window). > > I suggest removing iommu_pseries_table_update() from 6/7 and do > iommu_table_free() + iommu_init_table() + set_iommu_table_base() with a > WARN_ON(pdev->dev.archdata.dma_offset>=SZ_4G), may be even do this all > in enable_ddw() where we know for sure if it is 1:1 mapping or just a > big window. Sure, I have yet to understand the full impact of this change, but I will implement this and give it a try. > > Out of curiosity - what page sizes does pHyp advertise in "query"? 64kB (page shift 0x10) > > > > I think it's a better approach to only change what is needed. > > > > > There is also iommu_table_clear() which does a different thing so you > > > need a better name. > > > > I agree. > > I had not noticed this other function before sending the patchset. What > > would be a better name though? __iommu_table_free()? > > > > > Second, iommu_table_free > > > use and it would be ok as we would only see this when hot-unplugging a > > > PE because we always kept the default window. > > > Btw you must be seeing these warnings now every time you create DDW with > > > these patches as at least the first page is reserved, do not you? > > > > It does not print a warning. > > I noticed other warnings, > > And what are these? tce_freemulti_pSeriesLP: plpar_tce_stuff failed [...] It's regarding the change in pagesize. Some places have the tceshift hardcoded as 12, tce_freemulti_pSeriesLP is one of them, and that is causing some errors. I wrote a patch fixing this, and I will include it in the next series. > > > but not this one from iommu_table_free(): > > /* verify that table contains no entries */ > > if (!bitmap_empty(tbl->it_ma > > p, tbl->it_size)) > > pr_warn("%s: Unexpected TCEs\n", __func__); > > > > Before that, iommu_table_release_pages(tbl) is supposed to clear the > > bitmap, so this only tests for a tce that is created in this short period. > > iommu_table_release_pages() only clears reserved pages - page 0 (just a > protection against NULL DMA pointers) and 32bit MMIO (these should not > be set for 64bit window). The "%s: Unexpected TCEs\n" is what checks for > actual mapped TCEs. > Oh, I haven't noticed that. Thanks for pointing! > > > Since we are replacing a table for a device which is still in the > > > system, we should not try messing with its DMA if it already has > > > mappings so the warning should become an error preventing DDW. It is > > > rather hard to trigger in practice but I could hack a driver to ask for > > > 32bit DMA mask first, map few pages and then ask for 64bit DMA mask, it > > > is not illegal, I think. So this needs a new helper - "bool > > > iommu_table_in_use(tbl)" - to use in enable_ddw(). Or I am overthinking > > > this?... Thanks, > > > > As of today, there seems to be nothing like that happening in the > > driver I am testing. > > I spoke to Brian King on slack, and he mentioned that at the point DDW > > is created there should be no allocations in place. > > Correct, there should not be. But it is also not a huge effort to fall > back if there are. True. > > > But I suppose some driver could try to do this. > > > > Maybe a better approach would be removing the mapping only if the > > default window is removed (at the end of enable_ddw, as an else to > > resetting the default DMA window), and having a way to add more > > mappings to those pools. But this last part doesn't look so simple, and > > it would be better to understand if it's necessary investing work in > > this. > > > > What do you think? > > Add iommu_table_in_use(tbl) and fail DDW if that says "yes". Seems good, I will include that on the next patchset. Still, I will try to implement that more complex approach in a future patchset, as it may come to be useful. Thank you for the feedback!