From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C76A3C4360C for ; Wed, 3 Mar 2021 16:09:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9AE1F64EE9 for ; Wed, 3 Mar 2021 16:09:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1384665AbhCCQIo (ORCPT ); Wed, 3 Mar 2021 11:08:44 -0500 Received: from mout.kundenserver.de ([212.227.126.131]:35417 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357955AbhCCLh3 (ORCPT ); Wed, 3 Mar 2021 06:37:29 -0500 Received: from mail-oo1-f53.google.com ([209.85.161.53]) by mrelayeu.kundenserver.de (mreue010 [213.165.67.97]) with ESMTPSA (Nemesis) id 1MhDIw-1llmG30Z06-00eLjx; Wed, 03 Mar 2021 09:44:25 +0100 Received: by mail-oo1-f53.google.com with SMTP id n19so5480150ooj.11; Wed, 03 Mar 2021 00:44:24 -0800 (PST) X-Gm-Message-State: AOAM531I5RlSxC1fMxpxwzkcsPAJpnaJFMGKoUsV2NBkaCYTCh1InN3k qgP85DrGx8q+yk/Tr4YR5dK1btjxZ7onBt2pcZY= X-Google-Smtp-Source: ABdhPJx+y1mutrjVZMV70H7WGHtEMsbqS/b6Pacx4upesFv3Pu2rH/lMvNVNl6MT1UBcioMptZHIJHVRvTvRXobxOCo= X-Received: by 2002:a4a:304a:: with SMTP id z10mr20492940ooz.26.1614761063729; Wed, 03 Mar 2021 00:44:23 -0800 (PST) MIME-Version: 1.0 References: <20210226140305.26356-1-nsaenzjulienne@suse.de> <20210226140305.26356-10-nsaenzjulienne@suse.de> <3a4cf13f-c098-9ff3-6c0e-2c94daae452b@arm.com> In-Reply-To: <3a4cf13f-c098-9ff3-6c0e-2c94daae452b@arm.com> From: Arnd Bergmann Date: Wed, 3 Mar 2021 09:44:07 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC 09/13] iommu/arm-smmu: Make use of dev_64bit_mmio_supported() To: Robin Murphy Cc: Nicolas Saenz Julienne , Linux ARM , DTML , "linux-kernel@vger.kernel.org" , Florian Fainelli , Rob Herring , Ard Biesheuvel , Christoph Hellwig , Neil Armstrong , David Woodhouse , Russell King - ARM Linux , Catalin Marinas , Will Deacon Content-Type: text/plain; charset="UTF-8" X-Provags-ID: V03:K1:7g5UKd+vyI/Ln8q53XNHCBMONOXbq19hSutGtG8ohJvxzZ6ytkq kDKPGw9czCqxBsRjWnuuxYuw9k2WHUYh7nCLZ9ZBl0Ewfxl7F8JE+wKMEALu8nxlviKb5TS KBn1bzpxt9sMFEgJXzAhY0L7UaooMx4V2fmEWKLyYjfIvgS3p4wfZ5TiYTmektItVml2YY+ ToZTI/taz2+ShGdyjwbTg== X-UI-Out-Filterresults: notjunk:1;V03:K0:kZG0h7I6VUE=:ssp5mlwCOJ1kjmmq4zkPlT 0gWoWVf4Gp4PKTzCrocK4x2LN239iQqXeSH+jY62zGYznlhXP6bQ/yBQbYDOkFbMYWwOUpRom vtPoyCEuMXm2NFWn0URAtVN+Kfdjq2Ia1v1yfQNdJfNhQZ0uNSj4PC9k7nKs//X97rgTBzQUO abzvvQkJrFnTK7bvrivTiMxiaseLAB1WYCeO26vtXQ3VXDyYdOcTeUsfbYxbeD7IiAQ9WyX2R UoulpO+iJrXVTR7rjlHV43XyuhP/FeFHhjPDrsHnktpNXcQQBhHMM0r6Nm+vnZD2zYJ6uY1rm aZ+hXa2qzNoAK9RIJPsI+LKg+UK2cRaVc6UAOlLvcGWGTbDwc+xm+tnGwm7IU0Mo+WcHjnF/B TcsWeqdsqacoJz9oR3dX4U/yGzFy2Xn/pv+tcuqMBYi9GjKoWIuaXaE5cyZ4xtGESpAILNej2 GB+G9LY9pMbjsbsutKT6slvzywCsxq0QFj6y6aohdCXXAe8uiht6WeaIdLM6zIIhQlzlV51e0 Ph7h6MymUqoWz+kW3jI2JNZbDTMICYVrG99LRxkSxuP Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 2, 2021 at 12:07 PM Robin Murphy wrote: > On 2021-02-26 14:03, Nicolas Saenz Julienne wrote: > > index d2a2d1bc58ba..997d13a21717 100644 > > --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h > > +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h > > @@ -477,15 +477,20 @@ static inline void arm_smmu_writel(struct arm_smmu_device *smmu, int page, > > { > > if (smmu->impl && unlikely(smmu->impl->write_reg)) > > smmu->impl->write_reg(smmu, page, offset, val); > > - else > > + else if (dev_64bit_mmio_supported(smmu->dev)) > > writel_relaxed(val, arm_smmu_page(smmu, page) + offset); > > + else > > + hi_lo_writeq_relaxed(val, arm_smmu_page(smmu, page) + offset); > > As Arnd pointed out, this is in completely the wrong place. Also, in > general it doesn't work if the implementation already needs a hook to > filter or override register accesses for any other reason. TBH I'm not > convinced that this isn't *more* of a mess than handling it on a > SoC-specific basis... I think the main problem for handling it in a SoC specific way is that there is no device-independent way to do a 64-bit store as two 32-bit stores: - some devices need hi_lo_writeq_relaxed(), others need lo_hi_writeq_relaxed(), and some absolutely require 64-bit stores and cannot work at all behind a broken PCI bus. - if the driver requires the store to be atomic, it needs to use a spinlock around the two writel(), but if the register is on a PCI bus or mapped with page attributes that allow posted writes (like arm64 ioremap), then you may need to read back the register before spin_unlock to serialize them properly. However, reading back an mmio register is slow and can have side-effects, so you can't put that in driver-independent code either. Arnd From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0B29C433E0 for ; Wed, 3 Mar 2021 17:46:55 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C8B8864ED7 for ; Wed, 3 Mar 2021 17:46:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C8B8864ED7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arndb.de Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kb8jNDIUAPKREG/qrTS3rBnbwI+XfkNYT0PnOdBdx6g=; b=WunkCsQHmtmCeLt4Z04nTR6fq IuQYkrEIOKIMRvfb6x4hej3GfCmc3F0bflFxyzR/t/PpnXRaW2rYAZRha+c4EBpSFgRsmMNProiwK sdIKRFnVfN2MUOYlIPwJonFtenqRrU5btv1cFrP2nuCTFZKBiqx1XxnWQG4ACQVBjVdTUKvfB4mLS ANe/0s7oGJ25aS0N9X6DMSo3rWHbyXKGhRDmHDhyRuX27o1UR7KIoKlciuIoBSg7XVzdAYwv9gMeN zhA149mRzGgUFAeOYsEonqafw3lZfQ4BWx0QHR6RW9z9MfcK5uGRY8FwbJNAS7LxRkxVvc3BDfvAC tkvoWywfw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHVXp-005rIJ-3i; Wed, 03 Mar 2021 17:44:21 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSwF-005Fle-4v for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:57:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:Subject:Message-ID: Date:From:In-Reply-To:References:MIME-Version:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=QHqEY/vEXCOh2goEyN+RFDbiez4INEuZ++/yjwiwrhY=; b=OvYE2FpCu6YILUPELwpH68sbgm 6f7qPKl+rR9z70wV6IHeIQUeFaH4fQQEXoE6foeZtQnTEOZatdW10MzJY0Z98iSxb16NDEzDJG3UG rL0xGQp+a8IluEo0tyuQohPpftS0cFOGQ9H6ftyJpfaZBKc1vnAPqPWyBBLjLFcG+kkisH83v7F2x U2ZHWh2x6aFQntVX2F1tUSfJxfbtrC66Psv8GHQRukm8g4Z2licPFJ89Mar4Op4gmnU/fNwJdACbh vUxgTN5oZssaBOO7RNRgVQyc7OSgfl21mZWDwCEqxR0WqVtmqbdbMf7Un8tR+Y72t41KvH6bUNz/1 i9MR6KLQ==; Received: from mout.kundenserver.de ([212.227.17.24]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHN7a-001syh-Vj for linux-arm-kernel@lists.infradead.org; Wed, 03 Mar 2021 08:44:44 +0000 Received: from mail-oo1-f46.google.com ([209.85.161.46]) by mrelayeu.kundenserver.de (mreue107 [213.165.67.113]) with ESMTPSA (Nemesis) id 1MlO5j-1leM7i3fPJ-00linD for ; Wed, 03 Mar 2021 09:44:28 +0100 Received: by mail-oo1-f46.google.com with SMTP id z22so5493777oop.0 for ; Wed, 03 Mar 2021 00:44:24 -0800 (PST) X-Gm-Message-State: AOAM532LmROD2vijIXRZuiEU1Li+JsEhdkAb6smJTxfiQWuj7c1F8bTF 74F/rErlW4D9vS+ZG7kLtVzzyt6qEWp3WntgmJQ= X-Google-Smtp-Source: ABdhPJx+y1mutrjVZMV70H7WGHtEMsbqS/b6Pacx4upesFv3Pu2rH/lMvNVNl6MT1UBcioMptZHIJHVRvTvRXobxOCo= X-Received: by 2002:a4a:304a:: with SMTP id z10mr20492940ooz.26.1614761063729; Wed, 03 Mar 2021 00:44:23 -0800 (PST) MIME-Version: 1.0 References: <20210226140305.26356-1-nsaenzjulienne@suse.de> <20210226140305.26356-10-nsaenzjulienne@suse.de> <3a4cf13f-c098-9ff3-6c0e-2c94daae452b@arm.com> In-Reply-To: <3a4cf13f-c098-9ff3-6c0e-2c94daae452b@arm.com> From: Arnd Bergmann Date: Wed, 3 Mar 2021 09:44:07 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC 09/13] iommu/arm-smmu: Make use of dev_64bit_mmio_supported() To: Robin Murphy Cc: Nicolas Saenz Julienne , Linux ARM , DTML , "linux-kernel@vger.kernel.org" , Florian Fainelli , Rob Herring , Ard Biesheuvel , Christoph Hellwig , Neil Armstrong , David Woodhouse , Russell King - ARM Linux , Catalin Marinas , Will Deacon X-Provags-ID: V03:K1:ckVYriFixZZZRQdsfhNQ9QVY8v7lPdpBHfZIIJPPvEyriZ3Nhu8 +QUhe6nBOBO+wGyPTxoFJSe4Srd/0CPqkkPoY9y9Cs5tqFoo7FvKuyNH3liLvNt75XZMLYU qLPQGnFujzD+XJqmO/i3Mk1obdgEaO97UIYVBzHOdi8uIJgUw0EvDu0I5nL6ZI8MGXTnxfb BzxyIRXZ8LiEfj/fcZhBQ== X-UI-Out-Filterresults: notjunk:1;V03:K0:+UPp+chjO3c=:2u95FqsmVx2sHab8opyjvU XKcjJdoEOg+wDcQDiy1A22JNyOrTpR9WdW6T9DbkByL52ZWotj9B0o9L4zEQWITYasE8nnOFt 8vuPubQ0VRxJWW0Hv03W5pZ74w5KjTYaT8dMYgM8k+4oOaVPRvEQfTUFlOGSARYfkLIJcfpp2 SmZ+PT/InOE4+1XkL5GeTgsaQb4EPYRjmgXild3jqIHNNrY910KPHl73cKXK2ujoNo0ghLbYg jWNUR7IHabtn4x3+zBFEElMTX4vAfKpAauS4lzadC/Erlnc53LniYAG2Dn9PC7fFhfES2WZ0P jTd774BhA8Cz9DyylNPR+QO6nISseTY9ryazUcN7Bc5EPMhAlvDGq8OifnK5Dbo69eXzWGleU 1ACjP2JDxYa6OZSKPuTRTeH7vV4pfC/yj9c8YEuXvGwKV9fTEPqmClOPZyWtl X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210303_084444_849658_CA7E6893 X-CRM114-Status: GOOD ( 21.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Mar 2, 2021 at 12:07 PM Robin Murphy wrote: > On 2021-02-26 14:03, Nicolas Saenz Julienne wrote: > > index d2a2d1bc58ba..997d13a21717 100644 > > --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h > > +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h > > @@ -477,15 +477,20 @@ static inline void arm_smmu_writel(struct arm_smmu_device *smmu, int page, > > { > > if (smmu->impl && unlikely(smmu->impl->write_reg)) > > smmu->impl->write_reg(smmu, page, offset, val); > > - else > > + else if (dev_64bit_mmio_supported(smmu->dev)) > > writel_relaxed(val, arm_smmu_page(smmu, page) + offset); > > + else > > + hi_lo_writeq_relaxed(val, arm_smmu_page(smmu, page) + offset); > > As Arnd pointed out, this is in completely the wrong place. Also, in > general it doesn't work if the implementation already needs a hook to > filter or override register accesses for any other reason. TBH I'm not > convinced that this isn't *more* of a mess than handling it on a > SoC-specific basis... I think the main problem for handling it in a SoC specific way is that there is no device-independent way to do a 64-bit store as two 32-bit stores: - some devices need hi_lo_writeq_relaxed(), others need lo_hi_writeq_relaxed(), and some absolutely require 64-bit stores and cannot work at all behind a broken PCI bus. - if the driver requires the store to be atomic, it needs to use a spinlock around the two writel(), but if the register is on a PCI bus or mapped with page attributes that allow posted writes (like arm64 ioremap), then you may need to read back the register before spin_unlock to serialize them properly. However, reading back an mmio register is slow and can have side-effects, so you can't put that in driver-independent code either. Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel