From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83133C43603 for ; Wed, 18 Dec 2019 16:08:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 502B624672 for ; Wed, 18 Dec 2019 16:08:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="rf+XD25p" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727469AbfLRQII (ORCPT ); Wed, 18 Dec 2019 11:08:08 -0500 Received: from mail-wm1-f68.google.com ([209.85.128.68]:54266 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726985AbfLRQIH (ORCPT ); Wed, 18 Dec 2019 11:08:07 -0500 Received: by mail-wm1-f68.google.com with SMTP id m24so2408443wmc.3 for ; Wed, 18 Dec 2019 08:08:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=HlbfX+nhA+yr8jwGVCf2vKWxFkPoGRCkCNeDk7rXv8M=; b=rf+XD25p+0eZtPn+Of3DpE7+VILCHxiI01e18MTPQvaI5Dx5ShYXNOuMZBBRGuWrW7 l1zm14YHYrXBbPr0m+9SGpjE2MuyavX8n33cvL3Vn8k8IL4M71hQ/J7q9ms5ff+SEIEs D5jVsZF/E/N2V61nstQEiUgxdDSwlTbs//LUlxEeE0qiWM8OEDDM5v3/A72LGQhxvJKl wXRcvQ5GlzwYd42FGdjPWyGqNoTYN1oDy1wSS9NJZTpGPtUVCev9S9YLPd506DGk4Isz x7/Z7s7Fg8Fa+HMicr3R0DyfE7Fpezg9WcqfnaGeDhTrWZHL/y5EBz2iTSThhjSqSTnK 5DzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=HlbfX+nhA+yr8jwGVCf2vKWxFkPoGRCkCNeDk7rXv8M=; b=dLAbfN2QxXyI9M4P0NslGjIu7ZRBDD6IBYJAuVqVZfUkVIbD72+eswNo3LTxSNTMaa MKBqsdCGYYgvXwUQziJRJS5j66eYrEI59exv/Cs7Cyww7LEG4jfD1nk60nHIFfymVEwG rZSYPB9Bfmp8rfR9l95mdQMGVmU+I4AsgFCq72aJxKDbBrL+nDuGkMRIyt0VFGVCFmIG B36GM8pE5L22Il7db5+O6bh6HnWfQbq+yh9r3U0fbcmXSXWt+z1b6TA3TGmTbWTbIyhj fjBy08Ofg4hagHtaHU0VEKXnX+fpLMva/M1DFv6dN7W+LIZbCBMT4vBXB52yyi0O13LB 675A== X-Gm-Message-State: APjAAAUrtMUS8XAv4RJnZBZyZUrmOq8CuqLGSfDoLEQewoYV7K+epFmX mv5yfSL8n0RfKM1T4B+46niuaUMA+vw= X-Google-Smtp-Source: APXvYqx/JAqyGPCzoycHN+vwCa6OM+w0zw+UC9aZwLHysrkzU1K2buIOHwW7fRDNS6a2Je7DE55Xgw== X-Received: by 2002:a7b:cbc9:: with SMTP id n9mr4172404wmi.89.1576685286095; Wed, 18 Dec 2019 08:08:06 -0800 (PST) Received: from myrica (adsl-84-227-176-239.adslplus.ch. [84.227.176.239]) by smtp.gmail.com with ESMTPSA id z187sm2951508wme.16.2019.12.18.08.08.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 08:08:05 -0800 (PST) Date: Wed, 18 Dec 2019 17:07:59 +0100 From: Jean-Philippe Brucker To: Auger Eric Cc: linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-acpi@vger.kernel.org, devicetree@vger.kernel.org, iommu@lists.linux-foundation.org, joro@8bytes.org, robh+dt@kernel.org, mark.rutland@arm.com, lorenzo.pieralisi@arm.com, guohanjun@huawei.com, sudeep.holla@arm.com, rjw@rjwysocki.net, lenb@kernel.org, will@kernel.org, robin.murphy@arm.com, bhelgaas@google.com, jonathan.cameron@huawei.com, zhangfei.gao@linaro.org Subject: Re: [PATCH v3 07/13] iommu/arm-smmu-v3: Add support for Substream IDs Message-ID: <20191218160759.GF2371701@myrica> References: <20191209180514.272727-1-jean-philippe@linaro.org> <20191209180514.272727-8-jean-philippe@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org On Tue, Dec 17, 2019 at 05:43:59PM +0100, Auger Eric wrote: > > -static void arm_smmu_write_ctx_desc(struct arm_smmu_device *smmu, > > - struct arm_smmu_s1_cfg *cfg) > > +static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, > > + int ssid, struct arm_smmu_ctx_desc *cd) > > { > > - u64 val; > > - __le64 *cdptr = cfg->table.ptr; > > - > > /* > > - * We don't need to issue any invalidation here, as we'll invalidate > > - * the STE when installing the new entry anyway. > > + * This function handles the following cases: > > + * > > + * (1) Install primary CD, for normal DMA traffic (SSID = 0). > > + * (2) Install a secondary CD, for SID+SSID traffic. > > + * (3) Update ASID of a CD. Atomically write the first 64 bits of the > > + * CD, then invalidate the old entry and mappings. > > + * (4) Remove a secondary CD. > I see arm_smmu_write_ctx_desc getting called with non null cd in > arm_smmu_attach_dev but I do not see it removed in the detach path? No we don't have to remove the primary CD, since detach clears the STE and frees the CD tables. > > */ > > - val = arm_smmu_cpu_tcr_to_cd(cfg->cd.tcr) | > > -#ifdef __BIG_ENDIAN > > - CTXDESC_CD_0_ENDI | > > -#endif > > - CTXDESC_CD_0_R | CTXDESC_CD_0_A | CTXDESC_CD_0_ASET | > > - CTXDESC_CD_0_AA64 | FIELD_PREP(CTXDESC_CD_0_ASID, cfg->cd.asid) | > > - CTXDESC_CD_0_V; > > + u64 val; > > + bool cd_live; > > + struct arm_smmu_device *smmu = smmu_domain->smmu; > > + __le64 *cdptr = smmu_domain->s1_cfg.table.ptr + ssid * > > + CTXDESC_CD_DWORDS;> > > - /* STALL_MODEL==0b10 && CD.S==0 is ILLEGAL */ > > - if (smmu->features & ARM_SMMU_FEAT_STALL_FORCE) > > - val |= CTXDESC_CD_0_S; > > + val = le64_to_cpu(cdptr[0]); > > + cd_live = !!(val & CTXDESC_CD_0_V); > > > > - cdptr[0] = cpu_to_le64(val); > > + if (!cd) { /* (4) */ > > + val = 0; > > + } else if (cd_live) { /* (3) */ > > + val &= ~CTXDESC_CD_0_ASID; > > + val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); > > + /* > > + * Until CD+TLB invalidation, both ASIDs may be used for tagging > > + * this substream's traffic > > + */ > > + } else { /* (1) and (2) */ > > + cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); > > + cdptr[2] = 0; > > + cdptr[3] = cpu_to_le64(cd->mair); > > + > > + /* > > + * STE is live, and the SMMU might fetch this CD at any > > + * time. Ensure that it observes the rest of the CD before we > > + * enable it. > Mostly, on't you want the invalid state to be seen? Sorry I didn't get your question. Without the sync, the SMMU could read an invalid address in CD[1], read V=1 in CD[0] and proceed with dereferencing the bogus pointer before seeing our update to CD[1]. To prevent this we follow the update procedure described in IHI0070Ca 3.21.3.1 (Configuration structure update procedure): Because the SMMU can read any reachable structure at any time, and is not required to read the double-words of the structure in order, Arm recommends that the following procedure is used to initialize structures: 1. Structure starts invalid, having V == 0. 2. Fill in all fields, leaving V == 0, then perform a DSB operation to ensure written data is observable from the SMMU. 3. Issue a CMD_CFGI_, as appropriate. 4. Issue a CMD_SYNC, and wait for completion. 5. Set V to 1, then perform a DSB operation to ensure write is observable by the SMMU. 6. Issue CMD_CFGI_, as appropriate. 7. Optionally issue a CMD_SYNC, and wait for completion. This must be done if a subsequent software operation, such as enabling device DMA, depends on the SMMU using the new structure. > > @@ -1664,6 +1739,7 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, > > if (s1_cfg) { > > BUG_ON(ste_live); > > dst[1] = cpu_to_le64( > > + FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) | > > FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) | > > FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) | > > FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) | > > @@ -1674,7 +1750,9 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, > > dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD); > > > > val |= (s1_cfg->table.ptr_dma & STRTAB_STE_0_S1CTXPTR_MASK) | > > - FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S1_TRANS); > > + FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S1_TRANS) | > > + FIELD_PREP(STRTAB_STE_0_S1CDMAX, s1_cfg->s1cdmax) | > > + FIELD_PREP(STRTAB_STE_0_S1FMT, s1_cfg->s1fmt); > Do I understand correctly that each time we update the CD, we also > update the STE. This is not wrong but theoretically this is not needed, > right? Only the primary CD. Once we add SVA (or auxiliary domains) we need to update secondary CDs while the STE is live. This series is just preparation for SVA support. I didn't include the full support because it requires another 25+ patches. > This also explains why you do not need the CMD_PREFETCH_CONFIG with ssid > because you invalidate the config per sid? Hm, I didn't add a CMD_PREFETCH_CONFIG to CDs because I don't know if it's useful. Maybe as future optimization. Thanks, Jean From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19CF5C2D0D2 for ; Wed, 18 Dec 2019 16:08:11 +0000 (UTC) Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E15862465E for ; Wed, 18 Dec 2019 16:08:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="rf+XD25p" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E15862465E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id C2FE0884DE; Wed, 18 Dec 2019 16:08:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7t76FTcQhavx; Wed, 18 Dec 2019 16:08:10 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by hemlock.osuosl.org (Postfix) with ESMTP id 146528715D; Wed, 18 Dec 2019 16:08:10 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id EB74BC1797; Wed, 18 Dec 2019 16:08:09 +0000 (UTC) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id DC566C077D for ; Wed, 18 Dec 2019 16:08:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id D836185A49 for ; Wed, 18 Dec 2019 16:08:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UGKJWh6Mx7tG for ; Wed, 18 Dec 2019 16:08:08 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by fraxinus.osuosl.org (Postfix) with ESMTPS id C573B85A0E for ; Wed, 18 Dec 2019 16:08:07 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id a5so2420489wmb.0 for ; Wed, 18 Dec 2019 08:08:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=HlbfX+nhA+yr8jwGVCf2vKWxFkPoGRCkCNeDk7rXv8M=; b=rf+XD25p+0eZtPn+Of3DpE7+VILCHxiI01e18MTPQvaI5Dx5ShYXNOuMZBBRGuWrW7 l1zm14YHYrXBbPr0m+9SGpjE2MuyavX8n33cvL3Vn8k8IL4M71hQ/J7q9ms5ff+SEIEs D5jVsZF/E/N2V61nstQEiUgxdDSwlTbs//LUlxEeE0qiWM8OEDDM5v3/A72LGQhxvJKl wXRcvQ5GlzwYd42FGdjPWyGqNoTYN1oDy1wSS9NJZTpGPtUVCev9S9YLPd506DGk4Isz x7/Z7s7Fg8Fa+HMicr3R0DyfE7Fpezg9WcqfnaGeDhTrWZHL/y5EBz2iTSThhjSqSTnK 5DzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=HlbfX+nhA+yr8jwGVCf2vKWxFkPoGRCkCNeDk7rXv8M=; b=FnCfyN4MvpJsGAEQuUFhYsbUgZaQgAu42WfJhXKupOMhpmh89s48fYId3v7QNo16R5 CMS20S4+/X78paRvI7wC75oMzcU5f6nelfm9XQKER/ZgjrR3CKeQ9PJJy0HNw4pEa34Y vBvmnIaKrp5+c036kTMDRFj2sxlYJmFPc2cHXwV2XNElsxMaqnNuvLAQTPuwrAR/tjgS gdDv0QbKCVz7yzKKchKh45o+niw9DpJidsYCzf1I6rhYa1HDjHc9EzlARixm4lLdLFje qJPoDkgvnKlOjHwVDwWH02Ott7ZShA+rf1MUeECEtZxUkBxZcanNYJ1ea//kXWt4hpZb 4Ajw== X-Gm-Message-State: APjAAAWMub8KhK0R6Kdc6499DUw+tatJg2qCaFrJqXvk0T+QScgKZEpd tbkR1QTzOFPFzEXAJv9peVuHww== X-Google-Smtp-Source: APXvYqx/JAqyGPCzoycHN+vwCa6OM+w0zw+UC9aZwLHysrkzU1K2buIOHwW7fRDNS6a2Je7DE55Xgw== X-Received: by 2002:a7b:cbc9:: with SMTP id n9mr4172404wmi.89.1576685286095; Wed, 18 Dec 2019 08:08:06 -0800 (PST) Received: from myrica (adsl-84-227-176-239.adslplus.ch. [84.227.176.239]) by smtp.gmail.com with ESMTPSA id z187sm2951508wme.16.2019.12.18.08.08.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 08:08:05 -0800 (PST) Date: Wed, 18 Dec 2019 17:07:59 +0100 From: Jean-Philippe Brucker To: Auger Eric Subject: Re: [PATCH v3 07/13] iommu/arm-smmu-v3: Add support for Substream IDs Message-ID: <20191218160759.GF2371701@myrica> References: <20191209180514.272727-1-jean-philippe@linaro.org> <20191209180514.272727-8-jean-philippe@linaro.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: mark.rutland@arm.com, devicetree@vger.kernel.org, linux-pci@vger.kernel.org, sudeep.holla@arm.com, rjw@rjwysocki.net, robin.murphy@arm.com, linux-acpi@vger.kernel.org, iommu@lists.linux-foundation.org, robh+dt@kernel.org, guohanjun@huawei.com, bhelgaas@google.com, zhangfei.gao@linaro.org, will@kernel.org, linux-arm-kernel@lists.infradead.org, lenb@kernel.org X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Tue, Dec 17, 2019 at 05:43:59PM +0100, Auger Eric wrote: > > -static void arm_smmu_write_ctx_desc(struct arm_smmu_device *smmu, > > - struct arm_smmu_s1_cfg *cfg) > > +static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, > > + int ssid, struct arm_smmu_ctx_desc *cd) > > { > > - u64 val; > > - __le64 *cdptr = cfg->table.ptr; > > - > > /* > > - * We don't need to issue any invalidation here, as we'll invalidate > > - * the STE when installing the new entry anyway. > > + * This function handles the following cases: > > + * > > + * (1) Install primary CD, for normal DMA traffic (SSID = 0). > > + * (2) Install a secondary CD, for SID+SSID traffic. > > + * (3) Update ASID of a CD. Atomically write the first 64 bits of the > > + * CD, then invalidate the old entry and mappings. > > + * (4) Remove a secondary CD. > I see arm_smmu_write_ctx_desc getting called with non null cd in > arm_smmu_attach_dev but I do not see it removed in the detach path? No we don't have to remove the primary CD, since detach clears the STE and frees the CD tables. > > */ > > - val = arm_smmu_cpu_tcr_to_cd(cfg->cd.tcr) | > > -#ifdef __BIG_ENDIAN > > - CTXDESC_CD_0_ENDI | > > -#endif > > - CTXDESC_CD_0_R | CTXDESC_CD_0_A | CTXDESC_CD_0_ASET | > > - CTXDESC_CD_0_AA64 | FIELD_PREP(CTXDESC_CD_0_ASID, cfg->cd.asid) | > > - CTXDESC_CD_0_V; > > + u64 val; > > + bool cd_live; > > + struct arm_smmu_device *smmu = smmu_domain->smmu; > > + __le64 *cdptr = smmu_domain->s1_cfg.table.ptr + ssid * > > + CTXDESC_CD_DWORDS;> > > - /* STALL_MODEL==0b10 && CD.S==0 is ILLEGAL */ > > - if (smmu->features & ARM_SMMU_FEAT_STALL_FORCE) > > - val |= CTXDESC_CD_0_S; > > + val = le64_to_cpu(cdptr[0]); > > + cd_live = !!(val & CTXDESC_CD_0_V); > > > > - cdptr[0] = cpu_to_le64(val); > > + if (!cd) { /* (4) */ > > + val = 0; > > + } else if (cd_live) { /* (3) */ > > + val &= ~CTXDESC_CD_0_ASID; > > + val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); > > + /* > > + * Until CD+TLB invalidation, both ASIDs may be used for tagging > > + * this substream's traffic > > + */ > > + } else { /* (1) and (2) */ > > + cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); > > + cdptr[2] = 0; > > + cdptr[3] = cpu_to_le64(cd->mair); > > + > > + /* > > + * STE is live, and the SMMU might fetch this CD at any > > + * time. Ensure that it observes the rest of the CD before we > > + * enable it. > Mostly, on't you want the invalid state to be seen? Sorry I didn't get your question. Without the sync, the SMMU could read an invalid address in CD[1], read V=1 in CD[0] and proceed with dereferencing the bogus pointer before seeing our update to CD[1]. To prevent this we follow the update procedure described in IHI0070Ca 3.21.3.1 (Configuration structure update procedure): Because the SMMU can read any reachable structure at any time, and is not required to read the double-words of the structure in order, Arm recommends that the following procedure is used to initialize structures: 1. Structure starts invalid, having V == 0. 2. Fill in all fields, leaving V == 0, then perform a DSB operation to ensure written data is observable from the SMMU. 3. Issue a CMD_CFGI_, as appropriate. 4. Issue a CMD_SYNC, and wait for completion. 5. Set V to 1, then perform a DSB operation to ensure write is observable by the SMMU. 6. Issue CMD_CFGI_, as appropriate. 7. Optionally issue a CMD_SYNC, and wait for completion. This must be done if a subsequent software operation, such as enabling device DMA, depends on the SMMU using the new structure. > > @@ -1664,6 +1739,7 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, > > if (s1_cfg) { > > BUG_ON(ste_live); > > dst[1] = cpu_to_le64( > > + FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) | > > FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) | > > FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) | > > FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) | > > @@ -1674,7 +1750,9 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, > > dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD); > > > > val |= (s1_cfg->table.ptr_dma & STRTAB_STE_0_S1CTXPTR_MASK) | > > - FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S1_TRANS); > > + FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S1_TRANS) | > > + FIELD_PREP(STRTAB_STE_0_S1CDMAX, s1_cfg->s1cdmax) | > > + FIELD_PREP(STRTAB_STE_0_S1FMT, s1_cfg->s1fmt); > Do I understand correctly that each time we update the CD, we also > update the STE. This is not wrong but theoretically this is not needed, > right? Only the primary CD. Once we add SVA (or auxiliary domains) we need to update secondary CDs while the STE is live. This series is just preparation for SVA support. I didn't include the full support because it requires another 25+ patches. > This also explains why you do not need the CMD_PREFETCH_CONFIG with ssid > because you invalidate the config per sid? Hm, I didn't add a CMD_PREFETCH_CONFIG to CDs because I don't know if it's useful. Maybe as future optimization. Thanks, Jean _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C503C2D0D4 for ; Wed, 18 Dec 2019 16:08:11 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E0DD7218AC for ; Wed, 18 Dec 2019 16:08:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Qc1rnUS0"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="rf+XD25p" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E0DD7218AC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vrfk/Q98Fjw1ahTydMHlN7dsHnqE2oLA2knxsKxyvDo=; b=Qc1rnUS0BQY0RF 5kBlczpGYkadmHDYC08mDnCOt8yaLCuxEGBTDvZZivbnp5gas6o7DKhspr2x//DYfwYoiS/g5ItGF Kay0BpH/iKkPYuecJ6YksRxqu8IUrOJ9A8asfPazxXnT0yywKnUxpWOTlJVSj53fOkUER0Smf4tAu mrmFt7adjyiKLFyUcU2GlxVzZed2jnzKRHINxBySjNuRF1fDqOZDTjv6XQ+5FkxgR9myg/phf6hW4 teHEDQSrOiVbH6sLHhbliuWIpjjhryWEKXHht35nTabySZV4OpRzmJH7t29n2437WpTBY9asxV0HJ tZESPjMO0yGwCMS1Jwow==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihbru-0008LG-9X; Wed, 18 Dec 2019 16:08:10 +0000 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1ihbrr-0008Kb-Kt for linux-arm-kernel@lists.infradead.org; Wed, 18 Dec 2019 16:08:09 +0000 Received: by mail-wm1-x342.google.com with SMTP id p9so2417047wmc.2 for ; Wed, 18 Dec 2019 08:08:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=HlbfX+nhA+yr8jwGVCf2vKWxFkPoGRCkCNeDk7rXv8M=; b=rf+XD25p+0eZtPn+Of3DpE7+VILCHxiI01e18MTPQvaI5Dx5ShYXNOuMZBBRGuWrW7 l1zm14YHYrXBbPr0m+9SGpjE2MuyavX8n33cvL3Vn8k8IL4M71hQ/J7q9ms5ff+SEIEs D5jVsZF/E/N2V61nstQEiUgxdDSwlTbs//LUlxEeE0qiWM8OEDDM5v3/A72LGQhxvJKl wXRcvQ5GlzwYd42FGdjPWyGqNoTYN1oDy1wSS9NJZTpGPtUVCev9S9YLPd506DGk4Isz x7/Z7s7Fg8Fa+HMicr3R0DyfE7Fpezg9WcqfnaGeDhTrWZHL/y5EBz2iTSThhjSqSTnK 5DzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=HlbfX+nhA+yr8jwGVCf2vKWxFkPoGRCkCNeDk7rXv8M=; b=fdP1/4aLdhvbnPuHeYM54Mb1+YogPQxiMHAMVrrYR8VFaKxXTIagqd8zycW6tJaftj x9ZKqljRSYLRFO/sExyWrI01nWXBJCG2ytnnd0zEZXoQVJVGm8L0EKUNYgm78AdE4FOC IpPPE0jsv1QspXD9SArog2dZkMSEarhCCI6qdEZnwewGhvmZwWnW7vtWCG7241jkb67s mCZLzoNZl3R7XNBDacCoEBbfAAXzidJznCwnKzLbSal8Vlunevww0HEEtL7iWIGz2VOq D+pWil3QbrW5eoPAtLW5sowNcIlz6928V6ed/vt+cLVsWKeES3emLuyS/f7VPPYj0DEm OVoQ== X-Gm-Message-State: APjAAAWs3aZ69OiUzjJ0dACvthP/G+SaHb8bQxT6g2RodlaBc+ILKoJB 9uYnGX1rSsWdmtuoEHSv7zkbAA== X-Google-Smtp-Source: APXvYqx/JAqyGPCzoycHN+vwCa6OM+w0zw+UC9aZwLHysrkzU1K2buIOHwW7fRDNS6a2Je7DE55Xgw== X-Received: by 2002:a7b:cbc9:: with SMTP id n9mr4172404wmi.89.1576685286095; Wed, 18 Dec 2019 08:08:06 -0800 (PST) Received: from myrica (adsl-84-227-176-239.adslplus.ch. [84.227.176.239]) by smtp.gmail.com with ESMTPSA id z187sm2951508wme.16.2019.12.18.08.08.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 08:08:05 -0800 (PST) Date: Wed, 18 Dec 2019 17:07:59 +0100 From: Jean-Philippe Brucker To: Auger Eric Subject: Re: [PATCH v3 07/13] iommu/arm-smmu-v3: Add support for Substream IDs Message-ID: <20191218160759.GF2371701@myrica> References: <20191209180514.272727-1-jean-philippe@linaro.org> <20191209180514.272727-8-jean-philippe@linaro.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191218_080807_697490_FBF6C3A7 X-CRM114-Status: GOOD ( 24.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, devicetree@vger.kernel.org, lorenzo.pieralisi@arm.com, linux-pci@vger.kernel.org, joro@8bytes.org, sudeep.holla@arm.com, rjw@rjwysocki.net, robin.murphy@arm.com, linux-acpi@vger.kernel.org, iommu@lists.linux-foundation.org, robh+dt@kernel.org, jonathan.cameron@huawei.com, guohanjun@huawei.com, bhelgaas@google.com, zhangfei.gao@linaro.org, will@kernel.org, linux-arm-kernel@lists.infradead.org, lenb@kernel.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Dec 17, 2019 at 05:43:59PM +0100, Auger Eric wrote: > > -static void arm_smmu_write_ctx_desc(struct arm_smmu_device *smmu, > > - struct arm_smmu_s1_cfg *cfg) > > +static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, > > + int ssid, struct arm_smmu_ctx_desc *cd) > > { > > - u64 val; > > - __le64 *cdptr = cfg->table.ptr; > > - > > /* > > - * We don't need to issue any invalidation here, as we'll invalidate > > - * the STE when installing the new entry anyway. > > + * This function handles the following cases: > > + * > > + * (1) Install primary CD, for normal DMA traffic (SSID = 0). > > + * (2) Install a secondary CD, for SID+SSID traffic. > > + * (3) Update ASID of a CD. Atomically write the first 64 bits of the > > + * CD, then invalidate the old entry and mappings. > > + * (4) Remove a secondary CD. > I see arm_smmu_write_ctx_desc getting called with non null cd in > arm_smmu_attach_dev but I do not see it removed in the detach path? No we don't have to remove the primary CD, since detach clears the STE and frees the CD tables. > > */ > > - val = arm_smmu_cpu_tcr_to_cd(cfg->cd.tcr) | > > -#ifdef __BIG_ENDIAN > > - CTXDESC_CD_0_ENDI | > > -#endif > > - CTXDESC_CD_0_R | CTXDESC_CD_0_A | CTXDESC_CD_0_ASET | > > - CTXDESC_CD_0_AA64 | FIELD_PREP(CTXDESC_CD_0_ASID, cfg->cd.asid) | > > - CTXDESC_CD_0_V; > > + u64 val; > > + bool cd_live; > > + struct arm_smmu_device *smmu = smmu_domain->smmu; > > + __le64 *cdptr = smmu_domain->s1_cfg.table.ptr + ssid * > > + CTXDESC_CD_DWORDS;> > > - /* STALL_MODEL==0b10 && CD.S==0 is ILLEGAL */ > > - if (smmu->features & ARM_SMMU_FEAT_STALL_FORCE) > > - val |= CTXDESC_CD_0_S; > > + val = le64_to_cpu(cdptr[0]); > > + cd_live = !!(val & CTXDESC_CD_0_V); > > > > - cdptr[0] = cpu_to_le64(val); > > + if (!cd) { /* (4) */ > > + val = 0; > > + } else if (cd_live) { /* (3) */ > > + val &= ~CTXDESC_CD_0_ASID; > > + val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); > > + /* > > + * Until CD+TLB invalidation, both ASIDs may be used for tagging > > + * this substream's traffic > > + */ > > + } else { /* (1) and (2) */ > > + cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); > > + cdptr[2] = 0; > > + cdptr[3] = cpu_to_le64(cd->mair); > > + > > + /* > > + * STE is live, and the SMMU might fetch this CD at any > > + * time. Ensure that it observes the rest of the CD before we > > + * enable it. > Mostly, on't you want the invalid state to be seen? Sorry I didn't get your question. Without the sync, the SMMU could read an invalid address in CD[1], read V=1 in CD[0] and proceed with dereferencing the bogus pointer before seeing our update to CD[1]. To prevent this we follow the update procedure described in IHI0070Ca 3.21.3.1 (Configuration structure update procedure): Because the SMMU can read any reachable structure at any time, and is not required to read the double-words of the structure in order, Arm recommends that the following procedure is used to initialize structures: 1. Structure starts invalid, having V == 0. 2. Fill in all fields, leaving V == 0, then perform a DSB operation to ensure written data is observable from the SMMU. 3. Issue a CMD_CFGI_, as appropriate. 4. Issue a CMD_SYNC, and wait for completion. 5. Set V to 1, then perform a DSB operation to ensure write is observable by the SMMU. 6. Issue CMD_CFGI_, as appropriate. 7. Optionally issue a CMD_SYNC, and wait for completion. This must be done if a subsequent software operation, such as enabling device DMA, depends on the SMMU using the new structure. > > @@ -1664,6 +1739,7 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, > > if (s1_cfg) { > > BUG_ON(ste_live); > > dst[1] = cpu_to_le64( > > + FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) | > > FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) | > > FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) | > > FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) | > > @@ -1674,7 +1750,9 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, > > dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD); > > > > val |= (s1_cfg->table.ptr_dma & STRTAB_STE_0_S1CTXPTR_MASK) | > > - FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S1_TRANS); > > + FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S1_TRANS) | > > + FIELD_PREP(STRTAB_STE_0_S1CDMAX, s1_cfg->s1cdmax) | > > + FIELD_PREP(STRTAB_STE_0_S1FMT, s1_cfg->s1fmt); > Do I understand correctly that each time we update the CD, we also > update the STE. This is not wrong but theoretically this is not needed, > right? Only the primary CD. Once we add SVA (or auxiliary domains) we need to update secondary CDs while the STE is live. This series is just preparation for SVA support. I didn't include the full support because it requires another 25+ patches. > This also explains why you do not need the CMD_PREFETCH_CONFIG with ssid > because you invalidate the config per sid? Hm, I didn't add a CMD_PREFETCH_CONFIG to CDs because I don't know if it's useful. Maybe as future optimization. Thanks, Jean _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel