From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BDEBC433B4 for ; Mon, 3 May 2021 10:19:46 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9562661153 for ; Mon, 3 May 2021 10:19:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9562661153 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Subject:Cc:To: From:Message-ID:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=a12Qp8cCS/J9jFkwQBtYcYHosSkVHbJbNVqLbtz2Ka4=; b=RkXOVcWaZjQXbgKuEXpVyyW4L FHjrJEClvlabXwmafuvIRiYwqpWN6h5BWwXLEmHF1Y/O7iaOLNr554Dr/MhKiRiODsYqcgC9x/Pz+ 0lgZA3HhiTa0J8VSAHmdzeVIcY20XtCObArx1nzGeh2e145EPEELGU69OBijLVuj1zzlKTNtADTL3 UE0tviL70qcE6XCC3mz4ZQRHSw6o0mwLo3qqRMTdPF6WaO87r4DTMeHHr7PnNYUQk1XKgpoly/rJe fQ5HXjzikgbxL+b4USyOcR35h8n/tnfae1xkQsKwzRDeNnMgkUuQy9TZhPTnzpqwCOymGXWIm3/RK Ch8h8/cjw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1ldVe7-00Ddi4-2m; Mon, 03 May 2021 10:17:47 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1ldVdt-00Ddh6-Ry for linux-arm-kernel@desiato.infradead.org; Mon, 03 May 2021 10:17:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=lzByfI/8+8ZnClRBfoxRHzQCK5HbBvhBXue+9rYMBRU=; b=ENkIUQPAwyQ4AMHLW9qILMm9F+ vYtU8pjgi9FZ/1+Zq5hXAebZ34Byhi6HLrRou4rYLhpGdG55aKhGyueTw+l2i2UcHvgK2jha6yaR2 SgUpjpJ+WM0mgafxpmw+1W3e1M7xW/6p3IKBtfe14c4rfujjVFcyq5yVtPJ/8uRlBKY/Lz67FyX2Q gTTysG1s2hLVBUK6H95dtWNgrH1suvok8x6kJih4o0tbmeDvbAXDWjtqGpoZNkxQMJ3ZmONBJ0Z3i yapxiVlal2fGJ8mwl96rcQ2fyve//BhEhhC7X6XDY3EWJIGWtRcXXf3PmD9N9fX85zopK7zP+3XC8 JkeS76vA==; Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1ldVdn-002yHN-Fk for linux-arm-kernel@lists.infradead.org; Mon, 03 May 2021 10:17:28 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2C3DB610E6; Mon, 3 May 2021 10:17:27 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=wait-a-minute.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1ldVdk-00AX3n-VA; Mon, 03 May 2021 11:17:25 +0100 Date: Mon, 03 May 2021 11:17:23 +0100 Message-ID: <87bl9sunnw.wl-maz@kernel.org> From: Marc Zyngier To: Vikram Sethi Cc: Shanker Donthineni , Alex Williamson , Will Deacon , Catalin Marinas , Christoffer Dall , "linux-arm-kernel@lists.infradead.org" , "kvmarm@lists.cs.columbia.edu" , "linux-kernel@vger.kernel.org" , "kvm@vger.kernel.org" , Jason Sequeira Subject: Re: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR region in VMA In-Reply-To: References: <20210429162906.32742-1-sdonthineni@nvidia.com> <20210429162906.32742-2-sdonthineni@nvidia.com> <20210429122840.4f98f78e@redhat.com> <470360a7-0242-9ae5-816f-13608f957bf6@nvidia.com> <20210429134659.321a5c3c@redhat.com> <87czucngdc.wl-maz@kernel.org> <1edb2c4e-23f0-5730-245b-fc6d289951e1@nvidia.com> <878s4zokll.wl-maz@kernel.org> <87eeeqvm1d.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: vsethi@nvidia.com, sdonthineni@nvidia.com, alex.williamson@redhat.com, will@kernel.org, catalin.marinas@arm.com, christoffer.dall@arm.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, jsequeira@nvidia.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210503_031727_591677_177AAEEC X-CRM114-Status: GOOD ( 43.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Vikram, On Sun, 02 May 2021 18:56:31 +0100, Vikram Sethi wrote: > > Hi Marc, > > > From: Marc Zyngier > > Hi Vikram, > > > > > The problem I see is that we have VM and userspace being written in terms > > of Write-Combine, which is: > > > > - loosely defined even on x86 > > > > - subject to interpretations in the way it maps to PCI > > > > - has no direct equivalent in the ARMv8 collection of memory > > attributes (and Normal_NC comes with speculation capabilities which > > strikes me as extremely undesirable on arbitrary devices) > > If speculation with Normal NC to prefetchable BARs in devices was a > problem, those devices would already be broken in baremetal with > ioremap_wc on arm64, and we would need quirks there to not do Normal > NC for them but Device GRE, and if such a quirk was needed on > baremetal, it could be picked up by vfio/KVM as well. But we haven't > seen any broken devices doing wc on baremetal on ARM64, have we? The lack of evidence does not equate to a proof, and your devices not misbehaving doesn't mean it is the right thing, specially when we have such a wide range of CPU and interconnect implementation. Which is why I really want an answer at the architecture level. Not a "it works for me" type of answer. Furthermore, as I replied to Shanker in a separate email, what Linux/arm64 does is pretty much irrelevant. KVM/arm64 implements the ARMv8 architecture, and it is at that level that we need to solve the problem. If, by enumerating the properties of Prefetchable, you can show that they are a strict superset of Normal_NC, I'm on board. I haven't seen such an enumeration so far. > I know we have tested NICs write combining on arm64 in baremetal, as > well as GPU and NVMe CMB without issues. > > Further, I don't see why speculation to non cacheble would be an > issue if prefetch without side effects is allowed by the device, > which is what a prefetchable BAR is. > If it is an issue for a device I would consider that a bug already needing a quirk in > Baremetal/host kernel already. > From PCI spec " A prefetchable address range may have write side effects, > but it may not have read side effects." Right, so we have made a small step in the direction of mapping "prefetchable" onto "Normal_NC", thanks for that. What about all the other properties (unaligned accesses, ordering, gathering)? > > How do we translate this into something consistent? I'd like to see an actual > > description of what we *really* expect from WC on prefetchable PCI regions, > > turn that into a documented definition agreed across architectures, and then > > we can look at implementing it with one memory type or another on arm64. > > > > Because once we expose that memory type at S2 for KVM guests, it > > becomes ABI and there is no turning back. So I want to get it right once and > > for all. > > > I agree that we need a precise definition for the Linux ioremap_wc > API wrt what drivers (kernel and userspace) can expect and whether > memset/memcpy is expected to work or not and whether aligned > accesses are a requirement. > To the extent ABI is set, I would think that the ABI is also already > set in the host kernel for arm64 WC = Normal NC, so why should that > not also be the ABI for same driver in VMs. KVM is an implementation of the ARM architecture, and doesn't really care about what WC is. If we come to the conclusion that Normal_NC is the natural match for Prefetchable attributes, than we're good and we can have Normal_NC being set by userspace, or even VFIO. But I don't want to set it only because "it works when bare-metal Linux uses it". Remember KVM doesn't only run Linux as guests. M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel