From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15D51C433ED for ; Mon, 3 May 2021 09:52:49 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6FD1261157 for ; Mon, 3 May 2021 09:52:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6FD1261157 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Subject:Cc:To: From:Message-ID:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=zpiFIcJFcoA2DB0x/8BxidlNdCGfQceKfXW3OfRFgNo=; b=j/SfV5Ibu1DJg/ZyybTx68IbS u0o5/E1LWl9GiXcMWBts4MFn818So8VHpE7VfPMFrRDwVGUnvCEhtjlFWnZ3JkWfM2ra+tBFYglwr m3FxsHY9iFLojiMRJkfJygcd9dvE6jGdW6GbsGUyWHUEnz2A8E9E77gSx7o6tKMiIpdO4pxukpZ/5 kiboqXVt2IhJZxpmLhk1sUa3SXJsE/tFzcRu0NJMLz7wBATx7+DnB9tmEedXpoCKSxbfEVq0ek8S4 GLLz5PHX0Xsk92ZKGi8hrAZeJv4KtCR/8GAUISCsh7y7KwZIHKwVXo+k2HqqjyF9wwPs5iMiiaTFo m3kjQrPGg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1ldVE4-00DaGw-Px; Mon, 03 May 2021 09:50:52 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1ldVE0-00DaFh-Jw for linux-arm-kernel@desiato.infradead.org; Mon, 03 May 2021 09:50:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=+GtRgp+3iknJ51B9eOOOsJsdhgi5ARBvRQi0Zs95ZF4=; b=VfUf4GFEU7qOrsQuKZno6kksnO GkVq4iK0tAxYZJgM+Nju3KWICbvgRz8Sg+Jr7ZPcGFeiSMhI+NYYtMT5RY5ldK4cswN3YeQyLuI4G 5nxvPK4I8ogg5xT1Vp+P6VZ1F6Vn5l7U/rBdQRL4oGVf0t5eeQpfXwASJAZs7ALv2RxYIjJ6e7q3A bJKZtsREArebZWBeSf8vwnBKJkTVLWvcPoF+N2CEDcF6Z/IwJx1Oxn8gPZxDGkgPBvWVOFT676xyK vWfFAM+2xkz89YcDJoDg2//6fFGBnmdeLkiT63liNuBSlLYtvKZIfj3faB+hE4FlZSfrmkK6HRQXo gdExxnDw==; Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1ldVDy-002xO7-07 for linux-arm-kernel@lists.infradead.org; Mon, 03 May 2021 09:50:47 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id F0DC860FE9; Mon, 3 May 2021 09:50:44 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=wait-a-minute.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1ldVDu-00AWmd-Pq; Mon, 03 May 2021 10:50:42 +0100 Date: Mon, 03 May 2021 10:50:41 +0100 Message-ID: <87czu8uowe.wl-maz@kernel.org> From: Marc Zyngier To: Shanker R Donthineni Cc: Vikram Sethi , Alex Williamson , Will Deacon , Catalin Marinas , "Christoffer\ Dall" , "linux-arm-kernel@lists.infradead.org" , "kvmarm@lists.cs.columbia.edu" , "linux-kernel@vger.kernel.org" , "kvm@vger.kernel.org" , Jason Sequeira Subject: Re: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR region in VMA In-Reply-To: <49e26646-9f05-ccb8-f5c1-73a92ab79972@nvidia.com> References: <20210429162906.32742-1-sdonthineni@nvidia.com> <20210429162906.32742-2-sdonthineni@nvidia.com> <20210429122840.4f98f78e@redhat.com> <470360a7-0242-9ae5-816f-13608f957bf6@nvidia.com> <20210429134659.321a5c3c@redhat.com> <87czucngdc.wl-maz@kernel.org> <1edb2c4e-23f0-5730-245b-fc6d289951e1@nvidia.com> <878s4zokll.wl-maz@kernel.org> <87eeeqvm1d.wl-maz@kernel.org> <49e26646-9f05-ccb8-f5c1-73a92ab79972@nvidia.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: sdonthineni@nvidia.com, vsethi@nvidia.com, alex.williamson@redhat.com, will@kernel.org, catalin.marinas@arm.com, christoffer.dall@arm.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, jsequeira@nvidia.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210503_025046_127632_DB95F9F9 X-CRM114-Status: GOOD ( 33.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Sat, 01 May 2021 12:36:11 +0100, Shanker R Donthineni wrote: > > Hi Marc, > > On 5/1/21 4:30 AM, Marc Zyngier wrote: > >> I think Device GRE has some practical problems. > >> 1. A lot of userspace code which is used to getting write combined > >> mappings to GPU memory from kernel drivers does memcpy/memset on it > >> which can insert ldp/stp which can crash on Device Memory Type. From > >> a quick search I didn't find a memcpy_io or memset_io in > >> glibc. Perhaps there are some other functions available, but a lot > >> of userspace applications that work on x86 and ARM baremetal won't > >> work on ARM VMs without such changes. Changes to all of userspace > >> may not always be practical, specially if linking to binaries > > This seems to go against what Alex was hinting at earlier, which is > > that unaligned accesses were not expected on prefetchable regions, and > > Shanker latter confirming that it was an actual bug. Where do we stand > > here? > > > We agreed to call it a driver bug if it's not following Linux > write-combining API ioremap_wc() semantics. So far I didn't find > whether unaligned accesses allowed or not for WC regions explicitly > in Linux documentation. And that's exactly the kind of problem I want clarification on before we add *anything* to KVM. Proper, unambiguous definition of what WC is on the CPU side, and how it maps onto PCI. Without such a definition, we're just driving blind. > Page faults due to driver unaligned accesses > in kernel space will be under driver control, we'll fix it. > > Driver uses the architecture agnostic functions that are available > in the Linux kernel and expecting the same behavior in VM vs > Baremetal. We would like to keep the driver implementation is > architecture-independent as much as possible and support VM > unaware. For ARM64, VM's ioremap_wc() definition doesn't match > baremetal. You are mixing two things: what Linux/arm64 gives to kernel drivers, and what KVM, as an implementation of the ARMv8 architecture, gives to virtual machines. There is zero reason for the two to match if there is no definition of what we need to provide. > We don't have any control over the userspace > applications/drivers/libraries as Vikram saying. Another example GCC > memset() function uses 'DC ZVA' which triggers an alignment fault if > the actual memory type is device_xxx. Again, you're talking about an application, and I'm talking about how to map a nebulous concept that originated on a foreign architecture onto something that is entirely different. So please drop the "that's how my SW works", and instead give me a good definition of what WC means in architectural terms. Thanks M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel