From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FA5DC35280 for ; Wed, 6 Apr 2022 00:17:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1583307AbiDEXwO (ORCPT ); Tue, 5 Apr 2022 19:52:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37236 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1457074AbiDEQCm (ORCPT ); Tue, 5 Apr 2022 12:02:42 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0204810CF0F for ; Tue, 5 Apr 2022 08:27:22 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id ACD8BB81E50 for ; Tue, 5 Apr 2022 15:27:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5077BC385A1; Tue, 5 Apr 2022 15:27:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1649172439; bh=/DCV2yNrKkFlY3sIbdS1XuH+VrVl+6Ab+68ybP9HpLo=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=MQXOeijMhEla1Cj7eLgmWmBBIFH+rD4Yhwlpz5yfgJEglmi5blaz/z/69bzWNy8Hp Kl8ydUGN/efnr+Anz3HD609aOSf8AXdRpRXkUcQU8BTWc6gu6i9IhO3JNGfI3zAv1t Rrtwe2LSGOGJJc5yhRqqTtJ+xjk+Z6ZSxQ8ViZwNCzAw5dZ/kojGe3S4DshwYwtGl7 e30G+K1sCT6iLyV/s37CXqa3kJskUB/MRMiPE39QzkcBOGFqHjS3eaAmO5eeNQCAb/ 7Wu9BcjYRSmVjbKRVCyqbzNucLjdfXMJS8I8tYYykk8PZOclYMyj4woISfh+zEjaBN ttOKK2YPvID4w== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nbl5Q-001sAH-Oh; Tue, 05 Apr 2022 16:27:16 +0100 Date: Tue, 05 Apr 2022 16:27:16 +0100 Message-ID: <87mtgzblez.wl-maz@kernel.org> From: Marc Zyngier To: Jason Gunthorpe Cc: xieming , sashal@kernel.org, catalin.marinas@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64 In-Reply-To: <20220404170202.GR64706@ziepe.ca> References: <20220401090828.614167-1-xieming@kylinos.cn> <87tubcbvgk.wl-maz@kernel.org> <20220404132405.GQ64706@ziepe.ca> <87o81gc3dc.wl-maz@kernel.org> <20220404170202.GR64706@ziepe.ca> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: jgg@ziepe.ca, xieming@kylinos.cn, sashal@kernel.org, catalin.marinas@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 04 Apr 2022 18:02:02 +0100, Jason Gunthorpe wrote: > > On Mon, Apr 04, 2022 at 03:47:11PM +0100, Marc Zyngier wrote: > > > I'm guessing it will turn into a SBSA like thing where the ARM ARM is > > > kind of vauge but a SOC has to implement Normal-NC in a certain way to > > > be functional for the server market. > > > > The main issue is that this equivalence isn't architected, so people > > can build whatever they want. SBSA means nothing to KVM (or Linux at > > large), and there is currently no way to describe which devices are > > safe to map as Normal-NC vs Device. > > And people have, we know of some ARM SOC's that don't work fully with > NORMAL_NC for this usage. That is already a problem for baremetal > Linux, let alone KVM.. > > That is why I likened it to SBSA - if you want to build a server SOC > that works with existing server software, you have to support > NORMAL_NC in this way. Even if it isn't architected. I see it the other way around. If it isn't architected (and in this case not even detectable in a scalable way), it simply isn't supportable by SW. > The KVM challenge, at least, is to support a CPU with working > NORMAL_NC to create VM that emulates the same CPU with working > NORMAL_NC. > > I didn't quite understand your other remarks though - is there a > problem here? It seems like yes from the other thread you pointed at? The main issue is that we have no idea what the behaviour is on a given implementation, and no way to even detect that for a given device, NORMAL_NC is a memory type that won't cause any issue. > I would think that KVM should mirror the process page table > configuration into the KVM page table and make this into a userspace > problem? But what makes it safe to do this the first place? There are tons of HW out there that will simply issue a SError if you generate an unaligned access targeting the right device, and letting userspace decide on this is simply not acceptable. > That turns it into a VFIO problem to negotiate with userspace and set > the proper pgprot. At least VFIO has a better chance than KVM to > consult DT or something to learn about the specific device's > properties. > > I don't know how VFIO/qemu/etc can make this all work automatically > correctly 100% of the time. It seems to me it is the same problem as > just basic baremetal "WC" is troubled on ARM in general today. Maybe > some tables and a command line option in qemu is the best we can hope > for. Having a firmware description of what can be mapped with what attributes would be pretty useful indeed. Not sure how that scales, but the platform definitely needs to advertise *something* so that we can allow userspace to say something. > > Long ago I asked that the ARM folks could come with some Linux > definition of all the WC-like modes and some arch calls to indicate > which one(s) should be used. Nobody seemed interested in doing that, > so the above SOC was left non-working in mainline Linux.. > > > We either have to take userspace's word for it, or rely on some other > > heuristics (do this for PCIe, but not anything else). None of which > > are entirely safe. Not to mention that no currently available CPU > > implements FEAT_DGH. > > DHG is an optimization, not a functional requirement. Currently > available CPUs use one of the more expensive barriers that are > architected to include DHG behavior. > > In any event, this is an important optimization. It is why ARMv9 is > introducing a new instruction specifically to optmize it. ARMv9? No, seems like it was introduced in the v8.7 time frame, and allowed retroactively from v8.0. N2 has it, but A510 doesn't, while V1 (an ARMv8.3 part) has it. But at least it is slowly creeping into implementations. Thanks, M. -- Without deviation from the norm, progress is not possible.