From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=DKIM_ADSP_CUSTOM_MED, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C23CC1975A for ; Wed, 25 Mar 2020 16:12:21 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4753320409 for ; Wed, 25 Mar 2020 16:12:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4753320409 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:39140 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jH8dg-0007qR-Er for qemu-devel@archiver.kernel.org; Wed, 25 Mar 2020 12:12:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49797) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jH8cw-00073w-AW for qemu-devel@nongnu.org; Wed, 25 Mar 2020 12:11:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jH8ct-0005fs-Sx for qemu-devel@nongnu.org; Wed, 25 Mar 2020 12:11:34 -0400 Received: from indium.canonical.com ([91.189.90.7]:49336) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jH8ct-0005dx-Nb for qemu-devel@nongnu.org; Wed, 25 Mar 2020 12:11:31 -0400 Received: from loganberry.canonical.com ([91.189.90.37]) by indium.canonical.com with esmtp (Exim 4.86_2 #2 (Debian)) id 1jH8cr-0006ES-Nr for ; Wed, 25 Mar 2020 16:11:29 +0000 Received: from loganberry.canonical.com (localhost [127.0.0.1]) by loganberry.canonical.com (Postfix) with ESMTP id 9462D2E80C9 for ; Wed, 25 Mar 2020 16:11:29 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Date: Wed, 25 Mar 2020 15:57:45 -0000 From: Alex Williamson To: qemu-devel@nongnu.org X-Launchpad-Notification-Type: bug X-Launchpad-Bug: product=qemu; status=New; importance=Undecided; assignee=None; X-Launchpad-Bug-Tags: arm passthrough tcg X-Launchpad-Bug-Information-Type: Public X-Launchpad-Bug-Private: no X-Launchpad-Bug-Security-Vulnerability: no X-Launchpad-Bug-Commenters: alex-l-williamson hrw pmaydell X-Launchpad-Bug-Reporter: Marcin Juszkiewicz (hrw) X-Launchpad-Bug-Modifier: Alex Williamson (alex-l-williamson) References: <158514404728.11288.8869885318197124821.malonedeb@soybean.canonical.com> Message-Id: <158515186595.7043.14755579395187622825.malone@wampee.canonical.com> Subject: [Bug 1869006] Re: PCIe cards passthrough to TCG guest works on 2GB of guest memory but fails on 4GB (vfio_dma_map invalid arg) X-Launchpad-Message-Rationale: Subscriber (QEMU) @qemu-devel-ml X-Launchpad-Message-For: qemu-devel-ml Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="3a6db24bbe7280ec09bae73384238390fcc98ad3"; Instance="production-secrets-lazr.conf" X-Launchpad-Hash: 3669ff8b0930c9b07a9662366153419a95827882 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 91.189.90.7 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Bug 1869006 <1869006@bugs.launchpad.net> Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is not related to the BARs, the mapping of the BARs into the guest is purely virtual and controlled by the guest. The issue is that the device needs to be able to DMA into guest RAM, and to do that transparently (ie. the guest doesn't know it's being virtualized), we need to map GPAs into the host IOMMU such that the guest interacts with the device in terms of GPAs, the host IOMMU translates that to HPAs. Thus the IOMMU needs to support GPA range of the guest as IOVA. However, there are ranges of IOVA space that the host IOMMU cannot map, for example the MSI range here is handled by the interrupt remmapper, not the DMA translation portion of the IOMMU (on physical ARM systems these are one-in-the-same, on x86 they are different components, using different mapping interfaces of the IOMMU). Therefore if the guest programmed the device to perform a DMA to 0xfee00000, the host IOMMU would see that as an MSI, not a DMA. When we do an x86 VM on and x86 host, both the host and the guest have complimentary reserved regions, which avoids this issue. Also, to expand on what I mentioned on IRC, every x86 host is going to have some reserved range below 4G for this purpose, but if the aarch64 VM has no requirements for memory below 4G, the starting GPA for the VM could be at or above 4G and avoid this issue. -- = You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1869006 Title: PCIe cards passthrough to TCG guest works on 2GB of guest memory but fails on 4GB (vfio_dma_map invalid arg) Status in QEMU: New Bug description: During one meeting coworker asked "did someone tried to passthrough PCIe card to other arch guest?" and I decided to check it. Plugged SATA and USB3 controllers into spare slots on mainboard and started playing. On 1GB VM instance it worked (both cold- and hot- plugged). On 4GB one it did not: B=C5=82=C4=85d podczas uruchamiania domeny: internal error: process exite= d while connecting to monitor: 2020-03-25T13:43:39.107524Z qemu-system-aarc= h64: -device vfio-pci,host=3D0000:29:00.0,id=3Dhostdev0,bus=3Dpci.3,addr=3D= 0x0: VFIO_MAP_DMA: -22 2020-03-25T13:43:39.107560Z qemu-system-aarch64: -device vfio-pci,host=3D= 0000:29:00.0,id=3Dhostdev0,bus=3Dpci.3,addr=3D0x0: vfio 0000:29:00.0: faile= d to setup container for group 28: memory listener initialization failed: R= egion mach-virt.ram: vfio_dma_map(0x563169753c80, 0x40000000, 0x100000000, = 0x7fb2a3e00000) =3D -22 (Invalid argument) Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_= wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tm= pcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", lin= e 66, in newfn ret =3D fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/object/domain.py", line 1279,= in startup self._backend.create() File "/usr/lib64/python3.8/site-packages/libvirt.py", line 1234, in cre= ate if ret =3D=3D -1: raise libvirtError ('virDomainCreate() failed', dom= =3Dself) libvirt.libvirtError: internal error: process exited while connecting to = monitor: 2020-03-25T13:43:39.107524Z qemu-system-aarch64: -device vfio-pci,= host=3D0000:29:00.0,id=3Dhostdev0,bus=3Dpci.3,addr=3D0x0: VFIO_MAP_DMA: -22 2020-03-25T13:43:39.107560Z qemu-system-aarch64: -device vfio-pci,host=3D= 0000:29:00.0,id=3Dhostdev0,bus=3Dpci.3,addr=3D0x0: vfio 0000:29:00.0: faile= d to setup container for group 28: memory listener initialization failed: R= egion mach-virt.ram: vfio_dma_map(0x563169753c80, 0x40000000, 0x100000000, = 0x7fb2a3e00000) =3D -22 (Invalid argument) = I played with memory and 3054 MB is maximum value possible to boot VM wit= h coldplugged host PCIe cards. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1869006/+subscriptions