From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35537) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dcUvo-0002dv-VL for qemu-devel@nongnu.org; Tue, 01 Aug 2017 07:01:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dcUvj-0002J4-Il for qemu-devel@nongnu.org; Tue, 01 Aug 2017 07:01:45 -0400 References: <1499633493-19865-1-git-send-email-eric.auger@redhat.com> From: Tomasz Nowicki Message-ID: Date: Tue, 1 Aug 2017 13:01:18 +0200 MIME-Version: 1.0 In-Reply-To: <1499633493-19865-1-git-send-email-eric.auger@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC v5 0/8] ARM SMMUv3 Emulation Support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Eric Auger , eric.auger.pro@gmail.com, peter.maydell@linaro.org, qemu-arm@nongnu.org, qemu-devel@nongnu.org, alex.williamson@redhat.com, prem.mallappa@gmail.com Cc: drjones@redhat.com, christoffer.dall@linaro.org, Radha.Chintakuntla@cavium.com, Sunil.Goutham@cavium.com, mohun106@gmail.com, tcain@qti.qualcomm.com, bharat.bhushan@nxp.com, mst@redhat.com, will.deacon@arm.com, jean-philippe.brucker@arm.com, robin.murphy@arm.com, peterx@redhat.com, edgar.iglesias@gmail.com, "Nair, Jayachandran" Hi Eric, Just letting you know that I am facing another issue with the following setup: 1. host (4.12 kernel & 64K page) and VM (4.12 kernel & 64K page) 2. QEMU + -netdev type=tap,ifname=tap,id=net0 -device virtio-net-pci,netdev=net0,iommu_platform,disable-modern=off,disable-legacy=on 2. On VM, I allocate some huge pages and run DPDK testpmd app: # echo 4 > /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages # ./dpdk/usertools/dpdk-devbind.py -b vfio-pci 0000:00:02.0 # ./dpdk/build/app/testpmd -l 0-13 -n 4 -w 0000:00:02.0 -- --disable-hw-vlan-filter --disable-rss -i EAL: Detected 14 lcore(s) EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:00:02.0 on NUMA socket -1 EAL: probe driver: 1af4:1041 net_virtio EAL: using IOMMU type 1 (Type 1) EAL: iommu_map_dma vaddr ffff20000000 size 80000000 iova 120000000 EAL: Can't write to PCI bar (0) : offset (12) EAL: Can't read from PCI bar (0) : offset (12) EAL: Can't read from PCI bar (0) : offset (12) EAL: Can't write to PCI bar (0) : offset (12) EAL: Can't read from PCI bar (0) : offset (12) EAL: Can't write to PCI bar (0) : offset (12) EAL: Can't read from PCI bar (0) : offset (0) EAL: Can't write to PCI bar (0) : offset (4) EAL: Can't write to PCI bar (0) : offset (14) EAL: Can't write to PCI bar (0) : offset (e) EAL: Can't read from PCI bar (0) : offset (c) EAL: Requested device 0000:00:02.0 cannot be used EAL: No probed ethernet devices Interactive-mode selected USER1: create a new mbuf pool : n=251456, size=2176, socket=0 When VM uses *4K pages* the same setup works fine. I will work on this but please let me know in case you already know what is going on. Thanks, Tomasz On 09.07.2017 22:51, Eric Auger wrote: > This series implements the emulation code for ARM SMMUv3. > This is the continuation of Prem's work [1]. > > This v5 mainly brings VFIO integration in DT mode. On guest kernel > side, this requires a quirk [1] to force TLB invalidation on map. > > The following changes also are noticeable: > - fix SMMU_CMDQ_CONS offset > - adds dma-coherent dt property which fixes the unhandled command > opcode bug. > - implements block PTE > > The smmu is instantiated when passing the smmu option to machvirt: > "-M virt-2.10,smmu" > > As I haven't split the code yet so that it can be easily reviewable > I don't expect deep reviews at this stage. Also the implementation may > be largely sub-optimal. > > Tested Use Cases: > - booted a guest in dt and acpi mode with an iommu_platform > virtio-net-pci device (using dma ops). Tested with the following > guest combinations: 4K page - 39 bit VA, 4K - 48b, 64K - 39b, > 64K - 48b. > - booted a guest (featuring [1]) with PCIe passthrough'ed PCIe devices: > - AMD Overdrive and igbvf passthrough (using gsi direct mapping) > - Cavium ThunderX and ixgbevf passthrough (using KVM MSI routing) > > Unfortunately I have not been able to run DPDK testpmd yet on guest side. > The problem I see is the user space driver dma-maps a huge area > and this causes plenty of CMDQ_OP_TLBI_NH_VA commands to be sent > (tlbi-on-map) which are sent for each page whereas the dma-map covers a > huge page. I will work on this issue for next version. > > Known limitations: > - no VMSAv8-32 suport > - no nested stage support (S1 + S2) > - no support for HYP mappings > - register fine emulation, commands, interrupts and errors were > not accurately tested. Handling is sufficient to run use cases > described hereafter though. > > Best Regards > > Eric > > This series can be found at: > v5: https://github.com/eauger/qemu/tree/v2.9-SMMU-v5 > v4: https://github.com/eauger/qemu/tree/v2.9-SMMU-v4 > > References: > [1] [RFC 0/2] arm-smmu-v3 tlbi-on-map option > [2] Prem's last iteration: > - https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg03531.html > > History: > v4 -> v5: > - initial_level now part of SMMUTransCfg > - smmu_page_walk_64 takes into account the max input size > - implement sys->iommu_ops.replay and sys->iommu_ops.notify_flag_changed > - smmuv3_translate: bug fix: don't walk on bypass > - smmu_update_qreg: fix PROD index update > - I did not yet address Peter's comments as the code is not mature enough > to be split into sub patches. > > v3 -> v4 [Eric]: > - page table walk rewritten to allow scan of the page table within a > range of IOVA. This prepares for VFIO integration and replay. > - configuration parsing partially reworked. > - do not advertise unsupported/untested features: S2, S1 + S2, HYP, > PRI, ATS, .. > - added ACPI table generation > - migrated to dynamic traces > - mingw compilation fix > > v2 -> v3 [Eric]: > - rebased on 2.9 > - mostly code and patch reorganization to ease the review process > - optional patches removed. They may be handled separately. I am currently > working on ACPI enablement. > - optional instantiation of the smmu in mach-virt > - removed [2/9] (fdt functions) since not mandated > - start splitting main patch into base and derived object > - no new function feature added > > v1 -> v2 [Prem]: > - Adopted review comments from Eric Auger > - Make SMMU_DPRINTF to internally call qemu_log > (since translation requests are too many, we need control > on the type of log we want) > - SMMUTransCfg modified to suite simplicity > - Change RegInfo to uint64 register array > - Code cleanup > - Test cleanups > - Reshuffled patches > > v0 -> v1 [Prem]: > - As per SMMUv3 spec 16.0 (only is_ste_consistant() is noticeable) > - Reworked register access/update logic > - Factored out translation code for > - single point bug fix > - sharing/removal in future > - (optional) Unit tests added, with PCI test device > - S1 with 4k/64k, S1+S2 with 4k/64k > - (S1 or S2) only can be verified by Linux 4.7 driver > - (optional) Priliminary ACPI support > > v0 [Prem]: > - Implements SMMUv3 spec 11.0 > - Supported for PCIe devices, > - Command Queue and Event Queue supported > - LPAE only, S1 is supported and Tested, S2 not tested > - BE mode Translation not supported > - IRQ support (legacy, no MSI) > - Tested with DPDK and e1000 > > > Eric Auger (5): > hw/arm/smmu-common: smmu base class > hw/arm/virt: Add 2.10 machine type > hw/arm/virt: Add tlbi-on-map property to the smmuv3 node > target/arm/kvm: Translate the MSI doorbell in kvm_arch_fixup_msi_route > hw/arm/smmuv3: VFIO integration > > Prem Mallappa (3): > hw/arm/smmuv3: smmuv3 emulation model > hw/arm/virt: Add SMMUv3 to the virt board > hw/arm/virt-acpi-build: Add smmuv3 node in IORT table > > default-configs/aarch64-softmmu.mak | 1 + > hw/arm/Makefile.objs | 1 + > hw/arm/smmu-common.c | 474 +++++++++++++ > hw/arm/smmu-internal.h | 89 +++ > hw/arm/smmuv3-internal.h | 651 ++++++++++++++++++ > hw/arm/smmuv3.c | 1256 +++++++++++++++++++++++++++++++++++ > hw/arm/trace-events | 54 ++ > hw/arm/virt-acpi-build.c | 56 +- > hw/arm/virt.c | 111 +++- > include/hw/acpi/acpi-defs.h | 15 + > include/hw/arm/smmu-common.h | 127 ++++ > include/hw/arm/smmuv3.h | 87 +++ > include/hw/arm/virt.h | 5 + > target/arm/kvm.c | 28 + > target/arm/trace-events | 3 + > 15 files changed, 2949 insertions(+), 9 deletions(-) > create mode 100644 hw/arm/smmu-common.c > create mode 100644 hw/arm/smmu-internal.h > create mode 100644 hw/arm/smmuv3-internal.h > create mode 100644 hw/arm/smmuv3.c > create mode 100644 include/hw/arm/smmu-common.h > create mode 100644 include/hw/arm/smmuv3.h >