From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ravi Kerur Subject: Re: IXGBE, IOMMU DMAR DRHD handling fault issue Date: Fri, 2 Feb 2018 12:21:42 -0800 Message-ID: References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <10911b54-57ee-370b-a4f7-f34accf4811e@intel.com> <3464b900-8648-c128-e959-dc60a8883a2d@intel.com> <0120c68f-cf42-5d8b-5600-0514f76209b0@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Cc: dev@dpdk.org To: "Burakov, Anatoly" Return-path: Received: from mail-lf0-f53.google.com (mail-lf0-f53.google.com [209.85.215.53]) by dpdk.org (Postfix) with ESMTP id 489A81E34 for ; Fri, 2 Feb 2018 21:21:44 +0100 (CET) Received: by mail-lf0-f53.google.com with SMTP id a204so33263611lfa.2 for ; Fri, 02 Feb 2018 12:21:44 -0800 (PST) In-Reply-To: <0120c68f-cf42-5d8b-5600-0514f76209b0@intel.com> List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly wrote: > On 01-Feb-18 7:26 PM, Ravi Kerur wrote: > >> >> >> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly < >> anatoly.burakov@intel.com > wrote: >> >> On 31-Jan-18 9:51 PM, Ravi Kerur wrote: >> >> >> Hi Anatoly, >> >> Thanks. I am following wiki link below which uses vIOMMU with >> DPDK as a use-case and instantiate VM as specified with Q35 >> chipset in Qemu. >> >> https://wiki.qemu.org/Features/VT-d >> >> >> Qemu-version is 2.11 >> Host kernel 4.9 >> Guest kernel 4.4 >> >> I can only guess that guest kernel needs an upgrade in my setup >> to work correctly, if versions on my setup rings a bell on not >> having support kindly let me know. >> >> When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on >> guest I get following error >> ... >> vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored >> ... >> >> in guest. >> >> Thanks. >> >> >> AFAIK kernel 4.4 should have noiommu mode - it was introduced in >> 3.1x days. However, in order for that to work, kernel also has to be >> built with this mode enabled. My guess is, whoever is the supplier >> of your kernel, did not do that. You should double-check the kernel >> configuration of your distribution. >> >> However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode >> - "regular" vfio should work fine. noiommu mode should only be >> needed if you know you don't have IOMMU enabled in your kernel, and >> even if you can't enable it, you can still use igb_uio. >> >> Hi Anatoly, >> >> Do you suggest I take this discussion to kvm/qemu mailing list as I am >> not sure which component has the issue? I check dmesg for BIOS physical >> memory map and address reported as fault by DMAR is reported by BIOS as >> usable on both host and vm. >> >> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr >> *33a128000 *[fault reason 06] PTE Read access is not set >> >> dmesg | grep BIOS >> [ 0.000000] e820: BIOS-provided physical RAM map: >> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] >> ACPI data >> [ 0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] >> ACPI NVS >> [ 0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] >> reserved >> [* 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] >> usable* >> * >> * >> Kindly let me know your inputs. >> >> Thanks. >> >> >> -- Thanks, >> Anatoly >> >> >> > The "PTE Read not set" error usually indicates that you are trying to use > a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio > when IOMMU is on). That, to me, indicates that you do have IOMMU emulation > enabled. > > I would go about it this way. > > First, i'd ensure that your VM has IOMMU emulation enabled and working. > You have mentioned that your QEMU version should have IOMMU emulation, so > let's assume that's the case. > > I am not sure of the exact command-line needed to activate the vIOMMU > emulation, but assuming your VM emulates an Intel processor, your kernel > command-line should have "iommu=on intel_iommu=on" in it. Check > /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above > values are not in there, add the above changes, do "update-grub" and reboot > your VM. > > If it already did have the necessary kernel configuration, do "dmesg | > grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU > is enabled and working in the kernel. > > After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it > should be working. Please bear in mind that all of that is how i would've > gone about it if i had similar problems on baremetal, but i'm hoping all of > it is applicable to VM's. So, either disable IOMMU and use igb_uio, or > enable IOMMU and use VFIO. Both will work. > > Thanks for your help so far on this. I have everything enabled as you have mentioned in your email. Below is the error from EAL when DPDK is run on a VM. I am referring EAL code to check why it is failing. ************VM DPDK errors*********** EAL: Detected 14 lcore(s) EAL: Probing VFIO support... EAL: VFIO support initialized EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: PCI device 0000:00:07.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 8086:10ed net_ixgbe_vf EAL: using IOMMU type 1 (Type 1) *EAL: RTE_IOVA_VA 140051367329792EAL: cannot set up DMA remapping, error 14 (Bad address)EAL: 0000:00:07.0 DMA remapping failed, error 14 (Bad address)* EAL: Requested device 0000:00:07.0 cannot be used EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: PCI device 0000:04:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: PCI device 0000:05:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: No probed ethernet devices Interactive-mode selected USER1: create a new mbuf pool : n=171456, size=2176, socket=0 ********************VM************ root@sriov-ravi-01:~# lsmod | grep vfio *vfio_pci 40960 1vfio_virqfd 16384 1 vfio_pcivfio_iommu_type1 20480 1vfio 28672 5 vfio_iommu_type1,vfio_pciirqbypass 16384 3 kvm,vfio_pci* root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR 00000001 BXPC 00000001) *[ 0.000000] DMAR: IOMMU enabled* [ 9.423839] DMAR: Host address width 39 [ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 [ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 12008c22260286 ecap f00f5e [ 9.427314] DMAR: ATSR flags: 0x1 [ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 [ 9.429255] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 11.167334] DMAR: No RMRR found [ 11.168855] DMAR: dmar0: Using Queued invalidation [ 12.119060] DMAR: Setting RMRR: [ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC [ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O root@sriov-ravi-01:~# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=* Network devices using kernel driver =================================== 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci Other Network devices ===================== Crypto devices using DPDK-compatible driver =========================================== Crypto devices using kernel driver ================================== Other Crypto devices ==================== Eventdev devices using DPDK-compatible driver ============================================= Eventdev devices using kernel driver ==================================== Other Eventdev devices ====================== Mempool devices using DPDK-compatible driver ============================================ Mempool devices using kernel driver =================================== Other Mempool devices ===================== root@sriov-ravi-01:~# root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i vfio [ 77.676711] VFIO - User Level meta-driver version: 0.3 [ 149.013806] VFIO - User Level meta-driver version: 0.3 [ 275.876440] VFIO - User Level meta-driver version: 0.3 Thanks, Ravi -- > Thanks, > Anatoly >