From mboxrd@z Thu Jan 1 00:00:00 1970 From: Suravee Suthikulanit Subject: Re: [PATCH for-4.6 0/5] xen: arm: Parse PCI DT nodes' ranges and interrupt-map Date: Tue, 17 Feb 2015 23:28:30 -0600 Message-ID: <54E422FE.4080803@amd.com> References: <1414144694.15687.31.camel@citrix.com> <54E168D6.6000502@amd.com> <54E1C2AA.3070504@linaro.org> <54E2AFCC.3090302@amd.com> <54E34592.2020803@linaro.org> <54E3472D.6030103@citrix.com> <54E3C21A.6080909@amd.com> <54E3DD71.6070001@amd.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <54E3DD71.6070001@amd.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Andrew Cooper , Julien Grall , Ian Campbell , Stefano Stabellini , xen-devel , Jan Beulich List-Id: xen-devel@lists.xenproject.org On 2/17/2015 6:31 PM, Suravee Suthikulanit wrote: > On 2/17/2015 4:35 PM, Suravee Suthikulanit wrote: >> On 2/17/2015 7:50 AM, Andrew Cooper wrote: >>> On 17/02/15 13:43, Julien Grall wrote: >>>> (CC Jan and Andrew) >>>> >>>> Hi Suravee, >>>> >>>> On 17/02/15 03:04, Suravee Suthikulanit wrote: >>>>> By the way, looking at the output of "xl dmesg", I saw the following >>>>> message: >>>>> >>>>> (XEN) DOM0: PCI host bridge /smb/pcie@f0000000 ranges: >>>>> (XEN) DOM0: IO 0xefff0000..0xefffffff -> 0x00000000 >>>>> (XEN) DOM0: MEM 0x40000000..0xbfffffff -> 0x40000000 >>>>> (XEN) DOM0: MEM 0x100000000..0x7fffffffff -> 0x100000000 >>>>> (XEN) DOM0: pci-host-generic f0000000.pcie: PCI host bridge to bus >>>>> 0000:00 >>>>> (XEN) DOM0: pci_bus 0000:00: root bus resource [bus 00-7f] >>>>> (XEN) DOM0: pci_bus 0000:00: root bus resource [io 0x0000-0xffff] >>>>> (XEN) DOM0: pci_bus 0000:00: root bus resource [mem >>>>> 0x40000000-0xbfffffff] >>>>> (XEN) DOM0: pci_bus 0000:00: root bus resource [mem >>>>> 0x100000000-0x7fffffffff] >>>>> (XEN) DOM0: pci 0000:00:00.0: of_irq_parse_pci() failed with rc=-19 >>>>> (XEN) do_physdev_op 16 cmd=25: not implemented yet >>>>> (XEN) do_physdev_op 16 cmd=15: not implemented yet >>>>> (XEN) DOM0: pci 0000:00:00.0: Failed to add - passthrough or MSI/MSI-X >>>>> might fail! >>>>> (XEN) DOM0: pci 0000:00:02.0: of_irq_parse_pci() failed with rc=-19 >>>>> (XEN) do_physdev_op 16 cmd=15: not implemented yet >>>>> (XEN) DOM0: pci 0000:00:02.0: Failed to add - passthrough or MSI/MSI-X >>>>> might fail! >>>>> (XEN) do_physdev_op 16 cmd=15: not implemented yet >>>>> (XEN) DOM0: pci 0000:00:02.1: Failed to add - passthrough or MSI/MSI-X >>>>> might fail! >>>>> (XEN) do_physdev_op 16 cmd=15: not implemented yet >>>>> (XEN) DOM0: pci 0000:01:00.0: Failed to add - passthrough or MSI/MSI-X >>>>> might fail! >>>>> (XEN) DOM0: pci 0000:01:00.1: of_irq_parse_pci() failed with rc=-22 >>>>> (XEN) do_physdev_op 16 cmd=15: not implemented yet >>>>> (XEN) DOM0: pci 0000:01:00.1: Failed to add - passthrough or MSI/MSI-X >>>>> might fail! >>>>> >>>>> IIUC, This is because xen_add_device() failed, and it seems to be >>>>> related to some hyper call not implemented. Not sure what is "cmd=15". >>>>> Any ideas? >>>> There is 2 commands not implemented in the log: >>>> * cmd 15: PHYSDEVOP_manage_pci_add >>>> * cmd 25: PHYSDEVOP_pci_device_add >>>> >>>> Linux fallbacks on the former because the later is not implemented. >>>> >>>> AFAICT, PHYSDEVOP_manage_pci_add should not be implemented for ARM >>>> because it doesn't support segment. I suspect that it's kept for legacy >>>> on older Xen x86. Maybe Jan or Andrew have more input on this? >>> >>> It needs to be kept for backwards compatibility in x86. >>> >>> All new code should use PHYSDEVOP_pci_device_add. >>> >>> ~Andrew >>> >> >> Ok, now that I look at the arch/arm/physdev.c, I don't think the code >> for supporting any of the PHYSDEVOP_xxx is there. That's probably why >> Xen complains. In contrast, arch/x86/physdev.c has most PHYSDEVOP_xxx >> already supported. >> >> My question is, are we supposed to be adding code to put the support in >> here? >> >> Thanks, >> >> Suravee. > > My guess is yes, and that would mean we need to enable building > drivers/pci.c when building arm code, which then open up a can of worms > with re-factoring MSI support code from x86 and etc. > > Suravee Actually, that seems to be more related to the PCI pass-through devices. Isn't the Cavium guys already done that work to support their PCI device pass-through? Anyways, at this point, I am able to generated Dom0 device tree with correct v2m node, and I can see Dom0 gicv2m driver probing and initializing correctly as it would on bare-metal. # Snippet from /sys/firmware/fdt showing dom0 GIC node interrupt-controller { compatible = "arm,gic-400", "arm,cortex-a15-gic"; #interrupt-cells = <0x3>; interrupt-controller; reg = <0x0 0xe1110000 0x0 0x1000 0x0 0xe112f000 0x0 0x2000>; phandle = <0x1>; #address-cells = <0x2>; #size-cells = <0x2>; ranges = <0x0 0x0 0x0 0xe1100000 0x0 0x100000>; v2m { compatible = "arm,gic-v2m-frame"; msi-controller; arm,msi-base-spi = <0x40>; arm,msi-num-spis = <0x100>; phandle = <0x5>; reg = <0x0 0x80000 0x0 0x1000>; }; }; linux:~ # dmesg | grep v2m [ 0.000000] GICv2m: Overriding V2M MSI_TYPER (base:64, num:256) [ 0.000000] GICv2m: Node v2m: range[0xe1180000:0xe1180fff], SPI[64:320] So, during setting up v2m in hypervisor, I also call route_irq_to_guest() for the all SPIs used for MSI (i.e. 64-320 on Seattle), which will force the MSIs to Dom0. However, we would need to figure out how to detach and re-route certain interrupts to a specific DomU in case of passing through PCI devices in the future. So, here's what I got: linux:~ # cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 0: 0 0 0 0 0 0 xen-dyn-event xenbus 3: 19872 19872 19870 19861 19866 20034 GIC 27 arch_timer 4: 91 1 1 1 1 1 GIC 31 events 13: 2397 0 0 0 0 0 GIC 387 e0300000.sata 14: 0 0 0 0 0 0 GIC 389 e1000000.i2c 15: 0 0 0 0 0 0 GIC 362 pl022 16: 0 0 0 0 0 0 GIC 361 pl022 46: 90 0 0 0 0 0 xen-percpu-virq hvc_console 47: 200 0 0 0 0 0 MSI 524288 enp1s0f0-TxRx-0 48: 56 0 0 0 0 0 MSI 524289 enp1s0f0-TxRx-1 49: 47 0 0 0 0 0 MSI 524290 enp1s0f0-TxRx-2 50: 45 0 0 0 0 0 MSI 524291 enp1s0f0-TxRx-3 51: 53 0 0 0 0 0 MSI 524292 enp1s0f0-TxRx-4 52: 48 0 0 0 0 0 MSI 524293 enp1s0f0-TxRx-5 53: 4 0 0 0 0 0 MSI 524294 enp1s0f0 linux:~ # ifconfig enp1s0f0 enp1s0f0 Link encap:Ethernet HWaddr 00:1B:21:55:7F:14 inet addr:10.236.19.5 Bcast:10.236.19.255 Mask:255.255.254.0 inet6 addr: fe80::21b:21ff:fe55:7f14/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:333 errors:0 dropped:0 overruns:0 frame:0 TX packets:111 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:33406 (32.6 Kb) TX bytes:14957 (14.6 Kb) linux:~ # lspci -vvv -s 1:0.0 01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-