xen does not support the 8G large bar

xen does not support the 8G large bar

[Gaofeng (GaoFeng, Euler)]

[-- Attachment #1.1: Type: text/plain, Size: 2912 bytes --]

Hi George,

I found that you have submitted a patch about "libxl,hvmloader: Don't relocate memory for MMIO hole". So, I have a "8G large bar PCI device(NVIDIA Tesla M60)" passthrough question to ask you.



Host passthrough PCI Device info:
lspci -vs 06:00.0
06:00.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
        Subsystem: NVIDIA Corporation Device 115e
        Flags: fast devsel, IRQ 40
        Memory at 93000000 (32-bit, non-prefetchable) [disabled] [size=16M]
        Memory at 23c00000000 (64-bit, prefetchable) [disabled] [size=8G]
        Memory at 23e00000000 (64-bit, prefetchable) [disabled] [size=32M]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [258] #1e
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Kernel driver in use: pciback
        Kernel modules: nvidiafb

Guest passthrough PCI Device info:
lspci -vs 00:05.0
00:05.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
        Subsystem: NVIDIA Corporation Device 115e
        Physical Slot: 5
        Flags: fast devsel, IRQ 36
        Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 200000000 (64-bit, prefetchable) [size=4G]
        Memory at f2000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel modules: nouveau, nvidiafb


passthrough to the guest, the large bar size only has 4G. so, the passthrough pci device does not work, windows 7/2008 R2 64bit vm has the same problem.

.

According to the analysis hvmloader code, find a problem:



if (is_64bar) {

                bar_data_upper = pci_readl(devfn, bar_reg + 4);

                pci_writel(devfn, bar_reg + 4, ~0);

                bar_sz_upper = pci_readl(devfn, bar_reg + 4);

                pci_writel(devfn, bar_reg + 4, bar_data_upper);

                bar_sz = (bar_sz_upper << 32) | bar_sz;

            }

            bar_sz &= ~(bar_sz - 1);



read from the pci device, bar_sz_upper is 0xffffffff, if the bar size is 8G, the bar_sz_upper should be 0xfffffffe.

Have you ever encountered a similar problem?



Env info:

device_model_version: qemu-xen

xen version: xen-4.6.1

Guest: RedHat-6.4-64

Passthrough PCI Device: NVIDIA Tesla M60(compute mode)

CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz



Thanks

Feng

[-- Attachment #1.2: Type: text/html, Size: 13126 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: xen does not support the 8G large bar

[Jan Beulich]

>>> On 11.08.16 at 05:30, <visen.gao@huawei.com> wrote:
> According to the analysis hvmloader code, find a problem:
> 
> if (is_64bar) {
>                 bar_data_upper = pci_readl(devfn, bar_reg + 4);
>                 pci_writel(devfn, bar_reg + 4, ~0);
>                 bar_sz_upper = pci_readl(devfn, bar_reg + 4);
>                 pci_writel(devfn, bar_reg + 4, bar_data_upper);
>                 bar_sz = (bar_sz_upper << 32) | bar_sz;
>             }
>             bar_sz &= ~(bar_sz - 1);
> 
> read from the pci device, bar_sz_upper is 0xffffffff, if the bar size is 8G, 
> the bar_sz_upper should be 0xfffffffe.

But that doesn't indicate a problem with hvmloader. Instead that
tells you that qemu isn't behaving correctly. First thing for you
to do is probably to try a newer qemu, or otherwise see whether
you can figure why qemu sends back 0xffffffff in this case
(perhaps by adding a little bit of logging to the respective code).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

答复: xen does not support the 8G large bar

[Gaofeng (GaoFeng, Euler)]

Hi Jan,
Thanks for your reply.
Qemu-xen seems that has problem for support 8G large Bar.
I think this patch is not perfect: https://xenbits.xen.org/gitweb/?p=qemu-xen.git;a=commitdiff;h=aabc8530c7ba2be89e21463f051056ad7c255e6e
Because I found upper 32bit bar of 8G large bar was not register.
xen_pt_config_init.c:456  xen_pt_bar_reg_write

case XEN_PT_BAR_FLAG_UPPER:
        bar_emu_mask = XEN_PT_BAR_ALLF;
        bar_ro_mask = r_size ? r_size - 1 : 0;
        break;
r_size is always 0. So, bar_sz_upper is always 0xffffffff, regardless of whether 64 bit BAR size is larger than 4G.
With Konrad 's help , I find a patch at qemu-xen-trad about sorppurt BAR size is larger than 4G, I will analyze this patch and see if it helps to me.

Feng

-----邮件原件-----
发件人: Jan Beulich [mailto:JBeulich@suse.com] 
发送时间: 2016年8月11日 17:35
收件人: Gaofeng (GaoFeng, Euler)
抄送: george.dunlap@eu.citrix.com; Fanhenglong; Hanweidong (Randy); Lianxueguo (Albert); Huangpeng (Peter); xen-devel@lists.xen.org
主题: Re: [Xen-devel] xen does not support the 8G large bar

>>> On 11.08.16 at 05:30, <visen.gao@huawei.com> wrote:
> According to the analysis hvmloader code, find a problem:
> 
> if (is_64bar) {
>                 bar_data_upper = pci_readl(devfn, bar_reg + 4);
>                 pci_writel(devfn, bar_reg + 4, ~0);
>                 bar_sz_upper = pci_readl(devfn, bar_reg + 4);
>                 pci_writel(devfn, bar_reg + 4, bar_data_upper);
>                 bar_sz = (bar_sz_upper << 32) | bar_sz;
>             }
>             bar_sz &= ~(bar_sz - 1);
> 
> read from the pci device, bar_sz_upper is 0xffffffff, if the bar size 
> is 8G, the bar_sz_upper should be 0xfffffffe.

But that doesn't indicate a problem with hvmloader. Instead that tells you that qemu isn't behaving correctly. First thing for you to do is probably to try a newer qemu, or otherwise see whether you can figure why qemu sends back 0xffffffff in this case (perhaps by adding a little bit of logging to the respective code).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: 答复: xen does not support the 8G large bar

[Jan Beulich]

>>> On 12.08.16 at 05:18, <visen.gao@huawei.com> wrote:
> Hi Jan,
> Thanks for your reply.
> Qemu-xen seems that has problem for support 8G large Bar.
> I think this patch is not perfect: 
> https://xenbits.xen.org/gitweb/?p=qemu-xen.git;a=commitdiff;h=aabc8530c7ba2b 
> e89e21463f051056ad7c255e6e
> Because I found upper 32bit bar of 8G large bar was not register.
> xen_pt_config_init.c:456  xen_pt_bar_reg_write
> 
> case XEN_PT_BAR_FLAG_UPPER:
>         bar_emu_mask = XEN_PT_BAR_ALLF;
>         bar_ro_mask = r_size ? r_size - 1 : 0;
>         break;
> r_size is always 0. So, bar_sz_upper is always 0xffffffff, regardless of 
> whether 64 bit BAR size is larger than 4G.

Depending on how qemu sets things up, the issue may simply be on
of wrong types being used: r->size is of type pcibus_t (which I
sincerely hope is a 64-bit type), yet r_size is uint32_t. But then
again r->size being pcibus_t suggests that d->io_regions[] fields
for the upper halves of 64-bit BARs may not get populated at all
(in which case a possible fix would likely be more involved). I'm not
a qemu expert, so I can't easily tell.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: xen does not support the 8G large bar

[Konrad Rzeszutek Wilk]

On August 10, 2016 11:30:37 PM EDT, "Gaofeng (GaoFeng, Euler)" <visen.gao@huawei.com> wrote:
>Hi George,
>
>I found that you have submitted a patch about "libxl,hvmloader: Don't
>relocate memory for MMIO hole". So, I have a "8G large bar PCI
>device(NVIDIA Tesla M60)" passthrough question to ask you.
>
>
>
>Host passthrough PCI Device info:
>lspci -vs 06:00.0
>06:00.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
>        Subsystem: NVIDIA Corporation Device 115e
>        Flags: fast devsel, IRQ 40
>    Memory at 93000000 (32-bit, non-prefetchable) [disabled] [size=16M]
>      Memory at 23c00000000 (64-bit, prefetchable) [disabled] [size=8G]
>     Memory at 23e00000000 (64-bit, prefetchable) [disabled] [size=32M]
>        Capabilities: [60] Power Management version 3
>        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
>        Capabilities: [78] Express Endpoint, MSI 00
>        Capabilities: [100] Virtual Channel
>        Capabilities: [250] Latency Tolerance Reporting
>        Capabilities: [258] #1e
>        Capabilities: [128] Power Budgeting <?>
>        Capabilities: [420] Advanced Error Reporting
>Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024
><?>
>        Capabilities: [900] #19
>        Kernel driver in use: pciback
>        Kernel modules: nvidiafb
>
>Guest passthrough PCI Device info:
>lspci -vs 00:05.0
>00:05.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
>        Subsystem: NVIDIA Corporation Device 115e
>        Physical Slot: 5
>        Flags: fast devsel, IRQ 36
>        Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
>        Memory at 200000000 (64-bit, prefetchable) [size=4G]
>        Memory at f2000000 (64-bit, prefetchable) [size=32M]
>        Capabilities: [60] Power Management version 3
>        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
>        Capabilities: [78] Express Endpoint, MSI 00
>        Kernel modules: nouveau, nvidiafb
>
>
>passthrough to the guest, the large bar size only has 4G. so, the
>passthrough pci device does not work, windows 7/2008 R2 64bit vm has
>the same problem.
>
>.
>
>According to the analysis hvmloader code, find a problem:
>
>
>
>if (is_64bar) {
>
>                bar_data_upper = pci_readl(devfn, bar_reg + 4);
>
>                pci_writel(devfn, bar_reg + 4, ~0);
>
>                bar_sz_upper = pci_readl(devfn, bar_reg + 4);
>
>                pci_writel(devfn, bar_reg + 4, bar_data_upper);
>
>                bar_sz = (bar_sz_upper << 32) | bar_sz;
>
>            }
>
>            bar_sz &= ~(bar_sz - 1);
>
>
>
>read from the pci device, bar_sz_upper is 0xffffffff, if the bar size
>is 8G, the bar_sz_upper should be 0xfffffffe.
>
>Have you ever encountered a similar problem?
>
>

Yes but only if you are using qemu-trad.

You need to have

https://github.com/xenserver/qemu-trad.pg/blob/master/master/0001-Add-64-bit-support-to-QEMU.patch

>
>Env info:
>
>device_model_version: qemu-xen
>
>xen version: xen-4.6.1
>
>Guest: RedHat-6.4-64
>
>Passthrough PCI Device: NVIDIA Tesla M60(compute mode)
>
>CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>
>
>
>Thanks
>
>Feng
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xen.org
>https://lists.xen.org/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: xen does not support the 8G large bar

[Gaofeng (GaoFeng, Euler)]

Hi Konrad,
Thanks for your reply.
I will analyze this patch and if I have other questions, I will discuss with you again.

Feng

On August 10, 2016 11:30:37 PM EDT, "Gaofeng (GaoFeng, Euler)" <visen.gao@huawei.com> wrote:
>Hi George,
>
>I found that you have submitted a patch about "libxl,hvmloader: Don't 
>relocate memory for MMIO hole". So, I have a "8G large bar PCI 
>device(NVIDIA Tesla M60)" passthrough question to ask you.
>
>
>
>Host passthrough PCI Device info:
>lspci -vs 06:00.0
>06:00.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
>        Subsystem: NVIDIA Corporation Device 115e
>        Flags: fast devsel, IRQ 40
>    Memory at 93000000 (32-bit, non-prefetchable) [disabled] [size=16M]
>      Memory at 23c00000000 (64-bit, prefetchable) [disabled] [size=8G]
>     Memory at 23e00000000 (64-bit, prefetchable) [disabled] [size=32M]
>        Capabilities: [60] Power Management version 3
>        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
>        Capabilities: [78] Express Endpoint, MSI 00
>        Capabilities: [100] Virtual Channel
>        Capabilities: [250] Latency Tolerance Reporting
>        Capabilities: [258] #1e
>        Capabilities: [128] Power Budgeting <?>
>        Capabilities: [420] Advanced Error Reporting
>Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 
><?>
>        Capabilities: [900] #19
>        Kernel driver in use: pciback
>        Kernel modules: nvidiafb
>
>Guest passthrough PCI Device info:
>lspci -vs 00:05.0
>00:05.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
>        Subsystem: NVIDIA Corporation Device 115e
>        Physical Slot: 5
>        Flags: fast devsel, IRQ 36
>        Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
>        Memory at 200000000 (64-bit, prefetchable) [size=4G]
>        Memory at f2000000 (64-bit, prefetchable) [size=32M]
>        Capabilities: [60] Power Management version 3
>        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
>        Capabilities: [78] Express Endpoint, MSI 00
>        Kernel modules: nouveau, nvidiafb
>
>
>passthrough to the guest, the large bar size only has 4G. so, the 
>passthrough pci device does not work, windows 7/2008 R2 64bit vm has 
>the same problem.
>
>.
>
>According to the analysis hvmloader code, find a problem:
>
>
>
>if (is_64bar) {
>
>                bar_data_upper = pci_readl(devfn, bar_reg + 4);
>
>                pci_writel(devfn, bar_reg + 4, ~0);
>
>                bar_sz_upper = pci_readl(devfn, bar_reg + 4);
>
>                pci_writel(devfn, bar_reg + 4, bar_data_upper);
>
>                bar_sz = (bar_sz_upper << 32) | bar_sz;
>
>            }
>
>            bar_sz &= ~(bar_sz - 1);
>
>
>
>read from the pci device, bar_sz_upper is 0xffffffff, if the bar size 
>is 8G, the bar_sz_upper should be 0xfffffffe.
>
>Have you ever encountered a similar problem?
>
>

Yes but only if you are using qemu-trad.

You need to have

https://github.com/xenserver/qemu-trad.pg/blob/master/master/0001-Add-64-bit-support-to-QEMU.patch

>
>Env info:
>
>device_model_version: qemu-xen
>
>xen version: xen-4.6.1
>
>Guest: RedHat-6.4-64
>
>Passthrough PCI Device: NVIDIA Tesla M60(compute mode)
>
>CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>
>
>
>Thanks
>
>Feng
>
>
>-----------------------------------------------------------------------
>-
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xen.org
>https://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel