* xen does not support the 8G large bar
@ 2016-08-11 3:30 Gaofeng (GaoFeng, Euler)
2016-08-11 9:35 ` Jan Beulich
2016-08-11 11:29 ` Konrad Rzeszutek Wilk
0 siblings, 2 replies; 6+ messages in thread
From: Gaofeng (GaoFeng, Euler) @ 2016-08-11 3:30 UTC (permalink / raw)
To: george.dunlap
Cc: Lianxueguo (Albert), Fanhenglong, Hanweidong (Randy),
Huangpeng (Peter),
xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 2912 bytes --]
Hi George,
I found that you have submitted a patch about "libxl,hvmloader: Don't relocate memory for MMIO hole". So, I have a "8G large bar PCI device(NVIDIA Tesla M60)" passthrough question to ask you.
Host passthrough PCI Device info:
lspci -vs 06:00.0
06:00.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
Subsystem: NVIDIA Corporation Device 115e
Flags: fast devsel, IRQ 40
Memory at 93000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Memory at 23c00000000 (64-bit, prefetchable) [disabled] [size=8G]
Memory at 23e00000000 (64-bit, prefetchable) [disabled] [size=32M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] #1e
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] #19
Kernel driver in use: pciback
Kernel modules: nvidiafb
Guest passthrough PCI Device info:
lspci -vs 00:05.0
00:05.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
Subsystem: NVIDIA Corporation Device 115e
Physical Slot: 5
Flags: fast devsel, IRQ 36
Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
Memory at 200000000 (64-bit, prefetchable) [size=4G]
Memory at f2000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Kernel modules: nouveau, nvidiafb
passthrough to the guest, the large bar size only has 4G. so, the passthrough pci device does not work, windows 7/2008 R2 64bit vm has the same problem.
.
According to the analysis hvmloader code, find a problem:
if (is_64bar) {
bar_data_upper = pci_readl(devfn, bar_reg + 4);
pci_writel(devfn, bar_reg + 4, ~0);
bar_sz_upper = pci_readl(devfn, bar_reg + 4);
pci_writel(devfn, bar_reg + 4, bar_data_upper);
bar_sz = (bar_sz_upper << 32) | bar_sz;
}
bar_sz &= ~(bar_sz - 1);
read from the pci device, bar_sz_upper is 0xffffffff, if the bar size is 8G, the bar_sz_upper should be 0xfffffffe.
Have you ever encountered a similar problem?
Env info:
device_model_version: qemu-xen
xen version: xen-4.6.1
Guest: RedHat-6.4-64
Passthrough PCI Device: NVIDIA Tesla M60(compute mode)
CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Thanks
Feng
[-- Attachment #1.2: Type: text/html, Size: 13126 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xen does not support the 8G large bar
2016-08-11 3:30 xen does not support the 8G large bar Gaofeng (GaoFeng, Euler)
@ 2016-08-11 9:35 ` Jan Beulich
2016-08-12 3:18 ` 答复: " Gaofeng (GaoFeng, Euler)
2016-08-11 11:29 ` Konrad Rzeszutek Wilk
1 sibling, 1 reply; 6+ messages in thread
From: Jan Beulich @ 2016-08-11 9:35 UTC (permalink / raw)
To: Euler) Gaofeng (GaoFeng
Cc: Hanweidong (Randy), george.dunlap, Huangpeng (Peter),
xen-devel, Lianxueguo (Albert),
Fanhenglong
>>> On 11.08.16 at 05:30, <visen.gao@huawei.com> wrote:
> According to the analysis hvmloader code, find a problem:
>
> if (is_64bar) {
> bar_data_upper = pci_readl(devfn, bar_reg + 4);
> pci_writel(devfn, bar_reg + 4, ~0);
> bar_sz_upper = pci_readl(devfn, bar_reg + 4);
> pci_writel(devfn, bar_reg + 4, bar_data_upper);
> bar_sz = (bar_sz_upper << 32) | bar_sz;
> }
> bar_sz &= ~(bar_sz - 1);
>
> read from the pci device, bar_sz_upper is 0xffffffff, if the bar size is 8G,
> the bar_sz_upper should be 0xfffffffe.
But that doesn't indicate a problem with hvmloader. Instead that
tells you that qemu isn't behaving correctly. First thing for you
to do is probably to try a newer qemu, or otherwise see whether
you can figure why qemu sends back 0xffffffff in this case
(perhaps by adding a little bit of logging to the respective code).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xen does not support the 8G large bar
2016-08-11 3:30 xen does not support the 8G large bar Gaofeng (GaoFeng, Euler)
2016-08-11 9:35 ` Jan Beulich
@ 2016-08-11 11:29 ` Konrad Rzeszutek Wilk
2016-08-12 6:58 ` Gaofeng (GaoFeng, Euler)
1 sibling, 1 reply; 6+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-08-11 11:29 UTC (permalink / raw)
To: Gaofeng (GaoFeng, Euler), george.dunlap
Cc: Lianxueguo (Albert), Fanhenglong, xen-devel, Huangpeng (Peter),
Hanweidong (Randy)
On August 10, 2016 11:30:37 PM EDT, "Gaofeng (GaoFeng, Euler)" <visen.gao@huawei.com> wrote:
>Hi George,
>
>I found that you have submitted a patch about "libxl,hvmloader: Don't
>relocate memory for MMIO hole". So, I have a "8G large bar PCI
>device(NVIDIA Tesla M60)" passthrough question to ask you.
>
>
>
>Host passthrough PCI Device info:
>lspci -vs 06:00.0
>06:00.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
> Subsystem: NVIDIA Corporation Device 115e
> Flags: fast devsel, IRQ 40
> Memory at 93000000 (32-bit, non-prefetchable) [disabled] [size=16M]
> Memory at 23c00000000 (64-bit, prefetchable) [disabled] [size=8G]
> Memory at 23e00000000 (64-bit, prefetchable) [disabled] [size=32M]
> Capabilities: [60] Power Management version 3
> Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
> Capabilities: [78] Express Endpoint, MSI 00
> Capabilities: [100] Virtual Channel
> Capabilities: [250] Latency Tolerance Reporting
> Capabilities: [258] #1e
> Capabilities: [128] Power Budgeting <?>
> Capabilities: [420] Advanced Error Reporting
>Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024
><?>
> Capabilities: [900] #19
> Kernel driver in use: pciback
> Kernel modules: nvidiafb
>
>Guest passthrough PCI Device info:
>lspci -vs 00:05.0
>00:05.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
> Subsystem: NVIDIA Corporation Device 115e
> Physical Slot: 5
> Flags: fast devsel, IRQ 36
> Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
> Memory at 200000000 (64-bit, prefetchable) [size=4G]
> Memory at f2000000 (64-bit, prefetchable) [size=32M]
> Capabilities: [60] Power Management version 3
> Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
> Capabilities: [78] Express Endpoint, MSI 00
> Kernel modules: nouveau, nvidiafb
>
>
>passthrough to the guest, the large bar size only has 4G. so, the
>passthrough pci device does not work, windows 7/2008 R2 64bit vm has
>the same problem.
>
>.
>
>According to the analysis hvmloader code, find a problem:
>
>
>
>if (is_64bar) {
>
> bar_data_upper = pci_readl(devfn, bar_reg + 4);
>
> pci_writel(devfn, bar_reg + 4, ~0);
>
> bar_sz_upper = pci_readl(devfn, bar_reg + 4);
>
> pci_writel(devfn, bar_reg + 4, bar_data_upper);
>
> bar_sz = (bar_sz_upper << 32) | bar_sz;
>
> }
>
> bar_sz &= ~(bar_sz - 1);
>
>
>
>read from the pci device, bar_sz_upper is 0xffffffff, if the bar size
>is 8G, the bar_sz_upper should be 0xfffffffe.
>
>Have you ever encountered a similar problem?
>
>
Yes but only if you are using qemu-trad.
You need to have
https://github.com/xenserver/qemu-trad.pg/blob/master/master/0001-Add-64-bit-support-to-QEMU.patch
>
>Env info:
>
>device_model_version: qemu-xen
>
>xen version: xen-4.6.1
>
>Guest: RedHat-6.4-64
>
>Passthrough PCI Device: NVIDIA Tesla M60(compute mode)
>
>CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>
>
>
>Thanks
>
>Feng
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xen.org
>https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* 答复: xen does not support the 8G large bar
2016-08-11 9:35 ` Jan Beulich
@ 2016-08-12 3:18 ` Gaofeng (GaoFeng, Euler)
2016-08-12 11:24 ` Jan Beulich
0 siblings, 1 reply; 6+ messages in thread
From: Gaofeng (GaoFeng, Euler) @ 2016-08-12 3:18 UTC (permalink / raw)
To: Jan Beulich
Cc: Hanweidong (Randy), george.dunlap, Huangpeng (Peter),
xen-devel, Lianxueguo (Albert),
Fanhenglong
Hi Jan,
Thanks for your reply.
Qemu-xen seems that has problem for support 8G large Bar.
I think this patch is not perfect: https://xenbits.xen.org/gitweb/?p=qemu-xen.git;a=commitdiff;h=aabc8530c7ba2be89e21463f051056ad7c255e6e
Because I found upper 32bit bar of 8G large bar was not register.
xen_pt_config_init.c:456 xen_pt_bar_reg_write
case XEN_PT_BAR_FLAG_UPPER:
bar_emu_mask = XEN_PT_BAR_ALLF;
bar_ro_mask = r_size ? r_size - 1 : 0;
break;
r_size is always 0. So, bar_sz_upper is always 0xffffffff, regardless of whether 64 bit BAR size is larger than 4G.
With Konrad 's help , I find a patch at qemu-xen-trad about sorppurt BAR size is larger than 4G, I will analyze this patch and see if it helps to me.
Feng
-----邮件原件-----
发件人: Jan Beulich [mailto:JBeulich@suse.com]
发送时间: 2016年8月11日 17:35
收件人: Gaofeng (GaoFeng, Euler)
抄送: george.dunlap@eu.citrix.com; Fanhenglong; Hanweidong (Randy); Lianxueguo (Albert); Huangpeng (Peter); xen-devel@lists.xen.org
主题: Re: [Xen-devel] xen does not support the 8G large bar
>>> On 11.08.16 at 05:30, <visen.gao@huawei.com> wrote:
> According to the analysis hvmloader code, find a problem:
>
> if (is_64bar) {
> bar_data_upper = pci_readl(devfn, bar_reg + 4);
> pci_writel(devfn, bar_reg + 4, ~0);
> bar_sz_upper = pci_readl(devfn, bar_reg + 4);
> pci_writel(devfn, bar_reg + 4, bar_data_upper);
> bar_sz = (bar_sz_upper << 32) | bar_sz;
> }
> bar_sz &= ~(bar_sz - 1);
>
> read from the pci device, bar_sz_upper is 0xffffffff, if the bar size
> is 8G, the bar_sz_upper should be 0xfffffffe.
But that doesn't indicate a problem with hvmloader. Instead that tells you that qemu isn't behaving correctly. First thing for you to do is probably to try a newer qemu, or otherwise see whether you can figure why qemu sends back 0xffffffff in this case (perhaps by adding a little bit of logging to the respective code).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: xen does not support the 8G large bar
2016-08-11 11:29 ` Konrad Rzeszutek Wilk
@ 2016-08-12 6:58 ` Gaofeng (GaoFeng, Euler)
0 siblings, 0 replies; 6+ messages in thread
From: Gaofeng (GaoFeng, Euler) @ 2016-08-12 6:58 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, george.dunlap
Cc: Lianxueguo (Albert), Fanhenglong, xen-devel, Huangpeng (Peter),
Hanweidong (Randy)
Hi Konrad,
Thanks for your reply.
I will analyze this patch and if I have other questions, I will discuss with you again.
Feng
On August 10, 2016 11:30:37 PM EDT, "Gaofeng (GaoFeng, Euler)" <visen.gao@huawei.com> wrote:
>Hi George,
>
>I found that you have submitted a patch about "libxl,hvmloader: Don't
>relocate memory for MMIO hole". So, I have a "8G large bar PCI
>device(NVIDIA Tesla M60)" passthrough question to ask you.
>
>
>
>Host passthrough PCI Device info:
>lspci -vs 06:00.0
>06:00.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
> Subsystem: NVIDIA Corporation Device 115e
> Flags: fast devsel, IRQ 40
> Memory at 93000000 (32-bit, non-prefetchable) [disabled] [size=16M]
> Memory at 23c00000000 (64-bit, prefetchable) [disabled] [size=8G]
> Memory at 23e00000000 (64-bit, prefetchable) [disabled] [size=32M]
> Capabilities: [60] Power Management version 3
> Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
> Capabilities: [78] Express Endpoint, MSI 00
> Capabilities: [100] Virtual Channel
> Capabilities: [250] Latency Tolerance Reporting
> Capabilities: [258] #1e
> Capabilities: [128] Power Budgeting <?>
> Capabilities: [420] Advanced Error Reporting
>Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024
><?>
> Capabilities: [900] #19
> Kernel driver in use: pciback
> Kernel modules: nvidiafb
>
>Guest passthrough PCI Device info:
>lspci -vs 00:05.0
>00:05.0 3D controller: NVIDIA Corporation Device 13f2 (rev a1)
> Subsystem: NVIDIA Corporation Device 115e
> Physical Slot: 5
> Flags: fast devsel, IRQ 36
> Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
> Memory at 200000000 (64-bit, prefetchable) [size=4G]
> Memory at f2000000 (64-bit, prefetchable) [size=32M]
> Capabilities: [60] Power Management version 3
> Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
> Capabilities: [78] Express Endpoint, MSI 00
> Kernel modules: nouveau, nvidiafb
>
>
>passthrough to the guest, the large bar size only has 4G. so, the
>passthrough pci device does not work, windows 7/2008 R2 64bit vm has
>the same problem.
>
>.
>
>According to the analysis hvmloader code, find a problem:
>
>
>
>if (is_64bar) {
>
> bar_data_upper = pci_readl(devfn, bar_reg + 4);
>
> pci_writel(devfn, bar_reg + 4, ~0);
>
> bar_sz_upper = pci_readl(devfn, bar_reg + 4);
>
> pci_writel(devfn, bar_reg + 4, bar_data_upper);
>
> bar_sz = (bar_sz_upper << 32) | bar_sz;
>
> }
>
> bar_sz &= ~(bar_sz - 1);
>
>
>
>read from the pci device, bar_sz_upper is 0xffffffff, if the bar size
>is 8G, the bar_sz_upper should be 0xfffffffe.
>
>Have you ever encountered a similar problem?
>
>
Yes but only if you are using qemu-trad.
You need to have
https://github.com/xenserver/qemu-trad.pg/blob/master/master/0001-Add-64-bit-support-to-QEMU.patch
>
>Env info:
>
>device_model_version: qemu-xen
>
>xen version: xen-4.6.1
>
>Guest: RedHat-6.4-64
>
>Passthrough PCI Device: NVIDIA Tesla M60(compute mode)
>
>CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>
>
>
>Thanks
>
>Feng
>
>
>-----------------------------------------------------------------------
>-
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xen.org
>https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 答复: xen does not support the 8G large bar
2016-08-12 3:18 ` 答复: " Gaofeng (GaoFeng, Euler)
@ 2016-08-12 11:24 ` Jan Beulich
0 siblings, 0 replies; 6+ messages in thread
From: Jan Beulich @ 2016-08-12 11:24 UTC (permalink / raw)
To: Euler) Gaofeng (GaoFeng
Cc: Hanweidong (Randy), george.dunlap, Huangpeng (Peter),
xen-devel, Lianxueguo (Albert),
Fanhenglong
>>> On 12.08.16 at 05:18, <visen.gao@huawei.com> wrote:
> Hi Jan,
> Thanks for your reply.
> Qemu-xen seems that has problem for support 8G large Bar.
> I think this patch is not perfect:
> https://xenbits.xen.org/gitweb/?p=qemu-xen.git;a=commitdiff;h=aabc8530c7ba2b
> e89e21463f051056ad7c255e6e
> Because I found upper 32bit bar of 8G large bar was not register.
> xen_pt_config_init.c:456 xen_pt_bar_reg_write
>
> case XEN_PT_BAR_FLAG_UPPER:
> bar_emu_mask = XEN_PT_BAR_ALLF;
> bar_ro_mask = r_size ? r_size - 1 : 0;
> break;
> r_size is always 0. So, bar_sz_upper is always 0xffffffff, regardless of
> whether 64 bit BAR size is larger than 4G.
Depending on how qemu sets things up, the issue may simply be on
of wrong types being used: r->size is of type pcibus_t (which I
sincerely hope is a 64-bit type), yet r_size is uint32_t. But then
again r->size being pcibus_t suggests that d->io_regions[] fields
for the upper halves of 64-bit BARs may not get populated at all
(in which case a possible fix would likely be more involved). I'm not
a qemu expert, so I can't easily tell.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-08-12 11:24 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-11 3:30 xen does not support the 8G large bar Gaofeng (GaoFeng, Euler)
2016-08-11 9:35 ` Jan Beulich
2016-08-12 3:18 ` 答复: " Gaofeng (GaoFeng, Euler)
2016-08-12 11:24 ` Jan Beulich
2016-08-11 11:29 ` Konrad Rzeszutek Wilk
2016-08-12 6:58 ` Gaofeng (GaoFeng, Euler)
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.