xl does not support specifying virtual function for passthrough device

xl does not support specifying virtual function for passthrough device

[Saurabh Mishra]

[-- Attachment #1.1: Type: text/plain, Size: 5210 bytes --]

>FYI qemu-upstream xen passhthrough currently also doesn't support
specifying a request
> for a certain dev/function nr.
> And it is vaguely related to the lacking ability to passthrough
multifunction devices as
> multifunction devices. (the xm + qemu_trad require you to specify the
virtual dev/func in
> this case, and additional some checks in libxl can't handle the
multifunction mask and bail
> out early)
> Sander

But it seems to work in XM toolchain. For example :-

pci = [ '0000:08:00.1=0,2=1,3=2,4=3@0a' ]


*ironpass3:~/sles11_sp2_gm # xm pci-list-assignable-devices*

0000:08:00.1

0000:08:00.2

0000:08:00.3

0000:08:00.4



*ironpass3:~/sles11_sp2_gm # xm create sles11_sp2_gm.xenconfig*

Using config file "./sles11_sp2_gm.xenconfig".

Started domain sles11_sp2_gm (id=5)



*ironpass3:~/sles11_sp2_gm # uname -a*

Linux ironpass3 3.0.93-0.8-xen #1 SMP Tue Aug 27 08:44:18 UTC 2013
(70ed288) x86_64 x86_64 x86_64 GNU/Linux

ironpass3:~/sles11_sp2_gm #

ironpass3:~/sles11_sp2_gm #

*ironpass3:~/sles11_sp2_gm # grep pci sles11_sp2_gm.xenconfig*

#pci = [ '0000:08:00.1=0@0a', '0000:08:00.2=0@0b', '0000:08:00.3=0@0c',
'0000:08:00.4=0@0d' ]

*pci = [ '0000:08:00.1=0,2=1,3=2,4=3@0a' ]*

xen_platform_pci=0

ironpass3:~/sles11_sp2_gm #

ironpass3:~/sles11_sp2_gm #

ironpass3:~/sles11_sp2_gm # lspci -nn -s 08:00

08:00.0 Co-processor [0b40]: Intel Corporation Device [8086:0434] (rev 10)

*08:00.1 Ethernet controller [0200]: Intel Corporation DH8900CC Series
Gigabit Network Connection [8086:0438] (rev 10)*

*08:00.2 Ethernet controller [0200]: Intel Corporation DH8900CC Series
Gigabit Network Connection [8086:0438] (rev 10)*

*08:00.3 Ethernet controller [0200]: Intel Corporation DH8900CC Series
Gigabit Network Connection [8086:0438] (rev 10)*

*08:00.4 Ethernet controller [0200]: Intel Corporation DH8900CC Series
Gigabit Network Connection [8086:0438] (rev 10)*

08:00.5 Co-processor [0b40]: Intel Corporation Device [8086:043e] (rev 10)

ironpass3:~/sles11_sp2_gm #

ironpass3:~/sles11_sp2_gm # xm console sles11_sp2_gm



linux-34o4:~ # lspci -nn

00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma]
[8086:1237] (rev 02)

00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA
[Natoma/Triton II] [8086:7000]

00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE
[Natoma/Triton II] [8086:7010]

00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI
[8086:7113] (rev 01)

00:02.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8]

*00:0a.0 Ethernet controller [0200]: Intel Corporation DH8900CC Series
Gigabit Network Connection [8086:0438] (rev 10)*

*00:0a.1 Ethernet controller [0200]: Intel Corporation DH8900CC Series
Gigabit Network Connection [8086:0438] (rev 10)*

*00:0a.2 Ethernet controller [0200]: Intel Corporation DH8900CC Series
Gigabit Network Connection [8086:0438] (rev 10)*

*00:0a.3 Ethernet controller [0200]: Intel Corporation DH8900CC Series
Gigabit Network Connection [8086:0438] (rev 10)*

linux-34o4:~ #

>Do you happen to know (or can you easily do the experiment to find out)
>what happens with '0000:07:11.6@1a' on xm? Does the guest see function 0
>or function 6? If it is 0 then I think we can safely say xl will not
>change, but if it is 6 we'll have to have a think...
>Ian.

Each PCI- device have to have function 0. So in the above case it gets 0.

However,  we may specify '0000:07:11.6=0@1a' and ''0000:07:11.8=1@1a' and
XM does handle it as expected.

However, xl does not allow us to specify device function in the guest.

Is there a way specifying virtual function slot can become a parameter?
After all, xm cfg file is supposed to be 100% compatible with 'xl'.

In Xen 4.2.2_06 version, is 'xl' preferred way to instantiate VMs? We
currently use 'xm' but wanted to move to 'xl' because it has better support
for NUMA alignment and memory backing.

Thanks,
/Saurabh

On Mon, Nov 4, 2013 at 7:45 AM, Ian Campbell <Ian.Campbell@citrix.com>wrote:

> create ^
> title it xl does not support specifying virtual function for passthrough
> device
> thanks
>
> On Thu, 2013-10-31 at 14:29 -0700, Saurabh Mishra wrote:
> > In XM toolchain, we use to put pci = [ '0000:07:11.6=0@1a' ] but looks
> > like in XL toolchain there is no way to specify function in the guest.
>
> The code is a bit confusing, but that does appear to be the case. libxl
> seems to have some degree of support but I can't see any code in xl (or
> the libxlu helper library function which parses the PCI options) that
> would support the =N@M syntax.
>
> Given the lack of xl support there's a good chance that the libxl
> support is relatively untested.
>
> > Kindly let me know.
> >
> >
> > XM  - pci = [ '0000:07:11.6=0@1a' ]
> > XL   - pci = [ '0000:07:11.6@1a' ]
> >
> >
> > In the guest VM we see that PCI device is getting 0 function in both
> > the cases however, it may not work the same way in the future.
>
> Do you happen to know (or can you easily do the experiment to find out)
> what happens with '0000:07:11.6@1a' on xm? Does the guest see function 0
> or function 6? If it is 0 then I think we can safely say xl will not
> change, but if it is 6 we'll have to have a think...
>
> Ian.
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 12023 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: xl does not support specifying virtual function for passthrough device

[Ian Campbell]

graft 22 ^
thanks

On Mon, 2013-11-04 at 10:29 -0800, Saurabh Mishra wrote:
> >FYI qemu-upstream xen passhthrough currently also doesn't support specifying a request
> > for a certain dev/function nr.
> But it seems to work in XM toolchain.

xm only supports the historical qemu-xen-traditional fork of qemu, not
the more modern qemu-xen ("upstream") stuff. xl supports both.

>  For example :-

> pci = [ '0000:08:00.1=0,2=1,3=2,4=3@0a' ]

Oh wow, so it is possible to assign multiple functions in one go?
(lib)xl certainly doesn't know how to do that!

> ironpass3:~/sles11_sp2_gm # xm pci-list-assignable-devices
> 0000:08:00.1
> 0000:08:00.2
> 0000:08:00.3
> 0000:08:00.4

> ironpass3:~/sles11_sp2_gm # xm create sles11_sp2_gm.xenconfig
> Using config file "./sles11_sp2_gm.xenconfig".
> Started domain sles11_sp2_gm (id=5)

> ironpass3:~/sles11_sp2_gm # grep pci sles11_sp2_gm.xenconfig
> #pci = [ '0000:08:00.1=0@0a', '0000:08:00.2=0@0b', '0000:08:00.3=0@0c', '0000:08:00.4=0@0d' ]
> pci = [ '0000:08:00.1=0,2=1,3=2,4=3@0a' ]
> xen_platform_pci=0

> ironpass3:~/sles11_sp2_gm # lspci -nn -s 08:00
> 08:00.0 Co-processor [0b40]: Intel Corporation Device [8086:0434] (rev 10)
> 08:00.1 Ethernet controller [0200]: Intel Corporation DH8900CC Series Gigabit Network Connection [8086:0438] (rev 10)
> 08:00.2 Ethernet controller [0200]: Intel Corporation DH8900CC Series Gigabit Network Connection [8086:0438] (rev 10)
> 08:00.3 Ethernet controller [0200]: Intel Corporation DH8900CC Series Gigabit Network Connection [8086:0438] (rev 10)
> 08:00.4 Ethernet controller [0200]: Intel Corporation DH8900CC Series Gigabit Network Connection [8086:0438] (rev 10)
> 08:00.5 Co-processor [0b40]: Intel Corporation Device [8086:043e] (rev 10)
> 
> ironpass3:~/sles11_sp2_gm # xm console sles11_sp2_gm
> linux-34o4:~ # lspci -nn
> 00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
> 00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
> 00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] [8086:7010]
> 00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 01)
> 00:02.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8]
> 00:0a.0 Ethernet controller [0200]: Intel Corporation DH8900CC Series Gigabit Network Connection [8086:0438] (rev 10)
> 00:0a.1 Ethernet controller [0200]: Intel Corporation DH8900CC Series Gigabit Network Connection [8086:0438] (rev 10)
> 00:0a.2 Ethernet controller [0200]: Intel Corporation DH8900CC Series Gigabit Network Connection [8086:0438] (rev 10)
> 00:0a.3 Ethernet controller [0200]: Intel Corporation DH8900CC Series Gigabit Network Connection [8086:0438] (rev 10)

> >Do you happen to know (or can you easily do the experiment to find out)
> >what happens with '0000:07:11.6@1a' on xm? Does the guest see function 0
> >or function 6? If it is 0 then I think we can safely say xl will not
> >change, but if it is 6 we'll have to have a think...

> Each PCI- device have to have function 0. So in the above case it gets 0.

OK, so if xm treats 0000:07:11.6@1a as creating 00:1a.0 in the guest
then I think we can safely say that xl will not change in this regard.

> However,  we may specify '0000:07:11.6=0@1a' and ''0000:07:11.8=1@1a'
> and XM does handle it as expected.

> However, xl does not allow us to specify device function in the guest.
> Is there a way specifying virtual function slot can become a
> parameter?

Not currently.

> After all, xm cfg file is supposed to be 100% compatible with 'xl'.
 
Right. This is a bug in xl IMHO. The reason I was asking about the XM
behaviour without was to determine the exact nature of what xl should be
doing.

I instantiated this issue as http://bugs.xenproject.org/xen/bug/22 in my
initial reply. It looks like you broke threading with your reply. The
control statement at the top of this mail ought to fix that up so that
this subthread is also recorded in the bug.

We would be happy to receive patches to fix this stuff up if you feel
able, I'm more than happy to mentor if need be.

> In Xen 4.2.2_06 version, is 'xl' preferred way to instantiate VMs? We
> currently use 'xm' but wanted to move to 'xl' because it has better
> support for NUMA alignment and memory backing.

xl became the default in 4.3 IIRC, although in reality xm has been
unmaintained for quite a while before this.

Ian.

Processed: Re: xl does not support specifying virtual function for passthrough device

[xen]

Processing commands for xen@bugs.xenproject.org:

> graft 22 ^
Graft `<CAMnwyJ2ZpAKbrF18xY9yjGJpoAS-Z4uywCwm_F_2UkuxAEjUow@mail.gmail.com>' onto #22
> thanks
Finished processing.

Modified/created Bugs:
 - 22: http://bugs.xenproject.org/xen/bug/22

---
Xen Hypervisor Bug Tracker
See http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen for information on reporting bugs
Contact xen-bugs-owner@bugs.xenproject.org with any infrastructure issues

Re: xl does not support specifying virtual function for passthrough device

[Saurabh Mishra]

[-- Attachment #1.1: Type: text/plain, Size: 4977 bytes --]

Hi Ian --

Thanks. Yes -- it's possible to assign multiple function in one pci line.

Is 'xl' being actively supported by Xen development community? and 'xm'
will not supported at all in Xen 4.2.2+?

SuSE SP3 pack has Xen 4.2.2

Thanks
/Saurabh


On Tue, Nov 5, 2013 at 2:20 AM, Ian Campbell <Ian.Campbell@citrix.com>wrote:

> graft 22 ^
> thanks
>
> On Mon, 2013-11-04 at 10:29 -0800, Saurabh Mishra wrote:
> > >FYI qemu-upstream xen passhthrough currently also doesn't support
> specifying a request
> > > for a certain dev/function nr.
> > But it seems to work in XM toolchain.
>
> xm only supports the historical qemu-xen-traditional fork of qemu, not
> the more modern qemu-xen ("upstream") stuff. xl supports both.
>
> >  For example :-
>
> > pci = [ '0000:08:00.1=0,2=1,3=2,4=3@0a' ]
>
> Oh wow, so it is possible to assign multiple functions in one go?
> (lib)xl certainly doesn't know how to do that!
>
> > ironpass3:~/sles11_sp2_gm # xm pci-list-assignable-devices
> > 0000:08:00.1
> > 0000:08:00.2
> > 0000:08:00.3
> > 0000:08:00.4
>
> > ironpass3:~/sles11_sp2_gm # xm create sles11_sp2_gm.xenconfig
> > Using config file "./sles11_sp2_gm.xenconfig".
> > Started domain sles11_sp2_gm (id=5)
>
> > ironpass3:~/sles11_sp2_gm # grep pci sles11_sp2_gm.xenconfig
> > #pci = [ '0000:08:00.1=0@0a', '0000:08:00.2=0@0b', '0000:08:00.3=0@0c',
> '0000:08:00.4=0@0d' ]
> > pci = [ '0000:08:00.1=0,2=1,3=2,4=3@0a' ]
> > xen_platform_pci=0
>
> > ironpass3:~/sles11_sp2_gm # lspci -nn -s 08:00
> > 08:00.0 Co-processor [0b40]: Intel Corporation Device [8086:0434] (rev
> 10)
> > 08:00.1 Ethernet controller [0200]: Intel Corporation DH8900CC Series
> Gigabit Network Connection [8086:0438] (rev 10)
> > 08:00.2 Ethernet controller [0200]: Intel Corporation DH8900CC Series
> Gigabit Network Connection [8086:0438] (rev 10)
> > 08:00.3 Ethernet controller [0200]: Intel Corporation DH8900CC Series
> Gigabit Network Connection [8086:0438] (rev 10)
> > 08:00.4 Ethernet controller [0200]: Intel Corporation DH8900CC Series
> Gigabit Network Connection [8086:0438] (rev 10)
> > 08:00.5 Co-processor [0b40]: Intel Corporation Device [8086:043e] (rev
> 10)
> >
> > ironpass3:~/sles11_sp2_gm # xm console sles11_sp2_gm
> > linux-34o4:~ # lspci -nn
> > 00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC
> [Natoma] [8086:1237] (rev 02)
> > 00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA
> [Natoma/Triton II] [8086:7000]
> > 00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE
> [Natoma/Triton II] [8086:7010]
> > 00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI
> [8086:7113] (rev 01)
> > 00:02.0 VGA compatible controller [0300]: Cirrus Logic GD 5446
> [1013:00b8]
> > 00:0a.0 Ethernet controller [0200]: Intel Corporation DH8900CC Series
> Gigabit Network Connection [8086:0438] (rev 10)
> > 00:0a.1 Ethernet controller [0200]: Intel Corporation DH8900CC Series
> Gigabit Network Connection [8086:0438] (rev 10)
> > 00:0a.2 Ethernet controller [0200]: Intel Corporation DH8900CC Series
> Gigabit Network Connection [8086:0438] (rev 10)
> > 00:0a.3 Ethernet controller [0200]: Intel Corporation DH8900CC Series
> Gigabit Network Connection [8086:0438] (rev 10)
>
> > >Do you happen to know (or can you easily do the experiment to find out)
> > >what happens with '0000:07:11.6@1a' on xm? Does the guest see function
> 0
> > >or function 6? If it is 0 then I think we can safely say xl will not
> > >change, but if it is 6 we'll have to have a think...
>
> > Each PCI- device have to have function 0. So in the above case it gets 0.
>
> OK, so if xm treats 0000:07:11.6@1a as creating 00:1a.0 in the guest
> then I think we can safely say that xl will not change in this regard.
>
> > However,  we may specify '0000:07:11.6=0@1a' and ''0000:07:11.8=1@1a'
> > and XM does handle it as expected.
>
> > However, xl does not allow us to specify device function in the guest.
> > Is there a way specifying virtual function slot can become a
> > parameter?
>
> Not currently.
>
> > After all, xm cfg file is supposed to be 100% compatible with 'xl'.
>
> Right. This is a bug in xl IMHO. The reason I was asking about the XM
> behaviour without was to determine the exact nature of what xl should be
> doing.
>
> I instantiated this issue as http://bugs.xenproject.org/xen/bug/22 in my
> initial reply. It looks like you broke threading with your reply. The
> control statement at the top of this mail ought to fix that up so that
> this subthread is also recorded in the bug.
>
> We would be happy to receive patches to fix this stuff up if you feel
> able, I'm more than happy to mentor if need be.
>
> > In Xen 4.2.2_06 version, is 'xl' preferred way to instantiate VMs? We
> > currently use 'xm' but wanted to move to 'xl' because it has better
> > support for NUMA alignment and memory backing.
>
> xl became the default in 4.3 IIRC, although in reality xm has been
> unmaintained for quite a while before this.
>
> Ian.
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 6370 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: xl does not support specifying virtual function for passthrough device

[Ian Campbell]

On Tue, 2013-11-05 at 16:05 -0800, Saurabh Mishra wrote:
> Hi Ian --
> 
> 
> Thanks. Yes -- it's possible to assign multiple function in one pci
> line.
> 
> 
> Is 'xl' being actively supported by Xen development community?

Yes, actively supported and developed.

>  and 'xm' will not supported at all in Xen 4.2.2+?

It's been effectively unmaintained for longer than that.

Ian.

Re: xl does not support specifying virtual function for passthrough device

[Jim Fehlig]

Saurabh Mishra wrote:
> Hi Ian --
>
> Thanks. Yes -- it's possible to assign multiple function in one pci line.
>
> Is 'xl' being actively supported by Xen development community? and
> 'xm' will not supported at all in Xen 4.2.2+?
>
> SuSE SP3 pack has Xen 4.2.2

I assume you mean SLES11 SP3?  The SLES11 series started with the
xm/xend toolstack and will live  long, happy life using it.  When the
SLES12 series comes to life, it will be using the new xl/libxl toolstack.

Regards,
Jim

Re: xl does not support specifying virtual function for passthrough device

[Saurabh Mishra]

[-- Attachment #1.1: Type: text/plain, Size: 647 bytes --]

Hi Jim --

Yes - I meant SuSE 11 SP3 pack.

/Saurabh


On Wed, Nov 6, 2013 at 7:28 AM, Jim Fehlig <jfehlig@suse.com> wrote:

> Saurabh Mishra wrote:
> > Hi Ian --
> >
> > Thanks. Yes -- it's possible to assign multiple function in one pci line.
> >
> > Is 'xl' being actively supported by Xen development community? and
> > 'xm' will not supported at all in Xen 4.2.2+?
> >
> > SuSE SP3 pack has Xen 4.2.2
>
> I assume you mean SLES11 SP3?  The SLES11 series started with the
> xm/xend toolstack and will live  long, happy life using it.  When the
> SLES12 series comes to life, it will be using the new xl/libxl toolstack.
>
> Regards,
> Jim
>
>

[-- Attachment #1.2: Type: text/html, Size: 1110 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel