[Query] Flow of PCI device dedicated to a domU

[Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

[-- Attachment #1.1: Type: text/plain, Size: 789 bytes --]

Hi,
I am trying to understand the flow of how a PCI device driver in domU works
after a PCI device is is assigned to a domU.

a) If a PCI device is assigned to a domU, this device has to be on a PCI
bus. So as per my view xen would have to somehow provide a PCI Controller
on which this device is attached.
=> Is my assumption correct ? If yes how it is done, No then also How
enumeration of this device happens in domU kernel

b) Is the Configuration space of the PCI device directly accessible to the
domU (assuming the kernel accesses it using memory map) ? If not then it is
trapped by xen

c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how
Xen does a translation from Physical MSI to guest MSI (where in code)

Please help me with these doubts


Regards
Manish

[-- Attachment #1.2: Type: text/html, Size: 944 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 1444 bytes --]

On 16/07/14 07:12, Xenbegn developer wrote:
> Hi,
> I am trying to understand the flow of how a PCI device driver in domU
> works after a PCI device is is assigned to a domU.
>
> a) If a PCI device is assigned to a domU, this device has to be on a
> PCI bus. So as per my view xen would have to somehow provide a PCI
> Controller on which this device is attached.
> => Is my assumption correct ? If yes how it is done, No then also How
> enumeration of this device happens in domU kernel

No.

PV guests have no PCI root ports/bridges; they use devices as single
entities knowing that Xen/dom0 takes care of the other bits. HVM guests
have their PCI devices attached to the virtual southbridge which is all
emulated by Qemu.

>
> b) Is the Configuration space of the PCI device directly accessible to
> the domU (assuming the kernel accesses it using memory map) ? If not
> then it is trapped by xen

All configuration space is trap+emulate in Xen, although almost all
operations permitted.

>
> c) Who assigns the MSI (addr + value) in the PCI device. If Xen then
> how Xen does a translation from Physical MSI to guest MSI (where in code)

Xen controls all interrupts on the system, which is why it needs to trap
all config accesses to notice when a domain is attempting to change the
interrupt information.  In that case, Xen fixes up its delivery of
interrupts to the guest, but leaves the underlying interrupt information
intact.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 3212 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

On 7/16/14, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> On 16/07/14 07:12, Xenbegn developer wrote:
>> Hi,
>> I am trying to understand the flow of how a PCI device driver in domU
>> works after a PCI device is is assigned to a domU.
>>
>> a) If a PCI device is assigned to a domU, this device has to be on a
>> PCI bus. So as per my view xen would have to somehow provide a PCI
>> Controller on which this device is attached.
>> => Is my assumption correct ? If yes how it is done, No then also How
>> enumeration of this device happens in domU kernel
>
> No.
>
> PV guests have no PCI root ports/bridges; they use devices as single
> entities knowing that Xen/dom0 takes care of the other bits. HVM guests
> have their PCI devices attached to the virtual southbridge which is all
> emulated by Qemu.
>
How would a then a PV guest know about existence of a PCI device if it
does not enumerate. Can you please elaborate on this.


>>
>> b) Is the Configuration space of the PCI device directly accessible to
>> the domU (assuming the kernel accesses it using memory map) ? If not
>> then it is trapped by xen
>
> All configuration space is trap+emulate in Xen, although almost all
> operations permitted.
>

>>
>> c) Who assigns the MSI (addr + value) in the PCI device. If Xen then
>> how Xen does a translation from Physical MSI to guest MSI (where in code)
>
> Xen controls all interrupts on the system, which is why it needs to trap
> all config accesses to notice when a domain is attempting to change the
> interrupt information.  In that case, Xen fixes up its delivery of
> interrupts to the guest, but leaves the underlying interrupt information
> intact.
>
> ~Andrew
>

Re: [Query] Flow of PCI device dedicated to a domU

[Andrew Cooper]

On 17/07/14 07:34, Xenbegn developer wrote:
> On 7/16/14, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> On 16/07/14 07:12, Xenbegn developer wrote:
>>> Hi,
>>> I am trying to understand the flow of how a PCI device driver in domU
>>> works after a PCI device is is assigned to a domU.
>>>
>>> a) If a PCI device is assigned to a domU, this device has to be on a
>>> PCI bus. So as per my view xen would have to somehow provide a PCI
>>> Controller on which this device is attached.
>>> => Is my assumption correct ? If yes how it is done, No then also How
>>> enumeration of this device happens in domU kernel
>> No.
>>
>> PV guests have no PCI root ports/bridges; they use devices as single
>> entities knowing that Xen/dom0 takes care of the other bits. HVM guests
>> have their PCI devices attached to the virtual southbridge which is all
>> emulated by Qemu.
>>
> How would a then a PV guest know about existence of a PCI device if it
> does not enumerate. Can you please elaborate on this.

The pcifront driver is capable of finding any PCI devices offered by
pciback in dom0.  It is all paravirtualised, and doesn't need explicit
hardware probing.

~Andrew

Re: [Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

[-- Attachment #1.1: Type: text/plain, Size: 1560 bytes --]

On Thu, Jul 17, 2014 at 3:12 PM, Andrew Cooper <andrew.cooper3@citrix.com>
wrote:

> On 17/07/14 07:34, Xenbegn developer wrote:
> > On 7/16/14, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> >> On 16/07/14 07:12, Xenbegn developer wrote:
> >>> Hi,
> >>> I am trying to understand the flow of how a PCI device driver in domU
> >>> works after a PCI device is is assigned to a domU.
> >>>
> >>> a) If a PCI device is assigned to a domU, this device has to be on a
> >>> PCI bus. So as per my view xen would have to somehow provide a PCI
> >>> Controller on which this device is attached.
> >>> => Is my assumption correct ? If yes how it is done, No then also How
> >>> enumeration of this device happens in domU kernel
> >> No.
> >>
> >> PV guests have no PCI root ports/bridges; they use devices as single
> >> entities knowing that Xen/dom0 takes care of the other bits. HVM guests
> >> have their PCI devices attached to the virtual southbridge which is all
> >> emulated by Qemu.
> >>
> > How would a then a PV guest know about existence of a PCI device if it
> > does not enumerate. Can you please elaborate on this.
>
> The pcifront driver is capable of finding any PCI devices offered by
> pciback in dom0.  It is all paravirtualised, and doesn't need explicit
> hardware probing.
>
> So does this flow happens internally in xen for x86
a) dom0 scans all the pci devices
b) Using libxl hypercalls a pci device is assigned to a domainU
c) When  domU boots the pci front driver would talk to pci back driver and
discover the PCI devices


~Andrew
>
>

[-- Attachment #1.2: Type: text/html, Size: 2461 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 1867 bytes --]

On 22/07/14 10:43, Xenbegn developer wrote:
>
>
>
> On Thu, Jul 17, 2014 at 3:12 PM, Andrew Cooper
> <andrew.cooper3@citrix.com <mailto:andrew.cooper3@citrix.com>> wrote:
>
>     On 17/07/14 07:34, Xenbegn developer wrote:
>     > On 7/16/14, Andrew Cooper <andrew.cooper3@citrix.com
>     <mailto:andrew.cooper3@citrix.com>> wrote:
>     >> On 16/07/14 07:12, Xenbegn developer wrote:
>     >>> Hi,
>     >>> I am trying to understand the flow of how a PCI device driver
>     in domU
>     >>> works after a PCI device is is assigned to a domU.
>     >>>
>     >>> a) If a PCI device is assigned to a domU, this device has to
>     be on a
>     >>> PCI bus. So as per my view xen would have to somehow provide a PCI
>     >>> Controller on which this device is attached.
>     >>> => Is my assumption correct ? If yes how it is done, No then
>     also How
>     >>> enumeration of this device happens in domU kernel
>     >> No.
>     >>
>     >> PV guests have no PCI root ports/bridges; they use devices as
>     single
>     >> entities knowing that Xen/dom0 takes care of the other bits.
>     HVM guests
>     >> have their PCI devices attached to the virtual southbridge
>     which is all
>     >> emulated by Qemu.
>     >>
>     > How would a then a PV guest know about existence of a PCI device
>     if it
>     > does not enumerate. Can you please elaborate on this.
>
>     The pcifront driver is capable of finding any PCI devices offered by
>     pciback in dom0.  It is all paravirtualised, and doesn't need explicit
>     hardware probing.
>
> So does this flow happens internally in xen for x86
> a) dom0 scans all the pci devices
> b) Using libxl hypercalls a pci device is assigned to a domainU
> c) When  domU boots the pci front driver would talk to pci back driver
> and discover the PCI devices

>From a basic point of view, yes.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 3850 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

[-- Attachment #1.1: Type: text/plain, Size: 1998 bytes --]

On Tue, Jul 22, 2014 at 3:23 PM, Andrew Cooper <andrew.cooper3@citrix.com>
wrote:

>  On 22/07/14 10:43, Xenbegn developer wrote:
>
>
>
>
> On Thu, Jul 17, 2014 at 3:12 PM, Andrew Cooper <andrew.cooper3@citrix.com>
> wrote:
>
>> On 17/07/14 07:34, Xenbegn developer wrote:
>> > On 7/16/14, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> >> On 16/07/14 07:12, Xenbegn developer wrote:
>> >>> Hi,
>> >>> I am trying to understand the flow of how a PCI device driver in domU
>> >>> works after a PCI device is is assigned to a domU.
>> >>>
>> >>> a) If a PCI device is assigned to a domU, this device has to be on a
>> >>> PCI bus. So as per my view xen would have to somehow provide a PCI
>> >>> Controller on which this device is attached.
>> >>> => Is my assumption correct ? If yes how it is done, No then also How
>> >>> enumeration of this device happens in domU kernel
>> >> No.
>> >>
>> >> PV guests have no PCI root ports/bridges; they use devices as single
>> >> entities knowing that Xen/dom0 takes care of the other bits. HVM guests
>> >> have their PCI devices attached to the virtual southbridge which is all
>> >> emulated by Qemu.
>> >>
>> > How would a then a PV guest know about existence of a PCI device if it
>> > does not enumerate. Can you please elaborate on this.
>>
>>  The pcifront driver is capable of finding any PCI devices offered by
>> pciback in dom0.  It is all paravirtualised, and doesn't need explicit
>> hardware probing.
>>
>>  So does this flow happens internally in xen for x86
> a) dom0 scans all the pci devices
>  b) Using libxl hypercalls a pci device is assigned to a domainU
>  c) When  domU boots the pci front driver would talk to pci back driver
> and discover the PCI devices
>
>
> From a basic point of view, yes.
>
> Two points here
a) Does Xen do a PCI enumeration ? if yes then what is the use case if dom0
is capable of enumeration.
b) What other cases which are beyond the "basic point of view" where there
is a different flow

> ~Andrew
>

[-- Attachment #1.2: Type: text/html, Size: 4419 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

[-- Attachment #1.1: Type: text/plain, Size: 2016 bytes --]

On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper <andrew.cooper3@citrix.com>
wrote:

>  On 16/07/14 07:12, Xenbegn developer wrote:
>
>     Hi,
>  I am trying to understand the flow of how a PCI device driver in domU
> works after a PCI device is is assigned to a domU.
>
>  a) If a PCI device is assigned to a domU, this device has to be on a PCI
> bus. So as per my view xen would have to somehow provide a PCI Controller
> on which this device is attached.
>  => Is my assumption correct ? If yes how it is done, No then also How
> enumeration of this device happens in domU kernel
>
>
> No.
>
> PV guests have no PCI root ports/bridges; they use devices as single
> entities knowing that Xen/dom0 takes care of the other bits. HVM guests
> have their PCI devices attached to the virtual southbridge which is all
> emulated by Qemu.
>
>
>
>  b) Is the Configuration space of the PCI device directly accessible to
> the domU (assuming the kernel accesses it using memory map) ? If not then
> it is trapped by xen
>
>
> All configuration space is trap+emulate in Xen, although almost all
> operations permitted.
>
>
>
>  c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how
> Xen does a translation from Physical MSI to guest MSI (where in code)
>
>
> Xen controls all interrupts on the system, which is why it needs to trap
> all config accesses to notice when a domain is attempting to change the
> interrupt information.  In that case, Xen fixes up its delivery of
> interrupts to the guest, but leaves the underlying interrupt information
> intact.
>
> ~Andrew
>

Linux kernel pci code has a function called __write_msi_msg. This method
writes to config space for MSI/MSIX
pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
                       msg->address_lo);

Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c
(guest_io_write) I dont see any MSI handling.
Can you please explain (point to the code) where domain is attempting to
change the interrupt information.

[-- Attachment #1.2: Type: text/html, Size: 3885 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 2523 bytes --]

On 22/07/14 10:50, Xenbegn developer wrote:
>
>
>
> On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper
> <andrew.cooper3@citrix.com <mailto:andrew.cooper3@citrix.com>> wrote:
>
>     On 16/07/14 07:12, Xenbegn developer wrote:
>>     Hi,
>>     I am trying to understand the flow of how a PCI device driver in
>>     domU works after a PCI device is is assigned to a domU.
>>
>>     a) If a PCI device is assigned to a domU, this device has to be
>>     on a PCI bus. So as per my view xen would have to somehow provide
>>     a PCI Controller on which this device is attached.
>>     => Is my assumption correct ? If yes how it is done, No then also
>>     How enumeration of this device happens in domU kernel
>
>     No.
>
>     PV guests have no PCI root ports/bridges; they use devices as
>     single entities knowing that Xen/dom0 takes care of the other
>     bits. HVM guests have their PCI devices attached to the virtual
>     southbridge which is all emulated by Qemu.
>
>
>>
>>     b) Is the Configuration space of the PCI device directly
>>     accessible to the domU (assuming the kernel accesses it using
>>     memory map) ? If not then it is trapped by xen
>
>     All configuration space is trap+emulate in Xen, although almost
>     all operations permitted.
>
>
>>
>>     c) Who assigns the MSI (addr + value) in the PCI device. If Xen
>>     then how Xen does a translation from Physical MSI to guest MSI
>>     (where in code)
>
>     Xen controls all interrupts on the system, which is why it needs
>     to trap all config accesses to notice when a domain is attempting
>     to change the interrupt information.  In that case, Xen fixes up
>     its delivery of interrupts to the guest, but leaves the underlying
>     interrupt information intact.
>
>     ~Andrew
>
>
> Linux kernel pci code has a function called __write_msi_msg. This
> method writes to config space for MSI/MSIX
> pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
>                        msg->address_lo);
>
> Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c
> (guest_io_write) I dont see any MSI handling.
> Can you please explain (point to the code) where domain is attempting
> to change the interrupt information.
>    
>

PV guests are expected to use PHYSOP hypercalls for interrupt
management.  Xen does not provide transparent emulation support for them.

HVM guests do get transparent emulation support, as from the point of
view of an HVM guest, they are talking to a real PCI device.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 6579 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

[-- Attachment #1.1: Type: text/plain, Size: 2638 bytes --]

On Tue, Jul 22, 2014 at 3:26 PM, Andrew Cooper <andrew.cooper3@citrix.com>
wrote:

>  On 22/07/14 10:50, Xenbegn developer wrote:
>
>
>
>
> On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper <andrew.cooper3@citrix.com>
> wrote:
>
>>  On 16/07/14 07:12, Xenbegn developer wrote:
>>
>>     Hi,
>>  I am trying to understand the flow of how a PCI device driver in domU
>> works after a PCI device is is assigned to a domU.
>>
>>  a) If a PCI device is assigned to a domU, this device has to be on a PCI
>> bus. So as per my view xen would have to somehow provide a PCI Controller
>> on which this device is attached.
>>  => Is my assumption correct ? If yes how it is done, No then also How
>> enumeration of this device happens in domU kernel
>>
>>
>>  No.
>>
>> PV guests have no PCI root ports/bridges; they use devices as single
>> entities knowing that Xen/dom0 takes care of the other bits. HVM guests
>> have their PCI devices attached to the virtual southbridge which is all
>> emulated by Qemu.
>>
>>
>>
>>  b) Is the Configuration space of the PCI device directly accessible to
>> the domU (assuming the kernel accesses it using memory map) ? If not then
>> it is trapped by xen
>>
>>
>>  All configuration space is trap+emulate in Xen, although almost all
>> operations permitted.
>>
>>
>>
>>  c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how
>> Xen does a translation from Physical MSI to guest MSI (where in code)
>>
>>
>>  Xen controls all interrupts on the system, which is why it needs to trap
>> all config accesses to notice when a domain is attempting to change the
>> interrupt information.  In that case, Xen fixes up its delivery of
>> interrupts to the guest, but leaves the underlying interrupt information
>> intact.
>>
>> ~Andrew
>>
>
>  Linux kernel pci code has a function called __write_msi_msg. This method
> writes to config space for MSI/MSIX
> pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
>                        msg->address_lo);
>
>  Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c
> (guest_io_write) I dont see any MSI handling.
>  Can you please explain (point to the code) where domain is attempting to
> change the interrupt information.
>
>
>
> PV guests are expected to use PHYSOP hypercalls for interrupt management.
> Xen does not provide transparent emulation support for them.
>
> HVM guests do get transparent emulation support, as from the point of view
> of an HVM guest, they are talking to a real PCI device.
>
> Can you please point to a device driver code which is calling PHYSOP calls
for setting an MSI Addr + MSI number

> ~Andrew
>

[-- Attachment #1.2: Type: text/html, Size: 7010 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 3203 bytes --]

On 22/07/14 11:13, Xenbegn developer wrote:
>
>
>
> On Tue, Jul 22, 2014 at 3:26 PM, Andrew Cooper
> <andrew.cooper3@citrix.com <mailto:andrew.cooper3@citrix.com>> wrote:
>
>     On 22/07/14 10:50, Xenbegn developer wrote:
>>
>>
>>
>>     On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper
>>     <andrew.cooper3@citrix.com <mailto:andrew.cooper3@citrix.com>> wrote:
>>
>>         On 16/07/14 07:12, Xenbegn developer wrote:
>>>         Hi,
>>>         I am trying to understand the flow of how a PCI device
>>>         driver in domU works after a PCI device is is assigned to a
>>>         domU.
>>>
>>>         a) If a PCI device is assigned to a domU, this device has to
>>>         be on a PCI bus. So as per my view xen would have to somehow
>>>         provide a PCI Controller on which this device is attached.
>>>         => Is my assumption correct ? If yes how it is done, No then
>>>         also How enumeration of this device happens in domU kernel
>>
>>         No.
>>
>>         PV guests have no PCI root ports/bridges; they use devices as
>>         single entities knowing that Xen/dom0 takes care of the other
>>         bits. HVM guests have their PCI devices attached to the
>>         virtual southbridge which is all emulated by Qemu.
>>
>>
>>>
>>>         b) Is the Configuration space of the PCI device directly
>>>         accessible to the domU (assuming the kernel accesses it
>>>         using memory map) ? If not then it is trapped by xen
>>
>>         All configuration space is trap+emulate in Xen, although
>>         almost all operations permitted.
>>
>>
>>>
>>>         c) Who assigns the MSI (addr + value) in the PCI device. If
>>>         Xen then how Xen does a translation from Physical MSI to
>>>         guest MSI (where in code)
>>
>>         Xen controls all interrupts on the system, which is why it
>>         needs to trap all config accesses to notice when a domain is
>>         attempting to change the interrupt information.  In that
>>         case, Xen fixes up its delivery of interrupts to the guest,
>>         but leaves the underlying interrupt information intact.
>>
>>         ~Andrew
>>
>>
>>     Linux kernel pci code has a function called __write_msi_msg. This
>>     method writes to config space for MSI/MSIX
>>     pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
>>                            msg->address_lo);
>>
>>     Now as per (b) and (c) it has to be trapped in xen. But in
>>     x86/traps.c (guest_io_write) I dont see any MSI handling.
>>     Can you please explain (point to the code) where domain is
>>     attempting to change the interrupt information.
>>        
>>
>
>     PV guests are expected to use PHYSOP hypercalls for interrupt
>     management.  Xen does not provide transparent emulation support
>     for them.
>
>     HVM guests do get transparent emulation support, as from the point
>     of view of an HVM guest, they are talking to a real PCI device.
>
> Can you please point to a device driver code which is calling PHYSOP
> calls for setting an MSI Addr + MSI number

No.  Partly because I don't know exactly, and mainly because this is
something you should trivially be able to search for.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 10921 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

[-- Attachment #1.1: Type: text/plain, Size: 3051 bytes --]

On Tue, Jul 22, 2014 at 4:03 PM, Andrew Cooper <andrew.cooper3@citrix.com>
wrote:

>  On 22/07/14 11:13, Xenbegn developer wrote:
>
>
>
>
> On Tue, Jul 22, 2014 at 3:26 PM, Andrew Cooper <andrew.cooper3@citrix.com>
> wrote:
>
>>   On 22/07/14 10:50, Xenbegn developer wrote:
>>
>>
>>
>>
>> On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper <andrew.cooper3@citrix.com
>> > wrote:
>>
>>>  On 16/07/14 07:12, Xenbegn developer wrote:
>>>
>>>     Hi,
>>>  I am trying to understand the flow of how a PCI device driver in domU
>>> works after a PCI device is is assigned to a domU.
>>>
>>>  a) If a PCI device is assigned to a domU, this device has to be on a
>>> PCI bus. So as per my view xen would have to somehow provide a PCI
>>> Controller on which this device is attached.
>>>  => Is my assumption correct ? If yes how it is done, No then also How
>>> enumeration of this device happens in domU kernel
>>>
>>>
>>>  No.
>>>
>>> PV guests have no PCI root ports/bridges; they use devices as single
>>> entities knowing that Xen/dom0 takes care of the other bits. HVM guests
>>> have their PCI devices attached to the virtual southbridge which is all
>>> emulated by Qemu.
>>>
>>>
>>>
>>>  b) Is the Configuration space of the PCI device directly accessible to
>>> the domU (assuming the kernel accesses it using memory map) ? If not then
>>> it is trapped by xen
>>>
>>>
>>>  All configuration space is trap+emulate in Xen, although almost all
>>> operations permitted.
>>>
>>>
>>>
>>>  c) Who assigns the MSI (addr + value) in the PCI device. If Xen then
>>> how Xen does a translation from Physical MSI to guest MSI (where in code)
>>>
>>>
>>>  Xen controls all interrupts on the system, which is why it needs to
>>> trap all config accesses to notice when a domain is attempting to change
>>> the interrupt information.  In that case, Xen fixes up its delivery of
>>> interrupts to the guest, but leaves the underlying interrupt information
>>> intact.
>>>
>>> ~Andrew
>>>
>>
>>  Linux kernel pci code has a function called __write_msi_msg. This
>> method writes to config space for MSI/MSIX
>> pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
>>                        msg->address_lo);
>>
>>  Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c
>> (guest_io_write) I dont see any MSI handling.
>>  Can you please explain (point to the code) where domain is attempting
>> to change the interrupt information.
>>
>>
>>
>>  PV guests are expected to use PHYSOP hypercalls for interrupt
>> management.  Xen does not provide transparent emulation support for them.
>>
>> HVM guests do get transparent emulation support, as from the point of
>> view of an HVM guest, they are talking to a real PCI device.
>>
>>  Can you please point to a device driver code which is calling PHYSOP
> calls for setting an MSI Addr + MSI number
>
>
> No.  Partly because I don't know exactly, and mainly because this is
> something you should trivially be able to search for.
>
> It is not that trivial. Can anoyone from Citrix reply ?

> ~Andrew
>

[-- Attachment #1.2: Type: text/html, Size: 11073 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

[-- Attachment #1.1: Type: text/plain, Size: 3387 bytes --]

Ian Can you please help in this. I am trying to understand how things are
done in x86 xen. Based on that will implement in ARM PCI passthrough


On Tue, Jul 22, 2014 at 4:21 PM, Xenbegn developer <xen.begn.dev@gmail.com>
wrote:

>
>
>
> On Tue, Jul 22, 2014 at 4:03 PM, Andrew Cooper <andrew.cooper3@citrix.com>
> wrote:
>
>>  On 22/07/14 11:13, Xenbegn developer wrote:
>>
>>
>>
>>
>> On Tue, Jul 22, 2014 at 3:26 PM, Andrew Cooper <andrew.cooper3@citrix.com
>> > wrote:
>>
>>>   On 22/07/14 10:50, Xenbegn developer wrote:
>>>
>>>
>>>
>>>
>>> On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper <
>>> andrew.cooper3@citrix.com> wrote:
>>>
>>>>  On 16/07/14 07:12, Xenbegn developer wrote:
>>>>
>>>>     Hi,
>>>>  I am trying to understand the flow of how a PCI device driver in domU
>>>> works after a PCI device is is assigned to a domU.
>>>>
>>>>  a) If a PCI device is assigned to a domU, this device has to be on a
>>>> PCI bus. So as per my view xen would have to somehow provide a PCI
>>>> Controller on which this device is attached.
>>>>  => Is my assumption correct ? If yes how it is done, No then also How
>>>> enumeration of this device happens in domU kernel
>>>>
>>>>
>>>>  No.
>>>>
>>>> PV guests have no PCI root ports/bridges; they use devices as single
>>>> entities knowing that Xen/dom0 takes care of the other bits. HVM guests
>>>> have their PCI devices attached to the virtual southbridge which is all
>>>> emulated by Qemu.
>>>>
>>>>
>>>>
>>>>  b) Is the Configuration space of the PCI device directly accessible to
>>>> the domU (assuming the kernel accesses it using memory map) ? If not then
>>>> it is trapped by xen
>>>>
>>>>
>>>>  All configuration space is trap+emulate in Xen, although almost all
>>>> operations permitted.
>>>>
>>>>
>>>>
>>>>  c) Who assigns the MSI (addr + value) in the PCI device. If Xen then
>>>> how Xen does a translation from Physical MSI to guest MSI (where in code)
>>>>
>>>>
>>>>  Xen controls all interrupts on the system, which is why it needs to
>>>> trap all config accesses to notice when a domain is attempting to change
>>>> the interrupt information.  In that case, Xen fixes up its delivery of
>>>> interrupts to the guest, but leaves the underlying interrupt information
>>>> intact.
>>>>
>>>> ~Andrew
>>>>
>>>
>>>  Linux kernel pci code has a function called __write_msi_msg. This
>>> method writes to config space for MSI/MSIX
>>> pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
>>>                        msg->address_lo);
>>>
>>>  Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c
>>> (guest_io_write) I dont see any MSI handling.
>>>  Can you please explain (point to the code) where domain is attempting
>>> to change the interrupt information.
>>>
>>>
>>>
>>>  PV guests are expected to use PHYSOP hypercalls for interrupt
>>> management.  Xen does not provide transparent emulation support for them.
>>>
>>> HVM guests do get transparent emulation support, as from the point of
>>> view of an HVM guest, they are talking to a real PCI device.
>>>
>>>  Can you please point to a device driver code which is calling PHYSOP
>> calls for setting an MSI Addr + MSI number
>>
>>
>> No.  Partly because I don't know exactly, and mainly because this is
>> something you should trivially be able to search for.
>>
>> It is not that trivial. Can anoyone from Citrix reply ?
>
>>  ~Andrew
>>
>
>

[-- Attachment #1.2: Type: text/html, Size: 11606 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Ian Campbell]

On Tue, 2014-07-22 at 17:03 +0530, Xenbegn developer wrote:

Please don't top post and please use plain text email, not HTML.

> Ian Can you please help in this. I am trying to understand how things
> are done in x86 xen. Based on that will implement in ARM PCI
> passthrough

I'm afraid I don't know exactly how MSIs work for X86 (neither PV nor
HVM). It not clear that ARM would use the same mechanisms anyway, since
ARM has a more advanced virtualisable interrupt controller available.

However I think you are getting rather ahead of yourself, before MSI
passthrough becomes an issue we need a) host support for GICv3 b) host
support for GICv3+ITS (ARM name for MSI).

Vijay Kilari is working on those a) is pretty much there, b) is still a
WIP AIUI.

We also lack some of the basic framework for doing passthrough on ARM.
Julien Grall is working on that for the platform device case, which will
provide many of the foundations for doing PCI passthrough.

Ian.

Re: [Query] Flow of PCI device dedicated to a domU

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5077 bytes --]

Please don't use HTML emails and please don't top post.

On Tue, 22 Jul 2014, Xenbegn developer wrote:
> Ian Can you please help in this. I am trying to understand how things are done in x86 xen. Based on that will
> implement in ARM PCI passthrough

The basic mechanism is that Xen assigns the PCI root complex/host
bridge to Dom0. Dom0 walks the bus and finds out the devices available.

When the user assigns a PCI device to a VM, the toolstack (xl/libxl)
issues hypercalls to remap interrupts and MMIO regions of that device
into the guest VM.
The device is added to the guest virtual PCI bus that is created by the
xen-pcifront driver at boot. PCI config space reads/writes go through
xen-pcifront that talks to xen-pciback in Dom0.

PCI bus enumeration is done by Dom0 at boot time. Dom0 issues hypercalls
to Xen to tell the hypervisor what devices are available. It would be
conceivable to do the enumeration directly in Xen if Xen had the
appropriate driver for the host bridge. It might be even better.

The main interaction between Xen and the config space of PCI devices is
about MSI programming. Xen programs MSIs directly because it is Xen that
is going to receive them and then inject them into guests.


If I were you I would base my work on Julien SMMU series and I would
ignore MSIs for now. I would try to get pcifront and pciback running and
then assign a PCI device to a guest, only using legacy interrupts.

After that you could try to add support for doing PCI config space reads
and writes in Xen. Then introduce MSI support but, as Ian wrote, you
would need to rebase your work on the GICv3 series. I don't think you
actually need full ITS support to be able to get MSI running though.
You might already have everything you need in terms of patches, but they
are still not applied to xen-unstable yet.



 
> On Tue, Jul 22, 2014 at 4:21 PM, Xenbegn developer <xen.begn.dev@gmail.com> wrote:
> 
> 
> 
>       On Tue, Jul 22, 2014 at 4:03 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>             On 22/07/14 11:13, Xenbegn developer wrote:
> 
> 
> 
>       On Tue, Jul 22, 2014 at 3:26 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>             On 22/07/14 10:50, Xenbegn developer wrote:
> 
> 
> 
>       On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>             On 16/07/14 07:12, Xenbegn developer wrote:
>       Hi,
> I am trying to understand the flow of how a PCI device driver in domU works
> after a PCI device is is assigned to a domU.
> 
> a) If a PCI device is assigned to a domU, this device has to be on a PCI
> bus. So as per my view xen would have to somehow provide a PCI Controller
> on which this device is attached.
> => Is my assumption correct ? If yes how it is done, No then also How
> enumeration of this device happens in domU kernel
> 
> 
> No.
> 
> PV guests have no PCI root ports/bridges; they use devices as single entities
> knowing that Xen/dom0 takes care of the other bits. HVM guests have their PCI
> devices attached to the virtual southbridge which is all emulated by Qemu.
> 
> 
> b) Is the Configuration space of the PCI device directly accessible to the
> domU (assuming the kernel accesses it using memory map) ? If not then it is
> trapped by xen
> 
> 
> All configuration space is trap+emulate in Xen, although almost all operations
> permitted.
> 
> 
> c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how
> Xen does a translation from Physical MSI to guest MSI (where in code)
> 
> 
> Xen controls all interrupts on the system, which is why it needs to trap all
> config accesses to notice when a domain is attempting to change the interrupt
> information.  In that case, Xen fixes up its delivery of interrupts to the guest,
> but leaves the underlying interrupt information intact.
> 
> ~Andrew
> 
> 
> Linux kernel pci code has a function called __write_msi_msg. This method writes to
> config space for MSI/MSIX
> pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
>                        msg->address_lo);
> 
> Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c (guest_io_write)
> I dont see any MSI handling.
> Can you please explain (point to the code) where domain is attempting to change the
> interrupt information.
>    
> 
> 
> PV guests are expected to use PHYSOP hypercalls for interrupt management.  Xen does not
> provide transparent emulation support for them.
> 
> HVM guests do get transparent emulation support, as from the point of view of an HVM guest,
> they are talking to a real PCI device.
> 
> Can you please point to a device driver code which is calling PHYSOP calls for setting an MSI Addr
> + MSI number
> 
> 
> No.  Partly because I don't know exactly, and mainly because this is something you should trivially be
> able to search for.
> 
> It is not that trivial. Can anoyone from Citrix reply ?
>       ~Andrew
> 
> 
> 
> 
> 

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Simon Martin]

Hello Stefano,

> The basic mechanism is that Xen assigns the PCI root complex/host
> bridge to Dom0. Dom0 walks the bus and finds out the devices available.

> When the user assigns a PCI device to a VM, the toolstack (xl/libxl)
> issues hypercalls to remap interrupts and MMIO regions of that device
> into the guest VM.

I am currently trying writing my own PCI front to access directly a
PCI device from my PV guest.

If I read this correctly you are saying that the MMIO region is
already mapped into my domain by xl? So when I go and read the BAR
register from the PCI configuration I can just go and access it
directly or do I have to update the VA mapping?

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: [Query] Flow of PCI device dedicated to a domU

[Simon Martin]

Hello Simon,

Tuesday, July 22, 2014, 4:56:57 PM, you wrote:

> Hello Stefano,

>> The basic mechanism is that Xen assigns the PCI root complex/host
>> bridge to Dom0. Dom0 walks the bus and finds out the devices available.

>> When the user assigns a PCI device to a VM, the toolstack (xl/libxl)
>> issues hypercalls to remap interrupts and MMIO regions of that device
>> into the guest VM.

> I am currently trying writing my own PCI front to access directly a
> PCI device from my PV guest.

> If I read this correctly you are saying that the MMIO region is
> already mapped into my domain by xl? So when I go and read the BAR
> register from the PCI configuration I can just go and access it
> directly or do I have to update the VA mapping?

Just a clarification here, when I say "go and access it" what I meant
to say was "go and access directly in my PV the memory at the BAR
address".

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: [Query] Flow of PCI device dedicated to a domU

[Ian Campbell]

On Tue, 2014-07-22 at 16:56 +0100, Simon Martin wrote:
> Hello Stefano,
> 
> > The basic mechanism is that Xen assigns the PCI root complex/host
> > bridge to Dom0. Dom0 walks the bus and finds out the devices available.
> 
> > When the user assigns a PCI device to a VM, the toolstack (xl/libxl)
> > issues hypercalls to remap interrupts and MMIO regions of that device
> > into the guest VM.
> 
> I am currently trying writing my own PCI front to access directly a
> PCI device from my PV guest.
> 
> If I read this correctly you are saying that the MMIO region is
> already mapped into my domain by xl? So when I go and read the BAR
> register from the PCI configuration I can just go and access it
> directly or do I have to update the VA mapping?

Stefano meant mapped in the p2m which under ARM or x86 HVM still means
you need to make a virtual mapping of those PFNs.

Under X86 PV "mapping" in this context really just means giving
permission to map the underlying MFNs, so you still need to create a
virtual mapping but they don't exist in your p2m until you put them
there.

Ian.

Re: [Query] Flow of PCI device dedicated to a domU

[Simon Martin]

Hello Ian,

>> If I read this correctly you are saying that the MMIO region is
>> already mapped into my domain by xl? So when I go and read the BAR
>> register from the PCI configuration I can just go and access it
>> directly or do I have to update the VA mapping?

> Stefano meant mapped in the p2m which under ARM or x86 HVM still means
> you need to make a virtual mapping of those PFNs.

> Under X86 PV "mapping" in this context really just means giving
> permission to map the underlying MFNs, so you still need to create a
> virtual mapping but they don't exist in your p2m until you put them
> there.

Thanks Ian, that is what I had understood from the mail that you sent
me on the thread I started about "Consuming PCI device in PV kernel".
Just wanted to confirm. I have just about worked my way up to
implementing the MMIO access now and wanted to make sure before I
carry on.

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: [Query] Flow of PCI device dedicated to a domU

[Stefano Stabellini]

On Tue, 22 Jul 2014, Simon Martin wrote:
> Hello Stefano,
> 
> > The basic mechanism is that Xen assigns the PCI root complex/host
> > bridge to Dom0. Dom0 walks the bus and finds out the devices available.
> 
> > When the user assigns a PCI device to a VM, the toolstack (xl/libxl)
> > issues hypercalls to remap interrupts and MMIO regions of that device
> > into the guest VM.
> 
> I am currently trying writing my own PCI front to access directly a
> PCI device from my PV guest.
> 
> If I read this correctly you are saying that the MMIO region is
> already mapped into my domain by xl?

Looking again at:

tools/libxl/libxl_pci.c:do_pci_add

for the PV guest case, the function gives ioport and mmio access
permissions to the guest and maps the irqs.

It doesn't directly map any MMIO regions but it allows the guest to do so.



> So when I go and read the BAR
> register from the PCI configuration I can just go and access it
> directly or do I have to update the VA mapping?

You do need to update the VA mapping, sorry for the confusion.

Re: [Query] Flow of PCI device dedicated to a domU

[Simon Martin]

Hello Stefano,


>> So when I go and read the BAR
>> register from the PCI configuration I can just go and access it
>> directly or do I have to update the VA mapping?

> You do need to update the VA mapping, sorry for the confusion.

Thanks for the clarification.

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: [Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

[-- Attachment #1.1: Type: text/plain, Size: 5522 bytes --]

On Tue, Jul 22, 2014 at 7:41 PM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:

> Please don't use HTML emails and please don't top post.
>
> On Tue, 22 Jul 2014, Xenbegn developer wrote:
> > Ian Can you please help in this. I am trying to understand how things
> are done in x86 xen. Based on that will
> > implement in ARM PCI passthrough
>
> The basic mechanism is that Xen assigns the PCI root complex/host
> bridge to Dom0. Dom0 walks the bus and finds out the devices available.
>
> When the user assigns a PCI device to a VM, the toolstack (xl/libxl)
> issues hypercalls to remap interrupts and MMIO regions of that device
> into the guest VM.
> The device is added to the guest virtual PCI bus that is created by the
> xen-pcifront driver at boot. PCI config space reads/writes go through
> xen-pcifront that talks to xen-pciback in Dom0.
>

[manish] In ARM can we do things a bit simpler way, like a hypervisor trap
when guest accesses configuration space rather than the pci-back pci-front
communication. Eliminating pci-front for config space access.

>
> PCI bus enumeration is done by Dom0 at boot time. Dom0 issues hypercalls
> to Xen to tell the hypervisor what devices are available. It would be
> conceivable to do the enumeration directly in Xen if Xen had the
> appropriate driver for the host bridge. It might be even better.
>
> The main interaction between Xen and the config space of PCI devices is
> about MSI programming. Xen programs MSIs directly because it is Xen that
> is going to receive them and then inject them into guests.
>
> [manish] Same logic as above

>
> If I were you I would base my work on Julien SMMU series and I would
> ignore MSIs for now. I would try to get pcifront and pciback running and
> then assign a PCI device to a guest, only using legacy interrupts.
>
> After that you could try to add support for doing PCI config space reads
> and writes in Xen. Then introduce MSI support but, as Ian wrote, you
> would need to rebase your work on the GICv3 series. I don't think you
> actually need full ITS support to be able to get MSI running though.
> You might already have everything you need in terms of patches, but they
> are still not applied to xen-unstable yet.
>
>
>
>
> > On Tue, Jul 22, 2014 at 4:21 PM, Xenbegn developer <
> xen.begn.dev@gmail.com> wrote:
> >
> >
> >
> >       On Tue, Jul 22, 2014 at 4:03 PM, Andrew Cooper <
> andrew.cooper3@citrix.com> wrote:
> >             On 22/07/14 11:13, Xenbegn developer wrote:
> >
> >
> >
> >       On Tue, Jul 22, 2014 at 3:26 PM, Andrew Cooper <
> andrew.cooper3@citrix.com> wrote:
> >             On 22/07/14 10:50, Xenbegn developer wrote:
> >
> >
> >
> >       On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper <
> andrew.cooper3@citrix.com> wrote:
> >             On 16/07/14 07:12, Xenbegn developer wrote:
> >       Hi,
> > I am trying to understand the flow of how a PCI device driver in domU
> works
> > after a PCI device is is assigned to a domU.
> >
> > a) If a PCI device is assigned to a domU, this device has to be on a PCI
> > bus. So as per my view xen would have to somehow provide a PCI Controller
> > on which this device is attached.
> > => Is my assumption correct ? If yes how it is done, No then also How
> > enumeration of this device happens in domU kernel
> >
> >
> > No.
> >
> > PV guests have no PCI root ports/bridges; they use devices as single
> entities
> > knowing that Xen/dom0 takes care of the other bits. HVM guests have
> their PCI
> > devices attached to the virtual southbridge which is all emulated by
> Qemu.
> >
> >
> > b) Is the Configuration space of the PCI device directly accessible to
> the
> > domU (assuming the kernel accesses it using memory map) ? If not then it
> is
> > trapped by xen
> >
> >
> > All configuration space is trap+emulate in Xen, although almost all
> operations
> > permitted.
> >
> >
> > c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how
> > Xen does a translation from Physical MSI to guest MSI (where in code)
> >
> >
> > Xen controls all interrupts on the system, which is why it needs to trap
> all
> > config accesses to notice when a domain is attempting to change the
> interrupt
> > information.  In that case, Xen fixes up its delivery of interrupts to
> the guest,
> > but leaves the underlying interrupt information intact.
> >
> > ~Andrew
> >
> >
> > Linux kernel pci code has a function called __write_msi_msg. This method
> writes to
> > config space for MSI/MSIX
> > pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
> >                        msg->address_lo);
> >
> > Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c
> (guest_io_write)
> > I dont see any MSI handling.
> > Can you please explain (point to the code) where domain is attempting to
> change the
> > interrupt information.
> >
> >
> >
> > PV guests are expected to use PHYSOP hypercalls for interrupt
> management.  Xen does not
> > provide transparent emulation support for them.
> >
> > HVM guests do get transparent emulation support, as from the point of
> view of an HVM guest,
> > they are talking to a real PCI device.
> >
> > Can you please point to a device driver code which is calling PHYSOP
> calls for setting an MSI Addr
> > + MSI number
> >
> >
> > No.  Partly because I don't know exactly, and mainly because this is
> something you should trivially be
> > able to search for.
> >
> > It is not that trivial. Can anoyone from Citrix reply ?
> >       ~Andrew
> >
> >
> >
> >
> >
>

[-- Attachment #1.2: Type: text/html, Size: 7251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 6861 bytes --]

Could you please use plain text in emails? HTML makes it harder for me
to read, as you can see below for yourself.


On Thu, 24 Jul 2014, Xenbegn developer wrote:
> On Tue, Jul 22, 2014 at 7:41 PM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
>       Please don't use HTML emails and please don't top post.
> 
>       On Tue, 22 Jul 2014, Xenbegn developer wrote:
>       > Ian Can you please help in this. I am trying to understand how things are done in x86 xen. Based on
>       that will
>       > implement in ARM PCI passthrough
> 
> The basic mechanism is that Xen assigns the PCI root complex/host
> bridge to Dom0. Dom0 walks the bus and finds out the devices available.
> 
> When the user assigns a PCI device to a VM, the toolstack (xl/libxl)
> issues hypercalls to remap interrupts and MMIO regions of that device
> into the guest VM.
> The device is added to the guest virtual PCI bus that is created by the
> xen-pcifront driver at boot. PCI config space reads/writes go through
> xen-pcifront that talks to xen-pciback in Dom0.
> 
> 
> [manish] In ARM can we do things a bit simpler way, like a hypervisor trap when guest accesses configuration space
> rather than the pci-back pci-front communication. Eliminating pci-front for config space access.

Actually you'll find that pcifront/pciback is the easiest possible
solution: no emulation is involved, no need to write hypervisor code.
Also both drivers already exist and are known to work well.  If you look
at the x86 emulation code for HVM guests, you'll that is way more
complex. I would avoid something like that for ARM if we can.


>       PCI bus enumeration is done by Dom0 at boot time. Dom0 issues hypercalls
>       to Xen to tell the hypervisor what devices are available. It would be
>       conceivable to do the enumeration directly in Xen if Xen had the
>       appropriate driver for the host bridge. It might be even better.
> 
>       The main interaction between Xen and the config space of PCI devices is
>       about MSI programming. Xen programs MSIs directly because it is Xen that
>       is going to receive them and then inject them into guests.
> 
> [manish] Same logic as above
> 
>       If I were you I would base my work on Julien SMMU series and I would
>       ignore MSIs for now. I would try to get pcifront and pciback running and
>       then assign a PCI device to a guest, only using legacy interrupts.
> 
>       After that you could try to add support for doing PCI config space reads
>       and writes in Xen. Then introduce MSI support but, as Ian wrote, you
>       would need to rebase your work on the GICv3 series. I don't think you
>       actually need full ITS support to be able to get MSI running though.
>       You might already have everything you need in terms of patches, but they
>       are still not applied to xen-unstable yet.
> 
> 
> 
> 
>       > On Tue, Jul 22, 2014 at 4:21 PM, Xenbegn developer <xen.begn.dev@gmail.com> wrote:
>       >
>       >
>       >
>       >       On Tue, Jul 22, 2014 at 4:03 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>       >             On 22/07/14 11:13, Xenbegn developer wrote:
>       >
>       >
>       >
>       >       On Tue, Jul 22, 2014 at 3:26 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>       >             On 22/07/14 10:50, Xenbegn developer wrote:
>       >
>       >
>       >
>       >       On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>       >             On 16/07/14 07:12, Xenbegn developer wrote:
>       >       Hi,
>       > I am trying to understand the flow of how a PCI device driver in domU works
>       > after a PCI device is is assigned to a domU.
>       >
>       > a) If a PCI device is assigned to a domU, this device has to be on a PCI
>       > bus. So as per my view xen would have to somehow provide a PCI Controller
>       > on which this device is attached.
>       > => Is my assumption correct ? If yes how it is done, No then also How
>       > enumeration of this device happens in domU kernel
>       >
>       >
>       > No.
>       >
>       > PV guests have no PCI root ports/bridges; they use devices as single entities
>       > knowing that Xen/dom0 takes care of the other bits. HVM guests have their PCI
>       > devices attached to the virtual southbridge which is all emulated by Qemu.
>       >
>       >
>       > b) Is the Configuration space of the PCI device directly accessible to the
>       > domU (assuming the kernel accesses it using memory map) ? If not then it is
>       > trapped by xen
>       >
>       >
>       > All configuration space is trap+emulate in Xen, although almost all operations
>       > permitted.
>       >
>       >
>       > c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how
>       > Xen does a translation from Physical MSI to guest MSI (where in code)
>       >
>       >
>       > Xen controls all interrupts on the system, which is why it needs to trap all
>       > config accesses to notice when a domain is attempting to change the interrupt
>       > information.  In that case, Xen fixes up its delivery of interrupts to the guest,
>       > but leaves the underlying interrupt information intact.
>       >
>       > ~Andrew
>       >
>       >
>       > Linux kernel pci code has a function called __write_msi_msg. This method writes to
>       > config space for MSI/MSIX
>       > pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
>       >                        msg->address_lo);
>       >
>       > Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c (guest_io_write)
>       > I dont see any MSI handling.
>       > Can you please explain (point to the code) where domain is attempting to change the
>       > interrupt information.
>       >    
>       >
>       >
>       > PV guests are expected to use PHYSOP hypercalls for interrupt management.  Xen does not
>       > provide transparent emulation support for them.
>       >
>       > HVM guests do get transparent emulation support, as from the point of view of an HVM guest,
>       > they are talking to a real PCI device.
>       >
>       > Can you please point to a device driver code which is calling PHYSOP calls for setting an MSI Addr
>       > + MSI number
>       >
>       >
>       > No.  Partly because I don't know exactly, and mainly because this is something you should trivially be
>       > able to search for.
>       >
>       > It is not that trivial. Can anoyone from Citrix reply ?
>       >       ~Andrew
>       >
>       >
>       >
>       >
>       >
> 
> 
> 
> 

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Xenbegn developer]

[-- Attachment #1.1: Type: text/plain, Size: 2218 bytes --]

On Tue, Jul 22, 2014 at 3:20 PM, Xenbegn developer <xen.begn.dev@gmail.com>
wrote:

>
>
>
> On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper <andrew.cooper3@citrix.com>
> wrote:
>
>>  On 16/07/14 07:12, Xenbegn developer wrote:
>>
>>     Hi,
>>  I am trying to understand the flow of how a PCI device driver in domU
>> works after a PCI device is is assigned to a domU.
>>
>>  a) If a PCI device is assigned to a domU, this device has to be on a PCI
>> bus. So as per my view xen would have to somehow provide a PCI Controller
>> on which this device is attached.
>>  => Is my assumption correct ? If yes how it is done, No then also How
>> enumeration of this device happens in domU kernel
>>
>>
>> No.
>>
>> PV guests have no PCI root ports/bridges; they use devices as single
>> entities knowing that Xen/dom0 takes care of the other bits. HVM guests
>> have their PCI devices attached to the virtual southbridge which is all
>> emulated by Qemu.
>>
>>
>>
>>  b) Is the Configuration space of the PCI device directly accessible to
>> the domU (assuming the kernel accesses it using memory map) ? If not then
>> it is trapped by xen
>>
>>
>> All configuration space is trap+emulate in Xen, although almost all
>> operations permitted.
>>
>>
>>
>>  c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how
>> Xen does a translation from Physical MSI to guest MSI (where in code)
>>
>>
>> Xen controls all interrupts on the system, which is why it needs to trap
>> all config accesses to notice when a domain is attempting to change the
>> interrupt information.  In that case, Xen fixes up its delivery of
>> interrupts to the guest, but leaves the underlying interrupt information
>> intact.
>>
>> ~Andrew
>>
>
> Linux kernel pci code has a function called __write_msi_msg. This method
> writes to config space for MSI/MSIX
> pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
>                        msg->address_lo);
>
> Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c
> (guest_io_write) I dont see any MSI handling.
> Can you please explain (point to the code) where domain is attempting to
> change the interrupt information.
>
>
> Stefano, Can you please comment on it.

[-- Attachment #1.2: Type: text/html, Size: 4319 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Ian Campbell]

On Thu, 2014-07-24 at 13:38 +0530, Xenbegn developer wrote:

>         
>         Can you please explain (point to the code) where domain is
>         attempting to change the interrupt information.

I'd recommend that you setup an x86 system with a passed through MSI
device and instrument it with printk etc so you can see how this all
fits together.

Ian.

Re: [Query] Flow of PCI device dedicated to a domU

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 3236 bytes --]

Please send email in plain text.


On Thu, 24 Jul 2014, Xenbegn developer wrote:
> On Tue, Jul 22, 2014 at 3:20 PM, Xenbegn developer <xen.begn.dev@gmail.com> wrote:
> 
> 
> 
>       On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>             On 16/07/14 07:12, Xenbegn developer wrote:
>       Hi,
> I am trying to understand the flow of how a PCI device driver in domU works after a PCI device is
> is assigned to a domU.
> 
> a) If a PCI device is assigned to a domU, this device has to be on a PCI bus. So as per my view xen
> would have to somehow provide a PCI Controller on which this device is attached.
> => Is my assumption correct ? If yes how it is done, No then also How enumeration of this device
> happens in domU kernel
> 
> 
> No.
> 
> PV guests have no PCI root ports/bridges; they use devices as single entities knowing that Xen/dom0 takes
> care of the other bits. HVM guests have their PCI devices attached to the virtual southbridge which is
> all emulated by Qemu.
> 
> 
> b) Is the Configuration space of the PCI device directly accessible to the domU (assuming the
> kernel accesses it using memory map) ? If not then it is trapped by xen
> 
> 
> All configuration space is trap+emulate in Xen, although almost all operations permitted.
> 
> 
> c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how Xen does a translation
> from Physical MSI to guest MSI (where in code)
> 
> 
> Xen controls all interrupts on the system, which is why it needs to trap all config accesses to notice
> when a domain is attempting to change the interrupt information.  In that case, Xen fixes up its delivery
> of interrupts to the guest, but leaves the underlying interrupt information intact.
> 
> ~Andrew
> 
> 
> Linux kernel pci code has a function called __write_msi_msg. This method writes to config space for MSI/MSIX
> pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
>                        msg->address_lo);
> 
> Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c (guest_io_write) I dont see any MSI
> handling.
> Can you please explain (point to the code) where domain is attempting to change the interrupt information.
>    
> 
> Stefano, Can you please comment on it.

Keep in mind that on x86 there are two ways to do PCI passthrough:

1) HVM guests, the PCI bridge is emulated, PCI accesses are trapped

2) PV guests, pcifront/pciback drivers are used, no emulation is done,
   no trapping

In ARM we would do 2). In the (2) case, x86/traps.c is not involved.

The guest enables MSIs via pcifront. pcifront would send the request to
pciback that would actually enable MSIs on the physical device.  See
drivers/xen/xen-pciback/pciback_ops.c:xen_pcibk_enable_msi.  The guest
kernel would map the MSI into a "pirq" to receive MSI notifications as
event channels.  No trapping is needed. See:
xen/arch/x86/msi.c:xen_setup_msi_irqs.  Event channels are a Xen
software interrupt mechanism, see drivers/xen/evtchn.c and
drivers/xen/events.  On ARM we would probably deliver physical MSIs as
virtual MSIs, thanks to the capabilities of the vgic.

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Query] Flow of PCI device dedicated to a domU

[Konrad Rzeszutek Wilk]

On Wed, Jul 16, 2014 at 11:42:05AM +0530, Xenbegn developer wrote:
> Hi,
> I am trying to understand the flow of how a PCI device driver in domU works
> after a PCI device is is assigned to a domU.

And by domU do you mean PV or HVM?

> 
> a) If a PCI device is assigned to a domU, this device has to be on a PCI
> bus. So as per my view xen would have to somehow provide a PCI Controller
> on which this device is attached.
> => Is my assumption correct ? If yes how it is done, No then also How
> enumeration of this device happens in domU kernel
> 
> b) Is the Configuration space of the PCI device directly accessible to the
> domU (assuming the kernel accesses it using memory map) ? If not then it is
> trapped by xen
> 
> c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how
> Xen does a translation from Physical MSI to guest MSI (where in code)
> 
> Please help me with these doubts
> 
> 
> Regards
> Manish

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel