SR-IOV: do we need to virtualize in Xen or rely on Dom0?

SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Oleksandr Andrushchenko]

Hi, all!

While working on PCI SR-IOV support for ARM I started porting [1] on top
of current PCI on ARM support [2]. The question I have for this series
is if we really need emulating SR-IOV code in Xen?

I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2 
patches)
and it "works for me": MSI support is still WIP, but I was able to see that
VFs are properly seen in the guest and BARs are properly programmed in p2m.

What I can't fully understand is if we can live with this approach or there
are use-cases I can't see.

Previously I've been told that this approach might not work on FreeBSD 
running
as Domain-0, but is seems that "PCI Passthrough is not supported 
(Xen/FreeBSD)"
anyways [4].

I also see ACRN hypervisor [5] implements SR-IOV inside it which makes 
me think I
miss some important use-case on x86 though.

I would like to ask for any advise with SR-IOV in hypervisor respect, 
any pointers
to documentation or any other source which might be handy in deciding if 
we do
need SR-IOV complexity in Xen.

And it does bring complexity if you compare [1] and [3])...

A bit of technical details on the approach implemented [3]:
1. We rely on PHYSDEVOP_pci_device_add
2. We rely on Domain-0 SR-IOV drivers to instantiate VFs
3. BARs are programmed in p2m implementing guest view on those (we have 
extended
vPCI code for that and this path is used for both "normal" devices and 
VFs the same way)
4. No need to trap PCI_SRIOV_CTRL
5. No need to wait 100ms in Xen before attempting to access VF registers 
when
enabling virtual functions on the PF - this is handled by Domain-0 itself.

Thank you in advance,
Oleksandr

[1] 
https://lists.xenproject.org/archives/html/xen-devel/2018-07/msg01494.html
[2] 
https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough
[3] https://github.com/xen-troops/xen/commits/pci_phase2
[4] https://wiki.freebsd.org/Xen
[5] https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Roger Pau Monné]

On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
> Hi, all!
> 
> While working on PCI SR-IOV support for ARM I started porting [1] on top
> of current PCI on ARM support [2]. The question I have for this series
> is if we really need emulating SR-IOV code in Xen?
> 
> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2 
> patches)
> and it "works for me": MSI support is still WIP, but I was able to see that
> VFs are properly seen in the guest and BARs are properly programmed in p2m.
> 
> What I can't fully understand is if we can live with this approach or there
> are use-cases I can't see.
> 
> Previously I've been told that this approach might not work on FreeBSD 
> running
> as Domain-0, but is seems that "PCI Passthrough is not supported 
> (Xen/FreeBSD)"
> anyways [4].

PCI passthorgh is not supported on FreeBSD dom0 because PCI
passthrough is not supported by Xen itself when using a PVH dom0, and
that's the only mode FreeBSD dom0 can use.

PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
to work. I however think this is not the proper way to implement
SR-IOV support.

> 
> I also see ACRN hypervisor [5] implements SR-IOV inside it which makes 
> me think I
> miss some important use-case on x86 though.
> 
> I would like to ask for any advise with SR-IOV in hypervisor respect, 
> any pointers
> to documentation or any other source which might be handy in deciding if 
> we do
> need SR-IOV complexity in Xen.
> 
> And it does bring complexity if you compare [1] and [3])...
> 
> A bit of technical details on the approach implemented [3]:
> 1. We rely on PHYSDEVOP_pci_device_add
> 2. We rely on Domain-0 SR-IOV drivers to instantiate VFs
> 3. BARs are programmed in p2m implementing guest view on those (we have 
> extended
> vPCI code for that and this path is used for both "normal" devices and 
> VFs the same way)
> 4. No need to trap PCI_SRIOV_CTRL
> 5. No need to wait 100ms in Xen before attempting to access VF registers 
> when
> enabling virtual functions on the PF - this is handled by Domain-0 itself.

I think the SR-IOV capability should be handled like any other PCI
capability, ie: like we currently handle MSI or MSI-X in vPCI.

It's likely that using some kind of hypercall in order to deal with
SR-IOV could make this easier to implement in Xen, but that just adds
more code to all OSes that want to run as the hardware domain.

OTOH if we properly trap accesses to the SR-IOV capability (like it
was proposed in [1] from your references) we won't have to modify OSes
that want to run as hardware domains in order to handle SR-IOV devices.

IMO going for the hypercall option seems easier now, but adds a burden
to all OSes that want to manage SR-IOV devices that will hurt us long
term.

Thanks, Roger.

> Thank you in advance,
> Oleksandr
> 
> [1] 
> https://lists.xenproject.org/archives/html/xen-devel/2018-07/msg01494.html
> [2] 
> https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough
> [3] https://github.com/xen-troops/xen/commits/pci_phase2
> [4] https://wiki.freebsd.org/Xen
> [5] https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Oleksandr Andrushchenko]

Hello, Roger!

On 10.06.21 10:54, Roger Pau Monné wrote:
> On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
>> Hi, all!
>>
>> While working on PCI SR-IOV support for ARM I started porting [1] on top
>> of current PCI on ARM support [2]. The question I have for this series
>> is if we really need emulating SR-IOV code in Xen?
>>
>> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2
>> patches)
>> and it "works for me": MSI support is still WIP, but I was able to see that
>> VFs are properly seen in the guest and BARs are properly programmed in p2m.
>>
>> What I can't fully understand is if we can live with this approach or there
>> are use-cases I can't see.
>>
>> Previously I've been told that this approach might not work on FreeBSD
>> running
>> as Domain-0, but is seems that "PCI Passthrough is not supported
>> (Xen/FreeBSD)"
>> anyways [4].
> PCI passthorgh is not supported on FreeBSD dom0 because PCI
> passthrough is not supported by Xen itself when using a PVH dom0, and
> that's the only mode FreeBSD dom0 can use.

So, it is still not clear to me: how and if PCI passthrough is supported

on FreeBSD, what are the scenarios and requirements for that?

>
> PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
> to work. I however think this is not the proper way to implement
> SR-IOV support.

I was not able to find any support for PHYSDEVOP_XXX in FreeBSD code,

could you please point me to where are these used?

If they are not, so how Xen under FreeBSD knows about PCI devices?

I am trying to extrapolate my knowledge of how Linux does that

(during PCI enumeration in Domain-0 we use hypercalls)

>
>> I also see ACRN hypervisor [5] implements SR-IOV inside it which makes
>> me think I
>> miss some important use-case on x86 though.
>>
>> I would like to ask for any advise with SR-IOV in hypervisor respect,
>> any pointers
>> to documentation or any other source which might be handy in deciding if
>> we do
>> need SR-IOV complexity in Xen.
>>
>> And it does bring complexity if you compare [1] and [3])...
>>
>> A bit of technical details on the approach implemented [3]:
>> 1. We rely on PHYSDEVOP_pci_device_add
>> 2. We rely on Domain-0 SR-IOV drivers to instantiate VFs
>> 3. BARs are programmed in p2m implementing guest view on those (we have
>> extended
>> vPCI code for that and this path is used for both "normal" devices and
>> VFs the same way)
>> 4. No need to trap PCI_SRIOV_CTRL
>> 5. No need to wait 100ms in Xen before attempting to access VF registers
>> when
>> enabling virtual functions on the PF - this is handled by Domain-0 itself.
> I think the SR-IOV capability should be handled like any other PCI
> capability, ie: like we currently handle MSI or MSI-X in vPCI.
>
> It's likely that using some kind of hypercall in order to deal with
> SR-IOV could make this easier to implement in Xen, but that just adds
> more code to all OSes that want to run as the hardware domain.

I didn't introduce any new, but PHYSDEVOP_pci_device_add was enough.

The rest I did in Xen itself wrt SR-IOV.

>
> OTOH if we properly trap accesses to the SR-IOV capability (like it
> was proposed in [1] from your references) we won't have to modify OSes
> that want to run as hardware domains in order to handle SR-IOV devices.

Out of curiosity, could you please name a few? I do understand that

we do want to support unmodified OSes and this is indeed important.

But, still what are the other OSes which do support Xen + PCI passthrough?

>
> IMO going for the hypercall option seems easier now, but adds a burden
> to all OSes that want to manage SR-IOV devices that will hurt us long
> term.

Again, I was able to make it somewhat work with PHYSDEVOP_pci_device_add only.

>
> Thanks, Roger.

Thank you for your valuable comments,

Oleksandr

>
>> Thank you in advance,
>> Oleksandr
>>
>> [1]
>> https://urldefense.com/v3/__https://lists.xenproject.org/archives/html/xen-devel/2018-07/msg01494.html__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CezmvSZGg$ [lists[.]xenproject[.]org]
>> [2]
>> https://urldefense.com/v3/__https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8Cf3Z8Yk7w$ [gitlab[.]com]
>> [3] https://urldefense.com/v3/__https://github.com/xen-troops/xen/commits/pci_phase2__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CfncIdgvw$ [github[.]com]
>> [4] https://urldefense.com/v3/__https://wiki.freebsd.org/Xen__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CceLOPdyg$ [wiki[.]freebsd[.]org]
>> [5] https://urldefense.com/v3/__https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CdjqHPxZQ$ [projectacrn[.]github[.]io]

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Jan Beulich]

On 10.06.2021 12:01, Oleksandr Andrushchenko wrote:
> On 10.06.21 10:54, Roger Pau Monné wrote:
>> OTOH if we properly trap accesses to the SR-IOV capability (like it
>> was proposed in [1] from your references) we won't have to modify OSes
>> that want to run as hardware domains in order to handle SR-IOV devices.
> 
> Out of curiosity, could you please name a few? I do understand that
> 
> we do want to support unmodified OSes and this is indeed important.
> 
> But, still what are the other OSes which do support Xen + PCI passthrough?

I think Roger saying "want" meant to cover ones which currently don't,
and which would have to undergo more extensive changes if they were to
be enabled.

Jan

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Oleksandr Andrushchenko]

Hi, Jan!

On 10.06.21 13:48, Jan Beulich wrote:
> On 10.06.2021 12:01, Oleksandr Andrushchenko wrote:
>> On 10.06.21 10:54, Roger Pau Monné wrote:
>>> OTOH if we properly trap accesses to the SR-IOV capability (like it
>>> was proposed in [1] from your references) we won't have to modify OSes
>>> that want to run as hardware domains in order to handle SR-IOV devices.
>> Out of curiosity, could you please name a few? I do understand that
>>
>> we do want to support unmodified OSes and this is indeed important.
>>
>> But, still what are the other OSes which do support Xen + PCI passthrough?
> I think Roger saying "want" meant to cover ones which currently don't,
> and which would have to undergo more extensive changes if they were to
> be enabled.

Fair enough. Do you think we would also need to re-work the existing code

in Xen to support normal devices (not SR-IOV), e.g. we currently rely on

PHYSDEVOP_XXX and other Linux specifics. And even if SR-IOV is implemented

in Xen this won't allow those OSes to stay unmodified, including FreeBSD.

Is my understanding correct?

>
> Jan
>
Thank you,

Oleksandr

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Jan Beulich]

On 10.06.2021 13:45, Oleksandr Andrushchenko wrote:
> Hi, Jan!
> 
> On 10.06.21 13:48, Jan Beulich wrote:
>> On 10.06.2021 12:01, Oleksandr Andrushchenko wrote:
>>> On 10.06.21 10:54, Roger Pau Monné wrote:
>>>> OTOH if we properly trap accesses to the SR-IOV capability (like it
>>>> was proposed in [1] from your references) we won't have to modify OSes
>>>> that want to run as hardware domains in order to handle SR-IOV devices.
>>> Out of curiosity, could you please name a few? I do understand that
>>>
>>> we do want to support unmodified OSes and this is indeed important.
>>>
>>> But, still what are the other OSes which do support Xen + PCI passthrough?
>> I think Roger saying "want" meant to cover ones which currently don't,
>> and which would have to undergo more extensive changes if they were to
>> be enabled.
> 
> Fair enough. Do you think we would also need to re-work the existing code
> 
> in Xen to support normal devices (not SR-IOV), e.g. we currently rely on
> 
> PHYSDEVOP_XXX and other Linux specifics.

Yes, work in that area would also be needed. For example we'd need to
scan buses / segments as they become accessible. Right now we only scan
segment 0, and even that's only possible because on x86 mmconfig is not
the only way to access config space.

> And even if SR-IOV is implemented
> 
> in Xen this won't allow those OSes to stay unmodified, including FreeBSD.

Of course, it's the nature of PVH (as opposed to HVM) that OSes need
modification. The question is the scope thereof.

Jan

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Oleksandr Andrushchenko]

On 10.06.21 15:02, Jan Beulich wrote:
> On 10.06.2021 13:45, Oleksandr Andrushchenko wrote:
>> Hi, Jan!
>>
>> On 10.06.21 13:48, Jan Beulich wrote:
>>> On 10.06.2021 12:01, Oleksandr Andrushchenko wrote:
>>>> On 10.06.21 10:54, Roger Pau Monné wrote:
>>>>> OTOH if we properly trap accesses to the SR-IOV capability (like it
>>>>> was proposed in [1] from your references) we won't have to modify OSes
>>>>> that want to run as hardware domains in order to handle SR-IOV devices.
>>>> Out of curiosity, could you please name a few? I do understand that
>>>>
>>>> we do want to support unmodified OSes and this is indeed important.
>>>>
>>>> But, still what are the other OSes which do support Xen + PCI passthrough?
>>> I think Roger saying "want" meant to cover ones which currently don't,
>>> and which would have to undergo more extensive changes if they were to
>>> be enabled.
>> Fair enough. Do you think we would also need to re-work the existing code
>>
>> in Xen to support normal devices (not SR-IOV), e.g. we currently rely on
>>
>> PHYSDEVOP_XXX and other Linux specifics.
> Yes, work in that area would also be needed. For example we'd need to
> scan buses / segments as they become accessible. Right now we only scan
> segment 0, and even that's only possible because on x86 mmconfig is not
> the only way to access config space.
>
>> And even if SR-IOV is implemented
>>
>> in Xen this won't allow those OSes to stay unmodified, including FreeBSD.
> Of course, it's the nature of PVH (as opposed to HVM) that OSes need
> modification. The question is the scope thereof.

Ok, then it seems I need to get [1] back into the picture.

I have modified vPCI code a lot for ARM support, so [1] will not apply

as is anymore and needs to be re-worked. But, still it can mostly be re-used


> Jan
>
Thank you,

Oleksandr

[1] https://lists.xenproject.org/archives/html/xen-devel/2018-07/msg01494.html

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Roger Pau Monné]

On Thu, Jun 10, 2021 at 10:01:16AM +0000, Oleksandr Andrushchenko wrote:
> Hello, Roger!
> 
> On 10.06.21 10:54, Roger Pau Monné wrote:
> > On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
> >> Hi, all!
> >>
> >> While working on PCI SR-IOV support for ARM I started porting [1] on top
> >> of current PCI on ARM support [2]. The question I have for this series
> >> is if we really need emulating SR-IOV code in Xen?
> >>
> >> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2
> >> patches)
> >> and it "works for me": MSI support is still WIP, but I was able to see that
> >> VFs are properly seen in the guest and BARs are properly programmed in p2m.
> >>
> >> What I can't fully understand is if we can live with this approach or there
> >> are use-cases I can't see.
> >>
> >> Previously I've been told that this approach might not work on FreeBSD
> >> running
> >> as Domain-0, but is seems that "PCI Passthrough is not supported
> >> (Xen/FreeBSD)"
> >> anyways [4].
> > PCI passthorgh is not supported on FreeBSD dom0 because PCI
> > passthrough is not supported by Xen itself when using a PVH dom0, and
> > that's the only mode FreeBSD dom0 can use.
> 
> So, it is still not clear to me: how and if PCI passthrough is supported
> 
> on FreeBSD, what are the scenarios and requirements for that?
> 
> >
> > PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
> > to work. I however think this is not the proper way to implement
> > SR-IOV support.
> 
> I was not able to find any support for PHYSDEVOP_XXX in FreeBSD code,
> 
> could you please point me to where are these used?

Those are not used on FreeBSD, because x86 PVHv2 dom0 doesn't
implement them anymore. They are implemented on Linux for x86 PV dom0,
AFAIK Arm doesn't use them either.

> If they are not, so how Xen under FreeBSD knows about PCI devices?

Xen scans the PCI bus itself, see scan_pci_devices.

> I am trying to extrapolate my knowledge of how Linux does that
> 
> (during PCI enumeration in Domain-0 we use hypercalls)
> 
> >
> >> I also see ACRN hypervisor [5] implements SR-IOV inside it which makes
> >> me think I
> >> miss some important use-case on x86 though.
> >>
> >> I would like to ask for any advise with SR-IOV in hypervisor respect,
> >> any pointers
> >> to documentation or any other source which might be handy in deciding if
> >> we do
> >> need SR-IOV complexity in Xen.
> >>
> >> And it does bring complexity if you compare [1] and [3])...
> >>
> >> A bit of technical details on the approach implemented [3]:
> >> 1. We rely on PHYSDEVOP_pci_device_add
> >> 2. We rely on Domain-0 SR-IOV drivers to instantiate VFs
> >> 3. BARs are programmed in p2m implementing guest view on those (we have
> >> extended
> >> vPCI code for that and this path is used for both "normal" devices and
> >> VFs the same way)
> >> 4. No need to trap PCI_SRIOV_CTRL
> >> 5. No need to wait 100ms in Xen before attempting to access VF registers
> >> when
> >> enabling virtual functions on the PF - this is handled by Domain-0 itself.
> > I think the SR-IOV capability should be handled like any other PCI
> > capability, ie: like we currently handle MSI or MSI-X in vPCI.
> >
> > It's likely that using some kind of hypercall in order to deal with
> > SR-IOV could make this easier to implement in Xen, but that just adds
> > more code to all OSes that want to run as the hardware domain.
> 
> I didn't introduce any new, but PHYSDEVOP_pci_device_add was enough.

Well, that would be 'new' on x86 PVH or Arm, as they don't implement
any PHYSDEVOP at the moment.

Long term we might need an hypercall to report dynamic MCFG regions,
but I haven't got around to it yet (and haven't found any system that
reports extra MCFG regions from ACPI AML).

> The rest I did in Xen itself wrt SR-IOV.
> 
> >
> > OTOH if we properly trap accesses to the SR-IOV capability (like it
> > was proposed in [1] from your references) we won't have to modify OSes
> > that want to run as hardware domains in order to handle SR-IOV devices.
> 
> Out of curiosity, could you please name a few? I do understand that
> 
> we do want to support unmodified OSes and this is indeed important.
> 
> But, still what are the other OSes which do support Xen + PCI passthrough?

NetBSD PV dom0 does support PCI passthrough, but I'm not sure that's
relevant.

We shouldn't focus on current users to come up with an interface,
but rather think how we want that interface to be.

As I said on the previous email my opinion is that unless not
technically possible we should just trap accesses to the SR-IOV
capability like we do for MSI(-X) and handle it transparently from a
guest PoV.

> >
> > IMO going for the hypercall option seems easier now, but adds a burden
> > to all OSes that want to manage SR-IOV devices that will hurt us long
> > term.
> 
> Again, I was able to make it somewhat work with PHYSDEVOP_pci_device_add only.

Sure, that's how it works on x86 PV hardware domain, so it's certainly
possible. My comments to avoid that route is not because it's not
technically feasible, but because I don't like the approach.

So far we have avoided PVH from having to implement any PHYSDEVOP
hypercall, and that's a design decision, not a coincidence. I'm in
favor of using the existing hardware interfaces for guests instead of
introducing custom Xen ones when technically feasible.

Thanks, Roger.

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Oleksandr Andrushchenko]

On 10.06.21 17:10, Roger Pau Monné wrote:
> On Thu, Jun 10, 2021 at 10:01:16AM +0000, Oleksandr Andrushchenko wrote:
>> Hello, Roger!
>>
>> On 10.06.21 10:54, Roger Pau Monné wrote:
>>> On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
>>>> Hi, all!
>>>>
>>>> While working on PCI SR-IOV support for ARM I started porting [1] on top
>>>> of current PCI on ARM support [2]. The question I have for this series
>>>> is if we really need emulating SR-IOV code in Xen?
>>>>
>>>> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2
>>>> patches)
>>>> and it "works for me": MSI support is still WIP, but I was able to see that
>>>> VFs are properly seen in the guest and BARs are properly programmed in p2m.
>>>>
>>>> What I can't fully understand is if we can live with this approach or there
>>>> are use-cases I can't see.
>>>>
>>>> Previously I've been told that this approach might not work on FreeBSD
>>>> running
>>>> as Domain-0, but is seems that "PCI Passthrough is not supported
>>>> (Xen/FreeBSD)"
>>>> anyways [4].
>>> PCI passthorgh is not supported on FreeBSD dom0 because PCI
>>> passthrough is not supported by Xen itself when using a PVH dom0, and
>>> that's the only mode FreeBSD dom0 can use.
>> So, it is still not clear to me: how and if PCI passthrough is supported
>>
>> on FreeBSD, what are the scenarios and requirements for that?
>>
>>> PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
>>> to work. I however think this is not the proper way to implement
>>> SR-IOV support.
>> I was not able to find any support for PHYSDEVOP_XXX in FreeBSD code,
>>
>> could you please point me to where are these used?
> Those are not used on FreeBSD, because x86 PVHv2 dom0 doesn't
> implement them anymore. They are implemented on Linux for x86 PV dom0,
> AFAIK Arm doesn't use them either.

Well, ARM didn't until we started implementing PCI passthrough [1].

It was previously discussed [2], "# Discovering PCI devices:" and proposed

to use PHYSDEVOP_pci_device_add.

Long story short, it is not easy for ARM to enumerate PCI devices in Xen as there is

no unified way of doing so: different platforms implement different PCI host bridges

which require complex initialization including clocks, power domains etc.

It was also discussed that PCI on ARM would want to support dom0less (DomB) setups,

so we should have some bootloader which will enumerate PCI devices for Xen beforehand

and Xen will only support ECAM-based host bridges.

Anyways, as the above does not exist yet, we use PHYSDEVOP_pci_device_add on ARM.

And we rely on Dom0 to initialize PCI host bridge, so Xen can also access PCI.

>
>> If they are not, so how Xen under FreeBSD knows about PCI devices?
> Xen scans the PCI bus itself, see scan_pci_devices.
See above, this is not yet available on ARM
>
>> I am trying to extrapolate my knowledge of how Linux does that
>>
>> (during PCI enumeration in Domain-0 we use hypercalls)
>>
>>>> I also see ACRN hypervisor [5] implements SR-IOV inside it which makes
>>>> me think I
>>>> miss some important use-case on x86 though.
>>>>
>>>> I would like to ask for any advise with SR-IOV in hypervisor respect,
>>>> any pointers
>>>> to documentation or any other source which might be handy in deciding if
>>>> we do
>>>> need SR-IOV complexity in Xen.
>>>>
>>>> And it does bring complexity if you compare [1] and [3])...
>>>>
>>>> A bit of technical details on the approach implemented [3]:
>>>> 1. We rely on PHYSDEVOP_pci_device_add
>>>> 2. We rely on Domain-0 SR-IOV drivers to instantiate VFs
>>>> 3. BARs are programmed in p2m implementing guest view on those (we have
>>>> extended
>>>> vPCI code for that and this path is used for both "normal" devices and
>>>> VFs the same way)
>>>> 4. No need to trap PCI_SRIOV_CTRL
>>>> 5. No need to wait 100ms in Xen before attempting to access VF registers
>>>> when
>>>> enabling virtual functions on the PF - this is handled by Domain-0 itself.
>>> I think the SR-IOV capability should be handled like any other PCI
>>> capability, ie: like we currently handle MSI or MSI-X in vPCI.
>>>
>>> It's likely that using some kind of hypercall in order to deal with
>>> SR-IOV could make this easier to implement in Xen, but that just adds
>>> more code to all OSes that want to run as the hardware domain.
>> I didn't introduce any new, but PHYSDEVOP_pci_device_add was enough.
> Well, that would be 'new' on x86 PVH or Arm, as they don't implement
> any PHYSDEVOP at the moment.
Agree for x86 PVH
>
> Long term we might need an hypercall to report dynamic MCFG regions,
> but I haven't got around to it yet (and haven't found any system that
> reports extra MCFG regions from ACPI AML).
Which means we'll need to modify guest OS
>
>> The rest I did in Xen itself wrt SR-IOV.
>>
>>> OTOH if we properly trap accesses to the SR-IOV capability (like it
>>> was proposed in [1] from your references) we won't have to modify OSes
>>> that want to run as hardware domains in order to handle SR-IOV devices.
>> Out of curiosity, could you please name a few? I do understand that
>>
>> we do want to support unmodified OSes and this is indeed important.
>>
>> But, still what are the other OSes which do support Xen + PCI passthrough?
> NetBSD PV dom0 does support PCI passthrough, but I'm not sure that's
> relevant.

That was just for me to understand where to look for the PCI passthrough

implementations and not to break something which I don't see

>
> We shouldn't focus on current users to come up with an interface,
> but rather think how we want that interface to be.
>
> As I said on the previous email my opinion is that unless not
> technically possible we should just trap accesses to the SR-IOV
> capability like we do for MSI(-X) and handle it transparently from a
> guest PoV.

Ok, I understand. It seems that Jan also supports your idea. So, I am not

against that, just trying to see the whole picture which is a bit bigger than ARM

>
>>> IMO going for the hypercall option seems easier now, but adds a burden
>>> to all OSes that want to manage SR-IOV devices that will hurt us long
>>> term.
>> Again, I was able to make it somewhat work with PHYSDEVOP_pci_device_add only.
> Sure, that's how it works on x86 PV hardware domain, so it's certainly
> possible. My comments to avoid that route is not because it's not
> technically feasible, but because I don't like the approach.

Unless we have some unified way of accessing PCI on ARM I am not sure

we can live without PHYSDEVOP_pci_device_add hypercall.

>
> So far we have avoided PVH from having to implement any PHYSDEVOP
> hypercall, and that's a design decision, not a coincidence. I'm in
> favor of using the existing hardware interfaces for guests instead of
> introducing custom Xen ones when technically feasible.

Unfortunately, on ARM (and I believe it may also happen on other

non-x86 platforms) there are new obstacles to this design. And if

we want Xen + PCI be supported on other than x86 platforms we have

to re-think the existing approach to include others in the game.

>
> Thanks, Roger.

Thank you,

Oleksandr

[1] https://gitlab.com/xen-project/fusa/xen-integration/-/blob/integration/pci-passthrough/xen/arch/arm/physdev.c

[2] https://www.mail-archive.com/xen-devel@lists.xenproject.org/msg77422.html

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Jan Beulich]

On 10.06.2021 17:33, Oleksandr Andrushchenko wrote:
> On 10.06.21 17:10, Roger Pau Monné wrote:
>> On Thu, Jun 10, 2021 at 10:01:16AM +0000, Oleksandr Andrushchenko wrote:
>>> On 10.06.21 10:54, Roger Pau Monné wrote:
>>>> On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
>>>>> Hi, all!
>>>>>
>>>>> While working on PCI SR-IOV support for ARM I started porting [1] on top
>>>>> of current PCI on ARM support [2]. The question I have for this series
>>>>> is if we really need emulating SR-IOV code in Xen?
>>>>>
>>>>> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2
>>>>> patches)
>>>>> and it "works for me": MSI support is still WIP, but I was able to see that
>>>>> VFs are properly seen in the guest and BARs are properly programmed in p2m.
>>>>>
>>>>> What I can't fully understand is if we can live with this approach or there
>>>>> are use-cases I can't see.
>>>>>
>>>>> Previously I've been told that this approach might not work on FreeBSD
>>>>> running
>>>>> as Domain-0, but is seems that "PCI Passthrough is not supported
>>>>> (Xen/FreeBSD)"
>>>>> anyways [4].
>>>> PCI passthorgh is not supported on FreeBSD dom0 because PCI
>>>> passthrough is not supported by Xen itself when using a PVH dom0, and
>>>> that's the only mode FreeBSD dom0 can use.
>>> So, it is still not clear to me: how and if PCI passthrough is supported
>>>
>>> on FreeBSD, what are the scenarios and requirements for that?
>>>
>>>> PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
>>>> to work. I however think this is not the proper way to implement
>>>> SR-IOV support.
>>> I was not able to find any support for PHYSDEVOP_XXX in FreeBSD code,
>>>
>>> could you please point me to where are these used?
>> Those are not used on FreeBSD, because x86 PVHv2 dom0 doesn't
>> implement them anymore. They are implemented on Linux for x86 PV dom0,
>> AFAIK Arm doesn't use them either.
> 
> Well, ARM didn't until we started implementing PCI passthrough [1].
> 
> It was previously discussed [2], "# Discovering PCI devices:" and proposed
> 
> to use PHYSDEVOP_pci_device_add.
> 
> Long story short, it is not easy for ARM to enumerate PCI devices in Xen as there is
> 
> no unified way of doing so: different platforms implement different PCI host bridges
> 
> which require complex initialization including clocks, power domains etc.

Just for my own understanding: If this isn't done by firmware, doesn't
this mean you can't boot an Arm system from e.g. a disk connected through
a PCI-based controller? Host bridge setup is definitely firmware's job on
x86 ...

Jan

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Oleksandr Andrushchenko]

On 11.06.21 09:45, Jan Beulich wrote:
> On 10.06.2021 17:33, Oleksandr Andrushchenko wrote:
>> On 10.06.21 17:10, Roger Pau Monné wrote:
>>> On Thu, Jun 10, 2021 at 10:01:16AM +0000, Oleksandr Andrushchenko wrote:
>>>> On 10.06.21 10:54, Roger Pau Monné wrote:
>>>>> On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
>>>>>> Hi, all!
>>>>>>
>>>>>> While working on PCI SR-IOV support for ARM I started porting [1] on top
>>>>>> of current PCI on ARM support [2]. The question I have for this series
>>>>>> is if we really need emulating SR-IOV code in Xen?
>>>>>>
>>>>>> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2
>>>>>> patches)
>>>>>> and it "works for me": MSI support is still WIP, but I was able to see that
>>>>>> VFs are properly seen in the guest and BARs are properly programmed in p2m.
>>>>>>
>>>>>> What I can't fully understand is if we can live with this approach or there
>>>>>> are use-cases I can't see.
>>>>>>
>>>>>> Previously I've been told that this approach might not work on FreeBSD
>>>>>> running
>>>>>> as Domain-0, but is seems that "PCI Passthrough is not supported
>>>>>> (Xen/FreeBSD)"
>>>>>> anyways [4].
>>>>> PCI passthorgh is not supported on FreeBSD dom0 because PCI
>>>>> passthrough is not supported by Xen itself when using a PVH dom0, and
>>>>> that's the only mode FreeBSD dom0 can use.
>>>> So, it is still not clear to me: how and if PCI passthrough is supported
>>>>
>>>> on FreeBSD, what are the scenarios and requirements for that?
>>>>
>>>>> PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
>>>>> to work. I however think this is not the proper way to implement
>>>>> SR-IOV support.
>>>> I was not able to find any support for PHYSDEVOP_XXX in FreeBSD code,
>>>>
>>>> could you please point me to where are these used?
>>> Those are not used on FreeBSD, because x86 PVHv2 dom0 doesn't
>>> implement them anymore. They are implemented on Linux for x86 PV dom0,
>>> AFAIK Arm doesn't use them either.
>> Well, ARM didn't until we started implementing PCI passthrough [1].
>>
>> It was previously discussed [2], "# Discovering PCI devices:" and proposed
>>
>> to use PHYSDEVOP_pci_device_add.
>>
>> Long story short, it is not easy for ARM to enumerate PCI devices in Xen as there is
>>
>> no unified way of doing so: different platforms implement different PCI host bridges
>>
>> which require complex initialization including clocks, power domains etc.
> Just for my own understanding: If this isn't done by firmware, doesn't
> this mean you can't boot an Arm system from e.g. a disk connected through
> a PCI-based controller? Host bridge setup is definitely firmware's job on
> x86 ...

On the platforms I work with: no, you can't. Well, it is possible to add PCI

support to the firmware, but we normally boot out of eMMC, SD, network and

all those are typically NOT PCI devices.

Even more. In my everyday work I don't use (need) any PCI device to run

the system at all.


> Jan
>
Thank you,

Oleksandr