[Xen-devel] PCIe IOMMU ACS support

[Xen-devel] PCIe IOMMU ACS support

[Roman Shaposhnik]

Hi!

I was going through how Xen support PCIe IOMMU ACS and
all I could find is this:
    https://github.com/xen-project/xen/blob/master/xen/drivers/passthrough/pci.c#L608
which looks to me as an attempt of enabling ACS opportunistically,
but still proceeding forward even if it fails.

Am I missing something here? IOW, does Xen try to do any kind of
fine grained ACS differentiation along the lines of what linux kernel
is doing:
    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/quirks.c#n4299

Thanks,
Roman.

Re: [Xen-devel] PCIe IOMMU ACS support

[Roger Pau Monné]

Adding the PCI and IOMMU maintainers.

On Mon, Mar 23, 2020 at 01:55:01PM -0700, Roman Shaposhnik wrote:
> Hi!
> 
> I was going through how Xen support PCIe IOMMU ACS and
> all I could find is this:
>     https://github.com/xen-project/xen/blob/master/xen/drivers/passthrough/pci.c#L608
> which looks to me as an attempt of enabling ACS opportunistically,
> but still proceeding forward even if it fails.

That's correct AFAICT. Xen will try to enable some features, but will
proceed normally if ACS is not available, or if some of the features
are not implemented.

Are you looking to ensure that all devices on the system have a
certain feature enabled?

Can you provide some more details about what you expect of ACS
handling?

Thanks, Roger.

Re: [Xen-devel] PCIe IOMMU ACS support

[Roman Shaposhnik]

On Wed, Mar 25, 2020 at 4:05 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> Adding the PCI and IOMMU maintainers.
>
> On Mon, Mar 23, 2020 at 01:55:01PM -0700, Roman Shaposhnik wrote:
> > Hi!
> >
> > I was going through how Xen support PCIe IOMMU ACS and
> > all I could find is this:
> >     https://github.com/xen-project/xen/blob/master/xen/drivers/passthrough/pci.c#L608
> > which looks to me as an attempt of enabling ACS opportunistically,
> > but still proceeding forward even if it fails.
>
> That's correct AFAICT. Xen will try to enable some features, but will
> proceed normally if ACS is not available, or if some of the features
> are not implemented.
>
> Are you looking to ensure that all devices on the system have a
> certain feature enabled?

My primary objective was to get some visibility into how Xen would
prevent two PCIe devices behind a common bridge from doing p2p
transactions (thus violating VM boundaries if those devices are
assigned to different domains).

It looks like Xen simply trusts the hardware.

> Can you provide some more details about what you expect of ACS
> handling?

I was actually surprised not to see IOMMU groups in the style of what
VFIO https://www.kernel.org/doc/Documentation/vfio.txt

Thanks,
Roman.

Re: [Xen-devel] PCIe IOMMU ACS support

[Paul Durrant]

> -----Original Message-----
> From: Roman Shaposhnik <roman@zededa.com>
> Sent: 26 March 2020 22:03
> To: Roger Pau Monné <roger.pau@citrix.com>
> Cc: xen-devel@lists.xenproject.org; Jan Beulich <jbeulich@suse.com>; Paul Durrant <paul@xen.org>;
> Kevin Tian <kevin.tian@intel.com>; Andrew Cooper <andrew.cooper3@citrix.com>
> Subject: Re: [Xen-devel] PCIe IOMMU ACS support
> 
> On Wed, Mar 25, 2020 at 4:05 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > Adding the PCI and IOMMU maintainers.
> >
> > On Mon, Mar 23, 2020 at 01:55:01PM -0700, Roman Shaposhnik wrote:
> > > Hi!
> > >
> > > I was going through how Xen support PCIe IOMMU ACS and
> > > all I could find is this:
> > >     https://github.com/xen-project/xen/blob/master/xen/drivers/passthrough/pci.c#L608
> > > which looks to me as an attempt of enabling ACS opportunistically,
> > > but still proceeding forward even if it fails.
> >
> > That's correct AFAICT. Xen will try to enable some features, but will
> > proceed normally if ACS is not available, or if some of the features
> > are not implemented.
> >
> > Are you looking to ensure that all devices on the system have a
> > certain feature enabled?
> 
> My primary objective was to get some visibility into how Xen would
> prevent two PCIe devices behind a common bridge from doing p2p
> transactions (thus violating VM boundaries if those devices are
> assigned to different domains).
> 
> It looks like Xen simply trusts the hardware.
> 
> > Can you provide some more details about what you expect of ACS
> > handling?
> 
> I was actually surprised not to see IOMMU groups in the style of what
> VFIO https://www.kernel.org/doc/Documentation/vfio.txt
> 

I did write a doc some time ago to present the issues facing Xen w.r.t. IOMMU and device pass-through. Hopefully you can see it at https://docs.google.com/document/d/12-z6JD41J_oNrCg_c0yAxGWg5ADBQ8_bSiP_NH6Hqwo/edit?usp=sharing

  Paul

Re: [Xen-devel] PCIe IOMMU ACS support

[Roman Shaposhnik]

On Fri, Mar 27, 2020 at 2:12 AM Paul Durrant <xadimgnik@gmail.com> wrote:
>
> > -----Original Message-----
> > From: Roman Shaposhnik <roman@zededa.com>
> > Sent: 26 March 2020 22:03
> > To: Roger Pau Monné <roger.pau@citrix.com>
> > Cc: xen-devel@lists.xenproject.org; Jan Beulich <jbeulich@suse.com>; Paul Durrant <paul@xen.org>;
> > Kevin Tian <kevin.tian@intel.com>; Andrew Cooper <andrew.cooper3@citrix.com>
> > Subject: Re: [Xen-devel] PCIe IOMMU ACS support
> >
> > On Wed, Mar 25, 2020 at 4:05 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> > >
> > > Adding the PCI and IOMMU maintainers.
> > >
> > > On Mon, Mar 23, 2020 at 01:55:01PM -0700, Roman Shaposhnik wrote:
> > > > Hi!
> > > >
> > > > I was going through how Xen support PCIe IOMMU ACS and
> > > > all I could find is this:
> > > >     https://github.com/xen-project/xen/blob/master/xen/drivers/passthrough/pci.c#L608
> > > > which looks to me as an attempt of enabling ACS opportunistically,
> > > > but still proceeding forward even if it fails.
> > >
> > > That's correct AFAICT. Xen will try to enable some features, but will
> > > proceed normally if ACS is not available, or if some of the features
> > > are not implemented.
> > >
> > > Are you looking to ensure that all devices on the system have a
> > > certain feature enabled?
> >
> > My primary objective was to get some visibility into how Xen would
> > prevent two PCIe devices behind a common bridge from doing p2p
> > transactions (thus violating VM boundaries if those devices are
> > assigned to different domains).
> >
> > It looks like Xen simply trusts the hardware.
> >
> > > Can you provide some more details about what you expect of ACS
> > > handling?
> >
> > I was actually surprised not to see IOMMU groups in the style of what
> > VFIO https://www.kernel.org/doc/Documentation/vfio.txt
> >
>
> I did write a doc some time ago to present the issues facing Xen w.r.t. IOMMU and device pass-through. Hopefully you can see it at https://docs.google.com/document/d/12-z6JD41J_oNrCg_c0yAxGWg5ADBQ8_bSiP_NH6Hqwo/edit?usp=sharing

Paul, this is *exactly* what I was asking about -- thanks for the link.

I guess the only question I have left is whether there was any follow up
regarding what you sketched out in the "IOMMU drivers" section?

Thanks,
Roman.