[Xen-devel] per-domain passthrough/iommu options

[Xen-devel] per-domain passthrough/iommu options

[Paul Durrant]

Hi,

  I sent a previous email [1] about enabling use of the IOMMU on a per-domain basis and am now a reasonable way into my patch series, which also allows for IOMMU option-setting (specifically shared EPT use) on a per-domain basis too. Before I post v1 I'd like to get some opinion on the what the xl.cfg options should look like.
  The simplest way for me to do things would be to have two new boolean options; something like:

'passthrough' - enable/disable pass-through support (i.e. use of the IOMMU)... can be implicitly enabled if there are pci or dt devices specified in the xl.cfg.
'no-sharept' - named to match the xen-cmdline option for turning off shared EPT. (EPT sharing currently defaults on globally).

  I think the former is probably ok, but thinking forward to a time where we might have vIOMMU (PV or emulated) the latter is probably not the right thing to use. So, another way might be to have an IOMMU page-table option... something like:

'iommu-pt = shared|sync'

  where 'shared' means use EPT mappings, and 'sync' means keep the P2M in sync. This could then be extended with 'viommu' later, meaning that there would be some form of vIOMMU exposed to the guest, be it emulated, PV or both. One drawback with this mechanism is that 'shared' is not always possible (e.g. on AMD h/w) so what should be done in that case? Should selecting that option be considered an error, or should there be a fall-back to 'sync'? The fall-back would be easier to deal with as then the option could just default to 'sync' if it was not specified.

  Thoughts? Other ideas?

  Cheers,

    Paul

[1] https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg01090.html

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] per-domain passthrough/iommu options

[Jan Beulich]

On 26.07.2019 14:29, Paul Durrant wrote:
>    I sent a previous email [1] about enabling use of the IOMMU on a per-domain basis and am now a reasonable way into my patch series, which also allows for IOMMU option-setting (specifically shared EPT use) on a per-domain basis too. Before I post v1 I'd like to get some opinion on the what the xl.cfg options should look like.
>    The simplest way for me to do things would be to have two new boolean options; something like:
> 
> 'passthrough' - enable/disable pass-through support (i.e. use of the IOMMU)... can be implicitly enabled if there are pci or dt devices specified in the xl.cfg.
> 'no-sharept' - named to match the xen-cmdline option for turning off shared EPT. (EPT sharing currently defaults on globally).
> 
>    I think the former is probably ok, but thinking forward to a time where we might have vIOMMU (PV or emulated) the latter is probably not the right thing to use. So, another way might be to have an IOMMU page-table option... something like:
> 
> 'iommu-pt = shared|sync'
> 
>    where 'shared' means use EPT mappings, and 'sync' means keep the P2M in sync. This could then be extended with 'viommu' later, meaning that there would be some form of vIOMMU exposed to the guest, be it emulated, PV or both. One drawback with this mechanism is that 'shared' is not always possible (e.g. on AMD h/w) so what should be done in that case? Should selecting that option be considered an error, or should there be a fall-back to 'sync'? The fall-back would be easier to deal with as then the option could just default to 'sync' if it was not specified.

The fall-back sounds reasonable to me (as long as that's properly
described in documentation). What I'm less happy with is the idea
of having two options instead of just one. But of course this may
be a result of how libxl wants to organize options. If there's no
restriction at that end, then how about

passthrough = off|sync|share-pt|viommu

? It would default to off when there are no devices listed in the
config, and to share-pt (with the fall-back to sync) when there is
at least one.

As to "sync" - how did you come to use this as the "opposite" of
"share-pt"? There's nothing asynchronous with shared page tables,
is there? Maybe "private-pt" or "separate-pt"? The option wouldn't
be used typically anyway, especially if alongside "off" there was
also an "on" variant, meaning the same as "share-pt".

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] per-domain passthrough/iommu options

[Paul Durrant]

> -----Original Message-----
> From: Jan Beulich <JBeulich@suse.com>
> Sent: 26 July 2019 13:57
> To: Paul Durrant <Paul.Durrant@citrix.com>
> Cc: xen-devel (xen-devel@lists.xenproject.org) <xen-devel@lists.xenproject.org>
> Subject: Re: [Xen-devel] per-domain passthrough/iommu options
> 
> On 26.07.2019 14:29, Paul Durrant wrote:
> >    I sent a previous email [1] about enabling use of the IOMMU on a per-domain basis and am now a
> reasonable way into my patch series, which also allows for IOMMU option-setting (specifically shared
> EPT use) on a per-domain basis too. Before I post v1 I'd like to get some opinion on the what the
> xl.cfg options should look like.
> >    The simplest way for me to do things would be to have two new boolean options; something like:
> >
> > 'passthrough' - enable/disable pass-through support (i.e. use of the IOMMU)... can be implicitly
> enabled if there are pci or dt devices specified in the xl.cfg.
> > 'no-sharept' - named to match the xen-cmdline option for turning off shared EPT. (EPT sharing
> currently defaults on globally).
> >
> >    I think the former is probably ok, but thinking forward to a time where we might have vIOMMU (PV
> or emulated) the latter is probably not the right thing to use. So, another way might be to have an
> IOMMU page-table option... something like:
> >
> > 'iommu-pt = shared|sync'
> >
> >    where 'shared' means use EPT mappings, and 'sync' means keep the P2M in sync. This could then be
> extended with 'viommu' later, meaning that there would be some form of vIOMMU exposed to the guest, be
> it emulated, PV or both. One drawback with this mechanism is that 'shared' is not always possible
> (e.g. on AMD h/w) so what should be done in that case? Should selecting that option be considered an
> error, or should there be a fall-back to 'sync'? The fall-back would be easier to deal with as then
> the option could just default to 'sync' if it was not specified.
> 
> The fall-back sounds reasonable to me (as long as that's properly
> described in documentation). What I'm less happy with is the idea
> of having two options instead of just one. But of course this may
> be a result of how libxl wants to organize options. If there's no
> restriction at that end, then how about
> 
> passthrough = off|sync|share-pt|viommu
> 
> ? It would default to off when there are no devices listed in the
> config, and to share-pt (with the fall-back to sync) when there is
> at least one.

Yes, that sounds like it would work.

> 
> As to "sync" - how did you come to use this as the "opposite" of
> "share-pt"?

Oh, that's just adopting the general naming used in Xen. For a domU it is either going to have 'need_sync' set in its domain_iommu structure, or EPT sharing will be active.
 
> There's nothing asynchronous with shared page tables,
> is there? Maybe "private-pt" or "separate-pt"? The option wouldn't
> be used typically anyway, especially if alongside "off" there was
> also an "on" variant, meaning the same as "share-pt".

Not sure how 'on' would co-exist with 'viommu'... the crucial difference is whether the p2m is shared or not and the currently the only option in the non-shared case, because we lack a viommu, is to keep the IOMMU mappings in sync with the P2M whenever the latter is updated. So, how about:

passthrough = off|sync-pt|share-pt|viommu

? I don't think 'private-pt' or 'separate-pt' really capture the fact that the page tables match the P2M. They could just as easily be taken to mean that they are populated using some other policy.

  Paul

> 
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] per-domain passthrough/iommu options

[Jan Beulich]

On 26.07.2019 15:39, Paul Durrant wrote:
> Not sure how 'on' would co-exist with 'viommu'... the crucial difference is whether the p2m is shared or not and the currently the only option in the non-shared case, because we lack a viommu, is to keep the IOMMU mappings in sync with the P2M whenever the latter is updated. So, how about:
> 
> passthrough = off|sync-pt|share-pt|viommu
> 
> ? I don't think 'private-pt' or 'separate-pt' really capture the fact that the page tables match the P2M. They could just as easily be taken to mean that they are populated using some other policy.

But haven't we recently seen that this fully lock-step population
of page tables isn't always correct (or at least desirable)? I
vaguely recall other comments to that effect too, from long ago.
I'd specifically want to avoid encoding into the interface here
that the two are exact mirrors of one another, now and forever.

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] per-domain passthrough/iommu options

[Paul Durrant]

> -----Original Message-----
> From: Jan Beulich <JBeulich@suse.com>
> Sent: 26 July 2019 15:02
> To: Paul Durrant <Paul.Durrant@citrix.com>
> Cc: xen-devel (xen-devel@lists.xenproject.org) <xen-devel@lists.xenproject.org>
> Subject: Re: [Xen-devel] per-domain passthrough/iommu options
> 
> On 26.07.2019 15:39, Paul Durrant wrote:
> > Not sure how 'on' would co-exist with 'viommu'... the crucial difference is whether the p2m is
> shared or not and the currently the only option in the non-shared case, because we lack a viommu, is
> to keep the IOMMU mappings in sync with the P2M whenever the latter is updated. So, how about:
> >
> > passthrough = off|sync-pt|share-pt|viommu
> >
> > ? I don't think 'private-pt' or 'separate-pt' really capture the fact that the page tables match the
> P2M. They could just as easily be taken to mean that they are populated using some other policy.
> 
> But haven't we recently seen that this fully lock-step population
> of page tables isn't always correct (or at least desirable)? I
> vaguely recall other comments to that effect too, from long ago.
> I'd specifically want to avoid encoding into the interface here
> that the two are exact mirrors of one another, now and forever.

How do you think we should express it. I agree that it's a bit awkward because of the difference between HVM and PV domains, but all we can do there really is document it I think, so perhaps the manpage could have something like:

'off'

IOMMU mappings are disabled for the domain and so hardware may not be passed through.

'sync-pt'

For a PV domain, all writable pages assigned to the domain are identity mapped by MFN in the IOMMU page tables. Thus a device driver running in the domain may program passthrough hardware for DMA using MFN values (i.e. host/machine frame numbers) looked up in its P2M.
For an HVM domain, all non-foreign RAM pages present in the P2M will be identity mapped by GFN in the IOMMU page tables. Thus a device driver running in the domain may program passthrough hardware using GFN values (i.e. guest physical frame numbers) without any further translation.

'share-pt'

This is unavailable for a PV domain. For an HVM domain, this option means that the IOMMU will be programmed to directly reference the P2M as its page tables. This availability of this option is hardware specific and thus, if it is specified for a domain running on hardware that does not allow it, 'sync-pt' will be used instead.

?

  Paul

> 
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] per-domain passthrough/iommu options

[Jan Beulich]

On 26.07.2019 16:26, Paul Durrant wrote:
>> From: Jan Beulich <JBeulich@suse.com>
>> Sent: 26 July 2019 15:02
>>
>> On 26.07.2019 15:39, Paul Durrant wrote:
>>> ? I don't think 'private-pt' or 'separate-pt' really capture the fact that the page tables match the
>> P2M. They could just as easily be taken to mean that they are populated using some other policy.
>>
>> But haven't we recently seen that this fully lock-step population
>> of page tables isn't always correct (or at least desirable)? I
>> vaguely recall other comments to that effect too, from long ago.
>> I'd specifically want to avoid encoding into the interface here
>> that the two are exact mirrors of one another, now and forever.
> 
> How do you think we should express it. I agree that it's a bit awkward because of the difference between HVM and PV domains, but all we can do there really is document it I think, so perhaps the manpage could have something like:

Sounds reasonable - it at least avoids making the behavior too
much spelled out with regard to the similarity of mappings between
IOMMU page tables and P2M. There's one issue though:

> 'off'
> 
> IOMMU mappings are disabled for the domain and so hardware may not be passed through.
> 
> 'sync-pt'
> 
> For a PV domain, all writable pages assigned to the domain are identity mapped by MFN in the IOMMU page tables. Thus a device driver running in the domain may program passthrough hardware for DMA using MFN values (i.e. host/machine frame numbers) looked up in its P2M.
> For an HVM domain, all non-foreign RAM pages present in the P2M will be identity mapped by GFN

Why "identity mapped" here? It's a GFN -> MFN mappingm, isn't it?

Jan

> in the IOMMU page tables. Thus a device driver running in the domain may program passthrough hardware using GFN values (i.e. guest physical frame numbers) without any further translation.
> 
> 'share-pt'
> 
> This is unavailable for a PV domain. For an HVM domain, this option means that the IOMMU will be programmed to directly reference the P2M as its page tables. This availability of this option is hardware specific and thus, if it is specified for a domain running on hardware that does not allow it, 'sync-pt' will be used instead.
> 
> ?
> 
>    Paul
> 
>>
>> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] per-domain passthrough/iommu options

[Paul Durrant]

> -----Original Message-----
> From: Jan Beulich <JBeulich@suse.com>
> Sent: 26 July 2019 15:34
> To: Paul Durrant <Paul.Durrant@citrix.com>
> Cc: xen-devel (xen-devel@lists.xenproject.org) <xen-devel@lists.xenproject.org>
> Subject: Re: [Xen-devel] per-domain passthrough/iommu options
> 
> On 26.07.2019 16:26, Paul Durrant wrote:
> >> From: Jan Beulich <JBeulich@suse.com>
> >> Sent: 26 July 2019 15:02
> >>
> >> On 26.07.2019 15:39, Paul Durrant wrote:
> >>> ? I don't think 'private-pt' or 'separate-pt' really capture the fact that the page tables match
> the
> >> P2M. They could just as easily be taken to mean that they are populated using some other policy.
> >>
> >> But haven't we recently seen that this fully lock-step population
> >> of page tables isn't always correct (or at least desirable)? I
> >> vaguely recall other comments to that effect too, from long ago.
> >> I'd specifically want to avoid encoding into the interface here
> >> that the two are exact mirrors of one another, now and forever.
> >
> > How do you think we should express it. I agree that it's a bit awkward because of the difference
> between HVM and PV domains, but all we can do there really is document it I think, so perhaps the
> manpage could have something like:
> 
> Sounds reasonable - it at least avoids making the behavior too
> much spelled out with regard to the similarity of mappings between
> IOMMU page tables and P2M. There's one issue though:
> 
> > 'off'
> >
> > IOMMU mappings are disabled for the domain and so hardware may not be passed through.
> >
> > 'sync-pt'
> >
> > For a PV domain, all writable pages assigned to the domain are identity mapped by MFN in the IOMMU
> page tables. Thus a device driver running in the domain may program passthrough hardware for DMA using
> MFN values (i.e. host/machine frame numbers) looked up in its P2M.
> > For an HVM domain, all non-foreign RAM pages present in the P2M will be identity mapped by GFN
> 
> Why "identity mapped" here? It's a GFN -> MFN mappingm, isn't it?

Yes... it's hard to express. What I want to say, of course, is that device drivers can use GFNs. Can you think of any other form of words that might be better?

  Paul

> 
> Jan
> 
> > in the IOMMU page tables. Thus a device driver running in the domain may program passthrough
> hardware using GFN values (i.e. guest physical frame numbers) without any further translation.
> >
> > 'share-pt'
> >
> > This is unavailable for a PV domain. For an HVM domain, this option means that the IOMMU will be
> programmed to directly reference the P2M as its page tables. This availability of this option is
> hardware specific and thus, if it is specified for a domain running on hardware that does not allow
> it, 'sync-pt' will be used instead.
> >
> > ?
> >
> >    Paul
> >
> >>
> >> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] per-domain passthrough/iommu options

[Jan Beulich]

On 26.07.2019 16:54, Paul Durrant wrote:
>> -----Original Message-----
>> From: Jan Beulich <JBeulich@suse.com>
>> Sent: 26 July 2019 15:34
>> To: Paul Durrant <Paul.Durrant@citrix.com>
>> Cc: xen-devel (xen-devel@lists.xenproject.org) <xen-devel@lists.xenproject.org>
>> Subject: Re: [Xen-devel] per-domain passthrough/iommu options
>>
>> On 26.07.2019 16:26, Paul Durrant wrote:
>>>> From: Jan Beulich <JBeulich@suse.com>
>>>> Sent: 26 July 2019 15:02
>>>>
>>>> On 26.07.2019 15:39, Paul Durrant wrote:
>>>>> ? I don't think 'private-pt' or 'separate-pt' really capture the fact that the page tables match
>> the
>>>> P2M. They could just as easily be taken to mean that they are populated using some other policy.
>>>>
>>>> But haven't we recently seen that this fully lock-step population
>>>> of page tables isn't always correct (or at least desirable)? I
>>>> vaguely recall other comments to that effect too, from long ago.
>>>> I'd specifically want to avoid encoding into the interface here
>>>> that the two are exact mirrors of one another, now and forever.
>>>
>>> How do you think we should express it. I agree that it's a bit awkward because of the difference
>> between HVM and PV domains, but all we can do there really is document it I think, so perhaps the
>> manpage could have something like:
>>
>> Sounds reasonable - it at least avoids making the behavior too
>> much spelled out with regard to the similarity of mappings between
>> IOMMU page tables and P2M. There's one issue though:
>>
>>> 'off'
>>>
>>> IOMMU mappings are disabled for the domain and so hardware may not be passed through.
>>>
>>> 'sync-pt'
>>>
>>> For a PV domain, all writable pages assigned to the domain are identity mapped by MFN in the IOMMU
>> page tables. Thus a device driver running in the domain may program passthrough hardware for DMA using
>> MFN values (i.e. host/machine frame numbers) looked up in its P2M.
>>> For an HVM domain, all non-foreign RAM pages present in the P2M will be identity mapped by GFN
>>
>> Why "identity mapped" here? It's a GFN -> MFN mappingm, isn't it?
> 
> Yes... it's hard to express. What I want to say, of course, is that device drivers can use GFNs. Can you think of any other form of words that might be better?

Just omit "identity" in the HVM related sentence?

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] per-domain passthrough/iommu options

[Paul Durrant]

> -----Original Message-----
> From: Jan Beulich <JBeulich@suse.com>
> Sent: 29 July 2019 12:11
> To: Paul Durrant <Paul.Durrant@citrix.com>
> Cc: xen-devel (xen-devel@lists.xenproject.org) <xen-devel@lists.xenproject.org>
> Subject: Re: [Xen-devel] per-domain passthrough/iommu options
> 
> On 26.07.2019 16:54, Paul Durrant wrote:
> >> -----Original Message-----
> >> From: Jan Beulich <JBeulich@suse.com>
> >> Sent: 26 July 2019 15:34
> >> To: Paul Durrant <Paul.Durrant@citrix.com>
> >> Cc: xen-devel (xen-devel@lists.xenproject.org) <xen-devel@lists.xenproject.org>
> >> Subject: Re: [Xen-devel] per-domain passthrough/iommu options
> >>
> >> On 26.07.2019 16:26, Paul Durrant wrote:
> >>>> From: Jan Beulich <JBeulich@suse.com>
> >>>> Sent: 26 July 2019 15:02
> >>>>
> >>>> On 26.07.2019 15:39, Paul Durrant wrote:
> >>>>> ? I don't think 'private-pt' or 'separate-pt' really capture the fact that the page tables match
> >> the
> >>>> P2M. They could just as easily be taken to mean that they are populated using some other policy.
> >>>>
> >>>> But haven't we recently seen that this fully lock-step population
> >>>> of page tables isn't always correct (or at least desirable)? I
> >>>> vaguely recall other comments to that effect too, from long ago.
> >>>> I'd specifically want to avoid encoding into the interface here
> >>>> that the two are exact mirrors of one another, now and forever.
> >>>
> >>> How do you think we should express it. I agree that it's a bit awkward because of the difference
> >> between HVM and PV domains, but all we can do there really is document it I think, so perhaps the
> >> manpage could have something like:
> >>
> >> Sounds reasonable - it at least avoids making the behavior too
> >> much spelled out with regard to the similarity of mappings between
> >> IOMMU page tables and P2M. There's one issue though:
> >>
> >>> 'off'
> >>>
> >>> IOMMU mappings are disabled for the domain and so hardware may not be passed through.
> >>>
> >>> 'sync-pt'
> >>>
> >>> For a PV domain, all writable pages assigned to the domain are identity mapped by MFN in the IOMMU
> >> page tables. Thus a device driver running in the domain may program passthrough hardware for DMA
> using
> >> MFN values (i.e. host/machine frame numbers) looked up in its P2M.
> >>> For an HVM domain, all non-foreign RAM pages present in the P2M will be identity mapped by GFN
> >>
> >> Why "identity mapped" here? It's a GFN -> MFN mappingm, isn't it?
> >
> > Yes... it's hard to express. What I want to say, of course, is that device drivers can use GFNs. Can
> you think of any other form of words that might be better?
> 
> Just omit "identity" in the HVM related sentence?

Ok, I guess that sounds ok. I'll do that.

  Paul

> 
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel