Superpages for VT-D

Superpages for VT-D

[Santosh Jodh]

[-- Attachment #1.1: Type: text/plain, Size: 963 bytes --]

I do not have a clear answer about use of super pages in VT-D for the IOMMU.

Here is from the release notes for 4.1:

http://blog.xen.org/index.php/2011/03/25/xen-4-1-releases/

Further, support for EPT/VTd 1GB/2MB super pages has been added to Xen, reducing the TLB overhead. EPT/VTd page table sharing simplifies the support for Intel's IOMMU<http://en.wikipedia.org/wiki/IOMMU> by allowing the CPU's Enhanced Page Table to be directly utilized by the VTd IOMMU.


Based on this check in, I also see the output from Xen on my system showing HAP page sizes as 4kB, 2MB, and 1GB - but it does not say which one is in use.
http://lists.xen.org/archives/html/xen-changelog/2012-03/msg00140.html



1.       Are superpages (2MB, 1GB) enabled by default in Xen? If enabled, what size?

2.       On Intel, if superpages are enabled, is that used for the IOMMU?

3.       If it is not enabled by default, how does one enable it?

Thanks,
Santosh

[-- Attachment #1.2: Type: text/html, Size: 4782 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Superpages for VT-D

[Tim Deegan]

At 09:18 -0700 on 13 Jul (1342171126), Santosh Jodh wrote:
> 1.       Are superpages (2MB, 1GB) enabled by default in Xen? If enabled, what size?

Yes, they are enabled; for guest memory, Xen uses whatever the tools ask
for, which (IIRC) is the largest available for HVM (falling back to
smaller pages if no large ones can be found) and 2MB for PV if the guest
supports PV superpages.

> 2.       On Intel, if superpages are enabled, is that used for the IOMMU?

Yes.  In fact the IOMMU code checks for IOMMU superpage support
separately; if the IOMMU and the MMU have the same superpage sizes, they
share a table; otherwise the p2m is duplicated in the IOMMU's format. 

Cheers,

Tim.

Re: Superpages for VT-D

[Santosh Jodh]

Thanks Tim.

I understand the value of using superpages for IOMMU. What are the other advantages of sharing the table between the MMU and IOMMU (other than saving table memory)?

Thanks,
Santosh

-----Original Message-----
From: Tim Deegan [mailto:tim@xen.org] 
Sent: Thursday, July 19, 2012 3:23 AM
To: Santosh Jodh
Cc: xen-devel@lists.xen.org; xiantao.zhang@intel.com; jun.nakajima@intel.com
Subject: Re: [Xen-devel] Superpages for VT-D

At 09:18 -0700 on 13 Jul (1342171126), Santosh Jodh wrote:
> 1.       Are superpages (2MB, 1GB) enabled by default in Xen? If enabled, what size?

Yes, they are enabled; for guest memory, Xen uses whatever the tools ask for, which (IIRC) is the largest available for HVM (falling back to smaller pages if no large ones can be found) and 2MB for PV if the guest supports PV superpages.

> 2.       On Intel, if superpages are enabled, is that used for the IOMMU?

Yes.  In fact the IOMMU code checks for IOMMU superpage support separately; if the IOMMU and the MMU have the same superpage sizes, they share a table; otherwise the p2m is duplicated in the IOMMU's format. 

Cheers,

Tim.

Re: Superpages for VT-D

[Tim Deegan]

At 07:01 -0700 on 19 Jul (1342681270), Santosh Jodh wrote:
> I understand the value of using superpages for IOMMU. What are the
> other advantages of sharing the table between the MMU and IOMMU (other
> than saving table memory)?

It's mostly the memory, but also it's faster to update one table than
two, and we don't have to worry about them getting out of sync with each
other.  I don't think there are any deeper reasons.

Cheers,

Tim.

Re: Superpages for VT-D

[Nakajima, Jun]

[-- Attachment #1.1: Type: text/plain, Size: 751 bytes --]

On Thu, Jul 19, 2012 at 7:53 AM, Tim Deegan <tim@xen.org> wrote:

> At 07:01 -0700 on 19 Jul (1342681270), Santosh Jodh wrote:
> > I understand the value of using superpages for IOMMU. What are the
> > other advantages of sharing the table between the MMU and IOMMU (other
> > than saving table memory)?
>
> It's mostly the memory, but also it's faster to update one table than
> two, and we don't have to worry about them getting out of sync with each
> other.  I don't think there are any deeper reasons.
>
> Cheers,
>
> Tim.
>
>
- The code can be simpler because you don't need to maintain the same info
"guest physical -> host physical" in two different places.
- Maybe cache utilization can be better

-- 
Jun
Intel Open Source Technology Center

[-- Attachment #1.2: Type: text/html, Size: 1182 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Superpages for VT-D

[Santosh Jodh]

[-- Attachment #1.1: Type: text/plain, Size: 1093 bytes --]

Is there an easy way to see the mapping sizes being used for a VM in the IOMMU?

Thanks,
Santosh

From: Nakajima, Jun [mailto:jun.nakajima@intel.com]
Sent: Friday, July 20, 2012 2:38 PM
To: Tim (Xen.org)
Cc: Santosh Jodh; xen-devel@lists.xen.org; xiantao.zhang@intel.com
Subject: Re: [Xen-devel] Superpages for VT-D

On Thu, Jul 19, 2012 at 7:53 AM, Tim Deegan <tim@xen.org<mailto:tim@xen.org>> wrote:
At 07:01 -0700 on 19 Jul (1342681270), Santosh Jodh wrote:
> I understand the value of using superpages for IOMMU. What are the
> other advantages of sharing the table between the MMU and IOMMU (other
> than saving table memory)?
It's mostly the memory, but also it's faster to update one table than
two, and we don't have to worry about them getting out of sync with each
other.  I don't think there are any deeper reasons.

Cheers,

Tim.

- The code can be simpler because you don't need to maintain the same info "guest physical -> host physical" in two different places.
- Maybe cache utilization can be better

--
Jun
Intel Open Source Technology Center


[-- Attachment #1.2: Type: text/html, Size: 4368 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Superpages for VT-D

[Tim Deegan]

At 10:56 -0700 on 23 Jul (1343040996), Santosh Jodh wrote:
> Is there an easy way to see the mapping sizes being used for a VM in the IOMMU?

Doesn't look like it.  If the IOMMU is sharing the EPT table you can use
the 'D' debug-key to dump the EPT tables, but I don't think there's an
equivalent for the IOMMU tables.  I'm sure it wouldn't be too hard to
do, though.

Tim.

Re: Superpages for VT-D

[Santosh Jodh]

I am going to try to add this support. 

It looks like a new iommu_ops handler would be needed that would do the actual work of dumping the entries - one for AMD and one for Intel. Am I reading this correctly? 

Or is it better to get the root_table + paging_mode (for AMD) and pgd_maddr + agaw (for Intel) and then do a generic dump?

Thanks,
Santosh

-----Original Message-----
From: Tim Deegan [mailto:tim@xen.org] 
Sent: Tuesday, July 24, 2012 12:42 PM
To: Santosh Jodh
Cc: Nakajima, Jun; xiantao.zhang@intel.com; xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Superpages for VT-D

At 10:56 -0700 on 23 Jul (1343040996), Santosh Jodh wrote:
> Is there an easy way to see the mapping sizes being used for a VM in the IOMMU?

Doesn't look like it.  If the IOMMU is sharing the EPT table you can use the 'D' debug-key to dump the EPT tables, but I don't think there's an equivalent for the IOMMU tables.  I'm sure it wouldn't be too hard to do, though.

Tim.

Re: Superpages for VT-D

[Tim Deegan]

At 13:44 -0700 on 31 Jul (1343742260), Santosh Jodh wrote:
> I am going to try to add this support. 
> 
> It looks like a new iommu_ops handler would be needed that would do
> the actual work of dumping the entries - one for AMD and one for
> Intel. Am I reading this correctly?

I think that's correct.

> Or is it better to get the root_table + paging_mode (for AMD) and
> pgd_maddr + agaw (for Intel) and then do a generic dump?

No; the iommu tabels are sufficiebntly arch-specific that it's best to
add arch-specific dump routines.

Cheers,

Tim.

Re: Superpages for VT-D

[Santosh Jodh]

Thanks for confirming. BTW, would the IOMMU ever have entries above domains max mapped pfn?

-----Original Message-----
From: Tim Deegan [mailto:tim@xen.org] 
Sent: Thursday, August 02, 2012 4:11 AM
To: Santosh Jodh
Cc: xiantao.zhang@intel.com; Nakajima, Jun; xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Superpages for VT-D

At 13:44 -0700 on 31 Jul (1343742260), Santosh Jodh wrote:
> I am going to try to add this support. 
> 
> It looks like a new iommu_ops handler would be needed that would do 
> the actual work of dumping the entries - one for AMD and one for 
> Intel. Am I reading this correctly?

I think that's correct.

> Or is it better to get the root_table + paging_mode (for AMD) and 
> pgd_maddr + agaw (for Intel) and then do a generic dump?

No; the iommu tabels are sufficiebntly arch-specific that it's best to add arch-specific dump routines.

Cheers,

Tim.

Re: Superpages for VT-D

[Tim Deegan]

At 08:56 -0700 on 02 Aug (1343897810), Santosh Jodh wrote:
> Thanks for confirming. BTW, would the IOMMU ever have entries above
> domains max mapped pfn?

I don't believe so, but potentially some grant-table-style operations
might want to map pages in for DMA that don't need to be mapped in for
CPU access. 

But if you're writing a dump routine from scratch it should be easy
enough (and more efficient) to dump all the entries with a depth-first
pass over the trie, rather than individually querying all frames up to
max_mapped_pfn.

Tim.

> -----Original Message-----
> From: Tim Deegan [mailto:tim@xen.org] 
> Sent: Thursday, August 02, 2012 4:11 AM
> To: Santosh Jodh
> Cc: xiantao.zhang@intel.com; Nakajima, Jun; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Superpages for VT-D
> 
> At 13:44 -0700 on 31 Jul (1343742260), Santosh Jodh wrote:
> > I am going to try to add this support. 
> > 
> > It looks like a new iommu_ops handler would be needed that would do 
> > the actual work of dumping the entries - one for AMD and one for 
> > Intel. Am I reading this correctly?
> 
> I think that's correct.
> 
> > Or is it better to get the root_table + paging_mode (for AMD) and 
> > pgd_maddr + agaw (for Intel) and then do a generic dump?
> 
> No; the iommu tabels are sufficiebntly arch-specific that it's best to add arch-specific dump routines.
> 
> Cheers,
> 
> Tim.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel