HYBRID: gnttab_map() to map shared frames..

HYBRID: gnttab_map() to map shared frames..

[Mukesh Rathor]

Hi,

I am trying to figure how to make gnttab_map() work for Hybrid with EPT.
The only way I am coming up is the HVM way, where I would have to get
mmio_addr via pci_resource_start() which means registering xen-platform-pci
driver. I know there will be implications later with migration, but thats 
later.

I can't think of any other way since the grant frames are coming from
xen heap and not in p2m table, other than some sort of
xen-heap-p2m-table or something ugly.

Am I on right track, or any other ideas?

thanks,
Mukesh

Re: HYBRID: gnttab_map() to map shared frames..

[Keir Fraser]

If you are running without autotranslate paging mode then the p2m will be
1:1 and there is no problem. If you are running autotranslate then yes you
probably need the HVM style of mapping.

 -- Keir

On 14/10/2011 02:18, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:

> Hi,
> 
> I am trying to figure how to make gnttab_map() work for Hybrid with EPT.
> The only way I am coming up is the HVM way, where I would have to get
> mmio_addr via pci_resource_start() which means registering xen-platform-pci
> driver. I know there will be implications later with migration, but thats
> later.
> 
> I can't think of any other way since the grant frames are coming from
> xen heap and not in p2m table, other than some sort of
> xen-heap-p2m-table or something ugly.
> 
> Am I on right track, or any other ideas?
> 
> thanks,
> Mukesh
> 

Re: HYBRID: gnttab_map() to map shared frames..

[Ian Campbell]

On Fri, 2011-10-14 at 08:50 +0100, Keir Fraser wrote:
> If you are running without autotranslate paging mode then the p2m will be
> 1:1 and there is no problem. If you are running autotranslate then yes you
> probably need the HVM style of mapping.

The issue is that with an HVM guest there is no guest visibility of MFN
space and PFN space is not managed by the hypervisor, so you need an
add_to_physmap style call in order to point the hypervisor at some PFN
space which it can use as the grant table. You then get this same PFN
space back from GNTTABOP_setup_table and from then on PV and HVM look
much the same. (I'm sure you know this, I'm just thinking out loud).

For a hybrid guest it's not quite true that PFN space is not managed by
the hypervisor -- we could, at least in theory, use the hypervisor hole
and "allocate" ourselves some pfn space for the grant table in there.
Couldn't we?

Ian.

> 
>  -- Keir
> 
> On 14/10/2011 02:18, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:
> 
> > Hi,
> > 
> > I am trying to figure how to make gnttab_map() work for Hybrid with EPT.
> > The only way I am coming up is the HVM way, where I would have to get
> > mmio_addr via pci_resource_start() which means registering xen-platform-pci
> > driver. I know there will be implications later with migration, but thats
> > later.
> > 
> > I can't think of any other way since the grant frames are coming from
> > xen heap and not in p2m table, other than some sort of
> > xen-heap-p2m-table or something ugly.
> > 
> > Am I on right track, or any other ideas?
> > 
> > thanks,
> > Mukesh
> > 
> 
> 

Re: HYBRID: gnttab_map() to map shared frames..

[Mukesh Rathor]

Hi Jeremy,

You have any thoughts on this? This is basically for hybrid running 
autotranslate paging mode.  I am trying to figure how to get unmapped
pfns to map shared grant table page/s from xen. HVM does this via
pci_resource_start() and gets io remapped address. However, in case of
pure PV, I am not sure how feasible this approach would be.

Is there way I can reserve few pfn's above max_pfn and use those
for my XENMAPSPACE_grant_table call?

thanks,
Mukesh

On Fri, 14 Oct 2011 08:50:33 +0100
Keir Fraser <keir.xen@gmail.com> wrote:

> If you are running without autotranslate paging mode then the p2m
> will be 1:1 and there is no problem. If you are running autotranslate
> then yes you probably need the HVM style of mapping.
> 
>  -- Keir
> 
> On 14/10/2011 02:18, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:
> 
> > Hi,
> > 
> > I am trying to figure how to make gnttab_map() work for Hybrid with
> > EPT. The only way I am coming up is the HVM way, where I would have
> > to get mmio_addr via pci_resource_start() which means registering
> > xen-platform-pci driver. I know there will be implications later
> > with migration, but thats later.
> > 
> > I can't think of any other way since the grant frames are coming
> > from xen heap and not in p2m table, other than some sort of
> > xen-heap-p2m-table or something ugly.
> > 
> > Am I on right track, or any other ideas?
> > 
> > thanks,
> > Mukesh
> > 
> 

Re: HYBRID: gnttab_map() to map shared frames..

[Ian Campbell]

On Tue, 2011-10-18 at 03:18 +0100, Mukesh Rathor wrote:
> Hi Jeremy,
> 
> You have any thoughts on this? This is basically for hybrid running 
> autotranslate paging mode.  I am trying to figure how to get unmapped
> pfns to map shared grant table page/s from xen. HVM does this via
> pci_resource_start() and gets io remapped address. However, in case of
> pure PV, I am not sure how feasible this approach would be.
> 
> Is there way I can reserve few pfn's above max_pfn and use those
> for my XENMAPSPACE_grant_table call?

I think we should avoid exposing any sort of magic PCI bus to PV/hybrid
guests.

Could GNTTABOP_setup_table return GFNs from the very top of the GFN
space? Perhaps even above what would be architecturally valid, although
perhaps that is incompatible with HAP. Anything above max_pfn would seem
to be valid for the hypervisor to place "magic" mappings in?

Another approach might be to take advantage of the fact that a modern PV
kernel will call XENMEM_memory_map to obtain an e820 describing its
pseudo-physical address space. You could arrange for there to be some
reserved space returned in that.

Ian.

Re: HYBRID: gnttab_map() to map shared frames..

[Mukesh Rathor]

On Tue, 18 Oct 2011 09:13:02 +0100
Ian Campbell <Ian.Campbell@citrix.com> wrote:
... 
> Could GNTTABOP_setup_table return GFNs from the very top of the GFN
> space? Perhaps even above what would be architecturally valid,
> although perhaps that is incompatible with HAP. Anything above
> max_pfn would seem to be valid for the hypervisor to place "magic"
> mappings in?

Well, what's confusing me for this is that there are few max 
pfn's inside the guest! The nr_pages being one for example, then 
there's max_pfn in shared_info.arch. I'm not sure if these two are in 
sync with max_pfn in mm/bootmem.c. Then, we would need to adjust EPT to add 
these pfns there. So that may put a damper on this.

> Another approach might be to take advantage of the fact that a modern
> PV kernel will call XENMEM_memory_map to obtain an e820 describing its
> pseudo-physical address space. You could arrange for there to be some
> reserved space returned in that.

I saw e820 code last night, and wondered if this might be the best option.
I am glad you mentioned it. I am not much familiar with e820 format, so 
studying it right now.

thanks,
Mukesh

Re: HYBRID: gnttab_map() to map shared frames..

[Ian Campbell]

On Tue, 2011-10-18 at 20:17 +0100, Mukesh Rathor wrote:
> On Tue, 18 Oct 2011 09:13:02 +0100
> Ian Campbell <Ian.Campbell@citrix.com> wrote:
> ... 
> > Could GNTTABOP_setup_table return GFNs from the very top of the GFN
> > space? Perhaps even above what would be architecturally valid,
> > although perhaps that is incompatible with HAP. Anything above
> > max_pfn would seem to be valid for the hypervisor to place "magic"
> > mappings in?
> 
> Well, what's confusing me for this is that there are few max 
> pfn's inside the guest! The nr_pages being one for example, then 
> there's max_pfn in shared_info.arch. I'm not sure if these two are in 
> sync with max_pfn in mm/bootmem.c. Then, we would need to adjust EPT to add 
> these pfns there. So that may put a damper on this.

What about using PFNs from right at the top, i.e. right up near
0xffffffff? I don't think there's any particular reason these special
PFNs need to be contiguous with the "regular" ones. This would limit the
total amount of RAM you could give a hybrid guest, but not by much.

> > Another approach might be to take advantage of the fact that a modern
> > PV kernel will call XENMEM_memory_map to obtain an e820 describing its
> > pseudo-physical address space. You could arrange for there to be some
> > reserved space returned in that.
> 
> I saw e820 code last night, and wondered if this might be the best option.
> I am glad you mentioned it. I am not much familiar with e820 format, so 
> studying it right now.
> 
> thanks,
> Mukesh
> 

Re: HYBRID: gnttab_map() to map shared frames..

[Mukesh Rathor]

On Wed, 19 Oct 2011 09:39:33 +0100
Ian Campbell <Ian.Campbell@citrix.com> wrote:

> On Tue, 2011-10-18 at 20:17 +0100, Mukesh Rathor wrote:
> > On Tue, 18 Oct 2011 09:13:02 +0100
> > Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > ... 
> > > Could GNTTABOP_setup_table return GFNs from the very top of the
> > > GFN space? Perhaps even above what would be architecturally valid,
> > > although perhaps that is incompatible with HAP. Anything above
> > > max_pfn would seem to be valid for the hypervisor to place "magic"
> > > mappings in?
> > 
> > Well, what's confusing me for this is that there are few max 
> > pfn's inside the guest! The nr_pages being one for example, then 
> > there's max_pfn in shared_info.arch. I'm not sure if these two are
> > in sync with max_pfn in mm/bootmem.c. Then, we would need to adjust
> > EPT to add these pfns there. So that may put a damper on this.
> 
> What about using PFNs from right at the top, i.e. right up near
> 0xffffffff? I don't think there's any particular reason these special
> PFNs need to be contiguous with the "regular" ones. This would limit
> the total amount of RAM you could give a hybrid guest, but not by
> much.

Yeah, I dinkered around a bit with e820 and realized it would be
migration headache, so just decided to do this. I allocate pfn's near
0xffffffff and map it. Things are fine, I see all page table entries
just fine, the entry is put into the p2m also via ept_set_entry, but
for some reason it's taking fault on accessing shared[0] with error
code 0xb that doesn't make sense. Even the mfn in xen is writable
page. 

PGD 77895067 PUD 77896067 PMD 77897067 PTE 80000ffffffdf063

I'll continue debugging.

thanks,
Mukesh

Re: Re: HYBRID: gnttab_map() to map shared frames..

[Tim Deegan]

At 19:29 -0700 on 19 Oct (1319052558), Mukesh Rathor wrote:
> > What about using PFNs from right at the top, i.e. right up near
> > 0xffffffff? I don't think there's any particular reason these special
> > PFNs need to be contiguous with the "regular" ones. This would limit
> > the total amount of RAM you could give a hybrid guest, but not by
> > much.
> 
> Yeah, I dinkered around a bit with e820 and realized it would be
> migration headache, so just decided to do this. I allocate pfn's near
> 0xffffffff and map it. Things are fine, I see all page table entries
> just fine, the entry is put into the p2m also via ept_set_entry, but
> for some reason it's taking fault on accessing shared[0] with error
> code 0xb that doesn't make sense. Even the mfn in xen is writable
> page. 
> 
> PGD 77895067 PUD 77896067 PMD 77897067 PTE 80000ffffffdf063

0xB means you've got a reserved bit set.  Is 0xfff_fffdf000 above the 
number of supported addressing bits on your CPU, by any chance?

Cheers,

Tim

Re: Re: HYBRID: gnttab_map() to map shared frames..

[Mukesh Rathor]

On Thu, 20 Oct 2011 11:39:12 +0100
Tim Deegan <tim@xen.org> wrote:

> At 19:29 -0700 on 19 Oct (1319052558), Mukesh Rathor wrote:
> > > What about using PFNs from right at the top, i.e. right up near
> > > 0xffffffff? I don't think there's any particular reason these
> > > special PFNs need to be contiguous with the "regular" ones. This
> > > would limit the total amount of RAM you could give a hybrid
> > > guest, but not by much.
> > 
> > Yeah, I dinkered around a bit with e820 and realized it would be
> > migration headache, so just decided to do this. I allocate pfn's
> > near 0xffffffff and map it. Things are fine, I see all page table
> > entries just fine, the entry is put into the p2m also via
> > ept_set_entry, but for some reason it's taking fault on accessing
> > shared[0] with error code 0xb that doesn't make sense. Even the mfn
> > in xen is writable page. 
> > 
> > PGD 77895067 PUD 77896067 PMD 77897067 PTE 80000ffffffdf063
> 
> 0xB means you've got a reserved bit set.  Is 0xfff_fffdf000 above the 
> number of supported addressing bits on your CPU, by any chance?
> 
> Cheers,
> 
> Tim

Ooops, forgot to PAGE_SHIFT..  got that working now...

thanks,
mukesh