Can't map the page referenced by HVM-DomU CR3 in Dom0

Can't map the page referenced by HVM-DomU CR3 in Dom0

[david]

Hi,

I'm trying to access the page containing the paging information for a 
DomU from Dom0.

I'm doing that by translating the address contained in the DomU CR3 
register with xc_translate_foreign_address (libxc) and try to map the
returned frame number with xc_map_foreign_range.

The problem is, that the return value from xc_translate_foreign_address 
is 0 (guest cr3 is 0x002f3000 in my case), which indicates an error 
(corresponding to the code comments). After some debugging I have 
discovered, that pte becomes 0 when level=2 and therefore the function 
returns 0 on line 79:

tools/libxc/xc_pagetab.c
69    /* Walk the pagetables */
70    for (level = pt_levels; level > 0; level--) {
71        paddr += ((virt & mask) >> (xc_ffs64(mask) - 1)) * size;
72        map = xc_map_foreign_range(xc_handle, dom, PAGE_SIZE, PROT_READ,
73                                   paddr >>PAGE_SHIFT);
74        if (!map)
75            return 0;
76        memcpy(&pte, map + (paddr & (PAGE_SIZE - 1)), size);
77        munmap(map, PAGE_SIZE);
78        if (!(pte & 1))
79            return 0;
80        paddr = pte & 0x000ffffffffff000ull;
...
...

I'm currently trying to examine why pte becomes 0. Is anyone familiar 
with this part of the code and can explain why it is not possible to map 
the page?

My setup:
Dom0: debian 2.6.32.26 x86_64
DomU: HVM Win XP SP2 32bit
xen-4.0.1


thanks for any hints,


david

Re: Can't map the page referenced by HVM-DomU CR3 in Dom0

[Tim Deegan]

At 14:45 +0100 on 15 Apr (1302878734), david wrote:
> I'm trying to access the page containing the paging information for a 
> DomU from Dom0.
> 
> I'm doing that by translating the address contained in the DomU CR3 
> register with xc_translate_foreign_address (libxc) and try to map the
> returned frame number with xc_map_foreign_range.
> 
> The problem is, that the return value from xc_translate_foreign_address 
> is 0 (guest cr3 is 0x002f3000 in my case), which indicates an error 
> (corresponding to the code comments). After some debugging I have 
> discovered, that pte becomes 0 when level=2 and therefore the function 
> returns 0 on line 79:

How often does this happen?  On every attempt or only from time to time?
Have you checked (say, from inside the guest) that the level-2 PTE isn't
actually zero?

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Can't map the page referenced by HVM-DomU CR3 in Dom0

[david]

On 04/18/2011 11:34 AM, Tim Deegan wrote:
> At 14:45 +0100 on 15 Apr (1302878734), david wrote:
>> I'm trying to access the page containing the paging information for a
>> DomU from Dom0.
>>
>> I'm doing that by translating the address contained in the DomU CR3
>> register with xc_translate_foreign_address (libxc) and try to map the
>> returned frame number with xc_map_foreign_range.
>>
>> The problem is, that the return value from xc_translate_foreign_address
>> is 0 (guest cr3 is 0x002f3000 in my case), which indicates an error
>> (corresponding to the code comments). After some debugging I have
>> discovered, that pte becomes 0 when level=2 and therefore the function
>> returns 0 on line 79:
>
> How often does this happen?  On every attempt or only from time to time?
> Have you checked (say, from inside the guest) that the level-2 PTE isn't
> actually zero?

hi,

it happens for every cr3 value. I made some quick and dirty code, which 
reads 10 different cr3 values and tries to map the corresponding page:

----------------------------------------------------------------------
...
...
int crfinder = 1;

if(crfinder == 1){
         int m;
         unsigned long cr3s[10] = {0};
         unsigned long mfn = 0;
         vcpu_guest_context_any_t *ctxt = 
malloc(sizeof(vcpu_guest_context_any_t));
         unsigned long cr3 = ctxt->c.ctrlreg[3];

         while(1 == 1){

                 xc_vcpu_getcontext(xcinterface, domain, 
dominfo.max_vcpu_id, ctxt);
                 cr3 = ctxt->c.ctrlreg[3];

                 for(m = 0; m < 10; m++){

                         //already stored?
                         if(cr3s[m] == cr3){

                                 break;
                         //checked all stored cr3 values?
                         }else if (cr3s[m] != 0){

                                 continue;
                         //obviously new one found
                         }else{
                                 cr3s[m] = cr3;
                                 printf("new cr3 found %08x, stored in 
%d\n", cr3, m);

                                 mfn = 
xc_translate_foreign_address(xcinterface, domain, 0, cr3s[m]);
                                 printf("calculated mfn %08d for address 
%08x\n", mfn, cr3s[m]);
                                 break;
                         }
                 }

                 if(m == 10)
                        return 0;
         }
}
----------------------------------------------------------------------

the corresponding output is:

new cr3 found 002f3000, stored in 0
calculated mfn 00000000 for address 002f3000
new cr3 found 06ac01a0, stored in 1
calculated mfn 00000000 for address 06ac01a0
new cr3 found 06ac0040, stored in 2
calculated mfn 00000000 for address 06ac0040
new cr3 found 06ac00a0, stored in 3
calculated mfn 00000000 for address 06ac00a0
new cr3 found 06ac01e0, stored in 4
calculated mfn 00000000 for address 06ac01e0
new cr3 found 06ac0320, stored in 5
calculated mfn 00000000 for address 06ac0320
new cr3 found 06ac02a0, stored in 6
calculated mfn 00000000 for address 06ac02a0
new cr3 found 06ac01c0, stored in 7
calculated mfn 00000000 for address 06ac01c0
new cr3 found 06ac0200, stored in 8
calculated mfn 00000000 for address 06ac0200
new cr3 found 06ac0060, stored in 9
calculated mfn 00000000 for address 06ac0060

so, every try to translate a cr3 address to a frame number (I don't know 
what's the correct wording for frame numbers in hvm domains, .. mfn?) 
ends in 0. Maybe it's a failure in my code? I can't find it currently :) 
... I'm trying now, to read the cr3 values inside the domain, to check 
if the values are the same.

greets,
david


>
> Cheers,
>
> Tim.
>

Re: Can't map the page referenced by HVM-DomU CR3 in Dom0

[Keir Fraser]

On 19/04/2011 16:44, "david" <david_n@gmx.at> wrote:

> On 04/18/2011 11:34 AM, Tim Deegan wrote:
>> At 14:45 +0100 on 15 Apr (1302878734), david wrote:
>>> I'm trying to access the page containing the paging information for a
>>> DomU from Dom0.
>>> 
>>> I'm doing that by translating the address contained in the DomU CR3
>>> register with xc_translate_foreign_address (libxc) and try to map the
>>> returned frame number with xc_map_foreign_range.
>>> 
>>> The problem is, that the return value from xc_translate_foreign_address
>>> is 0 (guest cr3 is 0x002f3000 in my case), which indicates an error
>>> (corresponding to the code comments). After some debugging I have
>>> discovered, that pte becomes 0 when level=2 and therefore the function
>>> returns 0 on line 79:
>> 
>> How often does this happen?  On every attempt or only from time to time?
>> Have you checked (say, from inside the guest) that the level-2 PTE isn't
>> actually zero?
> 
> hi,
> 
> it happens for every cr3 value. I made some quick and dirty code, which
> reads 10 different cr3 values and tries to map the corresponding page:

xc_translate_foreign_address() will convert a guest virtual address into a
guest physical address. It's not working out for you because guest cr3
values are already guest physical addresses. Thus the virtual-to-physical
conversion you subject the values to is meaningless.

 -- Keir

> ----------------------------------------------------------------------
> ...
> ...
> int crfinder = 1;
> 
> if(crfinder == 1){
>          int m;
>          unsigned long cr3s[10] = {0};
>          unsigned long mfn = 0;
>          vcpu_guest_context_any_t *ctxt =
> malloc(sizeof(vcpu_guest_context_any_t));
>          unsigned long cr3 = ctxt->c.ctrlreg[3];
> 
>          while(1 == 1){
> 
>                  xc_vcpu_getcontext(xcinterface, domain,
> dominfo.max_vcpu_id, ctxt);
>                  cr3 = ctxt->c.ctrlreg[3];
> 
>                  for(m = 0; m < 10; m++){
> 
>                          //already stored?
>                          if(cr3s[m] == cr3){
> 
>                                  break;
>                          //checked all stored cr3 values?
>                          }else if (cr3s[m] != 0){
> 
>                                  continue;
>                          //obviously new one found
>                          }else{
>                                  cr3s[m] = cr3;
>                                  printf("new cr3 found %08x, stored in
> %d\n", cr3, m);
> 
>                                  mfn =
> xc_translate_foreign_address(xcinterface, domain, 0, cr3s[m]);
>                                  printf("calculated mfn %08d for address
> %08x\n", mfn, cr3s[m]);
>                                  break;
>                          }
>                  }
> 
>                  if(m == 10)
>                         return 0;
>          }
> }
> ----------------------------------------------------------------------
> 
> the corresponding output is:
> 
> new cr3 found 002f3000, stored in 0
> calculated mfn 00000000 for address 002f3000
> new cr3 found 06ac01a0, stored in 1
> calculated mfn 00000000 for address 06ac01a0
> new cr3 found 06ac0040, stored in 2
> calculated mfn 00000000 for address 06ac0040
> new cr3 found 06ac00a0, stored in 3
> calculated mfn 00000000 for address 06ac00a0
> new cr3 found 06ac01e0, stored in 4
> calculated mfn 00000000 for address 06ac01e0
> new cr3 found 06ac0320, stored in 5
> calculated mfn 00000000 for address 06ac0320
> new cr3 found 06ac02a0, stored in 6
> calculated mfn 00000000 for address 06ac02a0
> new cr3 found 06ac01c0, stored in 7
> calculated mfn 00000000 for address 06ac01c0
> new cr3 found 06ac0200, stored in 8
> calculated mfn 00000000 for address 06ac0200
> new cr3 found 06ac0060, stored in 9
> calculated mfn 00000000 for address 06ac0060
> 
> so, every try to translate a cr3 address to a frame number (I don't know
> what's the correct wording for frame numbers in hvm domains, .. mfn?)
> ends in 0. Maybe it's a failure in my code? I can't find it currently :)
> ... I'm trying now, to read the cr3 values inside the domain, to check
> if the values are the same.
> 
> greets,
> david
> 
> 
>> 
>> Cheers,
>> 
>> Tim.
>> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Can't map the page referenced by HVM-DomU CR3 in Dom0

[david]

On 04/19/2011 06:26 PM, Keir Fraser wrote:
> On 19/04/2011 16:44, "david"<david_n@gmx.at>  wrote:
>
>> On 04/18/2011 11:34 AM, Tim Deegan wrote:
>>> At 14:45 +0100 on 15 Apr (1302878734), david wrote:
>>>> I'm trying to access the page containing the paging information for a
>>>> DomU from Dom0.
>>>>
>>>> I'm doing that by translating the address contained in the DomU CR3
>>>> register with xc_translate_foreign_address (libxc) and try to map the
>>>> returned frame number with xc_map_foreign_range.
>>>>
>>>> The problem is, that the return value from xc_translate_foreign_address
>>>> is 0 (guest cr3 is 0x002f3000 in my case), which indicates an error
>>>> (corresponding to the code comments). After some debugging I have
>>>> discovered, that pte becomes 0 when level=2 and therefore the function
>>>> returns 0 on line 79:
>>>
>>> How often does this happen?  On every attempt or only from time to time?
>>> Have you checked (say, from inside the guest) that the level-2 PTE isn't
>>> actually zero?
>>
>> hi,
>>
>> it happens for every cr3 value. I made some quick and dirty code, which
>> reads 10 different cr3 values and tries to map the corresponding page:
>
> xc_translate_foreign_address() will convert a guest virtual address into a
> guest physical address. It's not working out for you because guest cr3
> values are already guest physical addresses. Thus the virtual-to-physical
> conversion you subject the values to is meaningless.

ahhh, .. I see. I thought the translation is between guest physical and 
real physical address. The way to calculate the correct physical frame 
number is cr3 >> PAGE_SHIFT corresponding to xen_cr3_to_pfn(cr3) in 
public/arch-x86/xen-x86_64.h, right?

thanks for your help,

greets
david


>
>   -- Keir
>
>> ----------------------------------------------------------------------
>> ...
>> ...
>> int crfinder = 1;
>>
>> if(crfinder == 1){
>>           int m;
>>           unsigned long cr3s[10] = {0};
>>           unsigned long mfn = 0;
>>           vcpu_guest_context_any_t *ctxt =
>> malloc(sizeof(vcpu_guest_context_any_t));
>>           unsigned long cr3 = ctxt->c.ctrlreg[3];
>>
>>           while(1 == 1){
>>
>>                   xc_vcpu_getcontext(xcinterface, domain,
>> dominfo.max_vcpu_id, ctxt);
>>                   cr3 = ctxt->c.ctrlreg[3];
>>
>>                   for(m = 0; m<  10; m++){
>>
>>                           //already stored?
>>                           if(cr3s[m] == cr3){
>>
>>                                   break;
>>                           //checked all stored cr3 values?
>>                           }else if (cr3s[m] != 0){
>>
>>                                   continue;
>>                           //obviously new one found
>>                           }else{
>>                                   cr3s[m] = cr3;
>>                                   printf("new cr3 found %08x, stored in
>> %d\n", cr3, m);
>>
>>                                   mfn =
>> xc_translate_foreign_address(xcinterface, domain, 0, cr3s[m]);
>>                                   printf("calculated mfn %08d for address
>> %08x\n", mfn, cr3s[m]);
>>                                   break;
>>                           }
>>                   }
>>
>>                   if(m == 10)
>>                          return 0;
>>           }
>> }
>> ----------------------------------------------------------------------
>>
>> the corresponding output is:
>>
>> new cr3 found 002f3000, stored in 0
>> calculated mfn 00000000 for address 002f3000
>> new cr3 found 06ac01a0, stored in 1
>> calculated mfn 00000000 for address 06ac01a0
>> new cr3 found 06ac0040, stored in 2
>> calculated mfn 00000000 for address 06ac0040
>> new cr3 found 06ac00a0, stored in 3
>> calculated mfn 00000000 for address 06ac00a0
>> new cr3 found 06ac01e0, stored in 4
>> calculated mfn 00000000 for address 06ac01e0
>> new cr3 found 06ac0320, stored in 5
>> calculated mfn 00000000 for address 06ac0320
>> new cr3 found 06ac02a0, stored in 6
>> calculated mfn 00000000 for address 06ac02a0
>> new cr3 found 06ac01c0, stored in 7
>> calculated mfn 00000000 for address 06ac01c0
>> new cr3 found 06ac0200, stored in 8
>> calculated mfn 00000000 for address 06ac0200
>> new cr3 found 06ac0060, stored in 9
>> calculated mfn 00000000 for address 06ac0060
>>
>> so, every try to translate a cr3 address to a frame number (I don't know
>> what's the correct wording for frame numbers in hvm domains, .. mfn?)
>> ends in 0. Maybe it's a failure in my code? I can't find it currently :)
>> ... I'm trying now, to read the cr3 values inside the domain, to check
>> if the values are the same.
>>
>> greets,
>> david
>>
>>
>>>
>>> Cheers,
>>>
>>> Tim.
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
>

Re: Can't map the page referenced by HVM-DomU CR3 in Dom0

[Keir Fraser]

On 19/04/2011 22:58, "david" <david_n@gmx.at> wrote:

>> xc_translate_foreign_address() will convert a guest virtual address into a
>> guest physical address. It's not working out for you because guest cr3
>> values are already guest physical addresses. Thus the virtual-to-physical
>> conversion you subject the values to is meaningless.
> 
> ahhh, .. I see. I thought the translation is between guest physical and
> real physical address.

The toolstack deals entirely with guest physical addresses for HVM guests.
There's actually no way, and no need, to find out the underlying real
physical addresses.

> The way to calculate the correct physical frame
> number is cr3 >> PAGE_SHIFT corresponding to xen_cr3_to_pfn(cr3) in
> public/arch-x86/xen-x86_64.h, right?

Yes, although strictly speaking the xen_cr3_to_pfn() macro is intended to be
used only with PV guests' cr3 values (the x86_32 version of that macro is
not suitable for use with HVM cr3 values). Consider an HVM guest in 32-bit
PAE mode -- its CR3 value may not be page aligned.

 -- Keir