Consuming PCI device in PV kernel

Consuming PCI device in PV kernel

[Simon Martin]

Hi all,

My  PV  kernel is running nicely and the modded gntalloc kernel driver
is  working like a dream, thanks to all that have helped over the last
year.

The  next  challenge  is  writing  directly to an Ethernet NIC. I have
imported  this  into  my guest domain using PCI passthrough, however I
can't   find   any  documentation  on where to go from there. Is there
any documentation for this?

Regards.



-- 
Best regards,
 Simon                          mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Konrad Rzeszutek Wilk]

On Thu, Jul 03, 2014 at 12:13:38PM +0100, Simon Martin wrote:
> Hi all,
> 
> My  PV  kernel is running nicely and the modded gntalloc kernel driver
> is  working like a dream, thanks to all that have helped over the last
> year.
> 
> The  next  challenge  is  writing  directly to an Ethernet NIC. I have
> imported  this  into  my guest domain using PCI passthrough, however I
> can't   find   any  documentation  on where to go from there. Is there
> any documentation for this?

As in doing what? Writting to the MMIO region of the Ethernet NIC?
Take a look at Intel's DPDK.

> 
> Regards.
> 
> 
> 
> -- 
> Best regards,
>  Simon                          mailto:furryfuttock@gmail.com
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Consuming PCI device in PV kernel

[Simon Martin]

Thanks Konrad,

>> The  next  challenge  is  writing  directly to an Ethernet NIC. I have
>> imported  this  into  my guest domain using PCI passthrough, however I
>> can't   find   any  documentation  on where to go from there. Is there
>> any documentation for this?

> As in doing what? Writting to the MMIO region of the Ethernet NIC?
> Take a look at Intel's DPDK.

Interesting  link,  and I will need it in the future, however my first
hurdle  is  enumerating  and  then  mapping  the PCI device into my PV
domain.  I  am  reading through the Linux kernel initialisation at the
moment to see how it gains access to PCI devices.

I  can  see  the  PCI  device references in the Xenstore, but am still
working out how to go from that data to an available PCI device in the
PV domain.

Regards.

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hi all,

> I  can  see  the  PCI  device references in the Xenstore, but am still
> working out how to go from that data to an available PCI device in the
> PV domain.

Found the code in mini-os. Implementing now.

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Realtime access to PCI NIC

[Simon Martin]

Hi all,

I am reading through the mini-os pcifront.c as an example of how to
implement direct access to a PCI device. However looking at the code
all access seems to go via the PCI Backend driver, which means that I
am subject to the latencies and jitter of the Xen scheduling.

Reading the documentation about PCI Passthrough I imagined that I
would have direct access to the PCI hardware once the handover from
dom0 as performed. Is that correct? If so, how do I implement that?

I am using an Intel processor that has VT-d so that should be
possible.

Any suggestions?


-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Realtime access to PCI NIC

[Ian Campbell]

On Mon, 2014-07-07 at 13:21 +0100, Simon Martin wrote:
> Hi all,
> 
> I am reading through the mini-os pcifront.c as an example of how to
> implement direct access to a PCI device. However looking at the code
> all access seems to go via the PCI Backend driver, which means that I
> am subject to the latencies and jitter of the Xen scheduling.

Only PCI cfg reads/writes go via the backend, I think. Any MMIO space
(i.e. described in the PCI BARS) can be directly mapped, likewise for
the IO port spaces associated with the device.

I think PCI cfg space is only generally used during probing/setup etc.
Hopefully you don't need to poke it at runtime?

> Reading the documentation about PCI Passthrough I imagined that I
> would have direct access to the PCI hardware once the handover from
> dom0 as performed. Is that correct? If so, how do I implement that?
> 
> I am using an Intel processor that has VT-d so that should be
> possible.

Right, VT-d allows the MMIO stuff to be mapped to an HVM guest (PV
doesn't even need that), but unfortunately there are bits in the pci cfg
space which even a guest with the device passed to it shouldn't be able
to mess with (I don't know the details though).

Ian.

Re: Consuming PCI device in PV kernel

[Konrad Rzeszutek Wilk]

On Mon, Jul 07, 2014 at 12:22:33PM +0100, Simon Martin wrote:
> Hi all,
> 
> > I  can  see  the  PCI  device references in the Xenstore, but am still
> > working out how to go from that data to an available PCI device in the
> > PV domain.
> 
> Found the code in mini-os. Implementing now.

Why not use the Xen PCI frontend?
> 
> -- 
> Best regards,
>  Simon                            mailto:furryfuttock@gmail.com
> 

Re: Realtime access to PCI NIC

[Simon Martin]

Thanks Ian,

Tuesday, July 8, 2014, 3:46:25 PM, you wrote:


> Only PCI cfg reads/writes go via the backend, I think. Any MMIO space
> (i.e. described in the PCI BARS) can be directly mapped, likewise for
> the IO port spaces associated with the device.

> I think PCI cfg space is only generally used during probing/setup etc.
> Hopefully you don't need to poke it at runtime?

You have confirmed the conclusion that I was coming to!

> Right, VT-d allows the MMIO stuff to be mapped to an HVM guest (PV
> doesn't even need that), but unfortunately there are bits in the pci cfg
> space which even a guest with the device passed to it shouldn't be able
> to mess with (I don't know the details though).

Perfect.

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Konrad,

Tuesday, July 8, 2014, 4:01:36 PM, you wrote:

> Why not use the Xen PCI frontend?

I am porting an embedded realtime OS to Xen. This is a reasonably
simple OS and has no dynamic memory allocation (malloc), flat memory
map (no paging), etc.

Porting the Xen PCI frontend driver would force me to implement a lot
of stuff that I don't actually require.

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 630 bytes --]

Hey Simon!

On gio, 2014-07-10 at 08:54 +0100, Simon Martin wrote:
> I am porting an embedded realtime OS to Xen. This is a reasonably
> simple OS and has no dynamic memory allocation (malloc), flat memory
> map (no paging), etc.
> 
BTW, how's this going (if I can ask)? Any exciting news about this you
can share with the community? :-)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Dario,

> BTW, how's this going (if I can ask)? Any exciting news about this you
> can share with the community? :-)

Nothing   exciting   I'm   afraid,   just   lots  of  donkey work. The
micropv  project (similar to the Xen mini-os but a LOT lighter with no
dynamic   memory,   paging  etc)  is  coming  along  nicely.  Lots  of
improvements   about   talking   to   the  hypervisor and xenstore and
making  it  more implementation agnostic. The mirage-os guys seemed to
be interested in this when I first mentioned it. No idea if they still
are, it's been a while. Too many projects, too little time ;-)

My  PV  OS  is  sitting  quite  nicely in the hypervisor and I have my
context  switch  running  nicely  within  the  micropv.  One problem I
haven't  been  able  to  work  around  so  far is implementing lazy FP
context recovery, I do it on every context switch. This is crazy but I
get  FP  exceptions if I don't. As I have larger fish to fry, this one
is just flagged as something to come back to.

I had to modify one of the existing Linux kernel drivers (gntalloc) to
provide  more  functionality.  I  got  it  to  a state in which it was
accepted  by  Daniel De Graaf the original implementor. I now have to
push it upstream the Linux kernel maintainers.

As  a  general  comment  I find that the existing Xen documentation is
fine  for handling existing HVM/PV domain implementations, but you are
on  your  own  when it comes to implementing your own PV. If it wasn't
for  the  good  will  of the people on this mailing list I would never
have  been  able  to  get  where I am now. I know that I am in a small
niche  here,  and the existing documentation is fine for 99% of people
who are using Xen. If this is something that should be documented then
I  would  be  willing  to  have  a  stab at writing some wiki pages if
someone can clean up the things that I have misunderstood.

Regards, and once again, a big thanks to everyone

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Konrad Rzeszutek Wilk]

On Tue, Jul 15, 2014 at 09:37:56AM +0100, Simon Martin wrote:
> Hello Dario,
> 
> > BTW, how's this going (if I can ask)? Any exciting news about this you
> > can share with the community? :-)
> 
> Nothing   exciting   I'm   afraid,   just   lots  of  donkey work. The
> micropv  project (similar to the Xen mini-os but a LOT lighter with no
> dynamic   memory,   paging  etc)  is  coming  along  nicely.  Lots  of
> improvements   about   talking   to   the  hypervisor and xenstore and
> making  it  more implementation agnostic. The mirage-os guys seemed to
> be interested in this when I first mentioned it. No idea if they still
> are, it's been a while. Too many projects, too little time ;-)
> 
> My  PV  OS  is  sitting  quite  nicely in the hypervisor and I have my
> context  switch  running  nicely  within  the  micropv.  One problem I
> haven't  been  able  to  work  around  so  far is implementing lazy FP
> context recovery, I do it on every context switch. This is crazy but I
> get  FP  exceptions if I don't. As I have larger fish to fry, this one
> is just flagged as something to come back to.
> 
> I had to modify one of the existing Linux kernel drivers (gntalloc) to
> provide  more  functionality.  I  got  it  to  a state in which it was
> accepted  by  Daniel De Graaf the original implementor. I now have to
> push it upstream the Linux kernel maintainers.

Please CC xen-devel@lists.xenproject.org, boris.ostrovsky@oracle.com,
david.vrabel@citrix.com, and me when you get a chance to post it.

Please do also run it first through scripts/checkpatch.pl
> 
> As  a  general  comment  I find that the existing Xen documentation is
> fine  for handling existing HVM/PV domain implementations, but you are
> on  your  own  when it comes to implementing your own PV. If it wasn't
> for  the  good  will  of the people on this mailing list I would never
> have  been  able  to  get  where I am now. I know that I am in a small
> niche  here,  and the existing documentation is fine for 99% of people
> who are using Xen. If this is something that should be documented then
> I  would  be  willing  to  have  a  stab at writing some wiki pages if
> someone can clean up the things that I have misunderstood.

Would you be interested in expanding this Wiki a bit:

http://wiki.xen.org/wiki/X86_Paravirtualised_Memory_Management

or correcting it?
?
> 
> Regards, and once again, a big thanks to everyone

Thank you for working on it and I am looking forward to your patches!
<cue evil maniacal laughter> Fresh patches! :-)
> 
> -- 
> Best regards,
>  Simon                            mailto:furryfuttock@gmail.com
> 

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Simon,

Monday, July 7, 2014, 12:22:33 PM, you wrote:

> Found the code in mini-os. Implementing now.

I  now  have  a  working implementation of my own pcifront based on the
mini-os  implementation,  however I can't seem to find any examples or
documentation  anywhere on how to discover and map the PCI device into
my PV memory map.

Checking  the  Linux  pcifront  driver it seems to be doing IO via the
pciback  driver. This won't work in my situation, I need raw access to
the PCI device memory.

Looking  at  the  GPLPV  drivers as well to see if I can decipher from
them how to do this.

Any suggestions on how to implement direct access to the PCI device?

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Konrad Rzeszutek Wilk]

On Fri, Jul 18, 2014 at 03:37:13PM +0100, Simon Martin wrote:
> Hello Simon,
> 
> Monday, July 7, 2014, 12:22:33 PM, you wrote:
> 
> > Found the code in mini-os. Implementing now.
> 
> I  now  have  a  working implementation of my own pcifront based on the
> mini-os  implementation,  however I can't seem to find any examples or
> documentation  anywhere on how to discover and map the PCI device into
> my PV memory map.

Not sure what 'PV memory map' means? You mean the BAR values and such?
> 
> Checking  the  Linux  pcifront  driver it seems to be doing IO via the
> pciback  driver. This won't work in my situation, I need raw access to
> the PCI device memory.

As in to the configuration registers (which would also have the BAR
values)?
> 
> Looking  at  the  GPLPV  drivers as well to see if I can decipher from
> them how to do this.
> 
> Any suggestions on how to implement direct access to the PCI device?
> 
> -- 
> Best regards,
>  Simon                            mailto:furryfuttock@gmail.com
> 

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Konrad,

Friday, July 18, 2014, 8:09:42 PM, you wrote:

>> I  now  have  a  working implementation of my own pcifront based on the
>> mini-os  implementation,  however I can't seem to find any examples or
>> documentation  anywhere on how to discover and map the PCI device into
>> my PV memory map.

> Not sure what 'PV memory map' means? You mean the BAR values and such?

That's exactly what I mean. I assume that BAR entries must be written
pointing to an absolute memory area in dom0 which will then be mapped
into the PV as this provides the greatest flexibility. Is this
correct? Or do I just enable a memory area in the PV and map the BAR
to the buffers physical address?

Regards.

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Ian Campbell]

On Mon, 2014-07-21 at 11:13 +0100, Simon Martin wrote:
> Hello Konrad,
> 
> Friday, July 18, 2014, 8:09:42 PM, you wrote:
> 
> >> I  now  have  a  working implementation of my own pcifront based on the
> >> mini-os  implementation,  however I can't seem to find any examples or
> >> documentation  anywhere on how to discover and map the PCI device into
> >> my PV memory map.
> 
> > Not sure what 'PV memory map' means? You mean the BAR values and such?
> 
> That's exactly what I mean. I assume that BAR entries must be written
> pointing to an absolute memory area in dom0 which will then be mapped
> into the PV as this provides the greatest flexibility. Is this
> correct? Or do I just enable a memory area in the PV and map the BAR
> to the buffers physical address?

IIRC from the guest PoV you just need to read the BAR values from the
CFG space and then use them without worrying about where they point,
i.e. by creating virtual address mappings of whatever MMIO address the
BAR gives you.

The initial setup of the BAR registers is handled either by the BIOS or
the dom0 kernel if it feels the need to do reassignment, you shouldn't
need to worry about it I don't think.

Ian.

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Ian,

> IIRC from the guest PoV you just need to read the BAR values from the
> CFG space and then use them without worrying about where they point,
> i.e. by creating virtual address mappings of whatever MMIO address the
> BAR gives you.

I  thought that I was mapping the BAR memory correctly, however when I
fixed  a  bug in my software I started getting errors and I eventually
tracked it down to mapping the MMIO memory.

I     have     tried     calling    HYPERCALL_update_va_mapping    and
HYPERCALL_mmu_update  and  they both fail. HYPERCALL_update_va_mapping
gives me the following in xl dmesg:

(XEN) mm.c:1700:d292v0 Bad L1 flags 400000

Looking at the code I can't see where bit 22 is being set!

Reading  the qemu PCI mapping code, it seems to call xc_domctl. I have
traced   down   through  this  into  the xen source and it seems to do
many  things that I don't have access to via straight hypercalls.

Google doesn't seem to help much on this one either.

Any idea on how to proceed with this?

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Ian Campbell]

On Fri, 2014-07-25 at 13:56 +0100, Simon Martin wrote:
> Hello Ian,
> 
> > IIRC from the guest PoV you just need to read the BAR values from the
> > CFG space and then use them without worrying about where they point,
> > i.e. by creating virtual address mappings of whatever MMIO address the
> > BAR gives you.
> 
> I  thought that I was mapping the BAR memory correctly, however when I
> fixed  a  bug in my software I started getting errors and I eventually
> tracked it down to mapping the MMIO memory.
> 
> I     have     tried     calling    HYPERCALL_update_va_mapping    and
> HYPERCALL_mmu_update  and  they both fail. HYPERCALL_update_va_mapping
> gives me the following in xl dmesg:
> 
> (XEN) mm.c:1700:d292v0 Bad L1 flags 400000
> 
> Looking at the code I can't see where bit 22 is being set!

Without seeing you code I've no idea about this (even if I could see it,
I dunno...)

> Reading  the qemu PCI mapping code, it seems to call xc_domctl. I have
> traced   down   through  this  into  the xen source and it seems to do
> many  things that I don't have access to via straight hypercalls.

domctl's are not for normal guest use.

So this is probably Qemu doing device model things. For a PV domain
either the toolstack has already done them, or they don't need doing or
they are something you do yourself via a different interface.

Ian.

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Ian,

Thanks for the response.

>> I     have     tried     calling    HYPERCALL_update_va_mapping    and
>> HYPERCALL_mmu_update  and  they both fail. HYPERCALL_update_va_mapping
>> gives me the following in xl dmesg:
>> 
>> (XEN) mm.c:1700:d292v0 Bad L1 flags 400000
>> 
>> Looking at the code I can't see where bit 22 is being set!

> Without seeing you code I've no idea about this (even if I could see it,
> I dunno...)

Bit 22 is being set in Xen code. This is what I am doing:

void *micropv_remap_page(uint64_t physical_address, uint64_t machine_address, int readonly)
{
    // Update the mapping. I have had problems using a readonly mapping, however I'm not sure whether that was to do with
    // this call, or the page that was being mapped.
    int rc = HYPERVISOR_update_va_mapping(physical_address, __pte(machine_address | (readonly ? L1_PROT_RO : L1_PROT)), UVMF_INVLPG);
    if (rc)
    {
        PRINTK("HYPERVISOR_update_va_mapping returns %i", rc);
        return NULL;
    }
    else
    {
        PRINTK("Machine Address %lx mapped to Physical Address %lx", machine_address, physical_address);
        return (void*)physical_address;
    }
}

This is the call when I try to map the PCI BAR.

        ptr = micropv_remap_page((uint64_t)buffer, pci_device.bar[0].memory.address << 4, 0);
        if (!ptr)
        {
                PRINTK("Error mapping PCI MMIO");
                goto fail;
        }

Where buffer is a page (4096) aligned buffer, one page long and
pci_device.bar.memory.address  is  the  top  28  bits  of  the PCI BAR
register (hence it is left shifted to give the correct address).


-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Ian Campbell]

On Fri, 2014-07-25 at 14:21 +0100, Simon Martin wrote:
> Hello Ian,
> 
> Thanks for the response.
> 
> >> I     have     tried     calling    HYPERCALL_update_va_mapping    and
> >> HYPERCALL_mmu_update  and  they both fail. HYPERCALL_update_va_mapping
> >> gives me the following in xl dmesg:
> >> 
> >> (XEN) mm.c:1700:d292v0 Bad L1 flags 400000
> >> 
> >> Looking at the code I can't see where bit 22 is being set!
> 
> > Without seeing you code I've no idea about this (even if I could see it,
> > I dunno...)
> 
> Bit 22 is being set in Xen code.

Hopefully someone who knows the x86 PV stuff can chime in then, I looked
but can't see what this represents.

>  This is what I am doing:
> 
> void *micropv_remap_page(uint64_t physical_address, uint64_t machine_address, int readonly)
> {
>     // Update the mapping. I have had problems using a readonly mapping, however I'm not sure whether that was to do with
>     // this call, or the page that was being mapped.
>     int rc = HYPERVISOR_update_va_mapping(physical_address, __pte(machine_address | (readonly ? L1_PROT_RO : L1_PROT)), UVMF_INVLPG);

The first argument should be the virtual address, I think. But maybe you
have a 1:1 mapping so this doesn't matter?

>     if (rc)
>     {
>         PRINTK("HYPERVISOR_update_va_mapping returns %i", rc);
>         return NULL;
>     }
>     else
>     {
>         PRINTK("Machine Address %lx mapped to Physical Address %lx", machine_address, physical_address);
>         return (void*)physical_address;
>     }
> }
> 
> This is the call when I try to map the PCI BAR.
> 
>         ptr = micropv_remap_page((uint64_t)buffer, pci_device.bar[0].memory.address << 4, 0);

Are you sure about that BAR decode? Don't you need to at least mask the
bottom 4 bits?

You might also need to mask it to a page boundary, depending on what the
address is...

I'm not 100% sure about that shift, it might already be encoded in the
BAR register due to the layout. Sometimes h/w people arrange it that
way... IOW you might only need to mask and not shift. 

Ian.

Re: Consuming PCI device in PV kernel

[Andrew Cooper]

On 25/07/14 14:37, Ian Campbell wrote:
> On Fri, 2014-07-25 at 14:21 +0100, Simon Martin wrote:
>> Hello Ian,
>>
>> Thanks for the response.
>>
>>>> I     have     tried     calling    HYPERCALL_update_va_mapping    and
>>>> HYPERCALL_mmu_update  and  they both fail. HYPERCALL_update_va_mapping
>>>> gives me the following in xl dmesg:
>>>>
>>>> (XEN) mm.c:1700:d292v0 Bad L1 flags 400000
>>>>
>>>> Looking at the code I can't see where bit 22 is being set!
>>> Without seeing you code I've no idea about this (even if I could see it,
>>> I dunno...)
>> Bit 22 is being set in Xen code.
> Hopefully someone who knows the x86 PV stuff can chime in then, I looked
> but can't see what this represents.

I suspect this would be:

/* Bit 22 of a 24-bit flag mask. This corresponds to bit 62 of a pte.*/
#define _PAGE_GNTTAB (1U<<22)

from include/asm-x86/x86_64/page.h

So you are attempting to update a pte which already has a granted page
in it.

~Andrew

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Andrew,

>>>>> Looking at the code I can't see where bit 22 is being set!
>>>> Without seeing you code I've no idea about this (even if I could see it,
>>>> I dunno...)
>>> Bit 22 is being set in Xen code.
>> Hopefully someone who knows the x86 PV stuff can chime in then, I looked
>> but can't see what this represents.

> I suspect this would be:

> /* Bit 22 of a 24-bit flag mask. This corresponds to bit 62 of a pte.*/
> #define _PAGE_GNTTAB (1U<<22)

> from include/asm-x86/x86_64/page.h

> So you are attempting to update a pte which already has a granted page
> in it.

That's  weird  as  I have only one domU running, so nothing else could
have claimed the PCI memory area. The PCI has been passed through as I
see the mapping in xenstore.

Should  I be using a different hypercall to gain access to this memory
area?


-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Andrew Cooper]

On 25/07/14 15:20, Simon Martin wrote:
> Hello Andrew,
>
>>>>>> Looking at the code I can't see where bit 22 is being set!
>>>>> Without seeing you code I've no idea about this (even if I could see it,
>>>>> I dunno...)
>>>> Bit 22 is being set in Xen code.
>>> Hopefully someone who knows the x86 PV stuff can chime in then, I looked
>>> but can't see what this represents.
>> I suspect this would be:
>> /* Bit 22 of a 24-bit flag mask. This corresponds to bit 62 of a pte.*/
>> #define _PAGE_GNTTAB (1U<<22)
>> from include/asm-x86/x86_64/page.h
>> So you are attempting to update a pte which already has a granted page
>> in it.
> That's  weird  as  I have only one domU running, so nothing else could
> have claimed the PCI memory area. The PCI has been passed through as I
> see the mapping in xenstore.
>
> Should  I be using a different hypercall to gain access to this memory
> area?
>
>

Once you grant map a page from another domain, your domain no longer
owns the PTE.  You should either avoid that PTE, or grant unmap the
foreign frame first.

This indicates that you have memory management issues/corruption in
whichever domain is receiving the update_va errors.

~Andrew

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Ian,

>>  This is what I am doing:
>> 
>> void *micropv_remap_page(uint64_t physical_address, uint64_t machine_address, int readonly)
>> {
>>     // Update the mapping. I have had problems using a readonly mapping, however I'm not sure whether that was to do with
>>     // this call, or the page that was being mapped.
>>     int rc = HYPERVISOR_update_va_mapping(physical_address, __pte(machine_address | (readonly ? L1_PROT_RO : L1_PROT)), UVMF_INVLPG);

> The first argument should be the virtual address, I think. But maybe you
> have a 1:1 mapping so this doesn't matter?

I   don't   think   so.  I  inverted the parameter order and it failed
during  the memory address validation. Also this is the same call as I
use for mapping the shared_info at start of day.

>>
>>         ptr = micropv_remap_page((uint64_t)buffer, pci_device.bar[0].memory.address << 4, 0);

> Are you sure about that BAR decode? Don't you need to at least mask the
> bottom 4 bits?

Yes  I  am sure on this. The decoded addresses look correct. I removed
it and get this output from xl dmesg

(XEN) mm.c:828:d304v0 pg_owner 304 l1e_owner 304, but real_pg_owner 0
(XEN) mm.c:899:d304v0 Error getting mfn f7d0 (pfn d2bd) from L1 entry 000000000f7d0027 for l1e_owner=304, pg_owner=304

> You might also need to mask it to a page boundary, depending on what the
> address is...

The BAR is page aligned the value is f7d00000


-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Andrew,

Friday, July 25, 2014, 3:25:35 PM, you wrote:

> Once you grant map a page from another domain, your domain no longer
> owns the PTE.  You should either avoid that PTE, or grant unmap the
> foreign frame first.

I  have  not  grant mapped this. I have just called update_va_mapping.
Should I be using grant?

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Consuming PCI device in PV kernel

[Ian Campbell]

On Fri, 2014-07-25 at 15:30 +0100, Simon Martin wrote:
> Hello Andrew,
> 
> Friday, July 25, 2014, 3:25:35 PM, you wrote:
> 
> > Once you grant map a page from another domain, your domain no longer
> > owns the PTE.  You should either avoid that PTE, or grant unmap the
> > foreign frame first.
> 
> I  have  not  grant mapped this. I have just called update_va_mapping.
> Should I be using grant?

Andrew is saying that the mapping which is currently at the virtual
address is *already* a grant mapping, not that you are trying to grant
map something here. Grant maps need to be unmapped with the grant unmap
hypercall, not just by rewriting the PTE. (IIRC).

You should not be using grants to map PCI IO space, no.

Ian.

Re: Consuming PCI device in PV kernel

[Simon Martin]

Hello Ian,

>> > Once you grant map a page from another domain, your domain no longer
>> > owns the PTE.  You should either avoid that PTE, or grant unmap the
>> > foreign frame first.
>> 
>> I  have  not  grant mapped this. I have just called update_va_mapping.
>> Should I be using grant?

> Andrew is saying that the mapping which is currently at the virtual
> address is *already* a grant mapping, not that you are trying to grant
> map something here. Grant maps need to be unmapped with the grant unmap
> hypercall, not just by rewriting the PTE. (IIRC).

> You should not be using grants to map PCI IO space, no.

Thanks  Ian. I'm off traveling next week so I'll pick up on this again
in August. I wonder what has grant mapped this memory area....

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com