PV drivers and zero copying

PV drivers and zero copying

[Oleksandr Andrushchenko]

Hi, all!

The aim of this mail is to highlight and discuss possible approaches to
implementing zero copying for PV drivers. Rationale behind this is that 
there
are use-cases when drivers operate with big shared buffers, e.g. 
display, when
memory copying from front’s buffer into back’s one may significantly hit
performance of the system (for example, for para-virtual display running 
at full
HD resolution at 60Hz it is approximately 475MB/sec).

Assumptions (which actually fit ARM platforms, but can be extended to other
platforms as well): Dom0 is a 1:1 mapped privileged domain, runs backend
driver/software DomU is an unprivileged domain without 1:1 memory 
mapping, runs
frontend driver

Buffer origin: while implementing zero copying the buffer allocation can 
happen
either on DomU’s end or Dom0’s one depending on the use-case and HW
capabilities/availability: When DomU allocates: It cannot guarantee physical
memory continuity of the buffers allocated Dom0’s HW *can* handle 
non-contiguous
memory buffers allocated by DomU for memory operations (DMA, for 
example), e.g.
either with IOMMU help or by any other means (HW block’s own MMU).  When 
Dom0
allocates as it is mapped 1:1 it can allocate physically contiguous memory
Dom0’s HW *cannot* handle non-contiguous memory buffers allocated by 
DomU for
memory operations by any means.

1 Sharing with granted references
==================================

1-1 Buffer allocated @DomU
--------------------------
@DomU
     alloc_xenballooned_pages(nr_pages, pages);
     cur_ref = gnttab_claim_grant_reference(&priv_gref_head);
     gnttab_grant_foreign_access_ref(cur_ref, otherend_id, ...);
     <pass grant_ref_t[] to Dom0>
@Dom0
     alloc_xenballooned_pages(nr_pages, pages);
     gnttab_set_map_op(&map_ops[i], addr, GNTMAP_host_map | 
GNTMAP_device_map,
         grefs[i], otherend_id);
     gnttab_map_refs(map_ops, NULL, pages, nr_pages);

1-2 Buffer allocated @Dom0
--------------------------
@Dom0
     <the code below is equivalent to xen_alloc_ballooned_pages without
      PV MMU support as seen in the balloon driver, the difference is that
      pages are explicitly allocated to be used for DMA>
     dma_alloc_wc(dev, size, &dev_addr, GFP_KERNEL | __GFP_NOWARN);
     HYPERVISOR_memory_op(XENMEM_populate_physmap, &reservation);
     cur_ref = gnttab_claim_grant_reference(&priv_gref_head);
     gnttab_grant_foreign_access_ref(cur_ref, otherend_id, ...);
     <pass grant_ref_t[] to DomU>
@Dom0
     alloc_xenballooned_pages(nr_pages, pages);
     gnttab_set_map_op(&map_ops[i], addr, GNTMAP_host_map, grefs[i], 
otherend_id);
     gnttab_map_refs(map_ops, NULL, pages, nr_pages);

2 Sharing with page transfers (GNTTABOP_transfer)
==================================================
FIXME: This use-case seems to be only needed while allocating physically
contiguous buffers at Dom0. For the reverse path 1-1 method can be used.

This approach relies on GNTTABOP_transfer API: “transfer <frame> to a 
foreign
domain. The foreign domain has previously registered its interest in the
transfer via <domid, ref>”, for full documentation see [1]. The process of
transferring pages is explained by Christopher Clark at [2] and is 
available as
implementation at [3], [4]. The relevant logic is in: 
xen/common/grant_table.c :
gnttab_transfer.

Basic workflow explained to me by Christopher:
- The mfn starts as owned by the sending domain, and that domain removes any
   mappings of it from its page tables. Xen will enforce that the 
reference count
must be low enough for the transfer to succeed.
- The receiving domain indicates interest for receiving a page by writing an
   entry in its grant table.
- You'll need to communicate the grant ref from the receiver to the 
sender (eg.
   via xenstore or another existing channel)
- The sending domain invokes the hypercall, with the grant ref from the
   receiving domain.
- The sending domain notifies the receiving domain somehow that the 
transfer has
   completed. (eg. send an event or via xenstore)
- Once the transfer has completed, the receiving domain will need to map the
   newly assigned page.
- Note: For the transfer, the receiving domain must have enough headroom to
   receive the new page, which means it must not have allocated all of 
its memory
quota already prior to the transfer. Typically this can be ensured by 
freeing
enough memory back to Xen before writing the grant ref.

3 Sharing with page exchange (XENMEM_exchange)
==============================================

This API was pointed to me by Stefano Stabellini as one of the possible 
ways to
achieve zero copying and share physically contiguous buffers. It is used 
by x86
SWIOTLB code (xen_create_contiguous_region, [5]), but as per my 
understanding
this API cannot be used on ARM as of now [6].  Conclusion: not an option 
for ARM
at the moment

Comparison for display use-case
===============================

1 Number of grant references used
1-1 grant references: nr_pages
1-2 GNTTABOP_transfer: nr_pages
1-3 XENMEM_exchange: not an option

2 Effect of DomU crash on Dom0 (its mapped pages)
2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully 
recovered
2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost 
for Dom0
2-3 XENMEM_exchange: not an option

3 Security issues from sharing Dom0 pages to DomU
1-1 grant references: none
1-2 GNTTABOP_transfer: none
1-3 XENMEM_exchange: not an option

At the moment approach 1 with granted references seems to be a winner for
sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.

Conclusion
==========

I would like to get some feedback from the community on which approach 
is more
suitable for sharing large buffers and to have a clear vision on cons 
and pros
of each one: please feel free to add other metrics I missed and correct 
the ones
I commented on.  I would appreciate help on comparing approaches 2 and 3 
as I
have little knowledge of these APIs (2 seems to be addressed by 
Christopher, and
3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).

Thank you,

Oleksandr

[1] 
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/grant_table.h;h=018036e825f8f2999812cdb089f7fa2195789231;hb=HEAD#l414
[2] https://xenbits.xen.org/docs/4.9-testing/misc/grant-tables.txt
[3] 
https://xenbits.xen.org/hg/linux-2.6.18-xen.hg/file/7d14715efcac/drivers/xen/netfront
[4] 
https://xenbits.xen.org/hg/linux-2.6.18-xen.hg/file/7d14715efcac/drivers/xen/netback
[5] 
http://elixir.free-electrons.com/linux/latest/source/arch/x86/xen/mmu_pv.c#L2618
[6] 
https://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02110.html


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Paul Durrant]

> -----Original Message-----
[snip]
> Comparison for display use-case
> ===============================
> 
> 1 Number of grant references used
> 1-1 grant references: nr_pages
> 1-2 GNTTABOP_transfer: nr_pages
> 1-3 XENMEM_exchange: not an option
> 
> 2 Effect of DomU crash on Dom0 (its mapped pages)
> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
> recovered
> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
> for Dom0
> 2-3 XENMEM_exchange: not an option
> 
> 3 Security issues from sharing Dom0 pages to DomU
> 1-1 grant references: none
> 1-2 GNTTABOP_transfer: none
> 1-3 XENMEM_exchange: not an option
> 
> At the moment approach 1 with granted references seems to be a winner for
> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
> 
> Conclusion
> ==========
> 
> I would like to get some feedback from the community on which approach
> is more
> suitable for sharing large buffers and to have a clear vision on cons
> and pros
> of each one: please feel free to add other metrics I missed and correct
> the ones
> I commented on.  I would appreciate help on comparing approaches 2 and 3
> as I
> have little knowledge of these APIs (2 seems to be addressed by
> Christopher, and
> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
> 

Hi,

  I once implemented a scheme where network frontends used memory granted from backends and this hit quite a few problems:

- If domU is allowed to grant map memory from dom0, there is not currently a way to forcibly take it back (so I don't think you're quite correct in 2-1 above... but I may have missed something). Hence the domU can hold dom0's memory to ransom. (In the network case this was avoided by using grant table v2 'copy-only' grants).
- If you end up having to grant buffers which do not originate in dom0 (i.e. they were grant mapped from another domU) then this creates similar problems with one domU holding another domU's memory to ransom, even when using copy-only grants. I don’t think this would be an issue in your use-case.
- Currently the default grant table size is 32 pages and it may not take that many guests using a protocol where dom0 grants memory to domU to exhaust dom0's grant table (depending on how many grants-per-domU the protocol allows). If you're intending to grant large buffers then you may need quite a few grants (since they are per-4k-chunk) to do this, so you might run into this limit.

  Cheers,

    Paul
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Oleksandr Andrushchenko]

Hi, Paul!


On 07/31/2017 12:03 PM, Paul Durrant wrote:
>> -----Original Message-----
> [snip]
>> Comparison for display use-case
>> ===============================
>>
>> 1 Number of grant references used
>> 1-1 grant references: nr_pages
>> 1-2 GNTTABOP_transfer: nr_pages
>> 1-3 XENMEM_exchange: not an option
>>
>> 2 Effect of DomU crash on Dom0 (its mapped pages)
>> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
>> recovered
>> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
>> for Dom0
>> 2-3 XENMEM_exchange: not an option
>>
>> 3 Security issues from sharing Dom0 pages to DomU
>> 1-1 grant references: none
>> 1-2 GNTTABOP_transfer: none
>> 1-3 XENMEM_exchange: not an option
>>
>> At the moment approach 1 with granted references seems to be a winner for
>> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
>>
>> Conclusion
>> ==========
>>
>> I would like to get some feedback from the community on which approach
>> is more
>> suitable for sharing large buffers and to have a clear vision on cons
>> and pros
>> of each one: please feel free to add other metrics I missed and correct
>> the ones
>> I commented on.  I would appreciate help on comparing approaches 2 and 3
>> as I
>> have little knowledge of these APIs (2 seems to be addressed by
>> Christopher, and
>> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
>>
> Hi,
>
>    I once implemented a scheme where network frontends used memory granted from backends and this hit quite a few problems:
>
> - If domU is allowed to grant map memory from dom0, there is not currently a way to forcibly take it back (so I don't think you're quite correct in 2-1 above... but I may have missed something).
This is where I was not 100% confident, so you seem to be right here.
>   Hence the domU can hold dom0's memory to ransom.
Yes, this is the problem
>   (In the network case this was avoided by using grant table v2 'copy-only' grants).
> - If you end up having to grant buffers which do not originate in dom0 (i.e. they were grant mapped from another domU) then this creates similar problems with one domU holding another domU's memory to ransom, even when using copy-only grants. I don’t think this would be an issue in your use-case.
In our use-cases we will only share from Dom0 to DomU in case Dom0 is 
1:1 mapped
Otherwise, HW should be capable of dealing with non-contiguous memory
> - Currently the default grant table size is 32 pages and it may not take that many guests using a protocol where dom0 grants memory to domU to exhaust dom0's grant table
Yes, I know about this limitation, thank you
>   (depending on how many grants-per-domU the protocol allows). If you're intending to grant large buffers then you may need quite a few grants (since they are per-4k-chunk) to do this, so you might run into this limit.
>
>    Cheers,
>
>      Paul
Thank you for comments,
so in both cases (with grant refs or memory transfer) there is no way to 
cleanly
recover from DomU crash (either grants cannot be returned or memory goes 
to the
hypervisor, not Dom0).
Is this correct?

Thank you,
Oleksandr

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Julien Grall]

(+ Joao)

On 31/07/17 09:34, Oleksandr Andrushchenko wrote:
> Hi, all!

Hi Oleksandr,

> The aim of this mail is to highlight and discuss possible approaches to
> implementing zero copying for PV drivers. Rationale behind this is that
> there
> are use-cases when drivers operate with big shared buffers, e.g.
> display, when
> memory copying from front’s buffer into back’s one may significantly hit
> performance of the system (for example, for para-virtual display running
> at full
> HD resolution at 60Hz it is approximately 475MB/sec).
>
> Assumptions (which actually fit ARM platforms, but can be extended to other
> platforms as well): Dom0 is a 1:1 mapped privileged domain, runs backend
> driver/software DomU is an unprivileged domain without 1:1 memory
> mapping, runs
> frontend driver

I would rather avoid to stick with this assumption on ARM. This was only 
meant to be a workaround for platform without IOMMU (see [1]) and we 
will get into trouble when using IOMMU.

For instance, there are no requirement to have the IOMMU supporting as 
many as address bits than the processor. So 1:1 mapping here will not be 
an option.

>
> Buffer origin: while implementing zero copying the buffer allocation can
> happen
> either on DomU’s end or Dom0’s one depending on the use-case and HW
> capabilities/availability: When DomU allocates: It cannot guarantee
> physical
> memory continuity of the buffers allocated Dom0’s HW *can* handle
> non-contiguous
> memory buffers allocated by DomU for memory operations (DMA, for
> example), e.g.
> either with IOMMU help or by any other means (HW block’s own MMU).  When
> Dom0
> allocates as it is mapped 1:1 it can allocate physically contiguous memory
> Dom0’s HW *cannot* handle non-contiguous memory buffers allocated by
> DomU for
> memory operations by any means.

I am not sure to follow this. How zero copy is related to 1:1 mapping? 
Is it because you have hardware that does not support scatter/gather IO 
or IOMMU?

>
> 1 Sharing with granted references
> ==================================
>
> 1-1 Buffer allocated @DomU
> --------------------------
> @DomU
>     alloc_xenballooned_pages(nr_pages, pages);
>     cur_ref = gnttab_claim_grant_reference(&priv_gref_head);
>     gnttab_grant_foreign_access_ref(cur_ref, otherend_id, ...);
>     <pass grant_ref_t[] to Dom0>
> @Dom0
>     alloc_xenballooned_pages(nr_pages, pages);
>     gnttab_set_map_op(&map_ops[i], addr, GNTMAP_host_map |
> GNTMAP_device_map,
>         grefs[i], otherend_id);
>     gnttab_map_refs(map_ops, NULL, pages, nr_pages);
>
> 1-2 Buffer allocated @Dom0
> --------------------------
> @Dom0
>     <the code below is equivalent to xen_alloc_ballooned_pages without
>      PV MMU support as seen in the balloon driver, the difference is that
>      pages are explicitly allocated to be used for DMA>
>     dma_alloc_wc(dev, size, &dev_addr, GFP_KERNEL | __GFP_NOWARN);
>     HYPERVISOR_memory_op(XENMEM_populate_physmap, &reservation);
>     cur_ref = gnttab_claim_grant_reference(&priv_gref_head);
>     gnttab_grant_foreign_access_ref(cur_ref, otherend_id, ...);
>     <pass grant_ref_t[] to DomU>
> @Dom0
>     alloc_xenballooned_pages(nr_pages, pages);
>     gnttab_set_map_op(&map_ops[i], addr, GNTMAP_host_map, grefs[i],
> otherend_id);
>     gnttab_map_refs(map_ops, NULL, pages, nr_pages);
>
> 2 Sharing with page transfers (GNTTABOP_transfer)
> ==================================================
> FIXME: This use-case seems to be only needed while allocating physically
> contiguous buffers at Dom0. For the reverse path 1-1 method can be used.
>
> This approach relies on GNTTABOP_transfer API: “transfer <frame> to a
> foreign
> domain. The foreign domain has previously registered its interest in the
> transfer via <domid, ref>”, for full documentation see [1]. The process of
> transferring pages is explained by Christopher Clark at [2] and is
> available as
> implementation at [3], [4]. The relevant logic is in:
> xen/common/grant_table.c :
> gnttab_transfer.
>
> Basic workflow explained to me by Christopher:
> - The mfn starts as owned by the sending domain, and that domain removes
> any
>   mappings of it from its page tables. Xen will enforce that the
> reference count
> must be low enough for the transfer to succeed.
> - The receiving domain indicates interest for receiving a page by
> writing an
>   entry in its grant table.
> - You'll need to communicate the grant ref from the receiver to the
> sender (eg.
>   via xenstore or another existing channel)
> - The sending domain invokes the hypercall, with the grant ref from the
>   receiving domain.
> - The sending domain notifies the receiving domain somehow that the
> transfer has
>   completed. (eg. send an event or via xenstore)
> - Once the transfer has completed, the receiving domain will need to map
> the
>   newly assigned page.
> - Note: For the transfer, the receiving domain must have enough headroom to
>   receive the new page, which means it must not have allocated all of
> its memory
> quota already prior to the transfer. Typically this can be ensured by
> freeing
> enough memory back to Xen before writing the grant ref.
>
> 3 Sharing with page exchange (XENMEM_exchange)
> ==============================================
>
> This API was pointed to me by Stefano Stabellini as one of the possible
> ways to
> achieve zero copying and share physically contiguous buffers. It is used
> by x86
> SWIOTLB code (xen_create_contiguous_region, [5]), but as per my
> understanding
> this API cannot be used on ARM as of now [6].  Conclusion: not an option
> for ARM
> at the moment
>
> Comparison for display use-case
> ===============================
>
> 1 Number of grant references used
> 1-1 grant references: nr_pages
> 1-2 GNTTABOP_transfer: nr_pages
> 1-3 XENMEM_exchange: not an option
>
> 2 Effect of DomU crash on Dom0 (its mapped pages)
> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
> recovered
> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
> for Dom0
> 2-3 XENMEM_exchange: not an option
>
> 3 Security issues from sharing Dom0 pages to DomU
> 1-1 grant references: none
> 1-2 GNTTABOP_transfer: none
> 1-3 XENMEM_exchange: not an option
>
> At the moment approach 1 with granted references seems to be a winner for
> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
>
> Conclusion
> ==========
>
> I would like to get some feedback from the community on which approach
> is more
> suitable for sharing large buffers and to have a clear vision on cons
> and pros
> of each one: please feel free to add other metrics I missed and correct
> the ones
> I commented on.  I would appreciate help on comparing approaches 2 and 3
> as I
> have little knowledge of these APIs (2 seems to be addressed by
> Christopher, and
> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
>
> Thank you,
>
> Oleksandr
>
> [1]
> https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/grant_table.h;h=018036e825f8f2999812cdb089f7fa2195789231;hb=HEAD#l414
>
> [2] https://xenbits.xen.org/docs/4.9-testing/misc/grant-tables.txt
> [3]
> https://xenbits.xen.org/hg/linux-2.6.18-xen.hg/file/7d14715efcac/drivers/xen/netfront
>
> [4]
> https://xenbits.xen.org/hg/linux-2.6.18-xen.hg/file/7d14715efcac/drivers/xen/netback
>
> [5]
> http://elixir.free-electrons.com/linux/latest/source/arch/x86/xen/mmu_pv.c#L2618
>
> [6]
> https://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02110.html
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Andrew Cooper]

On 31/07/17 10:03, Paul Durrant wrote:
>> -----Original Message-----
> [snip]
>> Comparison for display use-case
>> ===============================
>>
>> 1 Number of grant references used
>> 1-1 grant references: nr_pages
>> 1-2 GNTTABOP_transfer: nr_pages
>> 1-3 XENMEM_exchange: not an option
>>
>> 2 Effect of DomU crash on Dom0 (its mapped pages)
>> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
>> recovered
>> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
>> for Dom0
>> 2-3 XENMEM_exchange: not an option
>>
>> 3 Security issues from sharing Dom0 pages to DomU
>> 1-1 grant references: none
>> 1-2 GNTTABOP_transfer: none
>> 1-3 XENMEM_exchange: not an option
>>
>> At the moment approach 1 with granted references seems to be a winner for
>> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
>>
>> Conclusion
>> ==========
>>
>> I would like to get some feedback from the community on which approach
>> is more
>> suitable for sharing large buffers and to have a clear vision on cons
>> and pros
>> of each one: please feel free to add other metrics I missed and correct
>> the ones
>> I commented on.  I would appreciate help on comparing approaches 2 and 3
>> as I
>> have little knowledge of these APIs (2 seems to be addressed by
>> Christopher, and
>> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
>>
> Hi,
>
>   I once implemented a scheme where network frontends used memory granted from backends and this hit quite a few problems:
>
> - If domU is allowed to grant map memory from dom0, there is not currently a way to forcibly take it back (so I don't think you're quite correct in 2-1 above... but I may have missed something). Hence the domU can hold dom0's memory to ransom. (In the network case this was avoided by using grant table v2 'copy-only' grants).
> - If you end up having to grant buffers which do not originate in dom0 (i.e. they were grant mapped from another domU) then this creates similar problems with one domU holding another domU's memory to ransom, even when using copy-only grants. I don’t think this would be an issue in your use-case.
> - Currently the default grant table size is 32 pages and it may not take that many guests using a protocol where dom0 grants memory to domU to exhaust dom0's grant table (depending on how many grants-per-domU the protocol allows). If you're intending to grant large buffers then you may need quite a few grants (since they are per-4k-chunk) to do this, so you might run into this limit.

To follow up on what Paul said, google for XenServer Receive Side Copy. 
The issue is that it looks very attractive (from an offloading things
out of dom0 perspective), and does work at small scale, but the failure
cases are far more tricky than we imagined, and you run dom0 out of
grants very quickly, which puts a hard upper bound on scalability.  It
is high on the list of "worst mistakes we put into production".

It is not safe at all for dom0 to grant frames to domU without dom0
having a mechanism to revoke the grant.  (There was work looking into
this in the past, but it suffered from a lack of free time.)

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Oleksandr Andrushchenko]

On 07/31/2017 12:31 PM, Andrew Cooper wrote:
> On 31/07/17 10:03, Paul Durrant wrote:
>>> -----Original Message-----
>> [snip]
>>> Comparison for display use-case
>>> ===============================
>>>
>>> 1 Number of grant references used
>>> 1-1 grant references: nr_pages
>>> 1-2 GNTTABOP_transfer: nr_pages
>>> 1-3 XENMEM_exchange: not an option
>>>
>>> 2 Effect of DomU crash on Dom0 (its mapped pages)
>>> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
>>> recovered
>>> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
>>> for Dom0
>>> 2-3 XENMEM_exchange: not an option
>>>
>>> 3 Security issues from sharing Dom0 pages to DomU
>>> 1-1 grant references: none
>>> 1-2 GNTTABOP_transfer: none
>>> 1-3 XENMEM_exchange: not an option
>>>
>>> At the moment approach 1 with granted references seems to be a winner for
>>> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
>>>
>>> Conclusion
>>> ==========
>>>
>>> I would like to get some feedback from the community on which approach
>>> is more
>>> suitable for sharing large buffers and to have a clear vision on cons
>>> and pros
>>> of each one: please feel free to add other metrics I missed and correct
>>> the ones
>>> I commented on.  I would appreciate help on comparing approaches 2 and 3
>>> as I
>>> have little knowledge of these APIs (2 seems to be addressed by
>>> Christopher, and
>>> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
>>>
>> Hi,
>>
>>    I once implemented a scheme where network frontends used memory granted from backends and this hit quite a few problems:
>>
>> - If domU is allowed to grant map memory from dom0, there is not currently a way to forcibly take it back (so I don't think you're quite correct in 2-1 above... but I may have missed something). Hence the domU can hold dom0's memory to ransom. (In the network case this was avoided by using grant table v2 'copy-only' grants).
>> - If you end up having to grant buffers which do not originate in dom0 (i.e. they were grant mapped from another domU) then this creates similar problems with one domU holding another domU's memory to ransom, even when using copy-only grants. I don’t think this would be an issue in your use-case.
>> - Currently the default grant table size is 32 pages and it may not take that many guests using a protocol where dom0 grants memory to domU to exhaust dom0's grant table (depending on how many grants-per-domU the protocol allows). If you're intending to grant large buffers then you may need quite a few grants (since they are per-4k-chunk) to do this, so you might run into this limit.
> To follow up on what Paul said, google for XenServer Receive Side Copy.
I will, thank you
> The issue is that it looks very attractive (from an offloading things
> out of dom0 perspective), and does work at small scale, but the failure
> cases are far more tricky than we imagined, and you run dom0 out of
> grants very quickly, which puts a hard upper bound on scalability.  It
> is high on the list of "worst mistakes we put into production".
>
> It is not safe at all for dom0 to grant frames to domU without dom0
> having a mechanism to revoke the grant.
Other than that, are there any other concerns, e.g. from
security POV?
>   (There was work looking into
> this in the past, but it suffered from a lack of free time.)
>
> ~Andrew
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
Thank you,
Oleksandr

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Oleksandr Andrushchenko]

Hi, Julien!


On 07/31/2017 12:24 PM, Julien Grall wrote:
> (+ Joao)
>
> On 31/07/17 09:34, Oleksandr Andrushchenko wrote:
>> Hi, all!
>
> Hi Oleksandr,
>
>> The aim of this mail is to highlight and discuss possible approaches to
>> implementing zero copying for PV drivers. Rationale behind this is that
>> there
>> are use-cases when drivers operate with big shared buffers, e.g.
>> display, when
>> memory copying from front’s buffer into back’s one may significantly hit
>> performance of the system (for example, for para-virtual display running
>> at full
>> HD resolution at 60Hz it is approximately 475MB/sec).
>>
>> Assumptions (which actually fit ARM platforms, but can be extended to 
>> other
>> platforms as well): Dom0 is a 1:1 mapped privileged domain, runs backend
>> driver/software DomU is an unprivileged domain without 1:1 memory
>> mapping, runs
>> frontend driver
>
> I would rather avoid to stick with this assumption on ARM. This was 
> only meant to be a workaround for platform without IOMMU (see [1]) and 
> we will get into trouble when using IOMMU.
You are correct, thank you
>
> For instance, there are no requirement to have the IOMMU supporting as 
> many as address bits than the processor. So 1:1 mapping here will not 
> be an option.
>
>>
>> Buffer origin: while implementing zero copying the buffer allocation can
>> happen
>> either on DomU’s end or Dom0’s one depending on the use-case and HW
>> capabilities/availability: When DomU allocates: It cannot guarantee
>> physical
>> memory continuity of the buffers allocated Dom0’s HW *can* handle
>> non-contiguous
>> memory buffers allocated by DomU for memory operations (DMA, for
>> example), e.g.
>> either with IOMMU help or by any other means (HW block’s own MMU).  When
>> Dom0
>> allocates as it is mapped 1:1 it can allocate physically contiguous 
>> memory
>> Dom0’s HW *cannot* handle non-contiguous memory buffers allocated by
>> DomU for
>> memory operations by any means.
>
> I am not sure to follow this. How zero copy is related to 1:1 mapping? 
> Is it because you have hardware that does not support scatter/gather 
> IO or IOMMU?
yes, you got it right
>
>>
>> 1 Sharing with granted references
>> ==================================
>>
>> 1-1 Buffer allocated @DomU
>> --------------------------
>> @DomU
>>     alloc_xenballooned_pages(nr_pages, pages);
>>     cur_ref = gnttab_claim_grant_reference(&priv_gref_head);
>>     gnttab_grant_foreign_access_ref(cur_ref, otherend_id, ...);
>>     <pass grant_ref_t[] to Dom0>
>> @Dom0
>>     alloc_xenballooned_pages(nr_pages, pages);
>>     gnttab_set_map_op(&map_ops[i], addr, GNTMAP_host_map |
>> GNTMAP_device_map,
>>         grefs[i], otherend_id);
>>     gnttab_map_refs(map_ops, NULL, pages, nr_pages);
>>
>> 1-2 Buffer allocated @Dom0
>> --------------------------
>> @Dom0
>>     <the code below is equivalent to xen_alloc_ballooned_pages without
>>      PV MMU support as seen in the balloon driver, the difference is 
>> that
>>      pages are explicitly allocated to be used for DMA>
>>     dma_alloc_wc(dev, size, &dev_addr, GFP_KERNEL | __GFP_NOWARN);
>>     HYPERVISOR_memory_op(XENMEM_populate_physmap, &reservation);
>>     cur_ref = gnttab_claim_grant_reference(&priv_gref_head);
>>     gnttab_grant_foreign_access_ref(cur_ref, otherend_id, ...);
>>     <pass grant_ref_t[] to DomU>
>> @Dom0
>>     alloc_xenballooned_pages(nr_pages, pages);
>>     gnttab_set_map_op(&map_ops[i], addr, GNTMAP_host_map, grefs[i],
>> otherend_id);
>>     gnttab_map_refs(map_ops, NULL, pages, nr_pages);
>>
>> 2 Sharing with page transfers (GNTTABOP_transfer)
>> ==================================================
>> FIXME: This use-case seems to be only needed while allocating physically
>> contiguous buffers at Dom0. For the reverse path 1-1 method can be used.
>>
>> This approach relies on GNTTABOP_transfer API: “transfer <frame> to a
>> foreign
>> domain. The foreign domain has previously registered its interest in the
>> transfer via <domid, ref>”, for full documentation see [1]. The 
>> process of
>> transferring pages is explained by Christopher Clark at [2] and is
>> available as
>> implementation at [3], [4]. The relevant logic is in:
>> xen/common/grant_table.c :
>> gnttab_transfer.
>>
>> Basic workflow explained to me by Christopher:
>> - The mfn starts as owned by the sending domain, and that domain removes
>> any
>>   mappings of it from its page tables. Xen will enforce that the
>> reference count
>> must be low enough for the transfer to succeed.
>> - The receiving domain indicates interest for receiving a page by
>> writing an
>>   entry in its grant table.
>> - You'll need to communicate the grant ref from the receiver to the
>> sender (eg.
>>   via xenstore or another existing channel)
>> - The sending domain invokes the hypercall, with the grant ref from the
>>   receiving domain.
>> - The sending domain notifies the receiving domain somehow that the
>> transfer has
>>   completed. (eg. send an event or via xenstore)
>> - Once the transfer has completed, the receiving domain will need to map
>> the
>>   newly assigned page.
>> - Note: For the transfer, the receiving domain must have enough 
>> headroom to
>>   receive the new page, which means it must not have allocated all of
>> its memory
>> quota already prior to the transfer. Typically this can be ensured by
>> freeing
>> enough memory back to Xen before writing the grant ref.
>>
>> 3 Sharing with page exchange (XENMEM_exchange)
>> ==============================================
>>
>> This API was pointed to me by Stefano Stabellini as one of the possible
>> ways to
>> achieve zero copying and share physically contiguous buffers. It is used
>> by x86
>> SWIOTLB code (xen_create_contiguous_region, [5]), but as per my
>> understanding
>> this API cannot be used on ARM as of now [6].  Conclusion: not an option
>> for ARM
>> at the moment
>>
>> Comparison for display use-case
>> ===============================
>>
>> 1 Number of grant references used
>> 1-1 grant references: nr_pages
>> 1-2 GNTTABOP_transfer: nr_pages
>> 1-3 XENMEM_exchange: not an option
>>
>> 2 Effect of DomU crash on Dom0 (its mapped pages)
>> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
>> recovered
>> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
>> for Dom0
>> 2-3 XENMEM_exchange: not an option
>>
>> 3 Security issues from sharing Dom0 pages to DomU
>> 1-1 grant references: none
>> 1-2 GNTTABOP_transfer: none
>> 1-3 XENMEM_exchange: not an option
>>
>> At the moment approach 1 with granted references seems to be a winner 
>> for
>> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
>>
>> Conclusion
>> ==========
>>
>> I would like to get some feedback from the community on which approach
>> is more
>> suitable for sharing large buffers and to have a clear vision on cons
>> and pros
>> of each one: please feel free to add other metrics I missed and correct
>> the ones
>> I commented on.  I would appreciate help on comparing approaches 2 and 3
>> as I
>> have little knowledge of these APIs (2 seems to be addressed by
>> Christopher, and
>> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
>>
>> Thank you,
>>
>> Oleksandr
>>
>> [1]
>> https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/grant_table.h;h=018036e825f8f2999812cdb089f7fa2195789231;hb=HEAD#l414 
>>
>>
>> [2] https://xenbits.xen.org/docs/4.9-testing/misc/grant-tables.txt
>> [3]
>> https://xenbits.xen.org/hg/linux-2.6.18-xen.hg/file/7d14715efcac/drivers/xen/netfront 
>>
>>
>> [4]
>> https://xenbits.xen.org/hg/linux-2.6.18-xen.hg/file/7d14715efcac/drivers/xen/netback 
>>
>>
>> [5]
>> http://elixir.free-electrons.com/linux/latest/source/arch/x86/xen/mmu_pv.c#L2618 
>>
>>
>> [6]
>> https://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02110.html 
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>
Thank you,
Oleksandr

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Julien Grall]

On 31/07/17 10:46, Oleksandr Andrushchenko wrote:
> Hi, Julien!
>
>
> On 07/31/2017 12:24 PM, Julien Grall wrote:
>> (+ Joao)
>>
>> On 31/07/17 09:34, Oleksandr Andrushchenko wrote:
>>> Hi, all!
>>
>> Hi Oleksandr,
>>
>>> The aim of this mail is to highlight and discuss possible approaches to
>>> implementing zero copying for PV drivers. Rationale behind this is that
>>> there
>>> are use-cases when drivers operate with big shared buffers, e.g.
>>> display, when
>>> memory copying from front’s buffer into back’s one may significantly hit
>>> performance of the system (for example, for para-virtual display running
>>> at full
>>> HD resolution at 60Hz it is approximately 475MB/sec).
>>>
>>> Assumptions (which actually fit ARM platforms, but can be extended to
>>> other
>>> platforms as well): Dom0 is a 1:1 mapped privileged domain, runs backend
>>> driver/software DomU is an unprivileged domain without 1:1 memory
>>> mapping, runs
>>> frontend driver
>>
>> I would rather avoid to stick with this assumption on ARM. This was
>> only meant to be a workaround for platform without IOMMU (see [1]) and
>> we will get into trouble when using IOMMU.
> You are correct, thank you
>>
>> For instance, there are no requirement to have the IOMMU supporting as
>> many as address bits than the processor. So 1:1 mapping here will not
>> be an option.
>>
>>>
>>> Buffer origin: while implementing zero copying the buffer allocation can
>>> happen
>>> either on DomU’s end or Dom0’s one depending on the use-case and HW
>>> capabilities/availability: When DomU allocates: It cannot guarantee
>>> physical
>>> memory continuity of the buffers allocated Dom0’s HW *can* handle
>>> non-contiguous
>>> memory buffers allocated by DomU for memory operations (DMA, for
>>> example), e.g.
>>> either with IOMMU help or by any other means (HW block’s own MMU).  When
>>> Dom0
>>> allocates as it is mapped 1:1 it can allocate physically contiguous
>>> memory
>>> Dom0’s HW *cannot* handle non-contiguous memory buffers allocated by
>>> DomU for
>>> memory operations by any means.
>>
>> I am not sure to follow this. How zero copy is related to 1:1 mapping?
>> Is it because you have hardware that does not support scatter/gather
>> IO or IOMMU?
> yes, you got it right

Do you have any example of hardware? What are the performance you 
require with them?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Oleksandr Andrushchenko]

On 07/31/2017 12:47 PM, Julien Grall wrote:
>
>
> On 31/07/17 10:46, Oleksandr Andrushchenko wrote:
>> Hi, Julien!
>>
>>
>> On 07/31/2017 12:24 PM, Julien Grall wrote:
>>> (+ Joao)
>>>
>>> On 31/07/17 09:34, Oleksandr Andrushchenko wrote:
>>>> Hi, all!
>>>
>>> Hi Oleksandr,
>>>
>>>> The aim of this mail is to highlight and discuss possible 
>>>> approaches to
>>>> implementing zero copying for PV drivers. Rationale behind this is 
>>>> that
>>>> there
>>>> are use-cases when drivers operate with big shared buffers, e.g.
>>>> display, when
>>>> memory copying from front’s buffer into back’s one may 
>>>> significantly hit
>>>> performance of the system (for example, for para-virtual display 
>>>> running
>>>> at full
>>>> HD resolution at 60Hz it is approximately 475MB/sec).
>>>>
>>>> Assumptions (which actually fit ARM platforms, but can be extended to
>>>> other
>>>> platforms as well): Dom0 is a 1:1 mapped privileged domain, runs 
>>>> backend
>>>> driver/software DomU is an unprivileged domain without 1:1 memory
>>>> mapping, runs
>>>> frontend driver
>>>
>>> I would rather avoid to stick with this assumption on ARM. This was
>>> only meant to be a workaround for platform without IOMMU (see [1]) and
>>> we will get into trouble when using IOMMU.
>> You are correct, thank you
>>>
>>> For instance, there are no requirement to have the IOMMU supporting as
>>> many as address bits than the processor. So 1:1 mapping here will not
>>> be an option.
>>>
>>>>
>>>> Buffer origin: while implementing zero copying the buffer 
>>>> allocation can
>>>> happen
>>>> either on DomU’s end or Dom0’s one depending on the use-case and HW
>>>> capabilities/availability: When DomU allocates: It cannot guarantee
>>>> physical
>>>> memory continuity of the buffers allocated Dom0’s HW *can* handle
>>>> non-contiguous
>>>> memory buffers allocated by DomU for memory operations (DMA, for
>>>> example), e.g.
>>>> either with IOMMU help or by any other means (HW block’s own MMU).  
>>>> When
>>>> Dom0
>>>> allocates as it is mapped 1:1 it can allocate physically contiguous
>>>> memory
>>>> Dom0’s HW *cannot* handle non-contiguous memory buffers allocated by
>>>> DomU for
>>>> memory operations by any means.
>>>
>>> I am not sure to follow this. How zero copy is related to 1:1 mapping?
>>> Is it because you have hardware that does not support scatter/gather
>>> IO or IOMMU?
>> yes, you got it right
>
> Do you have any example of hardware? What are the performance you 
> require with them?
>
Currently our target is Renesas R-Car Gen3
At the moment I don't have clean requirements, but
ideally, PV driver introduces 0% performance drop
Some time soon I will have numbers on running display/GPU
with and without zero-copy - will keep updated
> Cheers,
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Julien Grall]

On 31/07/17 10:52, Oleksandr Andrushchenko wrote:
> On 07/31/2017 12:47 PM, Julien Grall wrote:
>> On 31/07/17 10:46, Oleksandr Andrushchenko wrote:
>> Do you have any example of hardware? What are the performance you
>> require with them?
>>
> Currently our target is Renesas R-Car Gen3
> At the moment I don't have clean requirements, but
> ideally, PV driver introduces 0% performance drop
> Some time soon I will have numbers on running display/GPU
> with and without zero-copy - will keep updated

PV driver with 0% performance drop sounds a stretch target. But this is 
does not answer to my question. Do you have any hardware that does not 
support scatter/gather or not protected by an IOMMU that will be 
interfaced with PV drivers?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Oleksandr Andrushchenko]

On 07/31/2017 01:04 PM, Julien Grall wrote:
>
>
> On 31/07/17 10:52, Oleksandr Andrushchenko wrote:
>> On 07/31/2017 12:47 PM, Julien Grall wrote:
>>> On 31/07/17 10:46, Oleksandr Andrushchenko wrote:
>>> Do you have any example of hardware? What are the performance you
>>> require with them?
>>>
>> Currently our target is Renesas R-Car Gen3
>> At the moment I don't have clean requirements, but
>> ideally, PV driver introduces 0% performance drop
>> Some time soon I will have numbers on running display/GPU
>> with and without zero-copy - will keep updated
>
> PV driver with 0% performance drop sounds a stretch target.
It is, but we should always get as close as possible
> But this is does not answer to my question. Do you have any hardware 
> that does not support scatter/gather
AFAIK display driver which doesn't support scatter-gather on our platform
(BSP based on 4.9 kernel, rcar_du uses DRM CMA - DRM contiguous memory 
allocator)
Anyways, for pages above 4GB even scatter-gather will not help
devices with 32-bit DMA
> or not protected by an IOMMU that will be interfaced with PV drivers?
>
As per my understanding, IOMMU is solely owned by the hypervisor now
and there is no API to tell Xen from Dom0 to setup IOMMU for such
a buffer (pages), so display HW can do DMA with that buffer.
Thus, Dom0 has no means to do that work and make PV driver produce
buffers which can be used by the real HW driver without bounce buffering.
> Cheers,
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Joao Martins]

Hey Oleksandr,

On 07/31/2017 09:34 AM, Oleksandr Andrushchenko wrote:
> Hi, all!
> 
[snip]
> 
> Comparison for display use-case
> ===============================
> 
> 1 Number of grant references used
> 1-1 grant references: nr_pages
> 1-2 GNTTABOP_transfer: nr_pages
> 1-3 XENMEM_exchange: not an option
> 
> 2 Effect of DomU crash on Dom0 (its mapped pages)
> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully 
> recovered
> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost 
> for Dom0
> 2-3 XENMEM_exchange: not an option
> 
> 3 Security issues from sharing Dom0 pages to DomU
> 1-1 grant references: none
> 1-2 GNTTABOP_transfer: none
> 1-3 XENMEM_exchange: not an option
> 
> At the moment approach 1 with granted references seems to be a winner for
> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
> 
> Conclusion
> ==========
> 
> I would like to get some feedback from the community on which approach 
> is more
> suitable for sharing large buffers and to have a clear vision on cons 
> and pros
> of each one: please feel free to add other metrics I missed and correct 
> the ones
> I commented on.  I would appreciate help on comparing approaches 2 and 3 
> as I
> have little knowledge of these APIs (2 seems to be addressed by 
> Christopher, and
> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).

Depending on your performance/memory requirements - there could be another
option which is to keep the guest mapped on Domain-0 (what was discussed with
Zerogrant session[0][1] that will be formally proposed in the next month or so).
But that would only solve the grant maps/unmaps/copies done on Domain-0 (given
the numbers you pasted a bit ago, you might not really need to go to such extents)

[0]
http://schd.ws/hosted_files/xendeveloperanddesignsummit2017/05/zerogrant_spec.pdf
[1]
http://schd.ws/hosted_files/xendeveloperanddesignsummit2017/a8/zerogrant_slides.pdf

For the buffers allocated on Dom0 and safely grant buffers from Dom0 to DomU
(which I am not so sure it is possible today :(), maybe a "contract" from DomU
provide a set of transferable pages that Dom0 holds on for each Dom-0 gref
provided to the guest (and assuming this is only a handful couple of guests as
grant table is not that big). IIUC, From what you pasted above on "Buffer
allocated @Dom0" sounds like Domain-0 could quickly ran out of pages/OOM (and
grants), if you're guest is misbehaving/buggy or malicious; *also* domain-0
grant table is a rather finite/small resource (even though you can override the
number of frames in the arguments).

Joao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Joao Martins]

On 07/31/2017 11:37 AM, Oleksandr Andrushchenko wrote:
> On 07/31/2017 01:04 PM, Julien Grall wrote:
>> On 31/07/17 10:52, Oleksandr Andrushchenko wrote:
>>> On 07/31/2017 12:47 PM, Julien Grall wrote:
>>>> On 31/07/17 10:46, Oleksandr Andrushchenko wrote:
>>>> Do you have any example of hardware? What are the performance you
>>>> require with them?
>>>>
>>> Currently our target is Renesas R-Car Gen3
>>> At the moment I don't have clean requirements, but
>>> ideally, PV driver introduces 0% performance drop
>>> Some time soon I will have numbers on running display/GPU
>>> with and without zero-copy - will keep updated
>>
>> PV driver with 0% performance drop sounds a stretch target.
> It is, but we should always get as close as possible
>> But this is does not answer to my question. Do you have any hardware 
>> that does not support scatter/gather
> AFAIK display driver which doesn't support scatter-gather on our platform
> (BSP based on 4.9 kernel, rcar_du uses DRM CMA - DRM contiguous memory 
> allocator)
> Anyways, for pages above 4GB even scatter-gather will not help
> devices with 32-bit DMA
>> or not protected by an IOMMU that will be interfaced with PV drivers?
>>
> As per my understanding, IOMMU is solely owned by the hypervisor now
> and there is no API to tell Xen from Dom0 to setup IOMMU for such
> a buffer (pages), so display HW can do DMA with that buffer.
> Thus, Dom0 has no means to do that work and make PV driver produce
> buffers which can be used by the real HW driver without bounce buffering.

Sounds like it is addressed by PV-IOMMU[0] which I think it will be resurrected
in the coming months as per the design session last hackaton.

[0] https://schd.ws/hosted_files/xendeveloperanddesignsummit2017/91/PV-IOMMU.txt

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Oleksandr Andrushchenko]

Hi, Joao!

On 07/31/2017 02:03 PM, Joao Martins wrote:
> Hey Oleksandr,
>
> On 07/31/2017 09:34 AM, Oleksandr Andrushchenko wrote:
>> Hi, all!
>>
> [snip]
>> Comparison for display use-case
>> ===============================
>>
>> 1 Number of grant references used
>> 1-1 grant references: nr_pages
>> 1-2 GNTTABOP_transfer: nr_pages
>> 1-3 XENMEM_exchange: not an option
>>
>> 2 Effect of DomU crash on Dom0 (its mapped pages)
>> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
>> recovered
>> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
>> for Dom0
>> 2-3 XENMEM_exchange: not an option
>>
>> 3 Security issues from sharing Dom0 pages to DomU
>> 1-1 grant references: none
>> 1-2 GNTTABOP_transfer: none
>> 1-3 XENMEM_exchange: not an option
>>
>> At the moment approach 1 with granted references seems to be a winner for
>> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
>>
>> Conclusion
>> ==========
>>
>> I would like to get some feedback from the community on which approach
>> is more
>> suitable for sharing large buffers and to have a clear vision on cons
>> and pros
>> of each one: please feel free to add other metrics I missed and correct
>> the ones
>> I commented on.  I would appreciate help on comparing approaches 2 and 3
>> as I
>> have little knowledge of these APIs (2 seems to be addressed by
>> Christopher, and
>> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
> Depending on your performance/memory requirements - there could be another
> option which is to keep the guest mapped on Domain-0 (what was discussed with
> Zerogrant session[0][1] that will be formally proposed in the next month or so).
Unfortunately I missed that session during the Summit
due to overlapping sessions
> But that would only solve the grant maps/unmaps/copies done on Domain-0 (given
> the numbers you pasted a bit ago, you might not really need to go to such extents)
>
> [0]
> http://schd.ws/hosted_files/xendeveloperanddesignsummit2017/05/zerogrant_spec.pdf
> [1]
> http://schd.ws/hosted_files/xendeveloperanddesignsummit2017/a8/zerogrant_slides.pdf
I will read these, thank you for the links
> For the buffers allocated on Dom0 and safely grant buffers from Dom0 to DomU
> (which I am not so sure it is possible today :()
We have this working in our setup for display (we have implemented
z-copy with grant references already)
> , maybe a "contract" from DomU
> provide a set of transferable pages that Dom0 holds on for each Dom-0 gref
> provided to the guest (and assuming this is only a handful couple of guests as
> grant table is not that big).
It is an option
>   IIUC, From what you pasted above on "Buffer
> allocated @Dom0" sounds like Domain-0 could quickly ran out of pages/OOM (and
> grants), if you're guest is misbehaving/buggy or malicious; *also* domain-0
> grant table is a rather finite/small resource (even though you can override the
> number of frames in the arguments).
Well, you are right. But, we are focusing on embedded appliances,
so those systems we use are not that "dynamic" with that respect.
Namely: we have fixed number of domains and their functionality
is well known, so we can do rather precise assumption on resource
usage.
> Joao
Thank you,
Oleksandr

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Joao Martins]

On 07/31/2017 12:41 PM, Oleksandr Andrushchenko wrote:
> Hi, Joao!
> 
> On 07/31/2017 02:03 PM, Joao Martins wrote:
>> Hey Oleksandr,
>>
>> On 07/31/2017 09:34 AM, Oleksandr Andrushchenko wrote:
>>> Hi, all!
>>>
>> [snip]
>>> Comparison for display use-case
>>> ===============================
>>>
>>> 1 Number of grant references used
>>> 1-1 grant references: nr_pages
>>> 1-2 GNTTABOP_transfer: nr_pages
>>> 1-3 XENMEM_exchange: not an option
>>>
>>> 2 Effect of DomU crash on Dom0 (its mapped pages)
>>> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
>>> recovered
>>> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
>>> for Dom0
>>> 2-3 XENMEM_exchange: not an option
>>>
>>> 3 Security issues from sharing Dom0 pages to DomU
>>> 1-1 grant references: none
>>> 1-2 GNTTABOP_transfer: none
>>> 1-3 XENMEM_exchange: not an option
>>>
>>> At the moment approach 1 with granted references seems to be a winner for
>>> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
>>>
>>> Conclusion
>>> ==========
>>>
>>> I would like to get some feedback from the community on which approach
>>> is more
>>> suitable for sharing large buffers and to have a clear vision on cons
>>> and pros
>>> of each one: please feel free to add other metrics I missed and correct
>>> the ones
>>> I commented on.  I would appreciate help on comparing approaches 2 and 3
>>> as I
>>> have little knowledge of these APIs (2 seems to be addressed by
>>> Christopher, and
>>> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
>> Depending on your performance/memory requirements - there could be another
>> option which is to keep the guest mapped on Domain-0 (what was discussed with
>> Zerogrant session[0][1] that will be formally proposed in the next month or so).
> Unfortunately I missed that session during the Summit
> due to overlapping sessions

Hmm - Zerocopy Rx (Dom0 -> DomU) would indeed be an interesting topic to bring up.

>> But that would only solve the grant maps/unmaps/copies done on Domain-0 (given
>> the numbers you pasted a bit ago, you might not really need to go to such extents)
>>
>> [0]
>> http://schd.ws/hosted_files/xendeveloperanddesignsummit2017/05/zerogrant_spec.pdf
>> [1]
>> http://schd.ws/hosted_files/xendeveloperanddesignsummit2017/a8/zerogrant_slides.pdf
> I will read these, thank you for the links
>> For the buffers allocated on Dom0 and safely grant buffers from Dom0 to DomU
>> (which I am not so sure it is possible today :()
> We have this working in our setup for display (we have implemented
> z-copy with grant references already)

Allow me to clarify :) I meant "possible to do it in a safely manner", IOW,
regarding what I mentioned below in following paragraphs. But your answer below
clarifies on that aspect.

>> , maybe a "contract" from DomU
>> provide a set of transferable pages that Dom0 holds on for each Dom-0 gref
>> provided to the guest (and assuming this is only a handful couple of guests as
>> grant table is not that big).
> It is an option
>>
>>   IIUC, From what you pasted above on "Buffer
>> allocated @Dom0" sounds like Domain-0 could quickly ran out of pages/OOM (and
>> grants), if you're guest is misbehaving/buggy or malicious; *also* domain-0
>> grant table is a rather finite/small resource (even though you can override the
>> number of frames in the arguments).
> Well, you are right. But, we are focusing on embedded appliances,
> so those systems we use are not that "dynamic" with that respect.
> Namely: we have fixed number of domains and their functionality
> is well known, so we can do rather precise assumption on resource
> usage.

Interesting! So here I presume backend trusts the frontend.

Cheers,
Joao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Oleksandr Andrushchenko]

On 07/31/2017 02:58 PM, Joao Martins wrote:
> On 07/31/2017 12:41 PM, Oleksandr Andrushchenko wrote:
>> Hi, Joao!
>>
>> On 07/31/2017 02:03 PM, Joao Martins wrote:
>>> Hey Oleksandr,
>>>
>>> On 07/31/2017 09:34 AM, Oleksandr Andrushchenko wrote:
>>>> Hi, all!
>>>>
>>> [snip]
>>>> Comparison for display use-case
>>>> ===============================
>>>>
>>>> 1 Number of grant references used
>>>> 1-1 grant references: nr_pages
>>>> 1-2 GNTTABOP_transfer: nr_pages
>>>> 1-3 XENMEM_exchange: not an option
>>>>
>>>> 2 Effect of DomU crash on Dom0 (its mapped pages)
>>>> 2-1 grant references: pages can be unmapped by Dom0, Dom0 is fully
>>>> recovered
>>>> 2-2 GNTTABOP_transfer: pages will be returned to the Hypervisor, lost
>>>> for Dom0
>>>> 2-3 XENMEM_exchange: not an option
>>>>
>>>> 3 Security issues from sharing Dom0 pages to DomU
>>>> 1-1 grant references: none
>>>> 1-2 GNTTABOP_transfer: none
>>>> 1-3 XENMEM_exchange: not an option
>>>>
>>>> At the moment approach 1 with granted references seems to be a winner for
>>>> sharing buffers both ways, e.g. Dom0 -> DomU and DomU -> Dom0.
>>>>
>>>> Conclusion
>>>> ==========
>>>>
>>>> I would like to get some feedback from the community on which approach
>>>> is more
>>>> suitable for sharing large buffers and to have a clear vision on cons
>>>> and pros
>>>> of each one: please feel free to add other metrics I missed and correct
>>>> the ones
>>>> I commented on.  I would appreciate help on comparing approaches 2 and 3
>>>> as I
>>>> have little knowledge of these APIs (2 seems to be addressed by
>>>> Christopher, and
>>>> 3 seems to be relevant to what Konrad/Stefano do WRT SWIOTLB).
>>> Depending on your performance/memory requirements - there could be another
>>> option which is to keep the guest mapped on Domain-0 (what was discussed with
>>> Zerogrant session[0][1] that will be formally proposed in the next month or so).
>> Unfortunately I missed that session during the Summit
>> due to overlapping sessions
> Hmm - Zerocopy Rx (Dom0 -> DomU) would indeed be an interesting topic to bring up.
>
it is, especially for the systems which require physically contiguous
buffers
>>> But that would only solve the grant maps/unmaps/copies done on Domain-0 (given
>>> the numbers you pasted a bit ago, you might not really need to go to such extents)
>>>
>>> [0]
>>> http://schd.ws/hosted_files/xendeveloperanddesignsummit2017/05/zerogrant_spec.pdf
>>> [1]
>>> http://schd.ws/hosted_files/xendeveloperanddesignsummit2017/a8/zerogrant_slides.pdf
>> I will read these, thank you for the links
>>> For the buffers allocated on Dom0 and safely grant buffers from Dom0 to DomU
>>> (which I am not so sure it is possible today :()
>> We have this working in our setup for display (we have implemented
>> z-copy with grant references already)
> Allow me to clarify :) I meant "possible to do it in a safely manner", IOW,
> regarding what I mentioned below in following paragraphs. But your answer below
> clarifies on that aspect.
good :)
>>> , maybe a "contract" from DomU
>>> provide a set of transferable pages that Dom0 holds on for each Dom-0 gref
>>> provided to the guest (and assuming this is only a handful couple of guests as
>>> grant table is not that big).
>> It is an option
>>>    IIUC, From what you pasted above on "Buffer
>>> allocated @Dom0" sounds like Domain-0 could quickly ran out of pages/OOM (and
>>> grants), if you're guest is misbehaving/buggy or malicious; *also* domain-0
>>> grant table is a rather finite/small resource (even though you can override the
>>> number of frames in the arguments).
>> Well, you are right. But, we are focusing on embedded appliances,
>> so those systems we use are not that "dynamic" with that respect.
>> Namely: we have fixed number of domains and their functionality
>> is well known, so we can do rather precise assumption on resource
>> usage.
> Interesting! So here I presume backend trusts the frontend.
yes, this is the case. What is more backend can make decision
on if to allow buffer allocation or reject the request
> Cheers,
> Joao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Stefano Stabellini]

On Mon, 31 Jul 2017, Oleksandr Andrushchenko wrote:
> 3 Sharing with page exchange (XENMEM_exchange)
> ==============================================
> 
> This API was pointed to me by Stefano Stabellini as one of the possible ways
> to
> achieve zero copying and share physically contiguous buffers. It is used by
> x86
> SWIOTLB code (xen_create_contiguous_region, [5]), but as per my understanding
> this API cannot be used on ARM as of now [6].  Conclusion: not an option for
> ARM
> at the moment

Let me elaborate on this. The purpose of XENMEM_exchange is to exchange
a number of memory pages with an equal number of contiguous memory
pages, possibly even under 4G. The original purpose of the hypercall was
to get DMA-able memory.

So far, it has only been used by Dom0 on x86. Dom0 on ARM doesn't need
it because it is mapped 1:1 by default and device assignment is not
allowed without an IOMMU. However it should work on ARM too, as the
implementation is all common code in Xen. Also, looking at the
implementation (xen/common/memory.c:memory_exchange) it would seem that
it can be called from a DomU too (but I have never tried).

Thus, if you have a platform without IOMMU and you disabled the IOMMU
checks in Xen to assign a device to a DomU anyway, then you could use
this hypercall from DomU to get memory under 4G to be used for DMA with
this device.

As far as I can tell XENMEM_exchange could help in the design of
zero-copy PV protocols only to address this specific use case:

- you have a frontend in DomU and a backend in Dom0
- pages shared by DomU get mapped in Dom0 and potentially used for DMA
- the device has under 4G DMA restrictions

Normally Dom0 maps a DomU page, then at the time of using the mapped
page for DMA it checks whether it is suitable for DMA (under 4G if the
device requires so). If it is not, Dom0 uses a bounce buffer borrowed
from the swiotlb. Obviously this introduces one or two memcpys.

Instead, if DomU calls XENMEM_exchange to get memory under 4G, and
shares one of the pages with Dom0 via PV frontends, then Dom0 wouldn't
have to use a bounce buffer to do DMA to this page.

Does it make sense?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Oleksandr Andrushchenko]

Hi, Stefano!

On 07/31/2017 11:28 PM, Stefano Stabellini wrote:
> On Mon, 31 Jul 2017, Oleksandr Andrushchenko wrote:
>> 3 Sharing with page exchange (XENMEM_exchange)
>> ==============================================
>>
>> This API was pointed to me by Stefano Stabellini as one of the possible ways
>> to
>> achieve zero copying and share physically contiguous buffers. It is used by
>> x86
>> SWIOTLB code (xen_create_contiguous_region, [5]), but as per my understanding
>> this API cannot be used on ARM as of now [6].  Conclusion: not an option for
>> ARM
>> at the moment
> Let me elaborate on this. The purpose of XENMEM_exchange is to exchange
> a number of memory pages with an equal number of contiguous memory
> pages, possibly even under 4G. The original purpose of the hypercall was
> to get DMA-able memory.
this is good to know
>
> So far, it has only been used by Dom0 on x86. Dom0 on ARM doesn't need
> it because it is mapped 1:1 by default and device assignment is not
> allowed without an IOMMU. However it should work on ARM too, as the
> implementation is all common code in Xen.
well, according to [6]:
"Currently XENMEM_exchange is not supported on ARM because the steal_page is
left unimplemented.

However, even if steal_page is implemented, the hypercall can't work for ARM
because:
     - Direct mapped domain is not supported
     - ARM doesn't have a M2P and therefore usage of mfn_to_gmfn is
     invalid"
And what I see at [7] is that it is still EOPNOTSUPP
So, yes, common code is usable for both ARM and x86, but
underlying support for ARM is till not there.
Please correct me if I am wrong here
>   Also, looking at the
> implementation (xen/common/memory.c:memory_exchange) it would seem that
> it can be called from a DomU too (but I have never tried).
good
> Thus, if you have a platform without IOMMU and you disabled the IOMMU
> checks in Xen to assign a device to a DomU anyway, then you could use
> this hypercall from DomU to get memory under 4G to be used for DMA with
> this device.
There is no real device assigned to DomU, but a PV frontend
>
> As far as I can tell XENMEM_exchange could help in the design of
> zero-copy PV protocols only to address this specific use case:
>
> - you have a frontend in DomU and a backend in Dom0
> - pages shared by DomU get mapped in Dom0 and potentially used for DMA
yes, this is crucial for zero copying in my case: DMA
> - the device has under 4G DMA restrictions
>
> Normally Dom0 maps a DomU page, then at the time of using the mapped
> page for DMA it checks whether it is suitable for DMA (under 4G if the
> device requires so). If it is not, Dom0 uses a bounce buffer borrowed
> from the swiotlb. Obviously this introduces one or two memcpys.
>
> Instead, if DomU calls XENMEM_exchange to get memory under 4G, and
> shares one of the pages with Dom0 via PV frontends, then Dom0 wouldn't
> have to use a bounce buffer to do DMA to this page.
>
> Does it make sense?
yes, it does, thank you, but [6], [7] :(


[7] 
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/mm.c;h=411bab1ea9f7f14789a134056ebff9f68fd4a4c7;hb=a15516c0cf21d7ac84799f1e2e500b0bb22d2300#l1161


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Stefano Stabellini]

On Tue, 1 Aug 2017, Oleksandr Andrushchenko wrote:
> Hi, Stefano!
> 
> On 07/31/2017 11:28 PM, Stefano Stabellini wrote:
> > On Mon, 31 Jul 2017, Oleksandr Andrushchenko wrote:
> > > 3 Sharing with page exchange (XENMEM_exchange)
> > > ==============================================
> > > 
> > > This API was pointed to me by Stefano Stabellini as one of the possible
> > > ways
> > > to
> > > achieve zero copying and share physically contiguous buffers. It is used
> > > by
> > > x86
> > > SWIOTLB code (xen_create_contiguous_region, [5]), but as per my
> > > understanding
> > > this API cannot be used on ARM as of now [6].  Conclusion: not an option
> > > for
> > > ARM
> > > at the moment
> > Let me elaborate on this. The purpose of XENMEM_exchange is to exchange
> > a number of memory pages with an equal number of contiguous memory
> > pages, possibly even under 4G. The original purpose of the hypercall was
> > to get DMA-able memory.
> this is good to know
> > 
> > So far, it has only been used by Dom0 on x86. Dom0 on ARM doesn't need
> > it because it is mapped 1:1 by default and device assignment is not
> > allowed without an IOMMU. However it should work on ARM too, as the
> > implementation is all common code in Xen.
> well, according to [6]:
> "Currently XENMEM_exchange is not supported on ARM because the steal_page is
> left unimplemented.
> 
> However, even if steal_page is implemented, the hypercall can't work for ARM
> because:
>     - Direct mapped domain is not supported
>     - ARM doesn't have a M2P and therefore usage of mfn_to_gmfn is
>     invalid"
> And what I see at [7] is that it is still EOPNOTSUPP
> So, yes, common code is usable for both ARM and x86, but
> underlying support for ARM is till not there.
> Please correct me if I am wrong here

Ops, I forgot about that! Implementing steal_page on ARM is not be a
problem, and direct mapped domains are not a concern in this scenario.

The issue is mfn_to_gmfn. However, we do not actually need mfn_to_gmfn
to implement xen/common/memory.c:memory_exchange as Julien pointed out
in http://marc.info/?l=xen-devel&m=145037009127660.

Julien, Jan, two years have passed. Do you think we can find a way to
make that old series work for everybody?


> >   Also, looking at the
> > implementation (xen/common/memory.c:memory_exchange) it would seem that
> > it can be called from a DomU too (but I have never tried).
> good
> > Thus, if you have a platform without IOMMU and you disabled the IOMMU
> > checks in Xen to assign a device to a DomU anyway, then you could use
> > this hypercall from DomU to get memory under 4G to be used for DMA with
> > this device.
> There is no real device assigned to DomU, but a PV frontend
> > 
> > As far as I can tell XENMEM_exchange could help in the design of
> > zero-copy PV protocols only to address this specific use case:
> > 
> > - you have a frontend in DomU and a backend in Dom0
> > - pages shared by DomU get mapped in Dom0 and potentially used for DMA
> yes, this is crucial for zero copying in my case: DMA
> > - the device has under 4G DMA restrictions
> > 
> > Normally Dom0 maps a DomU page, then at the time of using the mapped
> > page for DMA it checks whether it is suitable for DMA (under 4G if the
> > device requires so). If it is not, Dom0 uses a bounce buffer borrowed
> > from the swiotlb. Obviously this introduces one or two memcpys.
> > 
> > Instead, if DomU calls XENMEM_exchange to get memory under 4G, and
> > shares one of the pages with Dom0 via PV frontends, then Dom0 wouldn't
> > have to use a bounce buffer to do DMA to this page.
> > 
> > Does it make sense?
> yes, it does, thank you, but [6], [7] :(
> 
> [7]
> https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/mm.c;h=411bab1ea9f7f14789a134056ebff9f68fd4a4c7;hb=a15516c0cf21d7ac84799f1e2e500b0bb22d2300#l1161


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV drivers and zero copying

[Julien Grall]

Hi,

On 01/08/17 20:07, Stefano Stabellini wrote:
> On Tue, 1 Aug 2017, Oleksandr Andrushchenko wrote:
>> Hi, Stefano!
>>
>> On 07/31/2017 11:28 PM, Stefano Stabellini wrote:
>>> On Mon, 31 Jul 2017, Oleksandr Andrushchenko wrote:
>>>> 3 Sharing with page exchange (XENMEM_exchange)
>>>> ==============================================
>>>>
>>>> This API was pointed to me by Stefano Stabellini as one of the possible
>>>> ways
>>>> to
>>>> achieve zero copying and share physically contiguous buffers. It is used
>>>> by
>>>> x86
>>>> SWIOTLB code (xen_create_contiguous_region, [5]), but as per my
>>>> understanding
>>>> this API cannot be used on ARM as of now [6].  Conclusion: not an option
>>>> for
>>>> ARM
>>>> at the moment
>>> Let me elaborate on this. The purpose of XENMEM_exchange is to exchange
>>> a number of memory pages with an equal number of contiguous memory
>>> pages, possibly even under 4G. The original purpose of the hypercall was
>>> to get DMA-able memory.
>> this is good to know
>>>
>>> So far, it has only been used by Dom0 on x86. Dom0 on ARM doesn't need
>>> it because it is mapped 1:1 by default and device assignment is not
>>> allowed without an IOMMU. However it should work on ARM too, as the
>>> implementation is all common code in Xen.
>> well, according to [6]:
>> "Currently XENMEM_exchange is not supported on ARM because the steal_page is
>> left unimplemented.
>>
>> However, even if steal_page is implemented, the hypercall can't work for ARM
>> because:
>>     - Direct mapped domain is not supported
>>     - ARM doesn't have a M2P and therefore usage of mfn_to_gmfn is
>>     invalid"
>> And what I see at [7] is that it is still EOPNOTSUPP
>> So, yes, common code is usable for both ARM and x86, but
>> underlying support for ARM is till not there.
>> Please correct me if I am wrong here
>
> Ops, I forgot about that! Implementing steal_page on ARM is not be a
> problem, and direct mapped domains are not a concern in this scenario.
>
> The issue is mfn_to_gmfn. However, we do not actually need mfn_to_gmfn
> to implement xen/common/memory.c:memory_exchange as Julien pointed out
> in http://marc.info/?l=xen-devel&m=145037009127660.
>
> Julien, Jan, two years have passed. Do you think we can find a way to
> make that old series work for everybody?

I looked at the series and AFAICT Jan was pushing towards M2P.

I've already made my arguments in another thread (see [1]) why I don't 
think M2P is a solution for ARM.

I really want to kill the use of M2P on ARM because I think this using 
too much memory for the benefits it brings.

I don't have much time to investigate more on XENMEM_exchange, so I 
would be grateful if someone pick up that task.

Cheers,

[1] 
https://lists.xenproject.org/archives/html/xen-devel/2017-05/msg01370.html

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel