Re: VIRTIO - compatibility with different virtualization solutions

[Andres Lagar-Cavilla <andreslc@gridcentric.ca>]

> On Thu, Feb 20, 2014 at 06:50:59PM -0800, Anthony Liguori wrote:
>> On Wed, Feb 19, 2014 at 5:31 PM, Rusty Russell <rusty@au1.ibm.com> wrote:
>>> Anthony Liguori <anthony@codemonkey.ws> writes:
>>>> On Tue, Feb 18, 2014 at 4:26 PM, Rusty Russell <rusty@au1.ibm.com> wrote:
>>>>> Daniel Kiper <daniel.kiper@oracle.com> writes:
>>>>>> Hi,
>>>>>> 
>>>>>> Below you could find a summary of work in regards to VIRTIO compatibility with
>>>>>> different virtualization solutions. It was done mainly from Xen point of view
>>>>>> but results are quite generic and can be applied to wide spectrum
>>>>>> of virtualization platforms.
>>>>> 
>>>>> Hi Daniel,
>>>>> 
>>>>>        Sorry for the delayed response, I was pondering...  CC changed
>>>>> to virtio-dev.
>>>>> 
>>>>> From a standard POV: It's possible to abstract out the where we use
>>>>> 'physical address' for 'address handle'.  It's also possible to define
>>>>> this per-platform (ie. Xen-PV vs everyone else).  This is sane, since
>>>>> Xen-PV is a distinct platform from x86.
>>>> 
>>>> I'll go even further and say that "address handle" doesn't make sense too.
>>> 
>>> I was trying to come up with a unique term, I wasn't trying to define
>>> semantics :)
>> 
>> Understood, that wasn't really directed at you.
>> 
>>> There are three debates here now: (1) what should the standard say, and
>> 
>> The standard should say, "physical address"
>> 
>>> (2) how would Linux implement it,
>> 
>> Linux should use the PCI DMA API.
>> 
>>> (3) should we use each platform's PCI
>>> IOMMU.
>> 
>> Just like any other PCI device :-)
>> 
>>>> Just using grant table references is not enough to make virtio work
>>>> well under Xen.  You really need to use bounce buffers ala persistent
>>>> grants.
>>> 
>>> Wait, if you're using bounce buffers, you didn't make it "work well"!
>> 
>> Preaching to the choir man...  but bounce buffering is proven to be
>> faster than doing grant mappings on every request.  xen-blk does
>> bounce buffering by default and I suspect netfront is heading that
>> direction soon.
>> 
> 
> FWIW Annie Li @ Oracle once implemented a persistent map prototype for
> netfront and the result was not satisfying.
> 
>> It would be a lot easier to simply have a global pool of grant tables
>> that effectively becomes the DMA pool.  Then the DMA API can bounce
>> into that pool and those addresses can be placed on the ring.
>> 
>> It's a little different for Xen because now the backends have to deal
>> with physical addresses but the concept is still the same.
>> 
> 
> How would you apply this to Xen's security model? How can hypervisor
> effectively enforce access control? "Handle" and "physical address" are
> essentially not the same concept, otherwise you wouldn't have proposed
> this change. Not saying I'm against this change, just this description
> is too vague for me to understand the bigger picture.

I might be missing something trivial. But the burden of enforcing visibility of memory only for handles befalls on the hypervisor. Taking KVM for example, the whole RAM of a guest is a vma in the mm of the faulting qemu process. That's KVM's way of doing things. "Handles" could be pfns for all that model cares, and translation+mapping from handles to actual guest RAM addresses is trivially O(1). And there's no guest control over ram visibility, and that's happy KVM.

Xen, on the other hand, can encode a 64 bit grant handle in the "__u64 addr" field of a virtio descriptor. The negotiation happens up front, the flags field is set to signal the guest is encoding handles in there. Once the Xen virtio backend gets that descriptor out of the ring, what is left is not all that different from what netback/blkback/gntdev do today with a ring request.

I'm obviously glossing over serious details (e.g. negotiation of what u64 addr means), but I what I'm going at is that I fail to understand why whole RAM visibility is a requirement for virtio. It seems to me to be a requirement for KVM and other hypervisors, while virtio is a transport and sync mechanism for high(er) level IO descriptors.

Can someone please clarify why "under Xen, you really need to use bounce buffers ala persistent grants?" Is that a performance need to avoid backend side repeated mapping and TLB junking? Granted. But why would it be a correctness need? Guest side grant table works requires no hyper calls in the data path.

If I am rewinding the conversation, feel free to ignore, but I'm not feeling a lot of clarity in the dialogue right now.

Thanks
Andres

> 
> But a downside for sure is that if we go with this change we then have
> to maintain two different paths in backend. However small the difference
> is it is still a burden.


> 
> Wei.
>