[Qemu-devel] Re: [RFC] vhost-blk implementation

["Michael S. Tsirkin" <mst@redhat.com>]

On Wed, Mar 24, 2010 at 10:58:50AM -0700, Badari Pulavarty wrote:
> Michael S. Tsirkin wrote:
>> On Tue, Mar 23, 2010 at 12:55:07PM -0700, Badari Pulavarty wrote:
>>   
>>> Michael S. Tsirkin wrote:
>>>     
>>>> On Tue, Mar 23, 2010 at 10:57:33AM -0700, Badari Pulavarty wrote:
>>>>         
>>>>> Michael S. Tsirkin wrote:
>>>>>             
>>>>>> On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote:
>>>>>>                   
>>>>>>> Write Results:
>>>>>>> ==============
>>>>>>>
>>>>>>> I see degraded IO performance when doing sequential IO write
>>>>>>> tests with vhost-blk compared to virtio-blk.
>>>>>>>
>>>>>>> # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct
>>>>>>>
>>>>>>> I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with
>>>>>>> vhost-blk. Wondering why ?
>>>>>>>                         
>>>>>> Try to look and number of interrupts and/or number of exits.
>>>>>>                   
>>>>> I checked interrupts and IO exits - there is no major noticeable  
>>>>>  difference between
>>>>> vhost-blk and virtio-blk scenerios.
>>>>>             
>>>>>> It could also be that you are overrunning some queue.
>>>>>>
>>>>>> I don't see any exit mitigation strategy in your patch:
>>>>>> when there are already lots of requests in a queue, it's usually
>>>>>> a good idea to disable notifications and poll the
>>>>>> queue as requests complete. That could help performance.
>>>>>>                   
>>>>> Do you mean poll eventfd for new requests instead of waiting for 
>>>>> new  notifications ?
>>>>> Where do you do that in vhost-net code ?
>>>>>             
>>>> vhost_disable_notify does this.
>>>>
>>>>         
>>>>> Unlike network socket, since we are dealing with a file, there is 
>>>>> no  ->poll support for it.
>>>>> So I can't poll for the data. And also, Issue I am having is on 
>>>>> the   write() side.
>>>>>             
>>>> Not sure I understand.
>>>>
>>>>         
>>>>> I looked at it some more - I see 512K write requests on the
>>>>> virtio-queue  in both vhost-blk and virtio-blk cases. Both qemu or
>>>>> vhost is doing synchronous  writes to page cache (there is no write
>>>>> batching in qemu that is affecting this  case).  I still puzzled on
>>>>> why virtio-blk outperforms vhost-blk.
>>>>>
>>>>> Thanks,
>>>>> Badari
>>>>>             
>>>> If you say the number of requests is the same, we are left with:
>>>> - requests are smaller for some reason?
>>>> - something is causing retries?
>>>>         
>>> No. IO requests sizes are exactly same (512K) in both cases. There 
>>> are  no retries or
>>> errors in both cases. One thing I am not clear is - for some reason   
>>> guest kernel
>>> could push more data into virtio-ring in case of virtio-blk vs   
>>> vhost-blk. Is this possible ?
>>> Does guest gets to run much sooner in virtio-blk case than vhost-blk 
>>> ?  Sorry, if its dumb question -
>>> I don't understand  all the vhost details :(
>>>
>>> Thanks,
>>> Badari
>>>     
>>
>> BTW, did you put the backend in non-blocking mode?
>> As I said, vhost net passes non-blocking flag to
>> socket backend, but vfs_write/read that you use does
>> not have an option to do this.
>>
>> So we'll need to extend the backend to fix that,
>> but just for demo purposes, you could set non-blocking
>> mode on the file from userspace.
>>
>>   
> Michael,
>
> Atleast I understand why the performance difference now.. My debug
> code is changed the behaviour of virtio-blk which confused me.
>
> 1) virtio-blk is able to batch up writes from various requests.
> 2) virtio-blk offloads the writes to different thread
>
> Where as vhost-blk, I do each request syncrhonously. Hence
> the difference. You are right - i have to make backend async.
> I will working on handing off work to in-kernel threads.
> I am not sure about IO completion handling and calling
> vhost_add_used_and_signal() out of context. But let
> me take a stab at it and ask your help if I run into
> issues.
>
> Thanks,
> Badari
>


The way I did it for vhost net, requests are synchronous
but non-blocking. So if it can't be done directly,
I delay it.

-- 
MST