linux-aio usable?

linux-aio usable?

[Bernhard Schmidt]

Hi,

sorry for this pretty generic question, I did not find any real pros and
cons on the net anywhere, but I might just have missed them.

In a pure x86_64 environment (~2.6.32 vanilla kernel, 0.12.3 qemu-kvm),
is enabling linux-aio in KVM a good idea? What are the
advantages/disadvantages? Are there any potential pitfalls? 

The reason I'm asking is that there has been some traffic on the list
about it, so it seems to be something people want to get working.
qemu-kvm in Ubuntu Lucid is currently not compiled with that option.
I've made a local version with aio and it seems to work fine (and
performs a bit better at first glance).

Is there any reason one should not compile that feature by default?
Does it do anything if not explicitly run with aio=native?

Best Regards,
Bernhard

Re: linux-aio usable?

[Avi Kivity]

On 03/08/2010 03:46 AM, Bernhard Schmidt wrote:
> Hi,
>
> sorry for this pretty generic question, I did not find any real pros and
> cons on the net anywhere, but I might just have missed them.
>
> In a pure x86_64 environment (~2.6.32 vanilla kernel, 0.12.3 qemu-kvm),
> is enabling linux-aio in KVM a good idea?

Yes.

> What are the
> advantages/disadvantages?

It's faster.

> Are there any potential pitfalls?
>    

It won't work well unless running on a block device (partition or LVM).

> The reason I'm asking is that there has been some traffic on the list
> about it, so it seems to be something people want to get working.
> qemu-kvm in Ubuntu Lucid is currently not compiled with that option.
> I've made a local version with aio and it seems to work fine (and
> performs a bit better at first glance).
>
> Is there any reason one should not compile that feature by default?
>    

Not to my knowledge.

> Does it do anything if not explicitly run with aio=native?
>    

IIUC, no.

-- 
error compiling committee.c: too many arguments to function

Re: linux-aio usable?

[Bernhard Schmidt]

On Mon, Mar 08, 2010 at 11:10:29AM +0200, Avi Kivity wrote:

> >Are there any potential pitfalls?
> It won't work well unless running on a block device (partition or LVM).

What does "work well" mean in this context? Potential dataloss?

> >Is there any reason one should not compile that feature by default?
> Not to my knowledge.

Thanks, I've filed a bug with Ubuntu to get it enabled.

Bernhard

Re: linux-aio usable?

[Avi Kivity]

On 03/08/2010 11:48 AM, Bernhard Schmidt wrote:
> On Mon, Mar 08, 2010 at 11:10:29AM +0200, Avi Kivity wrote:
>
>    
>>> Are there any potential pitfalls?
>>>        
>> It won't work well unless running on a block device (partition or LVM).
>>      
> What does "work well" mean in this context? Potential dataloss?
>
>    

No, it becomes synchronous (=extra slow).

-- 
error compiling committee.c: too many arguments to function

Re: linux-aio usable?

[Christoph Hellwig]

On Mon, Mar 08, 2010 at 11:10:29AM +0200, Avi Kivity wrote:
>> Are there any potential pitfalls?
>>    
>
> It won't work well unless running on a block device (partition or LVM).

It will actually work well on pre-allocated filesystem images, at least
on XFS and NFS.

The real pitfal is that cache=none is required for kernel support as
it only supports O_DIRECT.

>> Is there any reason one should not compile that feature by default?

It's compiled by default if libaio and it's development headers are
found.

>> Does it do anything if not explicitly run with aio=native?

No.

Re: linux-aio usable?

[Dustin Kirkland]

On Mon, Mar 8, 2010 at 3:48 AM, Avi Kivity <avi@redhat.com> wrote:
> On 03/08/2010 11:48 AM, Bernhard Schmidt wrote:
>>
>> On Mon, Mar 08, 2010 at 11:10:29AM +0200, Avi Kivity wrote:
>>>> Are there any potential pitfalls?
>>>
>>> It won't work well unless running on a block device (partition or LVM).
>>
>> What does "work well" mean in this context? Potential dataloss?
>
> No, it becomes synchronous (=extra slow).

But for this to happen, the user would have had to consciously enter
into the situation by creating/using a non block device,
non-pre-allocated backing disk AND specify the aio=native option,
correct?

:-Dustin

Re: linux-aio usable?

[Avi Kivity]

On 03/08/2010 04:25 PM, Dustin Kirkland wrote:
> On Mon, Mar 8, 2010 at 3:48 AM, Avi Kivity<avi@redhat.com>  wrote:
>    
>> On 03/08/2010 11:48 AM, Bernhard Schmidt wrote:
>>      
>>> On Mon, Mar 08, 2010 at 11:10:29AM +0200, Avi Kivity wrote:
>>>        
>>>>> Are there any potential pitfalls?
>>>>>            
>>>> It won't work well unless running on a block device (partition or LVM).
>>>>          
>>> What does "work well" mean in this context? Potential dataloss?
>>>        
>> No, it becomes synchronous (=extra slow).
>>      
> But for this to happen, the user would have had to consciously enter
> into the situation by creating/using a non block device,
> non-pre-allocated backing disk AND specify the aio=native option,
> correct?
>
>    

I thought there was some autodetection involved, but perhaps I just 
imagined it.

-- 
error compiling committee.c: too many arguments to function

Re: linux-aio usable?

[Anthony Liguori]

On 03/08/2010 08:26 AM, Avi Kivity wrote:
> On 03/08/2010 04:25 PM, Dustin Kirkland wrote:
>> On Mon, Mar 8, 2010 at 3:48 AM, Avi Kivity<avi@redhat.com>  wrote:
>>> On 03/08/2010 11:48 AM, Bernhard Schmidt wrote:
>>>> On Mon, Mar 08, 2010 at 11:10:29AM +0200, Avi Kivity wrote:
>>>>>> Are there any potential pitfalls?
>>>>> It won't work well unless running on a block device (partition or 
>>>>> LVM).
>>>> What does "work well" mean in this context? Potential dataloss?
>>> No, it becomes synchronous (=extra slow).
>> But for this to happen, the user would have had to consciously enter
>> into the situation by creating/using a non block device,
>> non-pre-allocated backing disk AND specify the aio=native option,
>> correct?
>>
>
> I thought there was some autodetection involved, but perhaps I just 
> imagined it.

There's no autodetection.

linux-aio support in the kernel downgrades to synchronous IO if the 
underlying storage does not support linux-aio.  There is no indication 
to userspace that this has happened.

If this happens, besides having a slow guest, the guest VCPU will be 
starved during the I/O requests potentially resulting in things like 
soft lockups and time drift.

Generally, speaking, linux-aio will work well under the following 
circumstances:

  - cache=off is specified
  - the underlying file system is XFS or you are using a block device

We cannot detect this reliably though so it's really up to the user to 
decide whether to use it.  We're working on improving the linux-aio 
kernel interface though to eliminate this detectability problem after 
which, we can enable it in a more automatic fashion.

Regards,

Anthony Liguori

Re: linux-aio usable?

[Avi Kivity]

On 03/08/2010 06:28 PM, Anthony Liguori wrote:
>> I thought there was some autodetection involved, but perhaps I just 
>> imagined it.
>
>
> There's no autodetection.
>
> linux-aio support in the kernel downgrades to synchronous IO if the 
> underlying storage does not support linux-aio.  There is no indication 
> to userspace that this has happened.
>
> If this happens, besides having a slow guest, the guest VCPU will be 
> starved during the I/O requests potentially resulting in things like 
> soft lockups and time drift.
>
> Generally, speaking, linux-aio will work well under the following 
> circumstances:
>
>  - cache=off is specified
>  - the underlying file system is XFS or you are using a block device
>
> We cannot detect this reliably though so it's really up to the user to 
> decide whether to use it.  We're working on improving the linux-aio 
> kernel interface though to eliminate this detectability problem after 
> which, we can enable it in a more automatic fashion.

Well, the common case of cache=none on a block device certainly can be 
autodetected.

-- 
error compiling committee.c: too many arguments to function

Re: linux-aio usable?

[Michael Tokarev]

Avi Kivity wrote:
> On 03/08/2010 03:46 AM, Bernhard Schmidt wrote:
>> Hi,
>>
>> sorry for this pretty generic question, I did not find any real pros and
>> cons on the net anywhere, but I might just have missed them.
>>
>> In a pure x86_64 environment (~2.6.32 vanilla kernel, 0.12.3 qemu-kvm),
>> is enabling linux-aio in KVM a good idea?
> 
> Yes.

Apparently that does not quite work.  I just re-compiled kvm with
--enable-linux-aio (actually I just installed libaio-dev on debian
and qemu-kvm's configure picked it up automatically), and tried
a guest.  But any I/O fails.

  kvm-0.12.3 ... -drive file=/dev/sda10,if=virtio,cache=none,aio=native

(/dev/sda10 is a (spare) partition on my hard drive I use for testing).
Here's the resulting dmesg in the guest (2.6.32):

 vdb:
end_request: I/O error, dev vdb, sector 0
Buffer I/O error on device vdb, logical block 0
Buffer I/O error on device vdb, logical block 1
Buffer I/O error on device vdb, logical block 2
Buffer I/O error on device vdb, logical block 3
Buffer I/O error on device vdb, logical block 4
Buffer I/O error on device vdb, logical block 5
Buffer I/O error on device vdb, logical block 6
Buffer I/O error on device vdb, logical block 7
end_request: I/O error, dev vdb, sector 0
Buffer I/O error on device vdb, logical block 0
Buffer I/O error on device vdb, logical block 1
end_request: I/O error, dev vdb, sector 0
 unable to read partition table

And any I/O - be it reads of writes - fails.

I see some aio_submit() etc are happening in strace,
but no errors.  Unfortunately my strace does not
decode io_*() routines.

# fgrep io_ trc
...
1227  io_submit(4152147968, 1, {...})   = 1
1226  io_getevents(-142819328, 1, 128, {...}{0, 0}) = 1
1227  io_submit(4152147968, 1, {...})   = 1
1226  io_getevents(-142819328, 1, 128, {...}{0, 0}) = 1
1227  io_submit(4152147968, 1, {...})   = 1
1226  io_getevents(-142819328, 1, 128, {...}{0, 0}) = 1
...

Oh well.... ;)

/mjt

Re: linux-aio usable?

[Michael Tokarev]

Michael Tokarev wrote:
[]
> Apparently that does not quite work.  I just re-compiled kvm with
> --enable-linux-aio (actually I just installed libaio-dev on debian
> and qemu-kvm's configure picked it up automatically), and tried
> a guest.  But any I/O fails.

It has nothing to do with kvm.  It is compat_ioctl32 in the kernel
wrt aio calls.  Historically I've a 64bit kernel with 32bit userland,
and tried 32bit kvm too, and that does not work.  But 64bit kvm works
just fine with aio, and the performance numbers are indeed better.

Thanks!

/mjt

Re: linux-aio usable?

[Nikola Ciprich]

> It's faster.
Hi Avi,
Could You give some rough estimate on how much faster?
I'm stuck with glibc-2.5 now, but I'm always eager to improve performance,
so I wonder if it would make sense to either port eventfd + aio stuff, or
switch to glibc-2.8 for me...



-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: linux-aio usable?

[Brian Jackson]

On Monday 08 March 2010 03:27:36 pm Nikola Ciprich wrote:
> > It's faster.
> 
> Hi Avi,
> Could You give some rough estimate on how much faster?
> I'm stuck with glibc-2.5 now, but I'm always eager to improve performance,
> so I wonder if it would make sense to either port eventfd + aio stuff, or
> switch to glibc-2.8 for me...


I saw approx. 10% improvement in sequential i/o. Random i/o was only 
marginally faster in our setup. We generally have problems with random i/o 
here... Something to do with our setup.

Re: linux-aio usable?

[Avi Kivity]

On 03/08/2010 10:19 PM, Michael Tokarev wrote:
> Michael Tokarev wrote:
> []
>    
>> Apparently that does not quite work.  I just re-compiled kvm with
>> --enable-linux-aio (actually I just installed libaio-dev on debian
>> and qemu-kvm's configure picked it up automatically), and tried
>> a guest.  But any I/O fails.
>>      
> It has nothing to do with kvm.  It is compat_ioctl32 in the kernel
> wrt aio calls.  Historically I've a 64bit kernel with 32bit userland,
> and tried 32bit kvm too, and that does not work.  But 64bit kvm works
> just fine with aio, and the performance numbers are indeed better.
>
>    

Can you elaborate?  This sounds like a bug that wants to be fixed.

-- 
error compiling committee.c: too many arguments to function

Re: linux-aio usable?

[Avi Kivity]

On 03/08/2010 11:27 PM, Nikola Ciprich wrote:
>> It's faster.
>>      
> Hi Avi,
> Could You give some rough estimate on how much faster?
>    

The standard "it depends on the workload".

> I'm stuck with glibc-2.5 now, but I'm always eager to improve performance,
> so I wonder if it would make sense to either port eventfd + aio stuff, or
> switch to glibc-2.8 for me...
>    

Switching to a modern setup should be much easier and safer.  Esp. a 
modern kernel.

-- 
error compiling committee.c: too many arguments to function

Re: linux-aio usable?

[Michael Tokarev]

Avi Kivity wrote:
> On 03/08/2010 10:19 PM, Michael Tokarev wrote:
>> Michael Tokarev wrote:
>> []
>>   
>>> Apparently that does not quite work.  I just re-compiled kvm with
>>> --enable-linux-aio (actually I just installed libaio-dev on debian
>>> and qemu-kvm's configure picked it up automatically), and tried
>>> a guest.  But any I/O fails.
>>>      
>> It has nothing to do with kvm.  It is compat_ioctl32 in the kernel
>> wrt aio calls.  Historically I've a 64bit kernel with 32bit userland,
>> and tried 32bit kvm too, and that does not work.  But 64bit kvm works
>> just fine with aio, and the performance numbers are indeed better.
> 
> Can you elaborate?  This sounds like a bug that wants to be fixed.

http://thread.gmane.org/gmane.linux.kernel.aio.general/2891

It's missing compat_ioctl for some of the aio opcodes, namely
it's PREADV and PWRITE - the only ones used by kvm and the only
ones missing in kernel.

As far as i can see, current code converts the iocb array just
fine, but does not touch iovec array used with p{read,write}v.

/mjt

Re: linux-aio usable?

[Avi Kivity]

On 03/09/2010 11:19 AM, Michael Tokarev wrote:
>
>> Can you elaborate?  This sounds like a bug that wants to be fixed.
>>      
> http://thread.gmane.org/gmane.linux.kernel.aio.general/2891
>
> It's missing compat_ioctl for some of the aio opcodes, namely
> it's PREADV and PWRITE - the only ones used by kvm and the only
> ones missing in kernel.
>
> As far as i can see, current code converts the iocb array just
> fine, but does not touch iovec array used with p{read,write}v.
>    

Yikes, looks pretty bad.

-- 
error compiling committee.c: too many arguments to function

Re: linux-aio usable?

[Christoph Hellwig]

On Tue, Mar 09, 2010 at 12:05:49PM +0200, Avi Kivity wrote:
>> http://thread.gmane.org/gmane.linux.kernel.aio.general/2891
>>
>> It's missing compat_ioctl for some of the aio opcodes, namely
>> it's PREADV and PWRITE - the only ones used by kvm and the only
>> ones missing in kernel.
>>
>> As far as i can see, current code converts the iocb array just
>> fine, but does not touch iovec array used with p{read,write}v.
>>    
>
> Yikes, looks pretty bad.

No need for compat_ioctl handlers as there are no ioctls involved
in aio, we just need better compat versions of io_submit, which
by itself is a bit of an ioctl-like multiplexer.

Adding the vector compat might be a bit messy, but shouldn't be
too hard given that we already deal with this for native
(p)readv/(p)writev.