raw disks no longer work in latest kvm (kvm-88 was fine)

raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

Hi,

With qemu-kvm-0.12.3:
./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]
[    1.882843]  vdc:
[    2.365154] udev: starting version 146
[    2.693768] end_request: I/O error, dev vdc, sector 126
[    2.693772] Buffer I/O error on device vdc, logical block 126
[    2.693775] Buffer I/O error on device vdc, logical block 127
[    2.693777] Buffer I/O error on device vdc, logical block 128
[    2.693779] Buffer I/O error on device vdc, logical block 129
[    2.693781] Buffer I/O error on device vdc, logical block 130
[    2.693783] Buffer I/O error on device vdc, logical block 131
[    2.693785] Buffer I/O error on device vdc, logical block 132
[    2.693787] Buffer I/O error on device vdc, logical block 133
[    2.693788] Buffer I/O error on device vdc, logical block 134
[    2.693814] end_request: I/O error, dev vdc, sector 0
[    3.144870] end_request: I/O error, dev vdc, sector 0
[    3.499377] end_request: I/O error, dev vdc, sector 0
[    3.523247] end_request: I/O error, dev vdc, sector 0
[    3.547130] end_request: I/O error, dev vdc, sector 0
[    3.550076] end_request: I/O error, dev vdc, sector 0

Works fine with kvm-88:
cp  /usr/src/KVM/kvm-88/pc-bios/*bin ./
cp /usr/src/KVM/kvm-88/x86_64-softmmu/qemu-system-x86_64 ./
./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]

[    1.650274]  vdc: unknown partition table
[  112.704164] EXT4-fs (vdc): mounted filesystem with ordered data mode

I've tried running as root, using the block device directly (as shown 
above) rather than using a softlink, etc..

Something broke.
Host and guest are both running 2.6.33 and latest KVM.

Cheers
Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Michael Tokarev]

Antoine Martin wrote:
> Hi,
> 
> With qemu-kvm-0.12.3:
> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]
> [    1.882843]  vdc:
> [    2.365154] udev: starting version 146
> [    2.693768] end_request: I/O error, dev vdc, sector 126
> [    2.693772] Buffer I/O error on device vdc, logical block 126
> [    2.693775] Buffer I/O error on device vdc, logical block 127
> [    2.693777] Buffer I/O error on device vdc, logical block 128
..
> [    3.550076] end_request: I/O error, dev vdc, sector 0
> 
> Works fine with kvm-88:
> cp  /usr/src/KVM/kvm-88/pc-bios/*bin ./
> cp /usr/src/KVM/kvm-88/x86_64-softmmu/qemu-system-x86_64 ./
> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]
> 
> [    1.650274]  vdc: unknown partition table
> [  112.704164] EXT4-fs (vdc): mounted filesystem with ordered data mode
> 
> I've tried running as root, using the block device directly (as shown
> above) rather than using a softlink, etc..
> 
> Something broke.
> Host and guest are both running 2.6.33 and latest KVM.

https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 03/07/2010 04:28 AM, Michael Tokarev wrote:
> Antoine Martin wrote:
>    
>> Hi,
>>
>> With qemu-kvm-0.12.3:
>> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]
>> [    1.882843]  vdc:
>> [    2.365154] udev: starting version 146
>> [    2.693768] end_request: I/O error, dev vdc, sector 126
>> [    2.693772] Buffer I/O error on device vdc, logical block 126
>> [    2.693775] Buffer I/O error on device vdc, logical block 127
>> [    2.693777] Buffer I/O error on device vdc, logical block 128
>>      
> ..
>    
>> [    3.550076] end_request: I/O error, dev vdc, sector 0
>>
>> Works fine with kvm-88:
>> cp  /usr/src/KVM/kvm-88/pc-bios/*bin ./
>> cp /usr/src/KVM/kvm-88/x86_64-softmmu/qemu-system-x86_64 ./
>> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]
>>
>> [    1.650274]  vdc: unknown partition table
>> [  112.704164] EXT4-fs (vdc): mounted filesystem with ordered data mode
>>
>> I've tried running as root, using the block device directly (as shown
>> above) rather than using a softlink, etc..
>>
>> Something broke.
>> Host and guest are both running 2.6.33 and latest KVM.
>>      
> https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599
>    
The initial report is almost 8 weeks old!
Is data-corruption and data loss somehow less important than the 
hundreds of patches that have been submitted since?? Or is there a fix 
somewhere I've missed?

Cheers
Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Michael Tokarev]

Antoine Martin wrote:
[]
>> https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599
>>
>>    
> The initial report is almost 8 weeks old!
> Is data-corruption and data loss somehow less important than the
> hundreds of patches that have been submitted since?? Or is there a fix
> somewhere I've missed?

I can only guess that the info collected so far is not sufficient to
understand what's going on: except of "I/O error writing block NNN"
we does not have anything at all.  So it's impossible to know where
the problem is.

Please don't blame on people (me included: i suffer from this same
issue just like you).  We need something constructive.

And by the way, as stated in my comment attached to that bug (I'm
user mjtsf at sourceforge), error-behavour=remount-ro avoids data
corruption (this is not unique for the issue in question but for
general case of i/o errors and filesystem operations).

/mjt

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Gleb Natapov]

On Sun, Mar 07, 2010 at 03:48:23AM +0700, Antoine Martin wrote:
> Hi,
> 
> With qemu-kvm-0.12.3:
> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]
> [    1.882843]  vdc:
> [    2.365154] udev: starting version 146
> [    2.693768] end_request: I/O error, dev vdc, sector 126
> [    2.693772] Buffer I/O error on device vdc, logical block 126
> [    2.693775] Buffer I/O error on device vdc, logical block 127
> [    2.693777] Buffer I/O error on device vdc, logical block 128
> [    2.693779] Buffer I/O error on device vdc, logical block 129
> [    2.693781] Buffer I/O error on device vdc, logical block 130
> [    2.693783] Buffer I/O error on device vdc, logical block 131
> [    2.693785] Buffer I/O error on device vdc, logical block 132
> [    2.693787] Buffer I/O error on device vdc, logical block 133
> [    2.693788] Buffer I/O error on device vdc, logical block 134
> [    2.693814] end_request: I/O error, dev vdc, sector 0
> [    3.144870] end_request: I/O error, dev vdc, sector 0
> [    3.499377] end_request: I/O error, dev vdc, sector 0
> [    3.523247] end_request: I/O error, dev vdc, sector 0
> [    3.547130] end_request: I/O error, dev vdc, sector 0
> [    3.550076] end_request: I/O error, dev vdc, sector 0
> 
> Works fine with kvm-88:
> cp  /usr/src/KVM/kvm-88/pc-bios/*bin ./
> cp /usr/src/KVM/kvm-88/x86_64-softmmu/qemu-system-x86_64 ./
> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]
> 
> [    1.650274]  vdc: unknown partition table
> [  112.704164] EXT4-fs (vdc): mounted filesystem with ordered data mode
> 
> I've tried running as root, using the block device directly (as
> shown above) rather than using a softlink, etc..
> 
> Something broke.
> Host and guest are both running 2.6.33 and latest KVM.
> 
Are you sure you have write access to the block device?

--
			Gleb.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Christoph Hellwig]

On Sun, Mar 07, 2010 at 12:32:38PM +0300, Michael Tokarev wrote:
> Antoine Martin wrote:
> []
> >> https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599
> >>
> >>    
> > The initial report is almost 8 weeks old!
> > Is data-corruption and data loss somehow less important than the
> > hundreds of patches that have been submitted since?? Or is there a fix
> > somewhere I've missed?
> 
> I can only guess that the info collected so far is not sufficient to
> understand what's going on: except of "I/O error writing block NNN"
> we does not have anything at all.  So it's impossible to know where
> the problem is.

Actually it is, and the bug has been fixed long ago in:

commit e2a305fb13ff0f5cf6ff805555aaa90a5ed5954c
Author: Christoph Hellwig <hch@lst.de>
Date:   Tue Jan 26 14:49:08 2010 +0100

    block: avoid creating too large iovecs in multiwrite_merge


I've asked for it be added to the -stable series but that hasn't
happened so far.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 03/07/2010 05:00 PM, Christoph Hellwig wrote:
> On Sun, Mar 07, 2010 at 12:32:38PM +0300, Michael Tokarev wrote:
>    
>> Antoine Martin wrote:
>> []
>>      
>>>> https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599
>>>>
>>>>
>>>>          
>>> The initial report is almost 8 weeks old!
>>> Is data-corruption and data loss somehow less important than the
>>> hundreds of patches that have been submitted since?? Or is there a fix
>>> somewhere I've missed?
>>>        
>> I can only guess that the info collected so far is not sufficient to
>> understand what's going on: except of "I/O error writing block NNN"
>> we does not have anything at all.  So it's impossible to know where
>> the problem is.
>>      
> Actually it is, and the bug has been fixed long ago in:
>
> commit e2a305fb13ff0f5cf6ff805555aaa90a5ed5954c
> Author: Christoph Hellwig<hch@lst.de>
> Date:   Tue Jan 26 14:49:08 2010 +0100
>
>      block: avoid creating too large iovecs in multiwrite_merge
>    
Hmmm.
Like most of you, I had assumed that the data corruption bug (which I 
have also encountered on this raw partition as it is well over 1TB) 
would be the same as the "cannot open drive" bug I have described above, 
but unfortunately I still cannot use 0.12.3 with raw disks with this 
patch applied.
This is the patch, right?:
http://www.mail-archive.com/qemu-devel@nongnu.org/msg24129.html

So there is something else at play. And just for the record:
1) kvm-88 works fine *with the exact same setup*
2) I've tried running as root
3) The raw disk mounts fine from the host.
So I *know* the problem is with kvm. I wouldn't post to the list without 
triple checking that.

I have also just tested with another raw partition which is much smaller 
(1GB) and the same thing still occurs: kvm-88 works and qemu-kvm-0.12.3 
does not.
So I think that it is fair to assume that this new problem is unrelated 
to the partition size.
> I've asked for it be added to the -stable series but that hasn't
> happened so far.
>    
If "eating-your-data" doesn't make it to -stable, what does??
Even if it did, I would have thought this kind of bug should warrant a 
big warning sign somewhere.

Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

[snip]
>
> So there is something else at play. And just for the record:
> 1) kvm-88 works fine *with the exact same setup*
> 2) I've tried running as root
> 3) The raw disk mounts fine from the host.
> So I *know* the problem is with kvm. I wouldn't post to the list 
> without triple checking that.
>
> I have also just tested with another raw partition which is much 
> smaller (1GB) and the same thing still occurs: kvm-88 works and 
> qemu-kvm-0.12.3 does not.
> So I think that it is fair to assume that this new problem is 
> unrelated to the partition size.
I have narrowed it down to the "io-thread" option:
* rebuilding older versions of qemu without "--enable-io-thread" causes 
the bug (guest cannot open raw partition)
* qemu-kvm-0.12.3 cannot be built with "--enable-io-thread" over here:
   LINK  x86_64-softmmu/qemu-system-x86_64
kvm-all.o: In function `qemu_mutex_lock_iothread':
/usr/src/KVM/qemu-kvm-0.12.3/qemu-kvm.c:2532: multiple definition of 
`qemu_mutex_lock_iothread'
vl.o:/usr/src/KVM/qemu-kvm-0.12.3/vl.c:3772: first defined here
[..]
Which I have reported as part of another unsolved issue here:
http://www.mail-archive.com/kvm@vger.kernel.org/msg27663.html

Why not using the io-thread would prevent qemu from opening the raw 
partition is beyond me.

Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Michael Tokarev]

Antoine Martin wrote:
> [snip]
>>
>> So there is something else at play. And just for the record:
>> 1) kvm-88 works fine *with the exact same setup*
>> 2) I've tried running as root
>> 3) The raw disk mounts fine from the host.
>> So I *know* the problem is with kvm. I wouldn't post to the list
>> without triple checking that.
>>
>> I have also just tested with another raw partition which is much
>> smaller (1GB) and the same thing still occurs: kvm-88 works and
>> qemu-kvm-0.12.3 does not.
>> So I think that it is fair to assume that this new problem is
>> unrelated to the partition size.
> I have narrowed it down to the "io-thread" option:
> * rebuilding older versions of qemu without "--enable-io-thread" causes
> the bug (guest cannot open raw partition)
> * qemu-kvm-0.12.3 cannot be built with "--enable-io-thread" over here:
>   LINK  x86_64-softmmu/qemu-system-x86_64
> kvm-all.o: In function `qemu_mutex_lock_iothread':
> /usr/src/KVM/qemu-kvm-0.12.3/qemu-kvm.c:2532: multiple definition of
> `qemu_mutex_lock_iothread'
> vl.o:/usr/src/KVM/qemu-kvm-0.12.3/vl.c:3772: first defined here
> [..]
> Which I have reported as part of another unsolved issue here:
> http://www.mail-archive.com/kvm@vger.kernel.org/msg27663.html
> 
> Why not using the io-thread would prevent qemu from opening the raw
> partition is beyond me.

Ok, this is in fact different problem, not the one I referred you
initially (which was in fact good too, because apparently Christoph
solved that bug for me and for other Debian users, thank you!).

In your case, recalling your initial email:

> With qemu-kvm-0.12.3:
> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]
> [    1.882843]  vdc:
> [    2.365154] udev: starting version 146
> [    2.693768] end_request: I/O error, dev vdc, sector 126
> [    2.693772] Buffer I/O error on device vdc, logical block 126
> [    2.693775] Buffer I/O error on device vdc, logical block 127
> [    2.693777] Buffer I/O error on device vdc, logical block 128
...

the problem happens right at startup, it can't read _anything_
at all from the disk.  In my case, the problem is intermittent
and happens under high load only, hence the big difference.

But anyway, this is something which should be easy to find
out.  Run kvm under `strace -f' and see how it opens the
device, or find out with lsof what filedescriptor corresponds
to the file in question (in running kvm instance) and see
flags in /proc/$kvm_pid/fdinfo/$fdnum.

I guess it can't open the image in read-write mode somehow.

By the way, iothread doesn't really work in kvm, as far
as I can see.

Thanks.

/mjt

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 03/07/2010 12:00 PM, Christoph Hellwig wrote:
>
>> I can only guess that the info collected so far is not sufficient to
>> understand what's going on: except of "I/O error writing block NNN"
>> we does not have anything at all.  So it's impossible to know where
>> the problem is.
>>      
> Actually it is, and the bug has been fixed long ago in:
>
> commit e2a305fb13ff0f5cf6ff805555aaa90a5ed5954c
> Author: Christoph Hellwig<hch@lst.de>
> Date:   Tue Jan 26 14:49:08 2010 +0100
>
>      block: avoid creating too large iovecs in multiwrite_merge
>
>
> I've asked for it be added to the -stable series but that hasn't
> happened so far.
>    

Anthony, this looks critical.

-- 
error compiling committee.c: too many arguments to function

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

[snip]
> Ok, this is in fact different problem, not the one I referred you
> initially (which was in fact good too, because apparently Christoph
> solved that bug for me and for other Debian users, thank you!).
>    
Yes, I believe I had this issue too originally, which is why I was 
testing 0.12.3 to see if it would fix the problem...  and I hit this new 
seemingly unrelated snag.
> In your case, recalling your initial email:
>    
>> With qemu-kvm-0.12.3:
>> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..]
>> [    1.882843]  vdc:
>> [    2.365154] udev: starting version 146
>> [    2.693768] end_request: I/O error, dev vdc, sector 126
>> [    2.693772] Buffer I/O error on device vdc, logical block 126
>> [    2.693775] Buffer I/O error on device vdc, logical block 127
>> [    2.693777] Buffer I/O error on device vdc, logical block 128
>>      
> ...
>
> the problem happens right at startup, it can't read _anything_
> at all from the disk.  In my case, the problem is intermittent
> and happens under high load only, hence the big difference.
>
> But anyway, this is something which should be easy to find
> out.  Run kvm under `strace -f' and see how it opens the
>    
> device, or find out with lsof what filedescriptor corresponds
> to the file in question (in running kvm instance) and see
> flags in /proc/$kvm_pid/fdinfo/$fdnum.
>    
[...]
stat("./vm/var_fs", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 41), ...}) = 0
open("./vm/var_fs", O_RDWR|O_DIRECT|O_CLOEXEC) = 12
lseek(12, 0, SEEK_END)      = 1321851815424
[..]
So it opens it the device without problems.

The only things that stands out is this before the "read failed" message:
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid 
argument)


===================
Below is the full grep for "(12,"
[pid  9097] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9097] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9097] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9097] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9097] lseek(12, 0, SEEK_SET)      = 0
[pid  9097] read(12, 
"\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 
512) = 512
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9111] pread(12, 
"\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 
4096, 0) = 4096
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9157] pread(12, 0x7fa4f80b13d0, 64512, 0) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END <unfinished ...>
[pid  9137] pread(12, 0x7fa5020c53d0, 64512, 64512) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9138] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9143] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9123] pread(12, 0x7fa5090defd0, 16384, 0) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9126] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9125] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9127] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9128] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9129] pread(12, 
"?=\321\301\250\357\215\236\325{N\306\246\346=\23\266b\3556z\376\234\251\v,cG\371\302\340~"..., 
512, 4096) = 512
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9146] pread(12, 
"\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 
2048, 0) = 2048
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9115] pread(12, 0x7fa50d0e6fd0, 16384, 0) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9117] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9116] pread(12, 
"=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 
512, 1321851748352) = 512
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9118] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9119] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9120] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9153] pread(12, 0x7fa4fa0c0fd0, 16384, 0) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9132] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9154] pread(12, 
"=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 
512, 1321851748352) = 512
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9155] pread(12, "iQ\35 
\271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 512, 
1321851805696) = 512
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9133] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9134] pread(12, 
"?=\321\301\250\357\215\236\325{N\306\246\346=\23\266b\3556z\376\234\251\v,cG\371\302\340~"..., 
512, 4096) = 512
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9156] pread(12, 
"\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 
2048, 0) = 2048
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9117] pread(12, 0x7fa50c0e4fd0, 16384, 0) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9116] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9118] pread(12, 
"=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 
512, 1321851748352) = 512
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9119] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9120] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9121] pread(12, 
"?=\321\301\250\357\215\236\325{N\306\246\346=\23\266b\3556z\376\234\251\v,cG\371\302\340~"..., 
512, 4096) = 512
[pid  9098] lseek(12, 0, SEEK_END <unfinished ...>
[pid  9122] pread(12, 
"\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 
2048, 0) = 2048
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9207] pread(12, 0x7fa50e0e8fd0, 16384, 0) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9201] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9199] pread(12, 
"=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 
512, 1321851748352) = 512
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9210] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9208] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9203] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9200] pread(12, 0x7fa5098e37d0, 2048, 0) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9200] pread(12, 0x7fa5098dffd0, 16384, 0) = -1 EINVAL (Invalid 
argument)
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9209] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9202] pread(12, 
"=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 
512, 1321851748352) = 512
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9207] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9201] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9199] pread(12, <unfinished ...>
[pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
[pid  9210] pread(12, <unfinished ...>

Cheers
Antoine
> I guess it can't open the image in read-write mode somehow.
>
> By the way, iothread doesn't really work in kvm, as far
> as I can see.
>
> Thanks.
>
> /mjt
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 03/07/2010 07:11 PM, Antoine Martin wrote:
>>
>> the problem happens right at startup, it can't read _anything_
>> at all from the disk.  In my case, the problem is intermittent
>> and happens under high load only, hence the big difference.
>>
>> But anyway, this is something which should be easy to find
>> out.  Run kvm under `strace -f' and see how it opens the
>>    device, or find out with lsof what filedescriptor corresponds
>> to the file in question (in running kvm instance) and see
>> flags in /proc/$kvm_pid/fdinfo/$fdnum.
> ...
> [...]
> stat("./vm/var_fs", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 41), 
> ...}) = 0
> open("./vm/var_fs", O_RDWR|O_DIRECT|O_CLOEXEC) = 12
> lseek(12, 0, SEEK_END)      = 1321851815424
> [..]
> So it opens it the device without problems.
>
> The only things that stands out is this before the "read failed" message:
> [pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid  9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid 
> argument)
>

The buffer is unaligned here, yet the file was opened with O_DIRECT 
(cache=none).  This is strange, since alignment is not related to disk size.

-- 
error compiling committee.c: too many arguments to function

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Christoph Hellwig]

On Sun, Mar 07, 2010 at 07:18:40PM +0200, Avi Kivity wrote:
>> The only things that stands out is this before the "read failed" message:
>> [pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
>> [pid  9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid  
>> argument)
>>
>
> The buffer is unaligned here, yet the file was opened with O_DIRECT  
> (cache=none).  This is strange, since alignment is not related to disk 
> size.

The other interesting thing is that it's using pread - which means
it's a too old kernel to use preadv and thus a not very well tested
codepath with current qemu.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 03/07/2010 07:21 PM, Christoph Hellwig wrote:
> On Sun, Mar 07, 2010 at 07:18:40PM +0200, Avi Kivity wrote:
>    
>>> The only things that stands out is this before the "read failed" message:
>>> [pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
>>> [pid  9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid
>>> argument)
>>>
>>>        
>> The buffer is unaligned here, yet the file was opened with O_DIRECT
>> (cache=none).  This is strange, since alignment is not related to disk
>> size.
>>      
> The other interesting thing is that it's using pread - which means
> it's a too old kernel to use preadv and thus a not very well tested
> codepath with current qemu.
>    

It may also be that glibc is emulating preadv, incorrectly.

Antoine, can you check this?  ltrace may help, or run 'strings libc.so | 
grep pread'.

-- 
error compiling committee.c: too many arguments to function

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Christoph Hellwig]

On Sun, Mar 07, 2010 at 07:30:06PM +0200, Avi Kivity wrote:
> It may also be that glibc is emulating preadv, incorrectly.

I've done a quick audit of all pathes leading to pread and all seem
to align correctly.  So either a broken glibc emulation or something
else outside the block layer seems likely.

> Antoine, can you check this?  ltrace may help, or run 'strings libc.so |  
> grep pread'.

Or just add an

#undef CONFIG_PREADV

just before the first

#ifdef CONFIG_PREADV

in posix-aio-compat.c and see if that works.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 03/08/2010 12:30 AM, Avi Kivity wrote:
> On 03/07/2010 07:21 PM, Christoph Hellwig wrote:
>> On Sun, Mar 07, 2010 at 07:18:40PM +0200, Avi Kivity wrote:
>>>> The only things that stands out is this before the "read failed" 
>>>> message:
>>>> [pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
>>>> [pid  9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid
>>>> argument)
>>>>
>>> The buffer is unaligned here, yet the file was opened with O_DIRECT
>>> (cache=none).  This is strange, since alignment is not related to disk
>>> size.
Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none"

Side question: this is the right thing to do for raw partitions, right?
>> The other interesting thing is that it's using pread - which means
>> it's a too old kernel to use preadv and thus a not very well tested
>> codepath with current qemu.
Too old?, I am confused: both host and guest kernels are 2.6.33!
I built KVM against the 2.6.30 headers though.
>
> It may also be that glibc is emulating preadv, incorrectly.
Not sure how to do that.
>
> Antoine, can you check this?  ltrace may help,
This the only part that looked slightly relevant:
[pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0
[pid 26883] malloc(48)                           = 0x2a38d60
[pid 26883] memset(0x2a38d60, '\000', 48)        = 0x2a38d60
[pid 26883] open64("./vm/media_fs", 540674, 00)  = 12
[pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0
[pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600

> or run 'strings libc.so | grep pread'.
>
strings /lib/libc.so.6 | grep pread
preadv
preadv64
pread
__pread64_chk
__pread64
__pread_chk

Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine) [SOLVED]

[Antoine Martin]

On 03/08/2010 12:34 AM, Christoph Hellwig wrote:
> On Sun, Mar 07, 2010 at 07:30:06PM +0200, Avi Kivity wrote:
>
>> It may also be that glibc is emulating preadv, incorrectly.
>>
> I've done a quick audit of all pathes leading to pread and all seem
> to align correctly.  So either a broken glibc emulation or something
> else outside the block layer seems likely.
>
>
>> Antoine, can you check this?  ltrace may help, or run 'strings libc.so |
>> grep pread'.
>>
> Or just add an
>
> #undef CONFIG_PREADV
>
> just before the first
>
> #ifdef CONFIG_PREADV
>
> in posix-aio-compat.c and see if that works.
>
It does indeed! qemu-kvm-0.12.3 is now seeing my partition again. woohoo!
So, PREAD makes it break... how, where? What does this mean?

Thanks a lot!
Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 03/07/2010 08:01 PM, Antoine Martin wrote:
> On 03/08/2010 12:30 AM, Avi Kivity wrote:
>> On 03/07/2010 07:21 PM, Christoph Hellwig wrote:
>>> On Sun, Mar 07, 2010 at 07:18:40PM +0200, Avi Kivity wrote:
>>>>> The only things that stands out is this before the "read failed" 
>>>>> message:
>>>>> [pid  9098] lseek(12, 0, SEEK_END)      = 1321851815424
>>>>> [pid  9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid
>>>>> argument)
>>>>>
>>>> The buffer is unaligned here, yet the file was opened with O_DIRECT
>>>> (cache=none).  This is strange, since alignment is not related to disk
>>>> size.
> Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none"
>
> Side question: this is the right thing to do for raw partitions, right?

The rightest.

>>> The other interesting thing is that it's using pread - which means
>>> it's a too old kernel to use preadv and thus a not very well tested
>>> codepath with current qemu.
> Too old?, I am confused: both host and guest kernels are 2.6.33!
> I built KVM against the 2.6.30 headers though.

You need to build qemu against the 2.6.33 headers ('make 
headers-install').  But after we fix this, please.

>>
>> It may also be that glibc is emulating preadv, incorrectly.
> Not sure how to do that.
>>
>> Antoine, can you check this?  ltrace may help,
> This the only part that looked slightly relevant:
> [pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0
> [pid 26883] malloc(48)                           = 0x2a38d60
> [pid 26883] memset(0x2a38d60, '\000', 48)        = 0x2a38d60
> [pid 26883] open64("./vm/media_fs", 540674, 00)  = 12
> [pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0
> [pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600


Where's pread/preadv?  Did you use -f?

>
>> or run 'strings libc.so | grep pread'.
>>
> strings /lib/libc.so.6 | grep pread
> preadv
> preadv64
> pread
> __pread64_chk
> __pread64
> __pread_chk
>

So it does seem glibc emulates preadv.

Perhaps https://bugzilla.redhat.com/show_bug.cgi?id=563103 ?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: raw disks no longer work in latest kvm (kvm-88 was fine) [SOLVED]

[Avi Kivity]

On 03/07/2010 08:43 PM, Antoine Martin wrote:
>>
>>> Antoine, can you check this?  ltrace may help, or run 'strings 
>>> libc.so |
>>> grep pread'.
>>>
>> Or just add an
>>
>> #undef CONFIG_PREADV
>>
>> just before the first
>>
>> #ifdef CONFIG_PREADV
>>
>> in posix-aio-compat.c and see if that works.
>>
>
> It does indeed! qemu-kvm-0.12.3 is now seeing my partition again. woohoo!
> So, PREAD makes it break... how, where? What does this mean?
>

No - preadv emulation in glibc breaks it.  Likely the bug I mentioned in 
the other email on this thread.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

[snip]
>>>> The other interesting thing is that it's using pread - which means
>>>> it's a too old kernel to use preadv and thus a not very well tested
>>>> codepath with current qemu.
>> Too old?, I am confused: both host and guest kernels are 2.6.33!
>> I built KVM against the 2.6.30 headers though.
>
> You need to build qemu against the 2.6.33 headers ('make 
> headers-install').  But after we fix this, please.
OK.
I'm signing off for today, but I will test whatever patches/suggestions 
you send my way.
>
>>>
>>> It may also be that glibc is emulating preadv, incorrectly.
>> Not sure how to do that.
>>>
>>> Antoine, can you check this?  ltrace may help,
>> This the only part that looked slightly relevant:
>> [pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0
>> [pid 26883] malloc(48)                           = 0x2a38d60
>> [pid 26883] memset(0x2a38d60, '\000', 48)        = 0x2a38d60
>> [pid 26883] open64("./vm/media_fs", 540674, 00)  = 12
>> [pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0
>> [pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600
>
>
> Where's pread/preadv?  Did you use -f?
Yes I did.
ltrace -f bash ./vm/start.sh >& trace.log
>>> or run 'strings libc.so | grep pread'.
>>>
>> strings /lib/libc.so.6 | grep pread
>> preadv
>> preadv64
>> pread
>> __pread64_chk
>> __pread64
>> __pread_chk
>>
>
> So it does seem glibc emulates preadv.
>
> Perhaps https://bugzilla.redhat.com/show_bug.cgi?id=563103 ?
This a gentoo box... but yes, this does look similar doesn't it?

Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Asdo]

Avi Kivity wrote:
> On 03/07/2010 08:01 PM, Antoine Martin wrote:
>>
>> Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none"
>>
>> Side question: this is the right thing to do for raw partitions, right?
>
> The rightest.

Isn't cache=writeback now safe on virtio-blk since 2.6.32?
Doesn't it provide better performances?

Also that looks like a raw file to me, not a partition... do they follow 
the same optimization "rule"?

Thank you

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 03/07/2010 09:07 PM, Antoine Martin wrote:
>>>> Antoine, can you check this?  ltrace may help,
>>> This the only part that looked slightly relevant:
>>> [pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0
>>> [pid 26883] malloc(48)                           = 0x2a38d60
>>> [pid 26883] memset(0x2a38d60, '\000', 48)        = 0x2a38d60
>>> [pid 26883] open64("./vm/media_fs", 540674, 00)  = 12
>>> [pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0
>>> [pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600
>>
>>
>> Where's pread/preadv?  Did you use -f?
>
> Yes I did.
> ltrace -f bash ./vm/start.sh >& trace.log

Well, some variant of pread should have shown up.  Maybe it's the ltrace 
-f race.

>>>> or run 'strings libc.so | grep pread'.
>>>>
>>> strings /lib/libc.so.6 | grep pread
>>> preadv
>>> preadv64
>>> pread
>>> __pread64_chk
>>> __pread64
>>> __pread_chk
>>>
>>
>> So it does seem glibc emulates preadv.
>>
>> Perhaps https://bugzilla.redhat.com/show_bug.cgi?id=563103 ?
> This a gentoo box... but yes, this does look similar doesn't it?

What version of glibc do you have installed?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 03/08/2010 02:09 AM, Asdo wrote:
> Avi Kivity wrote:
>> On 03/07/2010 08:01 PM, Antoine Martin wrote:
>>>
>>> Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none"
>>>
>>> Side question: this is the right thing to do for raw partitions, right?
>>
>> The rightest.
>
> Isn't cache=writeback now safe on virtio-blk since 2.6.32?
> Doesn't it provide better performances?
I thought it was best to let the guest deal with it? (single cache)
> Also that looks like a raw file to me, not a partition... do they 
> follow the same optimization "rule"?
This is a device (/dev/sdc9) in the chroot, don't let the name fool you ;)

Antoine

>
> Thank you
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 03/07/2010 09:09 PM, Asdo wrote:
> Avi Kivity wrote:
>> On 03/07/2010 08:01 PM, Antoine Martin wrote:
>>>
>>> Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none"
>>>
>>> Side question: this is the right thing to do for raw partitions, right?
>>
>> The rightest.
>
> Isn't cache=writeback now safe on virtio-blk since 2.6.32?

Yes.

> Doesn't it provide better performances?

No.  The guest will do its own caching, so unless the guest is really 
short of memory, you aren't gaining much; and the copying will hurt.

>
> Also that looks like a raw file to me, not a partition... do they 
> follow the same optimization "rule"?

More or less.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 03/08/2010 02:10 AM, Avi Kivity wrote:
> On 03/07/2010 09:07 PM, Antoine Martin wrote:
>>>>> Antoine, can you check this?  ltrace may help,
>>>> This the only part that looked slightly relevant:
>>>> [pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0
>>>> [pid 26883] malloc(48)                           = 0x2a38d60
>>>> [pid 26883] memset(0x2a38d60, '\000', 48)        = 0x2a38d60
>>>> [pid 26883] open64("./vm/media_fs", 540674, 00)  = 12
>>>> [pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0
>>>> [pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600
>>>
>>>
>>> Where's pread/preadv?  Did you use -f?
>>
>> Yes I did.
>> ltrace -f bash ./vm/start.sh >& trace.log
>
> Well, some variant of pread should have shown up.  Maybe it's the 
> ltrace -f race.
FYI: qemu booted fully under strace, but not under ltrace...

>>>>> or run 'strings libc.so | grep pread'.
>>>>>
>>>> strings /lib/libc.so.6 | grep pread
>>>> preadv
>>>> preadv64
>>>> pread
>>>> __pread64_chk
>>>> __pread64
>>>> __pread_chk
>>>>
>>>
>>> So it does seem glibc emulates preadv.
>>>
>>> Perhaps https://bugzilla.redhat.com/show_bug.cgi?id=563103 ?
>> This a gentoo box... but yes, this does look similar doesn't it?
>
> What version of glibc do you have installed?
Latest stable:
sys-devel/gcc-4.3.4
sys-libs/glibc-2.10.1-r1

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 03/07/2010 09:13 PM, Antoine Martin wrote:
>> What version of glibc do you have installed?
>
> Latest stable:
> sys-devel/gcc-4.3.4
> sys-libs/glibc-2.10.1-r1
>

$ git show glibc-2.10~108 | head
commit e109c6124fe121618e42ba882e2a0af6e97b8efc
Author: Ulrich Drepper <drepper@redhat.com>
Date:   Fri Apr 3 19:57:16 2009 +0000

     * misc/Makefile (routines): Add preadv, preadv64, pwritev, pwritev64.

         * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 for
         GLIBC_2.10.
         * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64.
         * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for preadv

You might get away with rebuilding glibc against the 2.6.33 headers.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 03/08/2010 02:17 AM, Avi Kivity wrote:
> On 03/07/2010 09:13 PM, Antoine Martin wrote:
>>> What version of glibc do you have installed?
>>
>> Latest stable:
>> sys-devel/gcc-4.3.4
>> sys-libs/glibc-2.10.1-r1
>>
>
> $ git show glibc-2.10~108 | head
> commit e109c6124fe121618e42ba882e2a0af6e97b8efc
> Author: Ulrich Drepper <drepper@redhat.com>
> Date:   Fri Apr 3 19:57:16 2009 +0000
>
>     * misc/Makefile (routines): Add preadv, preadv64, pwritev, pwritev64.
>
>         * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 for
>         GLIBC_2.10.
>         * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64.
>         * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for 
> preadv
>
> You might get away with rebuilding glibc against the 2.6.33 headers.
>
The latest kernel headers available in gentoo (and they're masked unstable):
sys-kernel/linux-headers-2.6.32

So I think I will just keep using Christoph's patch until .33 hits portage.
Unless there's any reason not to? I would rather keep my system "clean".
I can try it though, if that helps you clear things up?

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 03/07/2010 09:25 PM, Antoine Martin wrote:
> On 03/08/2010 02:17 AM, Avi Kivity wrote:
>> On 03/07/2010 09:13 PM, Antoine Martin wrote:
>>>> What version of glibc do you have installed?
>>>
>>> Latest stable:
>>> sys-devel/gcc-4.3.4
>>> sys-libs/glibc-2.10.1-r1
>>>
>>
>> $ git show glibc-2.10~108 | head
>> commit e109c6124fe121618e42ba882e2a0af6e97b8efc
>> Author: Ulrich Drepper <drepper@redhat.com>
>> Date:   Fri Apr 3 19:57:16 2009 +0000
>>
>>     * misc/Makefile (routines): Add preadv, preadv64, pwritev, 
>> pwritev64.
>>
>>         * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 for
>>         GLIBC_2.10.
>>         * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64.
>>         * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for 
>> preadv
>>
>> You might get away with rebuilding glibc against the 2.6.33 headers.
>>
> The latest kernel headers available in gentoo (and they're masked 
> unstable):
> sys-kernel/linux-headers-2.6.32
>
> So I think I will just keep using Christoph's patch until .33 hits 
> portage.
> Unless there's any reason not to? I would rather keep my system "clean".
> I can try it though, if that helps you clear things up?

preadv/pwritev was actually introduced in 2.6.30.  Perhaps you last 
build glibc before that?  If so, a rebuild may be all that's necessary.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Anthony Liguori]

On 03/07/2010 10:21 AM, Avi Kivity wrote:
> On 03/07/2010 12:00 PM, Christoph Hellwig wrote:
>>
>>> I can only guess that the info collected so far is not sufficient to
>>> understand what's going on: except of "I/O error writing block NNN"
>>> we does not have anything at all.  So it's impossible to know where
>>> the problem is.
>> Actually it is, and the bug has been fixed long ago in:
>>
>> commit e2a305fb13ff0f5cf6ff805555aaa90a5ed5954c
>> Author: Christoph Hellwig<hch@lst.de>
>> Date:   Tue Jan 26 14:49:08 2010 +0100
>>
>>      block: avoid creating too large iovecs in multiwrite_merge
>>
>>
>> I've asked for it be added to the -stable series but that hasn't
>> happened so far.
>
> Anthony, this looks critical.
>

It's in stable now.  Sounds like a good time to do a 0.12.4.

Regards,

Anthony Liguori

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 03/08/2010 02:35 AM, Avi Kivity wrote:
> On 03/07/2010 09:25 PM, Antoine Martin wrote:
>> On 03/08/2010 02:17 AM, Avi Kivity wrote:
>>> On 03/07/2010 09:13 PM, Antoine Martin wrote:
>>>>> What version of glibc do you have installed?
>>>>
>>>> Latest stable:
>>>> sys-devel/gcc-4.3.4
>>>> sys-libs/glibc-2.10.1-r1
>>>>
>>>
>>> $ git show glibc-2.10~108 | head
>>> commit e109c6124fe121618e42ba882e2a0af6e97b8efc
>>> Author: Ulrich Drepper <drepper@redhat.com>
>>> Date:   Fri Apr 3 19:57:16 2009 +0000
>>>
>>>     * misc/Makefile (routines): Add preadv, preadv64, pwritev, 
>>> pwritev64.
>>>
>>>         * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 
>>> for
>>>         GLIBC_2.10.
>>>         * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64.
>>>         * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for 
>>> preadv
>>>
>>> You might get away with rebuilding glibc against the 2.6.33 headers.
>>>
>> The latest kernel headers available in gentoo (and they're masked 
>> unstable):
>> sys-kernel/linux-headers-2.6.32
>>
>> So I think I will just keep using Christoph's patch until .33 hits 
>> portage.
>> Unless there's any reason not to? I would rather keep my system "clean".
>> I can try it though, if that helps you clear things up?
>
> preadv/pwritev was actually introduced in 2.6.30.  Perhaps you last 
> build glibc before that?  If so, a rebuild may be all that's necessary.
>
To be certain, I've rebuilt qemu-kvm against:
linux-headers-2.6.33 + glibc-2.10.1-r1 (both freshly built)
And still no go!
I'm still having to use the patch which disables preadv unconditionally...

Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 03/13/2010 11:51 AM, Antoine Martin wrote:
>> preadv/pwritev was actually introduced in 2.6.30.  Perhaps you last 
>> build glibc before that?  If so, a rebuild may be all that's necessary.
>>
>
> To be certain, I've rebuilt qemu-kvm against:
> linux-headers-2.6.33 + glibc-2.10.1-r1 (both freshly built)
> And still no go!
> I'm still having to use the patch which disables preadv 
> unconditionally...

What does strace show?  Is the kernel's preadv called?

Maybe you have a glibc that has broken emulated preadv and no kernel 
preadv support.

-- 
error compiling committee.c: too many arguments to function

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

Antoine Martin wrote:
> On 03/08/2010 02:35 AM, Avi Kivity wrote:
>> On 03/07/2010 09:25 PM, Antoine Martin wrote:
>>> On 03/08/2010 02:17 AM, Avi Kivity wrote:
>>>> On 03/07/2010 09:13 PM, Antoine Martin wrote:
>>>>>> What version of glibc do you have installed?
>>>>>
>>>>> Latest stable:
>>>>> sys-devel/gcc-4.3.4
>>>>> sys-libs/glibc-2.10.1-r1
>>>>>
>>>>
>>>> $ git show glibc-2.10~108 | head
>>>> commit e109c6124fe121618e42ba882e2a0af6e97b8efc
>>>> Author: Ulrich Drepper <drepper@redhat.com>
>>>> Date:   Fri Apr 3 19:57:16 2009 +0000
>>>>
>>>>     * misc/Makefile (routines): Add preadv, preadv64, pwritev,
>>>> pwritev64.
>>>>
>>>>         * misc/Versions: Export preadv, preadv64, pwritev, pwritev64
>>>> for
>>>>         GLIBC_2.10.
>>>>         * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64.
>>>>         * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for
>>>> preadv
>>>>
>>>> You might get away with rebuilding glibc against the 2.6.33 headers.
>>>>
>>> The latest kernel headers available in gentoo (and they're masked
>>> unstable):
>>> sys-kernel/linux-headers-2.6.32
>>>
>>> So I think I will just keep using Christoph's patch until .33 hits
>>> portage.
>>> Unless there's any reason not to? I would rather keep my system "clean".
>>> I can try it though, if that helps you clear things up?
>>
>> preadv/pwritev was actually introduced in 2.6.30.  Perhaps you last
>> build glibc before that?  If so, a rebuild may be all that's necessary.
>>
> To be certain, I've rebuilt qemu-kvm against:
> linux-headers-2.6.33 + glibc-2.10.1-r1 (both freshly built)
> And still no go!
> I'm still having to use the patch which disables preadv unconditionally...

Better late than never, here's the relevant part of the strace (for the
unpatched case where it fails):

stat("./fs", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 41), ...}) = 0
open("./fs", O_RDWR|O_DIRECT|O_CLOEXEC) = 12

lseek(12, 0, SEEK_END)                  = 1321851815424
[pid 31266] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31266] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31266] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31266] lseek(12, 0, SEEK_SET)      = 0
[pid 31266] read(12,
"\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"...,
512) = 512
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12, "iQ\35
\271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 4096,
1321851805696) = 4096
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31273] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31271] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31331] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31294] pread(12, "iQ\35
\271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 512,
1321851805696) = 512
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31298] pread(12,
"\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"...,
512, 0) = 512
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31307] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31295] pread(12,
"\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"...,
2048, 0) = 2048
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31330] pread(12,
"=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"...,
512, 1321851748352) = 512
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31331] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31294] pread(12,
"\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"...,
512, 0) = 512
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
[pid 31298] pread(12,  <unfinished ...>
[pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424



Antoine

> 
> Antoine
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

Bump.

Now that qemu is less likely to eat my data, " *[Qemu-devel] [PATCH 4/8] 
block: fix sector comparism in*"
http://marc.info/?l=qemu-devel&m=127436114712437

I thought I would try using the raw 1.5TB partition again with KVM, 
still no go.
I am still having to use:

#undef CONFIG_PREADV

Host and guest kernel version is 2.6.34, headers 2.6.33, glibc 2.10.1-r1
qemu-kvm 0.12.4 + patch above.

Who do I need to bug? glibc? kvm?

Thanks
Antoine


On 04/09/2010 05:00 AM, Antoine Martin wrote:
>
> Antoine Martin wrote:
>    
>> On 03/08/2010 02:35 AM, Avi Kivity wrote:
>>      
>>> On 03/07/2010 09:25 PM, Antoine Martin wrote:
>>>        
>>>> On 03/08/2010 02:17 AM, Avi Kivity wrote:
>>>>          
>>>>> On 03/07/2010 09:13 PM, Antoine Martin wrote:
>>>>>            
>>>>>>> What version of glibc do you have installed?
>>>>>>>                
>>>>>> Latest stable:
>>>>>> sys-devel/gcc-4.3.4
>>>>>> sys-libs/glibc-2.10.1-r1
>>>>>>
>>>>>>              
>>>>> $ git show glibc-2.10~108 | head
>>>>> commit e109c6124fe121618e42ba882e2a0af6e97b8efc
>>>>> Author: Ulrich Drepper<drepper@redhat.com>
>>>>> Date:   Fri Apr 3 19:57:16 2009 +0000
>>>>>
>>>>>      * misc/Makefile (routines): Add preadv, preadv64, pwritev,
>>>>> pwritev64.
>>>>>
>>>>>          * misc/Versions: Export preadv, preadv64, pwritev, pwritev64
>>>>> for
>>>>>          GLIBC_2.10.
>>>>>          * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64.
>>>>>          * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for
>>>>> preadv
>>>>>
>>>>> You might get away with rebuilding glibc against the 2.6.33 headers.
>>>>>
>>>>>            
>>>> The latest kernel headers available in gentoo (and they're masked
>>>> unstable):
>>>> sys-kernel/linux-headers-2.6.32
>>>>
>>>> So I think I will just keep using Christoph's patch until .33 hits
>>>> portage.
>>>> Unless there's any reason not to? I would rather keep my system "clean".
>>>> I can try it though, if that helps you clear things up?
>>>>          
>>> preadv/pwritev was actually introduced in 2.6.30.  Perhaps you last
>>> build glibc before that?  If so, a rebuild may be all that's necessary.
>>>
>>>        
>> To be certain, I've rebuilt qemu-kvm against:
>> linux-headers-2.6.33 + glibc-2.10.1-r1 (both freshly built)
>> And still no go!
>> I'm still having to use the patch which disables preadv unconditionally...
>>      
> Better late than never, here's the relevant part of the strace (for the
> unpatched case where it fails):
>
> stat("./fs", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 41), ...}) = 0
> open("./fs", O_RDWR|O_DIRECT|O_CLOEXEC) = 12
>
> lseek(12, 0, SEEK_END)                  = 1321851815424
> [pid 31266] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31266] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31266] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31266] lseek(12, 0, SEEK_SET)      = 0
> [pid 31266] read(12,
> "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"...,
> 512) = 512
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12, "iQ\35
> \271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 4096,
> 1321851805696) = 4096
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31273] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31271] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31331] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31294] pread(12, "iQ\35
> \271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 512,
> 1321851805696) = 512
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31298] pread(12,
> "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"...,
> 512, 0) = 512
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31307] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31295] pread(12,
> "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"...,
> 2048, 0) = 2048
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31330] pread(12,
> "=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"...,
> 512, 1321851748352) = 512
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31331] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31294] pread(12,
> "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"...,
> 512, 0) = 512
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
> [pid 31298] pread(12,<unfinished ...>
> [pid 31267] lseek(12, 0, SEEK_END)      = 1321851815424
>
>
>
> Antoine
>
>    
>> Antoine
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>      
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Michael Tokarev]

22.05.2010 14:44, Antoine Martin wrote:
> Bump.
>
> Now that qemu is less likely to eat my data, " *[Qemu-devel] [PATCH 4/8]
> block: fix sector comparism in*"
> http://marc.info/?l=qemu-devel&m=127436114712437
>
> I thought I would try using the raw 1.5TB partition again with KVM,
> still no go.

Hm.  I don't have so much diskspace (my largest is 750Gb, whole disk),
but I created 1.5Tb sparse lvm volume.  It appears to work for me,
even 32bit version of qemu-kvm-0.12.4 (with the mentioned patch applied).

> I am still having to use:
>
> #undef CONFIG_PREADV
>
> Host and guest kernel version is 2.6.34, headers 2.6.33, glibc 2.10.1-r1
> qemu-kvm 0.12.4 + patch above.

eglibc-2.10.2-6, kernel #2.6.34.0-amd64,
kernel headers 2.6.32-11~bpo50+1
(debian)

> Who do I need to bug? glibc? kvm?

are you running 32bit userspace and 64bit kernel
by a chance?  If yes that's a kernel prob, see
http://thread.gmane.org/gmane.linux.kernel.aio.general/2891
(the fix will be in 2.6.35 hopefully, now it's in
Andrew Morton's tree).

If not, well, I don't know ;)

/mjt

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 05/22/2010 06:17 PM, Michael Tokarev wrote:
> 22.05.2010 14:44, Antoine Martin wrote:
>> Bump.
>>
>> Now that qemu is less likely to eat my data, " *[Qemu-devel] [PATCH 4/8]
>> block: fix sector comparism in*"
>> http://marc.info/?l=qemu-devel&m=127436114712437
>>
>> I thought I would try using the raw 1.5TB partition again with KVM,
>> still no go.
>
> Hm.  I don't have so much diskspace (my largest is 750Gb, whole disk),
> but I created 1.5Tb sparse lvm volume.  It appears to work for me,
> even 32bit version of qemu-kvm-0.12.4 (with the mentioned patch applied).
>
>> I am still having to use:
>>
>> #undef CONFIG_PREADV
>>
>> Host and guest kernel version is 2.6.34, headers 2.6.33, glibc 2.10.1-r1
>> qemu-kvm 0.12.4 + patch above.
>
> eglibc-2.10.2-6, kernel #2.6.34.0-amd64,
> kernel headers 2.6.32-11~bpo50+1
> (debian)
>
>> Who do I need to bug? glibc? kvm?
>
> are you running 32bit userspace and 64bit kernel
> by a chance?  If yes that's a kernel prob, see
> http://thread.gmane.org/gmane.linux.kernel.aio.general/2891
> (the fix will be in 2.6.35 hopefully, now it's in
> Andrew Morton's tree).
>
> If not, well, I don't know ;)
I'm not: 64-bit host and 64-bit guest.

Thanks anyway.
Antoine
>
> /mjt
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 05/22/2010 06:35 PM, Antoine Martin wrote:
> On 05/22/2010 06:17 PM, Michael Tokarev wrote:
>> 22.05.2010 14:44, Antoine Martin wrote:
>>> Bump.
>>>
>>> Now that qemu is less likely to eat my data, " *[Qemu-devel] [PATCH 
>>> 4/8]
>>> block: fix sector comparism in*"
>>> http://marc.info/?l=qemu-devel&m=127436114712437
>>>
>>> I thought I would try using the raw 1.5TB partition again with KVM,
>>> still no go.
>>
>> Hm.  I don't have so much diskspace (my largest is 750Gb, whole disk),
>> but I created 1.5Tb sparse lvm volume.  It appears to work for me,
>> even 32bit version of qemu-kvm-0.12.4 (with the mentioned patch 
>> applied).
>>
>>> I am still having to use:
>>>
>>> #undef CONFIG_PREADV
>>>
>>> Host and guest kernel version is 2.6.34, headers 2.6.33, glibc 
>>> 2.10.1-r1
>>> qemu-kvm 0.12.4 + patch above.
>>
>> eglibc-2.10.2-6, kernel #2.6.34.0-amd64,
>> kernel headers 2.6.32-11~bpo50+1
>> (debian)
>>
>>> Who do I need to bug? glibc? kvm?
>>
>> are you running 32bit userspace and 64bit kernel
>> by a chance?  If yes that's a kernel prob, see
>> http://thread.gmane.org/gmane.linux.kernel.aio.general/2891
>> (the fix will be in 2.6.35 hopefully, now it's in
>> Andrew Morton's tree).
>>
>> If not, well, I don't know ;)
> I'm not: 64-bit host and 64-bit guest.
Just to be sure, I've tested that patch and still no joy:
   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
   /dev/vdc: read failed after 0 of 2048 at 0: Input/output error
pread is still broken on this big raw disk.

Antoine



>
> Thanks anyway.
> Antoine
>>
>> /mjt
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 05/23/2010 11:53 AM, Antoine Martin wrote:
> I'm not: 64-bit host and 64-bit guest.
> Just to be sure, I've tested that patch and still no joy:
>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>   /dev/vdc: read failed after 0 of 2048 at 0: Input/output error
> pread is still broken on this big raw disk.
>

Can you summarize your environment and reproducer again?  The thread is 
so long it is difficult to understand exactly what you are testing.

-- 
error compiling committee.c: too many arguments to function

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 05/23/2010 06:57 PM, Avi Kivity wrote:
> On 05/23/2010 11:53 AM, Antoine Martin wrote:
>> I'm not: 64-bit host and 64-bit guest.
>> Just to be sure, I've tested that patch and still no joy:
>>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>>   /dev/vdc: read failed after 0 of 2048 at 0: Input/output error
>> pread is still broken on this big raw disk.
>>
>
> Can you summarize your environment and reproducer again?  The thread 
> is so long it is difficult to understand exactly what you are testing.
>
Sure thing:

Description of the problem:
A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my 
case), this fails with pread enabled, works with it disabled.

What has been tested:
Host has had 2.6.31.x and now 2.6.34 - with no improvement.
Guest has had numerous kernel versions and patches applied for testing 
(from 2.6.31.x to 2.6.34) - none made any difference. (although the 
recent patch stopped the large partition from getting corrupted by the 
merged requests bug.. yay!)
It was once thought that glibc needed to be rebuilt against newer kernel 
headers, done that too, and rebuilt qemu-kvm afterwards. (was 2.6.28 
then 2.6.31 and now 2.6.33)

What has been reported: straces, kernel error messages as above.
Here is the qemu command line (sanitized - removed the virtual disks 
which work fine and network config):
qemu -clock dynticks -m 256 -L ./ -kernel ./bzImage-2.6.34-aio -append 
"root=/dev/vda" -nographic -drive file=/dev/sda9,if=virtio,cache=none

Let me know if you need any other details. I can also arrange ssh access 
to the system if needed.

Cheers
Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 05/23/2010 05:07 PM, Antoine Martin wrote:
> On 05/23/2010 06:57 PM, Avi Kivity wrote:
>> On 05/23/2010 11:53 AM, Antoine Martin wrote:
>>> I'm not: 64-bit host and 64-bit guest.
>>> Just to be sure, I've tested that patch and still no joy:
>>>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>>>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>>>   /dev/vdc: read failed after 0 of 2048 at 0: Input/output error
>>> pread is still broken on this big raw disk.
>>>
>>
>> Can you summarize your environment and reproducer again?  The thread 
>> is so long it is difficult to understand exactly what you are testing.
>>
> Sure thing:
>
> Description of the problem:
> A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my 
> case), this fails with pread enabled, works with it disabled.

Did you mean: preadv?

>
> What has been tested:
> Host has had 2.6.31.x and now 2.6.34 - with no improvement.
> Guest has had numerous kernel versions and patches applied for testing 
> (from 2.6.31.x to 2.6.34) - none made any difference. (although the 
> recent patch stopped the large partition from getting corrupted by the 
> merged requests bug.. yay!)
> It was once thought that glibc needed to be rebuilt against newer 
> kernel headers, done that too, and rebuilt qemu-kvm afterwards. (was 
> 2.6.28 then 2.6.31 and now 2.6.33)
>
> What has been reported: straces, kernel error messages as above.
> Here is the qemu command line (sanitized - removed the virtual disks 
> which work fine and network config):
> qemu -clock dynticks -m 256 -L ./ -kernel ./bzImage-2.6.34-aio -append 
> "root=/dev/vda" -nographic -drive file=/dev/sda9,if=virtio,cache=none
>
> Let me know if you need any other details. I can also arrange ssh 
> access to the system if needed.

And: 64-bit host kernel, 64-bit host userspace, yes?

Does aio=native help?  How about if=ide?

-- 
error compiling committee.c: too many arguments to function

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 05/23/2010 09:18 PM, Avi Kivity wrote:
> On 05/23/2010 05:07 PM, Antoine Martin wrote:
>> On 05/23/2010 06:57 PM, Avi Kivity wrote:
>>> On 05/23/2010 11:53 AM, Antoine Martin wrote:
>>>> I'm not: 64-bit host and 64-bit guest.
>>>> Just to be sure, I've tested that patch and still no joy:
>>>>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>>>>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>>>>   /dev/vdc: read failed after 0 of 2048 at 0: Input/output error
>>>> pread is still broken on this big raw disk.
>>>>
>>>
>>> Can you summarize your environment and reproducer again?  The thread 
>>> is so long it is difficult to understand exactly what you are testing.
>>>
>> Sure thing:
>>
>> Description of the problem:
>> A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my 
>> case), this fails with pread enabled, works with it disabled.
>
> Did you mean: preadv?
Yes, here's what makes it work ok (as suggested by Christoph earlier in 
the thread) in posix-aio-compat.c:
#undef CONFIG_PREADV
#ifdef CONFIG_PREADV
static int preadv_present = 1;
#else
static int preadv_present = 0;
#endif
>>
>> What has been tested:
>> Host has had 2.6.31.x and now 2.6.34 - with no improvement.
>> Guest has had numerous kernel versions and patches applied for 
>> testing (from 2.6.31.x to 2.6.34) - none made any difference. 
>> (although the recent patch stopped the large partition from getting 
>> corrupted by the merged requests bug.. yay!)
>> It was once thought that glibc needed to be rebuilt against newer 
>> kernel headers, done that too, and rebuilt qemu-kvm afterwards. (was 
>> 2.6.28 then 2.6.31 and now 2.6.33)
>>
>> What has been reported: straces, kernel error messages as above.
>> Here is the qemu command line (sanitized - removed the virtual disks 
>> which work fine and network config):
>> qemu -clock dynticks -m 256 -L ./ -kernel ./bzImage-2.6.34-aio 
>> -append "root=/dev/vda" -nographic -drive 
>> file=/dev/sda9,if=virtio,cache=none
>>
>> Let me know if you need any other details. I can also arrange ssh 
>> access to the system if needed.
>
> And: 64-bit host kernel, 64-bit host userspace, yes?
Yes. (and 64-bit guest/userspace too)
>
> Does aio=native help?
It does indeed!
>   How about if=ide?
Will test with another kernel and report back (this one doesn't have any 
non-virtio drivers)

Thanks
Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

>>   How about if=ide?
> Will test with another kernel and report back (this one doesn't have 
> any non-virtio drivers)
Can anyone tell me which kernel module I need for "if=ide"? Google was 
no help here.
(before I include dozens of unnecessary modules in my slimmed down and 
non modular kernel)

Thanks
Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 05/23/2010 05:53 PM, Antoine Martin wrote:
>
>>>   How about if=ide?
>> Will test with another kernel and report back (this one doesn't have 
>> any non-virtio drivers)
> Can anyone tell me which kernel module I need for "if=ide"? Google was 
> no help here.
> (before I include dozens of unnecessary modules in my slimmed down and 
> non modular kernel)

ATA_PIIX probably.

-- 
error compiling committee.c: too many arguments to function

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 05/23/2010 09:43 PM, Antoine Martin wrote:
> On 05/23/2010 09:18 PM, Avi Kivity wrote:
>> On 05/23/2010 05:07 PM, Antoine Martin wrote:
>>> On 05/23/2010 06:57 PM, Avi Kivity wrote:
>>>> On 05/23/2010 11:53 AM, Antoine Martin wrote:
>>>>> I'm not: 64-bit host and 64-bit guest.
>>>>> Just to be sure, I've tested that patch and still no joy:
>>>>>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>>>>>   /dev/vdc: read failed after 0 of 512 at 0: Input/output error
>>>>>   /dev/vdc: read failed after 0 of 2048 at 0: Input/output error
>>>>> pread is still broken on this big raw disk.
>>>>>
>>>>
>>>> Can you summarize your environment and reproducer again?  The 
>>>> thread is so long it is difficult to understand exactly what you 
>>>> are testing.
>>>>
>>> Sure thing:
>>>
>>> Description of the problem:
>>> A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my 
>>> case), this fails with pread enabled, works with it disabled.
>>
>> Did you mean: preadv?
> Yes, here's what makes it work ok (as suggested by Christoph earlier 
> in the thread) in posix-aio-compat.c:
> #undef CONFIG_PREADV
> #ifdef CONFIG_PREADV
> static int preadv_present = 1;
> #else
> static int preadv_present = 0;
> #endif
>>>
>>> What has been tested:
>>> Host has had 2.6.31.x and now 2.6.34 - with no improvement.
>>> Guest has had numerous kernel versions and patches applied for 
>>> testing (from 2.6.31.x to 2.6.34) - none made any difference. 
>>> (although the recent patch stopped the large partition from getting 
>>> corrupted by the merged requests bug.. yay!)
>>> It was once thought that glibc needed to be rebuilt against newer 
>>> kernel headers, done that too, and rebuilt qemu-kvm afterwards. (was 
>>> 2.6.28 then 2.6.31 and now 2.6.33)
>>>
>>> What has been reported: straces, kernel error messages as above.
>>> Here is the qemu command line (sanitized - removed the virtual disks 
>>> which work fine and network config):
>>> qemu -clock dynticks -m 256 -L ./ -kernel ./bzImage-2.6.34-aio 
>>> -append "root=/dev/vda" -nographic -drive 
>>> file=/dev/sda9,if=virtio,cache=none
>>>
>>> Let me know if you need any other details. I can also arrange ssh 
>>> access to the system if needed.
>>
>> And: 64-bit host kernel, 64-bit host userspace, yes?
> Yes. (and 64-bit guest/userspace too)
>>
>> Does aio=native help?
> It does indeed!
>>   How about if=ide?
> Will test with another kernel and report back (this one doesn't have 
> any non-virtio drivers)
That also works fine.

I guess this tells you that the problem is confined to 
raw-disk+virtio+aio-threaded

I'm switching all my VMs to aio=native, but feel free to send patches if 
you want me to re-test aio=threaded

Thanks
Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Avi Kivity]

On 05/23/2010 05:43 PM, Antoine Martin wrote:
>>> Description of the problem:
>>> A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my 
>>> case), this fails with pread enabled, works with it disabled.
>>
>> Did you mean: preadv?
>
> Yes, here's what makes it work ok (as suggested by Christoph earlier 
> in the thread) in posix-aio-compat.c:
> #undef CONFIG_PREADV
> #ifdef CONFIG_PREADV
> static int preadv_present = 1;
> #else
> static int preadv_present = 0;
> #endif

When preadv_present=1, does strace -fF show preadv being called?

If so, the kernel's preadv() is broken.

If not, glibc's preadv() emulation is broken.

>>
>> Does aio=native help?
> It does indeed!

That's recommended anyway on raw partitions with cache=off.

-- 
error compiling committee.c: too many arguments to function

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 05/23/2010 10:12 PM, Avi Kivity wrote:
> On 05/23/2010 05:43 PM, Antoine Martin wrote:
>>>> Description of the problem:
>>>> A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my 
>>>> case), this fails with pread enabled, works with it disabled.
>>>
>>> Did you mean: preadv?
>>
>> Yes, here's what makes it work ok (as suggested by Christoph earlier 
>> in the thread) in posix-aio-compat.c:
>> #undef CONFIG_PREADV
>> #ifdef CONFIG_PREADV
>> static int preadv_present = 1;
>> #else
>> static int preadv_present = 0;
>> #endif
>
> When preadv_present=1, does strace -fF show preadv being called?
There were some pread() but no preadv() in the strace BTW.
> If so, the kernel's preadv() is broken.
>
> If not, glibc's preadv() emulation is broken.
OK, I found what's causing it: chroot
I was testing all this in a chroot, I always do and I didn't think of 
mentioning it, sorry about that.
Running it non-chrooted works in all cases, including aio!=native

Why does it work in a chroot for the other options (aio=native, if=ide, 
etc) but not for aio!=native??
Looks like I am misunderstanding the semantics of chroot...

>>>
>>> Does aio=native help?
>> It does indeed!
>
> That's recommended anyway on raw partitions with cache=off.
>
What about non-raw partitions (or/and with cache=on)?

Antoine

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Stefan Hajnoczi]

On Sun, May 23, 2010 at 5:18 PM, Antoine Martin <antoine@nagafix.co.uk> wrote:
> Why does it work in a chroot for the other options (aio=native, if=ide, etc)
> but not for aio!=native??
> Looks like I am misunderstanding the semantics of chroot...

It might not be the chroot() semantics but the environment inside that
chroot, like the glibc.  Have you compared strace inside and outside
the chroot?

Stefan

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Antoine Martin]

On 05/24/2010 12:47 AM, Stefan Hajnoczi wrote:
> On Sun, May 23, 2010 at 5:18 PM, Antoine Martin <antoine@nagafix.co.uk> wrote:
>> Why does it work in a chroot for the other options (aio=native, if=ide, etc)
>> but not for aio!=native??
>> Looks like I am misunderstanding the semantics of chroot...
> 
> It might not be the chroot() semantics but the environment inside that
> chroot, like the glibc.  Have you compared strace inside and outside
> the chroot?
Reverting to a static build also fixes the issue: aio=threads works.
Definitely something fishy going on with glibc library loading.
(I've checked glibc, libaio were up to date in the chroot - nothing
blatant in the strace)

Can someone explain the aio options?
All I can find is this:
# qemu-system-x86_64 -h | grep -i aio
       [,addr=A][,id=name][,aio=threads|native]
I assume it means the aio=threads emulates the kernel's aio with
separate threads? And is therefore likely to be slower, right?
Is there a reason why aio=native is not the default? Shouldn't
aio=threads be the fallback?

Cheers
Antoine



> 
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Stefan Hajnoczi]

On Sat, May 29, 2010 at 10:42 AM, Antoine Martin <antoine@nagafix.co.uk> wrote:
> Can someone explain the aio options?
> All I can find is this:
> # qemu-system-x86_64 -h | grep -i aio
>       [,addr=A][,id=name][,aio=threads|native]
> I assume it means the aio=threads emulates the kernel's aio with
> separate threads? And is therefore likely to be slower, right?
> Is there a reason why aio=native is not the default? Shouldn't
> aio=threads be the fallback?

aio=threads uses posix-aio-compat.c, a POSIX AIO-like implementation
using a thread pool.  Each thread services queued I/O requests using
blocking syscalls (e.g. preadv()/pwritev()).

aio=native uses Linux libaio, the native (non-POSIX) AIO interface.

I would expect that aio=native is faster but benchmarks show that this
isn't true for all workloads.

Stefan

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Christoph Hellwig]

On Sat, May 29, 2010 at 04:42:59PM +0700, Antoine Martin wrote:
> Can someone explain the aio options?
> All I can find is this:
> # qemu-system-x86_64 -h | grep -i aio
>        [,addr=A][,id=name][,aio=threads|native]
> I assume it means the aio=threads emulates the kernel's aio with
> separate threads? And is therefore likely to be slower, right?
> Is there a reason why aio=native is not the default? Shouldn't
> aio=threads be the fallback?

The kernel AIO support is unfortunately not a very generic API.
It only supports O_DIRECT I/O (cache=none for qemu), and if used on
a filesystems it might still block if we need to perform block
allocations.  We could probably make it the default for block devices,
but I'm not a big fan of these kind of conditional defaults.

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Christoph Hellwig]

On Sat, May 29, 2010 at 10:55:18AM +0100, Stefan Hajnoczi wrote:
> I would expect that aio=native is faster but benchmarks show that this
> isn't true for all workloads.

In what benchmark do you see worse results for aio=native compared to
aio=threads?

Re: raw disks no longer work in latest kvm (kvm-88 was fine)

[Stefan Hajnoczi]

On Sat, May 29, 2010 at 11:34 AM, Christoph Hellwig <hch@infradead.org> wrote:
> In what benchmark do you see worse results for aio=native compared to
> aio=threads?

Sequential reads using 4 concurrent dd if=/dev/vdb iflag=direct
of=/dev/null bs=8k processes.  2 vcpu guest with 4 GB RAM, virtio
block devices, cache=none.  Host storage is a striped LVM volume.
Host kernel kvm.git and qemu-kvm.git userspace.

aio=native and aio=threads each run 3 times.

Result: aio=native has 15% lower throughput than aio=threads.

I haven't looked into this so I don't know what is causes these results.

Stefan