* raw disks no longer work in latest kvm (kvm-88 was fine) @ 2010-03-06 20:48 Antoine Martin 2010-03-06 21:28 ` Michael Tokarev 2010-03-07 9:36 ` Gleb Natapov 0 siblings, 2 replies; 51+ messages in thread From: Antoine Martin @ 2010-03-06 20:48 UTC (permalink / raw) To: kvm Hi, With qemu-kvm-0.12.3: ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] [ 1.882843] vdc: [ 2.365154] udev: starting version 146 [ 2.693768] end_request: I/O error, dev vdc, sector 126 [ 2.693772] Buffer I/O error on device vdc, logical block 126 [ 2.693775] Buffer I/O error on device vdc, logical block 127 [ 2.693777] Buffer I/O error on device vdc, logical block 128 [ 2.693779] Buffer I/O error on device vdc, logical block 129 [ 2.693781] Buffer I/O error on device vdc, logical block 130 [ 2.693783] Buffer I/O error on device vdc, logical block 131 [ 2.693785] Buffer I/O error on device vdc, logical block 132 [ 2.693787] Buffer I/O error on device vdc, logical block 133 [ 2.693788] Buffer I/O error on device vdc, logical block 134 [ 2.693814] end_request: I/O error, dev vdc, sector 0 [ 3.144870] end_request: I/O error, dev vdc, sector 0 [ 3.499377] end_request: I/O error, dev vdc, sector 0 [ 3.523247] end_request: I/O error, dev vdc, sector 0 [ 3.547130] end_request: I/O error, dev vdc, sector 0 [ 3.550076] end_request: I/O error, dev vdc, sector 0 Works fine with kvm-88: cp /usr/src/KVM/kvm-88/pc-bios/*bin ./ cp /usr/src/KVM/kvm-88/x86_64-softmmu/qemu-system-x86_64 ./ ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] [ 1.650274] vdc: unknown partition table [ 112.704164] EXT4-fs (vdc): mounted filesystem with ordered data mode I've tried running as root, using the block device directly (as shown above) rather than using a softlink, etc.. Something broke. Host and guest are both running 2.6.33 and latest KVM. Cheers Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-06 20:48 raw disks no longer work in latest kvm (kvm-88 was fine) Antoine Martin @ 2010-03-06 21:28 ` Michael Tokarev 2010-03-07 4:14 ` Antoine Martin 2010-03-07 9:36 ` Gleb Natapov 1 sibling, 1 reply; 51+ messages in thread From: Michael Tokarev @ 2010-03-06 21:28 UTC (permalink / raw) To: Antoine Martin; +Cc: kvm Antoine Martin wrote: > Hi, > > With qemu-kvm-0.12.3: > ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] > [ 1.882843] vdc: > [ 2.365154] udev: starting version 146 > [ 2.693768] end_request: I/O error, dev vdc, sector 126 > [ 2.693772] Buffer I/O error on device vdc, logical block 126 > [ 2.693775] Buffer I/O error on device vdc, logical block 127 > [ 2.693777] Buffer I/O error on device vdc, logical block 128 .. > [ 3.550076] end_request: I/O error, dev vdc, sector 0 > > Works fine with kvm-88: > cp /usr/src/KVM/kvm-88/pc-bios/*bin ./ > cp /usr/src/KVM/kvm-88/x86_64-softmmu/qemu-system-x86_64 ./ > ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] > > [ 1.650274] vdc: unknown partition table > [ 112.704164] EXT4-fs (vdc): mounted filesystem with ordered data mode > > I've tried running as root, using the block device directly (as shown > above) rather than using a softlink, etc.. > > Something broke. > Host and guest are both running 2.6.33 and latest KVM. https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599 ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-06 21:28 ` Michael Tokarev @ 2010-03-07 4:14 ` Antoine Martin 2010-03-07 9:32 ` Michael Tokarev 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-03-07 4:14 UTC (permalink / raw) To: Michael Tokarev, kvm On 03/07/2010 04:28 AM, Michael Tokarev wrote: > Antoine Martin wrote: > >> Hi, >> >> With qemu-kvm-0.12.3: >> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] >> [ 1.882843] vdc: >> [ 2.365154] udev: starting version 146 >> [ 2.693768] end_request: I/O error, dev vdc, sector 126 >> [ 2.693772] Buffer I/O error on device vdc, logical block 126 >> [ 2.693775] Buffer I/O error on device vdc, logical block 127 >> [ 2.693777] Buffer I/O error on device vdc, logical block 128 >> > .. > >> [ 3.550076] end_request: I/O error, dev vdc, sector 0 >> >> Works fine with kvm-88: >> cp /usr/src/KVM/kvm-88/pc-bios/*bin ./ >> cp /usr/src/KVM/kvm-88/x86_64-softmmu/qemu-system-x86_64 ./ >> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] >> >> [ 1.650274] vdc: unknown partition table >> [ 112.704164] EXT4-fs (vdc): mounted filesystem with ordered data mode >> >> I've tried running as root, using the block device directly (as shown >> above) rather than using a softlink, etc.. >> >> Something broke. >> Host and guest are both running 2.6.33 and latest KVM. >> > https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599 > The initial report is almost 8 weeks old! Is data-corruption and data loss somehow less important than the hundreds of patches that have been submitted since?? Or is there a fix somewhere I've missed? Cheers Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 4:14 ` Antoine Martin @ 2010-03-07 9:32 ` Michael Tokarev 2010-03-07 10:00 ` Christoph Hellwig 0 siblings, 1 reply; 51+ messages in thread From: Michael Tokarev @ 2010-03-07 9:32 UTC (permalink / raw) To: Antoine Martin; +Cc: kvm Antoine Martin wrote: [] >> https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599 >> >> > The initial report is almost 8 weeks old! > Is data-corruption and data loss somehow less important than the > hundreds of patches that have been submitted since?? Or is there a fix > somewhere I've missed? I can only guess that the info collected so far is not sufficient to understand what's going on: except of "I/O error writing block NNN" we does not have anything at all. So it's impossible to know where the problem is. Please don't blame on people (me included: i suffer from this same issue just like you). We need something constructive. And by the way, as stated in my comment attached to that bug (I'm user mjtsf at sourceforge), error-behavour=remount-ro avoids data corruption (this is not unique for the issue in question but for general case of i/o errors and filesystem operations). /mjt ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 9:32 ` Michael Tokarev @ 2010-03-07 10:00 ` Christoph Hellwig 2010-03-07 13:12 ` Antoine Martin 2010-03-07 16:21 ` Avi Kivity 0 siblings, 2 replies; 51+ messages in thread From: Christoph Hellwig @ 2010-03-07 10:00 UTC (permalink / raw) To: Michael Tokarev; +Cc: Antoine Martin, kvm On Sun, Mar 07, 2010 at 12:32:38PM +0300, Michael Tokarev wrote: > Antoine Martin wrote: > [] > >> https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599 > >> > >> > > The initial report is almost 8 weeks old! > > Is data-corruption and data loss somehow less important than the > > hundreds of patches that have been submitted since?? Or is there a fix > > somewhere I've missed? > > I can only guess that the info collected so far is not sufficient to > understand what's going on: except of "I/O error writing block NNN" > we does not have anything at all. So it's impossible to know where > the problem is. Actually it is, and the bug has been fixed long ago in: commit e2a305fb13ff0f5cf6ff805555aaa90a5ed5954c Author: Christoph Hellwig <hch@lst.de> Date: Tue Jan 26 14:49:08 2010 +0100 block: avoid creating too large iovecs in multiwrite_merge I've asked for it be added to the -stable series but that hasn't happened so far. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 10:00 ` Christoph Hellwig @ 2010-03-07 13:12 ` Antoine Martin 2010-03-07 13:52 ` Antoine Martin 2010-03-07 16:21 ` Avi Kivity 1 sibling, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-03-07 13:12 UTC (permalink / raw) To: Christoph Hellwig, kvm On 03/07/2010 05:00 PM, Christoph Hellwig wrote: > On Sun, Mar 07, 2010 at 12:32:38PM +0300, Michael Tokarev wrote: > >> Antoine Martin wrote: >> [] >> >>>> https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2933400&group_id=180599 >>>> >>>> >>>> >>> The initial report is almost 8 weeks old! >>> Is data-corruption and data loss somehow less important than the >>> hundreds of patches that have been submitted since?? Or is there a fix >>> somewhere I've missed? >>> >> I can only guess that the info collected so far is not sufficient to >> understand what's going on: except of "I/O error writing block NNN" >> we does not have anything at all. So it's impossible to know where >> the problem is. >> > Actually it is, and the bug has been fixed long ago in: > > commit e2a305fb13ff0f5cf6ff805555aaa90a5ed5954c > Author: Christoph Hellwig<hch@lst.de> > Date: Tue Jan 26 14:49:08 2010 +0100 > > block: avoid creating too large iovecs in multiwrite_merge > Hmmm. Like most of you, I had assumed that the data corruption bug (which I have also encountered on this raw partition as it is well over 1TB) would be the same as the "cannot open drive" bug I have described above, but unfortunately I still cannot use 0.12.3 with raw disks with this patch applied. This is the patch, right?: http://www.mail-archive.com/qemu-devel@nongnu.org/msg24129.html So there is something else at play. And just for the record: 1) kvm-88 works fine *with the exact same setup* 2) I've tried running as root 3) The raw disk mounts fine from the host. So I *know* the problem is with kvm. I wouldn't post to the list without triple checking that. I have also just tested with another raw partition which is much smaller (1GB) and the same thing still occurs: kvm-88 works and qemu-kvm-0.12.3 does not. So I think that it is fair to assume that this new problem is unrelated to the partition size. > I've asked for it be added to the -stable series but that hasn't > happened so far. > If "eating-your-data" doesn't make it to -stable, what does?? Even if it did, I would have thought this kind of bug should warrant a big warning sign somewhere. Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 13:12 ` Antoine Martin @ 2010-03-07 13:52 ` Antoine Martin 2010-03-07 15:15 ` Michael Tokarev 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-03-07 13:52 UTC (permalink / raw) To: Christoph Hellwig, kvm [snip] > > So there is something else at play. And just for the record: > 1) kvm-88 works fine *with the exact same setup* > 2) I've tried running as root > 3) The raw disk mounts fine from the host. > So I *know* the problem is with kvm. I wouldn't post to the list > without triple checking that. > > I have also just tested with another raw partition which is much > smaller (1GB) and the same thing still occurs: kvm-88 works and > qemu-kvm-0.12.3 does not. > So I think that it is fair to assume that this new problem is > unrelated to the partition size. I have narrowed it down to the "io-thread" option: * rebuilding older versions of qemu without "--enable-io-thread" causes the bug (guest cannot open raw partition) * qemu-kvm-0.12.3 cannot be built with "--enable-io-thread" over here: LINK x86_64-softmmu/qemu-system-x86_64 kvm-all.o: In function `qemu_mutex_lock_iothread': /usr/src/KVM/qemu-kvm-0.12.3/qemu-kvm.c:2532: multiple definition of `qemu_mutex_lock_iothread' vl.o:/usr/src/KVM/qemu-kvm-0.12.3/vl.c:3772: first defined here [..] Which I have reported as part of another unsolved issue here: http://www.mail-archive.com/kvm@vger.kernel.org/msg27663.html Why not using the io-thread would prevent qemu from opening the raw partition is beyond me. Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 13:52 ` Antoine Martin @ 2010-03-07 15:15 ` Michael Tokarev 2010-03-07 17:11 ` Antoine Martin 0 siblings, 1 reply; 51+ messages in thread From: Michael Tokarev @ 2010-03-07 15:15 UTC (permalink / raw) To: Antoine Martin; +Cc: kvm, Christoph Hellwig Antoine Martin wrote: > [snip] >> >> So there is something else at play. And just for the record: >> 1) kvm-88 works fine *with the exact same setup* >> 2) I've tried running as root >> 3) The raw disk mounts fine from the host. >> So I *know* the problem is with kvm. I wouldn't post to the list >> without triple checking that. >> >> I have also just tested with another raw partition which is much >> smaller (1GB) and the same thing still occurs: kvm-88 works and >> qemu-kvm-0.12.3 does not. >> So I think that it is fair to assume that this new problem is >> unrelated to the partition size. > I have narrowed it down to the "io-thread" option: > * rebuilding older versions of qemu without "--enable-io-thread" causes > the bug (guest cannot open raw partition) > * qemu-kvm-0.12.3 cannot be built with "--enable-io-thread" over here: > LINK x86_64-softmmu/qemu-system-x86_64 > kvm-all.o: In function `qemu_mutex_lock_iothread': > /usr/src/KVM/qemu-kvm-0.12.3/qemu-kvm.c:2532: multiple definition of > `qemu_mutex_lock_iothread' > vl.o:/usr/src/KVM/qemu-kvm-0.12.3/vl.c:3772: first defined here > [..] > Which I have reported as part of another unsolved issue here: > http://www.mail-archive.com/kvm@vger.kernel.org/msg27663.html > > Why not using the io-thread would prevent qemu from opening the raw > partition is beyond me. Ok, this is in fact different problem, not the one I referred you initially (which was in fact good too, because apparently Christoph solved that bug for me and for other Debian users, thank you!). In your case, recalling your initial email: > With qemu-kvm-0.12.3: > ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] > [ 1.882843] vdc: > [ 2.365154] udev: starting version 146 > [ 2.693768] end_request: I/O error, dev vdc, sector 126 > [ 2.693772] Buffer I/O error on device vdc, logical block 126 > [ 2.693775] Buffer I/O error on device vdc, logical block 127 > [ 2.693777] Buffer I/O error on device vdc, logical block 128 ... the problem happens right at startup, it can't read _anything_ at all from the disk. In my case, the problem is intermittent and happens under high load only, hence the big difference. But anyway, this is something which should be easy to find out. Run kvm under `strace -f' and see how it opens the device, or find out with lsof what filedescriptor corresponds to the file in question (in running kvm instance) and see flags in /proc/$kvm_pid/fdinfo/$fdnum. I guess it can't open the image in read-write mode somehow. By the way, iothread doesn't really work in kvm, as far as I can see. Thanks. /mjt ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 15:15 ` Michael Tokarev @ 2010-03-07 17:11 ` Antoine Martin 2010-03-07 17:18 ` Avi Kivity 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-03-07 17:11 UTC (permalink / raw) To: Michael Tokarev, kvm [snip] > Ok, this is in fact different problem, not the one I referred you > initially (which was in fact good too, because apparently Christoph > solved that bug for me and for other Debian users, thank you!). > Yes, I believe I had this issue too originally, which is why I was testing 0.12.3 to see if it would fix the problem... and I hit this new seemingly unrelated snag. > In your case, recalling your initial email: > >> With qemu-kvm-0.12.3: >> ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] >> [ 1.882843] vdc: >> [ 2.365154] udev: starting version 146 >> [ 2.693768] end_request: I/O error, dev vdc, sector 126 >> [ 2.693772] Buffer I/O error on device vdc, logical block 126 >> [ 2.693775] Buffer I/O error on device vdc, logical block 127 >> [ 2.693777] Buffer I/O error on device vdc, logical block 128 >> > ... > > the problem happens right at startup, it can't read _anything_ > at all from the disk. In my case, the problem is intermittent > and happens under high load only, hence the big difference. > > But anyway, this is something which should be easy to find > out. Run kvm under `strace -f' and see how it opens the > > device, or find out with lsof what filedescriptor corresponds > to the file in question (in running kvm instance) and see > flags in /proc/$kvm_pid/fdinfo/$fdnum. > [...] stat("./vm/var_fs", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 41), ...}) = 0 open("./vm/var_fs", O_RDWR|O_DIRECT|O_CLOEXEC) = 12 lseek(12, 0, SEEK_END) = 1321851815424 [..] So it opens it the device without problems. The only things that stands out is this before the "read failed" message: [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid argument) =================== Below is the full grep for "(12," [pid 9097] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9097] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9097] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9097] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9097] lseek(12, 0, SEEK_SET) = 0 [pid 9097] read(12, "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 512) = 512 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9111] pread(12, "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 4096, 0) = 4096 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9157] pread(12, 0x7fa4f80b13d0, 64512, 0) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END <unfinished ...> [pid 9137] pread(12, 0x7fa5020c53d0, 64512, 64512) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9138] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9143] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9123] pread(12, 0x7fa5090defd0, 16384, 0) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9126] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9125] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9127] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9128] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9129] pread(12, "?=\321\301\250\357\215\236\325{N\306\246\346=\23\266b\3556z\376\234\251\v,cG\371\302\340~"..., 512, 4096) = 512 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9146] pread(12, "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 2048, 0) = 2048 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9115] pread(12, 0x7fa50d0e6fd0, 16384, 0) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9117] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9116] pread(12, "=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 512, 1321851748352) = 512 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9118] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9119] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9120] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9153] pread(12, 0x7fa4fa0c0fd0, 16384, 0) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9132] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9154] pread(12, "=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 512, 1321851748352) = 512 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9155] pread(12, "iQ\35 \271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 512, 1321851805696) = 512 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9133] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9134] pread(12, "?=\321\301\250\357\215\236\325{N\306\246\346=\23\266b\3556z\376\234\251\v,cG\371\302\340~"..., 512, 4096) = 512 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9156] pread(12, "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 2048, 0) = 2048 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9117] pread(12, 0x7fa50c0e4fd0, 16384, 0) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9116] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9118] pread(12, "=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 512, 1321851748352) = 512 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9119] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9120] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9121] pread(12, "?=\321\301\250\357\215\236\325{N\306\246\346=\23\266b\3556z\376\234\251\v,cG\371\302\340~"..., 512, 4096) = 512 [pid 9098] lseek(12, 0, SEEK_END <unfinished ...> [pid 9122] pread(12, "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 2048, 0) = 2048 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9207] pread(12, 0x7fa50e0e8fd0, 16384, 0) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9201] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9199] pread(12, "=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 512, 1321851748352) = 512 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9210] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9208] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9203] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9200] pread(12, 0x7fa5098e37d0, 2048, 0) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9200] pread(12, 0x7fa5098dffd0, 16384, 0) = -1 EINVAL (Invalid argument) [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9209] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9202] pread(12, "=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 512, 1321851748352) = 512 [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9207] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9201] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9199] pread(12, <unfinished ...> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 [pid 9210] pread(12, <unfinished ...> Cheers Antoine > I guess it can't open the image in read-write mode somehow. > > By the way, iothread doesn't really work in kvm, as far > as I can see. > > Thanks. > > /mjt > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 17:11 ` Antoine Martin @ 2010-03-07 17:18 ` Avi Kivity 2010-03-07 17:21 ` Christoph Hellwig 0 siblings, 1 reply; 51+ messages in thread From: Avi Kivity @ 2010-03-07 17:18 UTC (permalink / raw) To: Antoine Martin; +Cc: Michael Tokarev, kvm On 03/07/2010 07:11 PM, Antoine Martin wrote: >> >> the problem happens right at startup, it can't read _anything_ >> at all from the disk. In my case, the problem is intermittent >> and happens under high load only, hence the big difference. >> >> But anyway, this is something which should be easy to find >> out. Run kvm under `strace -f' and see how it opens the >> device, or find out with lsof what filedescriptor corresponds >> to the file in question (in running kvm instance) and see >> flags in /proc/$kvm_pid/fdinfo/$fdnum. > ... > [...] > stat("./vm/var_fs", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 41), > ...}) = 0 > open("./vm/var_fs", O_RDWR|O_DIRECT|O_CLOEXEC) = 12 > lseek(12, 0, SEEK_END) = 1321851815424 > [..] > So it opens it the device without problems. > > The only things that stands out is this before the "read failed" message: > [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid > argument) > The buffer is unaligned here, yet the file was opened with O_DIRECT (cache=none). This is strange, since alignment is not related to disk size. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 17:18 ` Avi Kivity @ 2010-03-07 17:21 ` Christoph Hellwig 2010-03-07 17:30 ` Avi Kivity 0 siblings, 1 reply; 51+ messages in thread From: Christoph Hellwig @ 2010-03-07 17:21 UTC (permalink / raw) To: Avi Kivity; +Cc: Antoine Martin, Michael Tokarev, kvm On Sun, Mar 07, 2010 at 07:18:40PM +0200, Avi Kivity wrote: >> The only things that stands out is this before the "read failed" message: >> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 >> [pid 9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid >> argument) >> > > The buffer is unaligned here, yet the file was opened with O_DIRECT > (cache=none). This is strange, since alignment is not related to disk > size. The other interesting thing is that it's using pread - which means it's a too old kernel to use preadv and thus a not very well tested codepath with current qemu. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 17:21 ` Christoph Hellwig @ 2010-03-07 17:30 ` Avi Kivity 2010-03-07 17:34 ` Christoph Hellwig 2010-03-07 18:01 ` raw disks no longer work in latest kvm (kvm-88 was fine) Antoine Martin 0 siblings, 2 replies; 51+ messages in thread From: Avi Kivity @ 2010-03-07 17:30 UTC (permalink / raw) To: Christoph Hellwig; +Cc: Antoine Martin, Michael Tokarev, kvm On 03/07/2010 07:21 PM, Christoph Hellwig wrote: > On Sun, Mar 07, 2010 at 07:18:40PM +0200, Avi Kivity wrote: > >>> The only things that stands out is this before the "read failed" message: >>> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 >>> [pid 9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid >>> argument) >>> >>> >> The buffer is unaligned here, yet the file was opened with O_DIRECT >> (cache=none). This is strange, since alignment is not related to disk >> size. >> > The other interesting thing is that it's using pread - which means > it's a too old kernel to use preadv and thus a not very well tested > codepath with current qemu. > It may also be that glibc is emulating preadv, incorrectly. Antoine, can you check this? ltrace may help, or run 'strings libc.so | grep pread'. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 17:30 ` Avi Kivity @ 2010-03-07 17:34 ` Christoph Hellwig 2010-03-07 18:43 ` raw disks no longer work in latest kvm (kvm-88 was fine) [SOLVED] Antoine Martin 2010-03-07 18:01 ` raw disks no longer work in latest kvm (kvm-88 was fine) Antoine Martin 1 sibling, 1 reply; 51+ messages in thread From: Christoph Hellwig @ 2010-03-07 17:34 UTC (permalink / raw) To: Avi Kivity; +Cc: Christoph Hellwig, Antoine Martin, Michael Tokarev, kvm On Sun, Mar 07, 2010 at 07:30:06PM +0200, Avi Kivity wrote: > It may also be that glibc is emulating preadv, incorrectly. I've done a quick audit of all pathes leading to pread and all seem to align correctly. So either a broken glibc emulation or something else outside the block layer seems likely. > Antoine, can you check this? ltrace may help, or run 'strings libc.so | > grep pread'. Or just add an #undef CONFIG_PREADV just before the first #ifdef CONFIG_PREADV in posix-aio-compat.c and see if that works. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) [SOLVED] 2010-03-07 17:34 ` Christoph Hellwig @ 2010-03-07 18:43 ` Antoine Martin 2010-03-07 18:55 ` Avi Kivity 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-03-07 18:43 UTC (permalink / raw) To: Christoph Hellwig, kvm On 03/08/2010 12:34 AM, Christoph Hellwig wrote: > On Sun, Mar 07, 2010 at 07:30:06PM +0200, Avi Kivity wrote: > >> It may also be that glibc is emulating preadv, incorrectly. >> > I've done a quick audit of all pathes leading to pread and all seem > to align correctly. So either a broken glibc emulation or something > else outside the block layer seems likely. > > >> Antoine, can you check this? ltrace may help, or run 'strings libc.so | >> grep pread'. >> > Or just add an > > #undef CONFIG_PREADV > > just before the first > > #ifdef CONFIG_PREADV > > in posix-aio-compat.c and see if that works. > It does indeed! qemu-kvm-0.12.3 is now seeing my partition again. woohoo! So, PREAD makes it break... how, where? What does this mean? Thanks a lot! Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) [SOLVED] 2010-03-07 18:43 ` raw disks no longer work in latest kvm (kvm-88 was fine) [SOLVED] Antoine Martin @ 2010-03-07 18:55 ` Avi Kivity 0 siblings, 0 replies; 51+ messages in thread From: Avi Kivity @ 2010-03-07 18:55 UTC (permalink / raw) To: Antoine Martin; +Cc: Christoph Hellwig, kvm On 03/07/2010 08:43 PM, Antoine Martin wrote: >> >>> Antoine, can you check this? ltrace may help, or run 'strings >>> libc.so | >>> grep pread'. >>> >> Or just add an >> >> #undef CONFIG_PREADV >> >> just before the first >> >> #ifdef CONFIG_PREADV >> >> in posix-aio-compat.c and see if that works. >> > > It does indeed! qemu-kvm-0.12.3 is now seeing my partition again. woohoo! > So, PREAD makes it break... how, where? What does this mean? > No - preadv emulation in glibc breaks it. Likely the bug I mentioned in the other email on this thread. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 17:30 ` Avi Kivity 2010-03-07 17:34 ` Christoph Hellwig @ 2010-03-07 18:01 ` Antoine Martin 2010-03-07 18:47 ` Avi Kivity 1 sibling, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-03-07 18:01 UTC (permalink / raw) To: Avi Kivity; +Cc: Christoph Hellwig, Michael Tokarev, kvm On 03/08/2010 12:30 AM, Avi Kivity wrote: > On 03/07/2010 07:21 PM, Christoph Hellwig wrote: >> On Sun, Mar 07, 2010 at 07:18:40PM +0200, Avi Kivity wrote: >>>> The only things that stands out is this before the "read failed" >>>> message: >>>> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 >>>> [pid 9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid >>>> argument) >>>> >>> The buffer is unaligned here, yet the file was opened with O_DIRECT >>> (cache=none). This is strange, since alignment is not related to disk >>> size. Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none" Side question: this is the right thing to do for raw partitions, right? >> The other interesting thing is that it's using pread - which means >> it's a too old kernel to use preadv and thus a not very well tested >> codepath with current qemu. Too old?, I am confused: both host and guest kernels are 2.6.33! I built KVM against the 2.6.30 headers though. > > It may also be that glibc is emulating preadv, incorrectly. Not sure how to do that. > > Antoine, can you check this? ltrace may help, This the only part that looked slightly relevant: [pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0 [pid 26883] malloc(48) = 0x2a38d60 [pid 26883] memset(0x2a38d60, '\000', 48) = 0x2a38d60 [pid 26883] open64("./vm/media_fs", 540674, 00) = 12 [pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0 [pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600 > or run 'strings libc.so | grep pread'. > strings /lib/libc.so.6 | grep pread preadv preadv64 pread __pread64_chk __pread64 __pread_chk Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 18:01 ` raw disks no longer work in latest kvm (kvm-88 was fine) Antoine Martin @ 2010-03-07 18:47 ` Avi Kivity 2010-03-07 19:07 ` Antoine Martin 2010-03-07 19:09 ` Asdo 0 siblings, 2 replies; 51+ messages in thread From: Avi Kivity @ 2010-03-07 18:47 UTC (permalink / raw) To: Antoine Martin; +Cc: Christoph Hellwig, Michael Tokarev, kvm On 03/07/2010 08:01 PM, Antoine Martin wrote: > On 03/08/2010 12:30 AM, Avi Kivity wrote: >> On 03/07/2010 07:21 PM, Christoph Hellwig wrote: >>> On Sun, Mar 07, 2010 at 07:18:40PM +0200, Avi Kivity wrote: >>>>> The only things that stands out is this before the "read failed" >>>>> message: >>>>> [pid 9098] lseek(12, 0, SEEK_END) = 1321851815424 >>>>> [pid 9121] pread(12, 0x7fa50a0e47d0, 2048, 0) = -1 EINVAL (Invalid >>>>> argument) >>>>> >>>> The buffer is unaligned here, yet the file was opened with O_DIRECT >>>> (cache=none). This is strange, since alignment is not related to disk >>>> size. > Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none" > > Side question: this is the right thing to do for raw partitions, right? The rightest. >>> The other interesting thing is that it's using pread - which means >>> it's a too old kernel to use preadv and thus a not very well tested >>> codepath with current qemu. > Too old?, I am confused: both host and guest kernels are 2.6.33! > I built KVM against the 2.6.30 headers though. You need to build qemu against the 2.6.33 headers ('make headers-install'). But after we fix this, please. >> >> It may also be that glibc is emulating preadv, incorrectly. > Not sure how to do that. >> >> Antoine, can you check this? ltrace may help, > This the only part that looked slightly relevant: > [pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0 > [pid 26883] malloc(48) = 0x2a38d60 > [pid 26883] memset(0x2a38d60, '\000', 48) = 0x2a38d60 > [pid 26883] open64("./vm/media_fs", 540674, 00) = 12 > [pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0 > [pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600 Where's pread/preadv? Did you use -f? > >> or run 'strings libc.so | grep pread'. >> > strings /lib/libc.so.6 | grep pread > preadv > preadv64 > pread > __pread64_chk > __pread64 > __pread_chk > So it does seem glibc emulates preadv. Perhaps https://bugzilla.redhat.com/show_bug.cgi?id=563103 ? -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 18:47 ` Avi Kivity @ 2010-03-07 19:07 ` Antoine Martin 2010-03-07 19:10 ` Avi Kivity 2010-03-07 19:09 ` Asdo 1 sibling, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-03-07 19:07 UTC (permalink / raw) To: Avi Kivity, Christoph Hellwig, kvm [snip] >>>> The other interesting thing is that it's using pread - which means >>>> it's a too old kernel to use preadv and thus a not very well tested >>>> codepath with current qemu. >> Too old?, I am confused: both host and guest kernels are 2.6.33! >> I built KVM against the 2.6.30 headers though. > > You need to build qemu against the 2.6.33 headers ('make > headers-install'). But after we fix this, please. OK. I'm signing off for today, but I will test whatever patches/suggestions you send my way. > >>> >>> It may also be that glibc is emulating preadv, incorrectly. >> Not sure how to do that. >>> >>> Antoine, can you check this? ltrace may help, >> This the only part that looked slightly relevant: >> [pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0 >> [pid 26883] malloc(48) = 0x2a38d60 >> [pid 26883] memset(0x2a38d60, '\000', 48) = 0x2a38d60 >> [pid 26883] open64("./vm/media_fs", 540674, 00) = 12 >> [pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0 >> [pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600 > > > Where's pread/preadv? Did you use -f? Yes I did. ltrace -f bash ./vm/start.sh >& trace.log >>> or run 'strings libc.so | grep pread'. >>> >> strings /lib/libc.so.6 | grep pread >> preadv >> preadv64 >> pread >> __pread64_chk >> __pread64 >> __pread_chk >> > > So it does seem glibc emulates preadv. > > Perhaps https://bugzilla.redhat.com/show_bug.cgi?id=563103 ? This a gentoo box... but yes, this does look similar doesn't it? Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 19:07 ` Antoine Martin @ 2010-03-07 19:10 ` Avi Kivity 2010-03-07 19:13 ` Antoine Martin 0 siblings, 1 reply; 51+ messages in thread From: Avi Kivity @ 2010-03-07 19:10 UTC (permalink / raw) To: Antoine Martin; +Cc: Christoph Hellwig, kvm On 03/07/2010 09:07 PM, Antoine Martin wrote: >>>> Antoine, can you check this? ltrace may help, >>> This the only part that looked slightly relevant: >>> [pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0 >>> [pid 26883] malloc(48) = 0x2a38d60 >>> [pid 26883] memset(0x2a38d60, '\000', 48) = 0x2a38d60 >>> [pid 26883] open64("./vm/media_fs", 540674, 00) = 12 >>> [pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0 >>> [pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600 >> >> >> Where's pread/preadv? Did you use -f? > > Yes I did. > ltrace -f bash ./vm/start.sh >& trace.log Well, some variant of pread should have shown up. Maybe it's the ltrace -f race. >>>> or run 'strings libc.so | grep pread'. >>>> >>> strings /lib/libc.so.6 | grep pread >>> preadv >>> preadv64 >>> pread >>> __pread64_chk >>> __pread64 >>> __pread_chk >>> >> >> So it does seem glibc emulates preadv. >> >> Perhaps https://bugzilla.redhat.com/show_bug.cgi?id=563103 ? > This a gentoo box... but yes, this does look similar doesn't it? What version of glibc do you have installed? -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 19:10 ` Avi Kivity @ 2010-03-07 19:13 ` Antoine Martin 2010-03-07 19:17 ` Avi Kivity 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-03-07 19:13 UTC (permalink / raw) To: Avi Kivity; +Cc: Christoph Hellwig, kvm On 03/08/2010 02:10 AM, Avi Kivity wrote: > On 03/07/2010 09:07 PM, Antoine Martin wrote: >>>>> Antoine, can you check this? ltrace may help, >>>> This the only part that looked slightly relevant: >>>> [pid 26883] __xstat64(1, "./vm/media_fs", 0x7fff92e40500) = 0 >>>> [pid 26883] malloc(48) = 0x2a38d60 >>>> [pid 26883] memset(0x2a38d60, '\000', 48) = 0x2a38d60 >>>> [pid 26883] open64("./vm/media_fs", 540674, 00) = 12 >>>> [pid 26883] posix_memalign(0x7fff92e40560, 512, 16384, -1, 48) = 0 >>>> [pid 26883] lseek64(12, 0, 2, 0x7f404f2f3e60, 4) = 0x133c4820600 >>> >>> >>> Where's pread/preadv? Did you use -f? >> >> Yes I did. >> ltrace -f bash ./vm/start.sh >& trace.log > > Well, some variant of pread should have shown up. Maybe it's the > ltrace -f race. FYI: qemu booted fully under strace, but not under ltrace... >>>>> or run 'strings libc.so | grep pread'. >>>>> >>>> strings /lib/libc.so.6 | grep pread >>>> preadv >>>> preadv64 >>>> pread >>>> __pread64_chk >>>> __pread64 >>>> __pread_chk >>>> >>> >>> So it does seem glibc emulates preadv. >>> >>> Perhaps https://bugzilla.redhat.com/show_bug.cgi?id=563103 ? >> This a gentoo box... but yes, this does look similar doesn't it? > > What version of glibc do you have installed? Latest stable: sys-devel/gcc-4.3.4 sys-libs/glibc-2.10.1-r1 ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 19:13 ` Antoine Martin @ 2010-03-07 19:17 ` Avi Kivity 2010-03-07 19:25 ` Antoine Martin 0 siblings, 1 reply; 51+ messages in thread From: Avi Kivity @ 2010-03-07 19:17 UTC (permalink / raw) To: Antoine Martin; +Cc: Christoph Hellwig, kvm On 03/07/2010 09:13 PM, Antoine Martin wrote: >> What version of glibc do you have installed? > > Latest stable: > sys-devel/gcc-4.3.4 > sys-libs/glibc-2.10.1-r1 > $ git show glibc-2.10~108 | head commit e109c6124fe121618e42ba882e2a0af6e97b8efc Author: Ulrich Drepper <drepper@redhat.com> Date: Fri Apr 3 19:57:16 2009 +0000 * misc/Makefile (routines): Add preadv, preadv64, pwritev, pwritev64. * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 for GLIBC_2.10. * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64. * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for preadv You might get away with rebuilding glibc against the 2.6.33 headers. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 19:17 ` Avi Kivity @ 2010-03-07 19:25 ` Antoine Martin 2010-03-07 19:35 ` Avi Kivity 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-03-07 19:25 UTC (permalink / raw) To: Avi Kivity, Christoph Hellwig, kvm On 03/08/2010 02:17 AM, Avi Kivity wrote: > On 03/07/2010 09:13 PM, Antoine Martin wrote: >>> What version of glibc do you have installed? >> >> Latest stable: >> sys-devel/gcc-4.3.4 >> sys-libs/glibc-2.10.1-r1 >> > > $ git show glibc-2.10~108 | head > commit e109c6124fe121618e42ba882e2a0af6e97b8efc > Author: Ulrich Drepper <drepper@redhat.com> > Date: Fri Apr 3 19:57:16 2009 +0000 > > * misc/Makefile (routines): Add preadv, preadv64, pwritev, pwritev64. > > * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 for > GLIBC_2.10. > * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64. > * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for > preadv > > You might get away with rebuilding glibc against the 2.6.33 headers. > The latest kernel headers available in gentoo (and they're masked unstable): sys-kernel/linux-headers-2.6.32 So I think I will just keep using Christoph's patch until .33 hits portage. Unless there's any reason not to? I would rather keep my system "clean". I can try it though, if that helps you clear things up? ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 19:25 ` Antoine Martin @ 2010-03-07 19:35 ` Avi Kivity 2010-03-13 9:51 ` Antoine Martin 0 siblings, 1 reply; 51+ messages in thread From: Avi Kivity @ 2010-03-07 19:35 UTC (permalink / raw) To: Antoine Martin; +Cc: Christoph Hellwig, kvm On 03/07/2010 09:25 PM, Antoine Martin wrote: > On 03/08/2010 02:17 AM, Avi Kivity wrote: >> On 03/07/2010 09:13 PM, Antoine Martin wrote: >>>> What version of glibc do you have installed? >>> >>> Latest stable: >>> sys-devel/gcc-4.3.4 >>> sys-libs/glibc-2.10.1-r1 >>> >> >> $ git show glibc-2.10~108 | head >> commit e109c6124fe121618e42ba882e2a0af6e97b8efc >> Author: Ulrich Drepper <drepper@redhat.com> >> Date: Fri Apr 3 19:57:16 2009 +0000 >> >> * misc/Makefile (routines): Add preadv, preadv64, pwritev, >> pwritev64. >> >> * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 for >> GLIBC_2.10. >> * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64. >> * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for >> preadv >> >> You might get away with rebuilding glibc against the 2.6.33 headers. >> > The latest kernel headers available in gentoo (and they're masked > unstable): > sys-kernel/linux-headers-2.6.32 > > So I think I will just keep using Christoph's patch until .33 hits > portage. > Unless there's any reason not to? I would rather keep my system "clean". > I can try it though, if that helps you clear things up? preadv/pwritev was actually introduced in 2.6.30. Perhaps you last build glibc before that? If so, a rebuild may be all that's necessary. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 19:35 ` Avi Kivity @ 2010-03-13 9:51 ` Antoine Martin 2010-03-14 7:34 ` Avi Kivity 2010-04-08 22:00 ` Antoine Martin 0 siblings, 2 replies; 51+ messages in thread From: Antoine Martin @ 2010-03-13 9:51 UTC (permalink / raw) To: Avi Kivity; +Cc: Christoph Hellwig, kvm On 03/08/2010 02:35 AM, Avi Kivity wrote: > On 03/07/2010 09:25 PM, Antoine Martin wrote: >> On 03/08/2010 02:17 AM, Avi Kivity wrote: >>> On 03/07/2010 09:13 PM, Antoine Martin wrote: >>>>> What version of glibc do you have installed? >>>> >>>> Latest stable: >>>> sys-devel/gcc-4.3.4 >>>> sys-libs/glibc-2.10.1-r1 >>>> >>> >>> $ git show glibc-2.10~108 | head >>> commit e109c6124fe121618e42ba882e2a0af6e97b8efc >>> Author: Ulrich Drepper <drepper@redhat.com> >>> Date: Fri Apr 3 19:57:16 2009 +0000 >>> >>> * misc/Makefile (routines): Add preadv, preadv64, pwritev, >>> pwritev64. >>> >>> * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 >>> for >>> GLIBC_2.10. >>> * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64. >>> * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for >>> preadv >>> >>> You might get away with rebuilding glibc against the 2.6.33 headers. >>> >> The latest kernel headers available in gentoo (and they're masked >> unstable): >> sys-kernel/linux-headers-2.6.32 >> >> So I think I will just keep using Christoph's patch until .33 hits >> portage. >> Unless there's any reason not to? I would rather keep my system "clean". >> I can try it though, if that helps you clear things up? > > preadv/pwritev was actually introduced in 2.6.30. Perhaps you last > build glibc before that? If so, a rebuild may be all that's necessary. > To be certain, I've rebuilt qemu-kvm against: linux-headers-2.6.33 + glibc-2.10.1-r1 (both freshly built) And still no go! I'm still having to use the patch which disables preadv unconditionally... Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-13 9:51 ` Antoine Martin @ 2010-03-14 7:34 ` Avi Kivity 2010-04-08 22:00 ` Antoine Martin 1 sibling, 0 replies; 51+ messages in thread From: Avi Kivity @ 2010-03-14 7:34 UTC (permalink / raw) To: Antoine Martin; +Cc: Christoph Hellwig, kvm On 03/13/2010 11:51 AM, Antoine Martin wrote: >> preadv/pwritev was actually introduced in 2.6.30. Perhaps you last >> build glibc before that? If so, a rebuild may be all that's necessary. >> > > To be certain, I've rebuilt qemu-kvm against: > linux-headers-2.6.33 + glibc-2.10.1-r1 (both freshly built) > And still no go! > I'm still having to use the patch which disables preadv > unconditionally... What does strace show? Is the kernel's preadv called? Maybe you have a glibc that has broken emulated preadv and no kernel preadv support. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-13 9:51 ` Antoine Martin 2010-03-14 7:34 ` Avi Kivity @ 2010-04-08 22:00 ` Antoine Martin 2010-05-22 10:44 ` Antoine Martin 1 sibling, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-04-08 22:00 UTC (permalink / raw) To: Avi Kivity; +Cc: Christoph Hellwig, kvm Antoine Martin wrote: > On 03/08/2010 02:35 AM, Avi Kivity wrote: >> On 03/07/2010 09:25 PM, Antoine Martin wrote: >>> On 03/08/2010 02:17 AM, Avi Kivity wrote: >>>> On 03/07/2010 09:13 PM, Antoine Martin wrote: >>>>>> What version of glibc do you have installed? >>>>> >>>>> Latest stable: >>>>> sys-devel/gcc-4.3.4 >>>>> sys-libs/glibc-2.10.1-r1 >>>>> >>>> >>>> $ git show glibc-2.10~108 | head >>>> commit e109c6124fe121618e42ba882e2a0af6e97b8efc >>>> Author: Ulrich Drepper <drepper@redhat.com> >>>> Date: Fri Apr 3 19:57:16 2009 +0000 >>>> >>>> * misc/Makefile (routines): Add preadv, preadv64, pwritev, >>>> pwritev64. >>>> >>>> * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 >>>> for >>>> GLIBC_2.10. >>>> * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64. >>>> * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for >>>> preadv >>>> >>>> You might get away with rebuilding glibc against the 2.6.33 headers. >>>> >>> The latest kernel headers available in gentoo (and they're masked >>> unstable): >>> sys-kernel/linux-headers-2.6.32 >>> >>> So I think I will just keep using Christoph's patch until .33 hits >>> portage. >>> Unless there's any reason not to? I would rather keep my system "clean". >>> I can try it though, if that helps you clear things up? >> >> preadv/pwritev was actually introduced in 2.6.30. Perhaps you last >> build glibc before that? If so, a rebuild may be all that's necessary. >> > To be certain, I've rebuilt qemu-kvm against: > linux-headers-2.6.33 + glibc-2.10.1-r1 (both freshly built) > And still no go! > I'm still having to use the patch which disables preadv unconditionally... Better late than never, here's the relevant part of the strace (for the unpatched case where it fails): stat("./fs", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 41), ...}) = 0 open("./fs", O_RDWR|O_DIRECT|O_CLOEXEC) = 12 lseek(12, 0, SEEK_END) = 1321851815424 [pid 31266] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31266] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31266] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31266] lseek(12, 0, SEEK_SET) = 0 [pid 31266] read(12, "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 512) = 512 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, "iQ\35 \271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 4096, 1321851805696) = 4096 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31273] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31271] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31331] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31294] pread(12, "iQ\35 \271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 512, 1321851805696) = 512 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31298] pread(12, "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 512, 0) = 512 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31307] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31295] pread(12, "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 2048, 0) = 2048 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31330] pread(12, "=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., 512, 1321851748352) = 512 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31331] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31294] pread(12, "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., 512, 0) = 512 [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 [pid 31298] pread(12, <unfinished ...> [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 Antoine > > Antoine > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-04-08 22:00 ` Antoine Martin @ 2010-05-22 10:44 ` Antoine Martin 2010-05-22 11:17 ` Michael Tokarev 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-05-22 10:44 UTC (permalink / raw) To: Avi Kivity; +Cc: Christoph Hellwig, kvm Bump. Now that qemu is less likely to eat my data, " *[Qemu-devel] [PATCH 4/8] block: fix sector comparism in*" http://marc.info/?l=qemu-devel&m=127436114712437 I thought I would try using the raw 1.5TB partition again with KVM, still no go. I am still having to use: #undef CONFIG_PREADV Host and guest kernel version is 2.6.34, headers 2.6.33, glibc 2.10.1-r1 qemu-kvm 0.12.4 + patch above. Who do I need to bug? glibc? kvm? Thanks Antoine On 04/09/2010 05:00 AM, Antoine Martin wrote: > > Antoine Martin wrote: > >> On 03/08/2010 02:35 AM, Avi Kivity wrote: >> >>> On 03/07/2010 09:25 PM, Antoine Martin wrote: >>> >>>> On 03/08/2010 02:17 AM, Avi Kivity wrote: >>>> >>>>> On 03/07/2010 09:13 PM, Antoine Martin wrote: >>>>> >>>>>>> What version of glibc do you have installed? >>>>>>> >>>>>> Latest stable: >>>>>> sys-devel/gcc-4.3.4 >>>>>> sys-libs/glibc-2.10.1-r1 >>>>>> >>>>>> >>>>> $ git show glibc-2.10~108 | head >>>>> commit e109c6124fe121618e42ba882e2a0af6e97b8efc >>>>> Author: Ulrich Drepper<drepper@redhat.com> >>>>> Date: Fri Apr 3 19:57:16 2009 +0000 >>>>> >>>>> * misc/Makefile (routines): Add preadv, preadv64, pwritev, >>>>> pwritev64. >>>>> >>>>> * misc/Versions: Export preadv, preadv64, pwritev, pwritev64 >>>>> for >>>>> GLIBC_2.10. >>>>> * misc/sys/uio.h: Declare preadv, preadv64, pwritev, pwritev64. >>>>> * sysdeps/unix/sysv/linux/kernel-features.h: Add entries for >>>>> preadv >>>>> >>>>> You might get away with rebuilding glibc against the 2.6.33 headers. >>>>> >>>>> >>>> The latest kernel headers available in gentoo (and they're masked >>>> unstable): >>>> sys-kernel/linux-headers-2.6.32 >>>> >>>> So I think I will just keep using Christoph's patch until .33 hits >>>> portage. >>>> Unless there's any reason not to? I would rather keep my system "clean". >>>> I can try it though, if that helps you clear things up? >>>> >>> preadv/pwritev was actually introduced in 2.6.30. Perhaps you last >>> build glibc before that? If so, a rebuild may be all that's necessary. >>> >>> >> To be certain, I've rebuilt qemu-kvm against: >> linux-headers-2.6.33 + glibc-2.10.1-r1 (both freshly built) >> And still no go! >> I'm still having to use the patch which disables preadv unconditionally... >> > Better late than never, here's the relevant part of the strace (for the > unpatched case where it fails): > > stat("./fs", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 41), ...}) = 0 > open("./fs", O_RDWR|O_DIRECT|O_CLOEXEC) = 12 > > lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31266] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31266] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31266] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31266] lseek(12, 0, SEEK_SET) = 0 > [pid 31266] read(12, > "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., > 512) = 512 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12, "iQ\35 > \271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 4096, > 1321851805696) = 4096 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31273] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31271] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31331] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31294] pread(12, "iQ\35 > \271O\203vj\ve[Ni}\355\263\272\4#yMo\266.\341\21\340Y5\204\20"..., 512, > 1321851805696) = 512 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31298] pread(12, > "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., > 512, 0) = 512 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31307] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31295] pread(12, > "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., > 2048, 0) = 2048 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31330] pread(12, > "=\206\5\355\35\2\2610\33\271\355\300qm\2174K\366\340ng\23\311\210Gg\220m\27\33E\254"..., > 512, 1321851748352) = 512 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31331] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31294] pread(12, > "\240\246E\32\r\21\367c\212\316Xn\177e'\310}\234\1\273`\371\266\247\r\1nj\332\32\221\26"..., > 512, 0) = 512 > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > [pid 31298] pread(12,<unfinished ...> > [pid 31267] lseek(12, 0, SEEK_END) = 1321851815424 > > > > Antoine > > >> Antoine >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-22 10:44 ` Antoine Martin @ 2010-05-22 11:17 ` Michael Tokarev 2010-05-22 11:35 ` Antoine Martin 0 siblings, 1 reply; 51+ messages in thread From: Michael Tokarev @ 2010-05-22 11:17 UTC (permalink / raw) To: Antoine Martin; +Cc: Avi Kivity, Christoph Hellwig, kvm 22.05.2010 14:44, Antoine Martin wrote: > Bump. > > Now that qemu is less likely to eat my data, " *[Qemu-devel] [PATCH 4/8] > block: fix sector comparism in*" > http://marc.info/?l=qemu-devel&m=127436114712437 > > I thought I would try using the raw 1.5TB partition again with KVM, > still no go. Hm. I don't have so much diskspace (my largest is 750Gb, whole disk), but I created 1.5Tb sparse lvm volume. It appears to work for me, even 32bit version of qemu-kvm-0.12.4 (with the mentioned patch applied). > I am still having to use: > > #undef CONFIG_PREADV > > Host and guest kernel version is 2.6.34, headers 2.6.33, glibc 2.10.1-r1 > qemu-kvm 0.12.4 + patch above. eglibc-2.10.2-6, kernel #2.6.34.0-amd64, kernel headers 2.6.32-11~bpo50+1 (debian) > Who do I need to bug? glibc? kvm? are you running 32bit userspace and 64bit kernel by a chance? If yes that's a kernel prob, see http://thread.gmane.org/gmane.linux.kernel.aio.general/2891 (the fix will be in 2.6.35 hopefully, now it's in Andrew Morton's tree). If not, well, I don't know ;) /mjt ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-22 11:17 ` Michael Tokarev @ 2010-05-22 11:35 ` Antoine Martin 2010-05-23 8:53 ` Antoine Martin 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-05-22 11:35 UTC (permalink / raw) To: Michael Tokarev; +Cc: Avi Kivity, Christoph Hellwig, kvm On 05/22/2010 06:17 PM, Michael Tokarev wrote: > 22.05.2010 14:44, Antoine Martin wrote: >> Bump. >> >> Now that qemu is less likely to eat my data, " *[Qemu-devel] [PATCH 4/8] >> block: fix sector comparism in*" >> http://marc.info/?l=qemu-devel&m=127436114712437 >> >> I thought I would try using the raw 1.5TB partition again with KVM, >> still no go. > > Hm. I don't have so much diskspace (my largest is 750Gb, whole disk), > but I created 1.5Tb sparse lvm volume. It appears to work for me, > even 32bit version of qemu-kvm-0.12.4 (with the mentioned patch applied). > >> I am still having to use: >> >> #undef CONFIG_PREADV >> >> Host and guest kernel version is 2.6.34, headers 2.6.33, glibc 2.10.1-r1 >> qemu-kvm 0.12.4 + patch above. > > eglibc-2.10.2-6, kernel #2.6.34.0-amd64, > kernel headers 2.6.32-11~bpo50+1 > (debian) > >> Who do I need to bug? glibc? kvm? > > are you running 32bit userspace and 64bit kernel > by a chance? If yes that's a kernel prob, see > http://thread.gmane.org/gmane.linux.kernel.aio.general/2891 > (the fix will be in 2.6.35 hopefully, now it's in > Andrew Morton's tree). > > If not, well, I don't know ;) I'm not: 64-bit host and 64-bit guest. Thanks anyway. Antoine > > /mjt > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-22 11:35 ` Antoine Martin @ 2010-05-23 8:53 ` Antoine Martin 2010-05-23 11:57 ` Avi Kivity 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-05-23 8:53 UTC (permalink / raw) To: Michael Tokarev; +Cc: Avi Kivity, Christoph Hellwig, kvm On 05/22/2010 06:35 PM, Antoine Martin wrote: > On 05/22/2010 06:17 PM, Michael Tokarev wrote: >> 22.05.2010 14:44, Antoine Martin wrote: >>> Bump. >>> >>> Now that qemu is less likely to eat my data, " *[Qemu-devel] [PATCH >>> 4/8] >>> block: fix sector comparism in*" >>> http://marc.info/?l=qemu-devel&m=127436114712437 >>> >>> I thought I would try using the raw 1.5TB partition again with KVM, >>> still no go. >> >> Hm. I don't have so much diskspace (my largest is 750Gb, whole disk), >> but I created 1.5Tb sparse lvm volume. It appears to work for me, >> even 32bit version of qemu-kvm-0.12.4 (with the mentioned patch >> applied). >> >>> I am still having to use: >>> >>> #undef CONFIG_PREADV >>> >>> Host and guest kernel version is 2.6.34, headers 2.6.33, glibc >>> 2.10.1-r1 >>> qemu-kvm 0.12.4 + patch above. >> >> eglibc-2.10.2-6, kernel #2.6.34.0-amd64, >> kernel headers 2.6.32-11~bpo50+1 >> (debian) >> >>> Who do I need to bug? glibc? kvm? >> >> are you running 32bit userspace and 64bit kernel >> by a chance? If yes that's a kernel prob, see >> http://thread.gmane.org/gmane.linux.kernel.aio.general/2891 >> (the fix will be in 2.6.35 hopefully, now it's in >> Andrew Morton's tree). >> >> If not, well, I don't know ;) > I'm not: 64-bit host and 64-bit guest. Just to be sure, I've tested that patch and still no joy: /dev/vdc: read failed after 0 of 512 at 0: Input/output error /dev/vdc: read failed after 0 of 512 at 0: Input/output error /dev/vdc: read failed after 0 of 2048 at 0: Input/output error pread is still broken on this big raw disk. Antoine > > Thanks anyway. > Antoine >> >> /mjt >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 8:53 ` Antoine Martin @ 2010-05-23 11:57 ` Avi Kivity 2010-05-23 14:07 ` Antoine Martin 0 siblings, 1 reply; 51+ messages in thread From: Avi Kivity @ 2010-05-23 11:57 UTC (permalink / raw) To: Antoine Martin; +Cc: Michael Tokarev, Christoph Hellwig, kvm On 05/23/2010 11:53 AM, Antoine Martin wrote: > I'm not: 64-bit host and 64-bit guest. > Just to be sure, I've tested that patch and still no joy: > /dev/vdc: read failed after 0 of 512 at 0: Input/output error > /dev/vdc: read failed after 0 of 512 at 0: Input/output error > /dev/vdc: read failed after 0 of 2048 at 0: Input/output error > pread is still broken on this big raw disk. > Can you summarize your environment and reproducer again? The thread is so long it is difficult to understand exactly what you are testing. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 11:57 ` Avi Kivity @ 2010-05-23 14:07 ` Antoine Martin 2010-05-23 14:18 ` Avi Kivity 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-05-23 14:07 UTC (permalink / raw) To: Avi Kivity; +Cc: Michael Tokarev, Christoph Hellwig, kvm On 05/23/2010 06:57 PM, Avi Kivity wrote: > On 05/23/2010 11:53 AM, Antoine Martin wrote: >> I'm not: 64-bit host and 64-bit guest. >> Just to be sure, I've tested that patch and still no joy: >> /dev/vdc: read failed after 0 of 512 at 0: Input/output error >> /dev/vdc: read failed after 0 of 512 at 0: Input/output error >> /dev/vdc: read failed after 0 of 2048 at 0: Input/output error >> pread is still broken on this big raw disk. >> > > Can you summarize your environment and reproducer again? The thread > is so long it is difficult to understand exactly what you are testing. > Sure thing: Description of the problem: A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my case), this fails with pread enabled, works with it disabled. What has been tested: Host has had 2.6.31.x and now 2.6.34 - with no improvement. Guest has had numerous kernel versions and patches applied for testing (from 2.6.31.x to 2.6.34) - none made any difference. (although the recent patch stopped the large partition from getting corrupted by the merged requests bug.. yay!) It was once thought that glibc needed to be rebuilt against newer kernel headers, done that too, and rebuilt qemu-kvm afterwards. (was 2.6.28 then 2.6.31 and now 2.6.33) What has been reported: straces, kernel error messages as above. Here is the qemu command line (sanitized - removed the virtual disks which work fine and network config): qemu -clock dynticks -m 256 -L ./ -kernel ./bzImage-2.6.34-aio -append "root=/dev/vda" -nographic -drive file=/dev/sda9,if=virtio,cache=none Let me know if you need any other details. I can also arrange ssh access to the system if needed. Cheers Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 14:07 ` Antoine Martin @ 2010-05-23 14:18 ` Avi Kivity 2010-05-23 14:43 ` Antoine Martin 0 siblings, 1 reply; 51+ messages in thread From: Avi Kivity @ 2010-05-23 14:18 UTC (permalink / raw) To: Antoine Martin; +Cc: Michael Tokarev, Christoph Hellwig, kvm On 05/23/2010 05:07 PM, Antoine Martin wrote: > On 05/23/2010 06:57 PM, Avi Kivity wrote: >> On 05/23/2010 11:53 AM, Antoine Martin wrote: >>> I'm not: 64-bit host and 64-bit guest. >>> Just to be sure, I've tested that patch and still no joy: >>> /dev/vdc: read failed after 0 of 512 at 0: Input/output error >>> /dev/vdc: read failed after 0 of 512 at 0: Input/output error >>> /dev/vdc: read failed after 0 of 2048 at 0: Input/output error >>> pread is still broken on this big raw disk. >>> >> >> Can you summarize your environment and reproducer again? The thread >> is so long it is difficult to understand exactly what you are testing. >> > Sure thing: > > Description of the problem: > A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my > case), this fails with pread enabled, works with it disabled. Did you mean: preadv? > > What has been tested: > Host has had 2.6.31.x and now 2.6.34 - with no improvement. > Guest has had numerous kernel versions and patches applied for testing > (from 2.6.31.x to 2.6.34) - none made any difference. (although the > recent patch stopped the large partition from getting corrupted by the > merged requests bug.. yay!) > It was once thought that glibc needed to be rebuilt against newer > kernel headers, done that too, and rebuilt qemu-kvm afterwards. (was > 2.6.28 then 2.6.31 and now 2.6.33) > > What has been reported: straces, kernel error messages as above. > Here is the qemu command line (sanitized - removed the virtual disks > which work fine and network config): > qemu -clock dynticks -m 256 -L ./ -kernel ./bzImage-2.6.34-aio -append > "root=/dev/vda" -nographic -drive file=/dev/sda9,if=virtio,cache=none > > Let me know if you need any other details. I can also arrange ssh > access to the system if needed. And: 64-bit host kernel, 64-bit host userspace, yes? Does aio=native help? How about if=ide? -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 14:18 ` Avi Kivity @ 2010-05-23 14:43 ` Antoine Martin 2010-05-23 14:53 ` Antoine Martin ` (2 more replies) 0 siblings, 3 replies; 51+ messages in thread From: Antoine Martin @ 2010-05-23 14:43 UTC (permalink / raw) To: Avi Kivity; +Cc: Michael Tokarev, Christoph Hellwig, kvm On 05/23/2010 09:18 PM, Avi Kivity wrote: > On 05/23/2010 05:07 PM, Antoine Martin wrote: >> On 05/23/2010 06:57 PM, Avi Kivity wrote: >>> On 05/23/2010 11:53 AM, Antoine Martin wrote: >>>> I'm not: 64-bit host and 64-bit guest. >>>> Just to be sure, I've tested that patch and still no joy: >>>> /dev/vdc: read failed after 0 of 512 at 0: Input/output error >>>> /dev/vdc: read failed after 0 of 512 at 0: Input/output error >>>> /dev/vdc: read failed after 0 of 2048 at 0: Input/output error >>>> pread is still broken on this big raw disk. >>>> >>> >>> Can you summarize your environment and reproducer again? The thread >>> is so long it is difficult to understand exactly what you are testing. >>> >> Sure thing: >> >> Description of the problem: >> A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my >> case), this fails with pread enabled, works with it disabled. > > Did you mean: preadv? Yes, here's what makes it work ok (as suggested by Christoph earlier in the thread) in posix-aio-compat.c: #undef CONFIG_PREADV #ifdef CONFIG_PREADV static int preadv_present = 1; #else static int preadv_present = 0; #endif >> >> What has been tested: >> Host has had 2.6.31.x and now 2.6.34 - with no improvement. >> Guest has had numerous kernel versions and patches applied for >> testing (from 2.6.31.x to 2.6.34) - none made any difference. >> (although the recent patch stopped the large partition from getting >> corrupted by the merged requests bug.. yay!) >> It was once thought that glibc needed to be rebuilt against newer >> kernel headers, done that too, and rebuilt qemu-kvm afterwards. (was >> 2.6.28 then 2.6.31 and now 2.6.33) >> >> What has been reported: straces, kernel error messages as above. >> Here is the qemu command line (sanitized - removed the virtual disks >> which work fine and network config): >> qemu -clock dynticks -m 256 -L ./ -kernel ./bzImage-2.6.34-aio >> -append "root=/dev/vda" -nographic -drive >> file=/dev/sda9,if=virtio,cache=none >> >> Let me know if you need any other details. I can also arrange ssh >> access to the system if needed. > > And: 64-bit host kernel, 64-bit host userspace, yes? Yes. (and 64-bit guest/userspace too) > > Does aio=native help? It does indeed! > How about if=ide? Will test with another kernel and report back (this one doesn't have any non-virtio drivers) Thanks Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 14:43 ` Antoine Martin @ 2010-05-23 14:53 ` Antoine Martin 2010-05-23 14:56 ` Avi Kivity 2010-05-23 15:06 ` Antoine Martin 2010-05-23 15:12 ` Avi Kivity 2 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-05-23 14:53 UTC (permalink / raw) To: Avi Kivity; +Cc: Michael Tokarev, Christoph Hellwig, kvm >> How about if=ide? > Will test with another kernel and report back (this one doesn't have > any non-virtio drivers) Can anyone tell me which kernel module I need for "if=ide"? Google was no help here. (before I include dozens of unnecessary modules in my slimmed down and non modular kernel) Thanks Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 14:53 ` Antoine Martin @ 2010-05-23 14:56 ` Avi Kivity 0 siblings, 0 replies; 51+ messages in thread From: Avi Kivity @ 2010-05-23 14:56 UTC (permalink / raw) To: Antoine Martin; +Cc: Michael Tokarev, Christoph Hellwig, kvm On 05/23/2010 05:53 PM, Antoine Martin wrote: > >>> How about if=ide? >> Will test with another kernel and report back (this one doesn't have >> any non-virtio drivers) > Can anyone tell me which kernel module I need for "if=ide"? Google was > no help here. > (before I include dozens of unnecessary modules in my slimmed down and > non modular kernel) ATA_PIIX probably. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 14:43 ` Antoine Martin 2010-05-23 14:53 ` Antoine Martin @ 2010-05-23 15:06 ` Antoine Martin 2010-05-23 15:12 ` Avi Kivity 2 siblings, 0 replies; 51+ messages in thread From: Antoine Martin @ 2010-05-23 15:06 UTC (permalink / raw) To: Avi Kivity; +Cc: Michael Tokarev, Christoph Hellwig, kvm On 05/23/2010 09:43 PM, Antoine Martin wrote: > On 05/23/2010 09:18 PM, Avi Kivity wrote: >> On 05/23/2010 05:07 PM, Antoine Martin wrote: >>> On 05/23/2010 06:57 PM, Avi Kivity wrote: >>>> On 05/23/2010 11:53 AM, Antoine Martin wrote: >>>>> I'm not: 64-bit host and 64-bit guest. >>>>> Just to be sure, I've tested that patch and still no joy: >>>>> /dev/vdc: read failed after 0 of 512 at 0: Input/output error >>>>> /dev/vdc: read failed after 0 of 512 at 0: Input/output error >>>>> /dev/vdc: read failed after 0 of 2048 at 0: Input/output error >>>>> pread is still broken on this big raw disk. >>>>> >>>> >>>> Can you summarize your environment and reproducer again? The >>>> thread is so long it is difficult to understand exactly what you >>>> are testing. >>>> >>> Sure thing: >>> >>> Description of the problem: >>> A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my >>> case), this fails with pread enabled, works with it disabled. >> >> Did you mean: preadv? > Yes, here's what makes it work ok (as suggested by Christoph earlier > in the thread) in posix-aio-compat.c: > #undef CONFIG_PREADV > #ifdef CONFIG_PREADV > static int preadv_present = 1; > #else > static int preadv_present = 0; > #endif >>> >>> What has been tested: >>> Host has had 2.6.31.x and now 2.6.34 - with no improvement. >>> Guest has had numerous kernel versions and patches applied for >>> testing (from 2.6.31.x to 2.6.34) - none made any difference. >>> (although the recent patch stopped the large partition from getting >>> corrupted by the merged requests bug.. yay!) >>> It was once thought that glibc needed to be rebuilt against newer >>> kernel headers, done that too, and rebuilt qemu-kvm afterwards. (was >>> 2.6.28 then 2.6.31 and now 2.6.33) >>> >>> What has been reported: straces, kernel error messages as above. >>> Here is the qemu command line (sanitized - removed the virtual disks >>> which work fine and network config): >>> qemu -clock dynticks -m 256 -L ./ -kernel ./bzImage-2.6.34-aio >>> -append "root=/dev/vda" -nographic -drive >>> file=/dev/sda9,if=virtio,cache=none >>> >>> Let me know if you need any other details. I can also arrange ssh >>> access to the system if needed. >> >> And: 64-bit host kernel, 64-bit host userspace, yes? > Yes. (and 64-bit guest/userspace too) >> >> Does aio=native help? > It does indeed! >> How about if=ide? > Will test with another kernel and report back (this one doesn't have > any non-virtio drivers) That also works fine. I guess this tells you that the problem is confined to raw-disk+virtio+aio-threaded I'm switching all my VMs to aio=native, but feel free to send patches if you want me to re-test aio=threaded Thanks Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 14:43 ` Antoine Martin 2010-05-23 14:53 ` Antoine Martin 2010-05-23 15:06 ` Antoine Martin @ 2010-05-23 15:12 ` Avi Kivity 2010-05-23 16:18 ` Antoine Martin 2 siblings, 1 reply; 51+ messages in thread From: Avi Kivity @ 2010-05-23 15:12 UTC (permalink / raw) To: Antoine Martin; +Cc: Michael Tokarev, Christoph Hellwig, kvm On 05/23/2010 05:43 PM, Antoine Martin wrote: >>> Description of the problem: >>> A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my >>> case), this fails with pread enabled, works with it disabled. >> >> Did you mean: preadv? > > Yes, here's what makes it work ok (as suggested by Christoph earlier > in the thread) in posix-aio-compat.c: > #undef CONFIG_PREADV > #ifdef CONFIG_PREADV > static int preadv_present = 1; > #else > static int preadv_present = 0; > #endif When preadv_present=1, does strace -fF show preadv being called? If so, the kernel's preadv() is broken. If not, glibc's preadv() emulation is broken. >> >> Does aio=native help? > It does indeed! That's recommended anyway on raw partitions with cache=off. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 15:12 ` Avi Kivity @ 2010-05-23 16:18 ` Antoine Martin 2010-05-23 17:47 ` Stefan Hajnoczi 0 siblings, 1 reply; 51+ messages in thread From: Antoine Martin @ 2010-05-23 16:18 UTC (permalink / raw) To: Avi Kivity; +Cc: Michael Tokarev, Christoph Hellwig, kvm On 05/23/2010 10:12 PM, Avi Kivity wrote: > On 05/23/2010 05:43 PM, Antoine Martin wrote: >>>> Description of the problem: >>>> A guest tries to mount a large raw partition (1.5TB /dev/sda9 in my >>>> case), this fails with pread enabled, works with it disabled. >>> >>> Did you mean: preadv? >> >> Yes, here's what makes it work ok (as suggested by Christoph earlier >> in the thread) in posix-aio-compat.c: >> #undef CONFIG_PREADV >> #ifdef CONFIG_PREADV >> static int preadv_present = 1; >> #else >> static int preadv_present = 0; >> #endif > > When preadv_present=1, does strace -fF show preadv being called? There were some pread() but no preadv() in the strace BTW. > If so, the kernel's preadv() is broken. > > If not, glibc's preadv() emulation is broken. OK, I found what's causing it: chroot I was testing all this in a chroot, I always do and I didn't think of mentioning it, sorry about that. Running it non-chrooted works in all cases, including aio!=native Why does it work in a chroot for the other options (aio=native, if=ide, etc) but not for aio!=native?? Looks like I am misunderstanding the semantics of chroot... >>> >>> Does aio=native help? >> It does indeed! > > That's recommended anyway on raw partitions with cache=off. > What about non-raw partitions (or/and with cache=on)? Antoine ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 16:18 ` Antoine Martin @ 2010-05-23 17:47 ` Stefan Hajnoczi 2010-05-29 9:42 ` Antoine Martin 0 siblings, 1 reply; 51+ messages in thread From: Stefan Hajnoczi @ 2010-05-23 17:47 UTC (permalink / raw) To: Antoine Martin; +Cc: Avi Kivity, Michael Tokarev, Christoph Hellwig, kvm On Sun, May 23, 2010 at 5:18 PM, Antoine Martin <antoine@nagafix.co.uk> wrote: > Why does it work in a chroot for the other options (aio=native, if=ide, etc) > but not for aio!=native?? > Looks like I am misunderstanding the semantics of chroot... It might not be the chroot() semantics but the environment inside that chroot, like the glibc. Have you compared strace inside and outside the chroot? Stefan ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-23 17:47 ` Stefan Hajnoczi @ 2010-05-29 9:42 ` Antoine Martin 2010-05-29 9:55 ` Stefan Hajnoczi 2010-05-29 10:34 ` Christoph Hellwig 0 siblings, 2 replies; 51+ messages in thread From: Antoine Martin @ 2010-05-29 9:42 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: Avi Kivity, Michael Tokarev, Christoph Hellwig, kvm On 05/24/2010 12:47 AM, Stefan Hajnoczi wrote: > On Sun, May 23, 2010 at 5:18 PM, Antoine Martin <antoine@nagafix.co.uk> wrote: >> Why does it work in a chroot for the other options (aio=native, if=ide, etc) >> but not for aio!=native?? >> Looks like I am misunderstanding the semantics of chroot... > > It might not be the chroot() semantics but the environment inside that > chroot, like the glibc. Have you compared strace inside and outside > the chroot? Reverting to a static build also fixes the issue: aio=threads works. Definitely something fishy going on with glibc library loading. (I've checked glibc, libaio were up to date in the chroot - nothing blatant in the strace) Can someone explain the aio options? All I can find is this: # qemu-system-x86_64 -h | grep -i aio [,addr=A][,id=name][,aio=threads|native] I assume it means the aio=threads emulates the kernel's aio with separate threads? And is therefore likely to be slower, right? Is there a reason why aio=native is not the default? Shouldn't aio=threads be the fallback? Cheers Antoine > > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-29 9:42 ` Antoine Martin @ 2010-05-29 9:55 ` Stefan Hajnoczi 2010-05-29 10:34 ` Christoph Hellwig 2010-05-29 10:34 ` Christoph Hellwig 1 sibling, 1 reply; 51+ messages in thread From: Stefan Hajnoczi @ 2010-05-29 9:55 UTC (permalink / raw) To: Antoine Martin; +Cc: Avi Kivity, Michael Tokarev, Christoph Hellwig, kvm On Sat, May 29, 2010 at 10:42 AM, Antoine Martin <antoine@nagafix.co.uk> wrote: > Can someone explain the aio options? > All I can find is this: > # qemu-system-x86_64 -h | grep -i aio > [,addr=A][,id=name][,aio=threads|native] > I assume it means the aio=threads emulates the kernel's aio with > separate threads? And is therefore likely to be slower, right? > Is there a reason why aio=native is not the default? Shouldn't > aio=threads be the fallback? aio=threads uses posix-aio-compat.c, a POSIX AIO-like implementation using a thread pool. Each thread services queued I/O requests using blocking syscalls (e.g. preadv()/pwritev()). aio=native uses Linux libaio, the native (non-POSIX) AIO interface. I would expect that aio=native is faster but benchmarks show that this isn't true for all workloads. Stefan ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-29 9:55 ` Stefan Hajnoczi @ 2010-05-29 10:34 ` Christoph Hellwig 2010-05-29 11:06 ` Stefan Hajnoczi 0 siblings, 1 reply; 51+ messages in thread From: Christoph Hellwig @ 2010-05-29 10:34 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Antoine Martin, Avi Kivity, Michael Tokarev, Christoph Hellwig, kvm On Sat, May 29, 2010 at 10:55:18AM +0100, Stefan Hajnoczi wrote: > I would expect that aio=native is faster but benchmarks show that this > isn't true for all workloads. In what benchmark do you see worse results for aio=native compared to aio=threads? ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-29 10:34 ` Christoph Hellwig @ 2010-05-29 11:06 ` Stefan Hajnoczi 0 siblings, 0 replies; 51+ messages in thread From: Stefan Hajnoczi @ 2010-05-29 11:06 UTC (permalink / raw) To: Christoph Hellwig; +Cc: Antoine Martin, Avi Kivity, Michael Tokarev, kvm On Sat, May 29, 2010 at 11:34 AM, Christoph Hellwig <hch@infradead.org> wrote: > In what benchmark do you see worse results for aio=native compared to > aio=threads? Sequential reads using 4 concurrent dd if=/dev/vdb iflag=direct of=/dev/null bs=8k processes. 2 vcpu guest with 4 GB RAM, virtio block devices, cache=none. Host storage is a striped LVM volume. Host kernel kvm.git and qemu-kvm.git userspace. aio=native and aio=threads each run 3 times. Result: aio=native has 15% lower throughput than aio=threads. I haven't looked into this so I don't know what is causes these results. Stefan ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-05-29 9:42 ` Antoine Martin 2010-05-29 9:55 ` Stefan Hajnoczi @ 2010-05-29 10:34 ` Christoph Hellwig 1 sibling, 0 replies; 51+ messages in thread From: Christoph Hellwig @ 2010-05-29 10:34 UTC (permalink / raw) To: Antoine Martin Cc: Stefan Hajnoczi, Avi Kivity, Michael Tokarev, Christoph Hellwig, kvm On Sat, May 29, 2010 at 04:42:59PM +0700, Antoine Martin wrote: > Can someone explain the aio options? > All I can find is this: > # qemu-system-x86_64 -h | grep -i aio > [,addr=A][,id=name][,aio=threads|native] > I assume it means the aio=threads emulates the kernel's aio with > separate threads? And is therefore likely to be slower, right? > Is there a reason why aio=native is not the default? Shouldn't > aio=threads be the fallback? The kernel AIO support is unfortunately not a very generic API. It only supports O_DIRECT I/O (cache=none for qemu), and if used on a filesystems it might still block if we need to perform block allocations. We could probably make it the default for block devices, but I'm not a big fan of these kind of conditional defaults. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 18:47 ` Avi Kivity 2010-03-07 19:07 ` Antoine Martin @ 2010-03-07 19:09 ` Asdo 2010-03-07 19:11 ` Antoine Martin 2010-03-07 19:12 ` Avi Kivity 1 sibling, 2 replies; 51+ messages in thread From: Asdo @ 2010-03-07 19:09 UTC (permalink / raw) To: Avi Kivity; +Cc: Antoine Martin, Christoph Hellwig, Michael Tokarev, kvm Avi Kivity wrote: > On 03/07/2010 08:01 PM, Antoine Martin wrote: >> >> Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none" >> >> Side question: this is the right thing to do for raw partitions, right? > > The rightest. Isn't cache=writeback now safe on virtio-blk since 2.6.32? Doesn't it provide better performances? Also that looks like a raw file to me, not a partition... do they follow the same optimization "rule"? Thank you ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 19:09 ` Asdo @ 2010-03-07 19:11 ` Antoine Martin 2010-03-07 19:12 ` Avi Kivity 1 sibling, 0 replies; 51+ messages in thread From: Antoine Martin @ 2010-03-07 19:11 UTC (permalink / raw) To: Asdo; +Cc: Avi Kivity, Christoph Hellwig, Michael Tokarev, kvm On 03/08/2010 02:09 AM, Asdo wrote: > Avi Kivity wrote: >> On 03/07/2010 08:01 PM, Antoine Martin wrote: >>> >>> Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none" >>> >>> Side question: this is the right thing to do for raw partitions, right? >> >> The rightest. > > Isn't cache=writeback now safe on virtio-blk since 2.6.32? > Doesn't it provide better performances? I thought it was best to let the guest deal with it? (single cache) > Also that looks like a raw file to me, not a partition... do they > follow the same optimization "rule"? This is a device (/dev/sdc9) in the chroot, don't let the name fool you ;) Antoine > > Thank you > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 19:09 ` Asdo 2010-03-07 19:11 ` Antoine Martin @ 2010-03-07 19:12 ` Avi Kivity 1 sibling, 0 replies; 51+ messages in thread From: Avi Kivity @ 2010-03-07 19:12 UTC (permalink / raw) To: Asdo; +Cc: Antoine Martin, Christoph Hellwig, Michael Tokarev, kvm On 03/07/2010 09:09 PM, Asdo wrote: > Avi Kivity wrote: >> On 03/07/2010 08:01 PM, Antoine Martin wrote: >>> >>> Yes cache=none: "-drive file=./vm/var_fs,if=virtio,cache=none" >>> >>> Side question: this is the right thing to do for raw partitions, right? >> >> The rightest. > > Isn't cache=writeback now safe on virtio-blk since 2.6.32? Yes. > Doesn't it provide better performances? No. The guest will do its own caching, so unless the guest is really short of memory, you aren't gaining much; and the copying will hurt. > > Also that looks like a raw file to me, not a partition... do they > follow the same optimization "rule"? More or less. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 10:00 ` Christoph Hellwig 2010-03-07 13:12 ` Antoine Martin @ 2010-03-07 16:21 ` Avi Kivity 2010-03-08 16:45 ` Anthony Liguori 1 sibling, 1 reply; 51+ messages in thread From: Avi Kivity @ 2010-03-07 16:21 UTC (permalink / raw) To: Christoph Hellwig; +Cc: Michael Tokarev, Antoine Martin, kvm, Anthony Liguori On 03/07/2010 12:00 PM, Christoph Hellwig wrote: > >> I can only guess that the info collected so far is not sufficient to >> understand what's going on: except of "I/O error writing block NNN" >> we does not have anything at all. So it's impossible to know where >> the problem is. >> > Actually it is, and the bug has been fixed long ago in: > > commit e2a305fb13ff0f5cf6ff805555aaa90a5ed5954c > Author: Christoph Hellwig<hch@lst.de> > Date: Tue Jan 26 14:49:08 2010 +0100 > > block: avoid creating too large iovecs in multiwrite_merge > > > I've asked for it be added to the -stable series but that hasn't > happened so far. > Anthony, this looks critical. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-07 16:21 ` Avi Kivity @ 2010-03-08 16:45 ` Anthony Liguori 0 siblings, 0 replies; 51+ messages in thread From: Anthony Liguori @ 2010-03-08 16:45 UTC (permalink / raw) To: Avi Kivity Cc: Christoph Hellwig, Michael Tokarev, Antoine Martin, kvm, Anthony Liguori On 03/07/2010 10:21 AM, Avi Kivity wrote: > On 03/07/2010 12:00 PM, Christoph Hellwig wrote: >> >>> I can only guess that the info collected so far is not sufficient to >>> understand what's going on: except of "I/O error writing block NNN" >>> we does not have anything at all. So it's impossible to know where >>> the problem is. >> Actually it is, and the bug has been fixed long ago in: >> >> commit e2a305fb13ff0f5cf6ff805555aaa90a5ed5954c >> Author: Christoph Hellwig<hch@lst.de> >> Date: Tue Jan 26 14:49:08 2010 +0100 >> >> block: avoid creating too large iovecs in multiwrite_merge >> >> >> I've asked for it be added to the -stable series but that hasn't >> happened so far. > > Anthony, this looks critical. > It's in stable now. Sounds like a good time to do a 0.12.4. Regards, Anthony Liguori ^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: raw disks no longer work in latest kvm (kvm-88 was fine) 2010-03-06 20:48 raw disks no longer work in latest kvm (kvm-88 was fine) Antoine Martin 2010-03-06 21:28 ` Michael Tokarev @ 2010-03-07 9:36 ` Gleb Natapov 1 sibling, 0 replies; 51+ messages in thread From: Gleb Natapov @ 2010-03-07 9:36 UTC (permalink / raw) To: Antoine Martin; +Cc: kvm On Sun, Mar 07, 2010 at 03:48:23AM +0700, Antoine Martin wrote: > Hi, > > With qemu-kvm-0.12.3: > ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] > [ 1.882843] vdc: > [ 2.365154] udev: starting version 146 > [ 2.693768] end_request: I/O error, dev vdc, sector 126 > [ 2.693772] Buffer I/O error on device vdc, logical block 126 > [ 2.693775] Buffer I/O error on device vdc, logical block 127 > [ 2.693777] Buffer I/O error on device vdc, logical block 128 > [ 2.693779] Buffer I/O error on device vdc, logical block 129 > [ 2.693781] Buffer I/O error on device vdc, logical block 130 > [ 2.693783] Buffer I/O error on device vdc, logical block 131 > [ 2.693785] Buffer I/O error on device vdc, logical block 132 > [ 2.693787] Buffer I/O error on device vdc, logical block 133 > [ 2.693788] Buffer I/O error on device vdc, logical block 134 > [ 2.693814] end_request: I/O error, dev vdc, sector 0 > [ 3.144870] end_request: I/O error, dev vdc, sector 0 > [ 3.499377] end_request: I/O error, dev vdc, sector 0 > [ 3.523247] end_request: I/O error, dev vdc, sector 0 > [ 3.547130] end_request: I/O error, dev vdc, sector 0 > [ 3.550076] end_request: I/O error, dev vdc, sector 0 > > Works fine with kvm-88: > cp /usr/src/KVM/kvm-88/pc-bios/*bin ./ > cp /usr/src/KVM/kvm-88/x86_64-softmmu/qemu-system-x86_64 ./ > ./qemu-system-x86_64 [..] -drive file=/dev/sdc9,if=virtio,cache=none [..] > > [ 1.650274] vdc: unknown partition table > [ 112.704164] EXT4-fs (vdc): mounted filesystem with ordered data mode > > I've tried running as root, using the block device directly (as > shown above) rather than using a softlink, etc.. > > Something broke. > Host and guest are both running 2.6.33 and latest KVM. > Are you sure you have write access to the block device? -- Gleb. ^ permalink raw reply [flat|nested] 51+ messages in thread
end of thread, other threads:[~2010-05-29 11:06 UTC | newest] Thread overview: 51+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-03-06 20:48 raw disks no longer work in latest kvm (kvm-88 was fine) Antoine Martin 2010-03-06 21:28 ` Michael Tokarev 2010-03-07 4:14 ` Antoine Martin 2010-03-07 9:32 ` Michael Tokarev 2010-03-07 10:00 ` Christoph Hellwig 2010-03-07 13:12 ` Antoine Martin 2010-03-07 13:52 ` Antoine Martin 2010-03-07 15:15 ` Michael Tokarev 2010-03-07 17:11 ` Antoine Martin 2010-03-07 17:18 ` Avi Kivity 2010-03-07 17:21 ` Christoph Hellwig 2010-03-07 17:30 ` Avi Kivity 2010-03-07 17:34 ` Christoph Hellwig 2010-03-07 18:43 ` raw disks no longer work in latest kvm (kvm-88 was fine) [SOLVED] Antoine Martin 2010-03-07 18:55 ` Avi Kivity 2010-03-07 18:01 ` raw disks no longer work in latest kvm (kvm-88 was fine) Antoine Martin 2010-03-07 18:47 ` Avi Kivity 2010-03-07 19:07 ` Antoine Martin 2010-03-07 19:10 ` Avi Kivity 2010-03-07 19:13 ` Antoine Martin 2010-03-07 19:17 ` Avi Kivity 2010-03-07 19:25 ` Antoine Martin 2010-03-07 19:35 ` Avi Kivity 2010-03-13 9:51 ` Antoine Martin 2010-03-14 7:34 ` Avi Kivity 2010-04-08 22:00 ` Antoine Martin 2010-05-22 10:44 ` Antoine Martin 2010-05-22 11:17 ` Michael Tokarev 2010-05-22 11:35 ` Antoine Martin 2010-05-23 8:53 ` Antoine Martin 2010-05-23 11:57 ` Avi Kivity 2010-05-23 14:07 ` Antoine Martin 2010-05-23 14:18 ` Avi Kivity 2010-05-23 14:43 ` Antoine Martin 2010-05-23 14:53 ` Antoine Martin 2010-05-23 14:56 ` Avi Kivity 2010-05-23 15:06 ` Antoine Martin 2010-05-23 15:12 ` Avi Kivity 2010-05-23 16:18 ` Antoine Martin 2010-05-23 17:47 ` Stefan Hajnoczi 2010-05-29 9:42 ` Antoine Martin 2010-05-29 9:55 ` Stefan Hajnoczi 2010-05-29 10:34 ` Christoph Hellwig 2010-05-29 11:06 ` Stefan Hajnoczi 2010-05-29 10:34 ` Christoph Hellwig 2010-03-07 19:09 ` Asdo 2010-03-07 19:11 ` Antoine Martin 2010-03-07 19:12 ` Avi Kivity 2010-03-07 16:21 ` Avi Kivity 2010-03-08 16:45 ` Anthony Liguori 2010-03-07 9:36 ` Gleb Natapov
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.