Re: [Qemu-devel] use O_DIRECT to open disk images for IDE failed under xen-4.1.2 and qemu upstream

[Stefan Hajnoczi <stefanha@gmail.com>]

On Sat, May 25, 2013 at 07:32:55AM +0000, Gonglei (Arei) wrote:
> 
> 
> > -----Original Message-----
> > From: Stefan Hajnoczi [mailto:stefanha@gmail.com]
> > Sent: Friday, May 24, 2013 8:20 PM
> > To: Gonglei (Arei)
> > Cc: Stefano Stabellini; Hanweidong; Luonengjun; qemu-devel@nongnu.org;
> > Wangzhenguo; xen-devel@lists.xen.org; Huangweidong (Hardware)
> > Subject: Re: [Qemu-devel] use O_DIRECT to open disk images for IDE failed
> > under xen-4.1.2 and qemu upstream
> > 
> > On Fri, May 24, 2013 at 02:59:05AM +0000, Gonglei (Arei) wrote:
> > > Hi,
> > >
> > > >
> > > > On Thu, 23 May 2013, Gonglei (Arei) wrote:
> > > > > Hi, all
> > > > >
> > > > > I use O_DIRECT to open disk images for IDE, but I'm failed. After debug, I
> > get
> > > > the below logs:
> > > > > [2013-05-22 23:25:46] ide: CMD=c8
> > > > > [2013-05-22 23:25:46] bmdma: readb 0x00 : 0x08
> > > > > [2013-05-22 23:25:46] bmdma: writeb 0x00 : 0x09
> > > > > [2013-05-22 23:25:46] bmdma_cmd_writeb: 0x00000009
> > > > > [2013-05-22 23:25:46] bmdma: readb 0x02 : 0x01
> > > > > [2013-05-22 23:25:46] bmdma: readb 0x02 : 0x01
> > > > > [2013-05-22 23:25:46] bmdma: readb 0x02 : 0x01
> > > > > [2013-05-22 23:25:46] bmdma: readb 0x02 : 0x01
> > > > > [2013-05-22 23:25:46] bmdma: readb 0x02 : 0x01
> > > > > [2013-05-22 23:25:46] bmdma: readb 0x02 : 0x01
> > > > > [2013-05-22 23:26:39] bmdma: writeb 0x00 : 0x08
> > > > > [2013-05-22 23:26:39] bmdma_cmd_writeb: 0x00000008
> > > > > [2013-05-22 23:26:56] ====== offset:0 buf:0x7ff100f21c00 count:512
> > > > aio_offset:0
> > > > > [2013-05-22 23:31:30] ====== offset:0 buf:0x7ff100f21c00 count:512
> > > > aio_offset:0
> > > > > [2013-05-22 23:31:30] ====== handle_aiocb_rw_linear errno: -14
> > > > > [2013-05-22 23:31:30] ====== paio_complete errno=14
> > > > > [2013-05-22 23:31:30] ====== ide_dma_error!!!
> > > > > [2013-05-22 23:31:30] ide: read status addr=0x3f6 val=41
> > > > >
> > > > > QEMU command line :
> > > > > qemu-system-i386 -xen-domid 837 -chardev
> > > > socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-837,server,nowait -mon
> > > > chardev=libxl-cmd,mode=control -name suse11 -vnc 0.0.0.0:1 -serial pty
> > -boot
> > > > order=c -usb -usbdevice tablet -smp 2,maxcpus=2 -device
> > > > rtl8139,id=nic0,netdev=net0,mac=00:16:3e:13:d3:72 -netdev
> > > > type=tap,id=net0,ifname=tap837.0,downscript=no -M xenfv -m 2040 -drive
> > > >
> > file=/mnt/sdd/image/suse.image,if=ide,index=0,media=disk,format=raw,cache
> > > > =none
> > > > >
> > > > > errno 14 shows Bad Address. And I find QEMU_AIO_MISALIGNED flag bit
> > is
> > > > not set through debug.
> > > > >
> > > > >     /*
> > > > >      * If O_DIRECT is used the buffer needs to be aligned on a sector
> > > > >      * boundary.  Check if this is the case or tell the low-level
> > > > >      * driver that it needs to copy the buffer.
> > > > >      */
> > > > >     if ((bs->open_flags & BDRV_O_NOCACHE)) {
> > > > >         if (!bdrv_qiov_is_aligned(bs, qiov)) {    //if the address is
> > > > aligned-512, will no meet the conditions
> > > > >             type |= QEMU_AIO_MISALIGNED;
> > > > > #ifdef CONFIG_LINUX_AIO
> > > > >         } else if (s->use_aio) {
> > > > >             return laio_submit(bs, s->aio_ctx, s->fd, sector_num, qiov,
> > > > >                                nb_sectors, cb, opaque, type);
> > > > > #endif
> > > > >
> > > > > Next process:
> > > > > static ssize_t handle_aiocb_rw(struct qemu_paiocb *aiocb)
> > > > > {
> > > > >     ssize_t nbytes;
> > > > >     char *buf;
> > > > >
> > > > >     if (!(aiocb->aio_type & QEMU_AIO_MISALIGNED)) {
> > > > >         /*
> > > > >          * If there is just a single buffer, and it is properly aligned
> > > > >          * we can just use plain pread/pwrite without any problems.
> > > > >          */
> > > > >         if (aiocb->aio_niov == 1)
> > > > >              return handle_aiocb_rw_linear(aiocb,
> > > > aiocb->aio_iov->iov_base); //this way, and reports errno 14 next
> > > > >
> > > > > Anyone have a good method to resolve this bug? Thanks!
> > > >
> > > > I know that this is not the answer you are looking for but why do you
> > > > want to use O_DIRECT with IDE?
> > > > It should be perfectly safe to use write-back.
> > >
> > > A few days ago, I asked a question about the IDE FLUSH time of guest：
> > > http://lists.nongnu.org/archive/html/qemu-devel/2013-05/msg02642.html
> > >
> > > finally I found that because Qemu use write-back flag to open disk images by
> > default.
> > > so I hope to use O_DIRECT to avoid meeting that problem, but I'm failed
> > under Xen platform with Qemu upstream.
> > 
> > cache=none does not eliminate the need for flush.  If you want to do
> > that, try cache=writethrough or cache=directsync.
> > 
> I had tried it, but I got the same error.

Perhaps that's because the busy bit also needs to be set correctly for
other IDE commands (like read/write requests).

> > Regarding the EFAULT you are seeing, did you check if the I/O buffer
> > address is valid?  Try breaking in gdb and inspecting /proc/<pid>/maps
> > or just x <buffer-address> in gdb to see if the buffer is readable.
> > 
> > Stefan
> 
> Hi, Stefan
> 
> The follows are details in gdb:
> handle_aiocb_rw_linear (aiocb=0x13a0f80, buf=0x7f0a637d7c00 <Address 0x7f0a637d7c00 out of bounds>)
>     at /mnt/sdd/gonglei/xen-4.1.2/tools/qemu-xen/posix-aio-compat.c:213
> 213     {
> (gdb) bt
> #0  handle_aiocb_rw_linear (aiocb=0x13a0f80, buf=0x7f0a637d7c00 <Address 0x7f0a637d7c00 out of bounds>)
>     at /mnt/sdd/gonglei/xen-4.1.2/tools/qemu-xen/posix-aio-compat.c:213
> #1  0x000000000058ba4f in handle_aiocb_rw (aiocb=0x13a0f80) at /mnt/sdd/gonglei/xen-4.1.2/tools/qemu-xen/posix-aio-compat.c:254
> #2  0x000000000058bde2 in aio_thread (unused=0x0) at /mnt/sdd/gonglei/xen-4.1.2/tools/qemu-xen/posix-aio-compat.c:351
> #3  0x00007f0a6a1ec7b6 in start_thread () from /lib64/libpthread.so.0
> #4  0x00007f0a68f0e9cd in clone () from /lib64/libc.so.6
> #5  0x0000000000000000 in ?? ()
> (gdb) x 0x7f0a637d7c00
> 0x7f0a637d7c00: Cannot access memory at address 0x7f0a637d7c00
> (gdb) n
> 214         ssize_t offset = 0;
> (gdb) 
> 217         while (offset < aiocb->aio_nbytes) {
> (gdb) n
> 218              if (aiocb->aio_type & QEMU_AIO_WRITE)
> (gdb) 
> 224                  len = pread(aiocb->aio_fildes,
> (gdb) 
> 229              if (len == -1 && errno == EINTR)
> (gdb) 
> 231              else if (len == -1) {
> (gdb) 
> 232                  offset = -errno;
> (gdb) p errno
> $1 = 14
> 
> linux-CGRmYS:~ # cat /proc/21837/maps 
> 00400000-00846000 r-xp 00000000 08:05 28601                              /usr/lib/xen/bin/qemu-system-i386
> 00a45000-00a46000 r--p 00445000 08:05 28601                              /usr/lib/xen/bin/qemu-system-i386
> 00a46000-00a92000 rw-p 00446000 08:05 28601                              /usr/lib/xen/bin/qemu-system-i386
> 00a92000-01344000 rw-p 00000000 00:00 0                                  [heap]
> 01344000-01345000 rw-p 00000000 00:00 0                                  [heap]
> 01345000-01393000 rw-p 00000000 00:00 0                                  [heap]
> 01393000-01394000 rw-p 00000000 00:00 0                                  [heap]
> 01394000-013ef000 rw-p 00000000 00:00 0                                  [heap]
> 7f0a636d0000-7f0a637d0000 rw-s 00000000 00:03 4026532176                 /proc/xen/privcmd
> 7f0a637d0000-7f0a638d0000 rw-s 00000000 00:03 4026532176                 /proc/xen/privcmd

Interesting, the memory is mapped read/write here.  But it's shared
memory that belongs to /proc/xen/privcmd and maybe doesn't support
O_DIRECT?

I don't know the Xen side of things but why are we doing I/O to
/proc/xen/privcmd?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel