From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?B?U8OpYmFzdGllbiBSaWNjaW8=?= Subject: Re: xm/xl block-detach issue Date: Tue, 12 Jul 2011 07:21:38 +0200 Message-ID: <4E1BD9E2.6030709@swisscenter.com> References: <4E186702.6050601@swisscenter.com> <4E188FC3.1010009@swisscenter.com> <1310288759.2685.356.camel@ramone> <4E19961F.9070109@swisscenter.com> <1310321977.2685.370.camel@ramone> <4E1B2E5C.2020402@swisscenter.com> <1310408578.14003.78.camel@agari.van.xensource.com> <4E1B6CFC.9050209@swisscenter.com> <1310420912.16930.21.camel@agari.van.xensource.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0146377647==" Return-path: In-Reply-To: <1310420912.16930.21.camel@agari.van.xensource.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Daniel Stodden Cc: "dmeyer@cs.ubc.ca" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --===============0146377647== Content-Type: multipart/alternative; boundary="------------060104060900000608050402" This is a multi-part message in MIME format. --------------060104060900000608050402 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mail.swisslink.ch id p6C5LmMX029343 Hi again :) Going further with my investigations, I'm definitively now sure that=20 it's not an issue with ocfs2. Same behavior occures on a local file. There is really something going wrong with xm/xl. In order to continue=20 my tests I've tried to attach and .img (raw) file and this time not attaching it to dom0 but to a=20 running domU vm: no way with xl box# xl block-attach 2=20 file:/cloud/data2/images/xen/slackware.13-37.x86.20110428/slackware.13-37= .x86.20110428.img=20 xvdc w libxl_device_disk_add failed. with xm box# xm block-attach 2=20 file:/cloud/data2/images/xen/slackware.13-37.x86.20110428/slackware.13-37= .x86.20110428.img=20 xvdc w this worked. Now trying to list with xl : box# xl block-list 2 Vdev BE handle state evt-ch ring-ref BE-path Segmentation fault box# xm block-list 2 Vdev BE handle state evt-ch ring-ref BE-path 51712 0 0 4 23 8 /local/domain/0/backend/vbd/2/51712 51728 0 0 4 24 17 /local/domain/0/backend/vbd/2/51728 *51744 0 0 4 25 50 /local/domain/0/backend/vbd/2/5174= 4 * Unplugging with xm: box# xm block-detach 2 51744 box# xm block-list 2 Vdev BE handle state evt-ch ring-ref BE-path 51712 0 0 4 23 8 /local/domain/0/backend/vbd/2/51712 51728 0 0 4 24 17 /local/domain/0/backend/vbd/2/51728 Gone :) Summary: 1) block-attach / block-detach / block-list actions issued from xl and=20 xm are not having the same behaviors, xl tend to crash a lot and mess things. 2) attaching an .img file with xm and file:/ let me detach it with xm=20 block-detach 3) attaching a .vhd file with xm and tap:vhd:/ does *not* let me detach=20 it with xm block-detach I almost got rid of all specialities on my box, except maybe konrad's=20 2.6.39.2 kernel. This is a quite serious issue, am I the only one able=20 to reproduce it ? For information, we've tried the same action (xl block-attach and xl=20 block-detach of a vhd on XenServer 6.0 beta (project boston) and the same issue happens, can't detach it, then it throws ugly messages at console: Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: ------------[ cut here ]------------ Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: invalid opcode: 0000 [#1] SMP Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: last sysfs file:=20 /sys/devices/xen-backend/vbd-0-51712/statistics/rd_usecs Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Process xenwatch (pid: 48, ti=3Dee9d0000 task=3Dee8d50= 70=20 task.ti=3Dee9d0000) Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Stack: Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Call Trace: Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: Code: 88 ff ff e9 5c ff ff ff 89 44 24 04 c7 44 24 08=20 2e a1 44 c0 8b 4d ec 89 0c 24 e8 62 88 ff ff 89 f8 e8 db ee ff ff e9 f7=20 fe ff ff <0f> 0b eb fe 66 90 ba 98 00 00 00 b8 ba a0 44 c0 e8 61 48 e8 ff Message from syslogd@ at Mon Jul 11 21:13:43 2011 ... xenblade13 kernel: EIP: [] blkback_queue_start+0x2ca/0x2f0=20 SS:ESP 0069:ee9d1f38 So my conclusions are that something is not right with xl/xm commands=20 and block attaching/detaching. I guess XenServer is not using this method to attach/detach disks as attaching them with xe commands=20 and with xencenter do not show any problems. Could someone try it on their side and confirms if this is a confirmed=20 problem ? That would be nice to fix it if it's the case, as shipping xen=20 with a "russian roulette" enabled xm/xl is maybe not the best :) I can provide root access to my test machines if you want to try things=20 on the best :) Cheers, S=C3=A9bastien --------------060104060900000608050402 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mail.swisslink.ch id p6C5LmMX029343 Hi again :)

Going further with my investigations, I'm definitively now sure that it's not an issue with ocfs2.
Same behavior occures on a local file.

There is really something going wrong with xm/xl. In order to continue my tests I've tried to attach
and .img (raw) file and this time not attaching it to dom0 but to a running domU vm:

no way with xl

box# xl block-attach 2 file:/c= loud/data2/images/xen/slackware.13-37.x86.20110428/slackware.13-37.x86.20= 110428.img xvdc w
libxl_device_disk_add failed.

with xm

box# xm block-attach 2 file:/c= loud/data2/images/xen/slackware.13-37.x86.20110428/slackware.13-37.x86.20= 110428.img xvdc w

this worked. Now trying to list with xl :

box# xl block-list 2
Vdev=C2=A0 BE=C2=A0 handle state evt-ch ring-ref BE-path=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
Segmentation fault

box# xm block-list 2
Vdev=C2=A0 BE handle state evt-ch ring-ref BE-path
51712=C2=A0 0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 23=C2=A0=C2=A0=C2=A0=C2=A0 8=C2=A0=C2=A0=C2=A0=C2=A0 /local/domain/0/backend/vbd/2/51712=C2=A0
51728=C2=A0 0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 24=C2=A0=C2=A0=C2=A0=C2=A0 17=C2=A0=C2=A0=C2=A0 /local/domain/0/backend/vbd/2/51728=C2=A0
51744=C2=A0 0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 25=C2=A0=C2=A0=C2=A0=C2=A0 50=C2=A0=C2=A0=C2=A0 /local/domain/0/backend/vbd/2/51744=C2=A0

Unplugging with xm:

box# xm block-detach 2 51744

box# xm block-list 2
Vdev=C2=A0 BE handle state evt-ch ring-ref BE-path
51712=C2=A0 0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 23=C2=A0=C2=A0=C2=A0=C2=A0 8=C2=A0=C2=A0=C2=A0=C2=A0 /local/domain/0/backend/vbd/2/51712=C2=A0
51728=C2=A0 0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 24=C2=A0=C2=A0=C2=A0=C2=A0 17=C2=A0=C2=A0=C2=A0 /local/domain/0/backend/vbd/2/51728=C2=A0

Gone :)


Summary:

1) block-attach / block-detach / block-list actions issued from xl and xm are not having the same behaviors,
xl tend to crash a lot and mess things.

2) attaching an .img file with xm and file:/ let me detach it with xm block-detach

3) attaching=C2=A0 a .vhd file with xm and tap:vhd:/ does not = let me detach it with xm block-detach

I almost got rid of all specialities on my box, except maybe konrad's 2.6.39.2 kernel. This is a quite serious issue, am I the only one able to reproduce it ?

For information, we've tried the same action (xl block-attach and xl block-detach of a vhd on XenServer 6.0 beta (project boston) and the same
issue happens, can't detach it, then it throws ugly messages at console:

Message from syslogd@ at Mon Jul 11 21:13:43 2011 ...
xenblade13 kernel: ------------[ cut here ]------------

Message from syslogd@ at Mon Jul 11 21:13:43 2011 ...
xenblade13 kernel: invalid opcode: 0000 [#1] SMP

Message from syslogd@ at Mon Jul 11 21:13:43 2011 ...
xenblade13 kernel: last sysfs file: /sys/devices/xen-backend/vbd-0-51712/statistics/rd_usecs

Message from syslogd@ at Mon Jul 11 21:13:43 2011 ...
xenblade13 kernel: Process xenwatch (pid: 48, ti=3Dee9d0000 task=3Dee8d5070 task.ti=3Dee9d0000)

Message from syslogd@ at Mon Jul 11 21:13:43 2011 ...
xenblade13 kernel: Stack:

Message from syslogd@ at Mon Jul 11 21:13:43 2011 ...
xenblade13 kernel: Call Trace:

Message from syslogd@ at Mon Jul 11 21:13:43 2011 ...
xenblade13 kernel: Code: 88 ff ff e9 5c ff ff ff 89 44 24 04 c7 44 24 08 2e a1 44 c0 8b 4d ec 89 0c 24 e8 62 88 ff ff 89 f8 e8 db ee ff ff e9 f7 fe ff ff <0f> 0b eb fe 66 90 ba 98 00 00 00 b8 ba a0 44 c0 e8 61 48 e8 ff

Message from syslogd@ at Mon Jul 11 21:13:43 2011 ...
xenblade13 kernel: EIP: [<c02a985a>] blkback_queue_start+0x2ca/0x2f0 SS:ESP 0069:ee9d1f38


So my conclusions are that something is not right with xl/xm commands and block attaching/detaching. I guess XenServer is not using
this method to attach/detach disks as attaching them with xe commands and with xencenter do not show any problems.

Could someone try it on their side and confirms if this is a confirmed problem ? That would be nice to fix it if it's the case, as shipping xen with a "russian roulette" enabled xm/xl is maybe not the best :)

I can provide root access to my test machines if you want to try things on the best :)

Cheers,
S=C3=A9bastien






--------------060104060900000608050402-- --===============0146377647== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0146377647==--