2.6.28.7 domU crashes

2.6.28.7 domU crashes

[Sven Köhler]

Hi,

below is the output of the kernel just before it goes down and crashes. 
I have to use "xm destroy" to kill it.

I have tried the enable/disable the "Optimize for Size" setting in the 
kernel. I hopes that it's some kind of compiler bug because of the 
"invalid opcode" thing. But actually I don't a clue what this might be 
caused by.

Do you have any idea?

The crash is very reproducable.


Regards,
   Sven


-Os

------------[ cut here ]------------
kernel BUG at /usr/src/linux-2.6.28/drivers/block/xen-blkfront.c:243!
invalid opcode: 0000 [#1]
last sysfs file: /sys/kernel/uevent_seqnum
CPU 0
Pid: 4841, comm: ebuild Not tainted 2.6.28.7 #5
RIP: e030:[<ffffffff803f22ef>]  [<ffffffff803f22ef>] 
do_blkif_request+0x156/0x312
RSP: e02b:ffffffff806fee08  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000000
RDX: 0000000000000001 RSI: 000000000000000a RDI: 0000000000002fe0
RBP: ffff88001a96ba50 R08: ffffffff806337f0 R09: ffff88001a96d540
R10: 000000000000003e R11: ffffffff8024d9eb R12: ffff88001a8c66f0
R13: ffff88001a914000 R14: ffff88000ef2d7c0 R15: ffff88001664b788
FS:  00007f712f92a6d0(0000) GS:ffffffff80707000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f712e085024 CR3: 000000001a3de000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ebuild (pid: 4841, threadinfo ffff8800166aa000, task 
ffff8800167a4050)
Stack:
  0000000000000019 ffff88001a968800 ffff88001a914000 000005fc00000000
  000000000000000f 0000000000000002 0000000000000001 ffff88001a968820
  ffffffff00000001 ffff88001a968800 0000000000000000 ffff88001a96ba50
Call Trace:
  <IRQ> <0> [<ffffffff803a8d0a>] ? blk_invoke_request_fn+0x1e/0x3f
  [<ffffffff803f24c3>] ? kick_pending_request_queues+0x18/0x24
  [<ffffffff803f2f7a>] ? blkif_interrupt+0x1a0/0x1c2
  [<ffffffff80248d4b>] ? handle_IRQ_event+0x2c/0x61
  [<ffffffff8024a538>] ? handle_level_irq+0x5c/0xa0
  [<ffffffff802114d4>] ? do_IRQ+0x57/0xb3
  [<ffffffff803d2c68>] ? xen_evtchn_do_upcall+0x89/0xeb
  [<ffffffff8049f22e>] ? xen_do_hypervisor_callback+0x1e/0x30
  <EOI> <0>Code: 00 00 66 41 8b 46 2a 44 0f b7 e0 66 89 44 24 32 49 c1 
e4 04 4d 03 66 48 c7 44 24 34 00 00 00 00 e9 f4 00 00 00 80 7d 01 0b 75 
04 <0f> 0b eb fe 48 bf 00 00 00 00 00 1e 00 00 49 03 3c 24 48 b8 b7
RIP  [<ffffffff803f22ef>] do_blkif_request+0x156/0x312
  RSP <ffffffff806fee08>
Kernel panic - not syncing: Fatal exception in interrupt


-O2

------------[ cut here ]------------
kernel BUG at /usr/src/linux-2.6.28/drivers/block/xen-blkfront.c:243!
invalid opcode: 0000 [#1]
last sysfs file: /sys/kernel/uevent_seqnum
CPU 0
Pid: 4723, comm: ebuild Not tainted 2.6.28.7 #6
RIP: e030:[<ffffffff80449200>]  [<ffffffff80449200>] 
do_blkif_request+0x2f0/0x380
RSP: e02b:ffffffff80783df8  EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff88000505da40 RCX: ffff88000505da40
RDX: ffff88001a8c65a0 RSI: 000000000000000a RDI: 0000000000000520
RBP: ffff88001a96b3c0 R08: 0000000000002900 R09: 0000000000000000
R10: 0000000000000000 R11: 000000000000001a R12: 0000000000000520
R13: 0000000000000002 R14: ffff88001a8c65b0 R15: 0000000000000000
FS:  00007fd37d7596d0(0000) GS:ffffffff8078c000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000006c4900 CR3: 00000000162c9000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ebuild (pid: 4723, threadinfo ffff88001638c000, task 
ffff880016382450)
Stack:
  000000000000004c ffff88001a966800 ffff88001a914000 ffff88000501fd48
  000000008027c3d2 0000000000000015 ffff88001a914000 00000000803f0804
  ffff88000505da40 ffff88001a966820 ffffffff00000001 ffff88001a966800
Call Trace:
  <IRQ> <0> [<ffffffff803f0c73>] ? blk_invoke_request_fn+0x33/0x40
  [<ffffffff804492a8>] ? kick_pending_request_queues+0x18/0x30
  [<ffffffff80449ef3>] ? blkif_interrupt+0x1a3/0x1e0
  [<ffffffff80255639>] ? handle_IRQ_event+0x39/0x80
  [<ffffffff80257532>] ? handle_level_irq+0x52/0xc0
  [<ffffffff80212a3c>] ? do_IRQ+0x5c/0xd0
  [<ffffffff80423295>] ? xen_evtchn_do_upcall+0xa5/0xe0
  [<ffffffff802379db>] ? __do_softirq+0xab/0x140
  [<ffffffff805172ce>] ? xen_do_hypervisor_callback+0x1e/0x30
  <EOI> <0>Code: fa d0 00 00 00 48 8d bc 07 88 00 00 00 e8 89 c2 fb ff 
8b 7c 24 54 e8 20 93 fd ff ff 44 24 24 e9 3b fd ff ff 0f 0b eb fe 0f 1f 
00 <0f> 0b eb fe 48 8b 7c 24 30 48 8b 54 24 30 b9 0b 00 00 00 48 c7
RIP  [<ffffffff80449200>] do_blkif_request+0x2f0/0x380
  RSP <ffffffff80783df8>
Kernel panic - not syncing: Fatal exception in interrupt

Re: 2.6.28.7 domU crashes

[Jeremy Fitzhardinge]

Sven Köhler wrote:
> Hi,
>
> below is the output of the kernel just before it goes down and 
> crashes. I have to use "xm destroy" to kill it.
>
> I have tried the enable/disable the "Optimize for Size" setting in the 
> kernel. I hopes that it's some kind of compiler bug because of the 
> "invalid opcode" thing. But actually I don't a clue what this might be 
> caused by.
>
> Do you have any idea?
>
> The crash is very reproducable.

Looks like the crash fixed by the change below 
(c7241227f61ca6606a7fa3555391360d92bd8d9b in linux-2.6.git)

    J

>From c7241227f61ca6606a7fa3555391360d92bd8d9b Mon Sep 17 00:00:00 2001
From: Jens Axboe <jens.axboe@oracle.com>
Date: Mon, 16 Feb 2009 13:18:28 -0800
Subject: [PATCH] xen/blkfront: use blk_rq_map_sg to generate ring entries

On occasion, the request will apparently have more segments than we
fit into the ring. Jens says:

> The second problem is that the block layer then appears to create one
> too many segments, but from the dump it has rq->nr_phys_segments ==
> BLKIF_MAX_SEGMENTS_PER_REQUEST. I suspect the latter is due to
> xen-blkfront not handling the merging on its own. It should check that
> the new page doesn't form part of the previous page. The
> rq_for_each_segment() iterates all single bits in the request, not dma
> segments. The "easiest" way to do this is to call blk_rq_map_sg() and
> then iterate the mapped sg list. That will give you what you are
> looking for.

> Here's a test patch, compiles but otherwise untested. I spent more
> time figuring out how to enable XEN than to code it up, so YMMV!
> Probably the sg list wants to be put inside the ring and only
> initialized on allocation, then you can get rid of the sg on stack and
> sg_init_table() loop call in the function. I'll leave that, and the
> testing, to you.

[Moved sg array into info structure, and initialize once. -J]

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index 918ef72..b6c8ce2 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -40,6 +40,7 @@
 #include <linux/hdreg.h>
 #include <linux/cdrom.h>
 #include <linux/module.h>
+#include <linux/scatterlist.h>
 
 #include <xen/xenbus.h>
 #include <xen/grant_table.h>
@@ -82,6 +83,7 @@ struct blkfront_info
 	enum blkif_state connected;
 	int ring_ref;
 	struct blkif_front_ring ring;
+	struct scatterlist sg[BLKIF_MAX_SEGMENTS_PER_REQUEST];
 	unsigned int evtchn, irq;
 	struct request_queue *rq;
 	struct work_struct work;
@@ -204,12 +206,11 @@ static int blkif_queue_request(struct request *req)
 	struct blkfront_info *info = req->rq_disk->private_data;
 	unsigned long buffer_mfn;
 	struct blkif_request *ring_req;
-	struct req_iterator iter;
-	struct bio_vec *bvec;
 	unsigned long id;
 	unsigned int fsect, lsect;
-	int ref;
+	int i, ref;
 	grant_ref_t gref_head;
+	struct scatterlist *sg;
 
 	if (unlikely(info->connected != BLKIF_STATE_CONNECTED))
 		return 1;
@@ -238,12 +239,13 @@ static int blkif_queue_request(struct request *req)
 	if (blk_barrier_rq(req))
 		ring_req->operation = BLKIF_OP_WRITE_BARRIER;
 
-	ring_req->nr_segments = 0;
-	rq_for_each_segment(bvec, req, iter) {
-		BUG_ON(ring_req->nr_segments == BLKIF_MAX_SEGMENTS_PER_REQUEST);
-		buffer_mfn = pfn_to_mfn(page_to_pfn(bvec->bv_page));
-		fsect = bvec->bv_offset >> 9;
-		lsect = fsect + (bvec->bv_len >> 9) - 1;
+	ring_req->nr_segments = blk_rq_map_sg(req->q, req, info->sg);
+	BUG_ON(ring_req->nr_segments > BLKIF_MAX_SEGMENTS_PER_REQUEST);
+
+	for_each_sg(info->sg, sg, ring_req->nr_segments, i) {
+		buffer_mfn = pfn_to_mfn(page_to_pfn(sg_page(sg)));
+		fsect = sg->offset >> 9;
+		lsect = fsect + (sg->length >> 9) - 1;
 		/* install a grant reference. */
 		ref = gnttab_claim_grant_reference(&gref_head);
 		BUG_ON(ref == -ENOSPC);
@@ -254,16 +256,12 @@ static int blkif_queue_request(struct request *req)
 				buffer_mfn,
 				rq_data_dir(req) );
 
-		info->shadow[id].frame[ring_req->nr_segments] =
-				mfn_to_pfn(buffer_mfn);
-
-		ring_req->seg[ring_req->nr_segments] =
+		info->shadow[id].frame[i] = mfn_to_pfn(buffer_mfn);
+		ring_req->seg[i] =
 				(struct blkif_request_segment) {
 					.gref       = ref,
 					.first_sect = fsect,
 					.last_sect  = lsect };
-
-		ring_req->nr_segments++;
 	}
 
 	info->ring.req_prod_pvt++;
@@ -622,6 +620,8 @@ static int setup_blkring(struct xenbus_device *dev,
 	SHARED_RING_INIT(sring);
 	FRONT_RING_INIT(&info->ring, sring, PAGE_SIZE);
 
+	sg_init_table(info->sg, BLKIF_MAX_SEGMENTS_PER_REQUEST);
+
 	err = xenbus_grant_ring(dev, virt_to_mfn(info->ring.sring));
 	if (err < 0) {
 		free_page((unsigned long)sring);

Re: 2.6.28.7 domU crashes

[Sven Köhler]

[-- Attachment #1.1: Type: text/plain, Size: 665 bytes --]

Jeremy Fitzhardinge schrieb:
> Sven Köhler wrote:
>> Hi,
>>
>> below is the output of the kernel just before it goes down and
>> crashes. I have to use "xm destroy" to kill it.
>>
>> I have tried the enable/disable the "Optimize for Size" setting in the
>> kernel. I hopes that it's some kind of compiler bug because of the
>> "invalid opcode" thing. But actually I don't a clue what this might be
>> caused by.
>>
>> Do you have any idea?
>>
>> The crash is very reproducable.
> 
> Looks like the crash fixed by the change below
> (c7241227f61ca6606a7fa3555391360d92bd8d9b in linux-2.6.git)

I have tested 2.6.29-rc7. It works stable for me.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel