2.4.22-pre3 and reiserfs boot problem

2.4.22-pre3 and reiserfs boot problem

["Peter Lojkin"]

Hello,

I am not on the list so please CC me if replying...

I've found the problem, it's patch with description:

Fix potential IO hangs and increase interactiveness during heavy IO

http://linux.bkbits.net:8080/linux-2.4/user=mason/cset@1.1024?nav=!-|index.html|stats|!+|index.html|ChangeSet@-7d

After removing all changes from this cset, a had no problems
mounting big reiserfs volumes...

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Thu, 10 Jul 2003 19:49:20 +0400
"Peter Lojkin" <ia6432@inbox.ru> wrote:

> Hello,
> 
> I am not on the list so please CC me if replying...
> 
> I've found the problem, it's patch with description:
> 
> Fix potential IO hangs and increase interactiveness during heavy IO
> 
> http://linux.bkbits.net:8080/linux-2.4/user=mason/cset@1.1024?nav=!-|index.html|stats|!+|index.html|ChangeSet@-7d
> 
> After removing all changes from this cset, a had no problems
> mounting big reiserfs volumes...

Hello Marcelo,

can you please send me a separated patch for reversal to verify this.

Hello Chris,

if this is true I am willing to test other versions of the questionable patch
to solve the issue.

Thank you.
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Marcelo Tosatti]

On Thu, 10 Jul 2003, Stephan von Krawczynski wrote:

> On Thu, 10 Jul 2003 19:49:20 +0400
> "Peter Lojkin" <ia6432@inbox.ru> wrote:
>
> > Hello,
> >
> > I am not on the list so please CC me if replying...
> >
> > I've found the problem, it's patch with description:
> >
> > Fix potential IO hangs and increase interactiveness during heavy IO
> >
> > http://linux.bkbits.net:8080/linux-2.4/user=mason/cset@1.1024?nav=!-|index.html|stats|!+|index.html|ChangeSet@-7d
> >
> > After removing all changes from this cset, a had no problems
> > mounting big reiserfs volumes...
>
> Hello Marcelo,
>
> can you please send me a separated patch for reversal to verify this.

Its attached (iostalls). Use patch -R.

Also, when the hang happens, sysrq works right?

If so get us the tasks backtraces with sysrq, ok?

Re: 2.4.22-pre3 and reiserfs boot problem

[Marcelo Tosatti]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 960 bytes --]


I forgot to attach the patch. Now its attached, duh.

On Thu, 10 Jul 2003, Marcelo Tosatti wrote:

>
>
> On Thu, 10 Jul 2003, Stephan von Krawczynski wrote:
>
> > On Thu, 10 Jul 2003 19:49:20 +0400
> > "Peter Lojkin" <ia6432@inbox.ru> wrote:
> >
> > > Hello,
> > >
> > > I am not on the list so please CC me if replying...
> > >
> > > I've found the problem, it's patch with description:
> > >
> > > Fix potential IO hangs and increase interactiveness during heavy IO
> > >
> > > http://linux.bkbits.net:8080/linux-2.4/user=mason/cset@1.1024?nav=!-|index.html|stats|!+|index.html|ChangeSet@-7d
> > >
> > > After removing all changes from this cset, a had no problems
> > > mounting big reiserfs volumes...
> >
> > Hello Marcelo,
> >
> > can you please send me a separated patch for reversal to verify this.
>
> Its attached (iostalls). Use patch -R.
>
> Also, when the hang happens, sysrq works right?
>
> If so get us the tasks backtraces with sysrq, ok?
>
>

[-- Attachment #2: Type: TEXT/PLAIN, Size: 23910 bytes --]

# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
#	           ChangeSet	1.1023  -> 1.1024 
#	drivers/ide/ide-probe.c	1.17    -> 1.18   
#	include/linux/pagemap.h	1.19    -> 1.20   
#	      kernel/ksyms.c	1.72    -> 1.73   
#	include/linux/elevator.h	1.5     -> 1.6    
#	drivers/block/ll_rw_blk.c	1.45    -> 1.46   
#	include/linux/blkdev.h	1.23    -> 1.24   
#	 fs/reiserfs/inode.c	1.47    -> 1.48   
#	        mm/filemap.c	1.81    -> 1.82   
#	drivers/scsi/scsi_lib.c	1.16    -> 1.17   
#	 drivers/scsi/scsi.c	1.17    -> 1.18   
#	         fs/buffer.c	1.86    -> 1.87   
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/07/05	mason@suse.com	1.1024
# [PATCH] Fix potential IO hangs and increase interactiveness during heavy IO
# 
# io-stalls-10:
# 
# 
# ===== drivers/block/ll_rw_blk.c 1.45 vs edited =====
# --------------------------------------------
#
diff -Nru a/drivers/block/ll_rw_blk.c b/drivers/block/ll_rw_blk.c
--- a/drivers/block/ll_rw_blk.c	Thu Jul 10 14:52:38 2003
+++ b/drivers/block/ll_rw_blk.c	Thu Jul 10 14:52:38 2003
@@ -176,11 +176,12 @@
 {
 	int count = q->nr_requests;
 
-	count -= __blk_cleanup_queue(&q->rq[READ]);
-	count -= __blk_cleanup_queue(&q->rq[WRITE]);
+	count -= __blk_cleanup_queue(&q->rq);
 
 	if (count)
 		printk("blk_cleanup_queue: leaked requests (%d)\n", count);
+	if (atomic_read(&q->nr_sectors))
+		printk("blk_cleanup_queue: leaked sectors (%d)\n", atomic_read(&q->nr_sectors));
 
 	memset(q, 0, sizeof(*q));
 }
@@ -215,6 +216,24 @@
 }
 
 /**
+ * blk_queue_throttle_sectors - indicates you will call sector throttling funcs
+ * @q:       The queue which this applies to.
+ * @active:  A flag indication if you want sector throttling on
+ *
+ * Description:
+ * The sector throttling code allows us to put a limit on the number of
+ * sectors pending io to the disk at a given time, sending @active nonzero
+ * indicates you will call blk_started_sectors and blk_finished_sectors in
+ * addition to calling blk_started_io and blk_finished_io in order to
+ * keep track of the number of sectors in flight.
+ **/
+ 
+void blk_queue_throttle_sectors(request_queue_t * q, int active)
+{
+	q->can_throttle = active;
+}
+
+/**
  * blk_queue_make_request - define an alternate make_request function for a device
  * @q:  the request queue for the device to be affected
  * @mfn: the alternate make_request function
@@ -389,7 +408,7 @@
  *
  * Returns the (new) number of requests which the queue has available.
  */
-int blk_grow_request_list(request_queue_t *q, int nr_requests)
+int blk_grow_request_list(request_queue_t *q, int nr_requests, int max_queue_sectors)
 {
 	unsigned long flags;
 	/* Several broken drivers assume that this function doesn't sleep,
@@ -399,21 +418,34 @@
 	spin_lock_irqsave(&io_request_lock, flags);
 	while (q->nr_requests < nr_requests) {
 		struct request *rq;
-		int rw;
 
 		rq = kmem_cache_alloc(request_cachep, SLAB_ATOMIC);
 		if (rq == NULL)
 			break;
 		memset(rq, 0, sizeof(*rq));
 		rq->rq_status = RQ_INACTIVE;
-		rw = q->nr_requests & 1;
-		list_add(&rq->queue, &q->rq[rw].free);
-		q->rq[rw].count++;
+ 		list_add(&rq->queue, &q->rq.free);
+ 		q->rq.count++;
+
 		q->nr_requests++;
 	}
+
+ 	/*
+ 	 * Wakeup waiters after both one quarter of the
+ 	 * max-in-fligh queue and one quarter of the requests
+ 	 * are available again.
+ 	 */
+
 	q->batch_requests = q->nr_requests / 4;
 	if (q->batch_requests > 32)
 		q->batch_requests = 32;
+ 	q->batch_sectors = max_queue_sectors / 4;
+ 
+ 	q->max_queue_sectors = max_queue_sectors;
+ 
+ 	BUG_ON(!q->batch_sectors);
+ 	atomic_set(&q->nr_sectors, 0);
+
 	spin_unlock_irqrestore(&io_request_lock, flags);
 	return q->nr_requests;
 }
@@ -422,23 +454,27 @@
 {
 	struct sysinfo si;
 	int megs;		/* Total memory, in megabytes */
-	int nr_requests;
-
-	INIT_LIST_HEAD(&q->rq[READ].free);
-	INIT_LIST_HEAD(&q->rq[WRITE].free);
-	q->rq[READ].count = 0;
-	q->rq[WRITE].count = 0;
+ 	int nr_requests, max_queue_sectors = MAX_QUEUE_SECTORS;
+  
+ 	INIT_LIST_HEAD(&q->rq.free);
+	q->rq.count = 0;
 	q->nr_requests = 0;
 
 	si_meminfo(&si);
 	megs = si.totalram >> (20 - PAGE_SHIFT);
-	nr_requests = 128;
-	if (megs < 32)
-		nr_requests /= 2;
-	blk_grow_request_list(q, nr_requests);
+ 	nr_requests = MAX_NR_REQUESTS;
+ 	if (megs < 30) {
+  		nr_requests /= 2;
+ 		max_queue_sectors /= 2;
+ 	}
+ 	/* notice early if anybody screwed the defaults */
+ 	BUG_ON(!nr_requests);
+ 	BUG_ON(!max_queue_sectors);
+ 
+ 	blk_grow_request_list(q, nr_requests, max_queue_sectors);
+
+ 	init_waitqueue_head(&q->wait_for_requests);
 
-	init_waitqueue_head(&q->wait_for_requests[0]);
-	init_waitqueue_head(&q->wait_for_requests[1]);
 	spin_lock_init(&q->queue_lock);
 }
 
@@ -491,6 +527,8 @@
 	q->plug_tq.routine	= &generic_unplug_device;
 	q->plug_tq.data		= q;
 	q->plugged        	= 0;
+	q->can_throttle		= 0;
+
 	/*
 	 * These booleans describe the queue properties.  We set the
 	 * default (and most common) values here.  Other drivers can
@@ -511,9 +549,10 @@
 static struct request *get_request(request_queue_t *q, int rw)
 {
 	struct request *rq = NULL;
-	struct request_list *rl = q->rq + rw;
+	struct request_list *rl;
 
-	if (!list_empty(&rl->free)) {
+	rl = &q->rq;
+	if (!list_empty(&rl->free) && !blk_oversized_queue(q)) {
 		rq = blkdev_free_rq(&rl->free);
 		list_del(&rq->queue);
 		rl->count--;
@@ -522,34 +561,23 @@
 		rq->special = NULL;
 		rq->q = q;
 	}
-
 	return rq;
 }
 
 /*
- * Here's the request allocation design:
+ * Here's the request allocation design, low latency version:
  *
  * 1: Blocking on request exhaustion is a key part of I/O throttling.
  * 
  * 2: We want to be `fair' to all requesters.  We must avoid starvation, and
  *    attempt to ensure that all requesters sleep for a similar duration.  Hence
  *    no stealing requests when there are other processes waiting.
- * 
- * 3: We also wish to support `batching' of requests.  So when a process is
- *    woken, we want to allow it to allocate a decent number of requests
- *    before it blocks again, so they can be nicely merged (this only really
- *    matters if the process happens to be adding requests near the head of
- *    the queue).
- * 
- * 4: We want to avoid scheduling storms.  This isn't really important, because
- *    the system will be I/O bound anyway.  But it's easy.
- * 
- *    There is tension between requirements 2 and 3.  Once a task has woken,
- *    we don't want to allow it to sleep as soon as it takes its second request.
- *    But we don't want currently-running tasks to steal all the requests
- *    from the sleepers.  We handle this with wakeup hysteresis around
- *    0 .. batch_requests and with the assumption that request taking is much,
- *    much faster than request freeing.
+ *
+ * There used to be more here, attempting to allow a process to send in a
+ * number of requests once it has woken up.  But, there's no way to 
+ * tell if a process has just been woken up, or if it is a new process
+ * coming in to steal requests from the waiters.  So, we give up and force
+ * everyone to wait fairly.
  * 
  * So here's what we do:
  * 
@@ -561,28 +589,23 @@
  * 
  *  When a process wants a new request:
  * 
- *    b) If free_requests == 0, the requester sleeps in FIFO manner.
- * 
- *    b) If 0 <  free_requests < batch_requests and there are waiters,
- *       we still take a request non-blockingly.  This provides batching.
- *
- *    c) If free_requests >= batch_requests, the caller is immediately
- *       granted a new request.
+ *    b) If free_requests == 0, the requester sleeps in FIFO manner, and
+ *       the queue full condition is set.  The full condition is not
+ *       cleared until there are no longer any waiters.  Once the full
+ *       condition is set, all new io must wait, hopefully for a very
+ *       short period of time.
  * 
  *  When a request is released:
  * 
- *    d) If free_requests < batch_requests, do nothing.
- * 
- *    f) If free_requests >= batch_requests, wake up a single waiter.
+ *    c) If free_requests < batch_requests, do nothing.
  * 
- *   The net effect is that when a process is woken at the batch_requests level,
- *   it will be able to take approximately (batch_requests) requests before
- *   blocking again (at the tail of the queue).
- * 
- *   This all assumes that the rate of taking requests is much, much higher
- *   than the rate of releasing them.  Which is very true.
+ *    d) If free_requests >= batch_requests, wake up a single waiter.
  *
- * -akpm, Feb 2002.
+ *   As each waiter gets a request, he wakes another waiter.  We do this
+ *   to prevent a race where an unplug might get run before a request makes
+ *   it's way onto the queue.  The result is a cascade of wakeups, so delaying
+ *   the initial wakeup until we've got batch_requests available helps avoid
+ *   wakeups where there aren't any requests available yet.
  */
 
 static struct request *__get_request_wait(request_queue_t *q, int rw)
@@ -590,21 +613,37 @@
 	register struct request *rq;
 	DECLARE_WAITQUEUE(wait, current);
 
-	add_wait_queue(&q->wait_for_requests[rw], &wait);
+	add_wait_queue_exclusive(&q->wait_for_requests, &wait);
+
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
-		generic_unplug_device(q);
-		if (q->rq[rw].count == 0)
-			schedule();
 		spin_lock_irq(&io_request_lock);
+		if (blk_oversized_queue(q)) {
+			__generic_unplug_device(q);
+			spin_unlock_irq(&io_request_lock);
+			schedule();
+			spin_lock_irq(&io_request_lock);
+		}
 		rq = get_request(q, rw);
 		spin_unlock_irq(&io_request_lock);
 	} while (rq == NULL);
-	remove_wait_queue(&q->wait_for_requests[rw], &wait);
+	remove_wait_queue(&q->wait_for_requests, &wait);
 	current->state = TASK_RUNNING;
+
 	return rq;
 }
 
+static void get_request_wait_wakeup(request_queue_t *q, int rw)
+{
+	/*
+	 * avoid losing an unplug if a second __get_request_wait did the
+	 * generic_unplug_device while our __get_request_wait was running
+	 * w/o the queue_lock held and w/ our request out of the queue.
+	 */	
+	if (waitqueue_active(&q->wait_for_requests))
+		wake_up(&q->wait_for_requests);
+}
+
 /* RO fail safe mechanism */
 
 static long ro_bits[MAX_BLKDEV][8];
@@ -818,7 +857,6 @@
 void blkdev_release_request(struct request *req)
 {
 	request_queue_t *q = req->q;
-	int rw = req->cmd;
 
 	req->rq_status = RQ_INACTIVE;
 	req->q = NULL;
@@ -828,9 +866,17 @@
 	 * assume it has free buffers and check waiters
 	 */
 	if (q) {
-		list_add(&req->queue, &q->rq[rw].free);
-		if (++q->rq[rw].count >= q->batch_requests)
-			wake_up(&q->wait_for_requests[rw]);
+		int oversized_batch = 0;
+
+		if (q->can_throttle)
+			oversized_batch = blk_oversized_queue_batch(q);
+		q->rq.count++;
+		list_add(&req->queue, &q->rq.free);
+		if (q->rq.count >= q->batch_requests && !oversized_batch) {
+			smp_mb();
+			if (waitqueue_active(&q->wait_for_requests))
+				wake_up(&q->wait_for_requests);
+		}
 	}
 }
 
@@ -908,6 +954,7 @@
 	struct list_head *head, *insert_here;
 	int latency;
 	elevator_t *elevator = &q->elevator;
+	int should_wake = 0;
 
 	count = bh->b_size >> 9;
 	sector = bh->b_rsector;
@@ -948,7 +995,6 @@
 	 */
 	max_sectors = get_max_sectors(bh->b_rdev);
 
-again:
 	req = NULL;
 	head = &q->queue_head;
 	/*
@@ -957,7 +1003,9 @@
 	 */
 	spin_lock_irq(&io_request_lock);
 
+again:
 	insert_here = head->prev;
+
 	if (list_empty(head)) {
 		q->plug_device_fn(q, bh->b_rdev); /* is atomic */
 		goto get_rq;
@@ -976,6 +1024,7 @@
 			req->bhtail = bh;
 			req->nr_sectors = req->hard_nr_sectors += count;
 			blk_started_io(count);
+			blk_started_sectors(req, count);
 			drive_stat_acct(req->rq_dev, req->cmd, count, 0);
 			req_new_io(req, 1, count);
 			attempt_back_merge(q, req, max_sectors, max_segments);
@@ -998,6 +1047,7 @@
 			req->sector = req->hard_sector = sector;
 			req->nr_sectors = req->hard_nr_sectors += count;
 			blk_started_io(count);
+			blk_started_sectors(req, count);
 			drive_stat_acct(req->rq_dev, req->cmd, count, 0);
 			req_new_io(req, 1, count);
 			attempt_front_merge(q, head, req, max_sectors, max_segments);
@@ -1030,7 +1080,7 @@
 		 * See description above __get_request_wait()
 		 */
 		if (rw_ahead) {
-			if (q->rq[rw].count < q->batch_requests) {
+			if (q->rq.count < q->batch_requests || blk_oversized_queue_batch(q)) {
 				spin_unlock_irq(&io_request_lock);
 				goto end_io;
 			}
@@ -1042,6 +1092,9 @@
 			if (req == NULL) {
 				spin_unlock_irq(&io_request_lock);
 				freereq = __get_request_wait(q, rw);
+				head = &q->queue_head;
+				spin_lock_irq(&io_request_lock);
+				should_wake = 1;
 				goto again;
 			}
 		}
@@ -1064,10 +1117,13 @@
 	req->start_time = jiffies;
 	req_new_io(req, 0, count);
 	blk_started_io(count);
+	blk_started_sectors(req, count);
 	add_request(q, req, insert_here);
 out:
 	if (freereq)
 		blkdev_release_request(freereq);
+	if (should_wake)
+		get_request_wait_wakeup(q, rw);
 	spin_unlock_irq(&io_request_lock);
 	return 0;
 end_io:
@@ -1196,8 +1252,15 @@
 	bh->b_rdev = bh->b_dev;
 	bh->b_rsector = bh->b_blocknr * count;
 
+	get_bh(bh);
 	generic_make_request(rw, bh);
 
+	/* fix race condition with wait_on_buffer() */
+	smp_mb(); /* spin_unlock may have inclusive semantics */
+	if (waitqueue_active(&bh->b_wait))
+		wake_up(&bh->b_wait);
+
+	put_bh(bh);
 	switch (rw) {
 		case WRITE:
 			kstat.pgpgout += count;
@@ -1350,6 +1413,7 @@
 	if ((bh = req->bh) != NULL) {
 		nsect = bh->b_size >> 9;
 		blk_finished_io(nsect);
+		blk_finished_sectors(req, nsect);
 		req->bh = bh->b_reqnext;
 		bh->b_reqnext = NULL;
 		bh->b_end_io(bh, uptodate);
@@ -1509,6 +1573,7 @@
 EXPORT_SYMBOL(blk_get_queue);
 EXPORT_SYMBOL(blk_cleanup_queue);
 EXPORT_SYMBOL(blk_queue_headactive);
+EXPORT_SYMBOL(blk_queue_throttle_sectors);
 EXPORT_SYMBOL(blk_queue_make_request);
 EXPORT_SYMBOL(generic_make_request);
 EXPORT_SYMBOL(blkdev_release_request);
diff -Nru a/drivers/ide/ide-probe.c b/drivers/ide/ide-probe.c
--- a/drivers/ide/ide-probe.c	Thu Jul 10 14:52:38 2003
+++ b/drivers/ide/ide-probe.c	Thu Jul 10 14:52:38 2003
@@ -971,6 +971,7 @@
 
 	q->queuedata = HWGROUP(drive);
 	blk_init_queue(q, do_ide_request);
+	blk_queue_throttle_sectors(q, 1);
 }
 
 #undef __IRQ_HELL_SPIN
diff -Nru a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
--- a/drivers/scsi/scsi.c	Thu Jul 10 14:52:38 2003
+++ b/drivers/scsi/scsi.c	Thu Jul 10 14:52:38 2003
@@ -197,6 +197,7 @@
 
 	blk_init_queue(q, scsi_request_fn);
 	blk_queue_headactive(q, 0);
+	blk_queue_throttle_sectors(q, 1);
 	q->queuedata = (void *) SDpnt;
 }
 
diff -Nru a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
--- a/drivers/scsi/scsi_lib.c	Thu Jul 10 14:52:38 2003
+++ b/drivers/scsi/scsi_lib.c	Thu Jul 10 14:52:38 2003
@@ -378,6 +378,7 @@
 		if ((bh = req->bh) != NULL) {
 			nsect = bh->b_size >> 9;
 			blk_finished_io(nsect);
+			blk_finished_sectors(req, nsect);
 			req->bh = bh->b_reqnext;
 			bh->b_reqnext = NULL;
 			sectors -= nsect;
diff -Nru a/fs/buffer.c b/fs/buffer.c
--- a/fs/buffer.c	Thu Jul 10 14:52:38 2003
+++ b/fs/buffer.c	Thu Jul 10 14:52:38 2003
@@ -153,10 +153,23 @@
 	get_bh(bh);
 	add_wait_queue(&bh->b_wait, &wait);
 	do {
-		run_task_queue(&tq_disk);
 		set_task_state(tsk, TASK_UNINTERRUPTIBLE);
 		if (!buffer_locked(bh))
 			break;
+		/*
+		 * We must read tq_disk in TQ_ACTIVE after the
+		 * add_wait_queue effect is visible to other cpus.
+		 * We could unplug some line above it wouldn't matter
+		 * but we can't do that right after add_wait_queue
+		 * without an smp_mb() in between because spin_unlock
+		 * has inclusive semantics.
+		 * Doing it here is the most efficient place so we
+		 * don't do a suprious unplug if we get a racy
+		 * wakeup that make buffer_locked to return 0, and
+		 * doing it here avoids an explicit smp_mb() we
+		 * rely on the implicit one in set_task_state.
+		 */
+		run_task_queue(&tq_disk);
 		schedule();
 	} while (buffer_locked(bh));
 	tsk->state = TASK_RUNNING;
@@ -1516,6 +1529,9 @@
 
 	/* Done - end_buffer_io_async will unlock */
 	SetPageUptodate(page);
+
+	wakeup_page_waiters(page);
+
 	return 0;
 
 out:
@@ -1547,6 +1563,7 @@
 	} while (bh != head);
 	if (need_unlock)
 		UnlockPage(page);
+	wakeup_page_waiters(page);
 	return err;
 }
 
@@ -1774,6 +1791,8 @@
 		else
 			submit_bh(READ, bh);
 	}
+
+	wakeup_page_waiters(page);
 	
 	return 0;
 }
@@ -2400,6 +2419,7 @@
 		submit_bh(rw, bh);
 		bh = next;
 	} while (bh != head);
+	wakeup_page_waiters(page);
 	return 0;
 }
 
diff -Nru a/fs/reiserfs/inode.c b/fs/reiserfs/inode.c
--- a/fs/reiserfs/inode.c	Thu Jul 10 14:52:38 2003
+++ b/fs/reiserfs/inode.c	Thu Jul 10 14:52:38 2003
@@ -2080,6 +2080,7 @@
     */
     if (nr) {
         submit_bh_for_writepage(arr, nr) ;
+	wakeup_page_waiters(page);
     } else {
         UnlockPage(page) ;
     }
diff -Nru a/include/linux/blkdev.h b/include/linux/blkdev.h
--- a/include/linux/blkdev.h	Thu Jul 10 14:52:38 2003
+++ b/include/linux/blkdev.h	Thu Jul 10 14:52:38 2003
@@ -64,12 +64,6 @@
 typedef void (plug_device_fn) (request_queue_t *q, kdev_t device);
 typedef void (unplug_device_fn) (void *q);
 
-/*
- * Default nr free requests per queue, ll_rw_blk will scale it down
- * according to available RAM at init time
- */
-#define QUEUE_NR_REQUESTS	8192
-
 struct request_list {
 	unsigned int count;
 	struct list_head free;
@@ -80,7 +74,7 @@
 	/*
 	 * the queue request freelist, one for reads and one for writes
 	 */
-	struct request_list	rq[2];
+	struct request_list	rq;
 
 	/*
 	 * The total number of requests on each queue
@@ -93,6 +87,21 @@
 	int batch_requests;
 
 	/*
+	 * The total number of 512byte blocks on each queue
+	 */
+	atomic_t nr_sectors;
+
+	/*
+	 * Batching threshold for sleep/wakeup decisions
+	 */
+	int batch_sectors;
+
+	/*
+	 * The max number of 512byte blocks on each queue
+	 */
+	int max_queue_sectors;
+
+	/*
 	 * Together with queue_head for cacheline sharing
 	 */
 	struct list_head	queue_head;
@@ -118,13 +127,21 @@
 	/*
 	 * Boolean that indicates whether this queue is plugged or not.
 	 */
-	char			plugged;
+	int			plugged:1;
 
 	/*
 	 * Boolean that indicates whether current_request is active or
 	 * not.
 	 */
-	char			head_active;
+	int			head_active:1;
+
+	/*
+	 * Boolean that indicates you will use blk_started_sectors
+	 * and blk_finished_sectors in addition to blk_started_io
+	 * and blk_finished_io.  It enables the throttling code to 
+	 * help keep the sectors in flight to a reasonable value
+	 */
+	int			can_throttle:1;
 
 	unsigned long		bounce_pfn;
 
@@ -137,7 +154,7 @@
 	/*
 	 * Tasks wait here for free read and write requests
 	 */
-	wait_queue_head_t	wait_for_requests[2];
+	wait_queue_head_t	wait_for_requests;
 };
 
 #define blk_queue_plugged(q)	(q)->plugged
@@ -221,10 +238,11 @@
 /*
  * Access functions for manipulating queue properties
  */
-extern int blk_grow_request_list(request_queue_t *q, int nr_requests);
+extern int blk_grow_request_list(request_queue_t *q, int nr_requests, int max_queue_sectors);
 extern void blk_init_queue(request_queue_t *, request_fn_proc *);
 extern void blk_cleanup_queue(request_queue_t *);
 extern void blk_queue_headactive(request_queue_t *, int);
+extern void blk_queue_throttle_sectors(request_queue_t *, int);
 extern void blk_queue_make_request(request_queue_t *, make_request_fn *);
 extern void generic_unplug_device(void *);
 extern inline int blk_seg_merge_ok(struct buffer_head *, struct buffer_head *);
@@ -243,6 +261,8 @@
 
 #define MAX_SEGMENTS 128
 #define MAX_SECTORS 255
+#define MAX_QUEUE_SECTORS (4 << (20 - 9)) /* 4 mbytes when full sized */
+#define MAX_NR_REQUESTS 1024 /* 1024k when in 512 units, normally min is 1M in 1k units */
 
 #define PageAlignSize(size) (((size) + PAGE_SIZE -1) & PAGE_MASK)
 
@@ -268,8 +288,50 @@
 	return retval;
 }
 
+static inline int blk_oversized_queue(request_queue_t * q)
+{
+	if (q->can_throttle)
+		return atomic_read(&q->nr_sectors) > q->max_queue_sectors;
+	return q->rq.count == 0;
+}
+
+static inline int blk_oversized_queue_batch(request_queue_t * q)
+{
+	return atomic_read(&q->nr_sectors) > q->max_queue_sectors - q->batch_sectors;
+}
+
 #define blk_finished_io(nsects)	do { } while (0)
 #define blk_started_io(nsects)	do { } while (0)
+
+static inline void blk_started_sectors(struct request *rq, int count)
+{
+	request_queue_t *q = rq->q;
+	if (q && q->can_throttle) {
+		atomic_add(count, &q->nr_sectors);
+		if (atomic_read(&q->nr_sectors) < 0) {
+			printk("nr_sectors is %d\n", atomic_read(&q->nr_sectors));
+			BUG();
+		}
+	}
+}
+
+static inline void blk_finished_sectors(struct request *rq, int count)
+{
+	request_queue_t *q = rq->q;
+	if (q && q->can_throttle) {
+		atomic_sub(count, &q->nr_sectors);
+		
+		smp_mb();
+		if (q->rq.count >= q->batch_requests && !blk_oversized_queue_batch(q)) {
+			if (waitqueue_active(&q->wait_for_requests))
+				wake_up(&q->wait_for_requests);
+		}
+		if (atomic_read(&q->nr_sectors) < 0) {
+			printk("nr_sectors is %d\n", atomic_read(&q->nr_sectors));
+			BUG();
+		}
+	}
+}
 
 static inline unsigned int blksize_bits(unsigned int size)
 {
diff -Nru a/include/linux/elevator.h b/include/linux/elevator.h
--- a/include/linux/elevator.h	Thu Jul 10 14:52:38 2003
+++ b/include/linux/elevator.h	Thu Jul 10 14:52:38 2003
@@ -80,7 +80,7 @@
 	return latency;
 }
 
-#define ELV_LINUS_SEEK_COST	16
+#define ELV_LINUS_SEEK_COST	1
 
 #define ELEVATOR_NOOP							\
 ((elevator_t) {								\
@@ -93,8 +93,8 @@
 
 #define ELEVATOR_LINUS							\
 ((elevator_t) {								\
-	2048,				/* read passovers */		\
-	8192,				/* write passovers */		\
+	128,				/* read passovers */		\
+	512,				/* write passovers */		\
 									\
 	elevator_linus_merge,		/* elevator_merge_fn */		\
 	elevator_linus_merge_req,	/* elevator_merge_req_fn */	\
diff -Nru a/include/linux/pagemap.h b/include/linux/pagemap.h
--- a/include/linux/pagemap.h	Thu Jul 10 14:52:38 2003
+++ b/include/linux/pagemap.h	Thu Jul 10 14:52:38 2003
@@ -97,6 +97,8 @@
 		___wait_on_page(page);
 }
 
+extern void FASTCALL(wakeup_page_waiters(struct page * page));
+
 /*
  * Returns locked page at given index in given cache, creating it if needed.
  */
diff -Nru a/kernel/ksyms.c b/kernel/ksyms.c
--- a/kernel/ksyms.c	Thu Jul 10 14:52:38 2003
+++ b/kernel/ksyms.c	Thu Jul 10 14:52:38 2003
@@ -296,6 +296,7 @@
 EXPORT_SYMBOL(filemap_fdatawait);
 EXPORT_SYMBOL(lock_page);
 EXPORT_SYMBOL(unlock_page);
+EXPORT_SYMBOL(wakeup_page_waiters);
 
 /* device registration */
 EXPORT_SYMBOL(register_chrdev);
diff -Nru a/mm/filemap.c b/mm/filemap.c
--- a/mm/filemap.c	Thu Jul 10 14:52:38 2003
+++ b/mm/filemap.c	Thu Jul 10 14:52:38 2003
@@ -810,6 +810,20 @@
 	return &wait[hash];
 }
 
+/*
+ * This must be called after every submit_bh with end_io
+ * callbacks that would result into the blkdev layer waking
+ * up the page after a queue unplug.
+ */
+void wakeup_page_waiters(struct page * page)
+{
+	wait_queue_head_t * head;
+
+	head = page_waitqueue(page);
+	if (waitqueue_active(head))
+		wake_up(head);
+}
+
 /* 
  * Wait for a page to get unlocked.
  *

Re: 2.4.22-pre3 and reiserfs boot problem

[Marcelo Tosatti]

On Fri, 11 Jul 2003, Stephan von Krawczynski wrote:

> On 11 Jul 2003 09:15:21 -0400
> Chris Mason <mason@suse.com> wrote:
>
> > > Andrea, Chris, any idea of why this is happening?
> >
> > My first guess is that blk_oversized_queue is false but there aren't any
> > requests left.  That will pretty much spin in __get_request_wait with
> > irqs off, which sounds similar to what he's hitting.
> >
> > I think we need this hunk even if it doesn't fix his problem.
>
> Strike!
> Your patch solves my problem. I applied it on 2.4.22-pre4 and it now works just
> like -pre2 did.
> Great Chris, compared to the pretty minimal input I could give ...

The fix is already in my BK tree.

Thanks a lot Chris and Stephan.

Re: 2.4.22-pre3 and reiserfs boot problem

[Chris Mason]

On Fri, 2003-07-11 at 09:38, Stephan von Krawczynski wrote:
> On 11 Jul 2003 09:15:21 -0400
> Chris Mason <mason@suse.com> wrote:
> 
> > > Andrea, Chris, any idea of why this is happening?
> > 
> > My first guess is that blk_oversized_queue is false but there aren't any
> > requests left.  That will pretty much spin in __get_request_wait with
> > irqs off, which sounds similar to what he's hitting.
> > 
> > I think we need this hunk even if it doesn't fix his problem.
> 
> Strike!
> Your patch solves my problem. I applied it on 2.4.22-pre4 and it now works just
> like -pre2 did.
> Great Chris, compared to the pretty minimal input I could give ...

Thanks for the quick test, you've been hugely helpful.

-chris

Re: 2.4.22-pre3 and reiserfs boot problem

[Chris Mason]

On Fri, 2003-07-11 at 09:27, Stephan von Krawczynski wrote:
>  
> > My first guess is that blk_oversized_queue is false but there aren't any
> > requests left.  That will pretty much spin in __get_request_wait with
> > irqs off, which sounds similar to what he's hitting.
> 
> Why is it I am only seeing it on a big device of 320 GB? Even 60GB mount
> without problems...

Honestly not sure.  I could make a better guess if you got a stack trace
with nmi_watchdog on, so we could see which part of the reiserfs mount
process was causing problems.

The bug would be easier to hit with smaller blocksizes, I'm running
tests now with ext and 1k blocksizes.

-chris

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On 11 Jul 2003 09:15:21 -0400
Chris Mason <mason@suse.com> wrote:

> > Andrea, Chris, any idea of why this is happening?
> 
> My first guess is that blk_oversized_queue is false but there aren't any
> requests left.  That will pretty much spin in __get_request_wait with
> irqs off, which sounds similar to what he's hitting.
> 
> I think we need this hunk even if it doesn't fix his problem.

Strike!
Your patch solves my problem. I applied it on 2.4.22-pre4 and it now works just
like -pre2 did.
Great Chris, compared to the pretty minimal input I could give ...

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On 11 Jul 2003 09:15:21 -0400
Chris Mason <mason@suse.com> wrote:

> On Thu, 2003-07-10 at 14:01, Marcelo Tosatti wrote:
> > On Thu, 10 Jul 2003, Stephan von Krawczynski wrote:
> > 
> > > On Thu, 10 Jul 2003 20:20:02 +0400
> > > "Peter Lojkin" <ia6432@inbox.ru> wrote:
> > >
> > > > Hello,
> > > >
> > > > here is exact patch i've used. i made it by cutting pre2-pre3 diff,
> > > > so apply it o top of 2.4.22-pre3 with -R option to patch...
> > >
> > > Hello Peter
> > > Hello Marcelo
> > >
> > > I can confirm that pre3 works when reversing the attached patch. Thanks
> > > very much, Peter.
> > 
> > Fine Stephan. Now can youplease get us the task backtraces from sysrq when
> > the hang happens?
> > 
> > Andrea, Chris, any idea of why this is happening?
> 
> My first guess is that blk_oversized_queue is false but there aren't any
> requests left.  That will pretty much spin in __get_request_wait with
> irqs off, which sounds similar to what he's hitting.

Why is it I am only seeing it on a big device of 320 GB? Even 60GB mount
without problems...

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Chris Mason]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

On Thu, 2003-07-10 at 14:01, Marcelo Tosatti wrote:
> On Thu, 10 Jul 2003, Stephan von Krawczynski wrote:
> 
> > On Thu, 10 Jul 2003 20:20:02 +0400
> > "Peter Lojkin" <ia6432@inbox.ru> wrote:
> >
> > > Hello,
> > >
> > > here is exact patch i've used. i made it by cutting pre2-pre3 diff,
> > > so apply it o top of 2.4.22-pre3 with -R option to patch...
> >
> > Hello Peter
> > Hello Marcelo
> >
> > I can confirm that pre3 works when reversing the attached patch. Thanks very
> > much, Peter.
> 
> Fine Stephan. Now can youplease get us the task backtraces from sysrq when
> the hang happens?
> 
> Andrea, Chris, any idea of why this is happening?

My first guess is that blk_oversized_queue is false but there aren't any
requests left.  That will pretty much spin in __get_request_wait with
irqs off, which sounds similar to what he's hitting.

I think we need this hunk even if it doesn't fix his problem.

Stephan, if this patch doesn't help, could you please boot with
nmi_watchdog=1?  An earlier email said sysrq wasn't working, so we'll
probably need the nmi_watchdog to get a backtrace.

-chris


[-- Attachment #2: io-stalls-11-inc.diff --]
[-- Type: text/plain, Size: 455 bytes --]

===== drivers/block/ll_rw_blk.c 1.46 vs edited =====
--- 1.46/drivers/block/ll_rw_blk.c	Fri Jul  4 13:35:08 2003
+++ edited/drivers/block/ll_rw_blk.c	Fri Jul 11 08:30:54 2003
@@ -618,7 +618,7 @@
 	do {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		spin_lock_irq(&io_request_lock);
-		if (blk_oversized_queue(q)) {
+		if (blk_oversized_queue(q) || q->rq.count == 0) {
 			__generic_unplug_device(q);
 			spin_unlock_irq(&io_request_lock);
 			schedule();

Re: 2.4.22-pre3 and reiserfs boot problem

[Anders Karlsson]

[-- Attachment #1: Type: text/plain, Size: 3549 bytes --]

On Thu, 2003-07-10 at 16:39, Carl-Daniel Hailfinger wrote:

> Was there any disk activity after it became unresponsive? If not, please
> provide a (partially) decoded SysRq-T. I'm only interested in the decoded
> stack trace of the hung process (it should have a "D" after the process name).

Right, I have collected the details that was asked for. Carl-Daniel
suggested a way of tricking the system into doing an fsck without
tampering with the filesystem itself. Please find below the data I
copied out and decoded. If there is anything else I can do, let me know.
If there is any data about my system you require, let me know.

Output from Alt SysRq P:

Pid: 160, comm:        fsck.ext3
EIP: 0010:[<c01b0f43>]  CPU: 0  EFLAGS: 00000246   Not Tainted
EAX: 00000000 EBX: 00000000 ECX: c0338c00 EDX: 00000007
ESI: c0338c00 EDI: eee68000 EBP: eee69de8 DS: 0018 ES: 0018
CR0: 80050033 CR2: 400dcff0 CR3: 2ee6b000 CR4: 000006d0

Call trace: [<c01b1584>] [<c01b1bfb>] [<c01b1cca>] [<c0141643>]
[<c014178f>] [<c01417a4>] [<c0141887>] [<c01464e1>] [<c0141b61>]
[<c0108f83>]

Output from Alt SysRq T:

fsck.ext3      D  current   3808   160  149     (NOTLB)
Call Trace: [<c014178f>] [<c01417a4>] [<c0141887>] [<c01464e1>]
[<c0141b61>] [<c0108f83>]


ksymoops 2.4.8 on i686 2.4.22-pre3-ac1.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.22-pre3-ac1/ (default)
     -m /boot/System.map-2.4.22-pre3-ac1 (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Warning (compare_maps): mismatch on symbol zeroes  , ipsec says
f23fdf00, /lib/modules/2.4.22-pre3-ac1/kernel/net/ipsec/ipsec.o says
f23fdde0.  Ignoring
/lib/modules/2.4.22-pre3-ac1/kernel/net/ipsec/ipsec.o entry
Pid: 160, comm:        fsck.ext3
EIP: 0010:[<c01b0f43>]  CPU: 0  EFLAGS: 00000246   Not Tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EAX: 00000000 EBX: 00000000 ECX: c0338c00 EDX: 00000007
ESI: c0338c00 EDI: eee68000 EBP: eee69de8 DS: 0018 ES: 0018
Warning (Oops_set_regs): garbage 'DS: 0018 ES: 0018' at end of register
line ignored
CR0: 80050033 CR2: 400dcff0 CR3: 2ee6b000 CR4: 000006d0
Call trace: [<c01b1584>] [<c01b1bfb>] [<c01b1cca>] [<c0141643>]
[<c014178f>] [<c01417a4>] [<c0141887>] [<c01464e1>] [<c0141b61>]
[<c0108f83>]
Warning (Oops_read): Code line not seen, dumping what data is available


>>EIP; c01b0f43 <__get_request_wait+ae/f6>   <=====

>>ECX; c0338c00 <ide_hwifs+c0/2af8>
>>ESI; c0338c00 <ide_hwifs+c0/2af8>
>>EDI; eee68000 <_end+2eb17bf4/304bcc54>
>>EBP; eee69de8 <_end+2eb199dc/304bcc54>

Trace; c01b1584 <__make_request+167/71a>
Trace; c01b1bfb <generic_make_request+c4/13a>
Trace; c01b1cca <submit_bh+59/a0>
Trace; c0141643 <write_locked_buffers+2a/36>
Trace; c014178f <write_some_buffers+140/142>
Trace; c01417a4 <write_unlocked_buffers+13/1d>
Trace; c0141887 <sync_buffers+1a/75>
Trace; c01464e1 <__block_fsync+2f/6c>
Trace; c0141b61 <sys_fsync+93/de>
Trace; c0108f83 <system_call+33/38>


4 warnings issued.  Results may not be reliable.



-- 
Anders Karlsson <anders@trudheim.com>
Trudheim Technology Limited

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 2.4.22-pre3 and reiserfs boot problem

[Marcelo Tosatti]

On Thu, 10 Jul 2003, Stephan von Krawczynski wrote:

> On Thu, 10 Jul 2003 20:20:02 +0400
> "Peter Lojkin" <ia6432@inbox.ru> wrote:
>
> > Hello,
> >
> > here is exact patch i've used. i made it by cutting pre2-pre3 diff,
> > so apply it o top of 2.4.22-pre3 with -R option to patch...
>
> Hello Peter
> Hello Marcelo
>
> I can confirm that pre3 works when reversing the attached patch. Thanks very
> much, Peter.

Fine Stephan. Now can youplease get us the task backtraces from sysrq when
the hang happens?

Andrea, Chris, any idea of why this is happening?

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

On Thu, 10 Jul 2003 20:20:02 +0400
"Peter Lojkin" <ia6432@inbox.ru> wrote:

> Hello,
> 
> here is exact patch i've used. i made it by cutting pre2-pre3 diff,
> so apply it o top of 2.4.22-pre3 with -R option to patch...

Hello Peter
Hello Marcelo

I can confirm that pre3 works when reversing the attached patch. Thanks very
much, Peter.

Regards,
Stephan



[-- Attachment #2: ll_rw_cset.diff.bz2 --]
[-- Type: application/x-bzip2, Size: 6826 bytes --]

Re: 2.4.22-pre3 and reiserfs boot problem

[Carl-Daniel Hailfinger]

Anders Karlsson wrote:
> Hi,
> 
> Apologies for chipping in, but I saw something similar to what was
> described in the thread. I'm running 2.4.22-pre3-ac1 with the FreeS/WAN
> 2.0.1 patches and noticed last night that when booting this kernel, if
> an ext3 filesystem had exceeded its mount count and required checking,
> the e2fsck process would hang sometime during the fsck and the system
> would become unresponsive, but SysRq would still work. Alt-SysRq-P would

Was there any disk activity after it became unresponsive? If not, please
provide a (partially) decoded SysRq-T. I'm only interested in the decoded
stack trace of the hung process (it should have a "D" after the process name).

> show e2fsck and some register details. I did not note them down, but
> booting 2.4.21-rc7-ac1 and letting that kernel check the filesystem
> would work. Booting back into 2.4.22-pre3-ac1 would then also work.
> 
> This might or might not be related to the original problem. I do use
> nmi_watchdog=1, NMI count is 1 presently, so I guess that works. The ram
> is memtested, so that is not an issue, heavy filesystem usage works
> normally, it was just e2fsck that would not work. I have not tried -pre2
> or -pre4 yet, but that is on the cards.
> 
> If there is anything I can try, let me know.


Carl-Daniel
-- 
http://www.hailfinger.org/

Re: 2.4.22-pre3 and reiserfs boot problem

[Anders Karlsson]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

Hi,

Apologies for chipping in, but I saw something similar to what was
described in the thread. I'm running 2.4.22-pre3-ac1 with the FreeS/WAN
2.0.1 patches and noticed last night that when booting this kernel, if
an ext3 filesystem had exceeded its mount count and required checking,
the e2fsck process would hang sometime during the fsck and the system
would become unresponsive, but SysRq would still work. Alt-SysRq-P would
show e2fsck and some register details. I did not note them down, but
booting 2.4.21-rc7-ac1 and letting that kernel check the filesystem
would work. Booting back into 2.4.22-pre3-ac1 would then also work.

This might or might not be related to the original problem. I do use
nmi_watchdog=1, NMI count is 1 presently, so I guess that works. The ram
is memtested, so that is not an issue, heavy filesystem usage works
normally, it was just e2fsck that would not work. I have not tried -pre2
or -pre4 yet, but that is on the cards.

If there is anything I can try, let me know.

-- 
Anders Karlsson <anders@trudheim.com>
Trudheim Technology Limited

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Thu, 10 Jul 2003 16:44:28 +0200
Carl-Daniel Hailfinger <c-d.hailfinger.kernel.2003@gmx.net> wrote:

> > I have currently nmi_watchdog=1 which works (NMI interrupts show up during
> > normal operation), but there is no backtrace visible or producable during
> > the freeze, sorry.
> 
> How is that possible? If the NMI watchdog works but doesn't fire, the
> lockup should respond to SysRq-T. Could you please try SysRq-T *before*
> the hang just to verify that it would work?

Well, the thing I don't really understand about SysRq-X is that the output is
visible by dmesg-command, but I cannot see anything on the console (single user
tested). This means it may well work in the hang-case, but as I cannot execute
dmesg I will never see it...

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Carl-Daniel Hailfinger]

Stephan von Krawczynski wrote:
> On Thu, 10 Jul 2003 16:20:14 +0200
> Carl-Daniel Hailfinger <c-d.hailfinger.kernel.2003@gmx.net> wrote:
> 
> 
>>Since your hangs are not even traceable with SysRq, please try to boot
>>with nmi_watchdog=1 and if that doesn't work (dmesg will complain)
>>nmi_watchdog=2. About 15 seconds after the hang your box should print a
>>backtrace.
> 
> 
> I have currently nmi_watchdog=1 which works (NMI interrupts show up during
> normal operation), but there is no backtrace visible or producable during the
> freeze, sorry.

How is that possible? If the NMI watchdog works but doesn't fire, the
lockup should respond to SysRq-T. Could you please try SysRq-T *before*
the hang just to verify that it would work?


Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Thu, 10 Jul 2003 16:20:14 +0200
Carl-Daniel Hailfinger <c-d.hailfinger.kernel.2003@gmx.net> wrote:

> Since your hangs are not even traceable with SysRq, please try to boot
> with nmi_watchdog=1 and if that doesn't work (dmesg will complain)
> nmi_watchdog=2. About 15 seconds after the hang your box should print a
> backtrace.

I have currently nmi_watchdog=1 which works (NMI interrupts show up during
normal operation), but there is no backtrace visible or producable during the
freeze, sorry.

-- 
Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Carl-Daniel Hailfinger]

Having read the whole thread, I came up with a few ideas.

The following patches in -pre3 could perhaps have to do something with
your problems:
  o (resend) collected semaphore fixes and semtimedop
  o Fix potential IO hangs and increase interactiveness during heavy IO
  o fix false sharing of mm info
  o fix up semops and return, allow timedop
  o mremap VM_LOCKED move_vma
  o remove io_apic_modify - this doesnt work on some APICs
  o small setup-pci cleanups

Since your hangs are not even traceable with SysRq, please try to boot
with nmi_watchdog=1 and if that doesn't work (dmesg will complain)
nmi_watchdog=2. About 15 seconds after the hang your box should print a
backtrace.

As a last resort you could mount the fs from UML.

I suggest you try the nmi_watchdog thing first.


Carl-Daniel
-- 
http://www.hailfinger.org/

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Thu, 10 Jul 2003 08:54:02 -0300 (BRT)
Marcelo Tosatti <marcelo@conectiva.com.br> wrote:

> > If you want me to try others, just send them to me like these three.
> 
> Stephan,
> 
> First of all, thanks a lot for your help. Not everyone is willing to
> debug/test problems like you do. This is very important for us.
> 
> Well, we now know reiserfs patches in 2.4.22-pre are not the problem.
> 
> 2.4.21 is OK (does not crash when mounting) correct?

Everything up to and including 2.4.22-pre2 is working without problems.
Under 2.4.22-pre3 I can reiserfsck the partition and see i/o going on for
minutes without troubles.
But I cannot mount it, the box completely hangs about 1 second after mount try.
I see the drive leds blinking shortly and that's it.

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Wed, 9 Jul 2003 14:18:37 -0300 (BRT)
Marcelo Tosatti <marcelo@conectiva.com.br> wrote:

> Can you please revert them (with patch -R) and try to reproduce the
> problem?
> 
> Thanks

In addition I can tell you that the problem is also visible with -pre4

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Marcelo Tosatti]

On Thu, 10 Jul 2003, Stephan von Krawczynski wrote:

> On Wed, 9 Jul 2003 14:18:37 -0300 (BRT)
> Marcelo Tosatti <marcelo@conectiva.com.br> wrote:
>
> > > PS to Marcelo:
> > > There is a problem with 2.4.22-pre3. I cannot mount a reiserfs
> > > data-partition with 320 GB size located on a 3ware RAID. It just hangs the
> > > box, during init or any runlevel I tried. It is completely reproducable,
> > > but debugreiserfs on the partition and reiserfsck both show no problems at
> > > all ... The things mounts flawlessly under 2.4.22-pre2 and below.
> >
> > There are no 3ware changes in pre3. So it must be reiserfs or something
> > else. Lets try reverting the reiserfs patches to see if they are the
> > cause?
> >
> > Attached are files rei1, rei2, and rei3 (all gzip compressed).
>
> I reverted all three patches and the problem stays just the same. I guess this
> makes it a lot likely that the problem lies elsewhere.
>
> If you want me to try others, just send them to me like these three.

Stephan,

First of all, thanks a lot for your help. Not everyone is willing to
debug/test problems like you do. This is very important for us.

Well, we now know reiserfs patches in 2.4.22-pre are not the problem.

2.4.21 is OK (does not crash when mounting) correct?

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Wed, 9 Jul 2003 14:18:37 -0300 (BRT)
Marcelo Tosatti <marcelo@conectiva.com.br> wrote:

> > PS to Marcelo:
> > There is a problem with 2.4.22-pre3. I cannot mount a reiserfs
> > data-partition with 320 GB size located on a 3ware RAID. It just hangs the
> > box, during init or any runlevel I tried. It is completely reproducable,
> > but debugreiserfs on the partition and reiserfsck both show no problems at
> > all ... The things mounts flawlessly under 2.4.22-pre2 and below.
> 
> There are no 3ware changes in pre3. So it must be reiserfs or something
> else. Lets try reverting the reiserfs patches to see if they are the
> cause?
> 
> Attached are files rei1, rei2, and rei3 (all gzip compressed).

I reverted all three patches and the problem stays just the same. I guess this
makes it a lot likely that the problem lies elsewhere.

If you want me to try others, just send them to me like these three.

Thanks for your support
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Marcelo Tosatti]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1121 bytes --]



On Wed, 9 Jul 2003, Stephan von Krawczynski wrote:

> On Wed, 9 Jul 2003 18:11:11 +0400
> ok, I did the reiserfsck and it works flawlessly. No errors no problems no
> hang.
> I tried mount afterwards and it still hangs.
> Is there some recent change around the mount procedure itself? maybe it is
> really unrelated to reiserfs and 3ware...
>
> Regards,
> Stephan
>
> PS to Marcelo:
> There is a problem with 2.4.22-pre3. I cannot mount a reiserfs data-partition
> with 320 GB size located on a 3ware RAID. It just hangs the box, during init or
> any runlevel I tried. It is completely reproducable, but debugreiserfs on the
> partition and reiserfsck both show no problems at all ...
> The things mounts flawlessly under 2.4.22-pre2 and below.

There are no 3ware changes in pre3. So it must be reiserfs or something
else. Lets try reverting the reiserfs patches to see if they are the
cause?

Attached are files rei1, rei2, and rei3 (all gzip compressed).

They are the three reiserfs changesets which have been included in -pre3.

Can you please revert them (with patch -R) and try to reproduce the
problem?

Thanks

[-- Attachment #2: Type: APPLICATION/x-gzip, Size: 21109 bytes --]

[-- Attachment #3: Type: APPLICATION/x-gzip, Size: 4360 bytes --]

[-- Attachment #4: Type: APPLICATION/x-gzip, Size: 17422 bytes --]

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Wed, 9 Jul 2003 18:40:15 +0300
Ville Herva <vherva@niksula.hut.fi> wrote:

>> > ok, I did the reiserfsck and it works flawlessly. No errors no problems no
> > hang.
> > I tried mount afterwards and it still hangs.
> > Is there some recent change around the mount procedure itself? maybe it is
> > really unrelated to reiserfs and 3ware...
> 
> Is it just this partition or any reiserfs fs on 3ware?

Hm, unfortunately I can't tell, I have no other partition available on 3ware
...

> > Oleg Drokin <green@namesys.com> wrote:
> > > Then next step would be probably to try and mount the partition from
> > > usermodelinux if you are able to conduct such a test.
> 
> It it possible to mount raw partitions with UML?

Hm, I never tried UML. I really wonder if there is nobody else with 3ware and
reiserfs available for re-checking 2.4.22-pre3. Only to see if this is a
singular problem or reproducable elsewhere.

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Wed, 9 Jul 2003 18:11:11 +0400
Oleg Drokin <green@namesys.com> wrote:

> Hello!
> 
> On Wed, Jul 09, 2003 at 03:58:03PM +0200, Stephan von Krawczynski wrote:
> > > > Step one is to figure out if the problem is reiserfs or 3ware.  Instead
> > > > of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null
> > > > and see if you still hang.
> > > > This will read the FS metadata and should generate enough io to trigger
> > > > the hang if it is a 3ware problem.
> > Ok, I tried this. debugreiserfs runs without any problems. Disks show quite
> > an activity, the whole thing lasts 1-2 minutes.
> > mount afterwards shows the same hang.
> 
> Hm.
> 
> > > Or if this one suceeds, then may be reiserfsck --check /dev/xxxx to get
> > > journal replayed. This is in case access pattern matters.
> > I can try that, too. What do you expect to see?
> 
> Well, it will either hang or not, I think.
> It it won't hang, this will complicate matters.
> Then next step would be probably to try and mount the partition from
> usermodelinux if you are able to conduct such a test.
> I am still pretty skeptical about the possibility that recent reiserfs
> changes broke stuff.
> 
> Bye,
>     Oleg

ok, I did the reiserfsck and it works flawlessly. No errors no problems no
hang.
I tried mount afterwards and it still hangs.
Is there some recent change around the mount procedure itself? maybe it is
really unrelated to reiserfs and 3ware...

Regards,
Stephan

PS to Marcelo:
There is a problem with 2.4.22-pre3. I cannot mount a reiserfs data-partition
with 320 GB size located on a 3ware RAID. It just hangs the box, during init or
any runlevel I tried. It is completely reproducable, but debugreiserfs on the
partition and reiserfsck both show no problems at all ...
The things mounts flawlessly under 2.4.22-pre2 and below.

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Wed, 9 Jul 2003 16:01:40 +0200
Vincent Touquet <vincent.touquet@pandora.be> wrote:

> Which mainboard do you use ?
> I'm having endless pain with a 3ware raid and Tyan mainboards, so much I
> really really want to replace it with anything else that is stable
> (probably a single Intel P4).
> 
> v

Sorry, the problem only arises with 2.4.22-pre3, and nothing before. So it is
definitely software-related.

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Oleg Drokin]

Hello!

On Wed, Jul 09, 2003 at 03:58:03PM +0200, Stephan von Krawczynski wrote:
> > > Step one is to figure out if the problem is reiserfs or 3ware.  Instead
> > > of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null
> > > and see if you still hang.
> > > This will read the FS metadata and should generate enough io to trigger
> > > the hang if it is a 3ware problem.
> Ok, I tried this. debugreiserfs runs without any problems. Disks show quite an
> activity, the whole thing lasts 1-2 minutes.
> mount afterwards shows the same hang.

Hm.

> > Or if this one suceeds, then may be reiserfsck --check /dev/xxxx to get
> > journal replayed. This is in case access pattern matters.
> I can try that, too. What do you expect to see?

Well, it will either hang or not, I think.
It it won't hang, this will complicate matters.
Then next step would be probably to try and mount the partition from usermodelinux if you are able
to conduct such a test.
I am still pretty skeptical about the possibility that recent reiserfs changes broke stuff.

Bye,
    Oleg

Re: 2.4.22-pre3 and reiserfs boot problem

[Vincent Touquet]

On Wed, Jul 09, 2003 at 02:01:38PM +0200, Stephan von Krawczynski wrote:
>I tried to produce some useful output but failed. Additionals I found:
>
>- pre3-ac1 has the same problem
>- the box _hangs_ in fact, no sysrq is working.
>  (you need hw-reset to revive the box)
>- I can see no disk activity on the 3ware RAID in question
>- It always shows up, completely reproducable
>- It shows during boot and during single- or multiuser (mount from console)
>
>Regards,
>Stephan

Which mainboard do you use ?
I'm having endless pain with a 3ware raid and Tyan mainboards, so much I
really really want to replace it with anything else that is stable
(probably a single Intel P4).

v

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On Wed, 9 Jul 2003 17:48:37 +0400
Oleg Drokin <green@namesys.com> wrote:

> Hello!
> 
> On Wed, Jul 09, 2003 at 09:36:04AM -0400, Chris Mason wrote:
> > > > Is reiserfs on your root drive?  If not can you please boot into single
> > > > user mode, enable sysrq, try the mount again and get the decoded output
> > > > from sysrq-p and sysrq-t if it hangs.
> > > I tried to produce some useful output but failed. Additionals I found:
> > > - pre3-ac1 has the same problem
> > > - the box _hangs_ in fact, no sysrq is working.
> > >   (you need hw-reset to revive the box)
> 
> This complete hang is _very_ suspicious. Usually you cannot achieve such
> results without touching hardware.

Well, I did a few more tries and it looks like this:
- enter mount command
- short blinking on the RAID disks
- about 1/2 second later box hangs
 
> > Step one is to figure out if the problem is reiserfs or 3ware.  Instead
> > of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null
> > and see if you still hang.
> > This will read the FS metadata and should generate enough io to trigger
> > the hang if it is a 3ware problem.

Ok, I tried this. debugreiserfs runs without any problems. Disks show quite an
activity, the whole thing lasts 1-2 minutes.

mount afterwards shows the same hang.

> Or if this one suceeds, then may be reiserfsck --check /dev/xxxx to get
> journal replayed. This is in case access pattern matters.

I can try that, too. What do you expect to see?

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Oleg Drokin]

Hello!

On Wed, Jul 09, 2003 at 09:36:04AM -0400, Chris Mason wrote:
> > > Is reiserfs on your root drive?  If not can you please boot into single
> > > user mode, enable sysrq, try the mount again and get the decoded output
> > > from sysrq-p and sysrq-t if it hangs.
> > I tried to produce some useful output but failed. Additionals I found:
> > - pre3-ac1 has the same problem
> > - the box _hangs_ in fact, no sysrq is working.
> >   (you need hw-reset to revive the box)

This complete hang is _very_ suspicious. Usually you cannot achieve such
results without touching hardware.

> > - I can see no disk activity on the 3ware RAID in question

After the lockup? This is kind of expected ;)

> > - It always shows up, completely reproducable
> > - It shows during boot and during single- or multiuser (mount from console)
> Step one is to figure out if the problem is reiserfs or 3ware.  Instead
> of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null
> and see if you still hang.
> This will read the FS metadata and should generate enough io to trigger
> the hang if it is a 3ware problem.

Or if this one suceeds, then may be reiserfsck --check /dev/xxxx to get
journal replayed. This is in case access pattern matters.

Bye,
    Oleg

Re: 2.4.22-pre3 and reiserfs boot problem

[Chris Mason]

On Wed, 2003-07-09 at 08:01, Stephan von Krawczynski wrote:
> On 06 Jul 2003 14:13:44 -0400
> Chris Mason <mason@suse.com> wrote:
> 
> > [...]
> > Is reiserfs on your root drive?  If not can you please boot into single
> > user mode, enable sysrq, try the mount again and get the decoded output
> > from sysrq-p and sysrq-t if it hangs.
> > 
> > -chris
> 
> Hello Chris,
> 
> I tried to produce some useful output but failed. Additionals I found:
> 
> - pre3-ac1 has the same problem
> - the box _hangs_ in fact, no sysrq is working.
>   (you need hw-reset to revive the box)
> - I can see no disk activity on the 3ware RAID in question
> - It always shows up, completely reproducable
> - It shows during boot and during single- or multiuser (mount from console)

Step one is to figure out if the problem is reiserfs or 3ware.  Instead
of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null
and see if you still hang.

This will read the FS metadata and should generate enough io to trigger
the hang if it is a 3ware problem.

(I'm on vacation for a few days, so Oleg is cc'd)

-chris

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On 06 Jul 2003 14:13:44 -0400
Chris Mason <mason@suse.com> wrote:

> [...]
> Is reiserfs on your root drive?  If not can you please boot into single
> user mode, enable sysrq, try the mount again and get the decoded output
> from sysrq-p and sysrq-t if it hangs.
> 
> -chris

Hello Chris,

I tried to produce some useful output but failed. Additionals I found:

- pre3-ac1 has the same problem
- the box _hangs_ in fact, no sysrq is working.
  (you need hw-reset to revive the box)
- I can see no disk activity on the 3ware RAID in question
- It always shows up, completely reproducable
- It shows during boot and during single- or multiuser (mount from console)

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

Hello Chris,

I have to correct myself regarding the problem. It does _not_ arise on
aic-driven scsi-disk, but on 3ware-driven RAID5 with 320 GB data. I tried to
mount it by hand, but that does not work either. It does not look like there is
a lot of work going on on the hds while mounting (no LEDs). 
Can anyone reproduce with equal setup?
I try to come up with further information...
-- 
Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

On 06 Jul 2003 14:13:44 -0400
Chris Mason <mason@suse.com> wrote:

> On Sun, 2003-07-06 at 12:34, Stephan von Krawczynski wrote:
> > Hello,
> > 
> > I just tried 2.4.22-pre3 and found out I cannot boot my test box any more.
> > It halts at:
> > 
> > reiserfs: found format "3.6" with standard journal
> > 
> > on a partition located on aic7xxx based hd. 
> > 
> > Booting the box with pre2 works perfectly well.
> > Anything I should try? What information is needed?
> 
> Same config here (reiserfs 3.6.x, aic7xxx drive), works without
> problems.  Is reiserfs or aic7xxx compiled as a module in your setup? 
> If so did you remember to install the new modules and mkinitrd (if
> required).

Hello Chris,

no, there is no modules involved here. The problem arises not on my root
partition (which happens to be reiserfs, too), that seems to be ok. Problems
seem to come up on sda4, a simple data partition.
 
> Is reiserfs on your root drive?  If not can you please boot into single
> user mode, enable sysrq, try the mount again and get the decoded output
> from sysrq-p and sysrq-t if it hangs.
> 
> -chris

I am going to try with single user tomorrow morning.

Regards,
Stephan

Re: 2.4.22-pre3 and reiserfs boot problem

[Chris Mason]

On Sun, 2003-07-06 at 12:34, Stephan von Krawczynski wrote:
> Hello,
> 
> I just tried 2.4.22-pre3 and found out I cannot boot my test box any more. It
> halts at:
> 
> reiserfs: found format "3.6" with standard journal
> 
> on a partition located on aic7xxx based hd. 
> 
> Booting the box with pre2 works perfectly well.
> Anything I should try? What information is needed?

Same config here (reiserfs 3.6.x, aic7xxx drive), works without
problems.  Is reiserfs or aic7xxx compiled as a module in your setup? 
If so did you remember to install the new modules and mkinitrd (if
required).

Is reiserfs on your root drive?  If not can you please boot into single
user mode, enable sysrq, try the mount again and get the decoded output
from sysrq-p and sysrq-t if it hangs.

-chris

2.4.22-pre3 and reiserfs boot problem

[Stephan von Krawczynski]

Hello,

I just tried 2.4.22-pre3 and found out I cannot boot my test box any more. It
halts at:

reiserfs: found format "3.6" with standard journal

on a partition located on aic7xxx based hd. 

Booting the box with pre2 works perfectly well.
Anything I should try? What information is needed?

Regards,
Stephan