I/O problems in 2.4.19/2.4.20/2.4.21-rc3

I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[Christian Klose]

Hello all :-)

I have a problem since Linux Kernel 2.4.19. Copying huge amount of data gives 
me pauses where pauses are disk io pauses, keyboard does not accept input and 
mouse won't move. This depends, sometimes those pauses are 1 to 2 seconds, 
sometimes even more up to 15 seconds where I can not do anything with my 
linux but waiting :-(

For the past weeks I've searched google alot and found almost the same reports 
there but with imho no real fixes. I've tried many different kernel patches 
I've found while searching google (2.4-aa, 2.4-ck, 2.4-wolk, 2.4-rmap) but 
none of them fixes the problem I've experienced. The kernel patches 2.4-ck 
and 2.4-wolk are very good, those pauses are almost gone, but also the 
throughput is horribly decreased. Yesterday, mcp from #kernelnewbies told me, 
that this is the decrease of nr_requests to 32 (or maybe 4? I don't remember 
exactly). I've also tried 2.4-aa patch because I've read about his lowlatency 
elevator which should fix these pauses. Unfortunately the pauses are still 
there and also a decrease in throughput :-(

I've switched my desktop machine back from 2.4.20 to 2.4.18 because these 
pauses are really annoying me. I wonder what changes were made to 2.4.19 
causing these pauses. Please don't get me wrong but it seems so that the Linux 
Kernel is not ready for desktop yet, and I even wonder about guaranteed io 
throughput for serverusage (please read down below)

This is also not a problem of my hardware. I've tried the same scenario on 
almost 20 different machines in my company, starting with a small 500mhz cpu 
and udma 100 intel controller up to a Pentium 4 with 2,4 GHz with an Adaptec 
U160 scsi controller and u160 scsi disks with software raid-0 and raid-1.

Carl-Daniel Hailfinger has been very helpfull yesterday on #kernelnewbies 
trying to track this behaviour down. He advised me to use SysRq-T and to 
press this key combination while there are io pauses. Maybe he will find out 
what's going on there :-)



Beside that, I've also noticed that there is no guaranteed io throughput while 
copying data in 2.4.18 up to 2.4.21-rc3. My machine has 512MB of memory and 
512MB swap. Right after bootup of linux, there is guaranteed io throughput 
until the memory is almost completely used (with buffers or cache? ... 
/proc/meminfo tells me so) and linux starts to swap. After this, copying data 
starts up with 30mb per second and goes down real fast to 1mb per second and 
even more worse down to ~250kb per second, goes up to 10mb per second and so 
on, so this varies alot.


Anyway, I've also read about kernel 2.5 and that this kernel should fix all of 
the above I've mentioned. So by reading all these great oppinions about 
kernel 2.5 I've tried it out last week and I just have to say that I cannot 
see any advantages, at least not for these 2 cases I've mentioned :-(((

Is it just me or are there many others noticing this too?


Please excuse my bad english but I hope everyone understands me :-)


PS: Should I CC this to Marcello Tosatti and Linus Torvalds too? I haven't 
done this yet but maybe it may help because both are the maintainers of 2.4/ 
2.5 (at least that's what I've found in google). Sorry, I am using Linux 
since ~ 1 1/2 years now and my knowledge about the Linux Kernel is not that 
big.


Thank you so much and have a nice weekend :-)

bye, Chris

Re: I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[Christian Klose]

On Friday 23 May 2003 15:00, Christian Klose wrote:

Hello all again :-)

> I have a problem since Linux Kernel 2.4.19. Copying huge amount of data
> gives me pauses where pauses are disk io pauses, keyboard does not accept
> input and mouse won't move. This depends, sometimes those pauses are 1 to 2
> seconds, sometimes even more up to 15 seconds where I can not do anything
> with my linux but waiting :-(
I forgot to mention that this is filesystem independant. ext2, ext3, reiserfs; 
always same problem.

bye, Chris

Re: I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[Marc-Christian Petersen]

[-- Attachment #1: Type: text/plain, Size: 620 bytes --]

On Friday 23 May 2003 15:46, Christian Klose wrote:

Moin Christian,

> > I have a problem since Linux Kernel 2.4.19. Copying huge amount of data
> > gives me pauses where pauses are disk io pauses, keyboard does not accept
> > input and mouse won't move. This depends, sometimes those pauses are 1 to
> > 2 seconds, sometimes even more up to 15 seconds where I can not do
> > anything with my linux but waiting :-(
> I forgot to mention that this is filesystem independant. ext2, ext3,
> reiserfs; always same problem.
it _seems_ the offending diff is the attached one. Went in into .19-pre5.

More to come.

ciao, Marc

[-- Attachment #2: pre4-to-pre5-ll_rw_blk.c --]
[-- Type: text/x-csrc, Size: 4865 bytes --]

diff -urN linux-2.4.19-pre4/drivers/block/ll_rw_blk.c linux-2.4.19-pre5/drivers/block/ll_rw_blk.c
--- linux-2.4.19-pre4/drivers/block/ll_rw_blk.c	Fri Mar 29 14:49:08 2002
+++ linux-2.4.19-pre5/drivers/block/ll_rw_blk.c	Fri Mar 29 14:49:29 2002
@@ -117,16 +117,6 @@
  */
 int * max_sectors[MAX_BLKDEV];
 
-/*
- * The total number of requests in each queue.
- */
-static int queue_nr_requests;
-
-/*
- * The threshold around which we make wakeup decisions
- */
-static int batch_requests;
-
 static inline int get_max_sectors(kdev_t dev)
 {
 	if (!max_sectors[MAJOR(dev)])
@@ -180,7 +170,7 @@
  **/
 void blk_cleanup_queue(request_queue_t * q)
 {
-	int count = queue_nr_requests;
+	int count = q->nr_requests;
 
 	count -= __blk_cleanup_queue(&q->rq[READ]);
 	count -= __blk_cleanup_queue(&q->rq[WRITE]);
@@ -330,31 +320,64 @@
 	spin_unlock_irqrestore(&io_request_lock, flags);
 }
 
+/** blk_grow_request_list
+ *  @q: The &request_queue_t
+ *  @nr_requests: how many requests are desired
+ *
+ * More free requests are added to the queue's free lists, bringing
+ * the total number of requests to @nr_requests.
+ *
+ * The requests are added equally to the request queue's read
+ * and write freelists.
+ *
+ * This function can sleep.
+ *
+ * Returns the (new) number of requests which the queue has available.
+ */
+int blk_grow_request_list(request_queue_t *q, int nr_requests)
+{
+	spin_lock_irq(&io_request_lock);
+	while (q->nr_requests < nr_requests) {
+		struct request *rq;
+		int rw;
+
+		spin_unlock_irq(&io_request_lock);
+		rq = kmem_cache_alloc(request_cachep, SLAB_KERNEL);
+		spin_lock_irq(&io_request_lock);
+		if (rq == NULL)
+			break;
+		memset(rq, 0, sizeof(*rq));
+		rq->rq_status = RQ_INACTIVE;
+		rw = q->nr_requests & 1;
+		list_add(&rq->queue, &q->rq[rw].free);
+		q->rq[rw].count++;
+		q->nr_requests++;
+	}
+	q->batch_requests = q->nr_requests / 4;
+	if (q->batch_requests > 32)
+		q->batch_requests = 32;
+	spin_unlock_irq(&io_request_lock);
+	return q->nr_requests;
+}
+
 static void blk_init_free_list(request_queue_t *q)
 {
-	struct request *rq;
-	int i;
+	struct sysinfo si;
+	int megs;		/* Total memory, in megabytes */
+	int nr_requests;
 
 	INIT_LIST_HEAD(&q->rq[READ].free);
 	INIT_LIST_HEAD(&q->rq[WRITE].free);
 	q->rq[READ].count = 0;
 	q->rq[WRITE].count = 0;
+	q->nr_requests = 0;
 
-	/*
-	 * Divide requests in half between read and write
-	 */
-	for (i = 0; i < queue_nr_requests; i++) {
-		rq = kmem_cache_alloc(request_cachep, SLAB_KERNEL);
-		if (rq == NULL) {
-			/* We'll get a `leaked requests' message from blk_cleanup_queue */
-			printk(KERN_EMERG "blk_init_free_list: error allocating requests\n");
-			break;
-		}
-		memset(rq, 0, sizeof(struct request));
-		rq->rq_status = RQ_INACTIVE;
-		list_add(&rq->queue, &q->rq[i&1].free);
-		q->rq[i&1].count++;
-	}
+	si_meminfo(&si);
+	megs = si.totalram >> (20 - PAGE_SHIFT);
+	nr_requests = 128;
+	if (megs < 32)
+		nr_requests /= 2;
+	blk_grow_request_list(q, nr_requests);
 
 	init_waitqueue_head(&q->wait_for_requests[0]);
 	init_waitqueue_head(&q->wait_for_requests[1]);
@@ -610,7 +633,7 @@
 	 */
 	if (q) {
 		list_add(&req->queue, &q->rq[rw].free);
-		if (++q->rq[rw].count >= batch_requests &&
+		if (++q->rq[rw].count >= q->batch_requests &&
 				waitqueue_active(&q->wait_for_requests[rw]))
 			wake_up(&q->wait_for_requests[rw]);
 	}
@@ -802,7 +825,7 @@
 		 * See description above __get_request_wait()
 		 */
 		if (rw_ahead) {
-			if (q->rq[rw].count < batch_requests) {
+			if (q->rq[rw].count < q->batch_requests) {
 				spin_unlock_irq(&io_request_lock);
 				goto end_io;
 			}
@@ -958,6 +981,7 @@
 		BUG();
 
 	set_bit(BH_Req, &bh->b_state);
+	set_bit(BH_Launder, &bh->b_state);
 
 	/*
 	 * First step, 'identity mapping' - RAID or LVM might
@@ -1149,12 +1173,9 @@
 	blkdev_release_request(req);
 }
 
-#define MB(kb)	((kb) << 10)
-
 int __init blk_dev_init(void)
 {
 	struct blk_dev_struct *dev;
-	int total_ram;
 
 	request_cachep = kmem_cache_create("blkdev_requests",
 					   sizeof(struct request),
@@ -1170,22 +1191,6 @@
 	memset(max_readahead, 0, sizeof(max_readahead));
 	memset(max_sectors, 0, sizeof(max_sectors));
 
-	total_ram = nr_free_pages() << (PAGE_SHIFT - 10);
-
-	/*
-	 * Free request slots per queue.
-	 * (Half for reads, half for writes)
-	 */
-	queue_nr_requests = 64;
-	if (total_ram > MB(32))
-		queue_nr_requests = 128;
-
-	/*
-	 * Batch frees according to queue length
-	 */
-	batch_requests = queue_nr_requests/4;
-	printk("block: %d slots per queue, batch=%d\n", queue_nr_requests, batch_requests);
-
 #ifdef CONFIG_AMIGA_Z2RAM
 	z2_init();
 #endif
@@ -1296,6 +1301,7 @@
 EXPORT_SYMBOL(io_request_lock);
 EXPORT_SYMBOL(end_that_request_first);
 EXPORT_SYMBOL(end_that_request_last);
+EXPORT_SYMBOL(blk_grow_request_list);
 EXPORT_SYMBOL(blk_init_queue);
 EXPORT_SYMBOL(blk_get_queue);
 EXPORT_SYMBOL(blk_cleanup_queue);

Re: I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[Marc-Christian Petersen]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

On Friday 23 May 2003 15:46, Christian Klose wrote:

Hi Christian,

> > I have a problem since Linux Kernel 2.4.19. Copying huge amount of data
> > gives me pauses where pauses are disk io pauses, keyboard does not accept
> > input and mouse won't move. This depends, sometimes those pauses are 1 to
> > 2 seconds, sometimes even more up to 15 seconds where I can not do
> > anything with my linux but waiting :-(
> I forgot to mention that this is filesystem independant. ext2, ext3,
> reiserfs; always same problem.
You've mentioned that you also tried 2.5 and you don't see any difference 
there. I am very interrested to hear what version of 2.5 you tried.

I am using 2.5.69 + some patches and it does not suck or behave brain dead 
like 2.4 does. For me I've quit with 2.4 from now on.

2.5 still has many things to do / to fix but 2.5 is _active_ developed and 
fixes _are_ going into 2.5 where 2.4 does not accept real bug fixes.

If you want to give 2.5.69 a try, please apply the attached patch if you want 
to use it for your desktop. It gives you more interactivity.

ciao, Marc

[-- Attachment #2: sched-interactivity.patch --]
[-- Type: text/x-diff, Size: 368 bytes --]

--- old/kernel/sched.c	2003-05-24 14:45:57.000000000 +0200
+++ 2.5-mcp/kernel/sched.c	2003-05-24 16:18:42.000000000 +0200
@@ -65,7 +65,7 @@
  * they expire.
  */
 #define MIN_TIMESLICE		( 10 * HZ / 1000)
-#define MAX_TIMESLICE		(200 * HZ / 1000)
+#define MAX_TIMESLICE		( 10 * HZ / 1000)
 #define CHILD_PENALTY		50
 #define PARENT_PENALTY		100
 #define EXIT_WEIGHT		3

Re: I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[William Lee Irwin III]

On Sat, May 24, 2003 at 04:19:40PM +0200, Marc-Christian Petersen wrote:
> --- old/kernel/sched.c	2003-05-24 14:45:57.000000000 +0200
> +++ 2.5-mcp/kernel/sched.c	2003-05-24 16:18:42.000000000 +0200
> @@ -65,7 +65,7 @@
>   * they expire.
>   */
>  #define MIN_TIMESLICE		( 10 * HZ / 1000)
> -#define MAX_TIMESLICE		(200 * HZ / 1000)
> +#define MAX_TIMESLICE		( 10 * HZ / 1000)
>  #define CHILD_PENALTY		50
>  #define PARENT_PENALTY		100
>  #define EXIT_WEIGHT		3


This looks highly suspicious as it essentially removes dynamic timeslice
sizing. If this fixes something, then dynamic timeslice heuristics are
going wrong somewhere that should be properly described and handled, not
this kind of shenanigan.


-- wli

Re: I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[Christian Klose]

On Saturday 24 May 2003 16:28, William Lee Irwin III wrote:

Hi wli,

> > --- old/kernel/sched.c	2003-05-24 14:45:57.000000000 +0200
> > +++ 2.5-mcp/kernel/sched.c	2003-05-24 16:18:42.000000000 +0200
> > @@ -65,7 +65,7 @@
> >   * they expire.
> >   */
> >  #define MIN_TIMESLICE		( 10 * HZ / 1000)
> > -#define MAX_TIMESLICE		(200 * HZ / 1000)
> > +#define MAX_TIMESLICE		( 10 * HZ / 1000)
> >  #define CHILD_PENALTY		50
> >  #define PARENT_PENALTY		100
> >  #define EXIT_WEIGHT		3

> This looks highly suspicious as it essentially removes dynamic timeslice
> sizing. If this fixes something, then dynamic timeslice heuristics are
> going wrong somewhere that should be properly described and handled, not
> this kind of shenanigan.
I somewhat agree with you but this "properly described" are all the bug 
reports on lkml containing "bad interactivity in 2.5, cpu starving in 2.5" 
and such...

This isn't a shenanigan, at least not for the interactivity for a desktop. 
This is a workaround for users who are complaining about bad interactivity in 
2.5!

ciao, Marc

Re: I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[Marc-Christian Petersen]

Hi Christian,

> I somewhat agree with you but this "properly described" are all the bug
> reports on lkml containing "bad interactivity in 2.5, cpu starving in 2.5"
> and such...
>
> This isn't a shenanigan, at least not for the interactivity for a desktop.
> This is a workaround for users who are complaining about bad interactivity
> in 2.5!
>
> ciao, Marc
err, could you please tell me why you are writing "ciao, Marc" like I do in my 
emails if you are pretending beeing Christian Klose??

Re: I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[Con Kolivas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, 25 May 2003 10:43, Christian Klose wrote:
> On Saturday 24 May 2003 16:28, William Lee Irwin III wrote:
>
> Hi wli,
>
> > > --- old/kernel/sched.c	2003-05-24 14:45:57.000000000 +0200
> > > +++ 2.5-mcp/kernel/sched.c	2003-05-24 16:18:42.000000000 +0200
> > > @@ -65,7 +65,7 @@
> > >   * they expire.
> > >   */
> > >  #define MIN_TIMESLICE		( 10 * HZ / 1000)
> > > -#define MAX_TIMESLICE		(200 * HZ / 1000)
> > > +#define MAX_TIMESLICE		( 10 * HZ / 1000)
> > >  #define CHILD_PENALTY		50
> > >  #define PARENT_PENALTY		100
> > >  #define EXIT_WEIGHT		3
> >
> > This looks highly suspicious as it essentially removes dynamic timeslice
> > sizing. If this fixes something, then dynamic timeslice heuristics are
> > going wrong somewhere that should be properly described and handled, not
> > this kind of shenanigan.
>
> I somewhat agree with you but this "properly described" are all the bug
> reports on lkml containing "bad interactivity in 2.5, cpu starving in 2.5"
> and such...
>
> This isn't a shenanigan, at least not for the interactivity for a desktop.
> This is a workaround for users who are complaining about bad interactivity
> in 2.5!
>
> ciao, Marc

Even though you're not Marc I do agree with you. The problem is well described 
as either poor interactivity (the window wiggle test) or starvation in the 
presence of certain scheduler hogs (for whatever reason) since the 
interactivity patch from mingo. Dropping the max timeslice is a bandaid but 
destroys priority based timeslice scheduling. Dropping the min timeslice will 
bring this back, but at some point the timeslice will be so low that low 
priority cpu intensive tasks will spend most of their time cache trashing.

A reasonable compromise for the desktop would be
min 5 
max 25
but some granularity will be lost in the different sizes of timeslices at 
different priorities.

is there any point having longer timeslices than this?

Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+0Bv6F6dfvkL3i1gRAhpfAKCG3fjkK02lYbAs1p3978rSL/PYAQCcCeK7
gHqR6bgrITE3CSjKCqntw+g=
=rq1o
-----END PGP SIGNATURE-----

Re: I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[William Lee Irwin III]

On Sun, May 25, 2003 at 11:27:20AM +1000, Con Kolivas wrote:
> Even though you're not Marc I do agree with you. The problem is well
> described as either poor interactivity (the window wiggle test) or
> starvation in the presence of certain scheduler hogs (for whatever
> reason) since the interactivity patch from mingo. Dropping the max
> timeslice is a bandaid but destroys priority based timeslice
> scheduling. Dropping the min timeslice will bring this back, but at
> some point the timeslice will be so low that low priority cpu
> intensive tasks will spend most of their time cache trashing.

The fact that it's a "bandaid" and that it "destroys priority-based
timeslice scheduling" makes it a shenanigan. If you're having problems
solved by capping timeslices, you have someone's timeslice and/or
priority growing too large for some reason.

It'd be far better to help figure out what went wrong.


-- wli

Re: I/O problems in 2.4.19/2.4.20/2.4.21-rc3

[Con Kolivas]

On Sun, 25 May 2003 14:28, William Lee Irwin III wrote:
> On Sun, May 25, 2003 at 11:27:20AM +1000, Con Kolivas wrote:
> > Even though you're not Marc I do agree with you. The problem is well
> > described as either poor interactivity (the window wiggle test) or
> > starvation in the presence of certain scheduler hogs (for whatever
> > reason) since the interactivity patch from mingo. Dropping the max
> > timeslice is a bandaid but destroys priority based timeslice
> > scheduling. Dropping the min timeslice will bring this back, but at
> > some point the timeslice will be so low that low priority cpu
> > intensive tasks will spend most of their time cache trashing.
>
> The fact that it's a "bandaid" and that it "destroys priority-based
> timeslice scheduling" makes it a shenanigan. If you're having problems

I don't disagree it's a shenanigan.

> solved by capping timeslices, you have someone's timeslice and/or
> priority growing too large for some reason.

So there is a benefit to timeslices being as large as 200ms? I'll take your 
word for it.

> It'd be far better to help figure out what went wrong.

Love to help. No idea where to begin. All we can do is report what helps the 
symptoms and hope those in the know can decipher it from that.

Con