Re: all processes waiting in TASK_UNINTERRUPTIBLE state

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[James Stevenson]

Hi again

i have a stack now an

#0  schedule () at sched.c:536
#1  0x1002f932 in __wait_on_buffer (bh=0x50eb16e4) at  buffer.c:157
#2  0x10036f46 in block_read (filp=0x5009787c, buf=0x80c08f0
	"¤\201", count=8192, ppos=0x5009789c) at
	/home/mistral/dev/kernel/linux-2.4.5-um9/include/linux/locks.h:20
#3  0x1002eb4b in sys_read (fd=3, buf=0x80c00f0 "¤\201", count=8192)
	at read_write.c:133
#4  0x100fb807 in execute_syscall (regs={regs = {3, 135004400, 8192,
	1283476480, 0, 3212835652, 4294967258, 43, 43, 0, 0, 3,
	1074582884, 35, 582, 3212835588, 43}}) at syscall_kern.c:332
#5  0x100fb926 in syscall_handler (unused=0x0) at
	syscall_user.c:80

this should still be on #umldebug on irc.openproject.net
for the next few ours if anybodys intresting at taking a look though
gdb via a bot.

	James

-- 
---------------------------------------------
Web: http://www.stev.org
Mobile: +44 07779080838
E-Mail: mistral@stev.org
  8:30pm  up 2 days, 42 min,  4 users,  load average: 1.00, 0.94, 0.64

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Jeff Dike]

To recap the story so far, a number of people have reported seeing processes 
hang indefinitely in TASK_UNINTERRUPTIBLE in __wait_on_buffer or __lock_page, 
both on physical boxes and under UML.  It was found to be reproducable under 
UML, but not on a native kernel as far as I know.

I've fixed this problem in UML.

The short story :

The bug was UML-specific and specific in such a way that I don't think it's 
possible to find the bug in the native kernel by making analogies from the UML 
bug.

The long story :

First, two pieces of background information
	- UML's ubd block driver performs asynchronous I/O by using a separate thread 
to perform the I/O.  The driver's request routine writes the request to the 
I/O thread over a file descriptor.  The thread performs the request by calling 
either read() or write() on the host.  When that call returns, it writes the 
results back to UML, causing a SIGIO.  That goes through the normal IRQ system 
and ends up in the ubd interrupt handler, which finishes the request, and, if 
the device request queue isn't empty, starts the next request.
	- A couple of weeks ago, I made a change to reduce the number of clock ticks 
that UML loses under load.  The clock is implemented with SIGALRM and 
SIGVTALRM.  The change involved leaving those signals enabled all the time, 
and having the handler decide whether it was safe to invoke the timer irq.  If 
not, it bumped a missing_ticks counter and returned.  The missing ticks are 
fully accounted for the next time the handler finds that it can call into the 
irq system.

The ubd request routine runs with interrupts off, but now, the alarms could at 
least fire, even if they didn't do anything but increment a counter.  This 
occasionally caused the write of the I/O request from the request routine to 
the I/O thread to return -EINTR.  The return value wasn't checked, so the I/O 
thread didn't get the request and the request routine had no idea that 
anything was wrong.  This caused disk I/O to permanently shut down, so pending 
requests stayed pending, and processes waiting on them waited forever.

The key piece of this bug was a signal causing a crucial communication between 
two UML threads to be lost.  I don't see any analogies between this and 
anything that happens on a physical system, so it looks to me like the problem 
that people are seeing there is completely different.

Thanks to James Stevenson for figuring out how to reproduce the bug under UML, 
and to Daniel Phillips and Al Viro for help in tracking this problem from the 
fs and mm systems into the block and driver layers.

				Jeff

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Daniel Phillips]

On Friday 29 June 2001 22:40, Jeff Dike wrote:
> The bug was UML-specific and specific in such a way that I don't think it's
> possible to find the bug in the native kernel by making analogies from the
> UML bug.

Heh, too bad, there goes that chance to show uml bagging a major kernel bug.  
But it's just a matter of time...

--
Daniel

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Jeff Dike]

phillips@bonn-fries.net said:
> Heh, too bad, there goes that chance to show uml bagging a major
> kernel bug.   But it's just a matter of time... 

Yeah, hopefully it's already prevented a number of them.  But those are a lot 
harder to count...

				Jeff

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Andrea Arcangeli]

On Tue, Jun 26, 2001 at 10:47:12AM -0400, Bulent Abali wrote:
> Andrea,
> I would like try your patch but so far I can trigger the bug only when
> running TUX 2.0-B6 which runs on 2.4.5-ac4.  /bulent
> 

to run tux you can apply those patches in `ls` order to 2.4.6pre5aa1:

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.6pre5aa1/30_tux/*

Andrea

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Bulent Abali]

>> I am running in to a problem, seemingly a deadlock situation, where
almost
>> all the processes end up in the TASK_UNINTERRUPTIBLE state.   All the
>
>could you try to reproduce with this patch applied on top of
>2.4.6pre5aa1 or 2.4.6pre5 vanilla?

Andrea,
I would like try your patch but so far I can trigger the bug only when
running TUX 2.0-B6 which runs on 2.4.5-ac4.  /bulent

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Andrea Arcangeli]

On Mon, Jun 25, 2001 at 10:10:38AM -0400, Bulent Abali wrote:
> 
> keywords:  tux, aic7xxx, 2.4.5-ac4, specweb99, __wait_on_page, __lock_page
> 
> Greetings,
> 
> I am running in to a problem, seemingly a deadlock situation, where almost
> all the processes end up in the TASK_UNINTERRUPTIBLE state.   All the

could you try to reproduce with this patch applied on top of
2.4.6pre5aa1 or 2.4.6pre5 vanilla?

diff -urN 2.4.6pre5aa1/include/linux/swap.h 2.4.6pre5aa1-backout-page_launder/include/linux/swap.h
--- 2.4.6pre5aa1/include/linux/swap.h	Sun Jun 24 02:06:13 2001
+++ 2.4.6pre5aa1-backout-page_launder/include/linux/swap.h	Sun Jun 24 21:37:12 2001
@@ -205,16 +205,6 @@
 	page->zone->inactive_dirty_pages++; \
 }
 
-/* Like the above, but add us after the bookmark. */
-#define add_page_to_inactive_dirty_list_marker(page) { \
-	DEBUG_ADD_PAGE \
-	ZERO_PAGE_BUG \
-	SetPageInactiveDirty(page); \
-	list_add(&(page)->lru, marker_lru); \
-	nr_inactive_dirty_pages++; \
-	page->zone->inactive_dirty_pages++; \
-}
-
 #define add_page_to_inactive_clean_list(page) { \
 	DEBUG_ADD_PAGE \
 	ZERO_PAGE_BUG \
diff -urN 2.4.6pre5aa1/mm/vmscan.c 2.4.6pre5aa1-backout-page_launder/mm/vmscan.c
--- 2.4.6pre5aa1/mm/vmscan.c	Sun Jun 24 01:41:09 2001
+++ 2.4.6pre5aa1-backout-page_launder/mm/vmscan.c	Sun Jun 24 21:37:11 2001
@@ -407,7 +407,7 @@
 /**
  * page_launder - clean dirty inactive pages, move to inactive_clean list
  * @gfp_mask: what operations we are allowed to do
- * @sync: are we allowed to do synchronous IO in emergencies ?
+ * @sync: should we wait synchronously for the cleaning of pages
  *
  * When this function is called, we are most likely low on free +
  * inactive_clean pages. Since we want to refill those pages as
@@ -426,61 +426,23 @@
 #define MAX_LAUNDER 		(4 * (1 << page_cluster))
 #define CAN_DO_IO		(gfp_mask & __GFP_IO)
 #define CAN_DO_BUFFERS		(gfp_mask & __GFP_BUFFER)
-#define marker_lru		(&marker_page_struct.lru)
 int page_launder(int gfp_mask, int sync)
 {
-	static int cannot_free_pages;
 	int launder_loop, maxscan, cleaned_pages, maxlaunder;
 	struct list_head * page_lru;
 	struct page * page;
 
-	/* Our bookmark of where we are in the inactive_dirty list. */
-	struct page marker_page_struct = { zone: NULL };
-
 	launder_loop = 0;
 	maxlaunder = 0;
 	cleaned_pages = 0;
 
 dirty_page_rescan:
 	spin_lock(&pagemap_lru_lock);
-	/*
-	 * By not scanning all inactive dirty pages we'll write out
-	 * really old dirty pages before evicting newer clean pages.
-	 * This should cause some LRU behaviour if we have a large
-	 * amount of inactive pages (due to eg. drop behind).
-	 *
-	 * It also makes us accumulate dirty pages until we have enough
-	 * to be worth writing to disk without causing excessive disk
-	 * seeks and eliminates the infinite penalty clean pages incurred
-	 * vs. dirty pages.
-	 */
-	maxscan = nr_inactive_dirty_pages / 4;
-	if (launder_loop)
-		maxscan *= 2;
-	list_add_tail(marker_lru, &inactive_dirty_list);
-	for (;;) {
-		page_lru = marker_lru->prev;
-		if (page_lru == &inactive_dirty_list)
-			break;
-		if (--maxscan < 0)
-			break;
-		if (!free_shortage())
-			break;
-
+	maxscan = nr_inactive_dirty_pages;
+	while ((page_lru = inactive_dirty_list.prev) != &inactive_dirty_list &&
+				maxscan-- > 0) {
 		page = list_entry(page_lru, struct page, lru);
 
-		/* Move the bookmark backwards.. */
-		list_del(marker_lru);
-		list_add_tail(marker_lru, page_lru);
-
-		/* Don't waste CPU if chances are we cannot free anything. */
-		if (launder_loop && maxlaunder < 0 && cannot_free_pages)
-			break;
-
-		/* Skip other people's marker pages. */
-		if (!page->zone)
-			continue;
-
 		/* Wrong page on list?! (list corruption, should not happen) */
 		if (!PageInactiveDirty(page)) {
 			printk("VM: page_launder, wrong page on list.\n");
@@ -492,6 +454,7 @@
 
 		/* Page is or was in use?  Move it to the active list. */
 		if (PageReferenced(page) || page->age > 0 ||
+				page->zone->free_pages > page->zone->pages_high ||
 				(!page->buffers && page_count(page) > 1) ||
 				page_ramdisk(page)) {
 			del_page_from_inactive_dirty_list(page);
@@ -501,9 +464,11 @@
 
 		/*
 		 * The page is locked. IO in progress?
-		 * Skip the page, we'll take a look when it unlocks.
+		 * Move it to the back of the list.
 		 */
 		if (TryLockPage(page)) {
+			list_del(page_lru);
+			list_add(page_lru, &inactive_dirty_list);
 			continue;
 		}
 
@@ -517,8 +482,10 @@
 			if (!writepage)
 				goto page_active;
 
-			/* First time through? Skip the page. */
+			/* First time through? Move it to the back of the list */
 			if (!launder_loop || !CAN_DO_IO) {
+				list_del(page_lru);
+				list_add(page_lru, &inactive_dirty_list);
 				UnlockPage(page);
 				continue;
 			}
@@ -531,8 +498,6 @@
 			writepage(page);
 			page_cache_release(page);
 
-			maxlaunder--;
-
 			/* And re-start the thing.. */
 			spin_lock(&pagemap_lru_lock);
 			continue;
@@ -560,9 +525,9 @@
 			spin_unlock(&pagemap_lru_lock);
 
 			/* Will we do (asynchronous) IO? */
-			if (launder_loop && maxlaunder-- == 0 && sync)
+			if (launder_loop && maxlaunder == 0 && sync)
 				wait = 2;	/* Synchrounous IO */
-			else if (launder_loop && maxlaunder > 0)
+			else if (launder_loop && maxlaunder-- > 0)
 				wait = 1;	/* Async IO */
 			else
 				wait = 0;	/* No IO */
@@ -579,7 +544,7 @@
 
 			/* The buffers were not freed. */
 			if (!clearedbuf) {
-				add_page_to_inactive_dirty_list_marker(page);
+				add_page_to_inactive_dirty_list(page);
 
 			/* The page was only in the buffer cache. */
 			} else if (!page->mapping) {
@@ -635,8 +600,6 @@
 			UnlockPage(page);
 		}
 	}
-	/* Remove our marker. */
-	list_del(marker_lru);
 	spin_unlock(&pagemap_lru_lock);
 
 	/*
@@ -652,29 +615,16 @@
 	 */
 	if ((CAN_DO_IO || CAN_DO_BUFFERS) && !launder_loop && free_shortage()) {
 		launder_loop = 1;
-		/*
-		 * If we, or the previous process running page_launder(),
-		 * managed to free any pages we never do synchronous IO.
-		 */
-		if (cleaned_pages || !cannot_free_pages)
+		/* If we cleaned pages, never do synchronous IO. */
+		if (cleaned_pages)
 			sync = 0;
-		/* Else, do synchronous IO (if we are allowed to). */
-		else if (sync)
-			sync = 1;
 		/* We only do a few "out of order" flushes. */
 		maxlaunder = MAX_LAUNDER;
-		/* Let bdflush take care of the rest. */
+		/* Kflushd takes care of the rest. */
 		wakeup_bdflush(0);
 		goto dirty_page_rescan;
 	}
 
-	/*
-	 * If we failed to free pages (because all pages are dirty)
-	 * we remember this for the next time. This will prevent us
-	 * from wasting too much CPU here.
-	 */
-	cannot_free_pages = !cleaned_pages;
-
 	/* Return the number of pages moved to the inactive_clean list. */
 	return cleaned_pages;
 }
@@ -899,7 +849,7 @@
 	 * list, so this is a relatively cheap operation.
 	 */
 	if (free_shortage()) {
-		ret += page_launder(gfp_mask, 1);
+		ret += page_launder(gfp_mask, user);
 		shrink_dcache_memory(DEF_PRIORITY, gfp_mask);
 		shrink_icache_memory(DEF_PRIORITY, gfp_mask);
 	}
Andrea

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Rik van Riel]

On Tue, 26 Jun 2001, Christian Ehrhardt wrote:

> > I've seen this under UML, Rik van Riel has seen it on a physical box, and we 
> > suspect that they're the same problem (i.e. mine isn't a UML-specific bug).
> 
> Could it be smbfs?

No. I've seen the hang on pure ex2.

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Christian Ehrhardt]

On Mon, Jun 25, 2001 at 12:05:14PM -0500, Jeff Dike wrote:
> abali@us.ibm.com said:
> > I am running in to a problem, seemingly a deadlock situation, where
> > almost all the processes end up in the TASK_UNINTERRUPTIBLE state.
> > All the process eventually stop responding, including login shell, no
> > screen updates, keyboard etc.  Can ping and sysrq key works.   I
> > traced the tasks through sysrq-t key.  The processors are in the idle
> > state.  Tasks all seem to get stuck in the __wait_on_page or
> > __lock_page.
> 
> I've seen this under UML, Rik van Riel has seen it on a physical box, and we 
> suspect that they're the same problem (i.e. mine isn't a UML-specific bug).
> 
> I've done some poking at the problem, but haven't really learned anything 
> except that something is locking pages and not unlocking them.  Figuring out 
> who that is was going to be my next step.

Could it be smbfs? The following piece of code from smb_writepage
looks like it could return with the page locked:


      static int
      smb_writepage(struct page *page)
      {

	      /*   ....    */

	      /* easy case */
	      if (page->index < end_index)
		      goto do_it;
	      /* things got complicated... */
	      offset = inode->i_size & (PAGE_CACHE_SIZE-1);
	      /* OK, are we completely out? */
	      if (page->index >= end_index+1 || !offset)
		      return -EIO;           <=====   This looks bad!
      do_it:
	      get_page(page);
	      err = smb_writepage_sync(inode, page, 0, offset);
	      SetPageUptodate(page);
	      UnlockPage(page);
	      put_page(page);
	      return err;
      }


   regards     Christian

-- 
THAT'S ALL FOLKS!

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[James Stevenson]

Hi

i have been looking at it a lot over the past few days i seem to be the
person who can trigger it easyest.

over the past couple of days i have been running with the
#define WAITQUEUE_DEBUG 1
no problems seem to have appeared there though and the bug still triggers.

On Mon, 25 Jun 2001, Jeff Dike wrote:

> abali@us.ibm.com said:
> > I am running in to a problem, seemingly a deadlock situation, where
> > almost all the processes end up in the TASK_UNINTERRUPTIBLE state.
> > All the process eventually stop responding, including login shell, no
> > screen updates, keyboard etc.  Can ping and sysrq key works.   I
> > traced the tasks through sysrq-t key.  The processors are in the idle
> > state.  Tasks all seem to get stuck in the __wait_on_page or
> > __lock_page.

i also seem to get ut ub __wait_on_buffer and ___wait_on_page

	James
-- 
---------------------------------------------
Web: http://www.stev.org
Mobile: +44 07779080838
E-Mail: mistral@stev.org
  8:00pm  up 2 days, 12 min,  4 users,  load average: 1.41, 0.38, 0.40

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Jeff Dike]

abali@us.ibm.com said:
> Can you give more details? 

BTW, here are a couple of the UML stacks:

#3  0x1000895d in schedule () at sched.c:669
#4 0x1001c9bd in ___wait_on_page (page=0x50025740) at filemap.c:641
#5  0x1001d286 in do_generic_file_read (filp=0x50d4c674, ppos=0x50d4c694,
    desc=0x50d1bea8, actor=0x1001d410 <file_read_actor>) at 
/home/mistral/dev/kernel/linux-2.4.5-um-06-15/include/linux/pagemap.h:97
#6  0x1001d528 in generic_file_read (filp=0x50d4c674, buf=0x401b7000 <Address 
0x401b7000 out of bounds>, count=4096, ppos=0x50d4c694) at filemap.c:1302
#7  0x1002ba2b in sys_read (fd=6, buf=0x401b7000 <Address 0x401b7000 out of 
bounds>, count=4096) at read_write.c:133

#3  0x1000895d in schedule () at sched.c:669
#4  0x1001c9bd in ___wait_on_page (page=0x50015bb4) at filemap.c:641
#5  0x1001a2e1 in do_swap_page (mm=0x5009ea34, vma=0x50f83dac, 
address=3212835782, page_table=0x50d5cffc, entry={val = 202240}, 
write_access=0) at /home/mistral/dev/kernel/linux-2.4.5-um7/include/linux/pagem
ap.h:97
#6  0x1001a733 in handle_mm_fault (mm=0x5009ea34, vma=0x50f83dac, 
address=3212835782, write_access=0) at memory.c:1307
#7  0x10012c40 in access_one_page (mm=0x5009ea34, vma=0x50f83dac, 
addr=3212835782, buf=0x5153e000, len=15, write=0) at ptrace.c:78
#8  0x10012cd8 in access_mm (mm=0x5009ea34, vma=0x50f83dac, addr=3212835782, 
buf=0x5153e000, len=15, write=0) at ptrace.c:104
#9  0x10012d5b in access_process_vm (tsk=0x50db0000, addr=3212835782, 
buf=0x5153e000, len=15, write=0) at ptrace.c:147
#10 0x1004656b in proc_pid_cmdline (task=0x50db0000, buffer=0x5153e000 "74 66 
64 8 0 58 14\n") at base.c:157
#11 0x10046914 in proc_info_read (file=0x50d4eefc, buf=0xbf7ff0b0 
"/proc/117/cmdline", count=2047, ppos=0x50d4ef1c) at base.c:288
#12 0x1002ba2b in sys_read (fd=6, buf=0xbf7ff0b0 "/proc/117/cmdline", 
count=2047) at read_write.c:133

There are a few more at http://www.geocrawler.com/lists/3/SourceForge/709/0/598
8636/

				Jeff

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Jeff Dike]

abali@us.ibm.com said:
> Can you give more details?  Was there an aic7xxx scsi driver on the
> box? run_task_queue(&tq_disk) should eventually unlock those pages but
> they remain locked.  I am trying to narrow it down to fs/buffer code
> or the SCSI driver aic7xxx in my case.

Rik would be the one to tell you whether there was an aic7xxx driver on the 
physical box.  There obviously isn't one on UML, so if we're looking at the 
same bug, it's in the generic code.

				Jeff

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Rik van Riel]

On Mon, 25 Jun 2001, Jeff Dike wrote:
> abali@us.ibm.com said:
> > Can you give more details?  Was there an aic7xxx scsi driver on the
> > box? run_task_queue(&tq_disk) should eventually unlock those pages but
> > they remain locked.  I am trying to narrow it down to fs/buffer code
> > or the SCSI driver aic7xxx in my case.
> 
> Rik would be the one to tell you whether there was an aic7xxx driver
> on the physical box.  There obviously isn't one on UML, so if we're
> looking at the same bug, it's in the generic code.

The box has as AIC-7880U controller. OTOH, my dual P5 also has
an AIC7xxx controller and I've never seen the problem there...

On our quad Xeon this problem really seems to be phase-of-moon
related; it hasn't shown up in the last 5 days or so under heavy
stress testing, but when the kernel is compiled just a little bit
different it doesn't happen. ;)

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Jeff Dike]

abali@us.ibm.com said:
> I am running in to a problem, seemingly a deadlock situation, where
> almost all the processes end up in the TASK_UNINTERRUPTIBLE state.
> All the process eventually stop responding, including login shell, no
> screen updates, keyboard etc.  Can ping and sysrq key works.   I
> traced the tasks through sysrq-t key.  The processors are in the idle
> state.  Tasks all seem to get stuck in the __wait_on_page or
> __lock_page.

I've seen this under UML, Rik van Riel has seen it on a physical box, and we 
suspect that they're the same problem (i.e. mine isn't a UML-specific bug).

I've done some poking at the problem, but haven't really learned anything 
except that something is locking pages and not unlocking them.  Figuring out 
who that is was going to be my next step.

If anyone is interested in poking around a UML in this state (i.e. you get all 
the niceties of gdb), let me know.  I think I can probably oblige.

				Jeff

Re: all processes waiting in TASK_UNINTERRUPTIBLE state

[Bulent Abali]

>abali@us.ibm.com said:
>> I am running in to a problem, seemingly a deadlock situation, where
>> almost all the processes end up in the TASK_UNINTERRUPTIBLE state.
>> All the process eventually stop responding, including login shell, no
>> screen updates, keyboard etc.  Can ping and sysrq key works.   I
>> traced the tasks through sysrq-t key.  The processors are in the idle
>> state.  Tasks all seem to get stuck in the __wait_on_page or
>> __lock_page.
>
>I've seen this under UML, Rik van Riel has seen it on a physical box, and
we
>suspect that they're the same problem (i.e. mine isn't a UML-specific
bug).

Can you give more details?  Was there an aic7xxx scsi driver on the box?
run_task_queue(&tq_disk) should eventually unlock those pages
but they remain locked.  I am trying to narrow it down to fs/buffer
code or the SCSI driver aic7xxx in my case. Thanks. /bulent

all processes waiting in TASK_UNINTERRUPTIBLE state

[Bulent Abali]

keywords:  tux, aic7xxx, 2.4.5-ac4, specweb99, __wait_on_page, __lock_page

Greetings,

I am running in to a problem, seemingly a deadlock situation, where almost
all the processes end up in the TASK_UNINTERRUPTIBLE state.   All the
process eventually stop responding, including login shell, no screen
updates, keyboard etc.  Can ping and sysrq key works.   I traced the tasks
through sysrq-t key.  The processors are in the idle state.  Tasks all seem
to get stuck in the __wait_on_page or __lock_page.  It appears from the
source that they are waiting for pages to be unlocked.   run_task_queue
(&tq_disk) should eventually cause pages to unlock but it doesn't happen.
Anybody familiar with this problem or have seen it before?  Thanks for any
comments.
Bulent

Here are the conditions:
Dual PIII, 1GHz, 1GB of memory,  aic7xxx scsi driver, acenic eth.
This occurs while TUX  (2.4.5-B6) webserver is being driven by SPECWeb99
benchmark at a rate of 800 c/s.  The system is very busy doing disk and
network I/O.  Problem occurs sometimes in an hour and sometimes 10-20 hours
in to the running.

Bulent


Process: 0, {             swapper}
EIP: 0010:[<c010524d>] CPU: 1 EFLAGS: 00000246
EAX: 00000000 EBX: c0105220 ECX: c2afe000 EDX: 00000025
ESI: c2afe000 EDI: c2afe000 EBP: c0105220 DS: 0018 ES: 0018
CR0: 8005003b CR2: 08049df0 CR3: 268e0000 CR4: 000006d0
Call Trace: [<c01052d2>] [<c0119186>] [<c01192fb>]
SysRq : Show Regs

Process: 0, {             swapper}
EIP: 0010:[<c010524d>] CPU: 0 EFLAGS: 00000246
EAX: 00000000 EBX: c0105220 ECX: c030a000 EDX: 00000000
ESI: c030a000 EDI: c030a000 EBP: c0105220 DS: 0018 ES: 0018
CR0: 8005003b CR2: 08049f7c CR3: 37a63000 CR4: 000006d0
Call Trace: [<c01052d2>] [<c0105000>] [<c01001cf>]
SysRq : Show Regs

EIP: 0010:[<c010524d>] CPU: 1 EFLAGS: 00000246
Using defaults from ksymoops -t elf32-i386 -a i386
EAX: 00000000 EBX: c0105220 ECX: c2afe000 EDX: 00000025
ESI: c2afe000 EDI: c2afe000 EBP: c0105220 DS: 0018 ES: 0018
CR0: 8005003b CR2: 08049df0 CR3: 268e0000 CR4: 000006d0
Call Trace: [<c01052d2>] [<c0119186>] [<c01192fb>]

EIP: 0010:[<c010524d>] CPU: 0 EFLAGS: 00000246
EAX: 00000000 EBX: c0105220 ECX: c030a000 EDX: 00000000
ESI: c030a000 EDI: c030a000 EBP: c0105220 DS: 0018 ES: 0018
CR0: 8005003b CR2: 08049f7c CR3: 37a63000 CR4: 000006d0
Call Trace: [<c01052d2>] [<c0105000>] [<c01001cf>]

>>EIP; c010524d <default_idle+2d/40>   <=====
Trace; c01052d2 <cpu_idle+52/70>
Trace; c0119186 <__call_console_drivers+46/60>
Trace; c01192fb <call_console_drivers+eb/100>

>>EIP; c010524d <default_idle+2d/40>   <=====
Trace; c01052d2 <cpu_idle+52/70>
Trace; c0105000 <prepare_namespace+0/10>
Trace; c01001cf <L6+0/2>

=================

SysRq : Show Memory
Mem-info:
Free pages:        4300kB (   792kB HighMem)
( Active: 200434, inactive_dirty: 26808, inactive_clean: 1472, free: 1075
(574 1148 1722) )
24*4kB 15*8kB 2*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 728kB)
493*4kB 3*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 0*4096kB = 2780kB)
0*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB
0*4096kB = 792kB)
Swap cache: add 2711, delete 643, find 5301/6721
Free swap:       2087996kB
253932 pages of RAM
24556 pages of HIGHMEM
7212 reserved pages
221419 pages shared
2068 pages swap cached
0 pages in page table cache
Buffer memory:    12164kB
    CLEAN: 2322 buffers, 9276 kbyte, 3 used (last=2322), 2 locked, 0
protected, 0 dirty
   LOCKED: 405 buffers, 1608 kbyte, 39 used (last=404), 348 locked, 0
protected, 0 dirty
    DIRTY: 322 buffers, 1288 kbyte, 0 used (last=0), 322 locked, 0
protected, 322 dirty

=====================

async IO 0/2  D 00000013     0  1061   1059          1062       (NOTLB)
Call Trace: [<c012e121>] [<c012f059>] [<c02614d7>] [<c0258c44>]
[<c02588c0>]
   [<c025c65a>] [<c0256848>] [<c0258478>] [<c0105636>] [<c02582a0>]

Trace; c012e121 <___wait_on_page+91/c0>
Trace; c012f059 <do_generic_file_read+449/7d0>
Trace; c02614d7 <send_abuf+27/30>
Trace; c0258c44 <generic_send_file+84/100>
Trace; c02588c0 <sock_send_actor+0/1a0>
Trace; c025c65a <http_send_body+6a/100>
Trace; c0256848 <tux_schedule_atom+18/20>
Trace; c0258478 <cachemiss_thread+1d8/350>
Trace; c0105636 <kernel_thread+26/30>
Trace; c02582a0 <cachemiss_thread+0/350>


==================

bash          D C2AE541C     0   920    912                     (NOTLB)
Call Trace: [<c012e1e1>] [<c012e04d>] [<c016b880>] [<c012fdac>]
[<c012a76a>]
   [<c012a8cb>] [<c0110018>] [<c02709c7>] [<c0113ed0>] [<c0114106>]
[<c0195494>]
   [<c01417d2>] [<c011e25b>] [<c0113ed0>] [<c01075b8>

Trace; c012e1e1 <__lock_page+91/c0>
Trace; c012e04d <read_cluster_nonblocking+17d/1c0>
Trace; c016b880 <ext2_get_block+0/5b0>
Trace; c012fdac <filemap_nopage+3fc/5b0>
Trace; c012a49a <do_swap_page+23a/2f0>
Trace; c012a76a <do_no_page+8a/150>
Trace; c012a8cb <handle_mm_fault+9b/150>
Trace; c021814c <sock_sendmsg+6c/90>
Trace; c0113ed0 <do_page_fault+0/550>
Trace; c0114106 <do_page_fault+236/550>
Trace; c0118aa5 <do_syslog+1e5/820>
Trace; c01417d2 <sys_read+c2/d0>
Trace; c011e25b <do_softirq+6b/a0>
Trace; c0113ed0 <do_page_fault+0/550>
Trace; c01075b8 <error_code+34/3c>



void ___wait_on_page(struct page *page)
{
        struct task_struct *tsk = current;
        DECLARE_WAITQUEUE(wait, tsk);

        add_wait_queue(&page->wait, &wait);
        do {
                sync_page(page);
                set_task_state(tsk, TASK_UNINTERRUPTIBLE);
                if (!PageLocked(page))
                        break;
                run_task_queue(&tq_disk);
                schedule();
        } while (PageLocked(page));
        tsk->state = TASK_RUNNING;
        remove_wait_queue(&page->wait, &wait);
}

static void __lock_page(struct page *page)
{
        struct task_struct *tsk = current;
        DECLARE_WAITQUEUE(wait, tsk);

        add_wait_queue_exclusive(&page->wait, &wait);
        for (;;) {
                sync_page(page);
                set_task_state(tsk, TASK_UNINTERRUPTIBLE);
                if (PageLocked(page)) {
                        run_task_queue(&tq_disk);
                        schedule();
                        continue;
                }
                if (!TryLockPage(page))
                        break;
        }
        tsk->state = TASK_RUNNING;
        remove_wait_queue(&page->wait, &wait);
}