Re: Kernel deadlock using nbd over acenic driver

[Steven Whitehouse <steve@gw.chygwyn.com>]

Hi,

thanks for the info. I'm starting to form some ideas of what the problem
with nbd might be. Here is my initial idea of what might be going on:

 1. Something happens which starts lots of I/O (e.g. the ext3/xfs journal
    flush that Xiangping says usually triggers the problem)
 2. One of the threads doing the writes blocks running the device I/O
    queue and causing nbd_send_req(), nbd_xmit() to block in the 
    sendmsg() call (trying to allocate memory GFP_NOIO). I think we
    only need to have each memory zones free pages just below pages_min
    at the right time to trigger this.
 3. Since bdflush will most likely be running it waits for the dirty
    blocks its submitted to finish being written back to the
    nbd device to finish.

So something to try is this, in nbd_send_req() add the lines:

	if (current->flags & PF_MEMALLOC == 0) {
		current->flags |= PF_MEMALLOC;
		we_set_memalloc = 1;
	}

before the first nbd_xmit() call and

	if (we_set_memalloc)
		current->flags &= ~PF_MEMALLOC;

at the end just before the return; rememebring to declare the variable:

int we_set_memalloc = 0;

at the top of the function. We know that since the box stays responsive to
pings that there must be some free memory, so I suspect some kind of
"priority inversion" is at work here.

Another interesting idea... if we changed the icmp receive function so that
it leaked all echo request packets rather than recycling them we could
measure the free memory after the box has hung by seeing how many times
we can ping it before it stops replying. An interesting way to
measure free memory, but probably quite effective :-)

It looks like adding the line:

	if (icmph->type == ICMP_ECHO) return;

just after (icmp_pointers[icmph->type].handler)(skb); in icmp.c:icmp_rcv()
should do the trick in this case.

Those are my current thoughts. Let me know if you are sure that some of
what I've said is right/wrong, otherwise I'll have a go tomorrow at
trying to prove some of it on my test system here,

Steve.