From: Reuben Farrelly <reuben-lkml@reub.net>
To: Tejun Heo <htejun@gmail.com>
Cc: Ric Wheeler <ric@emc.com>, Jens Axboe <axboe@suse.de>,
Andrew Morton <akpm@osdl.org>,
neilb@suse.de, mingo@elte.hu, linux-kernel@vger.kernel.org,
Jeff Garzik <jgarzik@pobox.com>
Subject: Re: 2.6.15-mm2
Date: Fri, 13 Jan 2006 08:26:42 +1300 [thread overview]
Message-ID: <43C6AD72.2010101@reub.net> (raw)
In-Reply-To: <20060112135533.GA29675@htj.dyndns.org>
Hi,
On 13/01/2006 2:55 a.m., Tejun Heo wrote:
> Hello, again.
>
> On Fri, Jan 13, 2006 at 01:39:18AM +1300, Reuben Farrelly wrote:
>>
>> On 13/01/2006 1:31 a.m., Ric Wheeler wrote:
>>> Reuben Farrelly wrote:
>>>> On 13/01/2006 12:18 a.m., Tejun Heo wrote:
>>>>> On Thu, Jan 12, 2006 at 09:38:48PM +1300, Reuben Farrelly wrote:
>>>>> [--snip--]
>>>>>
>>>>>> [start_ordered ] f7e8a708 -> c1b028fc,c1b029a4,c1b02a4c infl=1
>>>>>> [start_ordered ] f74b0e00 0 48869571 8 8 1 1 c1ba9000
>>>>>> [start_ordered ] BIO f74b0e00 48869571 4096
>>>>>> [start_ordered ] ordered=31 in_flight=1
>>>>>> [blk_do_ordered ] start_ordered f7e8a708->00000000
>>>>>> [blk_do_ordered ] seq=02 f74ccd98->f74ccd98
>>>>>> [blk_do_ordered ] seq=02 f74ccd98->f74ccd98
>>>>>> [blk_do_ordered ] seq=02 c1b028fc->00000000
>>>>>> [blk_do_ordered ] seq=02 c1b028fc->00000000
>>>>>> [blk_do_ordered ] seq=02 c1b028fc->00000000
>>>>>
>>>>> Yeap, this one is the offending one. 0xf74ccd98 got requeued in front
>>>>> of pre-flush while draining and when it finished it didn't complete
>>>>> draining thus hanging the queue. It seems like it's some kind of
>>>>> special request which probably fails and got retried. Are you using
>>>>> SMART or something which issues special commands to drives?
>>>>
>>>> No SMART, although I should be (rebuilt the system a few months
>>>> ago..and must
>>>> have missed it).
>>>>
>>>> Are there any other things which could be contributing to this?
>>>> <scratches head>
>>>>
>>> Could this be hdparm or something tweaking the drive write cache
>>> settings, etc?
>> hdparm isn't configured on the box by me or called by initscripts in Fedora
>> either, AFAIK.
>>
>
> This is the offending part of your new log.
>
> [02 start_ordered ] c1b36120 -> c1b35904,c1b359ac,c1b35a54 ordcolor=1 infl=1
> [02 start_ordered ] f7eb91c0 0 68436682 8 8 1 1 f7dc0000
> [02 start_ordered ] BIO f7eb91c0 68436682 4096
> [02 start_ordered ] ordered=31 in_flight=1
> [02 blk_do_ordered ] start_ordered c1b36120->00000000
> [02 blk_do_ordered ] seq=02 f7e53660->f7e53660 (flags=0x32888)
> [02 elv_completed_request ] seq=01 rq=f7dd7ba0 (flags=0x2000b44) infl=0
> [02 blk_do_ordered ] seq=02 f7e53660->f7e53660 (flags=0x32b88)
> [02 blk_do_ordered ] seq=02 c1b35904->00000000 (flags=0x0)
> [na flush_dry_bio_endio ] BIO c19c7580 48869579 4096
> [na end_that_request_last ] !ELVPRIV c1b3526c 000003d9
> [02 blk_do_ordered ] seq=02 c1b35904->00000000 (flags=0x0)
> [02 elv_completed_request ] seq=01 unacc f7e53660 (flags=0x32b88) infl=0
> [na end_that_request_last ] !ELVPRIV c1b35314 02002318
> [02 blk_do_ordered ] seq=02 c1b35904->00000000 (flags=0x0)
>
> And I was wrong, it wasn't special command being requeued. What
> happens here is....
>
> 1. fs requests are happily being processed
>
> 2. barrier request comes at the head of the queue
>
> 3. ordered code interprets it into three request sequence, a fs
> request is still in flight, so it wait for the queue to be drained.
>
> 4. a REQ_SPECIAL | REQ_BLOCK_PC | REQ_QUIET request gets queued at
> the head of the queue. (I have no idea where this comes from. sd
> driver doesn't even handle PC requests. It will be just failed.
> Some kind of hardware management stuff trying to probe MMC
> devices?)
>
> 5. the in-flight fs request finishes, in_flight is now zero but the
> head of queue is not the ordered sequence. It determines draining
> isn't complete yet.
>
> 6. the special request from #4 got issued and completed, but due to
> my stupid mistake, special requests don't check for draining
> completion condition.
>
> 7. The queue is stuck now. SORRY. My apologies.
>
> Reuben, can you please test the following patch? It's against -mm2
> but should apply to -mm3 too. If you confirm this one, I'll submit to
> Jens & Andrew with proper explanations and stuff. Thanks a lot for
> all your time and trouble.
>
>
> diff --git a/block/elevator.c b/block/elevator.c
> index 1b5b5d9..f905e47 100644
> --- a/block/elevator.c
> +++ b/block/elevator.c
> @@ -615,23 +615,23 @@ void elv_completed_request(request_queue
> * request is released from the driver, io must be done
> */
> if (blk_account_rq(rq)) {
> - struct request *first_rq = list_entry_rq(q->queue_head.next);
> -
> q->in_flight--;
> + if (blk_sorted_rq(rq) && e->ops->elevator_completed_req_fn)
> + e->ops->elevator_completed_req_fn(q, rq);
> + }
>
> - /*
> - * Check if the queue is waiting for fs requests to be
> - * drained for flush sequence.
> - */
> - if (q->ordseq && q->in_flight == 0 &&
> + /*
> + * Check if the queue is waiting for fs requests to be
> + * drained for flush sequence.
> + */
> + if (unlikely(q->ordseq)) {
> + struct request *first_rq = list_entry_rq(q->queue_head.next);
> + if (q->in_flight == 0 &&
> blk_ordered_cur_seq(q) == QUEUE_ORDSEQ_DRAIN &&
> blk_ordered_req_seq(first_rq) > QUEUE_ORDSEQ_DRAIN) {
> blk_ordered_complete_seq(q, QUEUE_ORDSEQ_DRAIN, 0);
> q->request_fn(q);
> }
> -
> - if (blk_sorted_rq(rq) && e->ops->elevator_completed_req_fn)
> - e->ops->elevator_completed_req_fn(q, rq);
> }
> }
Indeed that seems to fix it. I've just booted -mm3 and it came up with no
problems at all.
Many thanks for the fix Tejun :)
reuben
next prev parent reply other threads:[~2006-01-12 19:26 UTC|newest]
Thread overview: 133+ messages / expand[flat|nested] mbox.gz Atom feed top
2006-01-07 13:22 2.6.15-mm2 Andrew Morton
2006-01-07 13:23 ` 2.6.15-mm2 Andrew Morton
2006-01-07 15:05 ` 2.6.15-mm2 Reuben Farrelly
2006-01-07 21:31 ` 2.6.15-mm2 Andrew Morton
2006-01-07 22:06 ` 2.6.15-mm2 Reuben Farrelly
2006-01-07 23:15 ` 2.6.15-mm2 Reuben Farrelly
2006-01-07 23:40 ` 2.6.15-mm2 Andrew Morton
2006-01-10 10:15 ` 2.6.15-mm2 Reuben Farrelly
2006-01-10 10:30 ` 2.6.15-mm2 Andrew Morton
2006-01-10 10:58 ` 2.6.15-mm2 Reuben Farrelly
2006-01-10 10:47 ` 2.6.15-mm2 Ingo Molnar
2006-01-10 10:52 ` 2.6.15-mm2 Ingo Molnar
2006-01-10 10:58 ` 2.6.15-mm2 Ingo Molnar
2006-01-10 11:34 ` 2.6.15-mm2 Ingo Molnar
2006-01-10 12:28 ` 2.6.15-mm2 Reuben Farrelly
2006-01-10 12:42 ` 2.6.15-mm2 Andrew Morton
2006-01-10 13:16 ` 2.6.15-mm2 Ingo Molnar
2006-01-11 4:16 ` 2.6.15-mm2 Neil Brown
2006-01-11 5:15 ` 2.6.15-mm2 Reuben Farrelly
2006-01-11 5:30 ` 2.6.15-mm2 Andrew Morton
2006-01-11 5:30 ` 2.6.15-mm2 Andrew Morton
2006-01-11 10:49 ` 2.6.15-mm2 Reuben Farrelly
2006-01-11 11:05 ` 2.6.15-mm2 Andrew Morton
2006-01-11 11:13 ` 2.6.15-mm2 Jens Axboe
2006-01-11 11:40 ` 2.6.15-mm2 Reuben Farrelly
2006-01-11 11:56 ` 2.6.15-mm2 Jens Axboe
2006-01-11 14:39 ` 2.6.15-mm2 Reuben Farrelly
2006-01-11 14:52 ` 2.6.15-mm2 Jens Axboe
2006-01-11 14:55 ` 2.6.15-mm2 Jens Axboe
2006-01-11 19:23 ` 2.6.15-mm2 Reuben Farrelly
2006-01-11 19:45 ` 2.6.15-mm2 Jens Axboe
2006-01-11 19:53 ` 2.6.15-mm2 Jens Axboe
2006-01-12 3:49 ` 2.6.15-mm2 Reuben Farrelly
2006-01-12 8:00 ` 2.6.15-mm2 Tejun Heo
2006-01-12 8:22 ` 2.6.15-mm2 Jens Axboe
[not found] ` <43C61598.7050004@reub.net>
2006-01-12 11:18 ` 2.6.15-mm2 Tejun Heo
2006-01-12 12:05 ` 2.6.15-mm2 Reuben Farrelly
2006-01-12 12:31 ` 2.6.15-mm2 Ric Wheeler
2006-01-12 12:39 ` 2.6.15-mm2 Reuben Farrelly
2006-01-12 13:55 ` 2.6.15-mm2 Tejun Heo
2006-01-12 14:10 ` 2.6.15-mm2 Jens Axboe
2006-01-12 14:20 ` 2.6.15-mm2 Tejun Heo
2006-01-12 19:26 ` Reuben Farrelly [this message]
2006-01-12 20:32 ` 2.6.15-mm2 Andrew Morton
2006-01-12 20:51 ` 2.6.15-mm2 Jeff Garzik
2006-01-13 4:49 ` 2.6.15-mm2 Reuben Farrelly
2006-01-11 21:44 ` 2.6.15-mm2 Neil Brown
2006-01-12 7:35 ` 2.6.15-mm2 Jens Axboe
2006-01-07 15:08 ` 2.6.15-mm2 Jesper Juhl
2006-01-09 17:47 ` 2.6.15-mm2 Jesper Juhl
2006-01-09 17:57 ` 2.6.15-mm2 Dave Jones
2006-01-09 18:01 ` 2.6.15-mm2 Jesper Juhl
2006-01-09 18:24 ` 2.6.15-mm2 Hugh Dickins
2006-01-09 18:48 ` 2.6.15-mm2 Jesper Juhl
2006-01-09 19:16 ` 2.6.15-mm2 Hugh Dickins
2006-01-09 19:21 ` 2.6.15-mm2 Hugh Dickins
2006-01-09 19:39 ` 2.6.15-mm2 Jesper Juhl
2006-01-09 20:15 ` 2.6.15-mm Hugh Dickins
2006-01-09 20:30 ` 2.6.15-mm Jesper Juhl
2006-01-09 20:41 ` 2.6.15-mm Hugh Dickins
2006-01-09 20:46 ` [PATCH] fix Jesper's sg_page_free Bad page states Hugh Dickins
2006-01-09 20:44 ` 2.6.15-mm Mike Christie
2006-01-09 21:04 ` 2.6.15-mm Hugh Dickins
2006-01-07 16:20 ` 2.6.15-mm2: why is __get_page_state() global again? Adrian Bunk
2006-01-07 18:00 ` [-mm patch] drivers/block/amiflop.c: fix compilation Adrian Bunk
2006-01-07 18:19 ` [-mm patch] drivers/acpi/: make two functions static Adrian Bunk
2006-01-07 18:21 ` [-mm patch] kernel/synchro-test.c: make 5 " Adrian Bunk
2006-01-07 19:31 ` 2.6.15-mm2 Brice Goglin
2006-01-07 21:04 ` 2.6.15-mm2 Dave Jones
2006-01-07 21:26 ` 2.6.15-mm2 Brice Goglin
2006-01-07 21:29 ` 2.6.15-mm2 David S. Miller
2006-01-07 21:41 ` 2.6.15-mm2 Arjan van de Ven
2006-01-07 21:42 ` 2.6.15-mm2 Dave Jones
2006-01-07 21:50 ` 2.6.15-mm2 Brice Goglin
2006-01-07 22:13 ` 2.6.15-mm2 Dave Jones
2006-01-07 22:26 ` 2.6.15-mm2 Brice Goglin
2006-01-11 18:41 ` 2.6.15-mm2 Brice Goglin
2006-01-11 20:29 ` 2.6.15-mm2 Dave Jones
2006-01-11 21:50 ` 2.6.15-mm2 Dave Airlie
2006-01-11 21:56 ` 2.6.15-mm2 Dave Jones
2006-01-11 23:50 ` 2.6.15-mm2 Dave Airlie
2006-01-12 10:58 ` 2.6.15-mm2 Ulrich Mueller
2006-01-12 17:11 ` 2.6.15-mm2 Dave Jones
2006-01-12 18:11 ` 2.6.15-mm2 Ulrich Mueller
2006-01-12 20:37 ` 2.6.15-mm2 Dave Airlie
2006-01-12 21:03 ` 2.6.15-mm2 Alan Hourihane
2006-01-12 22:02 ` 2.6.15-mm2 Dave Airlie
2006-01-13 8:32 ` 2.6.15-mm2 Alan Hourihane
2006-01-13 16:49 ` 2.6.15-mm2 Dave Jones
2006-01-12 19:12 ` 2.6.15-mm2 Brice Goglin
2006-01-12 19:21 ` 2.6.15-mm2 Dave Jones
2006-01-07 22:58 ` 2.6.15-mm2 Andrew Morton
2006-01-07 23:38 ` 2.6.15-mm2 Brice Goglin
2006-01-08 12:24 ` 2.6.15-mm2 Andrew Morton
2006-01-08 14:39 ` 2.6.15-mm2 Brice Goglin
2006-01-08 18:56 ` 2.6.15-mm2 Andrew Morton
2006-01-08 12:28 ` 2.6.15-mm2 Andrew Morton
2006-01-08 14:14 ` 2.6.15-mm2 Brice Goglin
2006-01-07 20:51 ` Badness in __mutex_unlock_slowpath Andrew James Wade
2006-01-07 21:13 ` Arjan van de Ven
2006-01-08 8:53 ` Ingo Molnar
2006-01-07 21:06 ` 2.6.15-mm2: alpha broken Alexey Dobriyan
2006-01-07 23:48 ` Andrew Morton
2006-01-08 0:45 ` [PATCH -mm] fixup *at syscalls additions (alpha, sparc64) Alexey Dobriyan
2006-01-08 0:54 ` [PATCH -mm] Fixup arch/alpha/mm/init.c compilation Alexey Dobriyan
2006-01-08 12:31 ` 2.6.15-mm2: alpha broken Alexey Dobriyan
2006-01-11 2:24 ` Paul Jackson
2006-01-13 14:11 ` Adrian Bunk
2006-01-13 15:52 ` Paul Jackson
2006-01-13 16:37 ` Al Viro
2006-01-13 18:10 ` Paul Jackson
2006-01-13 18:19 ` Randy.Dunlap
2006-01-13 19:05 ` Thomas Gleixner
2006-01-13 21:08 ` Adrian Bunk
2006-01-13 21:12 ` Randy.Dunlap
2006-01-13 21:32 ` Adrian Bunk
2006-01-13 21:52 ` Paul Jackson
2006-01-13 22:18 ` Andrew Morton
2006-01-13 19:26 ` Andrew Morton
2006-01-13 21:05 ` Adrian Bunk
2006-01-08 0:40 ` 2.6.15-mm2 Alexander Gran
[not found] ` <200601080139.34774@zodiac.zodiac.dnsalias.org>
[not found] ` <20060107175056.3d7a2895.akpm@osdl.org>
2006-01-10 0:30 ` 2.6.15-mm2 Alexander Gran
2006-01-10 1:22 ` 2.6.15-mm2 Andrew Morton
2006-01-10 21:20 ` 2.6.15-mm2 Serge E. Hallyn
2006-01-07 21:51 2.6.15-mm2 Chuck Ebbert
2006-01-07 22:01 2.6.15-mm2 Chuck Ebbert
2006-01-08 8:16 2.6.15-mm2 Brown, Len
2006-01-08 14:23 ` 2.6.15-mm2 Brice Goglin
2006-01-08 8:19 2.6.15-mm2 Brown, Len
2006-01-08 9:40 ` 2.6.15-mm2 Reuben Farrelly
2006-01-08 17:58 2.6.15-mm2 Brown, Len
2006-01-08 18:08 2.6.15-mm2 Brown, Len
2006-01-08 18:18 2.6.15-mm2 Brown, Len
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=43C6AD72.2010101@reub.net \
--to=reuben-lkml@reub.net \
--cc=akpm@osdl.org \
--cc=axboe@suse.de \
--cc=htejun@gmail.com \
--cc=jgarzik@pobox.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=neilb@suse.de \
--cc=ric@emc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).