From: "Schmid, Carsten" <Carsten_Schmid@mentor.com>
To: Davidlohr Bueso <dave@stgolabs.net>,
Peter Zijlstra <peterz@infradead.org>
Cc: "mingo@redhat.com" <mingo@redhat.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"walken@google.com" <walken@google.com>
Subject: AW: Crash in fair scheduler
Date: Fri, 6 Dec 2019 10:11:25 +0000 [thread overview]
Message-ID: <1575627084926.26450@mentor.com> (raw)
> Von: Davidlohr Bueso [mailto:dave@stgolabs.net]
> Gesendet: Donnerstag, 5. Dezember 2019 18:41
>
> Yeah I had never seen this either, and would expect the world to fall
> appart if leftmost is buggy (much less a one time occurance), but the
> following certainly raises a red flag:
>
> &cfs_rq->tasks_timeline->rb_leftmost
> tasks_timeline = {
> rb_root = {
> rb_node = 0xffff99a9502e0d10
> },
> rb_leftmost = 0x0
> },
>
Meanwhile i am diving a bit deeper into the kernel dump.
I can see that for this rb_root we have a node structure with 2 nodes:
crash> p -x *(struct rb_node *)0xffff99a9502e0d10
$7 = {
__rb_parent_color = 0xffff99a9502e0d10, <- points to SELF
rb_right = 0xffff99a9502e0d10, <- points to self
rb_left = 0xffff99a9502e1990 <- and we have a node left
}
The rb_left node:
crash> p -x *(struct rb_node *)0xffff99a9502e1990
$6 = {
__rb_parent_color = 0xffff99a9502e0d11, <- points to the rb_root node (bit 0 is color)
rb_right = 0x0, <- no leaf
rb_left = 0x0 <- no leaf
}
I'm currently trying to extract the information what se (scheduling entity)
covers these nodes.
Anyway, the cfs_rq->tasks_timeline.rb_leftmost should point to 0xffff99a9502e1990
as far as i understand the rb_tree, right?
> >
> >I suppose one approach is to add code to both __enqueue_entity() and
> >__dequeue_entity() that compares ->rb_leftmost to the result of
> >rb_first(). That'd incur some overhead but it'd double check the logic.
>
> We could benefit from improved debugging in rbtrees, not only the cached
> flavor. Perhaps we can start with the following -- this would at least
> let us know if the case where the tree is non-empty and leftmost is nil
> was hit, whether in the scheduler or another user...
>
> Thanks,
> Davidlohr
>
That's what i will do too, add some debugging stuff.
Add that to the project i'm on here, not upstream; and try
to log as much debug data as possible if a similar case occurs again.
But as rb_tree is excessively used i need to be careful where
to add debug code due to performance impact.
The approach you do with a configurable rb_tree debug
might help me here, yes; i would have taken a similar approach.
Thanks,
Carsten
next reply other threads:[~2019-12-06 10:11 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-06 10:11 Schmid, Carsten [this message]
-- strict thread matches above, loose matches on Subject: below --
2019-12-03 9:11 Crash in fair scheduler Schmid, Carsten
2019-12-03 10:30 ` Valentin Schneider
2019-12-03 10:40 ` Dietmar Eggemann
2019-12-03 11:09 ` Valentin Schneider
2019-12-03 15:08 ` Dietmar Eggemann
2019-12-03 15:57 ` AW: " Schmid, Carsten
2019-12-03 10:30 ` Peter Zijlstra
2019-12-03 10:51 ` AW: " Schmid, Carsten
2019-12-03 14:01 ` Peter Zijlstra
2019-12-05 10:56 ` AW: " Schmid, Carsten
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1575627084926.26450@mentor.com \
--to=carsten_schmid@mentor.com \
--cc=dave@stgolabs.net \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=walken@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).