From: "Jim Schutt" <jaschut@sandia.gov>
To: Sage Weil <sage@newdream.net>
Cc: Gregory Farnum <gregory.farnum@dreamhost.com>,
"ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: cosd multi-second stalls cause "wrongly marked me down"
Date: Wed, 2 Mar 2011 08:21:02 -0700 [thread overview]
Message-ID: <1299079262.4750.19.camel@sale659.sandia.gov> (raw)
In-Reply-To: <Pine.LNX.4.64.1103011646140.32424@cobra.newdream.net>
On Tue, 2011-03-01 at 17:53 -0700, Sage Weil wrote:
> Hi Jim,
>
> We've fixed a few different bugs over the last week that were causing
> heartbeat issues.
Great!
> Nothing that explains why we would see the hang that
> you did, but other problems that caused the same 'wrongly marked me down'
> issue. Are you still seeing this problem with the latest 'next' and/or
> 'master' branch?
I've been trying to isolate this on the stable branch
since my last posting - I can still reproduce at will
with my 96 osd test, but I haven't made much progress
at tracking down what is going wrong.
>
> Also, if you don't mind reproducing, can you post a larger segment of the
> log?
Sure. I've got some extra debug printing going in
my tree - the most interesting is a patch to log
queue, operation, and total elapsed times in
dispatch_entry() - it makes is really easy to
find when things go wrong.
I'll try to reproduce with master and post logs.
Is it OK for me to add my extra debug patches for
that? I'll post them with the logs if so.
> The really interesting question is what the heartbeat thread
> (heartbeat_entry()) is doing during this period that tick() is blocked up,
> since that's the thread that's responsible for sending the ping messages
> to peer OSDs.
One of the things I am seeing is handle_osd_ping()
getting stalled, but I haven't been able to track
down why.
I'll see if I see the same signature with master,
and post logs.
-- Jim
>
> Thanks!
> sage
>
>
next prev parent reply other threads:[~2011-03-02 15:21 UTC|newest]
Thread overview: 94+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-02-16 21:25 cosd multi-second stalls cause "wrongly marked me down" Jim Schutt
2011-02-16 21:37 ` Wido den Hollander
2011-02-16 21:51 ` Jim Schutt
2011-02-16 21:40 ` Gregory Farnum
2011-02-16 21:50 ` Jim Schutt
2011-02-17 0:50 ` Sage Weil
2011-02-17 0:54 ` Sage Weil
2011-02-17 15:46 ` Jim Schutt
2011-02-17 16:11 ` Sage Weil
2011-02-17 23:31 ` Jim Schutt
2011-02-18 7:13 ` Sage Weil
2011-02-18 17:04 ` Jim Schutt
2011-02-18 17:15 ` Gregory Farnum
2011-02-18 18:41 ` Jim Schutt
2011-02-18 19:07 ` Colin McCabe
2011-02-18 20:48 ` Jim Schutt
2011-02-18 20:58 ` Sage Weil
2011-02-18 21:09 ` Jim Schutt
2011-03-09 16:02 ` Jim Schutt
2011-03-09 17:07 ` Gregory Farnum
2011-03-09 18:36 ` Jim Schutt
2011-03-09 19:37 ` Gregory Farnum
2011-03-10 23:09 ` Jim Schutt
2011-03-10 23:21 ` Sage Weil
2011-03-10 23:32 ` Jim Schutt
2011-03-10 23:40 ` Sage Weil
2011-03-11 14:51 ` Jim Schutt
2011-03-11 18:26 ` Jim Schutt
2011-03-11 18:37 ` Jim Schutt
2011-03-11 18:37 ` Sage Weil
2011-03-11 18:51 ` Jim Schutt
2011-03-11 19:09 ` Gregory Farnum
2011-03-11 19:13 ` Yehuda Sadeh Weinraub
2011-03-11 19:17 ` Yehuda Sadeh Weinraub
2011-03-11 19:16 ` Jim Schutt
2011-03-11 21:13 ` Jim Schutt
2011-03-11 21:37 ` Sage Weil
2011-03-11 22:21 ` Jim Schutt
2011-03-11 22:26 ` Jim Schutt
2011-03-11 22:45 ` Sage Weil
2011-03-11 23:29 ` Jim Schutt
2011-03-30 21:26 ` Jim Schutt
2011-03-30 21:55 ` Sage Weil
2011-03-31 14:16 ` Jim Schutt
2011-03-31 16:25 ` Sage Weil
2011-03-31 17:00 ` Jim Schutt
2011-03-31 17:10 ` Jim Schutt
2011-03-31 17:24 ` Sage Weil
2011-03-31 18:08 ` Jim Schutt
2011-03-31 18:41 ` Sage Weil
2011-04-01 22:38 ` Jim Schutt
2011-02-23 17:52 ` Jim Schutt
2011-02-23 18:12 ` Gregory Farnum
2011-02-23 18:54 ` Sage Weil
2011-02-23 19:12 ` Gregory Farnum
2011-02-23 19:23 ` Jim Schutt
2011-02-23 20:27 ` Gregory Farnum
2011-03-02 0:53 ` Sage Weil
2011-03-02 15:21 ` Jim Schutt [this message]
2011-03-02 17:10 ` Sage Weil
2011-03-02 20:54 ` Jim Schutt
2011-03-02 21:45 ` Sage Weil
2011-03-02 21:59 ` Jim Schutt
2011-03-02 22:57 ` Jim Schutt
2011-03-02 23:20 ` Gregory Farnum
2011-03-02 23:25 ` Jim Schutt
2011-03-02 23:33 ` Gregory Farnum
2011-03-03 2:26 ` Colin McCabe
2011-03-03 20:03 ` Jim Schutt
2011-03-03 20:47 ` Jim Schutt
2011-03-03 20:55 ` Yehuda Sadeh Weinraub
2011-03-03 21:45 ` Jim Schutt
2011-03-03 22:22 ` Sage Weil
2011-03-03 22:34 ` Jim Schutt
2011-03-03 21:53 ` Colin McCabe
2011-03-03 23:06 ` Jim Schutt
2011-03-03 23:30 ` Colin McCabe
2011-03-03 23:37 ` Jim Schutt
2011-03-03 5:03 ` Sage Weil
2011-03-03 16:35 ` Jim Schutt
2011-03-03 17:28 ` Jim Schutt
2011-03-03 18:04 ` Sage Weil
2011-03-03 18:42 ` Jim Schutt
2011-03-03 18:51 ` Sage Weil
2011-03-03 19:39 ` Jim Schutt
2011-04-08 16:23 ` Jim Schutt
2011-04-08 20:50 ` Sage Weil
2011-04-08 22:11 ` Jim Schutt
2011-04-08 23:10 ` Colin McCabe
2011-04-11 14:41 ` Jim Schutt
2011-04-11 16:25 ` Sage Weil
2011-04-11 20:14 ` Jim Schutt
2011-04-11 21:18 ` Jim Schutt
2011-04-11 23:23 ` Sage Weil
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1299079262.4750.19.camel@sale659.sandia.gov \
--to=jaschut@sandia.gov \
--cc=ceph-devel@vger.kernel.org \
--cc=gregory.farnum@dreamhost.com \
--cc=sage@newdream.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.