Re: cosd multi-second stalls cause "wrongly marked me down"

From: Colin McCabe <cmccabe@alumni.cmu.edu>
To: Jim Schutt <jaschut@sandia.gov>
Cc: Sage Weil <sage@newdream.net>,
	Gregory Farnum <gregory.farnum@dreamhost.com>,
	"ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: cosd multi-second stalls cause "wrongly marked me down"
Date: Wed, 2 Mar 2011 18:26:37 -0800	[thread overview]
Message-ID: <AANLkTikU_F_mRaYC6xE5V9OHaDbyjWJ7fnpVCXkz=tK=@mail.gmail.com> (raw)
In-Reply-To: <1299106628.4750.62.camel@sale659.sandia.gov>

Hi Jim,

We have seen this problem before. The usual suspects are the oom
killer (grep for "out of memory" in syslog).
Unfortunately, SIGKILL is uncatchable and that's what the OOM killer sends.

Another problem that can prevent core files from being generated is
bad ulimit -c settings or a bad setting for core_pattern and friends.
One problem I have a lot too is that the partition I'm writing core
files to fills up.

If none of that works, it's possible that someone is calling exit()
somewhere. You can attach a gdb to the process and put a breakpoint on
exit() to see if this is going on. There's a lot of "your foo is not
bar enough, I hate your config, exit(1)" type code that gets executed
while the daemon is starting up. It sounds like you should be past
that point, though.

Colin

On Wed, Mar 2, 2011 at 2:57 PM, Jim Schutt <jaschut@sandia.gov> wrote:
>
> On Wed, 2011-03-02 at 14:59 -0700, Jim Schutt wrote:
>>
>> On Wed, 2011-03-02 at 14:45 -0700, Sage Weil wrote:
>> > On Wed, 2 Mar 2011, Jim Schutt wrote:
>> > >
>> > > On Wed, 2011-03-02 at 10:10 -0700, Sage Weil wrote:
>> > > > > I'll see if I see the same signature with master,
>> > > > > and post logs.
>> > > >
>> > > > Thanks!  Keep us posted.
>> > >
>> > > Hmmm, I'm not having much luck with master (commit
>> > > 0fb5ef2ce92 + extra debugging) on a 96-osd filesystem;
>> > > lots of dead OSDs during startup.
>> >
>> > Commit c916905a8a14029653aae45f0a9fb6c9b4c39e05 (master) should fix this.
>>
>> I try it out, thanks!
>
> I don't get any more core files with master commit 67355779ecc.
> Now my cosds just die - no stack trace in the log, no core
> file, nothing in syslog or dmesg ...
>
> I'm not sure how to track down what's happening here...
>
> -- Jim
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html