Re: Nightly "Kernel branch not found" errors

From: Gregory Farnum <greg@gregs42.com>
To: Sage Weil <sage@newdream.net>
Cc: Ilya Dryomov <idryomov@gmail.com>, Zack Cerza <zack@redhat.com>,
	Andrew Schoen <aschoen@redhat.com>, ceph-qa <ceph-qa@ceph.com>,
	"ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: Nightly "Kernel branch not found" errors
Date: Sun, 31 May 2015 15:28:57 -0700	[thread overview]
Message-ID: <CAC6JEv-v+jq6ba1TeyknBqQD9Lb2DiF5LNeh8pcxJt1sJFeJUg@mail.gmail.com> (raw)
In-Reply-To: <alpine.DEB.2.00.1505311518340.6462@cobra.newdream.net>

On Sun, May 31, 2015 at 3:19 PM, Sage Weil <sage@newdream.net> wrote:
> On Sun, 31 May 2015, Gregory Farnum wrote:
>> On Sun, May 31, 2015 at 1:40 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
>> > On Sun, May 31, 2015 at 10:54 PM, Gregory Farnum <greg@gregs42.com> wrote:
>> >> We are getting this error in what looks like everything that specifies
>> >> the testing kernel. (That turns out to be almost all of the FS tests
>> >> and a surprising number of the non-rados runs; e.g. rgw.) I've checked
>> >> that the testing branch of ceph-client.git still exists and when
>> >> looking at the teuthology git logs the recent ping-pong of commits on
>> >> kernel flavors et al stand out. Any ideas? :)
>> >
>> > Which lab is this in?  Latest rgw and fs in sepia look fine.
>> >
>> > Kernel task was broken the entire last week but in a different way -
>> > when scheduled with teuthology-suite it wouldn't install anything even
>> > if you told it to install e.g. testing.  I fixed that on Friday.
>> >
>> > Looks like this is coming from create_initial_config().  Could be an
>> > environment issue, like down gitbuilders or a problem with requests
>> > module?  Kernel branch is checked before the others so it may not have
>> > anything to do with it at all.
>>
>> It's happening across labs for tests that were supposed to be
>> scheduled starting on the 29 (at least, that I've noticed). I think
>> this issue is before they get into pulpito, which is why the latest fs
>> suite run there was scheduled on May 27. :(
>> Looking at the gitbuliders I do see that the CentOS6 testing branch is
>> red and rhel7 appears to be down...maybe we're checking on more of
>> them now and then failing when those don't appear? :/
>
> We're half-way through creating the centos7 kernel builder to replace the
> rhel ones so I expect things are broken on the rpm side.  Not sure if
> that is the root cause here, but we can probably wait for that to get
> fixed first before looking further

*None* of our FS tests are running while this problem persists (and
they're not alone). That's not the sort of thing we can wait on...
Maybe we have some gratuitous non-use of the testing kernel we can
remove (I'm not sure), but that's the sort of thing that needs to be
discussed across teams so we can deal with it proactively instead of
just finding out when the nightlies start failing.
-Greg