Nightly "Kernel branch not found" errors

Nightly "Kernel branch not found" errors

[Gregory Farnum]

We are getting this error in what looks like everything that specifies
the testing kernel. (That turns out to be almost all of the FS tests
and a surprising number of the non-rados runs; e.g. rgw.) I've checked
that the testing branch of ceph-client.git still exists and when
looking at the teuthology git logs the recent ping-pong of commits on
kernel flavors et al stand out. Any ideas? :)
-Greg

On Fri, May 29, 2015 at 11:10 PM,  <teuthology@ceph.com> wrote:
> Kernel branch 'testing' not found
> _______________________________________________
> Ceph-qa mailing list
> Ceph-qa@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com

Re: Nightly "Kernel branch not found" errors

[Ilya Dryomov]

On Sun, May 31, 2015 at 10:54 PM, Gregory Farnum <greg@gregs42.com> wrote:
> We are getting this error in what looks like everything that specifies
> the testing kernel. (That turns out to be almost all of the FS tests
> and a surprising number of the non-rados runs; e.g. rgw.) I've checked
> that the testing branch of ceph-client.git still exists and when
> looking at the teuthology git logs the recent ping-pong of commits on
> kernel flavors et al stand out. Any ideas? :)

Which lab is this in?  Latest rgw and fs in sepia look fine.

Kernel task was broken the entire last week but in a different way -
when scheduled with teuthology-suite it wouldn't install anything even
if you told it to install e.g. testing.  I fixed that on Friday.

Looks like this is coming from create_initial_config().  Could be an
environment issue, like down gitbuilders or a problem with requests
module?  Kernel branch is checked before the others so it may not have
anything to do with it at all.

In any case, I'm half asleep so somebody else should look into this ;)

Thanks,

                Ilya

Re: Nightly "Kernel branch not found" errors

[Gregory Farnum]

On Sun, May 31, 2015 at 1:40 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
> On Sun, May 31, 2015 at 10:54 PM, Gregory Farnum <greg@gregs42.com> wrote:
>> We are getting this error in what looks like everything that specifies
>> the testing kernel. (That turns out to be almost all of the FS tests
>> and a surprising number of the non-rados runs; e.g. rgw.) I've checked
>> that the testing branch of ceph-client.git still exists and when
>> looking at the teuthology git logs the recent ping-pong of commits on
>> kernel flavors et al stand out. Any ideas? :)
>
> Which lab is this in?  Latest rgw and fs in sepia look fine.
>
> Kernel task was broken the entire last week but in a different way -
> when scheduled with teuthology-suite it wouldn't install anything even
> if you told it to install e.g. testing.  I fixed that on Friday.
>
> Looks like this is coming from create_initial_config().  Could be an
> environment issue, like down gitbuilders or a problem with requests
> module?  Kernel branch is checked before the others so it may not have
> anything to do with it at all.

It's happening across labs for tests that were supposed to be
scheduled starting on the 29 (at least, that I've noticed). I think
this issue is before they get into pulpito, which is why the latest fs
suite run there was scheduled on May 27. :(
Looking at the gitbuliders I do see that the CentOS6 testing branch is
red and rhel7 appears to be down...maybe we're checking on more of
them now and then failing when those don't appear? :/
-Greg

Re: Nightly "Kernel branch not found" errors

[Sage Weil]

On Sun, 31 May 2015, Gregory Farnum wrote:
> On Sun, May 31, 2015 at 1:40 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
> > On Sun, May 31, 2015 at 10:54 PM, Gregory Farnum <greg@gregs42.com> wrote:
> >> We are getting this error in what looks like everything that specifies
> >> the testing kernel. (That turns out to be almost all of the FS tests
> >> and a surprising number of the non-rados runs; e.g. rgw.) I've checked
> >> that the testing branch of ceph-client.git still exists and when
> >> looking at the teuthology git logs the recent ping-pong of commits on
> >> kernel flavors et al stand out. Any ideas? :)
> >
> > Which lab is this in?  Latest rgw and fs in sepia look fine.
> >
> > Kernel task was broken the entire last week but in a different way -
> > when scheduled with teuthology-suite it wouldn't install anything even
> > if you told it to install e.g. testing.  I fixed that on Friday.
> >
> > Looks like this is coming from create_initial_config().  Could be an
> > environment issue, like down gitbuilders or a problem with requests
> > module?  Kernel branch is checked before the others so it may not have
> > anything to do with it at all.
> 
> It's happening across labs for tests that were supposed to be
> scheduled starting on the 29 (at least, that I've noticed). I think
> this issue is before they get into pulpito, which is why the latest fs
> suite run there was scheduled on May 27. :(
> Looking at the gitbuliders I do see that the CentOS6 testing branch is
> red and rhel7 appears to be down...maybe we're checking on more of
> them now and then failing when those don't appear? :/

We're half-way through creating the centos7 kernel builder to replace the 
rhel ones so I expect things are broken on the rpm side.  Not sure if 
that is the root cause here, but we can probably wait for that to get 
fixed first before looking further

s

Re: Nightly "Kernel branch not found" errors

[Gregory Farnum]

On Sun, May 31, 2015 at 3:19 PM, Sage Weil <sage@newdream.net> wrote:
> On Sun, 31 May 2015, Gregory Farnum wrote:
>> On Sun, May 31, 2015 at 1:40 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
>> > On Sun, May 31, 2015 at 10:54 PM, Gregory Farnum <greg@gregs42.com> wrote:
>> >> We are getting this error in what looks like everything that specifies
>> >> the testing kernel. (That turns out to be almost all of the FS tests
>> >> and a surprising number of the non-rados runs; e.g. rgw.) I've checked
>> >> that the testing branch of ceph-client.git still exists and when
>> >> looking at the teuthology git logs the recent ping-pong of commits on
>> >> kernel flavors et al stand out. Any ideas? :)
>> >
>> > Which lab is this in?  Latest rgw and fs in sepia look fine.
>> >
>> > Kernel task was broken the entire last week but in a different way -
>> > when scheduled with teuthology-suite it wouldn't install anything even
>> > if you told it to install e.g. testing.  I fixed that on Friday.
>> >
>> > Looks like this is coming from create_initial_config().  Could be an
>> > environment issue, like down gitbuilders or a problem with requests
>> > module?  Kernel branch is checked before the others so it may not have
>> > anything to do with it at all.
>>
>> It's happening across labs for tests that were supposed to be
>> scheduled starting on the 29 (at least, that I've noticed). I think
>> this issue is before they get into pulpito, which is why the latest fs
>> suite run there was scheduled on May 27. :(
>> Looking at the gitbuliders I do see that the CentOS6 testing branch is
>> red and rhel7 appears to be down...maybe we're checking on more of
>> them now and then failing when those don't appear? :/
>
> We're half-way through creating the centos7 kernel builder to replace the
> rhel ones so I expect things are broken on the rpm side.  Not sure if
> that is the root cause here, but we can probably wait for that to get
> fixed first before looking further

*None* of our FS tests are running while this problem persists (and
they're not alone). That's not the sort of thing we can wait on...
Maybe we have some gratuitous non-use of the testing kernel we can
remove (I'm not sure), but that's the sort of thing that needs to be
discussed across teams so we can deal with it proactively instead of
just finding out when the nightlies start failing.
-Greg

Re: Nightly "Kernel branch not found" errors

[Ilya Dryomov]

On Mon, Jun 1, 2015 at 1:19 AM, Sage Weil <sage@newdream.net> wrote:
> On Sun, 31 May 2015, Gregory Farnum wrote:
>> On Sun, May 31, 2015 at 1:40 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
>> > On Sun, May 31, 2015 at 10:54 PM, Gregory Farnum <greg@gregs42.com> wrote:
>> >> We are getting this error in what looks like everything that specifies
>> >> the testing kernel. (That turns out to be almost all of the FS tests
>> >> and a surprising number of the non-rados runs; e.g. rgw.) I've checked
>> >> that the testing branch of ceph-client.git still exists and when
>> >> looking at the teuthology git logs the recent ping-pong of commits on
>> >> kernel flavors et al stand out. Any ideas? :)
>> >
>> > Which lab is this in?  Latest rgw and fs in sepia look fine.
>> >
>> > Kernel task was broken the entire last week but in a different way -
>> > when scheduled with teuthology-suite it wouldn't install anything even
>> > if you told it to install e.g. testing.  I fixed that on Friday.
>> >
>> > Looks like this is coming from create_initial_config().  Could be an
>> > environment issue, like down gitbuilders or a problem with requests
>> > module?  Kernel branch is checked before the others so it may not have
>> > anything to do with it at all.
>>
>> It's happening across labs for tests that were supposed to be
>> scheduled starting on the 29 (at least, that I've noticed). I think
>> this issue is before they get into pulpito, which is why the latest fs
>> suite run there was scheduled on May 27. :(
>> Looking at the gitbuliders I do see that the CentOS6 testing branch is
>> red and rhel7 appears to be down...maybe we're checking on more of
>> them now and then failing when those don't appear? :/
>
> We're half-way through creating the centos7 kernel builder to replace the
> rhel ones so I expect things are broken on the rpm side.  Not sure if
> that is the root cause here, but we can probably wait for that to get
> fixed first before looking further

I think it's definitely related to this.  commit f2ce5e1ed3d4 ("Treat
RHEL as CentOS when scheduling") by Zack makes it so centos7
gitbuilders are poked in case distro wasn't specified explicitly, which
is how I assume all those runs are scheduled.  kernel centos6
gitbuilder is down and AFAICT the centos7 one doesn't exist.

Thanks,

                Ilya

Re: Nightly "Kernel branch not found" errors

[Ilya Dryomov]

On Mon, Jun 1, 2015 at 1:28 AM, Gregory Farnum <greg@gregs42.com> wrote:
> On Sun, May 31, 2015 at 3:19 PM, Sage Weil <sage@newdream.net> wrote:
>> On Sun, 31 May 2015, Gregory Farnum wrote:
>>> On Sun, May 31, 2015 at 1:40 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
>>> > On Sun, May 31, 2015 at 10:54 PM, Gregory Farnum <greg@gregs42.com> wrote:
>>> >> We are getting this error in what looks like everything that specifies
>>> >> the testing kernel. (That turns out to be almost all of the FS tests
>>> >> and a surprising number of the non-rados runs; e.g. rgw.) I've checked
>>> >> that the testing branch of ceph-client.git still exists and when
>>> >> looking at the teuthology git logs the recent ping-pong of commits on
>>> >> kernel flavors et al stand out. Any ideas? :)
>>> >
>>> > Which lab is this in?  Latest rgw and fs in sepia look fine.
>>> >
>>> > Kernel task was broken the entire last week but in a different way -
>>> > when scheduled with teuthology-suite it wouldn't install anything even
>>> > if you told it to install e.g. testing.  I fixed that on Friday.
>>> >
>>> > Looks like this is coming from create_initial_config().  Could be an
>>> > environment issue, like down gitbuilders or a problem with requests
>>> > module?  Kernel branch is checked before the others so it may not have
>>> > anything to do with it at all.
>>>
>>> It's happening across labs for tests that were supposed to be
>>> scheduled starting on the 29 (at least, that I've noticed). I think
>>> this issue is before they get into pulpito, which is why the latest fs
>>> suite run there was scheduled on May 27. :(
>>> Looking at the gitbuliders I do see that the CentOS6 testing branch is
>>> red and rhel7 appears to be down...maybe we're checking on more of
>>> them now and then failing when those don't appear? :/
>>
>> We're half-way through creating the centos7 kernel builder to replace the
>> rhel ones so I expect things are broken on the rpm side.  Not sure if
>> that is the root cause here, but we can probably wait for that to get
>> fixed first before looking further
>
> *None* of our FS tests are running while this problem persists (and
> they're not alone). That's not the sort of thing we can wait on...
> Maybe we have some gratuitous non-use of the testing kernel we can
> remove (I'm not sure), but that's the sort of thing that needs to be
> discussed across teams so we can deal with it proactively instead of
> just finding out when the nightlies start failing.

It's not the kernel to blame here, so unless there is a specific reason
let's not remove use of testing kernel.

What I think we need to do is to make sure that teuthology test suite
includes "teuthology-suite --dry-run ..." with "--kernel testing" at
the very least.  It would be good to also verify that testing kernel is
actually going to be installed by the kernel task to avoid last week's
breakage, but that's a matter of adding unit tests and is a different
issue.

Unless I'm missing something, the only problem here is really that the
commit I mentioned in another mail got pushed before the gitbuilder was
setup.

Thanks,

                Ilya