* 12.2.3 QE Luminous validation status
@ 2018-02-13 21:36 Yuri Weinstein
2018-02-13 21:41 ` Casey Bodley
` (5 more replies)
0 siblings, 6 replies; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-13 21:36 UTC (permalink / raw)
To: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, Jeff Layton,
ceph-qe-team, Deza, Alfredo, Andrew Schoen
Details of this release summarized here
http://tracker.ceph.com/issues/22665#note-4
The following suites included:
rados
rgw
rbd
krbd
fs
kcephfs
multimds
knfs
hadoop - EXCLUDED
samba - EXCLUDED
ceph-deploy
ceph-disk
upgrade/client-upgrade-hammer (luminous)
upgrade/client-upgrade-kraken (luminous)
upgrade/client-upgrade-jewel (luminous)
upgrade/jewel-x (luminous)
upgrade/kraken-x (luminous)
upgrade/luminous-x (master) - EXCLUDED
powercycle
ceph-ansible
ceph-volume
(please speak up if something is missing)
Please see all details in the ticket and add comments in the tracker.
Seeking approval from the dev leads.
Issues:
rados - passed, Josh pls confirm approval.
rgw - Casey, Abhishek - do we want to ad//merge
https://github.com/ceph/ceph/pull/20407 ?
rbd, krbd - approved by Jason
fs, kcephfs, multimds - approved by Patrick
knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
ceph-deploy - approved by Vasu
upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
into this release ?
upgrade/kraken-x (luminous) - some jobs still rerunning
upgrade/luminous-x (master) - Sage you to exclude this suite for now?
ceph-volume - pending approval from Alfredo, Andrew
Pls reply
Thx
YuriW
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
@ 2018-02-13 21:41 ` Casey Bodley
2018-02-13 21:59 ` Jeff Layton
2018-02-13 22:42 ` Nathan Cutler
` (4 subsequent siblings)
5 siblings, 1 reply; 15+ messages in thread
From: Casey Bodley @ 2018-02-13 21:41 UTC (permalink / raw)
To: Yuri Weinstein, Sage Weil, Durgin, Josh, Dillaman, Jason,
Sadeh-Weinraub, Yehuda, John Spray, Karol Mroz, Patrick Donnelly,
Development, Ceph, Lekshmanan, Abhishek, Nathan Cutler,
Ilya Dryomov, Jeff Layton, ceph-qe-team, Deza, Alfredo,
Andrew Schoen
On 02/13/2018 04:36 PM, Yuri Weinstein wrote:
> Details of this release summarized here
> http://tracker.ceph.com/issues/22665#note-4
>
> The following suites included:
>
> rados
> rgw
> rbd
> krbd
> fs
> kcephfs
> multimds
> knfs
> hadoop - EXCLUDED
> samba - EXCLUDED
> ceph-deploy
> ceph-disk
> upgrade/client-upgrade-hammer (luminous)
> upgrade/client-upgrade-kraken (luminous)
> upgrade/client-upgrade-jewel (luminous)
> upgrade/jewel-x (luminous)
> upgrade/kraken-x (luminous)
> upgrade/luminous-x (master) - EXCLUDED
> powercycle
> ceph-ansible
> ceph-volume
> (please speak up if something is missing)
>
> Please see all details in the ticket and add comments in the tracker.
>
> Seeking approval from the dev leads.
>
> Issues:
>
> rados - passed, Josh pls confirm approval.
>
> rgw - Casey, Abhishek - do we want to ad//merge
> https://github.com/ceph/ceph/pull/20407 ?
yes please - that one was identified as a qa suite fix for a multisite
test failure in the last round of testing
>
> rbd, krbd - approved by Jason
>
> fs, kcephfs, multimds - approved by Patrick
>
> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>
> ceph-deploy - approved by Vasu
>
> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> into this release ?
>
> upgrade/kraken-x (luminous) - some jobs still rerunning
>
> upgrade/luminous-x (master) - Sage you to exclude this suite for now?
>
> ceph-volume - pending approval from Alfredo, Andrew
>
> Pls reply
>
> Thx
> YuriW
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-13 21:41 ` Casey Bodley
@ 2018-02-13 21:59 ` Jeff Layton
0 siblings, 0 replies; 15+ messages in thread
From: Jeff Layton @ 2018-02-13 21:59 UTC (permalink / raw)
To: Casey Bodley, Yuri Weinstein, Sage Weil, Durgin, Josh, Dillaman,
Jason, Sadeh-Weinraub, Yehuda, John Spray, Karol Mroz,
Patrick Donnelly, Development, Ceph, Lekshmanan, Abhishek,
Nathan Cutler, Ilya Dryomov, ceph-qe-team, Deza, Alfredo,
Andrew Schoen
On Tue, 2018-02-13 at 16:41 -0500, Casey Bodley wrote:
>
> On 02/13/2018 04:36 PM, Yuri Weinstein wrote:
> > Details of this release summarized here
> > http://tracker.ceph.com/issues/22665#note-4
> >
> > The following suites included:
> >
> > rados
> > rgw
> > rbd
> > krbd
> > fs
> > kcephfs
> > multimds
> > knfs
> > hadoop - EXCLUDED
> > samba - EXCLUDED
> > ceph-deploy
> > ceph-disk
> > upgrade/client-upgrade-hammer (luminous)
> > upgrade/client-upgrade-kraken (luminous)
> > upgrade/client-upgrade-jewel (luminous)
> > upgrade/jewel-x (luminous)
> > upgrade/kraken-x (luminous)
> > upgrade/luminous-x (master) - EXCLUDED
> > powercycle
> > ceph-ansible
> > ceph-volume
> > (please speak up if something is missing)
> >
> > Please see all details in the ticket and add comments in the tracker.
> >
> > Seeking approval from the dev leads.
> >
> > Issues:
> >
> > rados - passed, Josh pls confirm approval.
> >
> > rgw - Casey, Abhishek - do we want to ad//merge
> > https://github.com/ceph/ceph/pull/20407 ?
>
> yes please - that one was identified as a qa suite fix for a multisite
> test failure in the last round of testing
>
> >
> > rbd, krbd - approved by Jason
> >
> > fs, kcephfs, multimds - approved by Patrick
> >
> > knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
> >
So, we have two recent test runs, both with 2 failures:
The first run had one failure due to OSD_DOWN being in the logs, and
another due to what looks like a softlockup in the kernel on one of the
nodes. No stack trace to go with the softlockup, so I have no idea what
went wrong there. The second run just shows two OSD_DOWN failures. For
now I don't see anything that looks directly related to knfsd on either
of these.
Yuri looked and saw that we have a lot of failing runs on this suite so
it may be that this is just a broken test. I'll look more closely
tomorrow and see if I can figure out whether that's the case or whether
there is a real bug here.
Cheers,
> > ceph-deploy - approved by Vasu
> >
> > upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> > into this release ?
> >
> > upgrade/kraken-x (luminous) - some jobs still rerunning
> >
> > upgrade/luminous-x (master) - Sage you to exclude this suite for now?
> >
> > ceph-volume - pending approval from Alfredo, Andrew
> >
> > Pls reply
> >
> > Thx
> > YuriW
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
Jeff Layton <jlayton@redhat.com>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
2018-02-13 21:41 ` Casey Bodley
@ 2018-02-13 22:42 ` Nathan Cutler
2018-02-14 16:52 ` Yuri Weinstein
2018-02-13 23:14 ` Josh Durgin
` (3 subsequent siblings)
5 siblings, 1 reply; 15+ messages in thread
From: Nathan Cutler @ 2018-02-13 22:42 UTC (permalink / raw)
To: Yuri Weinstein, Sage Weil, Durgin, Josh, Dillaman, Jason,
Sadeh-Weinraub, Yehuda, John Spray, Karol Mroz, Patrick Donnelly,
Development, Ceph, Lekshmanan, Abhishek, Ilya Dryomov,
Jeff Layton, ceph-qe-team, Deza, Alfredo, Andrew Schoen
Hi Yuri,
> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> into this release ?
The fix is limited to the kraken branch and was just merged. Can you
repeat this run please?
Nathan
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
2018-02-13 21:41 ` Casey Bodley
2018-02-13 22:42 ` Nathan Cutler
@ 2018-02-13 23:14 ` Josh Durgin
2018-02-14 11:00 ` Abhishek
` (2 subsequent siblings)
5 siblings, 0 replies; 15+ messages in thread
From: Josh Durgin @ 2018-02-13 23:14 UTC (permalink / raw)
To: Yuri Weinstein
Cc: Sage Weil, Jason Dillaman, Yehuda Sadeh-Weinraub, John Spray,
Karol Mroz, Patrick Donnelly, Ceph Development,
Abhishek Lekshmanan, Nathan Cutler, Ilya Dryomov, Jeff Layton,
ceph-qe-team, Alfredo Deza, Andrew Schoen
> rados - passed, Josh pls confirm approval.
Yup, look good to me.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
` (2 preceding siblings ...)
2018-02-13 23:14 ` Josh Durgin
@ 2018-02-14 11:00 ` Abhishek
2018-02-14 16:39 ` Yuri Weinstein
2018-02-14 18:15 ` Jeff Layton
5 siblings, 0 replies; 15+ messages in thread
From: Abhishek @ 2018-02-14 11:00 UTC (permalink / raw)
To: Yuri Weinstein
Cc: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, Jeff Layton,
ceph-qe-team, Deza, Alfredo, Andrew Schoen, ceph-devel-owner
On 2018-02-13 22:36, Yuri Weinstein wrote:
> Details of this release summarized here
> http://tracker.ceph.com/issues/22665#note-4
>
> The following suites included:
>
> rados
> rgw
> rbd
> krbd
> fs
> kcephfs
> multimds
> knfs
> hadoop - EXCLUDED
> samba - EXCLUDED
> ceph-deploy
> ceph-disk
> upgrade/client-upgrade-hammer (luminous)
> upgrade/client-upgrade-kraken (luminous)
> upgrade/client-upgrade-jewel (luminous)
> upgrade/jewel-x (luminous)
> upgrade/kraken-x (luminous)
> upgrade/luminous-x (master) - EXCLUDED
> powercycle
> ceph-ansible
> ceph-volume
> (please speak up if something is missing)
>
> Please see all details in the ticket and add comments in the tracker.
>
> Seeking approval from the dev leads.
>
> Issues:
>
> rados - passed, Josh pls confirm approval.
>
> rgw - Casey, Abhishek - do we want to ad//merge
> https://github.com/ceph/ceph/pull/20407 ?
>
> rbd, krbd - approved by Jason
>
> fs, kcephfs, multimds - approved by Patrick
>
> knfs - pending review/approval from Jeff
> (http://tracker.ceph.com/issues/22995)
>
> ceph-deploy - approved by Vasu
>
> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> into this release ?
>
> upgrade/kraken-x (luminous) - some jobs still rerunning
>
> upgrade/luminous-x (master) - Sage you to exclude this suite for now?
>
> ceph-volume - pending approval from Alfredo, Andrew
Alfredo mentioned he's working on a critical fix that needs to go in for
12.2.3, waiting for merge + approval
PS. for some reason replies to this thread from my email always failed.
Hopefully we don't get multiple copies of the same message.
>
> Pls reply
>
> Thx
> YuriW
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
` (3 preceding siblings ...)
2018-02-14 11:00 ` Abhishek
@ 2018-02-14 16:39 ` Yuri Weinstein
2018-02-14 16:46 ` Alfredo Deza
2018-02-14 18:15 ` Jeff Layton
5 siblings, 1 reply; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-14 16:39 UTC (permalink / raw)
To: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, Jeff Layton,
ceph-qe-team, Deza, Alfredo, Andrew Schoen
re: upgrade/kraken-x we need kraken fix merged for
http://tracker.ceph.com/issues/22740
Jason, will you approve or merge that PR for 12.2.3?
On Tue, Feb 13, 2018 at 1:36 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Details of this release summarized here
> http://tracker.ceph.com/issues/22665#note-4
>
> The following suites included:
>
> rados
> rgw
> rbd
> krbd
> fs
> kcephfs
> multimds
> knfs
> hadoop - EXCLUDED
> samba - EXCLUDED
> ceph-deploy
> ceph-disk
> upgrade/client-upgrade-hammer (luminous)
> upgrade/client-upgrade-kraken (luminous)
> upgrade/client-upgrade-jewel (luminous)
> upgrade/jewel-x (luminous)
> upgrade/kraken-x (luminous)
> upgrade/luminous-x (master) - EXCLUDED
> powercycle
> ceph-ansible
> ceph-volume
> (please speak up if something is missing)
>
> Please see all details in the ticket and add comments in the tracker.
>
> Seeking approval from the dev leads.
>
> Issues:
>
> rados - passed, Josh pls confirm approval.
>
> rgw - Casey, Abhishek - do we want to ad//merge
> https://github.com/ceph/ceph/pull/20407 ?
>
> rbd, krbd - approved by Jason
>
> fs, kcephfs, multimds - approved by Patrick
>
> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>
> ceph-deploy - approved by Vasu
>
> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> into this release ?
>
> upgrade/kraken-x (luminous) - some jobs still rerunning
>
> upgrade/luminous-x (master) - Sage you to exclude this suite for now?
>
> ceph-volume - pending approval from Alfredo, Andrew
>
> Pls reply
>
> Thx
> YuriW
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-14 16:39 ` Yuri Weinstein
@ 2018-02-14 16:46 ` Alfredo Deza
0 siblings, 0 replies; 15+ messages in thread
From: Alfredo Deza @ 2018-02-14 16:46 UTC (permalink / raw)
To: Yuri Weinstein
Cc: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, Jeff Layton,
ceph-qe-team, Andrew Schoen
On Wed, Feb 14, 2018 at 11:39 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
> re: upgrade/kraken-x we need kraken fix merged for
> http://tracker.ceph.com/issues/22740
>
> Jason, will you approve or merge that PR for 12.2.3?
>
> On Tue, Feb 13, 2018 at 1:36 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
>> Details of this release summarized here
>> http://tracker.ceph.com/issues/22665#note-4
>>
>> The following suites included:
>>
>> rados
>> rgw
>> rbd
>> krbd
>> fs
>> kcephfs
>> multimds
>> knfs
>> hadoop - EXCLUDED
>> samba - EXCLUDED
>> ceph-deploy
>> ceph-disk
>> upgrade/client-upgrade-hammer (luminous)
>> upgrade/client-upgrade-kraken (luminous)
>> upgrade/client-upgrade-jewel (luminous)
>> upgrade/jewel-x (luminous)
>> upgrade/kraken-x (luminous)
>> upgrade/luminous-x (master) - EXCLUDED
>> powercycle
>> ceph-ansible
>> ceph-volume
>> (please speak up if something is missing)
>>
>> Please see all details in the ticket and add comments in the tracker.
>>
>> Seeking approval from the dev leads.
>>
>> Issues:
>>
>> rados - passed, Josh pls confirm approval.
>>
>> rgw - Casey, Abhishek - do we want to ad//merge
>> https://github.com/ceph/ceph/pull/20407 ?
>>
>> rbd, krbd - approved by Jason
>>
>> fs, kcephfs, multimds - approved by Patrick
>>
>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>
>> ceph-deploy - approved by Vasu
>>
>> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
>> into this release ?
>>
>> upgrade/kraken-x (luminous) - some jobs still rerunning
>>
>> upgrade/luminous-x (master) - Sage you to exclude this suite for now?
>>
>> ceph-volume - pending approval from Alfredo, Andrew
ceph-volume is good to go, both PRs got everything passing and merged today
https://github.com/ceph/ceph/pull/20429
https://github.com/ceph/ceph/pull/20438
>>
>> Pls reply
>>
>> Thx
>> YuriW
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-13 22:42 ` Nathan Cutler
@ 2018-02-14 16:52 ` Yuri Weinstein
0 siblings, 0 replies; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-14 16:52 UTC (permalink / raw)
To: Nathan Cutler
Cc: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Ilya Dryomov, Jeff Layton, ceph-qe-team,
Deza, Alfredo, Andrew Schoen
All green, great work Nathan !
http://pulpito.ceph.com/yuriw-2018-02-14_16:28:05-upgrade:client-upgrade-kraken-luminous-distro-basic-smithi/
On Tue, Feb 13, 2018 at 2:42 PM, Nathan Cutler <ncutler@suse.cz> wrote:
> Hi Yuri,
>
>> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
>> into this release ?
>
>
> The fix is limited to the kraken branch and was just merged. Can you repeat
> this run please?
>
> Nathan
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
` (4 preceding siblings ...)
2018-02-14 16:39 ` Yuri Weinstein
@ 2018-02-14 18:15 ` Jeff Layton
2018-02-15 16:55 ` Yuri Weinstein
5 siblings, 1 reply; 15+ messages in thread
From: Jeff Layton @ 2018-02-14 18:15 UTC (permalink / raw)
To: Yuri Weinstein, Sage Weil, Durgin, Josh, Dillaman, Jason,
Sadeh-Weinraub, Yehuda, John Spray, Karol Mroz, Patrick Donnelly,
Development, Ceph, Lekshmanan, Abhishek, Nathan Cutler,
Ilya Dryomov, ceph-qe-team, Deza, Alfredo, Andrew Schoen
On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>
> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>
Ok, I've flogged it about as far as I can. The problem is that we see
OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
it just goes unresponsive to pings for a little while.
That's about as far as I can carry it -- at this point it'd be nice to
have someone with more familiarity with the OSD code take a look and see
what they can tell.
FWIW, these tests seem to routinely fail on smithi, but we have at least
one run on OVH hosts that passed. This leads be to believe that it's
something specific to smithi: flaky hw maybe? or possibly load related?
--
Jeff Layton <jlayton@redhat.com>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-14 18:15 ` Jeff Layton
@ 2018-02-15 16:55 ` Yuri Weinstein
2018-02-15 16:58 ` Jason Dillaman
0 siblings, 1 reply; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-15 16:55 UTC (permalink / raw)
To: Jeff Layton
Cc: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
Deza, Alfredo, Andrew Schoen
Outstanding issues:
upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
Jason, Sage approve
knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
Abhishek, Alfredo - assuming all agreed and Sage approves, we can
publish 12.2.3 any time.
Thx
YuriW
On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>
>>
>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>
>
> Ok, I've flogged it about as far as I can. The problem is that we see
> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
> it just goes unresponsive to pings for a little while.
>
> That's about as far as I can carry it -- at this point it'd be nice to
> have someone with more familiarity with the OSD code take a look and see
> what they can tell.
>
> FWIW, these tests seem to routinely fail on smithi, but we have at least
> one run on OVH hosts that passed. This leads be to believe that it's
> something specific to smithi: flaky hw maybe? or possibly load related?
>
> --
> Jeff Layton <jlayton@redhat.com>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-15 16:55 ` Yuri Weinstein
@ 2018-02-15 16:58 ` Jason Dillaman
2018-02-15 17:33 ` Yuri Weinstein
0 siblings, 1 reply; 15+ messages in thread
From: Jason Dillaman @ 2018-02-15 16:58 UTC (permalink / raw)
To: Yuri Weinstein
Cc: Jeff Layton, Sage Weil, Durgin, Josh, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
Deza, Alfredo, Andrew Schoen
Yuri,
On Thu, Feb 15, 2018 at 11:55 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Outstanding issues:
>
> upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
> Jason, Sage approve
Do you have a link to the associated failed test run? I would have
expected this to have been fixed indirectly by
https://github.com/ceph/ceph/pull/20053 as a workaround for the broken
kraken builders.
> knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
>
> Abhishek, Alfredo - assuming all agreed and Sage approves, we can
> publish 12.2.3 any time.
>
> Thx
> YuriW
>
> On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
>> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>>
>>>
>>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>>
>>
>> Ok, I've flogged it about as far as I can. The problem is that we see
>> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
>> it just goes unresponsive to pings for a little while.
>>
>> That's about as far as I can carry it -- at this point it'd be nice to
>> have someone with more familiarity with the OSD code take a look and see
>> what they can tell.
>>
>> FWIW, these tests seem to routinely fail on smithi, but we have at least
>> one run on OVH hosts that passed. This leads be to believe that it's
>> something specific to smithi: flaky hw maybe? or possibly load related?
>>
>> --
>> Jeff Layton <jlayton@redhat.com>
--
Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-15 16:58 ` Jason Dillaman
@ 2018-02-15 17:33 ` Yuri Weinstein
2018-02-15 17:57 ` Jason Dillaman
0 siblings, 1 reply; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-15 17:33 UTC (permalink / raw)
To: Dillaman, Jason
Cc: Jeff Layton, Sage Weil, Durgin, Josh, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
Deza, Alfredo, Andrew Schoen
Jason
http://pulpito.ceph.com/yuriw-2018-02-13_21:14:53-upgrade:kraken-x-luminous-distro-basic-smithi/
On Thu, Feb 15, 2018 at 8:58 AM, Jason Dillaman <jdillama@redhat.com> wrote:
> Yuri,
>
> On Thu, Feb 15, 2018 at 11:55 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>> Outstanding issues:
>>
>> upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
>> Jason, Sage approve
>
> Do you have a link to the associated failed test run? I would have
> expected this to have been fixed indirectly by
> https://github.com/ceph/ceph/pull/20053 as a workaround for the broken
> kraken builders.
>
>> knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
>>
>> Abhishek, Alfredo - assuming all agreed and Sage approves, we can
>> publish 12.2.3 any time.
>>
>> Thx
>> YuriW
>>
>> On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
>>> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>>>
>>>>
>>>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>>>
>>>
>>> Ok, I've flogged it about as far as I can. The problem is that we see
>>> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
>>> it just goes unresponsive to pings for a little while.
>>>
>>> That's about as far as I can carry it -- at this point it'd be nice to
>>> have someone with more familiarity with the OSD code take a look and see
>>> what they can tell.
>>>
>>> FWIW, these tests seem to routinely fail on smithi, but we have at least
>>> one run on OVH hosts that passed. This leads be to believe that it's
>>> something specific to smithi: flaky hw maybe? or possibly load related?
>>>
>>> --
>>> Jeff Layton <jlayton@redhat.com>
>
>
>
> --
> Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-15 17:33 ` Yuri Weinstein
@ 2018-02-15 17:57 ` Jason Dillaman
2018-02-16 1:22 ` Yuri Weinstein
0 siblings, 1 reply; 15+ messages in thread
From: Jason Dillaman @ 2018-02-15 17:57 UTC (permalink / raw)
To: Yuri Weinstein
Cc: Jeff Layton, Sage Weil, Durgin, Josh, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
Deza, Alfredo, Andrew Schoen
Thanks -- just looks like that test needs the same blacklist applied
that PR 20053 performed on the other kraken-x upgrade test (or we just
fix the broken builders for kraken so the broken test can be yanked).
On Thu, Feb 15, 2018 at 12:33 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Jason
>
> http://pulpito.ceph.com/yuriw-2018-02-13_21:14:53-upgrade:kraken-x-luminous-distro-basic-smithi/
>
>
> On Thu, Feb 15, 2018 at 8:58 AM, Jason Dillaman <jdillama@redhat.com> wrote:
>> Yuri,
>>
>> On Thu, Feb 15, 2018 at 11:55 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>>> Outstanding issues:
>>>
>>> upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
>>> Jason, Sage approve
>>
>> Do you have a link to the associated failed test run? I would have
>> expected this to have been fixed indirectly by
>> https://github.com/ceph/ceph/pull/20053 as a workaround for the broken
>> kraken builders.
>>
>>> knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
>>>
>>> Abhishek, Alfredo - assuming all agreed and Sage approves, we can
>>> publish 12.2.3 any time.
>>>
>>> Thx
>>> YuriW
>>>
>>> On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
>>>> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>>>>
>>>>>
>>>>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>>>>
>>>>
>>>> Ok, I've flogged it about as far as I can. The problem is that we see
>>>> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
>>>> it just goes unresponsive to pings for a little while.
>>>>
>>>> That's about as far as I can carry it -- at this point it'd be nice to
>>>> have someone with more familiarity with the OSD code take a look and see
>>>> what they can tell.
>>>>
>>>> FWIW, these tests seem to routinely fail on smithi, but we have at least
>>>> one run on OVH hosts that passed. This leads be to believe that it's
>>>> something specific to smithi: flaky hw maybe? or possibly load related?
>>>>
>>>> --
>>>> Jeff Layton <jlayton@redhat.com>
>>
>>
>>
>> --
>> Jason
--
Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: 12.2.3 QE Luminous validation status
2018-02-15 17:57 ` Jason Dillaman
@ 2018-02-16 1:22 ` Yuri Weinstein
0 siblings, 0 replies; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-16 1:22 UTC (permalink / raw)
To: Dillaman, Jason
Cc: Jeff Layton, Sage Weil, Durgin, Josh, Sadeh-Weinraub, Yehuda,
John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
Deza, Alfredo, Andrew Schoen
Outstanding issues remaining:
upgrade/kraken-x => http://tracker.ceph.com/issues/22740 fixed by
https://github.com/ceph/ceph/pull/20451 (thanks for clues Jason!),
Abhishek assigned to you for merge
knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
On Thu, Feb 15, 2018 at 9:57 AM, Jason Dillaman <jdillama@redhat.com> wrote:
> Thanks -- just looks like that test needs the same blacklist applied
> that PR 20053 performed on the other kraken-x upgrade test (or we just
> fix the broken builders for kraken so the broken test can be yanked).
>
> On Thu, Feb 15, 2018 at 12:33 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
>> Jason
>>
>> http://pulpito.ceph.com/yuriw-2018-02-13_21:14:53-upgrade:kraken-x-luminous-distro-basic-smithi/
>>
>>
>> On Thu, Feb 15, 2018 at 8:58 AM, Jason Dillaman <jdillama@redhat.com> wrote:
>>> Yuri,
>>>
>>> On Thu, Feb 15, 2018 at 11:55 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>>>> Outstanding issues:
>>>>
>>>> upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
>>>> Jason, Sage approve
>>>
>>> Do you have a link to the associated failed test run? I would have
>>> expected this to have been fixed indirectly by
>>> https://github.com/ceph/ceph/pull/20053 as a workaround for the broken
>>> kraken builders.
>>>
>>>> knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
>>>>
>>>> Abhishek, Alfredo - assuming all agreed and Sage approves, we can
>>>> publish 12.2.3 any time.
>>>>
>>>> Thx
>>>> YuriW
>>>>
>>>> On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
>>>>> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>>>>>
>>>>>>
>>>>>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>>>>>
>>>>>
>>>>> Ok, I've flogged it about as far as I can. The problem is that we see
>>>>> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
>>>>> it just goes unresponsive to pings for a little while.
>>>>>
>>>>> That's about as far as I can carry it -- at this point it'd be nice to
>>>>> have someone with more familiarity with the OSD code take a look and see
>>>>> what they can tell.
>>>>>
>>>>> FWIW, these tests seem to routinely fail on smithi, but we have at least
>>>>> one run on OVH hosts that passed. This leads be to believe that it's
>>>>> something specific to smithi: flaky hw maybe? or possibly load related?
>>>>>
>>>>> --
>>>>> Jeff Layton <jlayton@redhat.com>
>>>
>>>
>>>
>>> --
>>> Jason
>
>
>
> --
> Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2018-02-16 1:22 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
2018-02-13 21:41 ` Casey Bodley
2018-02-13 21:59 ` Jeff Layton
2018-02-13 22:42 ` Nathan Cutler
2018-02-14 16:52 ` Yuri Weinstein
2018-02-13 23:14 ` Josh Durgin
2018-02-14 11:00 ` Abhishek
2018-02-14 16:39 ` Yuri Weinstein
2018-02-14 16:46 ` Alfredo Deza
2018-02-14 18:15 ` Jeff Layton
2018-02-15 16:55 ` Yuri Weinstein
2018-02-15 16:58 ` Jason Dillaman
2018-02-15 17:33 ` Yuri Weinstein
2018-02-15 17:57 ` Jason Dillaman
2018-02-16 1:22 ` Yuri Weinstein
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.