All of lore.kernel.org
 help / color / mirror / Atom feed
* 12.2.3 QE Luminous validation status
@ 2018-02-13 21:36 Yuri Weinstein
  2018-02-13 21:41 ` Casey Bodley
                   ` (5 more replies)
  0 siblings, 6 replies; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-13 21:36 UTC (permalink / raw)
  To: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, Jeff Layton,
	ceph-qe-team, Deza, Alfredo, Andrew Schoen

Details of this release summarized here
http://tracker.ceph.com/issues/22665#note-4

The following suites included:

rados
rgw
rbd
krbd
fs
kcephfs
multimds
knfs
hadoop - EXCLUDED
samba - EXCLUDED
ceph-deploy
ceph-disk
upgrade/client-upgrade-hammer (luminous)
upgrade/client-upgrade-kraken (luminous)
upgrade/client-upgrade-jewel (luminous)
upgrade/jewel-x (luminous)
upgrade/kraken-x (luminous)
upgrade/luminous-x (master) - EXCLUDED
powercycle
ceph-ansible
ceph-volume
(please speak up if something is missing)

Please see all details in the ticket and add comments in the tracker.

Seeking approval from the dev leads.

Issues:

rados - passed, Josh pls confirm approval.

rgw - Casey, Abhishek - do we want to ad//merge
https://github.com/ceph/ceph/pull/20407 ?

rbd, krbd - approved by Jason

fs, kcephfs, multimds - approved by Patrick

knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)

ceph-deploy - approved by Vasu

upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
into this release ?

upgrade/kraken-x (luminous) - some jobs still rerunning

upgrade/luminous-x (master) - Sage you to exclude this suite for now?

ceph-volume - pending approval from  Alfredo, Andrew

Pls reply

Thx
YuriW

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
@ 2018-02-13 21:41 ` Casey Bodley
  2018-02-13 21:59   ` Jeff Layton
  2018-02-13 22:42 ` Nathan Cutler
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 15+ messages in thread
From: Casey Bodley @ 2018-02-13 21:41 UTC (permalink / raw)
  To: Yuri Weinstein, Sage Weil, Durgin, Josh, Dillaman, Jason,
	Sadeh-Weinraub, Yehuda, John Spray, Karol Mroz, Patrick Donnelly,
	Development, Ceph, Lekshmanan, Abhishek, Nathan Cutler,
	Ilya Dryomov, Jeff Layton, ceph-qe-team, Deza, Alfredo,
	Andrew Schoen



On 02/13/2018 04:36 PM, Yuri Weinstein wrote:
> Details of this release summarized here
> http://tracker.ceph.com/issues/22665#note-4
>
> The following suites included:
>
> rados
> rgw
> rbd
> krbd
> fs
> kcephfs
> multimds
> knfs
> hadoop - EXCLUDED
> samba - EXCLUDED
> ceph-deploy
> ceph-disk
> upgrade/client-upgrade-hammer (luminous)
> upgrade/client-upgrade-kraken (luminous)
> upgrade/client-upgrade-jewel (luminous)
> upgrade/jewel-x (luminous)
> upgrade/kraken-x (luminous)
> upgrade/luminous-x (master) - EXCLUDED
> powercycle
> ceph-ansible
> ceph-volume
> (please speak up if something is missing)
>
> Please see all details in the ticket and add comments in the tracker.
>
> Seeking approval from the dev leads.
>
> Issues:
>
> rados - passed, Josh pls confirm approval.
>
> rgw - Casey, Abhishek - do we want to ad//merge
> https://github.com/ceph/ceph/pull/20407 ?

yes please - that one was identified as a qa suite fix for a multisite 
test failure in the last round of testing

>
> rbd, krbd - approved by Jason
>
> fs, kcephfs, multimds - approved by Patrick
>
> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>
> ceph-deploy - approved by Vasu
>
> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> into this release ?
>
> upgrade/kraken-x (luminous) - some jobs still rerunning
>
> upgrade/luminous-x (master) - Sage you to exclude this suite for now?
>
> ceph-volume - pending approval from  Alfredo, Andrew
>
> Pls reply
>
> Thx
> YuriW
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-13 21:41 ` Casey Bodley
@ 2018-02-13 21:59   ` Jeff Layton
  0 siblings, 0 replies; 15+ messages in thread
From: Jeff Layton @ 2018-02-13 21:59 UTC (permalink / raw)
  To: Casey Bodley, Yuri Weinstein, Sage Weil, Durgin, Josh, Dillaman,
	Jason, Sadeh-Weinraub, Yehuda, John Spray, Karol Mroz,
	Patrick Donnelly, Development, Ceph, Lekshmanan, Abhishek,
	Nathan Cutler, Ilya Dryomov, ceph-qe-team, Deza, Alfredo,
	Andrew Schoen

On Tue, 2018-02-13 at 16:41 -0500, Casey Bodley wrote:
> 
> On 02/13/2018 04:36 PM, Yuri Weinstein wrote:
> > Details of this release summarized here
> > http://tracker.ceph.com/issues/22665#note-4
> > 
> > The following suites included:
> > 
> > rados
> > rgw
> > rbd
> > krbd
> > fs
> > kcephfs
> > multimds
> > knfs
> > hadoop - EXCLUDED
> > samba - EXCLUDED
> > ceph-deploy
> > ceph-disk
> > upgrade/client-upgrade-hammer (luminous)
> > upgrade/client-upgrade-kraken (luminous)
> > upgrade/client-upgrade-jewel (luminous)
> > upgrade/jewel-x (luminous)
> > upgrade/kraken-x (luminous)
> > upgrade/luminous-x (master) - EXCLUDED
> > powercycle
> > ceph-ansible
> > ceph-volume
> > (please speak up if something is missing)
> > 
> > Please see all details in the ticket and add comments in the tracker.
> > 
> > Seeking approval from the dev leads.
> > 
> > Issues:
> > 
> > rados - passed, Josh pls confirm approval.
> > 
> > rgw - Casey, Abhishek - do we want to ad//merge
> > https://github.com/ceph/ceph/pull/20407 ?
> 
> yes please - that one was identified as a qa suite fix for a multisite 
> test failure in the last round of testing
> 
> > 
> > rbd, krbd - approved by Jason
> > 
> > fs, kcephfs, multimds - approved by Patrick
> > 
> > knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
> > 

So, we have two recent test runs, both with 2 failures:

The first run had one failure due to OSD_DOWN being in the logs, and
another due to what looks like a softlockup in the kernel on one of the
nodes. No stack trace to go with the softlockup, so I have no idea what
went wrong there. The second run just shows two OSD_DOWN failures. For
now I don't see anything that looks directly related to knfsd on either
of these.

Yuri looked and saw that we have a lot of failing runs on this suite so
it may be that this is just a broken test. I'll look more closely
tomorrow and see if I can figure out whether that's the case or whether
there is a real bug here.

Cheers,
 
> > ceph-deploy - approved by Vasu
> > 
> > upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> > into this release ?
> > 
> > upgrade/kraken-x (luminous) - some jobs still rerunning
> > 
> > upgrade/luminous-x (master) - Sage you to exclude this suite for now?
> > 
> > ceph-volume - pending approval from  Alfredo, Andrew
> > 
> > Pls reply
> > 
> > Thx
> > YuriW
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

-- 
Jeff Layton <jlayton@redhat.com>

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
  2018-02-13 21:41 ` Casey Bodley
@ 2018-02-13 22:42 ` Nathan Cutler
  2018-02-14 16:52   ` Yuri Weinstein
  2018-02-13 23:14 ` Josh Durgin
                   ` (3 subsequent siblings)
  5 siblings, 1 reply; 15+ messages in thread
From: Nathan Cutler @ 2018-02-13 22:42 UTC (permalink / raw)
  To: Yuri Weinstein, Sage Weil, Durgin, Josh, Dillaman, Jason,
	Sadeh-Weinraub, Yehuda, John Spray, Karol Mroz, Patrick Donnelly,
	Development, Ceph, Lekshmanan, Abhishek, Ilya Dryomov,
	Jeff Layton, ceph-qe-team, Deza, Alfredo, Andrew Schoen

Hi Yuri,

> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> into this release ?

The fix is limited to the kraken branch and was just merged. Can you 
repeat this run please?

Nathan

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
  2018-02-13 21:41 ` Casey Bodley
  2018-02-13 22:42 ` Nathan Cutler
@ 2018-02-13 23:14 ` Josh Durgin
  2018-02-14 11:00 ` Abhishek
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 15+ messages in thread
From: Josh Durgin @ 2018-02-13 23:14 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Sage Weil, Jason Dillaman, Yehuda Sadeh-Weinraub, John Spray,
	Karol Mroz, Patrick Donnelly, Ceph Development,
	Abhishek Lekshmanan, Nathan Cutler, Ilya Dryomov, Jeff Layton,
	ceph-qe-team, Alfredo Deza, Andrew Schoen

> rados - passed, Josh pls confirm approval.

Yup, look good to me.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
                   ` (2 preceding siblings ...)
  2018-02-13 23:14 ` Josh Durgin
@ 2018-02-14 11:00 ` Abhishek
  2018-02-14 16:39 ` Yuri Weinstein
  2018-02-14 18:15 ` Jeff Layton
  5 siblings, 0 replies; 15+ messages in thread
From: Abhishek @ 2018-02-14 11:00 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, Jeff Layton,
	ceph-qe-team, Deza, Alfredo, Andrew Schoen, ceph-devel-owner

On 2018-02-13 22:36, Yuri Weinstein wrote:
> Details of this release summarized here
> http://tracker.ceph.com/issues/22665#note-4
> 
> The following suites included:
> 
> rados
> rgw
> rbd
> krbd
> fs
> kcephfs
> multimds
> knfs
> hadoop - EXCLUDED
> samba - EXCLUDED
> ceph-deploy
> ceph-disk
> upgrade/client-upgrade-hammer (luminous)
> upgrade/client-upgrade-kraken (luminous)
> upgrade/client-upgrade-jewel (luminous)
> upgrade/jewel-x (luminous)
> upgrade/kraken-x (luminous)
> upgrade/luminous-x (master) - EXCLUDED
> powercycle
> ceph-ansible
> ceph-volume
> (please speak up if something is missing)
> 
> Please see all details in the ticket and add comments in the tracker.
> 
> Seeking approval from the dev leads.
> 
> Issues:
> 
> rados - passed, Josh pls confirm approval.
> 
> rgw - Casey, Abhishek - do we want to ad//merge
> https://github.com/ceph/ceph/pull/20407 ?
> 
> rbd, krbd - approved by Jason
> 
> fs, kcephfs, multimds - approved by Patrick
> 
> knfs - pending review/approval from Jeff 
> (http://tracker.ceph.com/issues/22995)
> 
> ceph-deploy - approved by Vasu
> 
> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> into this release ?
> 
> upgrade/kraken-x (luminous) - some jobs still rerunning
> 
> upgrade/luminous-x (master) - Sage you to exclude this suite for now?
> 
> ceph-volume - pending approval from  Alfredo, Andrew

Alfredo mentioned he's working on a critical fix that needs to go in for
12.2.3, waiting for merge + approval

PS. for some reason replies to this thread from my email always failed. 
Hopefully we don't get multiple copies of the same message.

> 
> Pls reply
> 
> Thx
> YuriW
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
                   ` (3 preceding siblings ...)
  2018-02-14 11:00 ` Abhishek
@ 2018-02-14 16:39 ` Yuri Weinstein
  2018-02-14 16:46   ` Alfredo Deza
  2018-02-14 18:15 ` Jeff Layton
  5 siblings, 1 reply; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-14 16:39 UTC (permalink / raw)
  To: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, Jeff Layton,
	ceph-qe-team, Deza, Alfredo, Andrew Schoen

re: upgrade/kraken-x we need kraken fix merged for
http://tracker.ceph.com/issues/22740

Jason, will you approve or merge that PR for 12.2.3?

On Tue, Feb 13, 2018 at 1:36 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Details of this release summarized here
> http://tracker.ceph.com/issues/22665#note-4
>
> The following suites included:
>
> rados
> rgw
> rbd
> krbd
> fs
> kcephfs
> multimds
> knfs
> hadoop - EXCLUDED
> samba - EXCLUDED
> ceph-deploy
> ceph-disk
> upgrade/client-upgrade-hammer (luminous)
> upgrade/client-upgrade-kraken (luminous)
> upgrade/client-upgrade-jewel (luminous)
> upgrade/jewel-x (luminous)
> upgrade/kraken-x (luminous)
> upgrade/luminous-x (master) - EXCLUDED
> powercycle
> ceph-ansible
> ceph-volume
> (please speak up if something is missing)
>
> Please see all details in the ticket and add comments in the tracker.
>
> Seeking approval from the dev leads.
>
> Issues:
>
> rados - passed, Josh pls confirm approval.
>
> rgw - Casey, Abhishek - do we want to ad//merge
> https://github.com/ceph/ceph/pull/20407 ?
>
> rbd, krbd - approved by Jason
>
> fs, kcephfs, multimds - approved by Patrick
>
> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>
> ceph-deploy - approved by Vasu
>
> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
> into this release ?
>
> upgrade/kraken-x (luminous) - some jobs still rerunning
>
> upgrade/luminous-x (master) - Sage you to exclude this suite for now?
>
> ceph-volume - pending approval from  Alfredo, Andrew
>
> Pls reply
>
> Thx
> YuriW

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-14 16:39 ` Yuri Weinstein
@ 2018-02-14 16:46   ` Alfredo Deza
  0 siblings, 0 replies; 15+ messages in thread
From: Alfredo Deza @ 2018-02-14 16:46 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, Jeff Layton,
	ceph-qe-team, Andrew Schoen

On Wed, Feb 14, 2018 at 11:39 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
> re: upgrade/kraken-x we need kraken fix merged for
> http://tracker.ceph.com/issues/22740
>
> Jason, will you approve or merge that PR for 12.2.3?
>
> On Tue, Feb 13, 2018 at 1:36 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
>> Details of this release summarized here
>> http://tracker.ceph.com/issues/22665#note-4
>>
>> The following suites included:
>>
>> rados
>> rgw
>> rbd
>> krbd
>> fs
>> kcephfs
>> multimds
>> knfs
>> hadoop - EXCLUDED
>> samba - EXCLUDED
>> ceph-deploy
>> ceph-disk
>> upgrade/client-upgrade-hammer (luminous)
>> upgrade/client-upgrade-kraken (luminous)
>> upgrade/client-upgrade-jewel (luminous)
>> upgrade/jewel-x (luminous)
>> upgrade/kraken-x (luminous)
>> upgrade/luminous-x (master) - EXCLUDED
>> powercycle
>> ceph-ansible
>> ceph-volume
>> (please speak up if something is missing)
>>
>> Please see all details in the ticket and add comments in the tracker.
>>
>> Seeking approval from the dev leads.
>>
>> Issues:
>>
>> rados - passed, Josh pls confirm approval.
>>
>> rgw - Casey, Abhishek - do we want to ad//merge
>> https://github.com/ceph/ceph/pull/20407 ?
>>
>> rbd, krbd - approved by Jason
>>
>> fs, kcephfs, multimds - approved by Patrick
>>
>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>
>> ceph-deploy - approved by Vasu
>>
>> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
>> into this release ?
>>
>> upgrade/kraken-x (luminous) - some jobs still rerunning
>>
>> upgrade/luminous-x (master) - Sage you to exclude this suite for now?
>>
>> ceph-volume - pending approval from  Alfredo, Andrew
ceph-volume is good to go, both PRs got everything passing and merged today

https://github.com/ceph/ceph/pull/20429
https://github.com/ceph/ceph/pull/20438
>>
>> Pls reply
>>
>> Thx
>> YuriW

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-13 22:42 ` Nathan Cutler
@ 2018-02-14 16:52   ` Yuri Weinstein
  0 siblings, 0 replies; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-14 16:52 UTC (permalink / raw)
  To: Nathan Cutler
  Cc: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Ilya Dryomov, Jeff Layton, ceph-qe-team,
	Deza, Alfredo, Andrew Schoen

All green, great work Nathan !
http://pulpito.ceph.com/yuriw-2018-02-14_16:28:05-upgrade:client-upgrade-kraken-luminous-distro-basic-smithi/

On Tue, Feb 13, 2018 at 2:42 PM, Nathan Cutler <ncutler@suse.cz> wrote:
> Hi Yuri,
>
>> upgrade/client-upgrade-kraken - Nathan, do/can we include your fix
>> into this release ?
>
>
> The fix is limited to the kraken branch and was just merged. Can you repeat
> this run please?
>
> Nathan
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
                   ` (4 preceding siblings ...)
  2018-02-14 16:39 ` Yuri Weinstein
@ 2018-02-14 18:15 ` Jeff Layton
  2018-02-15 16:55   ` Yuri Weinstein
  5 siblings, 1 reply; 15+ messages in thread
From: Jeff Layton @ 2018-02-14 18:15 UTC (permalink / raw)
  To: Yuri Weinstein, Sage Weil, Durgin, Josh, Dillaman, Jason,
	Sadeh-Weinraub, Yehuda, John Spray, Karol Mroz, Patrick Donnelly,
	Development, Ceph, Lekshmanan, Abhishek, Nathan Cutler,
	Ilya Dryomov, ceph-qe-team, Deza, Alfredo, Andrew Schoen

On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:

> 
> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
> 

Ok, I've flogged it about as far as I can. The problem is that we see
OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
it just goes unresponsive to pings for a little while.

That's about as far as I can carry it -- at this point it'd be nice to
have someone with more familiarity with the OSD code take a look and see
what they can tell.

FWIW, these tests seem to routinely fail on smithi, but we have at least
one run on OVH hosts that passed. This leads be to believe that it's
something specific to smithi: flaky hw maybe? or possibly load related?

-- 
Jeff Layton <jlayton@redhat.com>

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-14 18:15 ` Jeff Layton
@ 2018-02-15 16:55   ` Yuri Weinstein
  2018-02-15 16:58     ` Jason Dillaman
  0 siblings, 1 reply; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-15 16:55 UTC (permalink / raw)
  To: Jeff Layton
  Cc: Sage Weil, Durgin, Josh, Dillaman, Jason, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
	Deza, Alfredo, Andrew Schoen

Outstanding issues:

upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
Jason, Sage approve
knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.

Abhishek, Alfredo - assuming all agreed and Sage approves, we can
publish 12.2.3 any time.

Thx
YuriW

On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>
>>
>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>
>
> Ok, I've flogged it about as far as I can. The problem is that we see
> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
> it just goes unresponsive to pings for a little while.
>
> That's about as far as I can carry it -- at this point it'd be nice to
> have someone with more familiarity with the OSD code take a look and see
> what they can tell.
>
> FWIW, these tests seem to routinely fail on smithi, but we have at least
> one run on OVH hosts that passed. This leads be to believe that it's
> something specific to smithi: flaky hw maybe? or possibly load related?
>
> --
> Jeff Layton <jlayton@redhat.com>

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-15 16:55   ` Yuri Weinstein
@ 2018-02-15 16:58     ` Jason Dillaman
  2018-02-15 17:33       ` Yuri Weinstein
  0 siblings, 1 reply; 15+ messages in thread
From: Jason Dillaman @ 2018-02-15 16:58 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Jeff Layton, Sage Weil, Durgin, Josh, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
	Deza, Alfredo, Andrew Schoen

Yuri,

On Thu, Feb 15, 2018 at 11:55 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Outstanding issues:
>
> upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
> Jason, Sage approve

Do you have a link to the associated failed test run? I would have
expected this to have been fixed indirectly by
https://github.com/ceph/ceph/pull/20053 as a workaround for the broken
kraken builders.

> knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
>
> Abhishek, Alfredo - assuming all agreed and Sage approves, we can
> publish 12.2.3 any time.
>
> Thx
> YuriW
>
> On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
>> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>>
>>>
>>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>>
>>
>> Ok, I've flogged it about as far as I can. The problem is that we see
>> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
>> it just goes unresponsive to pings for a little while.
>>
>> That's about as far as I can carry it -- at this point it'd be nice to
>> have someone with more familiarity with the OSD code take a look and see
>> what they can tell.
>>
>> FWIW, these tests seem to routinely fail on smithi, but we have at least
>> one run on OVH hosts that passed. This leads be to believe that it's
>> something specific to smithi: flaky hw maybe? or possibly load related?
>>
>> --
>> Jeff Layton <jlayton@redhat.com>



-- 
Jason

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-15 16:58     ` Jason Dillaman
@ 2018-02-15 17:33       ` Yuri Weinstein
  2018-02-15 17:57         ` Jason Dillaman
  0 siblings, 1 reply; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-15 17:33 UTC (permalink / raw)
  To: Dillaman, Jason
  Cc: Jeff Layton, Sage Weil, Durgin, Josh, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
	Deza, Alfredo, Andrew Schoen

Jason

http://pulpito.ceph.com/yuriw-2018-02-13_21:14:53-upgrade:kraken-x-luminous-distro-basic-smithi/


On Thu, Feb 15, 2018 at 8:58 AM, Jason Dillaman <jdillama@redhat.com> wrote:
> Yuri,
>
> On Thu, Feb 15, 2018 at 11:55 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>> Outstanding issues:
>>
>> upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
>> Jason, Sage approve
>
> Do you have a link to the associated failed test run? I would have
> expected this to have been fixed indirectly by
> https://github.com/ceph/ceph/pull/20053 as a workaround for the broken
> kraken builders.
>
>> knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
>>
>> Abhishek, Alfredo - assuming all agreed and Sage approves, we can
>> publish 12.2.3 any time.
>>
>> Thx
>> YuriW
>>
>> On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
>>> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>>>
>>>>
>>>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>>>
>>>
>>> Ok, I've flogged it about as far as I can. The problem is that we see
>>> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
>>> it just goes unresponsive to pings for a little while.
>>>
>>> That's about as far as I can carry it -- at this point it'd be nice to
>>> have someone with more familiarity with the OSD code take a look and see
>>> what they can tell.
>>>
>>> FWIW, these tests seem to routinely fail on smithi, but we have at least
>>> one run on OVH hosts that passed. This leads be to believe that it's
>>> something specific to smithi: flaky hw maybe? or possibly load related?
>>>
>>> --
>>> Jeff Layton <jlayton@redhat.com>
>
>
>
> --
> Jason

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-15 17:33       ` Yuri Weinstein
@ 2018-02-15 17:57         ` Jason Dillaman
  2018-02-16  1:22           ` Yuri Weinstein
  0 siblings, 1 reply; 15+ messages in thread
From: Jason Dillaman @ 2018-02-15 17:57 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Jeff Layton, Sage Weil, Durgin, Josh, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
	Deza, Alfredo, Andrew Schoen

Thanks -- just looks like that test needs the same blacklist applied
that PR 20053 performed on the other kraken-x upgrade test (or we just
fix the broken builders for kraken so the broken test can be yanked).

On Thu, Feb 15, 2018 at 12:33 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Jason
>
> http://pulpito.ceph.com/yuriw-2018-02-13_21:14:53-upgrade:kraken-x-luminous-distro-basic-smithi/
>
>
> On Thu, Feb 15, 2018 at 8:58 AM, Jason Dillaman <jdillama@redhat.com> wrote:
>> Yuri,
>>
>> On Thu, Feb 15, 2018 at 11:55 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>>> Outstanding issues:
>>>
>>> upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
>>> Jason, Sage approve
>>
>> Do you have a link to the associated failed test run? I would have
>> expected this to have been fixed indirectly by
>> https://github.com/ceph/ceph/pull/20053 as a workaround for the broken
>> kraken builders.
>>
>>> knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
>>>
>>> Abhishek, Alfredo - assuming all agreed and Sage approves, we can
>>> publish 12.2.3 any time.
>>>
>>> Thx
>>> YuriW
>>>
>>> On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
>>>> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>>>>
>>>>>
>>>>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>>>>
>>>>
>>>> Ok, I've flogged it about as far as I can. The problem is that we see
>>>> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
>>>> it just goes unresponsive to pings for a little while.
>>>>
>>>> That's about as far as I can carry it -- at this point it'd be nice to
>>>> have someone with more familiarity with the OSD code take a look and see
>>>> what they can tell.
>>>>
>>>> FWIW, these tests seem to routinely fail on smithi, but we have at least
>>>> one run on OVH hosts that passed. This leads be to believe that it's
>>>> something specific to smithi: flaky hw maybe? or possibly load related?
>>>>
>>>> --
>>>> Jeff Layton <jlayton@redhat.com>
>>
>>
>>
>> --
>> Jason



-- 
Jason

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 12.2.3 QE Luminous validation status
  2018-02-15 17:57         ` Jason Dillaman
@ 2018-02-16  1:22           ` Yuri Weinstein
  0 siblings, 0 replies; 15+ messages in thread
From: Yuri Weinstein @ 2018-02-16  1:22 UTC (permalink / raw)
  To: Dillaman, Jason
  Cc: Jeff Layton, Sage Weil, Durgin, Josh, Sadeh-Weinraub, Yehuda,
	John Spray, Karol Mroz, Patrick Donnelly, Development, Ceph,
	Lekshmanan, Abhishek, Nathan Cutler, Ilya Dryomov, ceph-qe-team,
	Deza, Alfredo, Andrew Schoen

Outstanding issues remaining:

upgrade/kraken-x => http://tracker.ceph.com/issues/22740 fixed by
https://github.com/ceph/ceph/pull/20451 (thanks for clues Jason!),
Abhishek assigned to you for merge

knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.



On Thu, Feb 15, 2018 at 9:57 AM, Jason Dillaman <jdillama@redhat.com> wrote:
> Thanks -- just looks like that test needs the same blacklist applied
> that PR 20053 performed on the other kraken-x upgrade test (or we just
> fix the broken builders for kraken so the broken test can be yanked).
>
> On Thu, Feb 15, 2018 at 12:33 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
>> Jason
>>
>> http://pulpito.ceph.com/yuriw-2018-02-13_21:14:53-upgrade:kraken-x-luminous-distro-basic-smithi/
>>
>>
>> On Thu, Feb 15, 2018 at 8:58 AM, Jason Dillaman <jdillama@redhat.com> wrote:
>>> Yuri,
>>>
>>> On Thu, Feb 15, 2018 at 11:55 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>>>> Outstanding issues:
>>>>
>>>> upgrade/kraken-x => http://tracker.ceph.com/issues/22740 assuming
>>>> Jason, Sage approve
>>>
>>> Do you have a link to the associated failed test run? I would have
>>> expected this to have been fixed indirectly by
>>> https://github.com/ceph/ceph/pull/20053 as a workaround for the broken
>>> kraken builders.
>>>
>>>> knfs => http://tracker.ceph.com/issues/22995 Sage pls approve.
>>>>
>>>> Abhishek, Alfredo - assuming all agreed and Sage approves, we can
>>>> publish 12.2.3 any time.
>>>>
>>>> Thx
>>>> YuriW
>>>>
>>>> On Wed, Feb 14, 2018 at 10:15 AM, Jeff Layton <jlayton@redhat.com> wrote:
>>>>> On Tue, 2018-02-13 at 13:36 -0800, Yuri Weinstein wrote:
>>>>>
>>>>>>
>>>>>> knfs - pending review/approval from Jeff (http://tracker.ceph.com/issues/22995)
>>>>>>
>>>>>
>>>>> Ok, I've flogged it about as far as I can. The problem is that we see
>>>>> OSD_DOWN in the logs after they run. It doesn't seem to have crashed --
>>>>> it just goes unresponsive to pings for a little while.
>>>>>
>>>>> That's about as far as I can carry it -- at this point it'd be nice to
>>>>> have someone with more familiarity with the OSD code take a look and see
>>>>> what they can tell.
>>>>>
>>>>> FWIW, these tests seem to routinely fail on smithi, but we have at least
>>>>> one run on OVH hosts that passed. This leads be to believe that it's
>>>>> something specific to smithi: flaky hw maybe? or possibly load related?
>>>>>
>>>>> --
>>>>> Jeff Layton <jlayton@redhat.com>
>>>
>>>
>>>
>>> --
>>> Jason
>
>
>
> --
> Jason

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2018-02-16  1:22 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-13 21:36 12.2.3 QE Luminous validation status Yuri Weinstein
2018-02-13 21:41 ` Casey Bodley
2018-02-13 21:59   ` Jeff Layton
2018-02-13 22:42 ` Nathan Cutler
2018-02-14 16:52   ` Yuri Weinstein
2018-02-13 23:14 ` Josh Durgin
2018-02-14 11:00 ` Abhishek
2018-02-14 16:39 ` Yuri Weinstein
2018-02-14 16:46   ` Alfredo Deza
2018-02-14 18:15 ` Jeff Layton
2018-02-15 16:55   ` Yuri Weinstein
2018-02-15 16:58     ` Jason Dillaman
2018-02-15 17:33       ` Yuri Weinstein
2018-02-15 17:57         ` Jason Dillaman
2018-02-16  1:22           ` Yuri Weinstein

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.