All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: hammer 0.94.10 QE status
       [not found] <CAMMFjmHCAM6wmygytBDqwh89UOEex24_GGqrLJvXcOUS+QEg9A@mail.gmail.com>
@ 2017-02-01 17:03 ` Ilya Dryomov
  2017-02-02 10:01   ` Ilya Dryomov
  2017-02-01 19:12 ` John Spray
  1 sibling, 1 reply; 21+ messages in thread
From: Ilya Dryomov @ 2017-02-01 17:03 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Development, Ceph, Nathan Cutler, Weil, Sage, Durgin, Josh, Just,
	Samuel, Dillaman, Jason, Gregory Farnum, John Spray,
	Sadeh-Weinraub, Yehuda

On Wed, Feb 1, 2017 at 5:52 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Here is snapshot of hammer 0.94.10 point release testing status.
>
> Suites were not tested on ubuntu 16.06 due to
> http://tracker.ceph.com/issues/18382 and as a result were forced to test on
> ubuntu 14.04 and centos 7.2 (note we don't have smithi with centos 7.3).
>
> There was an additional noise created by fact that since we tested last
> hammer release in August 2016 we moved from gitbuilders to shaman build
> system and it resulted in issues with some packages.
>
> Details of tests can be found here
> http://tracker.ceph.com/issues/17151#note-61
>
> ======================
> rados ubuntu - packaging issues, need approval - Josh
> rados centos - packaging issues, need approval - Josh
> ======================
> rbd - PASSED
> ======================
> rgw ubuntu - PASSED
> rgw centos - valgrind issues http://tracker.ceph.com/issues/18735, need
> approval Yehuda, Sage
> ======================
> fs ubuntu - looks similar to v0.94.8, need Greg, John approval
> fs centos - noise? need Greg, John review/approval
> ======================
> krbd ubuntu - looks like the same in v0.94.8
> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
> krbd centos -  looks like the same in v0.94.8
> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya

I'll need to look at

http://pulpito.ceph.com/teuthology-2017-01-26_20:58:56-krbd-master-testing-basic-smithi/752997
http://pulpito.ceph.com/teuthology-2017-01-26_21:01:15-krbd-master-testing-basic-smithi/753036

These are unexpected.

Thanks,

                Ilya

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
       [not found] <CAMMFjmHCAM6wmygytBDqwh89UOEex24_GGqrLJvXcOUS+QEg9A@mail.gmail.com>
  2017-02-01 17:03 ` hammer 0.94.10 QE status Ilya Dryomov
@ 2017-02-01 19:12 ` John Spray
  1 sibling, 0 replies; 21+ messages in thread
From: John Spray @ 2017-02-01 19:12 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Development, Ceph, Nathan Cutler, Weil, Sage, Durgin, Josh, Just,
	Samuel, Dillaman, Jason, Ilya Dryomov, Gregory Farnum,
	Sadeh-Weinraub, Yehuda

On Wed, Feb 1, 2017 at 4:52 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Here is snapshot of hammer 0.94.10 point release testing status.
>
> Suites were not tested on ubuntu 16.06 due to
> http://tracker.ceph.com/issues/18382 and as a result were forced to test on
> ubuntu 14.04 and centos 7.2 (note we don't have smithi with centos 7.3).
>
> There was an additional noise created by fact that since we tested last
> hammer release in August 2016 we moved from gitbuilders to shaman build
> system and it resulted in issues with some packages.
>
> Details of tests can be found here
> http://tracker.ceph.com/issues/17151#note-61
>
> ======================
> rados ubuntu - packaging issues, need approval - Josh
> rados centos - packaging issues, need approval - Josh
> ======================
> rbd - PASSED
> ======================
> rgw ubuntu - PASSED
> rgw centos - valgrind issues http://tracker.ceph.com/issues/18735, need
> approval Yehuda, Sage
> ======================
> fs ubuntu - looks similar to v0.94.8, need Greg, John approval
> fs centos - noise? need Greg, John review/approval

The MDL_WriteError one is the bug from when ENOSPC stuff was
backported wrongly, the FileJournal/FileStore assertions are from
btrfs.

Good to go.

John



> ======================
> krbd ubuntu - looks like the same in v0.94.8
> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
> krbd centos -  looks like the same in v0.94.8
> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
> ======================
> kcephfs - PASSED
> ======================
> knfs - PASSED
> ======================
> hadoop - PASSED
> ======================
> samba - packaging issues, need approval - Greg, Sage
> ======================
> rest  - PASSED
> ======================
> ceph-deploy - Vasu is fixing, need approval Sage
> ======================
> upgrade/client-upgrade - PASSED
> ======================
> upgrade/firefly-x (hammer) - need Sage approval. maybe deprecate?
> ======================
> upgrade/hammer - PASSED
> ======================
> upgrade/hammer-x (jewel) - FAILED, need Sage approval. maybe deprecate?
> ======================
> powercycle - PASSED
> ======================
>
> Thx
> YuriW

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-01 17:03 ` hammer 0.94.10 QE status Ilya Dryomov
@ 2017-02-02 10:01   ` Ilya Dryomov
  2017-02-02 11:17     ` Nathan Cutler
  0 siblings, 1 reply; 21+ messages in thread
From: Ilya Dryomov @ 2017-02-02 10:01 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Development, Ceph, Nathan Cutler, Weil, Sage, Durgin, Josh, Just,
	Samuel, Dillaman, Jason, Gregory Farnum, John Spray,
	Sadeh-Weinraub, Yehuda

On Wed, Feb 1, 2017 at 6:03 PM, Ilya Dryomov <idryomov@gmail.com> wrote:
> On Wed, Feb 1, 2017 at 5:52 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
>> Here is snapshot of hammer 0.94.10 point release testing status.
>>
>> Suites were not tested on ubuntu 16.06 due to
>> http://tracker.ceph.com/issues/18382 and as a result were forced to test on
>> ubuntu 14.04 and centos 7.2 (note we don't have smithi with centos 7.3).
>>
>> There was an additional noise created by fact that since we tested last
>> hammer release in August 2016 we moved from gitbuilders to shaman build
>> system and it resulted in issues with some packages.
>>
>> Details of tests can be found here
>> http://tracker.ceph.com/issues/17151#note-61
>>
>> ======================
>> rados ubuntu - packaging issues, need approval - Josh
>> rados centos - packaging issues, need approval - Josh
>> ======================
>> rbd - PASSED
>> ======================
>> rgw ubuntu - PASSED
>> rgw centos - valgrind issues http://tracker.ceph.com/issues/18735, need
>> approval Yehuda, Sage
>> ======================
>> fs ubuntu - looks similar to v0.94.8, need Greg, John approval
>> fs centos - noise? need Greg, John review/approval
>> ======================
>> krbd ubuntu - looks like the same in v0.94.8
>> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
>> krbd centos -  looks like the same in v0.94.8
>> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
>
> I'll need to look at
>
> http://pulpito.ceph.com/teuthology-2017-01-26_20:58:56-krbd-master-testing-basic-smithi/752997
> http://pulpito.ceph.com/teuthology-2017-01-26_21:01:15-krbd-master-testing-basic-smithi/753036
>
> These are unexpected.

Turned out to be a buggy test -- hammer is missing an infernalis
commit f217865cf7c4 ("test_librbd_fsx: invalidate before discard in
krbd mode").

Approved.

Thanks,

                Ilya

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-02 10:01   ` Ilya Dryomov
@ 2017-02-02 11:17     ` Nathan Cutler
  2017-02-07 21:47       ` Yuri Weinstein
  2017-02-14 17:05       ` Yuri Weinstein
  0 siblings, 2 replies; 21+ messages in thread
From: Nathan Cutler @ 2017-02-02 11:17 UTC (permalink / raw)
  To: Ilya Dryomov, Yuri Weinstein
  Cc: Development, Ceph, Weil, Sage, Durgin, Josh, Just, Samuel,
	Dillaman, Jason, Gregory Farnum, John Spray, Sadeh-Weinraub,
	Yehuda

> Turned out to be a buggy test -- hammer is missing an infernalis
> commit f217865cf7c4 ("test_librbd_fsx: invalidate before discard in
> krbd mode").

I gave it a try; please take a look:

* https://github.com/ceph/ceph/pull/13230

Nathan

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-02 11:17     ` Nathan Cutler
@ 2017-02-07 21:47       ` Yuri Weinstein
  2017-02-08 21:14         ` Yuri Weinstein
  2017-02-14 17:05       ` Yuri Weinstein
  1 sibling, 1 reply; 21+ messages in thread
From: Yuri Weinstein @ 2017-02-07 21:47 UTC (permalink / raw)
  To: Nathan Cutler
  Cc: Ilya Dryomov, Development, Ceph, Weil, Sage, Durgin, Josh, Just,
	Samuel, Dillaman, Jason, Gregory Farnum, John Spray,
	Sadeh-Weinraub, Yehuda

See updated status with approvals http://tracker.ceph.com/issues/17151#note-61

Outstanding suites requiring approval:

rgw
samba
upgrade/firefly-x (hammer)
upgrade/hammer-x (jewel)

Thx
YuriW

On Thu, Feb 2, 2017 at 3:17 AM, Nathan Cutler <ncutler@suse.cz> wrote:
>> Turned out to be a buggy test -- hammer is missing an infernalis
>> commit f217865cf7c4 ("test_librbd_fsx: invalidate before discard in
>> krbd mode").
>
>
> I gave it a try; please take a look:
>
> * https://github.com/ceph/ceph/pull/13230
>
> Nathan

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-07 21:47       ` Yuri Weinstein
@ 2017-02-08 21:14         ` Yuri Weinstein
       [not found]           ` <CAObvjwPb_PosEnGOXP2tV4aWjnTyed7-275svyK+vu4uxYdTzQ@mail.gmail.com>
  0 siblings, 1 reply; 21+ messages in thread
From: Yuri Weinstein @ 2017-02-08 21:14 UTC (permalink / raw)
  To: Nathan Cutler
  Cc: Ilya Dryomov, Development, Ceph, Weil, Sage, Durgin, Josh, Just,
	Samuel, Dillaman, Jason, Gregory Farnum, John Spray,
	Sadeh-Weinraub, Yehuda

upgrade/hammer-x (jewel) PASSED now!

Outstanding suites requiring approval:

rgw
samba
upgrade/firefly-x (hammer)

Sage?

Thx
YuriW

On Tue, Feb 7, 2017 at 1:47 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> See updated status with approvals http://tracker.ceph.com/issues/17151#note-61
>
> Outstanding suites requiring approval:
>
> rgw
> samba
> upgrade/firefly-x (hammer)
> upgrade/hammer-x (jewel)
>
> Thx
> YuriW
>
> On Thu, Feb 2, 2017 at 3:17 AM, Nathan Cutler <ncutler@suse.cz> wrote:
>>> Turned out to be a buggy test -- hammer is missing an infernalis
>>> commit f217865cf7c4 ("test_librbd_fsx: invalidate before discard in
>>> krbd mode").
>>
>>
>> I gave it a try; please take a look:
>>
>> * https://github.com/ceph/ceph/pull/13230
>>
>> Nathan

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
       [not found]           ` <CAObvjwPb_PosEnGOXP2tV4aWjnTyed7-275svyK+vu4uxYdTzQ@mail.gmail.com>
@ 2017-02-13 16:24             ` Sage Weil
  2017-02-14 14:30               ` Orit Wasserman
  0 siblings, 1 reply; 21+ messages in thread
From: Sage Weil @ 2017-02-13 16:24 UTC (permalink / raw)
  To: Tamilarasi Muthamizhan
  Cc: Yuri Weinstein, Nathan Cutler, Ilya Dryomov, Development, Ceph,
	Durgin, Josh, Just, Samuel, Dillaman, Jason, Gregory Farnum,
	John Spray, Sadeh-Weinraub, Yehuda

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1656 bytes --]

Let's ignore samba here too.

The rgw failures are all valgrind... can Matt or Yehuda take a look and 
make sure they're not new/problematic?

Thanks!
sage

On Thu, 9 Feb 2017, Tamilarasi Muthamizhan wrote:

> Yuri, lets skip firefly-x upgrade issue. 
> 
> On Wed, Feb 8, 2017 at 1:14 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
>       upgrade/hammer-x (jewel) PASSED now!
> 
>       Outstanding suites requiring approval:
> 
>       rgw
>       samba
>       upgrade/firefly-x (hammer)
> 
>       Sage?
> 
>       Thx
>       YuriW
> 
>       On Tue, Feb 7, 2017 at 1:47 PM, Yuri Weinstein
>       <yweinste@redhat.com> wrote:
>       > See updated status with approvals
>       http://tracker.ceph.com/issues/17151#note-61
>       >
>       > Outstanding suites requiring approval:
>       >
>       > rgw
>       > samba
>       > upgrade/firefly-x (hammer)
>       > upgrade/hammer-x (jewel)
>       >
>       > Thx
>       > YuriW
>       >
>       > On Thu, Feb 2, 2017 at 3:17 AM, Nathan Cutler
>       <ncutler@suse.cz> wrote:
>       >>> Turned out to be a buggy test -- hammer is missing an
>       infernalis
>       >>> commit f217865cf7c4 ("test_librbd_fsx: invalidate before
>       discard in
>       >>> krbd mode").
>       >>
>       >>
>       >> I gave it a try; please take a look:
>       >>
>       >> * https://github.com/ceph/ceph/pull/13230
>       >>
>       >> Nathan
>       --
>       To unsubscribe from this list: send the line "unsubscribe
>       ceph-devel" in
>       the body of a message to majordomo@vger.kernel.org
>       More majordomo info at 
>       http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> 

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-13 16:24             ` Sage Weil
@ 2017-02-14 14:30               ` Orit Wasserman
  2017-02-14 15:04                 ` Nathan Cutler
  0 siblings, 1 reply; 21+ messages in thread
From: Orit Wasserman @ 2017-02-14 14:30 UTC (permalink / raw)
  To: Sage Weil
  Cc: Tamilarasi Muthamizhan, Yuri Weinstein, Nathan Cutler,
	Ilya Dryomov, Development, Ceph, Durgin, Josh, Just, Samuel,
	Dillaman, Jason, Gregory Farnum, John Spray, Sadeh-Weinraub,
	Yehuda

On Mon, Feb 13, 2017 at 6:24 PM, Sage Weil <sweil@redhat.com> wrote:
> Let's ignore samba here too.
>
> The rgw failures are all valgrind... can Matt or Yehuda take a look and
> make sure they're not new/problematic?
>

They are all resulting of linking with tcmalloc in notcmalloc builds
that valgrind doesn't like.
There is an open tracker issue: http://tracker.ceph.com/issues/18744

Orit
> Thanks!
> sage
>
> On Thu, 9 Feb 2017, Tamilarasi Muthamizhan wrote:
>
>> Yuri, lets skip firefly-x upgrade issue.
>>
>> On Wed, Feb 8, 2017 at 1:14 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
>>       upgrade/hammer-x (jewel) PASSED now!
>>
>>       Outstanding suites requiring approval:
>>
>>       rgw
>>       samba
>>       upgrade/firefly-x (hammer)
>>
>>       Sage?
>>
>>       Thx
>>       YuriW
>>
>>       On Tue, Feb 7, 2017 at 1:47 PM, Yuri Weinstein
>>       <yweinste@redhat.com> wrote:
>>       > See updated status with approvals
>>       http://tracker.ceph.com/issues/17151#note-61
>>       >
>>       > Outstanding suites requiring approval:
>>       >
>>       > rgw
>>       > samba
>>       > upgrade/firefly-x (hammer)
>>       > upgrade/hammer-x (jewel)
>>       >
>>       > Thx
>>       > YuriW
>>       >
>>       > On Thu, Feb 2, 2017 at 3:17 AM, Nathan Cutler
>>       <ncutler@suse.cz> wrote:
>>       >>> Turned out to be a buggy test -- hammer is missing an
>>       infernalis
>>       >>> commit f217865cf7c4 ("test_librbd_fsx: invalidate before
>>       discard in
>>       >>> krbd mode").
>>       >>
>>       >>
>>       >> I gave it a try; please take a look:
>>       >>
>>       >> * https://github.com/ceph/ceph/pull/13230
>>       >>
>>       >> Nathan
>>       --
>>       To unsubscribe from this list: send the line "unsubscribe
>>       ceph-devel" in
>>       the body of a message to majordomo@vger.kernel.org
>>       More majordomo info at
>>       http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>>

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-14 14:30               ` Orit Wasserman
@ 2017-02-14 15:04                 ` Nathan Cutler
  2017-02-14 17:46                   ` Yuri Weinstein
  0 siblings, 1 reply; 21+ messages in thread
From: Nathan Cutler @ 2017-02-14 15:04 UTC (permalink / raw)
  To: Orit Wasserman, Sage Weil
  Cc: Tamilarasi Muthamizhan, Yuri Weinstein, Ilya Dryomov,
	Development, Ceph, Durgin, Josh, Just, Samuel, Dillaman, Jason,
	Gregory Farnum, John Spray, Sadeh-Weinraub, Yehuda

> They are all resulting of linking with tcmalloc in notcmalloc builds
> that valgrind doesn't like.
> There is an open tracker issue: http://tracker.ceph.com/issues/18744

Note that these failures would probably disappear on re-run, because Dan 
fixed the underlying build issue.

Nathan

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-02 11:17     ` Nathan Cutler
  2017-02-07 21:47       ` Yuri Weinstein
@ 2017-02-14 17:05       ` Yuri Weinstein
  1 sibling, 0 replies; 21+ messages in thread
From: Yuri Weinstein @ 2017-02-14 17:05 UTC (permalink / raw)
  To: Nathan Cutler
  Cc: Ilya Dryomov, Development, Ceph, Weil, Sage, Durgin, Josh, Just,
	Samuel, Dillaman, Jason, Gregory Farnum, John Spray,
	Sadeh-Weinraub, Yehuda

Let's retest !

On Thu, Feb 2, 2017 at 3:17 AM, Nathan Cutler <ncutler@suse.cz> wrote:
>> Turned out to be a buggy test -- hammer is missing an infernalis
>> commit f217865cf7c4 ("test_librbd_fsx: invalidate before discard in
>> krbd mode").
>
>
> I gave it a try; please take a look:
>
> * https://github.com/ceph/ceph/pull/13230
>
> Nathan

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-14 15:04                 ` Nathan Cutler
@ 2017-02-14 17:46                   ` Yuri Weinstein
  2017-02-17  5:06                     ` Nathan Cutler
  0 siblings, 1 reply; 21+ messages in thread
From: Yuri Weinstein @ 2017-02-14 17:46 UTC (permalink / raw)
  To: Nathan Cutler
  Cc: Orit Wasserman, Sage Weil, Tamilarasi Muthamizhan, Ilya Dryomov,
	Development, Ceph, Durgin, Josh, Just, Samuel, Dillaman, Jason,
	Gregory Farnum, John Spray, Sadeh-Weinraub, Yehuda

I am running tests and so far they are still showing the same rgw issues

http://pulpito.ceph.com/teuthology-2017-02-14_16:21:46-rgw-hammer-distro-basic-smithi/


On Tue, Feb 14, 2017 at 7:04 AM, Nathan Cutler <ncutler@suse.cz> wrote:
>> They are all resulting of linking with tcmalloc in notcmalloc builds
>> that valgrind doesn't like.
>> There is an open tracker issue: http://tracker.ceph.com/issues/18744
>
>
> Note that these failures would probably disappear on re-run, because Dan
> fixed the underlying build issue.
>
> Nathan

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-14 17:46                   ` Yuri Weinstein
@ 2017-02-17  5:06                     ` Nathan Cutler
  2017-02-17 14:14                       ` Sage Weil
  2017-02-17 18:55                       ` Gregory Farnum
  0 siblings, 2 replies; 21+ messages in thread
From: Nathan Cutler @ 2017-02-17  5:06 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Orit Wasserman, Sage Weil, Tamilarasi Muthamizhan, Ilya Dryomov,
	Development, Ceph, Durgin, Josh, Just, Samuel, Dillaman, Jason,
	Gregory Farnum, John Spray, Sadeh-Weinraub, Yehuda

 From re-reading this thread I determined that the only two outstanding 
issues are upgrade/firefly-x and rgw+centos (can anyone confirm?)

I don't think upgrade/firefly-x should be a blocker. Rationale: as of 
the jewel release, firefly clusters were supposed to be upgraded to at 
least hammer. As of the kraken release, support of hammer has been 
focused on facilitating users to upgrade from hammer to jewel. In other 
words, firefly clusters should already have been upgraded and upgrading 
from firefly to hammer is no longer supported.

As for the rgw valgrind/libtcmalloc failures on centos, I made a 
wip-hammer-baseline branch based on "hammer" but with a different SHA1 
("git commit --amend --reset-author") and pushed it to ceph/ceph-ci.git 
so Shaman will re-build it. This should ensure that "flavor=notcmalloc" 
really means what it says. Re-running rgw+centos on this 
wip-hammer-baseline branch will tell us more.


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-17  5:06                     ` Nathan Cutler
@ 2017-02-17 14:14                       ` Sage Weil
  2017-02-17 18:55                       ` Gregory Farnum
  1 sibling, 0 replies; 21+ messages in thread
From: Sage Weil @ 2017-02-17 14:14 UTC (permalink / raw)
  To: Nathan Cutler
  Cc: Yuri Weinstein, Orit Wasserman, Tamilarasi Muthamizhan,
	Ilya Dryomov, Development, Ceph, Durgin, Josh, Just, Samuel,
	Dillaman, Jason, Gregory Farnum, John Spray, Sadeh-Weinraub,
	Yehuda

On Fri, 17 Feb 2017, Nathan Cutler wrote:
> From re-reading this thread I determined that the only two outstanding issues
> are upgrade/firefly-x and rgw+centos (can anyone confirm?)
> 
> I don't think upgrade/firefly-x should be a blocker. Rationale: as of the
> jewel release, firefly clusters were supposed to be upgraded to at least
> hammer. As of the kraken release, support of hammer has been focused on
> facilitating users to upgrade from hammer to jewel. In other words, firefly
> clusters should already have been upgraded and upgrading from firefly to
> hammer is no longer supported.
> 
> As for the rgw valgrind/libtcmalloc failures on centos, I made a
> wip-hammer-baseline branch based on "hammer" but with a different SHA1 ("git
> commit --amend --reset-author") and pushed it to ceph/ceph-ci.git so Shaman
> will re-build it. This should ensure that "flavor=notcmalloc" really means
> what it says. Re-running rgw+centos on this wip-hammer-baseline branch will
> tell us more.

I do'nt think these are a blocker either.. it's a problem with the 
hammer 'notcmalloc' builds with are only for teuthology valgrind testing.

sage

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-17  5:06                     ` Nathan Cutler
  2017-02-17 14:14                       ` Sage Weil
@ 2017-02-17 18:55                       ` Gregory Farnum
  2017-02-17 19:52                         ` Nathan Cutler
  1 sibling, 1 reply; 21+ messages in thread
From: Gregory Farnum @ 2017-02-17 18:55 UTC (permalink / raw)
  To: Nathan Cutler
  Cc: Yuri Weinstein, Orit Wasserman, Sage Weil,
	Tamilarasi Muthamizhan, Ilya Dryomov, Durgin, Josh, Just, Samuel,
	Dillaman, Jason, John Spray, Sadeh-Weinraub, Yehuda, ceph-devel

On Thu, Feb 16, 2017 at 9:06 PM, Nathan Cutler <ncutler@suse.cz> wrote:
> From re-reading this thread I determined that the only two outstanding
> issues are upgrade/firefly-x and rgw+centos (can anyone confirm?)
>
> I don't think upgrade/firefly-x should be a blocker. Rationale: as of the
> jewel release, firefly clusters were supposed to be upgraded to at least
> hammer. As of the kraken release, support of hammer has been focused on
> facilitating users to upgrade from hammer to jewel. In other words, firefly
> clusters should already have been upgraded and upgrading from firefly to
> hammer is no longer supported.

Woah, that can't be the case. Just because we're not supporting
firefly by putting out new releases doesn't mean it's acceptable to
prevent users from upgrading it to newer LTSes!

I've no idea what's broken there, so eg more valgrind failures are not
a big deal, but if the upgrade itself is failing in some way that's
going to be a problem. o_O
-Greg

>
> As for the rgw valgrind/libtcmalloc failures on centos, I made a
> wip-hammer-baseline branch based on "hammer" but with a different SHA1 ("git
> commit --amend --reset-author") and pushed it to ceph/ceph-ci.git so Shaman
> will re-build it. This should ensure that "flavor=notcmalloc" really means
> what it says. Re-running rgw+centos on this wip-hammer-baseline branch will
> tell us more.
>

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-17 18:55                       ` Gregory Farnum
@ 2017-02-17 19:52                         ` Nathan Cutler
  2017-02-17 23:21                           ` Gregory Farnum
  0 siblings, 1 reply; 21+ messages in thread
From: Nathan Cutler @ 2017-02-17 19:52 UTC (permalink / raw)
  To: Gregory Farnum
  Cc: Yuri Weinstein, Orit Wasserman, Sage Weil,
	Tamilarasi Muthamizhan, Ilya Dryomov, Durgin, Josh, Just, Samuel,
	Dillaman, Jason, John Spray, Sadeh-Weinraub, Yehuda, ceph-devel

> I've no idea what's broken there, so eg more valgrind failures are not
> a big deal, but if the upgrade itself is failing in some way that's
> going to be a problem. o_O

Hi Greg,

Let me try to clarify :-)

If someone reports problems upgrading their firefly cluster to hammer, 
and the problem is in firefly, will we publish a new firefly release?

If someone reports problems upgrading their dumpling cluster to firefly, 
and the problem is in dumpling, will we publish a new dumpling release?

I don't know the answer to those questions, but such a release -- if 
published -- would be "out-of-band" with regard to what is written in 
https://github.com/ceph/ceph/blob/master/doc/releases.rst#understanding-the-release-cycle 
- in particular this:

"Every other stable releases is a LTS (Long Term Stable) and will 
receive updates until two LTS are published. For instance Dumpling is 
retired when Hammer is published, Firefly is retired when Jewel is 
published etc."

Nathan

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-17 19:52                         ` Nathan Cutler
@ 2017-02-17 23:21                           ` Gregory Farnum
  2017-02-18  8:19                             ` Nathan Cutler
  0 siblings, 1 reply; 21+ messages in thread
From: Gregory Farnum @ 2017-02-17 23:21 UTC (permalink / raw)
  To: Nathan Cutler
  Cc: Yuri Weinstein, Orit Wasserman, Sage Weil,
	Tamilarasi Muthamizhan, Ilya Dryomov, Durgin, Josh, Just, Samuel,
	Dillaman, Jason, John Spray, Sadeh-Weinraub, Yehuda, ceph-devel

On Fri, Feb 17, 2017 at 11:52 AM, Nathan Cutler <ncutler@suse.cz> wrote:
>> I've no idea what's broken there, so eg more valgrind failures are not
>> a big deal, but if the upgrade itself is failing in some way that's
>> going to be a problem. o_O
>
>
> Hi Greg,
>
> Let me try to clarify :-)
>
> If someone reports problems upgrading their firefly cluster to hammer, and
> the problem is in firefly, will we publish a new firefly release?
>
> If someone reports problems upgrading their dumpling cluster to firefly, and
> the problem is in dumpling, will we publish a new dumpling release?

Probably not. But, we do know that upgrades work now and so any new
failures must be a failure of the new version of Hammer. That, we
would hopefully avoid releasing, and would definitely issue a fix
release for. :) We can't strand users!

>
> I don't know the answer to those questions, but such a release -- if
> published -- would be "out-of-band" with regard to what is written in
> https://github.com/ceph/ceph/blob/master/doc/releases.rst#understanding-the-release-cycle
> - in particular this:
>
> "Every other stable releases is a LTS (Long Term Stable) and will receive
> updates until two LTS are published. For instance Dumpling is retired when
> Hammer is published, Firefly is retired when Jewel is published etc."
>
> Nathan

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-17 23:21                           ` Gregory Farnum
@ 2017-02-18  8:19                             ` Nathan Cutler
  2017-02-18  8:39                               ` Nathan Cutler
  0 siblings, 1 reply; 21+ messages in thread
From: Nathan Cutler @ 2017-02-18  8:19 UTC (permalink / raw)
  To: Gregory Farnum
  Cc: Yuri Weinstein, Orit Wasserman, Sage Weil,
	Tamilarasi Muthamizhan, Ilya Dryomov, Durgin, Josh, Just, Samuel,
	Dillaman, Jason, John Spray, Sadeh-Weinraub, Yehuda, ceph-devel

>> If someone reports problems upgrading their firefly cluster to hammer, and
>> the problem is in firefly, will we publish a new firefly release?
>>
>> If someone reports problems upgrading their dumpling cluster to firefly, and
>> the problem is in dumpling, will we publish a new dumpling release?
>
> Probably not. But, we do know that upgrades work now and so any new
> failures must be a failure of the new version of Hammer. That, we
> would hopefully avoid releasing, and would definitely issue a fix
> release for. :) We can't strand users!

Several people have looked at the upgrade/firefly-x (hammer) failures 
and nobody has found anything that wasn't an issue with the tests 
themselves. Maybe you could take a look, too?

http://pulpito.ceph.com/teuthology-2017-01-30_20:52:09-upgrade:firefly-x-hammer-distro-basic-vps/

Note that this QE run was done against the old gitbuilder build system.
Yuri has some special technique for achieving that ;-)

This is because the old builds that are needed to run the tests were/are 
missing in the new build system. Injecting the old gitbuilder builds 
into the new system is apparently possible, but a manual hit-or-miss 
process. See http://tracker.ceph.com/issues/18089

(David Galloway has been _very_ helpful there, but the issues persist.)

This issue is affecting the jewel rados suite, even, since that suite 
includes a couple tests that install various dumpling and firefly 
releases and then upgrade them.

Thanks,
Nathan

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-18  8:19                             ` Nathan Cutler
@ 2017-02-18  8:39                               ` Nathan Cutler
  0 siblings, 0 replies; 21+ messages in thread
From: Nathan Cutler @ 2017-02-18  8:39 UTC (permalink / raw)
  To: Gregory Farnum
  Cc: Yuri Weinstein, Orit Wasserman, Sage Weil,
	Tamilarasi Muthamizhan, Ilya Dryomov, Durgin, Josh, Just, Samuel,
	Dillaman, Jason, John Spray, Sadeh-Weinraub, Yehuda, ceph-devel

> Several people have looked at the upgrade/firefly-x (hammer) failures
> and nobody has found anything that wasn't an issue with the tests
> themselves. Maybe you could take a look, too?
>
> http://pulpito.ceph.com/teuthology-2017-01-30_20:52:09-upgrade:firefly-x-hammer-distro-basic-vps/

There are two failed and three dead jobs. The two failures are

ceph-objectstore-tool apply-layout-settings failed with 1

My totally untested theory: the client is not getting upgraded from 
firefly to hammer, and as of 
https://github.com/ceph/ceph/commit/e4038a0c ceph_manager.py has the 
client running the "apply-layout-settings" subcommand of 
ceph-objectstore-tool which is missing in firefly.

PR with fix for review: https://github.com/ceph/ceph/pull/13513 (against 
hammer for now, but the same code is jewel, kraken, and master as well)

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-01 16:55 Yuri Weinstein
  2017-02-01 20:36 ` Sage Weil
@ 2017-02-01 23:47 ` Vasu Kulkarni
  1 sibling, 0 replies; 21+ messages in thread
From: Vasu Kulkarni @ 2017-02-01 23:47 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Development, Ceph, Nathan Cutler, Weil, Sage, Durgin, Josh, Just,
	Samuel, Dillaman, Jason, Ilya Dryomov, Gregory Farnum,
	John Spray, Sadeh-Weinraub, Yehuda

Sorry resending as text

On Wed, Feb 1, 2017 at 8:55 AM, Yuri Weinstein <yweinste@redhat.com> wrote:
>
> Here is snapshot of hammer 0.94.10 point release testing status.
>
> Suites were not tested on ubuntu 16.06 due to
> http://tracker.ceph.com/issues/18382 and as a result were forced to
> test on ubuntu 14.04 and centos 7.2 (note we don't have smithi with
> centos 7.3).
>
> There was an additional noise created by fact that since we tested
> last hammer release in August 2016 we moved from gitbuilders to shaman
> build system and it resulted in issues with some packages.
>
> Details of tests can be found here http://tracker.ceph.com/issues/17151#note-61
>
> ======================
> rados ubuntu - packaging issues, need approval - Josh
> rados centos - packaging issues, need approval - Josh
> ======================
> rbd - PASSED
> ======================
> rgw ubuntu - PASSED
> rgw centos - valgrind issues http://tracker.ceph.com/issues/18735,
> need approval Yehuda, Sage
> ======================
> fs ubuntu - looks similar to v0.94.8, need Greg, John approval
> fs centos - noise? need Greg, John review/approval
> ======================
> krbd ubuntu - looks like the same in v0.94.8
> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
> krbd centos -  looks like the same in v0.94.8
> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
> ======================
> kcephfs - PASSED
> ======================
> knfs - PASSED
> ======================
> hadoop - PASSED
> ======================
> samba - packaging issues, need approval - Greg, Sage
> ======================
> rest  - PASSED
> ======================
> ceph-deploy - Vasu is fixing, need approval Sage

This one is difficult to test due to package split that happened after
hammer+,  It is problem only for *new* installs of hammer.
Old version of ceph-deploy(1.5.27) works but it uses gitbuilder
instead of shaman, so we can test that only with some hacks. you
will have to pass on this for now, once its released we could check
the install with --stable flag instead of --dev flag.

>
> ======================
> upgrade/client-upgrade - PASSED
> ======================
> upgrade/firefly-x (hammer) - need Sage approval. maybe deprecate?
> ======================
> upgrade/hammer - PASSED
> ======================
> upgrade/hammer-x (jewel) - FAILED, need Sage approval. maybe deprecate?
> ======================
> powercycle - PASSED
> ======================
>
> Thx
> YuriW
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: hammer 0.94.10 QE status
  2017-02-01 16:55 Yuri Weinstein
@ 2017-02-01 20:36 ` Sage Weil
  2017-02-01 23:47 ` Vasu Kulkarni
  1 sibling, 0 replies; 21+ messages in thread
From: Sage Weil @ 2017-02-01 20:36 UTC (permalink / raw)
  To: Yuri Weinstein
  Cc: Development, Ceph, Nathan Cutler, Durgin, Josh, Just, Samuel,
	Dillaman, Jason, Ilya Dryomov, Gregory Farnum, John Spray,
	Sadeh-Weinraub, Yehuda

On Wed, 1 Feb 2017, Yuri Weinstein wrote:
> Here is snapshot of hammer 0.94.10 point release testing status.
> 
> Suites were not tested on ubuntu 16.06 due to
> http://tracker.ceph.com/issues/18382 and as a result were forced to
> test on ubuntu 14.04 and centos 7.2 (note we don't have smithi with
> centos 7.3).
> 
> There was an additional noise created by fact that since we tested
> last hammer release in August 2016 we moved from gitbuilders to shaman
> build system and it resulted in issues with some packages.
> 
> Details of tests can be found here http://tracker.ceph.com/issues/17151#note-61
> 
> ======================
> rados ubuntu - packaging issues, need approval - Josh
> rados centos - packaging issues, need approval - Josh
> ======================
> rbd - PASSED
> ======================
> rgw ubuntu - PASSED
> rgw centos - valgrind issues http://tracker.ceph.com/issues/18735,
> need approval Yehuda, Sage
> ======================
> fs ubuntu - looks similar to v0.94.8, need Greg, John approval
> fs centos - noise? need Greg, John review/approval
> ======================
> krbd ubuntu - looks like the same in v0.94.8
> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
> krbd centos -  looks like the same in v0.94.8
> http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
> ======================
> kcephfs - PASSED
> ======================
> knfs - PASSED
> ======================
> hadoop - PASSED
> ======================
> samba - packaging issues, need approval - Greg, Sage

ok to skip

> ======================
> rest  - PASSED
> ======================
> ceph-deploy - Vasu is fixing, need approval Sage

what's the story here?

> ======================
> upgrade/client-upgrade - PASSED
> ======================
> upgrade/firefly-x (hammer) - need Sage approval. maybe deprecate?

somethign is wrong with teh teuthology test, it's looping dumping ops in 
flight.  i don't have time to look at it, anybody else want to take a 
look?

> ======================
> upgrade/hammer - PASSED
> ======================
> upgrade/hammer-x (jewel) - FAILED, need Sage approval. maybe deprecate?

Looks like this test suite is broken, fixing here:

	https://github.com/ceph/ceph/pull/13222

> ======================
> powercycle - PASSED
> ======================
> 
> Thx
> YuriW
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

^ permalink raw reply	[flat|nested] 21+ messages in thread

* hammer 0.94.10 QE status
@ 2017-02-01 16:55 Yuri Weinstein
  2017-02-01 20:36 ` Sage Weil
  2017-02-01 23:47 ` Vasu Kulkarni
  0 siblings, 2 replies; 21+ messages in thread
From: Yuri Weinstein @ 2017-02-01 16:55 UTC (permalink / raw)
  To: Development, Ceph, Nathan Cutler, Weil, Sage, Durgin, Josh, Just,
	Samuel, Dillaman, Jason, Ilya Dryomov, Gregory Farnum,
	John Spray, Sadeh-Weinraub, Yehuda

Here is snapshot of hammer 0.94.10 point release testing status.

Suites were not tested on ubuntu 16.06 due to
http://tracker.ceph.com/issues/18382 and as a result were forced to
test on ubuntu 14.04 and centos 7.2 (note we don't have smithi with
centos 7.3).

There was an additional noise created by fact that since we tested
last hammer release in August 2016 we moved from gitbuilders to shaman
build system and it resulted in issues with some packages.

Details of tests can be found here http://tracker.ceph.com/issues/17151#note-61

======================
rados ubuntu - packaging issues, need approval - Josh
rados centos - packaging issues, need approval - Josh
======================
rbd - PASSED
======================
rgw ubuntu - PASSED
rgw centos - valgrind issues http://tracker.ceph.com/issues/18735,
need approval Yehuda, Sage
======================
fs ubuntu - looks similar to v0.94.8, need Greg, John approval
fs centos - noise? need Greg, John review/approval
======================
krbd ubuntu - looks like the same in v0.94.8
http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
krbd centos -  looks like the same in v0.94.8
http://tracker.ceph.com/issues/15895#note-33 need approval Jason, Ilya
======================
kcephfs - PASSED
======================
knfs - PASSED
======================
hadoop - PASSED
======================
samba - packaging issues, need approval - Greg, Sage
======================
rest  - PASSED
======================
ceph-deploy - Vasu is fixing, need approval Sage
======================
upgrade/client-upgrade - PASSED
======================
upgrade/firefly-x (hammer) - need Sage approval. maybe deprecate?
======================
upgrade/hammer - PASSED
======================
upgrade/hammer-x (jewel) - FAILED, need Sage approval. maybe deprecate?
======================
powercycle - PASSED
======================

Thx
YuriW

^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2017-02-18  8:39 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <CAMMFjmHCAM6wmygytBDqwh89UOEex24_GGqrLJvXcOUS+QEg9A@mail.gmail.com>
2017-02-01 17:03 ` hammer 0.94.10 QE status Ilya Dryomov
2017-02-02 10:01   ` Ilya Dryomov
2017-02-02 11:17     ` Nathan Cutler
2017-02-07 21:47       ` Yuri Weinstein
2017-02-08 21:14         ` Yuri Weinstein
     [not found]           ` <CAObvjwPb_PosEnGOXP2tV4aWjnTyed7-275svyK+vu4uxYdTzQ@mail.gmail.com>
2017-02-13 16:24             ` Sage Weil
2017-02-14 14:30               ` Orit Wasserman
2017-02-14 15:04                 ` Nathan Cutler
2017-02-14 17:46                   ` Yuri Weinstein
2017-02-17  5:06                     ` Nathan Cutler
2017-02-17 14:14                       ` Sage Weil
2017-02-17 18:55                       ` Gregory Farnum
2017-02-17 19:52                         ` Nathan Cutler
2017-02-17 23:21                           ` Gregory Farnum
2017-02-18  8:19                             ` Nathan Cutler
2017-02-18  8:39                               ` Nathan Cutler
2017-02-14 17:05       ` Yuri Weinstein
2017-02-01 19:12 ` John Spray
2017-02-01 16:55 Yuri Weinstein
2017-02-01 20:36 ` Sage Weil
2017-02-01 23:47 ` Vasu Kulkarni

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.