ceph-qa-suite branching (merge it into ceph.git?)

ceph-qa-suite branching (merge it into ceph.git?)

[John Spray]

Lately, we've instances where ceph.git and ceph-qa-suite.git got
slightly out of sync, as we were adding new stuff and interface
changes to ceph (especially in cephfs).

We used to have two repos (ceph and teuthology), now we have three
(ceph, ceph-qa-suite and teuthology).  Splitting tests out of
teuthology was a good thing, but maybe they should have gone into the
ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
pretty much mirror what we do with ceph, with master vs jewel etc.

Historically we had a comparatively static set of workloads in the qa
suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
much with ceph changes.  But these days we're adding much more
detailed tests, so there's more effort to keep the two in sync.

I would personally love to be able to have a single PR that contained
my code and the tests for it.  What if after Jewel we pulled all of
ceph-qa-suite into the ceph repo?

We could still enable folks running test changes without having to
rebuild ceph packages: the suite sha1 selected when running a
teuthology suite could still be different from that used for
installing ceph, it's just that it would fetch that sha1 from the ceph
repo instead of from a separate repo.

John

Re: ceph-qa-suite branching (merge it into ceph.git?)

[Sage Weil]

On Wed, 13 Apr 2016, John Spray wrote:
> Lately, we've instances where ceph.git and ceph-qa-suite.git got
> slightly out of sync, as we were adding new stuff and interface
> changes to ceph (especially in cephfs).
> 
> We used to have two repos (ceph and teuthology), now we have three
> (ceph, ceph-qa-suite and teuthology).  Splitting tests out of
> teuthology was a good thing, but maybe they should have gone into the
> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
> pretty much mirror what we do with ceph, with master vs jewel etc.
> 
> Historically we had a comparatively static set of workloads in the qa
> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
> much with ceph changes.  But these days we're adding much more
> detailed tests, so there's more effort to keep the two in sync.
> 
> I would personally love to be able to have a single PR that contained
> my code and the tests for it.  What if after Jewel we pulled all of
> ceph-qa-suite into the ceph repo?
> 
> We could still enable folks running test changes without having to
> rebuild ceph packages: the suite sha1 selected when running a
> teuthology suite could still be different from that used for
> installing ceph, it's just that it would fetch that sha1 from the ceph
> repo instead of from a separate repo.

This sounds pretty appealing to me...

sage

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Gregory Farnum]

On Wed, Apr 13, 2016 at 6:23 AM, Loic Dachary <loic@dachary.org> wrote:
> +1 : I agree it would be a good thing.
>
> The reason why it would help to merge ceph-qa-suite in ceph is because we have no proper methods or tools to work with interdependent repositories. The problem is not unique to Ceph: every Free Software developer end up bug fixing and adding features to dependencies (ceph-qa-suite in the case of Ceph but also jerasure, rocksdb, s3test etc.). It will take a long time to resolve that more general problem and I don't know about an effort in that direction. Do you ?
>
> Cheers
>
> On 13/04/2016 12:52, John Spray wrote:
>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>> slightly out of sync, as we were adding new stuff and interface
>> changes to ceph (especially in cephfs).
>>
>> We used to have two repos (ceph and teuthology), now we have three
>> (ceph, ceph-qa-suite and teuthology).  Splitting tests out of
>> teuthology was a good thing, but maybe they should have gone into the
>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>
>> Historically we had a comparatively static set of workloads in the qa
>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>> much with ceph changes.  But these days we're adding much more
>> detailed tests, so there's more effort to keep the two in sync.
>>
>> I would personally love to be able to have a single PR that contained
>> my code and the tests for it.  What if after Jewel we pulled all of
>> ceph-qa-suite into the ceph repo?
>>
>> We could still enable folks running test changes without having to
>> rebuild ceph packages: the suite sha1 selected when running a
>> teuthology suite could still be different from that used for
>> installing ceph, it's just that it would fetch that sha1 from the ceph
>> repo instead of from a separate repo.

We do have qa-suite tests that don't necessarily make a lot of sense
to have in ceph.git. Samba, kernel NFS, ganesha some day. That doesn't
mean we shouldn't merge them, but it popped into my head.
-Greg

Fwd: ceph-qa-suite branching (merge it into ceph.git?)

[Vasu Kulkarni]

Sorry for spam, the first one was rejected due to text/html format.

On Wed, Apr 13, 2016 at 3:52 AM, John Spray <jspray@redhat.com> wrote:
>
> Lately, we've instances where ceph.git and ceph-qa-suite.git got
> slightly out of sync, as we were adding new stuff and interface
> changes to ceph (especially in cephfs).
>
> We used to have two repos (ceph and teuthology), now we have three
> (ceph, ceph-qa-suite and teuthology).


we also have s3 suites in its own repo, we have big files in
ceph.com/qa which i believe is the right place.

>
> Splitting tests out of
> teuthology was a good thing, but maybe they should have gone into the
> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
> pretty much mirror what we do with ceph, with master vs jewel etc.
>
> Historically we had a comparatively static set of workloads in the qa
> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
> much with ceph changes.  But these days we're adding much more
> detailed tests, so there's more effort to keep the two in sync.
>
> I would personally love to be able to have a single PR that contained
> my code and the tests for it.  What if after Jewel we pulled all of
> ceph-qa-suite into the ceph repo?


I understand the advantage of having same sha between builds and
tests, but how are we going to guarantee ceph-builds in different
places(one built on other than ceph.com) will have same sha?
why not move ceph tests from ceph.git into ceph-qa-suite,  Ideally
this is not an issue with any major release, jewel workunits are
supposed to work with jewel ceph,
only in rare cases it may cause some issue but easily fixable. All the
dev runs can point to any qa-sha using the cli option so much less of
an issue in rare case.

>
> We could still enable folks running test changes without having to
> rebuild ceph packages: the suite sha1 selected when running a
> teuthology suite could still be different from that used for
> installing ceph, it's just that it would fetch that sha1 from the ceph
> repo instead of from a separate repo.
>
> John
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph-qa-suite branching (merge it into ceph.git?)

[Samuel Just]

This seems tidy to me.  I'd love to be able to merge a PR with it's
ceph-qa-suite PR "atomically".
-Sam

On Wed, Apr 13, 2016 at 11:10 AM, Vasu Kulkarni <vakulkar@redhat.com> wrote:
> Sorry for spam, the first one was rejected due to text/html format.
>
> On Wed, Apr 13, 2016 at 3:52 AM, John Spray <jspray@redhat.com> wrote:
>>
>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>> slightly out of sync, as we were adding new stuff and interface
>> changes to ceph (especially in cephfs).
>>
>> We used to have two repos (ceph and teuthology), now we have three
>> (ceph, ceph-qa-suite and teuthology).
>
>
> we also have s3 suites in its own repo, we have big files in
> ceph.com/qa which i believe is the right place.
>
>>
>> Splitting tests out of
>> teuthology was a good thing, but maybe they should have gone into the
>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>
>> Historically we had a comparatively static set of workloads in the qa
>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>> much with ceph changes.  But these days we're adding much more
>> detailed tests, so there's more effort to keep the two in sync.
>>
>> I would personally love to be able to have a single PR that contained
>> my code and the tests for it.  What if after Jewel we pulled all of
>> ceph-qa-suite into the ceph repo?
>
>
> I understand the advantage of having same sha between builds and
> tests, but how are we going to guarantee ceph-builds in different
> places(one built on other than ceph.com) will have same sha?
> why not move ceph tests from ceph.git into ceph-qa-suite,  Ideally
> this is not an issue with any major release, jewel workunits are
> supposed to work with jewel ceph,
> only in rare cases it may cause some issue but easily fixable. All the
> dev runs can point to any qa-sha using the cli option so much less of
> an issue in rare case.
>
>>
>> We could still enable folks running test changes without having to
>> rebuild ceph packages: the suite sha1 selected when running a
>> teuthology suite could still be different from that used for
>> installing ceph, it's just that it would fetch that sha1 from the ceph
>> repo instead of from a separate repo.
>>
>> John
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[John Spray]

On Wed, Apr 13, 2016 at 2:58 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> On Wed, Apr 13, 2016 at 6:23 AM, Loic Dachary <loic@dachary.org> wrote:
>> +1 : I agree it would be a good thing.
>>
>> The reason why it would help to merge ceph-qa-suite in ceph is because we have no proper methods or tools to work with interdependent repositories. The problem is not unique to Ceph: every Free Software developer end up bug fixing and adding features to dependencies (ceph-qa-suite in the case of Ceph but also jerasure, rocksdb, s3test etc.). It will take a long time to resolve that more general problem and I don't know about an effort in that direction. Do you ?
>>
>> Cheers
>>
>> On 13/04/2016 12:52, John Spray wrote:
>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>>> slightly out of sync, as we were adding new stuff and interface
>>> changes to ceph (especially in cephfs).
>>>
>>> We used to have two repos (ceph and teuthology), now we have three
>>> (ceph, ceph-qa-suite and teuthology).  Splitting tests out of
>>> teuthology was a good thing, but maybe they should have gone into the
>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>>
>>> Historically we had a comparatively static set of workloads in the qa
>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>>> much with ceph changes.  But these days we're adding much more
>>> detailed tests, so there's more effort to keep the two in sync.
>>>
>>> I would personally love to be able to have a single PR that contained
>>> my code and the tests for it.  What if after Jewel we pulled all of
>>> ceph-qa-suite into the ceph repo?
>>>
>>> We could still enable folks running test changes without having to
>>> rebuild ceph packages: the suite sha1 selected when running a
>>> teuthology suite could still be different from that used for
>>> installing ceph, it's just that it would fetch that sha1 from the ceph
>>> repo instead of from a separate repo.
>
> We do have qa-suite tests that don't necessarily make a lot of sense
> to have in ceph.git. Samba, kernel NFS, ganesha some day. That doesn't
> mean we shouldn't merge them, but it popped into my head.

Fair point.  I think that because those other tasks depend in turn on
the ceph tasks, we still ultimately benefit from having them in one
place.

It's also possible that in the long run things like the samba tests
become a bit more "smart" in a way that's more tightly coupled to
ceph, e.g. checking the resulting state inside ceph after doing things
in the samba/ganesha layer, at which point we'd enjoy having them in
the same place as the main body of test code.

John

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Samuel Just]

It also doesn't seem like it would actually present a problem in any case.
-Sam

On Wed, Apr 13, 2016 at 11:31 AM, John Spray <jspray@redhat.com> wrote:
> On Wed, Apr 13, 2016 at 2:58 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>> On Wed, Apr 13, 2016 at 6:23 AM, Loic Dachary <loic@dachary.org> wrote:
>>> +1 : I agree it would be a good thing.
>>>
>>> The reason why it would help to merge ceph-qa-suite in ceph is because we have no proper methods or tools to work with interdependent repositories. The problem is not unique to Ceph: every Free Software developer end up bug fixing and adding features to dependencies (ceph-qa-suite in the case of Ceph but also jerasure, rocksdb, s3test etc.). It will take a long time to resolve that more general problem and I don't know about an effort in that direction. Do you ?
>>>
>>> Cheers
>>>
>>> On 13/04/2016 12:52, John Spray wrote:
>>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>>>> slightly out of sync, as we were adding new stuff and interface
>>>> changes to ceph (especially in cephfs).
>>>>
>>>> We used to have two repos (ceph and teuthology), now we have three
>>>> (ceph, ceph-qa-suite and teuthology).  Splitting tests out of
>>>> teuthology was a good thing, but maybe they should have gone into the
>>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>>>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>>>
>>>> Historically we had a comparatively static set of workloads in the qa
>>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>>>> much with ceph changes.  But these days we're adding much more
>>>> detailed tests, so there's more effort to keep the two in sync.
>>>>
>>>> I would personally love to be able to have a single PR that contained
>>>> my code and the tests for it.  What if after Jewel we pulled all of
>>>> ceph-qa-suite into the ceph repo?
>>>>
>>>> We could still enable folks running test changes without having to
>>>> rebuild ceph packages: the suite sha1 selected when running a
>>>> teuthology suite could still be different from that used for
>>>> installing ceph, it's just that it would fetch that sha1 from the ceph
>>>> repo instead of from a separate repo.
>>
>> We do have qa-suite tests that don't necessarily make a lot of sense
>> to have in ceph.git. Samba, kernel NFS, ganesha some day. That doesn't
>> mean we shouldn't merge them, but it popped into my head.
>
> Fair point.  I think that because those other tasks depend in turn on
> the ceph tasks, we still ultimately benefit from having them in one
> place.
>
> It's also possible that in the long run things like the samba tests
> become a bit more "smart" in a way that's more tightly coupled to
> ceph, e.g. checking the resulting state inside ceph after doing things
> in the samba/ganesha layer, at which point we'd enjoy having them in
> the same place as the main body of test code.
>
> John
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Sage Weil]

On Wed, 13 Apr 2016, Samuel Just wrote:
> It also doesn't seem like it would actually present a problem in any case.

The reason we didn't do this before was because we wanted to revise tests 
independently of the thing being tested.  But as John points out, 
specifying a test branch should accomplish that, at least for testing.

It might be nice to preserve the ability specify an alternate repo with 
totally out-of-tree tests.  Maybe that can be done with a simple reorg of 
the repo, like putting everything under qa/, so that tasks/ and suites/ 
don't appear at the top level (of ceph.git or ceph-qa-suite.git).

sage



> -Sam
> 
> On Wed, Apr 13, 2016 at 11:31 AM, John Spray <jspray@redhat.com> wrote:
> > On Wed, Apr 13, 2016 at 2:58 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> >> On Wed, Apr 13, 2016 at 6:23 AM, Loic Dachary <loic@dachary.org> wrote:
> >>> +1 : I agree it would be a good thing.
> >>>
> >>> The reason why it would help to merge ceph-qa-suite in ceph is because we have no proper methods or tools to work with interdependent repositories. The problem is not unique to Ceph: every Free Software developer end up bug fixing and adding features to dependencies (ceph-qa-suite in the case of Ceph but also jerasure, rocksdb, s3test etc.). It will take a long time to resolve that more general problem and I don't know about an effort in that direction. Do you ?
> >>>
> >>> Cheers
> >>>
> >>> On 13/04/2016 12:52, John Spray wrote:
> >>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
> >>>> slightly out of sync, as we were adding new stuff and interface
> >>>> changes to ceph (especially in cephfs).
> >>>>
> >>>> We used to have two repos (ceph and teuthology), now we have three
> >>>> (ceph, ceph-qa-suite and teuthology).  Splitting tests out of
> >>>> teuthology was a good thing, but maybe they should have gone into the
> >>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
> >>>> pretty much mirror what we do with ceph, with master vs jewel etc.
> >>>>
> >>>> Historically we had a comparatively static set of workloads in the qa
> >>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
> >>>> much with ceph changes.  But these days we're adding much more
> >>>> detailed tests, so there's more effort to keep the two in sync.
> >>>>
> >>>> I would personally love to be able to have a single PR that contained
> >>>> my code and the tests for it.  What if after Jewel we pulled all of
> >>>> ceph-qa-suite into the ceph repo?
> >>>>
> >>>> We could still enable folks running test changes without having to
> >>>> rebuild ceph packages: the suite sha1 selected when running a
> >>>> teuthology suite could still be different from that used for
> >>>> installing ceph, it's just that it would fetch that sha1 from the ceph
> >>>> repo instead of from a separate repo.
> >>
> >> We do have qa-suite tests that don't necessarily make a lot of sense
> >> to have in ceph.git. Samba, kernel NFS, ganesha some day. That doesn't
> >> mean we shouldn't merge them, but it popped into my head.
> >
> > Fair point.  I think that because those other tasks depend in turn on
> > the ceph tasks, we still ultimately benefit from having them in one
> > place.
> >
> > It's also possible that in the long run things like the samba tests
> > become a bit more "smart" in a way that's more tightly coupled to
> > ceph, e.g. checking the resulting state inside ceph after doing things
> > in the samba/ganesha layer, at which point we'd enjoy having them in
> > the same place as the main body of test code.
> >
> > John
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> _______________________________________________
> Ceph-qa mailing list
> Ceph-qa@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com
> 
> 

Re: ceph-qa-suite branching (merge it into ceph.git?)

[John Spray]

On Wed, Apr 13, 2016 at 7:02 PM, Vasu Kulkarni <vakulkar@redhat.com> wrote:
>
>
> On Wed, Apr 13, 2016 at 3:52 AM, John Spray <jspray@redhat.com> wrote:
>>
>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>> slightly out of sync, as we were adding new stuff and interface
>> changes to ceph (especially in cephfs).
>>
>> We used to have two repos (ceph and teuthology), now we have three
>> (ceph, ceph-qa-suite and teuthology).
>
>
> we also have s3 suites in its own repo, we have big files in ceph.com/qa
> which i believe is the right place.

I'm a bit ignorant of the s3 tests, is there a particular reason they
live separately to the main ceph-qa-suite?

As for the stuff in ceph.com/qa, I of course am not suggesting putting
those bit binaries into the main git repo.  It is sort of annoying
that they live out there on a web server, but it's a separate issue
from the ceph/ceph-qa-suite one.

>> Splitting tests out of
>> teuthology was a good thing, but maybe they should have gone into the
>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>
>> Historically we had a comparatively static set of workloads in the qa
>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>> much with ceph changes.  But these days we're adding much more
>> detailed tests, so there's more effort to keep the two in sync.
>>
>> I would personally love to be able to have a single PR that contained
>> my code and the tests for it.  What if after Jewel we pulled all of
>> ceph-qa-suite into the ceph repo?
>
>
> I understand the advantage of having same sha between builds and tests, but
> how are we going to guarantee ceph-builds in different places(one built on
> other than ceph.com) will have same sha?

The thing that needs to match is the sha1 of the ceph code you're
running, and the sha1 you use when pulling the code to test it with.

I believe you currently have a situation where you have a built RPM
but you're not sure what ceph sha1 it came from, so you're pulling
e.g. the jewel branch of ceph-qa-suite rather than a specific sha1.
That needs fixing independently of ceph/ceph-qa-suite separation: you
have to run the right revision of the tests, irrespective of where
they're getting pulled from.

Basically, anyone generating packages with versions that don't map to
a ceph sha1 will need to have a way of remembering what ceph sha1
their packages correspond to.  This is not a new requirement, it's the
same issue that was discussed on IRC recently.

> why not move ceph tests from ceph.git into ceph-qa-suite,  Ideally this is
> not an issue with any major release, jewel workunits are supposed to work
> with jewel ceph,

That would not help, because you still have the same issue of changes
in ceph being out of step with changes in ceph-qa-suite.

I think your misconception here is to assume that running tests from
the same branch as the code is sufficient.  You need to run tests from
the same *revision*.  Otherwise, when I add a new test to reproduce a
bug, and fix the bug at the same time, then you will get failures if
you run the newer commit against the older code.

John

> only in rare cases it may cause some issue but easily fixable. All the dev
> runs can point to any qa-sha using the cli option so much less of an issue
> in rare case.



>>
>> We could still enable folks running test changes without having to
>> rebuild ceph packages: the suite sha1 selected when running a
>> teuthology suite could still be different from that used for
>> installing ceph, it's just that it would fetch that sha1 from the ceph
>> repo instead of from a separate repo.
>>
>> John
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[John Spray]

On Wed, Apr 13, 2016 at 7:38 PM, Sage Weil <sweil@redhat.com> wrote:
> On Wed, 13 Apr 2016, Samuel Just wrote:
>> It also doesn't seem like it would actually present a problem in any case.
>
> The reason we didn't do this before was because we wanted to revise tests
> independently of the thing being tested.  But as John points out,
> specifying a test branch should accomplish that, at least for testing.
>
> It might be nice to preserve the ability specify an alternate repo with
> totally out-of-tree tests.  Maybe that can be done with a simple reorg of
> the repo, like putting everything under qa/, so that tasks/ and suites/
> don't appear at the top level (of ceph.git or ceph-qa-suite.git).

Should be doable, teuthology will in any case still need to know how
to pull old-style separate-repo setups for when it is asked to
schedule a suite against a <= jewel branch.  Might be simplest to just
enable teuthology to have a suite path prefix in addition to its repo
address, so that you could tell it "get the suites from repo X in
/foo/bar" and avoid hardcoding it.  Then for older branches it would
be "github.com/ceph/ceph-qa-suite path /" and for newer stuff it would
be "github.com/ceph/ceph path /wherever"

John

> sage
>
>
>
>> -Sam
>>
>> On Wed, Apr 13, 2016 at 11:31 AM, John Spray <jspray@redhat.com> wrote:
>> > On Wed, Apr 13, 2016 at 2:58 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>> >> On Wed, Apr 13, 2016 at 6:23 AM, Loic Dachary <loic@dachary.org> wrote:
>> >>> +1 : I agree it would be a good thing.
>> >>>
>> >>> The reason why it would help to merge ceph-qa-suite in ceph is because we have no proper methods or tools to work with interdependent repositories. The problem is not unique to Ceph: every Free Software developer end up bug fixing and adding features to dependencies (ceph-qa-suite in the case of Ceph but also jerasure, rocksdb, s3test etc.). It will take a long time to resolve that more general problem and I don't know about an effort in that direction. Do you ?
>> >>>
>> >>> Cheers
>> >>>
>> >>> On 13/04/2016 12:52, John Spray wrote:
>> >>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>> >>>> slightly out of sync, as we were adding new stuff and interface
>> >>>> changes to ceph (especially in cephfs).
>> >>>>
>> >>>> We used to have two repos (ceph and teuthology), now we have three
>> >>>> (ceph, ceph-qa-suite and teuthology).  Splitting tests out of
>> >>>> teuthology was a good thing, but maybe they should have gone into the
>> >>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>> >>>> pretty much mirror what we do with ceph, with master vs jewel etc.
>> >>>>
>> >>>> Historically we had a comparatively static set of workloads in the qa
>> >>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>> >>>> much with ceph changes.  But these days we're adding much more
>> >>>> detailed tests, so there's more effort to keep the two in sync.
>> >>>>
>> >>>> I would personally love to be able to have a single PR that contained
>> >>>> my code and the tests for it.  What if after Jewel we pulled all of
>> >>>> ceph-qa-suite into the ceph repo?
>> >>>>
>> >>>> We could still enable folks running test changes without having to
>> >>>> rebuild ceph packages: the suite sha1 selected when running a
>> >>>> teuthology suite could still be different from that used for
>> >>>> installing ceph, it's just that it would fetch that sha1 from the ceph
>> >>>> repo instead of from a separate repo.
>> >>
>> >> We do have qa-suite tests that don't necessarily make a lot of sense
>> >> to have in ceph.git. Samba, kernel NFS, ganesha some day. That doesn't
>> >> mean we shouldn't merge them, but it popped into my head.
>> >
>> > Fair point.  I think that because those other tasks depend in turn on
>> > the ceph tasks, we still ultimately benefit from having them in one
>> > place.
>> >
>> > It's also possible that in the long run things like the samba tests
>> > become a bit more "smart" in a way that's more tightly coupled to
>> > ceph, e.g. checking the resulting state inside ceph after doing things
>> > in the samba/ganesha layer, at which point we'd enjoy having them in
>> > the same place as the main body of test code.
>> >
>> > John
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> _______________________________________________
>> Ceph-qa mailing list
>> Ceph-qa@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com
>>
>>

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Gregory Farnum]

On Wed, Apr 13, 2016 at 11:38 AM, Sage Weil <sweil@redhat.com> wrote:
> On Wed, 13 Apr 2016, Samuel Just wrote:
>> It also doesn't seem like it would actually present a problem in any case.
>
> The reason we didn't do this before was because we wanted to revise tests
> independently of the thing being tested.  But as John points out,
> specifying a test branch should accomplish that, at least for testing.
>
> It might be nice to preserve the ability specify an alternate repo with
> totally out-of-tree tests.  Maybe that can be done with a simple reorg of
> the repo, like putting everything under qa/, so that tasks/ and suites/
> don't appear at the top level (of ceph.git or ceph-qa-suite.git).

I shudder to suggest it, but would we get what we're really after by
including a ceph-qa-suite submodule in ceph.git, and having the
teuthology scheduling refer to that? Then we can update ceph-qa-suite
independently while still having an atomic switch for the tests that
goes into the same Ceph branch which makes the changes requiring them.
-Greg

Re: ceph-qa-suite branching (merge it into ceph.git?)

[Vasu Kulkarni]

+1 for "atomic" PR, this should solve the problem.

On Wed, Apr 13, 2016 at 11:23 AM, Samuel Just <sjust@redhat.com> wrote:
> This seems tidy to me.  I'd love to be able to merge a PR with it's
> ceph-qa-suite PR "atomically".
> -Sam
>
> On Wed, Apr 13, 2016 at 11:10 AM, Vasu Kulkarni <vakulkar@redhat.com> wrote:
>> Sorry for spam, the first one was rejected due to text/html format.
>>
>> On Wed, Apr 13, 2016 at 3:52 AM, John Spray <jspray@redhat.com> wrote:
>>>
>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>>> slightly out of sync, as we were adding new stuff and interface
>>> changes to ceph (especially in cephfs).
>>>
>>> We used to have two repos (ceph and teuthology), now we have three
>>> (ceph, ceph-qa-suite and teuthology).
>>
>>
>> we also have s3 suites in its own repo, we have big files in
>> ceph.com/qa which i believe is the right place.
>>
>>>
>>> Splitting tests out of
>>> teuthology was a good thing, but maybe they should have gone into the
>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>>
>>> Historically we had a comparatively static set of workloads in the qa
>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>>> much with ceph changes.  But these days we're adding much more
>>> detailed tests, so there's more effort to keep the two in sync.
>>>
>>> I would personally love to be able to have a single PR that contained
>>> my code and the tests for it.  What if after Jewel we pulled all of
>>> ceph-qa-suite into the ceph repo?
>>
>>
>> I understand the advantage of having same sha between builds and
>> tests, but how are we going to guarantee ceph-builds in different
>> places(one built on other than ceph.com) will have same sha?
>> why not move ceph tests from ceph.git into ceph-qa-suite,  Ideally
>> this is not an issue with any major release, jewel workunits are
>> supposed to work with jewel ceph,
>> only in rare cases it may cause some issue but easily fixable. All the
>> dev runs can point to any qa-sha using the cli option so much less of
>> an issue in rare case.
>>
>>>
>>> We could still enable folks running test changes without having to
>>> rebuild ceph packages: the suite sha1 selected when running a
>>> teuthology suite could still be different from that used for
>>> installing ceph, it's just that it would fetch that sha1 from the ceph
>>> repo instead of from a separate repo.
>>>
>>> John
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Sage Weil]

On Wed, 13 Apr 2016, Gregory Farnum wrote:
> On Wed, Apr 13, 2016 at 11:38 AM, Sage Weil <sweil@redhat.com> wrote:
> > On Wed, 13 Apr 2016, Samuel Just wrote:
> >> It also doesn't seem like it would actually present a problem in any case.
> >
> > The reason we didn't do this before was because we wanted to revise tests
> > independently of the thing being tested.  But as John points out,
> > specifying a test branch should accomplish that, at least for testing.
> >
> > It might be nice to preserve the ability specify an alternate repo with
> > totally out-of-tree tests.  Maybe that can be done with a simple reorg of
> > the repo, like putting everything under qa/, so that tasks/ and suites/
> > don't appear at the top level (of ceph.git or ceph-qa-suite.git).
> 
> I shudder to suggest it, but would we get what we're really after by
> including a ceph-qa-suite submodule in ceph.git, and having the
> teuthology scheduling refer to that? Then we can update ceph-qa-suite
> independently while still having an atomic switch for the tests that
> goes into the same Ceph branch which makes the changes requiring them.

Not really.  The workflow for updating submodules is really painful.  
Something like

 - commit to sub.git
 - push to branch in public sub.git
 - commit to main.git updating submodule sha
 - push to branch in main.git
 ...
 - merge main.git branch
 - merge sub.git branch (if you remember)

...and there are no "merge" semantics if there are racing sub.git updates.

Submodules only really work for things that are infrequently updated...

sage

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Samuel Just]

Seems like we just do a subtree merge into the ceph.git and call it good.
-Sam

On Wed, Apr 13, 2016 at 12:00 PM, Sage Weil <sweil@redhat.com> wrote:
> On Wed, 13 Apr 2016, Gregory Farnum wrote:
>> On Wed, Apr 13, 2016 at 11:38 AM, Sage Weil <sweil@redhat.com> wrote:
>> > On Wed, 13 Apr 2016, Samuel Just wrote:
>> >> It also doesn't seem like it would actually present a problem in any case.
>> >
>> > The reason we didn't do this before was because we wanted to revise tests
>> > independently of the thing being tested.  But as John points out,
>> > specifying a test branch should accomplish that, at least for testing.
>> >
>> > It might be nice to preserve the ability specify an alternate repo with
>> > totally out-of-tree tests.  Maybe that can be done with a simple reorg of
>> > the repo, like putting everything under qa/, so that tasks/ and suites/
>> > don't appear at the top level (of ceph.git or ceph-qa-suite.git).
>>
>> I shudder to suggest it, but would we get what we're really after by
>> including a ceph-qa-suite submodule in ceph.git, and having the
>> teuthology scheduling refer to that? Then we can update ceph-qa-suite
>> independently while still having an atomic switch for the tests that
>> goes into the same Ceph branch which makes the changes requiring them.
>
> Not really.  The workflow for updating submodules is really painful.
> Something like
>
>  - commit to sub.git
>  - push to branch in public sub.git
>  - commit to main.git updating submodule sha
>  - push to branch in main.git
>  ...
>  - merge main.git branch
>  - merge sub.git branch (if you remember)
>
> ...and there are no "merge" semantics if there are racing sub.git updates.
>
> Submodules only really work for things that are infrequently updated...
>
> sage

Re: ceph-qa-suite branching (merge it into ceph.git?)

[Vasu Kulkarni]

On Wed, Apr 13, 2016 at 11:46 AM, John Spray <jspray@redhat.com> wrote:
> On Wed, Apr 13, 2016 at 7:02 PM, Vasu Kulkarni <vakulkar@redhat.com> wrote:
>>
>>
>> On Wed, Apr 13, 2016 at 3:52 AM, John Spray <jspray@redhat.com> wrote:
>>>
>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>>> slightly out of sync, as we were adding new stuff and interface
>>> changes to ceph (especially in cephfs).
>>>
>>> We used to have two repos (ceph and teuthology), now we have three
>>> (ceph, ceph-qa-suite and teuthology).
>>
>>
>> we also have s3 suites in its own repo, we have big files in ceph.com/qa
>> which i believe is the right place.
>
> I'm a bit ignorant of the s3 tests, is there a particular reason they
> live separately to the main ceph-qa-suite?

Not sure, probably yehuda wanted to make it easier to test with
Amazon::S3 and Ceph

>
> As for the stuff in ceph.com/qa, I of course am not suggesting putting
> those bit binaries into the main git repo.  It is sort of annoying
> that they live out there on a web server, but it's a separate issue
> from the ceph/ceph-qa-suite one.
>
>>> Splitting tests out of
>>> teuthology was a good thing, but maybe they should have gone into the
>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>>
>>> Historically we had a comparatively static set of workloads in the qa
>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>>> much with ceph changes.  But these days we're adding much more
>>> detailed tests, so there's more effort to keep the two in sync.
>>>
>>> I would personally love to be able to have a single PR that contained
>>> my code and the tests for it.  What if after Jewel we pulled all of
>>> ceph-qa-suite into the ceph repo?
>>
>>
>> I understand the advantage of having same sha between builds and tests, but
>> how are we going to guarantee ceph-builds in different places(one built on
>> other than ceph.com) will have same sha?
>
> The thing that needs to match is the sha1 of the ceph code you're
> running, and the sha1 you use when pulling the code to test it with.
>
> I believe you currently have a situation where you have a built RPM
> but you're not sure what ceph sha1 it came from, so you're pulling
> e.g. the jewel branch of ceph-qa-suite rather than a specific sha1.
> That needs fixing independently of ceph/ceph-qa-suite separation: you
> have to run the right revision of the tests, irrespective of where
> they're getting pulled from.
>
> Basically, anyone generating packages with versions that don't map to
> a ceph sha1 will need to have a way of remembering what ceph sha1
> their packages correspond to.  This is not a new requirement, it's the
> same issue that was discussed on IRC recently.
>
>> why not move ceph tests from ceph.git into ceph-qa-suite,  Ideally this is
>> not an issue with any major release, jewel workunits are supposed to work
>> with jewel ceph,
>
> That would not help, because you still have the same issue of changes
> in ceph being out of step with changes in ceph-qa-suite.
>
> I think your misconception here is to assume that running tests from
> the same branch as the code is sufficient.  You need to run tests from
> the same *revision*.  Otherwise, when I add a new test to reproduce a
> bug, and fix the bug at the same time, then you will get failures if
> you run the newer commit against the older code.

This one I think figured out, I agree it needs to match for those rare
cases, but the option exists in teuthology
with overrides, just need to have right override. So hopefully it
shoudn't be an issue anymore.


>
> John
>
>> only in rare cases it may cause some issue but easily fixable. All the dev
>> runs can point to any qa-sha using the cli option so much less of an issue
>> in rare case.
>
>
>
>>>
>>> We could still enable folks running test changes without having to
>>> rebuild ceph packages: the suite sha1 selected when running a
>>> teuthology suite could still be different from that used for
>>> installing ceph, it's just that it would fetch that sha1 from the ceph
>>> repo instead of from a separate repo.
>>>
>>> John
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>

Re: ceph-qa-suite branching (merge it into ceph.git?)

[Josh Durgin]

On 04/13/2016 11:46 AM, John Spray wrote:
> On Wed, Apr 13, 2016 at 7:02 PM, Vasu Kulkarni <vakulkar@redhat.com> wrote:
>> we also have s3 suites in its own repo, we have big files in ceph.com/qa
>> which i believe is the right place.
>
> I'm a bit ignorant of the s3 tests, is there a particular reason they
> live separately to the main ceph-qa-suite?

s3 tests are more generic. They run against other s3 implementations,
and only a labeled subset work with rgw.

Josh

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Sage Weil]

On Wed, 13 Apr 2016, Vasu Kulkarni wrote:
> On Wed, Apr 13, 2016 at 11:46 AM, John Spray <jspray@redhat.com> wrote:
> > On Wed, Apr 13, 2016 at 7:02 PM, Vasu Kulkarni <vakulkar@redhat.com> wrote:
> >>
> >>
> >> On Wed, Apr 13, 2016 at 3:52 AM, John Spray <jspray@redhat.com> wrote:
> >>>
> >>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
> >>> slightly out of sync, as we were adding new stuff and interface
> >>> changes to ceph (especially in cephfs).
> >>>
> >>> We used to have two repos (ceph and teuthology), now we have three
> >>> (ceph, ceph-qa-suite and teuthology).
> >>
> >>
> >> we also have s3 suites in its own repo, we have big files in ceph.com/qa
> >> which i believe is the right place.
> >
> > I'm a bit ignorant of the s3 tests, is there a particular reason they
> > live separately to the main ceph-qa-suite?
> 
> Not sure, probably yehuda wanted to make it easier to test with
> Amazon::S3 and Ceph

The goal was to make them an independently useful set of functional tests 
for the S3 API, used not just by radosgw but also by other projects, so 
that we are not solely responsible for maintaining them.  I think it makes 
sense to keep them separate.

sage

Re: ceph-qa-suite branching (merge it into ceph.git?)

[Abhishek Lekshmanan]

John Spray writes:

> On Wed, Apr 13, 2016 at 7:02 PM, Vasu Kulkarni <vakulkar@redhat.com> wrote:
>>
>>
>> On Wed, Apr 13, 2016 at 3:52 AM, John Spray <jspray@redhat.com> wrote:
>>>
>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>>> slightly out of sync, as we were adding new stuff and interface
>>> changes to ceph (especially in cephfs).
>>>
>>> We used to have two repos (ceph and teuthology), now we have three
>>> (ceph, ceph-qa-suite and teuthology).
>>
>>
>> we also have s3 suites in its own repo, we have big files in ceph.com/qa
>> which i believe is the right place.
>
> I'm a bit ignorant of the s3 tests, is there a particular reason they
> live separately to the main ceph-qa-suite?

Technically s3-tests can have tests that do not all pass on rgw (which
is true mostly even now as we have a fails_on_rgw filter) and pass on
Amazon S3. I'm guessing this may be a reason they live outside of the
qa-suite, but I could be wrong

Abhishek
>
> As for the stuff in ceph.com/qa, I of course am not suggesting putting
> those bit binaries into the main git repo.  It is sort of annoying
> that they live out there on a web server, but it's a separate issue
> from the ceph/ceph-qa-suite one.
>
>>> Splitting tests out of
>>> teuthology was a good thing, but maybe they should have gone into the
>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>>
>>> Historically we had a comparatively static set of workloads in the qa
>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>>> much with ceph changes.  But these days we're adding much more
>>> detailed tests, so there's more effort to keep the two in sync.
>>>
>>> I would personally love to be able to have a single PR that contained
>>> my code and the tests for it.  What if after Jewel we pulled all of
>>> ceph-qa-suite into the ceph repo?
>>
>>
>> I understand the advantage of having same sha between builds and tests, but
>> how are we going to guarantee ceph-builds in different places(one built on
>> other than ceph.com) will have same sha?
>
> The thing that needs to match is the sha1 of the ceph code you're
> running, and the sha1 you use when pulling the code to test it with.
>
> I believe you currently have a situation where you have a built RPM
> but you're not sure what ceph sha1 it came from, so you're pulling
> e.g. the jewel branch of ceph-qa-suite rather than a specific sha1.
> That needs fixing independently of ceph/ceph-qa-suite separation: you
> have to run the right revision of the tests, irrespective of where
> they're getting pulled from.
>
> Basically, anyone generating packages with versions that don't map to
> a ceph sha1 will need to have a way of remembering what ceph sha1
> their packages correspond to.  This is not a new requirement, it's the
> same issue that was discussed on IRC recently.
>
>> why not move ceph tests from ceph.git into ceph-qa-suite,  Ideally this is
>> not an issue with any major release, jewel workunits are supposed to work
>> with jewel ceph,
>
> That would not help, because you still have the same issue of changes
> in ceph being out of step with changes in ceph-qa-suite.
>
> I think your misconception here is to assume that running tests from
> the same branch as the code is sufficient.  You need to run tests from
> the same *revision*.  Otherwise, when I add a new test to reproduce a
> bug, and fix the bug at the same time, then you will get failures if
> you run the newer commit against the older code.
>
> John
>
>> only in rare cases it may cause some issue but easily fixable. All the dev
>> runs can point to any qa-sha using the cli option so much less of an issue
>> in rare case.
>
>
>
>>>
>>> We could still enable folks running test changes without having to
>>> rebuild ceph packages: the suite sha1 selected when running a
>>> teuthology suite could still be different from that used for
>>> installing ceph, it's just that it would fetch that sha1 from the ceph
>>> repo instead of from a separate repo.
>>>
>>> John
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>


-- 
Abhishek

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Josh Durgin]

On 04/13/2016 11:38 AM, Sage Weil wrote:
> On Wed, 13 Apr 2016, Samuel Just wrote:
>> It also doesn't seem like it would actually present a problem in any case.
>
> The reason we didn't do this before was because we wanted to revise tests
> independently of the thing being tested.  But as John points out,
> specifying a test branch should accomplish that, at least for testing.
>
> It might be nice to preserve the ability specify an alternate repo with
> totally out-of-tree tests.  Maybe that can be done with a simple reorg of
> the repo, like putting everything under qa/, so that tasks/ and suites/
> don't appear at the top level (of ceph.git or ceph-qa-suite.git).

It'll also be quite helpful to specify an alternate repo (not just
branch) for testing suite changes without pushing to the main ceph.git
and triggering gitbuilder builds.

Josh

> sage
>
>
>
>> -Sam
>>
>> On Wed, Apr 13, 2016 at 11:31 AM, John Spray <jspray@redhat.com> wrote:
>>> On Wed, Apr 13, 2016 at 2:58 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>>>> On Wed, Apr 13, 2016 at 6:23 AM, Loic Dachary <loic@dachary.org> wrote:
>>>>> +1 : I agree it would be a good thing.
>>>>>
>>>>> The reason why it would help to merge ceph-qa-suite in ceph is because we have no proper methods or tools to work with interdependent repositories. The problem is not unique to Ceph: every Free Software developer end up bug fixing and adding features to dependencies (ceph-qa-suite in the case of Ceph but also jerasure, rocksdb, s3test etc.). It will take a long time to resolve that more general problem and I don't know about an effort in that direction. Do you ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On 13/04/2016 12:52, John Spray wrote:
>>>>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>>>>>> slightly out of sync, as we were adding new stuff and interface
>>>>>> changes to ceph (especially in cephfs).
>>>>>>
>>>>>> We used to have two repos (ceph and teuthology), now we have three
>>>>>> (ceph, ceph-qa-suite and teuthology).  Splitting tests out of
>>>>>> teuthology was a good thing, but maybe they should have gone into the
>>>>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>>>>>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>>>>>
>>>>>> Historically we had a comparatively static set of workloads in the qa
>>>>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>>>>>> much with ceph changes.  But these days we're adding much more
>>>>>> detailed tests, so there's more effort to keep the two in sync.
>>>>>>
>>>>>> I would personally love to be able to have a single PR that contained
>>>>>> my code and the tests for it.  What if after Jewel we pulled all of
>>>>>> ceph-qa-suite into the ceph repo?
>>>>>>
>>>>>> We could still enable folks running test changes without having to
>>>>>> rebuild ceph packages: the suite sha1 selected when running a
>>>>>> teuthology suite could still be different from that used for
>>>>>> installing ceph, it's just that it would fetch that sha1 from the ceph
>>>>>> repo instead of from a separate repo.
>>>>
>>>> We do have qa-suite tests that don't necessarily make a lot of sense
>>>> to have in ceph.git. Samba, kernel NFS, ganesha some day. That doesn't
>>>> mean we shouldn't merge them, but it popped into my head.
>>>
>>> Fair point.  I think that because those other tasks depend in turn on
>>> the ceph tasks, we still ultimately benefit from having them in one
>>> place.
>>>
>>> It's also possible that in the long run things like the samba tests
>>> become a bit more "smart" in a way that's more tightly coupled to
>>> ceph, e.g. checking the resulting state inside ceph after doing things
>>> in the samba/ganesha layer, at which point we'd enjoy having them in
>>> the same place as the main body of test code.
>>>
>>> John
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> _______________________________________________
>> Ceph-qa mailing list
>> Ceph-qa@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Samuel Just]

We need to do that anyway to make it easier to test ceph-qa-suite branches imho.
-Sam

On Wed, Apr 13, 2016 at 1:12 PM, Josh Durgin <jdurgin@redhat.com> wrote:
> On 04/13/2016 11:38 AM, Sage Weil wrote:
>>
>> On Wed, 13 Apr 2016, Samuel Just wrote:
>>>
>>> It also doesn't seem like it would actually present a problem in any
>>> case.
>>
>>
>> The reason we didn't do this before was because we wanted to revise tests
>> independently of the thing being tested.  But as John points out,
>> specifying a test branch should accomplish that, at least for testing.
>>
>> It might be nice to preserve the ability specify an alternate repo with
>> totally out-of-tree tests.  Maybe that can be done with a simple reorg of
>> the repo, like putting everything under qa/, so that tasks/ and suites/
>> don't appear at the top level (of ceph.git or ceph-qa-suite.git).
>
>
> It'll also be quite helpful to specify an alternate repo (not just
> branch) for testing suite changes without pushing to the main ceph.git
> and triggering gitbuilder builds.
>
> Josh
>
>
>> sage
>>
>>
>>
>>> -Sam
>>>
>>> On Wed, Apr 13, 2016 at 11:31 AM, John Spray <jspray@redhat.com> wrote:
>>>>
>>>> On Wed, Apr 13, 2016 at 2:58 PM, Gregory Farnum <gfarnum@redhat.com>
>>>> wrote:
>>>>>
>>>>> On Wed, Apr 13, 2016 at 6:23 AM, Loic Dachary <loic@dachary.org> wrote:
>>>>>>
>>>>>> +1 : I agree it would be a good thing.
>>>>>>
>>>>>> The reason why it would help to merge ceph-qa-suite in ceph is because
>>>>>> we have no proper methods or tools to work with interdependent repositories.
>>>>>> The problem is not unique to Ceph: every Free Software developer end up bug
>>>>>> fixing and adding features to dependencies (ceph-qa-suite in the case of
>>>>>> Ceph but also jerasure, rocksdb, s3test etc.). It will take a long time to
>>>>>> resolve that more general problem and I don't know about an effort in that
>>>>>> direction. Do you ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On 13/04/2016 12:52, John Spray wrote:
>>>>>>>
>>>>>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>>>>>>> slightly out of sync, as we were adding new stuff and interface
>>>>>>> changes to ceph (especially in cephfs).
>>>>>>>
>>>>>>> We used to have two repos (ceph and teuthology), now we have three
>>>>>>> (ceph, ceph-qa-suite and teuthology).  Splitting tests out of
>>>>>>> teuthology was a good thing, but maybe they should have gone into the
>>>>>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems
>>>>>>> to
>>>>>>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>>>>>>
>>>>>>> Historically we had a comparatively static set of workloads in the qa
>>>>>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>>>>>>> much with ceph changes.  But these days we're adding much more
>>>>>>> detailed tests, so there's more effort to keep the two in sync.
>>>>>>>
>>>>>>> I would personally love to be able to have a single PR that contained
>>>>>>> my code and the tests for it.  What if after Jewel we pulled all of
>>>>>>> ceph-qa-suite into the ceph repo?
>>>>>>>
>>>>>>> We could still enable folks running test changes without having to
>>>>>>> rebuild ceph packages: the suite sha1 selected when running a
>>>>>>> teuthology suite could still be different from that used for
>>>>>>> installing ceph, it's just that it would fetch that sha1 from the
>>>>>>> ceph
>>>>>>> repo instead of from a separate repo.
>>>>>
>>>>>
>>>>> We do have qa-suite tests that don't necessarily make a lot of sense
>>>>> to have in ceph.git. Samba, kernel NFS, ganesha some day. That doesn't
>>>>> mean we shouldn't merge them, but it popped into my head.
>>>>
>>>>
>>>> Fair point.  I think that because those other tasks depend in turn on
>>>> the ceph tasks, we still ultimately benefit from having them in one
>>>> place.
>>>>
>>>> It's also possible that in the long run things like the samba tests
>>>> become a bit more "smart" in a way that's more tightly coupled to
>>>> ceph, e.g. checking the resulting state inside ceph after doing things
>>>> in the samba/ganesha layer, at which point we'd enjoy having them in
>>>> the same place as the main body of test code.
>>>>
>>>> John
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> _______________________________________________
>>> Ceph-qa mailing list
>>> Ceph-qa@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Samuel Just]

Ah, I didn't know about that, I'll have to try it.
-Sam

On Wed, Apr 13, 2016 at 2:09 PM, Loic Dachary <loic@dachary.org> wrote:
>
>
> On 13/04/2016 22:12, Josh Durgin wrote:
>> On 04/13/2016 11:38 AM, Sage Weil wrote:
>>> On Wed, 13 Apr 2016, Samuel Just wrote:
>>>> It also doesn't seem like it would actually present a problem in any case.
>>>
>>> The reason we didn't do this before was because we wanted to revise tests
>>> independently of the thing being tested.  But as John points out,
>>> specifying a test branch should accomplish that, at least for testing.
>>>
>>> It might be nice to preserve the ability specify an alternate repo with
>>> totally out-of-tree tests.  Maybe that can be done with a simple reorg of
>>> the repo, like putting everything under qa/, so that tasks/ and suites/
>>> don't appear at the top level (of ceph.git or ceph-qa-suite.git).
>>
>> It'll also be quite helpful to specify an alternate repo (not just
>> branch) for testing suite changes without pushing to the main ceph.git
>> and triggering gitbuilder builds.
>
> We can keep the current --ceph-qa-suite-git-url argument for that. Or do you have something else in mind ?
>
>> Josh
>>
>>> sage
>>>
>>>
>>>
>>>> -Sam
>>>>
>>>> On Wed, Apr 13, 2016 at 11:31 AM, John Spray <jspray@redhat.com> wrote:
>>>>> On Wed, Apr 13, 2016 at 2:58 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>>>>>> On Wed, Apr 13, 2016 at 6:23 AM, Loic Dachary <loic@dachary.org> wrote:
>>>>>>> +1 : I agree it would be a good thing.
>>>>>>>
>>>>>>> The reason why it would help to merge ceph-qa-suite in ceph is because we have no proper methods or tools to work with interdependent repositories. The problem is not unique to Ceph: every Free Software developer end up bug fixing and adding features to dependencies (ceph-qa-suite in the case of Ceph but also jerasure, rocksdb, s3test etc.). It will take a long time to resolve that more general problem and I don't know about an effort in that direction. Do you ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> On 13/04/2016 12:52, John Spray wrote:
>>>>>>>> Lately, we've instances where ceph.git and ceph-qa-suite.git got
>>>>>>>> slightly out of sync, as we were adding new stuff and interface
>>>>>>>> changes to ceph (especially in cephfs).
>>>>>>>>
>>>>>>>> We used to have two repos (ceph and teuthology), now we have three
>>>>>>>> (ceph, ceph-qa-suite and teuthology).  Splitting tests out of
>>>>>>>> teuthology was a good thing, but maybe they should have gone into the
>>>>>>>> ceph tree instead of a new repo?  The ceph-qa-suite branching seems to
>>>>>>>> pretty much mirror what we do with ceph, with master vs jewel etc.
>>>>>>>>
>>>>>>>> Historically we had a comparatively static set of workloads in the qa
>>>>>>>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change
>>>>>>>> much with ceph changes.  But these days we're adding much more
>>>>>>>> detailed tests, so there's more effort to keep the two in sync.
>>>>>>>>
>>>>>>>> I would personally love to be able to have a single PR that contained
>>>>>>>> my code and the tests for it.  What if after Jewel we pulled all of
>>>>>>>> ceph-qa-suite into the ceph repo?
>>>>>>>>
>>>>>>>> We could still enable folks running test changes without having to
>>>>>>>> rebuild ceph packages: the suite sha1 selected when running a
>>>>>>>> teuthology suite could still be different from that used for
>>>>>>>> installing ceph, it's just that it would fetch that sha1 from the ceph
>>>>>>>> repo instead of from a separate repo.
>>>>>>
>>>>>> We do have qa-suite tests that don't necessarily make a lot of sense
>>>>>> to have in ceph.git. Samba, kernel NFS, ganesha some day. That doesn't
>>>>>> mean we shouldn't merge them, but it popped into my head.
>>>>>
>>>>> Fair point.  I think that because those other tasks depend in turn on
>>>>> the ceph tasks, we still ultimately benefit from having them in one
>>>>> place.
>>>>>
>>>>> It's also possible that in the long run things like the samba tests
>>>>> become a bit more "smart" in a way that's more tightly coupled to
>>>>> ceph, e.g. checking the resulting state inside ceph after doing things
>>>>> in the samba/ganesha layer, at which point we'd enjoy having them in
>>>>> the same place as the main body of test code.
>>>>>
>>>>> John
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> _______________________________________________
>>>> Ceph-qa mailing list
>>>> Ceph-qa@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>
> --
> Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Josh Durgin]

On 04/13/2016 02:09 PM, Loic Dachary wrote:
>
>
> On 13/04/2016 22:12, Josh Durgin wrote:
>> On 04/13/2016 11:38 AM, Sage Weil wrote:
>>> On Wed, 13 Apr 2016, Samuel Just wrote:
>>>> It also doesn't seem like it would actually present a problem in any case.
>>>
>>> The reason we didn't do this before was because we wanted to revise tests
>>> independently of the thing being tested.  But as John points out,
>>> specifying a test branch should accomplish that, at least for testing.
>>>
>>> It might be nice to preserve the ability specify an alternate repo with
>>> totally out-of-tree tests.  Maybe that can be done with a simple reorg of
>>> the repo, like putting everything under qa/, so that tasks/ and suites/
>>> don't appear at the top level (of ceph.git or ceph-qa-suite.git).
>>
>> It'll also be quite helpful to specify an alternate repo (not just
>> branch) for testing suite changes without pushing to the main ceph.git
>> and triggering gitbuilder builds.
>
> We can keep the current --ceph-qa-suite-git-url argument for that. Or do you have something else in mind ?

I wasn't aware of that - looks like it's just a teuthology-openstack
option. I was thinking we should have the same option for teuthology-suite.

Re: [Ceph-qa] ceph-qa-suite branching (merge it into ceph.git?)

[Loic Dachary]

On 13/04/2016 23:17, Josh Durgin wrote:
> On 04/13/2016 02:09 PM, Loic Dachary wrote:
>>
>>
>> On 13/04/2016 22:12, Josh Durgin wrote:
>>> On 04/13/2016 11:38 AM, Sage Weil wrote:
>>>> On Wed, 13 Apr 2016, Samuel Just wrote:
>>>>> It also doesn't seem like it would actually present a problem in any case.
>>>>
>>>> The reason we didn't do this before was because we wanted to revise tests
>>>> independently of the thing being tested.  But as John points out,
>>>> specifying a test branch should accomplish that, at least for testing.
>>>>
>>>> It might be nice to preserve the ability specify an alternate repo with
>>>> totally out-of-tree tests.  Maybe that can be done with a simple reorg of
>>>> the repo, like putting everything under qa/, so that tasks/ and suites/
>>>> don't appear at the top level (of ceph.git or ceph-qa-suite.git).
>>>
>>> It'll also be quite helpful to specify an alternate repo (not just
>>> branch) for testing suite changes without pushing to the main ceph.git
>>> and triggering gitbuilder builds.
>>
>> We can keep the current --ceph-qa-suite-git-url argument for that. Or do you have something else in mind ?
> 
> I wasn't aware of that - looks like it's just a teuthology-openstack
> option. I was thinking we should have the same option for teuthology-suite.

Right. The teuthology-openstack option sets the ceph_qa_suite_git_url configuration option from ~/.teuthology.yaml which is not OpenStack specific. A naive implementation in teuthology-suite would be to replace the value in ~/.teuthology.yaml in the same way teuthology-openstack does it. It would however be problematic for a long lived cluster with multiple users racing with each other. A better implementation would be to have that as part of the config.yaml.

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html