2 patch dependency

2 patch dependency

[Vijay Khemka]

Hi Brad,
We have issue of merging 2 commit which are dependent to each other. I am not sure how are we going address this. 
https://gerrit.openbmc-project.xyz/#/c/openbmc/meta-facebook/+/26629/ and 
https://gerrit.openbmc-project.xyz/#/c/openbmc/meta-phosphor/+/25651/

Regards
-Vijay

Re: 2 patch dependency

[Brad Bishop]

> On Nov 1, 2019, at 3:18 PM, Vijay Khemka <vijaykhemka@fb.com> wrote:
> 
> Hi Brad,
> We have issue of merging 2 commit which are dependent to each other. I am not sure how are we going address this. 
> https://gerrit.openbmc-project.xyz/#/c/openbmc/meta-facebook/+/26629/ and 
> https://gerrit.openbmc-project.xyz/#/c/openbmc/meta-phosphor/+/25651/
> 
> Regards
> -Vijay
> 

Hi Vijay

I’m working on this today:

https://gerrit.openbmc-project.xyz/c/openbmc/openbmc/+/26910

-brad

Re: 2 patch dependency

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

On Tue, Nov 05, 2019 at 09:27:21AM -0500, Brad Bishop wrote:
> 
> 
> > On Nov 1, 2019, at 3:18 PM, Vijay Khemka <vijaykhemka@fb.com> wrote:
> > 
> > We have issue of merging 2 commit which are dependent to each other. I am not sure how are we going address this. 
> 
> I’m working on this today:
> 
> https://gerrit.openbmc-project.xyz/c/openbmc/openbmc/+/26910

Do we have any sort of topic-based multi-repo testing?  If not, is there
any interest in developing it?

I implemented this on another project:
   * Whenever Jenkins was triggered by Gerrit, it looked at the
     GERRIT_TOPIC variable to determine if the change belonged to a
     topic.
   * If the commit was in a topic, Jenkins queried Gerrit for all open
     changes that belonged to the same topic.  Using `repo`, Jenkins
     would extract all of these changes into a working directory and
     `bitbake` to kick off a build.

This was useful both commits that spanned metas and for commits that
involved both code and recipe changes.  This was really useful for
changes that added a new dependency so you didn't have to add a
free-standing recipe update with a "trust me, the code is coming that is
going to need this dependency" kind of comment.

I know we're not using `repo` right now and so there is some work
involved in making this span the OpenBMC code repositories, but a simple
first step would be to get this working across the meta subtrees.  Would
this help changes like Vijay's so we don't need to involve a cross-meta
maintainer?

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: 2 patch dependency

[Andrew Geissler]

> On Nov 18, 2019, at 6:35 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
> On Tue, Nov 05, 2019 at 09:27:21AM -0500, Brad Bishop wrote:
>> 
>> 
>>> On Nov 1, 2019, at 3:18 PM, Vijay Khemka <vijaykhemka@fb.com> wrote:
>>> 
>>> We have issue of merging 2 commit which are dependent to each other. I am not sure how are we going address this. 
>> 
>> I’m working on this today:
>> 
>> https://gerrit.openbmc-project.xyz/c/openbmc/openbmc/+/26910
> 
> Do we have any sort of topic-based multi-repo testing?  If not, is there
> any interest in developing it?

I know in general we really try to encourage people to not cause
co-req issues across repositories. Making this easier for people
may cause more of it to happen.

Also, I believe we’re going back to just having everything in
openbmc/openbmc so that should alleviate a lot of our meta-*
co-req issues since they can all go in together again. Brad was
working on getting the gerrit plugin running that allows people
to maintain sub-directories in a repository. This will allow multiple
maintainers of openbmc/openbmc for their specific areas.

So to answer your question, I’m not sure. I think it could still
be useful to have between code repos and openbmc/openbmc
at times but I’d rather people just do the extra work to avoid
dependencies all together.

> 
> I implemented this on another project:
>   * Whenever Jenkins was triggered by Gerrit, it looked at the
>     GERRIT_TOPIC variable to determine if the change belonged to a
>     topic.
>   * If the commit was in a topic, Jenkins queried Gerrit for all open
>     changes that belonged to the same topic.  Using `repo`, Jenkins
>     would extract all of these changes into a working directory and
>     `bitbake` to kick off a build.
> 
> This was useful both commits that spanned metas and for commits that
> involved both code and recipe changes.  This was really useful for
> changes that added a new dependency so you didn't have to add a
> free-standing recipe update with a "trust me, the code is coming that is
> going to need this dependency" kind of comment.
> 
> I know we're not using `repo` right now and so there is some work
> involved in making this span the OpenBMC code repositories, but a simple
> first step would be to get this working across the meta subtrees.  Would
> this help changes like Vijay's so we don't need to involve a cross-meta
> maintainer?
> 
> -- 
> Patrick Williams

Re: 2 patch dependency

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1613 bytes --]

On Tue, Nov 19, 2019 at 09:45:34AM -0600, Andrew Geissler wrote:
> 
> I know in general we really try to encourage people to not cause
> co-req issues across repositories. Making this easier for people
> may cause more of it to happen.

I understand and agree.  I don't know how often this kind of breaking
behavior has happened, like these IPMI related changes, it was just
coincidentally at the top of my email queue as I've started digging
through.

> Also, I believe we’re going back to just having everything in
> openbmc/openbmc so that should alleviate a lot of our meta-*
> co-req issues since they can all go in together again. Brad was
> working on getting the gerrit plugin running that allows people
> to maintain sub-directories in a repository. This will allow multiple
> maintainers of openbmc/openbmc for their specific areas.

Ooh.  Was this talked about on the mailing list somewhere?  I've gone
back through at least August and haven't come across it.  Was there
an exploration of moving to `repo` instead?

> So to answer your question, I’m not sure. I think it could still
> be useful to have between code repos and openbmc/openbmc
> at times but I’d rather people just do the extra work to avoid
> dependencies all together.

Another "first step" might be to get a Jenkins job for code repositories
that builds the full openbmc/openbmc stack with the code change, so that
we can identify breaking changes like this before they are merged.
AFAIK, the code repo Jenkins jobs are still independent from the Yocto
recipes, correct?

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: 2 patch dependency

[Andrew Geissler]

> On Nov 19, 2019, at 10:54 AM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
> On Tue, Nov 19, 2019 at 09:45:34AM -0600, Andrew Geissler wrote:
>> 
>> I know in general we really try to encourage people to not cause
>> co-req issues across repositories. Making this easier for people
>> may cause more of it to happen.
> 
> I understand and agree.  I don't know how often this kind of breaking
> behavior has happened, like these IPMI related changes, it was just
> coincidentally at the top of my email queue as I've started digging
> through.
> 
>> Also, I believe we’re going back to just having everything in
>> openbmc/openbmc so that should alleviate a lot of our meta-*
>> co-req issues since they can all go in together again. Brad was
>> working on getting the gerrit plugin running that allows people
>> to maintain sub-directories in a repository. This will allow multiple
>> maintainers of openbmc/openbmc for their specific areas.
> 
> Ooh.  Was this talked about on the mailing list somewhere?  I've gone
> back through at least August and haven't come across it.  Was there
> an exploration of moving to `repo` instead?

May have only been IRC, I’ll let Brad handle this one. I just know
that my vote is to get away from subtree. It made the dev process
more difficult/complex and it doubled the required capacity of our CI
infrastructure.

> 
>> So to answer your question, I’m not sure. I think it could still
>> be useful to have between code repos and openbmc/openbmc
>> at times but I’d rather people just do the extra work to avoid
>> dependencies all together.
> 
> Another "first step" might be to get a Jenkins job for code repositories
> that builds the full openbmc/openbmc stack with the code change, so that
> we can identify breaking changes like this before they are merged.

Yes, this is definitely on the list. We run redfish validation on all
images and it continues to find good regressions but that’s always
after the code has been merged into bmcweb and pushed up to
meta-phosphor. We would like to at least enable this feature
(full flash image build) for bmcweb so we can get this
coverage earlier. Once we get rid of the meta-* layer CI
we should have enough compute power to do this.

> AFAIK, the code repo Jenkins jobs are still independent from the Yocto
> recipes, correct?

Yes, still independent. So we’d run that normal code repo CI
and if that passed, do a devtool or recipe update to build
the change into a bitbake image for further CI.

> 
> -- 
> Patrick Williams

Re: 2 patch dependency

[Brad Bishop]

Sorry - been a bit behind on email lately.

>> Ooh.  Was this talked about on the mailing list somewhere?  I've gone
>> back through at least August and haven't come across it.  Was there
>> an exploration of moving to `repo` instead?
> 
> May have only been IRC, I’ll let Brad handle this one. I just know
> that my vote is to get away from subtree. It made the dev process
> more difficult/complex and it doubled the required capacity of our CI
> infrastructure.

Right…I didn’t think this had left my head.  Apparently I told Andrew about it :-)

A brief history lesson for any that aren’t aware - the original workflow was patches were submitted to openbmc/openbmc for subtrees where we are as far upstream as you can get.  We moved from that to the workflow we have today because of a desire of mine and others for de-centralized ownership of metadata.

To enable that I proposed the same process that the Yocto project uses to aggregate the various sub-projects into the poky distribution.  This is how we wound up with the workflow we have today.  In hindsight this was a mistake.

The proposed change would simply be that we revert back to the old workflow, which was much simpler, and use the Gerrit plugin to implement the de-centralized ownership requirement.  There have been and will be a couple enhancements since we last used that workflow though:

1 - I track poky/meta-openembedded head. Last time we tracked the latest released version.  This has worked fine; I am not aware of a single instance of upstream breaking us or causing any instability.

2 - I would automate the subtree pushes. Last time this was done manually (and thus the subtrees were often stale).

As far as repo goes - in my mind that is something completely separate.  If anyone wants to maintain a repo manifest somewhere I don’t have a problem with that.  It doesn’t have to have anything to do with the patch submission workflow.

thx - brad

Re: 2 patch dependency

[Brad Bishop]

> Another "first step" might be to get a Jenkins job for code repositories
> that builds the full openbmc/openbmc stack with the code change, so that
> we can identify breaking changes like this before they are merged.
> AFAIK, the code repo Jenkins jobs are still independent from the Yocto
> recipes, correct?

FWIW I would love to get here.  Is there a way to do it without cratering the resources we have?

Re: 2 patch dependency

[Brad Bishop]

> On Nov 18, 2019, at 7:35 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
> On Tue, Nov 05, 2019 at 09:27:21AM -0500, Brad Bishop wrote:
>> 
>> 
>>> On Nov 1, 2019, at 3:18 PM, Vijay Khemka <vijaykhemka@fb.com> wrote:
>>> 
>>> We have issue of merging 2 commit which are dependent to each other. I am not sure how are we going address this. 
>> 
>> I’m working on this today:
>> 
>> https://gerrit.openbmc-project.xyz/c/openbmc/openbmc/+/26910
> 
> Do we have any sort of topic-based multi-repo testing?  If not, is there
> any interest in developing it?

I’m certainly interested in having this capability.

Re: 2 patch dependency

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 3105 bytes --]

Thanks Brad.

On Fri, Nov 22, 2019 at 10:44:16AM -0500, Brad Bishop wrote:

> A brief history lesson for any that aren’t aware - the original workflow was patches were submitted to openbmc/openbmc for subtrees where we are as far upstream as you can get.  We moved from that to the workflow we have today because of a desire of mine and others for de-centralized ownership of metadata.
> 
> To enable that I proposed the same process that the Yocto project uses to aggregate the various sub-projects into the poky distribution.  This is how we wound up with the workflow we have today.  In hindsight this was a mistake.

Can you elaborate briefly on the issues you've had with it?  It seems
like many metas with subtree, many metas with repo, or mono-repo all have
advantages and disadvantages and are just different ways people have
attempted to skin the same animal.  (There is also submodule, but
*yuck*...).

> The proposed change would simply be that we revert back to the old workflow, which was much simpler, and use the Gerrit plugin to implement the de-centralized ownership requirement.  There have been and will be a couple enhancements since we last used that workflow though:
> 
> 1 - I track poky/meta-openembedded head. Last time we tracked the latest released version.  This has worked fine; I am not aware of a single instance of upstream breaking us or causing any instability.

Is there any interest in supporting multiple Yocto releases at a time?
This would be one advantage of using repo.  You can have different
manifest files for the different Yocto branches.  That way we can keep
pointing to an LTS Yocto but with newer OpenBMC code (for people who
might desire that when putting OpenBMC into production).

> 
> 2 - I would automate the subtree pushes. Last time this was done manually (and thus the subtrees were often stale).

:+1:

> As far as repo goes - in my mind that is something completely separate.  If anyone wants to maintain a repo manifest somewhere I don’t have a problem with that.  It doesn’t have to have anything to do with the patch submission workflow.

I'm not really following how any of the options have a big impact on patch
submission workflow.  Most developers just contribute to whatever
repository is considered the "master" of where they're making changes.
If that is a subtree or a monorepo, does it really have much bearing?
(I guess it has impact in the cases where we need to span what are
currently subtrees?)

Having a repo manifest simplifies two things:
    1. Implementation of cross-repo CI (and the topic-based testing I
       mentioned) because there is a clear assembling of all the git
       repositories that might be impacted by a topic.

    2. A mechanism for developers to "get all the code".  I've heard
       complaints of developers having to manually assemble a collection
       of git-clones for all the different sub-projects they tend to
       work on.  A maintained manifest of all the openbmc/* repositories
       would alleviate this.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: 2 patch dependency

[Brad Bishop]

> On Nov 22, 2019, at 12:36 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
> Thanks Brad.
> 
> On Fri, Nov 22, 2019 at 10:44:16AM -0500, Brad Bishop wrote:
> 
>> A brief history lesson for any that aren’t aware - the original workflow was patches were submitted to openbmc/openbmc for subtrees where we are as far upstream as you can get.  We moved from that to the workflow we have today because of a desire of mine and others for de-centralized ownership of metadata.
>> 
>> To enable that I proposed the same process that the Yocto project uses to aggregate the various sub-projects into the poky distribution.  This is how we wound up with the workflow we have today.  In hindsight this was a mistake.
> 
> Can you elaborate briefly on the issues you've had with it?  

Well for one, the issue that started this thread - coreqs.  they are harder than they need to be with the current workflow.

Also the two step CI verification that comes with it is not good - it halves our already limited resources.  Although now that I think about it I don’t recall why we felt this was necessary…

Finally it is just awkward and lame to clone openbmc/openbmc, hack on it, and then have mess around with adding additional remotes and doing subtree cherry-picks, etc… to submit a patch.

> It seems 
> like many metas with subtree, many metas with repo, or mono-repo all have
> advantages and disadvantages and are just different ways people have
> attempted to skin the same animal.  

Could we not have all of them at the same time?

> 
>> The proposed change would simply be that we revert back to the old workflow, which was much simpler, and use the Gerrit plugin to implement the de-centralized ownership requirement.  There have been and will be a couple enhancements since we last used that workflow though:
>> 
>> 1 - I track poky/meta-openembedded head. Last time we tracked the latest released version.  This has worked fine; I am not aware of a single instance of upstream breaking us or causing any instability.
> 
> Is there any interest in supporting multiple Yocto releases at a time?

I don’t have a need for this.

> This would be one advantage of using repo.  

The word advantage implies repo would replace something else.  What do you want it to replace?

> You can have different
> manifest files for the different Yocto branches.  That way we can keep
> pointing to an LTS Yocto but with newer OpenBMC code (for people who
> might desire that when putting OpenBMC into production).

makes sense.

> 
> 
>> As far as repo goes - in my mind that is something completely separate.  If anyone wants to maintain a repo manifest somewhere I don’t have a problem with that.  It doesn’t have to have anything to do with the patch submission workflow.
> 
> I'm not really following how any of the options have a big impact on patch
> submission workflow.  

agreed.  In fact…is there anything you need from the community to start maintaining a repo manifest today?  I don’t think there is, is there?  Maybe a git repo to put it in?

> Most developers just contribute to whatever
> repository is considered the "master" of where they're making changes.
> If that is a subtree or a monorepo, does it really have much bearing?
> (I guess it has impact in the cases where we need to span what are
> currently subtrees?)
> 
> Having a repo manifest simplifies two things:
>    1. Implementation of cross-repo CI (and the topic-based testing I
>       mentioned) because there is a clear assembling of all the git
>       repositories that might be impacted by a topic.
> 
>    2. A mechanism for developers to "get all the code".  I've heard
>       complaints of developers having to manually assemble a collection
>       of git-clones for all the different sub-projects they tend to
>       work on.  A maintained manifest of all the openbmc/* repositories
>       would alleviate this.

I think you are driving at a couple things:

1 - patch submitters should continue to submit patches to meta-*, poky, meta-openembedded, etc.
2 - you start maintaining a repo manifest.
3 - everyone will abandon the current monorepo composed of subtrees.

Do I have it right?

What I’m asking for instead is:

1 - patch submitters submit patches to openbmc/openbmc (unless there is an upstream community)
2 - patches (immediately) find their way into meta-* via automation
3 - you start maintaining a repo manifest
4 - I continue to maintain a monorepo composed of subtrees (the ones with upstream communities) & directly submitted patches

thx - brad

Re: 2 patch dependency

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 6449 bytes --]

On Fri, Nov 22, 2019 at 01:41:14PM -0500, Brad Bishop wrote:
> 
> > On Nov 22, 2019, at 12:36 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> > 
> > Can you elaborate briefly on the issues you've had with it?  
> 
> Well for one, the issue that started this thread - coreqs.  they are harder than they need to be with the current workflow.
> 
> Also the two step CI verification that comes with it is not good - it halves our already limited resources.  Although now that I think about it I don’t recall why we felt this was necessary…

I understand the co-req issues.  We'll still have them between code
repositories (which was really the underlying issue in this case).  But,
eliminating the sub-trees certainly alleviates one introduction of them.

To be clear, I personally have no attachment to the sub-trees or mono-repo
from an openbmc/openbmc perspective.  Just trying to offer alternative
ideas before there are changes.

By "already limited resources", I assume you mean compute resources for
CI.  It seems with the companies involved in this project, this should
not be an issue... is this something the TSC should discuss?

> Finally it is just awkward and lame to clone openbmc/openbmc, hack on it, and then have mess around with adding additional remotes and doing subtree cherry-picks, etc… to submit a patch.

No disagreement at all here.

> > It seems 
> > like many metas with subtree, many metas with repo, or mono-repo all have
> > advantages and disadvantages and are just different ways people have
> > attempted to skin the same animal.  
> 
> Could we not have all of them at the same time?
> 

This is entirely a "without evidence" statement but my intuition would
be trying to do all of them gives you all the disadvantages and few of
the advantages...

Perhaps a higher-level discussion is what purpose do the meta sub-trees
serve and is it worth the project investment in them?

My recollection was that people asked for sub-trees (the old split back out
of openbmc/openbmc method) because they wanted to pick up some subset of
recipes to use in other projects.  Why should our project's day-to-day
activity be less efficient to satisfy that?  (+1 to mono-repo).

I recognize the subtree maintainers desire was a reason for moving to
that structure, which maybe this MAINTAINERS plugin addresses.

> > This would be one advantage of using repo.  
> 
> The word advantage implies repo would replace something else.  What do you want it to replace?

Good point.  A repo manifest could replace the current combined metas
that is openbmc/openbmc and keep today's subtree maintainer model as-is.
Again, I was just trying to present it as an alternative to going back
to the "old model" of openbmc/openbmc being the master.

> 
> > You can have different
> > manifest files for the different Yocto branches.  That way we can keep
> > pointing to an LTS Yocto but with newer OpenBMC code (for people who
> > might desire that when putting OpenBMC into production).
> 
> makes sense.
> 
> > 
> > 
> >> As far as repo goes - in my mind that is something completely separate.  If anyone wants to maintain a repo manifest somewhere I don’t have a problem with that.  It doesn’t have to have anything to do with the patch submission workflow.
> > 
> > I'm not really following how any of the options have a big impact on patch
> > submission workflow.  
> 
> agreed.  In fact…is there anything you need from the community to start maintaining a repo manifest today?  I don’t think there is, is there?  Maybe a git repo to put it in?
> 

There are two scopes to a potential repo-manifest (in the context of
this project specifically):

   1) A manifest that combines the meta-subtrees into a single
      directory tree.

   2) A manifest that combines openbmc/openbmc plus the project's code
      repositories into a single directory tree.

If we are going back to a "openbmc/openbmc is the primary source" model
#1 is pointless.  Also having an alternative Yocto branch is difficult
without #1.

Yes, #2 can be done today (with a new repository).  I think it would
mostly be a developer-help initially but could be rolled into
simplifying CI.  (We could also do #1 and #2.)

I'm mostly focused on #1 in this email thread because has impacts to
this subtree elimination proposal.  It is probably worth another thread
if we want to discuss #2 in more detail.

> > Most developers just contribute to whatever
> > repository is considered the "master" of where they're making changes.
> > If that is a subtree or a monorepo, does it really have much bearing?
> > (I guess it has impact in the cases where we need to span what are
> > currently subtrees?)
> > 
> > Having a repo manifest simplifies two things:
> >    1. Implementation of cross-repo CI (and the topic-based testing I
> >       mentioned) because there is a clear assembling of all the git
> >       repositories that might be impacted by a topic.
> > 
> >    2. A mechanism for developers to "get all the code".  I've heard
> >       complaints of developers having to manually assemble a collection
> >       of git-clones for all the different sub-projects they tend to
> >       work on.  A maintained manifest of all the openbmc/* repositories
> >       would alleviate this.
> 
> I think you are driving at a couple things:
> 
> 1 - patch submitters should continue to submit patches to meta-*, poky, meta-openembedded, etc.
> 2 - you start maintaining a repo manifest.
> 3 - everyone will abandon the current monorepo composed of subtrees.
> 
> Do I have it right?
> 
> What I’m asking for instead is:
> 
> 1 - patch submitters submit patches to openbmc/openbmc (unless there is an upstream community)
> 2 - patches (immediately) find their way into meta-* via automation
> 3 - you start maintaining a repo manifest
> 4 - I continue to maintain a monorepo composed of subtrees (the ones with upstream communities) & directly submitted patches
> 

I think we're on the same page with both of these last two "processes"
you wrote.  Hopefully my #1/#2 above clarifies my thinking.  Except "you
start" doesn't have to be *me*, but I could certainly pull together a
manifest file pretty quickly.  (As in, I don't need to 'maintain' it but
I can put in some effort to get the ball rolling.)

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]