* Upstreaming downstream Google BMC repositories @ 2021-01-07 1:49 Brandon Kim 2021-01-07 8:01 ` Paul Menzel 0 siblings, 1 reply; 12+ messages in thread From: Brandon Kim @ 2021-01-07 1:49 UTC (permalink / raw) To: OpenBMC (openbmc@lists.ozlabs.org) [-- Attachment #1: Type: text/plain, Size: 627 bytes --] Hi everyone, We're exploring ways of upstreaming some of the downstream repositories from Google to openbmc/* . Half, if not most of the downstream repositories are C++ daemons that are specific to Google so we didn't want to create a bunch of new openbmc/<repo> that no one would use. An idea that Ed gave me was having something like openbmc/google-misc repository for all these repositories and if there are any that seem useful to others, we can break it out into a different, separate repository in openbmc/* layer. Please let me know if this seems like a good idea and I'm open to other suggestions! Thanks, Brandon [-- Attachment #2: Type: text/html, Size: 786 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-07 1:49 Upstreaming downstream Google BMC repositories Brandon Kim @ 2021-01-07 8:01 ` Paul Menzel 2021-01-07 17:33 ` Benjamin Fair 0 siblings, 1 reply; 12+ messages in thread From: Paul Menzel @ 2021-01-07 8:01 UTC (permalink / raw) To: Brandon Kim; +Cc: openbmc Dear Brandon, Am 07.01.21 um 02:49 schrieb Brandon Kim: > We're exploring ways of upstreaming some of the downstream repositories > from Google to openbmc/* . > > Half, if not most of the downstream repositories are C++ daemons that are > specific to Google so we didn't want to create a bunch of new > openbmc/<repo> that no one would use. > > An idea that Ed gave me was having something like openbmc/google-misc > repository for all these repositories and if there are any that seem useful > to others, we can break it out into a different, separate repository in > openbmc/* layer. > > Please let me know if this seems like a good idea and I'm open to other > suggestions! Thank you very much for putting in the effort to make these repositories public. Using the openbmc/google-misc approach, how would the git history (commit log) be handled? Personally, I would prefer having small repositories as git makes that very easy to handle. Also it might save you time, as you do not have to think about what to do with the git history, and do not have to merge it. Kind regards, Paul ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-07 8:01 ` Paul Menzel @ 2021-01-07 17:33 ` Benjamin Fair 2021-01-07 18:25 ` Paul Menzel 0 siblings, 1 reply; 12+ messages in thread From: Benjamin Fair @ 2021-01-07 17:33 UTC (permalink / raw) To: Paul Menzel; +Cc: Brandon Kim, OpenBMC Maillist Hi Paul, On Thu, 7 Jan 2021 at 00:09, Paul Menzel <pmenzel@molgen.mpg.de> wrote: > > Dear Brandon, > > > Am 07.01.21 um 02:49 schrieb Brandon Kim: > > > We're exploring ways of upstreaming some of the downstream repositories > > from Google to openbmc/* . > > > > Half, if not most of the downstream repositories are C++ daemons that are > > specific to Google so we didn't want to create a bunch of new > > openbmc/<repo> that no one would use. > > > > An idea that Ed gave me was having something like openbmc/google-misc > > repository for all these repositories and if there are any that seem useful > > to others, we can break it out into a different, separate repository in > > openbmc/* layer. > > > > Please let me know if this seems like a good idea and I'm open to other > > suggestions! > > Thank you very much for putting in the effort to make these repositories > public. > > Using the openbmc/google-misc approach, how would the git history > (commit log) be handled? > > Personally, I would prefer having small repositories as git makes that > very easy to handle. Also it might save you time, as you do not have to > think about what to do with the git history, and do not have to merge it. We would most likely squash the history together, in case there's something confidential or private in the earlier commits. Many small repos would be easy to handle for us, but OpenBMC may not want to have lots of small Google-specific repos in their org as this may make it more cumbersome for others to find the relevant repos that they're interested in. There's also overhead for the project maintainers to create the relevant groups and permissions for each new repo. > > > Kind regards, > > Paul ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-07 17:33 ` Benjamin Fair @ 2021-01-07 18:25 ` Paul Menzel 2021-01-07 21:20 ` Ed Tanous 0 siblings, 1 reply; 12+ messages in thread From: Paul Menzel @ 2021-01-07 18:25 UTC (permalink / raw) To: Benjamin Fair; +Cc: Brandon Kim, openbmc Dear Benjamin, Am 07.01.21 um 18:33 schrieb Benjamin Fair: > On Thu, 7 Jan 2021 at 00:09, Paul Menzel <pmenzel@molgen.mpg.de> wrote: >> Am 07.01.21 um 02:49 schrieb Brandon Kim: >> >>> We're exploring ways of upstreaming some of the downstream repositories >>> from Google to openbmc/* . >>> >>> Half, if not most of the downstream repositories are C++ daemons that are >>> specific to Google so we didn't want to create a bunch of new >>> openbmc/<repo> that no one would use. >>> >>> An idea that Ed gave me was having something like openbmc/google-misc >>> repository for all these repositories and if there are any that seem useful >>> to others, we can break it out into a different, separate repository in >>> openbmc/* layer. >>> >>> Please let me know if this seems like a good idea and I'm open to other >>> suggestions! >> >> Thank you very much for putting in the effort to make these repositories >> public. >> >> Using the openbmc/google-misc approach, how would the git history >> (commit log) be handled? >> >> Personally, I would prefer having small repositories as git makes that >> very easy to handle. Also it might save you time, as you do not have to >> think about what to do with the git history, and do not have to merge it. > > We would most likely squash the history together, in case there's > something confidential or private in the earlier commits. Understood. If that could be avoided, and only the confidential stuff removed, that would be great, as the git history gives a lot of insight into design decisions. > Many small repos would be easy to handle for us, but OpenBMC may not > want to have lots of small Google-specific repos in their org as this > may make it more cumbersome for others to find the relevant repos that > they're interested in. Understood. On the other, with small repositories, they can only use the parts they need. > There's also overhead for the project maintainers to create the > relevant groups and permissions for each new repo. Please note, that Without knowing the contents of the repositories and the size, this is all just my opinion. If the OpenBMC “admins“ can easily create several repositories, I’d prefer that route. If it’s too much work for them, their preference should be chosen. Kind regards, Paul ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-07 18:25 ` Paul Menzel @ 2021-01-07 21:20 ` Ed Tanous 2021-01-11 21:13 ` Patrick Williams 0 siblings, 1 reply; 12+ messages in thread From: Ed Tanous @ 2021-01-07 21:20 UTC (permalink / raw) To: Paul Menzel; +Cc: Brandon Kim, Benjamin Fair, OpenBMC Maillist On Thu, Jan 7, 2021 at 10:26 AM Paul Menzel <pmenzel@molgen.mpg.de> wrote: > > > Dear Benjamin, > > > Am 07.01.21 um 18:33 schrieb Benjamin Fair: > > > On Thu, 7 Jan 2021 at 00:09, Paul Menzel <pmenzel@molgen.mpg.de> wrote: > > >> Am 07.01.21 um 02:49 schrieb Brandon Kim: > >> > >>> We're exploring ways of upstreaming some of the downstream repositories > >>> from Google to openbmc/* . > >>> > >>> Half, if not most of the downstream repositories are C++ daemons that are > >>> specific to Google so we didn't want to create a bunch of new > >>> openbmc/<repo> that no one would use. > >>> > >>> An idea that Ed gave me was having something like openbmc/google-misc > >>> repository for all these repositories and if there are any that seem useful > >>> to others, we can break it out into a different, separate repository in > >>> openbmc/* layer. > >>> > >>> Please let me know if this seems like a good idea and I'm open to other > >>> suggestions! > >> > >> Thank you very much for putting in the effort to make these repositories > >> public. > >> > >> Using the openbmc/google-misc approach, how would the git history > >> (commit log) be handled? > >> > >> Personally, I would prefer having small repositories as git makes that > >> very easy to handle. Also it might save you time, as you do not have to > >> think about what to do with the git history, and do not have to merge it. > > > > We would most likely squash the history together, in case there's > > something confidential or private in the earlier commits. > > Understood. If that could be avoided, and only the confidential stuff > removed, that would be great, as the git history gives a lot of insight > into design decisions. It should be noted, there isn't really much git history to speak of for the things we're looking at pushing. The intent was that we'd write good descriptive commit messages when each piece was pushed. With that said, I very much suspect that these things won't be terribly useful outside of google. If they are useful outside of google-built systems, they really don't belong in meta-google or google-misc. Some examples of things that would go in this repository. 1. The google public keys/certs. I would hope that non-google systems are using their own root certificates. 2. Disabling of ssh login flows. This is done in a very specific way that requires interfacing with out of network components and protocols that are specific to our systems. I'd be surprised if anyone found this useful. 3. In-band telemetry code implementing interfaces for interfacing to google infrastructure. These haven't been built yet, but will likely be a translation from the public facing APIs (Dbus/redfish/ipmi) to interface them to google infrastructure. it's unlikely anyone else would use this. > > > Many small repos would be easy to handle for us, but OpenBMC may not > > want to have lots of small Google-specific repos in their org as this > > may make it more cumbersome for others to find the relevant repos that > > they're interested in. > > Understood. On the other, with small repositories, they can only use the > parts they need. See above, if there are pieces that people want to use on non-google systems, they don't belong in meta-google. With that said, your statement is incorrect, recipes are not required to be 1:1 with repositories. Multiple recipes can point at subfolders of the same repository, allowing you to "use the parts they need" by simply adding recipes. With that said, this is not the intent, and I would much rather move code to a more common layer (meta-phosphor for example) rather than have non google systems including meta-google in their bblayers.conf. > > > There's also overhead for the project maintainers to create the > > relevant groups and permissions for each new repo. > Please note, that Without knowing the contents of the repositories and > the size, this is all just my opinion. If the OpenBMC “admins“ can > easily create several repositories, I’d prefer that route. Today creating new repos is non-trivial, and IMO we already have too many of them, which is causing a lot of thrash. I'd really like to see us start consolidating some of these (see my patches to consolidate all the meta-layers into openbmc/openbmc with the owners plugin) > If it’s too > much work for them, their preference should be chosen. > > > Kind regards, > > Paul ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-07 21:20 ` Ed Tanous @ 2021-01-11 21:13 ` Patrick Williams 2021-01-11 21:51 ` Ed Tanous 0 siblings, 1 reply; 12+ messages in thread From: Patrick Williams @ 2021-01-11 21:13 UTC (permalink / raw) To: Ed Tanous; +Cc: Paul Menzel, Brandon Kim, Benjamin Fair, OpenBMC Maillist [-- Attachment #1: Type: text/plain, Size: 6037 bytes --] Unfortunately we don't have a great policy on any of this. Hopefully it is something we can come up with a better definition on in the near future. On Thu, Jan 07, 2021 at 01:20:00PM -0800, Ed Tanous wrote: > On Thu, Jan 7, 2021 at 10:26 AM Paul Menzel <pmenzel@molgen.mpg.de> wrote: > > Am 07.01.21 um 18:33 schrieb Benjamin Fair: > > > On Thu, 7 Jan 2021 at 00:09, Paul Menzel <pmenzel@molgen.mpg.de> wrote: > > >> Am 07.01.21 um 02:49 schrieb Brandon Kim: > > >> > > >>> We're exploring ways of upstreaming some of the downstream repositories > > >>> from Google to openbmc/* . > > >>> > > >>> Half, if not most of the downstream repositories are C++ daemons that are > > >>> specific to Google so we didn't want to create a bunch of new > > >>> openbmc/<repo> that no one would use. Just out of curiousity, if they're not going to be useful to anyone except Google, what is the utility of getting them into an openbmc repo? (There are reasons but I don't want to put words in your mouths) > > >>> An idea that Ed gave me was having something like openbmc/google-misc > > >>> repository for all these repositories and if there are any that seem useful > > >>> to others, we can break it out into a different, separate repository in > > >>> openbmc/* layer. > > >>> > > >>> Please let me know if this seems like a good idea and I'm open to other > > >>> suggestions! I really dislike dumping-ground repositories. If we're going to allow company-specific "misc" repositories, I would really like a policy that they may *only* be referenced from that company's meta-layer. If anyone has any use in that code it really should be broken out into its own repository with a proper maintenance structure. > > >> Thank you very much for putting in the effort to make these repositories > > >> public. > > >> > > >> Using the openbmc/google-misc approach, how would the git history > > >> (commit log) be handled? > > >> > > >> Personally, I would prefer having small repositories as git makes that > > >> very easy to handle. Also it might save you time, as you do not have to > > >> think about what to do with the git history, and do not have to merge it. > > > > > > We would most likely squash the history together, in case there's > > > something confidential or private in the earlier commits. > > > > Understood. If that could be avoided, and only the confidential stuff > > removed, that would be great, as the git history gives a lot of insight > > into design decisions. > > It should be noted, there isn't really much git history to speak of > for the things we're looking at pushing. How was any code of significance developed without a git history? It is unfortunate we're going to lose all of this because of how often I tend to dig through 'git-blame' to understand the "why" on code. > Some examples of things that would go in this repository. > 1. The google public keys/certs. I would hope that non-google systems > are using their own root certificates. > 2. Disabling of ssh login flows. This is done in a very specific way > that requires interfacing with out of network components and protocols > that are specific to our systems. I'd be surprised if anyone found > this useful. > 3. In-band telemetry code implementing interfaces for interfacing to > google infrastructure. These haven't been built yet, but will likely > be a translation from the public facing APIs (Dbus/redfish/ipmi) to > interface them to google infrastructure. it's unlikely anyone else > would use this. These make me more curious on the value of opening them. > > > Many small repos would be easy to handle for us, but OpenBMC may not > > > want to have lots of small Google-specific repos in their org as this > > > may make it more cumbersome for others to find the relevant repos that > > > they're interested in. > > > > Understood. On the other, with small repositories, they can only use the > > parts they need. I'm more comfortable with others utilizing this code if it is in a small repo like "google-ssh-cert". As others find it valuable we can rename the repo. > See above, if there are pieces that people want to use on non-google > systems, they don't belong in meta-google. With that said, your > statement is incorrect, recipes are not required to be 1:1 with > repositories. Multiple recipes can point at subfolders of the same > repository, allowing you to "use the parts they need" by simply adding > recipes. With that said, this is not the intent, and I would much > rather move code to a more common layer (meta-phosphor for example) > rather than have non google systems including meta-google in their > bblayers.conf. > > > > > > There's also overhead for the project maintainers to create the > > > relevant groups and permissions for each new repo. > > Please note, that Without knowing the contents of the repositories and > > the size, this is all just my opinion. If the OpenBMC “admins“ can > > easily create several repositories, I’d prefer that route. > > Today creating new repos is non-trivial, and IMO we already have too > many of them, which is causing a lot of thrash. I'd really like to > see us start consolidating some of these (see my patches to > consolidate all the meta-layers into openbmc/openbmc with the owners > plugin) What do you mean by "thrash"? Ideally it should be cheap to create a repository. If there is significant overhead to create a repository with the current infrastructure we should document those challenges and look to improve them. I don't have any issue with consolidation of the meta-layers because those are effectively all built together anyhow. Right now I'm not in favor of consolidation of code repositories and we've even talked about splitting out some pieces (EM and fru-device come to mind to me). Can you quantify what the advantage of a big[ger] multi-function repositories are? -- Patrick Williams [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-11 21:13 ` Patrick Williams @ 2021-01-11 21:51 ` Ed Tanous 2021-01-14 23:17 ` Brandon Kim 0 siblings, 1 reply; 12+ messages in thread From: Ed Tanous @ 2021-01-11 21:51 UTC (permalink / raw) To: Patrick Williams Cc: OpenBMC Maillist, Paul Menzel, Ed Tanous, Brandon Kim, Benjamin Fair On Mon, Jan 11, 2021 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote: > > Unfortunately we don't have a great policy on any of this. Hopefully it > is something we can come up with a better definition on in the near > future. > > On Thu, Jan 07, 2021 at 01:20:00PM -0800, Ed Tanous wrote: > > On Thu, Jan 7, 2021 at 10:26 AM Paul Menzel <pmenzel@molgen.mpg.de> wrote: > > > Am 07.01.21 um 18:33 schrieb Benjamin Fair: > > > > On Thu, 7 Jan 2021 at 00:09, Paul Menzel <pmenzel@molgen.mpg.de> wrote: > > > >> Am 07.01.21 um 02:49 schrieb Brandon Kim: > > > >> > > > >>> We're exploring ways of upstreaming some of the downstream repositories > > > >>> from Google to openbmc/* . > > > >>> > > > >>> Half, if not most of the downstream repositories are C++ daemons that are > > > >>> specific to Google so we didn't want to create a bunch of new > > > >>> openbmc/<repo> that no one would use. > > Just out of curiousity, if they're not going to be useful to anyone > except Google, what is the utility of getting them into an openbmc repo? > (There are reasons but I don't want to put words in your mouths) A slight clarification: the theory is they're not useful unless you're building a machine that intends to live within the Google infrastructure (similar to zaius, or q71l). The meta layer _is_ useful outside of google, with companies that design and build the aforementioned platforms. Having the specific tweaks made available in the public means that the companies we work with can build 1:1 the image that we're operating with, report bugs against it more publicly, and we can share more code in the open, without resorting to public/private forks of OpenBMC for our own purposes which have their own problems as has been proven in the past. The other hope is that if we're wrong, and something within that repo is useful outside of google (seems likely it might happen for something), it's available with a public license and whoever finds it useful can simply move it to a common repo where others can use it, with minimal fuss, or asking us to upstream something we've built in the past. > > > > >>> An idea that Ed gave me was having something like openbmc/google-misc > > > >>> repository for all these repositories and if there are any that seem useful > > > >>> to others, we can break it out into a different, separate repository in > > > >>> openbmc/* layer. > > > >>> > > > >>> Please let me know if this seems like a good idea and I'm open to other > > > >>> suggestions! > > I really dislike dumping-ground repositories. If we're going to allow > company-specific "misc" repositories, I would really like a policy that > they may *only* be referenced from that company's meta-layer. That sounds completely reasonable, and in-line with our intent. > If anyone > has any use in that code it really should be broken out into its own > repository with a proper maintenance structure. +1. > > > > >> Thank you very much for putting in the effort to make these repositories > > > >> public. > > > >> > > > >> Using the openbmc/google-misc approach, how would the git history > > > >> (commit log) be handled? > > > >> > > > >> Personally, I would prefer having small repositories as git makes that > > > >> very easy to handle. Also it might save you time, as you do not have to > > > >> think about what to do with the git history, and do not have to merge it. > > > > > > > > We would most likely squash the history together, in case there's > > > > something confidential or private in the earlier commits. > > > > > > Understood. If that could be avoided, and only the confidential stuff > > > removed, that would be great, as the git history gives a lot of insight > > > into design decisions. > > > > It should be noted, there isn't really much git history to speak of > > for the things we're looking at pushing. > > How was any code of significance developed without a git history? It is > unfortunate we're going to lose all of this because of how often I tend > to dig through 'git-blame' to understand the "why" on code. A lot of the commits aren't really "openbmc worthy" in that they're not signed off, have no tested statement, might have > 72 character lines ect. They definitely do have a git history, and if the preference is that we push it with the messy history, we can certainly do it. I don't have a strong opinion here other than not wanting to rewrite and retest every commit we've had to these things. > > > Some examples of things that would go in this repository. > > 1. The google public keys/certs. I would hope that non-google systems > > are using their own root certificates. > > 2. Disabling of ssh login flows. This is done in a very specific way > > that requires interfacing with out of network components and protocols > > that are specific to our systems. I'd be surprised if anyone found > > this useful. > > 3. In-band telemetry code implementing interfaces for interfacing to > > google infrastructure. These haven't been built yet, but will likely > > be a translation from the public facing APIs (Dbus/redfish/ipmi) to > > interface them to google infrastructure. it's unlikely anyone else > > would use this. > > These make me more curious on the value of opening them. > > > > > Many small repos would be easy to handle for us, but OpenBMC may not > > > > want to have lots of small Google-specific repos in their org as this > > > > may make it more cumbersome for others to find the relevant repos that > > > > they're interested in. > > > > > > Understood. On the other, with small repositories, they can only use the > > > parts they need. > > I'm more comfortable with others utilizing this code if it is in a small > repo like "google-ssh-cert". As others find it valuable we can rename > the repo. Right now the process to create new repositories takes a very long time, and requires interaction with core maintainers for CI/gerrit/github/user groups setup. That model doesn't scale beyond what we have today if things like certs for every company needs its own repository. I don't see a way to make it scale to the "new repo for everything" model, unless you had some ideas there. In other tracks I've had good luck simply extracting the history from a subfolder, and pushing that to a new repo when things needed to be split out. > > > See above, if there are pieces that people want to use on non-google > > systems, they don't belong in meta-google. With that said, your > > statement is incorrect, recipes are not required to be 1:1 with > > repositories. Multiple recipes can point at subfolders of the same > > repository, allowing you to "use the parts they need" by simply adding > > recipes. With that said, this is not the intent, and I would much > > rather move code to a more common layer (meta-phosphor for example) > > rather than have non google systems including meta-google in their > > bblayers.conf. > > > > > > > > > There's also overhead for the project maintainers to create the > > > > relevant groups and permissions for each new repo. > > > Please note, that Without knowing the contents of the repositories and > > > the size, this is all just my opinion. If the OpenBMC “admins“ can > > > easily create several repositories, I’d prefer that route. > > > > Today creating new repos is non-trivial, and IMO we already have too > > many of them, which is causing a lot of thrash. I'd really like to > > see us start consolidating some of these (see my patches to > > consolidate all the meta-layers into openbmc/openbmc with the owners > > plugin) > > What do you mean by "thrash"? Ideally it should be cheap to create a > repository. Ideally, yes, but we're not there today, and I don't see a path to get us there. > If there is significant overhead to create a repository > with the current infrastructure we should document those challenges and > look to improve them. The biggest challenge is that a large percentage of the work needs to be done by relatively few people (those with core permissions), and they have their own things to get done, and rightfully aren't able to prioritize creating new repos/permissions/other stuff. This topic alone is probably worth an email thread, as it's worth trying to tackle; I'll try to get that written up. > > I don't have any issue with consolidation of the meta-layers because > those are effectively all built together anyhow. Right now I'm not in > favor of consolidation of code repositories and we've even talked about > splitting out some pieces (EM and fru-device come to mind to me). We talked about splitting those up? I'd be a little worried about that, as they're pretty intertwined in their dbus interfaces. We're currently having the problem where nearly any new feature in dbus-sensors needs a commit to entity-manager to add a new schema, and that's non-obvious, and requires a commit done in lockstep to get code through. I had actually considered the other day doing the opposite, and proposing merging entity-manager and dbus-sensors, but that's a different discussion. > Can > you quantify what the advantage of a big[ger] multi-function repositories > are? The biggest advantage is time-to-create new repositories and the significant reduction in autobump-like recipe changes that need to be made to keep everything in lockstep. > > -- > Patrick Williams ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-11 21:51 ` Ed Tanous @ 2021-01-14 23:17 ` Brandon Kim 2021-01-20 19:04 ` Ed Tanous 2021-01-30 22:19 ` Brad Bishop 0 siblings, 2 replies; 12+ messages in thread From: Brandon Kim @ 2021-01-14 23:17 UTC (permalink / raw) To: Ed Tanous; +Cc: OpenBMC Maillist, Ed Tanous, Benjamin Fair, Paul Menzel [-- Attachment #1: Type: text/plain, Size: 10236 bytes --] Hi everyone, Wanted to ping this thread to see if there were more concerns on creating an openbmc/google-misc repository. Thanks! Brandon On Mon, Jan 11, 2021 at 1:51 PM Ed Tanous <ed@tanous.net> wrote: > On Mon, Jan 11, 2021 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> > wrote: > > > > Unfortunately we don't have a great policy on any of this. Hopefully it > > is something we can come up with a better definition on in the near > > future. > > > > On Thu, Jan 07, 2021 at 01:20:00PM -0800, Ed Tanous wrote: > > > On Thu, Jan 7, 2021 at 10:26 AM Paul Menzel <pmenzel@molgen.mpg.de> > wrote: > > > > Am 07.01.21 um 18:33 schrieb Benjamin Fair: > > > > > On Thu, 7 Jan 2021 at 00:09, Paul Menzel <pmenzel@molgen.mpg.de> > wrote: > > > > >> Am 07.01.21 um 02:49 schrieb Brandon Kim: > > > > >> > > > > >>> We're exploring ways of upstreaming some of the downstream > repositories > > > > >>> from Google to openbmc/* . > > > > >>> > > > > >>> Half, if not most of the downstream repositories are C++ daemons > that are > > > > >>> specific to Google so we didn't want to create a bunch of new > > > > >>> openbmc/<repo> that no one would use. > > > > Just out of curiousity, if they're not going to be useful to anyone > > except Google, what is the utility of getting them into an openbmc repo? > > (There are reasons but I don't want to put words in your mouths) > > A slight clarification: the theory is they're not useful unless you're > building a machine that intends to live within the Google > infrastructure (similar to zaius, or q71l). The meta layer _is_ > useful outside of google, with companies that design and build the > aforementioned platforms. Having the specific tweaks made available > in the public means that the companies we work with can build 1:1 the > image that we're operating with, report bugs against it more publicly, > and we can share more code in the open, without resorting to > public/private forks of OpenBMC for our own purposes which have their > own problems as has been proven in the past. > > The other hope is that if we're wrong, and something within that repo > is useful outside of google (seems likely it might happen for > something), it's available with a public license and whoever finds it > useful can simply move it to a common repo where others can use it, > with minimal fuss, or asking us to upstream something we've built in > the past. > > > > > > > >>> An idea that Ed gave me was having something like > openbmc/google-misc > > > > >>> repository for all these repositories and if there are any that > seem useful > > > > >>> to others, we can break it out into a different, separate > repository in > > > > >>> openbmc/* layer. > > > > >>> > > > > >>> Please let me know if this seems like a good idea and I'm open > to other > > > > >>> suggestions! > > > > I really dislike dumping-ground repositories. If we're going to allow > > company-specific "misc" repositories, I would really like a policy that > > they may *only* be referenced from that company's meta-layer. > > That sounds completely reasonable, and in-line with our intent. > > > If anyone > > has any use in that code it really should be broken out into its own > > repository with a proper maintenance structure. > > +1. > > > > > > > >> Thank you very much for putting in the effort to make these > repositories > > > > >> public. > > > > >> > > > > >> Using the openbmc/google-misc approach, how would the git history > > > > >> (commit log) be handled? > > > > >> > > > > >> Personally, I would prefer having small repositories as git makes > that > > > > >> very easy to handle. Also it might save you time, as you do not > have to > > > > >> think about what to do with the git history, and do not have to > merge it. > > > > > > > > > > We would most likely squash the history together, in case there's > > > > > something confidential or private in the earlier commits. > > > > > > > > Understood. If that could be avoided, and only the confidential stuff > > > > removed, that would be great, as the git history gives a lot of > insight > > > > into design decisions. > > > > > > It should be noted, there isn't really much git history to speak of > > > for the things we're looking at pushing. > > > > How was any code of significance developed without a git history? It is > > unfortunate we're going to lose all of this because of how often I tend > > to dig through 'git-blame' to understand the "why" on code. > > A lot of the commits aren't really "openbmc worthy" in that they're > not signed off, have no tested statement, might have > 72 character > lines ect. They definitely do have a git history, and if the > preference is that we push it with the messy history, we can certainly > do it. I don't have a strong opinion here other than not wanting to > rewrite and retest every commit we've had to these things. > > > > > > Some examples of things that would go in this repository. > > > 1. The google public keys/certs. I would hope that non-google systems > > > are using their own root certificates. > > > 2. Disabling of ssh login flows. This is done in a very specific way > > > that requires interfacing with out of network components and protocols > > > that are specific to our systems. I'd be surprised if anyone found > > > this useful. > > > 3. In-band telemetry code implementing interfaces for interfacing to > > > google infrastructure. These haven't been built yet, but will likely > > > be a translation from the public facing APIs (Dbus/redfish/ipmi) to > > > interface them to google infrastructure. it's unlikely anyone else > > > would use this. > > > > These make me more curious on the value of opening them. > > > > > > > Many small repos would be easy to handle for us, but OpenBMC may > not > > > > > want to have lots of small Google-specific repos in their org as > this > > > > > may make it more cumbersome for others to find the relevant repos > that > > > > > they're interested in. > > > > > > > > Understood. On the other, with small repositories, they can only use > the > > > > parts they need. > > > > I'm more comfortable with others utilizing this code if it is in a small > > repo like "google-ssh-cert". As others find it valuable we can rename > > the repo. > > Right now the process to create new repositories takes a very long > time, and requires interaction with core maintainers for > CI/gerrit/github/user groups setup. That model doesn't scale beyond > what we have today if things like certs for every company needs its > own repository. I don't see a way to make it scale to the "new repo > for everything" model, unless you had some ideas there. > > In other tracks I've had good luck simply extracting the history from > a subfolder, and pushing that to a new repo when things needed to be > split out. > > > > > > See above, if there are pieces that people want to use on non-google > > > systems, they don't belong in meta-google. With that said, your > > > statement is incorrect, recipes are not required to be 1:1 with > > > repositories. Multiple recipes can point at subfolders of the same > > > repository, allowing you to "use the parts they need" by simply adding > > > recipes. With that said, this is not the intent, and I would much > > > rather move code to a more common layer (meta-phosphor for example) > > > rather than have non google systems including meta-google in their > > > bblayers.conf. > > > > > > > > > > > > There's also overhead for the project maintainers to create the > > > > > relevant groups and permissions for each new repo. > > > > Please note, that Without knowing the contents of the repositories > and > > > > the size, this is all just my opinion. If the OpenBMC “admins“ can > > > > easily create several repositories, I’d prefer that route. > > > > > > Today creating new repos is non-trivial, and IMO we already have too > > > many of them, which is causing a lot of thrash. I'd really like to > > > see us start consolidating some of these (see my patches to > > > consolidate all the meta-layers into openbmc/openbmc with the owners > > > plugin) > > > > What do you mean by "thrash"? Ideally it should be cheap to create a > > repository. > > Ideally, yes, but we're not there today, and I don't see a path to get us > there. > > > If there is significant overhead to create a repository > > with the current infrastructure we should document those challenges and > > look to improve them. > > The biggest challenge is that a large percentage of the work needs to > be done by relatively few people (those with core permissions), and > they have their own things to get done, and rightfully aren't able to > prioritize creating new repos/permissions/other stuff. This topic > alone is probably worth an email thread, as it's worth trying to > tackle; I'll try to get that written up. > > > > > I don't have any issue with consolidation of the meta-layers because > > those are effectively all built together anyhow. Right now I'm not in > > favor of consolidation of code repositories and we've even talked about > > splitting out some pieces (EM and fru-device come to mind to me). > > We talked about splitting those up? I'd be a little worried about > that, as they're pretty intertwined in their dbus interfaces. > > We're currently having the problem where nearly any new feature in > dbus-sensors needs a commit to entity-manager to add a new schema, and > that's non-obvious, and requires a commit done in lockstep to get code > through. I had actually considered the other day doing the opposite, > and proposing merging entity-manager and dbus-sensors, but that's a > different discussion. > > > Can > > you quantify what the advantage of a big[ger] multi-function repositories > > are? > > The biggest advantage is time-to-create new repositories and the > significant reduction in autobump-like recipe changes that need to be > made to keep everything in lockstep. > > > > > -- > > Patrick Williams > [-- Attachment #2: Type: text/html, Size: 12373 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-14 23:17 ` Brandon Kim @ 2021-01-20 19:04 ` Ed Tanous 2021-01-30 22:19 ` Brad Bishop 1 sibling, 0 replies; 12+ messages in thread From: Ed Tanous @ 2021-01-20 19:04 UTC (permalink / raw) To: Brandon Kim; +Cc: OpenBMC Maillist, Ed Tanous, Benjamin Fair, Paul Menzel No concerns from me. -Ed On Thu, Jan 14, 2021 at 3:18 PM Brandon Kim <brandonkim@google.com> wrote: > > Hi everyone, > > Wanted to ping this thread to see if there were more concerns on creating an openbmc/google-misc repository. > > Thanks! > Brandon > > On Mon, Jan 11, 2021 at 1:51 PM Ed Tanous <ed@tanous.net> wrote: >> >> On Mon, Jan 11, 2021 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote: >> > >> > Unfortunately we don't have a great policy on any of this. Hopefully it >> > is something we can come up with a better definition on in the near >> > future. >> > >> > On Thu, Jan 07, 2021 at 01:20:00PM -0800, Ed Tanous wrote: >> > > On Thu, Jan 7, 2021 at 10:26 AM Paul Menzel <pmenzel@molgen.mpg.de> wrote: >> > > > Am 07.01.21 um 18:33 schrieb Benjamin Fair: >> > > > > On Thu, 7 Jan 2021 at 00:09, Paul Menzel <pmenzel@molgen.mpg.de> wrote: >> > > > >> Am 07.01.21 um 02:49 schrieb Brandon Kim: >> > > > >> >> > > > >>> We're exploring ways of upstreaming some of the downstream repositories >> > > > >>> from Google to openbmc/* . >> > > > >>> >> > > > >>> Half, if not most of the downstream repositories are C++ daemons that are >> > > > >>> specific to Google so we didn't want to create a bunch of new >> > > > >>> openbmc/<repo> that no one would use. >> > >> > Just out of curiousity, if they're not going to be useful to anyone >> > except Google, what is the utility of getting them into an openbmc repo? >> > (There are reasons but I don't want to put words in your mouths) >> >> A slight clarification: the theory is they're not useful unless you're >> building a machine that intends to live within the Google >> infrastructure (similar to zaius, or q71l). The meta layer _is_ >> useful outside of google, with companies that design and build the >> aforementioned platforms. Having the specific tweaks made available >> in the public means that the companies we work with can build 1:1 the >> image that we're operating with, report bugs against it more publicly, >> and we can share more code in the open, without resorting to >> public/private forks of OpenBMC for our own purposes which have their >> own problems as has been proven in the past. >> >> The other hope is that if we're wrong, and something within that repo >> is useful outside of google (seems likely it might happen for >> something), it's available with a public license and whoever finds it >> useful can simply move it to a common repo where others can use it, >> with minimal fuss, or asking us to upstream something we've built in >> the past. >> >> > >> > > > >>> An idea that Ed gave me was having something like openbmc/google-misc >> > > > >>> repository for all these repositories and if there are any that seem useful >> > > > >>> to others, we can break it out into a different, separate repository in >> > > > >>> openbmc/* layer. >> > > > >>> >> > > > >>> Please let me know if this seems like a good idea and I'm open to other >> > > > >>> suggestions! >> > >> > I really dislike dumping-ground repositories. If we're going to allow >> > company-specific "misc" repositories, I would really like a policy that >> > they may *only* be referenced from that company's meta-layer. >> >> That sounds completely reasonable, and in-line with our intent. >> >> > If anyone >> > has any use in that code it really should be broken out into its own >> > repository with a proper maintenance structure. >> >> +1. >> >> > >> > > > >> Thank you very much for putting in the effort to make these repositories >> > > > >> public. >> > > > >> >> > > > >> Using the openbmc/google-misc approach, how would the git history >> > > > >> (commit log) be handled? >> > > > >> >> > > > >> Personally, I would prefer having small repositories as git makes that >> > > > >> very easy to handle. Also it might save you time, as you do not have to >> > > > >> think about what to do with the git history, and do not have to merge it. >> > > > > >> > > > > We would most likely squash the history together, in case there's >> > > > > something confidential or private in the earlier commits. >> > > > >> > > > Understood. If that could be avoided, and only the confidential stuff >> > > > removed, that would be great, as the git history gives a lot of insight >> > > > into design decisions. >> > > >> > > It should be noted, there isn't really much git history to speak of >> > > for the things we're looking at pushing. >> > >> > How was any code of significance developed without a git history? It is >> > unfortunate we're going to lose all of this because of how often I tend >> > to dig through 'git-blame' to understand the "why" on code. >> >> A lot of the commits aren't really "openbmc worthy" in that they're >> not signed off, have no tested statement, might have > 72 character >> lines ect. They definitely do have a git history, and if the >> preference is that we push it with the messy history, we can certainly >> do it. I don't have a strong opinion here other than not wanting to >> rewrite and retest every commit we've had to these things. >> >> > >> > > Some examples of things that would go in this repository. >> > > 1. The google public keys/certs. I would hope that non-google systems >> > > are using their own root certificates. >> > > 2. Disabling of ssh login flows. This is done in a very specific way >> > > that requires interfacing with out of network components and protocols >> > > that are specific to our systems. I'd be surprised if anyone found >> > > this useful. >> > > 3. In-band telemetry code implementing interfaces for interfacing to >> > > google infrastructure. These haven't been built yet, but will likely >> > > be a translation from the public facing APIs (Dbus/redfish/ipmi) to >> > > interface them to google infrastructure. it's unlikely anyone else >> > > would use this. >> > >> > These make me more curious on the value of opening them. >> > >> > > > > Many small repos would be easy to handle for us, but OpenBMC may not >> > > > > want to have lots of small Google-specific repos in their org as this >> > > > > may make it more cumbersome for others to find the relevant repos that >> > > > > they're interested in. >> > > > >> > > > Understood. On the other, with small repositories, they can only use the >> > > > parts they need. >> > >> > I'm more comfortable with others utilizing this code if it is in a small >> > repo like "google-ssh-cert". As others find it valuable we can rename >> > the repo. >> >> Right now the process to create new repositories takes a very long >> time, and requires interaction with core maintainers for >> CI/gerrit/github/user groups setup. That model doesn't scale beyond >> what we have today if things like certs for every company needs its >> own repository. I don't see a way to make it scale to the "new repo >> for everything" model, unless you had some ideas there. >> >> In other tracks I've had good luck simply extracting the history from >> a subfolder, and pushing that to a new repo when things needed to be >> split out. >> >> > >> > > See above, if there are pieces that people want to use on non-google >> > > systems, they don't belong in meta-google. With that said, your >> > > statement is incorrect, recipes are not required to be 1:1 with >> > > repositories. Multiple recipes can point at subfolders of the same >> > > repository, allowing you to "use the parts they need" by simply adding >> > > recipes. With that said, this is not the intent, and I would much >> > > rather move code to a more common layer (meta-phosphor for example) >> > > rather than have non google systems including meta-google in their >> > > bblayers.conf. >> > > >> > > > >> > > > > There's also overhead for the project maintainers to create the >> > > > > relevant groups and permissions for each new repo. >> > > > Please note, that Without knowing the contents of the repositories and >> > > > the size, this is all just my opinion. If the OpenBMC “admins“ can >> > > > easily create several repositories, I’d prefer that route. >> > > >> > > Today creating new repos is non-trivial, and IMO we already have too >> > > many of them, which is causing a lot of thrash. I'd really like to >> > > see us start consolidating some of these (see my patches to >> > > consolidate all the meta-layers into openbmc/openbmc with the owners >> > > plugin) >> > >> > What do you mean by "thrash"? Ideally it should be cheap to create a >> > repository. >> >> Ideally, yes, but we're not there today, and I don't see a path to get us there. >> >> > If there is significant overhead to create a repository >> > with the current infrastructure we should document those challenges and >> > look to improve them. >> >> The biggest challenge is that a large percentage of the work needs to >> be done by relatively few people (those with core permissions), and >> they have their own things to get done, and rightfully aren't able to >> prioritize creating new repos/permissions/other stuff. This topic >> alone is probably worth an email thread, as it's worth trying to >> tackle; I'll try to get that written up. >> >> > >> > I don't have any issue with consolidation of the meta-layers because >> > those are effectively all built together anyhow. Right now I'm not in >> > favor of consolidation of code repositories and we've even talked about >> > splitting out some pieces (EM and fru-device come to mind to me). >> >> We talked about splitting those up? I'd be a little worried about >> that, as they're pretty intertwined in their dbus interfaces. >> >> We're currently having the problem where nearly any new feature in >> dbus-sensors needs a commit to entity-manager to add a new schema, and >> that's non-obvious, and requires a commit done in lockstep to get code >> through. I had actually considered the other day doing the opposite, >> and proposing merging entity-manager and dbus-sensors, but that's a >> different discussion. >> >> > Can >> > you quantify what the advantage of a big[ger] multi-function repositories >> > are? >> >> The biggest advantage is time-to-create new repositories and the >> significant reduction in autobump-like recipe changes that need to be >> made to keep everything in lockstep. >> >> > >> > -- >> > Patrick Williams ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-14 23:17 ` Brandon Kim 2021-01-20 19:04 ` Ed Tanous @ 2021-01-30 22:19 ` Brad Bishop 2021-01-30 22:23 ` Ed Tanous 1 sibling, 1 reply; 12+ messages in thread From: Brad Bishop @ 2021-01-30 22:19 UTC (permalink / raw) To: Brandon Kim Cc: Paul Menzel, OpenBMC Maillist, Ed Tanous, Ed Tanous, Benjamin Fair On Thu, Jan 14, 2021 at 03:17:57PM -0800, Brandon Kim wrote: >Hi everyone, > >Wanted to ping this thread to see if there were more concerns on creating >an openbmc/google-misc repository. I finally created this today. Apologies to the Google team for the long delay. -brad ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-30 22:19 ` Brad Bishop @ 2021-01-30 22:23 ` Ed Tanous 2021-02-01 2:21 ` Brandon Kim 0 siblings, 1 reply; 12+ messages in thread From: Ed Tanous @ 2021-01-30 22:23 UTC (permalink / raw) To: Brad Bishop Cc: Paul Menzel, Brandon Kim, Ed Tanous, OpenBMC Maillist, Benjamin Fair On Sat, Jan 30, 2021 at 2:19 PM Brad Bishop <bradleyb@fuzziesquirrel.com> wrote: > > On Thu, Jan 14, 2021 at 03:17:57PM -0800, Brandon Kim wrote: > >Hi everyone, > > > >Wanted to ping this thread to see if there were more concerns on creating > >an openbmc/google-misc repository. > > I finally created this today. Apologies to the Google team for the long > delay. > > -brad Awesome; Thanks for getting that done (especially on a Saturday). -Ed ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Upstreaming downstream Google BMC repositories 2021-01-30 22:23 ` Ed Tanous @ 2021-02-01 2:21 ` Brandon Kim 0 siblings, 0 replies; 12+ messages in thread From: Brandon Kim @ 2021-02-01 2:21 UTC (permalink / raw) To: Ed Tanous Cc: Paul Menzel, OpenBMC Maillist, Brad Bishop, Ed Tanous, Benjamin Fair [-- Attachment #1: Type: text/plain, Size: 574 bytes --] Thank you Brad! On Sat, Jan 30, 2021 at 2:23 PM Ed Tanous <edtanous@google.com> wrote: > On Sat, Jan 30, 2021 at 2:19 PM Brad Bishop <bradleyb@fuzziesquirrel.com> > wrote: > > > > On Thu, Jan 14, 2021 at 03:17:57PM -0800, Brandon Kim wrote: > > >Hi everyone, > > > > > >Wanted to ping this thread to see if there were more concerns on > creating > > >an openbmc/google-misc repository. > > > > I finally created this today. Apologies to the Google team for the long > > delay. > > > > -brad > > Awesome; Thanks for getting that done (especially on a Saturday). > > -Ed > [-- Attachment #2: Type: text/html, Size: 1014 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-02-01 2:23 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-07 1:49 Upstreaming downstream Google BMC repositories Brandon Kim 2021-01-07 8:01 ` Paul Menzel 2021-01-07 17:33 ` Benjamin Fair 2021-01-07 18:25 ` Paul Menzel 2021-01-07 21:20 ` Ed Tanous 2021-01-11 21:13 ` Patrick Williams 2021-01-11 21:51 ` Ed Tanous 2021-01-14 23:17 ` Brandon Kim 2021-01-20 19:04 ` Ed Tanous 2021-01-30 22:19 ` Brad Bishop 2021-01-30 22:23 ` Ed Tanous 2021-02-01 2:21 ` Brandon Kim
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.