Can we improve the experience of working on OpenBMC?

Can we improve the experience of working on OpenBMC?

[Andrew Jeffery]

Hello everyone,

A few months back Ed kicked off a thread about changing how we work on OpenBMC
with the aim to improve the development rate and make it easier to on-board
people to the project:

https://lore.kernel.org/openbmc/CAH2-KxAJS_U8=meCxp8ue7n0bmnzeRpyZOPZpy0h1cFEbbz-HA@mail.gmail.com/

I felt that discussion splintered a bit, with a lot of back-and-forth about
details well down in the weeds. I found this hard to follow, and so put in some
work to try synthesise the discussion into desires for improving how we work,
and practical problems with what we do currently.

Below are the lists of these desires and problems. I think it would be good to
treat this as a survey to understand who feels strongly about what.

If you want to express support for a point then add a +1 below it. Feel free to
suggest new items for either list, or to further discuss a particular item
below its entry.

I feel that with information on what people feel strongly about we can
prioritise what we need address and work forwards from there.

For the record, the mind map that I used to generate these lists is here, which
contains further quotes and references to the original discussion thread:

https://github.com/amboar/openbmc-monorepo-discussion

Some of these might be closely related or considered duplicates of other list
items, but based on the discussions referenced above I felt they were distinct
enough to warrant separate entries.

# Desires

1. Easy sharing of a broad set of application and distro changes
2. Minimise reviews required to get a given feature or fix integrated into the distro build
3. Make fork maintenance easy
4. Provide one place to report bugs across the entire project
5. Minimise effort collecting project statistics
6. Make it easy to add new applications
7. Make it easy to refactor across the entire project
8. Support inclusive naming

# Problems

1. Yocto is hard
    1. Managing patch stacks is hard
    2. Yocto-specific tooling is hard
    3. Finding the right recipe file to inspect/modify is hard
    4. Yocto has too much documentation
2. OpenBMC has too much documentation
3. Querying design/implementation/bug properties across the project is hard
4. Coordinating breaking changes is hard
5. Coordinating tree-wide changes is hard
6. Identifying the right repo to file a bug against is hard
7. Transferring bugs between repos is hard
8. Bug reports are duplicated across repos
9. Bug reports are ignored
10. Working out where to submit a patch is hard
11. Getting patches reviewed is hard
12. New repo requests are bottle-necked

Cheers,

Andrew

Re: Can we improve the experience of working on OpenBMC?

[Brad Bishop]

Hi Andrew, thanks for poking at this.

On Wed, Jul 27, 2022 at 10:52:04AM +0930, Andrew Jeffery wrote:
>Hello everyone,
>
>A few months back Ed kicked off a thread about changing how we work on OpenBMC
>with the aim to improve the development rate and make it easier to on-board
>people to the project:
>
>https://lore.kernel.org/openbmc/CAH2-KxAJS_U8=meCxp8ue7n0bmnzeRpyZOPZpy0h1cFEbbz-HA@mail.gmail.com/
>
>I felt that discussion splintered a bit, with a lot of back-and-forth about
>details well down in the weeds. I found this hard to follow, and so put in some
>work to try synthesise the discussion into desires for improving how we work,
>and practical problems with what we do currently.
>
>Below are the lists of these desires and problems. I think it would be good to
>treat this as a survey to understand who feels strongly about what.
>
>If you want to express support for a point then add a +1 below it. Feel free to
>suggest new items for either list, or to further discuss a particular item
>below its entry.
>
>I feel that with information on what people feel strongly about we can
>prioritise what we need address and work forwards from there.
>
>For the record, the mind map that I used to generate these lists is here, which
>contains further quotes and references to the original discussion thread:
>
>https://github.com/amboar/openbmc-monorepo-discussion
>
>Some of these might be closely related or considered duplicates of other list
>items, but based on the discussions referenced above I felt they were distinct
>enough to warrant separate entries.
>
># Desires
>
>1. Easy sharing of a broad set of application and distro changes

+1

>2. Minimise reviews required to get a given feature or fix integrated into the distro build

+1

>3. Make fork maintenance easy

+1

>4. Provide one place to report bugs across the entire project
>5. Minimise effort collecting project statistics
>6. Make it easy to add new applications
>7. Make it easy to refactor across the entire project

+1

>8. Support inclusive naming

+1

>
># Problems
>
>1. Yocto is hard
>    1. Managing patch stacks is hard
>    2. Yocto-specific tooling is hard
>    3. Finding the right recipe file to inspect/modify is hard
>    4. Yocto has too much documentation
>2. OpenBMC has too much documentation

🙂 Really? 

>3. Querying design/implementation/bug properties across the project is hard

+1

>4. Coordinating breaking changes is hard

+1

>5. Coordinating tree-wide changes is hard

+1

>6. Identifying the right repo to file a bug against is hard
>7. Transferring bugs between repos is hard
>8. Bug reports are duplicated across repos
>9. Bug reports are ignored
>10. Working out where to submit a patch is hard
>11. Getting patches reviewed is hard
>12. New repo requests are bottle-necked
>
>Cheers,
>
>Andrew

Re: Can we improve the experience of working on OpenBMC?

[Andrew Jeffery]

On Fri, 29 Jul 2022, at 09:40, Brad Bishop wrote:
> Hi Andrew, thanks for poking at this.
>
> On Wed, Jul 27, 2022 at 10:52:04AM +0930, Andrew Jeffery wrote:
>>
>># Problems
>>
>>1. Yocto is hard
>>    1. Managing patch stacks is hard
>>    2. Yocto-specific tooling is hard
>>    3. Finding the right recipe file to inspect/modify is hard
>>    4. Yocto has too much documentation
>>2. OpenBMC has too much documentation
>
> 🙂 Really? 
>

Hey, I'm just the messenger 😅

But of course, I could have misinterpreted the conversation. The mind 
map has all the quotes and reference links.

I should have mentioned that the mind map I linked to on github was 
generated with Freemind in case anyone wants to see something more 
sensible than raw XML.

Andrew

Re: Can we improve the experience of working on OpenBMC?

[Alexander Amelkin]

Hi Andrew!

27.07.2022 04:22, Andrew Jeffery пишет:
> # Problems
>
> 1. Yocto is hard
>      1. Managing patch stacks is hard
>      2. Yocto-specific tooling is hard
>      3. Finding the right recipe file to inspect/modify is hard
>      4. Yocto has too much documentation
> 2. OpenBMC has too much documentation
> 3. Querying design/implementation/bug properties across the project is hard
> 4. Coordinating breaking changes is hard
> 5. Coordinating tree-wide changes is hard
> 6. Identifying the right repo to file a bug against is hard
> 7. Transferring bugs between repos is hard
> 8. Bug reports are duplicated across repos
> 9. Bug reports are ignored
> 10. Working out where to submit a patch is hard
> 11. Getting patches reviewed is hard
> 12. New repo requests are bottle-necked

To the list of the problems I would add:

13. D-Bus is hard for newcomers not familiar with it, best practices are 
not described,
inter-process synchronization when accessing large d-bus objects (like 
network interfaces)
is not inherent to d-bus, and auxiliary synchronization using standard 
POSIX means is neither
explicitly requested/advised, nor controlled/enforced. All that may lead 
(and have previously led) to
races and various other problems. Add to that the long and inconvenient 
prefixing that we've
discussed earlier in another thread where Brad has supported my point of 
those being useless
to the project.

14. D-Bus may become a bottleneck or a slowing factor (due to the 
context switching overhead) for
the situations when two processes are communicating actively. A standard 
POSIX IPC like pipes,
mq or shm could become a solution (with d-bus or any other method used 
as an aid to negotiate
names, keys, or whatever other credentials needed to access a common IPC).

WBR, Alexander

P.S. All in all, I think d-bus wasn't a good choice of IPC for a system 
running on a low-performance single-core ARM chip.

Re: Can we improve the experience of working on OpenBMC?

[Ed Tanous]

On Thu, Jul 28, 2022 at 5:11 PM Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
>
> >2. OpenBMC has too much documentation
>
> 🙂 Really?
>

I suspect this is an overgeneralization of what I intended to say.
OpenBMC has enough documentation to expect that new people on the
project would not be able to read all of it cover to cover before
pushing a patch, and to expect that every workflow is just a matter of
"go document that" or "Write a script to do that" overcomplicates
things for new folks.  This is not to say documentation is bad, or
"too much", but the organization to be able to find relevant
information quickly, and the "correctness" could certainly be improved
(I fully realize it's a difficult problem).  In the context of
monorepo, my point ws that less day-to-day developer documentation is
required, because we fall back to gits documentation, given we'd be
using it in a much simpler way for the end developer.

Re: Can we improve the experience of working on OpenBMC?

[Ed Tanous]

On Mon, Aug 1, 2022 at 2:27 PM Alexander Amelkin <a.amelkin@yadro.com> wrote:
>
> Hi Andrew!
>
> 27.07.2022 04:22, Andrew Jeffery пишет:
> > # Problems
> >
> > 1. Yocto is hard
> >      1. Managing patch stacks is hard
> >      2. Yocto-specific tooling is hard
> >      3. Finding the right recipe file to inspect/modify is hard
> >      4. Yocto has too much documentation
> > 2. OpenBMC has too much documentation
> > 3. Querying design/implementation/bug properties across the project is hard
> > 4. Coordinating breaking changes is hard
> > 5. Coordinating tree-wide changes is hard
> > 6. Identifying the right repo to file a bug against is hard
> > 7. Transferring bugs between repos is hard
> > 8. Bug reports are duplicated across repos
> > 9. Bug reports are ignored
> > 10. Working out where to submit a patch is hard
> > 11. Getting patches reviewed is hard
> > 12. New repo requests are bottle-necked
>
> To the list of the problems I would add:
>
> 13. D-Bus is hard for newcomers not familiar with it, best practices are
> not described,
> inter-process synchronization when accessing large d-bus objects (like
> network interfaces)
> is not inherent to d-bus, and auxiliary synchronization using standard
> POSIX means is neither
> explicitly requested/advised, nor controlled/enforced.
> All that may lead
> (and have previously led) to
> races and various other problems. Add to that the long and inconvenient
> prefixing that we've
> discussed earlier in another thread where Brad has supported my point of
> those being useless
> to the project.
>
> 14. D-Bus may become a bottleneck or a slowing factor (due to the
> context switching overhead) for
> the situations when two processes are communicating actively. A standard
> POSIX IPC like pipes,
> mq or shm could become a solution (with d-bus or any other method used
> as an aid to negotiate
> names, keys, or whatever other credentials needed to access a common IPC).

FWIW, in the original context of "a single repo would help with these
things" I don't think either of these would be helped with a
rearrangement of code.

With that said, lots of people dislike dbus.  There are performance
critical things (SOL, KVM, Virtual media) that have completely avoided
it.  If you have a proposal for something better, I'd highly recommend
writing up a design doc.

>
> WBR, Alexander
>
> P.S. All in all, I think d-bus wasn't a good choice of IPC for a system
> running on a low-performance single-core ARM chip.
>

So propose something better that solves the same problem within
OpenBMC?  Or, alternatively, there's u-bmc that from the sounds of
reading your above is closer to your ideal in terms of trade offs (no,
IPC, efficient point to point comms with grpc), it might be worth
looking into for either using directly, or porting some of the ideas
it encompases into OpenBMC.

Re: Can we improve the experience of working on OpenBMC?

[Alexander Amelkin]

>> 13. D-Bus is hard for newcomers not familiar with it, best practices are
>> not described,
>> inter-process synchronization when accessing large d-bus objects (like
>> network interfaces)
>> is not inherent to d-bus, and auxiliary synchronization using standard
>> POSIX means is neither
>> explicitly requested/advised, nor controlled/enforced.
>> All that may lead
>> (and have previously led) to
>> races and various other problems. Add to that the long and inconvenient
>> prefixing that we've
>> discussed earlier in another thread where Brad has supported my point of
>> those being useless
>> to the project.
>>
>> 14. D-Bus may become a bottleneck or a slowing factor (due to the
>> context switching overhead) for
>> the situations when two processes are communicating actively. A standard
>> POSIX IPC like pipes,
>> mq or shm could become a solution (with d-bus or any other method used
>> as an aid to negotiate
>> names, keys, or whatever other credentials needed to access a common IPC).
> FWIW, in the original context of "a single repo would help with these
> things" I don't think either of these would be helped with a
> rearrangement of code.

Well, I don't think a single repo would help actually if we want to
keep the system modular and versatile as it is (and what is definitely 
good).

> With that said, lots of people dislike dbus.  There are performance
> critical things (SOL, KVM, Virtual media) that have completely avoided
> it.  If you have a proposal for something better, I'd highly recommend
> writing up a design doc.

Yes, you're right that I'm not currently proposing anything specific, 
and that it
would surely require a start with a design document.

However, I thought the idea of Andrew's message was to outline the problems,
and it looks to me like you more or less agree that those are problems, 
don't you?

> Or, alternatively, there's u-bmc that from the sounds of
> reading your above is closer to your ideal in terms of trade offs (no,
> IPC, efficient point to point comms with grpc), it might be worth
> looking into for either using directly, or porting some of the ideas
> it encompases into OpenBMC. 

Thanks for the hint, I will look into that.

WBR, Alexander.

P.S. I feel like I must apologize for maybe sounding rude or somehow 
inappropriate sometimes.
As you may know, English is not my native language and I, despite the 
vocabulary and the ability
to construct more-or-less proper sentences, may often lack the ability 
to properly/politely express
my ideas or understand the tone and/or subtle meaning of what other 
people say or write.
Please forgive me for that, no offense or disrespect is ever implied on 
my side.

Re: Can we improve the experience of working on OpenBMC?

[Andrew Jeffery]

On Tue, 2 Aug 2022, at 06:57, Alexander Amelkin wrote:
> Hi Andrew!
>
> 27.07.2022 04:22, Andrew Jeffery пишет:
>> # Problems
>>
>> 1. Yocto is hard
>>      1. Managing patch stacks is hard
>>      2. Yocto-specific tooling is hard
>>      3. Finding the right recipe file to inspect/modify is hard
>>      4. Yocto has too much documentation
>> 2. OpenBMC has too much documentation
>> 3. Querying design/implementation/bug properties across the project is hard
>> 4. Coordinating breaking changes is hard
>> 5. Coordinating tree-wide changes is hard
>> 6. Identifying the right repo to file a bug against is hard
>> 7. Transferring bugs between repos is hard
>> 8. Bug reports are duplicated across repos
>> 9. Bug reports are ignored
>> 10. Working out where to submit a patch is hard
>> 11. Getting patches reviewed is hard
>> 12. New repo requests are bottle-necked
>
> To the list of the problems I would add:
>
> 13. D-Bus is hard for newcomers not familiar with it, best practices are 
> not described,
> inter-process synchronization when accessing large d-bus objects (like 
> network interfaces)
> is not inherent to d-bus, and auxiliary synchronization using standard 
> POSIX means is neither
> explicitly requested/advised, nor controlled/enforced. All that may lead 
> (and have previously led) to
> races and various other problems.
> Add to that the long and inconvenient 
> prefixing that we've
> discussed earlier in another thread where Brad has supported my point of 
> those being useless
> to the project.
>
> 14. D-Bus may become a bottleneck or a slowing factor (due to the 
> context switching overhead) for
> the situations when two processes are communicating actively. A standard 
> POSIX IPC like pipes,
> mq or shm could become a solution (with d-bus or any other method used 
> as an aid to negotiate
> names, keys, or whatever other credentials needed to access a common IPC).

So to confirm I've understood your concerns and to distill it down to 
specific things people can agree or disagree with, I have:

* DBus concepts are hard
* Designing for DBus is hard
* DBus use is awkward
* DBus is slow
* Unconventional, OpenBMC-specific DBus patterns are mostly tribal knowledge

Is that fair?

Andrew

Re: Can we improve the experience of working on OpenBMC?

[Andrew Jeffery]

On Tue, 2 Aug 2022, at 07:09, Ed Tanous wrote:
> On Mon, Aug 1, 2022 at 2:27 PM Alexander Amelkin <a.amelkin@yadro.com> wrote:
>>
>> 14. D-Bus may become a bottleneck or a slowing factor (due to the
>> context switching overhead) for
>> the situations when two processes are communicating actively. A standard
>> POSIX IPC like pipes,
>> mq or shm could become a solution (with d-bus or any other method used
>> as an aid to negotiate
>> names, keys, or whatever other credentials needed to access a common IPC).
>
> FWIW, in the original context of "a single repo would help with these
> things" I don't think either of these would be helped with a
> rearrangement of code.
>
> With that said, lots of people dislike dbus.  There are performance
> critical things (SOL, KVM, Virtual media) that have completely avoided
> it.  If you have a proposal for something better, I'd highly recommend
> writing up a design doc.

Yep, fixing this requires addressing specific concerns. Patches or 
design docs would be helpful, or even just a list of specific things 
that we think are a concern.

>
>>
>> WBR, Alexander
>>
>> P.S. All in all, I think d-bus wasn't a good choice of IPC for a system
>> running on a low-performance single-core ARM chip.

u-bmc might be something to look at here?

Andrew

Re: Can we improve the experience of working on OpenBMC?

[Jiaqing Zhao]

On 2022-07-27 09:22, Andrew Jeffery wrote> # Problems
> 
> 1. Yocto is hard
>     1. Managing patch stacks is hard
>     2. Yocto-specific tooling is hard
>     3. Finding the right recipe file to inspect/modify is hard
>     4. Yocto has too much documentation
> 2. OpenBMC has too much documentation
> 3. Querying design/implementation/bug properties across the project is hard
> 4. Coordinating breaking changes is hard
> 5. Coordinating tree-wide changes is hard
> 6. Identifying the right repo to file a bug against is hard
> 7. Transferring bugs between repos is hard
> 8. Bug reports are duplicated across repos
> 9. Bug reports are ignored
> 10. Working out where to submit a patch is hard
> 11. Getting patches reviewed is hard
> 12. New repo requests are bottle-necked

I'd personally like to add one item

* Submitting patches across multiple repos is hard, especially when they depends on each other