Swampd PID-driven Fan Controller

Swampd PID-driven Fan Controller

[Patrick Venture]

Per the requirements to have a fan controller for the quanta-q71l that
runs within the OpenBMC framework+, I developed the swampd
application.  It was designed to leverage sdbusplus and other OpenBMC
libraries and coding standards.

It supports:
* build-time yaml descriptions of the sensors and PID loops
* run-time configuration of the above
* run-time fan speed set-point control
* IPMI controlled manual mode
* passively and actively read sensors
* sensors that are from the host
* any variation on sysfs IO that is configured.

The design document is too large to really paste in this email, so
I'll summarize.

Each sensor is created based on the configuration provided.  Each PID
loop is created per zone.  Each zone lives on a separate thread (could
be merged into one thread that iterates through zones).  The zone
loops every 0.1s to steer the fans closer to their goal while every
second it adjusts the goal.

Some things I know OpenBMC won't like: it's multi-threaded.  It wasn't
built into the current fan-presence package.

I'm at a crossroads with this project because I'd like to make it
generally available.  Should I try to submit it for review on this
list, if so, how?  Or should I push it to Google's github projects
first?

+As well as other requirements.

Thanks,
Patrick

Re: Swampd PID-driven Fan Controller

[Patrick Venture]

Patrick brought it to my attention that PID wasn't defined:
https://en.wikipedia.org/wiki/PID_controller

-Patrick

On Tue, Jun 27, 2017 at 9:51 AM, Patrick Venture <venture@google.com> wrote:
> Per the requirements to have a fan controller for the quanta-q71l that
> runs within the OpenBMC framework+, I developed the swampd
> application.  It was designed to leverage sdbusplus and other OpenBMC
> libraries and coding standards.
>
> It supports:
> * build-time yaml descriptions of the sensors and PID loops
> * run-time configuration of the above
> * run-time fan speed set-point control
> * IPMI controlled manual mode
> * passively and actively read sensors
> * sensors that are from the host
> * any variation on sysfs IO that is configured.
>
> The design document is too large to really paste in this email, so
> I'll summarize.
>
> Each sensor is created based on the configuration provided.  Each PID
> loop is created per zone.  Each zone lives on a separate thread (could
> be merged into one thread that iterates through zones).  The zone
> loops every 0.1s to steer the fans closer to their goal while every
> second it adjusts the goal.
>
> Some things I know OpenBMC won't like: it's multi-threaded.  It wasn't
> built into the current fan-presence package.
>
> I'm at a crossroads with this project because I'd like to make it
> generally available.  Should I try to submit it for review on this
> list, if so, how?  Or should I push it to Google's github projects
> first?
>
> +As well as other requirements.
>
> Thanks,
> Patrick

Re: Swampd PID-driven Fan Controller

[Brad Bishop]

Hi Patrick

> On Jun 27, 2017, at 12:51 PM, Patrick Venture <venture@google.com> wrote:
> 
> Per the requirements to have a fan controller for the quanta-q71l that
> runs within the OpenBMC framework+, I developed the swampd
> application.  It was designed to leverage sdbusplus and other OpenBMC
> libraries and coding standards.

Sweet!  First let me say thanks for your willingness to share your
application.

> 
> It supports:
> * build-time yaml descriptions of the sensors and PID loops
> * run-time configuration of the above
> * run-time fan speed set-point control
> * IPMI controlled manual mode
> * passively and actively read sensors
> * sensors that are from the host
> * any variation on sysfs IO that is configured.

Sounds like a solid feature set.  Can you speak to what level of support
exists for things like hot plugging and/or degraded mode operation?

> 
> The design document is too large to really paste in this email, so
> I'll summarize.
> 
> Each sensor is created based on the configuration provided.  Each PID
> loop is created per zone.  Each zone lives on a separate thread (could
> be merged into one thread that iterates through zones).  The zone
> loops every 0.1s to steer the fans closer to their goal while every
> second it adjusts the goal.
> 
> Some things I know OpenBMC won't like: it's multi-threaded.  It wasn’t

True, we are trying to avoid multi-threaded applications, but if the
problem being solved by threads can’t reasonably be solved some other
way that seems fine.

> built into the current fan-presence package.

I don’t have any issue with this.

> 
> I'm at a crossroads with this project because I'd like to make it
> generally available.  Should I try to submit it for review on this
> list, if so, how?  Or should I push it to Google's github projects
> first?

I think this depends on a couple factors.

As you know, another team is also developing a fan control algorithm.
If the two algorithms are similar I don’t see any point in hosting two
applications that do the same thing.  It would just make it more confusing
for others coming to OpenBMC to pick one or the other for their platform.

Now, if each provides their own unique, fundamentally different fan control
strategy - each with a different set of pros/cons - that is exactly what we
had anticipated and bringing it in seems like a great idea.

For the second factor, I’m going to be a little frank and critical, because
my point is fundamental to the project goals and vision.  Please don’t take
this personally.

This is what is known in OSS circles as a code dump.  It isn’t OSS development.
OSS development happens...in the open.  The first three bullet points here touch
on this:

https://people.redhat.com/rjones/how-to-supply-code-to-open-source-projects/

In an OSS development model, the entire community is given the opportunity
to collaborate and shape the result of the end product, _as_it_is_developed_
through feedback and contributions.

I’m no zealot.  I completely understand that many times factors like product
delivery schedules, etc get in the way of this model, and thats fine.  Thats
why most products using OSS don’t ship with vanilla versions of the project
they are based on.

But the other fan control algorithm _is_ being developed in the open.  A lot of
folks have invested many many hours of peer review in it.

So the question becomes, are you prepared to submit your application to that
same level of peer review and reciprocate by investing the time required to
address the feedback and get it to a point where the community feels comfortable
assuming maintenance responsibility for it?

Thx - brad

> 
> +As well as other requirements.
> 
> Thanks,
> Patrick

Re: Swampd PID-driven Fan Controller

[Patrick Venture]

On Tue, Jun 27, 2017 at 3:34 PM, Brad Bishop
<bradleyb@fuzziesquirrel.com> wrote:
> Hi Patrick
>
>> On Jun 27, 2017, at 12:51 PM, Patrick Venture <venture@google.com> wrote:
>>
>> Per the requirements to have a fan controller for the quanta-q71l that
>> runs within the OpenBMC framework+, I developed the swampd
>> application.  It was designed to leverage sdbusplus and other OpenBMC
>> libraries and coding standards.
>
> Sweet!  First let me say thanks for your willingness to share your
> application.

No problem.
>
>>
>> It supports:
>> * build-time yaml descriptions of the sensors and PID loops
>> * run-time configuration of the above
>> * run-time fan speed set-point control
>> * IPMI controlled manual mode
>> * passively and actively read sensors
>> * sensors that are from the host
>> * any variation on sysfs IO that is configured.
>
> Sounds like a solid feature set.  Can you speak to what level of support
> exists for things like hot plugging and/or degraded mode operation?

Right now it's at ~80*% done, so I'm still working with people here to
design the best practices for what they want in failure modes.  It was
designed with input from a few engineers who have ties to its primary
use cases.  And there is a an expectation that things with this will
behave similarly to how the predecessors behaved.  So that drove a
fair amount, but there are certain failure modes this doesn't yet
handle that it needs to.  If the sensors stop reporting information,
for instance, it goes into a pre-configured fail-safe mode.


*Functionally complete but still has lots of room for improvement.

>
>>
>> The design document is too large to really paste in this email, so
>> I'll summarize.
>>
>> Each sensor is created based on the configuration provided.  Each PID
>> loop is created per zone.  Each zone lives on a separate thread (could
>> be merged into one thread that iterates through zones).  The zone
>> loops every 0.1s to steer the fans closer to their goal while every
>> second it adjusts the goal.
>>
>> Some things I know OpenBMC won't like: it's multi-threaded.  It wasn’t
>
> True, we are trying to avoid multi-threaded applications, but if the
> problem being solved by threads can’t reasonably be solved some other
> way that seems fine.
>
>> built into the current fan-presence package.
>
> I don’t have any issue with this.
>
>>
>> I'm at a crossroads with this project because I'd like to make it
>> generally available.  Should I try to submit it for review on this
>> list, if so, how?  Or should I push it to Google's github projects
>> first?
>
> I think this depends on a couple factors.
>
> As you know, another team is also developing a fan control algorithm.
> If the two algorithms are similar I don’t see any point in hosting two
> applications that do the same thing.  It would just make it more confusing
> for others coming to OpenBMC to pick one or the other for their platform.
>
> Now, if each provides their own unique, fundamentally different fan control
> strategy - each with a different set of pros/cons - that is exactly what we
> had anticipated and bringing it in seems like a great idea.

It does look like there has been some progress on the
phosphor-fan-presence/control.   Although I'd have to hack it anyway
to support Control.FanPwm because I lost that community argument.

There are definitely some differences.  However, some things can be
done in separate daemons -- as has been suggested.  As an alternative
to separate threads.

>
> For the second factor, I’m going to be a little frank and critical, because
> my point is fundamental to the project goals and vision.  Please don’t take
> this personally.
>
> This is what is known in OSS circles as a code dump.  It isn’t OSS development.
> OSS development happens...in the open.  The first three bullet points here touch
> on this:
>
> https://people.redhat.com/rjones/how-to-supply-code-to-open-source-projects/
>
> In an OSS development model, the entire community is given the opportunity
> to collaborate and shape the result of the end product, _as_it_is_developed_
> through feedback and contributions.
>
> I’m no zealot.  I completely understand that many times factors like product
> delivery schedules, etc get in the way of this model, and thats fine.  Thats
> why most products using OSS don’t ship with vanilla versions of the project
> they are based on.
>
> But the other fan control algorithm _is_ being developed in the open.  A lot of
> folks have invested many many hours of peer review in it.
>
> So the question becomes, are you prepared to submit your application to that
> same level of peer review and reciprocate by investing the time required to
> address the feedback and get it to a point where the community feels comfortable
> assuming maintenance responsibility for it?
>

I would have liked to develop this in the open, but as you touched on
it, schedules.  My platform is under a compressed schedule.  Under any
situation, I'll be responsible for maintaining this code base.  And
expanding it to support other platforms.  As implementing it wasn't
done in a vacuum, I feel fairly confident that outside contributions
at least in furthering the design to make it simpler or more modular
would be warranted.

Obviously, it was designed around very specific execution
requirements, which drove my adding an interval to phosphor-hwmon and
quite a few downstream hacks.  That said, those specific execution
requirements do have an impact on the ability to change the design
from input.  If the design suggestions or ideas violate the terms of
service that we're providing the data centers -- it can be forked,
effectively.

> Thx - brad
>
>>
>> +As well as other requirements.
>>
>> Thanks,
>> Patrick

Re: Swampd PID-driven Fan Controller

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2988 bytes --]

On Wed, Jun 28, 2017 at 06:13:27PM -0700, Patrick Venture wrote:
> I would have liked to develop this in the open, but as you touched on
> it, schedules.  My platform is under a compressed schedule.  Under any
> situation, I'll be responsible for maintaining this code base.  And
> expanding it to support other platforms.  As implementing it wasn't
> done in a vacuum, I feel fairly confident that outside contributions
> at least in furthering the design to make it simpler or more modular
> would be warranted.
> 
> Obviously, it was designed around very specific execution
> requirements, which drove my adding an interval to phosphor-hwmon and
> quite a few downstream hacks.  That said, those specific execution
> requirements do have an impact on the ability to change the design
> from input.  If the design suggestions or ideas violate the terms of
> service that we're providing the data centers -- it can be forked,
> effectively.

Patrick,

It was hard to tell from your initial email if your intentions were to
just give away some code you wrote (hence Brad's fears of a "code dump"
approach) or if you were trying to get this integrated as a repo we
would maintain long-term.  After discussions off-list it sounds like you
are wanting to do the work to integrate this into OpenBMC proper, which
is great.

One way of reading your original email was that you were wanting to host
this and maintain it in your own space and then add a recipe for the
package to openbmc/openbmc.  Since this hasn't come up before, I should
probably say formally I am really opposed to this.  I see it as a
difficulty long-term for us to have a repository that links against
phopshor-dbus-interfaces and recipe in openbmc, but the code itself is
not able to be updated by us.  It is not uncommon that we need to change
the dbus schema and the developer doing that is also generally responsible
for the changes across the repositories using the schema.  Having this
other code outside the project but that the project also relies on to
build an officially supported machine would make a dbus schema change
pretty painful.

My suggestion would be that we tackle this code in a few phases:

1. Place code on Github in your personal account for a high-level review
   by interested parties.

   1a.  We can take a look and get a feel for what lines up with
        existing architecture and where one of us needs to make changes.

2. We create a Gerrit repo for this code to live in long-term and you
   submit any changes from phase 1 there.

   2a.  We will then do a review and revisions per our normal process.

3. After merge, the code is now an officially supported repo that we
   will expect others to help maintain as, for example, APIs change.

At any point here, if you decide it is not going to be worth the effort,
we can stop or someone else can pick up where you leave off.  Does this
sound workable?

-- 
Patrick Williams

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Swampd PID-driven Fan Controller

[Patrick Venture]

On Thu, Jul 6, 2017 at 9:14 AM, Patrick Williams <patrick@stwcx.xyz> wrote:
> On Wed, Jun 28, 2017 at 06:13:27PM -0700, Patrick Venture wrote:
>> I would have liked to develop this in the open, but as you touched on
>> it, schedules.  My platform is under a compressed schedule.  Under any
>> situation, I'll be responsible for maintaining this code base.  And
>> expanding it to support other platforms.  As implementing it wasn't
>> done in a vacuum, I feel fairly confident that outside contributions
>> at least in furthering the design to make it simpler or more modular
>> would be warranted.
>>
>> Obviously, it was designed around very specific execution
>> requirements, which drove my adding an interval to phosphor-hwmon and
>> quite a few downstream hacks.  That said, those specific execution
>> requirements do have an impact on the ability to change the design
>> from input.  If the design suggestions or ideas violate the terms of
>> service that we're providing the data centers -- it can be forked,
>> effectively.
>
> Patrick,
>
> It was hard to tell from your initial email if your intentions were to
> just give away some code you wrote (hence Brad's fears of a "code dump"
> approach) or if you were trying to get this integrated as a repo we
> would maintain long-term.  After discussions off-list it sounds like you
> are wanting to do the work to integrate this into OpenBMC proper, which
> is great.

My plan is to work on integrating it in.  I've been preoccupied lately
with an upcoming deployment of OpenBMC: so basically all other things
are temporarily back-burner-ed.

>
> One way of reading your original email was that you were wanting to host
> this and maintain it in your own space and then add a recipe for the
> package to openbmc/openbmc.  Since this hasn't come up before, I should
> probably say formally I am really opposed to this.  I see it as a
> difficulty long-term for us to have a repository that links against
> phopshor-dbus-interfaces and recipe in openbmc, but the code itself is
> not able to be updated by us.  It is not uncommon that we need to change
> the dbus schema and the developer doing that is also generally responsible
> for the changes across the repositories using the schema.  Having this
> other code outside the project but that the project also relies on to
> build an officially supported machine would make a dbus schema change
> pretty painful.
>
> My suggestion would be that we tackle this code in a few phases:
>
> 1. Place code on Github in your personal account for a high-level review
>    by interested parties.

I have to reach out to the legal team to see if that's feasible.
There are guidelines for publishing to the Google owned repos, and
guidelines for personal open source projects, but not quite this type
of hybrid where it's non-personal project but isn't aiming to live
outside of the Google namespace.  So, i'll reach out and figure it
out.

>
>    1a.  We can take a look and get a feel for what lines up with
>         existing architecture and where one of us needs to make changes.
>
> 2. We create a Gerrit repo for this code to live in long-term and you
>    submit any changes from phase 1 there.
>
>    2a.  We will then do a review and revisions per our normal process.
>
> 3. After merge, the code is now an officially supported repo that we
>    will expect others to help maintain as, for example, APIs change.
>
> At any point here, if you decide it is not going to be worth the effort,
> we can stop or someone else can pick up where you leave off.  Does this
> sound workable?
>
It sounds workable.

> --
> Patrick Williams