Request for Feedback :: Time Mode setting in timemanager

Request for Feedback :: Time Mode setting in timemanager

[Vishwanatha Subbanna]

Hello,

Sending this email requesting feedback on one of the feature that we currently have in phosphor-timemanager.

Time manager uses TimeMode setting and can have either [NTP] or [Manual] as the valid options and are provided via xyz/openbmc_projects/settings/ for external users.

When the system power is off and BMC is in ready state, any changes to these settings are readily consumed by time manager daemon.

However, if the user changes the setting when the Host is booting, timemanager puts them in deferred state. Meaning, timemanager does not take the settings into effect until the Host is powered off.

So, if someone wants to move from [Manual] to [NTP] or vice-versa, when the Host is [On], they need to [power-off] the Host and power it back on.

This design was chosen because we wanted to give priority to Host. Some of us are asking me if we can make a change to take the setting changes in effect immediately, not caring the state of the Host.

Please could you help with your thoughts on this ?.. What is the Industry norm on this ?

Thank you,
!! Vishwa !!

Re: Request for Feedback :: Time Mode setting in timemanager

[Brad Bishop]

> On Feb 18, 2020, at 7:56 AM, Vishwanatha Subbanna <vishwa@linux.vnet.ibm.com> wrote:
> 
> Hello,
> 
> Sending this email requesting feedback on one of the feature that we currently have in phosphor-timemanager.
> 
> Time manager uses TimeMode setting and can have either [NTP] or [Manual] as the valid options and are provided via xyz/openbmc_projects/settings/ for external users.
> 
> When the system power is off and BMC is in ready state, any changes to these settings are readily consumed by time manager daemon.
> 
> However, if the user changes the setting when the Host is booting, timemanager puts them in deferred state. Meaning, timemanager does not take the settings into effect until the Host is powered off.

Can you elaborate on why it does this?

> 
> So, if someone wants to move from [Manual] to [NTP] or vice-versa, when the Host is [On], they need to [power-off] the Host and power it back on.

This seems less than ideal?  Would you agree?

> 
> This design was chosen because we wanted to give priority to Host.

What does it mean to give priority to the Host?  Are you trying to hide time changes in the time from the host?  Why?

> Some of us are asking me if we can make a change to take the setting changes in effect immediately, not caring the state of the Host.

Without additional background this is what seems intuitive to me.

> 
> Please could you help with your thoughts on this ?.. What is the Industry norm on this ?

FWIW on our (IBM) system designs we usually hook an RTC up to the BMC, and any host software needing a RTC has to get it via some in-band software interface.  I think I heard somewhere though that often in other systems designs the RTC is connected to the host processors and the BMC doesn’t have access to it.

Re: Request for Feedback :: Time Mode setting in timemanager

[Ryan Arnell]

[-- Attachment #1: Type: text/plain, Size: 2394 bytes --]

Thanks Brad and Vishwa for moving into the direction where we might not
require a reboot or host power off from moving to Manual to NTP or vise
versa.
We had a lot of user feedback who see this as a very inefficient task.
Goldman and David cc'ed here can also has a lot of customers who needs to
sync time often and they would not like to power off host every time that
happens.

Thanks again
Ryan

On Tue, Feb 18, 2020 at 8:43 AM Brad Bishop <bradleyb@fuzziesquirrel.com>
wrote:

>
>
> > On Feb 18, 2020, at 7:56 AM, Vishwanatha Subbanna <
> vishwa@linux.vnet.ibm.com> wrote:
> >
> > Hello,
> >
> > Sending this email requesting feedback on one of the feature that we
> currently have in phosphor-timemanager.
> >
> > Time manager uses TimeMode setting and can have either [NTP] or [Manual]
> as the valid options and are provided via xyz/openbmc_projects/settings/
> for external users.
> >
> > When the system power is off and BMC is in ready state, any changes to
> these settings are readily consumed by time manager daemon.
> >
> > However, if the user changes the setting when the Host is booting,
> timemanager puts them in deferred state. Meaning, timemanager does not take
> the settings into effect until the Host is powered off.
>
> Can you elaborate on why it does this?
>
> >
> > So, if someone wants to move from [Manual] to [NTP] or vice-versa, when
> the Host is [On], they need to [power-off] the Host and power it back on.
>
> This seems less than ideal?  Would you agree?
>
> >
> > This design was chosen because we wanted to give priority to Host.
>
> What does it mean to give priority to the Host?  Are you trying to hide
> time changes in the time from the host?  Why?
>
> > Some of us are asking me if we can make a change to take the setting
> changes in effect immediately, not caring the state of the Host.
>
> Without additional background this is what seems intuitive to me.
>
> >
> > Please could you help with your thoughts on this ?.. What is the
> Industry norm on this ?
>
> FWIW on our (IBM) system designs we usually hook an RTC up to the BMC, and
> any host software needing a RTC has to get it via some in-band software
> interface.  I think I heard somewhere though that often in other systems
> designs the RTC is connected to the host processors and the BMC doesn’t
> have access to it.

[-- Attachment #2: Type: text/html, Size: 3193 bytes --]

Re: Request for Feedback :: Time Mode setting in timemanager

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]

On Tue, Feb 18, 2020 at 09:40:53AM -0500, Brad Bishop wrote:
> 
> 
> > On Feb 18, 2020, at 7:56 AM, Vishwanatha Subbanna <vishwa@linux.vnet.ibm.com> wrote:
> > However, if the user changes the setting when the Host is booting, timemanager puts them in deferred state. Meaning, timemanager does not take the settings into effect until the Host is powered off.
> 
> Can you elaborate on why it does this?
> 
> > 
> > So, if someone wants to move from [Manual] to [NTP] or vice-versa, when the Host is [On], they need to [power-off] the Host and power it back on.
> 
> This seems less than ideal?  Would you agree?
> 
> > 
> > This design was chosen because we wanted to give priority to Host.
> 
> What does it mean to give priority to the Host?  Are you trying to hide time changes in the time from the host?  Why?
> 
> > Some of us are asking me if we can make a change to take the setting changes in effect immediately, not caring the state of the Host.
> 
> Without additional background this is what seems intuitive to me.
> 

Most of these design points came from considering how it might be best
for a cloud provider, like Rackspace, we were originally designing some
of this code for.

If I'm leasing the host processor from you, I don't necessarily trust
your time infrastructure and might want to use my own.  A compromised
time infrastructure can be used to get you to use expired SSL
certificates, for example.

With this in mind came all of these design points of "the host has
priority", "you may not change modes out from underneath a running host",
etc.

> > 
> > Please could you help with your thoughts on this ?.. What is the Industry norm on this ?
> 
> FWIW on our (IBM) system designs we usually hook an RTC up to the BMC, and any host software needing a RTC has to get it via some in-band software interface.  I think I heard somewhere though that often in other systems designs the RTC is connected to the host processors and the BMC doesn’t have access to it.

FB's OCP designs all have the RTC to the Host, so I'm not sure any of
this is applicable to us.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Request for Feedback :: Time Mode setting in timemanager

[Brad Bishop]

> On Feb 18, 2020, at 3:25 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
> On Tue, Feb 18, 2020 at 09:40:53AM -0500, Brad Bishop wrote:
>> 
>> 
>>> On Feb 18, 2020, at 7:56 AM, Vishwanatha Subbanna <vishwa@linux.vnet.ibm.com> wrote:
>>> However, if the user changes the setting when the Host is booting, timemanager puts them in deferred state. Meaning, timemanager does not take the settings into effect until the Host is powered off.
>> 
>> Can you elaborate on why it does this?
>> 
>>> 
>>> So, if someone wants to move from [Manual] to [NTP] or vice-versa, when the Host is [On], they need to [power-off] the Host and power it back on.
>> 
>> This seems less than ideal?  Would you agree?
>> 
>>> 
>>> This design was chosen because we wanted to give priority to Host.
>> 
>> What does it mean to give priority to the Host?  Are you trying to hide time changes in the time from the host?  Why?
>> 
>>> Some of us are asking me if we can make a change to take the setting changes in effect immediately, not caring the state of the Host.
>> 
>> Without additional background this is what seems intuitive to me.
>> 
> 
> Most of these design points came from considering how it might be best
> for a cloud provider, like Rackspace, we were originally designing some
> of this code for.
> 
> If I'm leasing the host processor from you, I don't necessarily trust
> your time infrastructure and might want to use my own.

Agreed but what does this have to do with what is going on, on the BMC?

Re: Request for Feedback :: Time Mode setting in timemanager

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

On Tue, Feb 18, 2020 at 03:34:02PM -0500, Brad Bishop wrote:
> > On Feb 18, 2020, at 3:25 PM, Patrick Williams <patrick@stwcx.xyz> wrote:

> > Most of these design points came from considering how it might be best
> > for a cloud provider, like Rackspace, we were originally designing some
> > of this code for.
> > 
> > If I'm leasing the host processor from you, I don't necessarily trust
> > your time infrastructure and might want to use my own.
> 
> Agreed but what does this have to do with what is going on, on the BMC?

When the BMC owns the hardware RTC, { Manual , Host } is the only mode
that allows the Host to utilize the RTC hardware without being subject
to the provider's time infrastructure.

What we talked about way back when this was implemented is that someone
super paranoid could use an inband IPMI call to get these settings,
confirm it was in { Manual , Host } mode, and know by design that it
won't ever change out from underneath without a reboot.  If it isn't
part of the design, I have no way of knowing if the provider (or someone
who has compromised their management infrastructure) has reverted time
control back to the BMC while I'm running.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Request for Feedback :: Time Mode setting in timemanager

[Brad Bishop]

>>> Please could you help with your thoughts on this ?.. What is the Industry norm on this ?
>> 
>> FWIW on our (IBM) system designs we usually hook an RTC up to the BMC, and any host software needing a RTC has to get it via some in-band software interface.  I think I heard somewhere though that often in other systems designs the RTC is connected to the host processors and the BMC doesn’t have access to it.
> 
> FB's OCP designs all have the RTC to the Host, so I'm not sure any of
> this is applicable to us.

Are there any down sides to designs like this?  I guess if NTP is not an option on the BMC, you are at the mercy of the host firmware if you want correct time.  If NTP is in use on the BMC does an RTC still do anything for you?

Re: Request for Feedback :: Time Mode setting in timemanager

[Brad Bishop]

> On Feb 18, 2020, at 3:52 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
>> oes this have to do with what is going on, on the BMC?
> 
> When the BMC owns the hardware RTC, { Manual , Host } is the only mode
> that allows the Host to utilize the RTC hardware without being subject
> to the provider's time infrastructure.

Would this be in a situation where the lessee can’t run NTP on the host?

Re: Request for Feedback :: Time Mode setting in timemanager

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

On Tue, Feb 18, 2020 at 04:01:58PM -0500, Brad Bishop wrote:
> 
> 
> > On Feb 18, 2020, at 3:52 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> > 
> >> oes this have to do with what is going on, on the BMC?
> > 
> > When the BMC owns the hardware RTC, { Manual , Host } is the only mode
> > that allows the Host to utilize the RTC hardware without being subject
> > to the provider's time infrastructure.
> 
> Would this be in a situation where the lessee can’t run NTP on the host?

IIRC, { NTP, Host } is an invalid combination.  When Owner is Host, we
allow the IPMI commands to set time.  How the Host got that time, we
cannot control at that point (NTP vs operator).

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Request for Feedback :: Time Mode setting in timemanager

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

On Tue, Feb 18, 2020 at 04:01:00PM -0500, Brad Bishop wrote:
> 
> >>> Please could you help with your thoughts on this ?.. What is the Industry norm on this ?
> >> 
> >> FWIW on our (IBM) system designs we usually hook an RTC up to the BMC, and any host software needing a RTC has to get it via some in-band software interface.  I think I heard somewhere though that often in other systems designs the RTC is connected to the host processors and the BMC doesn’t have access to it.
> > 
> > FB's OCP designs all have the RTC to the Host, so I'm not sure any of
> > this is applicable to us.
> 
> Are there any down sides to designs like this?  I guess if NTP is not an option on the BMC, you are at the mercy of the host firmware if you want correct time.  If NTP is in use on the BMC does an RTC still do anything for you?

There are times that the BMC doesn't know what time it is and reverts
back to 1970.  At least until the network and NTP are up again.  I'm not 
sure the history at why we arrived at this design, honestly.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Request for Feedback :: Time Mode setting in timemanager

[Vishwanatha Subbanna]

[-- Attachment #1: Type: text/html, Size: 2169 bytes --]

Re: Request for Feedback :: Time Mode setting in timemanager

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1332 bytes --]

On Wed, Feb 19, 2020 at 12:07:06PM +0530, Vishwanatha Subbanna wrote:
> Thanks for the good discussion on this.
> 
> Patrick, I see you mentioned the TimeOwner in your response. TimeOwner was
> another thing that was disliked by the users and I had sent an email couple
> months ago asking if anyone still needs it. I did not see anyone saying they
> need. I then proposed removing TimeOwner feature.
> 
> So, if we want to make it simpler, we would want to:
> 
> - Remove TimeOwner concept
> - Remove the deferred updates to Manual / NTP settings.
> 
> Please let me know if you see anything that might be affected by this ?.

I tried to provide history as best as I recollect, mostly to be helpful.

As I've mentioned elsewhere in this thread, Facebook servers use a
Host-owned RTC.  Unless someone can speak to non-Power servers that have
a BMC-owned RTC, this might only affect IBM(*).  You'll need to decide what
impact removing this feature may have to you and your customers.

If you'd like to put together a proposal and seek feedback on what the
potential side effects for specific customer classes might be, I can
certainly do that, but right now I don't really have enough information
on what the "new" design would be just from those two bullets.

(*) - And OpenPower.
-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Request for Feedback :: Time Mode setting in timemanager

[Vishwanatha Subbanna]

On deferring consuming the updates to Manual/NTP settings, so far, some of the customers have expressed dislike for this design. We don’t have this design in the traditional systems. What we have tho is that we don’t allow setting the time on BMC when the system power is on.  In a way, that behavior kind of matches with deferring the updates to Manual / NTP when the system power is on with the BMC based systems.

We are having this discussion internally. This mail was to see what the community feels about those settings.

Proposal for now is to:  *Remove the support for TimeOwner*. It will be as good as BOTH

Once I have some concrete data from the internal discussion, I will come back on the Manual/NTP settings.

Thanks,
!! Vishwa !!

> On 20-Feb-2020, at 10:03 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
> On Wed, Feb 19, 2020 at 12:07:06PM +0530, Vishwanatha Subbanna wrote:
>> Thanks for the good discussion on this.
>> 
>> Patrick, I see you mentioned the TimeOwner in your response. TimeOwner was
>> another thing that was disliked by the users and I had sent an email couple
>> months ago asking if anyone still needs it. I did not see anyone saying they
>> need. I then proposed removing TimeOwner feature.
>> 
>> So, if we want to make it simpler, we would want to:
>> 
>> - Remove TimeOwner concept
>> - Remove the deferred updates to Manual / NTP settings.
>> 
>> Please let me know if you see anything that might be affected by this ?.
> 
> I tried to provide history as best as I recollect, mostly to be helpful.
> 
> As I've mentioned elsewhere in this thread, Facebook servers use a
> Host-owned RTC.  Unless someone can speak to non-Power servers that have
> a BMC-owned RTC, this might only affect IBM(*).  You'll need to decide what
> impact removing this feature may have to you and your customers.
> 
> If you'd like to put together a proposal and seek feedback on what the
> potential side effects for specific customer classes might be, I can
> certainly do that, but right now I don't really have enough information
> on what the "new" design would be just from those two bullets.
> 
> (*) - And OpenPower.
> -- 
> Patrick Williams

Re: Request for Feedback :: Time Mode setting in timemanager

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

On Mon, Feb 24, 2020 at 11:38:56AM +0530, Vishwanatha Subbanna wrote:
> Proposal for now is to:  *Remove the support for TimeOwner*. It will be as good as BOTH

"TimeOwner = BOTH" today creates two virtual clocks from the physical
RTC by implementing the Host clock as an offset from the BMC clock,
doesn't it?  Is that going to continue to be the functionality with your
proposal or are you reverting back to a single physical clock where both
Host and BMC can update?

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Request for Feedback :: Time Mode setting in timemanager

[Lei YU]

On Tue, Feb 25, 2020 at 4:37 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Mon, Feb 24, 2020 at 11:38:56AM +0530, Vishwanatha Subbanna wrote:
> > Proposal for now is to:  *Remove the support for TimeOwner*. It will be as good as BOTH
>
> "TimeOwner = BOTH" today creates two virtual clocks from the physical
> RTC by implementing the Host clock as an offset from the BMC clock,
> doesn't it?  Is that going to continue to be the functionality with your
> proposal or are you reverting back to a single physical clock where both
> Host and BMC can update?

"TimeOnwer = BOTH" does not creates two virtual clocks, "TimeOwner =
Split" does.
"BOTH" effectively enables both BMC and the Host to set the "single" clock.

Re: Request for Feedback :: Time Mode setting in timemanager

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

On Tue, Feb 25, 2020 at 10:01:21AM +0800, Lei YU wrote:
> On Tue, Feb 25, 2020 at 4:37 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> >
> > On Mon, Feb 24, 2020 at 11:38:56AM +0530, Vishwanatha Subbanna wrote:
> > > Proposal for now is to:  *Remove the support for TimeOwner*. It will be as good as BOTH
> >
> > "TimeOwner = BOTH" today creates two virtual clocks from the physical
> > RTC by implementing the Host clock as an offset from the BMC clock,
> > doesn't it?  Is that going to continue to be the functionality with your
> > proposal or are you reverting back to a single physical clock where both
> > Host and BMC can update?
> 
> "TimeOnwer = BOTH" does not creates two virtual clocks, "TimeOwner =
> Split" does.
> "BOTH" effectively enables both BMC and the Host to set the "single" clock.

Got it, my mistake.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Request for Feedback :: Time Mode setting in timemanager

[Vishwanatha Subbanna]

Hi,

Further to the proposal I made to the community about removing TimeOwner feature, George Liu ( Thank you ) has put up a commit to :

- Remove TimeOwner feature :                          https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-time-manager/+/31278/
- Consume changes to NTP/MANUAL settings immediately: https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-time-manager/+/31284/

We have had many discussions with needed people here in IBM and we agree to the proposal. 

With that said: Both IPMI and PLDM will now use "/xyz/openbmc_project/time/bmc” as part of SetTime/GetTime instead of “/xyz/openbmc_project/time/host”

Thank you all for contributing into this discussion.

!! Vishwa !!

> On 25-Feb-2020, at 7:53 AM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
> On Tue, Feb 25, 2020 at 10:01:21AM +0800, Lei YU wrote:
>> On Tue, Feb 25, 2020 at 4:37 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>>> 
>>> On Mon, Feb 24, 2020 at 11:38:56AM +0530, Vishwanatha Subbanna wrote:
>>>> Proposal for now is to:  *Remove the support for TimeOwner*. It will be as good as BOTH
>>> 
>>> "TimeOwner = BOTH" today creates two virtual clocks from the physical
>>> RTC by implementing the Host clock as an offset from the BMC clock,
>>> doesn't it?  Is that going to continue to be the functionality with your
>>> proposal or are you reverting back to a single physical clock where both
>>> Host and BMC can update?
>> 
>> "TimeOnwer = BOTH" does not creates two virtual clocks, "TimeOwner =
>> Split" does.
>> "BOTH" effectively enables both BMC and the Host to set the "single" clock.
> 
> Got it, my mistake.
> 
> -- 
> Patrick Williams