Re: Bug#932845: TS-219 RTC issue with Debian Buster

Re: Bug#932845: TS-219 RTC issue with Debian Buster

[Uwe Kleine-König]

Hello Alexandre,

On Thu, Jul 25, 2019 at 04:31:49PM +0200, Oliver Hartkopp wrote:
> On 24.07.19 09:07, Uwe Kleine-König wrote:
> > On Tue, Jul 23, 2019 at 10:28:18PM +0200, Oliver Hartkopp wrote:
> > > I'm running a TS-119P+ and a TS-219P II Qnap NAS with Debian Buster.
> > > Both are now running a linux-image-4.19.0-5-marvell kernel.
> > > 
> > > But since my update from Linux 4.9 (Stretch) to Linux 4.19 (Buster) the
> > > hardware clock of both boxes refuse to work.
> > > 
> > > After some digging in kernel sources and re-installing Linux 4.9 on my
> > > Buster setup it turns out, that a change in the kernel config causes the
> > > problem:
> > > 
> > > 4.19.0-5-marvell -> CONFIG_RTC_DRV_S35390A=m   (fails)
> > > 
> > > 4.9.0-4-marvell -> CONFIG_RTC_DRV_S35390A=y    (works)
> > > 
> > > See details and solving process at:
> > > 
> > > https://marc.info/?l=linux-arm-kernel&m=156390875629259&w=2
> > > 
> > > Can you please revert the Kernel config parts for the RTC in a way that the
> > > RTC drivers are built into the marvell-arch kernel again instead of building
> > > them as modules?
> > 
> > They were switched to modules because the kernel image got too big to
> > fit into the flash storage of some machines. I assume when we switch to
> > built-in again the resulting problem is the bigger one.
> 
> I don't know which is the bigger problem here.
> 
> When the rtc driver is built as module it can not be operated with the
> hwclock tool from the util-linux package due to the missing rtc UIE support.
> 
> You finally have no hardware clock on these machines and must wait for ntp
> to shift time and date (my system always starts in February until ntp fixes
> the time).

For me it's obvious what is the bigger problem. Either you don't have
the correct time until ntp fixes it up for you, or others cannot install
a kernel update and so run a vulnerable OS.

> Maybe this problem is only relevant for the S35390A and PCF8563 chip which
> both lack the UIE support needed by hwclock. Both have only alarm triggers
> in a minute accuracy according to the driver source code.

AFAIK the rtc framework should then emulate this event somehow.

> So CONFIG_RTC_DRV_S35390A=y and CONFIG_RTC_DRV_PCF8563=y should bring back
> the hw clocks on these machines.
> 
> I assume using hwclock without UIE trigger code will impact the accuracy.
> 
> > > As described in the referenced description the hwclock tool does not work on
> > > the machines.
> > 
> > I'm not sure yet if this is a problem in hwclock() or the s35390a
> > driver.
> 
> In detail it is hwclock together with rtc drivers not supporting UIE trigger
> with a 1 second resolution.

@abelloni: Do you can shed some light here, how it is supposed to work?
Is hwclock just doing it wrong with non-PC clocks, or is there a kernel
bug to fix?

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: Bug#932845: TS-219 RTC issue with Debian Buster

[Oliver Hartkopp]

Hello Uwe,

On 26.07.19 09:27, Uwe Kleine-König wrote:
> Hello Alexandre,
> 
> On Thu, Jul 25, 2019 at 04:31:49PM +0200, Oliver Hartkopp wrote:
>> On 24.07.19 09:07, Uwe Kleine-König wrote:
>>> On Tue, Jul 23, 2019 at 10:28:18PM +0200, Oliver Hartkopp wrote:
>>>> I'm running a TS-119P+ and a TS-219P II Qnap NAS with Debian Buster.
>>>> Both are now running a linux-image-4.19.0-5-marvell kernel.
>>>>
>>>> But since my update from Linux 4.9 (Stretch) to Linux 4.19 (Buster) the
>>>> hardware clock of both boxes refuse to work.
>>>>
>>>> After some digging in kernel sources and re-installing Linux 4.9 on my
>>>> Buster setup it turns out, that a change in the kernel config causes the
>>>> problem:
>>>>
>>>> 4.19.0-5-marvell -> CONFIG_RTC_DRV_S35390A=m   (fails)
>>>>
>>>> 4.9.0-4-marvell -> CONFIG_RTC_DRV_S35390A=y    (works)
>>>>
>>>> See details and solving process at:
>>>>
>>>> https://marc.info/?l=linux-arm-kernel&m=156390875629259&w=2
>>>>
>>>> Can you please revert the Kernel config parts for the RTC in a way that the
>>>> RTC drivers are built into the marvell-arch kernel again instead of building
>>>> them as modules?
>>>
>>> They were switched to modules because the kernel image got too big to
>>> fit into the flash storage of some machines. I assume when we switch to
>>> built-in again the resulting problem is the bigger one.
>>
>> I don't know which is the bigger problem here.
>>
>> When the rtc driver is built as module it can not be operated with the
>> hwclock tool from the util-linux package due to the missing rtc UIE support.
>>
>> You finally have no hardware clock on these machines and must wait for ntp
>> to shift time and date (my system always starts in February until ntp fixes
>> the time).
> 
> For me it's obvious what is the bigger problem. Either you don't have
> the correct time until ntp fixes it up for you, or others cannot install
> a kernel update and so run a vulnerable OS.

That what I've written, NTP fixes the time for me. I have no problem 
with updating my kernels - in fact I was even able to flash an older 
kernel to figure out this rtc issue :-)

>> Maybe this problem is only relevant for the S35390A and PCF8563 chip which
>> both lack the UIE support needed by hwclock. Both have only alarm triggers
>> in a minute accuracy according to the driver source code.
> 
> AFAIK the rtc framework should then emulate this event somehow.

I don't think so. When the rtc chip is not able to trigger an event with 
a one second resolution - how can you emulate that?

>> So CONFIG_RTC_DRV_S35390A=y and CONFIG_RTC_DRV_PCF8563=y should bring back
>> the hw clocks on these machines.
>>
>> I assume using hwclock without UIE trigger code will impact the accuracy.
>>
>>>> As described in the referenced description the hwclock tool does not work on
>>>> the machines.
>>>
>>> I'm not sure yet if this is a problem in hwclock() or the s35390a
>>> driver.
>>
>> In detail it is hwclock together with rtc drivers not supporting UIE trigger
>> with a 1 second resolution.
> 
> @abelloni: Do you can shed some light here, how it is supposed to work?
> Is hwclock just doing it wrong with non-PC clocks, or is there a kernel
> bug to fix?
> 
> Best regards
> Uwe
> 

Best regards,
Oliver

Re: Bug#932845: TS-219 RTC issue with Debian Buster

[Alexandre Belloni]

On 26/07/2019 11:27:11+0200, Oliver Hartkopp wrote:
> Hello Uwe,
> 
> On 26.07.19 09:27, Uwe Kleine-König wrote:
> > Hello Alexandre,
> > 
> > On Thu, Jul 25, 2019 at 04:31:49PM +0200, Oliver Hartkopp wrote:
> > > On 24.07.19 09:07, Uwe Kleine-König wrote:
> > > > On Tue, Jul 23, 2019 at 10:28:18PM +0200, Oliver Hartkopp wrote:
> > > > > I'm running a TS-119P+ and a TS-219P II Qnap NAS with Debian Buster.
> > > > > Both are now running a linux-image-4.19.0-5-marvell kernel.
> > > > > 
> > > > > But since my update from Linux 4.9 (Stretch) to Linux 4.19 (Buster) the
> > > > > hardware clock of both boxes refuse to work.
> > > > > 
> > > > > After some digging in kernel sources and re-installing Linux 4.9 on my
> > > > > Buster setup it turns out, that a change in the kernel config causes the
> > > > > problem:
> > > > > 
> > > > > 4.19.0-5-marvell -> CONFIG_RTC_DRV_S35390A=m   (fails)
> > > > > 
> > > > > 4.9.0-4-marvell -> CONFIG_RTC_DRV_S35390A=y    (works)
> > > > > 
> > > > > See details and solving process at:
> > > > > 
> > > > > https://marc.info/?l=linux-arm-kernel&m=156390875629259&w=2
> > > > > 
> > > > > Can you please revert the Kernel config parts for the RTC in a way that the
> > > > > RTC drivers are built into the marvell-arch kernel again instead of building
> > > > > them as modules?
> > > > 
> > > > They were switched to modules because the kernel image got too big to
> > > > fit into the flash storage of some machines. I assume when we switch to
> > > > built-in again the resulting problem is the bigger one.
> > > 
> > > I don't know which is the bigger problem here.
> > > 
> > > When the rtc driver is built as module it can not be operated with the
> > > hwclock tool from the util-linux package due to the missing rtc UIE support.
> > > 
> > > You finally have no hardware clock on these machines and must wait for ntp
> > > to shift time and date (my system always starts in February until ntp fixes
> > > the time).
> > 
> > For me it's obvious what is the bigger problem. Either you don't have
> > the correct time until ntp fixes it up for you, or others cannot install
> > a kernel update and so run a vulnerable OS.
> 
> That what I've written, NTP fixes the time for me. I have no problem with
> updating my kernels - in fact I was even able to flash an older kernel to
> figure out this rtc issue :-)
> 
> > > Maybe this problem is only relevant for the S35390A and PCF8563 chip which
> > > both lack the UIE support needed by hwclock. Both have only alarm triggers
> > > in a minute accuracy according to the driver source code.
> > 
> > AFAIK the rtc framework should then emulate this event somehow.
> 
> I don't think so. When the rtc chip is not able to trigger an event with a
> one second resolution - how can you emulate that?
> 

CONFIG_RTC_INTF_DEV_UIE_EMUL emulates it by polling the RTC.

-- 
Alexandre Belloni, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: Bug#932845: TS-219 RTC issue with Debian Buster

[Oliver Hartkopp]

Hello Alexandre,

On 26.07.19 11:39, Alexandre Belloni wrote:

>>>> Maybe this problem is only relevant for the S35390A and PCF8563 chip which
>>>> both lack the UIE support needed by hwclock. Both have only alarm triggers
>>>> in a minute accuracy according to the driver source code.
>>>
>>> AFAIK the rtc framework should then emulate this event somehow.
>>
>> I don't think so. When the rtc chip is not able to trigger an event with a
>> one second resolution - how can you emulate that?
>>
> 
> CONFIG_RTC_INTF_DEV_UIE_EMUL emulates it by polling the RTC.
> 

Just booted my NAS box and /boot/config-4.19.0-5-marvell contains

# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set

Would be cool when this would make hwclock happy.

Best regards,
Oliver

Re: Bug#932845: TS-219 RTC issue with Debian Buster

[Oliver Hartkopp]

Just a thought:

There are some of these rtc drivers that set

rtc->rtc->uie_unsupported = 1;

in the case that they can't assign an irq line.

But others set

rtc->rtc->uie_unsupported = 1;

when they don't support an (alarm) trigger with 1 sec accuracy.

Wouldn't it make sense to put

+       select RTC_INTF_DEV
+       select RTC_INTF_DEV_UIE_EMUL

in the Kconfig entries of the latter devices?

Best regards,
Oliver


On 26.07.19 12:26, Oliver Hartkopp wrote:
> Hello Alexandre,
> 
> On 26.07.19 11:39, Alexandre Belloni wrote:
> 
>>>>> Maybe this problem is only relevant for the S35390A and PCF8563 
>>>>> chip which
>>>>> both lack the UIE support needed by hwclock. Both have only alarm 
>>>>> triggers
>>>>> in a minute accuracy according to the driver source code.
>>>>
>>>> AFAIK the rtc framework should then emulate this event somehow.
>>>
>>> I don't think so. When the rtc chip is not able to trigger an event 
>>> with a
>>> one second resolution - how can you emulate that?
>>>
>>
>> CONFIG_RTC_INTF_DEV_UIE_EMUL emulates it by polling the RTC.
>>
> 
> Just booted my NAS box and /boot/config-4.19.0-5-marvell contains
> 
> # CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
> 
> Would be cool when this would make hwclock happy.
> 
> Best regards,
> Oliver

Re: Bug#932845: TS-219 RTC issue with Debian Buster

[Trent Piepho]

On Fri, 2019-07-26 at 12:53 +0200, Oliver Hartkopp wrote:
> Just a thought:
> 
> There are some of these rtc drivers that set
> 
> rtc->rtc->uie_unsupported = 1;
> 
> in the case that they can't assign an irq line.
> 
> But others set
> 
> rtc->rtc->uie_unsupported = 1;
> 
> when they don't support an (alarm) trigger with 1 sec accuracy.
> 
> Wouldn't it make sense to put
> 
> +       select RTC_INTF_DEV
> +       select RTC_INTF_DEV_UIE_EMUL
> 
> in the Kconfig entries of the latter devices?

The hwclock in busybox does not use UIE.  Is it the util-linux version
that uses it?  Or systemd timedate?

I know that chrony's linux RTC support requires UIE, or UIE emulation,
to work.  chrony does not detect lack of this very well and the RTC
support just "doesn't happen" with no errors.  I had to strace it to
figure out it was waiting for UIE interrupts that never came.

Anyway, you don't really need UIE at all to use an rtc in a number of
ways.  The kernel "rtc to system clock on boot" feature doesn't need
it.  The kernel auto sync the rtc every 11 mins from NTP synced system
clock feature doesn't need it.  busybox hwclock doesn't need it.

So I suspect it's optional because it's not always needed.

Re: Bug#932845: TS-219 RTC issue with Debian Buster

[Oliver Hartkopp]

On 26/07/2019 20.12, Trent Piepho wrote:
> On Fri, 2019-07-26 at 12:53 +0200, Oliver Hartkopp wrote:
>> Just a thought:
>>
>> There are some of these rtc drivers that set
>>
>> rtc->rtc->uie_unsupported = 1;
>>
>> in the case that they can't assign an irq line.
>>
>> But others set
>>
>> rtc->rtc->uie_unsupported = 1;
>>
>> when they don't support an (alarm) trigger with 1 sec accuracy.
>>
>> Wouldn't it make sense to put
>>
>> +       select RTC_INTF_DEV
>> +       select RTC_INTF_DEV_UIE_EMUL
>>
>> in the Kconfig entries of the latter devices?
> 
> The hwclock in busybox does not use UIE.  Is it the util-linux version
> that uses it?  Or systemd timedate?

Yes, it is the util-linux version that invokes ioctl(rtc_fd, RTC_UIE_ON, 
0), see:

https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/tree/sys-utils/hwclock-rtc.c#n244

I documented the effect here:
https://marc.info/?l=linux-arm-kernel&m=156390875629259&w=2

> I know that chrony's linux RTC support requires UIE, or UIE emulation,
> to work.  chrony does not detect lack of this very well and the RTC
> support just "doesn't happen" with no errors.  I had to strace it to
> figure out it was waiting for UIE interrupts that never came.

"or UIE emulation" - sounds like a plan :-)

> Anyway, you don't really need UIE at all to use an rtc in a number of
> ways.  The kernel "rtc to system clock on boot" feature doesn't need
> it.

Right - for that reason the kernel sets the correct time when the 
rtc-s35390a driver is built-in.

When its loaded later as a module, the tool hwclock is used to retrieve 
the time which fails due to the missing UIE.

> The kernel auto sync the rtc every 11 mins from NTP synced system
> clock feature doesn't need it.  busybox hwclock doesn't need it.
> 
> So I suspect it's optional because it's not always needed.

hwclock works great in 'setting' the rtc (hwclock --systohc) but it 
fails to read it.

Regards,
Oliver