Re: [PATCH 0/7] TC-ETF support PTP clocks series

Re: [PATCH 0/7] TC-ETF support PTP clocks series

[Meisinger, Andreas]

Hello Thomas,
Sorry about the wrong format/style of my last mail, hope I did get it right this time.

Let me first point at an important thing because we did have discussions here about it too. As of the manpages Linux CLOCK_TAI seems to be defined as an nonsettable clock which does have the same behaviour as of international atomic time TAI. Yet if it's nonsettable it can't be linked or synchronized to the value of International Atomic Time?
On the other hand there seems to be code in place to change the CLOCK_TAI offset thus making it basically sort of "setable"?

> The user space daemon which does the correlation between these PTP domains and TAI is required in any case, so the magic
> clock TAI_PRIVATE is not having any advantage.

I think a userspace daemon handling the translation information between different clocks would be fine. I think it's not really that relevant who exactly does apply the translation. Kernel level might be a little bit more precise but I guess it'd depend on other factors too.
Yet all translation based approaches have in common, setting a clock, renders translations done in past invalid. It would be required to fix old translated values along with setting the clock. I assume we couldn't distinguish between "translated" values and genuine values after translation, so fixing them might not be possible at all.
In our usecase we do have a clock for network operation which can't be set. We can guarantee this because we are able to define the network not being operational when the defined time is not available 😉.
Having this defined the remaining option would  be the target clock to be set. As of your suggestion that would be CLOCK_TAI. So at this point "setable" or "nonsettable" would become important. Here "setable" would introduce a dependency between the clocks. Being independent from clocks outside of our control was exactly the reason to introduce the separate network clock. To me this means if CLOCK_TAI could be changed by anything it couldn't be the base clock for our usecase if it can't it might be a solution.

> Depending on the frequency drift between CLOCK_TAI and clock PTP/$N the timer expiry might be slightly inaccurate, but
> surely not more inaccurate than if that conversion is done purely in user space.
>
> The self rearming posix timers would work too, but the self rearming is based on CLOCK_TAI, so rounding errors and drift
> would be accumulative. So I'd rather stay away from them.

As of the time ranges typically used in tsn networks the drift error for single shot timers most likely isn't a big deal. But as you suggest I'd stay away from long running timers as well rearming ones too, I guess they'd be mostly unusable.

> If such a coordination exists, then the whole problem in the TSN stack is gone. The core can always operate on TAI and the
> network device which runs in a different time universe would use the same conversion data e.g. to queue a packet for HW
> based time triggered transmission. Again subject to slight inaccuracy, but it does not come with all the problems of dynamic
> clocks, locking issues etc. As the frequency drift between PTP domains is neither fast changing nor randomly jumping around
> the inaccuracy might even be a mostly academic problem.

These multiple conversion errors would happen even for applications aware of it's target timescale.
This might usually be an academic issue but for some of our usecases conversion errors aren’t allowed at all.
In any case adding conversion errors sounds strange as our goal is to improve precision.

I don't see a way to avoid conversion errors except of somehow passing the original timestamp down to network card level.
Right now there's only one timestamp in CLOCK_TAI format which is used by ETF as well as by network card thus causing trouble if time is not same there.
If we'd add an (optional) second timestamp to SKB which would have to be set to network card time we could avoid the conversion error. As we do know which network card will be used for sending the SKB we wouldn't need a clock identifier for the second timestamp.
For situations where the application is not aware of the network cards timespace it wouldn't provide the second timestamp. In these situations it'd behave identical to your suggestion. Here the CLOCK_TAI timestamp would be translated to network card time based on the information of the userspace daemon.

What's your opinion about a second timestamp?

Best regards
Andreas Meisinger

Siemens AG
Digital Industries
Process Automation
DI PA DCP
Gleiwitzer Str. 555
90475 Nürnberg, Deutschland
Tel.: +49 911 95822720
mailto:andreas.meisinger@siemens.com

Re: [PATCH 0/7] TC-ETF support PTP clocks series

[Thomas Gleixner]

Andreas,

On Wed, Oct 14 2020 at 09:12, Andreas Meisinger wrote:
> Sorry about the wrong format/style of my last mail, hope I did get it
> right this time.

No problem. Yes this looks better. The only thing which could be
improved is that your mail client fails to add

 In-Reply-To: <messageid>
 References: <msgid1> <msgid2> ...

headers and instead has the MS lookout specific

Thread-Topic: [PATCH 0/7] TC-ETF support PTP clocks series
Thread-Index: AdaiB8a+x+RhfhtwSZ+NKfvRdyiJkw==

headers. If you look at the lore archive you see the effect:

  https://lore.kernel.org/r/AM0PR10MB3073F9694ECAD4F612A86FA7FA050@AM0PR10MB3073.EURPRD10.PROD.OUTLOOK.COM

and that happens to all mail clients which use threading based on the
standard headers. There is config knob in lookout to enable them.

> Let me first point at an important thing because we did have
> discussions here about it too. As of the manpages Linux CLOCK_TAI
> seems to be defined as an nonsettable clock which does have the same
> behaviour as of international atomic time TAI. Yet if it's nonsettable
> it can't be linked or synchronized to the value of International
> Atomic Time?
>
> On the other hand there seems to be code in place to change the
> CLOCK_TAI offset thus making it basically sort of "setable"?

It obviously needs to be modifiable in some way, otherwise
synchronization to a master clock via PTP would not work at all.

But it cannot be set in the way of settimeofday()/clock_settime() like
CLOCK_REALTIME.

>> The user space daemon which does the correlation between these PTP
>> domains and TAI is required in any case, so the magic clock
>> TAI_PRIVATE is not having any advantage.

> I think a userspace daemon handling the translation information
> between different clocks would be fine. I think it's not really that
> relevant who exactly does apply the translation. Kernel level might be
> a little bit more precise but I guess it'd depend on other factors
> too.

Not really. The kernel still provides the timestamp pairs/triplets in the
best way the underlying hardware provides it. Some hardware can even
provide hardware assistet pairs of ART and PTP clock.

> Yet all translation based approaches have in common, setting a clock,
> renders translations done in past invalid. It would be required to fix
> old translated values along with setting the clock. I assume we
> couldn't distinguish between "translated" values and genuine values
> after translation, so fixing them might not be possible at all.

CLOCK_TAI is not really set after the initial sychronization. It's
frequency corrected without causing jumps. PTP daemon uses a PLL based
algorithm for that.

Of course this adjustment has side effects for translation.

> In our usecase we do have a clock for network operation which can't be
> set. We can guarantee this because we are able to define the network
> not being operational when the defined time is not available 😉.
> Having this defined the remaining option would be the target clock to
> be set. As of your suggestion that would be CLOCK_TAI. So at this
> point "setable" or "nonsettable" would become important. Here
> "setable" would introduce a dependency between the clocks. Being
> independent from clocks outside of our control was exactly the reason
> to introduce the separate network clock. To me this means if CLOCK_TAI
> could be changed by anything it couldn't be the base clock for our
> usecase if it can't it might be a solution.

It's under your control as system designer how you operate CLOCK_TAI.

If you use the PTP daemon then it will sync CLOCK_TAI to the PTP clock
of your choice. If you don't have PTP at all then you can use NTP to
sync to a time server, which is less accurate. You can use PPS or just
use nothing.

The kernel does not care which non-standard base time or frequency you
chose.

Applications which communicate over network might care if the other side
uses a differnet time universe. Log files which start at 1971 might be
interesting to analyse against the log file of your other system which
starts in 2020.

>> Depending on the frequency drift between CLOCK_TAI and clock PTP/$N
>> the timer expiry might be slightly inaccurate, but surely not more
>> inaccurate than if that conversion is done purely in user space.
>>
>> The self rearming posix timers would work too, but the self rearming
>> is based on CLOCK_TAI, so rounding errors and drift would be
>> accumulative. So I'd rather stay away from them.
>
> As of the time ranges typically used in tsn networks the drift error
> for single shot timers most likely isn't a big deal. But as you
> suggest I'd stay away from long running timers as well rearming ones
> too, I guess they'd be mostly unusable.

Depends. It's a matter of hardware properties, system requirements,
application/system designers decisions. So what you consider unusable
for your system might be perfectly fine for others.

If we add support for this type of correlation then of course these
things need to be documented.

> Right now there's only one timestamp in CLOCK_TAI format which is used
> by ETF as well as by network card thus causing trouble if time is not
> same there.
>
> If we'd add an (optional) second timestamp to SKB which would have to
> be set to network card time we could avoid the conversion error. As we
> do know which network card will be used for sending the SKB we
> wouldn't need a clock identifier for the second timestamp.  For
> situations where the application is not aware of the network cards
> timespace it wouldn't provide the second timestamp. In these
> situations it'd behave identical to your suggestion. Here the
> CLOCK_TAI timestamp would be translated to network card time based on
> the information of the userspace daemon.

That would work as long as the target PTP clock is correlated because
the TAI timestamp is still required to make all of this work.

There are probably quite some dragons lurking left and right if we go
there, but it looks like a possible option.

Thanks,

        tglx

Re: [PATCH 0/7] TC-ETF support PTP clocks series

[Thomas Gleixner]

Vinicius,

On Fri, Oct 02 2020 at 12:01, Vinicius Costa Gomes wrote:
> I think that there's an underlying problem/limitation that is the cause
> of the issue (or at least a step in the right direction) you are trying
> to solve: the issue is that PTP clocks can't be used as hrtimers.

That's only an issue if PTP time != CLOCK_TAI, which is insane to begin
with.

As I know that these insanities exists in real world setups, e.g. grand
clock masters which start at the epoch which causes complete disaster
when any of the slave devices booted earlier. Obviously people came
up with system designs which are even more insane.

> I didn't spend a lot of time thinking about how to solve this (the only
> thing that comes to mind is having a timecounter, or similar, "software
> view" over the PHC clock).

There are two aspects:

 1) What's the overall time coordination especially for applications?

    PTP is for a reason based on TAI which allows a universal
    representation of time. Strict monotonic, no time zones, no leap
    seconds, no bells and whistels.

    Using TAI in distributed systems solved a gazillion of hard problems
    in one go.

    TSN depends on PTP and that obviously makes CLOCK_TAI _the_ clock of
    choice for schedules and whatever is needed. It just solves the
    problem nicely and we spent a great amount of time to make
    application development for TSN reasonable and hardware agnostic.

    Now industry comes along and decides to introducde independent time
    universes. The result is a black hole for programmers because they
    now have to waste effort - again - on solving the incredibly hard
    problems of warping space and time.

    The amount of money saved by not having properly coordinated time
    bases in such systems is definitely marginal compared to the amount
    of non-sensical work required to fix it in software.

 2) How can an OS provide halfways usable interfaces to handle this
    trainwreck?

    Access to the various time universes is already available through
    the dynamic POSIX clocks. But these interfaces have been designed
    for the performance insensitive work of PTP daemons and not for the
    performance critical work of applications dealing with real-time
    requirements of all sorts.

    As these raw PTP clocks are hardware dependend and only known at
    boot / device discovery time they cannot be exposed to the kernel
    internaly in any sane way. Also the user space interface has to be
    dynamic which rules out the ability to assign fixed CLOCK_* ids.

    As a consequence these clocks cannot provide timers like the regular
    CLOCK_* variants do, which makes it insanely hard to develop sane
    and portable applications.

    What comes to my mind (without spending much thought on it) is:

       1) Utilize and extend the existing PTP mechanisms to calculate
          the time relationship between the system wide CLOCK_TAI and
          the uncoordinated time universe. As offset is a constant and
          frequency drift is not a high speed problem this can be done
          with a userspace daemon of some sorts.

        2) Provide CLOCK_TAI_PRIVATE which defaults to CLOCK_TAI,
           i.e. offset = 0 and frequency ratio = 1 : 1

        3) (Ab)use the existing time namespace to provide a mechanism to
           adjust the offset and frequency ratio of CLOCK_TAI_PRIVATE
           which is calculated by #1

           This is the really tricky part and comes with severe
           limitations:

             - We can't walk task list to find tasks which have their
               CLOCK_TAI_PRIVATE associated with a particular
               incarnation of PCH/PTP universe, so some sane referencing
               of the underlying parameters to convert TAI to
               TAI_PRIVATE and vice versa has to be found. Life time
               problems are going to be interesting to deal with.

             - An application cannot coordinate multiple PCH/PTP domains
               and has to restrict itself to pick ONE disjunct time
               universe.

               Whether that's a reasonable limitation I don't know
               simply because the information provided in this patch
               series is close to zero.

             - Preventing early timer expiration caused by frequency
               drift is not trivial either.

      TBH, just thinking about all of that makes me shudder and my knee
      jerk reaction is: NO WAY!

Why the heck can't hardware people and system designers finally
understand that time is not something they can define at their
own peril?

The "Let's solve it in software so I don't have to think about it"
design approach strikes again. This caused headaches for the past five
decades, but people obviously never learn.

That said, I'm open for solutions which are at least in the proximity of
sane, but that needs a lot more information about the use cases and the
implications and not just some handwavy 'we screwed up our system design
and therefore we need to inflict insanity on everyone' blurb.

Thanks,

        tglx

Re: [PATCH 0/7] TC-ETF support PTP clocks series

[Geva, Erez]

On 02/10/2020 21:01, Vinicius Costa Gomes wrote:
> Hi Erez,
>
> Erez Geva <erez.geva.ext@siemens.com> writes:
>
>> Add support for using PTP clock with
>>   Traffic control Earliest TxTime First (ETF) Qdisc.
>>
>> Why do we need ETF to use PTP clock?
>> Current ETF requires to synchronization the system clock
>>   to the PTP Hardware clock (PHC) we want to send through.
>> But there are cases that we can not synchronize the system clock with
>>   the desire NIC PHC.
>> 1. We use several NICs with several PTP domains that our device
>>      is not allowed to be PTP master.
>>     And we are not allowed to synchronize these PTP domains.
>> 2. We are using another clock source which we need for our system.
>>     Yet our device is not allowed to be PTP master.
>> Regardless of the exact topology, as the Linux tradition is to allow
>>   the user the freedom to choose, we propose a patch that will allow
>>   the user to configure the TC-ETF with a PTP clock as well as
>>   using the system clock.
>> * NOTE: we do encourage the users to synchronize the system clock with
>>    a PTP clock.
>>   As the ETF watchdog uses the system clock.
>>   Synchronizing the system clock with a PTP clock will probably
>>    reduce the frequency different of the PHC and the system clock.
>>   As sequence, the user might be able to reduce the ETF delta time
>>    and the packets latency cross the network.
>>
>> Follow the decision to derive a dynamic clock ID from the file description
>>   of an opened PTP clock device file.
>> We propose a simple way to use the dynamic clock ID with the ETF Qdisc.
>> We will submit a patch to the "tc" tool from the iproute2 project
>>   once this patch is accepted.
>>
>
> In addition to what Thomas said, I would like to add some thoughts
> (mostly re-wording some of Thomas' comments :-)).
>
> I think that there's an underlying problem/limitation that is the cause
> of the issue (or at least a step in the right direction) you are trying
> to solve: the issue is that PTP clocks can't be used as hrtimers.
>
> I didn't spend a lot of time thinking about how to solve this (the only
> thing that comes to mind is having a timecounter, or similar, "software
> view" over the PHC clock).
>
> Anyway, my feeling is that until this is solved, we would only be
> working around the problem, and creating even more hard to handle corner
> cases.
>
>
> Cheers,
>

You are right.

Thanks for the insight.

Erez

Re: [PATCH 0/7] TC-ETF support PTP clocks series

[Vinicius Costa Gomes]

Hi Erez,

Erez Geva <erez.geva.ext@siemens.com> writes:

> Add support for using PTP clock with
>  Traffic control Earliest TxTime First (ETF) Qdisc.
>
> Why do we need ETF to use PTP clock?
> Current ETF requires to synchronization the system clock
>  to the PTP Hardware clock (PHC) we want to send through.
> But there are cases that we can not synchronize the system clock with
>  the desire NIC PHC.
> 1. We use several NICs with several PTP domains that our device
>     is not allowed to be PTP master.
>    And we are not allowed to synchronize these PTP domains.
> 2. We are using another clock source which we need for our system.
>    Yet our device is not allowed to be PTP master.
> Regardless of the exact topology, as the Linux tradition is to allow
>  the user the freedom to choose, we propose a patch that will allow
>  the user to configure the TC-ETF with a PTP clock as well as
>  using the system clock.
> * NOTE: we do encourage the users to synchronize the system clock with
>   a PTP clock.
>  As the ETF watchdog uses the system clock.
>  Synchronizing the system clock with a PTP clock will probably
>   reduce the frequency different of the PHC and the system clock.
>  As sequence, the user might be able to reduce the ETF delta time
>   and the packets latency cross the network.
>
> Follow the decision to derive a dynamic clock ID from the file description
>  of an opened PTP clock device file.
> We propose a simple way to use the dynamic clock ID with the ETF Qdisc.
> We will submit a patch to the "tc" tool from the iproute2 project
>  once this patch is accepted.
>

In addition to what Thomas said, I would like to add some thoughts
(mostly re-wording some of Thomas' comments :-)).

I think that there's an underlying problem/limitation that is the cause
of the issue (or at least a step in the right direction) you are trying
to solve: the issue is that PTP clocks can't be used as hrtimers.

I didn't spend a lot of time thinking about how to solve this (the only
thing that comes to mind is having a timecounter, or similar, "software
view" over the PHC clock).

Anyway, my feeling is that until this is solved, we would only be
working around the problem, and creating even more hard to handle corner
cases.


Cheers,
-- 
Vinicius

[PATCH 0/7] TC-ETF support PTP clocks series

[Erez Geva]

Add support for using PTP clock with
 Traffic control Earliest TxTime First (ETF) Qdisc.

Why do we need ETF to use PTP clock?
Current ETF requires to synchronization the system clock
 to the PTP Hardware clock (PHC) we want to send through.
But there are cases that we can not synchronize the system clock with
 the desire NIC PHC.
1. We use several NICs with several PTP domains that our device
    is not allowed to be PTP master.
   And we are not allowed to synchronize these PTP domains.
2. We are using another clock source which we need for our system.
   Yet our device is not allowed to be PTP master.
Regardless of the exact topology, as the Linux tradition is to allow
 the user the freedom to choose, we propose a patch that will allow
 the user to configure the TC-ETF with a PTP clock as well as
 using the system clock.
* NOTE: we do encourage the users to synchronize the system clock with
  a PTP clock.
 As the ETF watchdog uses the system clock.
 Synchronizing the system clock with a PTP clock will probably
  reduce the frequency different of the PHC and the system clock.
 As sequence, the user might be able to reduce the ETF delta time
  and the packets latency cross the network.

Follow the decision to derive a dynamic clock ID from the file description
 of an opened PTP clock device file.
We propose a simple way to use the dynamic clock ID with the ETF Qdisc.
We will submit a patch to the "tc" tool from the iproute2 project
 once this patch is accepted.

The patches contain:
1. Add function to verify that a dynamic clock ID is derived
    from file description.
   The function follows the clock ID convention for dynamic clock ID
    for file description.
  The function will be used in the second patch.

2. Function to get main system oscillator calibration state.

3. Functions to get and put POSIX clock reference of
    a PTP Hardware Clock (PHC).
   The get function uses a dynamic clock ID created by application space.
   The purpose is that a module can hold a POSIX clock reference after the
    configuration application closed the PTP clock device file,
    and though the dynamic clock ID can not be used any further.
   The POSIX clock refereces are used by the TC-ETF.

4. A fix of the range check in qdisc_watchdog_schedule_range_ns().

5. During testing of ETF, we notice issue with the high-resolution timer
    the ETF Qdisc watchdog uses.
   The timer was set for a sleep of 300 nanoseconds but
    end up sleeping for 3 milliseconds.
   The problem happens when the timer is already active and
    the current expire is earlier then a new expire.
   So, we add a new TC schedule function that do not reprogram the timer
    under these conditions.
   The use of the function make sense as the Qdisc watchdog does act as
    watchdog.
   The Qdisc watchdog can expire earlier.
   However, if the watchdog is late, packets are dropped.

6. Add kernel configuration for TC-ETF watchdog range.
   As the range is characteristic of Hardware,
   that seems to be the proper way.

7. Add support for using PHC clock with TC-ETF.

Erez Geva (7):
  POSIX clock ID check function
  Function to retrieve main clock state
  Functions to fetch POSIX dynamic clock object
  Fix qdisc_watchdog_schedule_range_ns range check
  Traffic control using high-resolution timer issue
  TC-ETF code improvements
  TC-ETF support PTP clocks

 include/linux/posix-clock.h     |  39 +++++++++
 include/linux/posix-timers.h    |   5 ++
 include/linux/timex.h           |   1 +
 include/net/pkt_sched.h         |   2 +
 include/uapi/linux/net_tstamp.h |   5 ++
 kernel/time/posix-clock.c       |  76 ++++++++++++++++
 kernel/time/posix-timers.c      |   2 +-
 kernel/time/timekeeping.c       |   9 ++
 net/sched/Kconfig               |   8 ++
 net/sched/sch_api.c             |  36 +++++++-
 net/sched/sch_etf.c             | 148 +++++++++++++++++++++++++-------
 11 files changed, 298 insertions(+), 33 deletions(-)


base-commit: a1b8638ba1320e6684aa98233c15255eb803fac7
-- 
2.20.1