Re: [RFC 03/11] hte: Add tegra194 HTE kernel provider

[Dipen Patel <dipenp@nvidia.com>]

On 7/28/21 4:59 PM, Dipen Patel wrote:
> Thanks Kent for the review comment. My responses inline.
>
> On 7/1/21 7:21 AM, Kent Gibson wrote:
>> On Fri, Jun 25, 2021 at 04:55:24PM -0700, Dipen Patel wrote:
>>> Tegra194 device has multiple HTE instances also known as GTE
>>> (Generic hardware Timestamping Engine) which can timestamp subset of
>>> SoC lines/signals. This provider driver focuses on IRQ and GPIO lines
>>> and exposes timestamping ability on those lines to the consumers
>>> through HTE subsystem.
>>>
>>> Also, with this patch, added:
>>> - documentation about this provider and its capabilities at
>>> Documentation/hte.
>>> - Compilation support in Makefile and Kconfig
>>>
>>> Signed-off-by: Dipen Patel <dipenp@nvidia.com>
>>> ---
>>>  Documentation/hte/index.rst        |  21 ++
>>>  Documentation/hte/tegra194-hte.rst |  65 ++++
>>>  Documentation/index.rst            |   1 +
>>>  drivers/hte/Kconfig                |  12 +
>>>  drivers/hte/Makefile               |   1 +
>>>  drivers/hte/hte-tegra194.c         | 554 +++++++++++++++++++++++++++++
>>>  6 files changed, 654 insertions(+)
>>>  create mode 100644 Documentation/hte/index.rst
>>>  create mode 100644 Documentation/hte/tegra194-hte.rst
>>>  create mode 100644 drivers/hte/hte-tegra194.c
>>>
>>> diff --git a/Documentation/hte/index.rst b/Documentation/hte/index.rst
>>> new file mode 100644
>>> index 000000000000..f311ebec6b47
>>> --- /dev/null
>>> +++ b/Documentation/hte/index.rst
>>> @@ -0,0 +1,21 @@
>>> +.. SPDX-License-Identifier: GPL-2.0
>>> +
>>> +============================================
>>> +The Linux Hardware Timestamping Engine (HTE)
>>> +============================================
>>> +
>>> +The HTE Subsystem
>>> +=================
>>> +
>>> +.. toctree::
>>> +   :maxdepth: 1
>>> +
>>> +   hte
>>> +
>>> +HTE Tegra Provider
>>> +==================
>>> +
>>> +.. toctree::
>>> +   :maxdepth: 1
>>> +
>>> +   tegra194-hte
>>> \ No newline at end of file
>>> diff --git a/Documentation/hte/tegra194-hte.rst b/Documentation/hte/tegra194-hte.rst
>>> new file mode 100644
>>> index 000000000000..c23eaafcf080
>>> --- /dev/null
>>> +++ b/Documentation/hte/tegra194-hte.rst
>>> @@ -0,0 +1,65 @@
>>> +HTE Kernel provider driver
>>> +==========================
>>> +
>>> +Description
>>> +-----------
>>> +The Nvidia tegra194 chip has many hardware timestamping engine (HTE) instances
>>> +known as generic timestamping engine (GTE). This provider driver implements
>>> +two GTE instances 1) GPIO GTE and 2) IRQ GTE. The both GTEs instances get the
>>> +timestamp from the system counter TSC which has 31.25MHz clock rate, and the
>>> +driver converts clock tick rate to nano seconds before storing it as timestamp
>>> +value.
>>> +
>>> +GPIO GTE
>>> +--------
>>> +
>>> +This GTE instance help timestamps GPIO in real time, for that to happen GPIO
>>> +needs to be configured as input and IRQ needs to ba enabled as well. The only
>>> +always on (AON) gpio controller instance supports timestamping GPIOs in
>>> +realtime and it has 39 GPIO lines. There is also a dependency on AON GPIO
>>> +controller as it requires very specific bits to be set in GPIO config register.
>>> +It in a way creates cyclic dependency between GTE and GPIO controller. The GTE
>>> +GPIO functionality is accessed from the GPIOLIB. It can support both the in
>>> +kernel and userspace consumers. In the later case, requests go through GPIOLIB
>>> +CDEV framework. The below APIs are added in GPIOLIB framework to access HTE
>>> +subsystem and GPIO GTE for in kernel consumers.
>>> +
>>> +.. c:function:: int gpiod_hw_timestamp_control( struct gpio_desc *desc, bool enable )
>>> +
>>> +	To enable HTE on given GPIO line.
>>> +
>>> +.. c:function:: u64 gpiod_get_hw_timestamp( struct gpio_desc *desc, bool block )
>>> +
>>> +	To retrieve hardwre timestamp in nano seconds.
>>> +
>>> +.. c:function:: bool gpiod_is_hw_timestamp_enabled( const struct gpio_desc *desc )
>>> +
>>> +	To query if HTE is enabled on the given GPIO.
>>> +
>>> +There is hte-tegra194-gpio-test.c, located in ``drivers/hte/`` directory, test
>>> +driver which demonstrates above APIs for the Jetson AGX platform. For userspace
>>> +consumers, GPIO_V2_LINE_FLAG_EVENT_CLOCK_HARDWARE flag must be specifed during
>>> +IOCTL calls, refer ``tools/gpio/gpio-event-mon.c``, which returns the timestamp
>>> +in nano second.
>>> +
>> <snip>
>>
>>> +
>>> +static void tegra_hte_read_fifo(struct tegra_hte_soc *gs)
>>> +{
>>> +	u32 tsh, tsl, src, pv, cv, acv, slice, bit_index, line_id;
>>> +	u64 tsc;
>>> +	int dir;
>>> +	struct hte_ts_data el;
>>> +
>>> +	while ((tegra_hte_readl(gs, HTE_TESTATUS) >>
>>> +		HTE_TESTATUS_OCCUPANCY_SHIFT) &
>>> +		HTE_TESTATUS_OCCUPANCY_MASK) {
>>> +		tsh = tegra_hte_readl(gs, HTE_TETSCH);
>>> +		tsl = tegra_hte_readl(gs, HTE_TETSCL);
>>> +		tsc = (((u64)tsh << 32) | tsl);
>>> +
>>> +		src = tegra_hte_readl(gs, HTE_TESRC);
>>> +		slice = (src >> HTE_TESRC_SLICE_SHIFT) &
>>> +			    HTE_TESRC_SLICE_DEFAULT_MASK;
>>> +
>>> +		pv = tegra_hte_readl(gs, HTE_TEPCV);
>>> +		cv = tegra_hte_readl(gs, HTE_TECCV);
>>> +		acv = pv ^ cv;
>>> +		while (acv) {
>>> +			bit_index = __builtin_ctz(acv);
>>> +			if ((pv >> bit_index) & BIT(0))
>>> +				dir = HTE_EVENT_RISING_EDGE;
>>> +			else
>>> +				dir = HTE_EVENT_FALLING_EDGE;
>>> +
>>> +			line_id = bit_index + (slice << 5);
>>> +			el.dir = dir;
>>> +			el.tsc = tsc << HTE_TS_NS_SHIFT;
>>> +			hte_push_ts_ns_atomic(gs->chip, line_id, &el,
>>> +					      sizeof(el));
>>> +			acv &= ~BIT(bit_index);
>>> +		}
>>> +		tegra_hte_writel(gs, HTE_TECMD, HTE_TECMD_CMD_POP);
>>> +	}
>>> +}
>> What happens when the hte_push_ts_ns_atomic() fails?
>> The timestamp will be quietly dropped?
>> What happens when the interrupt corresponding to that dropped timestamp
>> asks for it?  The irq handler thread will block until it can get a
>> timestamp from the subsequent interrupt?
> Two things happen, 1) at the push, HTE core increments seq counter
>
> 2) If the consumer has provided callback, it will either call that callback
>
> with HTE_TS_DROPPED or HTE_TS_AVAIL. The seq counter gives indirect
>
> view of dropped ts. However, I see the problem with the consumers not
>
> providing callback, in that case, push_ts* API just wakes up process without
>
> indicating why (assuming notify variable is true or else there is a chance for
>
> the thread to block forever). One easy approach I can think of for now is to
>
> make callback mandatory (which is optional right now), I will have to rethink
>
> that scenario and will push corrected version next RFC version.
>
> Thanks for pointing out.
>
>> Which brings me back to the concern I have with the approach used in
>> the hte/gpiolib integration - how do you guarantee that the timestamp
>> returned by gpiod_get_hw_timestamp() corresponds to the irq interrupt
>> being handled, particularly in the face of errors such as:
>>  - overflows of the timestamp FIFO in the chip
> I currently do not have any indication mechanism as the providers
>
> I am dealing with right now does not have overflow hardware detection
>
> support. If the chip supports, it should be easy to integrate that feature.
>
> I will provide some hook function or change in push_* API to accommodate
>
> this in next version of RFC.
>
>>  - overflows of software FIFOs as here
> HTE core records sequence counter as well it callsback the consumer with
>
> HTE_TS_DROPPED.
>
>>  - lost interupts (if the hw generates interrupts faster than the CPU
>>    can service them)
> For this, I have no idea unless hardware supports some sort of mechanism
>
> to catch that. For the current providers, as soon as it detects changes on lines
>
> it captures TS in its hw fifo. Its interrupt gets generated based on threshold
>
> set in that hw fifo. This interrupt is different than the lines of actual device
>
> that is why I said I have no idea how we can tackle that. Let me know if there
>
> is any idea or reference of the codes which does tackle this.
>
>
> Regarding HTE/GPIOLIB integration comment:
>
> You are right, currently, I have only tsc field returned from struct hte_ts_data
>
> to gpiolib. If I can extend that to return hte_ts_data structure which has seq
>
> counter, which I believe can be used to track the overflow situation. The

The reason I only return timestamp and not other details like its seq

counter, is because to comply with line_event_timestamp since it returns

only u64. Not sure which is the best way to extend and bring out its seq.

>
> dropped scenario can be easily tracked if gpiolib can be notified with above
>
> mentioned DROP event through callback. If that is the case, is it ok to have
>
> some sort of callback per gpio in gpiolib?
>
>
> Any idea how I can integrate callback notification with gpiolib if you do not agree on
>
> above callback suggestion?
>
>> ?
>>
>> Cheers,
>> Kent.
>>