From: Dipen Patel <dipenp@nvidia.com>
To: Kent Gibson <warthog618@gmail.com>
Cc: <thierry.reding@gmail.com>, <jonathanh@nvidia.com>,
<linux-kernel@vger.kernel.org>, <linux-tegra@vger.kernel.org>,
<linux-gpio@vger.kernel.org>, <linus.walleij@linaro.org>,
<bgolaszewski@baylibre.com>, <devicetree@vger.kernel.org>,
<linux-doc@vger.kernel.org>, <robh+dt@kernel.org>
Subject: Re: [RFC 03/11] hte: Add tegra194 HTE kernel provider
Date: Tue, 3 Aug 2021 15:40:50 -0700 [thread overview]
Message-ID: <6429d26d-7835-c4f7-96b3-c8b0ba730187@nvidia.com> (raw)
In-Reply-To: <20210731154323.GA24906@sol>
On 7/31/21 8:43 AM, Kent Gibson wrote:
> On Wed, Jul 28, 2021 at 04:59:08PM -0700, Dipen Patel wrote:
>> Thanks Kent for the review comment. My responses inline.
>>
>> On 7/1/21 7:21 AM, Kent Gibson wrote:
>>> On Fri, Jun 25, 2021 at 04:55:24PM -0700, Dipen Patel wrote:
>>>> Tegra194 device has multiple HTE instances also known as GTE
>>>> (Generic hardware Timestamping Engine) which can timestamp subset of
>>>> SoC lines/signals. This provider driver focuses on IRQ and GPIO lines
>>>> and exposes timestamping ability on those lines to the consumers
>>>> through HTE subsystem.
>>>>
>>>> Also, with this patch, added:
>>>> - documentation about this provider and its capabilities at
>>>> Documentation/hte.
>>>> - Compilation support in Makefile and Kconfig
>>>>
>>>> Signed-off-by: Dipen Patel <dipenp@nvidia.com>
>>>> ---
>>>> Documentation/hte/index.rst | 21 ++
>>>> Documentation/hte/tegra194-hte.rst | 65 ++++
>>>> Documentation/index.rst | 1 +
>>>> drivers/hte/Kconfig | 12 +
>>>> drivers/hte/Makefile | 1 +
>>>> drivers/hte/hte-tegra194.c | 554 +++++++++++++++++++++++++++++
>>>> 6 files changed, 654 insertions(+)
>>>> create mode 100644 Documentation/hte/index.rst
>>>> create mode 100644 Documentation/hte/tegra194-hte.rst
>>>> create mode 100644 drivers/hte/hte-tegra194.c
>>>>
>>>> diff --git a/Documentation/hte/index.rst b/Documentation/hte/index.rst
>>>> new file mode 100644
>>>> index 000000000000..f311ebec6b47
>>>> --- /dev/null
>>>> +++ b/Documentation/hte/index.rst
>>>> @@ -0,0 +1,21 @@
>>>> +.. SPDX-License-Identifier: GPL-2.0
>>>> +
>>>> +============================================
>>>> +The Linux Hardware Timestamping Engine (HTE)
>>>> +============================================
>>>> +
>>>> +The HTE Subsystem
>>>> +=================
>>>> +
>>>> +.. toctree::
>>>> + :maxdepth: 1
>>>> +
>>>> + hte
>>>> +
>>>> +HTE Tegra Provider
>>>> +==================
>>>> +
>>>> +.. toctree::
>>>> + :maxdepth: 1
>>>> +
>>>> + tegra194-hte
>>>> \ No newline at end of file
>>>> diff --git a/Documentation/hte/tegra194-hte.rst b/Documentation/hte/tegra194-hte.rst
>>>> new file mode 100644
>>>> index 000000000000..c23eaafcf080
>>>> --- /dev/null
>>>> +++ b/Documentation/hte/tegra194-hte.rst
>>>> @@ -0,0 +1,65 @@
>>>> +HTE Kernel provider driver
>>>> +==========================
>>>> +
>>>> +Description
>>>> +-----------
>>>> +The Nvidia tegra194 chip has many hardware timestamping engine (HTE) instances
>>>> +known as generic timestamping engine (GTE). This provider driver implements
>>>> +two GTE instances 1) GPIO GTE and 2) IRQ GTE. The both GTEs instances get the
>>>> +timestamp from the system counter TSC which has 31.25MHz clock rate, and the
>>>> +driver converts clock tick rate to nano seconds before storing it as timestamp
>>>> +value.
>>>> +
>>>> +GPIO GTE
>>>> +--------
>>>> +
>>>> +This GTE instance help timestamps GPIO in real time, for that to happen GPIO
>>>> +needs to be configured as input and IRQ needs to ba enabled as well. The only
>>>> +always on (AON) gpio controller instance supports timestamping GPIOs in
>>>> +realtime and it has 39 GPIO lines. There is also a dependency on AON GPIO
>>>> +controller as it requires very specific bits to be set in GPIO config register.
>>>> +It in a way creates cyclic dependency between GTE and GPIO controller. The GTE
>>>> +GPIO functionality is accessed from the GPIOLIB. It can support both the in
>>>> +kernel and userspace consumers. In the later case, requests go through GPIOLIB
>>>> +CDEV framework. The below APIs are added in GPIOLIB framework to access HTE
>>>> +subsystem and GPIO GTE for in kernel consumers.
>>>> +
>>>> +.. c:function:: int gpiod_hw_timestamp_control( struct gpio_desc *desc, bool enable )
>>>> +
>>>> + To enable HTE on given GPIO line.
>>>> +
>>>> +.. c:function:: u64 gpiod_get_hw_timestamp( struct gpio_desc *desc, bool block )
>>>> +
>>>> + To retrieve hardwre timestamp in nano seconds.
>>>> +
>>>> +.. c:function:: bool gpiod_is_hw_timestamp_enabled( const struct gpio_desc *desc )
>>>> +
>>>> + To query if HTE is enabled on the given GPIO.
>>>> +
>>>> +There is hte-tegra194-gpio-test.c, located in ``drivers/hte/`` directory, test
>>>> +driver which demonstrates above APIs for the Jetson AGX platform. For userspace
>>>> +consumers, GPIO_V2_LINE_FLAG_EVENT_CLOCK_HARDWARE flag must be specifed during
>>>> +IOCTL calls, refer ``tools/gpio/gpio-event-mon.c``, which returns the timestamp
>>>> +in nano second.
>>>> +
>>> <snip>
>>>
>>>> +
>>>> +static void tegra_hte_read_fifo(struct tegra_hte_soc *gs)
>>>> +{
>>>> + u32 tsh, tsl, src, pv, cv, acv, slice, bit_index, line_id;
>>>> + u64 tsc;
>>>> + int dir;
>>>> + struct hte_ts_data el;
>>>> +
>>>> + while ((tegra_hte_readl(gs, HTE_TESTATUS) >>
>>>> + HTE_TESTATUS_OCCUPANCY_SHIFT) &
>>>> + HTE_TESTATUS_OCCUPANCY_MASK) {
>>>> + tsh = tegra_hte_readl(gs, HTE_TETSCH);
>>>> + tsl = tegra_hte_readl(gs, HTE_TETSCL);
>>>> + tsc = (((u64)tsh << 32) | tsl);
>>>> +
>>>> + src = tegra_hte_readl(gs, HTE_TESRC);
>>>> + slice = (src >> HTE_TESRC_SLICE_SHIFT) &
>>>> + HTE_TESRC_SLICE_DEFAULT_MASK;
>>>> +
>>>> + pv = tegra_hte_readl(gs, HTE_TEPCV);
>>>> + cv = tegra_hte_readl(gs, HTE_TECCV);
>>>> + acv = pv ^ cv;
>>>> + while (acv) {
>>>> + bit_index = __builtin_ctz(acv);
>>>> + if ((pv >> bit_index) & BIT(0))
>>>> + dir = HTE_EVENT_RISING_EDGE;
>>>> + else
>>>> + dir = HTE_EVENT_FALLING_EDGE;
>>>> +
>>>> + line_id = bit_index + (slice << 5);
>>>> + el.dir = dir;
>>>> + el.tsc = tsc << HTE_TS_NS_SHIFT;
>>>> + hte_push_ts_ns_atomic(gs->chip, line_id, &el,
>>>> + sizeof(el));
>>>> + acv &= ~BIT(bit_index);
>>>> + }
>>>> + tegra_hte_writel(gs, HTE_TECMD, HTE_TECMD_CMD_POP);
>>>> + }
>>>> +}
>>> What happens when the hte_push_ts_ns_atomic() fails?
>>> The timestamp will be quietly dropped?
>>> What happens when the interrupt corresponding to that dropped timestamp
>>> asks for it? The irq handler thread will block until it can get a
>>> timestamp from the subsequent interrupt?
>> Two things happen, 1) at the push, HTE core increments seq counter
>>
>> 2) If the consumer has provided callback, it will either call that callback
>>
>> with HTE_TS_DROPPED or HTE_TS_AVAIL. The seq counter gives indirect
>>
>> view of dropped ts. However, I see the problem with the consumers not
>>
>> providing callback, in that case, push_ts* API just wakes up process without
>>
>> indicating why (assuming notify variable is true or else there is a chance for
>>
>> the thread to block forever). One easy approach I can think of for now is to
>>
>> make callback mandatory (which is optional right now), I will have to rethink
>>
>> that scenario and will push corrected version next RFC version.
>>
>> Thanks for pointing out.
>>
> I'm not sure you understood my question, which was intended to
> demonstrate how an overflow here would break your gpio integration, but I
> am certain that I don't understand your answer.
>
> Using the callback to signal fifo overflow to the consumer is crazy.
> If the consumer is too busy to service the fifo then they probably wont
> be prepared to deal with the callback either. And the primary purpose of
> the fifo is to decouple the producer and consumer, so requiring a callback
> defeats the whole purpose of having the fifo there in the first place.
>
>>> Which brings me back to the concern I have with the approach used in
>>> the hte/gpiolib integration - how do you guarantee that the timestamp
>>> returned by gpiod_get_hw_timestamp() corresponds to the irq interrupt
>>> being handled, particularly in the face of errors such as:
>>> - overflows of the timestamp FIFO in the chip
>> I currently do not have any indication mechanism as the providers
>>
>> I am dealing with right now does not have overflow hardware detection
>>
>> support. If the chip supports, it should be easy to integrate that feature.
>>
>> I will provide some hook function or change in push_* API to accommodate
>>
>> this in next version of RFC.
>>
>>> - overflows of software FIFOs as here
>> HTE core records sequence counter as well it callsback the consumer with
>>
>> HTE_TS_DROPPED.
>>
>>> - lost interupts (if the hw generates interrupts faster than the CPU
>>> can service them)
>> For this, I have no idea unless hardware supports some sort of mechanism
>>
>> to catch that. For the current providers, as soon as it detects changes on lines
>>
>> it captures TS in its hw fifo. Its interrupt gets generated based on threshold
>>
>> set in that hw fifo. This interrupt is different than the lines of actual device
>>
>> that is why I said I have no idea how we can tackle that. Let me know if there
>>
>> is any idea or reference of the codes which does tackle this.
>>
> As far as I am aware there is no solution, given your suggested
> architecture.
>
> Your architecture is inherently fragile, as you try to use one stream
> of data (the timestamp fifo) to provide supplementary info for another
> (the physical irq). Guaranteeing that the two are synchronised is
> impossible - even if you can get them synced at some point, they can
> fall out of sync without any indication.
> That is a recipe for Ingenuity flight 6.
>
> My solution would be to use the hte timestamp fifo as the event source,
> rather than the physical irq. With only one event source the
> synchronisation problem disappears. As to how to implement that,
> gpiolib-cdev would request a line from the hte subsystem and register
> and event handler for it, much as it does currently with the irq
> subsystem.
Regarding "
much as it does currently with the irq
subsystem
" Statment, do you mean edge_irq_handler?
> That event handler would translate the hte events into gpio
> events.
>
> You still have to deal with possible fifo overflows, but if the fifo
> overflows result in discarding the oldest event, rather than the most
> recent, then everything comes out in the wash. If not then the final
> event in a burst may not correspond to the actual state so you need
> some additional mechanism to address that.
> Either way the consumer needs to be aware that events may be lost - but
> with the event seqno for consumers to detect those lost events we
> already have that covered.
>
>> Regarding HTE/GPIOLIB integration comment:
>>
>> You are right, currently, I have only tsc field returned from struct hte_ts_data
>>
>> to gpiolib. If I can extend that to return hte_ts_data structure which has seq
>>
>> counter, which I believe can be used to track the overflow situation. The
>>
>> dropped scenario can be easily tracked if gpiolib can be notified with above
>>
>> mentioned DROP event through callback. If that is the case, is it ok to have
>>
>> some sort of callback per gpio in gpiolib?
>>
> Even better if you can provide the whole struct hte_ts_data so we have
> the direction as well (assuming all hte providers provide direction?).
> Otherwise gpiolib-cdev may need to read the physical line state and that
> may have changed since the hardware captured the event.
> In the solution I outlined above, the hte_ts_data would be provided to
> the event handler registered by gpiolib-cdev.
How is this event handler different then cdev providing callback to
hte core? I am guessing even cdev registers event handler with HTE
it is some sort of function pointer so does callbacks.
> And in this case you could skip buffering the event in hte - it could be
> passed to the event handler as soon as it is read from the hardware -
> gpiolib-cdev does its own buffering of gpio events.
>
>> Any idea how I can integrate callback notification with gpiolib if you do not agree on
>>
>> above callback suggestion?
>>
> See above. But this is just my take, so I would get feedback from the
> relevant maintainers or SMEs before you act on anything suggested above.
>
> Cheers,
> Kent.
>
next prev parent reply other threads:[~2021-08-03 22:31 UTC|newest]
Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-25 23:55 [RFC 00/11] Intro to Hardware timestamping engine Dipen Patel
2021-06-25 23:55 ` [RFC 01/11] Documentation: Add HTE subsystem guide Dipen Patel
2021-07-04 18:55 ` Jonathan Cameron
2021-07-27 23:44 ` Dipen Patel
2021-08-01 15:24 ` Jonathan Cameron
2021-06-25 23:55 ` [RFC 02/11] drivers: Add HTE subsystem Dipen Patel
2021-06-27 17:40 ` Randy Dunlap
2021-07-04 20:15 ` Jonathan Cameron
2021-07-04 20:45 ` Jonathan Cameron
2021-07-28 5:12 ` Dipen Patel
2021-08-01 16:48 ` Jonathan Cameron
2021-07-28 4:38 ` Dipen Patel
2021-08-01 16:13 ` Jonathan Cameron
2021-09-14 5:43 ` Dipen Patel
2021-09-26 15:42 ` Jonathan Cameron
2021-07-05 7:30 ` Greg KH
2021-07-28 0:34 ` Dipen Patel
2021-06-25 23:55 ` [RFC 03/11] hte: Add tegra194 HTE kernel provider Dipen Patel
2021-07-01 14:21 ` Kent Gibson
2021-07-28 23:59 ` Dipen Patel
2021-07-30 7:01 ` Dipen Patel
2021-07-31 15:43 ` Kent Gibson
2021-08-03 22:40 ` Dipen Patel [this message]
2021-08-03 23:02 ` Kent Gibson
2021-08-07 2:41 ` Dipen Patel
2021-08-07 3:07 ` Kent Gibson
2021-08-07 4:52 ` Dipen Patel
2021-08-07 4:51 ` Kent Gibson
2021-08-07 5:35 ` Dipen Patel
2021-08-07 5:42 ` Kent Gibson
2021-08-07 5:47 ` Dipen Patel
2021-07-04 20:27 ` Jonathan Cameron
2021-07-29 2:42 ` Dipen Patel
2021-07-08 23:33 ` Michał Mirosław
2021-07-29 2:43 ` Dipen Patel
2021-06-25 23:55 ` [RFC 04/11] dt-bindings: Add HTE bindings Dipen Patel
2021-06-27 10:56 ` Linus Walleij
2021-07-30 1:32 ` Dipen Patel
2021-07-01 14:02 ` Rob Herring
2021-07-30 1:56 ` Dipen Patel
2021-07-01 15:54 ` Rob Herring
2021-07-30 1:58 ` Dipen Patel
2021-06-25 23:55 ` [RFC 05/11] hte: Add Tegra194 IRQ HTE test driver Dipen Patel
2021-06-27 17:42 ` Randy Dunlap
2021-06-25 23:55 ` [RFC 06/11] gpiolib: Add HTE support Dipen Patel
2021-06-27 11:41 ` Linus Walleij
2021-07-01 14:24 ` Kent Gibson
2021-07-30 2:25 ` Dipen Patel
2021-07-31 5:13 ` Kent Gibson
2021-06-25 23:55 ` [RFC 07/11] gpio: tegra186: Add HTE in gpio-tegra186 driver Dipen Patel
2021-06-25 23:55 ` [RFC 08/11] gpiolib: cdev: Add hardware timestamp clock type Dipen Patel
2021-06-27 11:38 ` Linus Walleij
2021-06-27 11:49 ` Linus Walleij
2021-07-30 3:16 ` Dipen Patel
2021-07-01 14:24 ` Kent Gibson
2021-07-30 3:07 ` Dipen Patel
2021-07-31 6:05 ` Kent Gibson
2021-08-03 22:41 ` Dipen Patel
2021-08-03 22:38 ` Kent Gibson
2021-07-09 8:30 ` Jon Hunter
2021-07-30 2:33 ` Dipen Patel
2021-08-03 16:42 ` Jon Hunter
2021-08-03 22:51 ` Dipen Patel
2021-08-03 23:09 ` Kent Gibson
2021-06-25 23:55 ` [RFC 09/11] tools: gpio: Add new hardware " Dipen Patel
2021-06-27 11:36 ` Linus Walleij
2021-07-30 3:17 ` Dipen Patel
2021-07-31 6:16 ` Kent Gibson
2021-08-11 9:11 ` Linus Walleij
2021-06-25 23:55 ` [RFC 10/11] hte: Add tegra GPIO HTE test driver Dipen Patel
2021-06-27 17:43 ` Randy Dunlap
2021-06-25 23:55 ` [RFC 11/11] MAINTAINERS: Added HTE Subsystem Dipen Patel
2021-06-27 13:07 ` [RFC 00/11] Intro to Hardware timestamping engine Andy Shevchenko
2021-06-27 14:40 ` Linus Walleij
2021-06-28 12:02 ` Andy Shevchenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6429d26d-7835-c4f7-96b3-c8b0ba730187@nvidia.com \
--to=dipenp@nvidia.com \
--cc=bgolaszewski@baylibre.com \
--cc=devicetree@vger.kernel.org \
--cc=jonathanh@nvidia.com \
--cc=linus.walleij@linaro.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-gpio@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=robh+dt@kernel.org \
--cc=thierry.reding@gmail.com \
--cc=warthog618@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).