All of lore.kernel.org
 help / color / mirror / Atom feed
* [Xenomai] rtdm_timer_start latency?
@ 2018-02-08 23:45 Lowell Gilbert
  2018-02-09 10:02 ` Philippe Gerum
  0 siblings, 1 reply; 10+ messages in thread
From: Lowell Gilbert @ 2018-02-08 23:45 UTC (permalink / raw)
  To: xenomai

Am I correct in assuming that when calling rtdm_timer_start(), I should
not be getting multi-second latencies before the first call to the timer
routine? Just checking before I dig in too far.

Thanks.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Xenomai] rtdm_timer_start latency?
  2018-02-08 23:45 [Xenomai] rtdm_timer_start latency? Lowell Gilbert
@ 2018-02-09 10:02 ` Philippe Gerum
  2018-02-09 10:11   ` Philippe Gerum
  0 siblings, 1 reply; 10+ messages in thread
From: Philippe Gerum @ 2018-02-09 10:02 UTC (permalink / raw)
  To: Lowell Gilbert, xenomai

On 02/09/2018 12:45 AM, Lowell Gilbert wrote:
> Am I correct in assuming that when calling rtdm_timer_start(), I should
> not be getting multi-second latencies before the first call to the timer
> routine? Just checking before I dig in too far.
> 

Of course not. I suspect an absolute expiry time is given with a
relative mode spec.

-- 
Philippe.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Xenomai] rtdm_timer_start latency?
  2018-02-09 10:02 ` Philippe Gerum
@ 2018-02-09 10:11   ` Philippe Gerum
  2018-02-09 14:50     ` Lowell Gilbert
  0 siblings, 1 reply; 10+ messages in thread
From: Philippe Gerum @ 2018-02-09 10:11 UTC (permalink / raw)
  To: Lowell Gilbert, xenomai

On 02/09/2018 11:02 AM, Philippe Gerum wrote:
> On 02/09/2018 12:45 AM, Lowell Gilbert wrote:
>> Am I correct in assuming that when calling rtdm_timer_start(), I should
>> not be getting multi-second latencies before the first call to the timer
>> routine? Just checking before I dig in too far.
>>
> 
> Of course not. I suspect an absolute expiry time is given with a
> relative mode spec.
> 

More generally, an absolute expiry time which is not based on the right
clock specified by the mode argument.

i.e.

expiry = rtdm_clock_monotonic(), mode = RTDM_TIMERMODE_REALTIME => BAD
expiry = rtdm_clock_realtime(), mode = RTDM_TIMERMODE_ABSOLUTE => BAD
expiry = rtdm_clock_realtime() or _monotonic(), mode =
RTDM_TIMERMODE_RELATIVE => BAD

-- 
Philippe.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Xenomai] rtdm_timer_start latency?
  2018-02-09 10:11   ` Philippe Gerum
@ 2018-02-09 14:50     ` Lowell Gilbert
  2018-02-09 15:00       ` Lowell Gilbert
  2018-02-12 13:41       ` Lowell Gilbert
  0 siblings, 2 replies; 10+ messages in thread
From: Lowell Gilbert @ 2018-02-09 14:50 UTC (permalink / raw)
  To: Philippe Gerum; +Cc: xenomai

Philippe Gerum <rpm@xenomai.org> writes:

> On 02/09/2018 11:02 AM, Philippe Gerum wrote:
>> On 02/09/2018 12:45 AM, Lowell Gilbert wrote:
>>> Am I correct in assuming that when calling rtdm_timer_start(), I should
>>> not be getting multi-second latencies before the first call to the timer
>>> routine? Just checking before I dig in too far.
>>>
>> 
>> Of course not. I suspect an absolute expiry time is given with a
>> relative mode spec.
>> 
>
> More generally, an absolute expiry time which is not based on the right
> clock specified by the mode argument.
>
> i.e.
>
> expiry = rtdm_clock_monotonic(), mode = RTDM_TIMERMODE_REALTIME => BAD
> expiry = rtdm_clock_realtime(), mode = RTDM_TIMERMODE_ABSOLUTE => BAD
> expiry = rtdm_clock_realtime() or _monotonic(), mode =
> RTDM_TIMERMODE_RELATIVE => BAD

Okay, that makes sense.

I was using 0 for expiry, because I really just wanted the periodic
wakeup. I don't remember why I didn't create a periodic task instead of
using a timer.

What clock do I want to use to get the timer started? Once it starts, it
runs fine; but it often takes two or three seconds before the first call
into the handler.

To put it another way, I'm trying to figure out what am I doing wrong in:
		ret = rtdm_timer_start(&pstate.collect.timer, rtdm_clock_read() + 1,
				       pstate.collect.period,
				       RTDM_TIMERMODE_RELATIVE);

Thanks again.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Xenomai] rtdm_timer_start latency?
  2018-02-09 14:50     ` Lowell Gilbert
@ 2018-02-09 15:00       ` Lowell Gilbert
  2018-02-12 13:41       ` Lowell Gilbert
  1 sibling, 0 replies; 10+ messages in thread
From: Lowell Gilbert @ 2018-02-09 15:00 UTC (permalink / raw)
  To: Lowell Gilbert; +Cc: xenomai

Lowell Gilbert <kludge@be-well.ilk.org> writes:

> To put it another way, I'm trying to figure out what am I doing wrong in:
> 		ret = rtdm_timer_start(&pstate.collect.timer, rtdm_clock_read() + 1,
> 				       pstate.collect.period,
> 				       RTDM_TIMERMODE_RELATIVE);

On second thought, and application of more coffee, that's clearly
wrong. My putting a zero in for the second parameter (as I did
originally) seems like what I would expect to work.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Xenomai] rtdm_timer_start latency?
  2018-02-09 14:50     ` Lowell Gilbert
  2018-02-09 15:00       ` Lowell Gilbert
@ 2018-02-12 13:41       ` Lowell Gilbert
  2018-02-12 14:25         ` Lange Norbert
  2018-02-13 17:04         ` Lowell Gilbert
  1 sibling, 2 replies; 10+ messages in thread
From: Lowell Gilbert @ 2018-02-12 13:41 UTC (permalink / raw)
  To: xenomai

Lowell Gilbert <kludge@be-well.ilk.org> writes:

> Philippe Gerum <rpm@xenomai.org> writes:
>
>> On 02/09/2018 11:02 AM, Philippe Gerum wrote:
>>> On 02/09/2018 12:45 AM, Lowell Gilbert wrote:
>>>> Am I correct in assuming that when calling rtdm_timer_start(), I should
>>>> not be getting multi-second latencies before the first call to the timer
>>>> routine? Just checking before I dig in too far.
>>>>
>>> 
>>> Of course not. I suspect an absolute expiry time is given with a
>>> relative mode spec.
>>> 
>>
>> More generally, an absolute expiry time which is not based on the right
>> clock specified by the mode argument.
>>
>> i.e.
>>
>> expiry = rtdm_clock_monotonic(), mode = RTDM_TIMERMODE_REALTIME => BAD
>> expiry = rtdm_clock_realtime(), mode = RTDM_TIMERMODE_ABSOLUTE => BAD
>> expiry = rtdm_clock_realtime() or _monotonic(), mode =
>> RTDM_TIMERMODE_RELATIVE => BAD
>
> Okay, that makes sense.
>
> I was using 0 for expiry, because I really just wanted the periodic
> wakeup. I don't remember why I didn't create a periodic task instead of
> using a timer.
>
> What clock do I want to use to get the timer started? Once it starts, it
> runs fine; but it often takes two or three seconds before the first call
> into the handler.
>
> To put it another way, I'm trying to figure out what am I doing wrong in:
> 		ret = rtdm_timer_start(&pstate.collect.timer, 1,
> 				       pstate.collect.period,
> 				       RTDM_TIMERMODE_RELATIVE);

I could just rewrite the code to use a periodic task, but I'd prefer to
understand what is actually wrong with my current code first, to be sure
that refactoring will really solve my problem.

Or I could have the timer run constantly, but that is just hacking
around the problem, not really solving it.

Thanks for all help, past and future.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Xenomai] rtdm_timer_start latency?
  2018-02-12 13:41       ` Lowell Gilbert
@ 2018-02-12 14:25         ` Lange Norbert
  2018-02-12 16:32           ` Lowell Gilbert
  2018-02-13 17:04         ` Lowell Gilbert
  1 sibling, 1 reply; 10+ messages in thread
From: Lange Norbert @ 2018-02-12 14:25 UTC (permalink / raw)
  To: Lowell Gilbert, xenomai

Hello,

this seems identical to an issue I have. I wanted to start a millisecond tick, aligned to seconds, but used the POSIX timerfd_settime.
Under Linux this works as intendent, under cobalt it doesn't.

I  expect that the first timeout would be somewhat close to the call, on a "whole" millisecond.

What happens is that the timer-function convert the (Nano-)seconds to the internal resolution. This has a small error, which normally won't matter for short timespans.
The problem here is, that the absolute time will get converted with a big (absolute) error - growing bigger the longer your system is up.

What you could do, is reading the current time and fixing it up to a whole tick close to it, name this prev_secondtime, then use this as initial value.
Now the driver will have to fixup nothing or only a small amount, keeping the error low.

You can also try fixing this at driver level, but I haven't dissected this carefully yet.
Basically the driver should calculate in the "source time" for as long as possible and only use the converted internal value for close events to keep conversion errors bounded.

Norbert

Highly experimental patch for ./kernel/cobalt/timer.c

--- timer.c.org 2017-12-18 10:09:37.283116000 +0100
+++ timer.c     2017-12-18 10:39:36.085302719 +0100
@@ -116,7 +116,7 @@
 {
        struct xnclock *clock = xntimer_clock(timer);
        xntimerq_t *q = xntimer_percpu_queue(timer);
-       xnticks_t date, now, delay, period;
+       xnticks_t date, now, delay, period, expired = 0;
        unsigned long gravity;
        int ret = 0;

@@ -141,6 +141,7 @@
        default: /* XN_ABSOLUTE || XN_REALTIME */
                date = xnclock_ns_to_ticks(clock, value);
                if ((xnsticks_t)(date - now) <= 0) {
+                       _ xnticks t nextvalue;
                        if (interval == XN_INFINITE)
                                return -ETIMEDOUT;
                        /*
@@ -148,9 +149,17 @@
                         * delivery, wait for the next shot on the
                         * periodic time line.
                         */
-                       delay = now - date;
+                       expired = xnarch_div64(xnclock_ticks_to_ns(now - date), interval);
+                       nextvalue = value + interval * (expired + 1);
+
+                       /*
+                        * Note: need to think about errors for big offsets,
+                        * iteration of the correction is likely necessary,
+                        * until the value is within the boundaries [now, now + period)
+                        */
+                       date = xnclock_ns_to_ticks(clock, nextvalue);
+
                        period = xnclock_ns_to_ticks(clock, interval);
-                       date += period * (xnarch_div64(delay, period) + 1);
                }
                break;
        }

 >-----Original Message-----
 >From: Xenomai [mailto:xenomai-bounces@xenomai.org] On Behalf Of Lowell
 >Gilbert
 >Sent: Montag, 12. Februar 2018 14:41
 >To: xenomai@xenomai.org
 >Subject: Re: [Xenomai] rtdm_timer_start latency?
 >
 >E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
 >EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
 >ATTACHMENTS.
 >
 >
 >Lowell Gilbert <kludge@be-well.ilk.org> writes:
 >
 >> Philippe Gerum <rpm@xenomai.org> writes:
 >>
 >>> On 02/09/2018 11:02 AM, Philippe Gerum wrote:
 >>>> On 02/09/2018 12:45 AM, Lowell Gilbert wrote:
 >>>>> Am I correct in assuming that when calling rtdm_timer_start(), I should
 >>>>> not be getting multi-second latencies before the first call to the timer
 >>>>> routine? Just checking before I dig in too far.
 >>>>>
 >>>>
 >>>> Of course not. I suspect an absolute expiry time is given with a
 >>>> relative mode spec.
 >>>>
 >>>
 >>> More generally, an absolute expiry time which is not based on the right
 >>> clock specified by the mode argument.
 >>>
 >>> i.e.
 >>>
 >>> expiry = rtdm_clock_monotonic(), mode = RTDM_TIMERMODE_REALTIME
 >=> BAD
 >>> expiry = rtdm_clock_realtime(), mode = RTDM_TIMERMODE_ABSOLUTE =>
 >BAD
 >>> expiry = rtdm_clock_realtime() or _monotonic(), mode =
 >>> RTDM_TIMERMODE_RELATIVE => BAD
 >>
 >> Okay, that makes sense.
 >>
 >> I was using 0 for expiry, because I really just wanted the periodic
 >> wakeup. I don't remember why I didn't create a periodic task instead of
 >> using a timer.
 >>
 >> What clock do I want to use to get the timer started? Once it starts, it
 >> runs fine; but it often takes two or three seconds before the first call
 >> into the handler.
 >>
 >> To put it another way, I'm trying to figure out what am I doing wrong in:
 >>               ret = rtdm_timer_start(&pstate.collect.timer, 1,
 >>                                      pstate.collect.period,
 >>                                      RTDM_TIMERMODE_RELATIVE);
 >
 >I could just rewrite the code to use a periodic task, but I'd prefer to
 >understand what is actually wrong with my current code first, to be sure
 >that refactoring will really solve my problem.
 >
 >Or I could have the timer run constantly, but that is just hacking
 >around the problem, not really solving it.
 >
 >Thanks for all help, past and future.
 >
 >_______________________________________________
 >Xenomai mailing list
 >Xenomai@xenomai.org
 >https://xenomai.org/mailman/listinfo/xenomai
________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Xenomai] rtdm_timer_start latency?
  2018-02-12 14:25         ` Lange Norbert
@ 2018-02-12 16:32           ` Lowell Gilbert
  0 siblings, 0 replies; 10+ messages in thread
From: Lowell Gilbert @ 2018-02-12 16:32 UTC (permalink / raw)
  To: Lange Norbert; +Cc: xenomai

Lange Norbert <norbert.lange@andritz.com> writes:

> this seems identical to an issue I have. I wanted to start a
> millisecond tick, aligned to seconds, but used the POSIX
> timerfd_settime.  Under Linux this works as intendent, under cobalt it
> doesn't.
>
> I expect that the first timeout would be somewhat close to the call,
> on a "whole" millisecond.
>
> What happens is that the timer-function convert the (Nano-)seconds to
> the internal resolution. This has a small error, which normally won't
> matter for short timespans.  The problem here is, that the absolute
> time will get converted with a big (absolute) error - growing bigger
> the longer your system is up.
>
> What you could do, is reading the current time and fixing it up to a
> whole tick close to it, name this prev_secondtime, then use this as
> initial value.  Now the driver will have to fixup nothing or only a
> small amount, keeping the error low.

It seems like a different problem to me, because I'm working in relative
mode, so such rounding shouldn't have a chance to get very large.

Also, I have a hardware timer that allows me to kick off much more often
than the system tick. Once the timer starts, it fires quite accurately
as often as every 10 microseconds.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Xenomai] rtdm_timer_start latency?
  2018-02-12 13:41       ` Lowell Gilbert
  2018-02-12 14:25         ` Lange Norbert
@ 2018-02-13 17:04         ` Lowell Gilbert
  2018-02-13 19:36           ` Auel, Kendall
  1 sibling, 1 reply; 10+ messages in thread
From: Lowell Gilbert @ 2018-02-13 17:04 UTC (permalink / raw)
  To: xenomai

Lowell Gilbert <kludge@be-well.ilk.org> writes:

> Lowell Gilbert <kludge@be-well.ilk.org> writes:
>
>> Philippe Gerum <rpm@xenomai.org> writes:
>>
>>> On 02/09/2018 11:02 AM, Philippe Gerum wrote:
>>>> On 02/09/2018 12:45 AM, Lowell Gilbert wrote:
>>>>> Am I correct in assuming that when calling rtdm_timer_start(), I should
>>>>> not be getting multi-second latencies before the first call to the timer
>>>>> routine? Just checking before I dig in too far.
>>>>>
>>>> 
>>>> Of course not. I suspect an absolute expiry time is given with a
>>>> relative mode spec.
>>>> 
>>>
>>> More generally, an absolute expiry time which is not based on the right
>>> clock specified by the mode argument.
>>>
>>> i.e.
>>>
>>> expiry = rtdm_clock_monotonic(), mode = RTDM_TIMERMODE_REALTIME => BAD
>>> expiry = rtdm_clock_realtime(), mode = RTDM_TIMERMODE_ABSOLUTE => BAD
>>> expiry = rtdm_clock_realtime() or _monotonic(), mode =
>>> RTDM_TIMERMODE_RELATIVE => BAD
>>
>> Okay, that makes sense.
>>
>> I was using 0 for expiry, because I really just wanted the periodic
>> wakeup. I don't remember why I didn't create a periodic task instead of
>> using a timer.
>>
>> What clock do I want to use to get the timer started? Once it starts, it
>> runs fine; but it often takes two or three seconds before the first call
>> into the handler.
>>
>> To put it another way, I'm trying to figure out what am I doing wrong in:
>> 		ret = rtdm_timer_start(&pstate.collect.timer, 1,
>> 				       pstate.collect.period,
>> 				       RTDM_TIMERMODE_RELATIVE);
>
> I could just rewrite the code to use a periodic task, but I'd prefer to
> understand what is actually wrong with my current code first, to be sure
> that refactoring will really solve my problem.
>
> Or I could have the timer run constantly, but that is just hacking
> around the problem, not really solving it.

I have a feature request that would require me to run the data collection
constantly (or at least whenever there's a consumer ready to receive it),
so that will ameliorate the problems for me. However, I'd prefer to
understand what's happening, because as far as I understand at the moment,
what I am doing *should* work.

Also, as a matter of design, is there any reason to prefer a periodic task
over a timer or vice versa? It can potentially cycle at fairly high rates
(a period as low as 10 microseconds), although expected use cases will not.

Be well.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [Xenomai] rtdm_timer_start latency?
  2018-02-13 17:04         ` Lowell Gilbert
@ 2018-02-13 19:36           ` Auel, Kendall
  0 siblings, 0 replies; 10+ messages in thread
From: Auel, Kendall @ 2018-02-13 19:36 UTC (permalink / raw)
  To: xenomai

I encountered a similar problem with a hardware clock implementation. Our implementation of "ns_to_ticks" for the xnclock struct had a very small truncation error. Immediately after booting the system, we didn't see a problem. In that case the absolute time was small, thus making the truncation error negligible. However, after the system had been up a day or two, relative timers had a very large initial delay (100s of ms). Eventually the startup delays were many seconds, if the clock had been running for days. For relative time durations, the truncation in the time conversion was also negligible, so after the initial delay there was no problem.

You might take a look at ...xenomai/clock.c, there are a number of #ifdef statements for choosing which math functions to use.

In our case, the truncation occurred with this code (in our custom driver for the hardware timer):

	ticks = xnarch_ulldiv(ns, 1000000000/TIMER_HZ, &rem);
	if (rem)
		ticks++;
	return ticks;

Note the TIMER_HZ value is just a little less than 40000. Thus the truncation error is in the range from 0.0000000 to 0.000025 of the original ns value. In our case it was about 0.000010. If we didn't have a random-looking clock frequency we would have never noticed the issue.

Anyway, to fix the problem, we changed it to:

	return xnarch_llimd(ns, TIMER_HZ, 1000000000);

I hope this helps,
Kendall

(PS - does anybody have a good MS-Outlook compatible tool for bottom-posting?)

> -----Original Message-----
> From: Xenomai [mailto:xenomai-bounces@xenomai.org] On Behalf Of
> Lowell Gilbert
> Sent: Tuesday, February 13, 2018 9:04 AM
> To: xenomai@xenomai.org
> Subject: Re: [Xenomai] rtdm_timer_start latency?
> 
> Lowell Gilbert <kludge@be-well.ilk.org> writes:
> 
> > Lowell Gilbert <kludge@be-well.ilk.org> writes:
> >
> >> Philippe Gerum <rpm@xenomai.org> writes:
> >>
> >>> On 02/09/2018 11:02 AM, Philippe Gerum wrote:
> >>>> On 02/09/2018 12:45 AM, Lowell Gilbert wrote:
> >>>>> Am I correct in assuming that when calling rtdm_timer_start(), I
> >>>>> should not be getting multi-second latencies before the first call
> >>>>> to the timer routine? Just checking before I dig in too far.
> >>>>>
> >>>>
> >>>> Of course not. I suspect an absolute expiry time is given with a
> >>>> relative mode spec.
> >>>>
> >>>
> >>> More generally, an absolute expiry time which is not based on the
> >>> right clock specified by the mode argument.
> >>>
> >>> i.e.
> >>>
> >>> expiry = rtdm_clock_monotonic(), mode =
> RTDM_TIMERMODE_REALTIME =>
> >>> BAD expiry = rtdm_clock_realtime(), mode =
> RTDM_TIMERMODE_ABSOLUTE
> >>> => BAD expiry = rtdm_clock_realtime() or _monotonic(), mode =
> >>> RTDM_TIMERMODE_RELATIVE => BAD
> >>
> >> Okay, that makes sense.
> >>
> >> I was using 0 for expiry, because I really just wanted the periodic
> >> wakeup. I don't remember why I didn't create a periodic task instead
> >> of using a timer.
> >>
> >> What clock do I want to use to get the timer started? Once it starts,
> >> it runs fine; but it often takes two or three seconds before the
> >> first call into the handler.
> >>
> >> To put it another way, I'm trying to figure out what am I doing wrong in:
> >> 		ret = rtdm_timer_start(&pstate.collect.timer, 1,
> >> 				       pstate.collect.period,
> >> 				       RTDM_TIMERMODE_RELATIVE);
> >
> > I could just rewrite the code to use a periodic task, but I'd prefer
> > to understand what is actually wrong with my current code first, to be
> > sure that refactoring will really solve my problem.
> >
> > Or I could have the timer run constantly, but that is just hacking
> > around the problem, not really solving it.
> 
> I have a feature request that would require me to run the data collection
> constantly (or at least whenever there's a consumer ready to receive it), so
> that will ameliorate the problems for me. However, I'd prefer to understand
> what's happening, because as far as I understand at the moment, what I am
> doing *should* work.
> 
> Also, as a matter of design, is there any reason to prefer a periodic task over
> a timer or vice versa? It can potentially cycle at fairly high rates (a period as
> low as 10 microseconds), although expected use cases will not.
> 
> Be well.
> 
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> https://xenomai.org/mailman/listinfo/xenomai


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2018-02-13 19:36 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-08 23:45 [Xenomai] rtdm_timer_start latency? Lowell Gilbert
2018-02-09 10:02 ` Philippe Gerum
2018-02-09 10:11   ` Philippe Gerum
2018-02-09 14:50     ` Lowell Gilbert
2018-02-09 15:00       ` Lowell Gilbert
2018-02-12 13:41       ` Lowell Gilbert
2018-02-12 14:25         ` Lange Norbert
2018-02-12 16:32           ` Lowell Gilbert
2018-02-13 17:04         ` Lowell Gilbert
2018-02-13 19:36           ` Auel, Kendall

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.