* What exactly does -F <freq> do?
@ 2022-03-05 0:51 Douglas Graham
2022-03-08 12:23 ` James Clark
0 siblings, 1 reply; 6+ messages in thread
From: Douglas Graham @ 2022-03-05 0:51 UTC (permalink / raw)
To: linux-perf-users
Hi,
The perf-record man page says this about the -F option:
-F, --freq=
Profile at this frequency.
And this about the -c option:
-c, --count=
Event period to sample.
A "perf stat sleep 1" on the system I'm interested in says:
1630370 cycles # 1.315 GHz
As I understand it, the cycles event is used by default, so a 100 Hz sample frequency should give the same results as a 13150000 cycles sample period.
And yet the results with "-F 100" are significantly different than those with "-c 13150000". Perf script output shows the same number of cycles on all samples collected with -c, eg:
tmoSchedulerTx 2281 [003] 32746.853052: 13150000 cycles: 55834b1e5c [unknown] (/opt/foo)
richw_commond 1950 [003] 32746.926042: 13150000 cycles: ffffffc010013504 do_el0_svc+0x44 ([kernel.kallsyms])
swapper 0 [000] 32746.946399: 13150000 cycles: ffffffc0107d1828 arch_cpu_idle+0x14 ([kernel.kallsyms])
richw_commond 1889 [003] 32747.002987: 13150000 cycles: ffffffc0107dba24 _raw_spin_unlock_irqrestore+0x14 ([kernel.kallsyms])
whereas the number of cycles bounces around in the perf -F output, eg:
tmoSchedulerTx 2281 [001] 32696.168404: 5904091 cycles: 55834b1988 [unknown] (/opt/foo)
startUp 2180 [001] 32696.209523: 6148607 cycles: 7fb235de68 operator delete@plt+0x8 (/usr/lib/foo.so.0.0.0)
swapper 0 [001] 32696.348061: 6416807 cycles: ffffffc0107dbaa8 _raw_spin_unlock_irq+0x18 ([kernel.kallsyms])
richw_commond 1889 [002] 32696.501004: 21730400 cycles: ffffffc0107dbaa8 _raw_spin_unlock_irq+0x18 ([kernel.kallsyms])
I can understand why perf would have to keep adjusting the number of events between samples if it is targeting an constant average sample frequency but using an event that occurs at a variable rate (eg: cache misses), but cycles *does* occur at a fixed rate. So why does the number of cycles between samples keep changing (often radically) like this? What exactly is perf aiming for?
Other statistical profilers (eg: gprof) sample at a fixed frequency, where you can assume that if the sampling frequency is 100Hz, then each sample represents roughly 10ms. A function that was sampled 100 times during a test run ran for roughly 1 second during that run. I think the same can be said if perf record is used with the -c option to ensure a constant sampling period. But I have no idea how to interpret the results of a "perf record -F 100". It almost looks like it's trying to capture 100 non-idle samples every second, but if that's what it's doing, I think the result can be wildly inaccurate. If it's constantly adjust the sample rate to try to ensure a fixed number of non-idle samples per second, then it will sample more often when the system is lightly loaded, and that will make whatever happened during that time look more CPU intensive than it really was, when analyzed together with the samples gathered when the system was busy.
Thanks,
Doug
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: What exactly does -F <freq> do?
2022-03-05 0:51 What exactly does -F <freq> do? Douglas Graham
@ 2022-03-08 12:23 ` James Clark
2022-03-09 1:05 ` Douglas Graham
0 siblings, 1 reply; 6+ messages in thread
From: James Clark @ 2022-03-08 12:23 UTC (permalink / raw)
To: Douglas Graham, linux-perf-users
On 05/03/2022 00:51, Douglas Graham wrote:
> Hi,
>
> The perf-record man page says this about the -F option:
>
> -F, --freq=
> Profile at this frequency.
>
> And this about the -c option:
>
> -c, --count=
> Event period to sample.
>
> A "perf stat sleep 1" on the system I'm interested in says:
>
> 1630370 cycles # 1.315 GHz
>
> As I understand it, the cycles event is used by default, so a 100 Hz sample frequency should give the same results as a 13150000 cycles sample period.
>
> And yet the results with "-F 100" are significantly different than those with "-c 13150000". Perf script output shows the same number of cycles on all samples collected with -c, eg:
>
> tmoSchedulerTx 2281 [003] 32746.853052: 13150000 cycles: 55834b1e5c [unknown] (/opt/foo)
> richw_commond 1950 [003] 32746.926042: 13150000 cycles: ffffffc010013504 do_el0_svc+0x44 ([kernel.kallsyms])
> swapper 0 [000] 32746.946399: 13150000 cycles: ffffffc0107d1828 arch_cpu_idle+0x14 ([kernel.kallsyms])
> richw_commond 1889 [003] 32747.002987: 13150000 cycles: ffffffc0107dba24 _raw_spin_unlock_irqrestore+0x14 ([kernel.kallsyms])
>
> whereas the number of cycles bounces around in the perf -F output, eg:
>
> tmoSchedulerTx 2281 [001] 32696.168404: 5904091 cycles: 55834b1988 [unknown] (/opt/foo)
> startUp 2180 [001] 32696.209523: 6148607 cycles: 7fb235de68 operator delete@plt+0x8 (/usr/lib/foo.so.0.0.0)
> swapper 0 [001] 32696.348061: 6416807 cycles: ffffffc0107dbaa8 _raw_spin_unlock_irq+0x18 ([kernel.kallsyms])
> richw_commond 1889 [002] 32696.501004: 21730400 cycles: ffffffc0107dbaa8 _raw_spin_unlock_irq+0x18 ([kernel.kallsyms])
>
> I can understand why perf would have to keep adjusting the number of events between samples if it is targeting an constant average sample frequency but using an event that occurs at a variable rate (eg: cache misses), but cycles *does* occur at a fixed rate. So why does the number of cycles between samples keep changing (often radically) like this? What exactly is perf aiming for?
>
> Other statistical profilers (eg: gprof) sample at a fixed frequency, where you can assume that if the sampling frequency is 100Hz, then each sample represents roughly 10ms. A function that was sampled 100 times during a test run ran for roughly 1 second during that run. I think the same can be said if perf record is used with the -c option to ensure a constant sampling period. But I have no idea how to interpret the results of a "perf record -F 100". It almost looks like it's trying to capture 100 non-idle samples every second, but if that's what it's doing, I think the result can be wildly inaccurate. If it's constantly adjust the sample rate to try to ensure a fixed number of non-idle samples per second, then it will sample more often when the system is lightly loaded, and that will make whatever happened during that time look more CPU intensive than it really was, when analyzed together with the samples gathered when the system was busy.
Yes you are right, perf only records when the process is scheduled on the CPU. If you're counting cycles
and that process isn't scheduled then cycles isn't increasing so you will get no samples.
If you have a process that has a lot of idle time then maybe -c is better than -F. It also sounds like you
are looking for a wall clock time profiler which perf isn't. You might also be able to play around with
-C or -a which change from tracking a single process to everything on that core/all cores. You still wouldn't get
sleeping samples, but you would get the samples padded out by whatever else was running on the core instead
of your sleeping process so -F might be more accurate.
I suppose perf is just trying to give an accurate representation of what the system is actually doing rather
than padding out with samples of processes that weren't even scheduled like a wall clock time profiler would do.
>
> Thanks,
> Doug
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: What exactly does -F <freq> do?
2022-03-08 12:23 ` James Clark
@ 2022-03-09 1:05 ` Douglas Graham
2022-03-09 11:03 ` James Clark
0 siblings, 1 reply; 6+ messages in thread
From: Douglas Graham @ 2022-03-09 1:05 UTC (permalink / raw)
To: James Clark, linux-perf-users
Thanks for the reply.
> Yes you are right, perf only records when the process is scheduled on
> the CPU. If you're counting cycles and that process isn't scheduled
> then cycles isn't increasing so you will get no samples.
Maybe there is something fundamental about how perf record works that I'm not getting.
I'm assuming that cycles is always incrementing, regardless of what is happening in
software or what thread is running. The only thing that might change its frequency is
CPU frequency scaling. Without freqency scaling, cycles is incremented at a constant rate. I assume
the way it works when I use -c <sample period> is that it writes the specified sample period to a
PMU register. When the number of elapsed cycles hits that limit, an interrupt is generated.
The ISR examines the state of the CPU. If it is idle, then no sample is saved. If a thread
is running, its program counter (and possibly a whole backtrace) is saved. So if the CPU
is 10% busy, only 10% of the sampling interrupts will actually wind up saving a sample.
You seem to be saying that the sampling interrupts are only generated when a thread is running
and that they all result in a sample being saved. I think this make sense when the event being
monitored is caused by something the thread does. For example, you won't get any cache
misses if nothing is running, so if your event is cache misses, you will never get a sampling interrupt
while the CPU is idle. But unless I'm missing something, cycles increments regardless of what
the software is doing, so you get those interrupts no matter what is happening in software.
If I'm right about cycles incrementing at a fixed rate, then the -c <sample_period> option will result
in a sampling interrupt at a fixed frequency. If the clock rate is known, then converting sample_period
to sample frequency and vice versa should be simple. And yet, -F <freq> seems to do something quite
different than -c <sample_period>.
> If you have a process that has a lot of idle time then maybe -c
> is better than -F. It also sounds like you are looking for a wall
> clock time profiler which perf isn't.
I am profiling a multithreaded application running on a quad core processor. I mainly just want to
make sure I understand what perf is actually doing so I know how to interpret the results. I know what
the results from a fixed sampling period mean, but I'm having a hard time wrapping my head around
what the results from -F mean. This guy: http://sandsoftwaresound.net/perf/perf-tut-profile-hw-events
seems to have come to the same conclusion.
I am already using -a.
Since most or all of this discussion seems to hinge on what happens when a core is idle, this
https://www.brendangregg.com/perf.html#FlameGraphs might be relevant. It contains some
mention of filtering out cpu_idle samples:
I frequently elide the cpu_idle threads in this way, to focus on the real threads that are consuming CPU resources
If I miss this step, the cpu_idle threads can often dominate the flame graph, squeezing the interesting code paths.
I don't see a lot of cpu_idle samples so maybe this is obsolete but it does imply that perf record -F did at one time
generate lots of idle samples.
Regards,
Doug
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: What exactly does -F <freq> do?
2022-03-09 1:05 ` Douglas Graham
@ 2022-03-09 11:03 ` James Clark
2022-03-09 18:36 ` Douglas Graham
0 siblings, 1 reply; 6+ messages in thread
From: James Clark @ 2022-03-09 11:03 UTC (permalink / raw)
To: Douglas Graham, linux-perf-users
On 09/03/2022 01:05, Douglas Graham wrote:
> Thanks for the reply.
>
> > Yes you are right, perf only records when the process is scheduled on
>> the CPU. If you're counting cycles and that process isn't scheduled
>> then cycles isn't increasing so you will get no samples.
>
> Maybe there is something fundamental about how perf record works that I'm not getting.
> I'm assuming that cycles is always incrementing, regardless of what is happening in
> software or what thread is running. The only thing that might change its frequency is
> CPU frequency scaling. Without freqency scaling, cycles is incremented at a constant rate. I assume
> the way it works when I use -c <sample period> is that it writes the specified sample period to a
> PMU register. When the number of elapsed cycles hits that limit, an interrupt is generated.
> The ISR examines the state of the CPU. If it is idle, then no sample is saved. If a thread
> is running, its program counter (and possibly a whole backtrace) is saved. So if the CPU
> is 10% busy, only 10% of the sampling interrupts will actually wind up saving a sample.
>
> You seem to be saying that the sampling interrupts are only generated when a thread is running
> and that they all result in a sample being saved. I think this make sense when the event being
> monitored is caused by something the thread does. For example, you won't get any cache
> misses if nothing is running, so if your event is cache misses, you will never get a sampling interrupt
> while the CPU is idle. But unless I'm missing something, cycles increments regardless of what
> the software is doing, so you get those interrupts no matter what is happening in software.
>
> If I'm right about cycles incrementing at a fixed rate, then the -c <sample_period> option will result
> in a sampling interrupt at a fixed frequency. If the clock rate is known, then converting sample_period
> to sample frequency and vice versa should be simple. And yet, -F <freq> seems to do something quite
> different than -c <sample_period>.
>
>> If you have a process that has a lot of idle time then maybe -c
>> is better than -F. It also sounds like you are looking for a wall
>> clock time profiler which perf isn't.
>
> I am profiling a multithreaded application running on a quad core processor. I mainly just want to
> make sure I understand what perf is actually doing so I know how to interpret the results. I know what
> the results from a fixed sampling period mean, but I'm having a hard time wrapping my head around
> what the results from -F mean. This guy: http://sandsoftwaresound.net/perf/perf-tut-profile-hw-events
> seems to have come to the same conclusion.
>
> I am already using -a.
In that case it does change some of my assumptions. But for this part of your message:
> A "perf stat sleep 1" on the system I'm interested in says:
> 1630370 cycles # 1.315 GHz
> As I understand it, the cycles event is used by default, so a 100 Hz sample frequency should give the same results as a 13150000 cycles sample period.
That command is not system wide or per-CPU which is what my assumption was based one (it's per process
mode). So it's not 1.6M cycles in 1 second, it's 1.6M cycles of when the sleep process was actively
launching and running, excluding the 1s of sleep. System wide would be 'perf stat -a -- sleep 1'
If you do some busy wait for 1 second you'd see significantly more cycles counted:
./perf stat -- stress -c 1 --timeout 1
Performance counter stats for 'stress -c 1 --timeout 1':
995.68 msec task-clock:u # 0.992 CPUs utilized
3,705,040,713 cycles:u # 3.721 GHz
>
> Since most or all of this discussion seems to hinge on what happens when a core is idle, this
> https://www.brendangregg.com/perf.html#FlameGraphs might be relevant. It contains some
> mention of filtering out cpu_idle samples:
>
> I frequently elide the cpu_idle threads in this way, to focus on the real threads that are consuming CPU resources
> If I miss this step, the cpu_idle threads can often dominate the flame graph, squeezing the interesting code paths.
>
> I don't see a lot of cpu_idle samples so maybe this is obsolete but it does imply that perf record -F did at one time
> generate lots of idle samples.
>
> Regards,
> Doug
^ permalink raw reply [flat|nested] 6+ messages in thread* RE: What exactly does -F <freq> do?
2022-03-09 11:03 ` James Clark
@ 2022-03-09 18:36 ` Douglas Graham
2022-03-10 8:52 ` Namhyung Kim
0 siblings, 1 reply; 6+ messages in thread
From: Douglas Graham @ 2022-03-09 18:36 UTC (permalink / raw)
To: James Clark, linux-perf-users
>> I am already using -a.
>
>In that case it does change some of my assumptions. But for this part of your message:
>
> > A "perf stat sleep 1" on the system I'm interested in says:
>
> > 1630370 cycles # 1.315 GHz
>
> > As I understand it, the cycles event is used by default, so a 100 Hz sample frequency
> > should give the same results as a 13150000 cycles sample period.
>
> That command is not system wide or per-CPU which is what my assumption was based one
> (it's per process mode). So it's not 1.6M cycles in 1 second, it's 1.6M cycles of when
> the sleep process was actively launching and running, excluding the 1s of sleep. System
> wide would be 'perf stat -a -- sleep 1'
It could just be me, but I think perf is in dire need of more comprehensive documentation.
Like, for example, how is that 1.315 GHz computed? That's actually the number I was using
to determine the sample period in cycles that should be equivalent to a 100Hz sample
frequency but now I'm not sure I know where that number comes from or what it means, and
the perf-stat man page is mum on the subject. The complete output of that command on this
quad core arm64 system is:
~# perf stat -- sleep 1
Performance counter stats for 'sleep 1':
1.15 msec task-clock # 0.001 CPUs utilized
1 context-switches # 0.872 K/sec
0 cpu-migrations # 0.000 K/sec
44 page-faults # 0.038 M/sec
1508958 cycles # 1.315 GHz
679359 instructions # 0.45 insn per cycle
86111 branches # 75.059 M/sec
10291 branch-misses # 11.95% of all branches
1.002371400 seconds time elapsed
0.002567000 seconds user
0.000000000 seconds sys
If I divide the number of cycles by the total cpu usage, I get 1508958/0.002567 == 588 MHz,
which is pretty far from 1.315 GHz. Where did 1.315 GHz come from?
And if I run a CPU intensive program like stress, I get:
# perf stat -- stress -c 1 --timeout 1
stress-ng: info: [9166] dispatching hogs: 1 cpu
stress-ng: info: [9166] successful run completed in 1.13s
Performance counter stats for 'stress -c 1 --timeout 1':
1135.88 msec task-clock # 0.991 CPUs utilized
156 context-switches # 0.137 K/sec
0 cpu-migrations # 0.000 K/sec
1253 page-faults # 0.001 M/sec
1513809696 cycles # 1.333 GHz
1546722405 instructions # 1.02 insn per cycle
175801202 branches # 154.772 M/sec
4725866 branch-misses # 2.69% of all branches
1.146401500 seconds time elapsed
1.128040000 seconds user
0.009161000 seconds sys
You're right that now I do see a lot more cycles, and also, dividing cycles by total CPU
usage gives something pretty close to the 1.333 GHz reported in the comment:
1513809696/(1.128040 + 0.009161) = 1.331 GHz.
The frequencies reported in these comments are quite close, so I guess they must mean
something. OTOH, they aren't identical, so I don't think they're just being read from some
table somewhere. They must be being computed, but how?
And just for fun, here's what the same two runs (sleep and stress) look like with the -a option:
# perf stat -a -- sleep 1
Performance counter stats for 'system wide':
4005.15 msec cpu-clock # 3.995 CPUs utilized
4945 context-switches # 0.001 M/sec
15 cpu-migrations # 0.004 K/sec
2 page-faults # 0.000 K/sec
78745230 cycles # 0.020 GHz
37676196 instructions # 0.48 insn per cycle
5155778 branches # 1.287 M/sec
749509 branch-misses # 14.54% of all branches
1.002497310 seconds time elapsed
# perf stat -a -- stress -c 1 --timeout 1
stress-ng: info: [9721] dispatching hogs: 1 cpu
stress-ng: info: [9721] successful run completed in 1.14s
Performance counter stats for 'system wide':
4598.27 msec cpu-clock # 3.991 CPUs utilized
7125 context-switches # 0.002 M/sec
14 cpu-migrations # 0.003 K/sec
1133 page-faults # 0.246 K/sec
1633500454 cycles # 0.355 GHz
1595921326 instructions # 0.98 insn per cycle
182601726 branches # 39.711 M/sec
5607300 branch-misses # 3.07% of all branches
1.152046320 seconds time elapsed
Now the frequencies displayed in the comments are quite different both from each other, and
from those displayed without -a.
I still have the feeling that I'm missing some important concept here. I'll keep trying
to figure out what it is.
Regards,
Doug
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: What exactly does -F <freq> do?
2022-03-09 18:36 ` Douglas Graham
@ 2022-03-10 8:52 ` Namhyung Kim
0 siblings, 0 replies; 6+ messages in thread
From: Namhyung Kim @ 2022-03-10 8:52 UTC (permalink / raw)
To: Douglas Graham; +Cc: James Clark, linux-perf-users
Hello,
On Wed, Mar 9, 2022 at 4:56 PM Douglas Graham
<douglas.graham@ericsson.com> wrote:
>
> >> I am already using -a.
> >
> >In that case it does change some of my assumptions. But for this part of your message:
> >
> > > A "perf stat sleep 1" on the system I'm interested in says:
> >
> > > 1630370 cycles # 1.315 GHz
> >
> > > As I understand it, the cycles event is used by default, so a 100 Hz sample frequency
> > > should give the same results as a 13150000 cycles sample period.
> >
> > That command is not system wide or per-CPU which is what my assumption was based one
> > (it's per process mode). So it's not 1.6M cycles in 1 second, it's 1.6M cycles of when
> > the sleep process was actively launching and running, excluding the 1s of sleep. System
> > wide would be 'perf stat -a -- sleep 1'
>
> It could just be me, but I think perf is in dire need of more comprehensive documentation.
> Like, for example, how is that 1.315 GHz computed? That's actually the number I was using
> to determine the sample period in cycles that should be equivalent to a 100Hz sample
> frequency but now I'm not sure I know where that number comes from or what it means, and
> the perf-stat man page is mum on the subject. The complete output of that command on this
> quad core arm64 system is:
>
> ~# perf stat -- sleep 1
>
> Performance counter stats for 'sleep 1':
>
> 1.15 msec task-clock # 0.001 CPUs utilized
> 1 context-switches # 0.872 K/sec
> 0 cpu-migrations # 0.000 K/sec
> 44 page-faults # 0.038 M/sec
> 1508958 cycles # 1.315 GHz
> 679359 instructions # 0.45 insn per cycle
> 86111 branches # 75.059 M/sec
> 10291 branch-misses # 11.95% of all branches
>
> 1.002371400 seconds time elapsed
>
> 0.002567000 seconds user
> 0.000000000 seconds sys
>
> If I divide the number of cycles by the total cpu usage, I get 1508958/0.002567 == 588 MHz,
> which is pretty far from 1.315 GHz. Where did 1.315 GHz come from?
I think it's cycles / task-clock.
>
> And if I run a CPU intensive program like stress, I get:
>
> # perf stat -- stress -c 1 --timeout 1
> stress-ng: info: [9166] dispatching hogs: 1 cpu
> stress-ng: info: [9166] successful run completed in 1.13s
>
> Performance counter stats for 'stress -c 1 --timeout 1':
>
> 1135.88 msec task-clock # 0.991 CPUs utilized
> 156 context-switches # 0.137 K/sec
> 0 cpu-migrations # 0.000 K/sec
> 1253 page-faults # 0.001 M/sec
> 1513809696 cycles # 1.333 GHz
> 1546722405 instructions # 1.02 insn per cycle
> 175801202 branches # 154.772 M/sec
> 4725866 branch-misses # 2.69% of all branches
>
> 1.146401500 seconds time elapsed
>
> 1.128040000 seconds user
> 0.009161000 seconds sys
>
> You're right that now I do see a lot more cycles, and also, dividing cycles by total CPU
> usage gives something pretty close to the 1.333 GHz reported in the comment:
>
> 1513809696/(1.128040 + 0.009161) = 1.331 GHz.
>
> The frequencies reported in these comments are quite close, so I guess they must mean
> something. OTOH, they aren't identical, so I don't think they're just being read from some
> table somewhere. They must be being computed, but how?
cycles / task-clock = 1513809696 / 1.13588 = 1332719738
>
> And just for fun, here's what the same two runs (sleep and stress) look like with the -a option:
>
> # perf stat -a -- sleep 1
>
> Performance counter stats for 'system wide':
>
> 4005.15 msec cpu-clock # 3.995 CPUs utilized
> 4945 context-switches # 0.001 M/sec
> 15 cpu-migrations # 0.004 K/sec
> 2 page-faults # 0.000 K/sec
> 78745230 cycles # 0.020 GHz
> 37676196 instructions # 0.48 insn per cycle
> 5155778 branches # 1.287 M/sec
> 749509 branch-misses # 14.54% of all branches
>
> 1.002497310 seconds time elapsed
>
> # perf stat -a -- stress -c 1 --timeout 1
> stress-ng: info: [9721] dispatching hogs: 1 cpu
> stress-ng: info: [9721] successful run completed in 1.14s
>
> Performance counter stats for 'system wide':
>
> 4598.27 msec cpu-clock # 3.991 CPUs utilized
> 7125 context-switches # 0.002 M/sec
> 14 cpu-migrations # 0.003 K/sec
> 1133 page-faults # 0.246 K/sec
> 1633500454 cycles # 0.355 GHz
> 1595921326 instructions # 0.98 insn per cycle
> 182601726 branches # 39.711 M/sec
> 5607300 branch-misses # 3.07% of all branches
>
> 1.152046320 seconds time elapsed
>
> Now the frequencies displayed in the comments are quite different both from each other, and
> from those displayed without -a.
For system-wide mode, it really depends on how much load
it has at the moment.
>
> I still have the feeling that I'm missing some important concept here. I'll keep trying
> to figure out what it is.
The sampling frequency can be inaccurate if the system is idle
or system load is fluctuating as it's not walltime based.
Thanks,
Namhyung
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-03-10 8:53 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-05 0:51 What exactly does -F <freq> do? Douglas Graham
2022-03-08 12:23 ` James Clark
2022-03-09 1:05 ` Douglas Graham
2022-03-09 11:03 ` James Clark
2022-03-09 18:36 ` Douglas Graham
2022-03-10 8:52 ` Namhyung Kim
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.