* [PATCH v3 1/2] perf test: Workload test of metric and metricgroups
@ 2021-09-17 18:42 Ian Rogers
2021-09-17 18:42 ` [PATCH v3 2/2] perf test: Workload test of all PMUs Ian Rogers
2021-09-24 19:09 ` [PATCH v3 1/2] perf test: Workload test of metric and metricgroups Arnaldo Carvalho de Melo
0 siblings, 2 replies; 10+ messages in thread
From: Ian Rogers @ 2021-09-17 18:42 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Mark Rutland, Alexander Shishkin, Jiri Olsa, Namhyung Kim,
Jin Yao, John Garry, Paul A . Clarke, linux-perf-users,
linux-kernel
Cc: eranian, Ian Rogers
Test every metric and metricgroup with 'true' as a workload. For
metrics, check that we see the metric printed or get unsupported. If the
'true' workload executes too quickly retry with 'perf bench internals
synthesize'.
v3. Fix test condition (thanks to Paul A. Clarke <pc@us.ibm.com>). Add a
fallback case of a larger workload so that we don't ignore "<not
counted>".
v2. Switched the workload to something faster.
Signed-off-by: Ian Rogers <irogers@google.com>
---
.../perf/tests/shell/stat_all_metricgroups.sh | 12 ++++++++++
tools/perf/tests/shell/stat_all_metrics.sh | 22 +++++++++++++++++++
2 files changed, 34 insertions(+)
create mode 100755 tools/perf/tests/shell/stat_all_metricgroups.sh
create mode 100755 tools/perf/tests/shell/stat_all_metrics.sh
diff --git a/tools/perf/tests/shell/stat_all_metricgroups.sh b/tools/perf/tests/shell/stat_all_metricgroups.sh
new file mode 100755
index 000000000000..de24d374ce24
--- /dev/null
+++ b/tools/perf/tests/shell/stat_all_metricgroups.sh
@@ -0,0 +1,12 @@
+#!/bin/sh
+# perf all metricgroups test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+for m in $(perf list --raw-dump metricgroups); do
+ echo "Testing $m"
+ perf stat -M "$m" true
+done
+
+exit 0
diff --git a/tools/perf/tests/shell/stat_all_metrics.sh b/tools/perf/tests/shell/stat_all_metrics.sh
new file mode 100755
index 000000000000..7f4ba3cad632
--- /dev/null
+++ b/tools/perf/tests/shell/stat_all_metrics.sh
@@ -0,0 +1,22 @@
+#!/bin/sh
+# perf all metrics test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+for m in $(perf list --raw-dump metrics); do
+ echo "Testing $m"
+ result=$(perf stat -M "$m" true 2>&1)
+ if [[ ! "$result" =~ "$m" ]] && [[ ! "$result" =~ "<not supported>" ]]; then
+ # We failed to see the metric and the events are support. Possibly the
+ # workload was too small so retry with something longer.
+ result=$(perf stat -M "$m" perf bench internals synthesize 2>&1)
+ if [[ ! "$result" =~ "$m" ]]; then
+ echo "Metric '$m' not printed in:"
+ echo "$result"
+ exit 1
+ fi
+ fi
+done
+
+exit 0
--
2.33.0.464.g1972c5931b-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v3 2/2] perf test: Workload test of all PMUs
2021-09-17 18:42 [PATCH v3 1/2] perf test: Workload test of metric and metricgroups Ian Rogers
@ 2021-09-17 18:42 ` Ian Rogers
2021-09-24 19:09 ` [PATCH v3 1/2] perf test: Workload test of metric and metricgroups Arnaldo Carvalho de Melo
1 sibling, 0 replies; 10+ messages in thread
From: Ian Rogers @ 2021-09-17 18:42 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Mark Rutland, Alexander Shishkin, Jiri Olsa, Namhyung Kim,
Jin Yao, John Garry, Paul A . Clarke, linux-perf-users,
linux-kernel
Cc: eranian, Ian Rogers
Iterate over the list of PMUs and run the 'true' workload on them. If
the event isn't printed then run the large 'perf bench internals
synthesize' workload and check the event is counted.
On a Skylake this test takes 1m15s mainly running the 'true' workload.
Suggested-by: John Garry <john.garry@huawei.com>
Signed-off-by: Ian Rogers <irogers@google.com>
---
tools/perf/tests/shell/stat_all_pmu.sh | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
create mode 100755 tools/perf/tests/shell/stat_all_pmu.sh
diff --git a/tools/perf/tests/shell/stat_all_pmu.sh b/tools/perf/tests/shell/stat_all_pmu.sh
new file mode 100755
index 000000000000..2de7fd0394fd
--- /dev/null
+++ b/tools/perf/tests/shell/stat_all_pmu.sh
@@ -0,0 +1,22 @@
+#!/bin/sh
+# perf all PMU test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+for p in $(perf list --raw-dump pmu); do
+ echo "Testing $p"
+ result=$(perf stat -e "$p" true 2>&1)
+ if [[ ! "$result" =~ "$p" ]] && [[ ! "$result" =~ "<not supported>" ]]; then
+ # We failed to see the event and it is supported. Possibly the workload was
+ # too small so retry with something longer.
+ result=$(perf stat -e "$p" perf bench internals synthesize 2>&1)
+ if [[ ! "$result" =~ "$p" ]]; then
+ echo "Event '$p' not printed in:"
+ echo "$result"
+ exit 1
+ fi
+ fi
+done
+
+exit 0
--
2.33.0.464.g1972c5931b-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups
2021-09-17 18:42 [PATCH v3 1/2] perf test: Workload test of metric and metricgroups Ian Rogers
2021-09-17 18:42 ` [PATCH v3 2/2] perf test: Workload test of all PMUs Ian Rogers
@ 2021-09-24 19:09 ` Arnaldo Carvalho de Melo
2021-09-24 19:39 ` John Garry
2022-01-12 12:24 ` John Garry
1 sibling, 2 replies; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2021-09-24 19:09 UTC (permalink / raw)
To: Ian Rogers
Cc: Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin,
Jiri Olsa, Namhyung Kim, Jin Yao, John Garry, Paul A . Clarke,
linux-perf-users, linux-kernel, eranian
Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
> Test every metric and metricgroup with 'true' as a workload. For
> metrics, check that we see the metric printed or get unsupported. If the
> 'true' workload executes too quickly retry with 'perf bench internals
> synthesize'.
>
> v3. Fix test condition (thanks to Paul A. Clarke <pc@us.ibm.com>). Add a
> fallback case of a larger workload so that we don't ignore "<not
> counted>".
> v2. Switched the workload to something faster.
Hi John, does your Reviewed-by stands for v3 too?
- Arnaldo
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
> .../perf/tests/shell/stat_all_metricgroups.sh | 12 ++++++++++
> tools/perf/tests/shell/stat_all_metrics.sh | 22 +++++++++++++++++++
> 2 files changed, 34 insertions(+)
> create mode 100755 tools/perf/tests/shell/stat_all_metricgroups.sh
> create mode 100755 tools/perf/tests/shell/stat_all_metrics.sh
>
> diff --git a/tools/perf/tests/shell/stat_all_metricgroups.sh b/tools/perf/tests/shell/stat_all_metricgroups.sh
> new file mode 100755
> index 000000000000..de24d374ce24
> --- /dev/null
> +++ b/tools/perf/tests/shell/stat_all_metricgroups.sh
> @@ -0,0 +1,12 @@
> +#!/bin/sh
> +# perf all metricgroups test
> +# SPDX-License-Identifier: GPL-2.0
> +
> +set -e
> +
> +for m in $(perf list --raw-dump metricgroups); do
> + echo "Testing $m"
> + perf stat -M "$m" true
> +done
> +
> +exit 0
> diff --git a/tools/perf/tests/shell/stat_all_metrics.sh b/tools/perf/tests/shell/stat_all_metrics.sh
> new file mode 100755
> index 000000000000..7f4ba3cad632
> --- /dev/null
> +++ b/tools/perf/tests/shell/stat_all_metrics.sh
> @@ -0,0 +1,22 @@
> +#!/bin/sh
> +# perf all metrics test
> +# SPDX-License-Identifier: GPL-2.0
> +
> +set -e
> +
> +for m in $(perf list --raw-dump metrics); do
> + echo "Testing $m"
> + result=$(perf stat -M "$m" true 2>&1)
> + if [[ ! "$result" =~ "$m" ]] && [[ ! "$result" =~ "<not supported>" ]]; then
> + # We failed to see the metric and the events are support. Possibly the
> + # workload was too small so retry with something longer.
> + result=$(perf stat -M "$m" perf bench internals synthesize 2>&1)
> + if [[ ! "$result" =~ "$m" ]]; then
> + echo "Metric '$m' not printed in:"
> + echo "$result"
> + exit 1
> + fi
> + fi
> +done
> +
> +exit 0
> --
> 2.33.0.464.g1972c5931b-goog
--
- Arnaldo
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups
2021-09-24 19:09 ` [PATCH v3 1/2] perf test: Workload test of metric and metricgroups Arnaldo Carvalho de Melo
@ 2021-09-24 19:39 ` John Garry
2021-09-28 18:55 ` Arnaldo Carvalho de Melo
2022-01-12 12:24 ` John Garry
1 sibling, 1 reply; 10+ messages in thread
From: John Garry @ 2021-09-24 19:39 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers
Cc: Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin,
Jiri Olsa, Namhyung Kim, Jin Yao, Paul A . Clarke,
linux-perf-users, linux-kernel, eranian
On 24/09/2021 20:09, Arnaldo Carvalho de Melo wrote:
> Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
>> Test every metric and metricgroup with 'true' as a workload. For
>> metrics, check that we see the metric printed or get unsupported. If the
>> 'true' workload executes too quickly retry with 'perf bench internals
>> synthesize'.
>>
>> v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
>> fallback case of a larger workload so that we don't ignore "<not
>> counted>".
>> v2. Switched the workload to something faster.
> Hi John, does your Reviewed-by stands for v3 too?
>
> - Arnaldo
>
Yeah,
Reviewed-by: John Garry <john.garry@huawei.com>
Thanks
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups
2021-09-24 19:39 ` John Garry
@ 2021-09-28 18:55 ` Arnaldo Carvalho de Melo
0 siblings, 0 replies; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2021-09-28 18:55 UTC (permalink / raw)
To: John Garry
Cc: Ian Rogers, Peter Zijlstra, Ingo Molnar, Mark Rutland,
Alexander Shishkin, Jiri Olsa, Namhyung Kim, Jin Yao,
Paul A . Clarke, linux-perf-users, linux-kernel, eranian
Em Fri, Sep 24, 2021 at 08:39:18PM +0100, John Garry escreveu:
> On 24/09/2021 20:09, Arnaldo Carvalho de Melo wrote:
> > Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
> > > Test every metric and metricgroup with 'true' as a workload. For
> > > metrics, check that we see the metric printed or get unsupported. If the
> > > 'true' workload executes too quickly retry with 'perf bench internals
> > > synthesize'.
> > > v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
> > > fallback case of a larger workload so that we don't ignore "<not
> > > counted>".
> > > v2. Switched the workload to something faster.
> > Hi John, does your Reviewed-by stands for v3 too?
> Yeah,
> Reviewed-by: John Garry <john.garry@huawei.com>
Thanks, applied.
- Arnaldo
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups
2021-09-24 19:09 ` [PATCH v3 1/2] perf test: Workload test of metric and metricgroups Arnaldo Carvalho de Melo
2021-09-24 19:39 ` John Garry
@ 2022-01-12 12:24 ` John Garry
2022-01-12 13:34 ` Arnaldo Carvalho de Melo
1 sibling, 1 reply; 10+ messages in thread
From: John Garry @ 2022-01-12 12:24 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Ian Rogers
Cc: Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin,
Jiri Olsa, Namhyung Kim, Jin Yao, Paul A . Clarke,
linux-perf-users, linux-kernel, eranian
On 24/09/2021 20:09, Arnaldo Carvalho de Melo wrote:
> Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
>> Test every metric and metricgroup with 'true' as a workload. For
>> metrics, check that we see the metric printed or get unsupported. If the
>> 'true' workload executes too quickly retry with 'perf bench internals
>> synthesize'.
>>
>> v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
>> fallback case of a larger workload so that we don't ignore "<not
>> counted>".
>> v2. Switched the workload to something faster.
Hi Ian,
I just noticed that this test fails on my broadwell machine.
I am using acme perf/core @ 09dd3c22daaf
metricgroup Memory_Bw fails, and it seems because of the "true" argument
to "perf stat" (or any argument, like sleep 1):
john@localhost:~/kernel-dev9/tools/perf> sudo ./perf stat -M Memory_BW
^C
Performance counter stats for 'system wide':
2,184 arb/event=0x84,umask=0x1/ # 0.26
DRAM_BW_Use
2,954,938 arb/event=0x81,umask=0x1/
736,368,852 ns duration_time
58,202,980 l1d_pend_miss.pending_cycles # 2.34 MLP
(80.11%)
136,293,194 l1d_pend_miss.pending
(19.89%)
736,368,852 ns duration_time
1,065,656 longest_lat_cache.miss # 0.09
L3_Cache_Fill_BW (39.71%)
736,368,852 ns duration_time
5,365,477 l2_lines_in.all # 0.47
L2_Cache_Fill_BW (59.80%)
736,368,852 ns duration_time
3,557,362 l1d.replacement # 0.31
L1D_Cache_Fill_BW (79.90%)
736,368,852 ns duration_time
0.736368852 seconds time elapsed
john@localhost:~/kernel-dev9/tools/perf> sudo ./perf stat -M Memory_BW true
Error:
The sys_perf_event_open() syscall returned with 22 (Invalid argument)
for event (arb/event=0x84,umask=0x1/).
/bin/dmesg | grep -i perf may provide additional information.
john@localhost:~/kernel-dev9/tools/perf>
Anyone any idea on this before I start digging?
Thanks,
John
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups
2022-01-12 12:24 ` John Garry
@ 2022-01-12 13:34 ` Arnaldo Carvalho de Melo
2022-01-12 17:32 ` John Garry
0 siblings, 1 reply; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2022-01-12 13:34 UTC (permalink / raw)
To: John Garry, Arnaldo Carvalho de Melo, Ian Rogers
Cc: Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin,
Jiri Olsa, Namhyung Kim, Jin Yao, Paul A . Clarke,
linux-perf-users, linux-kernel, eranian
On January 12, 2022 9:24:29 AM GMT-03:00, John Garry <john.garry@huawei.com> wrote:
>On 24/09/2021 20:09, Arnaldo Carvalho de Melo wrote:
>> Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
>>> Test every metric and metricgroup with 'true' as a workload. For
>>> metrics, check that we see the metric printed or get unsupported. If the
>>> 'true' workload executes too quickly retry with 'perf bench internals
>>> synthesize'.
>>>
>>> v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
>>> fallback case of a larger workload so that we don't ignore "<not
>>> counted>".
>>> v2. Switched the workload to something faster.
>
>Hi Ian,
>
>I just noticed that this test fails on my broadwell machine.
>
>I am using acme perf/core @ 09dd3c22daaf
Hi,
Can you try with tmp.perf/perf_cpu instead?
There's a patch there that maybe fixes this.
- Arnaldo
>
>metricgroup Memory_Bw fails, and it seems because of the "true" argument
>to "perf stat" (or any argument, like sleep 1):
>
>john@localhost:~/kernel-dev9/tools/perf> sudo ./perf stat -M Memory_BW
>^C
> Performance counter stats for 'system wide':
>
> 2,184 arb/event=0x84,umask=0x1/ # 0.26
>DRAM_BW_Use
> 2,954,938 arb/event=0x81,umask=0x1/
>
> 736,368,852 ns duration_time
>
> 58,202,980 l1d_pend_miss.pending_cycles # 2.34 MLP
> (80.11%)
> 136,293,194 l1d_pend_miss.pending
> (19.89%)
> 736,368,852 ns duration_time
>
> 1,065,656 longest_lat_cache.miss # 0.09
>L3_Cache_Fill_BW (39.71%)
> 736,368,852 ns duration_time
>
> 5,365,477 l2_lines_in.all # 0.47
>L2_Cache_Fill_BW (59.80%)
> 736,368,852 ns duration_time
>
> 3,557,362 l1d.replacement # 0.31
>L1D_Cache_Fill_BW (79.90%)
> 736,368,852 ns duration_time
>
>
> 0.736368852 seconds time elapsed
>
>
>john@localhost:~/kernel-dev9/tools/perf> sudo ./perf stat -M Memory_BW true
>Error:
>The sys_perf_event_open() syscall returned with 22 (Invalid argument)
>for event (arb/event=0x84,umask=0x1/).
>/bin/dmesg | grep -i perf may provide additional information.
>
>john@localhost:~/kernel-dev9/tools/perf>
>
>Anyone any idea on this before I start digging?
>
>Thanks,
>John
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups
2022-01-12 13:34 ` Arnaldo Carvalho de Melo
@ 2022-01-12 17:32 ` John Garry
2022-01-12 17:59 ` Arnaldo Carvalho de Melo
0 siblings, 1 reply; 10+ messages in thread
From: John Garry @ 2022-01-12 17:32 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Arnaldo Carvalho de Melo, Ian Rogers
Cc: Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin,
Jiri Olsa, Namhyung Kim, Jin Yao, Paul A . Clarke,
linux-perf-users, linux-kernel, eranian
On 12/01/2022 13:34, Arnaldo Carvalho de Melo wrote:
>>>> v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
>>>> fallback case of a larger workload so that we don't ignore "<not
>>>> counted>".
>>>> v2. Switched the workload to something faster.
>> Hi Ian,
>>
>> I just noticed that this test fails on my broadwell machine.
>>
>> I am using acme perf/core @ 09dd3c22daaf
> Hi,
>
> Can you try with tmp.perf/perf_cpu instead?
>
> There's a patch there that maybe fixes this.
Yeah, that (metricgroups) test passes on that branch. I assume it's the
test script -a change...
Cheers,
john
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups
2022-01-12 17:32 ` John Garry
@ 2022-01-12 17:59 ` Arnaldo Carvalho de Melo
2022-01-12 18:00 ` John Garry
0 siblings, 1 reply; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2022-01-12 17:59 UTC (permalink / raw)
To: John Garry
Cc: Arnaldo Carvalho de Melo, Ian Rogers, Peter Zijlstra,
Ingo Molnar, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Namhyung Kim, Jin Yao, Paul A . Clarke, linux-perf-users,
linux-kernel, eranian
Em Wed, Jan 12, 2022 at 05:32:56PM +0000, John Garry escreveu:
> On 12/01/2022 13:34, Arnaldo Carvalho de Melo wrote:
> > > > > v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
> > > > > fallback case of a larger workload so that we don't ignore "<not
> > > > > counted>".
> > > > > v2. Switched the workload to something faster.
> > > Hi Ian,
> > >
> > > I just noticed that this test fails on my broadwell machine.
> > >
> > > I am using acme perf/core @ 09dd3c22daaf
> > Hi,
> >
> > Can you try with tmp.perf/perf_cpu instead?
> >
> > There's a patch there that maybe fixes this.
>
> Yeah, that (metricgroups) test passes on that branch. I assume it's the test
> script -a change...
Can I get that as a Tested-by: you tag?
- Arnaldo
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups
2022-01-12 17:59 ` Arnaldo Carvalho de Melo
@ 2022-01-12 18:00 ` John Garry
0 siblings, 0 replies; 10+ messages in thread
From: John Garry @ 2022-01-12 18:00 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Arnaldo Carvalho de Melo, Ian Rogers, Peter Zijlstra,
Ingo Molnar, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Namhyung Kim, Jin Yao, Paul A . Clarke, linux-perf-users,
linux-kernel, eranian
On 12/01/2022 17:59, Arnaldo Carvalho de Melo wrote:
>>> There's a patch there that maybe fixes this.
>> Yeah, that (metricgroups) test passes on that branch. I assume it's the test
>> script -a change...
> Can I get that as a Tested-by: you tag?
Tested-by: John Garry <john.garry@huawei.com>
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2022-01-12 18:01 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-17 18:42 [PATCH v3 1/2] perf test: Workload test of metric and metricgroups Ian Rogers
2021-09-17 18:42 ` [PATCH v3 2/2] perf test: Workload test of all PMUs Ian Rogers
2021-09-24 19:09 ` [PATCH v3 1/2] perf test: Workload test of metric and metricgroups Arnaldo Carvalho de Melo
2021-09-24 19:39 ` John Garry
2021-09-28 18:55 ` Arnaldo Carvalho de Melo
2022-01-12 12:24 ` John Garry
2022-01-12 13:34 ` Arnaldo Carvalho de Melo
2022-01-12 17:32 ` John Garry
2022-01-12 17:59 ` Arnaldo Carvalho de Melo
2022-01-12 18:00 ` John Garry
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).