* Re: [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
@ 2019-05-15 12:59 Arnaldo Carvalho de Melo
2019-05-15 15:44 ` Alexey Budankov
0 siblings, 1 reply; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-05-15 12:59 UTC (permalink / raw)
To: Alexey Budankov
Cc: Arnaldo Carvalho de Melo, Jiri Olsa, Namhyung Kim,
Alexander Shishkin, Peter Zijlstra, Ingo Molnar, Andi Kleen,
linux-kernel
Em Wed, May 15, 2019 at 11:43:30AM +0300, Alexey Budankov escreveu:
> On 15.05.2019 0:46, Arnaldo Carvalho de Melo wrote:
> > Em Tue, May 14, 2019 at 05:20:41PM -0300, Arnaldo Carvalho de Melo escreveu:
> >> Em Mon, Mar 18, 2019 at 08:44:42PM +0300, Alexey Budankov escreveu:
> >>> Implemented -z,--compression_level[=<n>] option that enables compression
> >>> of mmaped kernel data buffers content in runtime during perf record
> >>> mode collection. Default option value is 1 (fastest compression).
> > <SNIP>
> >> [root@quaco ~]# perf record -z2
> >> ^C[ perf record: Woken up 1 times to write data ]
> >> 0x1746e0 [0x76]: failed to process type: 81 [Invalid argument]
> >> [ perf record: Captured and wrote 1.568 MB perf.data, compressed (original 0.452 MB, ratio is 3.995) ]
> >> [root@quaco ~]#
> > So, its the buildid processing at the end, so we can't do build-id
> > processing when using PERF_RECORD_COMPRESSED, otherwise we'd have to
> > uncompress at the end to find the PERF_RECORD_FORK/PERF_RECORD_MMAP,
> > etc.
> > [root@quaco ~]# perf record -z2 --no-buildid sleep 1
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.020 MB perf.data, compressed (original 0.001 MB, ratio is 2.153) ]
> > [root@quaco ~]# perf report -D | grep PERF_RECORD_COMP
> > 0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
> > Error:
> > failed to process sample
> > 0 0x4f40 [0x195]: PERF_RECORD_COMPRESSED
> > [root@quaco ~]#
> > I'll play with it tomorrow.
> Applied the whole patch set on top of the current perf/core
> and the whole thing functions as expected.
It doesn't, see the reported error above, these three lines, that
shouldn't be there:
0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
Error:
failed to process sample
That is because at this point in the patch series a record was
introduced that is not being handled by the build id processing done, by
default, at the end of the 'perf record' session, and, as explained
above, needs fixing so that when we do 'git bisect' looking for a non
expected "failed to process type: 81" kind of error, this doesn't
appear.
I added the changes below to this cset and will continue from there:
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index d84a4885e341..f8d21991f94c 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -2284,6 +2284,12 @@ int cmd_record(int argc, const char **argv)
"cgroup monitoring only available in system-wide mode");
}
+
+ if (rec->opts.comp_level != 0) {
+ pr_debug("Compression enabled, disabling build id collection at the end of the session\n");
+ rec->no_buildid = true;
+ }
+
if (rec->opts.record_switch_events &&
!perf_can_record_switch_events()) {
ui__error("kernel does not support recording context switch events\n");
---------------------------------------------------------------------------
[acme@quaco perf]$ perf record -z2 sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.001 MB perf.data, compressed (original 0.001 MB, ratio is 2.292) ]
[acme@quaco perf]$ perf record -v -z2 sleep 1
Compression enabled, disabling build id collection at the end of the session
Using CPUID GenuineIntel-6-8E-A
nr_cblocks: 0
affinity: SYS
mmap flush: 1
comp level: 2
mmap size 528384B
Couldn't start the BPF side band thread:
BPF programs starting from now on won't be annotatable
perf_event__synthesize_bpf_events: can't get next program: Operation not permitted
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.001 MB perf.data, compressed (original 0.001 MB, ratio is 2.305) ]
[acme@quaco perf]$
Will check if its possible to get rid of the following in this patch, to
keep bisection working for this case as well:
[acme@quaco perf]$ perf report -D | grep COMPRESS
0x1b8 [0x169]: failed to process type: 81 [Invalid argument]
Error:
failed to process sample
0 0x1b8 [0x169]: PERF_RECORD_COMPRESSED
[acme@quaco perf]$
- Arnaldo
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
2019-05-15 12:59 [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option Arnaldo Carvalho de Melo
@ 2019-05-15 15:44 ` Alexey Budankov
2019-05-17 15:01 ` Arnaldo Carvalho de Melo
0 siblings, 1 reply; 10+ messages in thread
From: Alexey Budankov @ 2019-05-15 15:44 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Jiri Olsa, Namhyung Kim, Alexander Shishkin, Peter Zijlstra,
Ingo Molnar, Andi Kleen, linux-kernel
On 15.05.2019 15:59, Arnaldo Carvalho de Melo wrote:
> Em Wed, May 15, 2019 at 11:43:30AM +0300, Alexey Budankov escreveu:
>> On 15.05.2019 0:46, Arnaldo Carvalho de Melo wrote:
>>> Em Tue, May 14, 2019 at 05:20:41PM -0300, Arnaldo Carvalho de Melo escreveu:
>>>> Em Mon, Mar 18, 2019 at 08:44:42PM +0300, Alexey Budankov escreveu:
>
>>>>> Implemented -z,--compression_level[=<n>] option that enables compression
>>>>> of mmaped kernel data buffers content in runtime during perf record
>>>>> mode collection. Default option value is 1 (fastest compression).
>
>>> <SNIP>
>
>>>> [root@quaco ~]# perf record -z2
>>>> ^C[ perf record: Woken up 1 times to write data ]
>>>> 0x1746e0 [0x76]: failed to process type: 81 [Invalid argument]
>>>> [ perf record: Captured and wrote 1.568 MB perf.data, compressed (original 0.452 MB, ratio is 3.995) ]
>
>>>> [root@quaco ~]#
>
>>> So, its the buildid processing at the end, so we can't do build-id
>>> processing when using PERF_RECORD_COMPRESSED, otherwise we'd have to
>>> uncompress at the end to find the PERF_RECORD_FORK/PERF_RECORD_MMAP,
>>> etc.
>
>>> [root@quaco ~]# perf record -z2 --no-buildid sleep 1
>>> [ perf record: Woken up 1 times to write data ]
>>> [ perf record: Captured and wrote 0.020 MB perf.data, compressed (original 0.001 MB, ratio is 2.153) ]
>>> [root@quaco ~]# perf report -D | grep PERF_RECORD_COMP
>>> 0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
>>> Error:
>>> failed to process sample
>>> 0 0x4f40 [0x195]: PERF_RECORD_COMPRESSED
>>> [root@quaco ~]#
>
>>> I'll play with it tomorrow.
>
>> Applied the whole patch set on top of the current perf/core
>> and the whole thing functions as expected.
>
> It doesn't, see the reported error above, these three lines, that
> shouldn't be there:
>
> 0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
> Error:
> failed to process sample
>
> That is because at this point in the patch series a record was
> introduced that is not being handled by the build id processing done, by
> default, at the end of the 'perf record' session, and, as explained
> above, needs fixing so that when we do 'git bisect' looking for a non
> expected "failed to process type: 81" kind of error, this doesn't
> appear.
>
> I added the changes below to this cset and will continue from there:
>
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index d84a4885e341..f8d21991f94c 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -2284,6 +2284,12 @@ int cmd_record(int argc, const char **argv)
> "cgroup monitoring only available in system-wide mode");
>
> }
> +
> + if (rec->opts.comp_level != 0) {
> + pr_debug("Compression enabled, disabling build id collection at the end of the session\n");
> + rec->no_buildid = true;
> + }
> +
> if (rec->opts.record_switch_events &&
> !perf_can_record_switch_events()) {
> ui__error("kernel does not support recording context switch events\n");
>
> ---------------------------------------------------------------------------
>
> [acme@quaco perf]$ perf record -z2 sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.001 MB perf.data, compressed (original 0.001 MB, ratio is 2.292) ]
> [acme@quaco perf]$ perf record -v -z2 sleep 1
> Compression enabled, disabling build id collection at the end of the session
> Using CPUID GenuineIntel-6-8E-A
> nr_cblocks: 0
> affinity: SYS
> mmap flush: 1
> comp level: 2
> mmap size 528384B
> Couldn't start the BPF side band thread:
> BPF programs starting from now on won't be annotatable
> perf_event__synthesize_bpf_events: can't get next program: Operation not permitted
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.001 MB perf.data, compressed (original 0.001 MB, ratio is 2.305) ]
> [acme@quaco perf]$
>
> Will check if its possible to get rid of the following in this patch, to
> keep bisection working for this case as well:
>
> [acme@quaco perf]$ perf report -D | grep COMPRESS
> 0x1b8 [0x169]: failed to process type: 81 [Invalid argument]
> Error:
> failed to process sample
> 0 0x1b8 [0x169]: PERF_RECORD_COMPRESSED
> [acme@quaco perf]$
Makes sense. Thanks.
~Alexey
>
> - Arnaldo
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
2019-05-15 15:44 ` Alexey Budankov
@ 2019-05-17 15:01 ` Arnaldo Carvalho de Melo
2019-05-17 16:56 ` Alexey Budankov
0 siblings, 1 reply; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-05-17 15:01 UTC (permalink / raw)
To: Alexey Budankov
Cc: Arnaldo Carvalho de Melo, Jiri Olsa, Namhyung Kim,
Alexander Shishkin, Peter Zijlstra, Ingo Molnar, Andi Kleen,
linux-kernel
Em Wed, May 15, 2019 at 06:44:29PM +0300, Alexey Budankov escreveu:
> On 15.05.2019 15:59, Arnaldo Carvalho de Melo wrote:
> > Em Wed, May 15, 2019 at 11:43:30AM +0300, Alexey Budankov escreveu:
> >> On 15.05.2019 0:46, Arnaldo Carvalho de Melo wrote:
> >>> Em Tue, May 14, 2019 at 05:20:41PM -0300, Arnaldo Carvalho de Melo escreveu:
> >>>> Em Mon, Mar 18, 2019 at 08:44:42PM +0300, Alexey Budankov escreveu:
> >
> >>>>> Implemented -z,--compression_level[=<n>] option that enables compression
> >>>>> of mmaped kernel data buffers content in runtime during perf record
> >>>>> mode collection. Default option value is 1 (fastest compression).
> >
> >>> <SNIP>
> >
> >>>> [root@quaco ~]# perf record -z2
> >>>> ^C[ perf record: Woken up 1 times to write data ]
> >>>> 0x1746e0 [0x76]: failed to process type: 81 [Invalid argument]
> >>>> [ perf record: Captured and wrote 1.568 MB perf.data, compressed (original 0.452 MB, ratio is 3.995) ]
> >
> >>>> [root@quaco ~]#
> >
> >>> So, its the buildid processing at the end, so we can't do build-id
> >>> processing when using PERF_RECORD_COMPRESSED, otherwise we'd have to
> >>> uncompress at the end to find the PERF_RECORD_FORK/PERF_RECORD_MMAP,
> >>> etc.
> >
> >>> [root@quaco ~]# perf record -z2 --no-buildid sleep 1
> >>> [ perf record: Woken up 1 times to write data ]
> >>> [ perf record: Captured and wrote 0.020 MB perf.data, compressed (original 0.001 MB, ratio is 2.153) ]
> >>> [root@quaco ~]# perf report -D | grep PERF_RECORD_COMP
> >>> 0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
> >>> Error:
> >>> failed to process sample
> >>> 0 0x4f40 [0x195]: PERF_RECORD_COMPRESSED
> >>> [root@quaco ~]#
> >
> >>> I'll play with it tomorrow.
> >
> >> Applied the whole patch set on top of the current perf/core
> >> and the whole thing functions as expected.
> >
> > It doesn't, see the reported error above, these three lines, that
> > shouldn't be there:
> >
> > 0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
> > Error:
> > failed to process sample
> >
> > That is because at this point in the patch series a record was
> > introduced that is not being handled by the build id processing done, by
> > default, at the end of the 'perf record' session, and, as explained
> > above, needs fixing so that when we do 'git bisect' looking for a non
> > expected "failed to process type: 81" kind of error, this doesn't
> > appear.
> >
> > I added the changes below to this cset and will continue from there:
> >
> > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > index d84a4885e341..f8d21991f94c 100644
> > --- a/tools/perf/builtin-record.c
> > +++ b/tools/perf/builtin-record.c
> > @@ -2284,6 +2284,12 @@ int cmd_record(int argc, const char **argv)
> > "cgroup monitoring only available in system-wide mode");
> >
> > }
> > +
> > + if (rec->opts.comp_level != 0) {
> > + pr_debug("Compression enabled, disabling build id collection at the end of the session\n");
> > + rec->no_buildid = true;
> > + }
> > +
> > if (rec->opts.record_switch_events &&
> > !perf_can_record_switch_events()) {
> > ui__error("kernel does not support recording context switch events\n");
> >
> > ---------------------------------------------------------------------------
> >
> > [acme@quaco perf]$ perf record -z2 sleep 1
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.001 MB perf.data, compressed (original 0.001 MB, ratio is 2.292) ]
> > [acme@quaco perf]$ perf record -v -z2 sleep 1
> > Compression enabled, disabling build id collection at the end of the session
> > Using CPUID GenuineIntel-6-8E-A
> > nr_cblocks: 0
> > affinity: SYS
> > mmap flush: 1
> > comp level: 2
> > mmap size 528384B
> > Couldn't start the BPF side band thread:
> > BPF programs starting from now on won't be annotatable
> > perf_event__synthesize_bpf_events: can't get next program: Operation not permitted
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.001 MB perf.data, compressed (original 0.001 MB, ratio is 2.305) ]
> > [acme@quaco perf]$
> >
> > Will check if its possible to get rid of the following in this patch, to
> > keep bisection working for this case as well:
> >
> > [acme@quaco perf]$ perf report -D | grep COMPRESS
> > 0x1b8 [0x169]: failed to process type: 81 [Invalid argument]
> > Error:
> > failed to process sample
> > 0 0x1b8 [0x169]: PERF_RECORD_COMPRESSED
> > [acme@quaco perf]$
>
> Makes sense. Thanks.
I did it yesterday, all is in my acme/perf/core branch, now testing it
together with the large pile of patches there accumulated while I was in
LSF/MM + vacations :-)
All have already passed through most of my test build containers, with
most of the distros that have libzstd being updated to include it, and
the make_minimal test build target was updated to build explicitely
disabling zstd, i.e. with NO_LIBZSTD=1, so that we test with/without it
in systems where it is installed and also in systems where zstd is not
even available.
- Arnaldo
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
2019-05-17 15:01 ` Arnaldo Carvalho de Melo
@ 2019-05-17 16:56 ` Alexey Budankov
0 siblings, 0 replies; 10+ messages in thread
From: Alexey Budankov @ 2019-05-17 16:56 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Jiri Olsa, Namhyung Kim, Alexander Shishkin, Peter Zijlstra,
Ingo Molnar, Andi Kleen, linux-kernel
On 17.05.2019 18:01, Arnaldo Carvalho de Melo wrote:
> Em Wed, May 15, 2019 at 06:44:29PM +0300, Alexey Budankov escreveu:
>> On 15.05.2019 15:59, Arnaldo Carvalho de Melo wrote:
<SNIP>
>>> Em Wed, May 15, 2019 at 11:43:30AM +0300, Alexey Budankov escreveu:
>>>> On 15.05.2019 0:46, Arnaldo Carvalho de Melo wrote:
>>>>> Em Tue, May 14, 2019 at 05:20:41PM -0300, Arnaldo Carvalho de Melo escreveu:
>>>>>> Em Mon, Mar 18, 2019 at 08:44:42PM +0300, Alexey Budankov escreveu:
>>>
>>>>>>> Implemented -z,--compression_level[=<n>] option that enables compression
>>>>>>> of mmaped kernel data buffers content in runtime during perf record
>>>>>>> mode collection. Default option value is 1 (fastest compression).
>>>
>>>>> <SNIP>
>>>
>>>>>> [root@quaco ~]# perf record -z2
>>>>>> ^C[ perf record: Woken up 1 times to write data ]
>>>>>> 0x1746e0 [0x76]: failed to process type: 81 [Invalid argument]
>>>>>> [ perf record: Captured and wrote 1.568 MB perf.data, compressed (original 0.452 MB, ratio is 3.995) ]
>>>
>>>>>> [root@quaco ~]#
>>>
>>>>> So, its the buildid processing at the end, so we can't do build-id
>>>>> processing when using PERF_RECORD_COMPRESSED, otherwise we'd have to
>>>>> uncompress at the end to find the PERF_RECORD_FORK/PERF_RECORD_MMAP,
>>>>> etc.
>>>
>>>>> [root@quaco ~]# perf record -z2 --no-buildid sleep 1
>>>>> [ perf record: Woken up 1 times to write data ]
>>>>> [ perf record: Captured and wrote 0.020 MB perf.data, compressed (original 0.001 MB, ratio is 2.153) ]
>>>>> [root@quaco ~]# perf report -D | grep PERF_RECORD_COMP
>>>>> 0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
>>>>> Error:
>>>>> failed to process sample
>>>>> 0 0x4f40 [0x195]: PERF_RECORD_COMPRESSED
>>>>> [root@quaco ~]#
>>>
>>>>> I'll play with it tomorrow.
>>>
>>>> Applied the whole patch set on top of the current perf/core
>>>> and the whole thing functions as expected.
>>>
>>> It doesn't, see the reported error above, these three lines, that
>>> shouldn't be there:
>>>
>>> 0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
>>> Error:
>>> failed to process sample
>>>
>>> That is because at this point in the patch series a record was
>>> introduced that is not being handled by the build id processing done, by
>>> default, at the end of the 'perf record' session, and, as explained
>>> above, needs fixing so that when we do 'git bisect' looking for a non
>>> expected "failed to process type: 81" kind of error, this doesn't
>>> appear.
>>>
>>> I added the changes below to this cset and will continue from there:
>>>
>>> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
>>> index d84a4885e341..f8d21991f94c 100644
>>> --- a/tools/perf/builtin-record.c
>>> +++ b/tools/perf/builtin-record.c
>>> @@ -2284,6 +2284,12 @@ int cmd_record(int argc, const char **argv)
>>> "cgroup monitoring only available in system-wide mode");
>>>
>>> }
>>> +
>>> + if (rec->opts.comp_level != 0) {
>>> + pr_debug("Compression enabled, disabling build id collection at the end of the session\n");
>>> + rec->no_buildid = true;
>>> + }
>>> +
>>> if (rec->opts.record_switch_events &&
>>> !perf_can_record_switch_events()) {
>>> ui__error("kernel does not support recording context switch events\n");
>>>
>>> ---------------------------------------------------------------------------
>>>
>>> [acme@quaco perf]$ perf record -z2 sleep 1
>>> [ perf record: Woken up 1 times to write data ]
>>> [ perf record: Captured and wrote 0.001 MB perf.data, compressed (original 0.001 MB, ratio is 2.292) ]
>>> [acme@quaco perf]$ perf record -v -z2 sleep 1
>>> Compression enabled, disabling build id collection at the end of the session
>>> Using CPUID GenuineIntel-6-8E-A
>>> nr_cblocks: 0
>>> affinity: SYS
>>> mmap flush: 1
>>> comp level: 2
>>> mmap size 528384B
>>> Couldn't start the BPF side band thread:
>>> BPF programs starting from now on won't be annotatable
>>> perf_event__synthesize_bpf_events: can't get next program: Operation not permitted
>>> [ perf record: Woken up 1 times to write data ]
>>> [ perf record: Captured and wrote 0.001 MB perf.data, compressed (original 0.001 MB, ratio is 2.305) ]
>>> [acme@quaco perf]$
>>>
>>> Will check if its possible to get rid of the following in this patch, to
>>> keep bisection working for this case as well:
>>>
>>> [acme@quaco perf]$ perf report -D | grep COMPRESS
>>> 0x1b8 [0x169]: failed to process type: 81 [Invalid argument]
>>> Error:
>>> failed to process sample
>>> 0 0x1b8 [0x169]: PERF_RECORD_COMPRESSED
>>> [acme@quaco perf]$
>>
>> Makes sense. Thanks.
>
> I did it yesterday, all is in my acme/perf/core branch, now testing it
> together with the large pile of patches there accumulated while I was in
> LSF/MM + vacations :-)
>
> All have already passed through most of my test build containers, with
> most of the distros that have libzstd being updated to include it, and
> the make_minimal test build target was updated to build explicitely
> disabling zstd, i.e. with NO_LIBZSTD=1, so that we test with/without it
> in systems where it is installed and also in systems where zstd is not
> even available.
Good news. Thanks!
~Alexey
>
> - Arnaldo
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
2019-05-14 21:46 ` Arnaldo Carvalho de Melo
@ 2019-05-15 8:43 ` Alexey Budankov
0 siblings, 0 replies; 10+ messages in thread
From: Alexey Budankov @ 2019-05-15 8:43 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Jiri Olsa, Namhyung Kim, Alexander Shishkin, Peter Zijlstra,
Ingo Molnar, Andi Kleen, linux-kernel
On 15.05.2019 0:46, Arnaldo Carvalho de Melo wrote:
> Em Tue, May 14, 2019 at 05:20:41PM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Mon, Mar 18, 2019 at 08:44:42PM +0300, Alexey Budankov escreveu:
>>>
>>> Implemented -z,--compression_level[=<n>] option that enables compression
>>> of mmaped kernel data buffers content in runtime during perf record
>>> mode collection. Default option value is 1 (fastest compression).
>
> <SNIP>
>
>> [root@quaco ~]# perf record -z2
>> ^C[ perf record: Woken up 1 times to write data ]
>> 0x1746e0 [0x76]: failed to process type: 81 [Invalid argument]
>> [ perf record: Captured and wrote 1.568 MB perf.data, compressed (original 0.452 MB, ratio is 3.995) ]
>>
>> [root@quaco ~]#
>
> So, its the buildid processing at the end, so we can't do build-id
> processing when using PERF_RECORD_COMPRESSED, otherwise we'd have to
> uncompress at the end to find the PERF_RECORD_FORK/PERF_RECORD_MMAP,
> etc.
>
> [root@quaco ~]# perf record -z2 --no-buildid sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.020 MB perf.data, compressed (original 0.001 MB, ratio is 2.153) ]
> [root@quaco ~]# perf report -D | grep PERF_RECORD_COMP
> 0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
> Error:
> failed to process sample
> 0 0x4f40 [0x195]: PERF_RECORD_COMPRESSED
> [root@quaco ~]#
>
> I'll play with it tomorrow.
Applied the whole patch set on top of the current perf/core
and the whole thing functions as expected.
~Alexey
>
> - Arnaldo
>
>> I've pushed what I have to the tmp.perf/core branch, please try to see
>> if I made any mistake in fixing up conflicts with BPF_PROG_INFO and
>> BPF_BTF header features. I'll continue tomorrow with 10-12/12.
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
2019-05-14 20:04 ` Arnaldo Carvalho de Melo
@ 2019-05-15 5:46 ` Alexey Budankov
0 siblings, 0 replies; 10+ messages in thread
From: Alexey Budankov @ 2019-05-15 5:46 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Jiri Olsa, Namhyung Kim, Alexander Shishkin, Peter Zijlstra,
Ingo Molnar, Andi Kleen, linux-kernel
On 14.05.2019 23:04, Arnaldo Carvalho de Melo wrote:
> Em Mon, Mar 18, 2019 at 08:44:42PM +0300, Alexey Budankov escreveu:
>>
>> Implemented -z,--compression_level[=<n>] option that enables compression
>> of mmaped kernel data buffers content in runtime during perf record
>> mode collection. Default option value is 1 (fastest compression).
>>
>> Compression overhead has been measured for serial and AIO streaming
>> when profiling matrix multiplication workload:
>>
>> -------------------------------------------------------------
>> | SERIAL | AIO-1 |
>> ----------------------------------------------------------------|
>
> Please don't have lines starting with --- in the cset comment log
> message, breaks scripts, fixing it up now.
Oops, will do my best about that. Thanks.
~Alexey
>
> - Arnaldo
>
>> |-z | OVH(x) | ratio(x) size(MiB) | OVH(x) | ratio(x) size(MiB) |
>> |---------------------------------------------------------------|
>> | 0 | 1,00 | 1,000 179,424 | 1,00 | 1,000 187,527 |
>> | 1 | 1,04 | 8,427 181,148 | 1,01 | 8,474 188,562 |
>> | 2 | 1,07 | 8,055 186,953 | 1,03 | 7,912 191,773 |
>> | 3 | 1,04 | 8,283 181,908 | 1,03 | 8,220 191,078 |
>> | 5 | 1,09 | 8,101 187,705 | 1,05 | 7,780 190,065 |
>> | 8 | 1,05 | 9,217 179,191 | 1,12 | 6,111 193,024 |
>> -----------------------------------------------------------------
>>
>> OVH = (Execution time with -z N) / (Execution time with -z 0)
>>
>> ratio - compression ratio
>> size - number of bytes that was compressed
>>
>> size ~= trace size x ratio
>>
>> Signed-off-by: Alexey Budankov <alexey.budankov@linux.intel.com>
>> ---
>> tools/perf/Documentation/perf-record.txt | 5 +++++
>> tools/perf/builtin-record.c | 25 ++++++++++++++++++++++++
>> 2 files changed, 30 insertions(+)
>>
>> diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
>> index 18fceb49434e..0567bacc2ae6 100644
>> --- a/tools/perf/Documentation/perf-record.txt
>> +++ b/tools/perf/Documentation/perf-record.txt
>> @@ -471,6 +471,11 @@ Also at some cases executing less trace write syscalls with bigger data size can
>> shorter than executing more trace write syscalls with smaller data size thus lowering
>> runtime profiling overhead.
>>
>> +-z::
>> +--compression-level[=n]::
>> +Produce compressed trace using specified level n (default: 1 - fastest compression,
>> +22 - smallest trace)
>> +
>> --all-kernel::
>> Configure all used events to run in kernel space.
>>
>> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
>> index 2e083891affa..7258f2964a3b 100644
>> --- a/tools/perf/builtin-record.c
>> +++ b/tools/perf/builtin-record.c
>> @@ -440,6 +440,26 @@ static int record__mmap_flush_parse(const struct option *opt,
>> return 0;
>> }
>>
>> +#ifdef HAVE_ZSTD_SUPPORT
>> +static unsigned int comp_level_default = 1;
>> +static int record__parse_comp_level(const struct option *opt,
>> + const char *str,
>> + int unset)
>> +{
>> + struct record_opts *opts = (struct record_opts *)opt->value;
>> +
>> + if (unset) {
>> + opts->comp_level = 0;
>> + } else {
>> + if (str)
>> + opts->comp_level = strtol(str, NULL, 0);
>> + if (!opts->comp_level)
>> + opts->comp_level = comp_level_default;
>> + }
>> +
>> + return 0;
>> +}
>> +#endif
>> static unsigned int comp_level_max = 22;
>>
>> static int record__comp_enabled(struct record *rec)
>> @@ -2169,6 +2189,11 @@ static struct option __record_options[] = {
>> OPT_CALLBACK(0, "affinity", &record.opts, "node|cpu",
>> "Set affinity mask of trace reading thread to NUMA node cpu mask or cpu of processed mmap buffer",
>> record__parse_affinity),
>> +#ifdef HAVE_ZSTD_SUPPORT
>> + OPT_CALLBACK_OPTARG('z', "compression-level", &record.opts, &comp_level_default,
>> + "n", "Produce compressed trace using specified level (default: 1 - fastest compression, 22 - smallest trace)",
>> + record__parse_comp_level),
>> +#endif
>> OPT_END()
>> };
>>
>> --
>> 2.20.1
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
2019-05-14 20:20 ` Arnaldo Carvalho de Melo
@ 2019-05-14 21:46 ` Arnaldo Carvalho de Melo
2019-05-15 8:43 ` Alexey Budankov
0 siblings, 1 reply; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-05-14 21:46 UTC (permalink / raw)
To: Alexey Budankov
Cc: Jiri Olsa, Namhyung Kim, Alexander Shishkin, Peter Zijlstra,
Ingo Molnar, Andi Kleen, linux-kernel
Em Tue, May 14, 2019 at 05:20:41PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Mar 18, 2019 at 08:44:42PM +0300, Alexey Budankov escreveu:
> >
> > Implemented -z,--compression_level[=<n>] option that enables compression
> > of mmaped kernel data buffers content in runtime during perf record
> > mode collection. Default option value is 1 (fastest compression).
<SNIP>
> [root@quaco ~]# perf record -z2
> ^C[ perf record: Woken up 1 times to write data ]
> 0x1746e0 [0x76]: failed to process type: 81 [Invalid argument]
> [ perf record: Captured and wrote 1.568 MB perf.data, compressed (original 0.452 MB, ratio is 3.995) ]
>
> [root@quaco ~]#
So, its the buildid processing at the end, so we can't do build-id
processing when using PERF_RECORD_COMPRESSED, otherwise we'd have to
uncompress at the end to find the PERF_RECORD_FORK/PERF_RECORD_MMAP,
etc.
[root@quaco ~]# perf record -z2 --no-buildid sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.020 MB perf.data, compressed (original 0.001 MB, ratio is 2.153) ]
[root@quaco ~]# perf report -D | grep PERF_RECORD_COMP
0x4f40 [0x195]: failed to process type: 81 [Invalid argument]
Error:
failed to process sample
0 0x4f40 [0x195]: PERF_RECORD_COMPRESSED
[root@quaco ~]#
I'll play with it tomorrow.
- Arnaldo
> I've pushed what I have to the tmp.perf/core branch, please try to see
> if I made any mistake in fixing up conflicts with BPF_PROG_INFO and
> BPF_BTF header features. I'll continue tomorrow with 10-12/12.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
2019-03-18 17:44 ` [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option Alexey Budankov
2019-05-14 20:04 ` Arnaldo Carvalho de Melo
@ 2019-05-14 20:20 ` Arnaldo Carvalho de Melo
2019-05-14 21:46 ` Arnaldo Carvalho de Melo
1 sibling, 1 reply; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-05-14 20:20 UTC (permalink / raw)
To: Alexey Budankov
Cc: Jiri Olsa, Namhyung Kim, Alexander Shishkin, Peter Zijlstra,
Ingo Molnar, Andi Kleen, linux-kernel
Em Mon, Mar 18, 2019 at 08:44:42PM +0300, Alexey Budankov escreveu:
>
> Implemented -z,--compression_level[=<n>] option that enables compression
> of mmaped kernel data buffers content in runtime during perf record
> mode collection. Default option value is 1 (fastest compression).
>
> Compression overhead has been measured for serial and AIO streaming
> when profiling matrix multiplication workload:
>
> -------------------------------------------------------------
> | SERIAL | AIO-1 |
> ----------------------------------------------------------------|
> |-z | OVH(x) | ratio(x) size(MiB) | OVH(x) | ratio(x) size(MiB) |
> |---------------------------------------------------------------|
> | 0 | 1,00 | 1,000 179,424 | 1,00 | 1,000 187,527 |
> | 1 | 1,04 | 8,427 181,148 | 1,01 | 8,474 188,562 |
> | 2 | 1,07 | 8,055 186,953 | 1,03 | 7,912 191,773 |
> | 3 | 1,04 | 8,283 181,908 | 1,03 | 8,220 191,078 |
> | 5 | 1,09 | 8,101 187,705 | 1,05 | 7,780 190,065 |
> | 8 | 1,05 | 9,217 179,191 | 1,12 | 6,111 193,024 |
> -----------------------------------------------------------------
>
> OVH = (Execution time with -z N) / (Execution time with -z 0)
>
> ratio - compression ratio
> size - number of bytes that was compressed
>
> size ~= trace size x ratio
[root@quaco ~]# perf record -z2
^C[ perf record: Woken up 1 times to write data ]
0x1746e0 [0x76]: failed to process type: 81 [Invalid argument]
[ perf record: Captured and wrote 1.568 MB perf.data, compressed (original 0.452 MB, ratio is 3.995) ]
[root@quaco ~]#
I've pushed what I have to the tmp.perf/core branch, please try to see
if I made any mistake in fixing up conflicts with BPF_PROG_INFO and
BPF_BTF header features. I'll continue tomorrow with 10-12/12.
- Arnaldo
> Signed-off-by: Alexey Budankov <alexey.budankov@linux.intel.com>
> ---
> tools/perf/Documentation/perf-record.txt | 5 +++++
> tools/perf/builtin-record.c | 25 ++++++++++++++++++++++++
> 2 files changed, 30 insertions(+)
>
> diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
> index 18fceb49434e..0567bacc2ae6 100644
> --- a/tools/perf/Documentation/perf-record.txt
> +++ b/tools/perf/Documentation/perf-record.txt
> @@ -471,6 +471,11 @@ Also at some cases executing less trace write syscalls with bigger data size can
> shorter than executing more trace write syscalls with smaller data size thus lowering
> runtime profiling overhead.
>
> +-z::
> +--compression-level[=n]::
> +Produce compressed trace using specified level n (default: 1 - fastest compression,
> +22 - smallest trace)
> +
> --all-kernel::
> Configure all used events to run in kernel space.
>
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index 2e083891affa..7258f2964a3b 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -440,6 +440,26 @@ static int record__mmap_flush_parse(const struct option *opt,
> return 0;
> }
>
> +#ifdef HAVE_ZSTD_SUPPORT
> +static unsigned int comp_level_default = 1;
> +static int record__parse_comp_level(const struct option *opt,
> + const char *str,
> + int unset)
> +{
> + struct record_opts *opts = (struct record_opts *)opt->value;
> +
> + if (unset) {
> + opts->comp_level = 0;
> + } else {
> + if (str)
> + opts->comp_level = strtol(str, NULL, 0);
> + if (!opts->comp_level)
> + opts->comp_level = comp_level_default;
> + }
> +
> + return 0;
> +}
> +#endif
> static unsigned int comp_level_max = 22;
>
> static int record__comp_enabled(struct record *rec)
> @@ -2169,6 +2189,11 @@ static struct option __record_options[] = {
> OPT_CALLBACK(0, "affinity", &record.opts, "node|cpu",
> "Set affinity mask of trace reading thread to NUMA node cpu mask or cpu of processed mmap buffer",
> record__parse_affinity),
> +#ifdef HAVE_ZSTD_SUPPORT
> + OPT_CALLBACK_OPTARG('z', "compression-level", &record.opts, &comp_level_default,
> + "n", "Produce compressed trace using specified level (default: 1 - fastest compression, 22 - smallest trace)",
> + record__parse_comp_level),
> +#endif
> OPT_END()
> };
>
> --
> 2.20.1
--
- Arnaldo
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
2019-03-18 17:44 ` [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option Alexey Budankov
@ 2019-05-14 20:04 ` Arnaldo Carvalho de Melo
2019-05-15 5:46 ` Alexey Budankov
2019-05-14 20:20 ` Arnaldo Carvalho de Melo
1 sibling, 1 reply; 10+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-05-14 20:04 UTC (permalink / raw)
To: Alexey Budankov
Cc: Jiri Olsa, Namhyung Kim, Alexander Shishkin, Peter Zijlstra,
Ingo Molnar, Andi Kleen, linux-kernel
Em Mon, Mar 18, 2019 at 08:44:42PM +0300, Alexey Budankov escreveu:
>
> Implemented -z,--compression_level[=<n>] option that enables compression
> of mmaped kernel data buffers content in runtime during perf record
> mode collection. Default option value is 1 (fastest compression).
>
> Compression overhead has been measured for serial and AIO streaming
> when profiling matrix multiplication workload:
>
> -------------------------------------------------------------
> | SERIAL | AIO-1 |
> ----------------------------------------------------------------|
Please don't have lines starting with --- in the cset comment log
message, breaks scripts, fixing it up now.
- Arnaldo
> |-z | OVH(x) | ratio(x) size(MiB) | OVH(x) | ratio(x) size(MiB) |
> |---------------------------------------------------------------|
> | 0 | 1,00 | 1,000 179,424 | 1,00 | 1,000 187,527 |
> | 1 | 1,04 | 8,427 181,148 | 1,01 | 8,474 188,562 |
> | 2 | 1,07 | 8,055 186,953 | 1,03 | 7,912 191,773 |
> | 3 | 1,04 | 8,283 181,908 | 1,03 | 8,220 191,078 |
> | 5 | 1,09 | 8,101 187,705 | 1,05 | 7,780 190,065 |
> | 8 | 1,05 | 9,217 179,191 | 1,12 | 6,111 193,024 |
> -----------------------------------------------------------------
>
> OVH = (Execution time with -z N) / (Execution time with -z 0)
>
> ratio - compression ratio
> size - number of bytes that was compressed
>
> size ~= trace size x ratio
>
> Signed-off-by: Alexey Budankov <alexey.budankov@linux.intel.com>
> ---
> tools/perf/Documentation/perf-record.txt | 5 +++++
> tools/perf/builtin-record.c | 25 ++++++++++++++++++++++++
> 2 files changed, 30 insertions(+)
>
> diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
> index 18fceb49434e..0567bacc2ae6 100644
> --- a/tools/perf/Documentation/perf-record.txt
> +++ b/tools/perf/Documentation/perf-record.txt
> @@ -471,6 +471,11 @@ Also at some cases executing less trace write syscalls with bigger data size can
> shorter than executing more trace write syscalls with smaller data size thus lowering
> runtime profiling overhead.
>
> +-z::
> +--compression-level[=n]::
> +Produce compressed trace using specified level n (default: 1 - fastest compression,
> +22 - smallest trace)
> +
> --all-kernel::
> Configure all used events to run in kernel space.
>
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index 2e083891affa..7258f2964a3b 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -440,6 +440,26 @@ static int record__mmap_flush_parse(const struct option *opt,
> return 0;
> }
>
> +#ifdef HAVE_ZSTD_SUPPORT
> +static unsigned int comp_level_default = 1;
> +static int record__parse_comp_level(const struct option *opt,
> + const char *str,
> + int unset)
> +{
> + struct record_opts *opts = (struct record_opts *)opt->value;
> +
> + if (unset) {
> + opts->comp_level = 0;
> + } else {
> + if (str)
> + opts->comp_level = strtol(str, NULL, 0);
> + if (!opts->comp_level)
> + opts->comp_level = comp_level_default;
> + }
> +
> + return 0;
> +}
> +#endif
> static unsigned int comp_level_max = 22;
>
> static int record__comp_enabled(struct record *rec)
> @@ -2169,6 +2189,11 @@ static struct option __record_options[] = {
> OPT_CALLBACK(0, "affinity", &record.opts, "node|cpu",
> "Set affinity mask of trace reading thread to NUMA node cpu mask or cpu of processed mmap buffer",
> record__parse_affinity),
> +#ifdef HAVE_ZSTD_SUPPORT
> + OPT_CALLBACK_OPTARG('z', "compression-level", &record.opts, &comp_level_default,
> + "n", "Produce compressed trace using specified level (default: 1 - fastest compression, 22 - smallest trace)",
> + record__parse_comp_level),
> +#endif
> OPT_END()
> };
>
> --
> 2.20.1
--
- Arnaldo
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option
2019-03-18 17:36 [PATCH v10 00/12] perf: enable compression of record mode trace to save storage space Alexey Budankov
@ 2019-03-18 17:44 ` Alexey Budankov
2019-05-14 20:04 ` Arnaldo Carvalho de Melo
2019-05-14 20:20 ` Arnaldo Carvalho de Melo
0 siblings, 2 replies; 10+ messages in thread
From: Alexey Budankov @ 2019-03-18 17:44 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Jiri Olsa, Namhyung Kim, Alexander Shishkin, Peter Zijlstra,
Ingo Molnar, Andi Kleen, linux-kernel
Implemented -z,--compression_level[=<n>] option that enables compression
of mmaped kernel data buffers content in runtime during perf record
mode collection. Default option value is 1 (fastest compression).
Compression overhead has been measured for serial and AIO streaming
when profiling matrix multiplication workload:
-------------------------------------------------------------
| SERIAL | AIO-1 |
----------------------------------------------------------------|
|-z | OVH(x) | ratio(x) size(MiB) | OVH(x) | ratio(x) size(MiB) |
|---------------------------------------------------------------|
| 0 | 1,00 | 1,000 179,424 | 1,00 | 1,000 187,527 |
| 1 | 1,04 | 8,427 181,148 | 1,01 | 8,474 188,562 |
| 2 | 1,07 | 8,055 186,953 | 1,03 | 7,912 191,773 |
| 3 | 1,04 | 8,283 181,908 | 1,03 | 8,220 191,078 |
| 5 | 1,09 | 8,101 187,705 | 1,05 | 7,780 190,065 |
| 8 | 1,05 | 9,217 179,191 | 1,12 | 6,111 193,024 |
-----------------------------------------------------------------
OVH = (Execution time with -z N) / (Execution time with -z 0)
ratio - compression ratio
size - number of bytes that was compressed
size ~= trace size x ratio
Signed-off-by: Alexey Budankov <alexey.budankov@linux.intel.com>
---
tools/perf/Documentation/perf-record.txt | 5 +++++
tools/perf/builtin-record.c | 25 ++++++++++++++++++++++++
2 files changed, 30 insertions(+)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 18fceb49434e..0567bacc2ae6 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -471,6 +471,11 @@ Also at some cases executing less trace write syscalls with bigger data size can
shorter than executing more trace write syscalls with smaller data size thus lowering
runtime profiling overhead.
+-z::
+--compression-level[=n]::
+Produce compressed trace using specified level n (default: 1 - fastest compression,
+22 - smallest trace)
+
--all-kernel::
Configure all used events to run in kernel space.
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 2e083891affa..7258f2964a3b 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -440,6 +440,26 @@ static int record__mmap_flush_parse(const struct option *opt,
return 0;
}
+#ifdef HAVE_ZSTD_SUPPORT
+static unsigned int comp_level_default = 1;
+static int record__parse_comp_level(const struct option *opt,
+ const char *str,
+ int unset)
+{
+ struct record_opts *opts = (struct record_opts *)opt->value;
+
+ if (unset) {
+ opts->comp_level = 0;
+ } else {
+ if (str)
+ opts->comp_level = strtol(str, NULL, 0);
+ if (!opts->comp_level)
+ opts->comp_level = comp_level_default;
+ }
+
+ return 0;
+}
+#endif
static unsigned int comp_level_max = 22;
static int record__comp_enabled(struct record *rec)
@@ -2169,6 +2189,11 @@ static struct option __record_options[] = {
OPT_CALLBACK(0, "affinity", &record.opts, "node|cpu",
"Set affinity mask of trace reading thread to NUMA node cpu mask or cpu of processed mmap buffer",
record__parse_affinity),
+#ifdef HAVE_ZSTD_SUPPORT
+ OPT_CALLBACK_OPTARG('z', "compression-level", &record.opts, &comp_level_default,
+ "n", "Produce compressed trace using specified level (default: 1 - fastest compression, 22 - smallest trace)",
+ record__parse_comp_level),
+#endif
OPT_END()
};
--
2.20.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2019-05-17 16:56 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-15 12:59 [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option Arnaldo Carvalho de Melo
2019-05-15 15:44 ` Alexey Budankov
2019-05-17 15:01 ` Arnaldo Carvalho de Melo
2019-05-17 16:56 ` Alexey Budankov
-- strict thread matches above, loose matches on Subject: below --
2019-03-18 17:36 [PATCH v10 00/12] perf: enable compression of record mode trace to save storage space Alexey Budankov
2019-03-18 17:44 ` [PATCH v10 09/12] perf record: implement -z,--compression_level[=<n>] option Alexey Budankov
2019-05-14 20:04 ` Arnaldo Carvalho de Melo
2019-05-15 5:46 ` Alexey Budankov
2019-05-14 20:20 ` Arnaldo Carvalho de Melo
2019-05-14 21:46 ` Arnaldo Carvalho de Melo
2019-05-15 8:43 ` Alexey Budankov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).