* [PATCH v2 01/87] trace-cmd library: Read option id with correct endian
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 02/87] trace-cmd report: Fix typos in error messages Tzvetomir Stoyanov (VMware)
` (85 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The id of a trace option is 2 bytes short integer. When reading it from
the trace file, use the tep handler associated with the file, if
available, to convert the option with the correct endian order.
A new helper function is introduced to read and convert 2 byte integer.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index ac57bc4f..0dbcdbdc 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -336,6 +336,18 @@ static char *read_string(struct tracecmd_input *handle)
return NULL;
}
+static int read2(struct tracecmd_input *handle, unsigned short *size)
+{
+ struct tep_handle *pevent = handle->pevent;
+ unsigned short data;
+
+ if (do_read_check(handle, &data, 2))
+ return -1;
+
+ *size = tep_read_number(pevent, &data, 2);
+ return 0;
+}
+
static int read4(struct tracecmd_input *handle, unsigned int *size)
{
struct tep_handle *pevent = handle->pevent;
@@ -2660,16 +2672,15 @@ static int handle_options(struct tracecmd_input *handle)
handle->options_start = lseek64(handle->fd, 0, SEEK_CUR);
for (;;) {
- if (do_read_check(handle, &option, 2))
+ if (read2(handle, &option))
return -1;
if (option == TRACECMD_OPTION_DONE)
break;
/* next 4 bytes is the size of the option */
- if (do_read_check(handle, &size, 4))
+ if (read4(handle, &size))
return -1;
- size = tep_read_number(handle->pevent, &size, 4);
buf = malloc(size);
if (!buf)
return -ENOMEM;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 02/87] trace-cmd report: Fix typos in error messages
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 01/87] trace-cmd library: Read option id with correct endian Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 03/87] tarce-cmd library: Fix version string memory leak Tzvetomir Stoyanov (VMware)
` (84 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Fixed typos in "trace-cmd report" messages that report reading and
parsing errors.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-read.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tracecmd/trace-read.c b/tracecmd/trace-read.c
index 6f43c1d2..31724b09 100644
--- a/tracecmd/trace-read.c
+++ b/tracecmd/trace-read.c
@@ -1290,7 +1290,7 @@ static void read_data_info(struct list_head *handle_list, enum output_type otype
die("error in reading buffer instance");
new_handle = tracecmd_buffer_instance_handle(handles->handle, i);
if (!new_handle) {
- warning("could not retreive handle %s", name);
+ warning("could not retrieve handle %s", name);
continue;
}
add_handle(new_handle, name);
@@ -1324,7 +1324,7 @@ static void read_data_info(struct list_head *handle_list, enum output_type otype
if (last_record) {
int cpu = last_record->cpu;
if (cpu >= last_handle->cpus)
- die("cpu %d creater than %d\n", cpu, last_handle->cpus);
+ die("cpu %d greater than %d\n", cpu, last_handle->cpus);
if (tscheck &&
last_handle->last_timestamp[cpu] > last_record->ts) {
errno = 0;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 03/87] tarce-cmd library: Fix version string memory leak
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 01/87] trace-cmd library: Read option id with correct endian Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 02/87] trace-cmd report: Fix typos in error messages Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 04/87] trace-cmd library: Fixed a memory leak on input handler close Tzvetomir Stoyanov (VMware)
` (83 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The version string is allocated when a VERSION option is processed, but
is never freed. Free it on input hanlder close.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 0dbcdbdc..9253bc37 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -3518,6 +3518,7 @@ void tracecmd_close(struct tracecmd_input *handle)
free(handle->cpu_data);
free(handle->uname);
free(handle->trace_clock);
+ free(handle->version);
close(handle->fd);
tracecmd_free_hooks(handle->hooks);
@@ -3959,6 +3960,7 @@ tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx)
new_handle->cpu_data = NULL;
new_handle->nr_buffers = 0;
new_handle->buffers = NULL;
+ new_handle->version = NULL;
new_handle->ref = 1;
if (handle->trace_clock) {
new_handle->trace_clock = strdup(handle->trace_clock);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 04/87] trace-cmd library: Fixed a memory leak on input handler close
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (2 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 03/87] tarce-cmd library: Fix version string memory leak Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 19:36 ` Steven Rostedt
2021-07-29 5:08 ` [PATCH v2 05/87] trace-cmd library: Fix possible memory corruption on processing a trace buffer Tzvetomir Stoyanov (VMware)
` (82 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When an input hanlder to a trace file is closed with tracecmd_close(),
the list with buffers is not freed. This leads to a memory leak. Added
logic to free that list.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 33 +++++++++++++++++++--------------
1 file changed, 19 insertions(+), 14 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 9253bc37..af11cbc6 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -3483,7 +3483,7 @@ void tracecmd_ref(struct tracecmd_input *handle)
*/
void tracecmd_close(struct tracecmd_input *handle)
{
- int cpu;
+ int i;
if (!handle)
return;
@@ -3496,21 +3496,21 @@ void tracecmd_close(struct tracecmd_input *handle)
if (--handle->ref)
return;
- for (cpu = 0; cpu < handle->cpus; cpu++) {
+ for (i = 0; i < handle->cpus; i++) {
/* The tracecmd_peek_data may have cached a record */
- free_next(handle, cpu);
- free_page(handle, cpu);
- if (handle->cpu_data && handle->cpu_data[cpu].kbuf) {
- kbuffer_free(handle->cpu_data[cpu].kbuf);
- if (handle->cpu_data[cpu].page_map)
- free_page_map(handle->cpu_data[cpu].page_map);
-
- if (handle->cpu_data[cpu].page_cnt)
+ free_next(handle, i);
+ free_page(handle, i);
+ if (handle->cpu_data && handle->cpu_data[i].kbuf) {
+ kbuffer_free(handle->cpu_data[i].kbuf);
+ if (handle->cpu_data[i].page_map)
+ free_page_map(handle->cpu_data[i].page_map);
+
+ if (handle->cpu_data[i].page_cnt)
tracecmd_warning("%d pages still allocated on cpu %d%s",
- handle->cpu_data[cpu].page_cnt, cpu,
- show_records(handle->cpu_data[cpu].pages,
- handle->cpu_data[cpu].nr_pages));
- free(handle->cpu_data[cpu].pages);
+ handle->cpu_data[i].page_cnt, i,
+ show_records(handle->cpu_data[i].pages,
+ handle->cpu_data[i].nr_pages));
+ free(handle->cpu_data[i].pages);
}
}
@@ -3521,6 +3521,11 @@ void tracecmd_close(struct tracecmd_input *handle)
free(handle->version);
close(handle->fd);
+ for (i = 0; i < handle->nr_buffers; i++)
+ free(handle->buffers[i].name);
+
+ free(handle->buffers);
+
tracecmd_free_hooks(handle->hooks);
handle->hooks = NULL;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 04/87] trace-cmd library: Fixed a memory leak on input handler close
2021-07-29 5:08 ` [PATCH v2 04/87] trace-cmd library: Fixed a memory leak on input handler close Tzvetomir Stoyanov (VMware)
@ 2021-07-29 19:36 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-07-29 19:36 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:08:36 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> When an input hanlder to a trace file is closed with tracecmd_close(),
> the list with buffers is not freed. This leads to a memory leak. Added
> logic to free that list.
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-input.c | 33 +++++++++++++++++++--------------
> 1 file changed, 19 insertions(+), 14 deletions(-)
>
> diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
> index 9253bc37..af11cbc6 100644
> --- a/lib/trace-cmd/trace-input.c
> +++ b/lib/trace-cmd/trace-input.c
> @@ -3483,7 +3483,7 @@ void tracecmd_ref(struct tracecmd_input *handle)
> */
> void tracecmd_close(struct tracecmd_input *handle)
> {
> - int cpu;
> + int i;
It's OK to add int i. There's no reason to rename cpu, as I believe the
"cpu" variable is still appropriate. The compiled code will have no
difference, as its a simple optimization to merge multiple local variables.
>
> if (!handle)
> return;
> @@ -3496,21 +3496,21 @@ void tracecmd_close(struct tracecmd_input *handle)
> if (--handle->ref)
> return;
>
> - for (cpu = 0; cpu < handle->cpus; cpu++) {
> + for (i = 0; i < handle->cpus; i++) {
> /* The tracecmd_peek_data may have cached a record */
> - free_next(handle, cpu);
> - free_page(handle, cpu);
> - if (handle->cpu_data && handle->cpu_data[cpu].kbuf) {
> - kbuffer_free(handle->cpu_data[cpu].kbuf);
> - if (handle->cpu_data[cpu].page_map)
> - free_page_map(handle->cpu_data[cpu].page_map);
> -
> - if (handle->cpu_data[cpu].page_cnt)
> + free_next(handle, i);
> + free_page(handle, i);
> + if (handle->cpu_data && handle->cpu_data[i].kbuf) {
> + kbuffer_free(handle->cpu_data[i].kbuf);
> + if (handle->cpu_data[i].page_map)
> + free_page_map(handle->cpu_data[i].page_map);
> +
> + if (handle->cpu_data[i].page_cnt)
> tracecmd_warning("%d pages still allocated on cpu %d%s",
> - handle->cpu_data[cpu].page_cnt, cpu,
> - show_records(handle->cpu_data[cpu].pages,
> - handle->cpu_data[cpu].nr_pages));
> - free(handle->cpu_data[cpu].pages);
> + handle->cpu_data[i].page_cnt, i,
> + show_records(handle->cpu_data[i].pages,
> + handle->cpu_data[i].nr_pages));
> + free(handle->cpu_data[i].pages);
One reason not to change it, is because I'm staring at this code trying
to figure out what logical change happened here. Then I realize, that
it's just a variable change.
Let's just add "int i" and keep "int cpu". Especially since I plan on
backporting this patch, and I don't need unnecessary conflicts to deal
with.
> }
> }
>
> @@ -3521,6 +3521,11 @@ void tracecmd_close(struct tracecmd_input *handle)
> free(handle->version);
> close(handle->fd);
>
> + for (i = 0; i < handle->nr_buffers; i++)
> + free(handle->buffers[i].name);
> +
> + free(handle->buffers);
I take it, this patch only does the above.
-- Steve
> +
> tracecmd_free_hooks(handle->hooks);
> handle->hooks = NULL;
>
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 05/87] trace-cmd library: Fix possible memory corruption on processing a trace buffer
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (3 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 04/87] trace-cmd library: Fixed a memory leak on input handler close Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 19:39 ` Steven Rostedt
2021-07-29 19:52 ` Steven Rostedt
2021-07-29 5:08 ` [PATCH v2 06/87] trace-cmd library: Add constructor and destructor Tzvetomir Stoyanov (VMware)
` (81 subsequent siblings)
86 siblings, 2 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When a new trace buffer is read from the trace file, a new input handler
is duplicated from the top one. Some of these data are poiters and
should not be duplicated, as it could lead to a memory corruption on
handler close.
Added a safety check to ensure requested buffer index is valid.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index af11cbc6..787d6825 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -3946,13 +3946,14 @@ struct tracecmd_input *
tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx)
{
struct tracecmd_input *new_handle;
- struct input_buffer_instance *buffer = &handle->buffers[indx];
+ struct input_buffer_instance *buffer;
size_t offset;
ssize_t ret;
if (indx >= handle->nr_buffers)
return NULL;
+ buffer = &handle->buffers[indx];
/*
* We make a copy of the current handle, but we substitute
* the cpu data with the cpu data for this buffer.
@@ -3966,6 +3967,7 @@ tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx)
new_handle->nr_buffers = 0;
new_handle->buffers = NULL;
new_handle->version = NULL;
+ new_handle->guest = NULL;
new_handle->ref = 1;
if (handle->trace_clock) {
new_handle->trace_clock = strdup(handle->trace_clock);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 05/87] trace-cmd library: Fix possible memory corruption on processing a trace buffer
2021-07-29 5:08 ` [PATCH v2 05/87] trace-cmd library: Fix possible memory corruption on processing a trace buffer Tzvetomir Stoyanov (VMware)
@ 2021-07-29 19:39 ` Steven Rostedt
2021-07-29 19:52 ` Steven Rostedt
1 sibling, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-07-29 19:39 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:08:37 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> When a new trace buffer is read from the trace file, a new input handler
> is duplicated from the top one. Some of these data are poiters and
BTW, I'm finding lots of typos in your change logs. What editor do you
use? Can you add a spell check to it?
For instance, I use vim for my change log editing, and have been running:
:set spell
Which will highlight all the misspelled words.
Thanks,
-- Steve
> should not be duplicated, as it could lead to a memory corruption on
> handler close.
> Added a safety check to ensure requested buffer index is valid.
^ permalink raw reply [flat|nested] 118+ messages in thread
* Re: [PATCH v2 05/87] trace-cmd library: Fix possible memory corruption on processing a trace buffer
2021-07-29 5:08 ` [PATCH v2 05/87] trace-cmd library: Fix possible memory corruption on processing a trace buffer Tzvetomir Stoyanov (VMware)
2021-07-29 19:39 ` Steven Rostedt
@ 2021-07-29 19:52 ` Steven Rostedt
1 sibling, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-07-29 19:52 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:08:37 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Added a safety check to ensure requested buffer index is valid.
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-input.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
> index af11cbc6..787d6825 100644
> --- a/lib/trace-cmd/trace-input.c
> +++ b/lib/trace-cmd/trace-input.c
> @@ -3946,13 +3946,14 @@ struct tracecmd_input *
> tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx)
> {
> struct tracecmd_input *new_handle;
> - struct input_buffer_instance *buffer = &handle->buffers[indx];
> + struct input_buffer_instance *buffer;
> size_t offset;
> ssize_t ret;
>
> if (indx >= handle->nr_buffers)
> return NULL;
>
> + buffer = &handle->buffers[indx];
This part is unneeded.
You could have indx = 10000000000000, and it wont bug.
Try it!
$ echo '
#include <stdio.h>
struct my_buffer {
int buf;
};
struct my_handle {
struct my_buffer *buffers;
};
int main() {
int indx = 10000000;
struct my_buffer buf;
struct my_handle hand = { .buffers = &buf };
struct my_handle *phand = &hand;
struct my_buffer *pbuf = &phand->buffers[indx];
printf("pbuf = %p\n", pbuf);
return 0;
}' > /tmp/blah
$ gcc -o /tmp/blah /tmp/blah.c -g -Wall
$ /tmp/blah
pbuf = 0x7ffe867b9b74
The reason is because we are getting the address of the indexed
location, and we are not dereferencing it. Thus, it is perfectly safe
to keep the code as is. There was no safety check added. Please remove
this hunk.
/me is reminded of the first X-Men movie, where Rogue warned Wolverine
about the guy that was about to stab him. Afterward, she said, "I saved
your life", and Wolverine replied "No you didn't.". As Rogue didn't
know that Wolverine had super healing powers where the knife would not
kill him. ;-)
-- Steve
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 06/87] trace-cmd library: Add constructor and destructor
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (4 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 05/87] trace-cmd library: Fix possible memory corruption on processing a trace buffer Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 20:06 ` Steven Rostedt
2021-07-29 5:08 ` [PATCH v2 07/87] trace-cmd library: Add cache functionality to network message handler Tzvetomir Stoyanov (VMware)
` (80 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added empty constructor and destructor routines of the trace-cmd
library. They will be used to initialize and free compression context of
the library.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-util.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c
index b0c98c72..0f49a21a 100644
--- a/lib/trace-cmd/trace-util.c
+++ b/lib/trace-cmd/trace-util.c
@@ -624,3 +624,13 @@ bool tracecmd_is_version_supported(unsigned int version)
return true;
return false;
}
+
+static void __attribute__ ((constructor)) tracecmd_lib_init(void)
+{
+
+}
+
+static void __attribute__((destructor)) tracecmd_lib_free(void)
+{
+
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 06/87] trace-cmd library: Add constructor and destructor
2021-07-29 5:08 ` [PATCH v2 06/87] trace-cmd library: Add constructor and destructor Tzvetomir Stoyanov (VMware)
@ 2021-07-29 20:06 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-07-29 20:06 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:08:38 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Added empty constructor and destructor routines of the trace-cmd
> library. They will be used to initialize and free compression context of
> the library.
This should be folded into the first patch they are used. Patches
should not be broken into just adding skeletons without usage.
-- Steve
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-util.c | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c
> index b0c98c72..0f49a21a 100644
> --- a/lib/trace-cmd/trace-util.c
> +++ b/lib/trace-cmd/trace-util.c
> @@ -624,3 +624,13 @@ bool tracecmd_is_version_supported(unsigned int version)
> return true;
> return false;
> }
> +
> +static void __attribute__ ((constructor)) tracecmd_lib_init(void)
> +{
> +
> +}
> +
> +static void __attribute__((destructor)) tracecmd_lib_free(void)
> +{
> +
> +}
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 07/87] trace-cmd library: Add cache functionality to network message handler
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (5 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 06/87] trace-cmd library: Add constructor and destructor Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 20:33 ` Steven Rostedt
2021-07-29 5:08 ` [PATCH v2 08/87] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
` (79 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Network message handler is used to send trace metadata through a network
socket, instead writing it into a trace file. There are use cases,
that could require to do a lseek() on the metadata. It is hard to
implement lseek on a network socket, that's why for such use cases a
cache to a local file is introduced. Once the metadata is constructed,
the local cache is send to the socket. A new library API is used to
enable the local cache:
tracecmd_msg_handle_cache()
The local cahce is flushed on the socket when the
tracecmd_msg_finish_sending_data()
is called.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 5 +
lib/trace-cmd/include/trace-cmd-local.h | 1 +
lib/trace-cmd/trace-msg.c | 131 +++++++++++++-----
3 files changed, 103 insertions(+), 34 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 6440084d..68715580 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -339,12 +339,16 @@ enum tracecmd_msg_flags {
};
/* for both client and server */
+#define MSG_CACHE_FILE "/tmp/trace_msg_cacheXXXXXX"
struct tracecmd_msg_handle {
int fd;
short cpu_count;
short version; /* Current protocol version */
unsigned long flags;
bool done;
+ bool cache;
+ int cfd;
+ char cfile[26]; /* strlen(MSG_CACHE_FILE) */
};
struct tracecmd_tsync_protos {
@@ -353,6 +357,7 @@ struct tracecmd_tsync_protos {
struct tracecmd_msg_handle *
tracecmd_msg_handle_alloc(int fd, unsigned long flags);
+int tracecmd_msg_handle_cache(struct tracecmd_msg_handle *msg_handle);
/* Closes the socket and frees the handle */
void tracecmd_msg_handle_close(struct tracecmd_msg_handle *msg_handle);
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 821b5cdb..7691cc05 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -31,5 +31,6 @@ void tracecmd_info(const char *fmt, ...);
#endif
#endif
+off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off_t offset, int whence);
#endif /* _TRACE_CMD_LOCAL_H */
diff --git a/lib/trace-cmd/trace-msg.c b/lib/trace-cmd/trace-msg.c
index 6667028e..e856fb33 100644
--- a/lib/trace-cmd/trace-msg.c
+++ b/lib/trace-cmd/trace-msg.c
@@ -154,33 +154,54 @@ static inline int msg_buf_len(struct tracecmd_msg *msg)
return ntohl(msg->hdr.size) - MSG_HDR_LEN - ntohl(msg->hdr.cmd_size);
}
-static int msg_write(int fd, struct tracecmd_msg *msg)
+static int __msg_write(int fd, struct tracecmd_msg *msg, bool network)
{
- int cmd = ntohl(msg->hdr.cmd);
int msg_size, data_size;
int ret;
-
- if (cmd < 0 || cmd >= MSG_NR_COMMANDS)
- return -EINVAL;
-
- dprint("msg send: %d (%s) [%d]\n",
- cmd, cmd_to_name(cmd), ntohl(msg->hdr.size));
-
+ int cmd;
+
+ if (network) {
+ cmd = ntohl(msg->hdr.cmd);
+ if (cmd < 0 || cmd >= MSG_NR_COMMANDS)
+ return -EINVAL;
+ dprint("msg send: %d (%s) [%d]\n",
+ cmd, cmd_to_name(cmd), ntohl(msg->hdr.size));
+ }
msg_size = MSG_HDR_LEN + ntohl(msg->hdr.cmd_size);
data_size = ntohl(msg->hdr.size) - msg_size;
if (data_size < 0)
return -EINVAL;
- ret = __do_write_check(fd, msg, msg_size);
- if (ret < 0)
- return ret;
-
+ if (network) {
+ ret = __do_write_check(fd, msg, msg_size);
+ if (ret < 0)
+ return ret;
+ }
if (!data_size)
return 0;
return __do_write_check(fd, msg->buf, data_size);
}
+__hidden off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off64_t offset, int whence)
+{
+ /* lseek works only if the handle is in cache mode,
+ * cannot seek on a network socket
+ */
+ if (!msg_handle->cache || msg_handle->cfd < 0)
+ return (off64_t)-1;
+ return lseek64(msg_handle->cfd, offset, whence);
+}
+
+static int msg_write(struct tracecmd_msg_handle *msg_handle, struct tracecmd_msg *msg)
+{
+ if (msg_handle->cache && msg_handle->cfd >= 0)
+ return __msg_write(msg_handle->cfd, msg, false);
+
+
+ return __msg_write(msg_handle->fd, msg, true);
+}
+
enum msg_trace_flags {
MSG_TRACE_USE_FIFOS = 1 << 0,
};
@@ -274,11 +295,11 @@ static void msg_free(struct tracecmd_msg *msg)
memset(msg, 0, sizeof(*msg));
}
-static int tracecmd_msg_send(int fd, struct tracecmd_msg *msg)
+static int tracecmd_msg_send(struct tracecmd_msg_handle *msg_handle, struct tracecmd_msg *msg)
{
int ret = 0;
- ret = msg_write(fd, msg);
+ ret = msg_write(msg_handle, msg);
if (ret < 0)
ret = -ECOMM;
@@ -287,11 +308,11 @@ static int tracecmd_msg_send(int fd, struct tracecmd_msg *msg)
return ret;
}
-static int msg_send_nofree(int fd, struct tracecmd_msg *msg)
+static int msg_send_nofree(struct tracecmd_msg_handle *msg_handle, struct tracecmd_msg *msg)
{
int ret = 0;
- ret = msg_write(fd, msg);
+ ret = msg_write(msg_handle, msg);
if (ret < 0)
ret = -ECOMM;
@@ -454,7 +475,7 @@ static int tracecmd_msg_send_notsupp(struct tracecmd_msg_handle *msg_handle)
struct tracecmd_msg msg;
tracecmd_msg_init(MSG_NOT_SUPP, &msg);
- return tracecmd_msg_send(msg_handle->fd, &msg);
+ return tracecmd_msg_send(msg_handle, &msg);
}
static int handle_unexpected_msg(struct tracecmd_msg_handle *msg_handle,
@@ -472,7 +493,6 @@ int tracecmd_msg_send_init_data(struct tracecmd_msg_handle *msg_handle,
unsigned int **client_ports)
{
struct tracecmd_msg msg;
- int fd = msg_handle->fd;
unsigned int *ports;
int i, cpus, ret;
char *p, *buf_end;
@@ -485,13 +505,13 @@ int tracecmd_msg_send_init_data(struct tracecmd_msg_handle *msg_handle,
if (ret < 0)
goto out;
- ret = tracecmd_msg_send(fd, &msg);
+ ret = tracecmd_msg_send(msg_handle, &msg);
if (ret < 0)
goto out;
msg_free(&msg);
- ret = tracecmd_msg_wait_for_msg(fd, &msg);
+ ret = tracecmd_msg_wait_for_msg(msg_handle->fd, &msg);
if (ret < 0)
goto out;
@@ -564,12 +584,57 @@ tracecmd_msg_handle_alloc(int fd, unsigned long flags)
handle->fd = fd;
handle->flags = flags;
+ handle->cfd = -1;
+ handle->cache = false;
return handle;
}
+int tracecmd_msg_handle_cache(struct tracecmd_msg_handle *msg_handle)
+{
+ if (msg_handle->cfd < 0) {
+ strcpy(msg_handle->cfile, MSG_CACHE_FILE);
+ msg_handle->cfd = mkstemp(msg_handle->cfile);
+ if (msg_handle->cfd < 0)
+ return -1;
+ }
+ msg_handle->cache = true;
+ return 0;
+}
+
+static int flush_cache(struct tracecmd_msg_handle *msg_handle)
+{
+ char buf[MSG_MAX_DATA_LEN];
+ int ret;
+ int fd;
+
+ if (!msg_handle->cache || msg_handle->cfd < 0)
+ return 0;
+ close(msg_handle->cfd);
+ msg_handle->cfd = -1;
+ fd = open(msg_handle->cfile, O_RDONLY);
+ if (fd < 0)
+ return -1;
+ do {
+ ret = read(fd, buf, MSG_MAX_DATA_LEN);
+ if (ret <= 0)
+ break;
+ ret = tracecmd_msg_data_send(msg_handle, buf, ret);
+ if (ret < 0)
+ break;
+ } while (ret >= 0);
+
+ close(fd);
+ unlink(msg_handle->cfile);
+ return ret;
+}
+
void tracecmd_msg_handle_close(struct tracecmd_msg_handle *msg_handle)
{
close(msg_handle->fd);
+ if (msg_handle->cfd >= 0) {
+ close(msg_handle->cfd);
+ unlink(msg_handle->cfile);
+ }
free(msg_handle);
}
@@ -666,7 +731,7 @@ int tracecmd_msg_send_port_array(struct tracecmd_msg_handle *msg_handle,
if (ret < 0)
return ret;
- ret = tracecmd_msg_send(msg_handle->fd, &msg);
+ ret = tracecmd_msg_send(msg_handle, &msg);
if (ret < 0)
return ret;
@@ -678,7 +743,7 @@ int tracecmd_msg_send_close_msg(struct tracecmd_msg_handle *msg_handle)
struct tracecmd_msg msg;
tracecmd_msg_init(MSG_CLOSE, &msg);
- return tracecmd_msg_send(msg_handle->fd, &msg);
+ return tracecmd_msg_send(msg_handle, &msg);
}
int tracecmd_msg_send_close_resp_msg(struct tracecmd_msg_handle *msg_handle)
@@ -686,14 +751,13 @@ int tracecmd_msg_send_close_resp_msg(struct tracecmd_msg_handle *msg_handle)
struct tracecmd_msg msg;
tracecmd_msg_init(MSG_CLOSE_RESP, &msg);
- return tracecmd_msg_send(msg_handle->fd, &msg);
+ return tracecmd_msg_send(msg_handle, &msg);
}
int tracecmd_msg_data_send(struct tracecmd_msg_handle *msg_handle,
const char *buf, int size)
{
struct tracecmd_msg msg;
- int fd = msg_handle->fd;
int n;
int ret;
int count = 0;
@@ -721,7 +785,7 @@ int tracecmd_msg_data_send(struct tracecmd_msg_handle *msg_handle,
memcpy(msg.buf, buf + count, n);
n = 0;
}
- ret = msg_write(fd, &msg);
+ ret = msg_write(msg_handle, &msg);
if (ret < 0)
break;
}
@@ -735,8 +799,10 @@ int tracecmd_msg_finish_sending_data(struct tracecmd_msg_handle *msg_handle)
struct tracecmd_msg msg;
int ret;
+ flush_cache(msg_handle);
+ msg_handle->cache = false;
tracecmd_msg_init(MSG_FIN_DATA, &msg);
- ret = tracecmd_msg_send(msg_handle->fd, &msg);
+ ret = tracecmd_msg_send(msg_handle, &msg);
if (ret < 0)
return ret;
return 0;
@@ -752,10 +818,7 @@ int tracecmd_msg_read_data(struct tracecmd_msg_handle *msg_handle, int ofd)
while (!tracecmd_msg_done(msg_handle)) {
ret = tracecmd_msg_recv_wait(msg_handle->fd, &msg);
if (ret < 0) {
- if (ret == -ETIMEDOUT)
- tracecmd_warning("Connection timed out\n");
- else
- tracecmd_warning("reading client");
+ tracecmd_warning("reading client %d (%s)", ret, strerror(ret));
return ret;
}
@@ -959,7 +1022,7 @@ int tracecmd_msg_send_trace_req(struct tracecmd_msg_handle *msg_handle,
if (ret < 0)
return ret;
- return tracecmd_msg_send(msg_handle->fd, &msg);
+ return tracecmd_msg_send(msg_handle, &msg);
}
static int get_trace_req_protos(char *buf, int length,
@@ -1151,7 +1214,7 @@ int tracecmd_msg_send_time_sync(struct tracecmd_msg_handle *msg_handle,
msg.hdr.size = htonl(ntohl(msg.hdr.size) + payload_size);
msg.buf = payload;
- return msg_send_nofree(msg_handle->fd, &msg);
+ return msg_send_nofree(msg_handle, &msg);
}
/**
@@ -1283,7 +1346,7 @@ int tracecmd_msg_send_trace_resp(struct tracecmd_msg_handle *msg_handle,
if (ret < 0)
return ret;
- return tracecmd_msg_send(msg_handle->fd, &msg);
+ return tracecmd_msg_send(msg_handle, &msg);
}
int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle *msg_handle,
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 07/87] trace-cmd library: Add cache functionality to network message handler
2021-07-29 5:08 ` [PATCH v2 07/87] trace-cmd library: Add cache functionality to network message handler Tzvetomir Stoyanov (VMware)
@ 2021-07-29 20:33 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-07-29 20:33 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:08:39 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Network message handler is used to send trace metadata through a network
> socket, instead writing it into a trace file. There are use cases,
> that could require to do a lseek() on the metadata. It is hard to
> implement lseek on a network socket, that's why for such use cases a
> cache to a local file is introduced. Once the metadata is constructed,
> the local cache is send to the socket. A new library API is used to
> enable the local cache:
> tracecmd_msg_handle_cache()
> The local cahce is flushed on the socket when the
> tracecmd_msg_finish_sending_data()
> is called.
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> .../include/private/trace-cmd-private.h | 5 +
> lib/trace-cmd/include/trace-cmd-local.h | 1 +
> lib/trace-cmd/trace-msg.c | 131 +++++++++++++-----
> 3 files changed, 103 insertions(+), 34 deletions(-)
>
> diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
> index 6440084d..68715580 100644
> --- a/lib/trace-cmd/include/private/trace-cmd-private.h
> +++ b/lib/trace-cmd/include/private/trace-cmd-private.h
> @@ -339,12 +339,16 @@ enum tracecmd_msg_flags {
> };
>
> /* for both client and server */
> +#define MSG_CACHE_FILE "/tmp/trace_msg_cacheXXXXXX"
> struct tracecmd_msg_handle {
> int fd;
> short cpu_count;
> short version; /* Current protocol version */
> unsigned long flags;
> bool done;
> + bool cache;
> + int cfd;
> + char cfile[26]; /* strlen(MSG_CACHE_FILE) */
char cfile[sizeof(MSG_CACHE_FILE)];
Then you don't need to worry about actual size, and prevent the bug you
have here.
strlen(MSG_CACHE_FILE) == 26, but when you add '\0' it's 27 bytes, and
you just overflowed the buffer.
sizeof(MSG_CACHE_FILE) returns 27, and you can use it directly, as it
is determined at compile time.
> };
>
> struct tracecmd_tsync_protos {
> @@ -353,6 +357,7 @@ struct tracecmd_tsync_protos {
>
> struct tracecmd_msg_handle *
> tracecmd_msg_handle_alloc(int fd, unsigned long flags);
> +int tracecmd_msg_handle_cache(struct tracecmd_msg_handle *msg_handle);
>
> /* Closes the socket and frees the handle */
> void tracecmd_msg_handle_close(struct tracecmd_msg_handle *msg_handle);
> diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
> index 821b5cdb..7691cc05 100644
> --- a/lib/trace-cmd/include/trace-cmd-local.h
> +++ b/lib/trace-cmd/include/trace-cmd-local.h
> @@ -31,5 +31,6 @@ void tracecmd_info(const char *fmt, ...);
> #endif
> #endif
>
> +off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off_t offset, int whence);
>
> #endif /* _TRACE_CMD_LOCAL_H */
> diff --git a/lib/trace-cmd/trace-msg.c b/lib/trace-cmd/trace-msg.c
> index 6667028e..e856fb33 100644
> --- a/lib/trace-cmd/trace-msg.c
> +++ b/lib/trace-cmd/trace-msg.c
> @@ -154,33 +154,54 @@ static inline int msg_buf_len(struct tracecmd_msg *msg)
> return ntohl(msg->hdr.size) - MSG_HDR_LEN - ntohl(msg->hdr.cmd_size);
> }
>
> -static int msg_write(int fd, struct tracecmd_msg *msg)
> +static int __msg_write(int fd, struct tracecmd_msg *msg, bool network)
> {
> - int cmd = ntohl(msg->hdr.cmd);
> int msg_size, data_size;
> int ret;
> -
> - if (cmd < 0 || cmd >= MSG_NR_COMMANDS)
> - return -EINVAL;
> -
> - dprint("msg send: %d (%s) [%d]\n",
> - cmd, cmd_to_name(cmd), ntohl(msg->hdr.size));
> -
> + int cmd;
> +
> + if (network) {
> + cmd = ntohl(msg->hdr.cmd);
> + if (cmd < 0 || cmd >= MSG_NR_COMMANDS)
> + return -EINVAL;
> + dprint("msg send: %d (%s) [%d]\n",
> + cmd, cmd_to_name(cmd), ntohl(msg->hdr.size));
> + }
> msg_size = MSG_HDR_LEN + ntohl(msg->hdr.cmd_size);
> data_size = ntohl(msg->hdr.size) - msg_size;
> if (data_size < 0)
> return -EINVAL;
>
> - ret = __do_write_check(fd, msg, msg_size);
> - if (ret < 0)
> - return ret;
> -
> + if (network) {
> + ret = __do_write_check(fd, msg, msg_size);
> + if (ret < 0)
> + return ret;
> + }
> if (!data_size)
> return 0;
>
> return __do_write_check(fd, msg->buf, data_size);
> }
>
> +__hidden off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off64_t offset, int whence)
> +{
> + /* lseek works only if the handle is in cache mode,
> + * cannot seek on a network socket
> + */
Nit, but change the above to:
/*
* lseek works only if the handle is in cache mode,
* cannot seek on a network socket
*/
as we keep with normal Linux comment style and not Linux networking
comment style.
> + if (!msg_handle->cache || msg_handle->cfd < 0)
> + return (off64_t)-1;
> + return lseek64(msg_handle->cfd, offset, whence);
> +}
> +
> +static int msg_write(struct tracecmd_msg_handle *msg_handle, struct tracecmd_msg *msg)
> +{
> + if (msg_handle->cache && msg_handle->cfd >= 0)
> + return __msg_write(msg_handle->cfd, msg, false);
> +
> +
> + return __msg_write(msg_handle->fd, msg, true);
> +}
> +
> enum msg_trace_flags {
> MSG_TRACE_USE_FIFOS = 1 << 0,
> };
> @@ -274,11 +295,11 @@ static void msg_free(struct tracecmd_msg *msg)
> memset(msg, 0, sizeof(*msg));
> }
>
> -static int tracecmd_msg_send(int fd, struct tracecmd_msg *msg)
> +static int tracecmd_msg_send(struct tracecmd_msg_handle *msg_handle, struct tracecmd_msg *msg)
> {
> int ret = 0;
>
> - ret = msg_write(fd, msg);
> + ret = msg_write(msg_handle, msg);
> if (ret < 0)
> ret = -ECOMM;
>
> @@ -287,11 +308,11 @@ static int tracecmd_msg_send(int fd, struct tracecmd_msg *msg)
> return ret;
> }
>
> -static int msg_send_nofree(int fd, struct tracecmd_msg *msg)
> +static int msg_send_nofree(struct tracecmd_msg_handle *msg_handle, struct tracecmd_msg *msg)
> {
> int ret = 0;
>
> - ret = msg_write(fd, msg);
> + ret = msg_write(msg_handle, msg);
> if (ret < 0)
> ret = -ECOMM;
>
> @@ -454,7 +475,7 @@ static int tracecmd_msg_send_notsupp(struct tracecmd_msg_handle *msg_handle)
> struct tracecmd_msg msg;
>
> tracecmd_msg_init(MSG_NOT_SUPP, &msg);
> - return tracecmd_msg_send(msg_handle->fd, &msg);
> + return tracecmd_msg_send(msg_handle, &msg);
> }
>
> static int handle_unexpected_msg(struct tracecmd_msg_handle *msg_handle,
> @@ -472,7 +493,6 @@ int tracecmd_msg_send_init_data(struct tracecmd_msg_handle *msg_handle,
> unsigned int **client_ports)
> {
> struct tracecmd_msg msg;
> - int fd = msg_handle->fd;
> unsigned int *ports;
> int i, cpus, ret;
> char *p, *buf_end;
> @@ -485,13 +505,13 @@ int tracecmd_msg_send_init_data(struct tracecmd_msg_handle *msg_handle,
> if (ret < 0)
> goto out;
>
> - ret = tracecmd_msg_send(fd, &msg);
> + ret = tracecmd_msg_send(msg_handle, &msg);
> if (ret < 0)
> goto out;
>
> msg_free(&msg);
>
> - ret = tracecmd_msg_wait_for_msg(fd, &msg);
> + ret = tracecmd_msg_wait_for_msg(msg_handle->fd, &msg);
> if (ret < 0)
> goto out;
>
> @@ -564,12 +584,57 @@ tracecmd_msg_handle_alloc(int fd, unsigned long flags)
>
> handle->fd = fd;
> handle->flags = flags;
> + handle->cfd = -1;
> + handle->cache = false;
> return handle;
> }
>
> +int tracecmd_msg_handle_cache(struct tracecmd_msg_handle *msg_handle)
> +{
> + if (msg_handle->cfd < 0) {
> + strcpy(msg_handle->cfile, MSG_CACHE_FILE);
And without the sizeof() change, here you overflow the allocated space.
> + msg_handle->cfd = mkstemp(msg_handle->cfile);
> + if (msg_handle->cfd < 0)
> + return -1;
> + }
We could unlink the file right away (see more below).
> + msg_handle->cache = true;
> + return 0;
> +}
> +
> +static int flush_cache(struct tracecmd_msg_handle *msg_handle)
> +{
> + char buf[MSG_MAX_DATA_LEN];
> + int ret;
> + int fd;
> +
> + if (!msg_handle->cache || msg_handle->cfd < 0)
> + return 0;
> + close(msg_handle->cfd);
> + msg_handle->cfd = -1;
> + fd = open(msg_handle->cfile, O_RDONLY);
Why the above dance? Instead (and if we unlink the file so it is gone
as soon as the application is done (for example on a crash), you can just do:
ret = lseek64(msg_handle->cfd, 0, SEEK_SET);
and that will rewind to the beginning of the file, where you can do the
read below.
By unlinking right after creation, you do not need to worry about
leftover temp files because the application exited before it could
clean it up.
> + if (fd < 0)
> + return -1;
> + do {
> + ret = read(fd, buf, MSG_MAX_DATA_LEN);
> + if (ret <= 0)
> + break;
> + ret = tracecmd_msg_data_send(msg_handle, buf, ret);
> + if (ret < 0)
> + break;
> + } while (ret >= 0);
> +
> + close(fd);
> + unlink(msg_handle->cfile);
> + return ret;
> +}
> +
> void tracecmd_msg_handle_close(struct tracecmd_msg_handle *msg_handle)
> {
> close(msg_handle->fd);
> + if (msg_handle->cfd >= 0) {
> + close(msg_handle->cfd);
> + unlink(msg_handle->cfile);
And by unlinking right away, you do not need to have multiple "unlinks"
all over the place.
-- Steve
> + }
> free(msg_handle);
> }
>
> @@ -666,7 +731,7 @@ int tracecmd_msg_send_port_array(struct
> tracecmd_msg_handle *msg_handle, if (ret < 0)
> return ret;
>
> - ret = tracecmd_msg_send(msg_handle->fd, &msg);
> + ret = tracecmd_msg_send(msg_handle, &msg);
> if (ret < 0)
> return ret;
>
> @@ -678,7 +743,7 @@ int tracecmd_msg_send_close_msg(struct
> tracecmd_msg_handle *msg_handle) struct tracecmd_msg msg;
>
> tracecmd_msg_init(MSG_CLOSE, &msg);
> - return tracecmd_msg_send(msg_handle->fd, &msg);
> + return tracecmd_msg_send(msg_handle, &msg);
> }
>
> int tracecmd_msg_send_close_resp_msg(struct tracecmd_msg_handle
> *msg_handle) @@ -686,14 +751,13 @@ int
> tracecmd_msg_send_close_resp_msg(struct tracecmd_msg_handle
> *msg_handle) struct tracecmd_msg msg;
> tracecmd_msg_init(MSG_CLOSE_RESP, &msg);
> - return tracecmd_msg_send(msg_handle->fd, &msg);
> + return tracecmd_msg_send(msg_handle, &msg);
> }
>
> int tracecmd_msg_data_send(struct tracecmd_msg_handle *msg_handle,
> const char *buf, int size)
> {
> struct tracecmd_msg msg;
> - int fd = msg_handle->fd;
> int n;
> int ret;
> int count = 0;
> @@ -721,7 +785,7 @@ int tracecmd_msg_data_send(struct
> tracecmd_msg_handle *msg_handle, memcpy(msg.buf, buf + count, n);
> n = 0;
> }
> - ret = msg_write(fd, &msg);
> + ret = msg_write(msg_handle, &msg);
> if (ret < 0)
> break;
> }
> @@ -735,8 +799,10 @@ int tracecmd_msg_finish_sending_data(struct
> tracecmd_msg_handle *msg_handle) struct tracecmd_msg msg;
> int ret;
>
> + flush_cache(msg_handle);
> + msg_handle->cache = false;
> tracecmd_msg_init(MSG_FIN_DATA, &msg);
> - ret = tracecmd_msg_send(msg_handle->fd, &msg);
> + ret = tracecmd_msg_send(msg_handle, &msg);
> if (ret < 0)
> return ret;
> return 0;
> @@ -752,10 +818,7 @@ int tracecmd_msg_read_data(struct
> tracecmd_msg_handle *msg_handle, int ofd) while
> (!tracecmd_msg_done(msg_handle)) { ret =
> tracecmd_msg_recv_wait(msg_handle->fd, &msg); if (ret < 0) {
> - if (ret == -ETIMEDOUT)
> - tracecmd_warning("Connection timed
> out\n");
> - else
> - tracecmd_warning("reading client");
> + tracecmd_warning("reading client %d (%s)",
> ret, strerror(ret)); return ret;
> }
>
> @@ -959,7 +1022,7 @@ int tracecmd_msg_send_trace_req(struct
> tracecmd_msg_handle *msg_handle, if (ret < 0)
> return ret;
>
> - return tracecmd_msg_send(msg_handle->fd, &msg);
> + return tracecmd_msg_send(msg_handle, &msg);
> }
>
> static int get_trace_req_protos(char *buf, int length,
> @@ -1151,7 +1214,7 @@ int tracecmd_msg_send_time_sync(struct
> tracecmd_msg_handle *msg_handle, msg.hdr.size =
> htonl(ntohl(msg.hdr.size) + payload_size);
> msg.buf = payload;
> - return msg_send_nofree(msg_handle->fd, &msg);
> + return msg_send_nofree(msg_handle, &msg);
> }
>
> /**
> @@ -1283,7 +1346,7 @@ int tracecmd_msg_send_trace_resp(struct
> tracecmd_msg_handle *msg_handle, if (ret < 0)
> return ret;
>
> - return tracecmd_msg_send(msg_handle->fd, &msg);
> + return tracecmd_msg_send(msg_handle, &msg);
> }
>
> int tracecmd_msg_recv_trace_resp(struct tracecmd_msg_handle
> *msg_handle,
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 08/87] trace-cmd library: Add support for compression algorithms
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (6 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 07/87] trace-cmd library: Add cache functionality to network message handler Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 21:02 ` Steven Rostedt
2021-07-29 5:08 ` [PATCH v2 09/87] trace-cmd list: Show supported " Tzvetomir Stoyanov (VMware)
` (78 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added infrastructure to trace-cmd library for compression. Introduced
various new APIs to work with this new functionality:
struct tracecmd_compression
tracecmd_compress_init()
tracecmd_compress_free()
tracecmd_compress_alloc()
tracecmd_compress_destroy()
tracecmd_compress_block()
tracecmd_uncompress_block()
tracecmd_compress_reset()
tracecmd_compress_read()
tracecmd_compress_pread()
tracecmd_compress_write()
tracecmd_compress_lseek()
tracecmd_compress_proto_get_name()
tracecmd_compress_is_supported()
tracecmd_compress_protos_get()
tracecmd_compress_proto_register()
tracecmd_compress_copy_from()
tracecmd_uncompress_copy_to()
tracecmd_uncompress_chunk()
tracecmd_load_chunks_info()
The compression algorithms are not part of this patch.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/Makefile | 1 +
.../include/private/trace-cmd-private.h | 39 +
lib/trace-cmd/include/trace-cmd-local.h | 3 +
lib/trace-cmd/trace-compress.c | 910 ++++++++++++++++++
lib/trace-cmd/trace-util.c | 4 +-
5 files changed, 955 insertions(+), 2 deletions(-)
create mode 100644 lib/trace-cmd/trace-compress.c
diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile
index 17600318..bab4322d 100644
--- a/lib/trace-cmd/Makefile
+++ b/lib/trace-cmd/Makefile
@@ -25,6 +25,7 @@ ifeq ($(VSOCK_DEFINED), 1)
OBJS += trace-timesync-ptp.o
OBJS += trace-timesync-kvm.o
endif
+OBJS += trace-compress.o
# Additional util objects
OBJS += trace-blk-hack.o
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 68715580..72729c57 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -468,6 +468,45 @@ void tracecmd_tsync_free(struct tracecmd_time_sync *tsync);
int tracecmd_write_guest_time_shift(struct tracecmd_output *handle,
struct tracecmd_time_sync *tsync);
+/* --- Compression --- */
+struct tracecmd_compress_chunk {
+ unsigned int size;
+ unsigned int zsize;
+ unsigned long long zoffset;
+ unsigned long long offset;
+};
+struct tracecmd_compression;
+struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version,
+ int fd, struct tep_handle *tep,
+ struct tracecmd_msg_handle *msg_handle);
+void tracecmd_compress_destroy(struct tracecmd_compression *handle);
+int tracecmd_compress_block(struct tracecmd_compression *handle);
+int tracecmd_uncompress_block(struct tracecmd_compression *handle);
+void tracecmd_compress_reset(struct tracecmd_compression *handle);
+int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len);
+int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset);
+int tracecmd_compress_write(struct tracecmd_compression *handle,
+ const void *data, unsigned long long size);
+off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence);
+int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress,
+ const char **name, const char **version);
+bool tracecmd_compress_is_supported(const char *name, const char *version);
+int tracecmd_compress_protos_get(char ***names, char ***versions);
+int tracecmd_compress_proto_register(const char *name, const char *version, int weight,
+ int (*compress)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes),
+ int (*uncompress)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes),
+ unsigned int (*comress_size)(unsigned int bytes),
+ bool (*is_supported)(const char *name, const char *version));
+int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size,
+ unsigned long long *read_size, unsigned long long *write_size);
+int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd,
+ unsigned long long *read_size, unsigned long long *write_size);
+int tracecmd_uncompress_chunk(struct tracecmd_compression *handle,
+ struct tracecmd_compress_chunk *chunk, char *data);
+int tracecmd_load_chunks_info(struct tracecmd_compression *handle,
+ struct tracecmd_compress_chunk **chunks_info);
/* --- Plugin handling --- */
extern struct tep_plugin_option trace_ftrace_options[];
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 7691cc05..2a622003 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -31,6 +31,9 @@ void tracecmd_info(const char *fmt, ...);
#endif
#endif
+void tracecmd_compress_init(void);
+void tracecmd_compress_free(void);
+
off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off_t offset, int whence);
#endif /* _TRACE_CMD_LOCAL_H */
diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c
new file mode 100644
index 00000000..5572acd6
--- /dev/null
+++ b/lib/trace-cmd/trace-compress.c
@@ -0,0 +1,910 @@
+// SPDX-License-Identifier: LGPL-2.1
+/*
+ * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com>
+ *
+ */
+#include <stdlib.h>
+#include <sys/time.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <unistd.h>
+
+#include "trace-cmd-private.h"
+#include "trace-cmd-local.h"
+
+struct compress_proto {
+ struct compress_proto *next;
+ char *proto_name;
+ char *proto_version;
+ int weight;
+
+ int (*compress_block)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes);
+ int (*uncompress_block)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes);
+ unsigned int (*compress_size)(unsigned int bytes);
+ bool (*is_supported)(const char *name, const char *version);
+};
+
+static struct compress_proto *proto_list;
+
+struct tracecmd_compression {
+ int fd;
+ unsigned int capacity;
+ unsigned long pointer;
+ char *buffer;
+ struct compress_proto *proto;
+ struct tep_handle *tep;
+ struct tracecmd_msg_handle *msg_handle;
+};
+
+static int read_fd(int fd, char *dst, int len)
+{
+ size_t size = 0;
+ int r;
+
+ do {
+ r = read(fd, dst+size, len);
+ if (r > 0) {
+ size += r;
+ len -= r;
+ } else
+ break;
+ } while (r > 0);
+
+ if (len)
+ return -1;
+ return size;
+}
+
+static long long write_fd(int fd, const void *data, size_t size)
+{
+ long long tot = 0;
+ long long w;
+
+ do {
+ w = write(fd, data + tot, size - tot);
+ tot += w;
+
+ if (!w)
+ break;
+ if (w < 0)
+ return w;
+ } while (tot != size);
+
+ return tot;
+}
+
+static long long do_write(struct tracecmd_compression *handle,
+ const void *data, unsigned long long size)
+{
+ int ret;
+
+ if (handle->msg_handle) {
+ ret = tracecmd_msg_data_send(handle->msg_handle, data, size);
+ if (ret)
+ return -1;
+ return size;
+ }
+ return write_fd(handle->fd, data, size);
+}
+
+/**
+ * tracecmd_compress_lseek - Move the read/write pointer into the compression buffer
+ * @handle: compression handler
+ * @offset: number of bytes to move the pointer, can be negative or positive
+ * @whence: the starting position of the pointer movement,
+ *
+ * Returns the new file pointer on success, or -1 in case of an error.
+ */
+off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence)
+{
+ unsigned long p, extend;
+ char *buf;
+
+ if (!handle || !handle->buffer)
+ return (off64_t)-1;
+
+ switch (whence) {
+ case SEEK_CUR:
+ p = handle->pointer + offset;
+ break;
+ case SEEK_END:
+ p = handle->capacity + offset;
+ break;
+ case SEEK_SET:
+ p = offset;
+ break;
+ default:
+ return (off64_t)-1;
+ }
+
+ if (p <= handle->capacity) {
+ handle->pointer = p;
+ } else {
+ extend = p - handle->capacity;
+ extend = extend < BUFSIZ ? BUFSIZ : extend;
+ buf = realloc(handle->buffer, handle->capacity + extend);
+ if (!buf)
+ return (off64_t)-1;
+ handle->buffer = buf;
+ handle->capacity += extend;
+ handle->pointer = p;
+ }
+
+ return p;
+}
+
+static int compress_read(struct tracecmd_compression *handle, char *dst, int len)
+{
+ int s;
+
+ if (handle->pointer + len > handle->capacity)
+ s = handle->capacity - handle->pointer;
+ else
+ s = len;
+ memcpy(dst, handle->buffer + handle->pointer, s);
+
+ return s;
+}
+
+/**
+ * tracecmd_compress_pread - pread() on compression buffer
+ * @handle: compression handler
+ * @dst: return, store the read data
+ * @len: length of data to be read
+ * @offset: offset in the buffer of data to be read
+ *
+ * Read a @len of data from the compression buffer at given @offset,
+ * without updating the buffer pointer.
+ *
+ * On success returns the number of bytes read, or -1 on failure.
+ */
+int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset)
+{
+ int ret;
+
+ if (!handle || !handle->buffer || offset > handle->capacity)
+ return -1;
+
+ ret = tracecmd_compress_lseek(handle, offset, SEEK_SET);
+ if (ret < 0)
+ return ret;
+ return compress_read(handle, dst, len);
+}
+
+/**
+ * tracecmd_compress_read - read() from compression buffer
+ * @handle: compression handler
+ * @dst: return, store the read data
+ * @len: length of data to be read
+ *
+ * Read a @len of data from the compression buffer
+ *
+ * On success returns the number of bytes read, or -1 on failure.
+ */
+int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len)
+{
+ int ret;
+
+ if (!handle || !handle->buffer)
+ return -1;
+
+ ret = compress_read(handle, dst, len);
+ if (ret > 0)
+ handle->pointer += ret;
+
+ return ret;
+}
+
+/**
+ * tracecmd_compress_reset - Reset the compression buffer
+ * @handle: compression handler
+ *
+ * Reset the compression buffer, any data currently in the buffer will be destroyed.
+ *
+ */
+void tracecmd_compress_reset(struct tracecmd_compression *handle)
+{
+ if (!handle)
+ return;
+
+ free(handle->buffer);
+ handle->buffer = NULL;
+ handle->pointer = 0;
+ handle->capacity = 0;
+}
+
+/**
+ * tracecmd_uncompress_block - uncompress a memory block
+ * @handle: compression handler
+ *
+ * Read compressed memory block from the file and uncompress it into internal buffer.
+ * The tracecmd_compress_read() can be used to read the uncompressed data from the buffer
+ *
+ * Returns 0 on success, or -1 in case of an error.
+ */
+int tracecmd_uncompress_block(struct tracecmd_compression *handle)
+{
+ unsigned int s_uncompressed;
+ unsigned int s_compressed;
+ char *bytes = NULL;
+ char buf[4];
+ int ret;
+
+ if (!handle || !handle->proto || !handle->proto->uncompress_block)
+ return -1;
+ tracecmd_compress_reset(handle);
+
+ if (read(handle->fd, buf, 4) != 4)
+ return -1;
+ s_compressed = tep_read_number(handle->tep, buf, 4);
+ if (read(handle->fd, buf, 4) != 4)
+ return -1;
+ s_uncompressed = tep_read_number(handle->tep, buf, 4);
+
+ handle->buffer = malloc(s_uncompressed);
+ if (!handle->buffer)
+ return -1;
+ bytes = malloc(s_compressed);
+ if (!bytes)
+ goto error;
+
+ if (read_fd(handle->fd, bytes, s_compressed) < 0)
+ goto error;
+ ret = handle->proto->uncompress_block(bytes, s_compressed,
+ handle->buffer, &s_uncompressed);
+ if (ret)
+ goto error;
+ free(bytes);
+ handle->pointer = 0;
+ handle->capacity = s_uncompressed;
+ return 0;
+error:
+ tracecmd_compress_reset(handle);
+ free(bytes);
+ return -1;
+}
+
+/**
+ * tracecmd_compress_block - compress a memory block
+ * @handle: compression handler
+ *
+ * Compress the content of the internal memory buffer and write the compressed data in the file
+ * The tracecmd_compress_write() can be used to write data into the internal memory buffer, before
+ * calling this API.
+ *
+ * Returns 0 on success, or -1 in case of an error.
+ */
+int tracecmd_compress_block(struct tracecmd_compression *handle)
+{
+ unsigned int size;
+ char *buf;
+ int endian4;
+ int ret;
+
+ if (!handle || !handle->proto ||
+ !handle->proto->compress_size || !handle->proto->compress_block)
+ return -1;
+
+ size = handle->proto->compress_size(handle->pointer);
+ buf = malloc(size);
+ if (!buf)
+ return -1;
+ ret = handle->proto->compress_block(handle->buffer, handle->pointer, buf, &size);
+ if (ret < 0)
+ goto out;
+ /* Write compressed data size */
+ endian4 = tep_read_number(handle->tep, &size, 4);
+ ret = do_write(handle, &endian4, 4);
+ if (ret != 4)
+ goto out;
+ /* Write uncompressed data size */
+ endian4 = tep_read_number(handle->tep, &handle->pointer, 4);
+ ret = do_write(handle, &endian4, 4);
+ if (ret != 4)
+ goto out;
+ /* Write compressed data */
+ ret = do_write(handle, buf, size);
+ ret = ((ret == size) ? 0 : -1);
+out:
+ tracecmd_compress_reset(handle);
+ free(buf);
+ return ret;
+}
+
+/**
+ * tracecmd_compress_write - write() to compression buffer
+ * @handle: compression handler
+ * @data: data to be written
+ * @size: size of @data
+ *
+ * Write @data of @size in the compression buffer
+ *
+ * Returns 0 on success, or -1 on failure.
+ */
+int tracecmd_compress_write(struct tracecmd_compression *handle,
+ const void *data, unsigned long long size)
+{
+ char *buf;
+ int extend;
+
+ if (!handle)
+ return -1;
+
+ if (handle->capacity < handle->pointer + size) {
+ extend = (handle->pointer + size) - handle->capacity;
+ extend = extend < BUFSIZ ? BUFSIZ : extend;
+ buf = realloc(handle->buffer, handle->capacity + extend);
+ if (!buf)
+ return -1;
+ handle->buffer = buf;
+ handle->capacity += extend;
+ }
+ memcpy(&handle->buffer[handle->pointer], data, size);
+ handle->pointer += size;
+ return 0;
+}
+
+/**
+ * tracecmd_compress_init - initialize the library with available compression algorithms
+ */
+void tracecmd_compress_init(void)
+{
+ struct timeval time;
+
+ gettimeofday(&time, NULL);
+ srand((time.tv_sec * 1000) + (time.tv_usec / 1000));
+
+#ifdef HAVE_ZLIB
+ tracecmd_zlib_init();
+#endif
+}
+
+static struct compress_proto *compress_proto_select(void)
+{
+ struct compress_proto *proto = proto_list;
+ struct compress_proto *selected = NULL;
+
+ while (proto) {
+ if (!selected || selected->weight > proto->weight)
+ selected = proto;
+ proto = proto->next;
+ }
+
+ return selected;
+}
+
+/**
+ * tracecmd_compress_alloc - Allocate a new compression context
+ * @name: name of the compression algorithm, if NULL - auto select the best available algorithm
+ * @version: version of the compression algorithm, can be NULL
+ * @fd: file descriptor for reading / writing data
+ * @tep: tep handler, used to encode the data
+ * @msg_handle: message handler, use it for reading / writing data instead of @fd
+ *
+ * Returns NULL on failure or pointer to allocated compression context.
+ * The returned context must be freed by tracecmd_compress_destroy()
+ */
+struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version,
+ int fd, struct tep_handle *tep,
+ struct tracecmd_msg_handle *msg_handle)
+{
+ struct tracecmd_compression *new;
+ struct compress_proto *proto;
+
+ if (name) {
+ proto = proto_list;
+ while (proto) {
+ if (proto->is_supported && proto->is_supported(name, version))
+ break;
+ proto = proto->next;
+ }
+ } else {
+ proto = compress_proto_select();
+ }
+ if (!proto)
+ return NULL;
+
+ new = calloc(1, sizeof(*new));
+ if (!new)
+ return NULL;
+ new->fd = fd;
+ new->tep = tep;
+ new->msg_handle = msg_handle;
+ new->proto = proto;
+ return new;
+}
+
+/**
+ * tracecmd_compress_destroy - Free a compression context
+ * @handle: handle to the compression context that will be freed
+ */
+void tracecmd_compress_destroy(struct tracecmd_compression *handle)
+{
+ tracecmd_compress_reset(handle);
+ free(handle);
+}
+
+/**
+ * tracecmd_compress_is_supported - check if compression algorithm with given name and
+ * version is supported
+ * @name: name of the compression algorithm.
+ * @version: version of the compression algorithm.
+ *
+ * Returns true if the algorithm with given name and version is supported or false if it is not.
+ */
+bool tracecmd_compress_is_supported(const char *name, const char *version)
+{
+ struct compress_proto *proto = proto_list;
+
+ if (!name)
+ return NULL;
+
+ while (proto) {
+ if (proto->is_supported && proto->is_supported(name, version))
+ return true;
+ proto = proto->next;
+ }
+ return false;
+}
+
+/**
+ * tracecmd_compress_proto_get_name - get name and version of compression algorithm
+ * @compress: compression handler.
+ * @name: return, name of the compression algorithm.
+ * @version: return, version of the compression algorithm.
+ *
+ * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the
+ * algorithm are stored in @name and @version. The returned strings must *not* be freed.
+ */
+int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress,
+ const char **name, const char **version)
+{
+ if (!compress || !compress->proto)
+ return -1;
+ if (name)
+ *name = compress->proto->proto_name;
+ if (version)
+ *version = compress->proto->proto_version;
+ return 0;
+}
+
+/**
+ * tracecmd_compress_proto_register - register a new compression algorithm
+ * @name: name of the compression algorithm.
+ * @version: version of the compression algorithm.
+ * @weight: weight of the compression algorithm, lower is better.
+ * @compress: compression hook, called to compress a memory block.
+ * @uncompress: uncompression hook, called to uncompress a memory block.
+ * @compress_size: hook, called to get the required minimum size of the buffer for compression
+ * given number of bytes.
+ * @is_supported: check hook, called to check if compression with given name and version is
+ * supported by this plugin.
+ *
+ * Returns 0 on success, or -1 in case of an error. If algorithm with given name and version is
+ * already registered, -1 is returned.
+ */
+int tracecmd_compress_proto_register(const char *name, const char *version, int weight,
+ int (*compress)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes),
+ int (*uncompress)(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes),
+ unsigned int (*compress_size)(unsigned int bytes),
+ bool (*is_supported)(const char *name, const char *version))
+{
+ struct compress_proto *new;
+
+ if (!name || !compress || !uncompress)
+ return -1;
+ if (tracecmd_compress_is_supported(name, version))
+ return -1;
+
+ new = calloc(1, sizeof(*new));
+ if (!new)
+ return -1;
+
+ new->proto_name = strdup(name);
+ if (!new->proto_name)
+ goto error;
+ new->proto_version = strdup(version);
+ if (!new->proto_version)
+ goto error;
+ new->compress_block = compress;
+ new->uncompress_block = uncompress;
+ new->compress_size = compress_size;
+ new->is_supported = is_supported;
+ new->weight = weight;
+ new->next = proto_list;
+ proto_list = new;
+ return 0;
+
+error:
+ free(new->proto_name);
+ free(new->proto_version);
+ free(new);
+ return -1;
+}
+
+/**
+ * tracecmd_compress_free - free the library resources, related to available compression algorithms
+ *
+ */
+void tracecmd_compress_free(void)
+{
+ struct compress_proto *proto = proto_list;
+ struct compress_proto *del;
+
+ while (proto) {
+ del = proto;
+ proto = proto->next;
+ free(del->proto_name);
+ free(del->proto_version);
+ free(del);
+ }
+ proto_list = NULL;
+}
+
+/**
+ * tracecmd_compress_protos_get - get a list of all supported compression algorithms and versions
+ * @names: return, array with names of all supported compression algorithms
+ * @versions: return, array with versions of all supported compression algorithms
+ *
+ * On success, the size of @names and @versions arrays is returned. Those arrays are allocated by
+ * the API and must be freed with free() by the caller. Both arrays are with same size, each name
+ * from @names corresponds to a version from @versions.
+ * On error -1 is returned and @names and @versions arrays are not allocated.
+ */
+int tracecmd_compress_protos_get(char ***names, char ***versions)
+{
+ struct compress_proto *proto = proto_list;
+ char **n = NULL;
+ char **v = NULL;
+ int c, i;
+
+ for (c = 0; proto; proto = proto->next)
+ c++;
+
+ if (c < 1)
+ return c;
+
+ n = calloc(c, sizeof(char *));
+ if (!n)
+ goto error;
+ v = calloc(c, sizeof(char *));
+ if (!v)
+ goto error;
+
+ proto = proto_list;
+ for (i = 0; i < c && proto; i++) {
+ n[i] = proto->proto_name;
+ v[i] = proto->proto_version;
+ proto = proto->next;
+ }
+
+ *names = n;
+ *versions = v;
+ return c;
+
+error:
+ free(n);
+ free(v);
+ return -1;
+}
+
+/**
+ * tracecmd_compress_copy_from - Copy and compress data from a file
+ * @handle: compression handler
+ * @fd: file descriptor to uncompressed data to copy from
+ * @chunk_size: size of one compression chunk
+ * @read_size: in - max bytes to read from @fd, 0 to read till the EOF
+ * out - size of the uncompressed data read from @fd
+ * @write_size: return, size of the compressed data written into @handle
+ *
+ * This function reads uncompressed data from given @fd, compresses the data using the @handle
+ * compression context and writes the compressed data into the fd associated with the @handle.
+ * The data is compressed on chunks with given @chunk_size size.
+ * The compressed data is written in the format:
+ * - 4 bytes, chunks count
+ * - for each chunk:
+ * - 4 bytes, size of compressed data in this chunk
+ * - 4 bytes, uncompressed size of the data in this chunk
+ * - data, bytes of <size of compressed data in this chunk>
+ *
+ * On success 0 is returned, @read_size and @write_size are updated with the size of
+ * read and written data.
+ */
+int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size,
+ unsigned long long *read_size, unsigned long long *write_size)
+{
+ unsigned int rchunk = 0;
+ unsigned int chunks = 0;
+ unsigned int wsize = 0;
+ unsigned int rsize = 0;
+ unsigned int rmax = 0;
+ unsigned int csize;
+ unsigned int size;
+ unsigned int all;
+ unsigned int r;
+ off64_t offset;
+ char *buf_from;
+ char *buf_to;
+ int endian4;
+ int ret;
+
+ if (!handle || !handle->proto ||
+ !handle->proto->compress_block || !handle->proto->compress_size)
+ return 0;
+ if (read_size)
+ rmax = *read_size;
+ csize = handle->proto->compress_size(chunk_size);
+ buf_from = malloc(chunk_size);
+ if (!buf_from)
+ return -1;
+ buf_to = malloc(csize);
+ if (!buf_to)
+ return -1;
+ /* save the initial offset and write 0 chunks */
+ offset = lseek64(handle->fd, 0, SEEK_CUR);
+ write_fd(handle->fd, &chunks, 4);
+
+ do {
+ all = 0;
+ if (rmax > 0 && (rmax - rsize) < chunk_size)
+ rchunk = (rmax - rsize);
+ else
+ rchunk = chunk_size;
+
+ do {
+ r = read(fd, buf_from + all, rchunk - all);
+ all += r;
+
+ if (r <= 0)
+ break;
+ } while (all != rchunk);
+
+
+ if (r < 0 || (rmax > 0 && rsize >= rmax))
+ break;
+ rsize += all;
+ size = csize;
+ if (all > 0) {
+ ret = handle->proto->compress_block(buf_from, all, buf_to, &size);
+ if (ret < 0) {
+ if (errno == EINTR)
+ continue;
+ break;
+ }
+ /* Write compressed data size */
+ endian4 = tep_read_number(handle->tep, &size, 4);
+ ret = write_fd(handle->fd, &endian4, 4);
+ if (ret != 4)
+ break;
+ /* Write uncompressed data size */
+ endian4 = tep_read_number(handle->tep, &all, 4);
+ ret = write_fd(handle->fd, &endian4, 4);
+ if (ret != 4)
+ break;
+ /* Write the compressed data */
+ ret = write_fd(handle->fd, buf_to, size);
+ if (ret != size)
+ break;
+ /* data + compress header */
+ wsize += (size + 8);
+ chunks++;
+ }
+ } while (all > 0);
+ free(buf_from);
+ free(buf_to);
+ if (all)
+ return -1;
+ if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1)
+ return -1;
+ endian4 = tep_read_number(handle->tep, &chunks, 4);
+ /* write chunks count*/
+ write_fd(handle->fd, &chunks, 4);
+ lseek64(handle->fd, offset, SEEK_SET);
+ if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1)
+ return -1;
+ if (read_size)
+ *read_size = rsize;
+ if (write_size)
+ *write_size = wsize;
+ return 0;
+}
+
+/**
+ * tracecmd_load_chunks_info - Read compression chunks information from the file
+ * @handle: compression handler
+ * @chunks_info: return, array with compression chunks information
+ *
+ * This function reads information of all compression chunks in the current compression block from
+ * the file and fills that information in a newly allocated array @chunks_info which is returned.
+ *
+ * On success count of compression chunks is returned. Array of that count is allocated and
+ * returned in @chunks_info. Each entry describes one compression chunk. On error -1 is returned.
+ * In case of success, @chunks_info must be freed by free().
+ */
+int tracecmd_load_chunks_info(struct tracecmd_compression *handle,
+ struct tracecmd_compress_chunk **chunks_info)
+{
+ struct tracecmd_compress_chunk *chunks = NULL;
+ unsigned long long size = 0;
+ unsigned int count = 0;
+ off64_t offset;
+ int ret = -1;
+ char buf[4];
+ int i;
+
+ if (!handle)
+ return -1;
+
+ offset = lseek64(handle->fd, 0, SEEK_CUR);
+ if (offset == (off64_t)-1)
+ return -1;
+
+ if (read(handle->fd, buf, 4) != 4)
+ return -1;
+ count = tep_read_number(handle->tep, buf, 4);
+ if (!count) {
+ ret = 0;
+ goto out;
+ }
+ chunks = calloc(count, sizeof(struct tracecmd_compress_chunk));
+ if (!chunks)
+ goto out;
+ for (i = 0; i < count; i++) {
+ chunks[i].zoffset = lseek64(handle->fd, 0, SEEK_CUR);
+ if (chunks[i].zoffset == (off_t)-1)
+ goto out;
+ if (read(handle->fd, buf, 4) != 4)
+ goto out;
+ chunks[i].zsize = tep_read_number(handle->tep, buf, 4);
+ chunks[i].offset = size;
+ if (read(handle->fd, buf, 4) != 4)
+ goto out;
+ chunks[i].size = tep_read_number(handle->tep, buf, 4);
+ size += chunks[i].size;
+ if (lseek64(handle->fd, chunks[i].zsize, SEEK_CUR) == (off64_t)-1)
+ goto out;
+ }
+
+ ret = count;
+out:
+ if (lseek64(handle->fd, offset, SEEK_SET) == (off64_t)-1)
+ ret = -1;
+
+ if (ret > 0 && chunks_info)
+ *chunks_info = chunks;
+ else
+ free(chunks);
+
+ return ret;
+}
+
+/**
+ * tracecmd_uncompress_chunk - Uncompress given compression chunk.
+ * @handle: compression handler
+ * @chunk: chunk, that will be uncompressed in @data
+ * @data: Preallocated memory for uncompressed data. Must have enough space to hold
+ * the uncompressed data
+ *
+ * This function uncompresses the chunk described by @chunk and stores the uncompressed data in
+ * the preallocated memory @data.
+ *
+ * On success 0 is returned and the uncompressed data is stored in @data. On error -1 is returned.
+ */
+int tracecmd_uncompress_chunk(struct tracecmd_compression *handle,
+ struct tracecmd_compress_chunk *chunk, char *data)
+{
+ char *bytes_in = NULL;
+ unsigned int size;
+ int ret = -1;
+
+ if (!handle || !handle->proto || !handle->proto->uncompress_block || !chunk || !data)
+ return -1;
+
+ if (lseek64(handle->fd, chunk->zoffset + 8, SEEK_SET) == (off_t)-1)
+ return -1;
+ bytes_in = malloc(chunk->zsize);
+ if (!bytes_in)
+ return -1;
+ if (read_fd(handle->fd, bytes_in, chunk->zsize) < 0)
+ goto out;
+ size = chunk->size;
+ if (handle->proto->uncompress_block(bytes_in, chunk->zsize, data, &size))
+ goto out;
+ ret = 0;
+out:
+ free(bytes_in);
+ return ret;
+}
+
+/**
+ * tracecmd_uncompress_copy_to - Uncompress data and copy to a file
+ * @handle: compression handler
+ * @fd: file descriptor to uncompressed data to copy into
+ * @read_size: return, size of the compressed data read from @handle
+ * @write_size: return, size of the uncompressed data written into @fd
+ *
+ * This function reads compressed data from the fd, associated with @handle, uncompresses it
+ * using the @handle compression context and writes the uncompressed data into the fd.
+ * The compressed data must be in the format:
+ * - 4 bytes, chunks count
+ * - for each chunk:
+ * - 4 bytes, size of compressed data in this chunk
+ * - 4 bytes, uncompressed size of the data in this chunk
+ * - data, bytes of <size of compressed data in this chunk>
+ *
+ * On success 0 is returned, @read_size and @write_size are updated with the size of
+ * read and written data.
+ */
+int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd,
+ unsigned long long *read_size, unsigned long long *write_size)
+{
+ unsigned int s_uncompressed;
+ unsigned int s_compressed;
+ unsigned int rsize = 0;
+ unsigned int wsize = 0;
+ char *bytes_out = NULL;
+ char *bytes_in = NULL;
+ int size_out = 0;
+ int size_in = 0;
+ int chunks;
+ char buf[4];
+ char *tmp;
+ int ret;
+
+ if (!handle || !handle->proto || !handle->proto->uncompress_block)
+ return -1;
+
+ if (read(handle->fd, buf, 4) != 4)
+ return -1;
+ chunks = tep_read_number(handle->tep, buf, 4);
+ rsize += 4;
+ while (chunks) {
+ if (read(handle->fd, buf, 4) != 4)
+ break;
+ s_compressed = tep_read_number(handle->tep, buf, 4);
+ rsize += 4;
+ if (read(handle->fd, buf, 4) != 4)
+ break;
+ s_uncompressed = tep_read_number(handle->tep, buf, 4);
+ rsize += 4;
+ if (!bytes_in || size_in < s_compressed) {
+ tmp = realloc(bytes_in, s_compressed);
+ if (!tmp)
+ break;
+ bytes_in = tmp;
+ size_in = s_compressed;
+ }
+
+ if (!bytes_out || size_out < s_uncompressed) {
+ tmp = realloc(bytes_out, s_uncompressed);
+ if (!tmp)
+ break;
+ bytes_out = tmp;
+ size_out = s_uncompressed;
+ }
+
+ if (read_fd(handle->fd, bytes_in, s_compressed) < 0)
+ break;
+ rsize += s_compressed;
+ ret = handle->proto->uncompress_block(bytes_in, s_compressed,
+ bytes_out, &s_uncompressed);
+ if (ret)
+ break;
+ write_fd(fd, bytes_out, s_uncompressed);
+ wsize += s_uncompressed;
+ chunks--;
+ }
+ free(bytes_in);
+ free(bytes_out);
+ if (chunks)
+ return -1;
+ if (read_size)
+ *read_size = rsize;
+ if (write_size)
+ *write_size = wsize;
+ return 0;
+}
diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c
index 0f49a21a..4e1b8427 100644
--- a/lib/trace-cmd/trace-util.c
+++ b/lib/trace-cmd/trace-util.c
@@ -627,10 +627,10 @@ bool tracecmd_is_version_supported(unsigned int version)
static void __attribute__ ((constructor)) tracecmd_lib_init(void)
{
-
+ tracecmd_compress_init();
}
static void __attribute__((destructor)) tracecmd_lib_free(void)
{
-
+ tracecmd_compress_free();
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 08/87] trace-cmd library: Add support for compression algorithms
2021-07-29 5:08 ` [PATCH v2 08/87] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
@ 2021-07-29 21:02 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-07-29 21:02 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:08:40 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Added infrastructure to trace-cmd library for compression. Introduced
> various new APIs to work with this new functionality:
> struct tracecmd_compression
> tracecmd_compress_init()
> tracecmd_compress_free()
> tracecmd_compress_alloc()
> tracecmd_compress_destroy()
> tracecmd_compress_block()
> tracecmd_uncompress_block()
> tracecmd_compress_reset()
> tracecmd_compress_read()
> tracecmd_compress_pread()
> tracecmd_compress_write()
> tracecmd_compress_lseek()
> tracecmd_compress_proto_get_name()
> tracecmd_compress_is_supported()
> tracecmd_compress_protos_get()
> tracecmd_compress_proto_register()
> tracecmd_compress_copy_from()
> tracecmd_uncompress_copy_to()
> tracecmd_uncompress_chunk()
> tracecmd_load_chunks_info()
Patch 6 should be folded into this patch.
>
> The compression algorithms are not part of this patch.
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/Makefile | 1 +
> .../include/private/trace-cmd-private.h | 39 +
> lib/trace-cmd/include/trace-cmd-local.h | 3 +
> lib/trace-cmd/trace-compress.c | 910 ++++++++++++++++++
> lib/trace-cmd/trace-util.c | 4 +-
> 5 files changed, 955 insertions(+), 2 deletions(-)
> create mode 100644 lib/trace-cmd/trace-compress.c
>
> diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile
> index 17600318..bab4322d 100644
> --- a/lib/trace-cmd/Makefile
> +++ b/lib/trace-cmd/Makefile
> @@ -25,6 +25,7 @@ ifeq ($(VSOCK_DEFINED), 1)
> OBJS += trace-timesync-ptp.o
> OBJS += trace-timesync-kvm.o
> endif
> +OBJS += trace-compress.o
>
> # Additional util objects
> OBJS += trace-blk-hack.o
> diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
> index 68715580..72729c57 100644
> --- a/lib/trace-cmd/include/private/trace-cmd-private.h
> +++ b/lib/trace-cmd/include/private/trace-cmd-private.h
> @@ -468,6 +468,45 @@ void tracecmd_tsync_free(struct tracecmd_time_sync *tsync);
> int tracecmd_write_guest_time_shift(struct tracecmd_output *handle,
> struct tracecmd_time_sync *tsync);
>
> +/* --- Compression --- */
> +struct tracecmd_compress_chunk {
> + unsigned int size;
> + unsigned int zsize;
> + unsigned long long zoffset;
> + unsigned long long offset;
I wonder if the above type should be "off64_t" as that is the official
type returned by lseek64().
> +};
> +struct tracecmd_compression;
> +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version,
> + int fd, struct tep_handle *tep,
> + struct tracecmd_msg_handle *msg_handle);
> +void tracecmd_compress_destroy(struct tracecmd_compression *handle);
> +int tracecmd_compress_block(struct tracecmd_compression *handle);
> +int tracecmd_uncompress_block(struct tracecmd_compression *handle);
> +void tracecmd_compress_reset(struct tracecmd_compression *handle);
> +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len);
> +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset);
> +int tracecmd_compress_write(struct tracecmd_compression *handle,
> + const void *data, unsigned long long size);
> +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence);
> +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress,
> + const char **name, const char **version);
> +bool tracecmd_compress_is_supported(const char *name, const char *version);
> +int tracecmd_compress_protos_get(char ***names, char ***versions);
> +int tracecmd_compress_proto_register(const char *name, const char *version, int weight,
> + int (*compress)(char *in, unsigned int in_bytes,
> + char *out, unsigned int *out_bytes),
> + int (*uncompress)(char *in, unsigned int in_bytes,
> + char *out, unsigned int *out_bytes),
> + unsigned int (*comress_size)(unsigned int bytes),
> + bool (*is_supported)(const char *name, const char *version));
> +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size,
> + unsigned long long *read_size, unsigned long long *write_size);
> +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd,
> + unsigned long long *read_size, unsigned long long *write_size);
> +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle,
> + struct tracecmd_compress_chunk *chunk, char *data);
> +int tracecmd_load_chunks_info(struct tracecmd_compression *handle,
> + struct tracecmd_compress_chunk **chunks_info);
> /* --- Plugin handling --- */
> extern struct tep_plugin_option trace_ftrace_options[];
>
> diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
> index 7691cc05..2a622003 100644
> --- a/lib/trace-cmd/include/trace-cmd-local.h
> +++ b/lib/trace-cmd/include/trace-cmd-local.h
> @@ -31,6 +31,9 @@ void tracecmd_info(const char *fmt, ...);
> #endif
> #endif
>
> +void tracecmd_compress_init(void);
> +void tracecmd_compress_free(void);
> +
> off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off_t offset, int whence);
>
> #endif /* _TRACE_CMD_LOCAL_H */
> diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c
> new file mode 100644
> index 00000000..5572acd6
> --- /dev/null
> +++ b/lib/trace-cmd/trace-compress.c
> @@ -0,0 +1,910 @@
> +// SPDX-License-Identifier: LGPL-2.1
> +/*
> + * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com>
> + *
> + */
> +#include <stdlib.h>
> +#include <sys/time.h>
> +#include <fcntl.h>
> +#include <errno.h>
> +#include <unistd.h>
> +
> +#include "trace-cmd-private.h"
> +#include "trace-cmd-local.h"
> +
> +struct compress_proto {
> + struct compress_proto *next;
> + char *proto_name;
> + char *proto_version;
> + int weight;
> +
> + int (*compress_block)(char *in, unsigned int in_bytes,
> + char *out, unsigned int *out_bytes);
> + int (*uncompress_block)(char *in, unsigned int in_bytes,
> + char *out, unsigned int *out_bytes);
Are both char * expected to be modified? That is, can we change
"char *in" to "const char *in"?
> + unsigned int (*compress_size)(unsigned int bytes);
> + bool (*is_supported)(const char *name, const char *version);
> +};
> +
> +static struct compress_proto *proto_list;
> +
> +struct tracecmd_compression {
> + int fd;
> + unsigned int capacity;
> + unsigned long pointer;
> + char *buffer;
> + struct compress_proto *proto;
> + struct tep_handle *tep;
> + struct tracecmd_msg_handle *msg_handle;
> +};
> +
> +static int read_fd(int fd, char *dst, int len)
> +{
> + size_t size = 0;
> + int r;
> +
> + do {
> + r = read(fd, dst+size, len);
> + if (r > 0) {
> + size += r;
> + len -= r;
> + } else
> + break;
> + } while (r > 0);
> +
> + if (len)
> + return -1;
> + return size;
> +}
> +
> +static long long write_fd(int fd, const void *data, size_t size)
> +{
> + long long tot = 0;
> + long long w;
> +
> + do {
> + w = write(fd, data + tot, size - tot);
> + tot += w;
> +
> + if (!w)
> + break;
> + if (w < 0)
> + return w;
> + } while (tot != size);
> +
> + return tot;
> +}
> +
> +static long long do_write(struct tracecmd_compression *handle,
> + const void *data, unsigned long long size)
> +{
> + int ret;
> +
> + if (handle->msg_handle) {
> + ret = tracecmd_msg_data_send(handle->msg_handle, data, size);
> + if (ret)
> + return -1;
> + return size;
> + }
> + return write_fd(handle->fd, data, size);
> +}
> +
> +/**
> + * tracecmd_compress_lseek - Move the read/write pointer into the compression buffer
> + * @handle: compression handler
> + * @offset: number of bytes to move the pointer, can be negative or positive
> + * @whence: the starting position of the pointer movement,
> + *
> + * Returns the new file pointer on success, or -1 in case of an error.
> + */
> +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence)
> +{
> + unsigned long p, extend;
> + char *buf;
> +
> + if (!handle || !handle->buffer)
> + return (off64_t)-1;
> +
> + switch (whence) {
> + case SEEK_CUR:
> + p = handle->pointer + offset;
> + break;
> + case SEEK_END:
> + p = handle->capacity + offset;
> + break;
> + case SEEK_SET:
> + p = offset;
> + break;
> + default:
> + return (off64_t)-1;
> + }
> +
> + if (p <= handle->capacity) {
> + handle->pointer = p;
> + } else {
> + extend = p - handle->capacity;
> + extend = extend < BUFSIZ ? BUFSIZ : extend;
I wonder if we should always keep the buffer a multiple of BUFSIZ. That
is, if extend is just one byte bigger than BUFSIZ, then we are going to
immediately extend again at the next write.
Instead, have:
extend = ((extend / BUFSIZ + 1) * BUFSIZ;
The above '/' is a integer divide, that rounds down.
(extend / BUFSIZ) will return the number of BUFSIZ that fit in
extend. Then we add 1 (as we want one more than is currently needed.
Multiply that number by BUFSIZ, and now we have a size that is
a multiple of BUFSIZ that can fully fit the extend.
This should help reduce the number of realloc()s that are required.
> + buf = realloc(handle->buffer, handle->capacity + extend);
> + if (!buf)
> + return (off64_t)-1;
> + handle->buffer = buf;
> + handle->capacity += extend;
> + handle->pointer = p;
> + }
> +
> + return p;
> +}
> +
> +static int compress_read(struct tracecmd_compression *handle, char *dst, int len)
> +{
> + int s;
> +
> + if (handle->pointer + len > handle->capacity)
> + s = handle->capacity - handle->pointer;
> + else
> + s = len;
> + memcpy(dst, handle->buffer + handle->pointer, s);
> +
> + return s;
> +}
> +
> +/**
> + * tracecmd_compress_pread - pread() on compression buffer
> + * @handle: compression handler
> + * @dst: return, store the read data
> + * @len: length of data to be read
> + * @offset: offset in the buffer of data to be read
> + *
> + * Read a @len of data from the compression buffer at given @offset,
> + * without updating the buffer pointer.
> + *
> + * On success returns the number of bytes read, or -1 on failure.
> + */
> +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset)
> +{
> + int ret;
> +
> + if (!handle || !handle->buffer || offset > handle->capacity)
> + return -1;
> +
> + ret = tracecmd_compress_lseek(handle, offset, SEEK_SET);
> + if (ret < 0)
> + return ret;
> + return compress_read(handle, dst, len);
> +}
> +
> +/**
> + * tracecmd_compress_read - read() from compression buffer
> + * @handle: compression handler
> + * @dst: return, store the read data
> + * @len: length of data to be read
> + *
> + * Read a @len of data from the compression buffer
> + *
> + * On success returns the number of bytes read, or -1 on failure.
> + */
> +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len)
> +{
> + int ret;
> +
> + if (!handle || !handle->buffer)
> + return -1;
> +
> + ret = compress_read(handle, dst, len);
> + if (ret > 0)
> + handle->pointer += ret;
> +
> + return ret;
> +}
> +
> +/**
> + * tracecmd_compress_reset - Reset the compression buffer
> + * @handle: compression handler
> + *
> + * Reset the compression buffer, any data currently in the buffer will be destroyed.
> + *
> + */
> +void tracecmd_compress_reset(struct tracecmd_compression *handle)
> +{
> + if (!handle)
> + return;
> +
> + free(handle->buffer);
> + handle->buffer = NULL;
> + handle->pointer = 0;
> + handle->capacity = 0;
> +}
> +
> +/**
> + * tracecmd_uncompress_block - uncompress a memory block
> + * @handle: compression handler
> + *
> + * Read compressed memory block from the file and uncompress it into internal buffer.
> + * The tracecmd_compress_read() can be used to read the uncompressed data from the buffer
> + *
> + * Returns 0 on success, or -1 in case of an error.
> + */
> +int tracecmd_uncompress_block(struct tracecmd_compression *handle)
> +{
> + unsigned int s_uncompressed;
> + unsigned int s_compressed;
> + char *bytes = NULL;
> + char buf[4];
> + int ret;
> +
> + if (!handle || !handle->proto || !handle->proto->uncompress_block)
> + return -1;
> + tracecmd_compress_reset(handle);
> +
> + if (read(handle->fd, buf, 4) != 4)
> + return -1;
> + s_compressed = tep_read_number(handle->tep, buf, 4);
> + if (read(handle->fd, buf, 4) != 4)
> + return -1;
> + s_uncompressed = tep_read_number(handle->tep, buf, 4);
> +
> + handle->buffer = malloc(s_uncompressed);
> + if (!handle->buffer)
> + return -1;
> + bytes = malloc(s_compressed);
> + if (!bytes)
> + goto error;
> +
> + if (read_fd(handle->fd, bytes, s_compressed) < 0)
> + goto error;
> + ret = handle->proto->uncompress_block(bytes, s_compressed,
> + handle->buffer, &s_uncompressed);
> + if (ret)
> + goto error;
> + free(bytes);
> + handle->pointer = 0;
> + handle->capacity = s_uncompressed;
> + return 0;
> +error:
> + tracecmd_compress_reset(handle);
> + free(bytes);
> + return -1;
> +}
> +
> +/**
> + * tracecmd_compress_block - compress a memory block
> + * @handle: compression handler
> + *
> + * Compress the content of the internal memory buffer and write the compressed data in the file
> + * The tracecmd_compress_write() can be used to write data into the internal memory buffer, before
> + * calling this API.
> + *
> + * Returns 0 on success, or -1 in case of an error.
> + */
> +int tracecmd_compress_block(struct tracecmd_compression *handle)
> +{
> + unsigned int size;
> + char *buf;
> + int endian4;
> + int ret;
> +
> + if (!handle || !handle->proto ||
> + !handle->proto->compress_size || !handle->proto->compress_block)
> + return -1;
> +
> + size = handle->proto->compress_size(handle->pointer);
> + buf = malloc(size);
> + if (!buf)
> + return -1;
> + ret = handle->proto->compress_block(handle->buffer, handle->pointer, buf, &size);
> + if (ret < 0)
> + goto out;
> + /* Write compressed data size */
> + endian4 = tep_read_number(handle->tep, &size, 4);
> + ret = do_write(handle, &endian4, 4);
> + if (ret != 4)
> + goto out;
> + /* Write uncompressed data size */
> + endian4 = tep_read_number(handle->tep, &handle->pointer, 4);
> + ret = do_write(handle, &endian4, 4);
> + if (ret != 4)
> + goto out;
> + /* Write compressed data */
> + ret = do_write(handle, buf, size);
> + ret = ((ret == size) ? 0 : -1);
> +out:
> + tracecmd_compress_reset(handle);
> + free(buf);
> + return ret;
> +}
> +
> +/**
> + * tracecmd_compress_write - write() to compression buffer
> + * @handle: compression handler
> + * @data: data to be written
> + * @size: size of @data
> + *
> + * Write @data of @size in the compression buffer
> + *
> + * Returns 0 on success, or -1 on failure.
> + */
> +int tracecmd_compress_write(struct tracecmd_compression *handle,
> + const void *data, unsigned long long size)
> +{
> + char *buf;
> + int extend;
> +
> + if (!handle)
> + return -1;
> +
> + if (handle->capacity < handle->pointer + size) {
> + extend = (handle->pointer + size) - handle->capacity;
> + extend = extend < BUFSIZ ? BUFSIZ : extend;
> + buf = realloc(handle->buffer, handle->capacity + extend);
> + if (!buf)
> + return -1;
Ug, you have the same allocation logic again. Please put this into a
helper function. And then, if you make a change into how it is updated,
you only need to do it in one place.
> + handle->buffer = buf;
> + handle->capacity += extend;
> + }
> + memcpy(&handle->buffer[handle->pointer], data, size);
> + handle->pointer += size;
> + return 0;
> +}
> +
> +/**
> + * tracecmd_compress_init - initialize the library with available compression algorithms
> + */
> +void tracecmd_compress_init(void)
> +{
> + struct timeval time;
> +
> + gettimeofday(&time, NULL);
> + srand((time.tv_sec * 1000) + (time.tv_usec / 1000));
> +
> +#ifdef HAVE_ZLIB
> + tracecmd_zlib_init();
> +#endif
The above looks like it belongs in the introduction of the zlib
compression patch.
> +}
> +
> +static struct compress_proto *compress_proto_select(void)
> +{
> + struct compress_proto *proto = proto_list;
> + struct compress_proto *selected = NULL;
> +
> + while (proto) {
> + if (!selected || selected->weight > proto->weight)
> + selected = proto;
> + proto = proto->next;
> + }
> +
> + return selected;
> +}
> +
> diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c
> index 0f49a21a..4e1b8427 100644
> --- a/lib/trace-cmd/trace-util.c
> +++ b/lib/trace-cmd/trace-util.c
> @@ -627,10 +627,10 @@ bool tracecmd_is_version_supported(unsigned int version)
>
> static void __attribute__ ((constructor)) tracecmd_lib_init(void)
> {
> -
> + tracecmd_compress_init();
> }
>
> static void __attribute__((destructor)) tracecmd_lib_free(void)
> {
> -
> + tracecmd_compress_free();
> }
As I said earlier. This patch should be the one that introduces these
handlers.
-- Steve
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 09/87] trace-cmd list: Show supported compression algorithms
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (7 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 08/87] trace-cmd library: Add support for compression algorithms Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 10/87] trace-cmd library: Internal helpers for compressing data Tzvetomir Stoyanov (VMware)
` (77 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Add new parameter "trace-cmd list -c" to show supported compression
algorithms.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
Documentation/trace-cmd/trace-cmd-list.1.txt | 3 +++
tracecmd/trace-list.c | 26 ++++++++++++++++++++
tracecmd/trace-usage.c | 1 +
3 files changed, 30 insertions(+)
diff --git a/Documentation/trace-cmd/trace-cmd-list.1.txt b/Documentation/trace-cmd/trace-cmd-list.1.txt
index a5c6b16c..b77e3460 100644
--- a/Documentation/trace-cmd/trace-cmd-list.1.txt
+++ b/Documentation/trace-cmd/trace-cmd-list.1.txt
@@ -71,6 +71,9 @@ OPTIONS
List defined clocks that can be used with trace-cmd record -C.
The one in brackets ([]) is the active clock.
+*-c*::
+ List the available trace file compression algorithms.
+
SEE ALSO
--------
trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
diff --git a/tracecmd/trace-list.c b/tracecmd/trace-list.c
index d060c810..900da73b 100644
--- a/tracecmd/trace-list.c
+++ b/tracecmd/trace-list.c
@@ -549,6 +549,24 @@ static void show_plugins(void)
tep_free(pevent);
}
+static void show_compression(void)
+{
+ char **versions, **names;
+ int c, i;
+
+ c = tracecmd_compress_protos_get(&names, &versions);
+ if (c <= 0) {
+ printf("No compression algorithms are supported\n");
+ return;
+ }
+ printf("Supported compression algorithms:\n");
+ for (i = 0; i < c; i++)
+ printf("\t%s, %s\n", names[i], versions[i]);
+
+ free(names);
+ free(versions);
+}
+
void trace_list(int argc, char **argv)
{
int events = 0;
@@ -562,6 +580,7 @@ void trace_list(int argc, char **argv)
int flags = 0;
int systems = 0;
int show_all = 1;
+ int compression = 0;
int i;
const char *arg;
const char *funcre = NULL;
@@ -626,6 +645,10 @@ void trace_list(int argc, char **argv)
systems = 1;
show_all = 0;
break;
+ case 'c':
+ compression = 1;
+ show_all = 0;
+ break;
case '-':
if (strcmp(argv[i], "--debug") == 0) {
tracecmd_set_debug(true);
@@ -670,6 +693,8 @@ void trace_list(int argc, char **argv)
show_clocks();
if (systems)
show_systems();
+ if (compression)
+ show_compression();
if (show_all) {
printf("event systems:\n");
show_systems();
@@ -679,6 +704,7 @@ void trace_list(int argc, char **argv)
show_tracers();
printf("\noptions:\n");
show_options();
+ show_compression();
}
return;
diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c
index 32b38bfd..f463465a 100644
--- a/tracecmd/trace-usage.c
+++ b/tracecmd/trace-usage.c
@@ -348,6 +348,7 @@ static struct usage_help usage_help[] = {
" -O list plugin options\n"
" -B list defined buffer instances\n"
" -C list the defined clocks (and active one)\n"
+ " -c list the supported trace file compression algorithms\n"
},
{
"restore",
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 10/87] trace-cmd library: Internal helpers for compressing data
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (8 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 09/87] trace-cmd list: Show supported " Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 11/87] trace-cmd library: Internal helpers for uncompressing data Tzvetomir Stoyanov (VMware)
` (76 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
New library internal helper functions are introduced, to add compression
functionality to the output trace handler.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/trace-cmd-local.h | 8 ++++
lib/trace-cmd/trace-output.c | 64 ++++++++++++++++++++++++-
2 files changed, 70 insertions(+), 2 deletions(-)
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 2a622003..14c4dd08 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -34,6 +34,14 @@ void tracecmd_info(const char *fmt, ...);
void tracecmd_compress_init(void);
void tracecmd_compress_free(void);
+int out_uncompress_block(struct tracecmd_output *handle);
+int out_compression_start(struct tracecmd_output *handle);
+int out_compression_end(struct tracecmd_output *handle);
+void out_compression_reset(struct tracecmd_output *handle);
+unsigned long long out_copy_fd_compress(struct tracecmd_output *handle,
+ int fd, unsigned long long max,
+ unsigned long long *write_size);
+
off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off_t offset, int whence);
#endif /* _TRACE_CMD_LOCAL_H */
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index a8de107c..552889c6 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -59,6 +59,8 @@ struct tracecmd_output {
unsigned long file_state;
unsigned long file_version;
size_t options_start;
+ bool do_compress;
+ struct tracecmd_compression *compress;
struct list_head options;
struct tracecmd_msg_handle *msg_handle;
@@ -77,15 +79,36 @@ struct list_event_system {
char *name;
};
-static stsize_t
-do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size)
+__hidden long long
+do_write_check(struct tracecmd_output *handle, const void *data, long long size)
{
+ if (handle->do_compress)
+ return tracecmd_compress_write(handle->compress, data, size);
+
if (handle->msg_handle)
return tracecmd_msg_data_send(handle->msg_handle, data, size);
return __do_write_check(handle->fd, data, size);
}
+static inline off64_t do_lseek(struct tracecmd_output *handle, off_t offset, int whence)
+{
+ if (handle->do_compress)
+ return tracecmd_compress_lseek(handle->compress, offset, whence);
+ else if (handle->msg_handle)
+ return msg_lseek(handle->msg_handle, offset, whence);
+ else
+ return lseek64(handle->fd, offset, whence);
+}
+
+static inline int do_preed(struct tracecmd_output *handle, void *dst, int len, off_t offset)
+{
+ if (handle->do_compress)
+ return tracecmd_compress_pread(handle->compress, dst, len, offset);
+ else
+ return pread(handle->fd, dst, len, offset);
+}
+
static short convert_endian_2(struct tracecmd_output *handle, short val)
{
if (!handle->pevent)
@@ -111,6 +134,43 @@ static unsigned long long convert_endian_8(struct tracecmd_output *handle,
return tep_read_number(handle->pevent, &val, 8);
}
+__hidden void out_compression_reset(struct tracecmd_output *handle)
+{
+ if (!handle->compress)
+ return;
+ tracecmd_compress_reset(handle->compress);
+ handle->do_compress = false;
+}
+
+__hidden int out_uncompress_block(struct tracecmd_output *handle)
+{
+ int ret = 0;
+
+ if (!handle->compress)
+ return 0;
+ ret = tracecmd_uncompress_block(handle->compress);
+ if (!ret)
+ handle->do_compress = true;
+ return ret;
+}
+
+__hidden int out_compression_start(struct tracecmd_output *handle)
+{
+ if (!handle->compress)
+ return 0;
+ tracecmd_compress_reset(handle->compress);
+ handle->do_compress = true;
+ return 0;
+}
+
+__hidden int out_compression_end(struct tracecmd_output *handle)
+{
+ if (!handle->compress)
+ return 0;
+ handle->do_compress = false;
+ return tracecmd_compress_block(handle->compress);
+}
+
/**
* tracecmd_set_quiet - Set if to print output to the screen
* @quiet: If non zero, print no output to the screen
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 11/87] trace-cmd library: Internal helpers for uncompressing data
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (9 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 10/87] trace-cmd library: Internal helpers for compressing data Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 12/87] trace-cmd library: Define trace file version 7 Tzvetomir Stoyanov (VMware)
` (75 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
New library internal helper functions are introduced, to add compression
functionality to the input trace handler.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/trace-cmd-local.h | 4 ++
lib/trace-cmd/trace-input.c | 53 +++++++++++++++++++++----
2 files changed, 49 insertions(+), 8 deletions(-)
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 14c4dd08..271c55a7 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -38,6 +38,10 @@ int out_uncompress_block(struct tracecmd_output *handle);
int out_compression_start(struct tracecmd_output *handle);
int out_compression_end(struct tracecmd_output *handle);
void out_compression_reset(struct tracecmd_output *handle);
+
+void in_uncompress_reset(struct tracecmd_input *handle);
+int in_uncompress_block(struct tracecmd_input *handle);
+
unsigned long long out_copy_fd_compress(struct tracecmd_output *handle,
int fd, unsigned long long max,
unsigned long long *write_size);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 787d6825..e759285b 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -136,6 +136,9 @@ struct tracecmd_input {
long long ts_offset;
struct tsc2nsec tsc_calc;
+ bool read_compress;
+ struct tracecmd_compression *compress;
+
struct host_trace_info host;
double ts2secs;
char * cpustats;
@@ -238,13 +241,13 @@ static const char *show_records(struct page **pages, int nr_pages)
static int init_cpu(struct tracecmd_input *handle, int cpu);
-static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size)
+static ssize_t do_read_fd(int fd, void *data, size_t size)
{
ssize_t tot = 0;
ssize_t r;
do {
- r = read(handle->fd, data + tot, size - tot);
+ r = read(fd, data + tot, size - tot);
tot += r;
if (!r)
@@ -256,12 +259,28 @@ static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size)
return tot;
}
+static inline int do_lseek(struct tracecmd_input *handle, int offset, int whence)
+{
+ if (handle->read_compress)
+ return tracecmd_compress_lseek(handle->compress, offset, whence);
+ else
+ return lseek(handle->fd, offset, whence);
+}
+
+static inline ssize_t do_read_compressed(struct tracecmd_input *handle, void *data, size_t size)
+{
+ if (handle->read_compress)
+ return tracecmd_compress_read(handle->compress, data, size);
+ else
+ return do_read_fd(handle->fd, data, size);
+}
+
static ssize_t
do_read_check(struct tracecmd_input *handle, void *data, size_t size)
{
ssize_t ret;
- ret = do_read(handle, data, size);
+ ret = do_read_compressed(handle, data, size);
if (ret < 0)
return ret;
if (ret != size)
@@ -279,10 +298,8 @@ static char *read_string(struct tracecmd_input *handle)
ssize_t r;
for (;;) {
- r = do_read(handle, buf, BUFSIZ);
- if (r < 0)
- goto fail;
- if (!r)
+ r = do_read_compressed(handle, buf, BUFSIZ);
+ if (r <= 0)
goto fail;
for (i = 0; i < r; i++) {
@@ -308,7 +325,7 @@ static char *read_string(struct tracecmd_input *handle)
}
/* move the file descriptor to the end of the string */
- r = lseek(handle->fd, -(r - (i+1)), SEEK_CUR);
+ r = do_lseek(handle, -(r - (i+1)), SEEK_CUR);
if (r < 0)
goto fail;
@@ -372,6 +389,26 @@ static int read8(struct tracecmd_input *handle, unsigned long long *size)
return 0;
}
+__hidden void in_uncompress_reset(struct tracecmd_input *handle)
+{
+ if (handle->compress) {
+ handle->read_compress = false;
+ tracecmd_compress_reset(handle->compress);
+ }
+}
+
+__hidden int in_uncompress_block(struct tracecmd_input *handle)
+{
+ int ret = 0;
+
+ if (handle->compress) {
+ ret = tracecmd_uncompress_block(handle->compress);
+ if (!ret)
+ handle->read_compress = true;
+ }
+ return ret;
+}
+
static int read_header_files(struct tracecmd_input *handle)
{
struct tep_handle *pevent = handle->pevent;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 12/87] trace-cmd library: Define trace file version 7
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (10 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 11/87] trace-cmd library: Internal helpers for uncompressing data Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 13/87] trace-cmd library: Refactor APIs for creating output handler Tzvetomir Stoyanov (VMware)
` (74 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added a define for file version 7, but keep the default file version to
6.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/private/trace-cmd-private.h | 3 +++
lib/trace-cmd/include/trace-cmd-local.h | 7 -------
lib/trace-cmd/trace-output.c | 2 +-
lib/trace-cmd/trace-util.c | 2 +-
tracecmd/trace-record.c | 2 ++
5 files changed, 7 insertions(+), 9 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 72729c57..c3b407a0 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -31,6 +31,9 @@ struct tep_plugin_list *trace_load_plugins(struct tep_handle *tep, int flags);
int *tracecmd_add_id(int *list, int id, int len);
+#define FILE_VERSION_DEFAULT 6
+#define FILE_VERSION_MIN 6
+#define FILE_VERSION_MAX 7
enum {
RINGBUF_TYPE_PADDING = 29,
RINGBUF_TYPE_TIME_EXTEND = 30,
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 271c55a7..ad3f7e1f 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -14,13 +14,6 @@ void tracecmd_warning(const char *fmt, ...);
void tracecmd_critical(const char *fmt, ...);
void tracecmd_info(const char *fmt, ...);
-/* trace.dat file format version */
-#define FILE_VERSION 6
-
-#define _STR(x) #x
-#define STR(x) _STR(x)
-#define FILE_VERSION_STRING STR(FILE_VERSION)
-
#ifndef htonll
# if __BYTE_ORDER == __LITTLE_ENDIAN
#define htonll(x) __bswap_64(x)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 552889c6..ae35c12d 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -976,7 +976,7 @@ static int select_file_version(struct tracecmd_output *handle,
if (ihandle)
handle->file_version = tracecmd_get_in_file_version(ihandle);
else
- handle->file_version = FILE_VERSION;
+ handle->file_version = FILE_VERSION_DEFAULT;
return 0;
}
diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c
index 4e1b8427..00246076 100644
--- a/lib/trace-cmd/trace-util.c
+++ b/lib/trace-cmd/trace-util.c
@@ -620,7 +620,7 @@ unsigned long long tracecmd_generate_traceid(void)
bool tracecmd_is_version_supported(unsigned int version)
{
- if (version <= FILE_VERSION)
+ if (version >= FILE_VERSION_MIN && version <= FILE_VERSION_MAX)
return true;
return false;
}
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index 1767a6c6..9780cedd 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -214,6 +214,7 @@ struct common_record_context {
int topt;
int run_command;
int saved_cmdlines_size;
+ int file_version;
};
static void add_reset_file(const char *file, const char *val, int prio)
@@ -5937,6 +5938,7 @@ static void init_common_record_context(struct common_record_context *ctx,
ctx->instance = &top_instance;
ctx->curr_cmd = curr_cmd;
local_cpu_count = tracecmd_count_cpus();
+ ctx->file_version = FILE_VERSION_DEFAULT;
init_top_instance();
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 13/87] trace-cmd library: Refactor APIs for creating output handler
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (11 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 12/87] trace-cmd library: Define trace file version 7 Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 14/87] trace-cmd library: Reuse within the library the function that checks file state Tzvetomir Stoyanov (VMware)
` (73 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In the trace-cmd library there are various APIs for allocating and
initializing output handler to a trace file. The existing APIs are use
case oriented, with a lot of parameters. Extending them for new use
cases, adding more input parameters, will make the library more complex
and not easy to use.
Almost all use case oriented APIs for output handler creation are
removed and replaced with new flow, which is easier to be extended with
new creation parameters.
Removed APIs:
tracecmd_create_init_fd_msg()
tracecmd_create_init_file_glob()
tracecmd_create_init_fd_glob()
tracecmd_create_init_file_override()
New APIs:
tracecmd_output_allocate()
tracecmd_output_set_msg()
tracecmd_output_set_trace_dir()
tracecmd_output_set_kallsyms()
tracecmd_output_set_from_input()
tracecmd_output_set_version()
tracecmd_output_write_init()
tracecmd_output_write_headers()
The new tracecmd_output_allocate() API allocates memory and performs
minimalinitialization of an output handler to a trace file. No data
is written in the file.
The tracecmd_output_set_...() APIs can be used to set various
parameters to the newly allocated output handler, that affect the way
the data is written into the file.
When the output handler is configured for the desired use case, the
tracecmd_output_write_init() is used to start writing to the file, it
writes initial magic bytes.
The tracecmd_output_write_headers() API is used to write the initial
headers into the file.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 29 +-
lib/trace-cmd/trace-output.c | 437 ++++++++++++------
tracecmd/trace-record.c | 64 ++-
tracecmd/trace-restore.c | 32 +-
4 files changed, 395 insertions(+), 167 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index c3b407a0..16dbed56 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -105,7 +105,8 @@ static inline int tracecmd_host_bigendian(void)
/* --- Opening and Reading the trace.dat file --- */
enum tracecmd_file_states {
- TRACECMD_FILE_INIT = 1,
+ TRACECMD_FILE_ALLOCATED = 0,
+ TRACECMD_FILE_INIT,
TRACECMD_FILE_HEADERS,
TRACECMD_FILE_FTRACE_EVENTS,
TRACECMD_FILE_ALL_EVENTS,
@@ -271,20 +272,22 @@ struct tracecmd_event_list {
struct tracecmd_option;
struct tracecmd_msg_handle;
-struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus);
-struct tracecmd_output *
-tracecmd_create_init_file_glob(const char *output_file,
- struct tracecmd_event_list *list);
+struct tracecmd_output *tracecmd_output_allocate(int fd);
+int tracecmd_output_set_msg(struct tracecmd_output *handler,
+ struct tracecmd_msg_handle *msg_handle);
+int tracecmd_output_set_trace_dir(struct tracecmd_output *handler, const char *tracing_dir);
+int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *kallsyms);
+int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle);
+int tracecmd_output_set_version(struct tracecmd_output *handler, int file_version);
+int tracecmd_output_write_init(struct tracecmd_output *handler);
+int tracecmd_output_write_headers(struct tracecmd_output *handler,
+ struct tracecmd_event_list *list);
+
+struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus,
+ int file_version);
struct tracecmd_output *tracecmd_create_init_fd(int fd);
-struct tracecmd_output *
-tracecmd_create_init_fd_glob(int fd, struct tracecmd_event_list *list);
-struct tracecmd_output *
-tracecmd_create_init_fd_msg(struct tracecmd_msg_handle *msg_handle,
- struct tracecmd_event_list *list);
+
struct tracecmd_output *tracecmd_create_init_file(const char *output_file);
-struct tracecmd_output *tracecmd_create_init_file_override(const char *output_file,
- const char *tracing_dir,
- const char *kallsyms);
struct tracecmd_option *tracecmd_add_option(struct tracecmd_output *handle,
unsigned short id, int size,
const void *data);
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index ae35c12d..e1373e80 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -31,11 +31,6 @@
typedef unsigned long long tsize_t;
typedef long long stsize_t;
-static struct tracecmd_event_list all_event_list = {
- .next = NULL,
- .glob = "all"
-};
-
struct tracecmd_option {
unsigned short id;
int size;
@@ -54,11 +49,14 @@ struct tracecmd_output {
int cpus;
struct tep_handle *pevent;
char *tracing_dir;
+ char *kallsyms;
+ struct tracecmd_event_list *events;
int nr_options;
bool quiet;
unsigned long file_state;
unsigned long file_version;
size_t options_start;
+ bool big_endian;
bool do_compress;
struct tracecmd_compression *compress;
@@ -769,7 +767,7 @@ static int read_event_files(struct tracecmd_output *handle,
break;
}
/* all events are listed, use a global glob */
- if (list)
+ if (!event_list || list)
event_list = &all_events;
systems = create_event_list(handle, event_list);
@@ -839,8 +837,7 @@ err:
tracecmd_warning("can't set kptr_restrict");
}
-static int read_proc_kallsyms(struct tracecmd_output *handle,
- const char *kallsyms)
+static int read_proc_kallsyms(struct tracecmd_output *handle)
{
unsigned int size, check_size, endian4;
const char *path = "/proc/kallsyms";
@@ -853,8 +850,8 @@ static int read_proc_kallsyms(struct tracecmd_output *handle,
return -1;
}
- if (kallsyms)
- path = kallsyms;
+ if (handle->kallsyms)
+ path = handle->kallsyms;
ret = stat(path, &st);
if (ret < 0) {
@@ -970,137 +967,272 @@ out_free:
return ret;
}
-static int select_file_version(struct tracecmd_output *handle,
- struct tracecmd_input *ihandle)
+/**
+ * tracecmd_output_allocate - allocate new output handler to a trace file
+ * @handle: file descriptor to an empty file, it can be -1 if the handler
+ * will not write to a file
+ *
+ * This API only allocates a handler and performs minimal initialization.
+ * No data is written in the file.
+ *
+ * Returns pointer to a newly allocated file descriptor, that can be used
+ * with other library APIs. In case of an error, NULL is returned. The returned
+ * handler must be freed with tracecmd_output_close() or tracecmd_output_free()
+ */
+struct tracecmd_output *tracecmd_output_allocate(int fd)
{
- if (ihandle)
- handle->file_version = tracecmd_get_in_file_version(ihandle);
+ struct tracecmd_output *handle;
+
+ handle = calloc(1, sizeof(*handle));
+ if (!handle)
+ return NULL;
+
+ handle->fd = fd;
+
+ handle->file_version = FILE_VERSION_DEFAULT;
+
+ handle->page_size = getpagesize();
+
+ if (tracecmd_host_bigendian())
+ handle->big_endian = true;
else
- handle->file_version = FILE_VERSION_DEFAULT;
+ handle->big_endian = false;
- return 0;
+ list_head_init(&handle->options);
+
+ handle->file_state = TRACECMD_FILE_ALLOCATED;
+
+ return handle;
}
-static struct tracecmd_output *
-create_file_fd(int fd, struct tracecmd_input *ihandle,
- const char *tracing_dir,
- const char *kallsyms,
- struct tracecmd_event_list *list,
- struct tracecmd_msg_handle *msg_handle)
+/**
+ * tracecmd_output_set_msg - associated an output file handler with network message handler
+ * @handle: output handler to a trace file.
+ * @msg_handle: network handler, allocated by tracecmd_msg_handle_alloc()
+ *
+ * This API associates an output file handler with a network stream. All subsequent API calls
+ * with this output file handler will send data over the network using the @msg_handle, instead
+ * of writing to a file.
+ * This API must be called after the handler file version is set and before
+ * tracecmd_output_write_init().
+ *
+ * Returns 0 on success, or -1 if the output file handler is not allocated or not in expected state.
+ */
+int tracecmd_output_set_msg(struct tracecmd_output *handler, struct tracecmd_msg_handle *msg_handle)
{
- struct tracecmd_output *handle;
- struct tep_handle *pevent;
- char buf[BUFSIZ];
- int endian4;
+ if (!handler || handler->file_state != TRACECMD_FILE_ALLOCATED)
+ return -1;
- handle = malloc(sizeof(*handle));
- if (!handle)
- return NULL;
- memset(handle, 0, sizeof(*handle));
+ handler->msg_handle = msg_handle;
+ /* Force messages to be cached in a temp file before sending through the socket */
+ if (handler->msg_handle && handler->file_version >= 7)
+ tracecmd_msg_handle_cache(handler->msg_handle);
- list_head_init(&handle->options);
+ return 0;
+}
- handle->fd = fd;
+/**
+ * tracecmd_output_set_trace_dir - Set a custom tracing dir, instead of system default
+ * @handle: output handler to a trace file.
+ * @tracing_dir: full path to a directory with tracing files
+ *
+ * This API associates an output file handler with a custom tracing directory, to be used when
+ * creating the trace file instead of the system default tracing directory.
+ * This API must be called before tracecmd_output_write_init().
+ *
+ * Returns 0 on success, or -1 if the output file handler is not allocated or not in expected state.
+ */
+int tracecmd_output_set_trace_dir(struct tracecmd_output *handler, const char *tracing_dir)
+{
+ if (!handler || handler->file_state != TRACECMD_FILE_ALLOCATED)
+ return -1;
+
+ free(handler->tracing_dir);
if (tracing_dir) {
- handle->tracing_dir = strdup(tracing_dir);
- if (!handle->tracing_dir)
- goto out_free;
- }
+ handler->tracing_dir = strdup(tracing_dir);
+ if (!handler->tracing_dir)
+ return -1;
+ } else
+ handler->tracing_dir = NULL;
- handle->msg_handle = msg_handle;
+ return 0;
+}
- if (select_file_version(handle, ihandle))
- goto out_free;
+/**
+ * tracecmd_output_set_kallsyms - Set a custom kernel symbols file, instead of system default
+ * @handle: output handler to a trace file.
+ * @tracing_dir: full path to a file with kernel symbols
+ *
+ * This API associates an output file handler with a custom kernel symbols file, to be used when
+ * creating the trace file instead of the system default kernel symbols file.
+ * This API must be called before tracecmd_output_write_init().
+ *
+ * Returns 0 on success, or -1 if the output file handler is not allocated or not in expected state.
+ */
+int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *kallsyms)
+{
+ if (!handler || handler->file_state != TRACECMD_FILE_ALLOCATED)
+ return -1;
- buf[0] = 23;
- buf[1] = 8;
- buf[2] = 68;
- memcpy(buf + 3, "tracing", 7);
+ free(handler->kallsyms);
+ if (kallsyms) {
+ handler->kallsyms = strdup(kallsyms);
+ if (!handler->kallsyms)
+ return -1;
+ } else
+ handler->kallsyms = NULL;
- if (do_write_check(handle, buf, 10))
- goto out_free;
+ return 0;
+}
- sprintf(buf, "%lu", handle->file_version);
- if (do_write_check(handle, buf, strlen(buf) + 1))
- goto out_free;
+/**
+ * tracecmd_output_set_from_input - Inherit parameters from an existing trace file
+ * @handle: output handler to a trace file.
+ * @ihandle: input handler to an existing trace file.
+ *
+ * This API copies parameters from input handler @ihandle, associated with an existing trace file,
+ * to the output handler @handle, associated with file that is going to be created.
+ * These parameters are copied:
+ * - tep handler
+ * - page size
+ * - file endian
+ * - file version
+ * - file compression protocol
+ * This API must be called before tracecmd_output_write_init().
+ *
+ * Returns 0 on success, or -1 if the output file handler is not allocated or not in expected state.
+ */
+int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle)
+{
+ if (!handler || !ihandle || handler->file_state != TRACECMD_FILE_ALLOCATED)
+ return -1;
- /* get endian and page size */
- if (ihandle) {
- pevent = tracecmd_get_tep(ihandle);
- /* Use the pevent of the ihandle for later writes */
- handle->pevent = tracecmd_get_tep(ihandle);
- tep_ref(pevent);
- if (tep_is_file_bigendian(pevent))
- buf[0] = 1;
- else
- buf[0] = 0;
- handle->page_size = tracecmd_page_size(ihandle);
- } else {
- if (tracecmd_host_bigendian())
- buf[0] = 1;
- else
- buf[0] = 0;
- handle->page_size = getpagesize();
- }
+ /* get endian, page size, file version and compression */
+ /* Use the pevent of the ihandle for later writes */
+ handler->pevent = tracecmd_get_tep(ihandle);
+ tep_ref(handler->pevent);
+ handler->page_size = tracecmd_page_size(ihandle);
+ handler->file_version = tracecmd_get_in_file_version(ihandle);
+ if (tep_is_file_bigendian(handler->pevent))
+ handler->big_endian = true;
+ else
+ handler->big_endian = false;
- if (do_write_check(handle, buf, 1))
- goto out_free;
- /* save size of long (this may not be what the kernel is) */
- buf[0] = sizeof(long);
- if (do_write_check(handle, buf, 1))
- goto out_free;
+ return 0;
+}
- endian4 = convert_endian_4(handle, handle->page_size);
- if (do_write_check(handle, &endian4, 4))
- goto out_free;
- handle->file_state = TRACECMD_FILE_INIT;
+/**
+ * tracecmd_output_set_version - Set file version of the output handler
+ * @handle: output handler to a trace file.
+ * @file_version: desired file version
+ *
+ * This API must be called before tracecmd_output_write_init().
+ *
+ * Returns 0 on success, or -1 if the output file handler is not allocated or not in expected state.
+ */
+int tracecmd_output_set_version(struct tracecmd_output *handler, int file_version)
+{
+ if (!handler || handler->file_state != TRACECMD_FILE_ALLOCATED)
+ return -1;
+ if (file_version < FILE_VERSION_MIN || file_version > FILE_VERSION_MAX)
+ return -1;
+ handler->file_version = file_version;
+ if (handler->file_version < 7)
+ handler->compress = NULL;
+ return 0;
+}
- if (ihandle)
- return handle;
- if (read_header_files(handle))
- goto out_free;
+/**
+ * tracecmd_output_write_init - Write the initial magics in the trace file
+ * @handle: output handler to a trace file.
+ *
+ * This API must be called after all tracecmd_output_set_...() APIs and before writing anything
+ * to the trace file. This initial information is written in the file:
+ * - initial file magic bytes
+ * - file version
+ * - data endian
+ * - long size
+ * - page size
+ * - compression header
+ *
+ * Returns 0 on success, or -1 if the output file handler is not allocated or not in expected state.
+ */
+int tracecmd_output_write_init(struct tracecmd_output *handler)
+{
+ char buf[BUFSIZ];
+ int endian4;
- if (read_ftrace_files(handle))
- goto out_free;
+ if (!handler || handler->file_state != TRACECMD_FILE_ALLOCATED)
+ return -1;
- if (read_event_files(handle, list))
- goto out_free;
+ buf[0] = 23;
+ buf[1] = 8;
+ buf[2] = 68;
+ memcpy(buf + 3, "tracing", 7);
- if (read_proc_kallsyms(handle, kallsyms))
- goto out_free;
+ if (do_write_check(handler, buf, 10))
+ return -1;
- if (read_ftrace_printk(handle))
- goto out_free;
+ sprintf(buf, "%lu", handler->file_version);
+ if (do_write_check(handler, buf, strlen(buf) + 1))
+ return -1;
- return handle;
+ if (handler->big_endian)
+ buf[0] = 1;
+ else
+ buf[0] = 0;
+ if (do_write_check(handler, buf, 1))
+ return -1;
- out_free:
- tracecmd_output_close(handle);
- return NULL;
+ /* save size of long (this may not be what the kernel is) */
+ buf[0] = sizeof(long);
+ if (do_write_check(handler, buf, 1))
+ return -1;
+
+ endian4 = convert_endian_4(handler, handler->page_size);
+ if (do_write_check(handler, &endian4, 4))
+ return -1;
+ handler->file_state = TRACECMD_FILE_INIT;
+ return 0;
}
-static struct tracecmd_output *create_file(const char *output_file,
- struct tracecmd_input *ihandle,
- const char *tracing_dir,
- const char *kallsyms,
- struct tracecmd_event_list *list)
+/**
+ * tracecmd_output_write_headers - Write the trace file headers
+ * @handle: output handler to a trace file.
+ * @list: desired events that will be included in the trace file.
+ * It can be NULL for all available events
+ *
+ * These headers are written in the file:
+ * - header files from the tracing directory
+ * - ftrace events from the tracing directory
+ * - event file from the tracing directory - all or only the one from @list
+ * - kernel symbols from the tracing directory
+ * - kernel printk strings from the tracing directory
+ *
+ * Returns 0 on success, or -1 in case of an error.
+ */
+int tracecmd_output_write_headers(struct tracecmd_output *handler,
+ struct tracecmd_event_list *list)
{
- struct tracecmd_output *handle;
- int fd;
-
- fd = open(output_file, O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE, 0644);
- if (fd < 0)
- return NULL;
-
- handle = create_file_fd(fd, ihandle, tracing_dir, kallsyms, list, NULL);
- if (!handle) {
- close(fd);
- unlink(output_file);
- }
+ if (!handler || handler->file_state < TRACECMD_FILE_ALLOCATED)
+ return -1;
- return handle;
+ /* Write init data, if not written yet */
+ if (handler->file_state < TRACECMD_FILE_INIT && tracecmd_output_write_init(handler))
+ return -1;
+ if (read_header_files(handler))
+ return -1;
+ if (read_ftrace_files(handler))
+ return -1;
+ if (read_event_files(handler, list))
+ return -1;
+ if (read_proc_kallsyms(handler))
+ return -1;
+ if (read_ftrace_printk(handler))
+ return -1;
+ return 0;
}
/**
@@ -1354,16 +1486,26 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle)
return 0;
}
-struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus)
+struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus,
+ int file_version)
{
struct tracecmd_output *handle;
char *path;
- int ret;
+ int fd;
- handle = create_file(output_file, NULL, NULL, NULL, &all_event_list);
- if (!handle)
+ fd = open(output_file, O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE, 0644);
+ if (fd < 0)
return NULL;
+ handle = tracecmd_output_allocate(fd);
+ if (!handle)
+ goto out_free;
+ if (file_version && tracecmd_output_set_version(handle, file_version))
+ goto out_free;
+ if (tracecmd_output_write_init(handle))
+ goto out_free;
+ if (tracecmd_output_write_headers(handle, NULL))
+ goto out_free;
/*
* Save the command lines;
*/
@@ -1376,8 +1518,7 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in
if (tracecmd_write_options(handle) < 0)
goto out_free;
- ret = check_out_state(handle, TRACECMD_FILE_CPU_LATENCY);
- if (ret < 0) {
+ if (!check_out_state(handle, TRACECMD_FILE_CPU_LATENCY)) {
tracecmd_warning("Cannot write latency data into the file, unexpected state 0x%X",
handle->file_state);
goto out_free;
@@ -1656,39 +1797,38 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd)
struct tracecmd_output *tracecmd_create_init_fd(int fd)
{
- return create_file_fd(fd, NULL, NULL, NULL, &all_event_list, NULL);
-}
-
-struct tracecmd_output *
-tracecmd_create_init_fd_msg(struct tracecmd_msg_handle *msg_handle,
- struct tracecmd_event_list *list)
-{
- return create_file_fd(msg_handle->fd, NULL, NULL, NULL, list, msg_handle);
-}
-
-struct tracecmd_output *
-tracecmd_create_init_fd_glob(int fd, struct tracecmd_event_list *list)
-{
- return create_file_fd(fd, NULL, NULL, NULL, list, NULL);
-}
+ struct tracecmd_output *out;
-struct tracecmd_output *
-tracecmd_create_init_file_glob(const char *output_file,
- struct tracecmd_event_list *list)
-{
- return create_file(output_file, NULL, NULL, NULL, list);
+ out = tracecmd_output_allocate(fd);
+ if (!out)
+ return NULL;
+ if (tracecmd_output_write_init(out))
+ goto error;
+ if (tracecmd_output_write_headers(out, NULL))
+ goto error;
+
+ return out;
+error:
+ tracecmd_output_close(out);
+ return NULL;
}
struct tracecmd_output *tracecmd_create_init_file(const char *output_file)
{
- return create_file(output_file, NULL, NULL, NULL, &all_event_list);
-}
+ struct tracecmd_output *handle;
+ int fd;
-struct tracecmd_output *tracecmd_create_init_file_override(const char *output_file,
- const char *tracing_dir,
- const char *kallsyms)
-{
- return create_file(output_file, NULL, tracing_dir, kallsyms, &all_event_list);
+ fd = open(output_file, O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE, 0644);
+ if (fd < 0)
+ return NULL;
+ handle = tracecmd_create_init_fd(fd);
+ if (!handle) {
+ close(fd);
+ unlink(output_file);
+ return NULL;
+ }
+
+ return handle;
}
/**
@@ -1704,11 +1844,18 @@ struct tracecmd_output *tracecmd_copy(struct tracecmd_input *ihandle,
const char *file)
{
struct tracecmd_output *handle;
+ int fd;
- handle = create_file(file, ihandle, NULL, NULL, &all_event_list);
- if (!handle)
+ fd = open(file, O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE, 0644);
+ if (fd < 0)
return NULL;
+ handle = tracecmd_output_allocate(fd);
+ if (!handle)
+ goto out_free;
+ if (tracecmd_output_set_from_input(handle, ihandle))
+ goto out_free;
+ tracecmd_output_write_init(handle);
if (tracecmd_copy_headers(ihandle, handle->fd, 0, 0) < 0)
goto out_free;
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index 9780cedd..8080d612 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -3690,6 +3690,27 @@ again:
static void add_options(struct tracecmd_output *handle, struct common_record_context *ctx);
+static struct tracecmd_output *create_net_output(struct common_record_context *ctx,
+ struct tracecmd_msg_handle *msg_handle)
+{
+ struct tracecmd_output *out;
+
+ out = tracecmd_output_allocate(-1);
+ if (!out)
+ return NULL;
+ if (ctx->file_version && tracecmd_output_set_version(out, ctx->file_version))
+ goto error;
+ if (tracecmd_output_set_msg(out, msg_handle))
+ goto error;
+ if (tracecmd_output_write_headers(out, listed_events))
+ goto error;
+
+ return out;
+error:
+ tracecmd_output_close(out);
+ return NULL;
+}
+
static struct tracecmd_msg_handle *
setup_connection(struct buffer_instance *instance, struct common_record_context *ctx)
{
@@ -3701,7 +3722,7 @@ setup_connection(struct buffer_instance *instance, struct common_record_context
/* Now create the handle through this socket */
if (msg_handle->version == V3_PROTOCOL) {
- network_handle = tracecmd_create_init_fd_msg(msg_handle, listed_events);
+ network_handle = create_net_output(ctx, msg_handle);
if (!network_handle)
goto error;
tracecmd_set_quiet(network_handle, quiet);
@@ -3719,10 +3740,13 @@ setup_connection(struct buffer_instance *instance, struct common_record_context
if (ret)
goto error;
} else {
- network_handle = tracecmd_create_init_fd_glob(msg_handle->fd,
- listed_events);
+ network_handle = tracecmd_output_allocate(msg_handle->fd);
if (!network_handle)
goto error;
+ if (tracecmd_output_set_version(network_handle, ctx->file_version))
+ goto error;
+ if (tracecmd_output_write_headers(network_handle, listed_events))
+ goto error;
tracecmd_set_quiet(network_handle, quiet);
}
@@ -4068,8 +4092,7 @@ static void setup_agent(struct buffer_instance *instance,
{
struct tracecmd_output *network_handle;
- network_handle = tracecmd_create_init_fd_msg(instance->msg_handle,
- listed_events);
+ network_handle = create_net_output(ctx, instance->msg_handle);
add_options(network_handle, ctx);
tracecmd_write_cmdlines(network_handle);
tracecmd_write_cpus(network_handle, instance->cpu_count);
@@ -4438,6 +4461,32 @@ static void write_guest_file(struct buffer_instance *instance)
free(temp_files);
}
+static struct tracecmd_output *create_output(struct common_record_context *ctx)
+{
+ struct tracecmd_output *out;
+ int fd;
+
+ fd = open(ctx->output, O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE, 0644);
+ if (fd < 0)
+ return NULL;
+
+ out = tracecmd_output_allocate(fd);
+ if (!out)
+ goto error;
+ if (ctx->file_version && tracecmd_output_set_version(out, ctx->file_version))
+ goto error;
+ if (tracecmd_output_write_headers(out, listed_events))
+ goto error;
+ return out;
+error:
+ if (out)
+ tracecmd_output_close(out);
+ else
+ close(fd);
+ unlink(ctx->output);
+ return NULL;
+}
+
static void record_data(struct common_record_context *ctx)
{
struct tracecmd_option **buffer_options;
@@ -4461,7 +4510,8 @@ static void record_data(struct common_record_context *ctx)
return;
if (latency) {
- handle = tracecmd_create_file_latency(ctx->output, local_cpu_count);
+ handle = tracecmd_create_file_latency(ctx->output, local_cpu_count,
+ ctx->file_version);
tracecmd_set_quiet(handle, quiet);
} else {
if (!local_cpu_count)
@@ -4492,7 +4542,7 @@ static void record_data(struct common_record_context *ctx)
touch_file(temp_files[i]);
}
- handle = tracecmd_create_init_file_glob(ctx->output, listed_events);
+ handle = create_output(ctx);
if (!handle)
die("Error creating output file");
tracecmd_set_quiet(handle, quiet);
diff --git a/tracecmd/trace-restore.c b/tracecmd/trace-restore.c
index 280a37f0..8d2fcae8 100644
--- a/tracecmd/trace-restore.c
+++ b/tracecmd/trace-restore.c
@@ -22,6 +22,35 @@
#include "trace-local.h"
+static struct tracecmd_output *create_output(const char *file,
+ const char *tracing_dir, const char *kallsyms)
+{
+ struct tracecmd_output *out;
+ int fd;
+
+ fd = open(file, O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE, 0644);
+ if (fd < 0)
+ return NULL;
+
+ out = tracecmd_output_allocate(fd);
+ if (!out)
+ goto error;
+ if (tracing_dir && tracecmd_output_set_trace_dir(out, tracing_dir))
+ goto error;
+ if (kallsyms && tracecmd_output_set_kallsyms(out, kallsyms))
+ goto error;
+ if (tracecmd_output_write_headers(out, NULL))
+ goto error;
+ return out;
+error:
+ if (out)
+ tracecmd_output_close(out);
+ else
+ close(fd);
+ unlink(file);
+ return NULL;
+}
+
void trace_restore (int argc, char **argv)
{
struct tracecmd_output *handle;
@@ -90,8 +119,7 @@ void trace_restore (int argc, char **argv)
usage(argv);
}
- handle = tracecmd_create_init_file_override(output, tracing_dir,
- kallsyms);
+ handle = create_output(output, tracing_dir, kallsyms);
if (!handle)
die("Unabled to create output file %s", output);
if (tracecmd_write_cmdlines(handle) < 0)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 14/87] trace-cmd library: Reuse within the library the function that checks file state.
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (12 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 13/87] trace-cmd library: Refactor APIs for creating output handler Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 15/87] trace-cmd library: New API to get the version of output handler Tzvetomir Stoyanov (VMware)
` (72 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Make the function, that checks if the next file state is valid, global
for the tracmd-library, so it can be reused. It is important the same check
logic to be used in the whole library.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/trace-cmd-local.h | 4 ++
lib/trace-cmd/trace-input.c | 5 +++
lib/trace-cmd/trace-output.c | 59 +++++++------------------
lib/trace-cmd/trace-util.c | 33 ++++++++++++++
4 files changed, 59 insertions(+), 42 deletions(-)
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index ad3f7e1f..f799ea8d 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -27,6 +27,10 @@ void tracecmd_info(const char *fmt, ...);
void tracecmd_compress_init(void);
void tracecmd_compress_free(void);
+bool check_file_state(unsigned long file_version, int current_state, int new_state);
+bool check_in_state(struct tracecmd_input *handle, int new_state);
+bool check_out_state(struct tracecmd_output *handle, int new_state);
+
int out_uncompress_block(struct tracecmd_output *handle);
int out_compression_start(struct tracecmd_output *handle);
int out_compression_end(struct tracecmd_output *handle);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index e759285b..8495ce35 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4259,3 +4259,8 @@ int tracecmd_enable_tsync(struct tracecmd_input *handle, bool enable)
return 0;
}
+
+__hidden bool check_in_state(struct tracecmd_input *handle, int new_state)
+{
+ return check_file_state(handle->file_version, handle->file_state, new_state);
+}
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index e1373e80..8e402d35 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -365,33 +365,6 @@ int tracecmd_ftrace_enable(int set)
return ret;
}
-static int check_out_state(struct tracecmd_output *handle, int new_state)
-{
- if (!handle)
- return -1;
-
- switch (new_state) {
- case TRACECMD_FILE_HEADERS:
- case TRACECMD_FILE_FTRACE_EVENTS:
- case TRACECMD_FILE_ALL_EVENTS:
- case TRACECMD_FILE_KALLSYMS:
- case TRACECMD_FILE_PRINTK:
- case TRACECMD_FILE_CMD_LINES:
- case TRACECMD_FILE_CPU_COUNT:
- case TRACECMD_FILE_OPTIONS:
- if (handle->file_state == (new_state - 1))
- return 0;
- break;
- case TRACECMD_FILE_CPU_LATENCY:
- case TRACECMD_FILE_CPU_FLYRECORD:
- if (handle->file_state == TRACECMD_FILE_OPTIONS)
- return 0;
- break;
- }
-
- return -1;
-}
-
static int read_header_files(struct tracecmd_output *handle)
{
tsize_t size, check_size, endian8;
@@ -400,7 +373,7 @@ static int read_header_files(struct tracecmd_output *handle)
int fd;
int ret;
- if (check_out_state(handle, TRACECMD_FILE_HEADERS) < 0) {
+ if (!check_out_state(handle, TRACECMD_FILE_HEADERS)) {
tracecmd_warning("Cannot read header files, unexpected state 0x%X",
handle->file_state);
return -1;
@@ -712,7 +685,7 @@ static int read_ftrace_files(struct tracecmd_output *handle)
struct tracecmd_event_list list = { .glob = "ftrace/*" };
int ret;
- if (check_out_state(handle, TRACECMD_FILE_FTRACE_EVENTS) < 0) {
+ if (!check_out_state(handle, TRACECMD_FILE_FTRACE_EVENTS)) {
tracecmd_warning("Cannot read ftrace files, unexpected state 0x%X",
handle->file_state);
return -1;
@@ -753,7 +726,7 @@ static int read_event_files(struct tracecmd_output *handle,
int endian4;
int ret;
- if (check_out_state(handle, TRACECMD_FILE_ALL_EVENTS) < 0) {
+ if (!check_out_state(handle, TRACECMD_FILE_ALL_EVENTS)) {
tracecmd_warning("Cannot read event files, unexpected state 0x%X",
handle->file_state);
return -1;
@@ -844,7 +817,7 @@ static int read_proc_kallsyms(struct tracecmd_output *handle)
struct stat st;
int ret;
- if (check_out_state(handle, TRACECMD_FILE_KALLSYMS) < 0) {
+ if (!check_out_state(handle, TRACECMD_FILE_KALLSYMS)) {
tracecmd_warning("Cannot read kallsyms, unexpected state 0x%X",
handle->file_state);
return -1;
@@ -889,7 +862,7 @@ static int read_ftrace_printk(struct tracecmd_output *handle)
char *path;
int ret;
- if (check_out_state(handle, TRACECMD_FILE_PRINTK) < 0) {
+ if (!check_out_state(handle, TRACECMD_FILE_PRINTK)) {
tracecmd_warning("Cannot read printk, unexpected state 0x%X",
handle->file_state);
return -1;
@@ -1326,11 +1299,10 @@ int tracecmd_write_cpus(struct tracecmd_output *handle, int cpus)
{
int ret;
- ret = check_out_state(handle, TRACECMD_FILE_CPU_COUNT);
- if (ret < 0) {
+ if (!check_out_state(handle, TRACECMD_FILE_CPU_COUNT)) {
tracecmd_warning("Cannot write CPU count into the file, unexpected state 0x%X",
handle->file_state);
- return ret;
+ return -1;
}
cpus = convert_endian_4(handle, cpus);
ret = do_write_check(handle, &cpus, 4);
@@ -1346,16 +1318,14 @@ int tracecmd_write_options(struct tracecmd_output *handle)
unsigned short option;
unsigned short endian2;
unsigned int endian4;
- int ret;
/* If already written, ignore */
if (handle->file_state == TRACECMD_FILE_OPTIONS)
return 0;
- ret = check_out_state(handle, TRACECMD_FILE_OPTIONS);
- if (ret < 0) {
+ if (!check_out_state(handle, TRACECMD_FILE_OPTIONS)) {
tracecmd_warning("Cannot write options into the file, unexpected state 0x%X",
handle->file_state);
- return ret;
+ return -1;
}
if (do_write_check(handle, "options ", 10))
@@ -1473,11 +1443,10 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle)
{
int ret;
- ret = check_out_state(handle, TRACECMD_FILE_CMD_LINES);
- if (ret < 0) {
+ if (!check_out_state(handle, TRACECMD_FILE_CMD_LINES)) {
tracecmd_warning("Cannot write command lines into the file, unexpected state 0x%X",
handle->file_state);
- return ret;
+ return -1;
}
ret = save_tracing_file_data(handle, "saved_cmdlines");
if (ret < 0)
@@ -1868,3 +1837,9 @@ out_free:
tracecmd_output_close(handle);
return NULL;
}
+
+__hidden bool check_out_state(struct tracecmd_output *handle, int new_state)
+{
+ return check_file_state(handle->file_version, handle->file_state, new_state);
+}
+
diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c
index 00246076..cda125c4 100644
--- a/lib/trace-cmd/trace-util.c
+++ b/lib/trace-cmd/trace-util.c
@@ -634,3 +634,36 @@ static void __attribute__((destructor)) tracecmd_lib_free(void)
{
tracecmd_compress_free();
}
+
+__hidden bool check_file_state(unsigned long file_version, int current_state, int new_state)
+{
+ if (file_version >= 7) {
+ if (current_state < TRACECMD_FILE_INIT)
+ return false;
+ return true;
+ }
+
+ switch (new_state) {
+ case TRACECMD_FILE_HEADERS:
+ case TRACECMD_FILE_FTRACE_EVENTS:
+ case TRACECMD_FILE_ALL_EVENTS:
+ case TRACECMD_FILE_KALLSYMS:
+ case TRACECMD_FILE_PRINTK:
+ case TRACECMD_FILE_CMD_LINES:
+ case TRACECMD_FILE_CPU_COUNT:
+ if (current_state == (new_state - 1))
+ return true;
+ break;
+ case TRACECMD_FILE_OPTIONS:
+ if (file_version < 7 && current_state == TRACECMD_FILE_CPU_COUNT)
+ return true;
+ break;
+ case TRACECMD_FILE_CPU_LATENCY:
+ case TRACECMD_FILE_CPU_FLYRECORD:
+ if (current_state == TRACECMD_FILE_OPTIONS)
+ return true;
+ break;
+ }
+
+ return false;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 15/87] trace-cmd library: New API to get the version of output handler
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (13 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 14/87] trace-cmd library: Reuse within the library the function that checks file state Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 16/87] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
` (71 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
There are cases where the version of the file, associated with given
output handler must be retrieved. Added a new API to get that version:
tracecmd_get_out_file_version()
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/private/trace-cmd-private.h | 1 +
lib/trace-cmd/trace-output.c | 8 ++++++++
2 files changed, 9 insertions(+)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 16dbed56..2fb15026 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -316,6 +316,7 @@ int tracecmd_append_buffer_cpu_data(struct tracecmd_output *handle,
int cpus, char * const *cpu_data_files);
struct tracecmd_output *tracecmd_get_output_handle_fd(int fd);
+unsigned long tracecmd_get_out_file_version(struct tracecmd_output *handle);
/* --- Reading the Fly Recorder Trace --- */
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 8e402d35..50a91cf4 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -1843,3 +1843,11 @@ __hidden bool check_out_state(struct tracecmd_output *handle, int new_state)
return check_file_state(handle->file_version, handle->file_state, new_state);
}
+/**
+ * tracecmd_get_out_file_version - return the trace.dat file version
+ * @handle: output handle for the trace.dat file
+ */
+unsigned long tracecmd_get_out_file_version(struct tracecmd_output *handle)
+{
+ return handle->file_version;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 16/87] trace-cmd library: Inherit compression algorithm from input file
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (14 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 15/87] trace-cmd library: New API to get the version of output handler Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 17/87] trace-cmd library: New API to configure compression on an output handler Tzvetomir Stoyanov (VMware)
` (70 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When a new trace file output handler is allocated, based on given trace
file input handler - use the same compression algorithm.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 2 ++
lib/trace-cmd/trace-input.c | 16 ++++++++++++++++
lib/trace-cmd/trace-output.c | 19 +++++++++++++++++++
3 files changed, 37 insertions(+)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 2fb15026..455ff680 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -239,6 +239,8 @@ tracecmd_get_cursor(struct tracecmd_input *handle, int cpu);
unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle);
size_t tracecmd_get_options_offset(struct tracecmd_input *handle);
+int tracecmd_get_file_compress_proto(struct tracecmd_input *handle,
+ const char **name, const char **version);
int tracecmd_ftrace_overrides(struct tracecmd_input *handle, struct tracecmd_ftrace *finfo);
bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 8495ce35..c5b4c32e 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4114,6 +4114,22 @@ unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle)
return handle->file_version;
}
+/**
+ * tracecmd_get_file_compress_proto - get name and version of compression algorithm,
+ * used to compress the trace file
+ * @handle: input handle for the trace.dat file
+ * @name: return, name of the compression algorithm.
+ * @version: return, version of the compression algorithm.
+ *
+ * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the
+ * algorithm are stored in @name and @version. The returned strings must *not* be freed.
+ */
+int tracecmd_get_file_compress_proto(struct tracecmd_input *handle,
+ const char **name, const char **version)
+{
+ return tracecmd_compress_proto_get_name(handle->compress, name, version);
+}
+
/**
* tracecmd_get_use_trace_clock - return use_trace_clock
* @handle: input handle for the trace.dat file
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 50a91cf4..42b193dc 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -1077,6 +1077,9 @@ int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *ka
*/
int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle)
{
+ const char *cname = NULL;
+ const char *cver = NULL;
+
if (!handler || !ihandle || handler->file_state != TRACECMD_FILE_ALLOCATED)
return -1;
@@ -1091,6 +1094,14 @@ int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct trace
else
handler->big_endian = false;
+ if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) {
+ handler->compress = tracecmd_compress_alloc(cname, cver, handler->fd,
+ handler->pevent, handler->msg_handle);
+ if (!handler->compress)
+ return -1;
+ if (handler->file_version < 7)
+ handler->file_version = 7;
+ }
return 0;
}
@@ -1716,6 +1727,8 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd)
{
struct tracecmd_output *handle = NULL;
struct tracecmd_input *ihandle;
+ const char *cname = NULL;
+ const char *cver = NULL;
int fd2;
/* Move the file descriptor to the beginning */
@@ -1754,6 +1767,12 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd)
handle->options_start = tracecmd_get_options_offset(ihandle);
list_head_init(&handle->options);
+ if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) {
+ handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd,
+ handle->pevent, handle->msg_handle);
+ if (!handle->compress)
+ goto out_free;
+ }
tracecmd_close(ihandle);
return handle;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 17/87] trace-cmd library: New API to configure compression on an output handler
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (15 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 16/87] trace-cmd library: Inherit compression algorithm from input file Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-08-05 21:15 ` Steven Rostedt
2021-07-29 5:08 ` [PATCH v2 18/87] trace-cmd record: Add compression to the trace context Tzvetomir Stoyanov (VMware)
` (69 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
New API can be used to configure compression algorithm on a output
handler to a trace file.
tracecmd_output_set_compression()
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 1 +
lib/trace-cmd/trace-output.c | 47 +++++++++++++++++++
2 files changed, 48 insertions(+)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 455ff680..ee224903 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -281,6 +281,7 @@ int tracecmd_output_set_trace_dir(struct tracecmd_output *handler, const char *t
int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *kallsyms);
int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle);
int tracecmd_output_set_version(struct tracecmd_output *handler, int file_version);
+int tracecmd_output_set_compression(struct tracecmd_output *handler, const char *compression);
int tracecmd_output_write_init(struct tracecmd_output *handler);
int tracecmd_output_write_headers(struct tracecmd_output *handler,
struct tracecmd_event_list *list);
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 42b193dc..23a10edc 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -219,6 +219,7 @@ void tracecmd_output_free(struct tracecmd_output *handle)
free(option);
}
free(handle->trace_clock);
+ tracecmd_compress_destroy(handle->compress);
free(handle);
}
@@ -1127,6 +1128,52 @@ int tracecmd_output_set_version(struct tracecmd_output *handler, int file_versio
return 0;
}
+/**
+ * tracecmd_output_set_compression - Set file compression algorithm of the output handler
+ * @handle: output handler to a trace file.
+ * @compression: name of the desired compression algorithm. Can be one of:
+ * - "none" - do not use compression
+ * - "all" - use the best available compression algorithm
+ * - or specific name of the desired compression algorithm
+ *
+ * This API must be called before tracecmd_output_write_init().
+ *
+ * Returns 0 on success, or -1 in case of an error:
+ * - the output file handler is not allocated or not in expected state.
+ * - the specified compression algorithm is not available
+ */
+int tracecmd_output_set_compression(struct tracecmd_output *handler, const char *compression)
+{
+ if (!handler || handler->file_state != TRACECMD_FILE_ALLOCATED)
+ return -1;
+
+ handler->compress = NULL;
+ if (compression && strcmp(compression, "none")) {
+ if (!strcmp(compression, "any")) {
+ handler->compress = tracecmd_compress_alloc(NULL, NULL, handler->fd,
+ handler->pevent,
+ handler->msg_handle);
+ if (!handler->compress)
+ tracecmd_warning("No compression algorithms are supported");
+ } else {
+ handler->compress = tracecmd_compress_alloc(compression, NULL, handler->fd,
+ handler->pevent,
+ handler->msg_handle);
+ if (!handler->compress) {
+ tracecmd_warning("Compression algorithm %s is not supported",
+ compression);
+ return -1;
+ }
+ }
+ }
+ if (handler->compress && handler->file_version < 7) {
+ handler->file_version = 7;
+ if (handler->msg_handle)
+ tracecmd_msg_handle_cache(handler->msg_handle);
+ }
+
+ return 0;
+}
/**
* tracecmd_output_write_init - Write the initial magics in the trace file
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 17/87] trace-cmd library: New API to configure compression on an output handler
2021-07-29 5:08 ` [PATCH v2 17/87] trace-cmd library: New API to configure compression on an output handler Tzvetomir Stoyanov (VMware)
@ 2021-08-05 21:15 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-05 21:15 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:08:49 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> + }
> + if (handler->compress && handler->file_version < 7) {
> + handler->file_version = 7;
> + if (handler->msg_handle)
> + tracecmd_msg_handle_cache(handler->msg_handle);
> + }
I was thinking that in all places that we check '7' for support of
compression, we should instead make that a macro.
/* The version that compression was added */
#define FILE_VERSION_COMPRESSION 7
[..]
if (handler->compress && handler->file_version < FILE_VERSION_COMPRESSION) {
handler->file_version = FILE_VERSION_COMPRESSION;
if (handler->msg_handle)
tracecmd_msg_handle_cache(handler->msg_handle);
}
Then we don't need to memorize what version compression was added in.
?
-- Steve
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 18/87] trace-cmd record: Add compression to the trace context
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (16 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 17/87] trace-cmd library: New API to configure compression on an output handler Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 19/87] trace-cmd library: Write compression header in the trace file Tzvetomir Stoyanov (VMware)
` (68 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
As the trace-cmd libray supports trace file compression, trace-cmd
record command should have a way to configure this functionality. Trace
context is extended to hold the compression algorithm, used to
compress the file.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-record.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index 8080d612..295fe633 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -199,6 +199,7 @@ struct common_record_context {
char *date2ts;
char *user;
const char *clock;
+ const char *compression;
struct tsc_nsec tsc2nsec;
int data_flags;
int tsync_loop_interval;
@@ -3702,6 +3703,12 @@ static struct tracecmd_output *create_net_output(struct common_record_context *c
goto error;
if (tracecmd_output_set_msg(out, msg_handle))
goto error;
+ if (ctx->compression) {
+ if (tracecmd_output_set_compression(out, ctx->compression))
+ goto error;
+ } else if (ctx->file_version >= 7) {
+ tracecmd_output_set_compression(out, "any");
+ }
if (tracecmd_output_write_headers(out, listed_events))
goto error;
@@ -3745,6 +3752,12 @@ setup_connection(struct buffer_instance *instance, struct common_record_context
goto error;
if (tracecmd_output_set_version(network_handle, ctx->file_version))
goto error;
+ if (ctx->compression) {
+ if (tracecmd_output_set_compression(network_handle, ctx->compression))
+ goto error;
+ } else if (ctx->file_version >= 7) {
+ tracecmd_output_set_compression(network_handle, "any");
+ }
if (tracecmd_output_write_headers(network_handle, listed_events))
goto error;
tracecmd_set_quiet(network_handle, quiet);
@@ -4475,6 +4488,12 @@ static struct tracecmd_output *create_output(struct common_record_context *ctx)
goto error;
if (ctx->file_version && tracecmd_output_set_version(out, ctx->file_version))
goto error;
+ if (ctx->compression) {
+ if (tracecmd_output_set_compression(out, ctx->compression))
+ goto error;
+ } else if (ctx->file_version >= 7) {
+ tracecmd_output_set_compression(out, "any");
+ }
if (tracecmd_output_write_headers(out, listed_events))
goto error;
return out;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 19/87] trace-cmd library: Write compression header in the trace file
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (17 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 18/87] trace-cmd record: Add compression to the trace context Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 20/87] trace-cmd library: Compress part of " Tzvetomir Stoyanov (VMware)
` (67 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
If there is a compression configured on the output file handler and if
the file version is at least 7, write compression header in the file.
The compression header is two null terminated strings - name and version
of the compression algorithm, used to compress some parts of the file.
The header is located after the page size in the file. The new header is
mandatory for trace files version 7. If no compression is used, the
string "none" is saved as name of the compression algorithm and empty
string as compression algorithm version.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 23a10edc..d348d6a5 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -941,6 +941,24 @@ out_free:
return ret;
}
+static int write_compression_header(struct tracecmd_output *handle)
+{
+ const char *name = NULL;
+ const char *ver = NULL;
+ int ret;
+
+ ret = tracecmd_compress_proto_get_name(handle->compress, &name, &ver);
+ if (ret < 0 || !name || !ver) {
+ name = "none";
+ ver = "";
+ }
+ if (do_write_check(handle, name, strlen(name) + 1))
+ return -1;
+ if (do_write_check(handle, ver, strlen(ver) + 1))
+ return -1;
+ return 0;
+}
+
/**
* tracecmd_output_allocate - allocate new output handler to a trace file
* @handle: file descriptor to an empty file, it can be -1 if the handler
@@ -1225,6 +1243,11 @@ int tracecmd_output_write_init(struct tracecmd_output *handler)
endian4 = convert_endian_4(handler, handler->page_size);
if (do_write_check(handler, &endian4, 4))
return -1;
+ if (handler->file_version >= 7) {
+ if (write_compression_header(handler))
+ return -1;
+ }
+
handler->file_state = TRACECMD_FILE_INIT;
return 0;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 20/87] trace-cmd library: Compress part of the trace file
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (18 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 19/87] trace-cmd library: Write compression header in the trace file Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-08-05 21:27 ` Steven Rostedt
2021-07-29 5:08 ` [PATCH v2 21/87] trace-cmd library: Add internal helper functon for writing headers before file sections Tzvetomir Stoyanov (VMware)
` (66 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Compress part of the trace.dat file metadata. If there is compression
support, compress these parts of the file:
- ftrace events format
- format of recorded events
- information of the mapping of function addresses to the function names
- trace_printk() format strings
- information of the mapping a PID to a process name
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 130 +++++++++++++++++++++++++----------
1 file changed, 92 insertions(+), 38 deletions(-)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index d348d6a5..5d3fd58f 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -366,12 +366,12 @@ int tracecmd_ftrace_enable(int set)
return ret;
}
-static int read_header_files(struct tracecmd_output *handle)
+static int read_header_files(struct tracecmd_output *handle, bool compress)
{
tsize_t size, check_size, endian8;
struct stat st;
char *path;
- int fd;
+ int fd = -1;
int ret;
if (!check_out_state(handle, TRACECMD_FILE_HEADERS)) {
@@ -384,26 +384,30 @@ static int read_header_files(struct tracecmd_output *handle)
if (!path)
return -1;
+ if (compress)
+ out_compression_start(handle);
ret = stat(path, &st);
if (ret < 0) {
/* old style did not show this info, just add zero */
put_tracing_file(path);
if (do_write_check(handle, "header_page", 12))
- return -1;
+ goto out_close;
size = 0;
if (do_write_check(handle, &size, 8))
- return -1;
+ goto out_close;
if (do_write_check(handle, "header_event", 13))
- return -1;
+ goto out_close;
if (do_write_check(handle, &size, 8))
- return -1;
+ goto out_close;
+ if (compress && out_compression_end(handle))
+ goto out_close;
return 0;
}
fd = open(path, O_RDONLY);
if (fd < 0) {
tracecmd_warning("can't read '%s'", path);
- return -1;
+ goto out_close;
}
/* unfortunately, you can not stat debugfs files for size */
@@ -419,18 +423,18 @@ static int read_header_files(struct tracecmd_output *handle)
if (size != check_size) {
tracecmd_warning("wrong size for '%s' size=%lld read=%lld", path, size, check_size);
errno = EINVAL;
- return -1;
+ goto out_close;
}
put_tracing_file(path);
path = get_tracing_file(handle, "events/header_event");
if (!path)
- return -1;
+ goto out_close;
fd = open(path, O_RDONLY);
if (fd < 0) {
tracecmd_warning("can't read '%s'", path);
- return -1;
+ goto out_close;
}
size = get_size_fd(fd);
@@ -444,16 +448,19 @@ static int read_header_files(struct tracecmd_output *handle)
close(fd);
if (size != check_size) {
tracecmd_warning("wrong size for '%s'", path);
- return -1;
+ goto out_close;
}
put_tracing_file(path);
-
+ if (compress && out_compression_end(handle))
+ goto out_close;
handle->file_state = TRACECMD_FILE_HEADERS;
return 0;
out_close:
- close(fd);
+ out_compression_reset(handle);
+ if (fd >= 0)
+ close(fd);
return -1;
}
@@ -680,7 +687,7 @@ create_event_list_item(struct tracecmd_output *handle,
tracecmd_warning("Insufficient memory");
}
-static int read_ftrace_files(struct tracecmd_output *handle)
+static int read_ftrace_files(struct tracecmd_output *handle, bool compress)
{
struct list_event_system *systems = NULL;
struct tracecmd_event_list list = { .glob = "ftrace/*" };
@@ -693,9 +700,15 @@ static int read_ftrace_files(struct tracecmd_output *handle)
}
create_event_list_item(handle, &systems, &list);
-
+ if (compress)
+ out_compression_start(handle);
ret = copy_event_system(handle, systems);
-
+ if (compress) {
+ if (!ret)
+ ret = out_compression_end(handle);
+ else
+ out_compression_reset(handle);
+ }
free_list_events(systems);
handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
@@ -717,7 +730,7 @@ create_event_list(struct tracecmd_output *handle,
}
static int read_event_files(struct tracecmd_output *handle,
- struct tracecmd_event_list *event_list)
+ struct tracecmd_event_list *event_list, bool compress)
{
struct list_event_system *systems;
struct list_event_system *slist;
@@ -748,7 +761,8 @@ static int read_event_files(struct tracecmd_output *handle,
for (slist = systems; slist; slist = slist->next)
count++;
-
+ if (compress)
+ out_compression_start(handle);
ret = -1;
endian4 = convert_endian_4(handle, count);
if (do_write_check(handle, &endian4, 4))
@@ -763,9 +777,19 @@ static int read_event_files(struct tracecmd_output *handle,
}
ret = copy_event_system(handle, slist);
}
-
- handle->file_state = TRACECMD_FILE_ALL_EVENTS;
+ if (ret)
+ goto out_free;
+ if (compress) {
+ ret = out_compression_end(handle);
+ if (ret)
+ goto out_free;
+ }
out_free:
+ if (!ret)
+ handle->file_state = TRACECMD_FILE_ALL_EVENTS;
+ else
+ out_compression_reset(handle);
+
free_list_events(systems);
return ret;
@@ -811,7 +835,7 @@ err:
tracecmd_warning("can't set kptr_restrict");
}
-static int read_proc_kallsyms(struct tracecmd_output *handle)
+static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress)
{
unsigned int size, check_size, endian4;
const char *path = "/proc/kallsyms";
@@ -827,19 +851,21 @@ static int read_proc_kallsyms(struct tracecmd_output *handle)
if (handle->kallsyms)
path = handle->kallsyms;
+ if (compress)
+ out_compression_start(handle);
ret = stat(path, &st);
if (ret < 0) {
/* not found */
size = 0;
endian4 = convert_endian_4(handle, size);
- if (do_write_check(handle, &endian4, 4))
- return -1;
- return 0;
+ ret = do_write_check(handle, &endian4, 4);
+ goto out;
}
size = get_size(path);
endian4 = convert_endian_4(handle, size);
- if (do_write_check(handle, &endian4, 4))
- return -1;
+ ret = do_write_check(handle, &endian4, 4);
+ if (ret)
+ goto out;
set_proc_kptr_restrict(0);
check_size = copy_file(handle, path);
@@ -847,16 +873,25 @@ static int read_proc_kallsyms(struct tracecmd_output *handle)
errno = EINVAL;
tracecmd_warning("error in size of file '%s'", path);
set_proc_kptr_restrict(1);
- return -1;
+ ret = -1;
+ goto out;
}
set_proc_kptr_restrict(1);
- handle->file_state = TRACECMD_FILE_KALLSYMS;
-
- return 0;
+ if (compress) {
+ ret = out_compression_end(handle);
+ if (ret)
+ goto out;
+ }
+out:
+ if (!ret)
+ handle->file_state = TRACECMD_FILE_KALLSYMS;
+ else
+ out_compression_reset(handle);
+ return ret;
}
-static int read_ftrace_printk(struct tracecmd_output *handle)
+static int read_ftrace_printk(struct tracecmd_output *handle, bool compress)
{
unsigned int size, check_size, endian4;
struct stat st;
@@ -873,6 +908,8 @@ static int read_ftrace_printk(struct tracecmd_output *handle)
if (!path)
return -1;
+ if (compress)
+ out_compression_start(handle);
ret = stat(path, &st);
if (ret < 0) {
/* not found */
@@ -894,11 +931,14 @@ static int read_ftrace_printk(struct tracecmd_output *handle)
}
out:
- handle->file_state = TRACECMD_FILE_PRINTK;
put_tracing_file(path);
+ if (compress && out_compression_end(handle))
+ return -1;
+ handle->file_state = TRACECMD_FILE_PRINTK;
return 0;
fail:
put_tracing_file(path);
+ out_compression_reset(handle);
return -1;
}
@@ -1270,21 +1310,25 @@ int tracecmd_output_write_init(struct tracecmd_output *handler)
int tracecmd_output_write_headers(struct tracecmd_output *handler,
struct tracecmd_event_list *list)
{
+ bool compress = false;
+
if (!handler || handler->file_state < TRACECMD_FILE_ALLOCATED)
return -1;
/* Write init data, if not written yet */
if (handler->file_state < TRACECMD_FILE_INIT && tracecmd_output_write_init(handler))
return -1;
- if (read_header_files(handler))
+ if (handler->compress)
+ compress = true;
+ if (read_header_files(handler, compress))
return -1;
- if (read_ftrace_files(handler))
+ if (read_ftrace_files(handler, compress))
return -1;
- if (read_event_files(handler, list))
+ if (read_event_files(handler, list, compress))
return -1;
- if (read_proc_kallsyms(handler))
+ if (read_proc_kallsyms(handler, compress))
return -1;
- if (read_ftrace_printk(handler))
+ if (read_ftrace_printk(handler, compress))
return -1;
return 0;
}
@@ -1529,9 +1573,19 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle)
handle->file_state);
return -1;
}
+
+ if (handle->compress)
+ out_compression_start(handle);
+
ret = save_tracing_file_data(handle, "saved_cmdlines");
- if (ret < 0)
+ if (ret < 0) {
+ out_compression_reset(handle);
return ret;
+ }
+
+ if (handle->compress && out_compression_end(handle))
+ return -1;
+
handle->file_state = TRACECMD_FILE_CMD_LINES;
return 0;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 20/87] trace-cmd library: Compress part of the trace file
2021-07-29 5:08 ` [PATCH v2 20/87] trace-cmd library: Compress part of " Tzvetomir Stoyanov (VMware)
@ 2021-08-05 21:27 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-05 21:27 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:08:52 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> @@ -384,26 +384,30 @@ static int read_header_files(struct tracecmd_output *handle)
> if (!path)
> return -1;
>
> + if (compress)
> + out_compression_start(handle);
> ret = stat(path, &st);
> if (ret < 0) {
> -
> + if (compress && out_compression_end(handle))
> + goto out_close;
> handle->file_state = TRACECMD_FILE_HEADERS;
>
> -
> + if (compress)
> + out_compression_start(handle);
> ret = copy_event_system(handle, systems);
> -
> + if (compress) {
> + if (!ret)
> + ret = out_compression_end(handle);
> + else
> + out_compression_reset(handle);
> + }
> -
> + if (compress)
> + out_compression_start(handle);
> ret = -1;
> endian4 = convert_endian_4(handle, count);
> if (do_write_check(handle, &endian4, 4))
> @@ -763,9 +777,19 @@ static int read_event_files(struct tracecmd_output *handle,
> }
> ret = copy_event_system(handle, slist);
> }
> -
> - handle->file_state = TRACECMD_FILE_ALL_EVENTS;
> + if (ret)
> + goto out_free;
> + if (compress) {
> + ret = out_compression_end(handle);
> + if (ret)
> + goto out_free;
> + }
> @@ -827,19 +851,21 @@ static int read_proc_kallsyms(struct tracecmd_output *handle)
> if (handle->kallsyms)
> path = handle->kallsyms;
>
> + if (compress)
> + out_compression_start(handle);
> -
> - return 0;
> + if (compress) {
> + ret = out_compression_end(handle);
> + if (ret)
> + goto out;
> + }
>
> + if (compress)
> + out_compression_start(handle);
> ret = stat(path, &st);
> if (ret < 0) {
> /* not found */
> @@ -894,11 +931,14 @@ static int read_ftrace_printk(struct tracecmd_output *handle)
> }
>
> out:
> - handle->file_state = TRACECMD_FILE_PRINTK;
> put_tracing_file(path);
> + if (compress && out_compression_end(handle))
> + return -1;
> + handle->file_state = TRACECMD_FILE_PRINTK;
> }
> +
> + if (handle->compress)
> + out_compression_start(handle);
> +
> ret = save_tracing_file_data(handle, "saved_cmdlines");
> - if (ret < 0)
> + if (ret < 0) {
> + out_compression_reset(handle);
> return ret;
> + }
> +
> + if (handle->compress && out_compression_end(handle))
> + return -1;
> +
> handle->file_state = TRACECMD_FILE_CMD_LINES;
> return 0;
> }
Pretty much all the out_compression_*() functions are enclosed in a:
if (compress)
out_compress_*()
condition.
Why not just add that to the parameter, and clean up the code where
this is used, because these branches are rather ugly and break up the
flow.
For example:
__hidden void out_compression_reset(struct tracecmd_output *handle, bool compress)
{
if (!compress || !handle->compress)
return;
tracecmd_compress_reset(handle->compress);
handle->do_compress = false;
}
__hidden int out_uncompress_block(struct tracecmd_output *handle, bool compress)
{
int ret = 0;
if (!compress || !handle->compress)
return 0;
ret = tracecmd_uncompress_block(handle->compress);
if (!ret)
handle->do_compress = true;
return ret;
}
__hidden int out_compression_start(struct tracecmd_output *handle, bool compress)
{
if (!compress || !handle->compress)
return 0;
tracecmd_compress_reset(handle->compress);
handle->do_compress = true;
return 0;
}
__hidden int out_compression_end(struct tracecmd_output *handle, bool compress)
{
if (!compress || !handle->compress)
return 0;
handle->do_compress = false;
return tracecmd_compress_block(handle->compress);
}
Then instead of all these:
if (compress)
out_compression_start(handle);
You just have:
out_compression_start(handle, compress);
-- Steve
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 21/87] trace-cmd library: Add internal helper functon for writing headers before file sections
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (19 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 20/87] trace-cmd library: Compress part of " Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 22/87] trace-cmd library: Write header " Tzvetomir Stoyanov (VMware)
` (65 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Introduce headers before each file section, in trace file version 7. The
section header has the following format:
<2 bytes>, header ID
<string>, null terminated ASCII string, description of the header
<2 bytes>, section flags:
1: the section is compressed
<4 bytes>, size of the section
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
include/trace-cmd/trace-cmd.h | 5 ++
.../include/private/trace-cmd-private.h | 1 +
lib/trace-cmd/include/trace-cmd-local.h | 4 ++
lib/trace-cmd/trace-output.c | 70 +++++++++++++++++++
4 files changed, 80 insertions(+)
diff --git a/include/trace-cmd/trace-cmd.h b/include/trace-cmd/trace-cmd.h
index 6984db86..6c76b928 100644
--- a/include/trace-cmd/trace-cmd.h
+++ b/include/trace-cmd/trace-cmd.h
@@ -15,6 +15,11 @@ enum tracecmd_open_flags {
TRACECMD_FL_LOAD_NO_PLUGINS = 1 << 0, /* Do not load plugins */
TRACECMD_FL_LOAD_NO_SYSTEM_PLUGINS = 1 << 1, /* Do not load system plugins */
};
+
+enum tracecmd_section_flags {
+ TRACECMD_SEC_FL_COMPRESS = 1 << 0, /* the section is compressed */
+};
+
struct tracecmd_input *tracecmd_open_head(const char *file, int flags);
struct tracecmd_input *tracecmd_open(const char *file, int flags);
struct tracecmd_input *tracecmd_open_fd(int fd, int flags);
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index ee224903..77948fbb 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -135,6 +135,7 @@ enum {
TRACECMD_OPTION_TIME_SHIFT,
TRACECMD_OPTION_GUEST,
TRACECMD_OPTION_TSC2NSEC,
+ TRACECMD_OPTION_MAX,
};
enum {
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index f799ea8d..2a23d9e0 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -42,6 +42,10 @@ int in_uncompress_block(struct tracecmd_input *handle);
unsigned long long out_copy_fd_compress(struct tracecmd_output *handle,
int fd, unsigned long long max,
unsigned long long *write_size);
+unsigned long long
+out_write_section_header(struct tracecmd_output *handle, unsigned short header_id,
+ char *description, enum tracecmd_section_flags flags, bool option);
+int out_update_section_header(struct tracecmd_output *handle, unsigned long long offset);
off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off_t offset, int whence);
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 5d3fd58f..74192ea2 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -366,6 +366,76 @@ int tracecmd_ftrace_enable(int set)
return ret;
}
+__hidden unsigned long long
+out_write_section_header(struct tracecmd_output *handle, unsigned short header_id,
+ char *description, enum tracecmd_section_flags flags, bool option)
+{
+ tsize_t endian8;
+ tsize_t offset;
+ short endian2;
+ int size;
+
+ if (header_id >= TRACECMD_OPTION_MAX)
+ return -1;
+ if (handle->file_version < 7)
+ return 0;
+ if (!handle->compress)
+ flags &= ~TRACECMD_SEC_FL_COMPRESS;
+ offset = do_lseek(handle, 0, SEEK_CUR);
+ if (option) {
+ endian8 = convert_endian_8(handle, offset);
+ if (!tracecmd_add_option(handle, header_id, 8, &endian8))
+ return -1;
+ }
+ /* Section ID */
+ endian2 = convert_endian_2(handle, header_id);
+ if (do_write_check(handle, &endian2, 2))
+ return (off_t)-1;
+
+ /* Section description */
+ if (do_write_check(handle, description, strlen(description) + 1))
+ return (off_t)-1;
+ /* Section flags */
+ endian2 = convert_endian_2(handle, flags);
+ if (do_write_check(handle, &endian2, 2))
+ return (off_t)-1;
+
+ offset = do_lseek(handle, 0, SEEK_CUR);
+ size = 0;
+ /* Reserve for section size */
+ if (do_write_check(handle, &size, 4))
+ return (off_t)-1;
+ return offset;
+}
+
+__hidden int out_update_section_header(struct tracecmd_output *handle, unsigned long long offset)
+{
+ unsigned long long current;
+ unsigned int endian4;
+ int size;
+
+ if (handle->file_version < 7 || offset == 0)
+ return 0;
+
+ current = do_lseek(handle, 0, SEEK_CUR);
+ /* The real size is the difference between the saved offset and
+ * the current offset - 4 bytes, the reserved space for the section size.
+ */
+ size = current - offset;
+ if (size < 4)
+ return -1;
+ size -= 4;
+ if (do_lseek(handle, offset, SEEK_SET) == (off_t)-1)
+ return -1;
+
+ endian4 = convert_endian_4(handle, size);
+ if (do_write_check(handle, &endian4, 4))
+ return -1;
+ if (do_lseek(handle, current, SEEK_SET) == (off_t)-1)
+ return -1;
+ return 0;
+}
+
static int read_header_files(struct tracecmd_output *handle, bool compress)
{
tsize_t size, check_size, endian8;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 22/87] trace-cmd library: Write header before file sections
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (20 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 21/87] trace-cmd library: Add internal helper functon for writing headers before file sections Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 23/87] trace-cmd library: Refactor the logic for writing trace data in the file Tzvetomir Stoyanov (VMware)
` (64 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Headers are added before these file secrtions, in trace
file version 7:
- ftrace events format
- format of recorded events
- information of the mapping of function addresses to the function names
- trace_printk() format strings
- information of the mapping a PID to a process name
New options are defined for each of these sections, holding
the section's offset into the trace file.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 6 ++
lib/trace-cmd/trace-output.c | 69 +++++++++++++++++++
2 files changed, 75 insertions(+)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 77948fbb..b193d6de 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -135,6 +135,12 @@ enum {
TRACECMD_OPTION_TIME_SHIFT,
TRACECMD_OPTION_GUEST,
TRACECMD_OPTION_TSC2NSEC,
+ TRACECMD_OPTION_HEADER_INFO,
+ TRACECMD_OPTION_FTRACE_EVENTS,
+ TRACECMD_OPTION_EVENT_FORMATS,
+ TRACECMD_OPTION_KALLSYMS,
+ TRACECMD_OPTION_PRINTK,
+ TRACECMD_OPTION_CMDLINES,
TRACECMD_OPTION_MAX,
};
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 74192ea2..a5c11eba 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -438,8 +438,10 @@ __hidden int out_update_section_header(struct tracecmd_output *handle, unsigned
static int read_header_files(struct tracecmd_output *handle, bool compress)
{
+ enum tracecmd_section_flags flags = 0;
tsize_t size, check_size, endian8;
struct stat st;
+ tsize_t offset;
char *path;
int fd = -1;
int ret;
@@ -454,6 +456,13 @@ static int read_header_files(struct tracecmd_output *handle, bool compress)
if (!path)
return -1;
+ if (compress)
+ flags |= TRACECMD_SEC_FL_COMPRESS;
+ offset = out_write_section_header(handle, TRACECMD_OPTION_HEADER_INFO,
+ "headers", flags, true);
+ if (offset == (off_t)-1)
+ return -1;
+
if (compress)
out_compression_start(handle);
ret = stat(path, &st);
@@ -469,6 +478,8 @@ static int read_header_files(struct tracecmd_output *handle, bool compress)
goto out_close;
if (do_write_check(handle, &size, 8))
goto out_close;
+ if (out_update_section_header(handle, offset))
+ goto out_close;
if (compress && out_compression_end(handle))
goto out_close;
return 0;
@@ -523,6 +534,8 @@ static int read_header_files(struct tracecmd_output *handle, bool compress)
put_tracing_file(path);
if (compress && out_compression_end(handle))
goto out_close;
+ if (out_update_section_header(handle, offset))
+ goto out_close;
handle->file_state = TRACECMD_FILE_HEADERS;
return 0;
@@ -759,8 +772,10 @@ create_event_list_item(struct tracecmd_output *handle,
static int read_ftrace_files(struct tracecmd_output *handle, bool compress)
{
+ enum tracecmd_section_flags flags = 0;
struct list_event_system *systems = NULL;
struct tracecmd_event_list list = { .glob = "ftrace/*" };
+ tsize_t offset;
int ret;
if (!check_out_state(handle, TRACECMD_FILE_FTRACE_EVENTS)) {
@@ -769,6 +784,13 @@ static int read_ftrace_files(struct tracecmd_output *handle, bool compress)
return -1;
}
+ if (compress)
+ flags |= TRACECMD_SEC_FL_COMPRESS;
+ offset = out_write_section_header(handle, TRACECMD_OPTION_FTRACE_EVENTS,
+ "ftrace events", flags, true);
+ if (offset == (off_t)-1)
+ return -1;
+
create_event_list_item(handle, &systems, &list);
if (compress)
out_compression_start(handle);
@@ -780,6 +802,10 @@ static int read_ftrace_files(struct tracecmd_output *handle, bool compress)
out_compression_reset(handle);
}
free_list_events(systems);
+ if (ret)
+ return ret;
+ if (out_update_section_header(handle, offset))
+ return -1;
handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
@@ -802,11 +828,13 @@ create_event_list(struct tracecmd_output *handle,
static int read_event_files(struct tracecmd_output *handle,
struct tracecmd_event_list *event_list, bool compress)
{
+ enum tracecmd_section_flags flags = 0;
struct list_event_system *systems;
struct list_event_system *slist;
struct tracecmd_event_list *list;
struct tracecmd_event_list all_events = { .glob = "*/*" };
int count = 0;
+ tsize_t offset;
int endian4;
int ret;
@@ -815,6 +843,13 @@ static int read_event_files(struct tracecmd_output *handle,
handle->file_state);
return -1;
}
+
+ if (compress)
+ flags |= TRACECMD_SEC_FL_COMPRESS;
+ offset = out_write_section_header(handle, TRACECMD_OPTION_EVENT_FORMATS,
+ "events format", flags, true);
+ if (offset == (off64_t)-1)
+ return -1;
/*
* If any of the list is the special keyword "all" then
* just do all files.
@@ -854,6 +889,8 @@ static int read_event_files(struct tracecmd_output *handle,
if (ret)
goto out_free;
}
+ ret = out_update_section_header(handle, offset);
+
out_free:
if (!ret)
handle->file_state = TRACECMD_FILE_ALL_EVENTS;
@@ -907,8 +944,10 @@ err:
static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress)
{
+ enum tracecmd_section_flags flags = 0;
unsigned int size, check_size, endian4;
const char *path = "/proc/kallsyms";
+ tsize_t offset;
struct stat st;
int ret;
@@ -921,6 +960,13 @@ static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress)
if (handle->kallsyms)
path = handle->kallsyms;
+ if (compress)
+ flags |= TRACECMD_SEC_FL_COMPRESS;
+ offset = out_write_section_header(handle, TRACECMD_OPTION_KALLSYMS,
+ "kallsyms", flags, true);
+ if (offset == (off64_t)-1)
+ return -1;
+
if (compress)
out_compression_start(handle);
ret = stat(path, &st);
@@ -953,6 +999,7 @@ static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress)
if (ret)
goto out;
}
+ ret = out_update_section_header(handle, offset);
out:
if (!ret)
handle->file_state = TRACECMD_FILE_KALLSYMS;
@@ -963,7 +1010,9 @@ out:
static int read_ftrace_printk(struct tracecmd_output *handle, bool compress)
{
+ enum tracecmd_section_flags flags = 0;
unsigned int size, check_size, endian4;
+ tsize_t offset;
struct stat st;
char *path;
int ret;
@@ -978,6 +1027,12 @@ static int read_ftrace_printk(struct tracecmd_output *handle, bool compress)
if (!path)
return -1;
+ if (compress)
+ flags |= TRACECMD_SEC_FL_COMPRESS;
+ offset = out_write_section_header(handle, TRACECMD_OPTION_PRINTK, "printk", flags, true);
+ if (offset == (off64_t)-1)
+ return -1;
+
if (compress)
out_compression_start(handle);
ret = stat(path, &st);
@@ -1004,6 +1059,8 @@ static int read_ftrace_printk(struct tracecmd_output *handle, bool compress)
put_tracing_file(path);
if (compress && out_compression_end(handle))
return -1;
+ if (out_update_section_header(handle, offset))
+ return -1;
handle->file_state = TRACECMD_FILE_PRINTK;
return 0;
fail:
@@ -1636,6 +1693,8 @@ tracecmd_add_buffer_option(struct tracecmd_output *handle, const char *name,
int tracecmd_write_cmdlines(struct tracecmd_output *handle)
{
+ enum tracecmd_section_flags flags = 0;
+ tsize_t offset;
int ret;
if (!check_out_state(handle, TRACECMD_FILE_CMD_LINES)) {
@@ -1644,6 +1703,13 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle)
return -1;
}
+ if (handle->compress)
+ flags |= TRACECMD_SEC_FL_COMPRESS;
+ offset = out_write_section_header(handle, TRACECMD_OPTION_CMDLINES,
+ "command lines", flags, true);
+ if (offset == (off_t)-1)
+ return -1;
+
if (handle->compress)
out_compression_start(handle);
@@ -1656,6 +1722,9 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle)
if (handle->compress && out_compression_end(handle))
return -1;
+ if (out_update_section_header(handle, offset))
+ return -1;
+
handle->file_state = TRACECMD_FILE_CMD_LINES;
return 0;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 23/87] trace-cmd library: Refactor the logic for writing trace data in the file
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (21 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 22/87] trace-cmd library: Write header " Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 24/87] trace-cmd library: Add local helper function for data compression Tzvetomir Stoyanov (VMware)
` (63 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When a trace buffer data are written in the trace file, the buffer
option in the file metadata is updated with the file offset of the
tracing data. Hide this logic into the trace-cmd library.
Added new APIs:
tracecmd_add_buffer_info()
tracecmd_write_buffer_info()
Changed APIs:
tracecmd_append_buffer_cpu_data()
Removed APIs:
tracecmd_add_buffer_option()
Refactored the internal logic of tracecmd_write_cpu_data() API to be
suitable for adding trace data compression. The size and the offset
of the trace data is saved in the file right after the data is written.
The old logic calculates the size and offset in advance, but when the
trace data is compressed it is hard to use that approach.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 10 +-
lib/trace-cmd/include/trace-cmd-local.h | 16 +
lib/trace-cmd/trace-output.c | 314 ++++++++++++------
tracecmd/trace-listen.c | 2 +-
tracecmd/trace-record.c | 18 +-
5 files changed, 240 insertions(+), 120 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index b193d6de..4e718c40 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -305,8 +305,8 @@ struct tracecmd_option *
tracecmd_add_option_v(struct tracecmd_output *handle,
unsigned short id, const struct iovec *vector, int count);
-struct tracecmd_option *tracecmd_add_buffer_option(struct tracecmd_output *handle,
- const char *name, int cpus);
+int tracecmd_add_buffer_info(struct tracecmd_output *handle, const char *name, int cpus);
+int tracecmd_write_buffer_info(struct tracecmd_output *handle);
int tracecmd_write_cpus(struct tracecmd_output *handle, int cpus);
int tracecmd_write_cmdlines(struct tracecmd_output *handle);
@@ -318,13 +318,11 @@ struct tracecmd_output *tracecmd_copy(struct tracecmd_input *ihandle,
const char *file);
int tracecmd_write_cpu_data(struct tracecmd_output *handle,
- int cpus, char * const *cpu_data_files);
+ int cpus, char * const *cpu_data_files, const char *buff_name);
int tracecmd_append_cpu_data(struct tracecmd_output *handle,
int cpus, char * const *cpu_data_files);
int tracecmd_append_buffer_cpu_data(struct tracecmd_output *handle,
- struct tracecmd_option *option,
- int cpus, char * const *cpu_data_files);
-
+ const char *name, int cpus, char * const *cpu_data_files);
struct tracecmd_output *tracecmd_get_output_handle_fd(int fd);
unsigned long tracecmd_get_out_file_version(struct tracecmd_output *handle);
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 2a23d9e0..e54ba7e1 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -24,6 +24,14 @@ void tracecmd_info(const char *fmt, ...);
#endif
#endif
+struct data_file_write {
+ unsigned long long file_size;
+ unsigned long long write_size;
+ unsigned long long soffset;
+ unsigned long long data_offset;
+ unsigned long long doffset;
+};
+
void tracecmd_compress_init(void);
void tracecmd_compress_free(void);
@@ -47,6 +55,14 @@ out_write_section_header(struct tracecmd_output *handle, unsigned short header_i
char *description, enum tracecmd_section_flags flags, bool option);
int out_update_section_header(struct tracecmd_output *handle, unsigned long long offset);
+struct cpu_data_source {
+ int fd;
+ int size;
+ off64_t offset;
+};
+
+int out_write_cpu_data(struct tracecmd_output *handle, int cpus,
+ struct cpu_data_source *data, const char *buff_name);
off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off_t offset, int whence);
#endif /* _TRACE_CMD_LOCAL_H */
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index a5c11eba..6a44a99b 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -39,6 +39,14 @@ struct tracecmd_option {
struct list_head list;
};
+struct tracecmd_buffer {
+ int cpus;
+ void *name;
+ tsize_t offset;
+ struct tracecmd_option *option;
+ struct list_head list;
+};
+
enum {
OUTPUT_FL_SEND_META = (1 << 0),
};
@@ -61,6 +69,7 @@ struct tracecmd_output {
struct tracecmd_compression *compress;
struct list_head options;
+ struct list_head buffers;
struct tracecmd_msg_handle *msg_handle;
char *trace_clock;
};
@@ -201,6 +210,7 @@ bool tracecmd_get_quiet(struct tracecmd_output *handle)
void tracecmd_output_free(struct tracecmd_output *handle)
{
struct tracecmd_option *option;
+ struct tracecmd_buffer *buffer;
if (!handle)
return;
@@ -211,6 +221,13 @@ void tracecmd_output_free(struct tracecmd_output *handle)
if (handle->pevent)
tep_unref(handle->pevent);
+ while (!list_empty(&handle->buffers)) {
+ buffer = container_of(handle->buffers.next,
+ struct tracecmd_buffer, list);
+ list_del(&buffer->list);
+ free(buffer->name);
+ free(buffer);
+ }
while (!list_empty(&handle->options)) {
option = container_of(handle->options.next,
struct tracecmd_option, list);
@@ -1158,6 +1175,7 @@ struct tracecmd_output *tracecmd_output_allocate(int fd)
handle->big_endian = false;
list_head_init(&handle->options);
+ list_head_init(&handle->buffers);
handle->file_state = TRACECMD_FILE_ALLOCATED;
@@ -1661,15 +1679,14 @@ int tracecmd_append_options(struct tracecmd_output *handle)
return 0;
}
-struct tracecmd_option *
-tracecmd_add_buffer_option(struct tracecmd_output *handle, const char *name,
- int cpus)
+static struct tracecmd_option *
+add_buffer_option(struct tracecmd_output *handle, const char *name, int cpus)
{
struct tracecmd_option *option;
char *buf;
int size = 8 + strlen(name) + 1;
- buf = malloc(size);
+ buf = calloc(1, size);
if (!buf) {
tracecmd_warning("Failed to malloc buffer");
return NULL;
@@ -1691,6 +1708,52 @@ tracecmd_add_buffer_option(struct tracecmd_output *handle, const char *name,
return option;
}
+int tracecmd_add_buffer_info(struct tracecmd_output *handle, const char *name, int cpus)
+{
+ struct tracecmd_buffer *buf;
+
+ buf = calloc(1, sizeof(struct tracecmd_buffer));
+ if (!buf)
+ return -1;
+ buf->name = strdup(name);
+ buf->cpus = cpus;
+ if (!buf->name) {
+ free(buf);
+ return -1;
+ }
+ list_add_tail(&buf->list, &handle->buffers);
+ return 0;
+}
+
+int tracecmd_write_buffer_info(struct tracecmd_output *handle)
+{
+ struct tracecmd_option *option;
+ struct tracecmd_buffer *buf;
+
+ list_for_each_entry(buf, &handle->buffers, list) {
+ option = add_buffer_option(handle, buf->name, buf->cpus);
+ if (!option)
+ return -1;
+ buf->option = option;
+ }
+
+ return 0;
+}
+
+static tsize_t get_buffer_file_offset(struct tracecmd_output *handle, const char *name)
+{
+ struct tracecmd_buffer *buf;
+
+ list_for_each_entry(buf, &handle->buffers, list) {
+ if (strlen(name) == strlen(buf->name) && !strcmp(name, buf->name)) {
+ if (!buf->option)
+ break;
+ return buf->option->offset;
+ }
+ }
+ return 0;
+}
+
int tracecmd_write_cmdlines(struct tracecmd_output *handle)
{
enum tracecmd_section_flags flags = 0;
@@ -1808,6 +1871,37 @@ out:
return ret;
}
+static int update_buffer_cpu_offset(struct tracecmd_output *handle,
+ const char *name, tsize_t offset)
+{
+ tsize_t b_offset;
+ tsize_t current;
+
+ b_offset = get_buffer_file_offset(handle, name);
+ if (!b_offset) {
+ tracecmd_warning("Cannot find description for buffer %s\n", name);
+ return -1;
+ }
+ current = do_lseek(handle, 0, SEEK_CUR);
+
+ /* Go to the option data, where will write the offest */
+ if (do_lseek(handle, b_offset, SEEK_SET) == (off64_t)-1) {
+ tracecmd_warning("could not seek to %lld\n", b_offset);
+ return -1;
+ }
+
+ if (do_write_check(handle, &offset, 8))
+ return -1;
+
+ /* Go back to end of file */
+ if (do_lseek(handle, current, SEEK_SET) == (off64_t)-1) {
+ tracecmd_warning("could not seek to %lld\n", offset);
+ return -1;
+ }
+ return 0;
+}
+
+
static char *get_clock(struct tracecmd_output *handle)
{
struct tracefs_instance *inst;
@@ -1827,120 +1921,158 @@ static char *get_clock(struct tracecmd_output *handle)
return handle->trace_clock;
}
-int tracecmd_write_cpu_data(struct tracecmd_output *handle,
- int cpus, char * const *cpu_data_files)
+
+__hidden int out_write_cpu_data(struct tracecmd_output *handle,
+ int cpus, struct cpu_data_source *data, const char *buff_name)
{
- off64_t *offsets = NULL;
- unsigned long long *sizes = NULL;
- off64_t offset;
+ struct data_file_write *data_files = NULL;
+ tsize_t data_offs, offset;
unsigned long long endian8;
- char *clock = NULL;
- off64_t check_size;
- char *file;
- struct stat st;
+ unsigned long long read_size;
+ char *clock;
int ret;
int i;
/* This can be called multiple times (when recording instances) */
ret = handle->file_state == TRACECMD_FILE_CPU_FLYRECORD ? 0 :
- check_out_state(handle, TRACECMD_FILE_CPU_FLYRECORD);
+ check_file_state(handle->file_version,
+ handle->file_state,
+ TRACECMD_FILE_CPU_FLYRECORD);
if (ret < 0) {
tracecmd_warning("Cannot write trace data into the file, unexpected state 0x%X",
handle->file_state);
goto out_free;
}
+ data_offs = do_lseek(handle, 0, SEEK_CUR);
if (do_write_check(handle, "flyrecord", 10))
goto out_free;
- offsets = malloc(sizeof(*offsets) * cpus);
- if (!offsets)
+ data_files = calloc(cpus, sizeof(struct data_file_write));
+ if (!data_files)
goto out_free;
- sizes = malloc(sizeof(*sizes) * cpus);
- if (!sizes)
- goto out_free;
-
- offset = lseek64(handle->fd, 0, SEEK_CUR);
- /* hold any extra data for data */
- offset += cpus * (16);
+ for (i = 0; i < cpus; i++) {
+ data_files[i].file_size = data[i].size;
+ /* Write 0 for trace data offset and size and store offsets of these fields */
+ if (handle->file_version < 7) {
+ endian8 = 0;
+ data_files[i].doffset = do_lseek(handle, 0, SEEK_CUR);
+ if (do_write_check(handle, &endian8, 8))
+ goto out_free;
+ data_files[i].soffset = do_lseek(handle, 0, SEEK_CUR);
+ if (do_write_check(handle, &endian8, 8))
+ goto out_free;
+ }
+ }
- /*
- * Unfortunately, the trace_clock data was placed after the
- * cpu data, and wasn't accounted for with the offsets.
- * We need to save room for the trace_clock file. This means
- * we need to find the size of it before we define the final
- * offsets.
- */
+ update_buffer_cpu_offset(handle, buff_name, data_offs);
clock = get_clock(handle);
- if (!clock)
+ if (clock && save_clock(handle, clock))
goto out_free;
- /* Save room for storing the size */
- offset += 8;
- offset += strlen(clock);
- /* 2 bytes for [] around the clock */
- offset += 2;
-
- /* Page align offset */
- offset = (offset + (handle->page_size - 1)) & ~(handle->page_size - 1);
for (i = 0; i < cpus; i++) {
- file = cpu_data_files[i];
- ret = stat(file, &st);
- if (ret < 0) {
- tracecmd_warning("can not stat '%s'", file);
+ data_files[i].data_offset = do_lseek(handle, 0, SEEK_CUR);
+ /* Page align offset */
+ data_files[i].data_offset = (data_files[i].data_offset + (handle->page_size - 1)) & ~(handle->page_size - 1);
+ data_files[i].data_offset = do_lseek(handle, data_files[i].data_offset, SEEK_SET);
+ if (data_files[i].data_offset == (off64_t)-1)
+ goto out_free;
+ if (!tracecmd_get_quiet(handle))
+ fprintf(stderr, "CPU%d data recorded at offset=0x%llx\n",
+ i, (unsigned long long) data_files[i].data_offset);
+ if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1)
goto out_free;
+ if (data[i].size) {
+ read_size = copy_file_fd(handle, data[i].fd);
+ if (read_size != data_files[i].file_size) {
+ errno = EINVAL;
+ tracecmd_warning("did not match size of %lld to %lld",
+ read_size, data_files[i].file_size);
+ goto out_free;
+ }
+ } else {
+ data_files[i].write_size = 0;
}
- offsets[i] = offset;
- sizes[i] = st.st_size;
- offset += st.st_size;
- offset = (offset + (handle->page_size - 1)) & ~(handle->page_size - 1);
- endian8 = convert_endian_8(handle, offsets[i]);
- if (do_write_check(handle, &endian8, 8))
+ /* Write the real CPU data offset in the file */
+ if (do_lseek(handle, data_files[i].doffset, SEEK_SET) == (off64_t)-1)
goto out_free;
- endian8 = convert_endian_8(handle, sizes[i]);
+ endian8 = convert_endian_8(handle, data_files[i].data_offset);
if (do_write_check(handle, &endian8, 8))
goto out_free;
- }
-
- if (save_clock(handle, clock))
- goto out_free;
-
- for (i = 0; i < cpus; i++) {
- if (!tracecmd_get_quiet(handle))
- fprintf(stderr, "CPU%d data recorded at offset=0x%llx\n",
- i, (unsigned long long) offsets[i]);
- offset = lseek64(handle->fd, offsets[i], SEEK_SET);
- if (offset == (off64_t)-1) {
- tracecmd_warning("could not seek to %lld\n", offsets[i]);
+ /* Write the real CPU data size in the file */
+ if (do_lseek(handle, data_files[i].soffset, SEEK_SET) == (off64_t)-1)
goto out_free;
- }
- check_size = copy_file(handle, cpu_data_files[i]);
- if (check_size != sizes[i]) {
- errno = EINVAL;
- tracecmd_warning("did not match size of %lld to %lld",
- check_size, sizes[i]);
+ endian8 = convert_endian_8(handle, data_files[i].write_size);
+ if (do_write_check(handle, &endian8, 8))
+ goto out_free;
+ offset = data_files[i].data_offset + data_files[i].write_size;
+ if (do_lseek(handle, offset, SEEK_SET) == (off64_t)-1)
goto out_free;
- }
if (!tracecmd_get_quiet(handle))
fprintf(stderr, " %llu bytes in size\n",
- (unsigned long long)check_size);
+ (unsigned long long)data_files[i].write_size);
}
- free(offsets);
- free(sizes);
+ if (do_lseek(handle, 0, SEEK_END) == (off64_t)-1)
+ goto out_free;
+ free(data_files);
handle->file_state = TRACECMD_FILE_CPU_FLYRECORD;
return 0;
out_free:
- free(offsets);
- free(sizes);
+ do_lseek(handle, 0, SEEK_END);
+ free(data_files);
return -1;
}
+int tracecmd_write_cpu_data(struct tracecmd_output *handle,
+ int cpus, char * const *cpu_data_files, const char *buff_name)
+{
+ struct cpu_data_source *data;
+ struct stat st;
+ int size = 0;
+ int ret;
+ int i;
+
+ data = calloc(cpus, sizeof(struct cpu_data_source));
+ if (!data)
+ return -1;
+ for (i = 0; i < cpus; i++)
+ data[i].fd = -1;
+ for (i = 0; i < cpus; i++) {
+ ret = stat(cpu_data_files[i], &st);
+ if (ret < 0) {
+ tracecmd_warning("can not stat '%s'", cpu_data_files[i]);
+ break;
+ }
+ data[i].fd = open(cpu_data_files[i], O_RDONLY);
+ if (data[i].fd < 0) {
+ tracecmd_warning("Can't read '%s'", data[i].fd);
+ break;
+ }
+
+ data[i].size = st.st_size;
+ data[i].offset = 0;
+ size += st.st_size;
+ }
+
+ if (i < cpus)
+ ret = -1;
+ else
+ ret = out_write_cpu_data(handle, cpus, data, buff_name);
+
+ for (i = 0; i < cpus; i++) {
+ if (data[i].fd >= 0)
+ close(data[i].fd);
+ }
+ free(data);
+ return ret;
+}
+
int tracecmd_append_cpu_data(struct tracecmd_output *handle,
int cpus, char * const *cpu_data_files)
{
@@ -1949,41 +2081,20 @@ int tracecmd_append_cpu_data(struct tracecmd_output *handle,
ret = tracecmd_write_cpus(handle, cpus);
if (ret)
return ret;
-
+ ret = tracecmd_write_buffer_info(handle);
+ if (ret)
+ return ret;
ret = tracecmd_write_options(handle);
if (ret)
return ret;
- return tracecmd_write_cpu_data(handle, cpus, cpu_data_files);
+ return tracecmd_write_cpu_data(handle, cpus, cpu_data_files, "");
}
int tracecmd_append_buffer_cpu_data(struct tracecmd_output *handle,
- struct tracecmd_option *option,
- int cpus, char * const *cpu_data_files)
+ const char *name, int cpus, char * const *cpu_data_files)
{
- tsize_t offset;
- stsize_t ret;
-
- offset = lseek64(handle->fd, 0, SEEK_CUR);
-
- /* Go to the option data, where will write the offest */
- ret = lseek64(handle->fd, option->offset, SEEK_SET);
- if (ret == (off64_t)-1) {
- tracecmd_warning("could not seek to %lld\n", option->offset);
- return -1;
- }
-
- if (do_write_check(handle, &offset, 8))
- return -1;
-
- /* Go back to end of file */
- ret = lseek64(handle->fd, offset, SEEK_SET);
- if (ret == (off64_t)-1) {
- tracecmd_warning("could not seek to %lld\n", offset);
- return -1;
- }
-
- return tracecmd_write_cpu_data(handle, cpus, cpu_data_files);
+ return tracecmd_write_cpu_data(handle, cpus, cpu_data_files, name);
}
struct tracecmd_output *tracecmd_get_output_handle_fd(int fd)
@@ -2029,6 +2140,7 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd)
handle->file_version = tracecmd_get_in_file_version(ihandle);
handle->options_start = tracecmd_get_options_offset(ihandle);
list_head_init(&handle->options);
+ list_head_init(&handle->buffers);
if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) {
handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd,
diff --git a/tracecmd/trace-listen.c b/tracecmd/trace-listen.c
index 0cb70b7d..d812145b 100644
--- a/tracecmd/trace-listen.c
+++ b/tracecmd/trace-listen.c
@@ -610,7 +610,7 @@ static int put_together_file(int cpus, int ofd, const char *node,
if (ret)
goto out;
}
- ret = tracecmd_write_cpu_data(handle, cpus, temp_files);
+ ret = tracecmd_write_cpu_data(handle, cpus, temp_files, "");
out:
tracecmd_output_close(handle);
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index 295fe633..350d2811 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -4187,7 +4187,6 @@ static void touch_file(const char *file)
}
static void append_buffer(struct tracecmd_output *handle,
- struct tracecmd_option *buffer_option,
struct buffer_instance *instance,
char **temp_files)
{
@@ -4215,7 +4214,7 @@ static void append_buffer(struct tracecmd_output *handle,
touch_file(temp_files[i]);
}
- tracecmd_append_buffer_cpu_data(handle, buffer_option,
+ tracecmd_append_buffer_cpu_data(handle, tracefs_instance_get_name(instance->tracefs),
cpu_count, temp_files);
for (i = 0; i < instance->cpu_count; i++) {
@@ -4465,7 +4464,7 @@ static void write_guest_file(struct buffer_instance *instance)
die("failed to allocate memory");
}
- if (tracecmd_write_cpu_data(handle, cpu_count, temp_files) < 0)
+ if (tracecmd_write_cpu_data(handle, cpu_count, temp_files, "") < 0)
die("failed to write CPU data");
tracecmd_output_close(handle);
@@ -4508,7 +4507,6 @@ error:
static void record_data(struct common_record_context *ctx)
{
- struct tracecmd_option **buffer_options;
struct tracecmd_output *handle;
struct buffer_instance *instance;
bool local = false;
@@ -4578,9 +4576,6 @@ static void record_data(struct common_record_context *ctx)
}
if (buffers) {
- buffer_options = malloc(sizeof(*buffer_options) * buffers);
- if (!buffer_options)
- die("Failed to allocate buffer options");
i = 0;
for_each_instance(instance) {
int cpus = instance->cpu_count != local_cpu_count ?
@@ -4588,10 +4583,9 @@ static void record_data(struct common_record_context *ctx)
if (instance->msg_handle)
continue;
-
- buffer_options[i++] = tracecmd_add_buffer_option(handle,
- tracefs_instance_get_name(instance->tracefs),
- cpus);
+ tracecmd_add_buffer_info(handle,
+ tracefs_instance_get_name(instance->tracefs),
+ cpus);
add_buffer_stat(handle, instance);
}
}
@@ -4626,7 +4620,7 @@ static void record_data(struct common_record_context *ctx)
if (instance->msg_handle)
continue;
print_stat(instance);
- append_buffer(handle, buffer_options[i++], instance, temp_files);
+ append_buffer(handle, instance, temp_files);
}
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 24/87] trace-cmd library: Add local helper function for data compression
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (22 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 23/87] trace-cmd library: Refactor the logic for writing trace data in the file Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-08-17 14:53 ` Steven Rostedt
2021-07-29 5:08 ` [PATCH v2 25/87] trace-cmd library: Compress the trace data Tzvetomir Stoyanov (VMware)
` (62 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The newly added helper functions read data from a file and compress it,
before writing into the trace file. The trace data is comressed in
chunks, which are page aligned. A new local define is introduced:
PAGES_IN_CHUNK
which can be used to tune how big a compression chunk is.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 70 ++++++++++++++++++++++++++++++++----
1 file changed, 64 insertions(+), 6 deletions(-)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 6a44a99b..90625c4e 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -285,18 +285,26 @@ static unsigned long get_size(const char *file)
return size;
}
-static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd)
+static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd, unsigned long long max)
{
+ tsize_t rsize = 0;
tsize_t size = 0;
char buf[BUFSIZ];
stsize_t r;
do {
- r = read(fd, buf, BUFSIZ);
+ if (max > 0 && (max - size) < BUFSIZ)
+ rsize = (max - size);
+ else
+ rsize = BUFSIZ;
+
+ r = read(fd, buf, rsize);
if (r > 0) {
size += r;
if (do_write_check(handle, buf, r))
return 0;
+ if (max > 0 && size >= max)
+ break;
}
} while (r > 0);
@@ -314,12 +322,62 @@ static tsize_t copy_file(struct tracecmd_output *handle,
tracecmd_warning("Can't read '%s'", file);
return 0;
}
- size = copy_file_fd(handle, fd);
+ size = copy_file_fd(handle, fd, 0);
close(fd);
return size;
}
+#define PAGES_IN_CHUNK 10
+__hidden unsigned long long out_copy_fd_compress(struct tracecmd_output *handle,
+ int fd, unsigned long long max,
+ unsigned long long *write_size)
+{
+ unsigned long long rsize = 0;
+ unsigned long long wsize = 0;
+ unsigned long long size;
+ int ret;
+
+ if (handle->compress) {
+ rsize = max;
+ ret = tracecmd_compress_copy_from(handle->compress, fd,
+ PAGES_IN_CHUNK * handle->page_size,
+ &rsize, &wsize);
+ if (ret < 0)
+ return 0;
+
+ size = rsize;
+ if (write_size)
+ *write_size = wsize;
+ } else {
+ size = copy_file_fd(handle, fd, max);
+ if (write_size)
+ *write_size = size;
+ }
+
+ return size;
+}
+
+static tsize_t copy_file_compress(struct tracecmd_output *handle,
+ const char *file, unsigned long long *write_size)
+{
+ int ret;
+ int fd;
+
+ fd = open(file, O_RDONLY);
+ if (fd < 0) {
+ tracecmd_warning("Can't read '%s'", file);
+ return 0;
+ }
+
+ ret = out_copy_fd_compress(handle, fd, 0, write_size);
+ if (!ret)
+ tracecmd_warning("Can't compress '%s'", file);
+
+ close(fd);
+ return ret;
+}
+
/*
* Finds the path to the debugfs/tracing
* Allocates the string and stores it.
@@ -516,7 +574,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress)
endian8 = convert_endian_8(handle, size);
if (do_write_check(handle, &endian8, 8))
goto out_close;
- check_size = copy_file_fd(handle, fd);
+ check_size = copy_file_fd(handle, fd, 0);
close(fd);
if (size != check_size) {
tracecmd_warning("wrong size for '%s' size=%lld read=%lld", path, size, check_size);
@@ -542,7 +600,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress)
endian8 = convert_endian_8(handle, size);
if (do_write_check(handle, &endian8, 8))
goto out_close;
- check_size = copy_file_fd(handle, fd);
+ check_size = copy_file_fd(handle, fd, 0);
close(fd);
if (size != check_size) {
tracecmd_warning("wrong size for '%s'", path);
@@ -1984,7 +2042,7 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle,
if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1)
goto out_free;
if (data[i].size) {
- read_size = copy_file_fd(handle, data[i].fd);
+ read_size = copy_file_fd(handle, data[i].fd, data[i].size);
if (read_size != data_files[i].file_size) {
errno = EINVAL;
tracecmd_warning("did not match size of %lld to %lld",
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 24/87] trace-cmd library: Add local helper function for data compression
2021-07-29 5:08 ` [PATCH v2 24/87] trace-cmd library: Add local helper function for data compression Tzvetomir Stoyanov (VMware)
@ 2021-08-17 14:53 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-17 14:53 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:08:56 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> The newly added helper functions read data from a file and compress it,
> before writing into the trace file. The trace data is comressed in
> chunks, which are page aligned. A new local define is introduced:
> PAGES_IN_CHUNK
> which can be used to tune how big a compression chunk is.
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-output.c | 70 ++++++++++++++++++++++++++++++++----
> 1 file changed, 64 insertions(+), 6 deletions(-)
>
> diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
> index 6a44a99b..90625c4e 100644
> --- a/lib/trace-cmd/trace-output.c
> +++ b/lib/trace-cmd/trace-output.c
> @@ -285,18 +285,26 @@ static unsigned long get_size(const char *file)
> return size;
> }
>
> -static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd)
> +static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd, unsigned long long max)
> {
> + tsize_t rsize = 0;
> tsize_t size = 0;
> char buf[BUFSIZ];
> stsize_t r;
>
> do {
> - r = read(fd, buf, BUFSIZ);
> + if (max > 0 && (max - size) < BUFSIZ)
> + rsize = (max - size);
> + else
> + rsize = BUFSIZ;
> +
> + r = read(fd, buf, rsize);
> if (r > 0) {
> size += r;
> if (do_write_check(handle, buf, r))
> return 0;
> + if (max > 0 && size >= max)
> + break;
> }
> } while (r > 0);
I think this would be a bit cleaner:
static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd, unsigned long long max)
{
tsize_t rsize = BUFSIZ;
tsize_t size = 0;
char buf[BUFSIZ];
stsize_t r;
do {
if (max && rsize > max)
rsize = max;
r = read(fd, buf, rsize);
if (r > 0) {
size += r;
if (do_write_check(handle, buf, r))
return 0;
if (max) {
max -= r;
if (!max)
break;
}
}
} while (r > 0);
return size;
}
>
> @@ -314,12 +322,62 @@ static tsize_t copy_file(struct tracecmd_output *handle,
> tracecmd_warning("Can't read '%s'", file);
> return 0;
> }
> - size = copy_file_fd(handle, fd);
> + size = copy_file_fd(handle, fd, 0);
> close(fd);
>
> return size;
> }
>
> +#define PAGES_IN_CHUNK 10
> +__hidden unsigned long long out_copy_fd_compress(struct tracecmd_output *handle,
The above name is also hard to understand. "out_copy_fd_compress"?
Would "copy_out_fd_compress()" be better?
Also, why is it __hidden and not static. it's not used outside this file.
If it gets used outside this file in a follow up patch, please convert
it from static to __hidden then.
-- Steve
> + int fd, unsigned long long max,
> + unsigned long long *write_size)
> +{
> + unsigned long long rsize = 0;
> + unsigned long long wsize = 0;
> + unsigned long long size;
> + int ret;
> +
> + if (handle->compress) {
> + rsize = max;
> + ret = tracecmd_compress_copy_from(handle->compress, fd,
> + PAGES_IN_CHUNK * handle->page_size,
> + &rsize, &wsize);
> + if (ret < 0)
> + return 0;
> +
> + size = rsize;
> + if (write_size)
> + *write_size = wsize;
> + } else {
> + size = copy_file_fd(handle, fd, max);
> + if (write_size)
> + *write_size = size;
> + }
> +
> + return size;
> +}
> +
> +static tsize_t copy_file_compress(struct tracecmd_output *handle,
> + const char *file, unsigned long long *write_size)
> +{
> + int ret;
> + int fd;
> +
> + fd = open(file, O_RDONLY);
> + if (fd < 0) {
> + tracecmd_warning("Can't read '%s'", file);
> + return 0;
> + }
> +
> + ret = out_copy_fd_compress(handle, fd, 0, write_size);
> + if (!ret)
> + tracecmd_warning("Can't compress '%s'", file);
> +
> + close(fd);
> + return ret;
> +}
> +
> /*
> * Finds the path to the debugfs/tracing
> * Allocates the string and stores it.
> @@ -516,7 +574,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress)
> endian8 = convert_endian_8(handle, size);
> if (do_write_check(handle, &endian8, 8))
> goto out_close;
> - check_size = copy_file_fd(handle, fd);
> + check_size = copy_file_fd(handle, fd, 0);
> close(fd);
> if (size != check_size) {
> tracecmd_warning("wrong size for '%s' size=%lld read=%lld", path, size, check_size);
> @@ -542,7 +600,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress)
> endian8 = convert_endian_8(handle, size);
> if (do_write_check(handle, &endian8, 8))
> goto out_close;
> - check_size = copy_file_fd(handle, fd);
> + check_size = copy_file_fd(handle, fd, 0);
> close(fd);
> if (size != check_size) {
> tracecmd_warning("wrong size for '%s'", path);
> @@ -1984,7 +2042,7 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle,
> if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1)
> goto out_free;
> if (data[i].size) {
> - read_size = copy_file_fd(handle, data[i].fd);
> + read_size = copy_file_fd(handle, data[i].fd, data[i].size);
> if (read_size != data_files[i].file_size) {
> errno = EINVAL;
> tracecmd_warning("did not match size of %lld to %lld",
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 25/87] trace-cmd library: Compress the trace data
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (23 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 24/87] trace-cmd library: Add local helper function for data compression Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 26/87] tarce-cmd library: Add multiple options sections in trace file version 7 Tzvetomir Stoyanov (VMware)
` (61 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
If the output file handler supports compression, use it to compress the
flyrecord and latency trace data.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 90625c4e..e232a678 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -1895,7 +1895,7 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in
if (!path)
goto out_free;
- copy_file(handle, path);
+ copy_file_compress(handle, path, NULL);
put_tracing_file(path);
@@ -2042,7 +2042,8 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle,
if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1)
goto out_free;
if (data[i].size) {
- read_size = copy_file_fd(handle, data[i].fd, data[i].size);
+ read_size = out_copy_fd_compress(handle, data[i].fd,
+ data[i].size, &data_files[i].write_size);
if (read_size != data_files[i].file_size) {
errno = EINVAL;
tracecmd_warning("did not match size of %lld to %lld",
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 26/87] tarce-cmd library: Add multiple options sections in trace file version 7
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (24 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 25/87] trace-cmd library: Compress the trace data Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:08 ` [PATCH v2 27/87] trace-cmd library: Do not write CPU count section in trace files " Tzvetomir Stoyanov (VMware)
` (60 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Introduced chain of options sections in trace file version 7. Extended
the "DONE" option to hold the offset into the file to the next options
section.
Format of trace file version 7 is extended with a new mandatory field
after the compression algorithm header:
<8 bytes>, unsigned long long integer - offset into the trace file
where the first options section is located. This allows to place this
section anywhere in the file. As all other sections have corresponding
options, describing their offsets into the trace file, this change makes
the structure of trace file version 7 flexible.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 110 ++++++++++++++++++++++++++++++-----
1 file changed, 96 insertions(+), 14 deletions(-)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index e232a678..6bb7f4de 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -63,7 +63,7 @@ struct tracecmd_output {
bool quiet;
unsigned long file_state;
unsigned long file_version;
- size_t options_start;
+ tsize_t options_start;
bool big_endian;
bool do_compress;
struct tracecmd_compression *compress;
@@ -1453,6 +1453,7 @@ int tracecmd_output_set_compression(struct tracecmd_output *handler, const char
*/
int tracecmd_output_write_init(struct tracecmd_output *handler)
{
+ unsigned long long offset;
char buf[BUFSIZ];
int endian4;
@@ -1489,6 +1490,11 @@ int tracecmd_output_write_init(struct tracecmd_output *handler)
if (handler->file_version >= 7) {
if (write_compression_header(handler))
return -1;
+ /* Write 0 as options offset and save its location */
+ offset = 0;
+ handler->options_start = do_lseek(handler, 0, SEEK_CUR);
+ if (do_write_check(handler, &offset, 8))
+ return -1;
}
handler->file_state = TRACECMD_FILE_INIT;
@@ -1563,7 +1569,7 @@ tracecmd_add_option_v(struct tracecmd_output *handle,
* We can only add options before tracing data were written.
* This may change in the future.
*/
- if (handle->file_state > TRACECMD_FILE_OPTIONS)
+ if (handle->file_version < 7 && handle->file_state > TRACECMD_FILE_OPTIONS)
return NULL;
for (i = 0; i < count; i++)
@@ -1576,8 +1582,7 @@ tracecmd_add_option_v(struct tracecmd_output *handle,
return NULL;
}
}
-
- option = malloc(sizeof(*option));
+ option = calloc(1, sizeof(*option));
if (!option) {
tracecmd_warning("Could not allocate space for option");
free(data);
@@ -1640,7 +1645,7 @@ int tracecmd_write_cpus(struct tracecmd_output *handle, int cpus)
return 0;
}
-int tracecmd_write_options(struct tracecmd_output *handle)
+static int write_options_v6(struct tracecmd_output *handle)
{
struct tracecmd_option *options;
unsigned short option;
@@ -1658,7 +1663,7 @@ int tracecmd_write_options(struct tracecmd_output *handle)
if (do_write_check(handle, "options ", 10))
return -1;
-
+ handle->options_start = do_lseek(handle, 0, SEEK_CUR);
list_for_each_entry(options, &handle->options, list) {
endian2 = convert_endian_2(handle, options->id);
if (do_write_check(handle, &endian2, 2))
@@ -1669,7 +1674,7 @@ int tracecmd_write_options(struct tracecmd_output *handle)
return -1;
/* Save the data location in case it needs to be updated */
- options->offset = lseek64(handle->fd, 0, SEEK_CUR);
+ options->offset = do_lseek(handle, 0, SEEK_CUR);
if (do_write_check(handle, options->data,
options->size))
@@ -1682,11 +1687,83 @@ int tracecmd_write_options(struct tracecmd_output *handle)
return -1;
handle->file_state = TRACECMD_FILE_OPTIONS;
+ return 0;
+}
+
+static int write_options_v7(struct tracecmd_output *handle)
+{
+ struct tracecmd_option *options;
+ unsigned long long endian8;
+ unsigned short endian2;
+ unsigned int endian4;
+ bool new = false;
+ tsize_t offset;
+
+ /* Check if there are unsaved options */
+ list_for_each_entry(options, &handle->options, list) {
+ if (!options->offset) {
+ new = true;
+ break;
+ }
+ }
+
+ if (!new)
+ return 0;
+ offset = do_lseek(handle, 0, SEEK_CUR);
+ /* Append to the previous options section, if any */
+ if (handle->options_start) {
+ if (do_lseek(handle, handle->options_start, SEEK_SET) == (off64_t)-1)
+ return -1;
+ endian8 = convert_endian_8(handle, offset);
+ if (do_write_check(handle, &endian8, 8))
+ return -1;
+ if (do_lseek(handle, offset, SEEK_SET) == (off_t)-1)
+ return -1;
+ }
+ offset = out_write_section_header(handle, TRACECMD_OPTION_DONE, "options", 0, false);
+ if (offset == (off_t)-1)
+ return -1;
+ list_for_each_entry(options, &handle->options, list) {
+ /* Option is already saved, skip it */
+ if (options->offset)
+ continue;
+ endian2 = convert_endian_2(handle, options->id);
+ if (do_write_check(handle, &endian2, 2))
+ return -1;
+ endian4 = convert_endian_4(handle, options->size);
+ if (do_write_check(handle, &endian4, 4))
+ return -1;
+ /* Save the data location */
+ options->offset = do_lseek(handle, 0, SEEK_CUR);
+ if (do_write_check(handle, options->data, options->size))
+ return -1;
+ }
+
+ endian2 = convert_endian_2(handle, TRACECMD_OPTION_DONE);
+ if (do_write_check(handle, &endian2, 2))
+ return -1;
+ endian4 = convert_endian_4(handle, 8);
+ if (do_write_check(handle, &endian4, 4))
+ return -1;
+ endian8 = 0;
+ handle->options_start = do_lseek(handle, 0, SEEK_CUR);
+ if (do_write_check(handle, &endian8, 8))
+ return -1;
+ if (out_update_section_header(handle, offset))
+ return -1;
return 0;
}
-int tracecmd_append_options(struct tracecmd_output *handle)
+int tracecmd_write_options(struct tracecmd_output *handle)
+{
+ if (handle->file_version < 7)
+ return write_options_v6(handle);
+ return write_options_v7(handle);
+}
+
+
+static int append_options_v6(struct tracecmd_output *handle)
{
struct tracecmd_option *options;
unsigned short option;
@@ -1701,14 +1778,12 @@ int tracecmd_append_options(struct tracecmd_output *handle)
*/
if (handle->file_state != TRACECMD_FILE_OPTIONS)
return -1;
-
- if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1)
+ if (do_lseek(handle, 0, SEEK_END) == -1)
return -1;
- offset = lseek64(handle->fd, -2, SEEK_CUR);
+ offset = do_lseek(handle, -2, SEEK_CUR);
if (offset == (off_t)-1)
return -1;
-
- r = pread(handle->fd, &option, 2, offset);
+ r = do_preed(handle, &option, 2, offset);
if (r != 2 || option != TRACECMD_OPTION_DONE)
return -1;
@@ -1722,7 +1797,7 @@ int tracecmd_append_options(struct tracecmd_output *handle)
return -1;
/* Save the data location in case it needs to be updated */
- options->offset = lseek64(handle->fd, 0, SEEK_CUR);
+ options->offset = do_lseek(handle, 0, SEEK_CUR);
if (do_write_check(handle, options->data,
options->size))
@@ -1737,6 +1812,13 @@ int tracecmd_append_options(struct tracecmd_output *handle)
return 0;
}
+int tracecmd_append_options(struct tracecmd_output *handle)
+{
+ if (handle->file_version < 7)
+ return append_options_v6(handle);
+ return write_options_v7(handle);
+}
+
static struct tracecmd_option *
add_buffer_option(struct tracecmd_output *handle, const char *name, int cpus)
{
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 27/87] trace-cmd library: Do not write CPU count section in trace files version 7
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (25 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 26/87] tarce-cmd library: Add multiple options sections in trace file version 7 Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:08 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 28/87] trace-cmd library: Move CPU flyrecord trace metadata into the buffer option, for trace file " Tzvetomir Stoyanov (VMware)
` (59 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:08 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Writing CPU count as a separate section in the trace file is redundant,
as there is already an option for that. Use that option in trace files
version 7.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 6bb7f4de..d60bd456 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -1637,10 +1637,16 @@ int tracecmd_write_cpus(struct tracecmd_output *handle, int cpus)
handle->file_state);
return -1;
}
- cpus = convert_endian_4(handle, cpus);
- ret = do_write_check(handle, &cpus, 4);
- if (ret < 0)
- return ret;
+
+ if (handle->file_version < 7) {
+ cpus = convert_endian_4(handle, cpus);
+ ret = do_write_check(handle, &cpus, 4);
+ if (ret < 0)
+ return ret;
+ } else {
+ tracecmd_add_option(handle, TRACECMD_OPTION_CPUCOUNT, sizeof(int), &cpus);
+ }
+
handle->file_state = TRACECMD_FILE_CPU_COUNT;
return 0;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 28/87] trace-cmd library: Move CPU flyrecord trace metadata into the buffer option, for trace file version 7
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (26 preceding siblings ...)
2021-07-29 5:08 ` [PATCH v2 27/87] trace-cmd library: Do not write CPU count section in trace files " Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-17 15:40 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 29/87] trace-cmd record: Append trace options after the trace data are written Tzvetomir Stoyanov (VMware)
` (58 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Extended the BUFFER trace option in trace file version 7 with CPU
flyrecord trace metadata. In the version 6 of the trace file, this metadata
is located at several places in the file. Part of the metadata is only for
the top trace instance, thus limiting per-instance configuration. Moving
all CPU trace related metadata in the BUFFER option simplifies the parsing
and makes per-instance configuration more flexible. In the new file
structure, the top instance is treated as any other instances.
The format of the extended BUFFER option is:
- offset of the buffer in the trace file
- name of the buffer
- trace clock, used in this buffer for events timestamps
- count of CPUs with trace data
- array, describing each CPU with trace data:
- CPU id
- offset of CPU trace data in the trace file
- size of the recorded CPU trace data
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 171 ++++++++++++++++++++++++++---------
1 file changed, 127 insertions(+), 44 deletions(-)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index d60bd456..21555d6d 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -1826,7 +1826,7 @@ int tracecmd_append_options(struct tracecmd_output *handle)
}
static struct tracecmd_option *
-add_buffer_option(struct tracecmd_output *handle, const char *name, int cpus)
+add_buffer_option_v6(struct tracecmd_output *handle, const char *name, int cpus)
{
struct tracecmd_option *option;
char *buf;
@@ -1876,8 +1876,11 @@ int tracecmd_write_buffer_info(struct tracecmd_output *handle)
struct tracecmd_option *option;
struct tracecmd_buffer *buf;
+ if (handle->file_version >= 7)
+ return 0;
+
list_for_each_entry(buf, &handle->buffers, list) {
- option = add_buffer_option(handle, buf->name, buf->cpus);
+ option = add_buffer_option_v6(handle, buf->name, buf->cpus);
if (!option)
return -1;
buf->option = option;
@@ -1938,6 +1941,98 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle)
return 0;
}
+static char *get_clock(struct tracecmd_output *handle)
+{
+ struct tracefs_instance *inst;
+
+ if (handle->trace_clock)
+ return handle->trace_clock;
+
+ /*
+ * If no clock is set on this handle, get the trace clock of
+ * the top instance in the handle's tracing dir
+ */
+ inst = tracefs_instance_alloc(handle->tracing_dir, NULL);
+ if (!inst)
+ return NULL;
+ handle->trace_clock = tracefs_get_clock(inst);
+ tracefs_instance_free(inst);
+ return handle->trace_clock;
+}
+
+__hidden struct tracecmd_option *
+out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name,
+ unsigned short id, unsigned long long data_offset,
+ int cpus, struct data_file_write *cpu_data)
+{
+ struct tracecmd_option *option;
+ int i, j = 0, k = 0;
+ int *cpu_ids = NULL;
+ struct iovec *vect;
+ char *clock;
+
+ if (handle->file_version < 7)
+ return NULL;
+
+ clock = get_clock(handle);
+
+ /* Buffer flyrecord option, v7:
+ * - trace data offset in the file
+ * - buffer name
+ * - buffer clock
+ * - CPU count
+ * - for each CPU:
+ * - CPU id
+ * - CPU trace data offset in the file
+ * - CPU trace data size
+ */
+
+ /* Buffer latency option, v7:
+ * - trace data offset in the file
+ * - buffer name
+ * - buffer clock
+ */
+
+ vect = calloc(5 + (cpus * 3), sizeof(struct iovec));
+ if (!vect)
+ return NULL;
+ if (cpus) {
+ cpu_ids = calloc(cpus, sizeof(int));
+ if (!cpu_ids) {
+ free(vect);
+ return NULL;
+ }
+ }
+ vect[j].iov_base = (void *) &data_offset;
+ vect[j++].iov_len = 8;
+ vect[j].iov_base = (void *) name;
+ vect[j++].iov_len = strlen(name) + 1;
+ vect[j].iov_base = (void *) clock;
+ vect[j++].iov_len = strlen(clock) + 1;
+ if (id == TRACECMD_OPTION_BUFFER) {
+ vect[j].iov_base = (void *) &k;
+ vect[j++].iov_len = 4;
+ for (i = 0; i < cpus; i++) {
+ if (!cpu_data[i].file_size)
+ continue;
+ cpu_ids[i] = i;
+ vect[j].iov_base = &cpu_ids[i];
+ vect[j++].iov_len = 4;
+ vect[j].iov_base = &cpu_data[i].data_offset;
+ vect[j++].iov_len = 8;
+ vect[j].iov_base = &cpu_data[i].write_size;
+ vect[j++].iov_len = 8;
+ k++;
+ }
+ }
+
+ option = tracecmd_add_option_v(handle, id, vect, j);
+ free(vect);
+ free(cpu_ids);
+
+ return option;
+}
+
struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus,
int file_version)
{
@@ -2017,7 +2112,7 @@ out:
return ret;
}
-static int update_buffer_cpu_offset(struct tracecmd_output *handle,
+static int update_buffer_cpu_offset_v6(struct tracecmd_output *handle,
const char *name, tsize_t offset)
{
tsize_t b_offset;
@@ -2047,27 +2142,6 @@ static int update_buffer_cpu_offset(struct tracecmd_output *handle,
return 0;
}
-
-static char *get_clock(struct tracecmd_output *handle)
-{
- struct tracefs_instance *inst;
-
- if (handle->trace_clock)
- return handle->trace_clock;
-
- /*
- * If no clock is set on this handle, get the trace clock of
- * the top instance in the handle's tracing dir
- */
- inst = tracefs_instance_alloc(handle->tracing_dir, NULL);
- if (!inst)
- return NULL;
- handle->trace_clock = tracefs_get_clock(inst);
- tracefs_instance_free(inst);
- return handle->trace_clock;
-}
-
-
__hidden int out_write_cpu_data(struct tracecmd_output *handle,
int cpus, struct cpu_data_source *data, const char *buff_name)
{
@@ -2091,7 +2165,7 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle,
}
data_offs = do_lseek(handle, 0, SEEK_CUR);
- if (do_write_check(handle, "flyrecord", 10))
+ if (handle->file_version < 7 && do_write_check(handle, "flyrecord", 10))
goto out_free;
data_files = calloc(cpus, sizeof(struct data_file_write));
@@ -2112,10 +2186,12 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle,
}
}
- update_buffer_cpu_offset(handle, buff_name, data_offs);
- clock = get_clock(handle);
- if (clock && save_clock(handle, clock))
- goto out_free;
+ if (handle->file_version < 7) {
+ update_buffer_cpu_offset_v6(handle, buff_name, data_offs);
+ clock = get_clock(handle);
+ if (clock && save_clock(handle, clock))
+ goto out_free;
+ }
for (i = 0; i < cpus; i++) {
data_files[i].data_offset = do_lseek(handle, 0, SEEK_CUR);
@@ -2142,26 +2218,33 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle,
data_files[i].write_size = 0;
}
- /* Write the real CPU data offset in the file */
- if (do_lseek(handle, data_files[i].doffset, SEEK_SET) == (off64_t)-1)
- goto out_free;
- endian8 = convert_endian_8(handle, data_files[i].data_offset);
- if (do_write_check(handle, &endian8, 8))
- goto out_free;
- /* Write the real CPU data size in the file */
- if (do_lseek(handle, data_files[i].soffset, SEEK_SET) == (off64_t)-1)
- goto out_free;
- endian8 = convert_endian_8(handle, data_files[i].write_size);
- if (do_write_check(handle, &endian8, 8))
- goto out_free;
- offset = data_files[i].data_offset + data_files[i].write_size;
- if (do_lseek(handle, offset, SEEK_SET) == (off64_t)-1)
- goto out_free;
+ if (handle->file_version < 7) {
+ /* Write the real CPU data offset in the file */
+ if (do_lseek(handle, data_files[i].doffset, SEEK_SET) == (off64_t)-1)
+ goto out_free;
+ endian8 = convert_endian_8(handle, data_files[i].data_offset);
+ if (do_write_check(handle, &endian8, 8))
+ goto out_free;
+ /* Write the real CPU data size in the file */
+ if (do_lseek(handle, data_files[i].soffset, SEEK_SET) == (off64_t)-1)
+ goto out_free;
+ endian8 = convert_endian_8(handle, data_files[i].write_size);
+ if (do_write_check(handle, &endian8, 8))
+ goto out_free;
+ offset = data_files[i].data_offset + data_files[i].write_size;
+ if (do_lseek(handle, offset, SEEK_SET) == (off64_t)-1)
+ goto out_free;
+ }
if (!tracecmd_get_quiet(handle))
fprintf(stderr, " %llu bytes in size\n",
(unsigned long long)data_files[i].write_size);
}
+ if (handle->file_version >= 7 &&
+ !out_add_buffer_option_v7(handle, buff_name,
+ TRACECMD_OPTION_BUFFER, data_offs, cpus, data_files))
+ goto out_free;
+
if (do_lseek(handle, 0, SEEK_END) == (off64_t)-1)
goto out_free;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 28/87] trace-cmd library: Move CPU flyrecord trace metadata into the buffer option, for trace file version 7
2021-07-29 5:09 ` [PATCH v2 28/87] trace-cmd library: Move CPU flyrecord trace metadata into the buffer option, for trace file " Tzvetomir Stoyanov (VMware)
@ 2021-08-17 15:40 ` Steven Rostedt
2021-09-02 13:20 ` Tzvetomir Stoyanov
0 siblings, 1 reply; 118+ messages in thread
From: Steven Rostedt @ 2021-08-17 15:40 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:00 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Extended the BUFFER trace option in trace file version 7 with CPU
> flyrecord trace metadata. In the version 6 of the trace file, this metadata
> is located at several places in the file. Part of the metadata is only for
> the top trace instance, thus limiting per-instance configuration. Moving
> all CPU trace related metadata in the BUFFER option simplifies the parsing
> and makes per-instance configuration more flexible. In the new file
> structure, the top instance is treated as any other instances.
> The format of the extended BUFFER option is:
> - offset of the buffer in the trace file
> - name of the buffer
> - trace clock, used in this buffer for events timestamps
> - count of CPUs with trace data
> - array, describing each CPU with trace data:
> - CPU id
> - offset of CPU trace data in the trace file
> - size of the recorded CPU trace data
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-output.c | 171 ++++++++++++++++++++++++++---------
> 1 file changed, 127 insertions(+), 44 deletions(-)
>
> diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
> index d60bd456..21555d6d 100644
> --- a/lib/trace-cmd/trace-output.c
> +++ b/lib/trace-cmd/trace-output.c
> @@ -1826,7 +1826,7 @@ int tracecmd_append_options(struct tracecmd_output *handle)
> }
>
> static struct tracecmd_option *
> -add_buffer_option(struct tracecmd_output *handle, const char *name, int cpus)
> +add_buffer_option_v6(struct tracecmd_output *handle, const char *name, int cpus)
> {
> struct tracecmd_option *option;
> char *buf;
> @@ -1876,8 +1876,11 @@ int tracecmd_write_buffer_info(struct tracecmd_output *handle)
> struct tracecmd_option *option;
> struct tracecmd_buffer *buf;
>
> + if (handle->file_version >= 7)
> + return 0;
> +
> list_for_each_entry(buf, &handle->buffers, list) {
> - option = add_buffer_option(handle, buf->name, buf->cpus);
> + option = add_buffer_option_v6(handle, buf->name, buf->cpus);
> if (!option)
> return -1;
> buf->option = option;
> @@ -1938,6 +1941,98 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle)
> return 0;
> }
>
> +static char *get_clock(struct tracecmd_output *handle)
> +{
> + struct tracefs_instance *inst;
> +
> + if (handle->trace_clock)
> + return handle->trace_clock;
> +
> + /*
> + * If no clock is set on this handle, get the trace clock of
> + * the top instance in the handle's tracing dir
> + */
> + inst = tracefs_instance_alloc(handle->tracing_dir, NULL);
> + if (!inst)
> + return NULL;
> + handle->trace_clock = tracefs_get_clock(inst);
> + tracefs_instance_free(inst);
> + return handle->trace_clock;
> +}
> +
> +__hidden struct tracecmd_option *
> +out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name,
> + unsigned short id, unsigned long long data_offset,
> + int cpus, struct data_file_write *cpu_data)
Why is this hidden, and not static?
I noticed that you are using "out_" as a normal prefix. Is that really
needed?
> +{
> + struct tracecmd_option *option;
> + int i, j = 0, k = 0;
> + int *cpu_ids = NULL;
> + struct iovec *vect;
> + char *clock;
> +
> + if (handle->file_version < 7)
> + return NULL;
> +
> + clock = get_clock(handle);
> +
> + /* Buffer flyrecord option, v7:
Nit, for consistency, use:
/*
* Buffer flyrecord option, v7:
> + * - trace data offset in the file
Is the above the "meta data". If so, perhaps we should call it that.
> + * - buffer name
> + * - buffer clock
> + * - CPU count
> + * - for each CPU:
> + * - CPU id
> + * - CPU trace data offset in the file
> + * - CPU trace data size
> + */
> +
> + /* Buffer latency option, v7:
> + * - trace data offset in the file
Is this the actual latency?
> + * - buffer name
> + * - buffer clock
> + */
> +
> + vect = calloc(5 + (cpus * 3), sizeof(struct iovec));
Comment what these magic numbers are: 5 and 3.
-- Steve
> + if (!vect)
> + return NULL;
> + if (cpus) {
> + cpu_ids = calloc(cpus, sizeof(int));
> + if (!cpu_ids) {
> + free(vect);
> + return NULL;
> + }
> + }
> + vect[j].iov_base = (void *) &data_offset;
> + vect[j++].iov_len = 8;
> + vect[j].iov_base = (void *) name;
> + vect[j++].iov_len = strlen(name) + 1;
> + vect[j].iov_base = (void *) clock;
> + vect[j++].iov_len = strlen(clock) + 1;
> + if (id == TRACECMD_OPTION_BUFFER) {
> + vect[j].iov_base = (void *) &k;
> + vect[j++].iov_len = 4;
> + for (i = 0; i < cpus; i++) {
> + if (!cpu_data[i].file_size)
> + continue;
> + cpu_ids[i] = i;
> + vect[j].iov_base = &cpu_ids[i];
> + vect[j++].iov_len = 4;
> + vect[j].iov_base = &cpu_data[i].data_offset;
> + vect[j++].iov_len = 8;
> + vect[j].iov_base = &cpu_data[i].write_size;
> + vect[j++].iov_len = 8;
> + k++;
> + }
> + }
> +
> + option = tracecmd_add_option_v(handle, id, vect, j);
> + free(vect);
> + free(cpu_ids);
> +
> + return option;
> +}
> +
^ permalink raw reply [flat|nested] 118+ messages in thread* Re: [PATCH v2 28/87] trace-cmd library: Move CPU flyrecord trace metadata into the buffer option, for trace file version 7
2021-08-17 15:40 ` Steven Rostedt
@ 2021-09-02 13:20 ` Tzvetomir Stoyanov
0 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov @ 2021-09-02 13:20 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linux Trace Devel
On Tue, Aug 17, 2021 at 6:40 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
[...]
> > +
> > +__hidden struct tracecmd_option *
> > +out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name,
> > + unsigned short id, unsigned long long data_offset,
> > + int cpus, struct data_file_write *cpu_data)
>
>
> Why is this hidden, and not static?
>
> I noticed that you are using "out_" as a normal prefix. Is that really
> needed?
I use "out_" and "in_" prefixes for internal library functions,
related to tracecmd_output and tracecmd_input handlers. For example,
there are:
int out_uncompress_block(struct tracecmd_output *handle);
int in_uncompress_block(struct tracecmd_input *handle);
>
>
[...]
--
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 29/87] trace-cmd record: Append trace options after the trace data are written
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (27 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 28/87] trace-cmd library: Move CPU flyrecord trace metadata into the buffer option, for trace file " Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 30/87] trace-cmd library: Add section header before flyrecord trace data Tzvetomir Stoyanov (VMware)
` (57 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In trace file version 7, the CPU trace metadata is saved in a trace
option. As this metadata has file offsets, which depend on the CPU trace
data, the option must be written in the file after the CPU trace data.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-record.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index 350d2811..f8c1c090 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -4466,6 +4466,9 @@ static void write_guest_file(struct buffer_instance *instance)
if (tracecmd_write_cpu_data(handle, cpu_count, temp_files, "") < 0)
die("failed to write CPU data");
+ if (tracecmd_get_out_file_version(handle) >= 7)
+ tracecmd_write_options(handle);
+
tracecmd_output_close(handle);
for (i = 0; i < cpu_count; i++)
@@ -4623,7 +4626,8 @@ static void record_data(struct common_record_context *ctx)
append_buffer(handle, instance, temp_files);
}
}
-
+ if (tracecmd_get_out_file_version(handle) >= 7)
+ tracecmd_write_options(handle);
free(temp_files);
}
if (!handle)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 30/87] trace-cmd library: Add section header before flyrecord trace data
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (28 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 29/87] trace-cmd record: Append trace options after the trace data are written Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 31/87] trace-cmd library: Fit CPU latency trace data in the new trace file version 7 format Tzvetomir Stoyanov (VMware)
` (56 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In trace file version 7, write a section header at the beginning of each
section with flyrecord trace data.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 21555d6d..30a6acf4 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -2146,10 +2146,12 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle,
int cpus, struct cpu_data_source *data, const char *buff_name)
{
struct data_file_write *data_files = NULL;
+ enum tracecmd_section_flags flags = 0;
tsize_t data_offs, offset;
unsigned long long endian8;
unsigned long long read_size;
char *clock;
+ char *str;
int ret;
int i;
@@ -2168,6 +2170,16 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle,
if (handle->file_version < 7 && do_write_check(handle, "flyrecord", 10))
goto out_free;
+ if (handle->compress)
+ flags |= TRACECMD_SEC_FL_COMPRESS;
+ if (asprintf(&str, "buffer flyrecord %s", buff_name) < 1)
+ goto out_free;
+
+ offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER, str, flags, false);
+ free(str);
+ if (offset == (off_t)-1)
+ goto out_free;
+
data_files = calloc(cpus, sizeof(struct data_file_write));
if (!data_files)
goto out_free;
@@ -2248,6 +2260,9 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle,
if (do_lseek(handle, 0, SEEK_END) == (off64_t)-1)
goto out_free;
+ if (out_update_section_header(handle, offset))
+ goto out_free;
+
free(data_files);
handle->file_state = TRACECMD_FILE_CPU_FLYRECORD;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 31/87] trace-cmd library: Fit CPU latency trace data in the new trace file version 7 format
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (29 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 30/87] trace-cmd library: Add section header before flyrecord trace data Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-17 15:44 ` Steven Rostedt
2021-08-19 19:10 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 32/87] trace-cmd library: Do not write CPUs with empty trace data Tzvetomir Stoyanov (VMware)
` (55 subsequent siblings)
86 siblings, 2 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file version 7 format is based on sections. To fit the latency
trace data in this structure, a new section and option for it is
defined:
BUFFER_LAT
It is similar to the BUFFER section which holds the flyrecord data, but
has a latency specific design. The BUFFER_LAT section has:
- section header, as all other sections
- compression of the trace data, optional
- corresponding trace option, pointing to the section
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 3 +-
lib/trace-cmd/trace-output.c | 36 ++++++++++++++++---
tracecmd/trace-record.c | 2 +-
3 files changed, 35 insertions(+), 6 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 4e718c40..f2868576 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -135,6 +135,7 @@ enum {
TRACECMD_OPTION_TIME_SHIFT,
TRACECMD_OPTION_GUEST,
TRACECMD_OPTION_TSC2NSEC,
+ TRACECMD_OPTION_BUFFER_LAT,
TRACECMD_OPTION_HEADER_INFO,
TRACECMD_OPTION_FTRACE_EVENTS,
TRACECMD_OPTION_EVENT_FORMATS,
@@ -294,7 +295,7 @@ int tracecmd_output_write_headers(struct tracecmd_output *handler,
struct tracecmd_event_list *list);
struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus,
- int file_version);
+ int file_version, const char *compression);
struct tracecmd_output *tracecmd_create_init_fd(int fd);
struct tracecmd_output *tracecmd_create_init_file(const char *output_file);
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 30a6acf4..83c3b9e7 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -2034,9 +2034,11 @@ out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name,
}
struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus,
- int file_version)
+ int file_version, const char *compression)
{
+ enum tracecmd_section_flags flags = 0;
struct tracecmd_output *handle;
+ tsize_t offset;
char *path;
int fd;
@@ -2049,6 +2051,12 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in
goto out_free;
if (file_version && tracecmd_output_set_version(handle, file_version))
goto out_free;
+ if (compression) {
+ if (tracecmd_output_set_compression(handle, compression))
+ goto out_free;
+ } else if (file_version >= 7) {
+ tracecmd_output_set_compression(handle, "any");
+ }
if (tracecmd_output_write_init(handle))
goto out_free;
if (tracecmd_output_write_headers(handle, NULL))
@@ -2061,7 +2069,8 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in
if (tracecmd_write_cpus(handle, cpus) < 0)
goto out_free;
-
+ if (tracecmd_write_buffer_info(handle) < 0)
+ goto out_free;
if (tracecmd_write_options(handle) < 0)
goto out_free;
@@ -2071,23 +2080,42 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in
goto out_free;
}
- if (do_write_check(handle, "latency ", 10))
+ if (handle->file_version < 7 && do_write_check(handle, "latency ", 10))
goto out_free;
path = get_tracing_file(handle, "trace");
if (!path)
goto out_free;
+ offset = do_lseek(handle, 0, SEEK_CUR);
+ if (handle->file_version >= 7 &&
+ !out_add_buffer_option_v7(handle, "", TRACECMD_OPTION_BUFFER_LAT, offset, 0, NULL))
+ goto out_free;
+ if (handle->compress)
+ flags |= TRACECMD_SEC_FL_COMPRESS;
+
+ offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER_LAT,
+ "buffer latency", flags, false);
+
copy_file_compress(handle, path, NULL);
+ if (out_update_section_header(handle, offset))
+ goto out_free;
put_tracing_file(path);
handle->file_state = TRACECMD_FILE_CPU_LATENCY;
+ if (tracecmd_get_out_file_version(handle) >= 7)
+ tracecmd_write_options(handle);
+
return handle;
out_free:
- tracecmd_output_close(handle);
+ if (handle)
+ tracecmd_output_close(handle);
+ else
+ close(fd);
+ unlink(output_file);
return NULL;
}
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index f8c1c090..2e8db8a4 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -4531,7 +4531,7 @@ static void record_data(struct common_record_context *ctx)
if (latency) {
handle = tracecmd_create_file_latency(ctx->output, local_cpu_count,
- ctx->file_version);
+ ctx->file_version, ctx->compression);
tracecmd_set_quiet(handle, quiet);
} else {
if (!local_cpu_count)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 31/87] trace-cmd library: Fit CPU latency trace data in the new trace file version 7 format
2021-07-29 5:09 ` [PATCH v2 31/87] trace-cmd library: Fit CPU latency trace data in the new trace file version 7 format Tzvetomir Stoyanov (VMware)
@ 2021-08-17 15:44 ` Steven Rostedt
2021-09-02 12:48 ` Tzvetomir Stoyanov
2021-08-19 19:10 ` Steven Rostedt
1 sibling, 1 reply; 118+ messages in thread
From: Steven Rostedt @ 2021-08-17 15:44 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:03 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Trace file version 7 format is based on sections. To fit the latency
> trace data in this structure, a new section and option for it is
> defined:
> BUFFER_LAT
> It is similar to the BUFFER section which holds the flyrecord data, but
> has a latency specific design. The BUFFER_LAT section has:
Actually, let's call it BUFFER_TEXT.
Although we do it for "latency" format, which I'm thinking we should
call it something different for v7, the real difference is that it's
text and not binary. Perhaps we can save other formats as "text".
In fact, I think v7 should allow for a mixture of TEXT and BINARY
buffers.
-- Steve
> - section header, as all other sections
> - compression of the trace data, optional
> - corresponding trace option, pointing to the section
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
^ permalink raw reply [flat|nested] 118+ messages in thread
* Re: [PATCH v2 31/87] trace-cmd library: Fit CPU latency trace data in the new trace file version 7 format
2021-08-17 15:44 ` Steven Rostedt
@ 2021-09-02 12:48 ` Tzvetomir Stoyanov
0 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov @ 2021-09-02 12:48 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linux Trace Devel
On Tue, Aug 17, 2021 at 6:44 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 29 Jul 2021 08:09:03 +0300
> "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
>
> > Trace file version 7 format is based on sections. To fit the latency
> > trace data in this structure, a new section and option for it is
> > defined:
> > BUFFER_LAT
> > It is similar to the BUFFER section which holds the flyrecord data, but
> > has a latency specific design. The BUFFER_LAT section has:
>
> Actually, let's call it BUFFER_TEXT.
>
> Although we do it for "latency" format, which I'm thinking we should
> call it something different for v7, the real difference is that it's
> text and not binary. Perhaps we can save other formats as "text".
>
> In fact, I think v7 should allow for a mixture of TEXT and BINARY
> buffers.
It does, there is no limitation to do a flyrecord in one buffer and
latency tracing in another. Although I cannot imagine a use case for
that.
>
> -- Steve
>
> > - section header, as all other sections
> > - compression of the trace data, optional
> > - corresponding trace option, pointing to the section
> >
> > Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
--
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
^ permalink raw reply [flat|nested] 118+ messages in thread
* Re: [PATCH v2 31/87] trace-cmd library: Fit CPU latency trace data in the new trace file version 7 format
2021-07-29 5:09 ` [PATCH v2 31/87] trace-cmd library: Fit CPU latency trace data in the new trace file version 7 format Tzvetomir Stoyanov (VMware)
2021-08-17 15:44 ` Steven Rostedt
@ 2021-08-19 19:10 ` Steven Rostedt
1 sibling, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 19:10 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:03 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> +++ b/lib/trace-cmd/include/private/trace-cmd-private.h
> @@ -135,6 +135,7 @@ enum {
> TRACECMD_OPTION_TIME_SHIFT,
> TRACECMD_OPTION_GUEST,
> TRACECMD_OPTION_TSC2NSEC,
> + TRACECMD_OPTION_BUFFER_LAT,
I just realized that this will break the parsing.
You can't add an enum in the middle. That will cause all the other
enums to shift in value, and these are already a file API.
> TRACECMD_OPTION_HEADER_INFO,
For example, if you saved a file before this patch, the
TRACECMD_OPTION_HEADER_INFO will be one number, and stored in the
trace.dat file.
Then if you try to read that same trace.dat file after this patch, it
will interpret the TRACECMD_OPTION_HEADER_INFO as
TRACECMD_OPTION_BUFFER_LAT, which will obviously be incorrect.
-- Steve
> TRACECMD_OPTION_FTRACE_EVENTS,
> TRACECMD_OPTION_EVENT_FORMATS,
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 32/87] trace-cmd library: Do not write CPUs with empty trace data
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (30 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 31/87] trace-cmd library: Fit CPU latency trace data in the new trace file version 7 format Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 33/87] trace-cmd library: Add macro to check file state on reading Tzvetomir Stoyanov (VMware)
` (54 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file version 7 has more flexible design, which allows to write only
CPUs with trace data in the file. If a CPU has no recorded trace data,
do not put it in the trace file version 7.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-output.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 83c3b9e7..87bbcf39 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -2335,8 +2335,10 @@ int tracecmd_write_cpu_data(struct tracecmd_output *handle,
if (i < cpus)
ret = -1;
- else
+ else if (size || handle->file_version < 7) /* do not write empty trace buffer in v7 files*/
ret = out_write_cpu_data(handle, cpus, data, buff_name);
+ else
+ ret = 0;
for (i = 0; i < cpus; i++) {
if (data[i].fd >= 0)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 33/87] trace-cmd library: Add macro to check file state on reading
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (31 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 32/87] trace-cmd library: Do not write CPUs with empty trace data Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 34/87] trace-cmd library: Introduce sections in trace file reading logic Tzvetomir Stoyanov (VMware)
` (53 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file version 7 has flexible structure. It allows reading almost
any part of the file at any time, unlike the version 6 file when reading
state must be validated at each step. Added a macro to handle these
checks.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index c5b4c32e..6872f6ef 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -166,6 +166,8 @@ struct tracecmd_input {
__thread struct tracecmd_input *tracecmd_curr_thread_handle;
+#define CHECK_READ_STATE(H, S) ((H)->file_version < 7 && (H)->file_state >= (S))
+
static int read_options_type(struct tracecmd_input *handle);
void tracecmd_set_flag(struct tracecmd_input *handle, int flag)
@@ -416,7 +418,7 @@ static int read_header_files(struct tracecmd_input *handle)
char *header;
char buf[BUFSIZ];
- if (handle->file_state >= TRACECMD_FILE_HEADERS)
+ if (CHECK_READ_STATE(handle, TRACECMD_FILE_HEADERS))
return 0;
if (do_read_check(handle, buf, 12))
@@ -622,7 +624,7 @@ static int read_ftrace_files(struct tracecmd_input *handle, const char *regex)
int unique;
int ret;
- if (handle->file_state >= TRACECMD_FILE_FTRACE_EVENTS)
+ if (CHECK_READ_STATE(handle, TRACECMD_FILE_FTRACE_EVENTS))
return 0;
if (regex) {
@@ -693,7 +695,7 @@ static int read_event_files(struct tracecmd_input *handle, const char *regex)
int unique;
int ret;
- if (handle->file_state >= TRACECMD_FILE_ALL_EVENTS)
+ if (CHECK_READ_STATE(handle, TRACECMD_FILE_ALL_EVENTS))
return 0;
if (regex) {
@@ -778,7 +780,7 @@ static int read_proc_kallsyms(struct tracecmd_input *handle)
unsigned int size;
char *buf;
- if (handle->file_state >= TRACECMD_FILE_KALLSYMS)
+ if (CHECK_READ_STATE(handle, TRACECMD_FILE_KALLSYMS))
return 0;
if (read4(handle, &size) < 0)
@@ -809,7 +811,7 @@ static int read_ftrace_printk(struct tracecmd_input *handle)
unsigned int size;
char *buf;
- if (handle->file_state >= TRACECMD_FILE_PRINTK)
+ if (CHECK_READ_STATE(handle, TRACECMD_FILE_PRINTK))
return 0;
if (read4(handle, &size) < 0)
@@ -855,7 +857,7 @@ static int read_cpus(struct tracecmd_input *handle)
{
unsigned int cpus;
- if (handle->file_state >= TRACECMD_FILE_CPU_COUNT)
+ if (CHECK_READ_STATE(handle, TRACECMD_FILE_CPU_COUNT))
return 0;
if (read4(handle, &cpus) < 0)
@@ -2861,7 +2863,7 @@ static int read_options_type(struct tracecmd_input *handle)
{
char buf[10];
- if (handle->file_state >= TRACECMD_FILE_CPU_LATENCY)
+ if (CHECK_READ_STATE(handle, TRACECMD_FILE_CPU_LATENCY))
return 0;
if (do_read_check(handle, buf, 10))
@@ -3026,7 +3028,7 @@ static int read_and_parse_cmdlines(struct tracecmd_input *handle)
unsigned long long size;
char *cmdlines;
- if (handle->file_state >= TRACECMD_FILE_CMD_LINES)
+ if (CHECK_READ_STATE(handle, TRACECMD_FILE_CMD_LINES))
return 0;
if (read_data_and_size(handle, &cmdlines, &size) < 0)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 34/87] trace-cmd library: Introduce sections in trace file reading logic
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (32 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 33/87] trace-cmd library: Add macro to check file state on reading Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 17:53 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 35/87] trace-cmd library: Initialize internal sections database on file read Tzvetomir Stoyanov (VMware)
` (52 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file version 7 is based on sections. Added an internal sections
database and new helper functions to add, read, open and close file
sections.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 71 +++++++++++++++++++++++++++++++++++++
1 file changed, 71 insertions(+)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 6872f6ef..e8eefb5c 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -114,6 +114,14 @@ struct tsc2nsec {
unsigned long long offset;
};
+struct file_section {
+ int id;
+ unsigned long long section_offset;
+ unsigned long long data_offset;
+ enum tracecmd_section_flags flags;
+ struct file_section *next;
+};
+
struct tracecmd_input {
struct tep_handle *pevent;
unsigned long file_state;
@@ -154,6 +162,7 @@ struct tracecmd_input {
struct hook_list *hooks;
struct pid_addr_maps *pid_maps;
/* file information */
+ struct file_section *sections;
size_t header_files_start;
size_t ftrace_files_start;
size_t event_files_start;
@@ -411,6 +420,61 @@ __hidden int in_uncompress_block(struct tracecmd_input *handle)
return ret;
}
+static struct file_section *get_section(struct tracecmd_input *handle, int id)
+{
+ struct file_section *sec = handle->sections;
+
+ while (sec) {
+ if (sec->id == id)
+ return sec;
+ sec = sec->next;
+ }
+ return NULL;
+}
+
+static struct file_section *open_section(struct tracecmd_input *handle, int id)
+{
+ struct file_section *sec = get_section(handle, id);
+
+ if (!sec)
+ return NULL;
+
+ if (lseek64(handle->fd, sec->data_offset, SEEK_SET) == (off64_t)-1)
+ return NULL;
+ if ((sec->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle))
+ return NULL;
+ return sec;
+}
+
+static void close_section(struct tracecmd_input *handle, struct file_section *sec)
+{
+ if (sec->flags & TRACECMD_SEC_FL_COMPRESS)
+ in_uncompress_reset(handle);
+}
+
+static int add_section(struct tracecmd_input *handle, int id, int flags,
+ unsigned long long soffset, unsigned long long doffset)
+{
+ struct file_section *sec = get_section(handle, id);
+
+ if (!sec) {
+ sec = calloc(1, sizeof(struct file_section));
+ if (!sec)
+ return -1;
+ sec->next = handle->sections;
+ handle->sections = sec;
+ }
+ sec->id = id;
+ if (soffset)
+ sec->section_offset = soffset;
+ if (doffset)
+ sec->data_offset = doffset;
+ if (flags > 0)
+ sec->flags = flags;
+ return 0;
+}
+
+
static int read_header_files(struct tracecmd_input *handle)
{
struct tep_handle *pevent = handle->pevent;
@@ -3522,6 +3586,7 @@ void tracecmd_ref(struct tracecmd_input *handle)
*/
void tracecmd_close(struct tracecmd_input *handle)
{
+ struct file_section *del_sec;
int i;
if (!handle)
@@ -3560,6 +3625,11 @@ void tracecmd_close(struct tracecmd_input *handle)
free(handle->version);
close(handle->fd);
+ while (handle->sections) {
+ del_sec = handle->sections;
+ handle->sections = handle->sections->next;
+ free(del_sec);
+ }
for (i = 0; i < handle->nr_buffers; i++)
free(handle->buffers[i].name);
@@ -4006,6 +4076,7 @@ tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx)
new_handle->nr_buffers = 0;
new_handle->buffers = NULL;
new_handle->version = NULL;
+ new_handle->sections = NULL;
new_handle->guest = NULL;
new_handle->ref = 1;
if (handle->trace_clock) {
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 34/87] trace-cmd library: Introduce sections in trace file reading logic
2021-07-29 5:09 ` [PATCH v2 34/87] trace-cmd library: Introduce sections in trace file reading logic Tzvetomir Stoyanov (VMware)
@ 2021-08-19 17:53 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 17:53 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:06 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Trace file version 7 is based on sections. Added an internal sections
> database and new helper functions to add, read, open and close file
> sections.
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-input.c | 71 +++++++++++++++++++++++++++++++++++++
> 1 file changed, 71 insertions(+)
>
> diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
> index 6872f6ef..e8eefb5c 100644
> --- a/lib/trace-cmd/trace-input.c
> +++ b/lib/trace-cmd/trace-input.c
> @@ -114,6 +114,14 @@ struct tsc2nsec {
> unsigned long long offset;
> };
>
> +struct file_section {
> + int id;
> + unsigned long long section_offset;
> + unsigned long long data_offset;
Should we have a size too?
> + enum tracecmd_section_flags flags;
> + struct file_section *next;
> +};
> +
> struct tracecmd_input {
> struct tep_handle *pevent;
> unsigned long file_state;
> @@ -154,6 +162,7 @@ struct tracecmd_input {
> struct hook_list *hooks;
> struct pid_addr_maps *pid_maps;
> /* file information */
> + struct file_section *sections;
> size_t header_files_start;
> size_t ftrace_files_start;
> size_t event_files_start;
> @@ -411,6 +420,61 @@ __hidden int in_uncompress_block(struct tracecmd_input *handle)
> return ret;
> }
>
> +static struct file_section *get_section(struct tracecmd_input *handle, int id)
> +{
> + struct file_section *sec = handle->sections;
> +
> + while (sec) {
> + if (sec->id == id)
> + return sec;
> + sec = sec->next;
> + }
The above could be a simple for loop.
for (sec = handle->sections; sec; sec = sec->next) {
if (sec->id == id);
return sec;
}
> + return NULL;
> +}
> +
> +static struct file_section *open_section(struct tracecmd_input *handle, int id)
I wonder if we should change the names to "section_open()", "section_close()" ...
This way the prefix "section_" is know to be a command for handling sections.
> +{
> + struct file_section *sec = get_section(handle, id);
> +
> + if (!sec)
> + return NULL;
> +
> + if (lseek64(handle->fd, sec->data_offset, SEEK_SET) == (off64_t)-1)
> + return NULL;
> + if ((sec->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle))
> + return NULL;
> + return sec;
> +}
> +
> +static void close_section(struct tracecmd_input *handle, struct file_section *sec)
> +{
> + if (sec->flags & TRACECMD_SEC_FL_COMPRESS)
> + in_uncompress_reset(handle);
> +}
> +
> +static int add_section(struct tracecmd_input *handle, int id, int flags,
> + unsigned long long soffset, unsigned long long doffset)
I'm still not sure what "soffest' is compared to "doffset".
> +{
> + struct file_section *sec = get_section(handle, id);
> +
> + if (!sec) {
> + sec = calloc(1, sizeof(struct file_section));
> + if (!sec)
> + return -1;
> + sec->next = handle->sections;
> + handle->sections = sec;
> + }
Hmm, we should probably have an "add" and "update" because blindly
changing an existing section may not be what was expected. That is,
going with the "section_" prefix.
section_add(...)
Would error if it already exists.
section_update(..)
Would expect it to already exist.
Could have an:
section_add_or_update(...)
Which would be the above, and lets the caller know that it will ether
add the section, or update it.
-- Steve
> + sec->id = id;
> + if (soffset)
> + sec->section_offset = soffset;
> + if (doffset)
> + sec->data_offset = doffset;
> + if (flags > 0)
> + sec->flags = flags;
> + return 0;
> +}
> +
> +
> static int read_header_files(struct tracecmd_input *handle)
> {
> struct tep_handle *pevent = handle->pevent;
> @@ -3522,6 +3586,7 @@ void tracecmd_ref(struct tracecmd_input *handle)
> */
> void tracecmd_close(struct tracecmd_input *handle)
> {
> + struct file_section *del_sec;
> int i;
>
> if (!handle)
> @@ -3560,6 +3625,11 @@ void tracecmd_close(struct tracecmd_input *handle)
> free(handle->version);
> close(handle->fd);
>
> + while (handle->sections) {
> + del_sec = handle->sections;
> + handle->sections = handle->sections->next;
> + free(del_sec);
> + }
> for (i = 0; i < handle->nr_buffers; i++)
> free(handle->buffers[i].name);
>
> @@ -4006,6 +4076,7 @@ tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx)
> new_handle->nr_buffers = 0;
> new_handle->buffers = NULL;
> new_handle->version = NULL;
> + new_handle->sections = NULL;
> new_handle->guest = NULL;
> new_handle->ref = 1;
> if (handle->trace_clock) {
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 35/87] trace-cmd library: Initialize internal sections database on file read
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (33 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 34/87] trace-cmd library: Introduce sections in trace file reading logic Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 17:57 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 36/87] trace-cmd library: Use sections database when reading parts of the trace file Tzvetomir Stoyanov (VMware)
` (51 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Add sections in internal database on file reading and parsing.
In trace file version 7, sections are initialized when parsing
corresponding trace options. In version 6 files, sections are retrieved
on file reading, as there they are in a fixed position file.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 55 +++++++++++++++++++++++++++----------
1 file changed, 40 insertions(+), 15 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index e8eefb5c..f5e13bbc 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -165,7 +165,6 @@ struct tracecmd_input {
struct file_section *sections;
size_t header_files_start;
size_t ftrace_files_start;
- size_t event_files_start;
size_t options_start;
size_t total_file_size;
@@ -485,6 +484,10 @@ static int read_header_files(struct tracecmd_input *handle)
if (CHECK_READ_STATE(handle, TRACECMD_FILE_HEADERS))
return 0;
+ if (handle->file_version < 7)
+ add_section(handle, TRACECMD_OPTION_HEADER_INFO, 0, 0,
+ lseek64(handle->fd, 0, SEEK_CUR));
+
if (do_read_check(handle, buf, 12))
return -1;
@@ -528,9 +531,6 @@ static int read_header_files(struct tracecmd_input *handle)
free(header);
- handle->ftrace_files_start =
- lseek64(handle->fd, 0, SEEK_CUR);
-
handle->file_state = TRACECMD_FILE_HEADERS;
return 0;
@@ -691,6 +691,10 @@ static int read_ftrace_files(struct tracecmd_input *handle, const char *regex)
if (CHECK_READ_STATE(handle, TRACECMD_FILE_FTRACE_EVENTS))
return 0;
+ if (handle->file_version < 7)
+ add_section(handle, TRACECMD_OPTION_FTRACE_EVENTS, 0, 0,
+ lseek64(handle->fd, 0, SEEK_CUR));
+
if (regex) {
sreg = &spreg;
ereg = &epreg;
@@ -729,9 +733,6 @@ static int read_ftrace_files(struct tracecmd_input *handle, const char *regex)
return -1;
}
- handle->event_files_start =
- lseek64(handle->fd, 0, SEEK_CUR);
-
if (sreg) {
regfree(sreg);
regfree(ereg);
@@ -762,6 +763,10 @@ static int read_event_files(struct tracecmd_input *handle, const char *regex)
if (CHECK_READ_STATE(handle, TRACECMD_FILE_ALL_EVENTS))
return 0;
+ if (handle->file_version < 7)
+ add_section(handle, TRACECMD_OPTION_EVENT_FORMATS, 0, 0,
+ lseek64(handle->fd, 0, SEEK_CUR));
+
if (regex) {
sreg = &spreg;
ereg = &epreg;
@@ -847,6 +852,11 @@ static int read_proc_kallsyms(struct tracecmd_input *handle)
if (CHECK_READ_STATE(handle, TRACECMD_FILE_KALLSYMS))
return 0;
+ if (handle->file_version < 7)
+ add_section(handle, TRACECMD_OPTION_KALLSYMS, 0, 0,
+ lseek64(handle->fd, 0, SEEK_CUR));
+
+
if (read4(handle, &size) < 0)
return -1;
if (!size)
@@ -878,6 +888,11 @@ static int read_ftrace_printk(struct tracecmd_input *handle)
if (CHECK_READ_STATE(handle, TRACECMD_FILE_PRINTK))
return 0;
+ if (handle->file_version < 7)
+ add_section(handle, TRACECMD_OPTION_PRINTK, 0, 0,
+ lseek64(handle->fd, 0, SEEK_CUR));
+
+
if (read4(handle, &size) < 0)
return -1;
if (!size)
@@ -2909,6 +2924,16 @@ static int handle_options(struct tracecmd_input *handle)
handle->tsc_calc.offset = tep_read_number(handle->pevent,
buf + 8, 8);
break;
+ case TRACECMD_OPTION_HEADER_INFO:
+ case TRACECMD_OPTION_FTRACE_EVENTS:
+ case TRACECMD_OPTION_EVENT_FORMATS:
+ case TRACECMD_OPTION_KALLSYMS:
+ case TRACECMD_OPTION_PRINTK:
+ case TRACECMD_OPTION_CMDLINES:
+ if (size < 8)
+ break;
+ add_section(handle, option, -1, tep_read_number(handle->pevent, buf, 8), 0);
+ break;
default:
tracecmd_warning("unknown option %d", option);
break;
@@ -3095,6 +3120,10 @@ static int read_and_parse_cmdlines(struct tracecmd_input *handle)
if (CHECK_READ_STATE(handle, TRACECMD_FILE_CMD_LINES))
return 0;
+ if (handle->file_version < 7)
+ add_section(handle, TRACECMD_OPTION_CMDLINES, 0, 0,
+ lseek64(handle->fd, 0, SEEK_CUR));
+
if (read_data_and_size(handle, &cmdlines, &size) < 0)
return -1;
cmdlines[size] = 0;
@@ -3384,6 +3413,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
struct tracecmd_input *handle;
char test[] = TRACECMD_MAGIC;
unsigned int page_size;
+ size_t offset;
char *version;
char buf[BUFSIZ];
unsigned long ver;
@@ -3444,14 +3474,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
read4(handle, &page_size);
handle->page_size = page_size;
- handle->header_files_start =
- lseek64(handle->fd, 0, SEEK_CUR);
-
- handle->total_file_size =
- lseek64(handle->fd, 0, SEEK_END);
-
- handle->header_files_start =
- lseek64(handle->fd, handle->header_files_start, SEEK_SET);
+ offset = lseek64(handle->fd, 0, SEEK_CUR);
+ handle->total_file_size = lseek64(handle->fd, 0, SEEK_END);
+ lseek64(handle->fd, offset, SEEK_SET);
handle->file_state = TRACECMD_FILE_INIT;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 36/87] trace-cmd library: Use sections database when reading parts of the trace file
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (34 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 35/87] trace-cmd library: Initialize internal sections database on file read Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 37/87] trace-cmd library: Set log size to the input tep handler when it is read from the file Tzvetomir Stoyanov (VMware)
` (50 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Utilize the internal database with file sections, when reading parts
of a trace file. This logic unifies the way trace file version 6 and 7
are processed.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 34 +++++++++++++++++++++-------------
1 file changed, 21 insertions(+), 13 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index f5e13bbc..bf56c92a 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -163,8 +163,6 @@ struct tracecmd_input {
struct pid_addr_maps *pid_maps;
/* file information */
struct file_section *sections;
- size_t header_files_start;
- size_t ftrace_files_start;
size_t options_start;
size_t total_file_size;
@@ -3301,21 +3299,26 @@ int tracecmd_make_pipe(struct tracecmd_input *handle, int cpu, int fd, int cpus)
*/
void tracecmd_print_events(struct tracecmd_input *handle, const char *regex)
{
- int ret;
+ struct file_section *sec;
if (!regex)
regex = ".*";
- if (!handle->ftrace_files_start) {
- lseek64(handle->fd, handle->header_files_start, SEEK_SET);
+ sec = open_section(handle, TRACECMD_OPTION_HEADER_INFO);
+ if (sec) {
read_header_files(handle);
+ close_section(handle, sec);
+ }
+ sec = open_section(handle, TRACECMD_OPTION_FTRACE_EVENTS);
+ if (sec) {
+ read_ftrace_files(handle, regex);
+ close_section(handle, sec);
+ }
+ sec = open_section(handle, TRACECMD_OPTION_EVENT_FORMATS);
+ if (sec) {
+ read_event_files(handle, regex);
+ close_section(handle, sec);
}
- ret = read_ftrace_files(handle, regex);
- if (ret < 0)
- return;
-
- read_event_files(handle, regex);
- return;
}
/* Show the cpu data stats */
@@ -3905,6 +3908,7 @@ int tracecmd_copy_headers(struct tracecmd_input *handle, int fd,
enum tracecmd_file_states start_state,
enum tracecmd_file_states end_state)
{
+ struct file_section *sec;
int ret;
if (!start_state)
@@ -3920,13 +3924,17 @@ int tracecmd_copy_headers(struct tracecmd_input *handle, int fd,
if (handle->file_state >= start_state) {
/* Set the handle to just before the start state */
- lseek64(handle->fd, handle->header_files_start, SEEK_SET);
+ sec = open_section(handle, TRACECMD_OPTION_HEADER_INFO);
+ if (!sec)
+ return -1;
/* Now that the file handle has moved, change its state */
handle->file_state = TRACECMD_FILE_INIT;
}
-
/* Try to bring the input up to the start state - 1 */
ret = tracecmd_read_headers(handle, start_state - 1);
+ if (sec)
+ close_section(handle, sec);
+
if (ret < 0)
goto out;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 37/87] trace-cmd library: Set log size to the input tep handler when it is read from the file
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (35 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 36/87] trace-cmd library: Use sections database when reading parts of the trace file Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 18:01 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 38/87] trace-cmd library: Fix possible memory leak in read_ftrace_files() Tzvetomir Stoyanov (VMware)
` (49 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Setting the long size to the input tep handler in tracecmd_read_headers()
API may be too late, as this tep handler is used to read and parse data
from the file before that. The most suitable place for that is
tracecmd_alloc_fd() API, right after reading the long size from the
file.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index bf56c92a..0ced15a8 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -976,8 +976,6 @@ int tracecmd_read_headers(struct tracecmd_input *handle,
if (ret < 0)
return -1;
- tep_set_long_size(handle->pevent, handle->long_size);
-
if (state <= handle->file_state)
return 0;
@@ -3473,6 +3471,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
do_read_check(handle, buf, 1);
handle->long_size = buf[0];
+ tep_set_long_size(handle->pevent, handle->long_size);
read4(handle, &page_size);
handle->page_size = page_size;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 37/87] trace-cmd library: Set log size to the input tep handler when it is read from the file
2021-07-29 5:09 ` [PATCH v2 37/87] trace-cmd library: Set log size to the input tep handler when it is read from the file Tzvetomir Stoyanov (VMware)
@ 2021-08-19 18:01 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 18:01 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:09 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Setting the long size to the input tep handler in tracecmd_read_headers()
> API may be too late, as this tep handler is used to read and parse data
> from the file before that. The most suitable place for that is
> tracecmd_alloc_fd() API, right after reading the long size from the
> file.
>
Typo in subject. I was expecting to see some log size get set ;-)
-- Steve
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-input.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
> index bf56c92a..0ced15a8 100644
> --- a/lib/trace-cmd/trace-input.c
> +++ b/lib/trace-cmd/trace-input.c
> @@ -976,8 +976,6 @@ int tracecmd_read_headers(struct tracecmd_input *handle,
> if (ret < 0)
> return -1;
>
> - tep_set_long_size(handle->pevent, handle->long_size);
> -
> if (state <= handle->file_state)
> return 0;
>
> @@ -3473,6 +3471,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
>
> do_read_check(handle, buf, 1);
> handle->long_size = buf[0];
> + tep_set_long_size(handle->pevent, handle->long_size);
>
> read4(handle, &page_size);
> handle->page_size = page_size;
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 38/87] trace-cmd library: Fix possible memory leak in read_ftrace_files()
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (36 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 37/87] trace-cmd library: Set log size to the input tep handler when it is read from the file Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 18:07 ` Steven Rostedt
2021-08-19 18:08 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 39/87] trace-cmd library: Fix possible memory leak in read_event_files() Tzvetomir Stoyanov (VMware)
` (48 subsequent siblings)
86 siblings, 2 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Some error paths in read_ftrace_files() may lead to a memory leak.
Improved the error handling of this internal function to avoid it.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 19 +++++++++++--------
1 file changed, 11 insertions(+), 8 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 0ced15a8..b36df98b 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -720,25 +720,28 @@ static int read_ftrace_files(struct tracecmd_input *handle, const char *regex)
}
}
- if (read4(handle, &count) < 0)
- return -1;
+ ret = read4(handle, &count);
+ if (ret < 0)
+ goto out;
for (i = 0; i < count; i++) {
- if (read8(handle, &size) < 0)
- return -1;
+ ret = read8(handle, &size);
+ if (ret < 0)
+ goto out;
ret = read_ftrace_file(handle, size, print_all, ereg);
if (ret < 0)
- return -1;
+ goto out;
}
+ handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
+ ret = 0;
+out:
if (sreg) {
regfree(sreg);
regfree(ereg);
}
- handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
-
- return 0;
+ return ret;
}
static int read_event_files(struct tracecmd_input *handle, const char *regex)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 38/87] trace-cmd library: Fix possible memory leak in read_ftrace_files()
2021-07-29 5:09 ` [PATCH v2 38/87] trace-cmd library: Fix possible memory leak in read_ftrace_files() Tzvetomir Stoyanov (VMware)
@ 2021-08-19 18:07 ` Steven Rostedt
2021-08-19 18:08 ` Steven Rostedt
1 sibling, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 18:07 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:10 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Some error paths in read_ftrace_files() may lead to a memory leak.
> Improved the error handling of this internal function to avoid it.
This looks more like a normal fix. Can it be moved to the front of the
queue? As it can be applied outside this patch set.
-- Steve
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-input.c | 19 +++++++++++--------
> 1 file changed, 11 insertions(+), 8 deletions(-)
>
> diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
> index 0ced15a8..b36df98b 100644
> --- a/lib/trace-cmd/trace-input.c
> +++ b/lib/trace-cmd/trace-input.c
> @@ -720,25 +720,28 @@ static int read_ftrace_files(struct tracecmd_input *handle, const char *regex)
> }
> }
>
> - if (read4(handle, &count) < 0)
> - return -1;
> + ret = read4(handle, &count);
> + if (ret < 0)
> + goto out;
>
> for (i = 0; i < count; i++) {
> - if (read8(handle, &size) < 0)
> - return -1;
> + ret = read8(handle, &size);
> + if (ret < 0)
> + goto out;
> ret = read_ftrace_file(handle, size, print_all, ereg);
> if (ret < 0)
> - return -1;
> + goto out;
> }
>
> + handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
> + ret = 0;
> +out:
> if (sreg) {
> regfree(sreg);
> regfree(ereg);
> }
>
> - handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
> -
> - return 0;
> + return ret;
> }
>
> static int read_event_files(struct tracecmd_input *handle, const char *regex)
^ permalink raw reply [flat|nested] 118+ messages in thread* Re: [PATCH v2 38/87] trace-cmd library: Fix possible memory leak in read_ftrace_files()
2021-07-29 5:09 ` [PATCH v2 38/87] trace-cmd library: Fix possible memory leak in read_ftrace_files() Tzvetomir Stoyanov (VMware)
2021-08-19 18:07 ` Steven Rostedt
@ 2021-08-19 18:08 ` Steven Rostedt
1 sibling, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 18:08 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:10 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Some error paths in read_ftrace_files() may lead to a memory leak.
> Improved the error handling of this internal function to avoid it.
>
>
In fact, all theses memory leak fixes should not even be part of this
patch set, and should be its own patchset to fix the current code.
If any of these is fixing code in this patch set, then simply fix the
patch it fixes.
-- Steve
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 39/87] trace-cmd library: Fix possible memory leak in read_event_files()
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (37 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 38/87] trace-cmd library: Fix possible memory leak in read_ftrace_files() Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 40/87] trace-cmd library: Fix possible memory leak in read_proc_kallsyms() Tzvetomir Stoyanov (VMware)
` (47 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Some error paths in read_event_files() may lead to a memory leak.
Improved the error handling of this internal function to avoid it.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 39 ++++++++++++++++++-------------------
1 file changed, 19 insertions(+), 20 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index b36df98b..e2e12014 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -747,7 +747,7 @@ out:
static int read_event_files(struct tracecmd_input *handle, const char *regex)
{
unsigned long long size;
- char *system;
+ char *system = NULL;
regex_t spreg;
regex_t epreg;
regex_t *sreg = NULL;
@@ -776,13 +776,16 @@ static int read_event_files(struct tracecmd_input *handle, const char *regex)
return -1;
}
- if (read4(handle, &systems) < 0)
- return -1;
+ ret = read4(handle, &systems);
+ if (ret < 0)
+ goto out;
for (i = 0; i < systems; i++) {
system = read_string(handle);
- if (!system)
- return -1;
+ if (!system) {
+ ret = -1;
+ goto out;
+ }
sys_printed = 0;
print_all = 0;
@@ -809,39 +812,35 @@ static int read_event_files(struct tracecmd_input *handle, const char *regex)
}
}
- if (read4(handle, &count) < 0)
- goto failed;
+ ret = read4(handle, &count);
+ if (ret < 0)
+ goto out;
for (x=0; x < count; x++) {
- if (read8(handle, &size) < 0)
- goto failed;
+ ret = read8(handle, &size);
+ if (ret < 0)
+ goto out;
ret = read_event_file(handle, system, size,
print_all, &sys_printed,
reg);
if (ret < 0)
- goto failed;
+ goto out;
}
free(system);
- }
-
- if (sreg) {
- regfree(sreg);
- regfree(ereg);
+ system = NULL;
}
handle->file_state = TRACECMD_FILE_ALL_EVENTS;
-
- return 0;
-
- failed:
+ ret = 0;
+ out:
if (sreg) {
regfree(sreg);
regfree(ereg);
}
free(system);
- return -1;
+ return ret;
}
static int read_proc_kallsyms(struct tracecmd_input *handle)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 40/87] trace-cmd library: Fix possible memory leak in read_proc_kallsyms()
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (38 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 39/87] trace-cmd library: Fix possible memory leak in read_event_files() Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 41/87] trace-cmd library: Fix possible memory leak in read_ftrace_printk() Tzvetomir Stoyanov (VMware)
` (46 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Some error paths in read_proc_kallsyms() may lead to a memory leak.
Improved the error handling of this internal function to avoid it.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 40 ++++++++++++++++++++-----------------
1 file changed, 22 insertions(+), 18 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index e2e12014..99aa33ae 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -845,9 +845,10 @@ static int read_event_files(struct tracecmd_input *handle, const char *regex)
static int read_proc_kallsyms(struct tracecmd_input *handle)
{
- struct tep_handle *pevent = handle->pevent;
+ struct tep_handle *tep = handle->pevent;
unsigned int size;
- char *buf;
+ char *buf = NULL;
+ int ret;
if (CHECK_READ_STATE(handle, TRACECMD_FILE_KALLSYMS))
return 0;
@@ -856,28 +857,31 @@ static int read_proc_kallsyms(struct tracecmd_input *handle)
add_section(handle, TRACECMD_OPTION_KALLSYMS, 0, 0,
lseek64(handle->fd, 0, SEEK_CUR));
-
- if (read4(handle, &size) < 0)
- return -1;
- if (!size)
- return 0; /* OK? */
+ ret = read4(handle, &size);
+ if (ret < 0)
+ goto out;
+ if (!size) {
+ handle->file_state = TRACECMD_FILE_KALLSYMS;
+ goto out; /* OK? */
+ }
buf = malloc(size+1);
- if (!buf)
- return -1;
- if (do_read_check(handle, buf, size)){
- free(buf);
- return -1;
+ if (!buf) {
+ ret = -1;
+ goto out;
}
- buf[size] = 0;
-
- tep_parse_kallsyms(pevent, buf);
+ ret = do_read_check(handle, buf, size);
+ if (ret < 0)
+ goto out;
- free(buf);
+ buf[size] = 0;
+ tep_parse_kallsyms(tep, buf);
handle->file_state = TRACECMD_FILE_KALLSYMS;
-
- return 0;
+ ret = 0;
+out:
+ free(buf);
+ return ret;
}
static int read_ftrace_printk(struct tracecmd_input *handle)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 41/87] trace-cmd library: Fix possible memory leak in read_ftrace_printk()
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (39 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 40/87] trace-cmd library: Fix possible memory leak in read_proc_kallsyms() Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 42/87] trace-cmd library: Fix possible memory leak in read_and_parse_cmdlines() Tzvetomir Stoyanov (VMware)
` (45 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Some error paths in read_ftrace_printk() may lead to a memory leak.
Improved the error handling of this internal function to avoid it.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 33 +++++++++++++++++++--------------
1 file changed, 19 insertions(+), 14 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 99aa33ae..5557e772 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -887,7 +887,8 @@ out:
static int read_ftrace_printk(struct tracecmd_input *handle)
{
unsigned int size;
- char *buf;
+ char *buf = NULL;
+ int ret;
if (CHECK_READ_STATE(handle, TRACECMD_FILE_PRINTK))
return 0;
@@ -896,29 +897,33 @@ static int read_ftrace_printk(struct tracecmd_input *handle)
add_section(handle, TRACECMD_OPTION_PRINTK, 0, 0,
lseek64(handle->fd, 0, SEEK_CUR));
-
- if (read4(handle, &size) < 0)
- return -1;
- if (!size)
- return 0; /* OK? */
+ ret = read4(handle, &size);
+ if (ret < 0)
+ goto out;
+ if (!size) {
+ handle->file_state = TRACECMD_FILE_PRINTK;
+ goto out; /* OK? */
+ }
buf = malloc(size + 1);
- if (!buf)
- return -1;
- if (do_read_check(handle, buf, size)) {
- free(buf);
- return -1;
+ if (!buf) {
+ ret = -1;
+ goto out;
}
+ ret = do_read_check(handle, buf, size);
+ if (ret < 0)
+ goto out;
buf[size] = 0;
tep_parse_printk_formats(handle->pevent, buf);
- free(buf);
-
handle->file_state = TRACECMD_FILE_PRINTK;
+ ret = 0;
- return 0;
+out:
+ free(buf);
+ return ret;
}
static int read_and_parse_cmdlines(struct tracecmd_input *handle);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 42/87] trace-cmd library: Fix possible memory leak in read_and_parse_cmdlines()
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (40 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 41/87] trace-cmd library: Fix possible memory leak in read_ftrace_printk() Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 43/87] trace-cmd library: Track maximum CPUs count in input handler Tzvetomir Stoyanov (VMware)
` (44 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Some error paths in read_and_parse_cmdlines() may lead to a memory leak.
Improved the error handling of this internal function to avoid it.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 20 +++++++++++++-------
1 file changed, 13 insertions(+), 7 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 5557e772..4324259a 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -3122,7 +3122,8 @@ static int read_and_parse_cmdlines(struct tracecmd_input *handle)
{
struct tep_handle *pevent = handle->pevent;
unsigned long long size;
- char *cmdlines;
+ char *cmdlines = NULL;
+ int ret;
if (CHECK_READ_STATE(handle, TRACECMD_FILE_CMD_LINES))
return 0;
@@ -3131,15 +3132,20 @@ static int read_and_parse_cmdlines(struct tracecmd_input *handle)
add_section(handle, TRACECMD_OPTION_CMDLINES, 0, 0,
lseek64(handle->fd, 0, SEEK_CUR));
- if (read_data_and_size(handle, &cmdlines, &size) < 0)
- return -1;
+ ret = read_data_and_size(handle, &cmdlines, &size);
+ if (ret < 0)
+ goto out;
+ if (!size) {
+ handle->file_state = TRACECMD_FILE_CMD_LINES;
+ goto out;
+ }
cmdlines[size] = 0;
tep_parse_saved_cmdlines(pevent, cmdlines);
- free(cmdlines);
-
handle->file_state = TRACECMD_FILE_CMD_LINES;
-
- return 0;
+ ret = 0;
+out:
+ free(cmdlines);
+ return ret;
}
static void extract_trace_clock(struct tracecmd_input *handle, char *line)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 43/87] trace-cmd library: Track maximum CPUs count in input handler
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (41 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 42/87] trace-cmd library: Fix possible memory leak in read_and_parse_cmdlines() Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 44/87] trace-cmd library: Set input handler default values in allocation function Tzvetomir Stoyanov (VMware)
` (43 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In trace file version 7, only CPUs with trace data are stored in the
file. Each instance may have its own CPU count, depending on collected
traces. As the main input handler is used by the top trace instance, the
CPU count there is for the top trace instance and may differ with cpu
counts of the other instances. Added a new "max_cpu" member of the input
handler, that tracks the maximum CPU count of all instances, recorded in
the file.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 4324259a..41f95874 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -133,6 +133,7 @@ struct tracecmd_input {
int long_size;
int page_size;
int page_map_size;
+ int max_cpu;
int cpus;
int ref;
int nr_buffers; /* buffer instances */
@@ -952,6 +953,7 @@ static int read_cpus(struct tracecmd_input *handle)
return -1;
handle->cpus = cpus;
+ handle->max_cpu = cpus;
tep_set_cpus(handle->pevent, handle->cpus);
handle->file_state = TRACECMD_FILE_CPU_COUNT;
@@ -2907,6 +2909,8 @@ static int handle_options(struct tracecmd_input *handle)
case TRACECMD_OPTION_CPUCOUNT:
cpus = *(int *)buf;
handle->cpus = tep_read_number(handle->pevent, &cpus, 4);
+ handle->max_cpu = handle->cpus;
+ tep_set_cpus(handle->pevent, handle->cpus);
break;
case TRACECMD_OPTION_PROCMAPS:
if (buf[size-1] == '\0')
@@ -4215,7 +4219,7 @@ int tracecmd_page_size(struct tracecmd_input *handle)
*/
int tracecmd_cpus(struct tracecmd_input *handle)
{
- return handle->cpus;
+ return handle->max_cpu;
}
/**
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 44/87] trace-cmd library: Set input handler default values in allocation function
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (42 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 43/87] trace-cmd library: Track maximum CPUs count in input handler Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 18:11 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 45/87] trace-cmd library: Read headers from trace file version 7 Tzvetomir Stoyanov (VMware)
` (42 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Set usecs flag by default when the input handler is allocated, it makes
more sense than setting it when options are handeled. This clean up is
needed for parsing version 7 trace files, where multiple options
sections may exist.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 41f95874..23eea0d1 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -2794,8 +2794,6 @@ static int handle_options(struct tracecmd_input *handle)
int cpus;
int ret;
- /* By default, use usecs, unless told otherwise */
- handle->flags |= TRACECMD_FL_IN_USECS;
handle->options_start = lseek64(handle->fd, 0, SEEK_CUR);
for (;;) {
@@ -3448,6 +3446,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
handle->fd = fd;
handle->ref = 1;
+ /* By default, use usecs, unless told otherwise */
+ handle->flags |= TRACECMD_FL_IN_USECS;
+
if (do_read_check(handle, buf, 3))
goto failed_read;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 44/87] trace-cmd library: Set input handler default values in allocation function
2021-07-29 5:09 ` [PATCH v2 44/87] trace-cmd library: Set input handler default values in allocation function Tzvetomir Stoyanov (VMware)
@ 2021-08-19 18:11 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 18:11 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:16 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> Set usecs flag by default when the input handler is allocated, it makes
> more sense than setting it when options are handeled. This clean up is
> needed for parsing version 7 trace files, where multiple options
> sections may exist.
Clean ups like this can also be moved to the front of the patch queue.
-- Steve
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-input.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
> index 41f95874..23eea0d1 100644
> --- a/lib/trace-cmd/trace-input.c
> +++ b/lib/trace-cmd/trace-input.c
> @@ -2794,8 +2794,6 @@ static int handle_options(struct tracecmd_input *handle)
> int cpus;
> int ret;
>
> - /* By default, use usecs, unless told otherwise */
> - handle->flags |= TRACECMD_FL_IN_USECS;
> handle->options_start = lseek64(handle->fd, 0, SEEK_CUR);
>
> for (;;) {
> @@ -3448,6 +3446,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
> handle->fd = fd;
> handle->ref = 1;
>
> + /* By default, use usecs, unless told otherwise */
> + handle->flags |= TRACECMD_FL_IN_USECS;
> +
> if (do_read_check(handle, buf, 3))
> goto failed_read;
>
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 45/87] trace-cmd library: Read headers from trace file version 7
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (43 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 44/87] trace-cmd library: Set input handler default values in allocation function Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 46/87] tarce-cmd library: Do not use local variables when reading CPU stat option Tzvetomir Stoyanov (VMware)
` (41 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file version 7 has more flexible structure than version 6, headers
are not located at fixed position in the file. A new logic is
implemented to read and parse the headers from this new format.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 150 ++++++++++++++++++++++++++++++++----
1 file changed, 136 insertions(+), 14 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 23eea0d1..f447c8d7 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -164,7 +164,8 @@ struct tracecmd_input {
struct pid_addr_maps *pid_maps;
/* file information */
struct file_section *sections;
- size_t options_start;
+ bool options_init;
+ unsigned long long options_start;
size_t total_file_size;
/* For custom profilers. */
@@ -960,19 +961,7 @@ static int read_cpus(struct tracecmd_input *handle)
return 0;
}
-/**
- * tracecmd_read_headers - read the header information from trace.dat
- * @handle: input handle for the trace.dat file
- * @state: The state to read up to or zero to read up to options.
- *
- * This reads the trace.dat file for various information. Like the
- * format of the ring buffer, event formats, ftrace formats, kallsyms
- * and printk. This may be called multiple times with different @state
- * values, to read partial data at a time. It will always continue
- * where it left off.
- */
-int tracecmd_read_headers(struct tracecmd_input *handle,
- enum tracecmd_file_states state)
+static int read_headers_v6(struct tracecmd_input *handle, enum tracecmd_file_states state)
{
int ret;
@@ -1038,6 +1027,139 @@ int tracecmd_read_headers(struct tracecmd_input *handle,
return 0;
}
+static int handle_options(struct tracecmd_input *handle);
+
+static int read_section_header(struct tracecmd_input *handle, unsigned short *id,
+ unsigned short *flags, unsigned int *size, char **description)
+{
+ unsigned short fl;
+ char *desc = NULL;
+ unsigned short sec_id;
+ unsigned int sz;
+
+ if (read2(handle, &sec_id))
+ return -1;
+ desc = read_string(handle);
+ if (!desc)
+ return -1;
+ if (read2(handle, &fl))
+ goto error;
+ if (read4(handle, &sz))
+ goto error;
+
+ if (id)
+ *id = sec_id;
+ if (flags)
+ *flags = fl;
+ if (size)
+ *size = sz;
+ if (description)
+ *description = desc;
+ else
+ free(desc);
+ return 0;
+
+error:
+ free(desc);
+ return -1;
+}
+
+static int handle_section(struct tracecmd_input *handle, struct file_section *section)
+{
+ unsigned short id, flags;
+ unsigned int size;
+ int ret;
+
+ if (lseek64(handle->fd, section->section_offset, SEEK_SET) == (off_t)-1)
+ return -1;
+ if (read_section_header(handle, &id, &flags, &size, NULL))
+ return -1;
+ section->flags = flags;
+ if (id != section->id)
+ return -1;
+
+ section->data_offset = lseek64(handle->fd, 0, SEEK_CUR);
+ if ((section->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle))
+ return -1;
+
+ switch (section->id) {
+ case TRACECMD_OPTION_HEADER_INFO:
+ ret = read_header_files(handle);
+ break;
+ case TRACECMD_OPTION_FTRACE_EVENTS:
+ ret = read_ftrace_files(handle, NULL);
+ break;
+ case TRACECMD_OPTION_EVENT_FORMATS:
+ ret = read_event_files(handle, NULL);
+ break;
+ case TRACECMD_OPTION_KALLSYMS:
+ ret = read_proc_kallsyms(handle);
+ break;
+ case TRACECMD_OPTION_PRINTK:
+ ret = read_ftrace_printk(handle);
+ break;
+ case TRACECMD_OPTION_CMDLINES:
+ ret = read_and_parse_cmdlines(handle);
+ break;
+ default:
+ ret = 0;
+ break;
+ }
+
+ if (section->flags & TRACECMD_SEC_FL_COMPRESS)
+ in_uncompress_reset(handle);
+
+ return ret;
+}
+
+static int read_headers_v7(struct tracecmd_input *handle)
+{
+ struct file_section *section;
+
+ if (handle->options_init)
+ return 0;
+
+ if (!handle->options_start)
+ return -1;
+
+ if (lseek64(handle->fd, handle->options_start, SEEK_SET) == (off64_t)-1) {
+ tracecmd_warning("Filed to goto options offset %lld", handle->options_start);
+ return -1;
+ }
+
+ if (handle_options(handle))
+ return -1;
+
+ section = handle->sections;
+ while (section) {
+ if (handle_section(handle, section))
+ return -1;
+ section = section->next;
+ }
+
+ handle->options_init = true;
+ return 0;
+}
+
+/**
+ * tracecmd_read_headers - read the header information from trace.dat
+ * @handle: input handle for the trace.dat file
+ * @state: The state to read up to or zero to read up to options.
+ *
+ * This reads the trace.dat file for various information. Like the
+ * format of the ring buffer, event formats, ftrace formats, kallsyms
+ * and printk. This may be called multiple times with different @state
+ * values, to read partial data at a time. It will always continue
+ * where it left off.
+ */
+int tracecmd_read_headers(struct tracecmd_input *handle,
+ enum tracecmd_file_states state)
+{
+ if (handle->file_version < 7)
+ return read_headers_v6(handle, state);
+ return read_headers_v7(handle);
+}
+
static unsigned long long calc_page_offset(struct tracecmd_input *handle,
unsigned long long offset)
{
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 46/87] tarce-cmd library: Do not use local variables when reading CPU stat option
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (44 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 45/87] trace-cmd library: Read headers from trace file version 7 Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 47/87] trace-cmd library: Read handle header and compression of the option section Tzvetomir Stoyanov (VMware)
` (40 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file version 7 can have more than one options section. Assuming
that all CPUSTAT options are in a single options section could be wrong,
that's why using local variable to track the CPUSTAT size is not
correct. Use input handler context instead.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index f447c8d7..d8c8b9b0 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -150,6 +150,7 @@ struct tracecmd_input {
struct host_trace_info host;
double ts2secs;
+ unsigned int cpustats_size;
char * cpustats;
char * uname;
char * version;
@@ -2909,7 +2910,6 @@ static int handle_options(struct tracecmd_input *handle)
unsigned short option;
unsigned int size;
char *cpustats = NULL;
- unsigned int cpustats_size = 0;
struct input_buffer_instance *buffer;
struct hook_list *hook;
char *buf;
@@ -2987,12 +2987,16 @@ static int handle_options(struct tracecmd_input *handle)
break;
case TRACECMD_OPTION_CPUSTAT:
buf[size-1] = '\n';
- cpustats = realloc(cpustats, cpustats_size + size + 1);
- if (!cpustats)
- return -ENOMEM;
- memcpy(cpustats + cpustats_size, buf, size);
- cpustats_size += size;
- cpustats[cpustats_size] = 0;
+ cpustats = realloc(handle->cpustats,
+ handle->cpustats_size + size + 1);
+ if (!cpustats) {
+ ret = -ENOMEM;
+ return ret;
+ }
+ memcpy(cpustats + handle->cpustats_size, buf, size);
+ handle->cpustats_size += size;
+ cpustats[handle->cpustats_size] = 0;
+ handle->cpustats = cpustats;
break;
case TRACECMD_OPTION_BUFFER:
/* A buffer instance is saved at the end of the file */
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 47/87] trace-cmd library: Read handle header and compression of the option section
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (45 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 46/87] tarce-cmd library: Do not use local variables when reading CPU stat option Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 48/87] trace-cmd library: Read extended BUFFER option Tzvetomir Stoyanov (VMware)
` (39 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In trace file version 7, options section has a section header and
possible compression of the data. Extend the options handling logic with
this new format.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 50 ++++++++++++++++++++++++++-----------
1 file changed, 35 insertions(+), 15 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index d8c8b9b0..e7f97561 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -2909,30 +2909,47 @@ static int handle_options(struct tracecmd_input *handle)
long long offset;
unsigned short option;
unsigned int size;
+ unsigned short id, flags;
char *cpustats = NULL;
struct input_buffer_instance *buffer;
struct hook_list *hook;
+ bool comperss = false;
char *buf;
int cpus;
int ret;
- handle->options_start = lseek64(handle->fd, 0, SEEK_CUR);
-
- for (;;) {
- if (read2(handle, &option))
+ if (handle->file_version < 7) {
+ handle->options_start = lseek64(handle->fd, 0, SEEK_CUR);
+ } else {
+ if (read_section_header(handle, &id, &flags, NULL, NULL))
return -1;
+ if (id != TRACECMD_OPTION_DONE)
+ return -1;
+ if (flags & TRACECMD_SEC_FL_COMPRESS)
+ comperss = true;
+ }
- if (option == TRACECMD_OPTION_DONE)
+ if (comperss && in_uncompress_block(handle))
+ return -1;
+ for (;;) {
+ ret = read2(handle, &option);
+ if (ret)
+ goto out;
+ if (handle->file_version < 7 && option == TRACECMD_OPTION_DONE)
break;
/* next 4 bytes is the size of the option */
- if (read4(handle, &size))
- return -1;
+ ret = read4(handle, &size);
+ if (ret)
+ goto out;
buf = malloc(size);
- if (!buf)
- return -ENOMEM;
- if (do_read_check(handle, buf, size))
- return -1;
+ if (!buf) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ ret = do_read_check(handle, buf, size);
+ if (ret)
+ goto out;
switch (option) {
case TRACECMD_OPTION_DATE:
@@ -2982,7 +2999,7 @@ static int handle_options(struct tracecmd_input *handle)
buf + 8, 4);
ret = tsync_cpu_offsets_load(handle, buf + 12, size - 12);
if (ret < 0)
- return ret;
+ goto out;
tracecmd_enable_tsync(handle, true);
break;
case TRACECMD_OPTION_CPUSTAT:
@@ -2991,7 +3008,7 @@ static int handle_options(struct tracecmd_input *handle)
handle->cpustats_size + size + 1);
if (!cpustats) {
ret = -ENOMEM;
- return ret;
+ goto out;
}
memcpy(cpustats + handle->cpustats_size, buf, size);
handle->cpustats_size += size;
@@ -3078,9 +3095,12 @@ static int handle_options(struct tracecmd_input *handle)
}
- handle->cpustats = cpustats;
+ ret = 0;
- return 0;
+out:
+ if (comperss)
+ in_uncompress_reset(handle);
+ return ret;
}
static int read_options_type(struct tracecmd_input *handle)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 48/87] trace-cmd library: Read extended BUFFER option
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (46 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 47/87] trace-cmd library: Read handle header and compression of the option section Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 18:54 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 49/87] trace-cmd library: Handle the extended DONE option Tzvetomir Stoyanov (VMware)
` (38 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The BUFFER option is extended in trace file version 7. It holds CPU
metadata related to the recorded CPU traces. Also, there is a new
BUFFER_LAT option for describing the latency trace data. A new logic
is implemented for these new options.
In trace file version 7, the top buffer is saved as other buffers in the
file, no special treatment. But saving the top buffer in the list of
buffers in the input handler causes problems. It breaks the legacy logic
of trace-cmd library users, which have special logic for trace buffers
processing. That's why "top_buffer" member is added in the input handler
structure, to hold the top buffer.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 116 ++++++++++++++++++++++++++++++------
1 file changed, 98 insertions(+), 18 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index e7f97561..57ae535a 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -74,9 +74,19 @@ struct cpu_data {
int pipe_fd;
};
+struct cpu_file_data {
+ int cpu;
+ unsigned long long offset;
+ unsigned long long size;
+};
+
struct input_buffer_instance {
char *name;
size_t offset;
+ char *clock;
+ bool latency;
+ int cpus;
+ struct cpu_file_data *cpu_data;
};
struct ts_offset_sample {
@@ -155,6 +165,7 @@ struct tracecmd_input {
char * uname;
char * version;
char * trace_clock;
+ struct input_buffer_instance top_buffer;
struct input_buffer_instance *buffers;
int parsing_failures;
struct guest_trace_info *guest;
@@ -2904,6 +2915,80 @@ tracecmd_search_task_map(struct tracecmd_input *handle,
return lib;
}
+#define save_read_number(R, C) \
+ do { \
+ if ((C) > size) \
+ return -1; \
+ (R) = tep_read_number(handle->pevent, (data + rsize), (C)); \
+ rsize += (C); \
+ size -= (C); \
+ } while (0)
+
+#define save_read_string(R) \
+ do { \
+ if (size < 1) \
+ return -1; \
+ (R) = strdup(data + rsize); \
+ if (!(R)) \
+ return -1; \
+ rsize += (strlen((R)) + 1); \
+ size -= (strlen((R)) + 1); \
+ if (size < 0) \
+ return -1; \
+ } while (0)
+
+static int handle_buffer_option(struct tracecmd_input *handle,
+ unsigned short id, char *data, int size)
+{
+ struct input_buffer_instance *buff;
+ unsigned long long offset;
+ int rsize = 0;
+ char *name;
+ int i;
+
+ save_read_number(offset, 8);
+ save_read_string(name);
+
+ if (*name == '\0') {
+ /* top buffer */
+ buff = &handle->top_buffer;
+ } else {
+ buff = realloc(handle->buffers, sizeof(*handle->buffers) * (handle->nr_buffers + 1));
+ if (!buff) {
+ free(name);
+ return -1;
+ }
+ handle->buffers = buff;
+ handle->nr_buffers++;
+
+ buff = &handle->buffers[handle->nr_buffers - 1];
+ }
+ memset(buff, 0, sizeof(struct input_buffer_instance));
+ buff->name = name;
+ buff->offset = offset;
+ if (handle->file_version < 7)
+ return 0;
+
+ /* file version >= 7 specific data */
+ save_read_string(buff->clock);
+ if (id == TRACECMD_OPTION_BUFFER) {
+ save_read_number(buff->cpus, 4);
+ if (!buff->cpus)
+ return 0;
+ buff->cpu_data = calloc(buff->cpus, sizeof(struct cpu_file_data));
+ if (!buff->cpu_data)
+ return -1;
+ for (i = 0; i < buff->cpus; i++) {
+ save_read_number(buff->cpu_data[i].cpu, 4);
+ save_read_number(buff->cpu_data[i].offset, 8);
+ save_read_number(buff->cpu_data[i].size, 8);
+ }
+ } else {
+ buff->latency = true;
+ }
+ return 0;
+}
+
static int handle_options(struct tracecmd_input *handle)
{
long long offset;
@@ -2911,7 +2996,6 @@ static int handle_options(struct tracecmd_input *handle)
unsigned int size;
unsigned short id, flags;
char *cpustats = NULL;
- struct input_buffer_instance *buffer;
struct hook_list *hook;
bool comperss = false;
char *buf;
@@ -3016,21 +3100,10 @@ static int handle_options(struct tracecmd_input *handle)
handle->cpustats = cpustats;
break;
case TRACECMD_OPTION_BUFFER:
- /* A buffer instance is saved at the end of the file */
- handle->nr_buffers++;
- handle->buffers = realloc(handle->buffers,
- sizeof(*handle->buffers) * handle->nr_buffers);
- if (!handle->buffers)
- return -ENOMEM;
- buffer = &handle->buffers[handle->nr_buffers - 1];
- buffer->name = strdup(buf + 8);
- if (!buffer->name) {
- free(handle->buffers);
- handle->buffers = NULL;
- return -ENOMEM;
- }
- offset = *(unsigned long long *)buf;
- buffer->offset = tep_read_number(handle->pevent, &offset, 8);
+ case TRACECMD_OPTION_BUFFER_LAT:
+ ret = handle_buffer_option(handle, option, buf, size);
+ if (ret < 0)
+ goto out;
break;
case TRACECMD_OPTION_TRACECLOCK:
if (!handle->ts2secs)
@@ -3825,9 +3898,15 @@ void tracecmd_close(struct tracecmd_input *handle)
handle->sections = handle->sections->next;
free(del_sec);
}
- for (i = 0; i < handle->nr_buffers; i++)
- free(handle->buffers[i].name);
+ free(handle->top_buffer.name);
+ free(handle->top_buffer.clock);
+ free(handle->top_buffer.cpu_data);
+ for (i = 0; i < handle->nr_buffers; i++) {
+ free(handle->buffers[i].name);
+ free(handle->buffers[i].clock);
+ free(handle->buffers[i].cpu_data);
+ }
free(handle->buffers);
tracecmd_free_hooks(handle->hooks);
@@ -4272,6 +4351,7 @@ tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx)
return NULL;
*new_handle = *handle;
+ memset(&new_handle->top_buffer, 0, sizeof(new_handle->top_buffer));
new_handle->cpu_data = NULL;
new_handle->nr_buffers = 0;
new_handle->buffers = NULL;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 48/87] trace-cmd library: Read extended BUFFER option
2021-07-29 5:09 ` [PATCH v2 48/87] trace-cmd library: Read extended BUFFER option Tzvetomir Stoyanov (VMware)
@ 2021-08-19 18:54 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 18:54 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:20 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> The BUFFER option is extended in trace file version 7. It holds CPU
> metadata related to the recorded CPU traces. Also, there is a new
> BUFFER_LAT option for describing the latency trace data. A new logic
> is implemented for these new options.
> In trace file version 7, the top buffer is saved as other buffers in the
> file, no special treatment. But saving the top buffer in the list of
> buffers in the input handler causes problems. It breaks the legacy logic
> of trace-cmd library users, which have special logic for trace buffers
> processing. That's why "top_buffer" member is added in the input handler
> structure, to hold the top buffer.
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-input.c | 116 ++++++++++++++++++++++++++++++------
> 1 file changed, 98 insertions(+), 18 deletions(-)
>
> diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
> index e7f97561..57ae535a 100644
> --- a/lib/trace-cmd/trace-input.c
> +++ b/lib/trace-cmd/trace-input.c
> @@ -74,9 +74,19 @@ struct cpu_data {
> int pipe_fd;
> };
>
> +struct cpu_file_data {
> + int cpu;
> + unsigned long long offset;
> + unsigned long long size;
> +};
> +
> struct input_buffer_instance {
> char *name;
> size_t offset;
> + char *clock;
> + bool latency;
> + int cpus;
> + struct cpu_file_data *cpu_data;
> };
>
> struct ts_offset_sample {
> @@ -155,6 +165,7 @@ struct tracecmd_input {
> char * uname;
> char * version;
> char * trace_clock;
> + struct input_buffer_instance top_buffer;
> struct input_buffer_instance *buffers;
> int parsing_failures;
> struct guest_trace_info *guest;
> @@ -2904,6 +2915,80 @@ tracecmd_search_task_map(struct tracecmd_input *handle,
> return lib;
> }
>
> +#define save_read_number(R, C) \
> + do { \
> + if ((C) > size) \
> + return -1; \
> + (R) = tep_read_number(handle->pevent, (data + rsize), (C)); \
> + rsize += (C); \
> + size -= (C); \
> + } while (0)
> +
> +#define save_read_string(R) \
> + do { \
> + if (size < 1) \
> + return -1; \
> + (R) = strdup(data + rsize); \
> + if (!(R)) \
> + return -1; \
> + rsize += (strlen((R)) + 1); \
> + size -= (strlen((R)) + 1); \
> + if (size < 0) \
> + return -1; \
> + } while (0)
The above should be inline functions, not macros. Just pass the address
of the values you want to modify.
> +
> +static int handle_buffer_option(struct tracecmd_input *handle,
> + unsigned short id, char *data, int size)
> +{
> + struct input_buffer_instance *buff;
> + unsigned long long offset;
> + int rsize = 0;
> + char *name;
> + int i;
> +
> + save_read_number(offset, 8);
> + save_read_string(name);
> +
> + if (*name == '\0') {
> + /* top buffer */
> + buff = &handle->top_buffer;
> + } else {
> + buff = realloc(handle->buffers, sizeof(*handle->buffers) * (handle->nr_buffers + 1));
> + if (!buff) {
> + free(name);
> + return -1;
> + }
> + handle->buffers = buff;
> + handle->nr_buffers++;
> +
> + buff = &handle->buffers[handle->nr_buffers - 1];
> + }
> + memset(buff, 0, sizeof(struct input_buffer_instance));
> + buff->name = name;
> + buff->offset = offset;
> + if (handle->file_version < 7)
> + return 0;
> +
> + /* file version >= 7 specific data */
> + save_read_string(buff->clock);
> + if (id == TRACECMD_OPTION_BUFFER) {
> + save_read_number(buff->cpus, 4);
> + if (!buff->cpus)
> + return 0;
> + buff->cpu_data = calloc(buff->cpus, sizeof(struct cpu_file_data));
> + if (!buff->cpu_data)
> + return -1;
> + for (i = 0; i < buff->cpus; i++) {
> + save_read_number(buff->cpu_data[i].cpu, 4);
> + save_read_number(buff->cpu_data[i].offset, 8);
> + save_read_number(buff->cpu_data[i].size, 8);
> + }
> + } else {
> + buff->latency = true;
> + }
> + return 0;
> +}
> +
> static int handle_options(struct tracecmd_input *handle)
> {
> long long offset;
> @@ -2911,7 +2996,6 @@ static int handle_options(struct tracecmd_input *handle)
> unsigned int size;
> unsigned short id, flags;
> char *cpustats = NULL;
> - struct input_buffer_instance *buffer;
> struct hook_list *hook;
> bool comperss = false;
> char *buf;
> @@ -3016,21 +3100,10 @@ static int handle_options(struct tracecmd_input *handle)
> handle->cpustats = cpustats;
> break;
> case TRACECMD_OPTION_BUFFER:
> - /* A buffer instance is saved at the end of the file */
> - handle->nr_buffers++;
> - handle->buffers = realloc(handle->buffers,
> - sizeof(*handle->buffers) * handle->nr_buffers);
> - if (!handle->buffers)
> - return -ENOMEM;
> - buffer = &handle->buffers[handle->nr_buffers - 1];
> - buffer->name = strdup(buf + 8);
> - if (!buffer->name) {
> - free(handle->buffers);
> - handle->buffers = NULL;
> - return -ENOMEM;
> - }
> - offset = *(unsigned long long *)buf;
> - buffer->offset = tep_read_number(handle->pevent, &offset, 8);
> + case TRACECMD_OPTION_BUFFER_LAT:
> + ret = handle_buffer_option(handle, option, buf, size);
> + if (ret < 0)
> + goto out;
> break;
> case TRACECMD_OPTION_TRACECLOCK:
> if (!handle->ts2secs)
> @@ -3825,9 +3898,15 @@ void tracecmd_close(struct tracecmd_input *handle)
> handle->sections = handle->sections->next;
> free(del_sec);
> }
> - for (i = 0; i < handle->nr_buffers; i++)
> - free(handle->buffers[i].name);
>
> + free(handle->top_buffer.name);
> + free(handle->top_buffer.clock);
> + free(handle->top_buffer.cpu_data);
> + for (i = 0; i < handle->nr_buffers; i++) {
> + free(handle->buffers[i].name);
> + free(handle->buffers[i].clock);
> + free(handle->buffers[i].cpu_data);
> + }
The freeing of the buffer should be a function that encapsulates this,
where all you need to do is:
free_buffer(&handle->top_buffer);
for (i = 0; i < handle->nr_buffers; i++)
free_buffer(&handle->buffers[i]);
-- Steve
> free(handle->buffers);
>
> tracecmd_free_hooks(handle->hooks);
> @@ -4272,6 +4351,7 @@ tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx)
> return NULL;
>
> *new_handle = *handle;
> + memset(&new_handle->top_buffer, 0, sizeof(new_handle->top_buffer));
> new_handle->cpu_data = NULL;
> new_handle->nr_buffers = 0;
> new_handle->buffers = NULL;
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 49/87] trace-cmd library: Handle the extended DONE option
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (47 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 48/87] trace-cmd library: Read extended BUFFER option Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 19:13 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 50/87] trace-cmd library: Read compression header Tzvetomir Stoyanov (VMware)
` (37 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In trace file version 7 the DONE option is extended to store the offset
in the file to the next options section. This way a list of options
sections can be stored in the file. Added logic to recursively read all
option sections from the file.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/trace-cmd-local.h | 1 +
lib/trace-cmd/trace-input.c | 31 +++++++++++++++++++++++++
lib/trace-cmd/trace-output.c | 2 +-
3 files changed, 33 insertions(+), 1 deletion(-)
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index e54ba7e1..0291b8de 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -64,5 +64,6 @@ struct cpu_data_source {
int out_write_cpu_data(struct tracecmd_output *handle, int cpus,
struct cpu_data_source *data, const char *buff_name);
off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off_t offset, int whence);
+unsigned long long get_last_option_offset(struct tracecmd_input *handle);
#endif /* _TRACE_CMD_LOCAL_H */
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 57ae535a..9ab20d64 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -178,6 +178,7 @@ struct tracecmd_input {
struct file_section *sections;
bool options_init;
unsigned long long options_start;
+ unsigned long long options_last_offset;
size_t total_file_size;
/* For custom profilers. */
@@ -2915,6 +2916,30 @@ tracecmd_search_task_map(struct tracecmd_input *handle,
return lib;
}
+__hidden unsigned long long get_last_option_offset(struct tracecmd_input *handle)
+{
+ return handle->options_last_offset;
+}
+
+static int handle_option_done(struct tracecmd_input *handle, char *buf, int size)
+{
+ unsigned long long offset;
+
+ if (size < 8)
+ return -1;
+ offset = lseek64(handle->fd, 0, SEEK_CUR);
+ if (offset >= size)
+ handle->options_last_offset = offset - size;
+ offset = tep_read_number(handle->pevent, buf, 8);
+ if (!offset)
+ return 0;
+
+ if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1)
+ return -1;
+
+ return handle_options(handle);
+}
+
#define save_read_number(R, C) \
do { \
if ((C) > size) \
@@ -3159,6 +3184,12 @@ static int handle_options(struct tracecmd_input *handle)
break;
add_section(handle, option, -1, tep_read_number(handle->pevent, buf, 8), 0);
break;
+ case TRACECMD_OPTION_DONE:
+ if (comperss)
+ in_uncompress_reset(handle);
+ ret = handle_option_done(handle, buf, size);
+ free(buf);
+ return ret;
default:
tracecmd_warning("unknown option %d", option);
break;
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 87bbcf39..01c1a4f5 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -2413,7 +2413,7 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd)
tep_ref(handle->pevent);
handle->page_size = tracecmd_page_size(ihandle);
handle->file_version = tracecmd_get_in_file_version(ihandle);
- handle->options_start = tracecmd_get_options_offset(ihandle);
+ handle->options_start = get_last_option_offset(ihandle);
list_head_init(&handle->options);
list_head_init(&handle->buffers);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 49/87] trace-cmd library: Handle the extended DONE option
2021-07-29 5:09 ` [PATCH v2 49/87] trace-cmd library: Handle the extended DONE option Tzvetomir Stoyanov (VMware)
@ 2021-08-19 19:13 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 19:13 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:21 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
\
> @@ -3159,6 +3184,12 @@ static int handle_options(struct tracecmd_input *handle)
> break;
> add_section(handle, option, -1, tep_read_number(handle->pevent, buf, 8), 0);
> break;
> + case TRACECMD_OPTION_DONE:
> + if (comperss)
s/comperss/compress/
And it needs to be fixed where it was introduced.
-- Steve
> + in_uncompress_reset(handle);
> + ret = handle_option_done(handle, buf, size);
> + free(buf);
> + return ret;
> default:
> tracecmd_warning("unknown option %d", option);
> break;
> diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
> index 87bbcf39..01c1a4f5 100644
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 50/87] trace-cmd library: Read compression header
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (48 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 49/87] trace-cmd library: Handle the extended DONE option Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 19:15 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 51/87] trace-cmd library: Extend the input handler with trace data decompression context Tzvetomir Stoyanov (VMware)
` (36 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file version 7 introduced new mandatory compression header,
storing information about the compression algorithm used to compress the
trace file. Added code to read that header and to initialize compression
context according to it.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 35 +++++++++++++++++++++++++++++++++--
1 file changed, 33 insertions(+), 2 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 9ab20d64..520d611f 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -3684,7 +3684,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
char test[] = TRACECMD_MAGIC;
unsigned int page_size;
size_t offset;
- char *version;
+ char *version = NULL;
+ char *zver = NULL;
+ char *zname = NULL;
char buf[BUFSIZ];
unsigned long ver;
@@ -3723,6 +3725,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
}
handle->file_version = ver;
free(version);
+ version = NULL;
if (do_read_check(handle, buf, 1))
goto failed_read;
@@ -3752,11 +3755,38 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
handle->total_file_size = lseek64(handle->fd, 0, SEEK_END);
lseek64(handle->fd, offset, SEEK_SET);
+ if (handle->file_version >= 7) {
+ zname = read_string(handle);
+ if (!zname)
+ goto failed_read;
+ zver = read_string(handle);
+ if (!zver)
+ goto failed_read;
+ if (strcmp(zname, "none")) {
+ handle->compress = tracecmd_compress_alloc(zname, zver,
+ handle->fd,
+ handle->pevent, NULL);
+ if (!handle->compress) {
+ tracecmd_warning("Unsupported file compression %s %s", zname, zver);
+ goto failed_read;
+ }
+ }
+ if (read8(handle, &(handle->options_start))) {
+ tracecmd_warning("Filed to read the offset of the first option section");
+ goto failed_read;
+ }
+ free(zname);
+ free(zver);
+ }
+
handle->file_state = TRACECMD_FILE_INIT;
return handle;
failed_read:
+ free(version);
+ free(zname);
+ free(zver);
free(handle);
return NULL;
@@ -3952,7 +3982,8 @@ void tracecmd_close(struct tracecmd_input *handle)
if (handle->flags & TRACECMD_FL_BUFFER_INSTANCE)
tracecmd_close(handle->parent);
else {
- /* Only main handle frees plugins and pevent */
+ /* Only main handle frees plugins, pevent and compression context */
+ tracecmd_compress_destroy(handle->compress);
tep_unload_plugins(handle->plugin_list, handle->pevent);
tep_free(handle->pevent);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 50/87] trace-cmd library: Read compression header
2021-07-29 5:09 ` [PATCH v2 50/87] trace-cmd library: Read compression header Tzvetomir Stoyanov (VMware)
@ 2021-08-19 19:15 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 19:15 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:22 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> @@ -3752,11 +3755,38 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
> handle->total_file_size = lseek64(handle->fd, 0, SEEK_END);
> lseek64(handle->fd, offset, SEEK_SET);
>
> + if (handle->file_version >= 7) {
In the next version, remove all comparisons to 7, and create feature
flags. Then in one place, where the version is decided on, it will set
the feature flags that are supported by that version. And everyplace
else, references the features, not the number. Because 7 is really
meaningless, and years from now, will just cause more confusion.
-- Steve
> + zname = read_string(handle);
> + if (!zname)
> + goto failed_read;
> + zver = read_string(handle);
> + if (!zver)
> + goto failed_read;
> + if (strcmp(zname, "none")) {
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 51/87] trace-cmd library: Extend the input handler with trace data decompression context
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (49 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 50/87] trace-cmd library: Read compression header Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 19:18 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 52/87] trace-cmd library: Initialize CPU data decompression logic Tzvetomir Stoyanov (VMware)
` (35 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The CPU tarce data is compressed in chunks, as chunk's size is multiple
trace pages. The input handler is extended with the necessary
structures, to control the data decompression. There are two approaches
for data decompression, both are supported and can be used in different
use cases:
- in-memory decompression, page by page.
- using a temporary file
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 66 ++++++++++++++++++++++++++++++-------
1 file changed, 54 insertions(+), 12 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 520d611f..6fb63c0f 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -29,6 +29,9 @@
#define COMMIT_MASK ((1 << 27) - 1)
+/* force uncompressing in memory */
+#define INMEMORY_DECOMPRESS
+
/* for debugging read instead of mmap */
static int force_read = 0;
@@ -54,6 +57,24 @@ struct page {
#endif
};
+struct zchunk_cache {
+ struct list_head list;
+ struct tracecmd_compress_chunk *chunk;
+ void *map;
+ int ref;
+};
+
+struct cpu_zdata {
+ /* uncompressed cpu data */
+ int fd;
+ char file[26]; /* strlen(COMPR_TEMP_FILE) */
+ unsigned int count;
+ unsigned int last_chunk;
+ struct list_head cache;
+ struct tracecmd_compress_chunk *chunks;
+};
+
+#define COMPR_TEMP_FILE "/tmp/trace_cpu_dataXXXXXX"
struct cpu_data {
/* the first two never change */
unsigned long long file_offset;
@@ -72,6 +93,7 @@ struct cpu_data {
int page_cnt;
int cpu;
int pipe_fd;
+ struct cpu_zdata compress;
};
struct cpu_file_data {
@@ -150,6 +172,8 @@ struct tracecmd_input {
bool use_trace_clock;
bool read_page;
bool use_pipe;
+ bool read_zpage; /* uncompress pages in memory, do not use tmp files */
+ bool cpu_compressed;
int file_version;
struct cpu_data *cpu_data;
long long ts_offset;
@@ -3284,7 +3308,7 @@ static int read_cpu_data(struct tracecmd_input *handle)
unsigned long long offset;
handle->cpu_data[cpu].cpu = cpu;
-
+ handle->cpu_data[cpu].compress.fd = -1;
handle->cpu_data[cpu].kbuf = kbuffer_alloc(long_size, endian);
if (!handle->cpu_data[cpu].kbuf)
goto out_free;
@@ -3701,6 +3725,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
/* By default, use usecs, unless told otherwise */
handle->flags |= TRACECMD_FL_IN_USECS;
+#ifdef INMEMORY_DECOMPRESS
+ handle->read_zpage = 1;
+#endif
if (do_read_check(handle, buf, 3))
goto failed_read;
@@ -3915,6 +3942,7 @@ void tracecmd_ref(struct tracecmd_input *handle)
*/
void tracecmd_close(struct tracecmd_input *handle)
{
+ struct zchunk_cache *cache;
struct file_section *del_sec;
int i;
@@ -3933,17 +3961,31 @@ void tracecmd_close(struct tracecmd_input *handle)
/* The tracecmd_peek_data may have cached a record */
free_next(handle, i);
free_page(handle, i);
- if (handle->cpu_data && handle->cpu_data[i].kbuf) {
- kbuffer_free(handle->cpu_data[i].kbuf);
- if (handle->cpu_data[i].page_map)
- free_page_map(handle->cpu_data[i].page_map);
-
- if (handle->cpu_data[i].page_cnt)
- tracecmd_warning("%d pages still allocated on cpu %d%s",
- handle->cpu_data[i].page_cnt, i,
- show_records(handle->cpu_data[i].pages,
- handle->cpu_data[i].nr_pages));
- free(handle->cpu_data[i].pages);
+ if (handle->cpu_data) {
+ if (handle->cpu_data[i].kbuf) {
+ kbuffer_free(handle->cpu_data[i].kbuf);
+ if (handle->cpu_data[i].page_map)
+ free_page_map(handle->cpu_data[i].page_map);
+
+ if (handle->cpu_data[i].page_cnt)
+ tracecmd_warning("%d pages still allocated on cpu %d%s",
+ handle->cpu_data[i].page_cnt, i,
+ show_records(handle->cpu_data[i].pages,
+ handle->cpu_data[i].nr_pages));
+ free(handle->cpu_data[i].pages);
+ }
+ if (handle->cpu_data[i].compress.fd >= 0) {
+ close(handle->cpu_data[i].compress.fd);
+ unlink(handle->cpu_data[i].compress.file);
+ }
+ while (!list_empty(&handle->cpu_data[i].compress.cache)) {
+ cache = container_of(handle->cpu_data[i].compress.cache.next,
+ struct zchunk_cache, list);
+ list_del(&cache->list);
+ free(cache->map);
+ free(cache);
+ }
+ free(handle->cpu_data[i].compress.chunks);
}
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 51/87] trace-cmd library: Extend the input handler with trace data decompression context
2021-07-29 5:09 ` [PATCH v2 51/87] trace-cmd library: Extend the input handler with trace data decompression context Tzvetomir Stoyanov (VMware)
@ 2021-08-19 19:18 ` Steven Rostedt
2021-09-02 12:46 ` Tzvetomir Stoyanov
0 siblings, 1 reply; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 19:18 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:23 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> The CPU tarce data is compressed in chunks, as chunk's size is multiple
typos.
> trace pages. The input handler is extended with the necessary
> structures, to control the data decompression. There are two approaches
> for data decompression, both are supported and can be used in different
> use cases:
> - in-memory decompression, page by page.
> - using a temporary file
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> lib/trace-cmd/trace-input.c | 66 ++++++++++++++++++++++++++++++-------
> 1 file changed, 54 insertions(+), 12 deletions(-)
>
> diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
> index 520d611f..6fb63c0f 100644
> --- a/lib/trace-cmd/trace-input.c
> +++ b/lib/trace-cmd/trace-input.c
> @@ -29,6 +29,9 @@
>
> #define COMMIT_MASK ((1 << 27) - 1)
>
> +/* force uncompressing in memory */
> +#define INMEMORY_DECOMPRESS
I wonder if we should just make this a variable in the handle, and
default it to in memory. Then it will be easy to change it from the
command line if we want to.
-- Steve
> +
> /* for debugging read instead of mmap */
> static int force_read = 0;
>
> @@ -54,6 +57,24 @@ struct page {
> #endif
> };
>
> +struct zchunk_cache {
> + struct list_head list;
> + struct tracecmd_compress_chunk *chunk;
> + void *map;
> + int ref;
> +};
> +
> +struct cpu_zdata {
> + /* uncompressed cpu data */
> + int fd;
> + char file[26]; /*
> strlen(COMPR_TEMP_FILE) */
> + unsigned int count;
> + unsigned int last_chunk;
> + struct list_head cache;
> + struct tracecmd_compress_chunk *chunks;
> +};
> +
> +#define COMPR_TEMP_FILE "/tmp/trace_cpu_dataXXXXXX"
> struct cpu_data {
> /* the first two never change */
> unsigned long long file_offset;
> @@ -72,6 +93,7 @@ struct cpu_data {
> int page_cnt;
> int cpu;
> int pipe_fd;
> + struct cpu_zdata compress;
> };
>
> struct cpu_file_data {
> @@ -150,6 +172,8 @@ struct tracecmd_input {
> bool use_trace_clock;
> bool read_page;
> bool use_pipe;
> + bool read_zpage; /* uncompress pages
> in memory, do not use tmp files */
> + bool cpu_compressed;
> int file_version;
> struct cpu_data *cpu_data;
> long long ts_offset;
> @@ -3284,7 +3308,7 @@ static int read_cpu_data(struct tracecmd_input
> *handle) unsigned long long offset;
>
> handle->cpu_data[cpu].cpu = cpu;
> -
> + handle->cpu_data[cpu].compress.fd = -1;
> handle->cpu_data[cpu].kbuf =
> kbuffer_alloc(long_size, endian); if (!handle->cpu_data[cpu].kbuf)
> goto out_free;
> @@ -3701,6 +3725,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int
> fd, int flags) /* By default, use usecs, unless told otherwise */
> handle->flags |= TRACECMD_FL_IN_USECS;
>
> +#ifdef INMEMORY_DECOMPRESS
> + handle->read_zpage = 1;
> +#endif
> if (do_read_check(handle, buf, 3))
> goto failed_read;
>
> @@ -3915,6 +3942,7 @@ void tracecmd_ref(struct tracecmd_input *handle)
> */
> void tracecmd_close(struct tracecmd_input *handle)
> {
> + struct zchunk_cache *cache;
> struct file_section *del_sec;
> int i;
>
> @@ -3933,17 +3961,31 @@ void tracecmd_close(struct tracecmd_input
> *handle) /* The tracecmd_peek_data may have cached a record */
> free_next(handle, i);
> free_page(handle, i);
> - if (handle->cpu_data && handle->cpu_data[i].kbuf) {
> - kbuffer_free(handle->cpu_data[i].kbuf);
> - if (handle->cpu_data[i].page_map)
> -
> free_page_map(handle->cpu_data[i].page_map); -
> - if (handle->cpu_data[i].page_cnt)
> - tracecmd_warning("%d pages still
> allocated on cpu %d%s",
> -
> handle->cpu_data[i].page_cnt, i,
> -
> show_records(handle->cpu_data[i].pages,
> -
> handle->cpu_data[i].nr_pages));
> - free(handle->cpu_data[i].pages);
> + if (handle->cpu_data) {
> + if (handle->cpu_data[i].kbuf) {
> +
> kbuffer_free(handle->cpu_data[i].kbuf);
> + if (handle->cpu_data[i].page_map)
> +
> free_page_map(handle->cpu_data[i].page_map); +
> + if (handle->cpu_data[i].page_cnt)
> + tracecmd_warning("%d pages
> still allocated on cpu %d%s",
> +
> handle->cpu_data[i].page_cnt, i,
> +
> show_records(handle->cpu_data[i].pages,
> +
> handle->cpu_data[i].nr_pages));
> + free(handle->cpu_data[i].pages);
> + }
> + if (handle->cpu_data[i].compress.fd >= 0) {
> +
> close(handle->cpu_data[i].compress.fd);
> +
> unlink(handle->cpu_data[i].compress.file);
> + }
> + while
> (!list_empty(&handle->cpu_data[i].compress.cache)) {
> + cache =
> container_of(handle->cpu_data[i].compress.cache.next,
> + struct
> zchunk_cache, list);
> + list_del(&cache->list);
> + free(cache->map);
> + free(cache);
> + }
> + free(handle->cpu_data[i].compress.chunks);
> }
> }
>
^ permalink raw reply [flat|nested] 118+ messages in thread* Re: [PATCH v2 51/87] trace-cmd library: Extend the input handler with trace data decompression context
2021-08-19 19:18 ` Steven Rostedt
@ 2021-09-02 12:46 ` Tzvetomir Stoyanov
0 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov @ 2021-09-02 12:46 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linux Trace Devel
On Thu, Aug 19, 2021 at 10:18 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 29 Jul 2021 08:09:23 +0300
> "Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
>
> > The CPU tarce data is compressed in chunks, as chunk's size is multiple
>
> typos.
>
> > trace pages. The input handler is extended with the necessary
> > structures, to control the data decompression. There are two approaches
> > for data decompression, both are supported and can be used in different
> > use cases:
> > - in-memory decompression, page by page.
> > - using a temporary file
> >
> > Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> > ---
> > lib/trace-cmd/trace-input.c | 66 ++++++++++++++++++++++++++++++-------
> > 1 file changed, 54 insertions(+), 12 deletions(-)
> >
> > diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
> > index 520d611f..6fb63c0f 100644
> > --- a/lib/trace-cmd/trace-input.c
> > +++ b/lib/trace-cmd/trace-input.c
> > @@ -29,6 +29,9 @@
> >
> > #define COMMIT_MASK ((1 << 27) - 1)
> >
> > +/* force uncompressing in memory */
> > +#define INMEMORY_DECOMPRESS
>
> I wonder if we should just make this a variable in the handle, and
> default it to in memory. Then it will be easy to change it from the
> command line if we want to.
There is "read_zpage", introduced in this patch. It controls whether
to use in memory decompression. The define is used to set the default
value of read_zpage, when the handler is allocated.
>
> -- Steve
>
> > +
> > /* for debugging read instead of mmap */
> > static int force_read = 0;
> >
> > @@ -54,6 +57,24 @@ struct page {
> > #endif
> > };
> >
> > +struct zchunk_cache {
> > + struct list_head list;
> > + struct tracecmd_compress_chunk *chunk;
> > + void *map;
> > + int ref;
> > +};
> > +
> > +struct cpu_zdata {
> > + /* uncompressed cpu data */
> > + int fd;
> > + char file[26]; /*
> > strlen(COMPR_TEMP_FILE) */
> > + unsigned int count;
> > + unsigned int last_chunk;
> > + struct list_head cache;
> > + struct tracecmd_compress_chunk *chunks;
> > +};
> > +
> > +#define COMPR_TEMP_FILE "/tmp/trace_cpu_dataXXXXXX"
> > struct cpu_data {
> > /* the first two never change */
> > unsigned long long file_offset;
> > @@ -72,6 +93,7 @@ struct cpu_data {
> > int page_cnt;
> > int cpu;
> > int pipe_fd;
> > + struct cpu_zdata compress;
> > };
> >
> > struct cpu_file_data {
> > @@ -150,6 +172,8 @@ struct tracecmd_input {
> > bool use_trace_clock;
> > bool read_page;
> > bool use_pipe;
> > + bool read_zpage; /* uncompress pages
> > in memory, do not use tmp files */
> > + bool cpu_compressed;
> > int file_version;
> > struct cpu_data *cpu_data;
> > long long ts_offset;
> > @@ -3284,7 +3308,7 @@ static int read_cpu_data(struct tracecmd_input
> > *handle) unsigned long long offset;
> >
> > handle->cpu_data[cpu].cpu = cpu;
> > -
> > + handle->cpu_data[cpu].compress.fd = -1;
> > handle->cpu_data[cpu].kbuf =
> > kbuffer_alloc(long_size, endian); if (!handle->cpu_data[cpu].kbuf)
> > goto out_free;
> > @@ -3701,6 +3725,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int
> > fd, int flags) /* By default, use usecs, unless told otherwise */
> > handle->flags |= TRACECMD_FL_IN_USECS;
> >
> > +#ifdef INMEMORY_DECOMPRESS
> > + handle->read_zpage = 1;
> > +#endif
> > if (do_read_check(handle, buf, 3))
> > goto failed_read;
> >
> > @@ -3915,6 +3942,7 @@ void tracecmd_ref(struct tracecmd_input *handle)
> > */
> > void tracecmd_close(struct tracecmd_input *handle)
> > {
> > + struct zchunk_cache *cache;
> > struct file_section *del_sec;
> > int i;
> >
> > @@ -3933,17 +3961,31 @@ void tracecmd_close(struct tracecmd_input
> > *handle) /* The tracecmd_peek_data may have cached a record */
> > free_next(handle, i);
> > free_page(handle, i);
> > - if (handle->cpu_data && handle->cpu_data[i].kbuf) {
> > - kbuffer_free(handle->cpu_data[i].kbuf);
> > - if (handle->cpu_data[i].page_map)
> > -
> > free_page_map(handle->cpu_data[i].page_map); -
> > - if (handle->cpu_data[i].page_cnt)
> > - tracecmd_warning("%d pages still
> > allocated on cpu %d%s",
> > -
> > handle->cpu_data[i].page_cnt, i,
> > -
> > show_records(handle->cpu_data[i].pages,
> > -
> > handle->cpu_data[i].nr_pages));
> > - free(handle->cpu_data[i].pages);
> > + if (handle->cpu_data) {
> > + if (handle->cpu_data[i].kbuf) {
> > +
> > kbuffer_free(handle->cpu_data[i].kbuf);
> > + if (handle->cpu_data[i].page_map)
> > +
> > free_page_map(handle->cpu_data[i].page_map); +
> > + if (handle->cpu_data[i].page_cnt)
> > + tracecmd_warning("%d pages
> > still allocated on cpu %d%s",
> > +
> > handle->cpu_data[i].page_cnt, i,
> > +
> > show_records(handle->cpu_data[i].pages,
> > +
> > handle->cpu_data[i].nr_pages));
> > + free(handle->cpu_data[i].pages);
> > + }
> > + if (handle->cpu_data[i].compress.fd >= 0) {
> > +
> > close(handle->cpu_data[i].compress.fd);
> > +
> > unlink(handle->cpu_data[i].compress.file);
> > + }
> > + while
> > (!list_empty(&handle->cpu_data[i].compress.cache)) {
> > + cache =
> > container_of(handle->cpu_data[i].compress.cache.next,
> > + struct
> > zchunk_cache, list);
> > + list_del(&cache->list);
> > + free(cache->map);
> > + free(cache);
> > + }
> > + free(handle->cpu_data[i].compress.chunks);
> > }
> > }
> >
>
--
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 52/87] trace-cmd library: Initialize CPU data decompression logic
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (50 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 51/87] trace-cmd library: Extend the input handler with trace data decompression context Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 53/87] trace-cmd library: Initialize CPU data for reading from version 7 trace files Tzvetomir Stoyanov (VMware)
` (34 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
On CPU data initialization stage, initialize decompression context for
both in-memory and temporary file decompression logics.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 70 +++++++++++++++++++++++++++++++++++--
1 file changed, 67 insertions(+), 3 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 6fb63c0f..f3783a98 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -1275,6 +1275,7 @@ static void *allocate_page_map(struct tracecmd_input *handle,
off64_t map_offset;
void *map;
int ret;
+ int fd;
if (handle->read_page) {
map = malloc(handle->page_size);
@@ -1314,12 +1315,15 @@ static void *allocate_page_map(struct tracecmd_input *handle,
map_size -= map_offset + map_size -
(cpu_data->file_offset + cpu_data->file_size);
+ if (cpu_data->compress.fd >= 0)
+ fd = cpu_data->compress.fd;
+ else
+ fd = handle->fd;
again:
page_map->size = map_size;
page_map->offset = map_offset;
- page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE,
- handle->fd, map_offset);
+ page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, fd, map_offset);
if (page_map->map == MAP_FAILED) {
/* Try a smaller map */
@@ -2511,16 +2515,76 @@ tracecmd_read_prev(struct tracecmd_input *handle, struct tep_record *record)
/* Not reached */
}
-static int init_cpu(struct tracecmd_input *handle, int cpu)
+static int init_cpu_zfile(struct tracecmd_input *handle, int cpu)
+{
+ struct cpu_data *cpu_data;
+ unsigned long long size;
+ off64_t offset;
+
+ cpu_data = &handle->cpu_data[cpu];
+ offset = lseek64(handle->fd, 0, SEEK_CUR);
+ if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1)
+ return -1;
+ strcpy(cpu_data->compress.file, COMPR_TEMP_FILE);
+ cpu_data->compress.fd = mkstemp(cpu_data->compress.file);
+ if (cpu_data->compress.fd < 0)
+ return -1;
+ if (tracecmd_uncompress_copy_to(handle->compress, cpu_data->compress.fd, NULL, &size))
+ return -1;
+ if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1)
+ return -1;
+ cpu_data->offset = 0;
+ cpu_data->file_offset = 0;
+ cpu_data->file_size = size;
+ cpu_data->size = size;
+ return 0;
+}
+
+static int init_cpu_zpage(struct tracecmd_input *handle, int cpu)
{
struct cpu_data *cpu_data = &handle->cpu_data[cpu];
+ int count;
int i;
+ if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1)
+ return -1;
+
+ count = tracecmd_load_chunks_info(handle->compress, &cpu_data->compress.chunks);
+ if (count < 0)
+ return -1;
+ cpu_data->compress.count = count;
+ cpu_data->compress.last_chunk = 0;
+
+ cpu_data->file_offset = 0;
+ cpu_data->file_size = 0;
+ for (i = 0; i < count; i++)
+ cpu_data->file_size += cpu_data->compress.chunks[i].size;
cpu_data->offset = cpu_data->file_offset;
cpu_data->size = cpu_data->file_size;
+ return 0;
+}
+
+static int init_cpu(struct tracecmd_input *handle, int cpu)
+{
+ struct cpu_data *cpu_data = &handle->cpu_data[cpu];
+ int ret;
+ int i;
+
+ if (handle->cpu_compressed && cpu_data->file_size > 0) {
+ if (handle->read_zpage)
+ ret = init_cpu_zpage(handle, cpu);
+ else
+ ret = init_cpu_zfile(handle, cpu);
+ if (ret)
+ return ret;
+ } else {
+ cpu_data->offset = cpu_data->file_offset;
+ cpu_data->size = cpu_data->file_size;
+ }
cpu_data->timestamp = 0;
list_head_init(&cpu_data->page_maps);
+ list_head_init(&cpu_data->compress.cache);
if (!cpu_data->size) {
printf("CPU %d is empty\n", cpu);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 53/87] trace-cmd library: Initialize CPU data for reading from version 7 trace files
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (51 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 52/87] trace-cmd library: Initialize CPU data decompression logic Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 54/87] trace-cmd library: Add logic for in-memory decompression Tzvetomir Stoyanov (VMware)
` (33 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In version 7 trace files, CPU trace data is written in slightly
different way than in version 6 files. Added new CPU data initialization
flow, to handle version 7 files:
- the top trace instance is saved in the same way as the other trace
instances.
- per CPU trace metadata is stored in the buffer option.
- trace data section has section header.
- trace data can be compressed.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 178 +++++++++++++++++++++++++-----------
1 file changed, 124 insertions(+), 54 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index f3783a98..3b58cb0a 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -3327,34 +3327,18 @@ static int read_options_type(struct tracecmd_input *handle)
return 0;
}
-static int read_cpu_data(struct tracecmd_input *handle)
+static int init_cpu_data(struct tracecmd_input *handle)
{
- struct tep_handle *pevent = handle->pevent;
enum kbuffer_long_size long_size;
enum kbuffer_endian endian;
- unsigned long long size;
unsigned long long max_size = 0;
unsigned long long pages;
- int cpus;
int cpu;
- /*
- * Check if this is a latency report or not.
- */
- if (handle->file_state == TRACECMD_FILE_CPU_LATENCY)
- return 1;
-
/* We expect this to be flyrecord */
if (handle->file_state != TRACECMD_FILE_CPU_FLYRECORD)
return -1;
- cpus = handle->cpus;
-
- handle->cpu_data = malloc(sizeof(*handle->cpu_data) * handle->cpus);
- if (!handle->cpu_data)
- return -1;
- memset(handle->cpu_data, 0, sizeof(*handle->cpu_data) * handle->cpus);
-
if (force_read)
handle->read_page = true;
@@ -3369,32 +3353,15 @@ static int read_cpu_data(struct tracecmd_input *handle)
endian = KBUFFER_ENDIAN_LITTLE;
for (cpu = 0; cpu < handle->cpus; cpu++) {
- unsigned long long offset;
-
- handle->cpu_data[cpu].cpu = cpu;
handle->cpu_data[cpu].compress.fd = -1;
handle->cpu_data[cpu].kbuf = kbuffer_alloc(long_size, endian);
if (!handle->cpu_data[cpu].kbuf)
goto out_free;
- if (tep_is_old_format(pevent))
+ if (tep_is_old_format(handle->pevent))
kbuffer_set_old_format(handle->cpu_data[cpu].kbuf);
- read8(handle, &offset);
- read8(handle, &size);
-
- handle->cpu_data[cpu].file_offset = offset;
- handle->cpu_data[cpu].file_size = size;
- if (size > max_size)
- max_size = size;
-
- if (size && (offset + size > handle->total_file_size)) {
- /* this happens if the file got truncated */
- printf("File possibly truncated. "
- "Need at least %llu, but file size is %zu.\n",
- offset + size, handle->total_file_size);
- errno = EINVAL;
- goto out_free;
- }
+ if (handle->cpu_data[cpu].file_size > max_size)
+ max_size = handle->cpu_data[cpu].file_size;
}
/* Calculate about a meg of pages for buffering */
@@ -3412,6 +3379,102 @@ static int read_cpu_data(struct tracecmd_input *handle)
goto out_free;
}
+ return 0;
+
+ out_free:
+ for ( ; cpu >= 0; cpu--) {
+ free_page(handle, cpu);
+ kbuffer_free(handle->cpu_data[cpu].kbuf);
+ handle->cpu_data[cpu].kbuf = NULL;
+ }
+ return -1;
+}
+
+static int init_buffer_cpu_data(struct tracecmd_input *handle, struct input_buffer_instance *buffer)
+{
+ unsigned long long offset;
+ unsigned long long size;
+ unsigned short id, flags;
+ int cpu;
+
+ if (handle->cpu_data)
+ return -1;
+
+ if (lseek64(handle->fd, buffer->offset, SEEK_SET) == (off_t)-1)
+ return -1;
+ if (read_section_header(handle, &id, &flags, NULL, NULL))
+ return -1;
+ if (flags & TRACECMD_SEC_FL_COMPRESS)
+ handle->cpu_compressed = true;
+ handle->file_state = TRACECMD_FILE_CPU_FLYRECORD;
+ handle->cpus = buffer->cpus;
+ if (handle->max_cpu < handle->cpus)
+ handle->max_cpu = handle->cpus;
+
+ handle->cpu_data = calloc(handle->cpus, sizeof(*handle->cpu_data));
+ if (!handle->cpu_data)
+ return -1;
+
+ for (cpu = 0; cpu < handle->cpus; cpu++) {
+ handle->cpu_data[cpu].cpu = buffer->cpu_data[cpu].cpu;
+ offset = buffer->cpu_data[cpu].offset;
+ size = buffer->cpu_data[cpu].size;
+ handle->cpu_data[cpu].file_offset = offset;
+ handle->cpu_data[cpu].file_size = size;
+ if (size && (offset + size > handle->total_file_size)) {
+ /* this happens if the file got truncated */
+ printf("File possibly truncated. "
+ "Need at least %llu, but file size is %zu.\n",
+ offset + size, handle->total_file_size);
+ errno = EINVAL;
+ return -1;
+ }
+ }
+
+ return init_cpu_data(handle);
+}
+
+static int read_cpu_data(struct tracecmd_input *handle)
+{
+ unsigned long long size;
+ int cpus;
+ int cpu;
+
+ /*
+ * Check if this is a latency report or not.
+ */
+ if (handle->file_state == TRACECMD_FILE_CPU_LATENCY)
+ return 1;
+
+ /* We expect this to be flyrecord */
+ if (handle->file_state != TRACECMD_FILE_CPU_FLYRECORD)
+ return -1;
+
+ cpus = handle->cpus;
+
+ handle->cpu_data = malloc(sizeof(*handle->cpu_data) * handle->cpus);
+ if (!handle->cpu_data)
+ return -1;
+ memset(handle->cpu_data, 0, sizeof(*handle->cpu_data) * handle->cpus);
+
+ for (cpu = 0; cpu < handle->cpus; cpu++) {
+ unsigned long long offset;
+
+ handle->cpu_data[cpu].cpu = cpu;
+ read8(handle, &offset);
+ read8(handle, &size);
+ handle->cpu_data[cpu].file_offset = offset;
+ handle->cpu_data[cpu].file_size = size;
+ if (size && (offset + size > handle->total_file_size)) {
+ /* this happens if the file got truncated */
+ printf("File possibly truncated. "
+ "Need at least %llu, but file size is %zu.\n",
+ offset + size, handle->total_file_size);
+ errno = EINVAL;
+ return -1;
+ }
+ }
+
/*
* It is possible that an option changed the number of CPUs.
* If that happened, then there's "empty" cpu data saved for
@@ -3431,15 +3494,7 @@ static int read_cpu_data(struct tracecmd_input *handle)
}
}
- return 0;
-
- out_free:
- for ( ; cpu >= 0; cpu--) {
- free_page(handle, cpu);
- kbuffer_free(handle->cpu_data[cpu].kbuf);
- handle->cpu_data[cpu].kbuf = NULL;
- }
- return -1;
+ return init_cpu_data(handle);
}
static int read_data_and_size(struct tracecmd_input *handle,
@@ -3542,14 +3597,7 @@ static int read_and_parse_trace_clock(struct tracecmd_input *handle,
return 0;
}
-/**
- * tracecmd_init_data - prepare reading the data from trace.dat
- * @handle: input handle for the trace.dat file
- *
- * This prepares reading the data from trace.dat. This is called
- * after tracecmd_read_headers() and before tracecmd_read_data().
- */
-int tracecmd_init_data(struct tracecmd_input *handle)
+static int init_data_v6(struct tracecmd_input *handle)
{
struct tep_handle *pevent = handle->pevent;
int ret;
@@ -3571,7 +3619,29 @@ int tracecmd_init_data(struct tracecmd_input *handle)
tracecmd_parse_trace_clock(handle, clock, 8);
}
}
+ return ret;
+}
+
+static int init_data_v7(struct tracecmd_input *handle)
+{
+ return init_buffer_cpu_data(handle, &handle->top_buffer);
+}
+
+/**
+ * tracecmd_init_data - prepare reading the data from trace.dat
+ * @handle: input handle for the trace.dat file
+ *
+ * This prepares reading the data from trace.dat. This is called
+ * after tracecmd_read_headers() and before tracecmd_read_data().
+ */
+int tracecmd_init_data(struct tracecmd_input *handle)
+{
+ int ret;
+ if (handle->file_version < 7)
+ ret = init_data_v6(handle);
+ else
+ ret = init_data_v7(handle);
tracecmd_blk_hack(handle);
return ret;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 54/87] trace-cmd library: Add logic for in-memory decompression
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (52 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 53/87] trace-cmd library: Initialize CPU data for reading from version 7 trace files Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 55/87] trace-cmd library: Handle latency trace in version 7 files Tzvetomir Stoyanov (VMware)
` (32 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
There are two approaches to read compressed trace data:
- use a temporary file to decompress entire trace data before reading
- use in-memory decompression of requested trace data chunk only
In-memory decompression seems to be more efficient, but selecting which
approach to use depends in the use case.
A compression chunk consists of multiple trace pages, that's why a small
cache with uncompressed chunks is implemented. The chunk stays in the
cache until there are pages which have reference to it.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 104 ++++++++++++++++++++++++++++++++++++
1 file changed, 104 insertions(+)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 3b58cb0a..4582b9bc 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -1266,6 +1266,105 @@ static void free_page_map(struct page_map *page_map)
free(page_map);
}
+#define CHUNK_CHECK_OFFSET(C, O) ((O) >= (C)->offset && (O) < ((C)->offset + (C)->size))
+static struct tracecmd_compress_chunk *get_zchunk(struct cpu_data *cpu, off64_t offset)
+{
+ struct cpu_zdata *cpuz = &cpu->compress;
+ int min, mid, max;
+
+ if (!cpuz->chunks)
+ return NULL;
+ if (offset > (cpuz->chunks[cpuz->count - 1].offset + cpuz->chunks[cpuz->count - 1].size))
+ return NULL;
+
+ /* check if the requested offset is in the last requested chunk or in the next chunk */
+ if (CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset))
+ return cpuz->chunks + cpuz->last_chunk;
+ cpuz->last_chunk++;
+ if (cpuz->last_chunk < cpuz->count &&
+ CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset))
+ return cpuz->chunks + cpuz->last_chunk;
+
+ /* do a binary search to find the chunk holding the given offset */
+ min = 0;
+ max = cpuz->count - 1;
+ mid = (min + max)/2;
+ while (min <= max) {
+ if (offset < cpuz->chunks[mid].offset)
+ max = mid - 1;
+ else if (offset > (cpuz->chunks[mid].offset + cpuz->chunks[mid].size))
+ min = mid + 1;
+ else
+ break;
+ mid = (min + max)/2;
+ }
+ cpuz->last_chunk = mid;
+ return cpuz->chunks + mid;
+}
+
+static void free_zpage(struct cpu_data *cpu_data, void *map)
+{
+ struct zchunk_cache *cache;
+
+ list_for_each_entry(cache, &cpu_data->compress.cache, list) {
+ if (map <= cache->map && map > (cache->map + cache->chunk->size))
+ goto found;
+ }
+ return;
+
+found:
+ cache->ref--;
+ if (cache->ref)
+ return;
+ list_del(&cache->list);
+ free(cache->map);
+ free(cache);
+}
+
+static void *read_zpage(struct tracecmd_input *handle, int cpu, off64_t offset)
+{
+ struct cpu_data *cpu_data = &handle->cpu_data[cpu];
+ struct tracecmd_compress_chunk *chunk;
+ struct zchunk_cache *cache;
+ void *map = NULL;
+ int pindex;
+ int size;
+
+ /* Look in the cache of already loaded chunks */
+ list_for_each_entry(cache, &cpu_data->compress.cache, list) {
+ if (CHUNK_CHECK_OFFSET(cache->chunk, offset)) {
+ cache->ref++;
+ goto out;
+ }
+ }
+
+ chunk = get_zchunk(cpu_data, offset);
+ if (!chunk)
+ return NULL;
+ size = handle->page_size > chunk->size ? handle->page_size : chunk->size;
+ map = malloc(size);
+ if (!map)
+ return NULL;
+ if (tracecmd_uncompress_chunk(handle->compress, chunk, map) < 0)
+ goto error;
+
+ cache = calloc(1, sizeof(struct zchunk_cache));
+ if (!cache)
+ goto error;
+ cache->ref = 1;
+ cache->chunk = chunk;
+ cache->map = map;
+ list_add(&cache->list, &cpu_data->compress.cache);
+
+ /* a chunk can hold multiple pages, get the requested one */
+out:
+ pindex = (offset - cache->chunk->offset) / handle->page_size;
+ return cache->map + (pindex * handle->page_size);
+error:
+ free(map);
+ return NULL;
+}
+
static void *allocate_page_map(struct tracecmd_input *handle,
struct page *page, int cpu, off64_t offset)
{
@@ -1277,6 +1376,9 @@ static void *allocate_page_map(struct tracecmd_input *handle,
int ret;
int fd;
+ if (handle->cpu_compressed && handle->read_zpage)
+ return read_zpage(handle, cpu, offset);
+
if (handle->read_page) {
map = malloc(handle->page_size);
if (!map)
@@ -1419,6 +1521,8 @@ static void __free_page(struct tracecmd_input *handle, struct page *page)
if (handle->read_page)
free(page->map);
+ else if (handle->read_zpage)
+ free_zpage(cpu_data, page->map);
else
free_page_map(page->page_map);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 55/87] trace-cmd library: Handle latency trace in version 7 files
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (53 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 54/87] trace-cmd library: Add logic for in-memory decompression Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 56/87] trace-cmd library: Handle buffer trace data init for " Tzvetomir Stoyanov (VMware)
` (31 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Latency trace data is saved the same was as flyrecord buffer data
in trace files version 7. There is a BUFFER_LAT option which hols the
latency specific trace metadata and points to the secrtion in the file
with the trace data. Extended the input hanlder with latency trace
context and added new logic to decompress and read the latency data.
A new API is added to read latency data:
tracecmd_latency_data_read()
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 2 +
lib/trace-cmd/trace-input.c | 91 ++++++++++++++++++-
2 files changed, 91 insertions(+), 2 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index f2868576..0a3e12cf 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -217,6 +217,8 @@ tracecmd_peek_data_ref(struct tracecmd_input *handle, int cpu)
return rec;
}
+int tracecmd_latency_data_read(struct tracecmd_input *handle, char **buf, size_t *size);
+
struct tep_record *
tracecmd_read_prev(struct tracecmd_input *handle, struct tep_record *record);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 4582b9bc..b9340160 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -175,6 +175,9 @@ struct tracecmd_input {
bool read_zpage; /* uncompress pages in memory, do not use tmp files */
bool cpu_compressed;
int file_version;
+ /* temporary file for uncompressed latency data */
+ struct cpu_zdata latz;
+
struct cpu_data *cpu_data;
long long ts_offset;
struct tsc2nsec tsc_calc;
@@ -3431,6 +3434,56 @@ static int read_options_type(struct tracecmd_input *handle)
return 0;
}
+int tracecmd_latency_data_read(struct tracecmd_input *handle, char **buf, size_t *size)
+{
+ struct cpu_zdata *zdata = &handle->latz;
+ void *data;
+ int rsize;
+ int fd = -1;
+ int id;
+
+ if (!handle || !buf || !size)
+ return -1;
+ if (handle->file_state != TRACECMD_FILE_CPU_LATENCY)
+ return -1;
+
+ if (!handle->cpu_compressed) {
+ fd = handle->fd;
+ } else if (!handle->read_zpage) {
+ if (zdata->fd < 0)
+ return -1;
+ fd = zdata->fd;
+ }
+
+ /* Read data from a file */
+ if (fd >= 0) {
+ if (!(*buf)) {
+ *size = BUFSIZ;
+ *buf = malloc(*size);
+ if (!(*buf))
+ return -1;
+ }
+ return do_read_fd(fd, *buf, *size);
+ }
+
+ /* Uncompress data in memory */
+ if (zdata->last_chunk >= zdata->count)
+ return 0;
+ id = zdata->last_chunk;
+ if (!*buf || *size < zdata->chunks[id].size) {
+ data = realloc(*buf, zdata->chunks[id].size);
+ if (!data)
+ return -1;
+ *buf = data;
+ *size = zdata->chunks[id].size;
+ }
+ if (tracecmd_uncompress_chunk(handle->compress, &zdata->chunks[id], *buf))
+ return -1;
+ rsize = zdata->chunks[id].size;
+ zdata->last_chunk++;
+ return rsize;
+}
+
static int init_cpu_data(struct tracecmd_input *handle)
{
enum kbuffer_long_size long_size;
@@ -3494,6 +3547,32 @@ static int init_cpu_data(struct tracecmd_input *handle)
return -1;
}
+int init_latency_data(struct tracecmd_input *handle)
+{
+ unsigned long long wsize;
+ int ret;
+
+ if (!handle->cpu_compressed)
+ return 0;
+
+ if (handle->read_zpage) {
+ handle->latz.count = tracecmd_load_chunks_info(handle->compress, &handle->latz.chunks);
+ if (handle->latz.count < 0)
+ return -1;
+ } else {
+ strcpy(handle->latz.file, COMPR_TEMP_FILE);
+ handle->latz.fd = mkstemp(handle->latz.file);
+ if (handle->latz.fd < 0)
+ return -1;
+ ret = tracecmd_uncompress_copy_to(handle->compress, handle->latz.fd, NULL, &wsize);
+ if (ret)
+ return -1;
+ lseek64(handle->latz.fd, 0, SEEK_SET);
+ }
+
+ return 0;
+}
+
static int init_buffer_cpu_data(struct tracecmd_input *handle, struct input_buffer_instance *buffer)
{
unsigned long long offset;
@@ -3510,6 +3589,10 @@ static int init_buffer_cpu_data(struct tracecmd_input *handle, struct input_buff
return -1;
if (flags & TRACECMD_SEC_FL_COMPRESS)
handle->cpu_compressed = true;
+ if (buffer->latency) {
+ handle->file_state = TRACECMD_FILE_CPU_LATENCY;
+ return init_latency_data(handle) == 0 ? 1 : -1;
+ }
handle->file_state = TRACECMD_FILE_CPU_FLYRECORD;
handle->cpus = buffer->cpus;
if (handle->max_cpu < handle->cpus)
@@ -3959,7 +4042,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags)
handle->fd = fd;
handle->ref = 1;
-
+ handle->latz.fd = -1;
/* By default, use usecs, unless told otherwise */
handle->flags |= TRACECMD_FL_IN_USECS;
@@ -4233,7 +4316,11 @@ void tracecmd_close(struct tracecmd_input *handle)
free(handle->trace_clock);
free(handle->version);
close(handle->fd);
-
+ free(handle->latz.chunks);
+ if (handle->latz.fd >= 0) {
+ close(handle->latz.fd);
+ unlink(handle->latz.file);
+ }
while (handle->sections) {
del_sec = handle->sections;
handle->sections = handle->sections->next;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 56/87] trace-cmd library: Handle buffer trace data init for version 7 files
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (54 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 55/87] trace-cmd library: Handle latency trace in version 7 files Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 57/87] trace-cmd report: Use the new latency API to read data Tzvetomir Stoyanov (VMware)
` (30 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
CPU data initialization is different for trace files version 6 and 7.
When a new input handler to trace buffer is created, initialize the CPU
data according to the file version.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 55 +++++++++++++++++++------------------
1 file changed, 29 insertions(+), 26 deletions(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index b9340160..2b98e6d8 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4809,34 +4809,37 @@ tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx)
new_handle->flags |= TRACECMD_FL_BUFFER_INSTANCE;
new_handle->pid_maps = NULL;
+ if (handle->file_version < 7) {
+ /* Save where we currently are */
+ offset = lseek64(handle->fd, 0, SEEK_CUR);
- /* Save where we currently are */
- offset = lseek64(handle->fd, 0, SEEK_CUR);
-
- ret = lseek64(handle->fd, buffer->offset, SEEK_SET);
- if (ret < 0) {
- tracecmd_warning("could not seek to buffer %s offset %ld\n",
- buffer->name, buffer->offset);
- goto error;
- }
-
- /*
- * read_options_type() is called right after the CPU count so update
- * file state accordingly.
- */
- new_handle->file_state = TRACECMD_FILE_CPU_COUNT;
- ret = read_options_type(new_handle);
- if (!ret)
- ret = read_cpu_data(new_handle);
- if (ret < 0) {
- tracecmd_warning("failed to read sub buffer %s\n", buffer->name);
- goto error;
- }
+ ret = lseek64(handle->fd, buffer->offset, SEEK_SET);
+ if (ret == (off64_t)-1) {
+ tracecmd_warning("could not seek to buffer %s offset %ld\n",
+ buffer->name, buffer->offset);
+ goto error;
+ }
+ /*
+ * read_options_type() is called right after the CPU count so update
+ * file state accordingly.
+ */
+ new_handle->file_state = TRACECMD_FILE_CPU_COUNT;
+ ret = read_options_type(new_handle);
+ if (!ret)
+ ret = read_cpu_data(new_handle);
- ret = lseek64(handle->fd, offset, SEEK_SET);
- if (ret < 0) {
- tracecmd_warning("could not seek to back to offset %ld\n", offset);
- goto error;
+ if (ret < 0) {
+ tracecmd_warning("failed to read sub buffer %s\n", buffer->name);
+ goto error;
+ }
+ ret = lseek64(handle->fd, offset, SEEK_SET);
+ if (ret < 0) {
+ tracecmd_warning("could not seek to back to offset %ld\n", offset);
+ goto error;
+ }
+ } else {
+ if (init_buffer_cpu_data(new_handle, buffer) < 0)
+ goto error;
}
return new_handle;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 57/87] trace-cmd report: Use the new latency API to read data
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (55 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 56/87] trace-cmd library: Handle buffer trace data init for " Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 58/87] trace-cmd report: Close input file handlers on exit Tzvetomir Stoyanov (VMware)
` (29 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When reading latency trace data, use the new API
tracecmd_latency_data_read()
It handles reading latency trace data from both version 6 and 7 trace
files.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-read.c | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/tracecmd/trace-read.c b/tracecmd/trace-read.c
index 31724b09..0ec2b2d1 100644
--- a/tracecmd/trace-read.c
+++ b/tracecmd/trace-read.c
@@ -948,18 +948,20 @@ void trace_show_data(struct tracecmd_input *handle, struct tep_record *record)
printf("\n");
}
-static void read_rest(void)
+static void read_latency(struct tracecmd_input *handle)
{
- char buf[BUFSIZ + 1];
+ char *buf = NULL;
+ size_t size = 0;
int r;
do {
- r = read(input_fd, buf, BUFSIZ);
- if (r > 0) {
- buf[r] = 0;
- printf("%s", buf);
- }
+ r = tracecmd_latency_data_read(handle, &buf, &size);
+ if (r > 0)
+ printf("%.*s", r, buf);
} while (r > 0);
+
+ printf("\n");
+ free(buf);
}
static int
@@ -1241,7 +1243,7 @@ static void read_data_info(struct list_head *handle_list, enum output_type otype
if (ret > 0) {
if (multi_inputs)
die("latency traces do not work with multiple inputs");
- read_rest();
+ read_latency(handles->handle);
return;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 58/87] trace-cmd report: Close input file handlers on exit
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (56 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 57/87] trace-cmd report: Use the new latency API to read data Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 59/87] trace-cmd report: Do not print empty buffer name Tzvetomir Stoyanov (VMware)
` (28 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When "trace-cmd report" is interrupted with "ctrl-c", close the input
hanlders to opened trace files, to delete any temporary files used when
reading the trace data.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-read.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/tracecmd/trace-read.c b/tracecmd/trace-read.c
index 0ec2b2d1..a36d72e4 100644
--- a/tracecmd/trace-read.c
+++ b/tracecmd/trace-read.c
@@ -1365,7 +1365,14 @@ struct tracecmd_input *read_trace_header(const char *file, int flags)
static void sig_end(int sig)
{
+ struct handle_list *handles;
+
fprintf(stderr, "trace-cmd: Received SIGINT\n");
+
+ list_for_each_entry(handles, &handle_list, list) {
+ tracecmd_close(handles->handle);
+ }
+
exit(0);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 59/87] trace-cmd report: Do not print empty buffer name
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (57 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 58/87] trace-cmd report: Close input file handlers on exit Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 19:21 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 60/87] trace-cmd report: Init the top trace instance earlier Tzvetomir Stoyanov (VMware)
` (27 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In trace file version 7, the top buffer is saved with its empty
file name, string "". When displaying it, filter those empty strings.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-read.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tracecmd/trace-read.c b/tracecmd/trace-read.c
index a36d72e4..1868a830 100644
--- a/tracecmd/trace-read.c
+++ b/tracecmd/trace-read.c
@@ -1175,7 +1175,7 @@ static void print_handle_file(struct handle_list *handles)
/* Only print file names if more than one file is read */
if (!multi_inputs && !instances)
return;
- if (handles->file)
+ if (handles->file && *handles->file != '\0')
printf("%*s: ", max_file_size, handles->file);
else
printf("%*s ", max_file_size, "");
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 59/87] trace-cmd report: Do not print empty buffer name
2021-07-29 5:09 ` [PATCH v2 59/87] trace-cmd report: Do not print empty buffer name Tzvetomir Stoyanov (VMware)
@ 2021-08-19 19:21 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 19:21 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:31 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> In trace file version 7, the top buffer is saved with its empty
> file name, string "". When displaying it, filter those empty strings.
This looks like it could have been folded into another patch.
-- Steve
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> tracecmd/trace-read.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tracecmd/trace-read.c b/tracecmd/trace-read.c
> index a36d72e4..1868a830 100644
> --- a/tracecmd/trace-read.c
> +++ b/tracecmd/trace-read.c
> @@ -1175,7 +1175,7 @@ static void print_handle_file(struct handle_list *handles)
> /* Only print file names if more than one file is read */
> if (!multi_inputs && !instances)
> return;
> - if (handles->file)
> + if (handles->file && *handles->file != '\0')
> printf("%*s: ", max_file_size, handles->file);
> else
> printf("%*s ", max_file_size, "");
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 60/87] trace-cmd report: Init the top trace instance earlier
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (58 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 59/87] trace-cmd report: Do not print empty buffer name Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 19:22 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 61/87] trace-cmd: Call additional APIs when creating trace file Tzvetomir Stoyanov (VMware)
` (26 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The instance must be initialized before calling the tracecmd_cpus() API,
as in trace files version 7, CPU count is set when the trace data are
initialized.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-read.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/tracecmd/trace-read.c b/tracecmd/trace-read.c
index 1868a830..cafceffe 100644
--- a/tracecmd/trace-read.c
+++ b/tracecmd/trace-read.c
@@ -1217,6 +1217,11 @@ static void read_data_info(struct list_head *handle_list, enum output_type otype
list_for_each_entry(handles, handle_list, list) {
int cpus;
+ if (!tracecmd_is_buffer_instance(handles->handle)) {
+ ret = tracecmd_init_data(handles->handle);
+ if (ret < 0)
+ die("failed to init data");
+ }
cpus = tracecmd_cpus(handles->handle);
handles->cpus = cpus;
handles->last_timestamp = calloc(cpus, sizeof(*handles->last_timestamp));
@@ -1227,9 +1232,6 @@ static void read_data_info(struct list_head *handle_list, enum output_type otype
if (tracecmd_is_buffer_instance(handles->handle))
continue;
- ret = tracecmd_init_data(handles->handle);
- if (ret < 0)
- die("failed to init data");
if (align_ts) {
ts = tracecmd_get_first_ts(handles->handle);
if (first || first_ts > ts)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 60/87] trace-cmd report: Init the top trace instance earlier
2021-07-29 5:09 ` [PATCH v2 60/87] trace-cmd report: Init the top trace instance earlier Tzvetomir Stoyanov (VMware)
@ 2021-08-19 19:22 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 19:22 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:32 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> The instance must be initialized before calling the tracecmd_cpus() API,
> as in trace files version 7, CPU count is set when the trace data are
> initialized.
This should have been an earlier patch.
-- Steve
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> tracecmd/trace-read.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/tracecmd/trace-read.c b/tracecmd/trace-read.c
> index 1868a830..cafceffe 100644
> --- a/tracecmd/trace-read.c
> +++ b/tracecmd/trace-read.c
> @@ -1217,6 +1217,11 @@ static void read_data_info(struct list_head *handle_list, enum output_type otype
> list_for_each_entry(handles, handle_list, list) {
> int cpus;
>
> + if (!tracecmd_is_buffer_instance(handles->handle)) {
> + ret = tracecmd_init_data(handles->handle);
> + if (ret < 0)
> + die("failed to init data");
> + }
> cpus = tracecmd_cpus(handles->handle);
> handles->cpus = cpus;
> handles->last_timestamp = calloc(cpus, sizeof(*handles->last_timestamp));
> @@ -1227,9 +1232,6 @@ static void read_data_info(struct list_head *handle_list, enum output_type otype
> if (tracecmd_is_buffer_instance(handles->handle))
> continue;
>
> - ret = tracecmd_init_data(handles->handle);
> - if (ret < 0)
> - die("failed to init data");
> if (align_ts) {
> ts = tracecmd_get_first_ts(handles->handle);
> if (first || first_ts > ts)
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 61/87] trace-cmd: Call additional APIs when creating trace file
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (59 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 60/87] trace-cmd report: Init the top trace instance earlier Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 62/87] trace-cmd dump: Add helpers for processing trace file version 7 Tzvetomir Stoyanov (VMware)
` (25 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When creating a trace file, two more APIs should be called, compared to
the old flow:
- tracecmd_write_buffer_info(), to write version 6 buffers metadata in
the file.
- tracecmd_write_options() after the trace data is written, for version
7 trace files, as the buffer metadata is appended to the options at
the end.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-listen.c | 6 ++++++
tracecmd/trace-record.c | 4 ++++
tracecmd/trace-restore.c | 3 ++-
tracecmd/trace-split.c | 3 +++
4 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/tracecmd/trace-listen.c b/tracecmd/trace-listen.c
index d812145b..3312a042 100644
--- a/tracecmd/trace-listen.c
+++ b/tracecmd/trace-listen.c
@@ -604,6 +604,9 @@ static int put_together_file(int cpus, int ofd, const char *node,
if (write_options) {
ret = tracecmd_write_cpus(handle, cpus);
+ if (ret)
+ goto out;
+ ret = tracecmd_write_buffer_info(handle);
if (ret)
goto out;
ret = tracecmd_write_options(handle);
@@ -612,6 +615,9 @@ static int put_together_file(int cpus, int ofd, const char *node,
}
ret = tracecmd_write_cpu_data(handle, cpus, temp_files, "");
+ if (!ret && tracecmd_get_out_file_version(handle) >= 7)
+ tracecmd_write_options(handle);
+
out:
tracecmd_output_close(handle);
for (cpu--; cpu >= 0; cpu--) {
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index 2e8db8a4..ddab7798 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -3738,6 +3738,9 @@ setup_connection(struct buffer_instance *instance, struct common_record_context
if (ret)
goto error;
ret = tracecmd_write_cpus(network_handle, instance->cpu_count);
+ if (ret)
+ goto error;
+ ret = tracecmd_write_buffer_info(network_handle);
if (ret)
goto error;
ret = tracecmd_write_options(network_handle);
@@ -4109,6 +4112,7 @@ static void setup_agent(struct buffer_instance *instance,
add_options(network_handle, ctx);
tracecmd_write_cmdlines(network_handle);
tracecmd_write_cpus(network_handle, instance->cpu_count);
+ tracecmd_write_buffer_info(network_handle);
tracecmd_write_options(network_handle);
tracecmd_msg_finish_sending_data(instance->msg_handle);
instance->network_handle = network_handle;
diff --git a/tracecmd/trace-restore.c b/tracecmd/trace-restore.c
index 8d2fcae8..fbda42dd 100644
--- a/tracecmd/trace-restore.c
+++ b/tracecmd/trace-restore.c
@@ -163,6 +163,7 @@ void trace_restore (int argc, char **argv)
if (tracecmd_append_cpu_data(handle, args, &argv[first_arg]) < 0)
die("failed to append data");
-
+ if (tracecmd_get_out_file_version(handle) >= 7)
+ tracecmd_write_options(handle);
return;
}
diff --git a/tracecmd/trace-split.c b/tracecmd/trace-split.c
index e4a0c3b3..0768329f 100644
--- a/tracecmd/trace-split.c
+++ b/tracecmd/trace-split.c
@@ -391,6 +391,9 @@ static double parse_file(struct tracecmd_input *handle,
if (tracecmd_append_cpu_data(ohandle, cpus, cpu_list) < 0)
die("Failed to append tracing data\n");
+ if (tracecmd_get_out_file_version(ohandle) >= 7)
+ tracecmd_write_options(ohandle);
+
current = end;
for (cpu = 0; cpu < cpus; cpu++) {
/* Set the tracecmd cursor to the next set of records */
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 62/87] trace-cmd dump: Add helpers for processing trace file version 7
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (60 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 61/87] trace-cmd: Call additional APIs when creating trace file Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 63/87] trace-cmd dump: Print compression header Tzvetomir Stoyanov (VMware)
` (24 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
New global variables are added, that hold trace file context:
- trace file version
- compression context
Also a few helper functions for reading compressed data.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 88 ++++++++++++++++++++++++++++++++-----------
1 file changed, 67 insertions(+), 21 deletions(-)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index 2334b57e..e150a17d 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -26,6 +26,9 @@
static struct tep_handle *tep;
static unsigned int trace_cpus;
static int has_clock;
+static unsigned long file_version;
+static bool read_compress;
+static struct tracecmd_compression *compress;
enum dump_items {
SUMMARY = (1 << 0),
@@ -52,46 +55,69 @@ enum dump_items verbosity;
tracecmd_plog(fmt, ##__VA_ARGS__); \
} while (0)
-static int read_file_string(int fd, char *dst, int len)
+static int read_fd(int fd, char *dst, int len)
{
size_t size = 0;
int r;
do {
- r = read(fd, dst+size, 1);
+ r = read(fd, dst+size, len);
if (r > 0) {
- size++;
- len--;
+ size += r;
+ len -= r;
} else
break;
- if (!dst[size - 1])
- break;
- } while (r > 0 && len);
+ } while (r > 0);
- if (!size || dst[size - 1])
+ if (len)
return -1;
- return 0;
+ return size;
}
-static int read_file_bytes(int fd, char *dst, int len)
+static int read_compressed(int fd, char *dst, int len)
+{
+
+ if (read_compress)
+ return tracecmd_compress_read(compress, dst, len);
+ return read_fd(fd, dst, len);
+}
+
+static int do_lseek(int fd, int offset, int whence)
+{
+ if (read_compress)
+ return tracecmd_compress_lseek(compress, offset, whence);
+ return lseek64(fd, offset, whence);
+}
+
+static int read_file_string(int fd, char *dst, int len)
{
size_t size = 0;
int r;
do {
- r = read(fd, dst+size, len);
+ r = read_compressed(fd, dst+size, 1);
if (r > 0) {
- size += r;
- len -= r;
+ size++;
+ len--;
} else
break;
- } while (r > 0);
+ if (!dst[size - 1])
+ break;
+ } while (r > 0 && len);
- if (len)
+ if (!size || dst[size - 1])
return -1;
return 0;
}
+static int read_file_bytes(int fd, char *dst, int len)
+{
+ int ret;
+
+ ret = read_compressed(fd, dst, len);
+ return ret < 0 ? ret : 0;
+}
+
static void read_dump_string(int fd, int size, enum dump_items id)
{
char buf[DUMP_SIZE];
@@ -146,7 +172,6 @@ static void dump_initial_format(int fd)
char magic[] = TRACECMD_MAGIC;
char buf[DUMP_SIZE];
int val4;
- unsigned long ver;
do_print(SUMMARY, "\t[Initial format]\n");
@@ -168,11 +193,11 @@ static void dump_initial_format(int fd)
die("no version string");
do_print(SUMMARY, "\t\t%s\t[Version]\n", buf);
- ver = strtol(buf, NULL, 10);
- if (!ver && errno)
+ file_version = strtol(buf, NULL, 10);
+ if (!file_version && errno)
die("Invalid file version string %s", buf);
- if (!tracecmd_is_version_supported(ver))
- die("Unsupported file version %lu", ver);
+ if (!tracecmd_is_version_supported(file_version))
+ die("Unsupported file version %lu", file_version);
/* get file endianness*/
if (read_file_bytes(fd, buf, 1))
@@ -234,6 +259,27 @@ static void dump_header_event(int fd)
read_dump_string(fd, size, HEAD_EVENT);
}
+static void uncompress_reset(void)
+{
+ if (compress && file_version >= 7) {
+ read_compress = false;
+ tracecmd_compress_reset(compress);
+ }
+}
+
+static int uncompress_block(void)
+{
+ int ret = 0;
+
+ if (compress && file_version >= 7) {
+ ret = tracecmd_uncompress_block(compress);
+ if (!ret)
+ read_compress = true;
+
+ }
+ return ret;
+}
+
static void dump_ftrace_events_format(int fd)
{
unsigned long long size;
@@ -578,7 +624,7 @@ static void dump_options(int fd)
default:
do_print(OPTIONS, " %d %d\t[Unknown option, size - skipping]\n",
option, size);
- lseek64(fd, size, SEEK_CUR);
+ do_lseek(fd, size, SEEK_CUR);
break;
}
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 63/87] trace-cmd dump: Print compression header
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (61 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 62/87] trace-cmd dump: Add helpers for processing trace file version 7 Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 64/87] trace-cmd dump: Add helpers for processing trace file sections Tzvetomir Stoyanov (VMware)
` (23 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Dump the compression header of trace file version 7
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index e150a17d..e6f774bd 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -217,6 +217,29 @@ static void dump_initial_format(int fd)
do_print(SUMMARY, "\t\t%d\t[Page size, bytes]\n", val4);
}
+static void dump_compress(int fd)
+{
+ char zname[DUMP_SIZE];
+ char zver[DUMP_SIZE];
+
+ if (file_version < 7)
+ return;
+
+ /* get compression header */
+ if (read_file_string(fd, zname, DUMP_SIZE))
+ die("no compression header");
+ if (read_file_string(fd, zver, DUMP_SIZE))
+ die("no compression version");
+ do_print((SUMMARY), "\t\t%s\t[Compression algorithm]\n", zname);
+ do_print((SUMMARY), "\t\t%s\t[Compression version]\n", zver);
+
+ if (strcmp(zname, "none")) {
+ compress = tracecmd_compress_alloc(zname, zver, fd, tep, NULL);
+ if (!compress)
+ die("cannot uncompress the file");
+ }
+}
+
static void dump_header_page(int fd)
{
unsigned long long size;
@@ -715,6 +738,7 @@ static void dump_file(const char *file)
do_print(SUMMARY, "\n Tracing meta data in file %s:\n", file);
dump_initial_format(fd);
+ dump_compress(fd);
dump_header_page(fd);
dump_header_event(fd);
dump_ftrace_events_format(fd);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 64/87] trace-cmd dump: Add helpers for processing trace file sections
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (62 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 63/87] trace-cmd dump: Print compression header Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 65/87] trace-cmd dump: Read recursively all options sections Tzvetomir Stoyanov (VMware)
` (22 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added a new local databse for storing sections, found in trace file
version 7.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index e6f774bd..010a8188 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -43,8 +43,18 @@ enum dump_items {
OPTIONS = (1 << 9),
FLYRECORD = (1 << 10),
CLOCK = (1 << 11),
+ SECTIONS = (1 << 12),
};
+struct file_section {
+ int id;
+ unsigned long long offset;
+ struct file_section *next;
+ enum dump_items verbosity;
+};
+
+static struct file_section *sections;
+
enum dump_items verbosity;
#define DUMP_CHECK(X) ((X) & verbosity)
@@ -723,6 +733,17 @@ static void dump_therest(int fd)
}
}
+static void free_sections(void)
+{
+ struct file_section *del;
+
+ while (sections) {
+ del = sections;
+ sections = sections->next;
+ free(del);
+ }
+}
+
static void dump_file(const char *file)
{
int fd;
@@ -748,7 +769,7 @@ static void dump_file(const char *file)
dump_cmdlines(fd);
dump_cpus_count(fd);
dump_therest(fd);
-
+ free_sections();
tep_free(tep);
tep = NULL;
close(fd);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 65/87] trace-cmd dump: Read recursively all options sections
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (63 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 64/87] trace-cmd dump: Add helpers for processing trace file sections Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 66/87] trace-cmd dump: Read extended BUFFER option Tzvetomir Stoyanov (VMware)
` (21 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file version 7 supports multiple options sections in the file. The
DONE option is extended to hold the offset to the next options section.
A new logic for handling the extended DONE option and to read recursively
all options sections is added.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 76 ++++++++++++++++++++++++++++++++++++++-----
1 file changed, 68 insertions(+), 8 deletions(-)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index 010a8188..f801e0b6 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -422,6 +422,30 @@ static void dump_option_string(int fd, int size, char *desc)
read_dump_string(fd, size, OPTIONS);
}
+static void dump_section_header(int fd, enum dump_items v, unsigned short *flags)
+{
+ unsigned long long offset;
+ unsigned short fl;
+ char buf[DUMP_SIZE];
+ unsigned short id;
+ int size;
+
+ offset = lseek64(fd, 0, SEEK_CUR);
+ if (read_file_number(fd, &id, 2))
+ die("cannot read the section id");
+ if (read_file_string(fd, buf, DUMP_SIZE))
+ die("no section description");
+ if (read_file_number(fd, &fl, 2))
+ die("cannot read the section flags");
+ if (read_file_number(fd, &size, 4))
+ die("cannot read section size");
+ do_print(v, "\t[Section %d @ %lld: \"%s\", flags 0x%X, %d bytes]\n",
+ id, offset, buf, fl, size);
+
+ if (flags)
+ *flags = fl;
+}
+
static void dump_option_buffer(int fd, int size)
{
unsigned long long offset;
@@ -591,25 +615,48 @@ void dump_option_tsc2nsec(int fd, int size)
do_print(OPTIONS, "%d %d %llu [multiplier, shift, offset]\n", mult, shift, offset);
}
-static void dump_options(int fd)
+static int dump_options_read(int fd);
+
+static int dump_option_done(int fd, int size)
{
+ unsigned long long offset;
+
+ do_print(OPTIONS, "\t\t[Option DONE, %d bytes]\n", size);
+
+ if (file_version < 7 || size < 8)
+ return 0;
+ if (read_file_number(fd, &offset, 8))
+ die("cannot read the next options offset");
+ do_print(OPTIONS, "%lld\n", offset);
+ if (!offset)
+ return 0;
+
+ if (lseek64(fd, offset, SEEK_SET) == (off_t)-1)
+ die("cannot goto next options offset %lld", offset);
+ return dump_options_read(fd);
+}
+
+static int dump_options_read(int fd)
+{
+ unsigned short flags = 0;
unsigned short option;
unsigned int size;
int count = 0;
+ if (file_version >= 7)
+ dump_section_header(fd, OPTIONS, &flags);
+ if ((flags & TRACECMD_SEC_FL_COMPRESS) && uncompress_block())
+ die("cannot uncompress file block");
+
for (;;) {
if (read_file_number(fd, &option, 2))
die("cannot read the option id");
- if (!option)
+ if (option == TRACECMD_OPTION_DONE && file_version < 7)
break;
if (read_file_number(fd, &size, 4))
die("cannot read the option size");
count++;
- if (!DUMP_CHECK(OPTIONS) && !DUMP_CHECK(CLOCK) && !DUMP_CHECK(SUMMARY)) {
- lseek64(fd, size, SEEK_CUR);
- continue;
- }
switch (option) {
case TRACECMD_OPTION_DATE:
dump_option_string(fd, size, "DATE");
@@ -621,7 +668,8 @@ static void dump_options(int fd)
dump_option_buffer(fd, size);
break;
case TRACECMD_OPTION_TRACECLOCK:
- dump_option_string(fd, size, "TRACECLOCK");
+ do_print(OPTIONS, "\t\t[Option TRACECLOCK, %d bytes]\n", size);
+ read_dump_string(fd, size, OPTIONS | CLOCK);
has_clock = 1;
break;
case TRACECMD_OPTION_UNAME:
@@ -654,6 +702,10 @@ static void dump_options(int fd)
case TRACECMD_OPTION_TSC2NSEC:
dump_option_tsc2nsec(fd, size);
break;
+ case TRACECMD_OPTION_DONE:
+ uncompress_reset();
+ count += dump_option_done(fd, size);
+ return count;
default:
do_print(OPTIONS, " %d %d\t[Unknown option, size - skipping]\n",
option, size);
@@ -661,8 +713,16 @@ static void dump_options(int fd)
break;
}
}
- do_print(SUMMARY, "\t[%d options]\n", count);
+ uncompress_reset();
+ return count;
+}
+
+static void dump_options(int fd)
+{
+ int count;
+ count = dump_options_read(fd);
+ do_print(SUMMARY|OPTIONS, "\t[%d options]\n", count);
}
static void dump_latency(int fd)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 66/87] trace-cmd dump: Read extended BUFFER option
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (64 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 65/87] trace-cmd dump: Read recursively all options sections Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 67/87] trace-cmd dump: Dump sections Tzvetomir Stoyanov (VMware)
` (20 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In trace file version 7 the BUFFER option is extended to hold a trace
metadata, related to the recorded instance. Also, a new BUFFER_LAT
option is added for latency trace data. Implemended logic for reading
and printing these extended options.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 64 +++++++++++++++++++++++++++++++++++++++----
1 file changed, 59 insertions(+), 5 deletions(-)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index f801e0b6..63d8bb81 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -446,19 +446,72 @@ static void dump_section_header(int fd, enum dump_items v, unsigned short *flags
*flags = fl;
}
-static void dump_option_buffer(int fd, int size)
+static void dump_option_buffer(int fd, unsigned short option, int size)
{
+ unsigned long long total_size = 0;
+ unsigned long long data_size;
+ unsigned long long current;
unsigned long long offset;
+ unsigned short flags;
+ char clock[DUMP_SIZE];
+ char name[DUMP_SIZE];
+ int cpus = 0;
+ int id;
+ int i;
if (size < 8)
die("broken buffer option with size %d", size);
if (read_file_number(fd, &offset, 8))
die("cannot read the offset of the buffer option");
+ if (read_file_string(fd, name, DUMP_SIZE))
+ die("cannot read the name of the buffer option");
+ if (file_version < 7) {
+ do_print(OPTIONS|FLYRECORD, "\t\t[Option BUFFER, %d bytes]\n", size);
+ do_print(OPTIONS|FLYRECORD, "%lld [offset]\n", offset);
+ do_print(OPTIONS|FLYRECORD, "\"%s\" [name]\n", name);
+ return;
+ }
+
+ current = lseek64(fd, 0, SEEK_CUR);
+ if (lseek64(fd, offset, SEEK_SET) == (off_t)-1)
+ die("cannot goto buffer offset %lld", offset);
+
+ dump_section_header(fd, FLYRECORD, &flags);
+
+ if (lseek64(fd, current, SEEK_SET) == (off_t)-1)
+ die("cannot go back to buffer option");
+
+ do_print(OPTIONS|FLYRECORD, "\t\t[Option BUFFER, %d bytes]\n", size);
+ do_print(OPTIONS|FLYRECORD, "%lld [offset]\n", offset);
+ do_print(OPTIONS|FLYRECORD, "\"%s\" [name]\n", name);
+
+ if (read_file_string(fd, clock, DUMP_SIZE))
+ die("cannot read clock of the buffer option");
+ do_print(OPTIONS|FLYRECORD, "\"%s\" [clock]\n", clock);
+ if (option == TRACECMD_OPTION_BUFFER) {
+ if (read_file_number(fd, &cpus, 4))
+ die("cannot read the cpu count of the buffer option");
+
+ do_print(OPTIONS|FLYRECORD, "%d [CPUs]:\n", cpus);
+ for (i = 0; i < cpus; i++) {
+ if (read_file_number(fd, &id, 4))
+ die("cannot read the id of cpu %d from the buffer option", i);
+ if (read_file_number(fd, &offset, 8))
+ die("cannot read the offset of cpu %d from the buffer option", i);
+ if (read_file_number(fd, &data_size, 8))
+ die("cannot read the data size of cpu %d from the buffer option", i);
+ total_size += data_size;
+ do_print(OPTIONS|FLYRECORD, " %d %lld\t%lld\t[id, data offset and size]\n",
+ id, offset, data_size);
+ }
+ do_print(SUMMARY, "\t\[buffer \"%s\", \"%s\" clock, "
+ "%d cpus, %lld bytes flyrecord data]\n",
+ name, clock, cpus, total_size);
+ } else {
+ do_print(SUMMARY, "\t\[buffer \"%s\", \"%s\" clock, latency data]\n", name, clock);
+ }
- do_print(OPTIONS, "\t\t[Option BUFFER, %d bytes]\n", size);
- do_print(OPTIONS, "%lld [offset]\n", offset);
- read_dump_string(fd, size - 8, OPTIONS);
}
static void dump_option_int(int fd, int size, char *desc)
@@ -665,7 +718,8 @@ static int dump_options_read(int fd)
dump_option_string(fd, size, "CPUSTAT");
break;
case TRACECMD_OPTION_BUFFER:
- dump_option_buffer(fd, size);
+ case TRACECMD_OPTION_BUFFER_LAT:
+ dump_option_buffer(fd, option, size);
break;
case TRACECMD_OPTION_TRACECLOCK:
do_print(OPTIONS, "\t\t[Option TRACECLOCK, %d bytes]\n", size);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 67/87] trace-cmd dump: Dump sections
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (65 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 66/87] trace-cmd dump: Read extended BUFFER option Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 68/87] trace-cmd dump: Dump trace file version 7 Tzvetomir Stoyanov (VMware)
` (19 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In trace file version 7, headers are stored as sections. This allows
their position in the file not to be fixed. Poters to these sections are
stored as trace options. Add logic to handle these new options:
HEADER_INFO
FTRACE_EVENTS
EVENT_FORMATS
KALLSYM
PRINTK
CMDLINES
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index 63d8bb81..571b6ac2 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -668,6 +668,24 @@ void dump_option_tsc2nsec(int fd, int size)
do_print(OPTIONS, "%d %d %llu [multiplier, shift, offset]\n", mult, shift, offset);
}
+static void dump_option_section(int fd, unsigned int size,
+ unsigned short id, char *desc, enum dump_items v)
+{
+ struct file_section *sec;
+
+ sec = calloc(1, sizeof(struct file_section));
+ if (!sec)
+ die("cannot allocate new section");
+ sec->next = sections;
+ sections = sec;
+ sec->id = id;
+ sec->verbosity = v;
+ if (read_file_number(fd, &sec->offset, 8))
+ die("cannot read the option %d offset", id);
+ do_print(OPTIONS, "\t\t[Option %s, %d bytes]\n", desc, size);
+ do_print(OPTIONS, "%lld\n", sec->offset);
+}
+
static int dump_options_read(int fd);
static int dump_option_done(int fd, int size)
@@ -756,6 +774,25 @@ static int dump_options_read(int fd)
case TRACECMD_OPTION_TSC2NSEC:
dump_option_tsc2nsec(fd, size);
break;
+ case TRACECMD_OPTION_HEADER_INFO:
+ dump_option_section(fd, size, option, "HEADERS", HEAD_PAGE | HEAD_EVENT);
+ break;
+ case TRACECMD_OPTION_FTRACE_EVENTS:
+ dump_option_section(fd, size, option, "FTRACE EVENTS", FTRACE_FORMAT);
+ break;
+ case TRACECMD_OPTION_EVENT_FORMATS:
+ dump_option_section(fd, size, option,
+ "EVENT FORMATS", EVENT_SYSTEMS | EVENT_FORMAT);
+ break;
+ case TRACECMD_OPTION_KALLSYMS:
+ dump_option_section(fd, size, option, "KALLSYMS", KALLSYMS);
+ break;
+ case TRACECMD_OPTION_PRINTK:
+ dump_option_section(fd, size, option, "PRINTK", TRACE_PRINTK);
+ break;
+ case TRACECMD_OPTION_CMDLINES:
+ dump_option_section(fd, size, option, "CMDLINES", CMDLINES);
+ break;
case TRACECMD_OPTION_DONE:
uncompress_reset();
count += dump_option_done(fd, size);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 68/87] trace-cmd dump: Dump trace file version 7
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (66 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 67/87] trace-cmd dump: Dump sections Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 69/87] trace-cmd dump: Dump sections content Tzvetomir Stoyanov (VMware)
` (18 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file version 7 has different structure than version 6. Separates
parsing both versions in different flows.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 38 +++++++++++++++++++++++++++++---------
1 file changed, 29 insertions(+), 9 deletions(-)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index 571b6ac2..9088d2f5 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -884,6 +884,31 @@ static void dump_therest(int fd)
}
}
+static void dump_v6_file(int fd)
+{
+ dump_header_page(fd);
+ dump_header_event(fd);
+ dump_ftrace_events_format(fd);
+ dump_events_format(fd);
+ dump_kallsyms(fd);
+ dump_printk(fd);
+ dump_cmdlines(fd);
+ dump_cpus_count(fd);
+ dump_therest(fd);
+}
+
+static void dump_v7_file(int fd)
+{
+ long long offset;
+
+ if (read_file_number(fd, &offset, 8))
+ die("cannot read offset of the first option section");
+
+ if (lseek64(fd, offset, SEEK_SET) == (off64_t)-1)
+ die("cannot goto options offset %lld", offset);
+ dump_options(fd);
+}
+
static void free_sections(void)
{
struct file_section *del;
@@ -911,15 +936,10 @@ static void dump_file(const char *file)
dump_initial_format(fd);
dump_compress(fd);
- dump_header_page(fd);
- dump_header_event(fd);
- dump_ftrace_events_format(fd);
- dump_events_format(fd);
- dump_kallsyms(fd);
- dump_printk(fd);
- dump_cmdlines(fd);
- dump_cpus_count(fd);
- dump_therest(fd);
+ if (file_version < 7)
+ dump_v6_file(fd);
+ else
+ dump_v7_file(fd);
free_sections();
tep_free(tep);
tep = NULL;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 69/87] trace-cmd dump: Dump sections content
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (67 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 68/87] trace-cmd dump: Dump trace file version 7 Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 70/87] trace-cmd dump: Add new argument --sections Tzvetomir Stoyanov (VMware)
` (17 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
In trace file version 7, dump the content of the headers content, that
are stored as sections. Added logic to dump these headers:
HEADER_INFO
FTRACE_EVENTS
EVENT_FORMATS
KALLSYM
PRINTK
CMDLINES
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 41 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index 9088d2f5..1dcc8c7c 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -686,6 +686,46 @@ static void dump_option_section(int fd, unsigned int size,
do_print(OPTIONS, "%lld\n", sec->offset);
}
+static void dump_sections(int fd)
+{
+ struct file_section *sec = sections;
+ unsigned short flags;
+
+ while (sec) {
+ if (lseek64(fd, sec->offset, SEEK_SET) == (off_t)-1)
+ die("cannot goto option offset %lld", sec->offset);
+
+ dump_section_header(fd, sec->verbosity, &flags);
+
+ if ((flags & TRACECMD_SEC_FL_COMPRESS) && uncompress_block())
+ die("cannot uncompress section block");
+
+ switch (sec->id) {
+ case TRACECMD_OPTION_HEADER_INFO:
+ dump_header_page(fd);
+ dump_header_event(fd);
+ break;
+ case TRACECMD_OPTION_FTRACE_EVENTS:
+ dump_ftrace_events_format(fd);
+ break;
+ case TRACECMD_OPTION_EVENT_FORMATS:
+ dump_events_format(fd);
+ break;
+ case TRACECMD_OPTION_KALLSYMS:
+ dump_kallsyms(fd);
+ break;
+ case TRACECMD_OPTION_PRINTK:
+ dump_printk(fd);
+ break;
+ case TRACECMD_OPTION_CMDLINES:
+ dump_cmdlines(fd);
+ break;
+ }
+ uncompress_reset();
+ sec = sec->next;
+ }
+}
+
static int dump_options_read(int fd);
static int dump_option_done(int fd, int size)
@@ -907,6 +947,7 @@ static void dump_v7_file(int fd)
if (lseek64(fd, offset, SEEK_SET) == (off64_t)-1)
die("cannot goto options offset %lld", offset);
dump_options(fd);
+ dump_sections(fd);
}
static void free_sections(void)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 70/87] trace-cmd dump: Add new argument --sections
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (68 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 69/87] trace-cmd dump: Dump sections content Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 71/87] trace-cmd dump: Align better the output of flyrecord dump Tzvetomir Stoyanov (VMware)
` (16 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The new "trace-cmd dump --sections" argument walks through the sections
from a trace file version 7 and prints their headers. The logic does
not rely on the options, describing these sections. It could be useful
to dump broken or incomplete trace files.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 71 +++++++++++++++++++++++++++++++++++++++++--
1 file changed, 68 insertions(+), 3 deletions(-)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index 1dcc8c7c..f966212a 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -686,7 +686,7 @@ static void dump_option_section(int fd, unsigned int size,
do_print(OPTIONS, "%lld\n", sec->offset);
}
-static void dump_sections(int fd)
+static void dump_sections(int fd, int count)
{
struct file_section *sec = sections;
unsigned short flags;
@@ -724,6 +724,7 @@ static void dump_sections(int fd)
uncompress_reset();
sec = sec->next;
}
+ do_print(SUMMARY|SECTIONS, "\t[%d sections]\n", count);
}
static int dump_options_read(int fd);
@@ -937,17 +938,76 @@ static void dump_v6_file(int fd)
dump_therest(fd);
}
+static int walk_v7_sections(int fd)
+{
+ unsigned long long offset, soffset;
+ unsigned short fl;
+ char buf[DUMP_SIZE];
+ unsigned short id;
+ int size, csize, rsize;
+ int count = 0;
+
+ offset = lseek64(fd, 0, SEEK_CUR);
+ do {
+ soffset = lseek64(fd, 0, SEEK_CUR);
+ if (read_file_number(fd, &id, 2))
+ break;
+ if (id >= TRACECMD_OPTION_MAX)
+ die("Unknown section id %d", id);
+ if (read_file_string(fd, buf, DUMP_SIZE))
+ die("cannot read section description");
+ if (read_file_number(fd, &fl, 2))
+ die("cannot read section flags");
+ if (read_file_number(fd, &size, 4))
+ die("cannot read section size");
+ if (size <= 4)
+ die("Section %d (%s) is too small, %d bytes", id, buf, size);
+ count++;
+ if (fl & TRACECMD_SEC_FL_COMPRESS) {
+ if (id == TRACECMD_OPTION_BUFFER ||
+ id == TRACECMD_OPTION_BUFFER_LAT) {
+ do_print(SECTIONS,
+ "\t[Section %2d @ %-16lld\t\"%s\", flags 0x%X, "
+ "%d compressed bytes]\n",
+ id, soffset, buf, fl, size);
+ } else {
+ if (read_file_number(fd, &csize, 4))
+ die("cannot read section size");
+ if (read_file_number(fd, &rsize, 4))
+ die("cannot read section size");
+ do_print(SECTIONS, "\t[Section %2d @ %-16lld\t\"%s\", flags 0x%X, "
+ "%d compressed, %d uncompressed]\n",
+ id, soffset, buf, fl, csize, rsize);
+ size -= 8;
+ }
+ } else {
+ do_print(SECTIONS, "\t[Section %2d @ %-16lld\t\"%s\", flags 0x%X, %d bytes]\n",
+ id, soffset, buf, fl, size);
+ }
+
+ if (lseek64(fd, size, SEEK_CUR) == (off_t)-1)
+ break;
+ } while (1);
+
+ if (lseek64(fd, offset, SEEK_SET) == (off_t)-1)
+ die("cannot restore the original file location");
+ return count;
+}
+
static void dump_v7_file(int fd)
{
long long offset;
+ int sections;
if (read_file_number(fd, &offset, 8))
die("cannot read offset of the first option section");
- if (lseek64(fd, offset, SEEK_SET) == (off64_t)-1)
+ sections = walk_v7_sections(fd);
+
+ if (lseek64(fd, offset, SEEK_SET) == (off_t)-1)
die("cannot goto options offset %lld", offset);
dump_options(fd);
- dump_sections(fd);
+ dump_sections(fd, sections);
}
static void free_sections(void)
@@ -988,6 +1048,7 @@ static void dump_file(const char *file)
}
enum {
+ OPT_sections = 241,
OPT_verbose = 242,
OPT_clock = 243,
OPT_all = 244,
@@ -1031,6 +1092,7 @@ void trace_dump(int argc, char **argv)
{"options", no_argument, NULL, OPT_options},
{"flyrecord", no_argument, NULL, OPT_flyrecord},
{"clock", no_argument, NULL, OPT_clock},
+ {"sections", no_argument, NULL, OPT_sections},
{"validate", no_argument, NULL, 'v'},
{"help", no_argument, NULL, '?'},
{"verbose", optional_argument, NULL, OPT_verbose},
@@ -1094,6 +1156,9 @@ void trace_dump(int argc, char **argv)
if (trace_set_verbose(optarg) < 0)
die("invalid verbose level %s", optarg);
break;
+ case OPT_sections:
+ verbosity |= SECTIONS;
+ break;
default:
usage(argv);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 71/87] trace-cmd dump: Align better the output of flyrecord dump
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (69 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 70/87] trace-cmd dump: Add new argument --sections Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 72/87] trace-cmd library: Add zlib compression algorithm Tzvetomir Stoyanov (VMware)
` (15 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The flyrecord dump prints various file offsets and sizes, that can be
huge numbers. Add format arguments to align better the output.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-dump.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tracecmd/trace-dump.c b/tracecmd/trace-dump.c
index f966212a..7c390022 100644
--- a/tracecmd/trace-dump.c
+++ b/tracecmd/trace-dump.c
@@ -898,7 +898,7 @@ static void dump_flyrecord(int fd)
die("cannot read the cpu %d offset", i);
if (read_file_number(fd, &cpu_size, 8))
die("cannot read the cpu %d size", i);
- do_print(FLYRECORD, "\t\t %lld %lld\t[offset, size of cpu %d]\n",
+ do_print(FLYRECORD, "\t %10.lld %10.lld\t[offset, size of cpu %d]\n",
cpu_offset, cpu_size, i);
}
dump_clock(fd);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 72/87] trace-cmd library: Add zlib compression algorithm
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (70 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 71/87] trace-cmd dump: Align better the output of flyrecord dump Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 73/87] trace-cmd library: Reuse local function that writes to output handler Tzvetomir Stoyanov (VMware)
` (14 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added implementation for zlib compression algorithm, using libz
libraray, if available.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
Makefile | 7 ++
lib/trace-cmd/Makefile | 7 ++
lib/trace-cmd/include/trace-cmd-local.h | 4 +
lib/trace-cmd/trace-compress-zlib.c | 109 ++++++++++++++++++++++++
tracecmd/Makefile | 4 +
5 files changed, 131 insertions(+)
create mode 100644 lib/trace-cmd/trace-compress-zlib.c
diff --git a/Makefile b/Makefile
index d6fd09d8..2bc325dc 100644
--- a/Makefile
+++ b/Makefile
@@ -311,6 +311,13 @@ ifeq ($(PERF_DEFINED), 1)
CFLAGS += -DPERF
endif
+ZLIB_INSTALLED := $(shell if (printf "$(pound)include <zlib.h>\n void main(){deflateInit(NULL, Z_BEST_COMPRESSION);}" | $(CC) -o /dev/null -x c - -lz >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi)
+ifeq ($(ZLIB_INSTALLED), 1)
+export ZLIB_INSTALLED
+CFLAGS += -DHAVE_ZLIB
+$(info Have zlib compression support)
+endif
+
CUNIT_INSTALLED := $(shell if (printf "$(pound)include <CUnit/Basic.h>\n void main(){CU_initialize_registry();}" | $(CC) -o /dev/null -x c - -lcunit >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi)
export CUNIT_INSTALLED
diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile
index bab4322d..1ee6ac4c 100644
--- a/lib/trace-cmd/Makefile
+++ b/lib/trace-cmd/Makefile
@@ -26,6 +26,9 @@ OBJS += trace-timesync-ptp.o
OBJS += trace-timesync-kvm.o
endif
OBJS += trace-compress.o
+ifeq ($(ZLIB_INSTALLED), 1)
+OBJS += trace-compress-zlib.o
+endif
# Additional util objects
OBJS += trace-blk-hack.o
@@ -47,6 +50,10 @@ $(LIBTRACECMD_STATIC): $(OBJS)
LIBS = $(LIBTRACEEVENT_LDLAGS) $(LIBTRACEFS_LDLAGS) -lpthread
+ifeq ($(ZLIB_INSTALLED), 1)
+LIBS += -lz
+endif
+
$(LIBTRACECMD_SHARED_VERSION): $(LIBTRACECMD_SHARED)
@ln -sf $(<F) $@
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 0291b8de..f8a335d4 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -24,6 +24,10 @@ void tracecmd_info(const char *fmt, ...);
#endif
#endif
+#ifdef HAVE_ZLIB
+int tracecmd_zlib_init(void);
+#endif
+
struct data_file_write {
unsigned long long file_size;
unsigned long long write_size;
diff --git a/lib/trace-cmd/trace-compress-zlib.c b/lib/trace-cmd/trace-compress-zlib.c
new file mode 100644
index 00000000..7f62bf0a
--- /dev/null
+++ b/lib/trace-cmd/trace-compress-zlib.c
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: LGPL-2.1
+/*
+ * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com>
+ *
+ */
+#include <stdlib.h>
+#include <dlfcn.h>
+#include <zlib.h>
+#include <errno.h>
+
+#include "trace-cmd-private.h"
+
+#define __ZLIB_NAME "zlib"
+#define __ZLIB_WEIGTH 10
+
+static int zlib_compress(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes)
+{
+ unsigned long out_size = *out_bytes;
+ int ret;
+
+ ret = compress2((unsigned char *)out, &out_size,
+ (unsigned char *)in, (unsigned long)in_bytes, Z_BEST_COMPRESSION);
+ *out_bytes = out_size;
+ errno = 0;
+ switch (ret) {
+ case Z_OK:
+ return 0;
+ case Z_BUF_ERROR:
+ errno = -ENOBUFS;
+ break;
+ case Z_MEM_ERROR:
+ errno = -ENOMEM;
+ break;
+ case Z_STREAM_ERROR:
+ errno = -EINVAL;
+ break;
+ default:
+ errno = -EFAULT;
+ break;
+ }
+
+ return -1;
+}
+
+static int zlib_decompress(char *in, unsigned int in_bytes,
+ char *out, unsigned int *out_bytes)
+{
+ unsigned long out_size = *out_bytes;
+ int ret;
+
+ ret = uncompress((unsigned char *)out, &out_size,
+ (unsigned char *)in, (unsigned long)in_bytes);
+ *out_bytes = out_size;
+ errno = 0;
+ switch (ret) {
+ case Z_OK:
+ return 0;
+ case Z_BUF_ERROR:
+ errno = -ENOBUFS;
+ break;
+ case Z_MEM_ERROR:
+ errno = -ENOMEM;
+ break;
+ case Z_DATA_ERROR:
+ errno = -EINVAL;
+ break;
+ default:
+ errno = -EFAULT;
+ break;
+ }
+
+ return -1;
+}
+
+static unsigned int zlib_compress_bound(unsigned int in_bytes)
+{
+ return compressBound(in_bytes);
+}
+
+static bool zlib_is_supported(const char *name, const char *version)
+{
+ const char *zver;
+
+ if (!name)
+ return false;
+ if (strlen(name) != strlen(__ZLIB_NAME) || strcmp(name, __ZLIB_NAME))
+ return false;
+
+ if (!version)
+ return true;
+
+ zver = zlibVersion();
+ if (!zver)
+ return false;
+
+ /* Compare the major version number */
+ if (atoi(version) <= atoi(zver))
+ return true;
+
+ return false;
+}
+
+int tracecmd_zlib_init(void)
+{
+ return tracecmd_compress_proto_register(__ZLIB_NAME, zlibVersion(), __ZLIB_WEIGTH,
+ zlib_compress, zlib_decompress,
+ zlib_compress_bound, zlib_is_supported);
+}
diff --git a/tracecmd/Makefile b/tracecmd/Makefile
index 80c69bbb..b7a23dc4 100644
--- a/tracecmd/Makefile
+++ b/tracecmd/Makefile
@@ -51,6 +51,10 @@ CONFIG_INCLUDES =
CONFIG_LIBS = -lrt -lpthread $(TRACE_LIBS)
CONFIG_FLAGS =
+ifeq ($(ZLIB_INSTALLED), 1)
+CONFIG_LIBS += -lz
+endif
+
all: $(TARGETS)
$(bdir):
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 73/87] trace-cmd library: Reuse local function that writes to output handler
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (71 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 72/87] trace-cmd library: Add zlib compression algorithm Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 74/87] trace-cmd library: Use output handler when copying data from input file Tzvetomir Stoyanov (VMware)
` (13 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Reuse do_write_check() internal helper function in the trace-cmd
library. It should be used when writing data to an output handler,
instead of writing to a fd, as it handles compression and network
contexts.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/trace-cmd-local.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index f8a335d4..c73b37a6 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -59,6 +59,8 @@ out_write_section_header(struct tracecmd_output *handle, unsigned short header_i
char *description, enum tracecmd_section_flags flags, bool option);
int out_update_section_header(struct tracecmd_output *handle, unsigned long long offset);
+long long do_write_check(struct tracecmd_output *handle, const void *data, long long size);
+
struct cpu_data_source {
int fd;
int size;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 74/87] trace-cmd library: Use output handler when copying data from input file
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (72 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 73/87] trace-cmd library: Reuse local function that writes to output handler Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 75/87] trace-cmd library: Handle version 7 files when copying headers between files Tzvetomir Stoyanov (VMware)
` (12 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When copying data between two trace files, use input and output
handlers, instead of raw file descriptors. Handlers take care of
file version and compression context.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 3 +-
lib/trace-cmd/trace-input.c | 145 +++++++++---------
lib/trace-cmd/trace-output.c | 2 +-
3 files changed, 78 insertions(+), 72 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 0a3e12cf..05a2ea01 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -179,7 +179,8 @@ int tracecmd_get_parsing_failures(struct tracecmd_input *handle);
int tracecmd_long_size(struct tracecmd_input *handle);
int tracecmd_page_size(struct tracecmd_input *handle);
int tracecmd_cpus(struct tracecmd_input *handle);
-int tracecmd_copy_headers(struct tracecmd_input *handle, int fd,
+int tracecmd_copy_headers(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle,
enum tracecmd_file_states start_state,
enum tracecmd_file_states end_state);
void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 2b98e6d8..2d2750ad 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4357,44 +4357,47 @@ void tracecmd_close(struct tracecmd_input *handle)
free(handle);
}
-static int read_copy_size8(struct tracecmd_input *handle, int fd, unsigned long long *size)
+static int read_copy_size8(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle, unsigned long long *size)
{
/* read size */
- if (do_read_check(handle, size, 8))
+ if (do_read_check(in_handle, size, 8))
return -1;
- if (__do_write_check(fd, size, 8))
+ if (do_write_check(out_handle, size, 8))
return -1;
- *size = tep_read_number(handle->pevent, size, 8);
+ *size = tep_read_number(in_handle->pevent, size, 8);
return 0;
}
-static int read_copy_size4(struct tracecmd_input *handle, int fd, unsigned int *size)
+static int read_copy_size4(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle,
+ unsigned int *size)
{
/* read size */
- if (do_read_check(handle, size, 4))
+ if (do_read_check(in_handle, size, 4))
return -1;
- if (__do_write_check(fd, size, 4))
+ if (do_write_check(out_handle, size, 4))
return -1;
- *size = tep_read_number(handle->pevent, size, 4);
+ *size = tep_read_number(in_handle->pevent, size, 4);
return 0;
}
-static int read_copy_data(struct tracecmd_input *handle,
- unsigned long long size, int fd)
+static int read_copy_data(struct tracecmd_input *in_handle,
+ unsigned long long size,
+ struct tracecmd_output *out_handle)
{
char *buf;
buf = malloc(size);
if (!buf)
return -1;
- if (do_read_check(handle, buf, size))
+ if (do_read_check(in_handle, buf, size))
goto failed_read;
- if (__do_write_check(fd, buf, size))
+ if (do_write_check(out_handle, buf, size))
goto failed_read;
free(buf);
@@ -4406,65 +4409,66 @@ static int read_copy_data(struct tracecmd_input *handle,
return -1;
}
-static int copy_header_files(struct tracecmd_input *handle, int fd)
+static int copy_header_files(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle)
{
unsigned long long size;
- if (handle->file_state != TRACECMD_FILE_HEADERS - 1)
+ if (in_handle->file_state != TRACECMD_FILE_HEADERS - 1)
return -1;
/* "header_page" */
- if (read_copy_data(handle, 12, fd) < 0)
+ if (read_copy_data(in_handle, 12, out_handle) < 0)
return -1;
- if (read_copy_size8(handle, fd, &size) < 0)
+ if (read_copy_size8(in_handle, out_handle, &size) < 0)
return -1;
- if (read_copy_data(handle, size, fd) < 0)
+ if (read_copy_data(in_handle, size, out_handle) < 0)
return -1;
/* "header_event" */
- if (read_copy_data(handle, 13, fd) < 0)
+ if (read_copy_data(in_handle, 13, out_handle) < 0)
return -1;
- if (read_copy_size8(handle, fd, &size) < 0)
+ if (read_copy_size8(in_handle, out_handle, &size) < 0)
return -1;
- if (read_copy_data(handle, size, fd) < 0)
+ if (read_copy_data(in_handle, size, out_handle) < 0)
return -1;
- handle->file_state = TRACECMD_FILE_HEADERS;
+ in_handle->file_state = TRACECMD_FILE_HEADERS;
return 0;
}
-static int copy_ftrace_files(struct tracecmd_input *handle, int fd)
+static int copy_ftrace_files(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
unsigned long long size;
unsigned int count;
unsigned int i;
- if (handle->file_state != TRACECMD_FILE_FTRACE_EVENTS - 1)
+ if (in_handle->file_state != TRACECMD_FILE_FTRACE_EVENTS - 1)
return -1;
- if (read_copy_size4(handle, fd, &count) < 0)
+ if (read_copy_size4(in_handle, out_handle, &count) < 0)
return -1;
for (i = 0; i < count; i++) {
- if (read_copy_size8(handle, fd, &size) < 0)
+ if (read_copy_size8(in_handle, out_handle, &size) < 0)
return -1;
- if (read_copy_data(handle, size, fd) < 0)
+ if (read_copy_data(in_handle, size, out_handle) < 0)
return -1;
}
- handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
+ in_handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
return 0;
}
-static int copy_event_files(struct tracecmd_input *handle, int fd)
+static int copy_event_files(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
unsigned long long size;
char *system;
@@ -4472,103 +4476,103 @@ static int copy_event_files(struct tracecmd_input *handle, int fd)
unsigned int count;
unsigned int i,x;
- if (handle->file_state != TRACECMD_FILE_ALL_EVENTS - 1)
+ if (in_handle->file_state != TRACECMD_FILE_ALL_EVENTS - 1)
return -1;
- if (read_copy_size4(handle, fd, &systems) < 0)
+ if (read_copy_size4(in_handle, out_handle, &systems) < 0)
return -1;
for (i = 0; i < systems; i++) {
- system = read_string(handle);
+ system = read_string(in_handle);
if (!system)
return -1;
- if (__do_write_check(fd, system, strlen(system) + 1)) {
+ if (do_write_check(out_handle, system, strlen(system) + 1)) {
free(system);
return -1;
}
free(system);
- if (read_copy_size4(handle, fd, &count) < 0)
+ if (read_copy_size4(in_handle, out_handle, &count) < 0)
return -1;
for (x=0; x < count; x++) {
- if (read_copy_size8(handle, fd, &size) < 0)
+ if (read_copy_size8(in_handle, out_handle, &size) < 0)
return -1;
- if (read_copy_data(handle, size, fd) < 0)
+ if (read_copy_data(in_handle, size, out_handle) < 0)
return -1;
}
}
- handle->file_state = TRACECMD_FILE_ALL_EVENTS;
+ in_handle->file_state = TRACECMD_FILE_ALL_EVENTS;
return 0;
}
-static int copy_proc_kallsyms(struct tracecmd_input *handle, int fd)
+static int copy_proc_kallsyms(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
unsigned int size;
- if (handle->file_state != TRACECMD_FILE_KALLSYMS - 1)
+ if (in_handle->file_state != TRACECMD_FILE_KALLSYMS - 1)
return -1;
- if (read_copy_size4(handle, fd, &size) < 0)
+ if (read_copy_size4(in_handle, out_handle, &size) < 0)
return -1;
if (!size)
return 0; /* OK? */
- if (read_copy_data(handle, size, fd) < 0)
+ if (read_copy_data(in_handle, size, out_handle) < 0)
return -1;
- handle->file_state = TRACECMD_FILE_KALLSYMS;
+ in_handle->file_state = TRACECMD_FILE_KALLSYMS;
return 0;
}
-static int copy_ftrace_printk(struct tracecmd_input *handle, int fd)
+static int copy_ftrace_printk(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
unsigned int size;
- if (handle->file_state != TRACECMD_FILE_PRINTK - 1)
+ if (in_handle->file_state != TRACECMD_FILE_PRINTK - 1)
return -1;
- if (read_copy_size4(handle, fd, &size) < 0)
+ if (read_copy_size4(in_handle, out_handle, &size) < 0)
return -1;
if (!size)
return 0; /* OK? */
- if (read_copy_data(handle, size, fd) < 0)
+ if (read_copy_data(in_handle, size, out_handle) < 0)
return -1;
- handle->file_state = TRACECMD_FILE_PRINTK;
+ in_handle->file_state = TRACECMD_FILE_PRINTK;
return 0;
}
-static int copy_command_lines(struct tracecmd_input *handle, int fd)
+static int copy_command_lines(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
unsigned long long size;
- if (handle->file_state != TRACECMD_FILE_CMD_LINES - 1)
+ if (!check_out_state(out_handle, TRACECMD_FILE_CMD_LINES))
return -1;
- if (read_copy_size8(handle, fd, &size) < 0)
+ if (read_copy_size8(in_handle, out_handle, &size) < 0)
return -1;
if (!size)
return 0; /* OK? */
- if (read_copy_data(handle, size, fd) < 0)
+ if (read_copy_data(in_handle, size, out_handle) < 0)
return -1;
- handle->file_state = TRACECMD_FILE_CMD_LINES;
+ in_handle->file_state = TRACECMD_FILE_CMD_LINES;
return 0;
}
/**
* tracecmd_copy_headers - Copy headers from a tracecmd_input handle to a file descriptor
- * @handle: input handle for the trace.dat file to copy from.
- * @fd: The file descriptor to copy to.
+ * @in_handle: input handle for the trace.dat file to copy from.
+ * @out_handle: output handle to the trace.dat file to copy to.
* @start_state: The file state to start copying from (zero for the beginnig)
* @end_state: The file state to stop at (zero for up to cmdlines)
*
@@ -4579,7 +4583,8 @@ static int copy_command_lines(struct tracecmd_input *handle, int fd)
* NOTE: The input handle is also modified, and ends at the end
* state as well.
*/
-int tracecmd_copy_headers(struct tracecmd_input *handle, int fd,
+int tracecmd_copy_headers(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle,
enum tracecmd_file_states start_state,
enum tracecmd_file_states end_state)
{
@@ -4597,71 +4602,71 @@ int tracecmd_copy_headers(struct tracecmd_input *handle, int fd,
if (end_state < TRACECMD_FILE_HEADERS)
return 0;
- if (handle->file_state >= start_state) {
+ if (in_handle->file_state >= start_state) {
/* Set the handle to just before the start state */
- sec = open_section(handle, TRACECMD_OPTION_HEADER_INFO);
+ sec = open_section(in_handle, TRACECMD_OPTION_HEADER_INFO);
if (!sec)
return -1;
/* Now that the file handle has moved, change its state */
- handle->file_state = TRACECMD_FILE_INIT;
+ in_handle->file_state = TRACECMD_FILE_INIT;
}
/* Try to bring the input up to the start state - 1 */
- ret = tracecmd_read_headers(handle, start_state - 1);
+ ret = tracecmd_read_headers(in_handle, start_state - 1);
if (sec)
- close_section(handle, sec);
+ close_section(in_handle, sec);
if (ret < 0)
goto out;
switch (start_state) {
case TRACECMD_FILE_HEADERS:
- ret = copy_header_files(handle, fd);
+ ret = copy_header_files(in_handle, out_handle);
if (ret < 0)
goto out;
/* fallthrough */
case TRACECMD_FILE_FTRACE_EVENTS:
/* handle's state is now updating with the copies */
- if (end_state <= handle->file_state)
+ if (end_state <= in_handle->file_state)
return 0;
- ret = copy_ftrace_files(handle, fd);
+ ret = copy_ftrace_files(in_handle, out_handle);
if (ret < 0)
goto out;
/* fallthrough */
case TRACECMD_FILE_ALL_EVENTS:
- if (end_state <= handle->file_state)
+ if (end_state <= in_handle->file_state)
return 0;
- ret = copy_event_files(handle, fd);
+ ret = copy_event_files(in_handle, out_handle);
if (ret < 0)
goto out;
/* fallthrough */
case TRACECMD_FILE_KALLSYMS:
- if (end_state <= handle->file_state)
+ if (end_state <= in_handle->file_state)
return 0;
- ret = copy_proc_kallsyms(handle, fd);
+ ret = copy_proc_kallsyms(in_handle, out_handle);
if (ret < 0)
goto out;
/* fallthrough */
case TRACECMD_FILE_PRINTK:
- if (end_state <= handle->file_state)
+ if (end_state <= in_handle->file_state)
return 0;
- ret = copy_ftrace_printk(handle, fd);
+ ret = copy_ftrace_printk(in_handle, out_handle);
if (ret < 0)
goto out;
/* fallthrough */
case TRACECMD_FILE_CMD_LINES:
- if (end_state <= handle->file_state)
+ if (end_state <= in_handle->file_state)
return 0;
- ret = copy_command_lines(handle, fd);
+ ret = copy_command_lines(in_handle, out_handle);
default:
break;
}
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 01c1a4f5..8499ab6f 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -2494,7 +2494,7 @@ struct tracecmd_output *tracecmd_copy(struct tracecmd_input *ihandle,
if (tracecmd_output_set_from_input(handle, ihandle))
goto out_free;
tracecmd_output_write_init(handle);
- if (tracecmd_copy_headers(ihandle, handle->fd, 0, 0) < 0)
+ if (tracecmd_copy_headers(ihandle, handle, 0, 0) < 0)
goto out_free;
handle->file_state = tracecmd_get_file_state(ihandle);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 75/87] trace-cmd library: Handle version 7 files when copying headers between files
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (73 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 74/87] trace-cmd library: Use output handler when copying data from input file Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 76/87] tarce-cmd library: Copy CPU count between trace files Tzvetomir Stoyanov (VMware)
` (11 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
When copying header data between trace files, handle the file versions.
Internal library functions for copying these headers are fixed to work
with trace file version 7:
copy_header_files
copy_ftrace_files
copy_event_files
copy_proc_kallsyms
copy_command_lines
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/trace-cmd-local.h | 1 +
lib/trace-cmd/trace-input.c | 191 ++++++++++++++++++++----
lib/trace-cmd/trace-output.c | 5 +
3 files changed, 167 insertions(+), 30 deletions(-)
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index c73b37a6..60520b07 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -51,6 +51,7 @@ void out_compression_reset(struct tracecmd_output *handle);
void in_uncompress_reset(struct tracecmd_input *handle);
int in_uncompress_block(struct tracecmd_input *handle);
+void out_set_file_state(struct tracecmd_output *handle, int new_state);
unsigned long long out_copy_fd_compress(struct tracecmd_output *handle,
int fd, unsigned long long max,
unsigned long long *write_size);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 2d2750ad..5ceb9756 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4412,161 +4412,292 @@ static int read_copy_data(struct tracecmd_input *in_handle,
static int copy_header_files(struct tracecmd_input *in_handle,
struct tracecmd_output *out_handle)
{
+ struct file_section *sec;
+ unsigned long long offset;
unsigned long long size;
- if (in_handle->file_state != TRACECMD_FILE_HEADERS - 1)
+ if (!check_out_state(out_handle, TRACECMD_FILE_HEADERS))
+ return -1;
+
+ sec = open_section(in_handle, TRACECMD_OPTION_HEADER_INFO);
+ if (!sec)
return -1;
+ offset = out_write_section_header(out_handle, TRACECMD_OPTION_HEADER_INFO,
+ "headers", TRACECMD_SEC_FL_COMPRESS, true);
+
+ out_compression_start(out_handle);
/* "header_page" */
if (read_copy_data(in_handle, 12, out_handle) < 0)
- return -1;
+ goto error;
if (read_copy_size8(in_handle, out_handle, &size) < 0)
- return -1;
+ goto error;
if (read_copy_data(in_handle, size, out_handle) < 0)
- return -1;
+ goto error;
/* "header_event" */
if (read_copy_data(in_handle, 13, out_handle) < 0)
- return -1;
+ goto error;
if (read_copy_size8(in_handle, out_handle, &size) < 0)
- return -1;
+ goto error;
if (read_copy_data(in_handle, size, out_handle) < 0)
- return -1;
+ goto error;
in_handle->file_state = TRACECMD_FILE_HEADERS;
+ if (out_compression_end(out_handle))
+ goto error;
+ out_set_file_state(out_handle, in_handle->file_state);
+ close_section(in_handle, sec);
+
+ if (out_update_section_header(out_handle, offset))
+ goto error;
return 0;
+error:
+ out_compression_reset(out_handle);
+ close_section(in_handle, sec);
+ return -1;
}
static int copy_ftrace_files(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
+ struct file_section *sec;
+ unsigned long long offset;
unsigned long long size;
unsigned int count;
unsigned int i;
- if (in_handle->file_state != TRACECMD_FILE_FTRACE_EVENTS - 1)
+ if (!check_out_state(out_handle, TRACECMD_FILE_FTRACE_EVENTS))
return -1;
- if (read_copy_size4(in_handle, out_handle, &count) < 0)
+ sec = open_section(in_handle, TRACECMD_OPTION_FTRACE_EVENTS);
+ if (!sec)
return -1;
+ offset = out_write_section_header(out_handle, TRACECMD_OPTION_FTRACE_EVENTS,
+ "ftrace events", TRACECMD_SEC_FL_COMPRESS, true);
+
+ out_compression_start(out_handle);
+
+ if (read_copy_size4(in_handle, out_handle, &count) < 0)
+ goto error;
for (i = 0; i < count; i++) {
if (read_copy_size8(in_handle, out_handle, &size) < 0)
- return -1;
+ goto error;
if (read_copy_data(in_handle, size, out_handle) < 0)
- return -1;
+ goto error;
}
in_handle->file_state = TRACECMD_FILE_FTRACE_EVENTS;
+ if (out_compression_end(out_handle))
+ goto error;
+ out_set_file_state(out_handle, in_handle->file_state);
+
+ close_section(in_handle, sec);
+
+ if (out_update_section_header(out_handle, offset))
+ goto error;
return 0;
+error:
+ out_compression_reset(out_handle);
+ close_section(in_handle, sec);
+ return -1;
}
static int copy_event_files(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
+ struct file_section *sec;
+ unsigned long long offset;
unsigned long long size;
char *system;
unsigned int systems;
unsigned int count;
unsigned int i,x;
- if (in_handle->file_state != TRACECMD_FILE_ALL_EVENTS - 1)
+ if (!check_out_state(out_handle, TRACECMD_FILE_ALL_EVENTS))
return -1;
- if (read_copy_size4(in_handle, out_handle, &systems) < 0)
+ sec = open_section(in_handle, TRACECMD_OPTION_EVENT_FORMATS);
+ if (!sec)
return -1;
+ offset = out_write_section_header(out_handle, TRACECMD_OPTION_EVENT_FORMATS,
+ "events format", TRACECMD_SEC_FL_COMPRESS, true);
+
+ out_compression_start(out_handle);
+
+ if (read_copy_size4(in_handle, out_handle, &systems) < 0)
+ goto error;
for (i = 0; i < systems; i++) {
system = read_string(in_handle);
if (!system)
- return -1;
+ goto error;
if (do_write_check(out_handle, system, strlen(system) + 1)) {
free(system);
- return -1;
+ goto error;
}
free(system);
if (read_copy_size4(in_handle, out_handle, &count) < 0)
- return -1;
+ goto error;
for (x=0; x < count; x++) {
if (read_copy_size8(in_handle, out_handle, &size) < 0)
- return -1;
+ goto error;
if (read_copy_data(in_handle, size, out_handle) < 0)
- return -1;
+ goto error;
}
}
in_handle->file_state = TRACECMD_FILE_ALL_EVENTS;
+ if (out_compression_end(out_handle))
+ goto error;
+ out_set_file_state(out_handle, in_handle->file_state);
+
+ close_section(in_handle, sec);
+
+ if (out_update_section_header(out_handle, offset))
+ goto error;
return 0;
+error:
+ out_compression_reset(out_handle);
+ close_section(in_handle, sec);
+ return -1;
}
static int copy_proc_kallsyms(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
+ struct file_section *sec;
+ unsigned long long offset;
unsigned int size;
- if (in_handle->file_state != TRACECMD_FILE_KALLSYMS - 1)
+ if (!check_out_state(out_handle, TRACECMD_FILE_KALLSYMS))
return -1;
+ sec = open_section(in_handle, TRACECMD_OPTION_KALLSYMS);
+ if (!sec)
+ return -1;
+ offset = out_write_section_header(out_handle, TRACECMD_OPTION_KALLSYMS,
+ "kallsyms", TRACECMD_SEC_FL_COMPRESS, true);
+ out_compression_start(out_handle);
if (read_copy_size4(in_handle, out_handle, &size) < 0)
- return -1;
+ goto error;
if (!size)
- return 0; /* OK? */
+ goto out; /* OK? */
if (read_copy_data(in_handle, size, out_handle) < 0)
- return -1;
-
+ goto error;
+out:
in_handle->file_state = TRACECMD_FILE_KALLSYMS;
+ if (out_compression_end(out_handle))
+ goto error;
+ out_set_file_state(out_handle, in_handle->file_state);
+
+ close_section(in_handle, sec);
+
+ if (out_update_section_header(out_handle, offset))
+ goto error;
return 0;
+error:
+ out_compression_reset(out_handle);
+ close_section(in_handle, sec);
+ return -1;
}
static int copy_ftrace_printk(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
+ struct file_section *sec;
+ unsigned long long offset;
unsigned int size;
- if (in_handle->file_state != TRACECMD_FILE_PRINTK - 1)
+ if (!check_out_state(out_handle, TRACECMD_FILE_PRINTK))
return -1;
- if (read_copy_size4(in_handle, out_handle, &size) < 0)
+ sec = open_section(in_handle, TRACECMD_OPTION_PRINTK);
+ if (!sec)
return -1;
+ offset = out_write_section_header(out_handle, TRACECMD_OPTION_PRINTK,
+ "printk", TRACECMD_SEC_FL_COMPRESS, true);
+
+ out_compression_start(out_handle);
+
+ if (read_copy_size4(in_handle, out_handle, &size) < 0)
+ goto error;
if (!size)
- return 0; /* OK? */
+ goto out; /* OK? */
if (read_copy_data(in_handle, size, out_handle) < 0)
- return -1;
+ goto error;
+out:
in_handle->file_state = TRACECMD_FILE_PRINTK;
+ if (out_compression_end(out_handle))
+ goto error;
+ out_set_file_state(out_handle, in_handle->file_state);
+
+ close_section(in_handle, sec);
+
+ if (out_update_section_header(out_handle, offset))
+ goto error;
return 0;
+error:
+ out_compression_reset(out_handle);
+ close_section(in_handle, sec);
+ return -1;
}
static int copy_command_lines(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
{
+ struct file_section *sec;
+ unsigned long long offset;
unsigned long long size;
if (!check_out_state(out_handle, TRACECMD_FILE_CMD_LINES))
return -1;
- if (read_copy_size8(in_handle, out_handle, &size) < 0)
+ sec = open_section(in_handle, TRACECMD_OPTION_CMDLINES);
+ if (!sec)
return -1;
+ offset = out_write_section_header(out_handle, TRACECMD_OPTION_CMDLINES,
+ "command lines", TRACECMD_SEC_FL_COMPRESS, true);
+
+ out_compression_start(out_handle);
+
+ if (read_copy_size8(in_handle, out_handle, &size) < 0)
+ goto error;
if (!size)
- return 0; /* OK? */
+ goto out; /* OK? */
if (read_copy_data(in_handle, size, out_handle) < 0)
- return -1;
+ goto error;
+out:
in_handle->file_state = TRACECMD_FILE_CMD_LINES;
+ if (out_compression_end(out_handle))
+ goto error;
+ out_set_file_state(out_handle, in_handle->file_state);
+
+ close_section(in_handle, sec);
+
+ if (out_update_section_header(out_handle, offset))
+ goto error;
return 0;
+error:
+ out_compression_reset(out_handle);
+ close_section(in_handle, sec);
+ return -1;
}
/**
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 8499ab6f..9972185a 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -2507,6 +2507,11 @@ out_free:
return NULL;
}
+__hidden void out_set_file_state(struct tracecmd_output *handle, int new_state)
+{
+ handle->file_state = new_state;
+}
+
__hidden bool check_out_state(struct tracecmd_output *handle, int new_state)
{
return check_file_state(handle->file_version, handle->file_state, new_state);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 76/87] tarce-cmd library: Copy CPU count between trace files
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (74 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 75/87] trace-cmd library: Handle version 7 files when copying headers between files Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 77/87] tarce-cmd library: New API to copy buffer description " Tzvetomir Stoyanov (VMware)
` (10 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The tracecmd_copy_headers() API should be able to copy CPU count also,
as it is part of the headers.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 40 +++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 5ceb9756..f4533f7e 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4700,6 +4700,33 @@ error:
return -1;
}
+static int copy_cpu_count(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
+{
+ unsigned int cpus;
+
+ if (!check_in_state(in_handle, TRACECMD_FILE_CPU_COUNT) ||
+ !check_out_state(out_handle, TRACECMD_FILE_CPU_COUNT))
+ return -1;
+
+ if (in_handle->file_version < 7) {
+ if (read4(in_handle, &cpus))
+ return -1;
+ } else {
+ cpus = in_handle->max_cpu;
+ }
+
+ if (tracecmd_get_out_file_version(out_handle) < 7) {
+ cpus = tep_read_number(in_handle->pevent, &cpus, 4);
+ if (do_write_check(out_handle, &cpus, 4))
+ return -1;
+ }
+
+ in_handle->file_state = TRACECMD_FILE_CPU_COUNT;
+ out_set_file_state(out_handle, in_handle->file_state);
+
+ return 0;
+}
+
/**
* tracecmd_copy_headers - Copy headers from a tracecmd_input handle to a file descriptor
* @in_handle: input handle for the trace.dat file to copy from.
@@ -4798,6 +4825,19 @@ int tracecmd_copy_headers(struct tracecmd_input *in_handle,
return 0;
ret = copy_command_lines(in_handle, out_handle);
+ if (ret < 0)
+ goto out;
+
+ /* fallthrough */
+ case TRACECMD_FILE_CPU_COUNT:
+ if (end_state <= in_handle->file_state)
+ return 0;
+
+ ret = copy_cpu_count(in_handle, out_handle);
+ if (ret < 0)
+ goto out;
+
+ /* fallthrough */
default:
break;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 77/87] tarce-cmd library: New API to copy buffer description between trace files
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (75 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 76/87] tarce-cmd library: Copy CPU count between trace files Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 78/87] tarce-cmd library: New API to copy options " Tzvetomir Stoyanov (VMware)
` (9 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
A new library API is introduced, to copy buffers description between
trace files:
tracecmd_copy_buffer_descr()
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/include/private/trace-cmd-private.h | 2 ++
lib/trace-cmd/trace-input.c | 13 +++++++++++++
2 files changed, 15 insertions(+)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 05a2ea01..58c17fe5 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -183,6 +183,8 @@ int tracecmd_copy_headers(struct tracecmd_input *in_handle,
struct tracecmd_output *out_handle,
enum tracecmd_file_states start_state,
enum tracecmd_file_states end_state);
+int tracecmd_copy_buffer_descr(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle);
void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index f4533f7e..76be1a02 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4846,6 +4846,19 @@ int tracecmd_copy_headers(struct tracecmd_input *in_handle,
return ret < 0 ? -1 : 0;
}
+int tracecmd_copy_buffer_descr(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle)
+{
+ int i;
+
+ if (tracecmd_get_out_file_version(out_handle) >= 7)
+ return 0;
+
+ for (i = 0; i < in_handle->nr_buffers; i++)
+ tracecmd_add_buffer_info(out_handle, in_handle->buffers[i].name, 0);
+ return tracecmd_write_buffer_info(out_handle);
+}
+
/**
* tracecmd_record_at_buffer_start - return true if record is first on subbuffer
* @handle: input handle for the trace.dat file
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 78/87] tarce-cmd library: New API to copy options between trace files
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (76 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 77/87] tarce-cmd library: New API to copy buffer description " Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 79/87] tarce-cmd library: New API to copy trace data " Tzvetomir Stoyanov (VMware)
` (8 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
A new library API is introduced, to copy options between trace files:
tracecmd_copy_options()
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 3 +
lib/trace-cmd/include/trace-cmd-local.h | 2 +
lib/trace-cmd/trace-input.c | 140 ++++++++++++++++++
lib/trace-cmd/trace-output.c | 29 ++++
4 files changed, 174 insertions(+)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 58c17fe5..8b730d1a 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -185,6 +185,8 @@ int tracecmd_copy_headers(struct tracecmd_input *in_handle,
enum tracecmd_file_states end_state);
int tracecmd_copy_buffer_descr(struct tracecmd_input *in_handle,
struct tracecmd_output *out_handle);
+int tracecmd_copy_options(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle);
void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
@@ -331,6 +333,7 @@ int tracecmd_append_buffer_cpu_data(struct tracecmd_output *handle,
const char *name, int cpus, char * const *cpu_data_files);
struct tracecmd_output *tracecmd_get_output_handle_fd(int fd);
unsigned long tracecmd_get_out_file_version(struct tracecmd_output *handle);
+unsigned long long tracecmd_get_out_file_offset(struct tracecmd_output *handle);
/* --- Reading the Fly Recorder Trace --- */
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index 60520b07..eb9d6de4 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -52,6 +52,8 @@ void in_uncompress_reset(struct tracecmd_input *handle);
int in_uncompress_block(struct tracecmd_input *handle);
void out_set_file_state(struct tracecmd_output *handle, int new_state);
+int out_save_options_offset(struct tracecmd_output *handle,
+ unsigned long long start);
unsigned long long out_copy_fd_compress(struct tracecmd_output *handle,
int fd, unsigned long long max,
unsigned long long *write_size);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index 76be1a02..be8c3e7c 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4859,6 +4859,146 @@ int tracecmd_copy_buffer_descr(struct tracecmd_input *in_handle,
return tracecmd_write_buffer_info(out_handle);
}
+static int copy_options_recursive(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle)
+{
+ unsigned short id, flags = 0;
+ unsigned short option, en2;
+ unsigned long long next;
+ unsigned int size, en4;
+ bool skip;
+
+ for (;;) {
+ if (do_read_check(in_handle, &option, 2))
+ return -1;
+ en2 = tep_read_number(in_handle->pevent, &option, 2);
+
+ if (en2 == TRACECMD_OPTION_DONE) {
+ if (in_handle->file_version < 7)
+ return 0;
+ }
+ /* next 4 bytes is the size of the option */
+ if (do_read_check(in_handle, &size, 4))
+ return -1;
+ en4 = tep_read_number(in_handle->pevent, &size, 4);
+ if (en2 == TRACECMD_OPTION_DONE) {
+ /* option done v7 */
+ if (en4 < 8)
+ return -1;
+ if (read8(in_handle, &next))
+ return -1;
+ if (!next)
+ break;
+ if (do_lseek(in_handle, next, SEEK_SET) == (off64_t)-1)
+ return -1;
+ if (read_section_header(in_handle, &id, &flags, NULL, NULL))
+ return -1;
+ if (id != TRACECMD_OPTION_DONE)
+ return -1;
+ if (flags & TRACECMD_SEC_FL_COMPRESS && in_uncompress_block(in_handle))
+ return -1;
+ return copy_options_recursive(in_handle, out_handle);
+ }
+ /* Do not copy these, as they have file specific offsets */
+ switch (en2) {
+ case TRACECMD_OPTION_BUFFER:
+ case TRACECMD_OPTION_BUFFER_LAT:
+ case TRACECMD_OPTION_HEADER_INFO:
+ case TRACECMD_OPTION_FTRACE_EVENTS:
+ case TRACECMD_OPTION_EVENT_FORMATS:
+ case TRACECMD_OPTION_KALLSYMS:
+ case TRACECMD_OPTION_PRINTK:
+ case TRACECMD_OPTION_CMDLINES:
+ skip = true;
+ break;
+ default:
+ skip = false;
+ break;
+ }
+ if (skip) {
+ do_lseek(in_handle, en4, SEEK_CUR);
+ continue;
+ }
+ if (do_write_check(out_handle, &option, 2))
+ return -1;
+ if (do_write_check(out_handle, &size, 4))
+ return -1;
+ if (read_copy_data(in_handle, en4, out_handle))
+ return -1;
+ }
+
+ return 0;
+}
+
+static int copy_options(struct tracecmd_input *in_handle, struct tracecmd_output *out_handle)
+{
+ unsigned long long offset, start;
+ unsigned short id, en2, flags = 0;
+ int tmp;
+
+ if (in_handle->file_version >= 7) {
+ if (read_section_header(in_handle, &id, &flags, NULL, NULL))
+ return -1;
+ if (id != TRACECMD_OPTION_DONE)
+ return -1;
+ if (flags & TRACECMD_SEC_FL_COMPRESS && in_uncompress_block(in_handle))
+ return -1;
+ }
+ start = tracecmd_get_out_file_offset(out_handle);
+ if (tracecmd_get_out_file_version(out_handle) < 7) {
+ if (do_write_check(out_handle, "options ", 10))
+ return -1;
+ }
+
+ offset = out_write_section_header(out_handle, TRACECMD_OPTION_DONE, "options", 0, false);
+
+ if (copy_options_recursive(in_handle, out_handle))
+ goto error;
+
+ id = TRACECMD_OPTION_DONE;
+ en2 = tep_read_number(in_handle->pevent, &id, 2);
+ if (do_write_check(out_handle, &en2, 2))
+ goto error;
+ if (tracecmd_get_out_file_version(out_handle) < 7) {
+ out_save_options_offset(out_handle, start);
+ } else {
+ tmp = 8;
+ if (do_write_check(out_handle, &tmp, 4))
+ goto error;
+ out_save_options_offset(out_handle, start);
+ start = 0;
+ if (do_write_check(out_handle, &start, 8))
+ goto error;
+ }
+ out_update_section_header(out_handle, offset);
+ if (flags & TRACECMD_SEC_FL_COMPRESS)
+ in_uncompress_reset(in_handle);
+ in_handle->file_state = TRACECMD_FILE_OPTIONS;
+ out_set_file_state(out_handle, in_handle->file_state);
+ /* Append local options */
+ return tracecmd_append_options(out_handle);
+
+error:
+ if (flags & TRACECMD_SEC_FL_COMPRESS)
+ in_uncompress_reset(in_handle);
+ return 0;
+}
+
+int tracecmd_copy_options(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle)
+{
+ if (!check_out_state(out_handle, TRACECMD_FILE_OPTIONS))
+ return -1;
+
+ if (!in_handle->options_start)
+ return 0;
+ if (lseek64(in_handle->fd, in_handle->options_start, SEEK_SET) == (off64_t)-1)
+ return -1;
+ if (copy_options(in_handle, out_handle) < 0)
+ return -1;
+ return 0;
+}
+
/**
* tracecmd_record_at_buffer_start - return true if record is first on subbuffer
* @handle: input handle for the trace.dat file
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index 9972185a..b02fad2f 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -2517,6 +2517,30 @@ __hidden bool check_out_state(struct tracecmd_output *handle, int new_state)
return check_file_state(handle->file_version, handle->file_state, new_state);
}
+__hidden int out_save_options_offset(struct tracecmd_output *handle, unsigned long long start)
+{
+ unsigned long long new, en8;
+
+ if (handle->file_version < 7) {
+ handle->options_start = start;
+ } else {
+ /* Append to the previous options section, if any */
+ if (!handle->options_start)
+ return -1;
+ new = do_lseek(handle, 0, SEEK_CUR);
+ if (do_lseek(handle, handle->options_start, SEEK_SET) == (off64_t)-1)
+ return -1;
+
+ en8 = convert_endian_8(handle, start);
+ if (do_write_check(handle, &en8, 8))
+ return -1;
+ handle->options_start = new;
+ if (do_lseek(handle, new, SEEK_SET) == (off64_t)-1)
+ return -1;
+ }
+ return 0;
+}
+
/**
* tracecmd_get_out_file_version - return the trace.dat file version
* @handle: output handle for the trace.dat file
@@ -2525,3 +2549,8 @@ unsigned long tracecmd_get_out_file_version(struct tracecmd_output *handle)
{
return handle->file_version;
}
+
+unsigned long long tracecmd_get_out_file_offset(struct tracecmd_output *handle)
+{
+ return do_lseek(handle, 0, SEEK_CUR);
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 79/87] tarce-cmd library: New API to copy trace data between trace files
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (77 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 78/87] tarce-cmd library: New API to copy options " Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 80/87] trace-cmd library: Extend tracecmd_copy() API Tzvetomir Stoyanov (VMware)
` (7 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
A new library API is introduced, to copy trace data between trace files:
tracecmd_copy_trace_data()
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 2 +
lib/trace-cmd/include/trace-cmd-local.h | 6 +
lib/trace-cmd/trace-input.c | 166 ++++++++++++++++++
lib/trace-cmd/trace-output.c | 38 ++++
4 files changed, 212 insertions(+)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 8b730d1a..3c3c98aa 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -187,6 +187,8 @@ int tracecmd_copy_buffer_descr(struct tracecmd_input *in_handle,
struct tracecmd_output *out_handle);
int tracecmd_copy_options(struct tracecmd_input *in_handle,
struct tracecmd_output *out_handle);
+int tracecmd_copy_trace_data(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle);
void tracecmd_set_flag(struct tracecmd_input *handle, int flag);
void tracecmd_clear_flag(struct tracecmd_input *handle, int flag);
unsigned long tracecmd_get_flags(struct tracecmd_input *handle);
diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h
index eb9d6de4..79c09fca 100644
--- a/lib/trace-cmd/include/trace-cmd-local.h
+++ b/lib/trace-cmd/include/trace-cmd-local.h
@@ -64,6 +64,11 @@ int out_update_section_header(struct tracecmd_output *handle, unsigned long long
long long do_write_check(struct tracecmd_output *handle, const void *data, long long size);
+struct tracecmd_option *
+out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name,
+ unsigned short id, unsigned long long data_offset,
+ int cpus, struct data_file_write *cpu_data);
+
struct cpu_data_source {
int fd;
int size;
@@ -72,6 +77,7 @@ struct cpu_data_source {
int out_write_cpu_data(struct tracecmd_output *handle, int cpus,
struct cpu_data_source *data, const char *buff_name);
+int out_write_emty_cpu_data(struct tracecmd_output *handle, int cpus);
off64_t msg_lseek(struct tracecmd_msg_handle *msg_handle, off_t offset, int whence);
unsigned long long get_last_option_offset(struct tracecmd_input *handle);
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index be8c3e7c..b66d876f 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -4999,6 +4999,172 @@ int tracecmd_copy_options(struct tracecmd_input *in_handle,
return 0;
}
+static int copy_trace_latency(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle, const char *buf_name)
+{
+ unsigned long long wsize;
+ unsigned long long offset;
+ int fd;
+
+ if (tracecmd_get_out_file_version(out_handle) < 7 &&
+ do_write_check(out_handle, "latency ", 10))
+ return -1;
+
+ offset = tracecmd_get_out_file_offset(out_handle);
+
+ if (tracecmd_get_out_file_version(out_handle) >= 7 &&
+ !out_add_buffer_option_v7(out_handle, buf_name, TRACECMD_OPTION_BUFFER_LAT, offset, 0, NULL))
+ return -1;
+
+ offset = out_write_section_header(out_handle, TRACECMD_OPTION_BUFFER_LAT,
+ "buffer latency", TRACECMD_SEC_FL_COMPRESS, false);
+
+ if (in_handle->latz.fd >= 0)
+ fd = in_handle->latz.fd;
+ else
+ fd = in_handle->fd;
+
+ if (!out_copy_fd_compress(out_handle, fd, 0, &wsize))
+ return -1;
+ if (out_update_section_header(out_handle, offset))
+ return -1;
+
+ out_set_file_state(out_handle, TRACECMD_FILE_CPU_LATENCY);
+ return 0;
+}
+
+static int copy_trace_flyrecord_data(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle, const char *buff_name)
+{
+ struct cpu_data_source *data;
+ int total_size = 0;
+ int ret;
+ int i;
+
+ data = calloc(in_handle->cpus, sizeof(struct cpu_data_source));
+ if (!data)
+ return -1;
+ for (i = 0; i < in_handle->cpus; i++) {
+ data[i].size = in_handle->cpu_data[i].file_size;
+ total_size += data[i].size;
+ if (in_handle->cpu_data[i].compress.fd >= 0) {
+ data[i].fd = in_handle->cpu_data[i].compress.fd;
+ data[i].offset = 0;
+ } else {
+ data[i].fd = in_handle->fd;
+ data[i].offset = in_handle->cpu_data[i].file_offset;
+ }
+ }
+ if (total_size || tracecmd_get_out_file_version(out_handle) < 7)
+ ret = out_write_cpu_data(out_handle, in_handle->cpus, data, buff_name);
+ else
+ ret = 0;
+ free(data);
+
+ return ret;
+}
+
+static int copy_flyrecord_buffer(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle, int index)
+{
+ struct tracecmd_input *instance;
+ const char *name;
+ int ret;
+
+ name = tracecmd_buffer_instance_name(in_handle, index);
+ if (!name)
+ return -1;
+
+ instance = tracecmd_buffer_instance_handle(in_handle, index);
+ if (!instance)
+ return -1;
+ if (!tracecmd_get_quiet(out_handle) && *name)
+ fprintf(stderr, "\nBuffer: %s\n\n", name);
+
+ if (in_handle->buffers[index].latency)
+ ret = copy_trace_latency(in_handle, out_handle, name);
+ else
+ ret = copy_trace_flyrecord_data(instance, out_handle, name);
+ tracecmd_close(instance);
+
+ return ret;
+}
+
+static int copy_trace_data_from_v6(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle)
+{
+ char buf[10];
+ int ret;
+ int i;
+
+ if (do_read_check(in_handle, buf, 10))
+ return -1;
+
+ if (strncmp(buf, "latency", 7) == 0)
+ in_handle->file_state = TRACECMD_FILE_CPU_LATENCY;
+ else if (strncmp(buf, "flyrecord", 9) == 0)
+ in_handle->file_state = TRACECMD_FILE_CPU_FLYRECORD;
+
+ tracecmd_init_data(in_handle);
+
+ if (in_handle->file_state == TRACECMD_FILE_CPU_LATENCY)
+ return copy_trace_latency(in_handle, out_handle, "");
+
+ /* top instance */
+ ret = copy_trace_flyrecord_data(in_handle, out_handle, "");
+ if (ret)
+ return ret;
+
+ for (i = 0; i < in_handle->nr_buffers; i++)
+ copy_flyrecord_buffer(in_handle, out_handle, i);
+ return 0;
+}
+
+static int copy_trace_data_from_v7(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle)
+{
+ int ret;
+ int i;
+
+ /* Force using temporary files for trace data decompression */
+ in_handle->read_zpage = false;
+ tracecmd_init_data(in_handle);
+
+ if (tracecmd_get_out_file_version(out_handle) < 7) {
+ /* copy top buffer in v6 files */
+ if (in_handle->top_buffer.latency)
+ ret = copy_trace_latency(in_handle, out_handle, in_handle->top_buffer.name);
+ else if (in_handle->top_buffer.cpus)
+ ret = copy_trace_flyrecord_data(in_handle, out_handle,
+ in_handle->top_buffer.name);
+ else
+ ret = out_write_emty_cpu_data(out_handle, in_handle->max_cpu);
+ if (ret)
+ return ret;
+ }
+
+ for (i = 0; i < in_handle->nr_buffers; i++)
+ copy_flyrecord_buffer(in_handle, out_handle, i);
+
+ return 0;
+}
+
+__hidden int tracecmd_copy_trace_data(struct tracecmd_input *in_handle,
+ struct tracecmd_output *out_handle)
+{
+ int ret;
+
+ if (!check_in_state(in_handle, TRACECMD_FILE_CPU_FLYRECORD) ||
+ !check_out_state(out_handle, TRACECMD_FILE_CPU_FLYRECORD))
+ return -1;
+ tracecmd_set_out_clock(out_handle, in_handle->trace_clock);
+ if (in_handle->file_version < 7)
+ ret = copy_trace_data_from_v6(in_handle, out_handle);
+ else
+ ret = copy_trace_data_from_v7(in_handle, out_handle);
+ return ret;
+}
+
/**
* tracecmd_record_at_buffer_start - return true if record is first on subbuffer
* @handle: input handle for the trace.dat file
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index b02fad2f..b718770d 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -2170,6 +2170,44 @@ static int update_buffer_cpu_offset_v6(struct tracecmd_output *handle,
return 0;
}
+__hidden int out_write_emty_cpu_data(struct tracecmd_output *handle, int cpus)
+{
+ unsigned long long zero = 0;
+ char *clock;
+ int ret;
+ int i;
+
+ if (handle->file_version >= 7)
+ return 0;
+
+ ret = handle->file_state == TRACECMD_FILE_CPU_FLYRECORD ? 0 :
+ check_file_state(handle->file_version,
+ handle->file_state,
+ TRACECMD_FILE_CPU_FLYRECORD);
+ if (ret < 0) {
+ tracecmd_warning("Cannot write trace data into the file, unexpected state 0x%X",
+ handle->file_state);
+ return ret;
+ }
+
+ if (do_write_check(handle, "flyrecord", 10))
+ return -1;
+
+ for (i = 0; i < cpus; i++) {
+ /* Write 0 for trace data offset and size */
+ if (do_write_check(handle, &zero, 8))
+ return -1;
+ if (do_write_check(handle, &zero, 8))
+ return -1;
+ }
+ clock = get_clock(handle);
+ if (clock && save_clock(handle, clock))
+ return -1;
+ handle->file_state = TRACECMD_FILE_CPU_FLYRECORD;
+ return 0;
+}
+
+
__hidden int out_write_cpu_data(struct tracecmd_output *handle,
int cpus, struct cpu_data_source *data, const char *buff_name)
{
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 80/87] trace-cmd library: Extend tracecmd_copy() API
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (78 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 79/87] tarce-cmd library: New API to copy trace data " Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 81/87] trace-cmd library: Set correct CPU to the record, retrieved with tracecmd_peek_data Tzvetomir Stoyanov (VMware)
` (6 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
The tracecmd_copy() API is used to copy data between trace files.
Extended it to support copying data bwteen files from different
versions. Added also functionality to copy all sections between the
files.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../include/private/trace-cmd-private.h | 5 ++-
lib/trace-cmd/trace-output.c | 38 +++++++++++++++----
tracecmd/trace-restore.c | 2 +-
tracecmd/trace-split.c | 2 +-
4 files changed, 36 insertions(+), 11 deletions(-)
diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 3c3c98aa..fd7b17e1 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -324,8 +324,9 @@ int tracecmd_write_options(struct tracecmd_output *handle);
int tracecmd_append_options(struct tracecmd_output *handle);
void tracecmd_output_close(struct tracecmd_output *handle);
void tracecmd_output_free(struct tracecmd_output *handle);
-struct tracecmd_output *tracecmd_copy(struct tracecmd_input *ihandle,
- const char *file);
+struct tracecmd_output *tracecmd_copy(struct tracecmd_input *ihandle, const char *file,
+ enum tracecmd_file_states state, int file_version,
+ const char *compression);
int tracecmd_write_cpu_data(struct tracecmd_output *handle,
int cpus, char * const *cpu_data_files, const char *buff_name);
diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c
index b718770d..e45369d3 100644
--- a/lib/trace-cmd/trace-output.c
+++ b/lib/trace-cmd/trace-output.c
@@ -2511,14 +2511,22 @@ struct tracecmd_output *tracecmd_create_init_file(const char *output_file)
* tracecmd_copy - copy the headers of one trace.dat file for another
* @ihandle: input handle of the trace.dat file to copy
* @file: the trace.dat file to create
+ * @state: what data will be copied from the source handle
+ * @compression: compression of the output file, can be one of:
+ * NULL - inherit compression from the input file
+ * "any" - compress the output file with the best available algorithm
+ * "none" - do not compress the output file
+ * algorithm_name - compress the output file with specified algorithm
*
* Reads the header information and creates a new trace data file
* with the same characteristics (events and all) and returns
* tracecmd_output handle to this new file.
*/
-struct tracecmd_output *tracecmd_copy(struct tracecmd_input *ihandle,
- const char *file)
+struct tracecmd_output *tracecmd_copy(struct tracecmd_input *ihandle, const char *file,
+ enum tracecmd_file_states state, int file_version,
+ const char *compression)
{
+ enum tracecmd_file_states fstate;
struct tracecmd_output *handle;
int fd;
@@ -2531,17 +2539,33 @@ struct tracecmd_output *tracecmd_copy(struct tracecmd_input *ihandle,
goto out_free;
if (tracecmd_output_set_from_input(handle, ihandle))
goto out_free;
+ if (file_version >= FILE_VERSION_MIN)
+ tracecmd_output_set_version(handle, file_version);
+ if (compression && tracecmd_output_set_compression(handle, compression))
+ goto out_free;
tracecmd_output_write_init(handle);
- if (tracecmd_copy_headers(ihandle, handle, 0, 0) < 0)
+ fstate = state > TRACECMD_FILE_CPU_COUNT ? TRACECMD_FILE_CPU_COUNT : state;
+ if (tracecmd_copy_headers(ihandle, handle, 0, fstate) < 0)
goto out_free;
-
- handle->file_state = tracecmd_get_file_state(ihandle);
-
+ if (tracecmd_copy_buffer_descr(ihandle, handle) < 0)
+ goto out_free;
+ if (state >= TRACECMD_FILE_OPTIONS &&
+ tracecmd_copy_options(ihandle, handle) < 0)
+ goto out_free;
+ if (state >= TRACECMD_FILE_CPU_LATENCY &&
+ tracecmd_copy_trace_data(ihandle, handle) < 0)
+ goto out_free;
+ if (handle->file_version >= 7)
+ tracecmd_write_options(handle);
/* The file is all ready to have cpu data attached */
return handle;
out_free:
- tracecmd_output_close(handle);
+ if (handle)
+ tracecmd_output_close(handle);
+ else
+ close(fd);
+ unlink(file);
return NULL;
}
diff --git a/tracecmd/trace-restore.c b/tracecmd/trace-restore.c
index fbda42dd..eedf6b49 100644
--- a/tracecmd/trace-restore.c
+++ b/tracecmd/trace-restore.c
@@ -153,7 +153,7 @@ void trace_restore (int argc, char **argv)
if (tracecmd_read_headers(ihandle, TRACECMD_FILE_CMD_LINES) < 0)
die("error reading file %s headers", input);
- handle = tracecmd_copy(ihandle, output);
+ handle = tracecmd_copy(ihandle, output, TRACECMD_FILE_CMD_LINES, 0, NULL);
tracecmd_close(ihandle);
} else
handle = tracecmd_create_init_file(output);
diff --git a/tracecmd/trace-split.c b/tracecmd/trace-split.c
index 0768329f..3759539c 100644
--- a/tracecmd/trace-split.c
+++ b/tracecmd/trace-split.c
@@ -348,7 +348,7 @@ static double parse_file(struct tracecmd_input *handle,
dir = dirname(output);
base = basename(output);
- ohandle = tracecmd_copy(handle, output_file);
+ ohandle = tracecmd_copy(handle, output_file, TRACECMD_FILE_CMD_LINES, 0, NULL);
cpus = tracecmd_cpus(handle);
cpu_data = malloc(sizeof(*cpu_data) * cpus);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 81/87] trace-cmd library: Set correct CPU to the record, retrieved with tracecmd_peek_data
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (79 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 80/87] trace-cmd library: Extend tracecmd_copy() API Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 82/87] trace-cmd: Add new subcommand "convert" Tzvetomir Stoyanov (VMware)
` (5 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace files version 7 stores only CPUs with trace data in the file,
empty CPUs are ommited. Fixed tracecmd_peek_data() to handle that case,
set the CPU in the record - do not use CPU indfex, but CPU id.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
lib/trace-cmd/trace-input.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c
index b66d876f..df2e999c 100644
--- a/lib/trace-cmd/trace-input.c
+++ b/lib/trace-cmd/trace-input.c
@@ -2403,7 +2403,7 @@ read_again:
record->ts = handle->cpu_data[cpu].timestamp;
record->size = kbuffer_event_size(kbuf);
- record->cpu = cpu;
+ record->cpu = handle->cpu_data[cpu].cpu;
record->data = data;
record->offset = handle->cpu_data[cpu].offset + index;
record->missed_events = kbuffer_missed_events(kbuf);
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 82/87] trace-cmd: Add new subcommand "convert"
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (80 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 81/87] trace-cmd library: Set correct CPU to the record, retrieved with tracecmd_peek_data Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 83/87] trace-cmd report: Add new parameters for version 7 trace files Tzvetomir Stoyanov (VMware)
` (4 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
There is a need to convert trace files between different versions. A new
"trace-cmd convert" subcommand is introduced, which reads and input
trace file and copies to an output trace file. Both files can be from
different versions.
trace-cmd convert -i <input file> -o <output file> --file-version <ver>
--compression <compression>
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/Makefile | 1 +
tracecmd/include/trace-local.h | 2 +
tracecmd/trace-cmd.c | 1 +
tracecmd/trace-convert.c | 106 +++++++++++++++++++++++++++++++++
tracecmd/trace-usage.c | 13 ++++
5 files changed, 123 insertions(+)
create mode 100644 tracecmd/trace-convert.c
diff --git a/tracecmd/Makefile b/tracecmd/Makefile
index b7a23dc4..56742f0a 100644
--- a/tracecmd/Makefile
+++ b/tracecmd/Makefile
@@ -36,6 +36,7 @@ TRACE_CMD_OBJS += trace-usage.o
TRACE_CMD_OBJS += trace-dump.o
TRACE_CMD_OBJS += trace-clear.o
TRACE_CMD_OBJS += trace-vm.o
+TRACE_CMD_OBJS += trace-convert.o
ifeq ($(VSOCK_DEFINED), 1)
TRACE_CMD_OBJS += trace-agent.o
diff --git a/tracecmd/include/trace-local.h b/tracecmd/include/trace-local.h
index e9a0aea8..13dab44c 100644
--- a/tracecmd/include/trace-local.h
+++ b/tracecmd/include/trace-local.h
@@ -110,6 +110,8 @@ void trace_usage(int argc, char **argv);
void trace_dump(int argc, char **argv);
+void trace_convert(int argc, char **argv);
+
int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
int cpus, int *fds,
int argc, char **argv, bool use_fifos,
diff --git a/tracecmd/trace-cmd.c b/tracecmd/trace-cmd.c
index 9fc126e4..a83a8d0b 100644
--- a/tracecmd/trace-cmd.c
+++ b/tracecmd/trace-cmd.c
@@ -133,6 +133,7 @@ struct command commands[] = {
{"list", trace_list},
{"help", trace_usage},
{"dump", trace_dump},
+ {"convert", trace_convert},
{"-h", trace_usage},
};
diff --git a/tracecmd/trace-convert.c b/tracecmd/trace-convert.c
new file mode 100644
index 00000000..2484d529
--- /dev/null
+++ b/tracecmd/trace-convert.c
@@ -0,0 +1,106 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2021, VMware, Tzvetomir Stoyanov <tz.stoyanov@gmail.com>
+ */
+#include <stdlib.h>
+#include <unistd.h>
+#include <getopt.h>
+#include <fcntl.h>
+#include <errno.h>
+
+#include "trace-local.h"
+#include "trace-cmd.h"
+#include "trace-cmd-private.h"
+
+static void convert_file(const char *in, const char *out, int file_version, char *compr)
+{
+ struct tracecmd_input *ihandle;
+ struct tracecmd_output *ohandle;
+
+ ihandle = tracecmd_open_head(in, 0);
+ if (!ihandle)
+ die("error reading %s", in);
+ ohandle = tracecmd_copy(ihandle, out, TRACECMD_FILE_CPU_FLYRECORD, file_version, compr);
+ if (!ohandle)
+ die("error writing %s", out);
+ tracecmd_output_close(ohandle);
+ tracecmd_close(ihandle);
+}
+
+enum {
+ OPT_file_version = 254,
+ OPT_comporession = 255,
+};
+
+void trace_convert(int argc, char **argv)
+{
+ char *input_file = NULL;
+ char *output_file = NULL;
+ char *compression = NULL;
+ int file_version = FILE_VERSION_DEFAULT;
+ int c;
+
+ if (argc < 2)
+ usage(argv);
+
+ if (strcmp(argv[1], "convert") != 0)
+ usage(argv);
+ for (;;) {
+ int option_index = 0;
+ static struct option long_options[] = {
+ {"compression", required_argument, NULL, OPT_comporession},
+ {"file-version", required_argument, NULL, OPT_file_version},
+ {"help", no_argument, NULL, '?'},
+ {NULL, 0, NULL, 0}
+ };
+
+ c = getopt_long (argc-1, argv+1, "+hi:o:", long_options, &option_index);
+ if (c == -1)
+ break;
+ switch (c) {
+ case 'i':
+ if (input_file)
+ die("Only one input file is supported, %s already set",
+ input_file);
+ input_file = optarg;
+ break;
+ case 'o':
+ if (output_file)
+ die("Only one output file is supported, %s already set",
+ output_file);
+ output_file = optarg;
+ break;
+ case OPT_comporession:
+ if (strcmp(optarg, "any") && strcmp(optarg, "none") &&
+ !tracecmd_compress_is_supported(optarg, NULL))
+ die("Compression algorithm %s is not supported", optarg);
+ compression = optarg;
+ break;
+ case OPT_file_version:
+ file_version = atoi(optarg);
+ if (file_version < FILE_VERSION_MIN || file_version > FILE_VERSION_MAX)
+ die("Unsupported file version %d, "
+ "supported versions are from %d to %d",
+ file_version, FILE_VERSION_MIN, FILE_VERSION_MAX);
+
+ break;
+ case 'h':
+ case '?':
+ default:
+ usage(argv);
+ }
+ }
+
+ if ((argc - optind) >= 2) {
+ if (output_file)
+ usage(argv);
+ output_file = argv[optind + 1];
+ }
+
+ if (!input_file)
+ input_file = DEFAULT_INPUT_FILE;
+ if (!output_file)
+ usage(argv);
+
+ convert_file(input_file, output_file, file_version, compression);
+}
diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c
index f463465a..9eb13ecb 100644
--- a/tracecmd/trace-usage.c
+++ b/tracecmd/trace-usage.c
@@ -407,6 +407,19 @@ static struct usage_help usage_help[] = {
" -h, --help show usage information\n"
" --verbose 'level' Set the desired log level\n"
},
+ {
+ "convert",
+ "convert trace file to different version",
+ " %s convert [options]\n"
+ " -i input file, default is trace.dat\n"
+ " -o output file, mandatory parameter.\n"
+ " The output file can be specified also as last argument of the command\n"
+ " --file-version set the desired trace file version\n"
+ " --compression compress the trace output file, one of these strings can be passed:\n"
+ " any - auto select the best available compression algorithm\n"
+ " none - do not compress the trace file\n"
+ " name - the name of the desired compression algorithms\n"
+ " available algorithms can be listed with trace-cmd list -c\n" },
{
NULL, NULL, NULL
}
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 83/87] trace-cmd report: Add new parameters for version 7 trace files
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (81 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 82/87] trace-cmd: Add new subcommand "convert" Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 19:26 ` Steven Rostedt
2021-07-29 5:09 ` [PATCH v2 84/87] trace-cmd: Update bash completion Tzvetomir Stoyanov (VMware)
` (3 subsequent siblings)
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
By default, "trace-cmd report" writes in trace file version 7 and no
trace file compression. A new parameters are added, which can be used to
set desired verrsion and compression of the output trace file.
"trace-cmd report --file-version <version> --compression <compression>"
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-record.c | 28 ++++++++++++++++++++++++++++
tracecmd/trace-usage.c | 7 ++++++-
2 files changed, 34 insertions(+), 1 deletion(-)
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index ddab7798..7013aee3 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -5810,6 +5810,8 @@ void init_top_instance(void)
}
enum {
+ OPT_file_ver = 237,
+ OPT_comporession = 238,
OPT_verbose = 239,
OPT_tsc2nsec = 240,
OPT_fork = 241,
@@ -6249,6 +6251,8 @@ static void parse_record_options(int argc,
{"tsc2nsec", no_argument, NULL, OPT_tsc2nsec},
{"poll", no_argument, NULL, OPT_poll},
{"verbose", optional_argument, NULL, OPT_verbose},
+ {"compression", required_argument, NULL, OPT_comporession},
+ {"file-version", required_argument, NULL, OPT_file_ver},
{NULL, 0, NULL, 0}
};
@@ -6674,6 +6678,30 @@ static void parse_record_options(int argc,
cmd_check_die(ctx, CMD_set, *(argv+1), "--poll");
recorder_flags |= TRACECMD_RECORD_POLL;
break;
+ case OPT_comporession:
+ cmd_check_die(ctx, CMD_start, *(argv+1), "--compression");
+ cmd_check_die(ctx, CMD_set, *(argv+1), "--compression");
+ cmd_check_die(ctx, CMD_extract, *(argv+1), "--compression");
+ cmd_check_die(ctx, CMD_stream, *(argv+1), "--compression");
+ cmd_check_die(ctx, CMD_profile, *(argv+1), "--compression");
+ if (strcmp(optarg, "any") && strcmp(optarg, "none") &&
+ !tracecmd_compress_is_supported(optarg, NULL))
+ die("Compression algorithm %s is not supported", optarg);
+ ctx->compression = strdup(optarg);
+ break;
+ case OPT_file_ver:
+ cmd_check_die(ctx, CMD_start, *(argv+1), "--file_version");
+ cmd_check_die(ctx, CMD_set, *(argv+1), "--file_version");
+ cmd_check_die(ctx, CMD_extract, *(argv+1), "--file_version");
+ cmd_check_die(ctx, CMD_stream, *(argv+1), "--file_version");
+ cmd_check_die(ctx, CMD_profile, *(argv+1), "--file_version");
+ ctx->file_version = atoi(optarg);
+ if (ctx->file_version < FILE_VERSION_MIN ||
+ ctx->file_version > FILE_VERSION_MAX)
+ die("Unsupported file version %d, "
+ "supported versions are from %d to %d",
+ ctx->file_version, FILE_VERSION_MIN, FILE_VERSION_MAX);
+ break;
case OPT_quiet:
case 'q':
quiet = true;
diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c
index 9eb13ecb..ec0af46f 100644
--- a/tracecmd/trace-usage.c
+++ b/tracecmd/trace-usage.c
@@ -69,7 +69,12 @@ static struct usage_help usage_help[] = {
" If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
" at the beginnig and at the end of the trace\n"
" --poll don't block while reading from the trace buffer\n"
- " --verbose 'level' Set the desired log level\n"
+ " --file-version set the desired trace file version\n"
+ " --compression compress the trace output file, one of these strings can be passed:\n"
+ " any - auto select the best available compression algorithm\n"
+ " none - do not compress the trace file\n"
+ " name - the name of the desired compression algorithms\n"
+ " available algorithms can be listed with trace-cmd list -c\n"
},
{
"set",
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 83/87] trace-cmd report: Add new parameters for version 7 trace files
2021-07-29 5:09 ` [PATCH v2 83/87] trace-cmd report: Add new parameters for version 7 trace files Tzvetomir Stoyanov (VMware)
@ 2021-08-19 19:26 ` Steven Rostedt
0 siblings, 0 replies; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 19:26 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
On Thu, 29 Jul 2021 08:09:55 +0300
"Tzvetomir Stoyanov (VMware)" <tz.stoyanov@gmail.com> wrote:
> By default, "trace-cmd report" writes in trace file version 7 and no
> trace file compression. A new parameters are added, which can be used to
> set desired verrsion and compression of the output trace file.
> "trace-cmd report --file-version <version> --compression <compression>"
>
> Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
> ---
> tracecmd/trace-record.c | 28 ++++++++++++++++++++++++++++
> tracecmd/trace-usage.c | 7 ++++++-
> 2 files changed, 34 insertions(+), 1 deletion(-)
>
> diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
> index ddab7798..7013aee3 100644
> --- a/tracecmd/trace-record.c
> +++ b/tracecmd/trace-record.c
> @@ -5810,6 +5810,8 @@ void init_top_instance(void)
> }
>
> enum {
> + OPT_file_ver = 237,
> + OPT_comporession = 238,
Fix spelling for all uses.
-- Steve
> OPT_verbose = 239,
> OPT_tsc2nsec = 240,
> OPT_fork = 241,
> @@ -6249,6 +6251,8 @@ static void parse_record_options(int argc,
> {"tsc2nsec", no_argument, NULL, OPT_tsc2nsec},
> {"poll", no_argument, NULL, OPT_poll},
> {"verbose", optional_argument, NULL, OPT_verbose},
> + {"compression", required_argument, NULL, OPT_comporession},
> + {"file-version", required_argument, NULL, OPT_file_ver},
> {NULL, 0, NULL, 0}
> };
>
> @@ -6674,6 +6678,30 @@ static void parse_record_options(int argc,
> cmd_check_die(ctx, CMD_set, *(argv+1), "--poll");
> recorder_flags |= TRACECMD_RECORD_POLL;
> break;
> + case OPT_comporession:
> + cmd_check_die(ctx, CMD_start, *(argv+1), "--compression");
> + cmd_check_die(ctx, CMD_set, *(argv+1), "--compression");
> + cmd_check_die(ctx, CMD_extract, *(argv+1), "--compression");
> + cmd_check_die(ctx, CMD_stream, *(argv+1), "--compression");
> + cmd_check_die(ctx, CMD_profile, *(argv+1), "--compression");
> + if (strcmp(optarg, "any") && strcmp(optarg, "none") &&
> + !tracecmd_compress_is_supported(optarg, NULL))
> + die("Compression algorithm %s is not supported", optarg);
> + ctx->compression = strdup(optarg);
> + break;
> + case OPT_file_ver:
> + cmd_check_die(ctx, CMD_start, *(argv+1), "--file_version");
> + cmd_check_die(ctx, CMD_set, *(argv+1), "--file_version");
> + cmd_check_die(ctx, CMD_extract, *(argv+1), "--file_version");
> + cmd_check_die(ctx, CMD_stream, *(argv+1), "--file_version");
> + cmd_check_die(ctx, CMD_profile, *(argv+1), "--file_version");
> + ctx->file_version = atoi(optarg);
> + if (ctx->file_version < FILE_VERSION_MIN ||
> + ctx->file_version > FILE_VERSION_MAX)
> + die("Unsupported file version %d, "
> + "supported versions are from %d to %d",
> + ctx->file_version, FILE_VERSION_MIN, FILE_VERSION_MAX);
> + break;
> case OPT_quiet:
> case 'q':
> quiet = true;
> diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c
> index 9eb13ecb..ec0af46f 100644
> --- a/tracecmd/trace-usage.c
> +++ b/tracecmd/trace-usage.c
> @@ -69,7 +69,12 @@ static struct usage_help usage_help[] = {
> " If 0 is specified, no loop is performed - timestamps offset is calculated only twice,"
> " at the beginnig and at the end of the trace\n"
> " --poll don't block while reading from the trace buffer\n"
> - " --verbose 'level' Set the desired log level\n"
> + " --file-version set the desired trace file version\n"
> + " --compression compress the trace output file, one of these strings can be passed:\n"
> + " any - auto select the best available compression algorithm\n"
> + " none - do not compress the trace file\n"
> + " name - the name of the desired compression algorithms\n"
> + " available algorithms can be listed with trace-cmd list -c\n"
> },
> {
> "set",
^ permalink raw reply [flat|nested] 118+ messages in thread
* [PATCH v2 84/87] trace-cmd: Update bash completion
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (82 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 83/87] trace-cmd report: Add new parameters for version 7 trace files Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 85/87] tarce-cmd: Man page for "trace-cmd convert" Tzvetomir Stoyanov (VMware)
` (2 subsequent siblings)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Added convert subcommand and its arguments to bash completion logic.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
tracecmd/trace-cmd.bash | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/tracecmd/trace-cmd.bash b/tracecmd/trace-cmd.bash
index b01c7a07..6639c143 100644
--- a/tracecmd/trace-cmd.bash
+++ b/tracecmd/trace-cmd.bash
@@ -64,6 +64,13 @@ plugin_options()
COMPREPLY=( $(compgen -W "${opts}" -- "${cur}") )
}
+compression_param()
+{
+ local opts=$(trace-cmd list -c | grep -v 'Supported' | cut -d "," -f1)
+ opts+=" any none "
+ COMPREPLY=( $(compgen -W "${opts}") )
+}
+
__trace_cmd_list_complete()
{
local prev=$1
@@ -181,6 +188,9 @@ __trace_cmd_record_complete()
cmd_options record "$cur"
fi
;;
+ --compression)
+ compression_param
+ ;;
*)
# stream start and profile do not show all options
cmd_options record "$cur"
@@ -222,6 +232,29 @@ __trace_cmd_dump_complete()
esac
}
+__trace_cmd_convert_complete()
+{
+ local prev=$1
+ local cur=$2
+ shift 2
+ local words=("$@")
+
+ case "$prev" in
+ -i)
+ __show_files
+ ;;
+ -o)
+ __show_files
+ ;;
+ --compression)
+ compression_param
+ ;;
+ *)
+ cmd_options convert "$cur"
+ ;;
+ esac
+}
+
__show_command_options()
{
local command="$1"
@@ -298,6 +331,10 @@ _trace_cmd_complete()
__trace_cmd_dump_complete "${prev}" "${cur}" ${words[@]}
return 0
;;
+ convert)
+ __trace_cmd_convert_complete "${prev}" "${cur}" ${words[@]}
+ return 0
+ ;;
*)
__show_command_options "$w" "${prev}" "${cur}"
;;
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 85/87] tarce-cmd: Man page for "trace-cmd convert"
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (83 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 84/87] trace-cmd: Update bash completion Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 86/87] tarce-cmd: Update record man page Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 87/87] trace-cmd: Document trace file version 7 Tzvetomir Stoyanov (VMware)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Documented new "trace-cmnd convert" subcommand.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
.../trace-cmd/trace-cmd-convert.1.txt | 65 +++++++++++++++++++
Documentation/trace-cmd/trace-cmd.1.txt | 4 +-
2 files changed, 68 insertions(+), 1 deletion(-)
create mode 100644 Documentation/trace-cmd/trace-cmd-convert.1.txt
diff --git a/Documentation/trace-cmd/trace-cmd-convert.1.txt b/Documentation/trace-cmd/trace-cmd-convert.1.txt
new file mode 100644
index 00000000..7c13cf3d
--- /dev/null
+++ b/Documentation/trace-cmd/trace-cmd-convert.1.txt
@@ -0,0 +1,65 @@
+TRACE-CMD-CONVERT(1)
+===================
+
+NAME
+----
+trace-cmd-convert - convert trace files
+
+SYNOPSIS
+--------
+*trace-cmd convert* ['OPTIONS'] ['output-file']
+
+DESCRIPTION
+-----------
+The trace-cmd(1) convert command converts trace file. It reads the input file and copies the data
+into an output file. The output file may be in different format, depending on the command line
+arguments.
+
+OPTIONS
+-------
+*-i* 'input-file'::
+ By default, trace-cmd convert will read the file 'trace.dat'. But the *-i*
+ option open up the given 'input-file' instead.
+
+*-o* 'out-file'::
+ The name of the output file, this parameter is mandatory. Note, the output file may also be
+ specified as the last item on the command line.
+
+*--file-version*::
+ Desired version of the output file. Supported versions are 6 or 7.
+
+*--compression*::
+ Compression of the trace output file, one of these strings can be passed:
+
+ 'any' - auto select the best available compression algorithm
+
+ 'none' - do not compress the trace file
+
+ 'name' - the name of the desired compression algorithms. Available algorithms can be listed with
+ trace-cmd list -c
+
+*--help*::
+ Print usage information.
+
+EXAMPLES
+--------
+
+# trace-cmd convert --compression any trace_compress.dat
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd.dat(1)
+
+AUTHOR
+------
+*Steven Rostedt* <rostedt@goodmis.org>, author of *trace-cmd*.
+*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
+
+RESOURCES
+---------
+https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/
+
+COPYING
+-------
+Copyright \(C) 2021 VMware. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
diff --git a/Documentation/trace-cmd/trace-cmd.1.txt b/Documentation/trace-cmd/trace-cmd.1.txt
index b330b4fe..27c6e973 100644
--- a/Documentation/trace-cmd/trace-cmd.1.txt
+++ b/Documentation/trace-cmd/trace-cmd.1.txt
@@ -58,6 +58,8 @@ COMMANDS
check-events - parse format strings for all trace events and return
whether all formats are parseable
+ convert - convert trace files
+
OPTIONS
-------
@@ -70,7 +72,7 @@ SEE ALSO
--------
trace-cmd-record(1), trace-cmd-report(1), trace-cmd-hist(1), trace-cmd-start(1),
trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1),
-trace-cmd-restore(1), trace-cmd-stack(1),
+trace-cmd-restore(1), trace-cmd-stack(1), trace-cmd-convert(1),
trace-cmd-split(1), trace-cmd-list(1), trace-cmd-listen(1),
trace-cmd.dat(5), trace-cmd-check-events(1) trace-cmd-stat(1)
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 86/87] tarce-cmd: Update record man page
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (84 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 85/87] tarce-cmd: Man page for "trace-cmd convert" Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-07-29 5:09 ` [PATCH v2 87/87] trace-cmd: Document trace file version 7 Tzvetomir Stoyanov (VMware)
86 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Documented new arguments of the "trace-cmd record" subcommand:
--compression
--file-version
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
Documentation/trace-cmd/trace-cmd-record.1.txt | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/Documentation/trace-cmd/trace-cmd-record.1.txt b/Documentation/trace-cmd/trace-cmd-record.1.txt
index 96c27108..b3620474 100644
--- a/Documentation/trace-cmd/trace-cmd-record.1.txt
+++ b/Documentation/trace-cmd/trace-cmd-record.1.txt
@@ -377,6 +377,19 @@ OPTIONS
trace-cmd record --verbose=warning
+*--file-version*::
+ Desired version of the output file. Supported versions are 6 or 7.
+
+*--compression*::
+ Compression of the trace output file, one of these strings can be passed:
+
+ 'any' - auto select the best available compression algorithm
+
+ 'none' - do not compress the trace file
+
+ 'name' - the name of the desired compression algorithms. Available algorithms can be listed with
+ trace-cmd list -c
+
EXAMPLES
--------
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* [PATCH v2 87/87] trace-cmd: Document trace file version 7
2021-07-29 5:08 [PATCH v2 00/87] Trace file version 7 Tzvetomir Stoyanov (VMware)
` (85 preceding siblings ...)
2021-07-29 5:09 ` [PATCH v2 86/87] tarce-cmd: Update record man page Tzvetomir Stoyanov (VMware)
@ 2021-07-29 5:09 ` Tzvetomir Stoyanov (VMware)
2021-08-19 19:33 ` Steven Rostedt
86 siblings, 1 reply; 118+ messages in thread
From: Tzvetomir Stoyanov (VMware) @ 2021-07-29 5:09 UTC (permalink / raw)
To: rostedt; +Cc: linux-trace-devel
Trace file versions 6 and 7 have a lot of differences. Added new man
page describing the version 7 and renamed the existing trace-cmd.dat
page for version 6 only.
Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
---
...e-cmd.dat.5.txt => trace-cmd.dat.v6.5.txt} | 8 +-
.../trace-cmd/trace-cmd.dat.v7.5.txt | 442 ++++++++++++++++++
2 files changed, 446 insertions(+), 4 deletions(-)
rename Documentation/trace-cmd/{trace-cmd.dat.5.txt => trace-cmd.dat.v6.5.txt} (98%)
create mode 100644 Documentation/trace-cmd/trace-cmd.dat.v7.5.txt
diff --git a/Documentation/trace-cmd/trace-cmd.dat.5.txt b/Documentation/trace-cmd/trace-cmd.dat.v6.5.txt
similarity index 98%
rename from Documentation/trace-cmd/trace-cmd.dat.5.txt
rename to Documentation/trace-cmd/trace-cmd.dat.v6.5.txt
index 8d285353..b412bfc7 100644
--- a/Documentation/trace-cmd/trace-cmd.dat.5.txt
+++ b/Documentation/trace-cmd/trace-cmd.dat.v6.5.txt
@@ -1,9 +1,9 @@
-TRACE-CMD.DAT(5)
-================
+TRACE-CMD.DAT.v6(5)
+===================
NAME
----
-trace-cmd.dat - trace-cmd file format
+trace-cmd.dat.v6 - trace-cmd version 6 file format
SYNOPSIS
--------
@@ -30,7 +30,7 @@ INITIAL FORMAT
"tracing"
The next set of characters contain a null '\0' terminated string
- that contains the version of the file (for example):
+ that contains the version of the file:
"6\0"
diff --git a/Documentation/trace-cmd/trace-cmd.dat.v7.5.txt b/Documentation/trace-cmd/trace-cmd.dat.v7.5.txt
new file mode 100644
index 00000000..36dae643
--- /dev/null
+++ b/Documentation/trace-cmd/trace-cmd.dat.v7.5.txt
@@ -0,0 +1,442 @@
+TRACE-CMD.DAT.v7(5)
+===================
+
+NAME
+----
+trace-cmd.dat.v7 - trace-cmd version 7 file format
+
+SYNOPSIS
+--------
+*trace-cmd.dat* ignore
+
+DESCRIPTION
+-----------
+The trace-cmd(1) utility produces a "trace.dat" file. The file may also
+be named anything depending if the user specifies a different output name,
+but it must have a certain binary format. The file is used
+by trace-cmd to save kernel traces into it and be able to extract
+the trace from it at a later point (see *trace-cmd-report(1)*).
+
+
+INITIAL FORMAT
+--------------
+
+ The first three bytes contain the magic value:
+
+ 0x17 0x08 0x44
+
+ The next 7 bytes contain the characters:
+
+ "tracing"
+
+ The next set of characters contain a null '\0' terminated string
+ that contains the version of the file:
+
+ "7\0"
+
+ The next 1 byte contains the flags for the file endianess:
+
+ 0 = little endian
+ 1 = big endian
+
+ The next byte contains the number of bytes per "long" value:
+
+ 4 - 32-bit long values
+ 8 - 64-bit long values
+
+ Note: This is the long size of the target's userspace. Not the
+ kernel space size.
+
+ [ Now all numbers are written in file defined endianess. ]
+
+ The next 4 bytes are a 32-bit word that defines what the traced
+ host machine page size was.
+
+ The compression algorithm header is written next:
+ "name\0version\0"
+ where "name" and "version" are strings, name and version of the
+ compression algorithm used to compress the trace file. If the name
+ is "none", the data in the file is not compressed.
+
+ The next 8 bytes are 64-bit integer, the offset within the file where
+ the first OPTIONS section is located.
+
+ The rest of the file consists of different sections. The only mandatory
+ is the first OPTIONS section, all others are optional. The location and
+ the order of the sections is not strict. Each section starts with a header:
+
+FORMAT OF THE SECTION HEADER
+----------------------------
+ <2 bytes> unsigned short integer, ID of the section.
+ <string> a null terminated ASCII string, description of the section.
+ <2 bytes> unsigned short integer, section flags:
+ 1 = the section is compressed.
+ <4 bytes> unsigned integer, size of the section in the file.
+
+ If the section is compressed, the above is the compressed size.
+ The section must be uncompressed on reading. The described format of
+ the sections refers to the uncompressed data.
+
+COMPRESSION FORMAT OF THE FILE SECTIONS
+---------------------------------------
+
+ Some of the sections in the file may be compressed with the compression algorithm,
+ specified in the compression algorithm header. Compressed sections have a compression
+ header, written after the section header and right before the compressed data:
+ <4 bytes> unsigned int, size of compressed data in this section.
+ <4 bytes> unsigned int, size of uncompressed data.
+ <data> binary compressed data, with the specified size.
+
+COMPRESSION FORMAT OF THE TRACE DATA
+------------------------------------
+
+ There are two special sections, BUFFER FLYRECORD and BUFFER LATENCY, containing
+ trace data. These sections may be compressed with the compression algorithm, specified
+ in the compression header. Usually the size of these sections is huge, that's why its
+ compression format is different from the other sections. The trace data is compressed
+ in chunks The size of one chunk is specified in the file creation time. The format
+ of compressed trace data is:
+ <4 bytes> unsigned int, count of chunks.
+ Follows the compressed chunks of given count. For each chunk:
+ <4 bytes> unsigned int, size of compressed data in this chunk.
+ <4 bytes> unsigned int, size of uncompressed data, aligned with the trace page size.
+ <data> binary compressed data, with the specified size.
+ These chunks must be uncompressed on reading. The described format of
+ trace data refers to the uncompressed data.
+
+OPTIONS SECTION
+---------------
+
+ Section ID: 0
+
+ This is the the only mandatory section in the file. There can be multiple
+ options sections, the first one is located at the offset specified right
+ after the compression algorithm header. The section consists of multiple
+ trace options, each option has the following format:
+ <2 bytes> unsigned short integer, ID of the option.
+ <4 bytes> unsigned integer, size of the option's data.
+ <binary data> bytes of the size specified above, data of the option.
+
+
+ Options, supported by the trace file version 7:
+
+ DONE: id 0, size 8
+ This option indicates the end of the options section, it is written
+ always as last option. The DONE option data is:
+ <8 bytes> long long unsigned integer, offset in the trace file where
+ the next options section is located. If this offset is 0, then there
+ are no more options sections.
+
+ DATE: id 1, size vary
+ The DATE option data is a null terminated ASCII string, which represents
+ the time difference between trace events timestamps and the Generic Time
+ of Day of the system.
+
+ CPUSTAT: id 2, size vary
+ The CPUSTAT option data is a null terminated ASCII string, the content of the
+ "per_cpu/cpu<id>/stats" file from the trace directory. There is a CPUSTAT option
+ for each CPU.
+
+ BUFFER: id 3, size vary
+ The BUFFER option describes the flyrecord trace data saved in the file, collected
+ from one trace instance. There is BUFFER option for each trace instance. The format
+ of the BUFFER data is:
+ <8 bytes> long long unsigned integer, offset in the trace file where the
+ BUFFER FLYRECORD section is located, containing flyrecord trace data.
+ <string> a null terminated ASCII string, name of the trace instance. Empty string ""
+ is saved as name of the top instance.
+ <string> a null terminated ASCII string, trace clock used for events timestamps in
+ this trace instance.
+ <4 bytes> unsigned integer, count of the CPUs with trace data.
+ For each CPU of the above count:
+ <4 bytes> unsigned integer, ID of the CPU.
+ <8 bytes> long long unsigned integer, offset in the trace file where the trace data
+ for this CPU is located.
+ <8 bytes> long long unsigned integer, size of the trace data for this CPU.
+
+ TRACECLOCK: id 4, size vary
+ The TRACECLOCK option data is a null terminated ASCII string, the content of the
+ "trace_clock" file from the trace directory.
+
+ UNAME: id 5, size vary
+ The UNAME option data is a null terminated ASCII string, identifying the system where
+ the trace data is collected. The string is retrieved by the uname() system call.
+
+ HOOK: id 6, size vary
+ The HOOK option data is a null terminated ASCII string, describing event hooks: custom
+ event matching to connect any two events together.
+
+ OFFSET: id 7, size vary
+ The OFFSET option data is a null terminated ASCII string, representing a fixed time that
+ is added to each event timestamp on reading.
+
+ CPUCOUNT: id 8, size 4
+ The CPUCOUNT option data is:
+ <4 bytes> unsigned integer, number of CPUs in the system.
+
+ VERSION: id 9, size vary
+ The VERSION option data is a null terminated ASCII string, representing the version of
+ the trace-cmd application, used to collect these trace logs.
+
+ PROCMAPS: id 10, size vary
+ The PROCMAPS option data is a null terminated ASCII string, representing the memory map
+ of each traced filtered process. The format of the string is, for each filtered process:
+ <procss ID> <libraries count> <process command> \n
+ <memory start address> <memory end address> <full path of the mapped library file> \n
+ ...
+ separate line for each library, used by this process
+ ...
+ ...
+
+ TRACEID: id 11, size 8
+ The TRACEID option data is a unique identifier of this tracing session:
+ <8 bytes> long long unsigned integer, trace session identifier.
+
+ TIME_SHIFT: id 12, size vary
+ The TIME_SHIFT option stores time synchronization information, collected during host and guest
+ tracing session. Usually it is saved in the guest trace file. This information is used to
+ synchronize guest with host events timestamps, when displaying all files from this tracing
+ session. The format of the TIME_SHIFT option data is:
+ <8 bytes> long long unsigned integer, trace identifier of the peer (usually the host).
+ <4 bytes> unsigned integer, flags specific to the time synchronization protocol, used in this
+ trace session.
+ <4 bytes> unsigned integer, number of traced CPUs. For each CPU, timestamps corrections
+ are recorded:
+ <4 bytes> unsigned integer, count of the recorded timestamps corrections for this CPU.
+ <array of unsigned long long integers of the above count>, times when the corrections are calculated
+ <array of unsigned long long integers of the above count>, corrections offsets
+ <array of unsigned long long integers of the above count>, corrections scaling ratio
+
+ GUEST: id 13, size vary
+ The GUEST option stores information about traced guests in this tracing session. Usually it is
+ saved in the host trace file. There is a separate GUEST option for each traced guest.
+ The information is used when displaying all files from this tracing session. The format of
+ the GUEST option data is:
+ <string> a null terminated ASCII string, name of the guest.
+ <8 bytes> long long unsigned integer, trace identifier of the guest for this session.
+ <4 bytes> unsigned integer, number of guest's CPUs. For each CPU:
+ <4 bytes> unsigned integer, ID of the CPU.
+ <4 bytes> unsigned integer, PID of the host task, emulating this guest CPU.
+
+ TSC2NSEC: id 14, size 16
+ The TSC2NSEC option stores information, used to convert TSC events timestamps to nanoseconds.
+ The format of the TSC2NSEC option data is:
+ <4 bytes> unsigned integer, time multiplier.
+ <4 bytes> unsigned integer, time shift.
+ <8 bytes> unsigned long long integer, time offset.
+
+ BUFFER_LAT: id 15, size
+ The BUFFER_LAT option describes the latency trace data saved in the file. The format
+ of the BUFFER_LAT data is:
+ <8 bytes> long long unsigned integer, offset in the trace file where the
+ BUFFER LATENCY section is located, containing latency trace data.
+ <string> a null terminated ASCII string, name of the trace instance. Empty string ""
+ is saved as name of the top instance.
+ <string> a null terminated ASCII string, trace clock used for events timestamps in
+ this trace instance.
+
+ HEADER_INFO: id 15, size 8
+ The HEADER_INFO option data is:
+ <8 bytes> long long unsigned integer, offset into the trace file where the HEADER INFO
+ section is located
+
+ FTRACE_EVENTS: id 16, size 8
+ The FTRACE_EVENTS option data is:
+ <8 bytes> long long unsigned integer, offset into the trace file where the
+ FTRACE EVENT FORMATS section is located.
+
+ EVENT_FORMATS: id 17, size 8
+ The EVENT_FORMATS option data is:
+ <8 bytes> long long unsigned integer, offset into the trace file where the EVENT FORMATS
+ section is located.
+
+ KALLSYMS: id 18, size 8
+ The KALLSYMS option data is:
+ <8 bytes> long long unsigned integer, offset into the trace file where the KALLSYMS
+ section is located.
+
+ PRINTK: id 19, size 8
+ The PRINTK option data is:
+ <8 bytes> long long unsigned integer, offset into the trace file where the TRACE_PRINTK
+ section is located.
+
+ CMDLINES: id 20, size 8
+ The CMDLINES option data is:
+ <8 bytes> long long unsigned integer, offset into the trace file where the
+ SAVED COMMAND LINES section is located.
+
+HEADER INFO SECTION
+-------------------
+
+ Section ID: 16
+
+ The first 12 bytes of the section, after the section header, contain the string:
+
+ "header_page\0"
+
+ The next 8 bytes are a 64-bit word containing the size of the
+ page header information stored next.
+
+ The next set of data is of the size read from the previous 8 bytes,
+ and contains the data retrieved from debugfs/tracing/events/header_page.
+
+ Note: The size of the second field \fBcommit\fR contains the target
+ kernel long size. For example:
+
+ field: local_t commit; offset:8; \fBsize:8;\fR signed:1;
+
+ shows the kernel has a 64-bit long.
+
+ The next 13 bytes contain the string:
+
+ "header_event\0"
+
+ The next 8 bytes are a 64-bit word containing the size of the
+ event header information stored next.
+
+ The next set of data is of the size read from the previous 8 bytes
+ and contains the data retrieved from debugfs/tracing/events/header_event.
+
+ This data allows the trace-cmd tool to know if the ring buffer format
+ of the kernel made any changes.
+
+FTRACE EVENT FORMATS SECTION
+----------------------------
+
+ Section ID: 17
+
+ Directly after the section header comes the information about
+ the Ftrace specific events. These are the events used by the Ftrace plugins
+ and are not enabled by the event tracing.
+
+ The next 4 bytes contain a 32-bit word of the number of Ftrace event
+ format files that are stored in the file.
+
+ For the number of times defined by the previous 4 bytes is the
+ following:
+
+ 8 bytes for the size of the Ftrace event format file.
+
+ The Ftrace event format file copied from the target machine:
+ debugfs/tracing/events/ftrace/<event>/format
+
+EVENT FORMATS SECTION
+---------------------
+
+ Section ID: 18
+
+ Directly after the section header comes the information about
+ the event layout.
+
+ The next 4 bytes are a 32-bit word containing the number of
+ event systems that are stored in the file. These are the
+ directories in debugfs/tracing/events excluding the \fBftrace\fR
+ directory.
+
+ For the number of times defined by the previous 4 bytes is the
+ following:
+
+ A null-terminated string containing the system name.
+
+ 4 bytes containing a 32-bit word containing the number
+ of events within the system.
+
+ For the number of times defined in the previous 4 bytes is the
+ following:
+
+ 8 bytes for the size of the event format file.
+
+ The event format file copied from the target machine:
+ debugfs/tracing/events/<system>/<event>/format
+
+KALLSYMS SECTION
+----------------
+
+ Section ID: 19
+
+ Directly after the section header comes the information of the mapping
+ of function addresses to the function names.
+
+ The next 4 bytes are a 32-bit word containing the size of the
+ data holding the function mappings.
+
+ The next set of data is of the size defined by the previous 4 bytes
+ and contains the information from the target machine's file:
+ /proc/kallsyms
+
+
+TRACE_PRINTK SECTION
+--------------------
+
+ Section ID: 20
+
+ If a developer used trace_printk() within the kernel, it may
+ store the format string outside the ring buffer.
+ This information can be found in:
+ debugfs/tracing/printk_formats
+
+ The next 4 bytes are a 32-bit word containing the size of the
+ data holding the printk formats.
+
+ The next set of data is of the size defined by the previous 4 bytes
+ and contains the information from debugfs/tracing/printk_formats.
+
+
+SAVED COMMAND LINES SECTION
+---------------------------
+
+ Section ID: 21
+
+ Directly after the section header comes the information mapping
+ a PID to a process name.
+
+ The next 8 bytes contain a 64-bit word that holds the size of the
+ data mapping the PID to a process name.
+
+ The next set of data is of the size defined by the previous 8 bytes
+ and contains the information from debugfs/tracing/saved_cmdlines.
+
+
+BUFFER FLYRECORD SECTION
+------------------------
+
+ This section contains flyrecord tracing data, collected in one trace instance.
+ The data is saved per CPU. Each BUFFER FLYRECORD section has a corresponding BUFFER
+ option, containing information about saved CPU's trace data. Padding is placed between
+ the section header and the CPU data, placing the CPU data at a page aligned (target page)
+ position in the file.
+
+ This data is copied directly from the Ftrace ring buffer and is of the
+ same format as the ring buffer specified by the event header files
+ loaded in the header format file.
+
+ The trace-cmd tool will try to \fBmmap(2)\fR the data page by page with the
+ target's page size if possible. If it fails to mmap, it will just read the
+ data instead.
+
+BUFFER LATENCY SECTION
+------------------------
+
+ This section contains latency tracing data, ASCII text taken from the
+ target's debugfs/tracing/trace file.
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
+trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1),
+trace-cmd-split(1), trace-cmd-list(1), trace-cmd-listen(1),
+trace-cmd.dat(5)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
--
2.31.1
^ permalink raw reply related [flat|nested] 118+ messages in thread* Re: [PATCH v2 87/87] trace-cmd: Document trace file version 7
2021-07-29 5:09 ` [PATCH v2 87/87] trace-cmd: Document trace file version 7 Tzvetomir Stoyanov (VMware)
@ 2021-08-19 19:33 ` Steven Rostedt
2021-09-02 13:07 ` Tzvetomir Stoyanov
0 siblings, 1 reply; 118+ messages in thread
From: Steven Rostedt @ 2021-08-19 19:33 UTC (permalink / raw)
To: Tzvetomir Stoyanov (VMware); +Cc: linux-trace-devel
OK, so I quickly went through the rest of the series, and this really
should have been at least 4 different patch sets. This series was
really hard to review, because it bounced around between clean ups,
sections, compressions, and conversions.
Thus, I'm going to set this entire series to "Changes Requested".
Please break this into at least 4 patch sets (here's 6 I would recommend).
Patch set 1: Clean ups.
Any change that was to make v6 work better with v7, or fix to v6 or
whatever. Should be first. It should be non-controversial, and we can
get that in right away.
Patch set 2: Sections:
This adds the notion of v7, but also adds all the work that has to do
with sections. Anything that has to do with compression, should be the
"non-compressed" case. The focus of this series would be to simply
break up the file for v7 where it is grouped by sections.
Patch set 3: Compression
This is where you can add all the changes that deal with compression.
Patch set 4: Conversion
This will add all the changes that are used to convert between v6 <-> v7.
Patch set 5: Dump
The changes here would be for dumping the output.
Patch set 6: Documentation.
And finally, add all the documentation at the end.
This will make it much easier to understand, and manage. Let's not have
a 87 patch series again. It's just too big to manage.
-- Steve
^ permalink raw reply [flat|nested] 118+ messages in thread
* Re: [PATCH v2 87/87] trace-cmd: Document trace file version 7
2021-08-19 19:33 ` Steven Rostedt
@ 2021-09-02 13:07 ` Tzvetomir Stoyanov
0 siblings, 0 replies; 118+ messages in thread
From: Tzvetomir Stoyanov @ 2021-09-02 13:07 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linux Trace Devel
On Thu, Aug 19, 2021 at 10:34 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> OK, so I quickly went through the rest of the series, and this really
> should have been at least 4 different patch sets. This series was
> really hard to review, because it bounced around between clean ups,
> sections, compressions, and conversions.
>
> Thus, I'm going to set this entire series to "Changes Requested".
>
> Please break this into at least 4 patch sets (here's 6 I would recommend).
>
> Patch set 1: Clean ups.
>
> Any change that was to make v6 work better with v7, or fix to v6 or
> whatever. Should be first. It should be non-controversial, and we can
> get that in right away.
>
>
> Patch set 2: Sections:
>
> This adds the notion of v7, but also adds all the work that has to do
> with sections. Anything that has to do with compression, should be the
> "non-compressed" case. The focus of this series would be to simply
> break up the file for v7 where it is grouped by sections.
>
> Patch set 3: Compression
>
> This is where you can add all the changes that deal with compression.
>
>
> Patch set 4: Conversion
>
> This will add all the changes that are used to convert between v6 <-> v7.
>
> Patch set 5: Dump
>
> The changes here would be for dumping the output.
>
> Patch set 6: Documentation.
>
> And finally, add all the documentation at the end.
>
> This will make it much easier to understand, and manage. Let's not have
> a 87 patch series again. It's just too big to manage.
>
> -- Steve
Thanks for reviewing this huge patch set, Steven! I'll break it into 6
patch sets, as you proposed.
--
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center
^ permalink raw reply [flat|nested] 118+ messages in thread