[PATCH 1/4] trace-cruncher: Add set_ftrace_loglevel()

[PATCH 1/4] trace-cruncher: Add set_ftrace_loglevel()

[Yordan Karadzhov (VMware)]

Signed-off-by: Yordan Karadzhov (VMware) <y.karadz@gmail.com>
---
 src/ftracepy-utils.c | 23 +++++++++++++++++++++++
 src/ftracepy-utils.h |  3 +++
 src/ftracepy.c       |  5 +++++
 3 files changed, 31 insertions(+)

diff --git a/src/ftracepy-utils.c b/src/ftracepy-utils.c
index dfb0669..8f4b50c 100644
--- a/src/ftracepy-utils.c
+++ b/src/ftracepy-utils.c
@@ -1801,6 +1801,29 @@ PyObject *PyFtrace_kprobe_is_enabled(PyObject *self, PyObject *args,
 	return event_is_enabled(instance, TC_SYS, event);
 }
 
+PyObject *PyFtrace_set_ftrace_loglevel(PyObject *self, PyObject *args,
+						       PyObject *kwargs)
+{
+	static char *kwlist[] = {"level", NULL};
+	int level;
+
+	if (!PyArg_ParseTupleAndKeywords(args,
+					 kwargs,
+					 "i",
+					 kwlist,
+					 &level)) {
+		return NULL;
+	}
+
+	if (level < 0)
+		level = 0;
+
+	tracefs_set_loglevel(level);
+	tep_set_loglevel(level);
+
+	Py_RETURN_NONE;
+}
+
 static bool set_fork_options(struct tracefs_instance *instance, bool enable)
 {
 	if (enable) {
diff --git a/src/ftracepy-utils.h b/src/ftracepy-utils.h
index d826427..5d7c19c 100644
--- a/src/ftracepy-utils.h
+++ b/src/ftracepy-utils.h
@@ -155,6 +155,9 @@ PyObject *PyFtrace_disable_kprobe(PyObject *self, PyObject *args,
 PyObject *PyFtrace_kprobe_is_enabled(PyObject *self, PyObject *args,
 						     PyObject *kwargs);
 
+PyObject *PyFtrace_set_ftrace_loglevel(PyObject *self, PyObject *args,
+						       PyObject *kwargs);
+
 PyObject *PyFtrace_trace_process(PyObject *self, PyObject *args,
 						 PyObject *kwargs);
 
diff --git a/src/ftracepy.c b/src/ftracepy.c
index e5fcd54..e3fec7b 100644
--- a/src/ftracepy.c
+++ b/src/ftracepy.c
@@ -269,6 +269,11 @@ static PyMethodDef ftracepy_methods[] = {
 	 METH_VARARGS | METH_KEYWORDS,
 	 "Check if kprobe event is enabled."
 	},
+	{"set_ftrace_loglevel",
+	 (PyCFunction) PyFtrace_set_ftrace_loglevel,
+	 METH_VARARGS | METH_KEYWORDS,
+	 "Set the verbose level of the ftrace libraries."
+	},
 	{"trace_process",
 	 (PyCFunction) PyFtrace_trace_process,
 	 METH_VARARGS | METH_KEYWORDS,
-- 
2.27.0

[PATCH 2/4] trace-cruncher: Add local_tep() to utils

[Yordan Karadzhov (VMware)]

The function can be used to retrieve the "tep" event of the
current (local) system.

Signed-off-by: Yordan Karadzhov (VMware) <y.karadz@gmail.com>
---
 tracecruncher/ft_utils.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tracecruncher/ft_utils.py b/tracecruncher/ft_utils.py
index 5b4f2f2..8c245b1 100644
--- a/tracecruncher/ft_utils.py
+++ b/tracecruncher/ft_utils.py
@@ -11,6 +11,15 @@ import ctypes
 from . import ftracepy as ft
 
 
+def local_tep():
+    """ Get the "tep" event of the current system (local).
+    """
+    tep = ft.tep_handle();
+    tep.init_local(dir=ft.dir());
+
+    return tep
+
+
 def find_event_id(system, event):
     """ Get the unique identifier of a trace event.
     """
-- 
2.27.0

[PATCH 3/4] trace-cruncher: Allow for detachable instances

[Yordan Karadzhov (VMware)]

 If the instance is created with "detached=True", the Python module is
 no longer responsible for destroying it when exiting. It is therefore
 up to the user free it explicitly, or to keep it active.

Signed-off-by: Yordan Karadzhov (VMware) <y.karadz@gmail.com>
---
 src/ftracepy-utils.c | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/src/ftracepy-utils.c b/src/ftracepy-utils.c
index 8f4b50c..d061817 100644
--- a/src/ftracepy-utils.c
+++ b/src/ftracepy-utils.c
@@ -421,6 +421,12 @@ static PyObject *tfs_list2py_list(char **list)
 
 struct instance_wrapper {
 	struct tracefs_instance *ptr;
+	bool detached;
+
+	/*
+	 * This name will be used only for searching. The actual name of
+	 * the instance is owned by the "tracefs_instance" object.
+	 */
 	const char *name;
 };
 
@@ -451,10 +457,12 @@ void instance_wrapper_free(void *ptr)
 
 	iw = ptr;
 	if (iw->ptr) {
-		if (tracefs_instance_destroy(iw->ptr) < 0)
-			fprintf(stderr,
-				"\ntfs_error: Failed to destroy instance '%s'.\n",
-				get_instance_name(iw->ptr));
+		if (!iw->detached) {
+			if (tracefs_instance_destroy(iw->ptr) < 0)
+				fprintf(stderr,
+					"\ntfs_error: Failed to destroy instance '%s'.\n",
+					get_instance_name(iw->ptr));
+		}
 
 		free(iw->ptr);
 	}
@@ -569,14 +577,16 @@ PyObject *PyFtrace_create_instance(PyObject *self, PyObject *args,
 	struct tracefs_instance *instance;
 	const char *name = NO_ARG;
 	int tracing_on = true;
+	int detached = false;
+	static char *kwlist[] = {"name", "tracing_on", "detached", NULL};
 
-	static char *kwlist[] = {"name", "tracing_on", NULL};
 	if (!PyArg_ParseTupleAndKeywords(args,
 					 kwargs,
-					 "|sp",
+					 "|spp",
 					 kwlist,
 					 &name,
-					 &tracing_on)) {
+					 &tracing_on,
+					 &detached)) {
 		return NULL;
 	}
 
@@ -600,6 +610,11 @@ PyObject *PyFtrace_create_instance(PyObject *self, PyObject *args,
 	}
 
 	iw->ptr = instance;
+	if (detached) {
+		printf("detached instance: %s\n", name);
+		iw->detached = detached;
+	}
+
 	iw_ptr = tsearch(iw, &instance_root, instance_compare);
 	if (!iw_ptr || !(*iw_ptr) || !(*iw_ptr)->ptr ||
 	    strcmp(tracefs_instance_get_name((*iw_ptr)->ptr), name) != 0) {
-- 
2.27.0

[PATCH 4/4] trace-cruncher: Allow for detachable kprobes

[Yordan Karadzhov (VMware)]

If a kprobe or kretprobe is created with "detached=True", the Python
module is no longer responsible for destroying it when exiting. It is
therefore up to the user free it explicitly, or to keep it active.

Signed-off-by: Yordan Karadzhov (VMware) <y.karadz@gmail.com>
---
 src/ftracepy-utils.c                  | 26 ++++++++++++++++----------
 tests/1_unit/test_01_ftracepy_unit.py |  9 +++++++++
 tracecruncher/ft_utils.py             |  8 ++++----
 3 files changed, 29 insertions(+), 14 deletions(-)

diff --git a/src/ftracepy-utils.c b/src/ftracepy-utils.c
index d061817..605f7eb 100644
--- a/src/ftracepy-utils.c
+++ b/src/ftracepy-utils.c
@@ -1570,16 +1570,18 @@ bool store_new_kprobe(const char *event)
 PyObject *PyFtrace_register_kprobe(PyObject *self, PyObject *args,
 						   PyObject *kwargs)
 {
-	static char *kwlist[] = {"event", "function", "probe", NULL};
+	static char *kwlist[] = {"event", "function", "probe", "detached", NULL};
 	const char *event, *function, *probe;
+	int detached = false;
 
 	if (!PyArg_ParseTupleAndKeywords(args,
 					 kwargs,
-					 "sss",
+					 "sss|p",
 					 kwlist,
 					 &event,
 					 &function,
-					 &probe)) {
+					 &probe,
+					 &detached)) {
 		return NULL;
 	}
 
@@ -1589,8 +1591,9 @@ PyObject *PyFtrace_register_kprobe(PyObject *self, PyObject *args,
 		return NULL;
 	}
 
-	if (!store_new_kprobe(event))
-		return NULL;
+	if (!detached)
+		if (!store_new_kprobe(event))
+			return NULL;
 
 	Py_RETURN_NONE;
 }
@@ -1598,16 +1601,18 @@ PyObject *PyFtrace_register_kprobe(PyObject *self, PyObject *args,
 PyObject *PyFtrace_register_kretprobe(PyObject *self, PyObject *args,
 						      PyObject *kwargs)
 {
-	static char *kwlist[] = {"event", "function", "probe", NULL};
+	static char *kwlist[] = {"event", "function", "probe", "detached", NULL};
 	const char *event, *function, *probe = "$retval";
+	int detached = false;
 
 	if (!PyArg_ParseTupleAndKeywords(args,
 					 kwargs,
-					 "ss|s",
+					 "ss|sp",
 					 kwlist,
 					 &event,
 					 &function,
-					 &probe)) {
+					 &probe,
+					 &detached)) {
 		return NULL;
 	}
 
@@ -1617,8 +1622,9 @@ PyObject *PyFtrace_register_kretprobe(PyObject *self, PyObject *args,
 		return NULL;
 	}
 
-	if (!store_new_kprobe(event))
-		return NULL;
+	if (!detached)
+		if (!store_new_kprobe(event))
+			return NULL;
 
 	Py_RETURN_NONE;
 }
diff --git a/tests/1_unit/test_01_ftracepy_unit.py b/tests/1_unit/test_01_ftracepy_unit.py
index 0d62da2..a7a6dec 100644
--- a/tests/1_unit/test_01_ftracepy_unit.py
+++ b/tests/1_unit/test_01_ftracepy_unit.py
@@ -479,6 +479,15 @@ class KprobeTestCase(unittest.TestCase):
         all_kprobes = ft.registered_kprobes()
         self.assertEqual(len(all_kprobes), 0)
 
+        ft.register_kprobe(event=evt1, function=evt1_func,
+                           probe=evt1_prove, detached=True)
+        ft.unregister_kprobe(event='ALL')
+        all_kprobes = ft.registered_kprobes()
+        self.assertEqual(len(all_kprobes), 1)
+        ft.unregister_kprobe(event='ALL', force=True)
+        all_kprobes = ft.registered_kprobes()
+        self.assertEqual(len(all_kprobes), 0)
+
 
     def test_enable_kprobe(self):
         evt1 = 'mkdir'
diff --git a/tracecruncher/ft_utils.py b/tracecruncher/ft_utils.py
index 8c245b1..a976885 100644
--- a/tracecruncher/ft_utils.py
+++ b/tracecruncher/ft_utils.py
@@ -171,12 +171,12 @@ class kprobe(kprobe_base):
                            offset=offset,
                            size=size)
 
-    def register(self):
+    def register(self, detached=False):
         """ Register this probe to Ftrace.
         """
         probe = ' '.join('{!s}={!s}'.format(key,val) for (key, val) in self.fields.items())
 
-        ft.register_kprobe(event=self.name, function=self.func, probe=probe);
+        ft.register_kprobe(event=self.name, function=self.func, probe=probe, detached=detached);
         self.evt_id = find_event_id(system=ft.tc_event_system(), event=self.name)
 
 
@@ -203,8 +203,8 @@ class kretval_probe(kprobe_base):
         """
         super().__init__(name, func)
 
-    def register(self):
+    def register(self, detached=False):
         """ Register this probe to Ftrace.
         """
-        ft.register_kprobe(event=self.name, function=self.func);
+        ft.register_kprobe(event=self.name, function=self.func, detached=detached);
         self.evt_id = find_event_id(system=ft.tc_event_system(), event=self.name)
-- 
2.27.0

Re: [PATCH 1/4] trace-cruncher: Add set_ftrace_loglevel()

[Steven Rostedt]

On Mon, 12 Jul 2021 15:32:39 +0300
"Yordan Karadzhov (VMware)" <y.karadz@gmail.com> wrote:

-ENOCOMMITMESSAGE

Please describe what you are doing and why. The subject is not enough.

Even on simple changes, there should be some commentary.

The rest looks fine.

-- Steve


> Signed-off-by: Yordan Karadzhov (VMware) <y.karadz@gmail.com>
> ---
>  src/ftracepy-utils.c | 23 +++++++++++++++++++++++
>  src/ftracepy-utils.h |  3 +++
>  src/ftracepy.c       |  5 +++++
>  3 files changed, 31 insertions(+)
> 
> diff --git a/src/ftracepy-utils.c b/src/ftracepy-utils.c
> index dfb0669..8f4b50c 100644
> --- a/src/ftracepy-utils.c
> +++ b/src/ftracepy-utils.c
> @@ -1801,6 +1801,29 @@ PyObject *PyFtrace_kprobe_is_enabled(PyObject *self, PyObject *args,
>  	return event_is_enabled(instance, TC_SYS, event);
>  }
>  
> +PyObject *PyFtrace_set_ftrace_loglevel(PyObject *self, PyObject *args,
> +						       PyObject *kwargs)
> +{
> +	static char *kwlist[] = {"level", NULL};
> +	int level;
> +
> +	if (!PyArg_ParseTupleAndKeywords(args,
> +					 kwargs,
> +					 "i",
> +					 kwlist,
> +					 &level)) {
> +		return NULL;
> +	}
> +
> +	if (level < 0)
> +		level = 0;
> +
> +	tracefs_set_loglevel(level);
> +	tep_set_loglevel(level);
> +
> +	Py_RETURN_NONE;
> +}
> +
>  static bool set_fork_options(struct tracefs_instance *instance, bool enable)
>  {
>  	if (enable) {
> diff --git a/src/ftracepy-utils.h b/src/ftracepy-utils.h
> index d826427..5d7c19c 100644
> --- a/src/ftracepy-utils.h
> +++ b/src/ftracepy-utils.h
> @@ -155,6 +155,9 @@ PyObject *PyFtrace_disable_kprobe(PyObject *self, PyObject *args,
>  PyObject *PyFtrace_kprobe_is_enabled(PyObject *self, PyObject *args,
>  						     PyObject *kwargs);
>  
> +PyObject *PyFtrace_set_ftrace_loglevel(PyObject *self, PyObject *args,
> +						       PyObject *kwargs);
> +
>  PyObject *PyFtrace_trace_process(PyObject *self, PyObject *args,
>  						 PyObject *kwargs);
>  
> diff --git a/src/ftracepy.c b/src/ftracepy.c
> index e5fcd54..e3fec7b 100644
> --- a/src/ftracepy.c
> +++ b/src/ftracepy.c
> @@ -269,6 +269,11 @@ static PyMethodDef ftracepy_methods[] = {
>  	 METH_VARARGS | METH_KEYWORDS,
>  	 "Check if kprobe event is enabled."
>  	},
> +	{"set_ftrace_loglevel",
> +	 (PyCFunction) PyFtrace_set_ftrace_loglevel,
> +	 METH_VARARGS | METH_KEYWORDS,
> +	 "Set the verbose level of the ftrace libraries."
> +	},
>  	{"trace_process",
>  	 (PyCFunction) PyFtrace_trace_process,
>  	 METH_VARARGS | METH_KEYWORDS,

Re: [PATCH 3/4] trace-cruncher: Allow for detachable instances

[Steven Rostedt]

On Mon, 12 Jul 2021 15:32:41 +0300
"Yordan Karadzhov (VMware)" <y.karadz@gmail.com> wrote:

> @@ -600,6 +610,11 @@ PyObject *PyFtrace_create_instance(PyObject *self, PyObject *args,
>  	}
>  
>  	iw->ptr = instance;
> +	if (detached) {
> +		printf("detached instance: %s\n", name);

Do we really need to print this? I would think this is up to the
calling function to print this or not.

-- Steve


> +		iw->detached = detached;
> +	}
> +

Re: [PATCH 4/4] trace-cruncher: Allow for detachable kprobes

[Steven Rostedt]

On Mon, 12 Jul 2021 15:32:42 +0300
"Yordan Karadzhov (VMware)" <y.karadz@gmail.com> wrote:


> @@ -1617,8 +1622,9 @@ PyObject *PyFtrace_register_kretprobe(PyObject *self, PyObject *args,
>  		return NULL;
>  	}
>  
> -	if (!store_new_kprobe(event))
> -		return NULL;
> +	if (!detached)
> +		if (!store_new_kprobe(event))
> +			return NULL;

I wonder if it would be more consistent and helpful if we created a
structure for kprobes like we do for instances, and this way we can
save the "detached" field in that structure, and not free it on destroy.

This way, even though the kprobes are detached it may still be needed
in the future to list all kprobes that were created by the application,
regardless if they are detached or not.

-- Steve

Re: [PATCH 3/4] trace-cruncher: Allow for detachable instances

[Yordan Karadzhov (VMware)]

On 23.07.21 г. 0:19, Steven Rostedt wrote:
> On Mon, 12 Jul 2021 15:32:41 +0300
> "Yordan Karadzhov (VMware)" <y.karadz@gmail.com> wrote:
> 
>> @@ -600,6 +610,11 @@ PyObject *PyFtrace_create_instance(PyObject *self, PyObject *args,
>>   	}
>>   
>>   	iw->ptr = instance;
>> +	if (detached) {
>> +		printf("detached instance: %s\n", name);
> 
> Do we really need to print this? I would think this is up to the
> calling function to print this or not.
>

I think there must be some notice that it is up to the user to take care 
about this new instance. This essentially detaches the instance from the 
garbage collection, which is quite something in the Python world.

Maybe we can change the message with something more appropriate?

Thanks!
Yordan

> -- Steve
> 
> 
>> +		iw->detached = detached;
>> +	}
>> +

Re: [PATCH 1/4] trace-cruncher: Add set_ftrace_loglevel()

[Yordan Karadzhov (VMware)]

On 23.07.21 г. 0:15, Steven Rostedt wrote:
> On Mon, 12 Jul 2021 15:32:39 +0300
> "Yordan Karadzhov (VMware)" <y.karadz@gmail.com> wrote:
> 
> -ENOCOMMITMESSAGE
> 
> Please describe what you are doing and why. The subject is not enough.
> 
> Even on simple changes, there should be some commentary.

There was a message for this commit. I guess I somehow deleted it with 
the last rebase. Will fix this in v2.

Thanks!
Yordan

> The rest looks fine.
> 
> -- Steve
> 
> 
>> Signed-off-by: Yordan Karadzhov (VMware) <y.karadz@gmail.com>
>> ---
>>   src/ftracepy-utils.c | 23 +++++++++++++++++++++++
>>   src/ftracepy-utils.h |  3 +++
>>   src/ftracepy.c       |  5 +++++
>>   3 files changed, 31 insertions(+)
>>
>> diff --git a/src/ftracepy-utils.c b/src/ftracepy-utils.c
>> index dfb0669..8f4b50c 100644
>> --- a/src/ftracepy-utils.c
>> +++ b/src/ftracepy-utils.c
>> @@ -1801,6 +1801,29 @@ PyObject *PyFtrace_kprobe_is_enabled(PyObject *self, PyObject *args,
>>   	return event_is_enabled(instance, TC_SYS, event);
>>   }
>>   
>> +PyObject *PyFtrace_set_ftrace_loglevel(PyObject *self, PyObject *args,
>> +						       PyObject *kwargs)
>> +{
>> +	static char *kwlist[] = {"level", NULL};
>> +	int level;
>> +
>> +	if (!PyArg_ParseTupleAndKeywords(args,
>> +					 kwargs,
>> +					 "i",
>> +					 kwlist,
>> +					 &level)) {
>> +		return NULL;
>> +	}
>> +
>> +	if (level < 0)
>> +		level = 0;
>> +
>> +	tracefs_set_loglevel(level);
>> +	tep_set_loglevel(level);
>> +
>> +	Py_RETURN_NONE;
>> +}
>> +
>>   static bool set_fork_options(struct tracefs_instance *instance, bool enable)
>>   {
>>   	if (enable) {
>> diff --git a/src/ftracepy-utils.h b/src/ftracepy-utils.h
>> index d826427..5d7c19c 100644
>> --- a/src/ftracepy-utils.h
>> +++ b/src/ftracepy-utils.h
>> @@ -155,6 +155,9 @@ PyObject *PyFtrace_disable_kprobe(PyObject *self, PyObject *args,
>>   PyObject *PyFtrace_kprobe_is_enabled(PyObject *self, PyObject *args,
>>   						     PyObject *kwargs);
>>   
>> +PyObject *PyFtrace_set_ftrace_loglevel(PyObject *self, PyObject *args,
>> +						       PyObject *kwargs);
>> +
>>   PyObject *PyFtrace_trace_process(PyObject *self, PyObject *args,
>>   						 PyObject *kwargs);
>>   
>> diff --git a/src/ftracepy.c b/src/ftracepy.c
>> index e5fcd54..e3fec7b 100644
>> --- a/src/ftracepy.c
>> +++ b/src/ftracepy.c
>> @@ -269,6 +269,11 @@ static PyMethodDef ftracepy_methods[] = {
>>   	 METH_VARARGS | METH_KEYWORDS,
>>   	 "Check if kprobe event is enabled."
>>   	},
>> +	{"set_ftrace_loglevel",
>> +	 (PyCFunction) PyFtrace_set_ftrace_loglevel,
>> +	 METH_VARARGS | METH_KEYWORDS,
>> +	 "Set the verbose level of the ftrace libraries."
>> +	},
>>   	{"trace_process",
>>   	 (PyCFunction) PyFtrace_trace_process,
>>   	 METH_VARARGS | METH_KEYWORDS,
> 

Re: [PATCH 3/4] trace-cruncher: Allow for detachable instances

[Steven Rostedt]

On Mon, 26 Jul 2021 12:04:51 +0300
"Yordan Karadzhov (VMware)" <y.karadz@gmail.com> wrote:

> I think there must be some notice that it is up to the user to take care 
> about this new instance. This essentially detaches the instance from the 
> garbage collection, which is quite something in the Python world.
> 
> Maybe we can change the message with something more appropriate?

Can't the documentation on the use case be sufficient?

I mean, if you have a script that does this, and you give this script
to your customers, it would be annoying to have this message appear to
your customers where they have no idea what it means.

If anything, it should only be printed if you enable some sort of
"debug level", like we started doing with the trace libraries.

-- Steve

Re: [PATCH 4/4] trace-cruncher: Allow for detachable kprobes

[Yordan Karadzhov]

On 26.07.21 г. 12:07, Yordan Karadzhov (VMware) wrote:
> 
> 
> On 23.07.21 г. 0:24, Steven Rostedt wrote:
>> On Mon, 12 Jul 2021 15:32:42 +0300
>> "Yordan Karadzhov (VMware)" <y.karadz@gmail.com> wrote:
>>
>>
>>> @@ -1617,8 +1622,9 @@ PyObject *PyFtrace_register_kretprobe(PyObject *self, PyObject *args,
>>>           return NULL;
>>>       }
>>> -    if (!store_new_kprobe(event))
>>> -        return NULL;
>>> +    if (!detached)
>>> +        if (!store_new_kprobe(event))
>>> +            return NULL;
>>
>> I wonder if it would be more consistent and helpful if we created a
>> structure for kprobes like we do for instances, and this way we can
>> save the "detached" field in that structure, and not free it on destroy.
>>
>> This way, even though the kprobes are detached it may still be needed
>> in the future to list all kprobes that were created by the application,
>> regardless if they are detached or not.
> 
> Yes, this make sense.
> 
> BTW we should discuss what is the best data structure to store the kprobes that were created. In the case of the 
> instances we really need to use a binary tree, because we identify the instances only by name (string) so we are forced 
> to do a lot of searches. Initially I assumed that we will need to do searches for kprobes as well. However, so far we 
> only do listing.
> 
> Maybe, we should switch from tree to linked list? This will greatly simplify everything.
> 

I am deferring this patch together with the previous one ("Allow for detachable instances") for possible redesign.

Thanks,
Yordan

> Thanks!
> Yordan
> 



> 
>>
>> -- Steve
>>