[PATCH v5 0/7] ndctl: cxl: add monitor support for trace events

[PATCH v5 0/7] ndctl: cxl: add monitor support for trace events

[Dave Jiang]

This patch series for ndctl implements the monitor command for the cxl tool.
The initial implementation will collect CXL trace events emitted by the
kernel. libtraceevent and libtracefs will be used to parse the trace
event buffer. The monitor will pend on an epoll fd and wait for new event
entries to be posted. The output will be in json format. By default the events
are emitted to stdio, but can also be logged to a file. Each event is converted
to a JSON object and logged as such. All the fields exported are read by the
monitor code and added to the JSON object.

v5:
- Remove unneeded header includes. (Vishal)
- Reformat using git clang-format. (Vishal)
- Change uuid string to 40 bytes allocation. (Vishal)
- Change cxl_event_to_json_callback() to cxl_event_to_json(). (Vishal)
- Change cxl_event_parse_cb() to cxl_event_parse(). (Vishal)
- Squash patch 3 & 4. (Vishal)
- Move common logging helper to util/log.c. (Vishal)
- Squash patch 5 & 7. (Vishal)
- Drop Alison's pid patch. (Vishal)
- Fixup man page. (Vishal)

v4:
- Fix num_to_json for less than int size (Ira)
- Use TEP_FIELD flags to determine data type. (Steve)
- Use tracefs_event_enable() instad of directly toggle sysfs via lib calls (Steve)
- Remove tracefs_trace_is_on() in disable() call (Steve)
- Update to use tep_data_pid() (Steve)

v3:
- Change uuid parsing from u8[] to uuid_t (Alison)
- Add event_name to event_ctx for filtering (Alison)
- Add event_pid to event_ctx for filtering (Alison)
- Add parse_event callback to event_ctx for filtering (Alison)

v2:
- Simplify logging functions (Nathan)
- Drop ndctl prefix (Vishal)
- Reduce to single trace event system (Alison)
- Add systemd startup file
- Add man page

---

Dave Jiang (7):
      ndctl: cxl: add helper function to parse trace event to json object
      ndctl: cxl: add helper to parse through all current events
      ndctl: cxl: add common function to enable/disable event trace
      ndctl: move common logging functions from ndctl/monitor.c to util/log.c
      ndctl: cxl: add monitor command for event trace events
      ndctl: cxl: add systemd service for monitor
      ndctl: cxl: add man page documentation for monitor


 Documentation/cxl/cxl-monitor.txt |  62 ++++++++
 Documentation/cxl/meson.build     |   1 +
 cxl/builtin.h                     |   1 +
 cxl/cxl-monitor.service           |   9 ++
 cxl/cxl.c                         |   1 +
 cxl/event_trace.c                 | 251 ++++++++++++++++++++++++++++++
 cxl/event_trace.h                 |  27 ++++
 cxl/meson.build                   |   8 +
 cxl/monitor.c                     | 215 +++++++++++++++++++++++++
 meson.build                       |   3 +
 ndctl.spec.in                     |   1 +
 ndctl/monitor.c                   |  41 +----
 util/log.c                        |  34 ++++
 util/log.h                        |   8 +-
 14 files changed, 624 insertions(+), 38 deletions(-)
 create mode 100644 Documentation/cxl/cxl-monitor.txt
 create mode 100644 cxl/cxl-monitor.service
 create mode 100644 cxl/event_trace.c
 create mode 100644 cxl/event_trace.h
 create mode 100644 cxl/monitor.c

--

[PATCH v5 1/7] ndctl: cxl: add helper function to parse trace event to json object

[Dave Jiang]

Add the helper function that parses a trace event captured by
libtraceevent in a tep handle. All the parsed fields are added to a json
object. The json object is added to the provided list in the input parameter.

Tested-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 cxl/event_trace.c |  193 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 cxl/event_trace.h |   14 ++++
 cxl/meson.build   |    2 +
 meson.build       |    1 
 4 files changed, 210 insertions(+)
 create mode 100644 cxl/event_trace.c
 create mode 100644 cxl/event_trace.h

diff --git a/cxl/event_trace.c b/cxl/event_trace.c
new file mode 100644
index 000000000000..3c9fb684139a
--- /dev/null
+++ b/cxl/event_trace.c
@@ -0,0 +1,193 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2022, Intel Corp. All rights reserved.
+#include <stdio.h>
+#include <errno.h>
+#include <json-c/json.h>
+#include <util/json.h>
+#include <util/util.h>
+#include <util/strbuf.h>
+#include <ccan/list/list.h>
+#include <uuid/uuid.h>
+#include <traceevent/event-parse.h>
+#include "event_trace.h"
+
+#define _GNU_SOURCE
+#include <string.h>
+
+static struct json_object *num_to_json(void *num, int elem_size, unsigned long flags)
+{
+	bool sign = flags & TEP_FIELD_IS_SIGNED;
+	int64_t val = 0;
+
+	/* special case 64 bit as the call depends on sign */
+	if (elem_size == 8) {
+		if (sign)
+			return json_object_new_int64(*(int64_t *)num);
+		else
+			return json_object_new_uint64(*(uint64_t *)num);
+	}
+
+	/* All others fit in a signed 64 bit */
+	switch (elem_size) {
+	case 1:
+		if (sign)
+			val = *(int8_t *)num;
+		else
+			val = *(uint8_t *)num;
+		break;
+	case 2:
+		if (sign)
+			val = *(int16_t *)num;
+		else
+			val = *(uint16_t *)num;
+		break;
+	case 4:
+		if (sign)
+			val = *(int32_t *)num;
+		else
+			val = *(uint32_t *)num;
+		break;
+	default:
+		/*
+		 * Odd sizes are converted in the kernel to one of the above.
+		 * It is an error to see them here.
+		 */
+		return NULL;
+	}
+
+	return json_object_new_int64(val);
+}
+
+static int cxl_event_to_json(struct tep_event *event, struct tep_record *record,
+			     struct list_head *jlist_head)
+{
+	struct json_object *jevent, *jobj, *jarray;
+	struct tep_format_field **fields;
+	struct jlist_node *jnode;
+	int i, j, rc = 0;
+
+	jnode = malloc(sizeof(*jnode));
+	if (!jnode)
+		return -ENOMEM;
+
+	jevent = json_object_new_object();
+	if (!jevent) {
+		rc = -ENOMEM;
+		goto err_jnode;
+	}
+	jnode->jobj = jevent;
+
+	fields = tep_event_fields(event);
+	if (!fields) {
+		rc = -ENOENT;
+		goto err_jevent;
+	}
+
+	jobj = json_object_new_string(event->system);
+	if (!jobj) {
+		rc = -ENOMEM;
+		goto err_jevent;
+	}
+	json_object_object_add(jevent, "system", jobj);
+
+	jobj = json_object_new_string(event->name);
+	if (!jobj) {
+		rc = -ENOMEM;
+		goto err_jevent;
+	}
+	json_object_object_add(jevent, "event", jobj);
+
+	jobj = json_object_new_uint64(record->ts);
+	if (!jobj) {
+		rc = -ENOMEM;
+		goto err_jevent;
+	}
+	json_object_object_add(jevent, "timestamp", jobj);
+
+	for (i = 0; fields[i]; i++) {
+		struct tep_format_field *f = fields[i];
+		int len;
+
+		if (f->flags & TEP_FIELD_IS_STRING) {
+			char *str;
+
+			str = tep_get_field_raw(NULL, event, f->name, record, &len, 0);
+			if (!str)
+				continue;
+
+			jobj = json_object_new_string(str);
+			if (!jobj) {
+				rc = -ENOMEM;
+				goto err_jevent;
+			}
+
+			json_object_object_add(jevent, f->name, jobj);
+		} else if (f->flags & TEP_FIELD_IS_ARRAY) {
+			unsigned char *data;
+			int chunks;
+
+			data = tep_get_field_raw(NULL, event, f->name, record, &len, 0);
+			if (!data)
+				continue;
+
+			jarray = json_object_new_array();
+			if (!jarray) {
+				rc = -ENOMEM;
+				goto err_jevent;
+			}
+
+			chunks = f->size / f->elementsize;
+			for (j = 0; j < chunks; j++) {
+				jobj = num_to_json(data, f->elementsize, f->flags);
+				if (!jobj) {
+					json_object_put(jarray);
+					return -ENOMEM;
+				}
+				json_object_array_add(jarray, jobj);
+				data += f->elementsize;
+			}
+
+			json_object_object_add(jevent, f->name, jarray);
+		} else { /* single number */
+			unsigned char *data;
+			char *tmp;
+
+			data = tep_get_field_raw(NULL, event, f->name, record, &len, 0);
+			if (!data)
+				continue;
+
+			/* check to see if we have a UUID */
+			tmp = strcasestr(f->type, "uuid_t");
+			if (tmp) {
+				char uuid[40];
+
+				uuid_unparse(data, uuid);
+				jobj = json_object_new_string(uuid);
+				if (!jobj) {
+					rc = -ENOMEM;
+					goto err_jevent;
+				}
+
+				json_object_object_add(jevent, f->name, jobj);
+				continue;
+			}
+
+			jobj = num_to_json(data, f->elementsize, f->flags);
+			if (!jobj) {
+				rc = -ENOMEM;
+				goto err_jevent;
+			}
+
+			json_object_object_add(jevent, f->name, jobj);
+		}
+	}
+
+	list_add_tail(jlist_head, &jnode->list);
+	return 0;
+
+err_jevent:
+	json_object_put(jevent);
+err_jnode:
+	free(jnode);
+	return rc;
+}
diff --git a/cxl/event_trace.h b/cxl/event_trace.h
new file mode 100644
index 000000000000..00975a0b5680
--- /dev/null
+++ b/cxl/event_trace.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2022 Intel Corporation. All rights reserved. */
+#ifndef __CXL_EVENT_TRACE_H__
+#define __CXL_EVENT_TRACE_H__
+
+#include <json-c/json.h>
+#include <ccan/list/list.h>
+
+struct jlist_node {
+	struct json_object *jobj;
+	struct list_node list;
+};
+
+#endif
diff --git a/cxl/meson.build b/cxl/meson.build
index f2474aaa6e2e..8c7733431613 100644
--- a/cxl/meson.build
+++ b/cxl/meson.build
@@ -7,6 +7,7 @@ cxl_src = [
   'memdev.c',
   'json.c',
   'filter.c',
+  'event_trace.c',
 ]
 
 cxl_tool = executable('cxl',
@@ -19,6 +20,7 @@ cxl_tool = executable('cxl',
     kmod,
     json,
     versiondep,
+    traceevent,
   ],
   install : true,
   install_dir : rootbindir,
diff --git a/meson.build b/meson.build
index 20a646d135c7..f611e0bdd7f3 100644
--- a/meson.build
+++ b/meson.build
@@ -142,6 +142,7 @@ kmod = dependency('libkmod')
 libudev = dependency('libudev')
 uuid = dependency('uuid')
 json = dependency('json-c')
+traceevent = dependency('libtraceevent')
 if get_option('docs').enabled()
   if get_option('asciidoctor').enabled()
     asciidoc = find_program('asciidoctor', required : true)

[PATCH v5 2/7] ndctl: cxl: add helper to parse through all current events

[Dave Jiang]

Add common function to iterate through and extract the events in the
current trace buffer. The function uses tracefs_iterate_raw_events() from
libtracefs to go through all the events loaded into a tep_handle. A
callback is provided to the API call in order to parse the event. For cxl
monitor, the "system" or category of trace event, in this case "cxl",
is provided in order to filter for the CXL events.

Tested-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 cxl/event_trace.c |   37 +++++++++++++++++++++++++++++++++++++
 cxl/event_trace.h |   10 ++++++++++
 cxl/meson.build   |    1 +
 meson.build       |    2 ++
 4 files changed, 50 insertions(+)

diff --git a/cxl/event_trace.c b/cxl/event_trace.c
index 3c9fb684139a..d7bbd3cf4946 100644
--- a/cxl/event_trace.c
+++ b/cxl/event_trace.c
@@ -9,6 +9,7 @@
 #include <ccan/list/list.h>
 #include <uuid/uuid.h>
 #include <traceevent/event-parse.h>
+#include <tracefs/tracefs.h>
 #include "event_trace.h"
 
 #define _GNU_SOURCE
@@ -191,3 +192,39 @@ err_jnode:
 	free(jnode);
 	return rc;
 }
+
+static int cxl_event_parse(struct tep_event *event, struct tep_record *record,
+			   int cpu, void *ctx)
+{
+	struct event_ctx *event_ctx = (struct event_ctx *)ctx;
+
+	/* Filter out all the events that the caller isn't interested in. */
+	if (strcmp(event->system, event_ctx->system) != 0)
+		return 0;
+
+	if (event_ctx->event_name) {
+		if (strcmp(event->name, event_ctx->event_name) != 0)
+			return 0;
+	}
+
+	if (event_ctx->parse_event)
+		return event_ctx->parse_event(event, record,
+					      &event_ctx->jlist_head);
+
+	return cxl_event_to_json(event, record, &event_ctx->jlist_head);
+}
+
+int cxl_parse_events(struct tracefs_instance *inst, struct event_ctx *ectx)
+{
+	struct tep_handle *tep;
+	int rc;
+
+	tep = tracefs_local_events(NULL);
+	if (!tep)
+		return -ENOMEM;
+
+	rc = tracefs_iterate_raw_events(tep, inst, NULL, 0, cxl_event_parse,
+					ectx);
+	tep_free(tep);
+	return rc;
+}
diff --git a/cxl/event_trace.h b/cxl/event_trace.h
index 00975a0b5680..e83737de0ad5 100644
--- a/cxl/event_trace.h
+++ b/cxl/event_trace.h
@@ -11,4 +11,14 @@ struct jlist_node {
 	struct list_node list;
 };
 
+struct event_ctx {
+	const char *system;
+	struct list_head jlist_head;
+	const char *event_name; /* optional */
+	int (*parse_event)(struct tep_event *event, struct tep_record *record,
+			   struct list_head *jlist_head); /* optional */
+};
+
+int cxl_parse_events(struct tracefs_instance *inst, struct event_ctx *ectx);
+
 #endif
diff --git a/cxl/meson.build b/cxl/meson.build
index 8c7733431613..c59876262e76 100644
--- a/cxl/meson.build
+++ b/cxl/meson.build
@@ -21,6 +21,7 @@ cxl_tool = executable('cxl',
     json,
     versiondep,
     traceevent,
+    tracefs,
   ],
   install : true,
   install_dir : rootbindir,
diff --git a/meson.build b/meson.build
index f611e0bdd7f3..c204c8ac52de 100644
--- a/meson.build
+++ b/meson.build
@@ -143,6 +143,8 @@ libudev = dependency('libudev')
 uuid = dependency('uuid')
 json = dependency('json-c')
 traceevent = dependency('libtraceevent')
+tracefs = dependency('libtracefs')
+
 if get_option('docs').enabled()
   if get_option('asciidoctor').enabled()
     asciidoc = find_program('asciidoctor', required : true)

[PATCH v5 3/7] ndctl: cxl: add common function to enable/disable event trace

[Dave Jiang]

Add a common function for cxl command to enable and disable event
tracing for the instance created. Will only enable "cxl" system.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 cxl/event_trace.c |   21 +++++++++++++++++++++
 cxl/event_trace.h |    3 +++
 2 files changed, 24 insertions(+)

diff --git a/cxl/event_trace.c b/cxl/event_trace.c
index d7bbd3cf4946..a973a1f62d35 100644
--- a/cxl/event_trace.c
+++ b/cxl/event_trace.c
@@ -228,3 +228,24 @@ int cxl_parse_events(struct tracefs_instance *inst, struct event_ctx *ectx)
 	tep_free(tep);
 	return rc;
 }
+
+int cxl_event_tracing_enable(struct tracefs_instance *inst, const char *system,
+		const char *event)
+{
+	int rc;
+
+	rc = tracefs_event_enable(inst, system, event);
+	if (rc == -1)
+		return -errno;
+
+	if (tracefs_trace_is_on(inst))
+		return 0;
+
+	tracefs_trace_on(inst);
+	return 0;
+}
+
+int cxl_event_tracing_disable(struct tracefs_instance *inst)
+{
+	return tracefs_trace_off(inst);
+}
diff --git a/cxl/event_trace.h b/cxl/event_trace.h
index e83737de0ad5..ec6267202c8b 100644
--- a/cxl/event_trace.h
+++ b/cxl/event_trace.h
@@ -20,5 +20,8 @@ struct event_ctx {
 };
 
 int cxl_parse_events(struct tracefs_instance *inst, struct event_ctx *ectx);
+int cxl_event_tracing_enable(struct tracefs_instance *inst, const char *system,
+		const char *event);
+int cxl_event_tracing_disable(struct tracefs_instance *inst);
 
 #endif

[PATCH v5 4/7] ndctl: move common logging functions from ndctl/monitor.c to util/log.c

[Dave Jiang]

Since cxl/monitor.c will be using same logging functions that
ndctl/monitor.c will be using, move common functions to util/log.c.

Signed-off-by: Dave Jiang <dave.jiang@gmail.com>
---
 ndctl/monitor.c |   41 ++++-------------------------------------
 util/log.c      |   34 ++++++++++++++++++++++++++++++++++
 util/log.h      |    8 +++++++-
 3 files changed, 45 insertions(+), 38 deletions(-)

diff --git a/ndctl/monitor.c b/ndctl/monitor.c
index 54678d6100b9..89903def63d4 100644
--- a/ndctl/monitor.c
+++ b/ndctl/monitor.c
@@ -33,7 +33,6 @@ static struct monitor {
 	const char *log;
 	const char *configs;
 	const char *dimm_event;
-	FILE *log_file;
 	bool daemon;
 	bool human;
 	bool verbose;
@@ -61,38 +60,6 @@ do { \
 			VERSION, __func__, __LINE__, ##__VA_ARGS__); \
 } while (0)
 
-static void log_syslog(struct log_ctx *ctx, int priority, const char *file,
-		int line, const char *fn, const char *format, va_list args)
-{
-	vsyslog(priority, format, args);
-}
-
-static void log_standard(struct log_ctx *ctx, int priority, const char *file,
-		int line, const char *fn, const char *format, va_list args)
-{
-	if (priority == 6)
-		vfprintf(stdout, format, args);
-	else
-		vfprintf(stderr, format, args);
-}
-
-static void log_file(struct log_ctx *ctx, int priority, const char *file,
-		int line, const char *fn, const char *format, va_list args)
-{
-	FILE *f = monitor.log_file;
-
-	if (priority != LOG_NOTICE) {
-		struct timespec ts;
-
-		clock_gettime(CLOCK_REALTIME, &ts);
-		fprintf(f, "[%10ld.%09ld] [%d] ", ts.tv_sec, ts.tv_nsec, getpid());
-		vfprintf(f, format, args);
-	} else
-		vfprintf(f, format, args);
-
-	fflush(f);
-}
-
 static struct json_object *dimm_event_to_json(struct monitor_dimm *mdimm)
 {
 	struct json_object *jevent, *jobj;
@@ -648,8 +615,8 @@ int cmd_monitor(int argc, const char **argv, struct ndctl_ctx *ctx)
 		else if (strncmp(monitor.log, "./standard", 10) == 0)
 			monitor.ctx.log_fn = log_standard;
 		else {
-			monitor.log_file = fopen(monitor.log, "a+");
-			if (!monitor.log_file) {
+			monitor.ctx.log_file = fopen(monitor.log, "a+");
+			if (!monitor.ctx.log_file) {
 				error("open %s failed\n", monitor.log);
 				rc = -errno;
 				goto out;
@@ -694,8 +661,8 @@ int cmd_monitor(int argc, const char **argv, struct ndctl_ctx *ctx)
 
 	rc = monitor_event(ctx, &mfa);
 out:
-	if (monitor.log_file)
-		fclose(monitor.log_file);
+	if (monitor.ctx.log_file)
+		fclose(monitor.ctx.log_file);
 	if (path)
 		free(path);
 	return rc;
diff --git a/util/log.c b/util/log.c
index 61ac509f4ac5..d4eef0a1fc5c 100644
--- a/util/log.c
+++ b/util/log.c
@@ -2,11 +2,45 @@
 // Copyright (C) 2016-2020, Intel Corporation. All rights reserved.
 #include <syslog.h>
 #include <stdlib.h>
+#include <unistd.h>
 #include <ctype.h>
 #include <string.h>
 #include <errno.h>
+#include <time.h>
 #include <util/log.h>
 
+void log_syslog(struct log_ctx *ctx, int priority, const char *file, int line,
+		const char *fn, const char *format, va_list args)
+{
+	vsyslog(priority, format, args);
+}
+
+void log_standard(struct log_ctx *ctx, int priority, const char *file, int line,
+		  const char *fn, const char *format, va_list args)
+{
+	if (priority == 6)
+		vfprintf(stdout, format, args);
+	else
+		vfprintf(stderr, format, args);
+}
+
+void log_file(struct log_ctx *ctx, int priority, const char *file, int line,
+	      const char *fn, const char *format, va_list args)
+{
+	FILE *f = ctx->log_file;
+
+	if (priority != LOG_NOTICE) {
+		struct timespec ts;
+
+		clock_gettime(CLOCK_REALTIME, &ts);
+		fprintf(f, "[%10ld.%09ld] [%d] ", ts.tv_sec, ts.tv_nsec,
+			getpid());
+	}
+
+	vfprintf(f, format, args);
+	fflush(f);
+}
+
 void do_log(struct log_ctx *ctx, int priority, const char *file,
 		int line, const char *fn, const char *format, ...)
 {
diff --git a/util/log.h b/util/log.h
index 28f1c7bfcea4..6e09b0dc6494 100644
--- a/util/log.h
+++ b/util/log.h
@@ -14,9 +14,15 @@ struct log_ctx {
 	log_fn log_fn;
 	const char *owner;
 	int log_priority;
+	FILE *log_file;
 };
 
-
+void log_syslog(struct log_ctx *ctx, int priority, const char *file, int line,
+		const char *fn, const char *format, va_list args);
+void log_standard(struct log_ctx *ctx, int priority, const char *file, int line,
+		  const char *fn, const char *format, va_list args);
+void log_file(struct log_ctx *ctx, int priority, const char *file, int line,
+	      const char *fn, const char *format, va_list args);
 void do_log(struct log_ctx *ctx, int priority, const char *file, int line,
 		const char *fn, const char *format, ...)
 	__attribute__((format(printf, 6, 7)));

[PATCH v5 5/7] ndctl: cxl: add monitor command for event trace events

[Dave Jiang]

Add function that creates an event trace instance and utilize the cxl event
trace common functions to extract interested events from the trace buffer.
The monitoring function will pend on an epoll fd and wait for new events
to appear in the trace buffer.

Connect the monitoring functionality to the cxl monitor command. Add basic
functionality to the cxl monitor command where it can be launched as a daemon
and logging can be designated to stdout or a file.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 cxl/builtin.h   |    1 
 cxl/cxl.c       |    1 
 cxl/meson.build |    1 
 cxl/monitor.c   |  215 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 218 insertions(+)
 create mode 100644 cxl/monitor.c

diff --git a/cxl/builtin.h b/cxl/builtin.h
index b28c2213993b..34c5cfb49051 100644
--- a/cxl/builtin.h
+++ b/cxl/builtin.h
@@ -22,4 +22,5 @@ int cmd_create_region(int argc, const char **argv, struct cxl_ctx *ctx);
 int cmd_enable_region(int argc, const char **argv, struct cxl_ctx *ctx);
 int cmd_disable_region(int argc, const char **argv, struct cxl_ctx *ctx);
 int cmd_destroy_region(int argc, const char **argv, struct cxl_ctx *ctx);
+int cmd_monitor(int argc, const char **argv, struct cxl_ctx *ctx);
 #endif /* _CXL_BUILTIN_H_ */
diff --git a/cxl/cxl.c b/cxl/cxl.c
index dd1be7a054a1..3be7026f43d3 100644
--- a/cxl/cxl.c
+++ b/cxl/cxl.c
@@ -76,6 +76,7 @@ static struct cmd_struct commands[] = {
 	{ "enable-region", .c_fn = cmd_enable_region },
 	{ "disable-region", .c_fn = cmd_disable_region },
 	{ "destroy-region", .c_fn = cmd_destroy_region },
+	{ "monitor", .c_fn = cmd_monitor },
 };
 
 int main(int argc, const char **argv)
diff --git a/cxl/meson.build b/cxl/meson.build
index c59876262e76..eb8b2b1070ed 100644
--- a/cxl/meson.build
+++ b/cxl/meson.build
@@ -8,6 +8,7 @@ cxl_src = [
   'json.c',
   'filter.c',
   'event_trace.c',
+  'monitor.c',
 ]
 
 cxl_tool = executable('cxl',
diff --git a/cxl/monitor.c b/cxl/monitor.c
new file mode 100644
index 000000000000..31e6f98f5299
--- /dev/null
+++ b/cxl/monitor.c
@@ -0,0 +1,215 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2022, Intel Corp. All rights reserved.
+/* Some bits copied from ndctl monitor code */
+#include <stdio.h>
+#include <unistd.h>
+#include <errno.h>
+#include <json-c/json.h>
+#include <libgen.h>
+#include <time.h>
+#include <dirent.h>
+#include <ccan/list/list.h>
+#include <util/json.h>
+#include <util/util.h>
+#include <util/parse-options.h>
+#include <util/parse-configs.h>
+#include <util/strbuf.h>
+#include <sys/epoll.h>
+#include <sys/stat.h>
+#include <traceevent/event-parse.h>
+#include <tracefs/tracefs.h>
+#include <cxl/libcxl.h>
+
+/* reuse the core log helpers for the monitor logger */
+#ifndef ENABLE_LOGGING
+#define ENABLE_LOGGING
+#endif
+#ifndef ENABLE_DEBUG
+#define ENABLE_DEBUG
+#endif
+#include <util/log.h>
+
+#include "event_trace.h"
+
+static const char *cxl_system = "cxl";
+const char *default_log = "/var/log/cxl-monitor.log";
+
+static struct monitor {
+	const char *log;
+	struct log_ctx ctx;
+	FILE *log_file;
+	bool human;
+	bool verbose;
+	bool daemon;
+} monitor;
+
+static int monitor_event(struct cxl_ctx *ctx)
+{
+	int fd, epollfd, rc = 0, timeout = -1;
+	struct epoll_event ev, *events;
+	struct tracefs_instance *inst;
+	struct event_ctx ectx;
+	int jflag;
+
+	events = calloc(1, sizeof(struct epoll_event));
+	if (!events) {
+		err(&monitor, "alloc for events error\n");
+		return -ENOMEM;
+	}
+
+	epollfd = epoll_create1(0);
+	if (epollfd == -1) {
+		rc = -errno;
+		err(&monitor, "epoll_create1() error: %d\n", rc);
+		goto epoll_err;
+	}
+
+	inst = tracefs_instance_create("cxl_monitor");
+	if (!inst) {
+		rc = -errno;
+		err(&monitor, "tracefs_instance_create( failed: %d\n", rc);
+		goto inst_err;
+	}
+
+	fd = tracefs_instance_file_open(inst, "trace_pipe", -1);
+	if (fd < 0) {
+		rc = fd;
+		err(&monitor, "tracefs_instance_file_open() err: %d\n", rc);
+		goto inst_file_err;
+	}
+
+	memset(&ev, 0, sizeof(ev));
+	ev.events = EPOLLIN;
+	ev.data.fd = fd;
+
+	if (epoll_ctl(epollfd, EPOLL_CTL_ADD, fd, &ev) != 0) {
+		rc = -errno;
+		err(&monitor, "epoll_ctl() error: %d\n", rc);
+		goto epoll_ctl_err;
+	}
+
+	rc = cxl_event_tracing_enable(inst, cxl_system, NULL);
+	if (rc < 0) {
+		err(&monitor, "cxl_trace_event_enable() failed: %d\n", rc);
+		goto event_en_err;
+	}
+
+	memset(&ectx, 0, sizeof(ectx));
+	ectx.system = cxl_system;
+	if (monitor.human)
+		jflag = JSON_C_TO_STRING_PRETTY;
+	else
+		jflag = JSON_C_TO_STRING_PLAIN;
+
+	while (1) {
+		struct jlist_node *jnode, *next;
+
+		rc = epoll_wait(epollfd, events, 1, timeout);
+		if (rc < 0) {
+			rc = -errno;
+			if (errno != EINTR)
+				err(&monitor, "epoll_wait error: %d\n", -errno);
+			break;
+		}
+
+		list_head_init(&ectx.jlist_head);
+		rc = cxl_parse_events(inst, &ectx);
+		if (rc < 0)
+			goto parse_err;
+
+		if (list_empty(&ectx.jlist_head))
+			continue;
+
+		list_for_each_safe(&ectx.jlist_head, jnode, next, list) {
+			notice(&monitor, "%s\n",
+				json_object_to_json_string_ext(jnode->jobj, jflag));
+			list_del(&jnode->list);
+			json_object_put(jnode->jobj);
+			free(jnode);
+		}
+	}
+
+parse_err:
+	rc = cxl_event_tracing_disable(inst);
+event_en_err:
+epoll_ctl_err:
+	close(fd);
+inst_file_err:
+	tracefs_instance_free(inst);
+inst_err:
+	close(epollfd);
+epoll_err:
+	free(events);
+	return rc;
+}
+
+int cmd_monitor(int argc, const char **argv, struct cxl_ctx *ctx)
+{
+	const struct option options[] = {
+		OPT_FILENAME('l', "log", &monitor.log,
+				"<file> | standard",
+				"where to output the monitor's notification"),
+		OPT_BOOLEAN('\0', "daemon", &monitor.daemon,
+				"run cxl monitor as a daemon"),
+		OPT_BOOLEAN('u', "human", &monitor.human,
+				"use human friendly output formats"),
+		OPT_BOOLEAN('v', "verbose", &monitor.verbose,
+				"emit extra debug messages to log"),
+		OPT_END(),
+	};
+	const char * const u[] = {
+		"cxl monitor [<options>]",
+		NULL
+	};
+	const char *prefix ="./";
+	int rc = 0, i;
+
+	argc = parse_options_prefix(argc, argv, prefix, options, u, 0);
+	for (i = 0; i < argc; i++)
+		error("unknown parameter \"%s\"\n", argv[i]);
+	if (argc)
+		usage_with_options(u, options);
+
+	log_init(&monitor.ctx, "cxl/monitor", "CXL_MONITOR_LOG");
+	monitor.ctx.log_fn = log_standard;
+
+	if (monitor.verbose)
+		monitor.ctx.log_priority = LOG_DEBUG;
+	else
+		monitor.ctx.log_priority = LOG_INFO;
+
+	if (monitor.log) {
+		if (strncmp(monitor.log, "./", 2) != 0)
+			fix_filename(prefix, (const char **)&monitor.log);
+		if (strncmp(monitor.log, "./standard", 10) == 0 && !monitor.daemon) {
+			monitor.ctx.log_fn = log_standard;
+		} else {
+			const char *log = monitor.log;
+
+			if (!monitor.log)
+				log = default_log;
+			monitor.log_file = fopen(log, "a+");
+			if (!monitor.log_file) {
+				rc = -errno;
+				error("open %s failed: %d\n", monitor.log, rc);
+				goto out;
+			}
+			monitor.ctx.log_fn = log_file;
+		}
+	}
+
+	if (monitor.daemon) {
+		if (daemon(0, 0) != 0) {
+			err(&monitor, "daemon start failed\n");
+			goto out;
+		}
+		info(&monitor, "cxl monitor daemon started.\n");
+	}
+
+	rc = monitor_event(ctx);
+
+out:
+	if (monitor.log_file)
+		fclose(monitor.log_file);
+	return rc;
+}

[PATCH v5 6/7] ndctl: cxl: add systemd service for monitor

[Dave Jiang]

Add a systemd service file for cxl monitor to start the monitoring service
on boot initialization. Add the installation setup for the service file.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 cxl/cxl-monitor.service |    9 +++++++++
 cxl/meson.build         |    4 ++++
 ndctl.spec.in           |    1 +
 3 files changed, 14 insertions(+)
 create mode 100644 cxl/cxl-monitor.service

diff --git a/cxl/cxl-monitor.service b/cxl/cxl-monitor.service
new file mode 100644
index 000000000000..66de39d883be
--- /dev/null
+++ b/cxl/cxl-monitor.service
@@ -0,0 +1,9 @@
+[Unit]
+Description=CXL Monitor Daemon
+
+[Service]
+Type=simple
+ExecStart=/usr/bin/cxl monitor
+
+[Install]
+WantedBy=multi-user.target
diff --git a/cxl/meson.build b/cxl/meson.build
index eb8b2b1070ed..fc2e946707a8 100644
--- a/cxl/meson.build
+++ b/cxl/meson.build
@@ -11,6 +11,10 @@ cxl_src = [
   'monitor.c',
 ]
 
+if get_option('systemd').enabled()
+  install_data('cxl-monitor.service', install_dir : systemdunitdir)
+endif
+
 cxl_tool = executable('cxl',
   cxl_src,
   include_directories : root_inc,
diff --git a/ndctl.spec.in b/ndctl.spec.in
index cfcafa2ba816..c883317c5ce7 100644
--- a/ndctl.spec.in
+++ b/ndctl.spec.in
@@ -194,6 +194,7 @@ fi
 %{_bindir}/cxl
 %{_mandir}/man1/cxl*
 %{bashcompdir}/cxl
+%{_unitdir}/cxl-monitor.service
 
 %files -n LNAME
 %defattr(-,root,root)

[PATCH v5 7/7] ndctl: cxl: add man page documentation for monitor

[Dave Jiang]

Add man page documentation to explain the usage of cxl monitor.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 Documentation/cxl/cxl-monitor.txt |   62 +++++++++++++++++++++++++++++++++++++
 Documentation/cxl/meson.build     |    1 +
 2 files changed, 63 insertions(+)
 create mode 100644 Documentation/cxl/cxl-monitor.txt

diff --git a/Documentation/cxl/cxl-monitor.txt b/Documentation/cxl/cxl-monitor.txt
new file mode 100644
index 000000000000..3fc992e4d4d9
--- /dev/null
+++ b/Documentation/cxl/cxl-monitor.txt
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-2.0
+
+cxl-monitor(1)
+==============
+
+NAME
+----
+cxl-monitor - Monitor the CXL trace events
+
+SYNOPSIS
+--------
+[verse]
+'cxl monitor' [<options>]
+
+DESCRIPTION
+-----------
+cxl-monitor is used for monitoring the CXL trace events emitted by
+the kernel and convert them to json objects and dumping the json format
+notifications to standard output or a logfile.
+
+EXAMPLES
+--------
+
+Run a monitor as a daemon to monitor events and output to a log file.
+[verse]
+cxl monitor --daemon --log=/var/log/cxl-monitor.log
+
+Run a monitor as a one-shot command and output the notifications to stdio.
+[verse]
+cxl monitor
+
+Run a monitor daemon as a system service
+[verse]
+systemctl start cxl-monitor.service
+
+OPTIONS
+-------
+-l::
+--log=::
+	Send log messages to the specified destination.
+	- "<file>":
+	  Send log messages to specified <file>. When fopen() is not able
+	  to open <file>, log messages will be forwarded to syslog.
+	- "standard":
+	  Send messages to standard output.
+
+The default log destination is '/var/log/cxl-monitor.log' if "--daemon" is specified,
+otherwise 'standard'. Note that standard and relative path for <file>
+will not work if "--daemon" is specified.
+
+--daemon::
+	Run a monitor as a daemon.
+
+include::verbose-option.txt[]
+
+include::human-option.txt[]
+
+include::../copyright.txt[]
+
+SEE ALSO
+--------
+linkcxl:cxl-list[1]
diff --git a/Documentation/cxl/meson.build b/Documentation/cxl/meson.build
index 147ea7130211..a6d77ab8cbc2 100644
--- a/Documentation/cxl/meson.build
+++ b/Documentation/cxl/meson.build
@@ -45,6 +45,7 @@ cxl_manpages = [
   'cxl-disable-region.txt',
   'cxl-enable-region.txt',
   'cxl-destroy-region.txt',
+  'cxl-monitor.txt',
 ]
 
 foreach man : cxl_manpages