[PATCH v3 00/11] trace-cmd: Allow agent to use networking

[PATCH v3 00/11] trace-cmd: Allow agent to use networking

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

There's no reason for trace-cmd agent not to be able to do the communication
over TCP instead of vsockets. Have trace-cmd record pass in a host name or
IP address (IPv4 or IPv6) to the -A option and if the cid or guest name
fails, then try host name or IP address.

This will fall back to the P2P time sync protocol and recordings from
different hosts can be synchronized.

Also add more debug prints when the --debug option is used. This proved
useful to debugging this code.

Changes since v2: https://lore.kernel.org/all/20220417184538.1044417-1-rostedt@goodmis.org/
 - Added comments about USE_UDP as being the default, as well as
     explicitly setting it to zero. (Tzvetomir Stoyanov)
 - Appended https://lore.kernel.org/all/20220417211315.1049221-1-rostedt@goodmis.org/
     to the patch set.
 - Added more warnings about the insecurities of using trace-cmd agent
     with the -N option. (Tzvetomir Stoyanov)


Changes since v1: https://lore.kernel.org/r/20220417011958.1013591-1-rostedt@goodmis.org
 - Rebased on the above dependency.
 - Added patch to allow compiling of agent without vsocket support.


Steven Rostedt (Google) (11):
  trace-cmd record: Move port_type into instance
  trace-cmd library: Add network roles for time sync
  trace-cmd record: Allow for ip connections to agents
  trace-cmd agent: Allow for ip connections from the agent
  trace-cmd library: Create tracecmd_debug() for debug printing
  trace-cmd: Add debug prints for network connections
  trace-cmd: Add print helpers to show connections
  trace-cmd: Override tracecmd_debug() to show thread id
  trace-cmd agent: Have agent work without vsockets available
  trace-cmd agent: Have -N take a host name
  trace-cmd agent: Add documentation

 Documentation/trace-cmd/trace-cmd-agent.1.txt |  62 ++++
 lib/trace-cmd/Makefile                        |   2 +-
 .../include/private/trace-cmd-private.h       |   5 +
 lib/trace-cmd/include/trace-tsync-local.h     |   6 +-
 lib/trace-cmd/trace-msg.c                     |  12 +-
 lib/trace-cmd/trace-timesync-ptp.c            |   4 +-
 lib/trace-cmd/trace-timesync.c                |   2 +
 lib/trace-cmd/trace-util.c                    |  12 +
 tracecmd/Makefile                             |   5 +-
 tracecmd/include/trace-local.h                |  25 +-
 tracecmd/trace-agent.c                        | 126 ++++++--
 tracecmd/trace-cmd.c                          |  20 +-
 tracecmd/trace-listen.c                       | 139 +++++++--
 tracecmd/trace-record.c                       | 269 ++++++++++++------
 tracecmd/trace-usage.c                        |   5 +-
 tracecmd/trace-vsock.c                        |  20 ++
 16 files changed, 562 insertions(+), 152 deletions(-)
 create mode 100644 Documentation/trace-cmd/trace-cmd-agent.1.txt

-- 
2.35.1

[PATCH v3 01/11] trace-cmd record: Move port_type into instance

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

Instead of one global method of communication, have the "port_type" be
part of the instance. This will allow for different instances to have
different connection types.

Link: https://lore.kernel.org/linux-trace-devel/20220417184538.1044417-2-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 tracecmd/include/trace-local.h |  3 ++-
 tracecmd/trace-record.c        | 41 ++++++++++++++++++----------------
 2 files changed, 24 insertions(+), 20 deletions(-)

diff --git a/tracecmd/include/trace-local.h b/tracecmd/include/trace-local.h
index aa1cb8d939bd..c75ebf7d7f17 100644
--- a/tracecmd/include/trace-local.h
+++ b/tracecmd/include/trace-local.h
@@ -43,7 +43,7 @@ extern int show_status;
 int trace_set_verbose(char *level);
 
 enum port_type {
-	USE_UDP,
+	USE_UDP			= 0,	/* Default setting */
 	USE_TCP,
 	USE_VSOCK
 };
@@ -285,6 +285,7 @@ struct buffer_instance {
 	int			*fds;
 	bool			use_fifos;
 
+	enum port_type		port_type;	/* Default to USE_UDP (zero) */
 	int			tsync_loop_interval;
 	struct tracecmd_time_sync *tsync;
 };
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index 674ec2c3ba65..14ba4ac5dc9e 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -90,8 +90,6 @@ static bool fork_process;
 /* Max size to let a per cpu file get */
 static int max_kb;
 
-static enum port_type port_type;
-
 static int do_ptrace;
 
 static int filter_task;
@@ -3119,7 +3117,7 @@ static void finish(int sig)
 	finished = 1;
 }
 
-static int connect_port(const char *host, unsigned int port)
+static int connect_port(const char *host, unsigned int port, enum port_type type)
 {
 	struct addrinfo hints;
 	struct addrinfo *results, *rp;
@@ -3128,17 +3126,17 @@ static int connect_port(const char *host, unsigned int port)
 
 	snprintf(buf, BUFSIZ, "%u", port);
 
-	if (port_type == USE_VSOCK)
+	if (type == USE_VSOCK)
 		return trace_vsock_open(atoi(host), port);
 
 	memset(&hints, 0, sizeof(hints));
 	hints.ai_family = AF_UNSPEC;
-	hints.ai_socktype = port_type == USE_TCP ? SOCK_STREAM : SOCK_DGRAM;
+	hints.ai_socktype = type == USE_TCP ? SOCK_STREAM : SOCK_DGRAM;
 
 	s = getaddrinfo(host, buf, &hints, &results);
 	if (s != 0)
 		die("connecting to %s server %s:%s",
-		    port_type == USE_TCP ? "TCP" : "UDP", host, buf);
+		    type == USE_TCP ? "TCP" : "UDP", host, buf);
 
 	for (rp = results; rp != NULL; rp = rp->ai_next) {
 		sfd = socket(rp->ai_family, rp->ai_socktype,
@@ -3152,7 +3150,7 @@ static int connect_port(const char *host, unsigned int port)
 
 	if (rp == NULL)
 		die("Can not connect to %s server %s:%s",
-		    port_type == USE_TCP ? "TCP" : "UDP", host, buf);
+		    type == USE_TCP ? "TCP" : "UDP", host, buf);
 
 	freeaddrinfo(results);
 
@@ -3359,7 +3357,8 @@ static int create_recorder(struct buffer_instance *instance, int cpu,
 			else
 				fd = do_accept(instance->fds[cpu]);
 		} else {
-			fd = connect_port(host, instance->client_ports[cpu]);
+			fd = connect_port(host, instance->client_ports[cpu],
+					  instance->port_type);
 		}
 		if (fd < 0)
 			die("Failed connecting to client");
@@ -3414,8 +3413,9 @@ static void check_first_msg_from_server(struct tracecmd_msg_handle *msg_handle)
 }
 
 static void communicate_with_listener_v1(struct tracecmd_msg_handle *msg_handle,
-					 unsigned int **client_ports)
+					 struct buffer_instance *instance)
 {
+	unsigned int *client_ports;
 	char buf[BUFSIZ];
 	ssize_t n;
 	int cpu, i;
@@ -3442,11 +3442,11 @@ static void communicate_with_listener_v1(struct tracecmd_msg_handle *msg_handle,
 	/* TODO, test for ipv4 */
 	if (page_size >= UDP_MAX_PACKET) {
 		warning("page size too big for UDP using TCP in live read");
-		port_type = USE_TCP;
+		instance->port_type = USE_TCP;
 		msg_handle->flags |= TRACECMD_MSG_FL_USE_TCP;
 	}
 
-	if (port_type == USE_TCP) {
+	if (instance->port_type == USE_TCP) {
 		/* Send one option */
 		write(msg_handle->fd, "1", 2);
 		/* Size 4 */
@@ -3457,8 +3457,8 @@ static void communicate_with_listener_v1(struct tracecmd_msg_handle *msg_handle,
 		/* No options */
 		write(msg_handle->fd, "0", 2);
 
-	*client_ports = malloc(local_cpu_count * sizeof(*client_ports));
-	if (!*client_ports)
+	client_ports = malloc(local_cpu_count * sizeof(*client_ports));
+	if (!client_ports)
 		die("Failed to allocate client ports for %d cpus", local_cpu_count);
 
 	/*
@@ -3476,8 +3476,10 @@ static void communicate_with_listener_v1(struct tracecmd_msg_handle *msg_handle,
 		if (i == BUFSIZ)
 			die("read bad port number");
 		buf[i] = 0;
-		(*client_ports)[cpu] = atoi(buf);
+		client_ports[cpu] = atoi(buf);
 	}
+
+	instance->client_ports = client_ports;
 }
 
 static void communicate_with_listener_v3(struct tracecmd_msg_handle *msg_handle,
@@ -3609,10 +3611,11 @@ static int connect_ip(char *thost)
 static struct tracecmd_msg_handle *setup_network(struct buffer_instance *instance)
 {
 	struct tracecmd_msg_handle *msg_handle = NULL;
+	enum port_type type = instance->port_type;
 	int sfd;
 
 again:
-	switch (port_type) {
+	switch (type) {
 	case USE_VSOCK:
 		sfd = connect_vsock(host);
 		break;
@@ -3634,7 +3637,7 @@ again:
 		msg_handle->version = V3_PROTOCOL;
 	}
 
-	switch (port_type) {
+	switch (type) {
 	case USE_TCP:
 		msg_handle->flags |= TRACECMD_MSG_FL_USE_TCP;
 		break;
@@ -3657,7 +3660,7 @@ again:
 	}
 
 	if (msg_handle->version == V1_PROTOCOL)
-		communicate_with_listener_v1(msg_handle, &instance->client_ports);
+		communicate_with_listener_v1(msg_handle, instance);
 
 	return msg_handle;
 }
@@ -6514,7 +6517,7 @@ static void parse_record_options(int argc,
 			if (ctx->output)
 				die("-V incompatible with -o");
 			host = optarg;
-			port_type = USE_VSOCK;
+			ctx->instance->port_type = USE_VSOCK;
 			break;
 		case 'm':
 			if (max_kb)
@@ -6532,7 +6535,7 @@ static void parse_record_options(int argc,
 			if (IS_EXTRACT(ctx))
 				ctx->topt = 1; /* Extract top instance also */
 			else
-				port_type = USE_TCP;
+				ctx->instance->port_type = USE_TCP;
 			break;
 		case 'b':
 			check_instance_die(ctx->instance, "-b");
-- 
2.35.1

[PATCH v3 02/11] trace-cmd library: Add network roles for time sync

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

Add TRACECMD_TIME_SYNC_ROLE_CLIENT and SERVER to distringuish from
guest/host to client/server.

Link: https://lore.kernel.org/linux-trace-devel/20220417184538.1044417-3-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 lib/trace-cmd/include/private/trace-cmd-private.h | 2 ++
 lib/trace-cmd/trace-timesync-ptp.c                | 4 +++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index f68d17bb8e1d..6e7b346cda07 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -480,6 +480,8 @@ enum{
 enum tracecmd_time_sync_role {
 	TRACECMD_TIME_SYNC_ROLE_HOST	= (1 << 0),
 	TRACECMD_TIME_SYNC_ROLE_GUEST	= (1 << 1),
+	TRACECMD_TIME_SYNC_ROLE_CLIENT	= (1 << 2),
+	TRACECMD_TIME_SYNC_ROLE_SERVER	= (1 << 3),
 };
 
 /* Timestamp synchronization flags */
diff --git a/lib/trace-cmd/trace-timesync-ptp.c b/lib/trace-cmd/trace-timesync-ptp.c
index 653d176e2e79..20e6e6f1ab59 100644
--- a/lib/trace-cmd/trace-timesync-ptp.c
+++ b/lib/trace-cmd/trace-timesync-ptp.c
@@ -702,7 +702,9 @@ int ptp_clock_sync_register(void)
 {
 	return tracecmd_tsync_proto_register(PTP_NAME, PTP_ACCURACY,
 					     TRACECMD_TIME_SYNC_ROLE_GUEST |
-					     TRACECMD_TIME_SYNC_ROLE_HOST,
+					     TRACECMD_TIME_SYNC_ROLE_HOST |
+					     TRACECMD_TIME_SYNC_ROLE_CLIENT |
+					     TRACECMD_TIME_SYNC_ROLE_SERVER,
 					     0, TRACECMD_TSYNC_FLAG_INTERPOLATE,
 					     ptp_clock_sync_init,
 					     ptp_clock_sync_free,
-- 
2.35.1

[PATCH v3 03/11] trace-cmd record: Allow for ip connections to agents

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

Have the -A option to trace-cmd record accept IP addresses (v4 or v6)
or host names that can connect to agents that are not running on guests
(or maybe they are!) but are running on other machines connecting to the
network.

Link: https://lore.kernel.org/linux-trace-devel/20220417184538.1044417-4-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 tracecmd/include/trace-local.h |   3 +
 tracecmd/trace-record.c        | 214 +++++++++++++++++++++++----------
 2 files changed, 151 insertions(+), 66 deletions(-)

diff --git a/tracecmd/include/trace-local.h b/tracecmd/include/trace-local.h
index c75ebf7d7f17..5ffd29124859 100644
--- a/tracecmd/include/trace-local.h
+++ b/tracecmd/include/trace-local.h
@@ -189,6 +189,7 @@ enum buffer_instance_flags {
 	BUFFER_FL_AGENT		= 1 << 3,
 	BUFFER_FL_HAS_CLOCK	= 1 << 4,
 	BUFFER_FL_TSC2NSEC	= 1 << 5,
+	BUFFER_FL_NETWORK	= 1 << 6,
 };
 
 struct func_list {
@@ -280,6 +281,7 @@ struct buffer_instance {
 	int			argc;
 	char			**argv;
 
+	struct addrinfo		*result;
 	unsigned int		cid;
 	unsigned int		port;
 	int			*fds;
@@ -302,6 +304,7 @@ extern struct buffer_instance *first_instance;
 
 #define is_agent(instance)	((instance)->flags & BUFFER_FL_AGENT)
 #define is_guest(instance)	((instance)->flags & BUFFER_FL_GUEST)
+#define is_network(instance)	((instance)->flags & BUFFER_FL_NETWORK)
 
 struct buffer_instance *allocate_instance(const char *name);
 void add_instance(struct buffer_instance *instance, int cpu_count);
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index 14ba4ac5dc9e..dfcf3d494053 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -83,6 +83,8 @@ static bool no_fifos;
 
 static char *host;
 
+static const char *gai_err;
+
 static bool quiet;
 
 static bool fork_process;
@@ -3117,26 +3119,33 @@ static void finish(int sig)
 	finished = 1;
 }
 
-static int connect_port(const char *host, unsigned int port, enum port_type type)
+static struct addrinfo *do_getaddrinfo(const char *host, unsigned int port,
+				       enum port_type type)
 {
+	struct addrinfo *results;
 	struct addrinfo hints;
-	struct addrinfo *results, *rp;
-	int s, sfd;
 	char buf[BUFSIZ];
+	int s;
 
 	snprintf(buf, BUFSIZ, "%u", port);
 
-	if (type == USE_VSOCK)
-		return trace_vsock_open(atoi(host), port);
-
 	memset(&hints, 0, sizeof(hints));
 	hints.ai_family = AF_UNSPEC;
 	hints.ai_socktype = type == USE_TCP ? SOCK_STREAM : SOCK_DGRAM;
 
 	s = getaddrinfo(host, buf, &hints, &results);
-	if (s != 0)
-		die("connecting to %s server %s:%s",
-		    type == USE_TCP ? "TCP" : "UDP", host, buf);
+	if (s != 0) {
+		gai_err = gai_strerror(s);
+		return NULL;
+	}
+
+	return results;
+}
+
+static int connect_addr(struct addrinfo *results)
+{
+	struct addrinfo *rp;
+	int sfd = -1;
 
 	for (rp = results; rp != NULL; rp = rp->ai_next) {
 		sfd = socket(rp->ai_family, rp->ai_socktype,
@@ -3149,11 +3158,33 @@ static int connect_port(const char *host, unsigned int port, enum port_type type
 	}
 
 	if (rp == NULL)
-		die("Can not connect to %s server %s:%s",
-		    type == USE_TCP ? "TCP" : "UDP", host, buf);
+		return -1;
+
+	return sfd;
+}
+
+static int connect_port(const char *host, unsigned int port, enum port_type type)
+{
+	struct addrinfo *results;
+	int sfd;
+
+	if (type == USE_VSOCK)
+		return trace_vsock_open(atoi(host), port);
+
+	results = do_getaddrinfo(host, port, type);
+
+	if (!results)
+		die("connecting to %s server %s:%u",
+		    type == USE_TCP ? "TCP" : "UDP", host, port);
+
+	sfd = connect_addr(results);
 
 	freeaddrinfo(results);
 
+	if (sfd < 0)
+		die("Can not connect to %s server %s:%u",
+		    type == USE_TCP ? "TCP" : "UDP", host, port);
+
 	return sfd;
 }
 
@@ -3207,14 +3238,22 @@ static void find_tasks(struct trace_guest *guest)
 	closedir(dir);
 }
 
-static char *parse_guest_name(char *gname, int *cid, int *port)
+static char *parse_guest_name(char *gname, int *cid, int *port,
+			      struct addrinfo **res)
 {
-	struct trace_guest *guest;
+	struct trace_guest *guest = NULL;
+	struct addrinfo *result;
+	char *ip = NULL;
 	char *p;
 
+	*res = NULL;
+
 	*port = -1;
-	p = strrchr(gname, ':');
-	if (p) {
+	for (p = gname + strlen(gname); p > gname; p--) {
+		if (*p == ':')
+			break;
+	}
+	if (p > gname) {
 		*p = '\0';
 		*port = atoi(p + 1);
 	}
@@ -3224,13 +3263,19 @@ static char *parse_guest_name(char *gname, int *cid, int *port)
 	if (p) {
 		*p = '\0';
 		*cid = atoi(p + 1);
-	} else if (is_digits(gname))
+	} else if (is_digits(gname)) {
 		*cid = atoi(gname);
+	} else {
+		/* Check if this is an IP address */
+		if (strstr(gname, ":") || strstr(gname, "."))
+			ip = gname;
+	}
 
-	if (*cid < 0)
+	if (!ip && *cid < 0)
 		read_qemu_guests();
 
-	guest = trace_get_guest(*cid, gname);
+	if (!ip)
+		guest = trace_get_guest(*cid, gname);
 	if (guest) {
 		*cid = guest->cid;
 		/* Mapping not found, search for them */
@@ -3239,6 +3284,13 @@ static char *parse_guest_name(char *gname, int *cid, int *port)
 		return guest->name;
 	}
 
+	/* Test to see if this is an internet address */
+	result = do_getaddrinfo(gname, *port, USE_TCP);
+	if (!result)
+		return NULL;
+
+	*res = result;
+
 	return gname;
 }
 
@@ -3280,6 +3332,7 @@ create_recorder_instance(struct buffer_instance *instance, const char *file, int
 			 int *brass)
 {
 	struct tracecmd_recorder *record;
+	struct addrinfo *result;
 	char *path;
 
 	if (is_guest(instance)) {
@@ -3288,7 +3341,17 @@ create_recorder_instance(struct buffer_instance *instance, const char *file, int
 
 		if (instance->use_fifos)
 			fd = instance->fds[cpu];
-		else
+		else if (is_network(instance)) {
+			result = do_getaddrinfo(instance->name,
+						instance->client_ports[cpu],
+						instance->port_type);
+			if (!result)
+				die("Failed to connect to %s port %d\n",
+				    instance->name,
+				    instance->client_ports[cpu]);
+			fd = connect_addr(result);
+			freeaddrinfo(result);
+		} else
 			fd = trace_vsock_open(instance->cid, instance->client_ports[cpu]);
 		if (fd < 0)
 			die("Failed to connect to agent");
@@ -3560,9 +3623,8 @@ static int connect_vsock(char *vhost)
 
 static int connect_ip(char *thost)
 {
-	struct addrinfo hints;
-	struct addrinfo *result, *rp;
-	int sfd, s;
+	struct addrinfo *result;
+	int sfd;
 	char *server;
 	char *port;
 	char *p;
@@ -3581,30 +3643,17 @@ static int connect_ip(char *thost)
 		port = strtok_r(NULL, ":", &p);
 	}
 
-	memset(&hints, 0, sizeof(hints));
-	hints.ai_family = AF_UNSPEC;
-	hints.ai_socktype = SOCK_STREAM;
+	result = do_getaddrinfo(server, atoi(port), USE_TCP);
+	if (!result)
+		die("getaddrinfo: %s", gai_err);
 
-	s = getaddrinfo(server, port, &hints, &result);
-	if (s != 0)
-		die("getaddrinfo: %s", gai_strerror(s));
+	sfd = connect_addr(result);
 
-	for (rp = result; rp != NULL; rp = rp->ai_next) {
-		sfd = socket(rp->ai_family, rp->ai_socktype,
-			     rp->ai_protocol);
-		if (sfd == -1)
-			continue;
-
-		if (connect(sfd, rp->ai_addr, rp->ai_addrlen) != -1)
-			break;
-		close(sfd);
-	}
+	freeaddrinfo(result);
 
-	if (!rp)
+	if (sfd < 0)
 		die("Can not connect to %s:%s", server, port);
 
-	freeaddrinfo(result);
-
 	return sfd;
 }
 
@@ -3960,20 +4009,26 @@ static int host_tsync(struct common_record_context *ctx,
 	if (!proto)
 		return -1;
 
-	guest = trace_get_guest(instance->cid, NULL);
-	if (guest == NULL)
-		return -1;
+	if (is_network(instance)) {
+		fd = connect_port(instance->name, tsync_port,
+				  instance->port_type);
+	} else {
+		guest = trace_get_guest(instance->cid, NULL);
+		if (guest == NULL)
+			return -1;
 
-	guest_pid = guest->pid;
-	start_mapping_vcpus(guest);
+		guest_pid = guest->pid;
+		start_mapping_vcpus(guest);
+		fd = trace_vsock_open(instance->cid, tsync_port);
+	}
 
-	fd = trace_vsock_open(instance->cid, tsync_port);
 	instance->tsync = tracecmd_tsync_with_guest(top_instance.trace_id,
 						    instance->tsync_loop_interval,
 						    fd, guest_pid,
 						    instance->cpu_count,
 						    proto, ctx->clock);
-	stop_mapping_vcpus(instance, guest);
+	if (!is_network(instance))
+		stop_mapping_vcpus(instance, guest);
 
 	if (!instance->tsync)
 		return -1;
@@ -3987,6 +4042,7 @@ static void connect_to_agent(struct common_record_context *ctx,
 	struct tracecmd_tsync_protos *protos = NULL;
 	int sd, ret, nr_fifos, nr_cpus, page_size;
 	struct tracecmd_msg_handle *msg_handle;
+	enum tracecmd_time_sync_role role;
 	char *tsync_protos_reply = NULL;
 	unsigned int tsync_port = 0;
 	unsigned int *ports;
@@ -3998,10 +4054,19 @@ static void connect_to_agent(struct common_record_context *ctx,
 		use_fifos = nr_fifos > 0;
 	}
 
-	sd = trace_vsock_open(instance->cid, instance->port);
-	if (sd < 0)
-		die("Failed to connect to vsocket @%u:%u",
-		    instance->cid, instance->port);
+	if (ctx->instance->result) {
+		role = TRACECMD_TIME_SYNC_ROLE_CLIENT;
+		sd = connect_addr(ctx->instance->result);
+		if (sd < 0)
+			die("Failed to connect to host %s:%u",
+			    instance->name, instance->port);
+	} else {
+		role = TRACECMD_TIME_SYNC_ROLE_HOST;
+		sd = trace_vsock_open(instance->cid, instance->port);
+		if (sd < 0)
+			die("Failed to connect to vsocket @%u:%u",
+			    instance->cid, instance->port);
+	}
 
 	msg_handle = tracecmd_msg_handle_alloc(sd, 0);
 	if (!msg_handle)
@@ -4011,8 +4076,7 @@ static void connect_to_agent(struct common_record_context *ctx,
 		instance->clock = tracefs_get_clock(NULL);
 
 	if (instance->tsync_loop_interval >= 0)
-		tracecmd_tsync_proto_getall(&protos, instance->clock,
-					    TRACECMD_TIME_SYNC_ROLE_HOST);
+		tracecmd_tsync_proto_getall(&protos, instance->clock, role);
 
 	ret = tracecmd_msg_send_trace_req(msg_handle, instance->argc,
 					  instance->argv, use_fifos,
@@ -4219,16 +4283,23 @@ static void append_buffer(struct tracecmd_output *handle,
 static void
 add_guest_info(struct tracecmd_output *handle, struct buffer_instance *instance)
 {
-	struct trace_guest *guest = trace_get_guest(instance->cid, NULL);
+	struct trace_guest *guest;
+	const char *name;
 	char *buf, *p;
 	int size;
 	int pid;
 	int i;
 
-	if (!guest)
-		return;
+	if (is_network(instance)) {
+		name = instance->name;
+	} else {
+		guest = trace_get_guest(instance->cid, NULL);
+		if (!guest)
+			return;
+		name = guest->name;
+	}
 
-	size = strlen(guest->name) + 1;
+	size = strlen(name) + 1;
 	size += sizeof(long long);	/* trace_id */
 	size += sizeof(int);		/* cpu count */
 	size += instance->cpu_count * 2 * sizeof(int);	/* cpu,pid pair */
@@ -4237,8 +4308,8 @@ add_guest_info(struct tracecmd_output *handle, struct buffer_instance *instance)
 	if (!buf)
 		return;
 	p = buf;
-	strcpy(p, guest->name);
-	p += strlen(guest->name) + 1;
+	strcpy(p, name);
+	p += strlen(name) + 1;
 
 	memcpy(p, &instance->trace_id, sizeof(long long));
 	p += sizeof(long long);
@@ -4246,10 +4317,11 @@ add_guest_info(struct tracecmd_output *handle, struct buffer_instance *instance)
 	memcpy(p, &instance->cpu_count, sizeof(int));
 	p += sizeof(int);
 	for (i = 0; i < instance->cpu_count; i++) {
-		if (i < guest->cpu_max)
-			pid = guest->cpu_pid[i];
-		else
-			pid = -1;
+		pid = -1;
+		if (!is_network(instance)) {
+			if (i < guest->cpu_max)
+				pid = guest->cpu_pid[i];
+		}
 		memcpy(p, &i, sizeof(int));
 		p += sizeof(int);
 		memcpy(p, &pid, sizeof(int));
@@ -6165,6 +6237,7 @@ static void parse_record_options(int argc,
 	const char *option;
 	struct event_list *event = NULL;
 	struct event_list *last_event = NULL;
+	struct addrinfo *result;
 	char *pids;
 	char *pid;
 	char *sav;
@@ -6304,8 +6377,8 @@ static void parse_record_options(int argc,
 			if (!IS_RECORD(ctx))
 				die("-A is only allowed for record operations");
 
-			name = parse_guest_name(optarg, &cid, &port);
-			if (cid == -1)
+			name = parse_guest_name(optarg, &cid, &port, &result);
+			if (cid == -1 && !result)
 				die("guest %s not found", optarg);
 			if (port == -1)
 				port = TRACE_AGENT_DEFAULT_PORT;
@@ -6321,7 +6394,16 @@ static void parse_record_options(int argc,
 				die("Failed to allocate guest name");
 
 			ctx->instance = allocate_instance(name);
+			if (!ctx->instance)
+				die("Failed to allocate instance");
+
+			if (result) {
+				ctx->instance->flags |= BUFFER_FL_NETWORK;
+				ctx->instance->port_type = USE_TCP;
+			}
+
 			ctx->instance->flags |= BUFFER_FL_GUEST;
+			ctx->instance->result = result;
 			ctx->instance->cid = cid;
 			ctx->instance->port = port;
 			ctx->instance->name = name;
-- 
2.35.1

[PATCH v3 04/11] trace-cmd agent: Allow for ip connections from the agent

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

Add a -N option to trace-cmd agent to listen on a network socket that
can be used by trace-cmd record -A to connect with.

Link: https://lore.kernel.org/linux-trace-devel/20220417184538.1044417-5-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 tracecmd/include/trace-local.h |  6 +++
 tracecmd/trace-agent.c         | 89 ++++++++++++++++++++++++++--------
 tracecmd/trace-listen.c        | 58 ++++++++++++++--------
 3 files changed, 113 insertions(+), 40 deletions(-)

diff --git a/tracecmd/include/trace-local.h b/tracecmd/include/trace-local.h
index 5ffd29124859..77d531d5056a 100644
--- a/tracecmd/include/trace-local.h
+++ b/tracecmd/include/trace-local.h
@@ -306,6 +306,12 @@ extern struct buffer_instance *first_instance;
 #define is_guest(instance)	((instance)->flags & BUFFER_FL_GUEST)
 #define is_network(instance)	((instance)->flags & BUFFER_FL_NETWORK)
 
+#define START_PORT_SEARCH 1500
+#define MAX_PORT_SEARCH 6000
+
+int trace_net_make(int port, enum port_type type);
+int trace_net_search(int start_port, int *sfd, enum port_type type);
+
 struct buffer_instance *allocate_instance(const char *name);
 void add_instance(struct buffer_instance *instance, int cpu_count);
 void update_first_instance(struct buffer_instance *instance, int topt);
diff --git a/tracecmd/trace-agent.c b/tracecmd/trace-agent.c
index 719d7126ca67..cd201ad948ad 100644
--- a/tracecmd/trace-agent.c
+++ b/tracecmd/trace-agent.c
@@ -41,6 +41,32 @@ static void make_vsocks(int nr, int *fds, unsigned int *ports)
 	}
 }
 
+static void make_net(int nr, int *fds, unsigned int *ports)
+{
+	int port;
+	int i, fd;
+	int start_port = START_PORT_SEARCH;
+
+	for (i = 0; i < nr; i++) {
+		port = trace_net_search(start_port, &fd, USE_TCP);
+		if (port < 0)
+			die("Failed to open socket");
+		if (listen(fd, 5) < 0)
+			die("Failed to listen on port %d\n", port);
+		fds[i] = fd;
+		ports[i] = port;
+		start_port = port + 1;
+	}
+}
+
+static void make_sockets(int nr, int *fds, unsigned int *ports, bool network)
+{
+	if (network)
+		return make_net(nr, fds, ports);
+	else
+		return make_vsocks(nr, fds, ports);
+}
+
 static int open_agent_fifos(int nr_cpus, int *fds)
 {
 	char path[PATH_MAX];
@@ -80,7 +106,7 @@ static char *get_clock(int argc, char **argv)
 	return NULL;
 }
 
-static void agent_handle(int sd, int nr_cpus, int page_size)
+static void agent_handle(int sd, int nr_cpus, int page_size, bool network)
 {
 	struct tracecmd_tsync_protos *tsync_protos = NULL;
 	struct tracecmd_time_sync *tsync = NULL;
@@ -117,17 +143,31 @@ static void agent_handle(int sd, int nr_cpus, int page_size)
 		use_fifos = false;
 
 	if (!use_fifos)
-		make_vsocks(nr_cpus, fds, ports);
+		make_sockets(nr_cpus, fds, ports, network);
 	if (tsync_protos && tsync_protos->names) {
-		if (get_vsocket_params(msg_handle->fd, &local_id,
-				       &remote_id)) {
-			warning("Failed to get local and remote ids");
-			/* Just make something up */
-			remote_id = -1;
-			local_id = -2;
+		if (network) {
+			/* For now just use something */
+			remote_id = 2;
+			local_id = 1;
+			tsync_port = trace_net_search(START_PORT_SEARCH, &fd, USE_TCP);
+			if (listen(fd, 5) < 0)
+				die("Failed to listen on %d\n", tsync_port);
+		} else {
+			if (get_vsocket_params(msg_handle->fd, &local_id,
+					       &remote_id)) {
+				warning("Failed to get local and remote ids");
+				/* Just make something up */
+				remote_id = -1;
+				local_id = -2;
+			}
+			fd = trace_vsock_make_any();
+			if (fd >= 0 &&
+			    trace_vsock_get_port(fd, &tsync_port) < 0) {
+				close(fd);
+				fd = -1;
+			}
 		}
-		fd = trace_vsock_make_any();
-		if (fd >= 0 && trace_vsock_get_port(fd, &tsync_port) >= 0) {
+		if (fd >= 0) {
 			tsync = tracecmd_tsync_with_host(fd, tsync_protos,
 							 get_clock(argc, argv),
 							 remote_id, local_id);
@@ -193,7 +233,7 @@ static pid_t do_fork()
 	return fork();
 }
 
-static void agent_serve(unsigned int port, bool do_daemon)
+static void agent_serve(unsigned int port, bool do_daemon, bool network)
 {
 	int sd, cd, nr_cpus;
 	unsigned int cid;
@@ -204,14 +244,21 @@ static void agent_serve(unsigned int port, bool do_daemon)
 	nr_cpus = tracecmd_count_cpus();
 	page_size = getpagesize();
 
-	sd = trace_vsock_make(port);
+	if (network) {
+		sd = trace_net_make(port, USE_TCP);
+		if (listen(sd, 5) < 0)
+			die("Failed to listen on %d\n", port);
+	} else
+		sd = trace_vsock_make(port);
 	if (sd < 0)
-		die("Failed to open vsocket");
+		die("Failed to open socket");
 	tracecmd_tsync_init();
 
-	cid = trace_vsock_local_cid();
-	if (cid >= 0)
-		printf("listening on @%u:%u\n", cid, port);
+	if (!network) {
+		cid = trace_vsock_local_cid();
+		if (cid >= 0)
+			printf("listening on @%u:%u\n", cid, port);
+	}
 
 	if (do_daemon && daemon(1, 0))
 		die("daemon");
@@ -231,7 +278,7 @@ static void agent_serve(unsigned int port, bool do_daemon)
 		if (pid == 0) {
 			close(sd);
 			signal(SIGCHLD, SIG_DFL);
-			agent_handle(cd, nr_cpus, page_size);
+			agent_handle(cd, nr_cpus, page_size, network);
 		}
 		if (pid > 0)
 			handler_pid = pid;
@@ -250,6 +297,7 @@ void trace_agent(int argc, char **argv)
 {
 	bool do_daemon = false;
 	unsigned int port = TRACE_AGENT_DEFAULT_PORT;
+	bool network = false;
 
 	if (argc < 2)
 		usage(argv);
@@ -267,7 +315,7 @@ void trace_agent(int argc, char **argv)
 			{NULL, 0, NULL, 0}
 		};
 
-		c = getopt_long(argc-1, argv+1, "+hp:D",
+		c = getopt_long(argc-1, argv+1, "+hp:DN",
 				long_options, &option_index);
 		if (c == -1)
 			break;
@@ -275,6 +323,9 @@ void trace_agent(int argc, char **argv)
 		case 'h':
 			usage(argv);
 			break;
+		case 'N':
+			network = true;
+			break;
 		case 'p':
 			port = atoi(optarg);
 			break;
@@ -296,5 +347,5 @@ void trace_agent(int argc, char **argv)
 	if (optind < argc-1)
 		usage(argv);
 
-	agent_serve(port, do_daemon);
+	agent_serve(port, do_daemon, network);
 }
diff --git a/tracecmd/trace-listen.c b/tracecmd/trace-listen.c
index 2338173fc8a6..8476fa51dadd 100644
--- a/tracecmd/trace-listen.c
+++ b/tracecmd/trace-listen.c
@@ -234,52 +234,68 @@ static int setup_vsock_port(int start_port, int *sfd)
 	return start_port;
 }
 
-#define START_PORT_SEARCH 1500
-#define MAX_PORT_SEARCH 6000
-
-static int bind_a_port(int start_port, int *sfd, enum port_type type)
+int trace_net_make(int port, enum port_type type)
 {
 	struct addrinfo hints;
 	struct addrinfo *result, *rp;
 	char buf[BUFSIZ];
+	int sd;
 	int s;
-	int num_port = start_port;
 
-	if (type == USE_VSOCK)
-		return setup_vsock_port(start_port, sfd);
- again:
-	snprintf(buf, BUFSIZ, "%d", num_port);
+	snprintf(buf, BUFSIZ, "%d", port);
 
 	memset(&hints, 0, sizeof(hints));
 	hints.ai_family = AF_UNSPEC;
-	hints.ai_socktype = type == USE_TCP ? SOCK_STREAM : SOCK_DGRAM;
 	hints.ai_flags = AI_PASSIVE;
 
+	switch (type) {
+	case USE_TCP:
+		hints.ai_socktype = SOCK_STREAM;
+		break;
+	case USE_UDP:
+		hints.ai_socktype = SOCK_DGRAM;
+		break;
+	default:
+		return -1;
+	}
+
 	s = getaddrinfo(NULL, buf, &hints, &result);
 	if (s != 0)
 		pdie("getaddrinfo: error opening socket");
 
 	for (rp = result; rp != NULL; rp = rp->ai_next) {
-		*sfd = socket(rp->ai_family, rp->ai_socktype,
-			      rp->ai_protocol);
-		if (*sfd < 0)
+		sd = socket(rp->ai_family, rp->ai_socktype,
+			    rp->ai_protocol);
+		if (sd < 0)
 			continue;
 
-		if (bind(*sfd, rp->ai_addr, rp->ai_addrlen) == 0)
+		if (bind(sd, rp->ai_addr, rp->ai_addrlen) == 0)
 			break;
 
-		close(*sfd);
+		close(sd);
 	}
+	freeaddrinfo(result);
+
+	if (rp == NULL)
+		return -1;
+
+	return sd;
+}
+
+int trace_net_search(int start_port, int *sfd, enum port_type type)
+{
+	int num_port = start_port;
 
-	if (rp == NULL) {
-		freeaddrinfo(result);
+	if (type == USE_VSOCK)
+		return setup_vsock_port(start_port, sfd);
+ again:
+	*sfd = trace_net_make(num_port, type);
+	if (*sfd < 0) {
 		if (++num_port > MAX_PORT_SEARCH)
 			pdie("No available ports to bind");
 		goto again;
 	}
 
-	freeaddrinfo(result);
-
 	return num_port;
 }
 
@@ -309,10 +325,10 @@ static int open_port(const char *node, const char *port, int *pid,
 	int num_port;
 
 	/*
-	 * bind_a_port() currently does not return an error, but if that
+	 * trace_net_search() currently does not return an error, but if that
 	 * changes in the future, we have a check for it now.
 	 */
-	num_port = bind_a_port(start_port, &sfd, type);
+	num_port = trace_net_search(start_port, &sfd, type);
 	if (num_port < 0)
 		return num_port;
 
-- 
2.35.1

[PATCH v3 05/11] trace-cmd library: Create tracecmd_debug() for debug printing

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

Move the dprint() of trace-msg.c into trace-utils.c and export it such
that other applications could use it as well. It is now renamed to
tracecmd_debug().

Link: https://lore.kernel.org/linux-trace-devel/20220417184538.1044417-6-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 lib/trace-cmd/include/private/trace-cmd-private.h |  3 +++
 lib/trace-cmd/trace-msg.c                         | 12 +-----------
 lib/trace-cmd/trace-util.c                        | 12 ++++++++++++
 3 files changed, 16 insertions(+), 11 deletions(-)

diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h
index 6e7b346cda07..766e0a762c2b 100644
--- a/lib/trace-cmd/include/private/trace-cmd-private.h
+++ b/lib/trace-cmd/include/private/trace-cmd-private.h
@@ -43,6 +43,9 @@ enum {
 	RINGBUF_TYPE_TIME_STAMP		= 31,
 };
 
+/* Can be overridden */
+void tracecmd_debug(const char *fmt, ...);
+
 void tracecmd_record_ref(struct tep_record *record);
 
 void tracecmd_set_debug(bool set_debug);
diff --git a/lib/trace-cmd/trace-msg.c b/lib/trace-cmd/trace-msg.c
index 6cf74f9b1c99..39465ade8ab3 100644
--- a/lib/trace-cmd/trace-msg.c
+++ b/lib/trace-cmd/trace-msg.c
@@ -31,17 +31,7 @@
 typedef __u32 u32;
 typedef __be32 be32;
 
-static inline void dprint(const char *fmt, ...)
-{
-	va_list ap;
-
-	if (!tracecmd_get_debug())
-		return;
-
-	va_start(ap, fmt);
-	vprintf(fmt, ap);
-	va_end(ap);
-}
+#define dprint(fmt, ...)	tracecmd_debug(fmt, ##__VA_ARGS__)
 
 /* Two (4k) pages is the max transfer for now */
 #define MSG_MAX_LEN			8192
diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c
index b5a0a1a601d4..9564c81a5c99 100644
--- a/lib/trace-cmd/trace-util.c
+++ b/lib/trace-cmd/trace-util.c
@@ -415,6 +415,18 @@ void __weak tracecmd_critical(const char *fmt, ...)
 	}
 }
 
+void __weak tracecmd_debug(const char *fmt, ...)
+{
+	va_list ap;
+
+	if (!tracecmd_get_debug())
+		return;
+
+	va_start(ap, fmt);
+	vprintf(fmt, ap);
+	va_end(ap);
+}
+
 #define LOG_BUF_SIZE 1024
 static void __plog(const char *prefix, const char *fmt, va_list ap, FILE *fp)
 {
-- 
2.35.1

[PATCH v3 06/11] trace-cmd: Add debug prints for network connections

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

To help debug the network agent and listener, add some debug prints that
are enabled when --debug option is used.

Link: https://lore.kernel.org/linux-trace-devel/20220417184538.1044417-7-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 lib/trace-cmd/trace-timesync.c | 2 ++
 tracecmd/trace-agent.c         | 3 +++
 tracecmd/trace-listen.c        | 4 ++++
 tracecmd/trace-record.c        | 9 +++++++++
 4 files changed, 18 insertions(+)

diff --git a/lib/trace-cmd/trace-timesync.c b/lib/trace-cmd/trace-timesync.c
index 14cf20c870cb..cc44af38e8ad 100644
--- a/lib/trace-cmd/trace-timesync.c
+++ b/lib/trace-cmd/trace-timesync.c
@@ -925,7 +925,9 @@ static void *tsync_agent_thread(void *data)
 	int sd;
 
 	while (true) {
+		tracecmd_debug("Listening on fd:%d\n", tsync->msg_handle->fd);
 		sd = accept(tsync->msg_handle->fd, NULL, NULL);
+		tracecmd_debug("Accepted fd:%d\n", sd);
 		if (sd < 0) {
 			if (errno == EINTR)
 				continue;
diff --git a/tracecmd/trace-agent.c b/tracecmd/trace-agent.c
index cd201ad948ad..819ac016612f 100644
--- a/tracecmd/trace-agent.c
+++ b/tracecmd/trace-agent.c
@@ -22,6 +22,8 @@
 #include "trace-local.h"
 #include "trace-msg.h"
 
+#define dprint(fmt, ...)	tracecmd_debug(fmt, ##__VA_ARGS__)
+
 static void make_vsocks(int nr, int *fds, unsigned int *ports)
 {
 	unsigned int port;
@@ -55,6 +57,7 @@ static void make_net(int nr, int *fds, unsigned int *ports)
 			die("Failed to listen on port %d\n", port);
 		fds[i] = fd;
 		ports[i] = port;
+		dprint("CPU[%d]: fd:%d port:%d\n", i, fd, port);
 		start_port = port + 1;
 	}
 }
diff --git a/tracecmd/trace-listen.c b/tracecmd/trace-listen.c
index 8476fa51dadd..3b446acea972 100644
--- a/tracecmd/trace-listen.c
+++ b/tracecmd/trace-listen.c
@@ -26,6 +26,8 @@
 #include "trace-local.h"
 #include "trace-msg.h"
 
+#define dprint(fmt, ...)	tracecmd_debug(fmt, ##__VA_ARGS__)
+
 #define MAX_OPTION_SIZE 4096
 
 #define _VAR_DIR_Q(dir)		#dir
@@ -279,6 +281,8 @@ int trace_net_make(int port, enum port_type type)
 	if (rp == NULL)
 		return -1;
 
+	dprint("Create listen port: %d fd:%d\n", port, sd);
+
 	return sd;
 }
 
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index dfcf3d494053..9c930920c89e 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -56,6 +56,8 @@
 #define FUNC_STACK_TRACE "func_stack_trace"
 #define TSC_CLOCK	"x86-tsc"
 
+#define dprint(fmt, ...)	tracecmd_debug(fmt, ##__VA_ARGS__)
+
 enum trace_type {
 	TRACE_TYPE_RECORD	= 1,
 	TRACE_TYPE_START	= (1 << 1),
@@ -3139,6 +3141,9 @@ static struct addrinfo *do_getaddrinfo(const char *host, unsigned int port,
 		return NULL;
 	}
 
+	dprint("Attached port %s: %d to results: %p\n",
+	       type == USE_TCP ? "TCP" : "UDP", port, results);
+
 	return results;
 }
 
@@ -3160,6 +3165,8 @@ static int connect_addr(struct addrinfo *results)
 	if (rp == NULL)
 		return -1;
 
+	dprint("connect results: %p with fd: %d\n", results, sfd);
+
 	return sfd;
 }
 
@@ -3193,7 +3200,9 @@ static int do_accept(int sd)
 	int cd;
 
 	for (;;) {
+		dprint("Wait on accept: %d\n", sd);
 		cd = accept(sd, NULL, NULL);
+		dprint("accepted: %d\n", cd);
 		if (cd < 0) {
 			if (errno == EINTR)
 				continue;
-- 
2.35.1

[PATCH v3 07/11] trace-cmd: Add print helpers to show connections

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

Add helper functions trace_net_print_connection() and
trace_vsock_print_connection() to print the host/port or cid/port that
has connected.

Link: https://lore.kernel.org/linux-trace-devel/20220417184538.1044417-8-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 tracecmd/include/trace-local.h |  6 ++++++
 tracecmd/trace-agent.c         | 14 ++++++++++++++
 tracecmd/trace-listen.c        | 22 ++++++++++++++++++++++
 tracecmd/trace-vsock.c         | 20 ++++++++++++++++++++
 4 files changed, 62 insertions(+)

diff --git a/tracecmd/include/trace-local.h b/tracecmd/include/trace-local.h
index 77d531d5056a..8c59595795a4 100644
--- a/tracecmd/include/trace-local.h
+++ b/tracecmd/include/trace-local.h
@@ -311,6 +311,7 @@ extern struct buffer_instance *first_instance;
 
 int trace_net_make(int port, enum port_type type);
 int trace_net_search(int start_port, int *sfd, enum port_type type);
+int trace_net_print_connection(int fd);
 
 struct buffer_instance *allocate_instance(const char *name);
 void add_instance(struct buffer_instance *instance, int cpu_count);
@@ -366,6 +367,7 @@ int get_vsocket_params(int fd, unsigned int *lcid, unsigned int *rcid);
 int trace_vsock_get_port(int sd, unsigned int *port);
 bool trace_vsock_can_splice_read(void);
 int trace_vsock_local_cid(void);
+int trace_vsock_print_connection(int fd);
 #else
 static inline int trace_vsock_open(unsigned int cid, unsigned int port)
 {
@@ -403,6 +405,10 @@ static inline int trace_vsock_local_cid(void)
 {
 	return -ENOTSUP;
 }
+static inline int trace_vsock_print_connection(int fd)
+{
+	return -1;
+}
 #endif /* VSOCK */
 
 /* No longer in event-utils.h */
diff --git a/tracecmd/trace-agent.c b/tracecmd/trace-agent.c
index 819ac016612f..59cecae770a6 100644
--- a/tracecmd/trace-agent.c
+++ b/tracecmd/trace-agent.c
@@ -109,6 +109,18 @@ static char *get_clock(int argc, char **argv)
 	return NULL;
 }
 
+static void trace_print_connection(int fd, bool network)
+{
+	int ret;
+
+	if (network)
+		ret = trace_net_print_connection(fd);
+	else
+		ret = trace_vsock_print_connection(fd);
+	if (ret < 0)
+		tracecmd_debug("Could not print connection fd:%d\n", fd);
+}
+
 static void agent_handle(int sd, int nr_cpus, int page_size, bool network)
 {
 	struct tracecmd_tsync_protos *tsync_protos = NULL;
@@ -273,6 +285,8 @@ static void agent_serve(unsigned int port, bool do_daemon, bool network)
 				continue;
 			die("accept");
 		}
+		if (tracecmd_get_debug())
+			trace_print_connection(cd, network);
 
 		if (handler_pid)
 			goto busy;
diff --git a/tracecmd/trace-listen.c b/tracecmd/trace-listen.c
index 3b446acea972..b7be761d032e 100644
--- a/tracecmd/trace-listen.c
+++ b/tracecmd/trace-listen.c
@@ -756,6 +756,28 @@ static int do_fork(int cfd)
 	return 0;
 }
 
+int trace_net_print_connection(int fd)
+{
+	char host[NI_MAXHOST], service[NI_MAXSERV];
+	struct sockaddr_storage net_addr;
+	socklen_t addr_len;
+
+	addr_len = sizeof(net_addr);
+	if (getpeername(fd, (struct sockaddr *)&net_addr, &addr_len))
+		return -1;
+
+	if (getnameinfo((struct sockaddr *)&net_addr, addr_len,
+			host, NI_MAXHOST,
+			service, NI_MAXSERV, NI_NUMERICSERV))
+		return -1;
+
+	if (tracecmd_get_debug())
+		tracecmd_debug("Connected to %s:%s fd:%d\n", host, service, fd);
+	else
+		tracecmd_plog("Connected to %s:%s\n", host, service);
+	return 0;
+}
+
 static int do_connection(int cfd, struct sockaddr *addr,
 			  socklen_t addr_len)
 {
diff --git a/tracecmd/trace-vsock.c b/tracecmd/trace-vsock.c
index d18ecb45004e..39294e7a2a3c 100644
--- a/tracecmd/trace-vsock.c
+++ b/tracecmd/trace-vsock.c
@@ -94,6 +94,26 @@ int get_vsocket_params(int fd, unsigned int *lcid, unsigned int *rcid)
 	return 0;
 }
 
+int trace_vsock_print_connection(int fd)
+{
+	struct sockaddr_vm vm_addr;
+	socklen_t addr_len;
+	int cid, port;
+
+	addr_len = sizeof(vm_addr);
+	if (getpeername(fd, (struct sockaddr *)&vm_addr, &addr_len))
+		return -1;
+	if (vm_addr.svm_family != AF_VSOCK)
+		return -1;
+	cid = vm_addr.svm_cid;
+	port = vm_addr.svm_port;
+	if (tracecmd_get_debug())
+		tracecmd_debug("Connected to @%u:%u fd:%d\n", cid, port, fd);
+	else
+		tracecmd_plog("Connected to @%u:%u\n", cid, port);
+	return 0;
+}
+
 static int try_splice_read_vsock(void)
 {
 	int ret, sd, brass[2];
-- 
2.35.1

[PATCH v3 08/11] trace-cmd: Override tracecmd_debug() to show thread id

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

To differentiate threads, have tracecmd_debug() show the thread id of
the debug prints.

Link: https://lore.kernel.org/linux-trace-devel/20220417184538.1044417-9-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 tracecmd/trace-cmd.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/tracecmd/trace-cmd.c b/tracecmd/trace-cmd.c
index a83a8d0bb9e4..3f3e69a7f1e2 100644
--- a/tracecmd/trace-cmd.c
+++ b/tracecmd/trace-cmd.c
@@ -10,12 +10,17 @@
 #include <dirent.h>
 #include <errno.h>
 #include <stdlib.h>
+#include <sys/syscall.h>
 
 #include "trace-local.h"
 
 int silence_warnings;
 int show_status;
 
+#ifndef gettid
+#define gettid() syscall(__NR_gettid)
+#endif
+
 void warning(const char *fmt, ...)
 {
 	va_list ap;
@@ -45,6 +50,19 @@ void *malloc_or_die(unsigned int size)
 	return data;
 }
 
+void tracecmd_debug(const char *fmt, ...)
+{
+	va_list ap;
+
+	if (!tracecmd_get_debug())
+		return;
+
+	va_start(ap, fmt);
+	printf("[%d] ", (int)gettid());
+	vprintf(fmt, ap);
+	va_end(ap);
+}
+
 static struct trace_log_severity {
 	int		id;
 	const char	*name;
-- 
2.35.1

[PATCH v3 09/11] trace-cmd agent: Have agent work without vsockets available

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

Compile agent code without vsockets available to allow it to work with
networking. This includes allowing the ptp time synchronization protocol
to be used.

Link: https://lore.kernel.org/linux-trace-devel/20220417184538.1044417-10-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 lib/trace-cmd/Makefile                    | 2 +-
 lib/trace-cmd/include/trace-tsync-local.h | 6 +-----
 tracecmd/Makefile                         | 5 ++---
 tracecmd/trace-cmd.c                      | 2 --
 tracecmd/trace-usage.c                    | 2 --
 5 files changed, 4 insertions(+), 13 deletions(-)

diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile
index da0ad4deeb4f..9374b163b5f3 100644
--- a/lib/trace-cmd/Makefile
+++ b/lib/trace-cmd/Makefile
@@ -21,8 +21,8 @@ ifeq ($(PERF_DEFINED), 1)
 OBJS += trace-perf.o
 endif
 OBJS += trace-timesync.o
-ifeq ($(VSOCK_DEFINED), 1)
 OBJS += trace-timesync-ptp.o
+ifeq ($(VSOCK_DEFINED), 1)
 OBJS += trace-timesync-kvm.o
 endif
 OBJS += trace-compress.o
diff --git a/lib/trace-cmd/include/trace-tsync-local.h b/lib/trace-cmd/include/trace-tsync-local.h
index 885c9f51d891..5bbc1db622c4 100644
--- a/lib/trace-cmd/include/trace-tsync-local.h
+++ b/lib/trace-cmd/include/trace-tsync-local.h
@@ -64,15 +64,11 @@ int tracecmd_tsync_proto_register(const char *proto_name, int accuracy, int role
 					      long long *, long long *, long long*,
 					      long long *, unsigned int));
 int tracecmd_tsync_proto_unregister(char *proto_name);
+int ptp_clock_sync_register(void);
 
 #ifdef VSOCK
-int ptp_clock_sync_register(void);
 int kvm_clock_sync_register(void);
 #else
-static inline int ptp_clock_sync_register(void)
-{
-	return 0;
-}
 static inline int kvm_clock_sync_register(void)
 {
 	return 0;
diff --git a/tracecmd/Makefile b/tracecmd/Makefile
index 13f7776e8e45..0114948fe385 100644
--- a/tracecmd/Makefile
+++ b/tracecmd/Makefile
@@ -37,11 +37,10 @@ TRACE_CMD_OBJS += trace-dump.o
 TRACE_CMD_OBJS += trace-clear.o
 TRACE_CMD_OBJS += trace-vm.o
 TRACE_CMD_OBJS += trace-convert.o
-TRACE_CMD_OBJS += trace-vsock.o
-
-ifeq ($(VSOCK_DEFINED), 1)
 TRACE_CMD_OBJS += trace-agent.o
 TRACE_CMD_OBJS += trace-setup-guest.o
+ifeq ($(VSOCK_DEFINED), 1)
+TRACE_CMD_OBJS += trace-vsock.o
 endif
 
 ALL_OBJS := $(TRACE_CMD_OBJS:%.o=$(bdir)/%.o)
diff --git a/tracecmd/trace-cmd.c b/tracecmd/trace-cmd.c
index 3f3e69a7f1e2..69800d26c5ee 100644
--- a/tracecmd/trace-cmd.c
+++ b/tracecmd/trace-cmd.c
@@ -127,10 +127,8 @@ struct command commands[] = {
 	{"hist", trace_hist},
 	{"mem", trace_mem},
 	{"listen", trace_listen},
-#ifdef VSOCK
 	{"agent", trace_agent},
 	{"setup-guest", trace_setup_guest},
-#endif
 	{"split", trace_split},
 	{"restore", trace_restore},
 	{"stack", trace_stack},
diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c
index ec6376557394..88eac10a4e75 100644
--- a/tracecmd/trace-usage.c
+++ b/tracecmd/trace-usage.c
@@ -325,7 +325,6 @@ static struct usage_help usage_help[] = {
 		"          -l logfile to write messages to.\n"
 		"          --verbose 'level' Set the desired log level\n"
 	},
-#ifdef VSOCK
 	{
 		"agent",
 		"listen on a vsocket for trace clients",
@@ -344,7 +343,6 @@ static struct usage_help usage_help[] = {
 		"          -g FIFOs group owner\n"
 		"          -a Attach FIFOs to guest VM config\n"
 	},
-#endif
 	{
 		"list",
 		"list the available events, plugins or options",
-- 
2.35.1

[PATCH v3 10/11] trace-cmd agent: Have -N take a host name

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

For security reasons, it is not safe to let a trace-cmd agent connect to
*any* host. Have the -N option take a host name and only connect to that
host. It still gives full control to any process on that host, but at
least the agent is not fully open to *any* machine.

Link: https://lore.kernel.org/linux-trace-devel/20220417211315.1049221-2-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 tracecmd/include/trace-local.h |  7 ++++-
 tracecmd/trace-agent.c         | 34 +++++++++++++++------
 tracecmd/trace-listen.c        | 55 ++++++++++++++++++++++++++++++++++
 tracecmd/trace-record.c        | 13 ++++++--
 4 files changed, 97 insertions(+), 12 deletions(-)

diff --git a/tracecmd/include/trace-local.h b/tracecmd/include/trace-local.h
index 8c59595795a4..e3fec1319880 100644
--- a/tracecmd/include/trace-local.h
+++ b/tracecmd/include/trace-local.h
@@ -122,7 +122,7 @@ void trace_convert(int argc, char **argv);
 int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
 		       int cpus, int *fds,
 		       int argc, char **argv, bool use_fifos,
-		       unsigned long long trace_id);
+		       unsigned long long trace_id, const char *host);
 
 struct hook_list;
 
@@ -267,6 +267,7 @@ struct buffer_instance {
 
 	struct tracecmd_msg_handle *msg_handle;
 	struct tracecmd_output *network_handle;
+	const char		*host;
 
 	struct pid_addr_maps	*pid_maps;
 
@@ -309,9 +310,13 @@ extern struct buffer_instance *first_instance;
 #define START_PORT_SEARCH 1500
 #define MAX_PORT_SEARCH 6000
 
+struct sockaddr_storage;
+
 int trace_net_make(int port, enum port_type type);
 int trace_net_search(int start_port, int *sfd, enum port_type type);
 int trace_net_print_connection(int fd);
+bool trace_net_cmp_connection(struct sockaddr_storage *addr, const char *name);
+bool trace_net_cmp_connection_fd(int fd, const char *name);
 
 struct buffer_instance *allocate_instance(const char *name);
 void add_instance(struct buffer_instance *instance, int cpu_count);
diff --git a/tracecmd/trace-agent.c b/tracecmd/trace-agent.c
index 59cecae770a6..f0723a6601a4 100644
--- a/tracecmd/trace-agent.c
+++ b/tracecmd/trace-agent.c
@@ -62,7 +62,8 @@ static void make_net(int nr, int *fds, unsigned int *ports)
 	}
 }
 
-static void make_sockets(int nr, int *fds, unsigned int *ports, bool network)
+static void make_sockets(int nr, int *fds, unsigned int *ports,
+			 const char * network)
 {
 	if (network)
 		return make_net(nr, fds, ports);
@@ -109,7 +110,7 @@ static char *get_clock(int argc, char **argv)
 	return NULL;
 }
 
-static void trace_print_connection(int fd, bool network)
+static void trace_print_connection(int fd, const char *network)
 {
 	int ret;
 
@@ -121,7 +122,7 @@ static void trace_print_connection(int fd, bool network)
 		tracecmd_debug("Could not print connection fd:%d\n", fd);
 }
 
-static void agent_handle(int sd, int nr_cpus, int page_size, bool network)
+static void agent_handle(int sd, int nr_cpus, int page_size, const char *network)
 {
 	struct tracecmd_tsync_protos *tsync_protos = NULL;
 	struct tracecmd_time_sync *tsync = NULL;
@@ -203,7 +204,7 @@ static void agent_handle(int sd, int nr_cpus, int page_size, bool network)
 		die("Failed to send trace response");
 
 	trace_record_agent(msg_handle, nr_cpus, fds, argc, argv,
-			   use_fifos, trace_id);
+			   use_fifos, trace_id, network);
 
 	if (tsync) {
 		tracecmd_tsync_with_host_stop(tsync);
@@ -248,14 +249,23 @@ static pid_t do_fork()
 	return fork();
 }
 
-static void agent_serve(unsigned int port, bool do_daemon, bool network)
+static void agent_serve(unsigned int port, bool do_daemon, const char *network)
 {
+	struct sockaddr_storage net_addr;
+	struct sockaddr *addr = NULL;
+	socklen_t *addr_len_p = NULL;
+	socklen_t addr_len = sizeof(net_addr);
 	int sd, cd, nr_cpus;
 	unsigned int cid;
 	pid_t pid;
 
 	signal(SIGCHLD, handle_sigchld);
 
+	if (network) {
+		addr = (struct sockaddr *)&net_addr;
+		addr_len_p = &addr_len;
+	}
+
 	nr_cpus = tracecmd_count_cpus();
 	page_size = getpagesize();
 
@@ -279,7 +289,7 @@ static void agent_serve(unsigned int port, bool do_daemon, bool network)
 		die("daemon");
 
 	for (;;) {
-		cd = accept(sd, NULL, NULL);
+		cd = accept(sd, addr, addr_len_p);
 		if (cd < 0) {
 			if (errno == EINTR)
 				continue;
@@ -288,6 +298,12 @@ static void agent_serve(unsigned int port, bool do_daemon, bool network)
 		if (tracecmd_get_debug())
 			trace_print_connection(cd, network);
 
+		if (network && !trace_net_cmp_connection(&net_addr, network)) {
+			dprint("Client does not match '%s'\n", network);
+			close(cd);
+			continue;
+		}
+
 		if (handler_pid)
 			goto busy;
 
@@ -314,7 +330,7 @@ void trace_agent(int argc, char **argv)
 {
 	bool do_daemon = false;
 	unsigned int port = TRACE_AGENT_DEFAULT_PORT;
-	bool network = false;
+	const char *network = NULL;
 
 	if (argc < 2)
 		usage(argv);
@@ -332,7 +348,7 @@ void trace_agent(int argc, char **argv)
 			{NULL, 0, NULL, 0}
 		};
 
-		c = getopt_long(argc-1, argv+1, "+hp:DN",
+		c = getopt_long(argc-1, argv+1, "+hp:DN:",
 				long_options, &option_index);
 		if (c == -1)
 			break;
@@ -341,7 +357,7 @@ void trace_agent(int argc, char **argv)
 			usage(argv);
 			break;
 		case 'N':
-			network = true;
+			network = optarg;
 			break;
 		case 'p':
 			port = atoi(optarg);
diff --git a/tracecmd/trace-listen.c b/tracecmd/trace-listen.c
index b7be761d032e..86d2b9e9deb2 100644
--- a/tracecmd/trace-listen.c
+++ b/tracecmd/trace-listen.c
@@ -756,6 +756,61 @@ static int do_fork(int cfd)
 	return 0;
 }
 
+bool trace_net_cmp_connection(struct sockaddr_storage *addr, const char *name)
+{
+	char host[NI_MAXHOST], nhost[NI_MAXHOST];
+	char service[NI_MAXSERV];
+	socklen_t addr_len = sizeof(*addr);
+	struct addrinfo *result, *rp;
+	struct addrinfo hints;
+	bool found = false;
+	int s;
+
+	if (getnameinfo((struct sockaddr *)addr, addr_len,
+			host, NI_MAXHOST,
+			service, NI_MAXSERV, NI_NUMERICSERV))
+		return -1;
+
+	if (strcmp(host, name) == 0)
+		return true;
+
+	/* Check other IPs that name could be for */
+
+	memset(&hints, 0, sizeof(hints));
+	hints.ai_family = AF_UNSPEC;
+	hints.ai_socktype = SOCK_STREAM;
+
+	/* Check other IPs that name could be for */
+	s = getaddrinfo(name, NULL, &hints, &result);
+	if (s != 0)
+		return false;
+
+	for (rp = result; rp != NULL; rp = rp->ai_next) {
+		if (getnameinfo(rp->ai_addr, rp->ai_addrlen,
+				nhost, NI_MAXHOST,
+				service, NI_MAXSERV, NI_NUMERICSERV))
+			continue;
+		if (strcmp(host, nhost) == 0) {
+			found = 1;
+			break;
+		}
+	}
+
+	freeaddrinfo(result);
+	return found;
+}
+
+bool trace_net_cmp_connection_fd(int fd, const char *name)
+{
+	struct sockaddr_storage addr;
+	socklen_t addr_len = sizeof(addr);
+
+	if (getpeername(fd, (struct sockaddr *)&addr, &addr_len))
+		return false;
+
+	return trace_net_cmp_connection(&addr, name);
+};
+
 int trace_net_print_connection(int fd)
 {
 	char host[NI_MAXHOST], service[NI_MAXSERV];
diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c
index 9c930920c89e..27c4e7ba6f3f 100644
--- a/tracecmd/trace-record.c
+++ b/tracecmd/trace-record.c
@@ -3426,8 +3426,16 @@ static int create_recorder(struct buffer_instance *instance, int cpu,
 		if (is_agent(instance)) {
 			if (instance->use_fifos)
 				fd = instance->fds[cpu];
-			else
+			else {
+ again:
 				fd = do_accept(instance->fds[cpu]);
+				if (instance->host &&
+				    !trace_net_cmp_connection_fd(fd, instance->host)) {
+					dprint("Client does not match '%s' for cpu:%d\n",
+					       instance->host, cpu);
+					goto again;
+				}
+			}
 		} else {
 			fd = connect_port(host, instance->client_ports[cpu],
 					  instance->port_type);
@@ -7275,7 +7283,7 @@ int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
 		       int cpus, int *fds,
 		       int argc, char **argv,
 		       bool use_fifos,
-		       unsigned long long trace_id)
+		       unsigned long long trace_id, const char *host)
 {
 	struct common_record_context ctx;
 	char **argv_plus;
@@ -7304,6 +7312,7 @@ int trace_record_agent(struct tracecmd_msg_handle *msg_handle,
 	ctx.instance->use_fifos = use_fifos;
 	ctx.instance->flags |= BUFFER_FL_AGENT;
 	ctx.instance->msg_handle = msg_handle;
+	ctx.instance->host = host;
 	msg_handle->version = V3_PROTOCOL;
 	top_instance.trace_id = trace_id;
 	record_trace(argc, argv, &ctx);
-- 
2.35.1

[PATCH v3 11/11] trace-cmd agent: Add documentation

[Steven Rostedt]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

Add a man page for trace-cmd agent, and also update the usage to include
the new -N option.

Link: https://lore.kernel.org/linux-trace-devel/20220417211315.1049221-3-rostedt@goodmis.org

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
 Documentation/trace-cmd/trace-cmd-agent.1.txt | 62 +++++++++++++++++++
 tracecmd/trace-usage.c                        |  3 +
 2 files changed, 65 insertions(+)
 create mode 100644 Documentation/trace-cmd/trace-cmd-agent.1.txt

diff --git a/Documentation/trace-cmd/trace-cmd-agent.1.txt b/Documentation/trace-cmd/trace-cmd-agent.1.txt
new file mode 100644
index 000000000000..f247d41dcdb8
--- /dev/null
+++ b/Documentation/trace-cmd/trace-cmd-agent.1.txt
@@ -0,0 +1,62 @@
+TRACE-CMD-AGENT(1)
+==================
+
+NAME
+----
+trace-cmd-agent - Run as an agent on a machine (to be controlled by another machine)
+
+SYNOPSIS
+--------
+*trace-cmd agent* ['OPTIONS']
+
+DESCRIPTION
+-----------
+The trace-cmd(1) agent listens over a vsocket (for virtual machines) or a TCP port
+for connections to control the tracing of the machine. The agent will then start
+tracing on the local machine and pass the data to the controlling connection.
+
+OPTIONS
+-------
+*-N* 'client'::
+    Listen over TCP instead of a vsocket. Must pass in a client host name or IP address
+    to allow connection to. It will only connect to the specified client. Note, any process
+    on that client can control the agent.
+
+    *This is a very insecure setting. Only use on a trusted network*
+    *Only use if the client machine is totally trusted*
+
+*-p* 'port'::
+    This option will specify the port to listen to.
+
+*-D*::
+    This options causes trace-cmd agent to go into a daemon mode and run in
+    the background.
+
+*--verbose*[='level']::
+     Set the log level. Supported log levels are "none", "critical", "error", "warning",
+     "info", "debug", "all" or their identifiers "0", "1", "2", "3", "4", "5", "6". Setting the log
+     level to specific value enables all logs from that and all previous levels.
+     The level will default to "info" if one is not specified.
+
+     Example: enable all critical, error and warning logs
+
+      trace-cmd listen --verbose=warning
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
+trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1),
+trace-cmd-split(1), trace-cmd-list(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c
index 88eac10a4e75..2cfa64f5aa0c 100644
--- a/tracecmd/trace-usage.c
+++ b/tracecmd/trace-usage.c
@@ -330,6 +330,9 @@ static struct usage_help usage_help[] = {
 		"listen on a vsocket for trace clients",
 		" %s agent -p port[-D]\n"
 		"          Creates a vsocket to listen for clients.\n"
+		"          -N Connect to IP via TCP instead of vsockets\n"
+		"             *** Insecure setting, only use on a trusted network ***\n"
+		"             ***   Only use if the client is totally trusted.    ***\n"
 		"          -p port number to listen on.\n"
 		"          -D run in daemon mode.\n"
 		"          --verbose 'level' Set the desired log level\n"
-- 
2.35.1