From: Stefano Garzarella <sgarzare@redhat.com> To: Steven Rostedt <rostedt@goodmis.org> Cc: LKML <linux-kernel@vger.kernel.org>, Stefan Hajnoczi <stefanha@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>, Jason Wang <jasowang@redhat.com>, "David S. Miller" <davem@davemloft.net>, Jakub Kicinski <kuba@kernel.org>, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, Joel Fernandes <joelaf@google.com>, Linux Trace Devel <linux-trace-devel@vger.kernel.org> Subject: Re: [RFC][PATCH] vhost/vsock: Add vsock_list file to map cid with vhost tasks Date: Fri, 7 May 2021 16:11:20 +0200 [thread overview] Message-ID: <20210507141120.ot6xztl4h5zyav2c@steredhat> (raw) In-Reply-To: <20210505163855.32dad8e7@gandalf.local.home> Hi Steven, On Wed, May 05, 2021 at 04:38:55PM -0400, Steven Rostedt wrote: >The new trace-cmd 3.0 (which is almost ready to be released) allows for >tracing between host and guests with timestamp synchronization such that >the events on the host and the guest can be interleaved in the proper order >that they occur. KernelShark now has a plugin that visualizes this >interaction. > >The implementation requires that the guest has a vsock CID assigned, and on >the guest a "trace-cmd agent" is running, that will listen on a port for >the CID. The on the host a "trace-cmd record -A guest@cid:port -e events" >can be called and the host will connect to the guest agent through the >cid/port pair and have the agent enable tracing on behalf of the host and >send the trace data back down to it. > >The problem is that there is no sure fire way to find the CID for a guest. >Currently, the user must know the cid, or we have a hack that looks for the >qemu process and parses the --guest-cid parameter from it. But this is >prone to error and does not work on other implementation (was told that >crosvm does not use qemu). For debug I think could be useful to link the vhost-vsock kthread to the CID, but for the user point of view, maybe is better to query the VM management layer, for example if you're using libvirt, you can easily do: $ virsh dumpxml fedora34 | grep cid <cid auto='yes' address='3'/> > >As I can not find a way to discover CIDs assigned to guests via any kernel >interface, I decided to create this one. Note, I'm not attached to it. If >there's a better way to do this, I would love to have it. But since I'm not >an expert in the networking layer nor virtio, I decided to stick to what I >know and add a debugfs interface that simply lists all the registered >CIDs >and the worker task that they are associated with. The worker task at >least has the PID of the task it represents. I honestly don't know if it's the best interface, like I said maybe for debugging it's fine, but if we want to expose it to the user in some way, we could support devlink/netlink to provide information about the vsock devices currently in use. > >Now I can find the cid / host process in charge of the guest pair: > > # cat /sys/kernel/debug/vsock_list > 3 vhost-1954:2002 > > # ps aux | grep 1954 > qemu 1954 9.9 21.3 1629092 796148 ? Sl 16:22 0:58 /usr/bin/qemu-kvm -name guest=Fedora21,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-Fedora21/master-key.aes -machine pc-1.2,accel=kvm,usb=off,dump-guest-core=off -cpu qemu64 -m 1000 -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 1eefeeb0-3ac7-07c1-926e-236908313b4c -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=32,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -blockdev {"driver":"host_device","filename":"/dev/mapper/vg_bxtest-GuestFedora","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"} -device ide-hd,bus=ide.0,unit=0,drive=libvirt-1- > format,id=ide0-0-0,bootindex=1 -netdev tap,fd=34,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:9f:e9:d5,bus=pci.0,addr=0x3 -netdev tap,fd=35,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ec:dc:6e,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev pipe,id=charchannel0,path=/var/lib/trace-cmd/virt/Fedora21/trace-pipe-cpu0 -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=trace-pipe-cpu0 -chardev pipe,id=charchannel1,path=/var/lib/trace-cmd/virt/Fedora21/trace-pipe-cpu1 -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=trace-pipe-cpu1 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -device vhost-vsock-pci,id=vsock0,guest-cid=3,vhostfd=16,bus=pci.0,addr=0x7 -msg > timestamp=on > root 2000 0.0 0.0 0 0 ? S 16:22 0:00 [kvm-pit/1954] > root 2002 0.0 0.0 0 0 ? S 16:22 0:00 [vhost-1954] > > >This is just an example of what I'm looking for. Just a way to find what >process is using what cid. > >Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org> >--- >diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c >index 5e78fb719602..4f03b25b23c1 100644 >--- a/drivers/vhost/vsock.c >+++ b/drivers/vhost/vsock.c >@@ -15,6 +15,7 @@ > #include <linux/virtio_vsock.h> > #include <linux/vhost.h> > #include <linux/hashtable.h> >+#include <linux/debugfs.h> > > #include <net/af_vsock.h> > #include "vhost.h" >@@ -900,6 +901,128 @@ static struct miscdevice vhost_vsock_misc = { > .fops = &vhost_vsock_fops, > }; > >+static struct dentry *vsock_file; >+ >+struct vsock_file_iter { >+ struct hlist_node *node; >+ int index; >+}; >+ >+ >+static void *vsock_next(struct seq_file *m, void *v, loff_t *pos) >+{ >+ struct vsock_file_iter *iter = v; >+ struct vhost_vsock *vsock; >+ >+ if (pos) >+ (*pos)++; >+ >+ if (iter->index >= (int)HASH_SIZE(vhost_vsock_hash)) >+ return NULL; >+ >+ if (iter->node) >+ iter->node = rcu_dereference_raw(hlist_next_rcu(iter->node)); >+ >+ for (;;) { >+ if (iter->node) { >+ vsock = hlist_entry_safe(rcu_dereference_raw(iter->node), >+ struct vhost_vsock, hash); >+ if (vsock->guest_cid) >+ break; >+ iter->node = rcu_dereference_raw(hlist_next_rcu(iter->node)); >+ continue; >+ } >+ iter->index++; >+ if (iter->index >= HASH_SIZE(vhost_vsock_hash)) >+ return NULL; >+ >+ iter->node = rcu_dereference_raw(hlist_first_rcu(&vhost_vsock_hash[iter->index])); >+ } >+ return iter; >+} >+ >+static void *vsock_start(struct seq_file *m, loff_t *pos) >+{ >+ struct vsock_file_iter *iter = m->private; >+ loff_t l = 0; >+ void *t; >+ >+ rcu_read_lock(); Instead of keeping this rcu lock between vsock_start() and vsock_stop(), maybe it's better to make a dump here of the bindings (pid/cid), save it in an array, and iterate it in vsock_next(). >+ >+ iter->index = -1; >+ iter->node = NULL; >+ t = vsock_next(m, iter, NULL); >+ >+ for (; iter->index < HASH_SIZE(vhost_vsock_hash) && l < *pos; >+ t = vsock_next(m, iter, &l)) >+ ; A while() maybe was more readable... Thanks, Stefano
WARNING: multiple messages have this Message-ID (diff)
From: Stefano Garzarella <sgarzare@redhat.com> To: Steven Rostedt <rostedt@goodmis.org> Cc: Joel Fernandes <joelaf@google.com>, kvm@vger.kernel.org, "Michael S. Tsirkin" <mst@redhat.com>, netdev@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>, virtualization@lists.linux-foundation.org, Linux Trace Devel <linux-trace-devel@vger.kernel.org>, Stefan Hajnoczi <stefanha@redhat.com>, Jakub Kicinski <kuba@kernel.org>, "David S. Miller" <davem@davemloft.net> Subject: Re: [RFC][PATCH] vhost/vsock: Add vsock_list file to map cid with vhost tasks Date: Fri, 7 May 2021 16:11:20 +0200 [thread overview] Message-ID: <20210507141120.ot6xztl4h5zyav2c@steredhat> (raw) In-Reply-To: <20210505163855.32dad8e7@gandalf.local.home> Hi Steven, On Wed, May 05, 2021 at 04:38:55PM -0400, Steven Rostedt wrote: >The new trace-cmd 3.0 (which is almost ready to be released) allows for >tracing between host and guests with timestamp synchronization such that >the events on the host and the guest can be interleaved in the proper order >that they occur. KernelShark now has a plugin that visualizes this >interaction. > >The implementation requires that the guest has a vsock CID assigned, and on >the guest a "trace-cmd agent" is running, that will listen on a port for >the CID. The on the host a "trace-cmd record -A guest@cid:port -e events" >can be called and the host will connect to the guest agent through the >cid/port pair and have the agent enable tracing on behalf of the host and >send the trace data back down to it. > >The problem is that there is no sure fire way to find the CID for a guest. >Currently, the user must know the cid, or we have a hack that looks for the >qemu process and parses the --guest-cid parameter from it. But this is >prone to error and does not work on other implementation (was told that >crosvm does not use qemu). For debug I think could be useful to link the vhost-vsock kthread to the CID, but for the user point of view, maybe is better to query the VM management layer, for example if you're using libvirt, you can easily do: $ virsh dumpxml fedora34 | grep cid <cid auto='yes' address='3'/> > >As I can not find a way to discover CIDs assigned to guests via any kernel >interface, I decided to create this one. Note, I'm not attached to it. If >there's a better way to do this, I would love to have it. But since I'm not >an expert in the networking layer nor virtio, I decided to stick to what I >know and add a debugfs interface that simply lists all the registered >CIDs >and the worker task that they are associated with. The worker task at >least has the PID of the task it represents. I honestly don't know if it's the best interface, like I said maybe for debugging it's fine, but if we want to expose it to the user in some way, we could support devlink/netlink to provide information about the vsock devices currently in use. > >Now I can find the cid / host process in charge of the guest pair: > > # cat /sys/kernel/debug/vsock_list > 3 vhost-1954:2002 > > # ps aux | grep 1954 > qemu 1954 9.9 21.3 1629092 796148 ? Sl 16:22 0:58 /usr/bin/qemu-kvm -name guest=Fedora21,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-Fedora21/master-key.aes -machine pc-1.2,accel=kvm,usb=off,dump-guest-core=off -cpu qemu64 -m 1000 -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 1eefeeb0-3ac7-07c1-926e-236908313b4c -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=32,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -blockdev {"driver":"host_device","filename":"/dev/mapper/vg_bxtest-GuestFedora","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"} -device ide-hd,bus=ide.0,unit=0,drive=libvirt-1 - > format,id=ide0-0-0,bootindex=1 -netdev tap,fd=34,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:9f:e9:d5,bus=pci.0,addr=0x3 -netdev tap,fd=35,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ec:dc:6e,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev pipe,id=charchannel0,path=/var/lib/trace-cmd/virt/Fedora21/trace-pipe-cpu0 -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=trace-pipe-cpu0 -chardev pipe,id=charchannel1,path=/var/lib/trace-cmd/virt/Fedora21/trace-pipe-cpu1 -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=trace-pipe-cpu1 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -device vhost-vsock-pci,id=vsock0,guest-cid=3,vhostfd=16,bus=pci.0,addr=0x7 -msg > timestamp=on > root 2000 0.0 0.0 0 0 ? S 16:22 0:00 [kvm-pit/1954] > root 2002 0.0 0.0 0 0 ? S 16:22 0:00 [vhost-1954] > > >This is just an example of what I'm looking for. Just a way to find what >process is using what cid. > >Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org> >--- >diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c >index 5e78fb719602..4f03b25b23c1 100644 >--- a/drivers/vhost/vsock.c >+++ b/drivers/vhost/vsock.c >@@ -15,6 +15,7 @@ > #include <linux/virtio_vsock.h> > #include <linux/vhost.h> > #include <linux/hashtable.h> >+#include <linux/debugfs.h> > > #include <net/af_vsock.h> > #include "vhost.h" >@@ -900,6 +901,128 @@ static struct miscdevice vhost_vsock_misc = { > .fops = &vhost_vsock_fops, > }; > >+static struct dentry *vsock_file; >+ >+struct vsock_file_iter { >+ struct hlist_node *node; >+ int index; >+}; >+ >+ >+static void *vsock_next(struct seq_file *m, void *v, loff_t *pos) >+{ >+ struct vsock_file_iter *iter = v; >+ struct vhost_vsock *vsock; >+ >+ if (pos) >+ (*pos)++; >+ >+ if (iter->index >= (int)HASH_SIZE(vhost_vsock_hash)) >+ return NULL; >+ >+ if (iter->node) >+ iter->node = rcu_dereference_raw(hlist_next_rcu(iter->node)); >+ >+ for (;;) { >+ if (iter->node) { >+ vsock = hlist_entry_safe(rcu_dereference_raw(iter->node), >+ struct vhost_vsock, hash); >+ if (vsock->guest_cid) >+ break; >+ iter->node = rcu_dereference_raw(hlist_next_rcu(iter->node)); >+ continue; >+ } >+ iter->index++; >+ if (iter->index >= HASH_SIZE(vhost_vsock_hash)) >+ return NULL; >+ >+ iter->node = rcu_dereference_raw(hlist_first_rcu(&vhost_vsock_hash[iter->index])); >+ } >+ return iter; >+} >+ >+static void *vsock_start(struct seq_file *m, loff_t *pos) >+{ >+ struct vsock_file_iter *iter = m->private; >+ loff_t l = 0; >+ void *t; >+ >+ rcu_read_lock(); Instead of keeping this rcu lock between vsock_start() and vsock_stop(), maybe it's better to make a dump here of the bindings (pid/cid), save it in an array, and iterate it in vsock_next(). >+ >+ iter->index = -1; >+ iter->node = NULL; >+ t = vsock_next(m, iter, NULL); >+ >+ for (; iter->index < HASH_SIZE(vhost_vsock_hash) && l < *pos; >+ t = vsock_next(m, iter, &l)) >+ ; A while() maybe was more readable... Thanks, Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2021-05-07 14:11 UTC|newest] Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-05-05 20:38 [RFC][PATCH] vhost/vsock: Add vsock_list file to map cid with vhost tasks Steven Rostedt 2021-05-05 20:38 ` Steven Rostedt 2021-05-06 1:03 ` Steven Rostedt 2021-05-06 1:03 ` Steven Rostedt 2021-05-07 14:11 ` Stefano Garzarella [this message] 2021-05-07 14:11 ` Stefano Garzarella 2021-05-07 14:40 ` Steven Rostedt 2021-05-07 14:40 ` Steven Rostedt 2021-05-07 15:43 ` Stefano Garzarella 2021-05-07 15:43 ` Stefano Garzarella 2021-05-07 16:09 ` Steven Rostedt 2021-05-07 16:09 ` Steven Rostedt 2021-05-08 18:32 ` Mike Christie 2021-05-08 18:32 ` Mike Christie 2021-05-13 15:57 ` Stefan Hajnoczi 2021-05-13 15:57 ` Stefan Hajnoczi 2021-05-13 16:08 ` Steven Rostedt 2021-05-13 16:08 ` Steven Rostedt 2021-05-06 1:54 kernel test robot
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210507141120.ot6xztl4h5zyav2c@steredhat \ --to=sgarzare@redhat.com \ --cc=davem@davemloft.net \ --cc=jasowang@redhat.com \ --cc=joelaf@google.com \ --cc=kuba@kernel.org \ --cc=kvm@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-trace-devel@vger.kernel.org \ --cc=mst@redhat.com \ --cc=netdev@vger.kernel.org \ --cc=rostedt@goodmis.org \ --cc=stefanha@redhat.com \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.