Shared value between host and guests

Shared value between host and guests

[Genevieve Bastien]

Hi,

I'm tracing kvm guests and their host and an issue we have is to
identify which host process corresponds to which guest trace so we can
follow the state of the virtual cpu on the host.

For our needs, we fixed this problem by adding a kernel module on host
and guest where hosts and guests make an hypercall received by the other
and sending a UID between them. By tracing those hypercalls and tracking
the uid value, we can know which thread on the host corresponds to which
guest machine.

But we would like to achieve this without having to add an additional
module for tracing purposes. I'm wondering if there's already a value
somewhere in the kernel that is uniquely shared between the host
process/threads and the corresponding guest. And if not, would it be
possible/desirable to add one? For instance, the bootid of the guest
could be sent to the host process at the start and stored in some system
file. The trace could then retrieve it and the analysis can figure out
who's who all the traces.

Fyi, this is the kind of virtual machine analyses we can obtain by
tracing guest and host:
http://archive.eclipse.org/tracecompass.incubator/doc/org.eclipse.tracecompass.incubator.virtual.machine.analysis.doc.user/Virtual-Machine-Analysis-User-Documentation.html 
We have more, but it is not yet documented unfortunately.

Thanks,
Geneviève

Re: Shared value between host and guests

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

On Fri, Jan 19, 2018 at 03:03:25PM -0500, Genevieve Bastien wrote:
> I'm tracing kvm guests and their host and an issue we have is to
> identify which host process corresponds to which guest trace so we can
> follow the state of the virtual cpu on the host.

There are uevents for the VM lifecycle that include the PID.  Please see
Linux commit 286de8f6ac9202f1c9012784639156c6ec386eb8 &
fdeaf7e3eb37c6dbc4b4ac97dbe1945d239eb788.

Did you check if these are appropriate for your use case?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: Shared value between host and guests

[Geneviève Bastien]

On 2018-01-23 12:21 PM, Stefan Hajnoczi wrote:
> On Fri, Jan 19, 2018 at 03:03:25PM -0500, Genevieve Bastien wrote:
>> I'm tracing kvm guests and their host and an issue we have is to
>> identify which host process corresponds to which guest trace so we can
>> follow the state of the virtual cpu on the host.
> There are uevents for the VM lifecycle that include the PID.  Please see
> Linux commit 286de8f6ac9202f1c9012784639156c6ec386eb8 &
> fdeaf7e3eb37c6dbc4b4ac97dbe1945d239eb788.
>
> Did you check if these are appropriate for your use case?
Thanks for the hint, I didn't know about those. But the uevents are only
for the host right? There is no specific value in debugfs for guest.

But I'll see if I can use some udev rules to trigger
<some-kvm-tracing-script> to achieve what I need.

Thanks,
Geneviève

Re: Shared value between host and guests

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote:
> On 2018-01-23 12:21 PM, Stefan Hajnoczi wrote:
> > On Fri, Jan 19, 2018 at 03:03:25PM -0500, Genevieve Bastien wrote:
> >> I'm tracing kvm guests and their host and an issue we have is to
> >> identify which host process corresponds to which guest trace so we can
> >> follow the state of the virtual cpu on the host.
> > There are uevents for the VM lifecycle that include the PID.  Please see
> > Linux commit 286de8f6ac9202f1c9012784639156c6ec386eb8 &
> > fdeaf7e3eb37c6dbc4b4ac97dbe1945d239eb788.
> >
> > Did you check if these are appropriate for your use case?
> Thanks for the hint, I didn't know about those. But the uevents are only
> for the host right? There is no specific value in debugfs for guest.

I'm not sure I understand the question.

Uevent are emitted on the host.  Given the PID field, it should be
possible to correlate them to a specific guest (e.g. by looking at
information from the user or VM management tools, or simply by looking
at /proc/$PID/cmdline for the QEMU -name argument that can be used to
identify guests on the host side).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: Shared value between host and guests

[Geneviève Bastien]

On 2018-01-25 04:29 AM, Stefan Hajnoczi wrote:
> On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote:
>> Thanks for the hint, I didn't know about those. But the uevents are only
>> for the host right? There is no specific value in debugfs for guest.
> I'm not sure I understand the question.
>
> Uevent are emitted on the host.  Given the PID field, it should be
> possible to correlate them to a specific guest (e.g. by looking at
> information from the user or VM management tools, or simply by looking
> at /proc/$PID/cmdline for the QEMU -name argument that can be used to
> identify guests on the host side).
>
> Stefan
With the information from the host and those uevents, we can indeed
easily figure out in our traces which processes are associated with a
guest and get its command line and a lot more information.

The missing piece is in the guest trace. For example, we have lttng
traces taken on the guest and the host. We may have multiple guests and
multiple hosts to trace. It's easy to know which traces are from hosts
from the events, but which are from guests? And which guest trace goes
with which of the kvm processes on which host? The guest trace contains
no information that can be linked to the host.

I was thinking that the guest could do a hypercall to the host after
bootup to share some unique information, for instance its bootid, that
the host could store somewhere. That information would be available in
the traces so that we can easily associate the guest with its host
process and the states of its virtual CPUs to that of the corresponding
threads, and much more.

I hope this describes our need a little better.

Thanks,
Geneviève (tahini on irc)

Re: Shared value between host and guests

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2836 bytes --]

On Thu, Jan 25, 2018 at 10:44:12AM -0500, Geneviève Bastien wrote:
> On 2018-01-25 04:29 AM, Stefan Hajnoczi wrote:
> > On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote:
> >> Thanks for the hint, I didn't know about those. But the uevents are only
> >> for the host right? There is no specific value in debugfs for guest.
> > I'm not sure I understand the question.
> >
> > Uevent are emitted on the host.  Given the PID field, it should be
> > possible to correlate them to a specific guest (e.g. by looking at
> > information from the user or VM management tools, or simply by looking
> > at /proc/$PID/cmdline for the QEMU -name argument that can be used to
> > identify guests on the host side).
> >
> > Stefan
> With the information from the host and those uevents, we can indeed
> easily figure out in our traces which processes are associated with a
> guest and get its command line and a lot more information.
> 
> The missing piece is in the guest trace. For example, we have lttng
> traces taken on the guest and the host. We may have multiple guests and
> multiple hosts to trace. It's easy to know which traces are from hosts
> from the events, but which are from guests? And which guest trace goes
> with which of the kvm processes on which host? The guest trace contains
> no information that can be linked to the host.
> 
> I was thinking that the guest could do a hypercall to the host after
> bootup to share some unique information, for instance its bootid, that
> the host could store somewhere. That information would be available in
> the traces so that we can easily associate the guest with its host
> process and the states of its virtual CPUs to that of the corresponding
> threads, and much more.
> 
> I hope this describes our need a little better.

The QEMU -uuid <uuid> option makes a UUID available to the guest via
SMBIOS and fw_cfg on x86.  Inside the guest you can print it like this:

  # dmidecode -s system-uuid
  01020304-0506-0708-090A-0B0C0D0E0F10

Maybe you can base the guest trace filename off the UUID:

  guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat

On the host you can either find the UUIDs in the libvirt domain XML:

  # virsh dump my-domain
  <domain ...>
    <uuid>0102030405060708090a0b0c0d0e0f10</uuid>
    ...

Or you can use the kvm.ko uevent to find the QEMU PID and then check
/proc/$PID/cmdline for the uuid.  The UUID can also be fetched via the
query-uuid QMP command if you don't want to search /proc/$PID/cmdline
for -uuid <uuid>.

Maybe you can base the host trace filename off the UUID too:

  host-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat

Then the ask of correlating traces becomes pretty easy for
post-processing scripts since they can look at the filenames.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: Shared value between host and guests

[Wanpeng Li]

2018-01-26 22:03 GMT+08:00 Stefan Hajnoczi <stefanha@gmail.com>:
> On Thu, Jan 25, 2018 at 10:44:12AM -0500, Geneviève Bastien wrote:
>> On 2018-01-25 04:29 AM, Stefan Hajnoczi wrote:
>> > On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote:
>> >> Thanks for the hint, I didn't know about those. But the uevents are only
>> >> for the host right? There is no specific value in debugfs for guest.
>> > I'm not sure I understand the question.
>> >
>> > Uevent are emitted on the host.  Given the PID field, it should be
>> > possible to correlate them to a specific guest (e.g. by looking at
>> > information from the user or VM management tools, or simply by looking
>> > at /proc/$PID/cmdline for the QEMU -name argument that can be used to
>> > identify guests on the host side).
>> >
>> > Stefan
>> With the information from the host and those uevents, we can indeed
>> easily figure out in our traces which processes are associated with a
>> guest and get its command line and a lot more information.
>>
>> The missing piece is in the guest trace. For example, we have lttng
>> traces taken on the guest and the host. We may have multiple guests and
>> multiple hosts to trace. It's easy to know which traces are from hosts
>> from the events, but which are from guests? And which guest trace goes
>> with which of the kvm processes on which host? The guest trace contains
>> no information that can be linked to the host.
>>
>> I was thinking that the guest could do a hypercall to the host after
>> bootup to share some unique information, for instance its bootid, that
>> the host could store somewhere. That information would be available in
>> the traces so that we can easily associate the guest with its host
>> process and the states of its virtual CPUs to that of the corresponding
>> threads, and much more.
>>
>> I hope this describes our need a little better.
>
> The QEMU -uuid <uuid> option makes a UUID available to the guest via
> SMBIOS and fw_cfg on x86.  Inside the guest you can print it like this:
>
>   # dmidecode -s system-uuid
>   01020304-0506-0708-090A-0B0C0D0E0F10
>
> Maybe you can base the guest trace filename off the UUID:
>
>   guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
>
> On the host you can either find the UUIDs in the libvirt domain XML:
>
>   # virsh dump my-domain
>   <domain ...>
>     <uuid>0102030405060708090a0b0c0d0e0f10</uuid>
>     ...
>
> Or you can use the kvm.ko uevent to find the QEMU PID and then check

How to use the kvm.ko uevent to find the QEMU PID?

Regards,
Wanpeng Li

Re: Shared value between host and guests

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3034 bytes --]

On Mon, Jan 29, 2018 at 08:59:16AM +0800, Wanpeng Li wrote:
> 2018-01-26 22:03 GMT+08:00 Stefan Hajnoczi <stefanha@gmail.com>:
> > On Thu, Jan 25, 2018 at 10:44:12AM -0500, Geneviève Bastien wrote:
> >> On 2018-01-25 04:29 AM, Stefan Hajnoczi wrote:
> >> > On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote:
> >> >> Thanks for the hint, I didn't know about those. But the uevents are only
> >> >> for the host right? There is no specific value in debugfs for guest.
> >> > I'm not sure I understand the question.
> >> >
> >> > Uevent are emitted on the host.  Given the PID field, it should be
> >> > possible to correlate them to a specific guest (e.g. by looking at
> >> > information from the user or VM management tools, or simply by looking
> >> > at /proc/$PID/cmdline for the QEMU -name argument that can be used to
> >> > identify guests on the host side).
> >> >
> >> > Stefan
> >> With the information from the host and those uevents, we can indeed
> >> easily figure out in our traces which processes are associated with a
> >> guest and get its command line and a lot more information.
> >>
> >> The missing piece is in the guest trace. For example, we have lttng
> >> traces taken on the guest and the host. We may have multiple guests and
> >> multiple hosts to trace. It's easy to know which traces are from hosts
> >> from the events, but which are from guests? And which guest trace goes
> >> with which of the kvm processes on which host? The guest trace contains
> >> no information that can be linked to the host.
> >>
> >> I was thinking that the guest could do a hypercall to the host after
> >> bootup to share some unique information, for instance its bootid, that
> >> the host could store somewhere. That information would be available in
> >> the traces so that we can easily associate the guest with its host
> >> process and the states of its virtual CPUs to that of the corresponding
> >> threads, and much more.
> >>
> >> I hope this describes our need a little better.
> >
> > The QEMU -uuid <uuid> option makes a UUID available to the guest via
> > SMBIOS and fw_cfg on x86.  Inside the guest you can print it like this:
> >
> >   # dmidecode -s system-uuid
> >   01020304-0506-0708-090A-0B0C0D0E0F10
> >
> > Maybe you can base the guest trace filename off the UUID:
> >
> >   guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
> >
> > On the host you can either find the UUIDs in the libvirt domain XML:
> >
> >   # virsh dump my-domain
> >   <domain ...>
> >     <uuid>0102030405060708090a0b0c0d0e0f10</uuid>
> >     ...
> >
> > Or you can use the kvm.ko uevent to find the QEMU PID and then check
> 
> How to use the kvm.ko uevent to find the QEMU PID?

Add a udev rule on the host for kvm.ko EVENT=create/destroy events.
These events have a PID=<pid> field that a script can use to record the
QEMU instance that is associated with this guest.

Please see virt/kvm/kvm_main.c:kvm_uevent_notify_change().

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

RFC: lttng-module: how to track the link between host and guest traces

[Genevieve Bastien]

[-- Attachment #1.1: Type: text/plain, Size: 4250 bytes --]

Hi all,


Here included is a conversation I had on the kvm mailing list regarding
how to link traces from host and guests when tracing virtual
environments. Following Stefan's last suggestions, here is a suggestion
on what can be done in lttng-modules, it's not intrusive and does not
require modifications to the kernel. Let me know what you think:


* If we want to trace guests _and_ host and linek them together, we
require to set the VM's -uuid parameter. I think virsh does that by
default anyway.

    - We can then obtain this uuid in lttng via
dmi_get_system_info(DMI_PRODUCT_UUID). We can add it to a statedump
event, or in the 'env' section of the metadata (which might be easier to
manager and does not require activating an additional event).

* Guest statedump: on host, we can access the guests by parsing the
struct dentry* kvm_debugfs_dir for entries with the <pid>-xx format. We
can then parse the /proc/<pid>/cmdline to obtain the uuid and associate
it to the guest.

* Guest lifetime during the trace: We can add a lttng-module that adds a
udev rule to intercept VM creation/destruction. We can trace those
events, adding the pid/uuid in the fields.


What do you think of this approach?


Thanks,

Geneviève



-------- Forwarded Message --------
Subject: 	Re: Shared value between host and guests
Date: 	Fri, 26 Jan 2018 14:03:48 +0000
From: 	Stefan Hajnoczi <stefanha@gmail.com>
To: 	Geneviève Bastien <gbastien@versatic.net>
CC: 	kvm@vger.kernel.org



On Thu, Jan 25, 2018 at 10:44:12AM -0500, Geneviève Bastien wrote:
> On 2018-01-25 04:29 AM, Stefan Hajnoczi wrote:
> > On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote:
> >> Thanks for the hint, I didn't know about those. But the uevents are only
> >> for the host right? There is no specific value in debugfs for guest.
> > I'm not sure I understand the question.
> >
> > Uevent are emitted on the host.  Given the PID field, it should be
> > possible to correlate them to a specific guest (e.g. by looking at
> > information from the user or VM management tools, or simply by looking
> > at /proc/$PID/cmdline for the QEMU -name argument that can be used to
> > identify guests on the host side).
> >
> > Stefan
> With the information from the host and those uevents, we can indeed
> easily figure out in our traces which processes are associated with a
> guest and get its command line and a lot more information.
> 
> The missing piece is in the guest trace. For example, we have lttng
> traces taken on the guest and the host. We may have multiple guests and
> multiple hosts to trace. It's easy to know which traces are from hosts
> from the events, but which are from guests? And which guest trace goes
> with which of the kvm processes on which host? The guest trace contains
> no information that can be linked to the host.
> 
> I was thinking that the guest could do a hypercall to the host after
> bootup to share some unique information, for instance its bootid, that
> the host could store somewhere. That information would be available in
> the traces so that we can easily associate the guest with its host
> process and the states of its virtual CPUs to that of the corresponding
> threads, and much more.
> 
> I hope this describes our need a little better.

The QEMU -uuid <uuid> option makes a UUID available to the guest via
SMBIOS and fw_cfg on x86.  Inside the guest you can print it like this:

  # dmidecode -s system-uuid
  01020304-0506-0708-090A-0B0C0D0E0F10

Maybe you can base the guest trace filename off the UUID:

  guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat

On the host you can either find the UUIDs in the libvirt domain XML:

  # virsh dump my-domain
  <domain ...>
    <uuid>0102030405060708090a0b0c0d0e0f10</uuid>
    ...

Or you can use the kvm.ko uevent to find the QEMU PID and then check
/proc/$PID/cmdline for the uuid.  The UUID can also be fetched via the
query-uuid QMP command if you don't want to search /proc/$PID/cmdline
for -uuid <uuid>.

Maybe you can base the host trace filename off the UUID too:

  host-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat

Then the ask of correlating traces becomes pretty easy for
post-processing scripts since they can look at the filenames.

Stefan


[-- Attachment #1.2: Type: text/html, Size: 5973 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Shared value between host and guests

[Geneviève Bastien]

On 2018-01-26 09:03 AM, Stefan Hajnoczi wrote:
>
> The QEMU -uuid <uuid> option makes a UUID available to the guest via
> SMBIOS and fw_cfg on x86.  Inside the guest you can print it like this:
>
>   # dmidecode -s system-uuid
>   01020304-0506-0708-090A-0B0C0D0E0F10
>
> Maybe you can base the guest trace filename off the UUID:
>
>   guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
>
> On the host you can either find the UUIDs in the libvirt domain XML:
>
>   # virsh dump my-domain
>   <domain ...>
>     <uuid>0102030405060708090a0b0c0d0e0f10</uuid>
>     ...
>
> Or you can use the kvm.ko uevent to find the QEMU PID and then check
> /proc/$PID/cmdline for the uuid.  The UUID can also be fetched via the
> query-uuid QMP command if you don't want to search /proc/$PID/cmdline
> for -uuid <uuid>.
>
> Maybe you can base the host trace filename off the UUID too:
>
>   host-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat
>
> Then the ask of correlating traces becomes pretty easy for
> post-processing scripts since they can look at the filenames.
>
> Stefan
I think this approach would work. Requiring users to set a uuid to their
VM if they plan on tracing it doesn't seem too bad a requirement.

Thanks for your help,
Geneviève