* [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single
@ 2022-12-06 12:58 chenxiang
2022-12-06 12:58 ` [RFC PATCH 1/2] KVM: debugfs: Add vcpu debugfs to record statstical data for every single vcpu chenxiang
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: chenxiang @ 2022-12-06 12:58 UTC (permalink / raw)
To: pbonzini, maz, james.morse; +Cc: kvm, linuxarm, Xiang Chen
From: Xiang Chen <chenxiang66@hisilicon.com>
Currently it only records statistical data for all vcpus, but we ofen want
to know statistical data for a single vcpu, there is no debugfs for that.
So add vcpu debugfs to record statstical data for every single vcpu, and
also enable vcpu debugfs for arm64.
After the change, those vcpu debugfs are as follows (we have 4 vcpu in the
vm):
[root@centos kvm]# cd 2025-14/
[root@centos 2025-14]# ls
blocking halt_wait_hist vcpu0
exits halt_wait_ns vcpu1
halt_attempted_poll halt_wakeup vcpu2
halt_poll_fail_hist hvc_exit_stat vcpu3
halt_poll_fail_ns mmio_exit_kernel vgic-state
halt_poll_invalid mmio_exit_user wfe_exit_stat
halt_poll_success_hist remote_tlb_flush wfi_exit_stat
halt_poll_success_ns remote_tlb_flush_requests
halt_successful_poll signal_exits
[root@centos 2025-14]# cat exits
124689
[root@centos 2025-14]# cat vcpu0/exits
52966
[root@centos 2025-14]# cat vcpu1/exits
21549
[root@centos 2025-14]# cat vcpu2/exits
43864
[root@centos 2025-14]# cat vcpu3/exits
6572
[root@centos 2025-14]# ls vcpu0
blocking halt_poll_invalid halt_wait_ns pid
exits halt_poll_success_hist halt_wakeup signal_exits
halt_attempted_poll halt_poll_success_ns hvc_exit_stat wfe_exit_stat
halt_poll_fail_hist halt_successful_poll mmio_exit_kernel wfi_exit_stat
halt_poll_fail_ns halt_wait_hist mmio_exit_user
Xiang Chen (2):
KVM: debugfs: Add vcpu debugfs to record statstical data for every
single vcpu
KVM: arm64: Enable __KVM_HAVE_ARCH_VCPU_DEBUGFS
arch/arm64/include/asm/kvm_host.h | 1 +
arch/arm64/kvm/arm.c | 4 +++
include/linux/kvm_host.h | 2 ++
virt/kvm/kvm_main.c | 62 +++++++++++++++++++++++++++++++++++++--
4 files changed, 67 insertions(+), 2 deletions(-)
--
2.8.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [RFC PATCH 1/2] KVM: debugfs: Add vcpu debugfs to record statstical data for every single vcpu
2022-12-06 12:58 [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single chenxiang
@ 2022-12-06 12:58 ` chenxiang
2022-12-06 12:58 ` [RFC PATCH 2/2] KVM: arm64: Enable __KVM_HAVE_ARCH_VCPU_DEBUGFS chenxiang
2022-12-07 8:21 ` [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single Marc Zyngier
2 siblings, 0 replies; 6+ messages in thread
From: chenxiang @ 2022-12-06 12:58 UTC (permalink / raw)
To: pbonzini, maz, james.morse; +Cc: kvm, linuxarm, Xiang Chen
From: Xiang Chen <chenxiang66@hisilicon.com>
Currently kvm supports debugfs for vm and vcpu. For vcpu debugfs, it
records statistical data for all vcpus not for every single vcpu. But
sometimes we want to know statistical data for every vcpu, there is no
vcpu debugfs for that. So add vcpu debugfs to record statistical
data for every single vcpu.
Signed-off-by: Xiang Chen <chenxiang66@hisilicon.com>
---
include/linux/kvm_host.h | 2 ++
virt/kvm/kvm_main.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 62 insertions(+), 2 deletions(-)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 597953f..25cd2e2 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1756,12 +1756,14 @@ static inline bool kvm_is_error_gpa(struct kvm *kvm, gpa_t gpa)
enum kvm_stat_kind {
KVM_STAT_VM,
KVM_STAT_VCPU,
+ KVM_STAT_VCPU_SINGLE,
};
struct kvm_stat_data {
struct kvm *kvm;
const struct _kvm_stats_desc *desc;
enum kvm_stat_kind kind;
+ unsigned long cpu_id;
};
struct _kvm_stats_desc {
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index baf7a83..0cca201 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1009,7 +1009,7 @@ static void kvm_destroy_vm_debugfs(struct kvm *kvm)
{
int i;
int kvm_debugfs_num_entries = kvm_vm_stats_header.num_desc +
- kvm_vcpu_stats_header.num_desc;
+ 2 * kvm_vcpu_stats_header.num_desc;
if (IS_ERR(kvm->debugfs_dentry))
return;
@@ -1032,7 +1032,7 @@ static int kvm_create_vm_debugfs(struct kvm *kvm, const char *fdname)
const struct _kvm_stats_desc *pdesc;
int i, ret = -ENOMEM;
int kvm_debugfs_num_entries = kvm_vm_stats_header.num_desc +
- kvm_vcpu_stats_header.num_desc;
+ 2 * kvm_vcpu_stats_header.num_desc;
if (!debugfs_initialized())
return 0;
@@ -3899,6 +3899,12 @@ static void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu)
{
struct dentry *debugfs_dentry;
char dir_name[ITOA_MAX_LEN * 2];
+ const struct _kvm_stats_desc *pdesc;
+ struct kvm_stat_data *stat_data;
+ struct kvm *kvm = vcpu->kvm;
+ int start = kvm_vm_stats_header.num_desc +
+ kvm_vcpu_stats_header.num_desc;
+ int i, j;
if (!debugfs_initialized())
return;
@@ -3909,7 +3915,29 @@ static void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu)
debugfs_create_file("pid", 0444, debugfs_dentry, vcpu,
&vcpu_get_pid_fops);
+ for (i = 0; i < kvm_vcpu_stats_header.num_desc; ++i) {
+ pdesc = &kvm_vcpu_stats_desc[i];
+ stat_data = kzalloc(sizeof(*stat_data), GFP_KERNEL_ACCOUNT);
+ if (!stat_data)
+ goto out_err;
+ stat_data->kvm = kvm;
+ stat_data->desc = pdesc;
+ stat_data->kind = KVM_STAT_VCPU_SINGLE;
+ stat_data->cpu_id = vcpu->vcpu_id;
+ kvm->debugfs_stat_data[i + start] = stat_data;
+ debugfs_create_file(pdesc->name, kvm_stats_debugfs_mode(pdesc),
+ debugfs_dentry, stat_data,
+ &stat_fops_per_vm);
+ }
+
kvm_arch_create_vcpu_debugfs(vcpu, debugfs_dentry);
+ return;
+out_err:
+ debugfs_remove_recursive(debugfs_dentry);
+ if (kvm->debugfs_stat_data)
+ for (j = i - 1; j >= 0; j--)
+ kfree(kvm->debugfs_stat_data[j + start]);
+ return;
}
#endif
@@ -5581,6 +5609,28 @@ static int kvm_clear_stat_per_vcpu(struct kvm *kvm, size_t offset)
return 0;
}
+static int kvm_get_stat_per_vcpu_single(struct kvm *kvm, size_t offset, unsigned long i, u64 *val)
+{
+ struct kvm_vcpu *vcpu;
+
+ *val = 0;
+
+ vcpu = kvm_get_vcpu(kvm, i);
+ *val += *(u64 *)((void *)(&vcpu->stat) + offset);
+
+ return 0;
+}
+
+static int kvm_clear_stat_per_vcpu_single(struct kvm *kvm, size_t offset, unsigned long i)
+{
+ struct kvm_vcpu *vcpu;
+
+ vcpu = kvm_get_vcpu(kvm, i);
+ *(u64 *)((void *)(&vcpu->stat) + offset) = 0;
+
+ return 0;
+}
+
static int kvm_stat_data_get(void *data, u64 *val)
{
int r = -EFAULT;
@@ -5595,6 +5645,10 @@ static int kvm_stat_data_get(void *data, u64 *val)
r = kvm_get_stat_per_vcpu(stat_data->kvm,
stat_data->desc->desc.offset, val);
break;
+ case KVM_STAT_VCPU_SINGLE:
+ r = kvm_get_stat_per_vcpu_single(stat_data->kvm,
+ stat_data->desc->desc.offset, stat_data->cpu_id, val);
+ break;
}
return r;
@@ -5617,6 +5671,10 @@ static int kvm_stat_data_clear(void *data, u64 val)
r = kvm_clear_stat_per_vcpu(stat_data->kvm,
stat_data->desc->desc.offset);
break;
+ case KVM_STAT_VCPU_SINGLE:
+ r = kvm_clear_stat_per_vcpu_single(stat_data->kvm,
+ stat_data->desc->desc.offset, stat_data->cpu_id);
+ break;
}
return r;
--
2.8.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [RFC PATCH 2/2] KVM: arm64: Enable __KVM_HAVE_ARCH_VCPU_DEBUGFS
2022-12-06 12:58 [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single chenxiang
2022-12-06 12:58 ` [RFC PATCH 1/2] KVM: debugfs: Add vcpu debugfs to record statstical data for every single vcpu chenxiang
@ 2022-12-06 12:58 ` chenxiang
2022-12-07 8:21 ` [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single Marc Zyngier
2 siblings, 0 replies; 6+ messages in thread
From: chenxiang @ 2022-12-06 12:58 UTC (permalink / raw)
To: pbonzini, maz, james.morse; +Cc: kvm, linuxarm, Xiang Chen
From: Xiang Chen <chenxiang66@hisilicon.com>
Sometimes statistical data of vcpus is helpful for locating some issues,
so enable vcpu debugfs for ARM64.
Signed-off-by: Xiang Chen <chenxiang66@hisilicon.com>
---
arch/arm64/include/asm/kvm_host.h | 1 +
arch/arm64/kvm/arm.c | 4 ++++
2 files changed, 5 insertions(+)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 35a159d..538da55 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -53,6 +53,7 @@
#define KVM_HAVE_MMU_RWLOCK
+#define __KVM_HAVE_ARCH_VCPU_DEBUGFS
/*
* Mode of operation configurable with kvm-arm.mode early param.
* See Documentation/admin-guide/kernel-parameters.txt for more information.
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 9c5573b..d230dfc 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -526,6 +526,10 @@ bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu)
return vcpu_mode_priv(vcpu);
}
+void kvm_arch_create_vcpu_debugfs(struct kvm_vcpu *vcpu, struct dentry *debugfs_dentry)
+{
+}
+
#ifdef CONFIG_GUEST_PERF_EVENTS
unsigned long kvm_arch_vcpu_get_ip(struct kvm_vcpu *vcpu)
{
--
2.8.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single
2022-12-06 12:58 [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single chenxiang
2022-12-06 12:58 ` [RFC PATCH 1/2] KVM: debugfs: Add vcpu debugfs to record statstical data for every single vcpu chenxiang
2022-12-06 12:58 ` [RFC PATCH 2/2] KVM: arm64: Enable __KVM_HAVE_ARCH_VCPU_DEBUGFS chenxiang
@ 2022-12-07 8:21 ` Marc Zyngier
2022-12-07 10:16 ` chenxiang (M)
2 siblings, 1 reply; 6+ messages in thread
From: Marc Zyngier @ 2022-12-07 8:21 UTC (permalink / raw)
To: chenxiang; +Cc: pbonzini, james.morse, kvm, linuxarm
On Tue, 06 Dec 2022 12:58:26 +0000,
chenxiang <chenxiang66@hisilicon.com> wrote:
>
> From: Xiang Chen <chenxiang66@hisilicon.com>
>
> Currently it only records statistical data for all vcpus, but we ofen want
> to know statistical data for a single vcpu, there is no debugfs for that.
> So add vcpu debugfs to record statstical data for every single vcpu, and
> also enable vcpu debugfs for arm64.
>
> After the change, those vcpu debugfs are as follows (we have 4 vcpu in the
> vm):
>
> [root@centos kvm]# cd 2025-14/
> [root@centos 2025-14]# ls
> blocking halt_wait_hist vcpu0
> exits halt_wait_ns vcpu1
> halt_attempted_poll halt_wakeup vcpu2
> halt_poll_fail_hist hvc_exit_stat vcpu3
> halt_poll_fail_ns mmio_exit_kernel vgic-state
> halt_poll_invalid mmio_exit_user wfe_exit_stat
> halt_poll_success_hist remote_tlb_flush wfi_exit_stat
> halt_poll_success_ns remote_tlb_flush_requests
> halt_successful_poll signal_exits
> [root@centos 2025-14]# cat exits
> 124689
> [root@centos 2025-14]# cat vcpu0/exits
> 52966
> [root@centos 2025-14]# cat vcpu1/exits
> 21549
> [root@centos 2025-14]# cat vcpu2/exits
> 43864
> [root@centos 2025-14]# cat vcpu3/exits
> 6572
> [root@centos 2025-14]# ls vcpu0
> blocking halt_poll_invalid halt_wait_ns pid
> exits halt_poll_success_hist halt_wakeup signal_exits
> halt_attempted_poll halt_poll_success_ns hvc_exit_stat wfe_exit_stat
> halt_poll_fail_hist halt_successful_poll mmio_exit_kernel wfi_exit_stat
> halt_poll_fail_ns halt_wait_hist mmio_exit_user
This is yet another example of "KVM doesn't give me the stats I want,
so let's pile more stats on top". This affects every users (counters
are not free), and hardly benefits anyone.
How about you instead add trace hooks that allows you to plumb your
own counters using BPF or another kernel module? This is what is stuff
is for, and we really don't need to create more ABI around that. At
least, the other stat-hungry folks out there would also be able to get
their own stuff, and normal users wouldn't be affected by it.
Thanks,
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single
2022-12-07 8:21 ` [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single Marc Zyngier
@ 2022-12-07 10:16 ` chenxiang (M)
2022-12-07 11:09 ` Marc Zyngier
0 siblings, 1 reply; 6+ messages in thread
From: chenxiang (M) @ 2022-12-07 10:16 UTC (permalink / raw)
To: Marc Zyngier; +Cc: pbonzini, james.morse, kvm, linuxarm
在 2022/12/7 16:21, Marc Zyngier 写道:
> On Tue, 06 Dec 2022 12:58:26 +0000,
> chenxiang <chenxiang66@hisilicon.com> wrote:
>> From: Xiang Chen <chenxiang66@hisilicon.com>
>>
>> Currently it only records statistical data for all vcpus, but we ofen want
>> to know statistical data for a single vcpu, there is no debugfs for that.
>> So add vcpu debugfs to record statstical data for every single vcpu, and
>> also enable vcpu debugfs for arm64.
>>
>> After the change, those vcpu debugfs are as follows (we have 4 vcpu in the
>> vm):
>>
>> [root@centos kvm]# cd 2025-14/
>> [root@centos 2025-14]# ls
>> blocking halt_wait_hist vcpu0
>> exits halt_wait_ns vcpu1
>> halt_attempted_poll halt_wakeup vcpu2
>> halt_poll_fail_hist hvc_exit_stat vcpu3
>> halt_poll_fail_ns mmio_exit_kernel vgic-state
>> halt_poll_invalid mmio_exit_user wfe_exit_stat
>> halt_poll_success_hist remote_tlb_flush wfi_exit_stat
>> halt_poll_success_ns remote_tlb_flush_requests
>> halt_successful_poll signal_exits
>> [root@centos 2025-14]# cat exits
>> 124689
>> [root@centos 2025-14]# cat vcpu0/exits
>> 52966
>> [root@centos 2025-14]# cat vcpu1/exits
>> 21549
>> [root@centos 2025-14]# cat vcpu2/exits
>> 43864
>> [root@centos 2025-14]# cat vcpu3/exits
>> 6572
>> [root@centos 2025-14]# ls vcpu0
>> blocking halt_poll_invalid halt_wait_ns pid
>> exits halt_poll_success_hist halt_wakeup signal_exits
>> halt_attempted_poll halt_poll_success_ns hvc_exit_stat wfe_exit_stat
>> halt_poll_fail_hist halt_successful_poll mmio_exit_kernel wfi_exit_stat
>> halt_poll_fail_ns halt_wait_hist mmio_exit_user
> This is yet another example of "KVM doesn't give me the stats I want,
> so let's pile more stats on top". This affects every users (counters
> are not free), and hardly benefits anyone.
Currently it already has vcpu debugfs on top, but it only records
statstical data for total vm
which is helpless for debug, for example, file exists records the number
of VM exist for all vcpus, before we encountered a
issue that there is something wrong with the thread of a vcpu which
doesn't VM exit but other vcpus are normal,
we can't get anything useful from current vcpu debugfs as the number of
exits still increase in current vcpu debugfs.
Compared with current vcpu debugfs, i think it is more useful to know
the statstical data for every vcpu and it benefits more.
>
> How about you instead add trace hooks that allows you to plumb your
> own counters using BPF or another kernel module? This is what is stuff
> is for, and we really don't need to create more ABI around that. At
> least, the other stat-hungry folks out there would also be able to get
> their own stuff, and normal users wouldn't be affected by it.
>
> Thanks,
>
> M.
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single
2022-12-07 10:16 ` chenxiang (M)
@ 2022-12-07 11:09 ` Marc Zyngier
0 siblings, 0 replies; 6+ messages in thread
From: Marc Zyngier @ 2022-12-07 11:09 UTC (permalink / raw)
To: chenxiang (M); +Cc: pbonzini, james.morse, kvm, linuxarm
On Wed, 07 Dec 2022 10:16:17 +0000,
"chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
>
>
>
> 在 2022/12/7 16:21, Marc Zyngier 写道:
> > On Tue, 06 Dec 2022 12:58:26 +0000,
> > chenxiang <chenxiang66@hisilicon.com> wrote:
> >> From: Xiang Chen <chenxiang66@hisilicon.com>
> >>
> >> Currently it only records statistical data for all vcpus, but we ofen want
> >> to know statistical data for a single vcpu, there is no debugfs for that.
> >> So add vcpu debugfs to record statstical data for every single vcpu, and
> >> also enable vcpu debugfs for arm64.
> >>
> >> After the change, those vcpu debugfs are as follows (we have 4 vcpu in the
> >> vm):
> >>
> >> [root@centos kvm]# cd 2025-14/
> >> [root@centos 2025-14]# ls
> >> blocking halt_wait_hist vcpu0
> >> exits halt_wait_ns vcpu1
> >> halt_attempted_poll halt_wakeup vcpu2
> >> halt_poll_fail_hist hvc_exit_stat vcpu3
> >> halt_poll_fail_ns mmio_exit_kernel vgic-state
> >> halt_poll_invalid mmio_exit_user wfe_exit_stat
> >> halt_poll_success_hist remote_tlb_flush wfi_exit_stat
> >> halt_poll_success_ns remote_tlb_flush_requests
> >> halt_successful_poll signal_exits
> >> [root@centos 2025-14]# cat exits
> >> 124689
> >> [root@centos 2025-14]# cat vcpu0/exits
> >> 52966
> >> [root@centos 2025-14]# cat vcpu1/exits
> >> 21549
> >> [root@centos 2025-14]# cat vcpu2/exits
> >> 43864
> >> [root@centos 2025-14]# cat vcpu3/exits
> >> 6572
> >> [root@centos 2025-14]# ls vcpu0
> >> blocking halt_poll_invalid halt_wait_ns pid
> >> exits halt_poll_success_hist halt_wakeup signal_exits
> >> halt_attempted_poll halt_poll_success_ns hvc_exit_stat wfe_exit_stat
> >> halt_poll_fail_hist halt_successful_poll mmio_exit_kernel wfi_exit_stat
> >> halt_poll_fail_ns halt_wait_hist mmio_exit_user
> > This is yet another example of "KVM doesn't give me the stats I want,
> > so let's pile more stats on top". This affects every users (counters
> > are not free), and hardly benefits anyone.
>
> Currently it already has vcpu debugfs on top, but it only records
> statstical data for total vm
> which is helpless for debug, for example, file exists records the
> number of VM exist for all vcpus, before we encountered a
> issue that there is something wrong with the thread of a vcpu which
> doesn't VM exit but other vcpus are normal,
> we can't get anything useful from current vcpu debugfs as the number
> of exits still increase in current vcpu debugfs.
> Compared with current vcpu debugfs, i think it is more useful to know
> the statstical data for every vcpu and it benefits more.
What is useful for you is useless for somebody else. And vice versa.
At the end of the day, this is *debug* stuff. Why should a normal user
pay a price for *debug* counters?
> > How about you instead add trace hooks that allows you to plumb your
> > own counters using BPF or another kernel module? This is what is stuff
> > is for, and we really don't need to create more ABI around that. At
> > least, the other stat-hungry folks out there would also be able to get
> > their own stuff, and normal users wouldn't be affected by it.
The answer to your problems is ^^^HERE^^^.
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-12-07 11:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-06 12:58 [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single chenxiang
2022-12-06 12:58 ` [RFC PATCH 1/2] KVM: debugfs: Add vcpu debugfs to record statstical data for every single vcpu chenxiang
2022-12-06 12:58 ` [RFC PATCH 2/2] KVM: arm64: Enable __KVM_HAVE_ARCH_VCPU_DEBUGFS chenxiang
2022-12-07 8:21 ` [RFC PATCH 0/2] Add vcpu debugfs to record statstical data for every single Marc Zyngier
2022-12-07 10:16 ` chenxiang (M)
2022-12-07 11:09 ` Marc Zyngier
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).