From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C56C3C282E1 for ; Thu, 25 Apr 2019 07:17:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7A01C217D7 for ; Thu, 25 Apr 2019 07:17:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=euphon.net header.i=fam@euphon.net header.b="e5slVkYB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389234AbfDYHR4 (ORCPT ); Thu, 25 Apr 2019 03:17:56 -0400 Received: from sender2-op-o12.zoho.com.cn ([163.53.93.243]:17701 "EHLO sender1.zoho.com.cn" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1730113AbfDYHR4 (ORCPT ); Thu, 25 Apr 2019 03:17:56 -0400 X-Greylist: delayed 905 seconds by postgrey-1.27 at vger.kernel.org; Thu, 25 Apr 2019 03:17:54 EDT ARC-Seal: i=1; a=rsa-sha256; t=1556175739; cv=none; d=zoho.com.cn; s=zohoarc; b=OIlQe0O8SqVQ2LQnnxRNme2ogOF0m58aZfRM2AFrJw3AZVeBTBkSvwbLlbqepAIjTqlfqHO9HBW3UtheGcw+upRkBcjzeqW18ZPt1MAdST0RuKbu8OAX/+HjDNjNipdLrbGffLyCeGDQtmpsPQRyzzP/Pc9nSpb9/G6eIRuGQko= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com.cn; s=zohoarc; t=1556175739; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results; bh=e0lDNvbRNjedhISZ0vePAMiJvQ+kRneLd73JrVGpm4g=; b=D2hyp1DgqpkD+jh5wUd5S1WLNv1D3gLvzwYXFdxvKoGWtTOqygmD8fdK9F7QuZNLtKNo3cc6ByuIDLPH/8buSKmi9eEP2/yug9sKc9crTK3ttwKmQl79T9JwDzhIxWZGQEch/O2LCcclot73cXYcT2tSTyJUQmak6sSTSDf74bg= ARC-Authentication-Results: i=1; mx.zoho.com.cn; dkim=pass header.i=euphon.net; spf=pass smtp.mailfrom=fam@euphon.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1556175739; s=zoho; d=euphon.net; i=fam@euphon.net; h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Content-Transfer-Encoding:Message-Id:References:To; l=2718; bh=e0lDNvbRNjedhISZ0vePAMiJvQ+kRneLd73JrVGpm4g=; b=e5slVkYBXsW82GOm67ypmCezbdlGFpxFOgXY30wCkIFi+semQ3JjOh2Ztyixrxwx 8DD5oHEeIgO0/dJfQ/oRUXkEFZChj57uQYgBh0SFd/C0zEEMVVSzHeS3fLnlMmYamy5 IFB8rQgTv/3s93nnJImUfyzDiH3Tz6JwIF61dilw= Received: from [10.2.201.103] (120.52.147.47 [120.52.147.47]) by mx.zoho.com.cn with SMTPS id 1556175736636618.8678966648743; Thu, 25 Apr 2019 15:02:16 +0800 (CST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: [PATCH] KVM: x86: add full vm-exit reason debug entries From: Fam Zheng In-Reply-To: <20190422205455.GI1236@linux.intel.com> Date: Thu, 25 Apr 2019 15:01:37 +0800 Cc: Fam Zheng , zhenwei pi , Paolo Bonzini , rkrcmar@redhat.com, mingo@redhat.com, kvm@vger.kernel.org, lifei.shirley@bytedance.com Content-Transfer-Encoding: quoted-printable Message-Id: References: <1555939499-30854-1-git-send-email-pizhenwei@bytedance.com> <20190422205455.GI1236@linux.intel.com> To: Sean Christopherson X-Mailer: Apple Mail (2.3445.104.8) X-ZohoCNMailClient: External Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org > On Apr 23, 2019, at 04:54, Sean Christopherson = wrote: >=20 > On Mon, Apr 22, 2019 at 09:24:59PM +0800, zhenwei pi wrote: >> Hit several typical cases of performance drop due to vm-exit: >> case 1, jemalloc calls madvise(void *addr, size_t length, = MADV_DONTNEED) in >> guest, IPI causes a lot of exits. >> case 2, atop collects IPC by perf hardware events in guest, vpmu & = rdpmc >> exits increase a lot. >> case 3, host memory compaction invalidates TDP and tdp_page_fault can = cause >> huge loss of performance. >> case 4, web services(written by golang) call futex and have higher = latency >> than host os environment. >>=20 >> Add more vm-exit reason debug entries, they are helpful to recognize >> performance drop reason. In this patch: >> 1, add more vm-exit reasons. >> 2, add wrmsr details. >> 3, add CR details. >> 4, add hypercall details. >>=20 >> Currently we can also implement the same result by bpf. >> Sample code (written by Fei Li): >> b =3D BPF(text=3D""" >>=20 >> struct kvm_msr_exit_info { >> u32 pid; >> u32 tgid; >> u32 msr_exit_ct; >> }; >> BPF_HASH(kvm_msr_exit, unsigned int, struct kvm_msr_exit_info, = 1024); >>=20 >> TRACEPOINT_PROBE(kvm, kvm_msr) { >> int ct =3D args->ecx; >> if (ct >=3D 0xffffffff) { >> return -1; >> } >>=20 >> u32 pid =3D bpf_get_current_pid_tgid() >> 32; >> u32 tgid =3D bpf_get_current_pid_tgid(); >>=20 >> struct kvm_msr_exit_info *exit_info; >> struct kvm_msr_exit_info init_exit_info =3D {}; >> exit_info =3D kvm_msr_exit.lookup(&ct); >> if (exit_info !=3D NULL) { >> exit_info->pid =3D pid; >> exit_info->tgid =3D tgid; >> exit_info->msr_exit_ct++; >> } else { >> init_exit_info.pid =3D pid; >> init_exit_info.tgid =3D tgid; >> init_exit_info.msr_exit_ct =3D 1; >> kvm_msr_exit.update(&ct, &init_exit_info); >> } >> return 0; >> } >> """) >>=20 >> Run wrmsr(MSR_IA32_TSCDEADLINE, val) benchmark in guest >> (CPU Intel Gold 5118): >> case 1, no bpf on host. ~1127 cycles/wrmsr. >> case 2, sample bpf on host with JIT. ~1223 cycles/wrmsr. --> = 8.5% >> case 3, sample bpf on host without JIT. ~1312 cycles/wrmsr. --> = 16.4% >>=20 >> So, debug entries are more efficient than the bpf method. >=20 > How much does host performance matter? E.g. does high overhead = interfere > with debug, is this something you want to have running at all times, = etc=E2=80=A6 The intention is to have a long running statictics for monitoring etc. = In general, using eBPF, ftrace etc. are okay for not-so-hot points, but = this one turned out to be very difficult to do efficiently without such = a code change. Fam=