From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2AF7C433B4 for ; Wed, 14 Apr 2021 22:25:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C78BC6117A for ; Wed, 14 Apr 2021 22:25:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232131AbhDNWZt convert rfc822-to-8bit (ORCPT ); Wed, 14 Apr 2021 18:25:49 -0400 Received: from mail.kernel.org ([198.145.29.99]:43234 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231207AbhDNWZt (ORCPT ); Wed, 14 Apr 2021 18:25:49 -0400 Received: from gandalf.local.home (cpe-66-24-58-225.stny.res.rr.com [66.24.58.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AD7EC600CD; Wed, 14 Apr 2021 22:25:26 +0000 (UTC) Date: Wed, 14 Apr 2021 18:25:24 -0400 From: Steven Rostedt To: Dario Faggioli Cc: Giuseppe Eletto , linux-trace-devel@vger.kernel.org, xen-devel@lists.xenproject.org, Enrico Bini Subject: Re: A KernelShark plugin for Xen traces analysis Message-ID: <20210414182524.0e51a520@gandalf.local.home> In-Reply-To: <28ac9c046cc521cbaef9c2ff56911cd7b3100ac4.camel@suse.com> References: <20210413114614.4971caff@gandalf.local.home> <28ac9c046cc521cbaef9c2ff56911cd7b3100ac4.camel@suse.com> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org On Thu, 15 Apr 2021 00:11:32 +0200 Dario Faggioli wrote: > Yes, basically, we can say that a Xen system has "its own trace-cmd". > It's called `xentrace`, you run it from Dom0 and you get a (binary) > file which contains a bunch of events. > > Not that differently from a trace-cmd's "trace.dat" file, but the > events in there comes from tracepoints within the hypervisor (which, of > course, use a different tracing mechanism than ftrace). Right, that's exactly what the ESX trace did as well. > > Perhaps we can update trace-cmd agent to work with > > Xen as well. Does xen implement vsock or some other way to > > communicate > > between the guests and the Dom0 kernel?  > > > Not vsock, AFAIK. But we probably can use something else/come up with > something new. > Yeah, we would like to have trace-cmd agent work with more than just vsock. Heck, we could just use networking as well. It's just a bit more overhead. > > And then on KernelShark, > > we have a KVM plugin in development that does this. But you can do > > the same > > with Xen. > > > I think that one of the trickiest aspects would be synchronizing the > timestamps in the 3 traces. > > *I guess* that the dom0 trace and the guest traces could at least use > the PTP algorithm that is currently implemented in the trace-cmd > patches (but not KVM specific one). For synch'ing the Xen trace with > them, well, I don't really know... We'd have to think about it. :-P Really, TSC is the way to go. All you would need to do is to have a way to map all the TSCs together. Assuming the xen trace has a unmodified TSC, and you can retrieve all the multipliers and shifts used for each guest, you then will have a synchronized TSC. Then only one guest or the xen HV needs to calculate the TSC to nanoseconds, and then have all use that. Probably would need to be the xen HV as it would be the one without a modified TSC. -- Steve From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21E00C433ED for ; Wed, 14 Apr 2021 22:25:42 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B5DEB600CD for ; Wed, 14 Apr 2021 22:25:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B5DEB600CD Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=goodmis.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.110850.211778 (Exim 4.92) (envelope-from ) id 1lWnwv-0005ml-Gh; Wed, 14 Apr 2021 22:25:29 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 110850.211778; Wed, 14 Apr 2021 22:25:29 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lWnwv-0005me-Dl; Wed, 14 Apr 2021 22:25:29 +0000 Received: by outflank-mailman (input) for mailman id 110850; Wed, 14 Apr 2021 22:25:28 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lWnwu-0005mZ-Lt for xen-devel@lists.xenproject.org; Wed, 14 Apr 2021 22:25:28 +0000 Received: from mail.kernel.org (unknown [198.145.29.99]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 021aaba3-48f0-4fd2-a6f1-283672b117e3; Wed, 14 Apr 2021 22:25:27 +0000 (UTC) Received: from gandalf.local.home (cpe-66-24-58-225.stny.res.rr.com [66.24.58.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AD7EC600CD; Wed, 14 Apr 2021 22:25:26 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 021aaba3-48f0-4fd2-a6f1-283672b117e3 Date: Wed, 14 Apr 2021 18:25:24 -0400 From: Steven Rostedt To: Dario Faggioli Cc: Giuseppe Eletto , linux-trace-devel@vger.kernel.org, xen-devel@lists.xenproject.org, Enrico Bini Subject: Re: A KernelShark plugin for Xen traces analysis Message-ID: <20210414182524.0e51a520@gandalf.local.home> In-Reply-To: <28ac9c046cc521cbaef9c2ff56911cd7b3100ac4.camel@suse.com> References: <20210413114614.4971caff@gandalf.local.home> <28ac9c046cc521cbaef9c2ff56911cd7b3100ac4.camel@suse.com> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu, 15 Apr 2021 00:11:32 +0200 Dario Faggioli wrote: =20 > Yes, basically, we can say that a Xen system has "its own trace-cmd". > It's called `xentrace`, you run it from Dom0 and you get a (binary) > file which contains a bunch of events. >=20 > Not that differently from a trace-cmd's "trace.dat" file, but the > events in there comes from tracepoints within the hypervisor (which, of > course, use a different tracing mechanism than ftrace). Right, that's exactly what the ESX trace did as well. > > Perhaps we can update trace-cmd agent to work with > > Xen as well. Does xen implement vsock or some other way to > > communicate > > between the guests and the Dom0 kernel?=C2=A0 > > =20 > Not vsock, AFAIK. But we probably can use something else/come up with > something new. >=20 Yeah, we would like to have trace-cmd agent work with more than just vsock. Heck, we could just use networking as well. It's just a bit more overhead. > > And then on KernelShark, > > we have a KVM plugin in development that does this. But you can do > > the same > > with Xen. > > =20 > I think that one of the trickiest aspects would be synchronizing the > timestamps in the 3 traces. >=20 > *I guess* that the dom0 trace and the guest traces could at least use > the PTP algorithm that is currently implemented in the trace-cmd > patches (but not KVM specific one). For synch'ing the Xen trace with > them, well, I don't really know... We'd have to think about it. :-P Really, TSC is the way to go. All you would need to do is to have a way to map all the TSCs together. Assuming the xen trace has a unmodified TSC, and you can retrieve all the multipliers and shifts used for each guest, you then will have a synchronized TSC. Then only one guest or the xen HV needs to calculate the TSC to nanoseconds, and then have all use that. Probably would need to be the xen HV as it would be the one without a modified TSC. -- Steve