From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED, USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D033AC4360F for ; Sat, 23 Mar 2019 12:02:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9CAD121902 for ; Sat, 23 Mar 2019 12:02:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dcMGrPnS" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727385AbfCWMBG (ORCPT ); Sat, 23 Mar 2019 08:01:06 -0400 Received: from mail-yw1-f67.google.com ([209.85.161.67]:41878 "EHLO mail-yw1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726118AbfCWMBG (ORCPT ); Sat, 23 Mar 2019 08:01:06 -0400 Received: by mail-yw1-f67.google.com with SMTP id l15so3357597ywe.8 for ; Sat, 23 Mar 2019 05:01:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=OxtWF8GmikN+/3mXH6py7vzZ6KycdJnIWvcyc/DIb6A=; b=dcMGrPnSUtoqjxhfLJs7nY564bSfH1rTmJgryZZZc5bjCCq5Zx1v5aT+6No6OV8i0M vJVJuveL26qPkVohNXJqTMoht3s4LOryV7lNBKJ2okYblwCXl0BvAQzNh0D3GCeDhsB8 4bEysLJ36jp8Eh6WMmnPWsjeiowp7KdOrcpgvF8kYMniZbqgdfdnYcuNDmy10kudcNPU AWJ3mjaM2hwh070PzYGPhhQKKbjoV49DjAD9BhfK+572drdGJuB+yfuXzMM+O2KGe62Y pXxp30PxGobEn2Y+kFZBJyVlkO10iiFmTvX5hEKFS5vXiiHddlb8qEdB29I2FkHVY74I HKvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=OxtWF8GmikN+/3mXH6py7vzZ6KycdJnIWvcyc/DIb6A=; b=PvDRt5foW9gefupWZ0nKD2vQfObRK8Lfgz0IDczdCVdvPu2iF1CndxgRF2EEISKy47 tMgYW9Xj7k+omjQltstMR8mMsyVBnAKJIXzGoolAYG7HeHjv9V5VR1NepFSr1FyOPwPn AH8wem3pi7/3SKGJ/1KxmlBqIkzEqcbTQbIEvGbyh3r4wKL1SPddqVmMGAi2Sl02vfPN t+HcsSTX9djyL7Y8/5eYYAHR5bz2ka0+oo4hJJ1LS44cNUR334za2v4wt9dZc+kEoDu2 +sbWV+zg3U3v2dPWr8Mju25EtVEZ/anGC6yBiM4jBgD4ykokMYBCFQu3PcxRAriTK4wK FEgg== X-Gm-Message-State: APjAAAWRMuFG94r9C6mcc1WuK1B6yLnqBr5mrBfsQ2eoNZoJo2Eio+Fr hqGZJ4aD+o2cQpRWof855xE= X-Google-Smtp-Source: APXvYqzn4FMMWily8FQT5LLExBeX86jm+dO3qtpoZvBNUPGdF+z0kvtC/HCmBZOQNLFdDbKu2182AA== X-Received: by 2002:a0d:c9c2:: with SMTP id l185mr12972134ywd.227.1553342465644; Sat, 23 Mar 2019 05:01:05 -0700 (PDT) Received: from centos-dev.localdomain (pool-173-66-89-81.washdc.fios.verizon.net. [173.66.89.81]) by smtp.gmail.com with ESMTPSA id g1sm3613882ywf.0.2019.03.23.05.01.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 23 Mar 2019 05:01:05 -0700 (PDT) Date: Sat, 23 Mar 2019 08:00:52 -0400 From: Ryan Thibodeaux To: Boris Ostrovsky Cc: xen-devel@lists.xenproject.org, linux-kernel@vger.kernel.org, oleksandr_andrushchenko@epam.com, tglx@linutronix.de, jgross@suse.com, ryan.thibodeaux@starlab.io Subject: Re: [PATCH] x86/xen: Add "xen_timer_slop" command line option Message-ID: <20190323120052.GB11100@centos-dev.localdomain> References: <1553279397-130201-1-git-send-email-ryan.thibodeaux@starlab.io> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Mar 22, 2019 at 06:10:16PM -0400, Boris Ostrovsky wrote: > On 3/22/19 2:29 PM, thibodux@gmail.com wrote: > > From: Ryan Thibodeaux > > > > Add a new command-line option "xen_timer_slop=" that sets the > > minimum delta of virtual Xen timers. This commit does not change the > > default timer slop value for virtual Xen timers. > > > > Lowering the timer slop value should improve the accuracy of virtual > > timers (e.g., better process dispatch latency), but it will likely > > increase the number of virtual timer interrupts (relative to the > > original slop setting). > > > > The original timer slop value has not changed since the introduction > > of the Xen-aware Linux kernel code. This commit provides users an > > opportunity to tune timer performance given the refinements to > > hardware and the Xen event channel processing. It also mirrors > > a feature in the Xen hypervisor - the "timer_slop" Xen command line > > option. > > Is there any data that shows effects of using this new parameter? > > -boris > For our own testing using "cyclictest" from the rt-tests project, lowering the timer slop helped produce the best test runs, especially in terms of maximum process dispatch latency (PDL). Here is the output from one such test that ran overnight. The Xen timer slop in this case was 10000 or 10 microseconds. ... [root@slop1 ~]# cset shield -c 3 [root@slop1 ~]# echo ; date ; echo ; \ ./rt-tests-1.3/cyclictest -p95 -a3 -t1 -m; echo ; date Thu Mar 14 19:45:36 UTC 2019 # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 0.00 0.02 0.00 1/91 4260 T: 0 ( 3212) P:95 I:1000 C:57077313 Min: 27 Act: 44 Avg: 43 Max: 145 ^C Fri Mar 15 11:36:53 UTC 2019 ... This test system was configured to use a TSC clocksource, disabled C states, and lowered the timer slop. I am not claiming the timer slop change was solely responsible for the best results. In other testing with the default timer slop setting of 100000 (100 microseconds), the average PDL would run slightly higher, but the spikes were much higher and more in number, often near the 1000s and happening multiple times per 10 minutes of testing. - Ryan From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ryan Thibodeaux Subject: Re: [PATCH] x86/xen: Add "xen_timer_slop" command line option Date: Sat, 23 Mar 2019 08:00:52 -0400 Message-ID: <20190323120052.GB11100@centos-dev.localdomain> References: <1553279397-130201-1-git-send-email-ryan.thibodeaux@starlab.io> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1h7fKl-0001nP-CM for xen-devel@lists.xenproject.org; Sat, 23 Mar 2019 12:01:07 +0000 Received: by mail-yw1-xc44.google.com with SMTP id x129so3061511ywc.1 for ; Sat, 23 Mar 2019 05:01:06 -0700 (PDT) Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" To: Boris Ostrovsky Cc: jgross@suse.com, oleksandr_andrushchenko@epam.com, linux-kernel@vger.kernel.org, ryan.thibodeaux@starlab.io, xen-devel@lists.xenproject.org, tglx@linutronix.de List-Id: xen-devel@lists.xenproject.org T24gRnJpLCBNYXIgMjIsIDIwMTkgYXQgMDY6MTA6MTZQTSAtMDQwMCwgQm9yaXMgT3N0cm92c2t5 IHdyb3RlOgo+IE9uIDMvMjIvMTkgMjoyOSBQTSwgdGhpYm9kdXhAZ21haWwuY29tIHdyb3RlOgo+ ID4gRnJvbTogUnlhbiBUaGlib2RlYXV4IDxyeWFuLnRoaWJvZGVhdXhAc3RhcmxhYi5pbz4KPiA+ Cj4gPiBBZGQgYSBuZXcgY29tbWFuZC1saW5lIG9wdGlvbiAieGVuX3RpbWVyX3Nsb3A9PElOVD4i IHRoYXQgc2V0cyB0aGUKPiA+IG1pbmltdW0gZGVsdGEgb2YgdmlydHVhbCBYZW4gdGltZXJzLiBU aGlzIGNvbW1pdCBkb2VzIG5vdCBjaGFuZ2UgdGhlCj4gPiBkZWZhdWx0IHRpbWVyIHNsb3AgdmFs dWUgZm9yIHZpcnR1YWwgWGVuIHRpbWVycy4KPiA+Cj4gPiBMb3dlcmluZyB0aGUgdGltZXIgc2xv cCB2YWx1ZSBzaG91bGQgaW1wcm92ZSB0aGUgYWNjdXJhY3kgb2YgdmlydHVhbAo+ID4gdGltZXJz IChlLmcuLCBiZXR0ZXIgcHJvY2VzcyBkaXNwYXRjaCBsYXRlbmN5KSwgYnV0IGl0IHdpbGwgbGlr ZWx5Cj4gPiBpbmNyZWFzZSB0aGUgbnVtYmVyIG9mIHZpcnR1YWwgdGltZXIgaW50ZXJydXB0cyAo cmVsYXRpdmUgdG8gdGhlCj4gPiBvcmlnaW5hbCBzbG9wIHNldHRpbmcpLgo+ID4KPiA+IFRoZSBv cmlnaW5hbCB0aW1lciBzbG9wIHZhbHVlIGhhcyBub3QgY2hhbmdlZCBzaW5jZSB0aGUgaW50cm9k dWN0aW9uCj4gPiBvZiB0aGUgWGVuLWF3YXJlIExpbnV4IGtlcm5lbCBjb2RlLiBUaGlzIGNvbW1p dCBwcm92aWRlcyB1c2VycyBhbgo+ID4gb3Bwb3J0dW5pdHkgdG8gdHVuZSB0aW1lciBwZXJmb3Jt YW5jZSBnaXZlbiB0aGUgcmVmaW5lbWVudHMgdG8KPiA+IGhhcmR3YXJlIGFuZCB0aGUgWGVuIGV2 ZW50IGNoYW5uZWwgcHJvY2Vzc2luZy4gSXQgYWxzbyBtaXJyb3JzCj4gPiBhIGZlYXR1cmUgaW4g dGhlIFhlbiBoeXBlcnZpc29yIC0gdGhlICJ0aW1lcl9zbG9wIiBYZW4gY29tbWFuZCBsaW5lCj4g PiBvcHRpb24uCj4gCj4gSXMgdGhlcmUgYW55IGRhdGEgdGhhdCBzaG93cyBlZmZlY3RzIG9mIHVz aW5nIHRoaXMgbmV3IHBhcmFtZXRlcj8KPiAKPiAtYm9yaXMKPiAKCkZvciBvdXIgb3duIHRlc3Rp bmcgdXNpbmcgImN5Y2xpY3Rlc3QiIGZyb20gdGhlIHJ0LXRlc3RzIHByb2plY3QsCmxvd2VyaW5n IHRoZSB0aW1lciBzbG9wIGhlbHBlZCBwcm9kdWNlIHRoZSBiZXN0IHRlc3QgcnVucywgZXNwZWNp YWxseQppbiB0ZXJtcyBvZiBtYXhpbXVtIHByb2Nlc3MgZGlzcGF0Y2ggbGF0ZW5jeSAoUERMKS4K CkhlcmUgaXMgdGhlIG91dHB1dCBmcm9tIG9uZSBzdWNoIHRlc3QgdGhhdCByYW4gb3Zlcm5pZ2h0 LiBUaGUgWGVuCnRpbWVyIHNsb3AgaW4gdGhpcyBjYXNlIHdhcyAxMDAwMCBvciAxMCBtaWNyb3Nl Y29uZHMuCgouLi4KW3Jvb3RAc2xvcDEgfl0jIGNzZXQgc2hpZWxkIC1jIDMKW3Jvb3RAc2xvcDEg fl0jIGVjaG8gOyBkYXRlIDsgZWNobyA7IFwKLi9ydC10ZXN0cy0xLjMvY3ljbGljdGVzdCAtcDk1 IC1hMyAtdDEgLW07IGVjaG8gOyBkYXRlCgpUaHUgTWFyIDE0IDE5OjQ1OjM2IFVUQyAyMDE5Cgoj IC9kZXYvY3B1X2RtYV9sYXRlbmN5IHNldCB0byAwdXMKcG9saWN5OiBmaWZvOiBsb2FkYXZnOiAw LjAwIDAuMDIgMC4wMCAxLzkxIDQyNjAKVDogMCAoIDMyMTIpIFA6OTUgSToxMDAwIEM6NTcwNzcz MTMgTWluOiAyNyBBY3Q6IDQ0IEF2ZzogNDMgTWF4OiAxNDUKXkMKRnJpIE1hciAxNSAxMTozNjo1 MyBVVEMgMjAxOQouLi4KClRoaXMgdGVzdCBzeXN0ZW0gd2FzIGNvbmZpZ3VyZWQgdG8gdXNlIGEg VFNDIGNsb2Nrc291cmNlLCBkaXNhYmxlZApDIHN0YXRlcywgYW5kIGxvd2VyZWQgdGhlIHRpbWVy IHNsb3AuIEkgYW0gbm90IGNsYWltaW5nIHRoZSB0aW1lcgpzbG9wIGNoYW5nZSB3YXMgc29sZWx5 IHJlc3BvbnNpYmxlIGZvciB0aGUgYmVzdCByZXN1bHRzLiBJbiBvdGhlcgp0ZXN0aW5nIHdpdGgg dGhlIGRlZmF1bHQgdGltZXIgc2xvcCBzZXR0aW5nIG9mIDEwMDAwMCAoMTAwCm1pY3Jvc2Vjb25k cyksIHRoZSBhdmVyYWdlIFBETCB3b3VsZCBydW4gc2xpZ2h0bHkgaGlnaGVyLCBidXQgdGhlCnNw aWtlcyB3ZXJlIG11Y2ggaGlnaGVyIGFuZCBtb3JlIGluIG51bWJlciwgb2Z0ZW4gbmVhciB0aGUg MTAwMHMKYW5kIGhhcHBlbmluZyBtdWx0aXBsZSB0aW1lcyBwZXIgMTAgbWludXRlcyBvZiB0ZXN0 aW5nLgoKLSBSeWFuCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZlbEBsaXN0cy54ZW5wcm9qZWN0Lm9y ZwpodHRwczovL2xpc3RzLnhlbnByb2plY3Qub3JnL21haWxtYW4vbGlzdGluZm8veGVuLWRldmVs