From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67D41C35280 for ; Wed, 2 Oct 2019 04:59:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 08CA3215EA for ; Wed, 2 Oct 2019 04:59:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oEVkss6V" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 08CA3215EA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 73F638E0005; Wed, 2 Oct 2019 00:59:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F0078E0001; Wed, 2 Oct 2019 00:59:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6064B8E0005; Wed, 2 Oct 2019 00:59:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0090.hostedemail.com [216.40.44.90]) by kanga.kvack.org (Postfix) with ESMTP id 408328E0001 for ; Wed, 2 Oct 2019 00:59:55 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id D18BA6D9A for ; Wed, 2 Oct 2019 04:59:54 +0000 (UTC) X-FDA: 75997642308.23.cake50_3968d0713c33d X-HE-Tag: cake50_3968d0713c33d X-Filterd-Recvd-Size: 6228 Received: from mail-io1-f68.google.com (mail-io1-f68.google.com [209.85.166.68]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Oct 2019 04:59:54 +0000 (UTC) Received: by mail-io1-f68.google.com with SMTP id q1so54009798ion.1 for ; Tue, 01 Oct 2019 21:59:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=utqwTvRLyjSsVfZRkt/xO+q4bNl5CsJkNoufFxflxJQ=; b=oEVkss6VR+9MCsjpkKg1eW3moidaorvQjMfZN1aobBTuZpUSiliyZAD/ZB+LSjwIzx ffJcXnSgJUT6H0k5VHNZ8N0lAqZTbi+fmGXl1utWTFytCzdYlsCJfaCKQiHMCWa3jbD/ HrmS9VKmNkxVQ3WvL5HS7dGXRTvtKi+Pn4ejI9bscexXo9IjPo9FfiiZNmwCoL7uWpiE 3EIqDGN7ic+YN6gruIZhX3E2bGkGijDJxJ1YZwQHkPyE2tbdAS5ZUKUNO2Knw+uAkba+ Pi+IOwyNQJGwJMIJmh2CvkWbUL8J/ILcEGUo/IeaaUHYG/ZHyg2Z6VcRhkHylVPJcl76 w+ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=utqwTvRLyjSsVfZRkt/xO+q4bNl5CsJkNoufFxflxJQ=; b=N0S8NvCVuIjhfEGaHM1Wqq90tXMd0KLK0ycz8/Bf4+sAvyh7sJuB8uqeGe59/8vxnH IVUBQ6H7RWA2/5Vl/QgijlHd9J6WhmZuNUSKWw3x36e6aiZYfWJfoaye24kdH3ILBgak 0PQqToPhz5dWlciH8CaHJ5P2Y/N/D1njCaMw9zA3wWwKZP10vlj7Kgxg17nbgnSPCSrY myodNLRmWVdXXyfy4v/97bEMOxCQUFG7EifAevaq/XCwS0tF0vLUGE17Hk2ClHuIctiq V42eSm+iy335CQra5xnZxvwE0F5KSMoZZm1gB6a6gMbDRnImoERESB4Ong0pdMH0nHmt asXQ== X-Gm-Message-State: APjAAAX6XmymIMNlQjNuHA6WhRJ1ZAe18AmtGcRT5y3dNNfFK0mdZgtd 7+a/qh4Tvp7WtxUJvToge5AmJtxOYAU4C03dcic= X-Google-Smtp-Source: APXvYqx06rpxC7ziYaQbDjJwNP8n17yaCIyqxnU/Wrn7tsZo+KpOiuVAlHvBbHJp0PTPCVEpajAbIk+YHTgN7/i1XLQ= X-Received: by 2002:a92:4a11:: with SMTP id m17mr1998877ilf.142.1569992393636; Tue, 01 Oct 2019 21:59:53 -0700 (PDT) MIME-Version: 1.0 References: <1569899984-16272-1-git-send-email-laoar.shao@gmail.com> <20191001144524.GB3321@techsingularity.net> In-Reply-To: <20191001144524.GB3321@techsingularity.net> From: Yafang Shao Date: Wed, 2 Oct 2019 12:59:14 +0800 Message-ID: Subject: Re: [PATCH v2] perf script python: integrate page reclaim analyze script To: Mel Gorman Cc: tonyj@suse.com, acme@kernel.org, Peter Zijlstra , mingo@redhat.com, alexander.shishkin@linux.intel.com, jolsa@redhat.com, namhyung@kernel.org, Andrew Morton , LKML , Linux MM , Tony Jones Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Oct 1, 2019 at 10:45 PM Mel Gorman wrote: > > On Mon, Sep 30, 2019 at 11:19:44PM -0400, Yafang Shao wrote: > > A new perf script page-reclaim is introduced in this patch. This new script > > is used to report the page reclaim details. The possible usage of this > > script is as bellow, > > - identify latency spike caused by direct reclaim > > - whehter the latency spike is relevant with pageout > > - why is page reclaim requested, i.e. whether it is because of memory > > fragmentation > > - page reclaim efficiency > > etc > > In the future we may also enhance it to analyze the memcg reclaim. > > > > Hi, > > I ended up not reviewing this patch in detail simply because I would > approach the same class of problem in an entirely different way today. > There is value in accumulating the stats in a report like this; > > > $ perf script report page-reclaim > > Direct reclaims: 4924 > > Direct latency (ms) total max avg min > > 177823.211 6378.977 36.114 0.051 > > Direct file reclaimed 22920 > > Direct file scanned 28306 > > Direct file sync write I/O 0 > > Direct file async write I/O 0 > > Direct anon reclaimed 212567 > > Direct anon scanned 1446854 > > Direct anon sync write I/O 0 > > Direct anon async write I/O 278325 > > Direct order 0 1 3 > > 4870 23 31 > > Wake kswapd requests 716 > > Wake order 0 1 > > 715 1 > > > > Kswapd reclaims: 9 > > However, the basic option I would prefer is having the raw latency > information for Direct latency that can be externally parsed by R or any > other statistical method. The reason why is because knowing the max latency > is not enough, I'd want to know the spread of latencies and whether they > were clustered at a point of time or spread out over long periods of > time. I would then build the higher-level reports on top if necessary. > > Today, I would also have considered getting the latency figures using eBPF > or systemtap instead although having perf do it may be useful too. That's > not universally popular though so at minimum I would have; > eBPF requires newer kernel, while there're still lots of servers running with old kernels. The systemtap is not convenient as it requires many debug packages, and it is still not stable enough to run on the product environment, for example, the systemtap deamon may exit without uninstalling the systemtap kernel module. > perf script record page-reclaim -- capture all page-reclaim tracepoints > perf script report page-reclaim -- For reclaim entry/exit, merge the two > tracepoints into one that reports latency. Dump the rest out > verbatim > > For latencies, I would externally post-process them until such time as I > found a common class of bug that needed a high-level report and then > build the perf script support for it. > This seem like a good suggestion. I will try to think about it. > Please note that I did not spot anything wrong with your script, it's > just that I would not use it myself in its current format for debugging > a reclaim-related problem. > > -- > Mel Gorman > SUSE Labs