From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755514AbcJTXKn (ORCPT ); Thu, 20 Oct 2016 19:10:43 -0400 Received: from mail-it0-f67.google.com ([209.85.214.67]:35458 "EHLO mail-it0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752483AbcJTXKk (ORCPT ); Thu, 20 Oct 2016 19:10:40 -0400 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: [RFC] scripts: Include postprocessing script for memory allocation tracing From: Janani Ravichandran In-Reply-To: <20161018131343.GJ12092@dhcp22.suse.cz> Date: Thu, 20 Oct 2016 18:10:37 -0500 Cc: Janani Ravichandran , linux-mm@kvack.org, linux-kernel@vger.kernel.org Message-Id: <4F0F918D-B98A-48EC-82ED-EE7D32F222EA@gmail.com> References: <20160912121635.GL14524@dhcp22.suse.cz> <0ACE5927-A6E5-4B49-891D-F990527A9F50@gmail.com> <20160919094224.GH10785@dhcp22.suse.cz> <20160923080709.GB4478@dhcp22.suse.cz> <2D27EF16-B63B-4516-A156-5E2FB675A1BB@gmail.com> <20161016073340.GA15839@dhcp22.suse.cz> <20161018131343.GJ12092@dhcp22.suse.cz> To: Michal Hocko X-Mailer: Apple Mail (2.2104) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by mail.home.local id u9KNAmw4029791 Michal, > On Oct 18, 2016, at 8:13 AM, Michal Hocko wrote: > >> > > yes, function_graph tracer will give you _some_ information but it will > not have the context you are looking for, right? See the following > example > > ------------------------------------------ > 0) x-www-b-22756 => x-termi-4083 > ------------------------------------------ > > 0) | __alloc_pages_nodemask() { > 0) | /* mm_page_alloc: page=ffffea000411b380 pfn=1066702 order=0 migratetype=0 gfp_flags=GFP_KERNEL */ > 0) 3.328 us | } > 3) | __alloc_pages_nodemask() { > 3) | /* mm_page_alloc: page=ffffea0008f1f6c0 pfn=2344923 order=0 migratetype=0 gfp_flags=GFP_KERNEL */ > 3) 1.011 us | } > 0) | __alloc_pages_nodemask() { > 0) | /* mm_page_alloc: page=ffffea000411b380 pfn=1066702 order=0 migratetype=0 gfp_flags=GFP_KERNEL */ > 0) 0.587 us | } > 3) | __alloc_pages_nodemask() { > 3) | /* mm_page_alloc: page=ffffea0008f1f6c0 pfn=2344923 order=0 migratetype=0 gfp_flags=GFP_KERNEL */ > 3) 1.125 us | } > > How do I know which process has performed those allocations? I know that > CPU0 should be running x-termi-4083 but what is running on other CPUs? > > Let me explain my usecase I am very interested in. Say I that a usespace > application is not performing well. I would like to see some statistics > about memory allocations performed for that app - are there few outliers > or the allocation stalls increase gradually? Where do we spend time during > that allocation? Reclaim LRU pages? Compaction or the slab shrinkers? > > To answer those questions I need to track particular events (alocation, > reclaim, compaction) to the process and know how long each step > took. Maybe we can reconstruct something from the above output but it is > a major PITA. If we either hard start/stop pairs for each step (which > we already do have for reclaim, compaction AFAIR) then this is an easy > scripting. Another option would be to have only a single tracepoint for > each step with a timing information. > > See my point? Yes, if we want to know what processes are running on what CPUs, echo funcgraph-proc > trace_options in the tracing directory should give us what we want. The bash script which is part of this patch does this kind of setup for you. As a result, the output you get is something like what you see here: https://github.com/Jananiravichandran/Analyzing-tracepoints/blob/master/no_tp_no_threshold.txt Does this answer your question? Let me know if otherwise. Janani. > -- > Michal Hocko > SUSE Labs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-f70.google.com (mail-it0-f70.google.com [209.85.214.70]) by kanga.kvack.org (Postfix) with ESMTP id A076A6B025E for ; Thu, 20 Oct 2016 19:10:40 -0400 (EDT) Received: by mail-it0-f70.google.com with SMTP id f129so137441273itc.7 for ; Thu, 20 Oct 2016 16:10:40 -0700 (PDT) Received: from mail-it0-x242.google.com (mail-it0-x242.google.com. [2607:f8b0:4001:c0b::242]) by mx.google.com with ESMTPS id a7si1522391ioe.155.2016.10.20.16.10.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Oct 2016 16:10:40 -0700 (PDT) Received: by mail-it0-x242.google.com with SMTP id k64so8496367itb.0 for ; Thu, 20 Oct 2016 16:10:40 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: [RFC] scripts: Include postprocessing script for memory allocation tracing From: Janani Ravichandran In-Reply-To: <20161018131343.GJ12092@dhcp22.suse.cz> Date: Thu, 20 Oct 2016 18:10:37 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <4F0F918D-B98A-48EC-82ED-EE7D32F222EA@gmail.com> References: <20160912121635.GL14524@dhcp22.suse.cz> <0ACE5927-A6E5-4B49-891D-F990527A9F50@gmail.com> <20160919094224.GH10785@dhcp22.suse.cz> <20160923080709.GB4478@dhcp22.suse.cz> <2D27EF16-B63B-4516-A156-5E2FB675A1BB@gmail.com> <20161016073340.GA15839@dhcp22.suse.cz> <20161018131343.GJ12092@dhcp22.suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: Janani Ravichandran , linux-mm@kvack.org, linux-kernel@vger.kernel.org Michal, > On Oct 18, 2016, at 8:13 AM, Michal Hocko wrote: >=20 >>=20 >=20 > yes, function_graph tracer will give you _some_ information but it = will > not have the context you are looking for, right? See the following > example >=20 > ------------------------------------------ > 0) x-www-b-22756 =3D> x-termi-4083=20 > ------------------------------------------ >=20 > 0) | __alloc_pages_nodemask() { > 0) | /* mm_page_alloc: page=3Dffffea000411b380 = pfn=3D1066702 order=3D0 migratetype=3D0 gfp_flags=3DGFP_KERNEL */ > 0) 3.328 us | } > 3) | __alloc_pages_nodemask() { > 3) | /* mm_page_alloc: page=3Dffffea0008f1f6c0 = pfn=3D2344923 order=3D0 migratetype=3D0 gfp_flags=3DGFP_KERNEL */ > 3) 1.011 us | } > 0) | __alloc_pages_nodemask() { > 0) | /* mm_page_alloc: page=3Dffffea000411b380 = pfn=3D1066702 order=3D0 migratetype=3D0 gfp_flags=3DGFP_KERNEL */ > 0) 0.587 us | } > 3) | __alloc_pages_nodemask() { > 3) | /* mm_page_alloc: page=3Dffffea0008f1f6c0 = pfn=3D2344923 order=3D0 migratetype=3D0 gfp_flags=3DGFP_KERNEL */ > 3) 1.125 us | } >=20 > How do I know which process has performed those allocations? I know = that > CPU0 should be running x-termi-4083 but what is running on other CPUs? >=20 > Let me explain my usecase I am very interested in. Say I that a = usespace > application is not performing well. I would like to see some = statistics > about memory allocations performed for that app - are there few = outliers > or the allocation stalls increase gradually? Where do we spend time = during > that allocation? Reclaim LRU pages? Compaction or the slab shrinkers? >=20 > To answer those questions I need to track particular events = (alocation, > reclaim, compaction) to the process and know how long each step > took. Maybe we can reconstruct something from the above output but it = is > a major PITA. If we either hard start/stop pairs for each step (which > we already do have for reclaim, compaction AFAIR) then this is an easy > scripting. Another option would be to have only a single tracepoint = for > each step with a timing information. >=20 > See my point? Yes, if we want to know what processes are running on what CPUs, echo funcgraph-proc > trace_options in the tracing directory should give = us what we want. The bash script which is part of this patch does this kind of setup for = you. As a result, the output you get is something like what you see here: = https://github.com/Jananiravichandran/Analyzing-tracepoints/blob/master/no= _tp_no_threshold.txt Does this answer your question? Let me know if otherwise. Janani. > --=20 > Michal Hocko > SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org