From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751170AbdE3NeB (ORCPT ); Tue, 30 May 2017 09:34:01 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:50904 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750890AbdE3Nd7 (ORCPT ); Tue, 30 May 2017 09:33:59 -0400 Authentication-Results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=fb.com; Date: Tue, 30 May 2017 14:33:35 +0100 From: Roman Gushchin To: Michal Hocko CC: Tetsuo Handa , Johannes Weiner , Vladimir Davydov , , , Subject: Re: [PATCH] mm,oom: add tracepoints for oom reaper-related events Message-ID: <20170530133335.GB28148@castle> References: <1496145932-18636-1-git-send-email-guro@fb.com> <20170530123415.GF7969@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20170530123415.GF7969@dhcp22.suse.cz> User-Agent: Mutt/1.5.24 (2015-08-30) X-Originating-IP: [2620:10d:c092:200::1:255a] X-ClientProxiedBy: DB6PR07CA0174.eurprd07.prod.outlook.com (10.166.153.156) To DM3PR15MB1082.namprd15.prod.outlook.com (10.166.160.136) X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM3PR15MB1082: X-MS-Office365-Filtering-Correlation-Id: 66a1763c-b245-4267-c5cc-08d4a7608058 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:DM3PR15MB1082; X-Microsoft-Exchange-Diagnostics: 1;DM3PR15MB1082;3:WzTL5VaMO8zTRVBBET/Cn+8ccgqRo73FaygVuFhsa4bzpKJwFsYMYuxrte7SUWpJmfcwn8xr19GKha2cGWQURFo6QzeCm3+bRJ4NwKRLJ6ttVZvAJ6MT2RdBjtLSj9zh48aMcvRzQGfVuqT08fp9FFt2jgaMSp57VnOpOB3yXCJ/wbKwQnh37rY0veJDqie2f5i7gSCdi0bIzehrG1k3G+++NJr0MZk1RX7gAX7OjrnkLTSjPqATGlEalb4ISSXxK2EvlWwDR+NmDvuhR/fCRIEzC0X6/daf7kbZcq+wDa04VQU7fgY9Tz5qGM+vWx46Bg7K/i82SV7PXCWdb57mqQ==;25:eE1oWUn8TitJOGK7MFRFxW5+NRFOPg9GPBJyObyHBPbFsXH93SWqCHwvjcQJ0gKWxhmEO8ZiFrec8lUWP5HegzgxZe+iwCZOkVlLlgzbFVfTcL2P5/hwNTFojoHeJJV0Z/FjHlDzcbcgDVlQCOg8b0QG7yrGAGK946XhSWAZi1tRCKP8MhFHT1zhX0rGcQXv54NflADXztQBy19AC1EWn3TsrrNqZK0vxqKSEsUzfhyH++MeNndt9rMVkuk2rl+znzcbUBuwVOVgYWH7SNiSGJk7UpJozcCxQWzHr+4CwRaXPWsixrGiFBcR9G73nZ6cNGJPUx3KEenb7Jt+c5JmwLqBA4T8apr0xXCtCsVfRou6RdpzRml0zQYcPz0AsFxnyPDPvdR1EiGfjaqT4Bcj5MOTMOFae/3AJyPdSOrM9GLjWJJgxmpU9425n7hwzOBPhphdq9NLNwa7/hzp3dKA/4yqsU7UUpVNJcFQ4ajl4Gg= X-Microsoft-Exchange-Diagnostics: 1;DM3PR15MB1082;31:FwcNRfDpo0F5woofcBfs7LIC1+gvTVxJNVF+ofnb+7De8l+KPvEPqYxPlA7FxeRTCoP5tKDENOcSN03weApllfx636THYJcM3HBtK8FnAuoOzMS96UA/VtL5v7EY9f8m5lKsvBGGLbuEdPuzj8mzp1trgFqoD540HC+PNCTO8Un8accyN06guJk470zPSm4PkYiqOYkqts4hIBCoXEinIAu8bF/iPNFmDQ+zVobPlHM=;20:tExduWRCQOsU+YX7iigOy4qmGeFaMmuI0+AE1+nhW1uwoP10MhjGONRA8FOVp0s2jmzNU+iMF6yncuFBNmMZzNyv7Jb7YBBpvf9QwjqmZYXUThR2NXxfreLdbE2C5c3p1ipnPfgNBfRSntIFX+iOU0KACFtD+PrG7ngF4PObZ+7k9fGO0aaPhXL4oLal7fZMTiBELjdSaDL8rJyvov3TGvwz71SMVp0o2gclJ2pknnwu34vyhap6PMtLraiVnZCScc+JFw8NZvqv92jaSnbURUyYvQ+Y1zQto47BV22iXZxv86fKTh2ju1xf1ZoAPx5MLVjKVpFm0bovvBmWahj3D3Zmsnfz37hQBgOfsXeRfydOSApjw779tlk5KlB+xBfDiyv7o3uwZqOXCESw9V3aiwxuLHp4blws6jJuto3M8m8fTMsNQZevBi4GXD4RYYsSa4FWo0rliw1cGVAQIa55CsQnhiqtYqsBEFX4cqX0fEpAGvnUKGz/kQWCc8Q98QkX X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123558100)(20161123562025)(20161123555025)(20161123564025)(6072148);SRVR:DM3PR15MB1082;BCL:0;PCL:0;RULEID:;SRVR:DM3PR15MB1082; X-Microsoft-Exchange-Diagnostics: 1;DM3PR15MB1082;4:PPHDlQwx2VzJMkCFPaxEqQLK0tEW7e6VHbD62ojOtf3scefrKMsUrLCWInatmOJP0IHeXS7FYtjLBRU53iJ49rW3DULK6/JlvZx5srJFnGffAtoLTnJC1sjDiVt6gxnxENYTcjUPnQcTpeVOsSr6MMqlTPY47uqLdf7pHiUE15XH9aG+YMHUZ3RgfsFjrrZpFIaSTnK0BfrpZBo4R/3s7RIx6hES0QpaX2aCYQ35kBIPBESLg27+nQPM4Ur9+xkWnAH3xGyJJEbFEBD9drQU8pHNL7MWS6qbGsX3UAY+Z8nrJe1MGwmmQDtwMD06McQxIJHw6K6f3qndwHms8lpj83Y7mmNJWX0X6xhzD26r2Jg0vV3nkJd69ucGynFb0RaTxMRZz4PDVruQeyG52rKB5ztxcwFFYQFLxIclG905AO5mhXjm2ARsXz6b1OL0bIXYnZ4jP/4Xg7shSUDyhulHylUiwOJrw4dkJqRZuNv0T7VwUMrgzgm+smtAgw0H3wBF4NbvSXpBGhSSC1VHzGY+3PhjLPD7oN7Jbp6G+xzy4lDGKMl7CAlQ8pGhHorgzOpM0xcD0gKFKC6VRGrU424XcqE9jq2iH9UP3715fqPR+qanxhFzu/fnSrCFVWPCS5mn1Dx3tSr0hkrTH/CMxUxUUVOvW3jGVyEF4wUcHWbyfRZaBLHShOO/MEiNTuO0mng0ucbX2pSHlnCMR11sf3wZqUDA2PQDGeajf5o6RIiBNvaXQAdgggY8NOcSzTyhZH/KKjyATmwz3cjVI6xGRrCC53ECNqBrAaYLQa1eUt2QzEQ= X-Forefront-PRVS: 032334F434 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(39450400003)(39400400002)(39850400002)(39410400002)(39840400002)(24454002)(377424004)(7736002)(189998001)(305945005)(54906002)(55016002)(81166006)(6306002)(33716001)(4326008)(5660300001)(4001350100001)(42186005)(86362001)(83506001)(25786009)(6496005)(6246003)(6666003)(54356999)(76176999)(53936002)(50986999)(6916009)(2950100002)(8676002)(9686003)(50466002)(1076002)(6116002)(229853002)(966005)(2906002)(33656002)(478600001)(47776003)(38730400002)(110136004)(18370500001)(142933001)(42262002);DIR:OUT;SFP:1102;SCL:1;SRVR:DM3PR15MB1082;H:castle;FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;DM3PR15MB1082;23:BRGB4ZVu0zHs0+fUfoCv44F6+sLE6JdaScwDzTWXz?= =?us-ascii?Q?+DQBTLT3vGLzY/tNtF1othSmYkhHpeoy2gnWb11uNUR397cmW7/ljoblX7c3?= =?us-ascii?Q?LplFaGmwDAbW8QGOMvXFpuLCXIXs2Ee0L9/OgnNY7YyVXnygv2CJ1dhMOdR6?= =?us-ascii?Q?ML/rgFZhV+xl7VM/nIYWWgJlZPW5PJNhHSlPUaWwZcbAoPPKYgaMEUiFaIvA?= =?us-ascii?Q?nTbz/+TfEsULzOJ9Oi83QEg2NXX3S7XMa0T83Fo9eVwYfJXkwcVlloPXu2kV?= =?us-ascii?Q?7PDMc4h+Wx0Ul4wloGBroTkHbAp2nlpcfuG7po9lCKntRu2eS4u9dDezfjPN?= =?us-ascii?Q?k7mWD9VOawpPee7KXykYTtLXB2TRg+XkHvkbQnoL2Z5F+DQ5h4V/G0tSyAHQ?= =?us-ascii?Q?CI1MQNfTUE//7gQEplhvEClcWAZYGn+aWsiH/67bklTUe+dZU27E31bxU2rN?= =?us-ascii?Q?x++oNJbGuXxVz0zdUPN+VDkvqQx1q5YfG5CeIOtq+u1rzrly6mF+8dds8PY2?= =?us-ascii?Q?Cm3ofRIiXtmoMWBBhzWJjvApvSpna7M3VqAmgky+6GoW2Au3U+LpVKc465v5?= =?us-ascii?Q?OlX5HEZb+oKC+dM2F8U64hld5gmm1ZmT8HERdretzbNubM6K//fyjnIHXK+1?= =?us-ascii?Q?ZZwf0c0AXac7Pj4c7sHPp2eVtltiGYQIfEl/cqVAgdv2POLL+QFILx9Flojs?= =?us-ascii?Q?zV68U1DPdC7HcejDvQRlCB0CIQP/ea/DFDbjBZQ6Xo0Se8oQqKGluaAcfFku?= =?us-ascii?Q?OPXWUUCCcoz08h7nmaFxotyOuiklJqpXCji52i+NpuOpW50/jNXDYjvVstWO?= =?us-ascii?Q?yZYr+4VjdtkQ7pLSXQL7CWdaeEptjkdFRyeYwdlT9B33cjR72gzNy91EL9pj?= =?us-ascii?Q?nG/GziPJ12/fKUkx7zglfRQMHFncjXCTQNksq3X6R87MgXittIK+atb7QmvW?= =?us-ascii?Q?LUbYoztowreJRPjktDFo6ZyvBQ8j+O+ZSKs82eoxMwhdPVGHOmCN1I3x2c7A?= =?us-ascii?Q?6hMbp+9+IvniPm1c9fK9UO0FpP+ikEFFe7ymZssbcgY/QIx4IYTRSCQdQ5j9?= =?us-ascii?Q?1aPz+GGgEf7Btdz8QBcXKFX03HAeBxrqTZ5Fqtv6o4LOvxJuhsikXoBMVmPL?= =?us-ascii?Q?IXRzLG1N9vQhbqkuAgRzSKQBvRkU9WmShTz8aBfAuFRhX4VhGfQL3n5FWIcu?= =?us-ascii?Q?7gIk6xvoH4M1+ir3s5AARMnVU0K9/VLV8N1dwKIAVZy1/uaLKch1plWeQ=3D?= =?us-ascii?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1;DM3PR15MB1082;6:ou+LWNGerr2cDXmJet3xUnceByStaBc3Uy8OuQ4ym5jGp2Awi+8v1QhFJtm7Ho3Y6yydV3ZuWkeUKUEvcItodewyBEbc33fN3QFjpvqsGkhMeBs/odXhNRFk1Ms44ZPipsBFlPutOYN4yZg1qSCg+u9now7ihqs9LVXuHcmyPouIuRNwrwNeHZrYdN9uIH0riP+x2yvvI7+tIPiOX4qOHYhxtQP+jH3LrBbzJAuIFIQ+Pth41rnx6GsvJJDlnBvpDca44pwpi87Vzamf+jxUzUqkTp4+M0TB5hhCOdZpPBwSBWIW5WSs66uunl8Ik+TnHhOOzRijf02cwShkTarCvyXtM/Rse6zcsCG5F8+h5q3ggXx0DOFvZkqk7u+EtR46xd85sDDVuiXgpGLNWp4uHGvGv+BVQVZX5aTr4Ivf0hbFjYXamEql40nKOWoAMppyz6eql4MeieGkaKciFrViffGk0LDyiRD1Z4EkDoa9JV46FADy5ApGpu1JQbLSsQ90+KRCk2IO2zVnZg4uVwifYw==;5:KFrjzMJuZMzGS3eqs2PRFMv0ej7L8UsQ4V3g9MSAvtArzD2hF725bxi9zEACiYlVXc2vtx+TSVg5CZt2Xbyi3kuuf/WNQeS3uLXsFfFPiKIVqJYi2p87YGX8K82OQh5JdWWGHj9Qylz0YRggihILZQ==;24:Fz8iavc5PN+5HiWGChTSFolkbGFN+Ut0y3Q60CS1pp9AYl6Iz95NYz4qgj5y5tg98JdWfFa8unjmaahin8tVit6lL/fgriJpTN14cRGdlhs= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM3PR15MB1082;7:6W6IYL9X8cTUqAJfXKZd8Tu3iXq9aJrZkVoZF26pVzl9MtHcCs7Ll9TJUycWyst8ZBJ1xieB2EAhSFDUBVr1+6krCUKuYxum9qBeBXjHUzqt7XGT+5uIyuut+unzNNwpJMYDvoBx+dcIhB9rthTSdKMNrXHnsAlvXcU5tZBcTZuCHhUGcd/WcrgUJuFJjX8bs5xJk1J7VN+XZzsJpCYFAbIedQZYrxBzxdHrc6DZSGoxbtft5e9VhBWS0Te1Yq7bt7h12N80wOc6H0ohHwu3omfrT8OcdG7PYoCPkWMOZ+sFOGvvPOtWyrMOV3JDwB9jzfxeGA9jVMJrEuxxwaUzPw==;20:Qjk0JvPR/YQlTVgNHEXVAt7GVmMnGoPL/TdOElnwXrqndoVCBYww/fgb7A1j41pZjGXzx3XhvGKlDhFSHmsNxRJR3UnRh5ECPSyLCKD/II1Rea9F4zDM+kJT0G5U7EieMU+wIgGATwBvobmY/+XeOUo2KK6DY0qOwJ5mhDat92Q= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2017 13:33:46.6676 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM3PR15MB1082 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-30_09:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, May 30, 2017 at 02:34:16PM +0200, Michal Hocko wrote: > On Tue 30-05-17 13:05:32, Roman Gushchin wrote: > > Add tracepoints to simplify the debugging of the oom reaper code. > > > > Trace the following events: > > 1) a process is marked as an oom victim, > > 2) a process is added to the oom reaper list, > > 3) the oom reaper starts reaping process's mm, > > 4) the oom reaper finished reaping, > > 5) the oom reaper skips reaping. > > I am not against but could you explain why the current printks are not > sufficient? We do not have any explicit printk for the 2) and 3) but > are those really necessary? We also don't have any printks for 1) and 2) if, for, instance, we call out_of_memory() and task_will_free_mem(current) returns true. > > In other words could you describe the situation when you found these > tracepoints more useful than what the kernel log offers already? During my work on cgroup-aware OOM killer and some issues discovered in process (which are described in https://lkml.org/lkml/2017/5/17/542; most important problem fixed by Tetsuo), I've found an existing debug output insufficient and sometimes too bulky. Suggested traces allowed me to debug issues like I've met (double invocation of oom_reaper, etc) much easier. Thanks! Roman From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f200.google.com (mail-pf0-f200.google.com [209.85.192.200]) by kanga.kvack.org (Postfix) with ESMTP id A07536B0279 for ; Tue, 30 May 2017 09:33:59 -0400 (EDT) Received: by mail-pf0-f200.google.com with SMTP id e131so95590417pfh.7 for ; Tue, 30 May 2017 06:33:59 -0700 (PDT) Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id h18si13335202pfd.167.2017.05.30.06.33.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 30 May 2017 06:33:58 -0700 (PDT) Date: Tue, 30 May 2017 14:33:35 +0100 From: Roman Gushchin Subject: Re: [PATCH] mm,oom: add tracepoints for oom reaper-related events Message-ID: <20170530133335.GB28148@castle> References: <1496145932-18636-1-git-send-email-guro@fb.com> <20170530123415.GF7969@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20170530123415.GF7969@dhcp22.suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: Tetsuo Handa , Johannes Weiner , Vladimir Davydov , kernel-team@fb.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org On Tue, May 30, 2017 at 02:34:16PM +0200, Michal Hocko wrote: > On Tue 30-05-17 13:05:32, Roman Gushchin wrote: > > Add tracepoints to simplify the debugging of the oom reaper code. > > > > Trace the following events: > > 1) a process is marked as an oom victim, > > 2) a process is added to the oom reaper list, > > 3) the oom reaper starts reaping process's mm, > > 4) the oom reaper finished reaping, > > 5) the oom reaper skips reaping. > > I am not against but could you explain why the current printks are not > sufficient? We do not have any explicit printk for the 2) and 3) but > are those really necessary? We also don't have any printks for 1) and 2) if, for, instance, we call out_of_memory() and task_will_free_mem(current) returns true. > > In other words could you describe the situation when you found these > tracepoints more useful than what the kernel log offers already? During my work on cgroup-aware OOM killer and some issues discovered in process (which are described in https://lkml.org/lkml/2017/5/17/542; most important problem fixed by Tetsuo), I've found an existing debug output insufficient and sometimes too bulky. Suggested traces allowed me to debug issues like I've met (double invocation of oom_reaper, etc) much easier. Thanks! Roman -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org