From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12FB6C2D0CE for ; Tue, 21 Jan 2020 09:00:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D290F217F4 for ; Tue, 21 Jan 2020 09:00:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D290F217F4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4DEE96B0003; Tue, 21 Jan 2020 04:00:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48E746B0005; Tue, 21 Jan 2020 04:00:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3570F6B0006; Tue, 21 Jan 2020 04:00:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0198.hostedemail.com [216.40.44.198]) by kanga.kvack.org (Postfix) with ESMTP id 1F7736B0003 for ; Tue, 21 Jan 2020 04:00:52 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id D57A9824805A for ; Tue, 21 Jan 2020 09:00:51 +0000 (UTC) X-FDA: 76401046302.30.bite38_248c440041d20 X-HE-Tag: bite38_248c440041d20 X-Filterd-Recvd-Size: 5588 Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Tue, 21 Jan 2020 09:00:51 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id u2so2079463wmc.3 for ; Tue, 21 Jan 2020 01:00:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=2lqF7a8plZ8V2KVyRIs80uv+I50WzzSKH8CLyPZhybY=; b=jAgXRD/+mtCFZxKg4ibrCiewR+3B/NS5VPQLmrcNY58lcP0INg4m6Ohee0tzcssvHj O3bkO7rkoMmH/b/o7k48bVGVwAuDAHMll+zq2rXUzcK70Hbc/7IF4MVr/wPv47ONlBqV 011/M8uKl89lemKJMCNG+8X8xGu4sOvTxUmxaEdduGuxt8gSzQYVTOZVcm0UuZbNCeZH yUSxA+JWXNSofy4uvcm7RpY9t9KUSb/vJNGd0Ej3xiLiyEYujjq9QJeLZxA6I8cgM4tY /qZPNJKpLeAoVoxaeePEUSPq6INuvbeu+FEJ17XlPXYim5jLDuKdj3ePUkrxxlJY+0wU OoXg== X-Gm-Message-State: APjAAAWGh45MClU+F80Ujs6o9ggZcYn2XWeyhOgbWIkIyMqSxpT1+5kn XbiKCGZ7m6Ncm6v9cFCUpiU= X-Google-Smtp-Source: APXvYqwwT2fszaSN+hspfYkrHGm4hGdKvDJG9i/v0amOZy7O+n/Kuo41UfeG82AoV+zBTIHM0CrJSg== X-Received: by 2002:a7b:c416:: with SMTP id k22mr3390379wmi.10.1579597250018; Tue, 21 Jan 2020 01:00:50 -0800 (PST) Received: from localhost (prg-ext-pat.suse.com. [213.151.95.130]) by smtp.gmail.com with ESMTPSA id j12sm55633547wrw.54.2020.01.21.01.00.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jan 2020 01:00:49 -0800 (PST) Date: Tue, 21 Jan 2020 10:00:48 +0100 From: Michal Hocko To: Cong Wang Cc: LKML , Andrew Morton , linux-mm , Mel Gorman , Vlastimil Babka Subject: Re: [PATCH] mm: avoid blocking lock_page() in kcompactd Message-ID: <20200121090048.GG29276@dhcp22.suse.cz> References: <20200109225646.22983-1-xiyou.wangcong@gmail.com> <20200110073822.GC29802@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon 20-01-20 14:48:05, Cong Wang wrote: > Hi, Michal > > On Thu, Jan 9, 2020 at 11:38 PM Michal Hocko wrote: > > > > [CC Mel and Vlastimil] > > > > On Thu 09-01-20 14:56:46, Cong Wang wrote: > > > We observed kcompactd hung at __lock_page(): > > > > > > INFO: task kcompactd0:57 blocked for more than 120 seconds. > > > Not tainted 4.19.56.x86_64 #1 > > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > kcompactd0 D 0 57 2 0x80000000 > > > Call Trace: > > > ? __schedule+0x236/0x860 > > > schedule+0x28/0x80 > > > io_schedule+0x12/0x40 > > > __lock_page+0xf9/0x120 > > > ? page_cache_tree_insert+0xb0/0xb0 > > > ? update_pageblock_skip+0xb0/0xb0 > > > migrate_pages+0x88c/0xb90 > > > ? isolate_freepages_block+0x3b0/0x3b0 > > > compact_zone+0x5f1/0x870 > > > kcompactd_do_work+0x130/0x2c0 > > > ? __switch_to_asm+0x35/0x70 > > > ? __switch_to_asm+0x41/0x70 > > > ? kcompactd_do_work+0x2c0/0x2c0 > > > ? kcompactd+0x73/0x180 > > > kcompactd+0x73/0x180 > > > ? finish_wait+0x80/0x80 > > > kthread+0x113/0x130 > > > ? kthread_create_worker_on_cpu+0x50/0x50 > > > ret_from_fork+0x35/0x40 > > > > > > which faddr2line maps to: > > > > > > migrate_pages+0x88c/0xb90: > > > lock_page at include/linux/pagemap.h:483 > > > (inlined by) __unmap_and_move at mm/migrate.c:1024 > > > (inlined by) unmap_and_move at mm/migrate.c:1189 > > > (inlined by) migrate_pages at mm/migrate.c:1419 > > > > > > Sometimes kcompactd eventually got out of this situation, sometimes not. > > > > What does this mean exactly? Who is holding the page lock? > > As I explained in other email, I didn't locate the process holding the page > lock before I sent out this patch, as I was fooled by /proc/X/stack. > > But now I got its stack trace with `perf`: > > ffffffffa722aa06 shrink_inactive_list > ffffffffa722b3d7 shrink_node_memcg > ffffffffa722b85f shrink_node > ffffffffa722bc89 do_try_to_free_pages > ffffffffa722c179 try_to_free_mem_cgroup_pages > ffffffffa7298703 try_charge > ffffffffa729a886 mem_cgroup_try_charge > ffffffffa720ec03 __add_to_page_cache_locked > ffffffffa720ee3a add_to_page_cache_lru > ffffffffa7312ddb iomap_readpages_actor > ffffffffa73133f7 iomap_apply > ffffffffa73135da iomap_readpages > ffffffffa722062e read_pages > ffffffffa7220b3f __do_page_cache_readahead > ffffffffa7210554 filemap_fault > ffffffffc039e41f __xfs_filemap_fault > ffffffffa724f5e7 __do_fault > ffffffffa724c5f2 __handle_mm_fault > ffffffffa724cbc6 handle_mm_fault > ffffffffa70a313e __do_page_fault > ffffffffa7a00dfe page_fault > > It got stuck somewhere along the call path of mem_cgroup_try_charge(), > and the trace events of mm_vmscan_lru_shrink_inactive() indicates this > too: So it seems that you are condending on the page lock. It is really unexpected that the reclaim would take that long though. Please try to enable more vmscan tracepoints to see where the time is spent. Thanks! -- Michal Hocko SUSE Labs