From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AA5CC2BBC7 for ; Fri, 10 Apr 2020 08:43:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DF888206F7 for ; Fri, 10 Apr 2020 08:43:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DF888206F7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=restena.lu Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 739C78E0034; Fri, 10 Apr 2020 04:43:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EAB48E0003; Fri, 10 Apr 2020 04:43:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 600038E0034; Fri, 10 Apr 2020 04:43:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0207.hostedemail.com [216.40.44.207]) by kanga.kvack.org (Postfix) with ESMTP id 49D3E8E0003 for ; Fri, 10 Apr 2020 04:43:46 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id F3F328248076 for ; Fri, 10 Apr 2020 08:43:45 +0000 (UTC) X-FDA: 76691307252.26.lead46_20d14b078425e X-HE-Tag: lead46_20d14b078425e X-Filterd-Recvd-Size: 3917 Received: from smtprelay.restena.lu (smtprelay.restena.lu [158.64.1.62]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Fri, 10 Apr 2020 08:43:45 +0000 (UTC) Received: from hemera.lan.sysophe.eu (unknown [IPv6:2001:a18:1:12::4]) by smtprelay.restena.lu (Postfix) with ESMTPSA id E40C540CEB; Fri, 10 Apr 2020 10:43:43 +0200 (CEST) Date: Fri, 10 Apr 2020 10:43:43 +0200 From: Bruno =?UTF-8?B?UHLDqW1vbnQ=?= To: Michal Hocko Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, Johannes Weiner , Vladimir Davydov , Chris Down Subject: Re: Memory CG and 5.1 to 5.6 uprade slows backup Message-ID: <20200410104343.5bcde519@hemera.lan.sysophe.eu> In-Reply-To: <20200410091525.287062fa@hemera.lan.sysophe.eu> References: <20200409112505.2e1fc150@hemera.lan.sysophe.eu> <20200409094615.GE18386@dhcp22.suse.cz> <20200409121733.1a5ba17c@hemera.lan.sysophe.eu> <20200409103400.GF18386@dhcp22.suse.cz> <20200409170926.182354c3@hemera.lan.sysophe.eu> <20200409152540.GP18386@dhcp22.suse.cz> <20200410091525.287062fa@hemera.lan.sysophe.eu> Organization: RESTENA Foundation X-Mailer: Claws Mail 3.16.0git939 (GTK+ 3.24.13; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Michal, Chris, Well, tar made me unhappy, it just collected list of files but not content from /sys/fs/cgroup/... But if I set memory.max =3D memory.high reclaim seems to work and memory pressure remains zero for the cg. If I set memory.max =3D $((memory.high + 128M)) memory pressure rises immediately (when memory.current ~=3D memory.high). Returning to memory.max=3Dmemory.high gets things running again and memory pressure starts dropping immediately. Could it be that the wrong limit of high/max is being used for reclaim? Bruno On Fri, 10 Apr 2020 09:15:25 +0200 Bruno Pr=C3=A9mont wrote: > Hi Michal, >=20 > On Thu, 9 Apr 2020 17:25:40 Michal Hocko wrote: > > Your earlier stat snapshot doesn't indicate a big problem with the > > reclaim though: > >=20 > > memory.stat:pgscan 47519855 > > memory.stat:pgsteal 44933838 > >=20 > > This tells the overall reclaim effectiveness was 94%. Could you try to > > gather snapshots with a 1s granularity starting before your run your > > backup to see how those numbers evolve? Ideally with timestamps to > > compare with the actual stall information. =20 >=20 > Attached is a long collection of > date memory.current memory.stat[pgscan] memory.stat[pgsteal] >=20 > It started while backup was running +/- smoothly with its memory.high > set to 4294967296 (4G instead of 2G) until backup finished around 20:22. >=20 > From system memory pressure RRD-graph I see pressure (around 60) > between about 19:50 to 20:10 while very small the rest of the time > (below 1). >=20 >=20 >=20 > I started a new backup run this morning grabbing full info snapshots of > backup cgroup at 1s interval in order to get a better/more complete > picture and CG's memory.high back to 2G limit. >=20 >=20 > I have the impression as if reclaim was somehow triggered not enough or > not strongly enough compared to the IO performed within the CG > (complete backup covers 130G of data, data being read in blocks of > 128kB at a smooth-running rate of ~7MiB/s). >=20 > > Another option would be to enable vmscan tracepoints but let's try with > > stats first. =20 >=20 >=20 > Bruno --=20 Bruno Pr=C3=A9mont Ing=C3=A9nieur syst=C3=A8me et d=C3=A9veloppements Fondation RESTENA 2, avenue de l'Universit=C3=A9 L-4365 Esch/Alzette T=C3=A9l: (+352) 424409 Fax: (+352) 422473 https://www.restena.lu https://www.dns.lu