From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: ** X-Spam-Status: No, score=2.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE492C3815B for ; Wed, 15 Apr 2020 12:17:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6D7F5214AF for ; Wed, 15 Apr 2020 12:17:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=magicleap.com header.i=@magicleap.com header.b="pMewSNmA"; dkim=pass (1024-bit key) header.d=magicleap.com header.i=@magicleap.com header.b="C4dm//SQ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6D7F5214AF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=magicleap.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C846B8E0016; Wed, 15 Apr 2020 08:17:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C35FB8E0001; Wed, 15 Apr 2020 08:17:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFC958E0016; Wed, 15 Apr 2020 08:17:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0060.hostedemail.com [216.40.44.60]) by kanga.kvack.org (Postfix) with ESMTP id 945538E0001 for ; Wed, 15 Apr 2020 08:17:56 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4CEBC180AD80F for ; Wed, 15 Apr 2020 12:17:56 +0000 (UTC) X-FDA: 76709990952.15.cord33_afe891904101 X-HE-Tag: cord33_afe891904101 X-Filterd-Recvd-Size: 16833 Received: from mx0a-001e9b01.pphosted.com (mx0b-001e9b01.pphosted.com [148.163.159.123]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Wed, 15 Apr 2020 12:17:55 +0000 (UTC) Received: from pps.filterd (m0088348.ppops.net [127.0.0.1]) by mx0b-001e9b01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 03FCHsee023091 for ; Wed, 15 Apr 2020 08:17:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=magicleap.com; h=mime-version : references : in-reply-to : from : date : message-id : subject : to : cc : content-type; s=pp09042018; bh=Ucs94/s0OCBIKvEGfhG0zgI08GuwORapetOXXQzG7j0=; b=pMewSNmAN1mlUV1jjmQAwUF2/EOjpLBtfB8qNQnVVTuGksdgYkwOC4uVp4Dtw34/rfcW Kpj/LIOlbAy42qcW8Re4QLYijTIJZr4qIc7GMFvW/kErwfp4xs/lAAJ7d9pmjxLJYTY7 Wo8VSNZ8eGAhqdTTQVF2Rji7w8QtVbL4RwmfitI28l/2ASfizy4HNFIYd3SZ/p+Wm17y 3EX8Yj9dQK8KkgKpybmN8dEpOxfAkZbBDczeIO9ELMFHWP92FgBMNymCu+liMs1MCjdH 5RcWi3scuOOhyG+gAyw9yCq8cpB5q2FDkjqBTXUUiNfBlrCncPShqkndRZAtryHkMhGA 8g== Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by mx0b-001e9b01.pphosted.com with ESMTP id 30dnbfrght-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=OK) for ; Wed, 15 Apr 2020 08:17:54 -0400 Received: by mail-qk1-f198.google.com with SMTP id k16so14769160qkk.16 for ; Wed, 15 Apr 2020 05:17:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=magicleap.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Ucs94/s0OCBIKvEGfhG0zgI08GuwORapetOXXQzG7j0=; b=C4dm//SQnEL2t6C9GA2mCEARb7Wbz+FMV9IbVCLKkf4Dp6z+cQmL3osGszQF3a4zHX xOvVIJpW+Ncv96wBWtILySvDt8wUzfsAb+BS90oZy+vyWEQvaHf+PTi2tOq6vyXhRiZM 0GSrvKiyUHRa9xfYWEcpgjjiI26O3CbVhJRfc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Ucs94/s0OCBIKvEGfhG0zgI08GuwORapetOXXQzG7j0=; b=OC8VXFVqHmrXatB2nfUac62/rpbCumlqIxJKufOmmrUDMdGSN0/hc3CSpwtNBcSJfx yyTRQzC8IOr136PVyNGWCVfc8uQ9CcgGUMANj21s89vA8An2nomdqYc32RVnCC+IaCkx byaL/mCdz/PmzYaQmyBtfenZcT5GOnAYeDk/fNK6CvJnP18XLQv9I8N2/KanSJFXKxo9 1Pt8coqT1Le5gp7FFLnVszJcq/L31bP+qnRTrYB4hcybm8BhRYJBqOWWNgl4RprfnAJz EMaZhas7WRAC+jqv2iImmoLcYnCB+UA5zuAS3TegdWejuwkDCYXp5mmgZ5JpGRMxyko9 7R/g== X-Gm-Message-State: AGi0PuYXfVk8ChFYlRyzuv7BvLHtvT8fQcg0N2mA07P3fmaRKeJX8yBr nheaf6XGAUx8MXPGtoF+fTa7eVp7LsH4d7NfcUeXCfs7jRxBI/rSutQrFmfBMYbwYX/rOb29ZV2 0OMeX6OnqSPqjqrmRRxoOnFxxxFs= X-Received: by 2002:a37:71c7:: with SMTP id m190mr26204994qkc.177.1586953073976; Wed, 15 Apr 2020 05:17:53 -0700 (PDT) X-Google-Smtp-Source: APiQypKNx0Dqosq3/FO+c0iB9h4P2f5XotMQi4LGmR2xJNr6BxAjGPtdRBXmBbXAwZUGuZ2dBs1SJRQqb+7pM6JriJk= X-Received: by 2002:a37:71c7:: with SMTP id m190mr26204966qkc.177.1586953073649; Wed, 15 Apr 2020 05:17:53 -0700 (PDT) MIME-Version: 1.0 References: <20200413215750.7239-1-lmoiseichuk@magicleap.com> <20200414113730.GH4629@dhcp22.suse.cz> <20200414184917.GT4629@dhcp22.suse.cz> <20200415075136.GY4629@dhcp22.suse.cz> In-Reply-To: <20200415075136.GY4629@dhcp22.suse.cz> From: Leonid Moiseichuk Date: Wed, 15 Apr 2020 08:17:42 -0400 Message-ID: Subject: Re: [PATCH 0/2] memcg, vmpressure: expose vmpressure controls To: Michal Hocko Cc: svc lmoiseichuk , Johannes Weiner , vdavydov.dev@gmail.com, tj@kernel.org, lizefan@huawei.com, cgroups@vger.kernel.org, akpm@linux-foundation.org, rientjes@google.com, minchan@kernel.org, vinmenon@codeaurora.org, andriy.shevchenko@linux.intel.com, anton.vorontsov@linaro.org, penberg@kernel.org, linux-mm@kvack.org Content-Type: multipart/alternative; boundary="0000000000003b059805a3534e83" X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-15_03:2020-04-14,2020-04-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 priorityscore=1501 lowpriorityscore=0 malwarescore=0 suspectscore=5 phishscore=0 clxscore=1015 mlxscore=0 impostorscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004150093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --0000000000003b059805a3534e83 Content-Type: text/plain; charset="UTF-8" As Chris Down stated cgroups v1 frozen, so no API changes in the mainline kernel. If opinions change in the future I can continue with polishing this change. I will focus on PSI bugs for swapless/zram swapped devices :) The rest is below. On Wed, Apr 15, 2020 at 3:51 AM Michal Hocko wrote: > On Tue 14-04-20 16:53:55, Leonid Moiseichuk wrote: > > It would be nice if you can specify exact numbers you like to see. > > You are proposing an interface which allows to tune thresholds from > userspace. Which suggests that you want to tune them. I am asking what > kind of tuning you are using and why cannot we use them as defaults in > the kernel. > Yes, this type of hack is obvious. But selecting some parameters in one moment of time might be not good later. Plus these patches can be applied by vendor to e.g. Android 8 or 9 who has no PSI and tweaked their own way. Some products stick to old versions of kernels. I made a docs in separate change to cover a wider set of older kernels. They are transparent, tested and working fine. > > On Tue, Apr 14, 2020 at 2:49 PM Michal Hocko wrote: > > > > > .... > > > > > As far I see numbers which vmpressure uses - they are closer to RSS of > > > > userspace processes for memory utilization. > > > > Default calibration in memory.pressure_level_medium as 60% makes 8GB > > > device > > > > hit memory threshold when RSS utilization > > > > reaches ~5 GB and that is a bit too early, I observe it happened > > > > immediately after boot. Reasonable level should be > > > > in the 70-80% range depending on SW preloaded on your device. > > > > > > I am not sure I follow. Levels are based on the reclaim ineffectivity > not > > > the overall memory utilization. So it takes to have only 40% reclaim > > > effectivity to trigger the medium level. While you are right that the > > > threshold for the event is pretty arbitrary I would like to hear why > > > that doesn't work in your environment. It shouldn't really depend on > the > > > amount of memory as this is a percentage, right? > > > > > It is not only depends from amount of memory or reclams but also what is > > software running. > > > > As I see from vmscan.c vmpressure activated from various shrink_node() > or, > > basically do_try_to_free_pages(). > > To hit this state you need to somehow lack memory due to various reasons, > > so the amount of memory plays a role here. > > In particular my case is very impacted by GPU (using CMA) consumption > which > > can easily take gigs. > > Apps can take gigabyte as well. > > So reclaiming will be quite often called in case of lack of memory (4K > > calls are possible). > > > > Handling level change will happen if the amount of scanned pages is more > > than window size, 512 is too little as now it is only 2 MB. > > So small slices are a source of false triggers. > > > > Next, pressure counted as > > unsigned long scale = scanned + reclaimed; > > pressure = scale - (reclaimed * scale / scanned); > > pressure = pressure * 100 / scale; > > Just to make this more obvious this is essentially > 100 * (1 - reclaimed/scanned) > > > Or for 512 pages (lets use minimal) it leads to reclaimed should be 204 > > pages for 60% threshold and 25 pages for 95% (as critical) > > > > In case of pressure happened (usually at 85% of memory used, and hittin > > critical level) > > I still find this very confusing because the amount of used memory is > not really important. It really only depends on the reclaim activity and > that is either the memcg or the global reclaim. And you are getting > critical levels only if the reclaim is failing to reclaim way too many > pages. > OK, agree from that point of view. But for larger systems reclaiming happens not so often and we can use larger window sizes to have better memory utilization approximation. > > > I rarely see something like closer to real numbers > > vmpressure_work_fn: scanned 545, reclaimed 144 <-- 73% > > vmpressure_work_fn: scanned 16283, reclaimed 2495 <-- same session but > 83% > > Most of the time it is looping between kswapd and lmkd reclaiming > failures, > > consuming quite a high amount of cpu. > > > > On vmscan calls everything looks as expected > > [ 312.410938] vmpressure: tree 0 scanned 4, reclaimed 2 > > [ 312.410939] vmpressure: tree 0 scanned 120, reclaimed 62 > > [ 312.410939] vmpressure: tree 1 scanned 2, reclaimed 1 > > [ 312.410940] vmpressure: tree 1 scanned 120, reclaimed 62 > > [ 312.410941] vmpressure: tree 0 scanned 0, reclaimed 0 > > This looks more like a problem of vmpressure implementation than > something you want to workaround by tuning to me. > Basically it is how it works - collect the scanned page and activate worker activity to update the current level. > > -- > Michal Hocko > SUSE Labs > -- With Best Wishes, Leonid --0000000000003b059805a3534e83 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
As Chris Down stated cgroups v1 frozen, so no API cha= nges in the mainline kernel.
If opinions change in=C2=A0the futur= e I can continue with polishing this change.
I will focus on PSI = bugs for swapless/zram swapped devices :)

The rest= is below.

On Wed, Apr 15, 2020 at 3:51 AM Michal Hocko <mhocko@kernel.org> wrot= e:
On Tue 14-04-= 20 16:53:55, Leonid Moiseichuk wrote:
> It would be nice if you can specify exact numbers you like to see.

You are proposing an interface which allows to tune thresholds from
userspace. Which suggests that you want to tune them. I am asking what
kind of tuning you are using and why cannot we use them as defaults in
the kernel.

Yes, this type of hack is o= bvious. But selecting some parameters in one moment of time might be not go= od later.
Plus these patches can be applied by vendor to e.g. And= roid 8 or 9 who has no PSI and tweaked their own way.
Some produc= ts stick to old versions of kernels. I made a docs in separate change to co= ver a wider set of older kernels.
They are transparent, tested an= d working fine.


> On Tue, Apr 14, 2020 at 2:49 PM Michal Hocko <mhocko@kernel.org> wrote:
>
> > ....
>
> > As far I see numbers which vmpressure uses - they are closer to R= SS of
> > > userspace processes for memory utilization.
> > > Default calibration in memory.pressure_level_medium as 60% m= akes 8GB
> > device
> > > hit memory threshold when RSS utilization
> > > reaches ~5 GB and that is a bit too early, I observe it happ= ened
> > > immediately after boot. Reasonable level should be
> > > in the 70-80% range depending on SW preloaded on your device= .
> >
> > I am not sure I follow. Levels are based on the reclaim ineffecti= vity not
> > the overall memory utilization. So it takes to have only 40% recl= aim
> > effectivity to trigger the medium level. While you are right that= the
> > threshold for the event is pretty arbitrary I would like to hear = why
> > that doesn't work in your environment. It shouldn't reall= y depend on the
> > amount of memory as this is a percentage, right?
> >
> It is not only depends from amount of memory or reclams but also what = is
> software running.
>
> As I see from vmscan.c vmpressure activated from various shrink_node()= =C2=A0 or,
> basically do_try_to_free_pages().
> To hit this state you need to somehow lack memory due to various reaso= ns,
> so the amount of memory plays a role here.
> In particular my case is very impacted by GPU (using CMA) consumption = which
> can easily take gigs.
> Apps can take gigabyte as well.
> So reclaiming will be quite often called in case of lack of memory (4K=
> calls are possible).
>
> Handling level change will happen if the amount of scanned pages is mo= re
> than window size, 512 is too little as now it is only 2 MB.
> So small slices are a source of false triggers.
>
> Next, pressure counted as
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unsigned long scale =3D scanned + rec= laimed;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0pressure =3D scale - (reclaimed * sca= le / scanned);
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0pressure =3D pressure * 100 / scale;<= br>
Just to make this more obvious this is essentially
=C2=A0 =C2=A0 =C2=A0 =C2=A0 100 * (1 - reclaimed/scanned)

> Or for 512 pages (lets use minimal) it leads to reclaimed should be 20= 4
> pages for 60% threshold and 25 pages for 95% (as critical)
>
> In case of pressure happened (usually at 85% of memory used, and hitti= n
> critical level)

I still find this very confusing because the amount of used memory is
not really important. It really only depends on the reclaim activity and that is either the memcg or the global reclaim. And you are getting
critical levels only if the reclaim is failing to reclaim way too many
pages.

OK, agree from that point of vi= ew.=C2=A0
But for larger systems reclaiming happens not so often = and we can use=C2=A0larger=C2=A0window=C2=A0sizes to have better memory uti= lization approximation.
=C2=A0

> I rarely see something like closer to real numbers
> vmpressure_work_fn: scanned 545, reclaimed 144=C2=A0 =C2=A0<-- 73%<= br> > vmpressure_work_fn: scanned 16283, reclaimed 2495=C2=A0 <-- same se= ssion but 83%
> Most of the time it is looping between kswapd and lmkd reclaiming fail= ures,
> consuming quite a high amount of cpu.
>
> On vmscan calls everything looks as expected
> [=C2=A0 312.410938] vmpressure: tree 0 scanned 4, reclaimed 2
> [=C2=A0 312.410939] vmpressure: tree 0 scanned 120, reclaimed 62
> [=C2=A0 312.410939] vmpressure: tree 1 scanned 2, reclaimed 1
> [=C2=A0 312.410940] vmpressure: tree 1 scanned 120, reclaimed 62
> [=C2=A0 312.410941] vmpressure: tree 0 scanned 0, reclaimed 0

This looks more like a problem of vmpressure implementation than
something you want to workaround by tuning to me.
Basi= cally it is how it works - collect the scanned page and activate worker act= ivity to update the current level.
=C2=A0

--
Michal Hocko
SUSE Labs


--
With Best Wishes,
Leonid
--0000000000003b059805a3534e83--