From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: ** X-Spam-Status: No, score=2.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6EC0C2BB1D for ; Tue, 14 Apr 2020 20:54:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4A4F82064A for ; Tue, 14 Apr 2020 20:54:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=magicleap.com header.i=@magicleap.com header.b="pE0kNpK8"; dkim=pass (1024-bit key) header.d=magicleap.com header.i=@magicleap.com header.b="IUT2rxOC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4A4F82064A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=magicleap.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D8C6A8E0003; Tue, 14 Apr 2020 16:54:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D13578E0001; Tue, 14 Apr 2020 16:54:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C02E08E0003; Tue, 14 Apr 2020 16:54:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A7EF78E0001 for ; Tue, 14 Apr 2020 16:54:09 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 63BD3824556B for ; Tue, 14 Apr 2020 20:54:09 +0000 (UTC) X-FDA: 76707663018.06.beam81_5e6b0731ca257 X-HE-Tag: beam81_5e6b0731ca257 X-Filterd-Recvd-Size: 16258 Received: from mx0a-001e9b01.pphosted.com (mx0b-001e9b01.pphosted.com [148.163.159.123]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Tue, 14 Apr 2020 20:54:08 +0000 (UTC) Received: from pps.filterd (m0088348.ppops.net [127.0.0.1]) by mx0b-001e9b01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 03EKqwh3025249 for ; Tue, 14 Apr 2020 16:54:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=magicleap.com; h=mime-version : references : in-reply-to : from : date : message-id : subject : to : cc : content-type; s=pp09042018; bh=q7sf2zZhI7hINgeOJoTzj0eLS1AarCrDwtE8BtI5Ryw=; b=pE0kNpK8ChkSuANt6SHd6EC47xzvoeOEK9ENWfiUvJHRyrt9+DlozJCYR4dw0TrfoXeT ARWql8nUIMN20D7ZgEDZ4XSxIyuS2khQy+zRjVkW+TMVTQpI1efXzzC25FY+C4PcTtpC /5CpZOsOGXQehC5cURxiC4icnU2lj6vzpUC0J6Cur6fUkgUlnvOpCO0hY0P5F7mNP9QH Ke3WVRl1GTc55l5Ae/Zar+LnFQfvzh2hVNrzO0WM3WsP/iyOIRxqz4YxY4kp7KfeFbfv ZmoQKoyekgkBmIYC67v3T9aEz17LHJ1KNENhC9BayiDyu9pekxZZrwvAaBFrToHWWyM6 0g== Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by mx0b-001e9b01.pphosted.com with ESMTP id 30b8rgjvcq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=OK) for ; Tue, 14 Apr 2020 16:54:07 -0400 Received: by mail-qt1-f198.google.com with SMTP id d17so10062501qto.6 for ; Tue, 14 Apr 2020 13:54:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=magicleap.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=q7sf2zZhI7hINgeOJoTzj0eLS1AarCrDwtE8BtI5Ryw=; b=IUT2rxOCKEI7qcjs0xWcsqIwfSXJgRELswkBaxHY6/U1Yxfr4cgxqUvbIHLlhffKf4 Wpz9+mVus/AZMb125Wjkqn+CzQqzSg+NqmjA9Dk3n34VebRKPGivSvTiOth4goFfjj01 8PwMajFtGXr5AaRPhoapOaP431+x6C1vfIqKE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=q7sf2zZhI7hINgeOJoTzj0eLS1AarCrDwtE8BtI5Ryw=; b=qlJH1uXf94sT/ZXovag4VHazbpW2MHPFTdisechA0ClShFtP+0ZGrZpGnIRjfhpV4i xyNtuaCWir91g0CBSIIpsvGRgAGbMLfZ2JH9z1JWOWm4l+CD6VRqb6ytBhyE1tFIrxSC AcTqYwBL/fEgHoA7sI8BFGV/lNPGT/A+9ywFoAqTwXUqdE6X92CbioqFog4j1giey9nm G0F+maH8KGoJ3R7UvPFo0YVitsK7siiGgGZ1xKDE29jlw6iKbgx/vXnTbOuL3c33H5MQ n6+VNlbYndbaqMTHixqVxwxkV4DOT4wRRrnD95NJgZxWBXWK9wUeSQqimI/oOnH+ms73 wA6g== X-Gm-Message-State: AGi0PuanzLzKGADQLXBvJFg2AxtwKYzvkgKxK8WcvF5oqfgSxBwC6+6H 1ucT2rDaHV/FfG5xs9qHAxYPcX46RvtfJbHfg7Qk/KLQ0DaP4y5kIS9ppsv4X3S1gfiRtHXVVLB 8v2Mq/StIt4BBUDP2a9VmyT9nxUM= X-Received: by 2002:a05:6214:227:: with SMTP id j7mr1944629qvt.85.1586897647100; Tue, 14 Apr 2020 13:54:07 -0700 (PDT) X-Google-Smtp-Source: APiQypLXWHh06dSh2E56lzg85urtMG6+G8jF2kAxIvFzhWDRtbak+ZrGitkRCAjzjpkBUogFzs0yBUwHi3MWjPtPnig= X-Received: by 2002:a05:6214:227:: with SMTP id j7mr1944598qvt.85.1586897646849; Tue, 14 Apr 2020 13:54:06 -0700 (PDT) MIME-Version: 1.0 References: <20200413215750.7239-1-lmoiseichuk@magicleap.com> <20200414113730.GH4629@dhcp22.suse.cz> <20200414184917.GT4629@dhcp22.suse.cz> In-Reply-To: <20200414184917.GT4629@dhcp22.suse.cz> From: Leonid Moiseichuk Date: Tue, 14 Apr 2020 16:53:55 -0400 Message-ID: Subject: Re: [PATCH 0/2] memcg, vmpressure: expose vmpressure controls To: Michal Hocko Cc: svc_lmoiseichuk@magicleap.com, hannes@cmpxchg.org, vdavydov.dev@gmail.com, tj@kernel.org, lizefan@huawei.com, cgroups@vger.kernel.org, akpm@linux-foundation.org, rientjes@google.com, minchan@kernel.org, vinmenon@codeaurora.org, andriy.shevchenko@linux.intel.com, anton.vorontsov@linaro.org, penberg@kernel.org, linux-mm@kvack.org Content-Type: multipart/alternative; boundary="0000000000008928ff05a34666b7" X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-14_10:2020-04-14,2020-04-14 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 lowpriorityscore=0 suspectscore=5 phishscore=0 adultscore=0 clxscore=1015 malwarescore=0 mlxlogscore=999 priorityscore=1501 spamscore=0 bulkscore=0 impostorscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004140148 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --0000000000008928ff05a34666b7 Content-Type: text/plain; charset="UTF-8" It would be nice if you can specify exact numbers you like to see. On Tue, Apr 14, 2020 at 2:49 PM Michal Hocko wrote: > .... > As far I see numbers which vmpressure uses - they are closer to RSS of > > userspace processes for memory utilization. > > Default calibration in memory.pressure_level_medium as 60% makes 8GB > device > > hit memory threshold when RSS utilization > > reaches ~5 GB and that is a bit too early, I observe it happened > > immediately after boot. Reasonable level should be > > in the 70-80% range depending on SW preloaded on your device. > > I am not sure I follow. Levels are based on the reclaim ineffectivity not > the overall memory utilization. So it takes to have only 40% reclaim > effectivity to trigger the medium level. While you are right that the > threshold for the event is pretty arbitrary I would like to hear why > that doesn't work in your environment. It shouldn't really depend on the > amount of memory as this is a percentage, right? > It is not only depends from amount of memory or reclams but also what is software running. As I see from vmscan.c vmpressure activated from various shrink_node() or, basically do_try_to_free_pages(). To hit this state you need to somehow lack memory due to various reasons, so the amount of memory plays a role here. In particular my case is very impacted by GPU (using CMA) consumption which can easily take gigs. Apps can take gigabyte as well. So reclaiming will be quite often called in case of lack of memory (4K calls are possible). Handling level change will happen if the amount of scanned pages is more than window size, 512 is too little as now it is only 2 MB. So small slices are a source of false triggers. Next, pressure counted as unsigned long scale = scanned + reclaimed; pressure = scale - (reclaimed * scale / scanned); pressure = pressure * 100 / scale; Or for 512 pages (lets use minimal) it leads to reclaimed should be 204 pages for 60% threshold and 25 pages for 95% (as critical) In case of pressure happened (usually at 85% of memory used, and hittin critical level) I rarely see something like closer to real numbers vmpressure_work_fn: scanned 545, reclaimed 144 <-- 73% vmpressure_work_fn: scanned 16283, reclaimed 2495 <-- same session but 83% Most of the time it is looping between kswapd and lmkd reclaiming failures, consuming quite a high amount of cpu. On vmscan calls everything looks as expected [ 312.410938] vmpressure: tree 0 scanned 4, reclaimed 2 [ 312.410939] vmpressure: tree 0 scanned 120, reclaimed 62 [ 312.410939] vmpressure: tree 1 scanned 2, reclaimed 1 [ 312.410940] vmpressure: tree 1 scanned 120, reclaimed 62 [ 312.410941] vmpressure: tree 0 scanned 0, reclaimed 0 > > > From another point of view having a memory.pressure_level_critical set to > > 95% may never happen as it comes to a level where an OOM killer already > > starts to kill processes, > > and in some cases it is even worse than the now removed Android low > memory > > killer. For such cases has sense to shift the threshold down to 85-90% to > > have device reliably > > handling low memory situations and not rely only on oom_score_adj hints. > > > > Next important parameter for tweaking is memory.pressure_window which has > > the sense to increase twice to reduce the number of activations of > userspace > > to save some power by reducing sensitivity. > > Could you be more specific, please? > That are parameters which most sensitive for tweaking for me. At least someone who use vmpressure will be able to tune up or down depending on combination apps. > > > For 12 and 16 GB devices the situation will be similar but worse, based > on > > fact in current settings they will hit medium memory usage when ~5 or 6.5 > > GB memory will be still free. > > > > > > > > > > Anyway, I have to confess I am not a big fan of this. vmpressure turned > > > out to be a very weak interface to measure the memory pressure. Not > only > > > it is not numa aware which makes it unusable on many systems it also > > > gives data way too late from the practice. > > > > > > Btw. why don't you use /proc/pressure/memory resp. its memcg > counterpart > > > to measure the memory pressure in the first place? > > > > > > > According to our checks PSI produced numbers only when swap enabled e.g. > > swapless device 75% RAM utilization: > > I believe you should discuss that with the people familiar with PSI > internals (Johannes already in the CC list). > Thanks for pointing, I will answer for his letters. > -- > Michal Hocko > SUSE Labs > -- With Best Wishes, Leonid --0000000000008928ff05a34666b7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
It would be nice if you can specify exact numbers you= like to see.

On Tue, Apr 14, 2020 at 2:49 PM Michal Hocko <mhocko@kernel.org> wrote:
....> As far I see numbers which vmpr= essure uses - they are closer to RSS of
> userspace processes for memory utilization.
> Default calibration in memory.pressure_level_medium as 60% makes 8GB d= evice
> hit memory threshold when RSS utilization
> reaches ~5 GB and that is a bit too early, I observe it happened
> immediately after boot. Reasonable level should be
> in the 70-80% range depending on SW preloaded on your device.

I am not sure I follow. Levels are based on the reclaim ineffectivity not the overall memory utilization. So it takes to have only 40% reclaim
effectivity to trigger the medium level. While you are right that the
threshold for the event is pretty arbitrary I would like to hear why
that doesn't work in your environment. It shouldn't really depend o= n the
amount of memory as this is a percentage, right?
It is= not only depends from amount of memory or reclams but also what is softwar= e running.

As I see from vmscan.c vmpressure activ= ated from various shrink_node()=C2=A0 or, basically do_try_to_free_pages().=
To hit this state you need to somehow lack memory due to various= reasons, so the amount of memory plays a role here.
In particula= r my case is very impacted=C2=A0by GPU (using CMA) consumption which can ea= sily=C2=A0take=C2=A0gigs.=C2=A0
Apps can take gigabyte as wel= l.
So reclaiming will be quite often called in case of lack of me= mory (4K calls are possible).

Handling level chang= e will happen if the amount of scanned pages is more than window size, 512 = is too little as now=C2=A0it is only 2 MB.
So small slices are a = source of false triggers.

Next, pressure counted a= s
=C2=A0 =C2=A0 =C2=A0 =C2=A0 unsigned long scale =3D scanned + r= eclaimed;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 pressure =3D scale - (r= eclaimed * scale / scanned);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 pressure =3D pr= essure * 100 / scale;
Or for 512 pages (lets use minimal) it = leads to reclaimed should be 204 pages for 60% threshold and 25 pages for 9= 5% (as critical)

In case of pressure happened (usu= ally at 85% of memory used, and hittin critical level) I rarely see somethi= ng like closer to real numbers
vmpressure_work_fn: scanned 545, r= eclaimed 144=C2=A0 =C2=A0<-- 73%
vmpressure_work_fn: scann= ed 16283, reclaimed 2495=C2=A0 <-- same session but 83%
Mo= st of the time it is looping between kswapd and lmkd reclaiming failures, c= onsuming quite a high amount of cpu.

On vmscan= calls everything looks as expected
[ =C2=A0312.410938] vmpressur= e: tree 0 scanned 4, reclaimed 2
[ =C2=A0312.410939] vmpressure: tree 0 = scanned 120, reclaimed 62
[ =C2=A0312.410939] vmpressure: tree 1 scanned= 2, reclaimed 1
[ =C2=A0312.410940] vmpressure: tree 1 scanned 120, recl= aimed 62
[ =C2=A0312.410941] vmpressure: tree 0 scanned 0, reclaimed 0
=C2=A0

> From another point of view having a memory.pressure_level_critical set= to
> 95% may never happen as it comes to a level where an OOM killer alread= y
> starts to kill processes,
> and in some cases it is even worse than the now removed Android low me= mory
> killer. For such cases has sense to shift the threshold down to 85-90%= to
> have device reliably
> handling low memory situations and not rely only on oom_score_adj hint= s.
>
> Next important parameter for tweaking is memory.pressure_window which = has
> the sense to increase twice to reduce the number of activations of use= rspace
> to save some power by reducing sensitivity.

Could you be more specific, please?
That are parameter= s which most sensitive for tweaking for me.
At least someone who = use vmpressure=C2=A0will be able to tune up or down depending on combinatio= n apps.

=C2=A0

> For 12 and 16 GB devices the situation will be similar but worse, base= d on
> fact in current settings they will hit medium memory usage when ~5 or = 6.5
> GB memory will be still free.
>
>
> >
> > Anyway, I have to confess I am not a big fan of this. vmpressure = turned
> > out to be a very weak interface to measure the memory pressure. N= ot only
> > it is not numa aware which makes it unusable on many systems it a= lso
> > gives data way too late from the practice.
> >
> > Btw. why don't you use /proc/pressure/memory resp. its memcg = counterpart
> > to measure the memory pressure in the first place?
> >
>
> According to our checks PSI produced numbers only when swap enabled e.= g.
> swapless device 75% RAM utilization:

I believe you should discuss that with the people familiar with PSI
internals (Johannes already in the CC list).

Thanks for pointing, I will answer for his letters.=C2=A0
--
Michal Hocko
SUSE Labs


--
With Best Wishes,
Leonid
--0000000000008928ff05a34666b7--