From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AF91C433EF for ; Mon, 27 Sep 2021 09:31:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6C18061157 for ; Mon, 27 Sep 2021 09:31:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233669AbhI0Jcj (ORCPT ); Mon, 27 Sep 2021 05:32:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:39356 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233683AbhI0Jcg (ORCPT ); Mon, 27 Sep 2021 05:32:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632735058; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K7NSp/vWgh0IQXSwoMraW7CkaYyKMLgroGH9C/clVag=; b=KWtgXikx+M+yWLCBQOgGhRWsrpaNCGbtE/wczN+bVtilwx3Le3JBkCuccCC5TixydqqIql iP0BaFR15SNMUrNYYBVdlS5KadEjHo7jgVU03YSWjez5Q9av7FYSusCuDnSoGK8xv1nDjp AeOJozRgMP8aj5rgmSp2Fjg2L223Njw= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-421-L6z1zNCwPnmzJCV3XmF6nQ-1; Mon, 27 Sep 2021 05:30:56 -0400 X-MC-Unique: L6z1zNCwPnmzJCV3XmF6nQ-1 Received: by mail-wr1-f69.google.com with SMTP id e1-20020adfa741000000b0015e424fdd01so13822125wrd.11 for ; Mon, 27 Sep 2021 02:30:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=K7NSp/vWgh0IQXSwoMraW7CkaYyKMLgroGH9C/clVag=; b=1Acxv3D68gOnKMASKd1ROof5VK/395GCmLLM694Qf/25TaijhcQ29wTUCOj+4feZ5D wj5DemPKSRA+62iWM3cDME71hievAnRvuxexWbgkqv7ZHgnE1bzTvDDRZH4CpRIZcwYN /KhQtVBcUxqC0rM7M6FYRy+QxFNQKuYXtnqSbYublLp92WSI/QfVkXh/MJAgZy9jOi03 maJsMQ56gWYOs/8xKlXDWRwRvHlk6NoE8lTn14GCNZKICqfCh/sTp1pvRX9xMXD8F5dx BT2gLJDC4ajs4pD4abtgrwPGm1DuUJkEi3ByoKcDhrnI1GeJwKiyel6vDbtrqNY58Zan NyNw== X-Gm-Message-State: AOAM533XQYO6QKZTIiabmtCC0kBTXK3HaDj0pDutdETDt5MNfP1HNvpD 1MhU0Eq49yZRILtMHt/VTnke+7bQa7Cn2bQqrmvq8/cEf6SBENsPqJ7kk13vJEI/nDmzxh9Kdh9 pUJnL0JDfYSDwLnOYccqqftzk X-Received: by 2002:a5d:58ec:: with SMTP id f12mr26326686wrd.24.1632735054900; Mon, 27 Sep 2021 02:30:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx6b8PsqrofWNdXOOdYGB7TMIOeoLXw6T/j3GuUxYOkzE46cemEL5lj+Ydtl2K8rifSKmavPg== X-Received: by 2002:a5d:58ec:: with SMTP id f12mr26326655wrd.24.1632735054634; Mon, 27 Sep 2021 02:30:54 -0700 (PDT) Received: from ?IPv6:2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123? ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id n186sm19979340wme.31.2021.09.27.02.30.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Sep 2021 02:30:54 -0700 (PDT) Message-ID: <8ee79b78c7f7f8fa18ec237494f09492ef60081f.camel@redhat.com> Subject: Re: [PATCH 0/6] mm: Remote LRU per-cpu pagevec cache/per-cpu page list drain support From: nsaenzju@redhat.com To: Thomas Gleixner , Peter Zijlstra , Vlastimil Babka Cc: akpm@linux-foundation.org, frederic@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, cl@linux.com, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, bigeasy@linutronix.de, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org Date: Mon, 27 Sep 2021 11:30:53 +0200 In-Reply-To: <87v92sgt3n.ffs@tglx> References: <20210921161323.607817-1-nsaenzju@redhat.com> <20210922112817.GO4323@worktop.programming.kicks-ass.net> <87v92sgt3n.ffs@tglx> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.40.4 (3.40.4-1.fc34) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Thomas, thanks for the in-depth review. On Thu, 2021-09-23 at 00:09 +0200, Thomas Gleixner wrote: > On Wed, Sep 22 2021 at 13:28, Peter Zijlstra wrote: > > On Tue, Sep 21, 2021 at 07:59:51PM +0200, Vlastimil Babka wrote: > > > > > These days the pcplist protection is done by local_lock, which solved > > > the RT concerns. Probably a stupid/infeasible idea, but maybe what you > > > want to achieve could be more generally solved at the local_lock level? > > > That on NOHZ_FULL CPUs, local_locks could have this mode where they > > > could synchronize with remote cpus? > > > > local_lock and spinlock have different rules, local_lock for example can > > never cause an irq inversion, unlike a spinlock. > > TBH, I really regret an attempt I made at some point in the RT > development to abuse local locks for this kind of cross CPU protections > because that led to yet another semantically ill defined construct. > > local locks are as the name says strictly local. IOW, they do not exist > for remote access. Just look at the !RT mapping: > > local_lock() -> preempt_disable() > local_lock_irq() -> local_irq_disable() > ... > > The only thing local_lock is addressing is the opaque nature of > preempt_disable(), local*_disable/save() protected critical sections, > which have per CPU BKL, i.e. undefined protection scope, semantics. > > If you really want cross CPU protection then using a regular spinlock in > a per CPU data structure is the right way to go. > > That makes it a bit akward vs. the code at hand which already introduced > local locks to replace the opaque preempt/local_irq disabled critical > sections with scoped local locks which in turn allows RT to substitute > them with strict CPU local spinlocks. > > But for clarity sake we really have to look at two different cases now: > > 1) Strict per CPU local protection > > That's what the code does today via local lock and this includes > RT where the locality is preserved via the local lock semantics > > I.e. for the !RT case the spinlock overhead is avoided > > 2) Global scoped per CPU protection > > That's what Nicolas is trying to achieve to be able to update > data structures cross CPU for the sake of avoiding smp function > calls or queuing work on remote CPUs for the NOHZ_FULL isolation > use case. > > That said, I completely understand Andrew's concerns versus these > distinctions and their test coverage. > > In consequence the real interesting question here is whether any of > these use cases are sensible to the extra overhead of #2. > > IOW, does it really matter whether the !RT and !NOHZ_FULL case take an > uncontended per CPU spinlock unconditionally instead of just disabling > preemption / interrupts? > > The same question arises vs. the remote work queuing. Does it really > matter? I think that a proper investigation is due to figure out whether > delegated work is really superiour vs. doing the same work locked from a > remote CPU occasionally. > > If you really think about it then the task which is initiating the work > is already in a slow path. So what's the tradeoff to make this path a > little bit slower due to remote access vs. scheduling work and thereby > disturbing the remote CPU which has a performance impact as well and in > the NOHZ_FULL case even a correctness impact? That's especially > questionable for situations where the initiator has to wait for > completion of the remote work. > > The changelogs and the cover letter have a distinct void vs. that which > means this is just another example of 'scratch my itch' changes w/o > proper justification. Point taken, I'll get to it. -- Nicolás Sáenz From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3632CC433EF for ; Mon, 27 Sep 2021 09:30:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E6B25611BD for ; Mon, 27 Sep 2021 09:30:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E6B25611BD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 8732A94000A; Mon, 27 Sep 2021 05:30:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8232D6B0075; Mon, 27 Sep 2021 05:30:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EC5E94000A; Mon, 27 Sep 2021 05:30:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5FC506B0074 for ; Mon, 27 Sep 2021 05:30:58 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 25E3E1818FCCE for ; Mon, 27 Sep 2021 09:30:58 +0000 (UTC) X-FDA: 78632834196.12.B11EEF5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id CF1E510328E3 for ; Mon, 27 Sep 2021 09:30:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632735057; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K7NSp/vWgh0IQXSwoMraW7CkaYyKMLgroGH9C/clVag=; b=HQFctji/TTHQw/uQsBEHFwTDI1vgu2r3sVn4lxOQVYCjhCP+hBmL2pYXW6NtkpYYvXc3Ew TtaH6HoichbJZBCuYPRiMiz3amkeD64wWVXOZuT1fHifdgCRIHPjlOGaXESw43xuwMFaaQ ZKqvpOJBwVJeo1kqKE5dqnJoU6bGsXs= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-456-RIpK47vGOwmCicSlPz-WFA-1; Mon, 27 Sep 2021 05:30:56 -0400 X-MC-Unique: RIpK47vGOwmCicSlPz-WFA-1 Received: by mail-wr1-f70.google.com with SMTP id z2-20020a5d4c82000000b0015b140e0562so13822805wrs.7 for ; Mon, 27 Sep 2021 02:30:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=K7NSp/vWgh0IQXSwoMraW7CkaYyKMLgroGH9C/clVag=; b=2aRqomSaGURnPb4mSzpULyDKTiB5mHG5y+ax030qDSjkvy52Q98mYD00vzyFkxQOtJ Xoj4znFfdgi2fboFxF3/Cp0x0NDZbbFM/Gq/Y3rT2mqee3sJkd5/SsOYM9PXKO1nStWc 0yxEw5ONuUo15fQ2rQWQIKFueEqRFuepveSu9lniaFMbO16bRewTqG7/QC7LbtN/+qWv kOedilaG8WdOlmdxYB98KD2hGM702Cxj7kAC9N2NWkuu/hcIka8RmijS66pHdI63VQuK zAxHjy85ru2SkYQ2CWvFmcpJk2b1JxkEI9bbo2ESeNDFltluNC6rzzn4xNCxsN2zejOU AGDA== X-Gm-Message-State: AOAM530UIxl+L3yjT1pqPw6oGAKcf8F8YRb4nhH6qP/mq8/r1ZtUs6YZ Wec9QjkpkRLzXOqUpDljKFbzG4boTQ3FTcrvi7uR02PNfmygu7qexmBaVvyP97iNvrT5c1lwb3V NDlKcXSoIgsY= X-Received: by 2002:a5d:58ec:: with SMTP id f12mr26326685wrd.24.1632735054900; Mon, 27 Sep 2021 02:30:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx6b8PsqrofWNdXOOdYGB7TMIOeoLXw6T/j3GuUxYOkzE46cemEL5lj+Ydtl2K8rifSKmavPg== X-Received: by 2002:a5d:58ec:: with SMTP id f12mr26326655wrd.24.1632735054634; Mon, 27 Sep 2021 02:30:54 -0700 (PDT) Received: from ?IPv6:2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123? ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id n186sm19979340wme.31.2021.09.27.02.30.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Sep 2021 02:30:54 -0700 (PDT) Message-ID: <8ee79b78c7f7f8fa18ec237494f09492ef60081f.camel@redhat.com> Subject: Re: [PATCH 0/6] mm: Remote LRU per-cpu pagevec cache/per-cpu page list drain support From: nsaenzju@redhat.com To: Thomas Gleixner , Peter Zijlstra , Vlastimil Babka Cc: akpm@linux-foundation.org, frederic@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, cl@linux.com, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, bigeasy@linutronix.de, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org Date: Mon, 27 Sep 2021 11:30:53 +0200 In-Reply-To: <87v92sgt3n.ffs@tglx> References: <20210921161323.607817-1-nsaenzju@redhat.com> <20210922112817.GO4323@worktop.programming.kicks-ass.net> <87v92sgt3n.ffs@tglx> User-Agent: Evolution 3.40.4 (3.40.4-1.fc34) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CF1E510328E3 X-Stat-Signature: feh1s9gbkn1tgyapoahgdabj5xzstj9w Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="HQFctji/"; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf13.hostedemail.com: domain of nsaenzju@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=nsaenzju@redhat.com X-HE-Tag: 1632735057-183698 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Thomas, thanks for the in-depth review. On Thu, 2021-09-23 at 00:09 +0200, Thomas Gleixner wrote: > On Wed, Sep 22 2021 at 13:28, Peter Zijlstra wrote: > > On Tue, Sep 21, 2021 at 07:59:51PM +0200, Vlastimil Babka wrote: > >=20 > > > These days the pcplist protection is done by local_lock, which solv= ed > > > the RT concerns. Probably a stupid/infeasible idea, but maybe what = you > > > want to achieve could be more generally solved at the local_lock le= vel? > > > That on NOHZ_FULL CPUs, local_locks could have this mode where they > > > could synchronize with remote cpus? > >=20 > > local_lock and spinlock have different rules, local_lock for example = can > > never cause an irq inversion, unlike a spinlock. >=20 > TBH, I really regret an attempt I made at some point in the RT > development to abuse local locks for this kind of cross CPU protections > because that led to yet another semantically ill defined construct. >=20 > local locks are as the name says strictly local. IOW, they do not exist > for remote access. Just look at the !RT mapping: >=20 > local_lock() -> preempt_disable() > local_lock_irq() -> local_irq_disable() > ... >=20 > The only thing local_lock is addressing is the opaque nature of > preempt_disable(), local*_disable/save() protected critical sections, > which have per CPU BKL, i.e. undefined protection scope, semantics. >=20 > If you really want cross CPU protection then using a regular spinlock i= n > a per CPU data structure is the right way to go. >=20 > That makes it a bit akward vs. the code at hand which already introduce= d > local locks to replace the opaque preempt/local_irq disabled critical > sections with scoped local locks which in turn allows RT to substitute > them with strict CPU local spinlocks. >=20 > But for clarity sake we really have to look at two different cases now: >=20 > 1) Strict per CPU local protection >=20 > That's what the code does today via local lock and this includes > RT where the locality is preserved via the local lock semantics >=20 > I.e. for the !RT case the spinlock overhead is avoided=20 >=20 > 2) Global scoped per CPU protection >=20 > That's what Nicolas is trying to achieve to be able to update > data structures cross CPU for the sake of avoiding smp function > calls or queuing work on remote CPUs for the NOHZ_FULL isolation > use case. >=20 > That said, I completely understand Andrew's concerns versus these > distinctions and their test coverage. >=20 > In consequence the real interesting question here is whether any of > these use cases are sensible to the extra overhead of #2. >=20 > IOW, does it really matter whether the !RT and !NOHZ_FULL case take an > uncontended per CPU spinlock unconditionally instead of just disabling > preemption / interrupts? >=20 > The same question arises vs. the remote work queuing. Does it really > matter? I think that a proper investigation is due to figure out whethe= r > delegated work is really superiour vs. doing the same work locked from = a > remote CPU occasionally. >=20 > If you really think about it then the task which is initiating the work > is already in a slow path. So what's the tradeoff to make this path a > little bit slower due to remote access vs. scheduling work and thereby > disturbing the remote CPU which has a performance impact as well and in > the NOHZ_FULL case even a correctness impact? That's especially > questionable for situations where the initiator has to wait for > completion of the remote work. >=20 > The changelogs and the cover letter have a distinct void vs. that which > means this is just another example of 'scratch my itch' changes w/o > proper justification. Point taken, I'll get to it. --=20 Nicol=C3=A1s S=C3=A1enz