From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C740C433FE for ; Wed, 22 Sep 2021 09:50:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 804E761100 for ; Wed, 22 Sep 2021 09:50:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234536AbhIVJvm (ORCPT ); Wed, 22 Sep 2021 05:51:42 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:50302 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234381AbhIVJvl (ORCPT ); Wed, 22 Sep 2021 05:51:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632304211; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HWU7csjt4Np/ai++4aBlLIWralnkqY/1oQmbIHwv+NI=; b=d5dMTWoT1IsIoBa24qiJLrxRzYkmxqb7a64hxf2Orokpg0+hoOeDF7bDTVUoZC1S15ta36 +HbKalklo/7rrMD4gmyyo0ri8PQJlAgwaNOevmN/RHRueJ1RK58m3M8dk+fwtsChnVJW+r L5WTax272SeW6QvavDFDErpm+83AX8k= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-33-fYT69TeDNiy5TWbbBeyZxA-1; Wed, 22 Sep 2021 05:50:10 -0400 X-MC-Unique: fYT69TeDNiy5TWbbBeyZxA-1 Received: by mail-wr1-f71.google.com with SMTP id r9-20020a5d4989000000b0015d0fbb8823so1600644wrq.18 for ; Wed, 22 Sep 2021 02:50:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=HWU7csjt4Np/ai++4aBlLIWralnkqY/1oQmbIHwv+NI=; b=4vfZi5WmzqqJNH7sEPTtrpJodb0HCVQZc6xOU1rjFgOHIz78hfibfZXYrqcIe/CaRR qqvxgAsQt5zwWTPAsbg8G8B+iCd2JoREGY5YlIhGTUZGxjkdNV7OPjRGKjXbcbOND8ZM fMEcs1jtcj0QrCW1E0t7ghrBbjVtTmn0F4tLEYjh4WhUaQMX4M0JKcMnIEukXGIdjjX0 w9+pa8sgXpERfmC4kxy5yxruvZ2pSHgtk9n15tuAmuE9f/goOZ946RHrXGALcNC9sPIR IU45Xl+wFx/+s8vAIbqocPvVdwelTYaFfkvXpnG5hSK7Kys1Ivqs4Q3TFtREspWwXig8 QkiQ== X-Gm-Message-State: AOAM533UvbDQKHO7cVlYXaYYQ3BFqAAn0PSeChcMIM7OUH+K9QNG+9Kn 80XXaIzWw8abEQTxvXL1b1Mkq4bHcjUpcWI6q8XYqAmeN4v9BQS2BEklTOUBu4T/CXkMOi98oxh 5feiRWAE6tmtd8rO1OCc57UVb X-Received: by 2002:a5d:47a3:: with SMTP id 3mr38586415wrb.334.1632304208910; Wed, 22 Sep 2021 02:50:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxSdlRuz+OyjwzCgzVR2s/BhfvS6G/FdxXQ7YW/wSm+ya8/shGeXeBe0T4XHcZG3G0y6rx1lQ== X-Received: by 2002:a5d:47a3:: with SMTP id 3mr38586391wrb.334.1632304208701; Wed, 22 Sep 2021 02:50:08 -0700 (PDT) Received: from ?IPv6:2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123? ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id f8sm1651573wrx.15.2021.09.22.02.50.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Sep 2021 02:50:08 -0700 (PDT) Message-ID: <7c8594d5943f99bc647951269606ae24dd1b37de.camel@redhat.com> Subject: Re: [PATCH 2/6] mm/swap: Introduce alternative per-cpu LRU cache locking From: nsaenzju@redhat.com To: Sebastian Andrzej Siewior Cc: Peter Zijlstra , akpm@linux-foundation.org, frederic@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, cl@linux.com, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org Date: Wed, 22 Sep 2021 11:50:07 +0200 In-Reply-To: <20210922092039.2j6efnkhmfxuzjnx@linutronix.de> References: <20210921161323.607817-1-nsaenzju@redhat.com> <20210921161323.607817-3-nsaenzju@redhat.com> <20210921220358.GN4323@worktop.programming.kicks-ass.net> <0ec733daf2daaf8a6f5b1fbf56676b9892d5bf73.camel@redhat.com> <20210922092039.2j6efnkhmfxuzjnx@linutronix.de> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.40.4 (3.40.4-1.fc34) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 2021-09-22 at 11:20 +0200, Sebastian Andrzej Siewior wrote: > On 2021-09-22 10:47:07 [+0200], nsaenzju@redhat.com wrote: > > > *why* use migrate_disable(), that's horrible! > > > > I was trying to be mindful of RT. They don't appreciate people taking spinlocks > > just after having disabled preemption. > > > > I think getting local_lock(&locks->local) is my only option then. But it adds > > an extra redundant spinlock in the RT+NOHZ_FULL case. > > spin_lock() does not disable preemption on PREEMPT_RT. You don't > disables preemption on purpose or did I miss that? Sorry my message wasn't clear. Adding code for context: + static inline void lru_cache_lock(struct lru_cache_locks *locks) + { + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + /* Avoid migration between this_cpu_ptr() and spin_lock() */ + migrate_disable(); IIUC PeterZ would've preferred that I disable preemption here. And what I meant to explain is that, given the spinlock below, I choose migrate_disable() over preempt_disable() to cater for RT. + spin_lock(this_cpu_ptr(&locks->spin)); + } else { So, to make both worlds happy, I think the only option left is using the local_lock (at the expense of extra overhead in the RT+NOHZ_FULL case): + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + /* Local lock avoids migration between this_cpu_ptr() and spin_lock() */ + local_lock(&locks->local); + spin_lock(this_cpu_ptr(&locks->spin)); + } else { -- Nicolás Sáenz From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C723C433F5 for ; Wed, 22 Sep 2021 09:50:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D7A85610A1 for ; Wed, 22 Sep 2021 09:50:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D7A85610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5AC846B006C; Wed, 22 Sep 2021 05:50:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 55C946B0072; Wed, 22 Sep 2021 05:50:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42438900002; Wed, 22 Sep 2021 05:50:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0098.hostedemail.com [216.40.44.98]) by kanga.kvack.org (Postfix) with ESMTP id 2FF316B006C for ; Wed, 22 Sep 2021 05:50:13 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D49A72DD88 for ; Wed, 22 Sep 2021 09:50:12 +0000 (UTC) X-FDA: 78614738664.11.7F8C6AB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 6C8C770000A0 for ; Wed, 22 Sep 2021 09:50:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632304211; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HWU7csjt4Np/ai++4aBlLIWralnkqY/1oQmbIHwv+NI=; b=d5dMTWoT1IsIoBa24qiJLrxRzYkmxqb7a64hxf2Orokpg0+hoOeDF7bDTVUoZC1S15ta36 +HbKalklo/7rrMD4gmyyo0ri8PQJlAgwaNOevmN/RHRueJ1RK58m3M8dk+fwtsChnVJW+r L5WTax272SeW6QvavDFDErpm+83AX8k= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-115-dgYr6PHzOX63ITmYnLrmpw-1; Wed, 22 Sep 2021 05:50:10 -0400 X-MC-Unique: dgYr6PHzOX63ITmYnLrmpw-1 Received: by mail-wr1-f72.google.com with SMTP id x7-20020a5d6507000000b0015dada209b1so1606549wru.15 for ; Wed, 22 Sep 2021 02:50:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=HWU7csjt4Np/ai++4aBlLIWralnkqY/1oQmbIHwv+NI=; b=5sUiFJvYKq1tSAdZ9QsVaeedqLhZMa5sVDq2lUyqA9VMJbFXP8ECQ/x2jkVLsQFaN0 m8n1GBdT551HTN0hUxkPM7kWAVIafN4ghdiv+vXlV0ta6DWq8SKlFGa/oRpRd97Pl5Vo yrgVKGtD/XmxHnqiG0xlWF6ipIEZ2cK3qXgAWY4EELtyufbxQ2hcNpBmTeAiiuWdf4xf f3NjX4tdtdS5PbfqF60l2k5L/KQj9qH5LQ++YDZR8y8BQelCDFOfumwQLuEM711YR4Ps Vks8gDNaGiIHCg5bq8bUpdf1grJ9J/P34LtOgSTCVcd5QNguKO102dbpLzf26+1nSCz0 FrYw== X-Gm-Message-State: AOAM531kS2oYlsNX6MM2GegKIjEEcbTZ2j3jSxR6opVU3Pii6BekCl14 mNlNUp2xX41Y/BfhHXPLATHWfTTjjpIZAzOQHMIietLwQfSzxJsUQ3QSopug+Vn8Z1rtv4t68ag bflBhPEUFrsE= X-Received: by 2002:a5d:47a3:: with SMTP id 3mr38586430wrb.334.1632304208928; Wed, 22 Sep 2021 02:50:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxSdlRuz+OyjwzCgzVR2s/BhfvS6G/FdxXQ7YW/wSm+ya8/shGeXeBe0T4XHcZG3G0y6rx1lQ== X-Received: by 2002:a5d:47a3:: with SMTP id 3mr38586391wrb.334.1632304208701; Wed, 22 Sep 2021 02:50:08 -0700 (PDT) Received: from ?IPv6:2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123? ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id f8sm1651573wrx.15.2021.09.22.02.50.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Sep 2021 02:50:08 -0700 (PDT) Message-ID: <7c8594d5943f99bc647951269606ae24dd1b37de.camel@redhat.com> Subject: Re: [PATCH 2/6] mm/swap: Introduce alternative per-cpu LRU cache locking From: nsaenzju@redhat.com To: Sebastian Andrzej Siewior Cc: Peter Zijlstra , akpm@linux-foundation.org, frederic@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, cl@linux.com, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org Date: Wed, 22 Sep 2021 11:50:07 +0200 In-Reply-To: <20210922092039.2j6efnkhmfxuzjnx@linutronix.de> References: <20210921161323.607817-1-nsaenzju@redhat.com> <20210921161323.607817-3-nsaenzju@redhat.com> <20210921220358.GN4323@worktop.programming.kicks-ass.net> <0ec733daf2daaf8a6f5b1fbf56676b9892d5bf73.camel@redhat.com> <20210922092039.2j6efnkhmfxuzjnx@linutronix.de> User-Agent: Evolution 3.40.4 (3.40.4-1.fc34) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d5dMTWoT; spf=none (imf27.hostedemail.com: domain of nsaenzju@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=nsaenzju@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6C8C770000A0 X-Stat-Signature: pps7ccj585yk6bgah1ezhbmmm5c85tdt X-HE-Tag: 1632304212-922177 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.173968, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 2021-09-22 at 11:20 +0200, Sebastian Andrzej Siewior wrote: > On 2021-09-22 10:47:07 [+0200], nsaenzju@redhat.com wrote: > > > *why* use migrate_disable(), that's horrible! > >=20 > > I was trying to be mindful of RT. They don't appreciate people taking= spinlocks > > just after having disabled preemption. > >=20 > > I think getting local_lock(&locks->local) is my only option then. But= it adds > > an extra redundant spinlock in the RT+NOHZ_FULL case. >=20 > spin_lock() does not disable preemption on PREEMPT_RT. You don't > disables preemption on purpose or did I miss that? Sorry my message wasn't clear. Adding code for context: + static inline void lru_cache_lock(struct lru_cache_locks *locks) + { + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + /* Avoid migration between this_cpu_ptr() and spin_lock() */ + migrate_disable(); IIUC PeterZ would've preferred that I disable preemption here. And what I= meant to explain is that, given the spinlock below, I choose migrate_disable() = over preempt_disable() to cater for RT. + spin_lock(this_cpu_ptr(&locks->spin)); + } else { So, to make both worlds happy, I think the only option left is using the local_lock (at the expense of extra overhead in the RT+NOHZ_FULL case): + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + /* Local lock avoids migration between this_cpu_ptr() and spin_lock()= */ + local_lock(&locks->local); + spin_lock(this_cpu_ptr(&locks->spin)); + } else { --=20 Nicol=C3=A1s S=C3=A1enz