From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67178C2D0ED for ; Mon, 30 Mar 2020 02:33:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2099B20735 for ; Mon, 30 Mar 2020 02:33:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="DeRVgPSq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2099B20735 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E4A2E6B0075; Sun, 29 Mar 2020 22:33:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D84A76B0078; Sun, 29 Mar 2020 22:33:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B88436B007B; Sun, 29 Mar 2020 22:33:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0053.hostedemail.com [216.40.44.53]) by kanga.kvack.org (Postfix) with ESMTP id A08866B0075 for ; Sun, 29 Mar 2020 22:33:43 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B0FA340D6 for ; Mon, 30 Mar 2020 02:33:43 +0000 (UTC) X-FDA: 76650457926.18.scale35_5aba6cedc2059 X-HE-Tag: scale35_5aba6cedc2059 X-Filterd-Recvd-Size: 5926 Received: from mail-qk1-f195.google.com (mail-qk1-f195.google.com [209.85.222.195]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:43 +0000 (UTC) Received: by mail-qk1-f195.google.com with SMTP id o10so17481387qki.10 for ; Sun, 29 Mar 2020 19:33:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TgVjjqxhrtlCFDIsHkuC1uTJ/LkUZOMYLnLhSTfKyFo=; b=DeRVgPSq3h4ir1leXjxInBG2h8bq9qGUaf0q/6P5vVer49r4RyYdVmEqgmRvSGQgiR Szt5lyIL+P4fZAYle1WQ6mbTytS2I6VlQ9plfFUPqx/C6Z/OXRGykLv5/yyq2ciXUrK3 uQEVTGkaN4TtsK0qXWZ7uhAXhcNSwwyMN8s6o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TgVjjqxhrtlCFDIsHkuC1uTJ/LkUZOMYLnLhSTfKyFo=; b=JW6E8mDV1rWkaug11/C66k7iTZQWRW9yYsZvKugASfcyAzH3wuftKz0OK4hT8rBZpI wmCiXm57pU8we1YJWUgXvIpVXXzQYERM1TDpYDC4IgKiHIVJYdm2+EePEcHZPobt+LZY /8pjvXaa4c24omq67Lnsjln6j6N0sIBnjyAQfCyFLFOimJrfTbfPJYLV8dxDqkZ1JtLn VHnnoeVeQuIxgTSZTeTxqLCoor+yK1fGIbKUr9DrxfSCzMFNjf2jCzRDAeU741ZdJTAw 0WDOZOHXX0xzp/HYn52I/3S2/r2cHZG+BAQ6dgCQZXIpZGmw+TmBATgMawo5LmRe11bT 8dPA== X-Gm-Message-State: ANhLgQ3m4wWMbXt/eVgQpi/La3OuBzimLDfVVV8EVCadDq+plJhuPRGJ +nOuSMC8m2dNyeI4MRWFqVdXZ88c1Zg= X-Google-Smtp-Source: ADFU+vunMDIJ4H3Pq+Krsea4u9kPIh06c+HxsJd5C4qvfZCMXYJheHm4aD9CNDZFXtrDOkjDRGKzpg== X-Received: by 2002:a05:620a:1189:: with SMTP id b9mr7758077qkk.236.1585535622481; Sun, 29 Mar 2020 19:33:42 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:41 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 11/18] rcu/tree: Introduce expedited_drain flag Date: Sun, 29 Mar 2020 22:32:41 -0400 Message-Id: <20200330023248.164994-12-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" It is used and set to true when the bulk array can not be maintained, it happens under low memory condition and memory pressure. In that case the drain work is scheduled right away and not after KFREE_DRAIN_JIFFIES. It tends to speed up the reclaim path. On the other hand, there is no data showing the difference yet. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8fbc8450284db..3b94526f490cb 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3128,14 +3128,16 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_= cpu *krcp, void *ptr) * due to memory pressure. * * Each kvfree_call_rcu() request is added to a batch. The batch will be= drained - * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the b= atch will - * be free'd in workqueue context. This allows us to: batch requests tog= ether to - * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rc= u() load. + * every KFREE_DRAIN_JIFFIES number of jiffies or can be scheduled right= away if + * a low memory is detected. All the objects in the batch will be free'd= in + * workqueue context. This allows us to: batch requests together to redu= ce the + * number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. */ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; + bool expedited_drain =3D false; void *ptr; =20 local_irq_save(flags); // For safely calling this_cpu_ptr(). @@ -3161,6 +3163,14 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_ca= llback_t func) head->func =3D func; head->next =3D krcp->head; krcp->head =3D head; + + /* + * There was an issue to place the pointer directly + * into array, due to memory pressure. Initiate an + * expedited drain to accelerate lazy invocation of + * appropriate free calls. + */ + expedited_drain =3D true; } =20 WRITE_ONCE(krcp->count, krcp->count + 1); @@ -3169,7 +3179,9 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_cal= lback_t func) if (rcu_scheduler_active =3D=3D RCU_SCHEDULER_RUNNING && !krcp->monitor_todo) { krcp->monitor_todo =3D true; - schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES); + + schedule_delayed_work(&krcp->monitor_work, + expedited_drain ? 0 : KFREE_DRAIN_JIFFIES); } =20 unlock_return: --=20 2.26.0.rc2.310.g2932bb562d-goog