From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.3 required=3.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89035C83000 for ; Tue, 28 Apr 2020 21:00:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3BA5B2137B for ; Tue, 28 Apr 2020 21:00:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="d6UTV9hy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3BA5B2137B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0150A8E0018; Tue, 28 Apr 2020 16:59:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EDF6D8E0007; Tue, 28 Apr 2020 16:59:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA51F8E0018; Tue, 28 Apr 2020 16:59:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0166.hostedemail.com [216.40.44.166]) by kanga.kvack.org (Postfix) with ESMTP id BE5958E0007 for ; Tue, 28 Apr 2020 16:59:54 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7B5532493 for ; Tue, 28 Apr 2020 20:59:54 +0000 (UTC) X-FDA: 76758480708.13.rings59_5678998eaaf2e X-HE-Tag: rings59_5678998eaaf2e X-Filterd-Recvd-Size: 9958 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:54 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id a21so191502ljj.11 for ; Tue, 28 Apr 2020 13:59:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=66vSg9Njt1sZCB7M+dj1UlLLZ6BR/zJPh4B9KJb+CwY=; b=d6UTV9hymGMssr4fjOWqvH+tmH1+vbRR4+2M9oD8d4EnJPh0SHe7iRFDylZiDMGjHA JlfOILnK2hmlYxoV47p3bLx2RydaXS6tJMVboKMWS3J8iNzcIpksYLgp/hd+/iSk52Bc FZhx5LS9U3kAaoLzsoB7gF3Q0P1aqNetJLXUwXOBjG3pjd0reRTStlPB2nNHtf9WfeoT 2EHBK9+AmhcjSLEoej6QDrJTRpP2yLaiB4jWVvMEkjn88sYmZ4aIjGiBMeZZ7I/uEEYs 9rUx8KLuPJKJ3kSF3zVDbspnnrRzPN9vo8j5JP4VGPefcIr7ikAmm1mmSg//aGSaCJU5 UzNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=66vSg9Njt1sZCB7M+dj1UlLLZ6BR/zJPh4B9KJb+CwY=; b=uF6s5Ie/aylwJbAn963oI+/wnnKh1VoJJysKFO2z1z4BC3jDhpl+ccap+kpizTIrVy GAs5PRzy+3TAXx8yCqHp9UWOkwiraTcG/jvN4ut3ovSHF9+mKuTcbOCJ1zZFBJR7+ZtE i2sISoAJ8tXtAKRt9ojZImEZkDlggOpLtajVIHVWj4cEvyxlrlq3wsj4Q9CC+d8P4gbG LmfplUBh5GDUV67wVrduhcrTYs+Wxr0XooRXHZUPnMexOWuODPnFtA7MHBA6CSPPImVo 0fAH+z7hOn/Q1opR3UQXBsc/JgnMWS+fk/CWcmd350PHuQIoEYL5H8D8eRIXbqg2lmCM zZzg== X-Gm-Message-State: AGi0PuYDHd4tW6IIOTUg4r/TgVEAeHPn/1iA09Gs4HOfBg3OPg45fAPO WnheqzhqZ00wXDtHANQsyzo= X-Google-Smtp-Source: APiQypJwT3gU3ag2Y4efeBkFASGSObeuAOyEJgCNhNbI+6xXdAvu4anqfWejz6vVzioHPMcJxp3dYQ== X-Received: by 2002:a2e:9818:: with SMTP id a24mr18725795ljj.126.1588107592738; Tue, 28 Apr 2020 13:59:52 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:51 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 19/24] rcu/tree: Support reclaim for head-less object Date: Tue, 28 Apr 2020 22:58:58 +0200 Message-Id: <20200428205903.61704-20-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Update the kvfree_call_rcu() with head-less support, it means an object without any rcu_head structure can be reclaimed after GP. To store pointers there are two chain-arrays maintained one for SLAB and another one is for vmalloc. Both types of objects(head-less variant and regular one) are placed there based on the type. It can be that maintaining of arrays becomes impossible due to high memory pressure. For such reason there is an emergency path. In that case objects with rcu_head inside are just queued building one way list. Later on that list is drained. As for head-less variant. Such objects do not have any rcu_head helper inside. Thus it is dynamically attached. As a result an object consists of back-pointer and regular rcu_head. It implies that emergency path can detect such object type, therefore they are tagged. So a back-pointer could be freed as well as dynamically attached wrapper. Even though such approach requires dynamic memory it needs only sizeof(unsigned long *) + sizeof(struct rcu_head) bytes, thus SLAB is used to obtain it. Finally if attaching of the rcu_head and queuing get failed, the current context has to follow might_sleep() annotation, thus below steps could be applied: a) wait until a grace period has elapsed; b) direct inlining of the kvfree() call. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) Co-developed-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 102 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 98 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 51726e4c3b4d..501cac02146d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3072,15 +3072,31 @@ static void kfree_rcu_work(struct work_struct *wo= rk) */ for (; head; head =3D next) { unsigned long offset =3D (unsigned long)head->func; - void *ptr =3D (void *)head - offset; + bool headless; + void *ptr; =20 next =3D head->next; + + /* We tag the headless object, if so adjust offset. */ + headless =3D (((unsigned long) head - offset) & BIT(0)); + if (headless) + offset -=3D 1; + + ptr =3D (void *) head - offset; + debug_rcu_head_unqueue((struct rcu_head *)ptr); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset); =20 - if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) + if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) { + /* + * If headless free the back-pointer first. + */ + if (headless) + kvfree((void *) *((unsigned long *) ptr)); + kvfree(ptr); + } =20 rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); @@ -3221,6 +3237,13 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_c= pu *krcp, void *ptr) if (IS_ENABLED(CONFIG_PREEMPT_RT)) return false; =20 + /* + * TODO: For one argument of kvfree_rcu() we can + * drop the lock and get the page in sleepable + * context. That would allow to maintain an array + * for the CONFIG_PREEMPT_RT as well. Thus we could + * get rid of dynamic rcu_head attaching code. + */ bnode =3D (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); } @@ -3244,6 +3267,23 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_c= pu *krcp, void *ptr) return true; } =20 +static inline struct rcu_head * +attach_rcu_head_to_object(void *obj) +{ + unsigned long *ptr; + + ptr =3D kmalloc(sizeof(unsigned long *) + + sizeof(struct rcu_head), GFP_NOWAIT | + __GFP_RECLAIM | /* can do direct reclaim. */ + __GFP_NORETRY | /* only lightweight one. */ + __GFP_NOWARN); /* no failure reports. */ + if (!ptr) + return NULL; + + ptr[0] =3D (unsigned long) obj; + return ((struct rcu_head *) ++ptr); +} + /* * Queue a request for lazy invocation of appropriate free routine after= a * grace period. Please note there are three paths are maintained, two a= re the @@ -3260,16 +3300,34 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_c= allback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; + bool success; void *ptr; =20 + if (head) { + ptr =3D (void *) head - (unsigned long) func; + } else { + /* + * Please note there is a limitation for the head-less + * variant, that is why there is a clear rule for such + * objects: + * + * it can be used from might_sleep() context only. For + * other places please embed an rcu_head to your data. + */ + might_sleep(); + ptr =3D (unsigned long *) func; + } + krcp =3D krc_this_cpu_lock(&flags); - ptr =3D (void *)head - (unsigned long)func; =20 /* Queue the object but don't yet schedule the batch. */ if (debug_rcu_head_queue(ptr)) { /* Probable double kfree_rcu(), just leak. */ WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", __func__, head); + + /* Mark as success and leave. */ + success =3D true; goto unlock_return; } =20 @@ -3277,10 +3335,34 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_c= allback_t func) * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (unlikely(!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr))) { + success =3D kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr); + if (!success) { + if (head =3D=3D NULL) { + /* + * Headless(one argument kvfree_rcu()) can sleep. + * Drop the lock and tack it back. So it can do + * direct lightweight reclaim. + */ + krc_this_cpu_unlock(krcp, flags); + head =3D attach_rcu_head_to_object(ptr); + krcp =3D krc_this_cpu_lock(&flags); + + if (head =3D=3D NULL) + goto unlock_return; + + /* + * Tag the headless object. Such objects have a + * back-pointer to the original allocated memory, + * that has to be freed as well as dynamically + * attached wrapper/head. + */ + func =3D (rcu_callback_t) (sizeof(unsigned long *) + 1); + } + head->func =3D func; head->next =3D krcp->head; krcp->head =3D head; + success =3D true; } =20 WRITE_ONCE(krcp->count, krcp->count + 1); @@ -3294,6 +3376,18 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_ca= llback_t func) =20 unlock_return: krc_this_cpu_unlock(krcp, flags); + + /* + * High memory pressure, so inline kvfree() after + * synchronize_rcu(). We can do it from might_sleep() + * context only, so the current CPU can pass the QS + * state. + */ + if (!success) { + debug_rcu_head_unqueue(ptr); + synchronize_rcu(); + kvfree(ptr); + } } EXPORT_SYMBOL_GPL(kvfree_call_rcu); =20 --=20 2.20.1