From: Santosh Shilimkar <santosh.shilimkar@oracle.com>
To: Praveen Kannoju <praveen.kannoju@oracle.com>
Cc: "David S . Miller" <davem@davemloft.net>,
"kuba@kernel.org" <kuba@kernel.org>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
"rds-devel@oss.oracle.com" <rds-devel@oss.oracle.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Rama Nichanamatlu <rama.nichanamatlu@oracle.com>,
Rajesh Sivaramasubramaniom
<rajesh.sivaramasubramaniom@oracle.com>
Subject: Re: [PATCH RFC] rds: ib: Reduce the contention caused by the asynchronous workers to flush the mr pool
Date: Tue, 18 Jan 2022 16:48:43 +0000 [thread overview]
Message-ID: <53D98F26-FC52-4F3E-9700-ED0312756785@oracle.com> (raw)
In-Reply-To: <1642517238-9912-1-git-send-email-praveen.kannoju@oracle.com>
> On Jan 18, 2022, at 6:47 AM, Praveen Kannoju <praveen.kannoju@oracle.com> wrote:
>
> This patch aims to reduce the number of asynchronous workers being spawned
> to execute the function "rds_ib_flush_mr_pool" during the high I/O
> situations. Synchronous call path's to this function "rds_ib_flush_mr_pool"
> will be executed without being disturbed. By reducing the number of
> processes contending to flush the mr pool, the total number of D state
> processes waiting to acquire the mutex lock will be greatly reduced, which
> otherwise were causing DB instance crash as the corresponding processes
> were not progressing while waiting to acquire the mutex lock.
>
> Signed-off-by: Praveen Kumar Kannoju <praveen.kannoju@oracle.com>
> —
>
[…]
> diff --git a/net/rds/ib_rdma.c b/net/rds/ib_rdma.c
> index 8f070ee..6b640b5 100644
> --- a/net/rds/ib_rdma.c
> +++ b/net/rds/ib_rdma.c
> @@ -393,6 +393,8 @@ int rds_ib_flush_mr_pool(struct rds_ib_mr_pool *pool,
> */
> dirty_to_clean = llist_append_to_list(&pool->drop_list, &unmap_list);
> dirty_to_clean += llist_append_to_list(&pool->free_list, &unmap_list);
> + WRITE_ONCE(pool->flush_ongoing, true);
> + smp_wmb();
> if (free_all) {
> unsigned long flags;
>
> @@ -430,6 +432,8 @@ int rds_ib_flush_mr_pool(struct rds_ib_mr_pool *pool,
> atomic_sub(nfreed, &pool->item_count);
>
> out:
> + WRITE_ONCE(pool->flush_ongoing, false);
> + smp_wmb();
> mutex_unlock(&pool->flush_lock);
> if (waitqueue_active(&pool->flush_wait))
> wake_up(&pool->flush_wait);
> @@ -507,8 +511,17 @@ void rds_ib_free_mr(void *trans_private, int invalidate)
>
> /* If we've pinned too many pages, request a flush */
> if (atomic_read(&pool->free_pinned) >= pool->max_free_pinned ||
> - atomic_read(&pool->dirty_count) >= pool->max_items / 5)
> - queue_delayed_work(rds_ib_mr_wq, &pool->flush_worker, 10);
> + atomic_read(&pool->dirty_count) >= pool->max_items / 5) {
> + smp_rmb();
You won’t need these explicit barriers since above atomic and write once already
issue them.
Other than that, it seems a good idea to me. With that fixed, feel free to add my
ack and repost.
Regards,
Santosh
next prev parent reply other threads:[~2022-01-18 16:49 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-18 14:47 [PATCH RFC] rds: ib: Reduce the contention caused by the asynchronous workers to flush the mr pool Praveen Kumar Kannoju
2022-01-18 16:48 ` Santosh Shilimkar [this message]
2022-01-18 18:00 ` Leon Romanovsky
2022-01-18 19:17 ` Jason Gunthorpe
2022-01-18 19:42 ` Santosh Shilimkar
2022-01-19 6:59 ` Leon Romanovsky
2022-01-19 11:46 ` Praveen Kannoju
2022-01-19 13:04 ` Jason Gunthorpe
2022-01-19 13:12 ` Praveen Kannoju
2022-01-19 13:17 ` Jason Gunthorpe
2022-01-19 14:08 ` Praveen Kannoju
2022-01-19 14:49 ` Jason Gunthorpe
2022-01-19 14:56 ` Leon Romanovsky
2022-01-20 8:00 ` Praveen Kannoju
2022-01-20 11:11 ` Leon Romanovsky
2022-01-20 11:57 ` Praveen Kannoju
2022-01-20 12:21 ` Leon Romanovsky
2022-01-20 12:27 ` Praveen Kannoju
-- strict thread matches above, loose matches on Subject: below --
2022-01-18 13:10 Praveen Kumar Kannoju
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53D98F26-FC52-4F3E-9700-ED0312756785@oracle.com \
--to=santosh.shilimkar@oracle.com \
--cc=davem@davemloft.net \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=praveen.kannoju@oracle.com \
--cc=rajesh.sivaramasubramaniom@oracle.com \
--cc=rama.nichanamatlu@oracle.com \
--cc=rds-devel@oss.oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).