[PATCH v3] IB/mlx5: Reduce max order of memory allocated for xlt update

[PATCH v3] IB/mlx5: Reduce max order of memory allocated for xlt update

[Praveen Kumar Kannoju]

To update xlt (during mlx5_ib_reg_user_mr()), the driver can request up to
1 MB (order-8) memory, depending on the size of the MR. This costly
allocation can sometimes take very long to return (a few seconds). This
causes the calling application to hang for a long time, especially when the
system is fragmented.  To avoid these long latency spikes, the calls the
higher order allocations need to fail faster in case they are not
available. In order to acheive this we need __GFP_NORETRY flag in the
gfp_mask before during fetching the free pages. This patch adds this flag
to the mask.

Signed-off-by: Praveen Kumar Kannoju <praveen.kannoju@oracle.com>
---
 drivers/infiniband/hw/mlx5/mr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index db05b0e..429e7aa6 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1028,7 +1028,7 @@ static void *mlx5_ib_alloc_xlt(size_t *nents, size_t ent_size, gfp_t gfp_mask)
 	 */
 	might_sleep();
 
-	gfp_mask |= __GFP_ZERO;
+	gfp_mask |= __GFP_ZERO | __GFP_NORETRY;
 
 	/*
 	 * If the system already has a suitable high order page then just use
-- 
1.8.3.1

Re: [PATCH v3] IB/mlx5: Reduce max order of memory allocated for xlt update

[Leon Romanovsky]

On Sat, Apr 03, 2021 at 04:53:55AM +0000, Praveen Kumar Kannoju wrote:
> To update xlt (during mlx5_ib_reg_user_mr()), the driver can request up to
> 1 MB (order-8) memory, depending on the size of the MR. This costly
> allocation can sometimes take very long to return (a few seconds). This
> causes the calling application to hang for a long time, especially when the
> system is fragmented.  To avoid these long latency spikes, the calls the
> higher order allocations need to fail faster in case they are not
> available. In order to acheive this we need __GFP_NORETRY flag in the
> gfp_mask before during fetching the free pages. This patch adds this flag
> to the mask.
> 
> Signed-off-by: Praveen Kumar Kannoju <praveen.kannoju@oracle.com>
> ---
>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 

Thanks a lot,
Acked-by: Leon Romanovsky <leonro@nvidia.com>

Re: [PATCH v3] IB/mlx5: Reduce max order of memory allocated for xlt update

[Jason Gunthorpe]

On Sat, Apr 03, 2021 at 04:53:55AM +0000, Praveen Kumar Kannoju wrote:
> To update xlt (during mlx5_ib_reg_user_mr()), the driver can request up to
> 1 MB (order-8) memory, depending on the size of the MR. This costly
> allocation can sometimes take very long to return (a few seconds). This
> causes the calling application to hang for a long time, especially when the
> system is fragmented.  To avoid these long latency spikes, the calls the
> higher order allocations need to fail faster in case they are not
> available. In order to acheive this we need __GFP_NORETRY flag in the
> gfp_mask before during fetching the free pages. This patch adds this flag
> to the mask.
> 
> Signed-off-by: Praveen Kumar Kannoju <praveen.kannoju@oracle.com>
> Acked-by: Leon Romanovsky <leonro@nvidia.com>
> ---
>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Applied to for-next, thanks

Jason