From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.0 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C17A9C433E0 for ; Tue, 4 Aug 2020 10:58:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B048820738 for ; Tue, 4 Aug 2020 10:58:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729549AbgHDK6q (ORCPT ); Tue, 4 Aug 2020 06:58:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729548AbgHDK6q (ORCPT ); Tue, 4 Aug 2020 06:58:46 -0400 X-Greylist: delayed 362 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Tue, 04 Aug 2020 03:58:46 PDT Received: from btbn.de (btbn.de [IPv6:2a01:4f8:162:63a9::2]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 976BFC06174A for ; Tue, 4 Aug 2020 03:58:46 -0700 (PDT) Received: from [IPv6:2001:16b8:64d7:4500:fc3b:cfd2:151e:7636] (200116b864d74500fc3bcfd2151e7636.dip.versatel-1u1.de [IPv6:2001:16b8:64d7:4500:fc3b:cfd2:151e:7636]) by btbn.de (Postfix) with ESMTPSA id B558D4BD30; Tue, 4 Aug 2020 12:52:30 +0200 (CEST) Subject: Re: NFS over RDMA issues on Linux 5.4 To: Leon Romanovsky , Chuck Lever Cc: Linux NFS Mailing List , linux-rdma References: <8a1087d3-9add-dfe1-da0c-edab74fcca51@rothenpieler.org> <20200804093635.GA4432@unreal> From: Timo Rothenpieler Message-ID: <92a5a932-b843-eed3-555e-7557ccc1f308@rothenpieler.org> Date: Tue, 4 Aug 2020 12:52:27 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <20200804093635.GA4432@unreal> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On 04.08.2020 11:36, Leon Romanovsky wrote: > On Mon, Aug 03, 2020 at 12:24:21PM -0400, Chuck Lever wrote: >> Hi Timo- >> >>> On Aug 3, 2020, at 11:05 AM, Timo Rothenpieler wrote: >>> >>> Hello, >>> >>> I have just deployed a new system with Mellanox ConnectX-4 VPI EDR IB cards and wanted to setup NFS over RDMA on it. >>> >>> However, while mounting the FS over RDMA works fine, actually using it results in the following messages absolutely hammering dmesg on both client and server: >>> >>>> https://gist.github.com/BtbN/9582e597b6581f552fa15982b0285b80#file-server-log >>> >>> The spam only stops once I forcibly reboot the client. The filesystem gets nowhere during all this. The retrans counter in nfsstat just keeps going up, nothing actually gets done. >>> >>> This is on Linux 5.4.54, using nfs-utils 2.4.3. >>> The mlx5 driver had enhanced-mode disabled in order to enable IPoIB connected mode with an MTU of 65520. >>> >>> Normal NFS 4.2 over tcp works perfectly fine on this setup, it's only when I mount via rdma that things go wrong. >>> >>> Is this an issue on my end, or did I run into a bug somewhere here? >>> Any pointers, patches and solutions to test are welcome. >> >> I haven't seen that failure mode here, so best I can recommend is >> keep investigating. I've copied linux-rdma in case they have any >> advice. > > The mentioning of IPoIB is a slightly confusing in the context of NFS-over-RDMA. > Are you running NFS over IPoIB? For all I'm aware, NFS over RDMA still needs an IP and port to be targeted to, so IPoIB is mandatory? At least the admin guide in the kernel says so. Right now I actually am running NFS over IPoIB (without RDMA), because of the issue at hand. And would like to turn on RDMA for enhanced performance. > From brief look on CQE error syndrome (local length error), the client sends wrong WQE. Does that point at an issue in the kernel code, or something I did wrong? The fstab entries for these mounts look like this: 10.110.10.200:/home /home nfs4 rw,rdma,port=20049,noatime,async,vers=4.2,_netdev 0 0 Is there anything more I can investigate? I tried turning connected mode off and lowering the mtu in turn, but that did not have any effect.