From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D0A7C3B186 for ; Wed, 12 Feb 2020 16:03:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E8BC621569 for ; Wed, 12 Feb 2020 16:03:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727582AbgBLQDa (ORCPT ); Wed, 12 Feb 2020 11:03:30 -0500 Received: from mail1.ugh.no ([178.79.162.34]:49078 "EHLO mail1.ugh.no" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726728AbgBLQDa (ORCPT ); Wed, 12 Feb 2020 11:03:30 -0500 Received: from localhost (localhost [127.0.0.1]) by mail1.ugh.no (Postfix) with ESMTP id ADC6424EAC9; Wed, 12 Feb 2020 17:03:28 +0100 (CET) Received: from mail1.ugh.no ([127.0.0.1]) by localhost (catastrophix.ugh.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uPMdJS4zdReN; Wed, 12 Feb 2020 17:03:27 +0100 (CET) Received: from [10.255.64.11] (unknown [185.176.245.143]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: andre@tomt.net) by mail.ugh.no (Postfix) with ESMTPSA id B710324EAB0; Wed, 12 Feb 2020 17:03:27 +0100 (CET) Subject: Re: [PATCH v2] xprtrdma: Fix DMA scatter-gather list mapping imbalance To: Chuck Lever Cc: Linux NFS Mailing List , iommu@lists.linux-foundation.org References: <158151473332.515306.1111360128438553868.stgit@morisot.1015granger.net> <869DC73D-190E-46AB-B8F8-1A394F92AF41@oracle.com> From: Andre Tomt Message-ID: <1d2693b1-b37f-c611-91c3-55b567be5274@tomt.net> Date: Wed, 12 Feb 2020 17:03:27 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: <869DC73D-190E-46AB-B8F8-1A394F92AF41@oracle.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org On 12.02.2020 14:48, Chuck Lever wrote: >> On Feb 12, 2020, at 8:43 AM, Chuck Lever wrote: >> >> The @nents value that was passed to ib_dma_map_sg() has to be passed >> to the matching ib_dma_unmap_sg() call. If ib_dma_map_sg() choses to >> concatenate sg entries, it will return a different nents value than >> it was passed. >> >> The bug was exposed by recent changes to the AMD IOMMU driver, which >> enabled sg entry concatenation. >> >> Looking all the way back to 4143f34e01e9 ("xprtrdma: Port to new >> memory registration API") and reviewing other kernel ULPs, it's not >> clear that the frwr_map() logic was ever correct for this case. >> >> Reported-by: Andre Tomt >> Suggested-by: Robin Murphy >> Signed-off-by: Chuck Lever >> --- >> include/trace/events/rpcrdma.h | 6 ++++-- >> net/sunrpc/xprtrdma/frwr_ops.c | 13 +++++++------ >> 2 files changed, 11 insertions(+), 8 deletions(-) >> >> Hi Andre, here's take 2, based on the trace data you sent me. >> Please let me know if this one fares any better. >> >> Changes since v1: >> - Ensure the correct nents value is passed to ib_map_mr_sg >> - Record the mr_nents value in the MR trace points Verified working (with the patch correction) in my environment, with some quick testing (mount + some random and bulk I/O) client, 5.5.3 + patch + amd iommu on = OK client, 5.5.3 + patch + amd iommu off = OK client, 5.6-rc1 + patch + amd iommu on = OK server, 5.5.3 + patch + intel iommu on = OK Thanks!