From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B0A1C43603 for ; Thu, 5 Dec 2019 14:36:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DD2FA21835 for ; Thu, 5 Dec 2019 14:36:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=blockbridge-com.20150623.gappssmtp.com header.i=@blockbridge-com.20150623.gappssmtp.com header.b="Mliv1jVw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729430AbfLEOgh (ORCPT ); Thu, 5 Dec 2019 09:36:37 -0500 Received: from mail-ot1-f65.google.com ([209.85.210.65]:39896 "EHLO mail-ot1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729240AbfLEOgh (ORCPT ); Thu, 5 Dec 2019 09:36:37 -0500 Received: by mail-ot1-f65.google.com with SMTP id 77so2752340oty.6 for ; Thu, 05 Dec 2019 06:36:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=blockbridge-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=nb1iCxXYbt2jkN74UTYTGkL9Q0crXuctu/rAfI3chic=; b=Mliv1jVwJONxPxX8a+HIrkjg+1+iqeWdjJtk5PSTsbTpVYpYK9rnTYc9IlzCWSFp6C hLRbRY6NqdzU1TbcsV4pre8NqQZU57la7/w15fp3tcEcKMpxuOPcs8i+1sk5KJrShNsT biPNn/LJyLs9i+ykuRTqjeBacKl/swEJlB6Ttljja+laCFHVPq84rZEPT6GzOuteNa5X NtRXQOicvkVx/5wUs5HbiQHw8WeBTesrNWlpIWpPwnrBCVL/bBz6AEsjSIZWz2tyzRe4 O5gjrF9HyKIJNiWKeygmUnfRMrGtpcZSR2dwY1LITYqQSbCWMO3MdSeGvBKjUesbOur8 hI0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=nb1iCxXYbt2jkN74UTYTGkL9Q0crXuctu/rAfI3chic=; b=SOIutYS7NdPH4bqKWNlOFuGlbgcgNbNKAr2fdznTZqaOmue3VRmbqPGzsvZK/NQCpN pXzQR4vPfqSXIHYimhubB/x7LBRk2xb2HTui6Ruhg8cZQjyVH/o/nJKKSNO0SmPmq23k MVREfihwQKtSHeHRmbP2ELnjciz0G6qcPOL2lzJe4BjyLI8wyeWNbHTwXE+gAVJrQJ9a PjchqMmV1RccuArkUy/bKxyHGFCl7LFmmgNDCUYBoe45CQfl07H6Bdab7t9ebulmoS9M rPwqtqix84mAGITYx08Mb0lZMu3+RI5FEJzuejxzH4KqWilaKP7gHN18XCtZfY4jY9Af L7Yw== X-Gm-Message-State: APjAAAXXwBtDH/yl2vOIjtnFxSqM1BwqCeAhC8Ofmg2R+5e+8TjrOoav pPxs45BaojRWtkLeNe1Hub1CYoixvtkgnjsbNvF+BQ== X-Google-Smtp-Source: APXvYqzGI+fRlB/GMXFu6Wwbd4QM3eRy5QYgmqsBwAn/vjLzUklIRe/Ogzd3OhFZVBzVUTgdbpdfwtlUMu6+ljI8xcQ= X-Received: by 2002:a05:6830:2141:: with SMTP id r1mr6342822otd.124.1575556596256; Thu, 05 Dec 2019 06:36:36 -0800 (PST) MIME-Version: 1.0 References: <20191203005849.GB25002@ming.t460p> <20191203031444.GB6245@ming.t460p> <20191203124558.GA22805@ming.t460p> <20191204010529.GA3910@ming.t460p> <20191204230225.GA26189@ming.t460p> In-Reply-To: From: Stephen Rust Date: Thu, 5 Dec 2019 09:36:25 -0500 Message-ID: Subject: Re: Data corruption in kernel 5.1+ with iSER attached ramdisk To: Sagi Grimberg Cc: Ming Lei , Rob Townley , Christoph Hellwig , Jens Axboe , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, linux-scsi@vger.kernel.org, martin.petersen@oracle.com, target-devel@vger.kernel.org, Doug Ledford , Jason Gunthorpe , Max Gurtovoy Content-Type: text/plain; charset="UTF-8" Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Hi Sagi, On Thu, Dec 5, 2019 at 4:17 AM Sagi Grimberg wrote: > > Just got this one (Thanks for CCing me Ming, been extremely busy > lately). No problem, thanks for looking into it! > So it looks from the report that this is the immediate-data and > unsolicited data-out flows, which indeed seem to violate the alignment > assumption. The reason is that isert post recv a contig rx_desc which > has both the headers and the data, and when it gets immediate_data it > will set the data sg to rx_desc+offset (which are the headers). > > Stephen, > As a work-around for now, you should turn off immediate-data in your LIO > target. I'll work on a fix. I have confirmed that turning off ImmediateData in the target (and reconnecting) is a successful workaround for this test case. All of the I/O as reported by bio_add_page() is aligned. Using the previously described bpftrace script with 512 offset: # bpftrace lio.bt Attaching 4 probes... 512 0 512 0 1 131071 4096 0 4096 0 0 0 4096 0 4096 0 0 8 4096 0 4096 0 0 131064 4096 0 4096 0 1 131064 4096 0 4096 0 0 0 4096 0 4096 0 0 8 512 0 512 0 0 131071 4096 0 4096 0 0 130944 4096 0 4096 0 0 131056 > Thanks for reporting! Please let me know if you need any additional information, or if I can assist further. I would be happy to test any patches when you are ready. Thanks, Steve