All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ming Lei <ming.lei@redhat.com>
To: Stephen Rust <srust@blockbridge.com>
Cc: Rob Townley <rob.townley@gmail.com>,
	Christoph Hellwig <hch@lst.de>, Jens Axboe <axboe@kernel.dk>,
	linux-block@vger.kernel.org, linux-rdma@vger.kernel.org,
	linux-scsi@vger.kernel.org, martin.petersen@oracle.com,
	target-devel@vger.kernel.org
Subject: Re: Data corruption in kernel 5.1+ with iSER attached ramdisk
Date: Tue, 3 Dec 2019 08:58:49 +0800	[thread overview]
Message-ID: <20191203005849.GB25002@ming.t460p> (raw)
In-Reply-To: <CAAFE1beMkvyRctGqpffd3o_QtDH0CrmQSb=fV4GzqMUXWzPyOw@mail.gmail.com>

On Mon, Dec 02, 2019 at 01:42:15PM -0500, Stephen Rust wrote:
> Hi Ming,
> 
> > I may get one machine with Mellanox NIC, is it easy to setup & reproduce
> > just in the local machine(both host and target are setup on same machine)?
> 
> Yes, I have reproduced locally on one machine (using the IP address of
> the Mellanox NIC as the target IP), with iser enabled on the target,
> and iscsiadm connected via iser.
> 
> e.g.:
> target:
> /iscsi/iqn.20.../0.0.0.0:3260> enable_iser true
> iSER enable now: True
> 
>   | |   o- portals
> ....................................................................................................
> [Portals: 1]
>   | |     o- 0.0.0.0:3260
> ...................................................................................................
> [iser]
> 
> client:
> # iscsiadm -m node -o update --targetname <target> -n
> iface.transport_name -v iser
> # iscsiadm -m node --targetname <target> --login
> # iscsiadm -m session
> iser: [3] 172.16.XX.XX:3260,1
> iqn.2003-01.org.linux-iscsi.x8664:sn.c46c084919b0 (non-flash)
> 
> > Please try to trace bio_add_page() a bit via 'bpftrace ./ilo.bt'.
> 
> Here is the output of this trace from a failed run:
> 
> # bpftrace lio.bt
> modprobe: FATAL: Module kheaders not found.
> Attaching 3 probes...
> 512 76
> 4096 0
> 4096 0
> 4096 0
> 4096 76

The above buffer might be the reason, 4096 is length, and 76 is the
offset, that means the added buffer crosses two pages, meantime the
buffer isn't aligned.

We need to figure out why the magic 76 offset is passed from target or
driver.

Please install bcc and collect the following log:

/usr/share/bcc/tools/trace -K 'bio_add_page ((arg4 & 512) != 0) "%d %d", arg3, arg4 '


Thanks,
Ming


WARNING: multiple messages have this Message-ID (diff)
From: Ming Lei <ming.lei@redhat.com>
To: Stephen Rust <srust@blockbridge.com>
Cc: Rob Townley <rob.townley@gmail.com>,
	Christoph Hellwig <hch@lst.de>, Jens Axboe <axboe@kernel.dk>,
	linux-block@vger.kernel.org, linux-rdma@vger.kernel.org,
	linux-scsi@vger.kernel.org, martin.petersen@oracle.com,
	target-devel@vger.kernel.org
Subject: Re: Data corruption in kernel 5.1+ with iSER attached ramdisk
Date: Tue, 03 Dec 2019 00:58:49 +0000	[thread overview]
Message-ID: <20191203005849.GB25002@ming.t460p> (raw)
In-Reply-To: <CAAFE1beMkvyRctGqpffd3o_QtDH0CrmQSb=fV4GzqMUXWzPyOw@mail.gmail.com>

On Mon, Dec 02, 2019 at 01:42:15PM -0500, Stephen Rust wrote:
> Hi Ming,
> 
> > I may get one machine with Mellanox NIC, is it easy to setup & reproduce
> > just in the local machine(both host and target are setup on same machine)?
> 
> Yes, I have reproduced locally on one machine (using the IP address of
> the Mellanox NIC as the target IP), with iser enabled on the target,
> and iscsiadm connected via iser.
> 
> e.g.:
> target:
> /iscsi/iqn.20.../0.0.0.0:3260> enable_iser true
> iSER enable now: True
> 
>   | |   o- portals
> ....................................................................................................
> [Portals: 1]
>   | |     o- 0.0.0.0:3260
> ...................................................................................................
> [iser]
> 
> client:
> # iscsiadm -m node -o update --targetname <target> -n
> iface.transport_name -v iser
> # iscsiadm -m node --targetname <target> --login
> # iscsiadm -m session
> iser: [3] 172.16.XX.XX:3260,1
> iqn.2003-01.org.linux-iscsi.x8664:sn.c46c084919b0 (non-flash)
> 
> > Please try to trace bio_add_page() a bit via 'bpftrace ./ilo.bt'.
> 
> Here is the output of this trace from a failed run:
> 
> # bpftrace lio.bt
> modprobe: FATAL: Module kheaders not found.
> Attaching 3 probes...
> 512 76
> 4096 0
> 4096 0
> 4096 0
> 4096 76

The above buffer might be the reason, 4096 is length, and 76 is the
offset, that means the added buffer crosses two pages, meantime the
buffer isn't aligned.

We need to figure out why the magic 76 offset is passed from target or
driver.

Please install bcc and collect the following log:

/usr/share/bcc/tools/trace -K 'bio_add_page ((arg4 & 512) != 0) "%d %d", arg3, arg4 '


Thanks,
Ming

  reply	other threads:[~2019-12-03  0:59 UTC|newest]

Thread overview: 60+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CAAFE1bd9wuuobpe4VK7Ty175j7mWT+kRmHCNhVD+6R8MWEAqmw@mail.gmail.com>
2019-11-28  1:57 ` Data corruption in kernel 5.1+ with iSER attached ramdisk Ming Lei
2019-11-28  1:57   ` Ming Lei
     [not found]   ` <CA+VdTb_-CGaPjKUQteKVFSGqDz-5o-tuRRkJYqt8B9iOQypiwQ@mail.gmail.com>
2019-11-28  2:58     ` Ming Lei
2019-11-28  2:58       ` Ming Lei
     [not found]       ` <CAAFE1bfsXsKGyw7SU_z4NanT+wmtuJT=XejBYbHHMCDQwm73sw@mail.gmail.com>
2019-11-28  4:25         ` Stephen Rust
2019-11-28  4:25           ` Stephen Rust
2019-11-28  5:51           ` Rob Townley
2019-11-28  5:51             ` Rob Townley
2019-11-28  9:12         ` Ming Lei
2019-11-28  9:12           ` Ming Lei
2019-12-02 18:42           ` Stephen Rust
2019-12-02 18:42             ` Stephen Rust
2019-12-03  0:58             ` Ming Lei [this message]
2019-12-03  0:58               ` Ming Lei
2019-12-03  3:04               ` Stephen Rust
2019-12-03  3:04                 ` Stephen Rust
2019-12-03  3:14                 ` Ming Lei
2019-12-03  3:14                   ` Ming Lei
2019-12-03  3:26                   ` Stephen Rust
2019-12-03  3:26                     ` Stephen Rust
2019-12-03  3:50                     ` Stephen Rust
2019-12-03  3:50                       ` Stephen Rust
2019-12-03 12:45                       ` Ming Lei
2019-12-03 12:45                         ` Ming Lei
2019-12-03 19:56                         ` Stephen Rust
2019-12-03 19:56                           ` Stephen Rust
2019-12-04  1:05                           ` Ming Lei
2019-12-04  1:05                             ` Ming Lei
2019-12-04 17:23                             ` Stephen Rust
2019-12-04 17:23                               ` Stephen Rust
2019-12-04 23:02                               ` Ming Lei
2019-12-04 23:02                                 ` Ming Lei
2019-12-05  0:16                                 ` Bart Van Assche
2019-12-05  0:16                                   ` Bart Van Assche
2019-12-05 14:44                                   ` Stephen Rust
2019-12-05 14:44                                     ` Stephen Rust
2019-12-05  2:28                                 ` Stephen Rust
2019-12-05  2:28                                   ` Stephen Rust
2019-12-05  3:05                                   ` Ming Lei
2019-12-05  3:05                                     ` Ming Lei
2019-12-05  9:17                                 ` Sagi Grimberg
2019-12-05  9:17                                   ` Sagi Grimberg
2019-12-05 14:36                                   ` Stephen Rust
2019-12-05 14:36                                     ` Stephen Rust
     [not found]                                   ` <CAAFE1beqFBQS_zVYEXFTD2qu8PAF9hBSW4j1k9ZD6MhU_gWg5Q@mail.gmail.com>
2020-03-25  0:15                                     ` Sagi Grimberg
2020-03-25  0:15                                       ` Sagi Grimberg
2020-03-30 17:08                                       ` Stephen Rust
2020-03-30 17:08                                         ` Stephen Rust
2020-03-31  1:07                                         ` Sagi Grimberg
2020-03-31  1:07                                           ` Sagi Grimberg
2020-04-01  0:38                                         ` Sagi Grimberg
2020-04-01  0:38                                           ` Sagi Grimberg
2020-04-02 20:03                                           ` Stephen Rust
2020-04-02 20:03                                             ` Stephen Rust
2020-04-02 22:16                                             ` Sagi Grimberg
2020-04-02 22:16                                               ` Sagi Grimberg
2019-12-04  2:39                           ` Ming Lei
2019-12-04  2:39                             ` Ming Lei
2019-12-03  4:15                     ` Ming Lei
2019-12-03  4:15                       ` Ming Lei

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191203005849.GB25002@ming.t460p \
    --to=ming.lei@redhat.com \
    --cc=axboe@kernel.dk \
    --cc=hch@lst.de \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=martin.petersen@oracle.com \
    --cc=rob.townley@gmail.com \
    --cc=srust@blockbridge.com \
    --cc=target-devel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.