Re: Data corruption when using multiple devices with NVMEoF TCP

[Sagi Grimberg <sagi@grimberg.me>]

> Okay, tried both v5.10 and latest 58cf05f597b0.
> 
> And same behavior
>   - data corruption on the initiator side when creating a raid-0 volume
> using 2 nvme-tcp devices;
>   - no data corruption either on local target side, or on initiator
> side but only using 1 nvme-tcp devoce.
> 
> A difference I can see on the max_sectors_kb is that, now on the
> target side, /sys/block/nvme*n1/queue/max_sectors_kb also becomes
> 1280.
> 

Thanks Hao,

I'm thinking we maybe have an issue with bio splitting/merge/cloning.

Question, if you build the raid0 in the target and expose that over
nvmet-tcp (with a single namespace), does the issue happen?

Also, would be interesting to add this patch and see if the following
print pops up, and if it correlates when you see the issue:

--
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 979ee31b8dd1..d0a68cdb374f 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -243,6 +243,9 @@ static void nvme_tcp_init_iter(struct 
nvme_tcp_request *req,
                 nsegs = bio_segments(bio);
                 size = bio->bi_iter.bi_size;
                 offset = bio->bi_iter.bi_bvec_done;
+               if (rq->bio != rq->biotail)
+                       pr_info("rq %d (%s) contains multiple bios bvec: 
nsegs %d size %d offset %ld\n",
+                               rq->tag, dir == WRITE ? "WRITE" : 
"READ", nsegs, size, offset);
         }

         iov_iter_bvec(&req->iter, dir, vec, nsegs, size);
--

I'll try to look further to understand if we have an issue there.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme