Re: Data corruption in kernel 5.1+ with iSER attached ramdisk

[Ming Lei <ming.lei@redhat.com>]

On Wed, Dec 04, 2019 at 09:28:43PM -0500, Stephen Rust wrote:
> Hi Ming,
> 
> Thanks for all your help and insight. I really appreciate it.
> 
> > > Presumably non-brd devices, ie: real scsi devices work for these test
> > > cases because they accept un-aligned buffers?
> >
> > Right, not every driver supports such un-aligned buffer.
> 
> Can you please clarify: does the block layer require that it is called
> with 512-byte aligned buffers? If that is the case, would it make
> sense for the block interface (bio_add_page() or other) to reject
> buffers that are not aligned?

The things is a bit complicated, see the following xfs commits:

f8f9ee479439 xfs: add kmem_alloc_io()
d916275aa4dd xfs: get allocation alignment from the buftarg

Which applies request queue's dma alignment limit which may be
smaller than 512. Before this report, xfs should be the only known
user of passing un-aligned buffer.

So we can't add the check in bio_add_page(), in which request queue
may not be available, also bio_add_page() is really hot path, and
people hates to add unnecessary code in this function.

IMO, it is better for all FS or users of bio_add_page() to pass
512 aligned buffer.

> 
> It seems that passing these buffers on to underlying drivers that
> don't support un-aligned buffers can result in silent data corruption.
> Perhaps it would be better to fail the I/O up front. This would also
> help future proof the block interface when changes/new target drivers
> are added.

It is a brd device, strictly speaking, it doesn't matter to fail the
I/O or whatever, given either way should cause data loss.

> 
> I'm also curious how these same unaligned buffers from iSER made it to
> brd and were written successfully in the pre "multi-page bvec" world.
> (Just trying to understand, if you have any thoughts, as this same
> test case worked fine in 4.14+ until 5.1)

I am pretty sure that brd never supports un-aligned buffer, and I have
no idea why 'multi-page bvec' helper can cause this issue. However, I
am happy to investigate further if you can run previous trace on pre
'multi-page bvec' kernel.

> 
> > I am not familiar with RDMA, but from the trace we have done so far,
> > it is highly related with iser driver.
> 
> Do you think it is fair to say that the iSER/block integration is
> causing corruption by using un-aligned buffers?

As you saw, XFS changed the un-aligned buffer into aligned one for
avoiding the issue, so I think it is pretty fair to say that.

Thanks, 
Ming