From: Mikulas Patocka <mpatocka@redhat.com>
To: Milan Broz <gmazyland@gmail.com>,
Bart Van Assche <bvanassche@acm.org>,
"Theodore Ts'o" <tytso@mit.edu>,
Changheun Lee <nanich.lee@samsung.com>
Cc: axboe@kernel.dk, yi.zhang@redhat.com, bgoncalv@redhat.com,
dm-crypt@saout.de, linux-kernel@vger.kernel.org,
alex_y_xu@yahoo.ca, ming.lei@redhat.com,
linux-block@vger.kernel.org, dm-devel@redhat.com,
linux-nvme@lists.infradead.org, jaegeuk@kernel.org,
linux-ext4@vger.kernel.org, hch@lst.de
Subject: Re: [dm-devel] regression: data corruption with ext4 on LUKS on nvme with torvalds master
Date: Thu, 13 May 2021 15:22:25 -0400 (EDT) [thread overview]
Message-ID: <alpine.LRH.2.02.2105131510330.21927@file01.intranet.prod.int.rdu2.redhat.com> (raw)
In-Reply-To: <0e7b0b6e-e78c-f22d-af8d-d7bdcb597bea@gmail.com>
> On 5/13/21 7:15 AM, Theodore Ts'o wrote:
> > On Thu, May 13, 2021 at 06:42:22PM +0900, Changheun Lee wrote:
> >>
> >> Problem might be casued by exhausting of memory. And memory exhausting
> >> would be caused by setting of small bio_max_size. Actually it was not
> >> reproduced in my VM environment at first. But, I reproduced same problem
> >> when bio_max_size is set with 8KB forced. Too many bio allocation would
> >> be occurred by setting of 8KB bio_max_size.
> >
> > Hmm... I'm not sure how to align your diagnosis with the symptoms in
> > the bug report. If we were limited by memory, that should slow down
> > the I/O, but we should still be making forward progress, no? And a
> > forced reboot should not result in data corruption, unless maybe there
> > was a missing check for a failed memory allocation, causing data to be
> > written to the wrong location, a missing error check leading to the
> > block or file system layer not noticing that a write had failed
> > (although again, memory exhaustion should not lead to failed writes;
> > it might slow us down, sure, but if writes are being failed, something
> > is Badly Going Wrong --- things like writes to the swap device or
> > writes by the page cleaner must succeed, or else Things Would Go Bad
> > In A Hurry).
>
> After the LUKS data corruption issue was reported I decided to take a
> look at the dm-crypt code. In that code I found the following:
>
> static void clone_init(struct dm_crypt_io *io, struct bio *clone)
> {
> struct crypt_config *cc = io->cc;
>
> clone->bi_private = io;
> clone->bi_end_io = crypt_endio;
> bio_set_dev(clone, cc->dev->bdev);
> clone->bi_opf = io->base_bio->bi_opf;
> }
> [ ... ]
> static struct bio *crypt_alloc_buffer(struct dm_crypt_io *io, unsigned size)
> {
> [ ... ]
> clone = bio_alloc_bioset(GFP_NOIO, nr_iovecs, &cc->bs);
> [ ... ]
> clone_init(io, clone);
> [ ... ]
> for (i = 0; i < nr_iovecs; i++) {
> [ ... ]
> bio_add_page(clone, page, len, 0);
>
> remaining_size -= len;
> }
> [ ... ]
> }
>
> My interpretation is that crypt_alloc_buffer() allocates a bio,
> associates it with the underlying device and clones a bio. The input bio
> may have a size up to UINT_MAX while the new limit for the size of the
> cloned bio is max_sectors * 512. That causes bio_add_page() to fail if
> the input bio is larger than max_sectors * 512, hence the data
> corruption. Please note that this is a guess only and that I'm not
> familiar with the dm-crypt code.
>
> Bart.
We already had problems with too large bios in dm-crypt and we fixed it by
adding this piece of code:
/*
* Check if bio is too large, split as needed.
*/
if (unlikely(bio->bi_iter.bi_size > (BIO_MAX_VECS << PAGE_SHIFT)) &&
(bio_data_dir(bio) == WRITE || cc->on_disk_tag_size))
dm_accept_partial_bio(bio, ((BIO_MAX_VECS << PAGE_SHIFT) >> SECTOR_SHIFT));
It will ask the device mapper to split the bio if it is too large. So,
crypt_alloc_buffer can't receive a bio that is larger than BIO_MAX_VECS <<
PAGE_SHIFT.
Mikulas
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
next parent reply other threads:[~2021-05-13 19:22 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <a01ab479-69e8-9395-7d24-9de1eec28aff@acm.org>
[not found] ` <0e7b0b6e-e78c-f22d-af8d-d7bdcb597bea@gmail.com>
2021-05-13 19:22 ` Mikulas Patocka [this message]
2021-05-13 21:18 ` [dm-devel] regression: data corruption with ext4 on LUKS on nvme with torvalds master Bart Van Assche
2021-05-14 9:43 ` Mikulas Patocka
2021-05-14 9:50 ` Mikulas Patocka
[not found] ` <CGME20210514104426epcas1p3ee2f22f8e18c961118795c356e6a14ae@epcas1p3.samsung.com>
2021-05-14 10:26 ` Changheun Lee
2021-07-09 20:45 ` Samuel Mendoza-Jonas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.LRH.2.02.2105131510330.21927@file01.intranet.prod.int.rdu2.redhat.com \
--to=mpatocka@redhat.com \
--cc=alex_y_xu@yahoo.ca \
--cc=axboe@kernel.dk \
--cc=bgoncalv@redhat.com \
--cc=bvanassche@acm.org \
--cc=dm-crypt@saout.de \
--cc=dm-devel@redhat.com \
--cc=gmazyland@gmail.com \
--cc=hch@lst.de \
--cc=jaegeuk@kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=ming.lei@redhat.com \
--cc=nanich.lee@samsung.com \
--cc=tytso@mit.edu \
--cc=yi.zhang@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).