From: David Sterba <dsterba@suse.cz>
To: Dmitriy Gorokh <Dmitriy.Gorokh@wdc.com>
Cc: "linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>
Subject: Re: [PATCH] btrfs: raid56: data corruption on a device removal
Date: Wed, 12 Dec 2018 16:53:52 +0100 [thread overview]
Message-ID: <20181212155352.GZ23615@twin.jikos.cz> (raw)
In-Reply-To: <66F8D435-4E51-4761-B6CF-BA96F4BC5986@wdc.com>
On Wed, Dec 12, 2018 at 12:25:55AM +0000, Dmitriy Gorokh wrote:
> I found that RAID5 or RAID6 filesystem might be got corrupted in the following scenario:
>
> 1. Create 4 disks RAID6 filesystem
> 2. Preallocate 16 10Gb files
> 3. Run fio: 'fio --name=testload --directory=./ --size=10G --numjobs=16 --bs=64k --iodepth=64 --rw=randrw --verify=sha256 --time_based --runtime=3600’
> 4. After few minutes pull out two drives: 'echo 1 > /sys/block/sdc/device/delete ; echo 1 > /sys/block/sdd/device/delete’
>
> About 5 of 10 times the test is run, it led to silent data corruption
> of a random extent, resulting in ‘IO Error’ and ‘csum failed’ messages
> while trying to read the affected file. It usually affects only small
> portion of the files and only one underlying extent of a file. When I
> converted logical address of the damaged extent to physical address
> and dumped a stripe directly from drives, I saw specific pattern,
> always the same when the issue occurs.
>
> I found that few bios which were being processed right during the
> drives removal, contained non zero bio->bi_iter.bi_done field despite
> of EIO bi_status. bi_sector field was also increased from original
> one by that 'bi_done' value. Looks like this is a quite rare
> condition. Subsequently, in the raid_rmw_end_io handler that failed
> bio can be translated to a wrong stripe number and fail wrong rbio.
Thanks for the analysis, sounds correct to me and the fix too. Would be
good if you can attach the logs or portions of the dumps you've used to
understand the problem, like the pattern you mention above.
next prev parent reply other threads:[~2018-12-12 15:54 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-12 0:25 [PATCH] btrfs: raid56: data corruption on a device removal Dmitriy Gorokh
2018-12-12 9:09 ` Johannes Thumshirn
2018-12-12 15:53 ` David Sterba [this message]
2018-12-14 17:48 ` [PATCH v2] " Dmitriy Gorokh
2018-12-26 0:15 ` Liu Bo
2019-01-04 16:49 ` David Sterba
2019-01-07 11:03 ` Johannes Thumshirn
2019-01-07 15:34 ` David Sterba
2019-01-10 16:49 ` Johannes Thumshirn
2019-01-11 8:08 ` Johannes Thumshirn
2019-01-11 9:26 ` Ming Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181212155352.GZ23615@twin.jikos.cz \
--to=dsterba@suse.cz \
--cc=Dmitriy.Gorokh@wdc.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).