Re: Simulating disk failure with a writeback cache

From: Josef Bacik <jbacik@fb.com>
To: Kent Overstreet <kent.overstreet@gmail.com>
Cc: "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
	"david@fromorbit.com" <david@fromorbit.com>,
	"hch@lst.de" <hch@lst.de>
Subject: Re: Simulating disk failure with a writeback cache
Date: Mon, 19 Dec 2016 10:27:34 -0500	[thread overview]
Message-ID: <1482161254.17731.2@smtp.office365.com> (raw)
In-Reply-To: <20161219030725.3kmsfu23sjypcnek@kmo-pixel>

On Sun, Dec 18, 2016 at 10:07 PM, Kent Overstreet 
<kent.overstreet@gmail.com> wrote:
> On Sun, Dec 18, 2016 at 05:51:11PM -0900, Kent Overstreet wrote:
>>  On Sun, Dec 18, 2016 at 09:19:57PM +0000, Josef Bacik wrote:
>>  > Eeesh these are older than I thought, I pushed to
>>  > https://github.com/josefbacik/fstests.git.  The fsx work is there 
>> and is
>>  > generic, the fsstress one has some btrfs specific stuff but you 
>> can just pull
>>  > that crap out and it'll work on anything.  Let me know if you 
>> need anything
>>  > else, thanks,
>> 
>>  Thanks
>> 
>>  I got your first fsx based test up and running, but - did you get 
>> it to pass
>>  with any existing filesystems?
>> 
>>  I finally figured out what I'm seeing, in _check_files() where it's 
>> looping over
>>  the mark: I changed it to so that it always checks the fsync marks 
>> in the order
>>  where it was created, but when it goes to check the very first mark 
>> it's getting
>>  the very last version of the file (at least, the file size is 
>> consistent with
>>  that).
>> 
>>  I just pushed what I'm working off of, but I don't think it's 
>> anything I
>>  broke...
>> 
>>  
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__evilpiepirate.org_git_xfstests.git&d=DgIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=sDzg6MvHymKOUgI8SFIm4Q&m=E_8Rycs0b-Vo8GLKPW3XOah6HMo1P1ixJKC56-np3BQ&s=xABKM88GhCUwdBnB8_7bzZM76O0U9tnQdK6p40kdXtU&e=
> 
> Just figured it out - log replay doesn't touch anything that hadn't 
> been written
> at that point in the log, so the newer journal entries were still 
> there. fun...
> 
> If I blow away the journal before replay, it works.

Oh sorry for some reason I only saw this reply but not your previous 
one about seeing the last version of the file.  Yeah that's kind of 
annoying, I'll make a note to fix the test to do a wipefs before doing 
the log replay.  Does using wipefs work as well as dd'ing the first bit 
of the disk for you?  Thanks,

Josef