From: Timothy Pearson <firstname.lastname@example.org> To: linux-btrfs <email@example.com> Subject: Re: Unusual crash -- data rolled back ~2 weeks? Date: Sat, 9 Nov 2019 16:48:05 -0600 (CST) Message-ID: <1000805314.68851.1573339685782.JavaMail.firstname.lastname@example.org> (raw) In-Reply-To: <344827358.67114.1573338809278.JavaMail.email@example.com> One item I did forget to mention here is that the underlying device was expanded online using "btrfs fi resize max /mount/path" at most a month before the failure -- I don't have the exact timestamps available, so there remains a possibility that the latest files on the currently mounted filesystem correspond to the filesystem as it was immediately prior to the resize operation. Again, any suggestions welcome. It took us a bit of time (and several large file restores) to realize the filesystem had rolled back vs just corrupted a few files, so while there does exist a raw copy of the filesystem it is tainted by being mounted and written to before the copy was taken. ----- Original Message ----- > From: "Timothy Pearson" <firstname.lastname@example.org> > To: "linux-btrfs" <email@example.com> > Sent: Saturday, November 9, 2019 4:33:29 PM > Subject: Unusual crash -- data rolled back ~2 weeks? > We just experienced a very unusual crash on a Linux 5.3 file server using NFS to > serve a BTRFS filesystem. NFS went into deadlock (D wait) with no apparent > underlying disk subsystem problems, and when the server was hard rebooted to > clear the D wait the BTRFS filesystem remounted itself in the state that it was > in approximately two weeks earlier (!). There was also significant corruption > of certain files (e.g. LDAP MDB and MySQL InnoDB) noted -- we restored from > backup for those files, but are concerned about the status of the entire > filesystem at this point. > > We do not use subvolumes, snapshots, or any of the advanced features of BTRFS > beyond the data checksumming. I am at a loss as to how BTRFS could suddenly > just "forget" about the past two weeks of written data and (mostly) cleanly > roll back on the next mount without even throwing any warnings in dmesg. > > Any thoughts on how this is possible, and if there is any chance of getting the > lost couple weeks of data back, would be appreciated. > > Thank you!
next prev parent reply index Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-11-09 22:33 Timothy Pearson 2019-11-09 22:48 ` Timothy Pearson [this message] 2019-11-10 3:38 ` Qu Wenruo 2019-11-10 6:47 ` Timothy Pearson 2019-11-10 6:54 ` Qu Wenruo 2019-11-10 7:18 ` Timothy Pearson 2019-11-10 7:45 ` Qu Wenruo 2019-11-10 7:48 ` Timothy Pearson 2019-11-10 10:02 ` Timothy Pearson 2019-11-10 20:10 ` Zygo Blaxell 2019-11-11 23:28 ` Timothy Pearson 2019-11-11 23:33 ` Timothy Pearson 2019-11-12 11:30 ` Chris Murphy 2019-11-10 8:04 ` Andrei Borzenkov
Reply instructions: You may reply publically to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1000805314.68851.1573339685782.JavaMail.firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Linux-BTRFS Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \ email@example.com public-inbox-index linux-btrfs Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs AGPL code for this site: git clone https://public-inbox.org/public-inbox.git