From: Timothy Pearson <firstname.lastname@example.org> To: email@example.com Subject: Unusual crash -- data rolled back ~2 weeks? Date: Sat, 9 Nov 2019 16:33:29 -0600 (CST) Message-ID: <344827358.67114.1573338809278.JavaMail.firstname.lastname@example.org> (raw) We just experienced a very unusual crash on a Linux 5.3 file server using NFS to serve a BTRFS filesystem. NFS went into deadlock (D wait) with no apparent underlying disk subsystem problems, and when the server was hard rebooted to clear the D wait the BTRFS filesystem remounted itself in the state that it was in approximately two weeks earlier (!). There was also significant corruption of certain files (e.g. LDAP MDB and MySQL InnoDB) noted -- we restored from backup for those files, but are concerned about the status of the entire filesystem at this point. We do not use subvolumes, snapshots, or any of the advanced features of BTRFS beyond the data checksumming. I am at a loss as to how BTRFS could suddenly just "forget" about the past two weeks of written data and (mostly) cleanly roll back on the next mount without even throwing any warnings in dmesg. Any thoughts on how this is possible, and if there is any chance of getting the lost couple weeks of data back, would be appreciated. Thank you!
next reply index Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-11-09 22:33 Timothy Pearson [this message] 2019-11-09 22:48 ` Timothy Pearson 2019-11-10 3:38 ` Qu Wenruo 2019-11-10 6:47 ` Timothy Pearson 2019-11-10 6:54 ` Qu Wenruo 2019-11-10 7:18 ` Timothy Pearson 2019-11-10 7:45 ` Qu Wenruo 2019-11-10 7:48 ` Timothy Pearson 2019-11-10 10:02 ` Timothy Pearson 2019-11-10 20:10 ` Zygo Blaxell 2019-11-11 23:28 ` Timothy Pearson 2019-11-11 23:33 ` Timothy Pearson 2019-11-12 11:30 ` Chris Murphy 2019-11-10 8:04 ` Andrei Borzenkov
Reply instructions: You may reply publically to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=344827358.67114.1573338809278.JavaMail.email@example.com \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Linux-BTRFS Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \ firstname.lastname@example.org public-inbox-index linux-btrfs Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs AGPL code for this site: git clone https://public-inbox.org/public-inbox.git