* filesystem dead, xfs_repair won't help
@ 2017-04-10 9:23 Avi Kivity
2017-04-10 9:42 ` Avi Kivity
` (2 more replies)
0 siblings, 3 replies; 32+ messages in thread
From: Avi Kivity @ 2017-04-10 9:23 UTC (permalink / raw)
To: linux-xfs
Today my kernel complained that in memory metadata is corrupt and
asked that I run xfs_repair. But xfs_repair doesn't like the
superblock and isn't able to find a secondary superblock.
Latest Fedora 25 kernel, new Intel NVMe drive (worked for a few weeks
without issue).
Anything I can do to recover the data?
^ permalink raw reply [flat|nested] 32+ messages in thread* Re: filesystem dead, xfs_repair won't help 2017-04-10 9:23 filesystem dead, xfs_repair won't help Avi Kivity @ 2017-04-10 9:42 ` Avi Kivity 2017-04-10 15:35 ` Brian Foster 2017-04-10 9:43 ` allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) L A Walsh 2017-04-10 15:49 ` filesystem dead, xfs_repair won't help Eric Sandeen 2 siblings, 1 reply; 32+ messages in thread From: Avi Kivity @ 2017-04-10 9:42 UTC (permalink / raw) To: linux-xfs On 04/10/2017 12:23 PM, Avi Kivity wrote: > Today my kernel complained that in memory metadata is corrupt and > asked that I run xfs_repair. But xfs_repair doesn't like the > superblock and isn't able to find a secondary superblock. > > Latest Fedora 25 kernel, new Intel NVMe drive (worked for a few weeks > without issue). > > Anything I can do to recover the data? Initial error: Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Metadata CRC error detected at xfs_agfl_read_verify+0xcd/0x100 [xfs], xfs_agfl block 0x2cb68e13 Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Unmount and run xfs_repair Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): First 64 bytes of corrupted metadata buffer: Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75400: 23 40 8f 28 5b 50 3a b4 f8 54 1e 31 97 f4 fe ed #@.([P:..T.1.... Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75410: 62 87 57 51 ee 9d 31 02 ec 2c 10 46 6c 93 db 09 b.WQ..1..,.Fl... Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75420: ae 7a ea b3 91 49 7e d3 99 a4 25 49 11 c5 8b be .z...I~...%I.... Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75430: e4 2e 14 d4 8a f8 5f 98 66 d8 67 72 ec c9 1a d5 ......_.f.gr.... Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): metadata I/O error: block 0x2cb68e13 ("xfs_trans_read_buf_map") error 74 numblks 1 Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): xfs_do_force_shutdown(0x8) called from line 236 of file fs/xfs/libxfs/xfs_defer.c. Return address = 0xffffffffc05bdbc6 Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Corruption of in-memory data detected. Shutting down filesystem Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Please umount the filesystem and rectify the problem(s) After restart: Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Mounting V5 Filesystem Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Starting recovery (logdev: internal) Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Metadata CRC error detected at xfs_agfl_read_verify+0xcd/0x100 [xfs], xfs_agfl block 0x2cb68e13 Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Unmount and run xfs_repair Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): First 64 bytes of corrupted metadata buffer: Apr 10 11:47:58 avi.cloudius-systems.com kernel: ffff9450761d4a00: 23 40 8f 28 5b 50 3a b4 f8 54 1e 31 97 f4 fe ed #@.([P:..T.1.... Apr 10 11:47:58 avi.cloudius-systems.com kernel: ffff9450761d4a10: 62 87 57 51 ee 9d 31 02 ec 2c 10 46 6c 93 db 09 b.WQ..1..,.Fl... Apr 10 11:47:58 avi.cloudius-systems.com kernel: ffff9450761d4a20: ae 7a ea b3 91 49 7e d3 99 a4 25 49 11 c5 8b be .z...I~...%I.... Apr 10 11:47:58 avi.cloudius-systems.com kernel: ffff9450761d4a30: e4 2e 14 d4 8a f8 5f 98 66 d8 67 72 ec c9 1a d5 ......_.f.gr.... Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): metadata I/O error: block 0x2cb68e13 ("xfs_trans_read_buf_map") error 74 numblks 1 Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Internal error xfs_trans_cancel at line 983 of file fs/xfs/xfs_trans.c. Caller xfs_efi_recover+0x18e/0x1c0 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: CPU: 3 PID: 1063 Comm: mount Not tainted 4.10.8-200.fc25.x86_64 #1 Apr 10 11:47:58 avi.cloudius-systems.com kernel: Hardware name: /DH77EB, BIOS EBH7710H.86A.0099.2013.0125.1400 01/25/2013 Apr 10 11:47:58 avi.cloudius-systems.com kernel: Call Trace: Apr 10 11:47:58 avi.cloudius-systems.com kernel: dump_stack+0x63/0x86 Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_error_report+0x3c/0x40 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? xfs_efi_recover+0x18e/0x1c0 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_trans_cancel+0xb6/0xe0 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_efi_recover+0x18e/0x1c0 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: xlog_recover_process_efi+0x2c/0x50 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: xlog_recover_process_intents.isra.42+0x122/0x160 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? xfs_reinit_percpu_counters+0x46/0x50 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: xlog_recover_finish+0x23/0xb0 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_log_mount_finish+0x29/0x50 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_mountfs+0x6ce/0x930 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_fs_fill_super+0x3ee/0x570 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: mount_bdev+0x178/0x1b0 Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? xfs_test_remount_options.isra.14+0x60/0x60 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_fs_mount+0x15/0x20 [xfs] Apr 10 11:47:58 avi.cloudius-systems.com kernel: mount_fs+0x38/0x150 Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? __alloc_percpu+0x15/0x20 Apr 10 11:47:58 avi.cloudius-systems.com kernel: vfs_kern_mount+0x67/0x130 Apr 10 11:47:58 avi.cloudius-systems.com kernel: do_mount+0x1dd/0xc50 Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? _copy_from_user+0x4e/0x80 Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? memdup_user+0x4f/0x70 Apr 10 11:47:58 avi.cloudius-systems.com kernel: SyS_mount+0x83/0xd0 Apr 10 11:47:58 avi.cloudius-systems.com kernel: do_syscall_64+0x67/0x180 Apr 10 11:47:58 avi.cloudius-systems.com kernel: entry_SYSCALL64_slow_path+0x25/0x25 Apr 10 11:47:58 avi.cloudius-systems.com kernel: RIP: 0033:0x7f5cb9a626fa Apr 10 11:47:58 avi.cloudius-systems.com kernel: RSP: 002b:00007ffeffa2c928 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 Apr 10 11:47:58 avi.cloudius-systems.com kernel: RAX: ffffffffffffffda RBX: 000055b59fd6f030 RCX: 00007f5cb9a626fa Apr 10 11:47:58 avi.cloudius-systems.com kernel: RDX: 000055b59fd6f210 RSI: 000055b59fd6f250 RDI: 000055b59fd6f230 Apr 10 11:47:58 avi.cloudius-systems.com kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000012 Apr 10 11:47:58 avi.cloudius-systems.com kernel: R10: 00000000c0ed0000 R11: 0000000000000246 R12: 000055b59fd6f230 Apr 10 11:47:58 avi.cloudius-systems.com kernel: R13: 000055b59fd6f210 R14: 0000000000000000 R15: 00000000ffffffff Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): xfs_do_force_shutdown(0x8) called from line 984 of file fs/xfs/xfs_trans.c. Return address = 0xffffffffc056324f Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Corruption of in-memory data detected. Shutting down filesystem Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Please umount the filesystem and rectify the problem(s) Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Failed to recover intents Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): log mount finish failed smart (note error at end; there were no kernel I/O errors from the block layer): $ sudo smartctl -a /dev/nvme0n1 smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.10.8-200.fc25.x86_64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: INTEL SSDPEKKW512G7 Serial Number: BTPY6313086D512F Firmware Version: PSF100C PCI Vendor/Subsystem ID: 0x8086 IEEE OUI Identifier: 0x5cd2e4 Controller ID: 1 Number of Namespaces: 1 Namespace 1 Size/Capacity: 512,110,190,592 [512 GB] Namespace 1 Formatted LBA Size: 512 Local Time is: Mon Apr 10 12:36:41 2017 IDT Firmware Updates (0x12): 1 Slot, no Reset required Optional Admin Commands (0x0006): Format Frmw_DL Optional NVM Commands (0x001e): Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Maximum Data Transfer Size: 32 Pages Warning Comp. Temp. Threshold: 70 Celsius Critical Comp. Temp. Threshold: 80 Celsius Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 9.00W - - 0 0 0 0 5 5 1 + 4.60W - - 1 1 1 1 30 30 2 + 3.80W - - 2 2 2 2 30 30 3 - 0.0700W - - 3 3 3 3 10000 300 4 - 0.0050W - - 4 4 4 4 2000 10000 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02, NSID 0x1) Critical Warning: 0x00 Temperature: 27 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 8,854,487 [4.53 TB] Data Units Written: 5,652,445 [2.89 TB] Host Read Commands: 446,901,662 Host Write Commands: 35,627,742 Controller Busy Time: 633 Power Cycles: 24 Power On Hours: 987 Unsafe Shutdowns: 16 Media and Data Integrity Errors: 1 Error Information Log Entries: 1 Warning Comp. Temperature Time: 11 Critical Comp. Temperature Time: 0 Error Information (NVMe Log 0x01, max 64 entries) Num ErrCount SQId CmdId Status PELoc LBA NSID VS 0 1 1 0x0000 0x0286 - 0 1 - ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-10 9:42 ` Avi Kivity @ 2017-04-10 15:35 ` Brian Foster 2017-04-11 7:46 ` Avi Kivity 0 siblings, 1 reply; 32+ messages in thread From: Brian Foster @ 2017-04-10 15:35 UTC (permalink / raw) To: Avi Kivity; +Cc: linux-xfs On Mon, Apr 10, 2017 at 12:42:33PM +0300, Avi Kivity wrote: > On 04/10/2017 12:23 PM, Avi Kivity wrote: > > Today my kernel complained that in memory metadata is corrupt and > > asked that I run xfs_repair. But xfs_repair doesn't like the > > superblock and isn't able to find a secondary superblock. > > > > Latest Fedora 25 kernel, new Intel NVMe drive (worked for a few weeks > > without issue). > > > > Anything I can do to recover the data? > Well I can't explain why you have a checksum error, but what do you mean that xfs_repair doesn't like the superblock? Can you provide the xfs_repair output? It seems strange for xfs_repair to not find the superblock of a filesystem that can otherwise run log recovery up until it encounters the buffer with a bad crc. It also might be useful to find out exactly what that error reported by smartctl means. Are you aware of whether it pre-existed the filesystem issue or not? Brian > > Initial error: > > Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Metadata CRC > error detected at xfs_agfl_read_verify+0xcd/0x100 [xfs], xfs_agfl block > 0x2cb68e13 > Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Unmount and > run xfs_repair > Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): First 64 > bytes of corrupted metadata buffer: > Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75400: 23 40 8f > 28 5b 50 3a b4 f8 54 1e 31 97 f4 fe ed #@.([P:..T.1.... > Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75410: 62 87 57 > 51 ee 9d 31 02 ec 2c 10 46 6c 93 db 09 b.WQ..1..,.Fl... > Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75420: ae 7a ea > b3 91 49 7e d3 99 a4 25 49 11 c5 8b be .z...I~...%I.... > Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75430: e4 2e 14 > d4 8a f8 5f 98 66 d8 67 72 ec c9 1a d5 ......_.f.gr.... > Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): metadata I/O > error: block 0x2cb68e13 ("xfs_trans_read_buf_map") error 74 numblks 1 > Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): > xfs_do_force_shutdown(0x8) called from line 236 of file > fs/xfs/libxfs/xfs_defer.c. Return address = 0xffffffffc05bdbc6 > Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Corruption > of in-memory data detected. Shutting down filesystem > Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Please > umount the filesystem and rectify the problem(s) > > > After restart: > > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Mounting V5 > Filesystem > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Starting > recovery (logdev: internal) > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Metadata CRC > error detected at xfs_agfl_read_verify+0xcd/0x100 [xfs], xfs_agfl block > 0x2cb68e13 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Unmount and > run xfs_repair > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): First 64 > bytes of corrupted metadata buffer: > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ffff9450761d4a00: 23 40 8f > 28 5b 50 3a b4 f8 54 1e 31 97 f4 fe ed #@.([P:..T.1.... > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ffff9450761d4a10: 62 87 57 > 51 ee 9d 31 02 ec 2c 10 46 6c 93 db 09 b.WQ..1..,.Fl... > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ffff9450761d4a20: ae 7a ea > b3 91 49 7e d3 99 a4 25 49 11 c5 8b be .z...I~...%I.... > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ffff9450761d4a30: e4 2e 14 > d4 8a f8 5f 98 66 d8 67 72 ec c9 1a d5 ......_.f.gr.... > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): metadata I/O > error: block 0x2cb68e13 ("xfs_trans_read_buf_map") error 74 numblks 1 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Internal > error xfs_trans_cancel at line 983 of file fs/xfs/xfs_trans.c. Caller > xfs_efi_recover+0x18e/0x1c0 [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: CPU: 3 PID: 1063 Comm: > mount Not tainted 4.10.8-200.fc25.x86_64 #1 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: Hardware name: > /DH77EB, BIOS EBH7710H.86A.0099.2013.0125.1400 01/25/2013 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: Call Trace: > Apr 10 11:47:58 avi.cloudius-systems.com kernel: dump_stack+0x63/0x86 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_error_report+0x3c/0x40 > [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? > xfs_efi_recover+0x18e/0x1c0 [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_trans_cancel+0xb6/0xe0 > [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_efi_recover+0x18e/0x1c0 > [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: > xlog_recover_process_efi+0x2c/0x50 [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: > xlog_recover_process_intents.isra.42+0x122/0x160 [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? > xfs_reinit_percpu_counters+0x46/0x50 [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: > xlog_recover_finish+0x23/0xb0 [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: > xfs_log_mount_finish+0x29/0x50 [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_mountfs+0x6ce/0x930 > [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: > xfs_fs_fill_super+0x3ee/0x570 [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: mount_bdev+0x178/0x1b0 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? > xfs_test_remount_options.isra.14+0x60/0x60 [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: xfs_fs_mount+0x15/0x20 > [xfs] > Apr 10 11:47:58 avi.cloudius-systems.com kernel: mount_fs+0x38/0x150 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? __alloc_percpu+0x15/0x20 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: vfs_kern_mount+0x67/0x130 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: do_mount+0x1dd/0xc50 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? > _copy_from_user+0x4e/0x80 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: ? memdup_user+0x4f/0x70 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: SyS_mount+0x83/0xd0 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: do_syscall_64+0x67/0x180 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: > entry_SYSCALL64_slow_path+0x25/0x25 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: RIP: 0033:0x7f5cb9a626fa > Apr 10 11:47:58 avi.cloudius-systems.com kernel: RSP: 002b:00007ffeffa2c928 > EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: RAX: ffffffffffffffda RBX: > 000055b59fd6f030 RCX: 00007f5cb9a626fa > Apr 10 11:47:58 avi.cloudius-systems.com kernel: RDX: 000055b59fd6f210 RSI: > 000055b59fd6f250 RDI: 000055b59fd6f230 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: RBP: 0000000000000000 R08: > 0000000000000000 R09: 0000000000000012 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: R10: 00000000c0ed0000 R11: > 0000000000000246 R12: 000055b59fd6f230 > Apr 10 11:47:58 avi.cloudius-systems.com kernel: R13: 000055b59fd6f210 R14: > 0000000000000000 R15: 00000000ffffffff > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): > xfs_do_force_shutdown(0x8) called from line 984 of file fs/xfs/xfs_trans.c. > Return address = 0xffffffffc056324f > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Corruption > of in-memory data detected. Shutting down filesystem > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Please > umount the filesystem and rectify the problem(s) > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): Failed to > recover intents > Apr 10 11:47:58 avi.cloudius-systems.com kernel: XFS (nvme0n1): log mount > finish failed > > > > smart (note error at end; there were no kernel I/O errors from the block > layer): > > $ sudo smartctl -a /dev/nvme0n1 > smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.10.8-200.fc25.x86_64] (local > build) > Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org > > === START OF INFORMATION SECTION === > Model Number: INTEL SSDPEKKW512G7 > Serial Number: BTPY6313086D512F > Firmware Version: PSF100C > PCI Vendor/Subsystem ID: 0x8086 > IEEE OUI Identifier: 0x5cd2e4 > Controller ID: 1 > Number of Namespaces: 1 > Namespace 1 Size/Capacity: 512,110,190,592 [512 GB] > Namespace 1 Formatted LBA Size: 512 > Local Time is: Mon Apr 10 12:36:41 2017 IDT > Firmware Updates (0x12): 1 Slot, no Reset required > Optional Admin Commands (0x0006): Format Frmw_DL > Optional NVM Commands (0x001e): Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat > Maximum Data Transfer Size: 32 Pages > Warning Comp. Temp. Threshold: 70 Celsius > Critical Comp. Temp. Threshold: 80 Celsius > > Supported Power States > St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat > 0 + 9.00W - - 0 0 0 0 5 5 > 1 + 4.60W - - 1 1 1 1 30 30 > 2 + 3.80W - - 2 2 2 2 30 30 > 3 - 0.0700W - - 3 3 3 3 10000 300 > 4 - 0.0050W - - 4 4 4 4 2000 10000 > > Supported LBA Sizes (NSID 0x1) > Id Fmt Data Metadt Rel_Perf > 0 + 512 0 0 > > === START OF SMART DATA SECTION === > SMART overall-health self-assessment test result: PASSED > > SMART/Health Information (NVMe Log 0x02, NSID 0x1) > Critical Warning: 0x00 > Temperature: 27 Celsius > Available Spare: 100% > Available Spare Threshold: 10% > Percentage Used: 0% > Data Units Read: 8,854,487 [4.53 TB] > Data Units Written: 5,652,445 [2.89 TB] > Host Read Commands: 446,901,662 > Host Write Commands: 35,627,742 > Controller Busy Time: 633 > Power Cycles: 24 > Power On Hours: 987 > Unsafe Shutdowns: 16 > Media and Data Integrity Errors: 1 > Error Information Log Entries: 1 > Warning Comp. Temperature Time: 11 > Critical Comp. Temperature Time: 0 > > Error Information (NVMe Log 0x01, max 64 entries) > Num ErrCount SQId CmdId Status PELoc LBA NSID VS > 0 1 1 0x0000 0x0286 - 0 1 - > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-10 15:35 ` Brian Foster @ 2017-04-11 7:46 ` Avi Kivity 2017-04-11 11:30 ` Emmanuel Florac 0 siblings, 1 reply; 32+ messages in thread From: Avi Kivity @ 2017-04-11 7:46 UTC (permalink / raw) To: Brian Foster; +Cc: linux-xfs On 04/10/2017 06:35 PM, Brian Foster wrote: > On Mon, Apr 10, 2017 at 12:42:33PM +0300, Avi Kivity wrote: >> On 04/10/2017 12:23 PM, Avi Kivity wrote: >>> Today my kernel complained that in memory metadata is corrupt and >>> asked that I run xfs_repair. But xfs_repair doesn't like the >>> superblock and isn't able to find a secondary superblock. >>> >>> Latest Fedora 25 kernel, new Intel NVMe drive (worked for a few weeks >>> without issue). >>> >>> Anything I can do to recover the data? > Well I can't explain why you have a checksum error, but what do you mean > that xfs_repair doesn't like the superblock? Can you provide the > xfs_repair output? > > It seems strange for xfs_repair to not find the superblock of a > filesystem that can otherwise run log recovery up until it encounters > the buffer with a bad crc. Sorry, should have done it earlier. $ sudo xfs_repair /dev/nvme0n1 Phase 1 - find and verify superblock... couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!! attempting to find secondary superblock... ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C In a previous run, after returning from lunch, xfs_repair did not find the secondary superblock. The superblock is there though: $ sudo file -s /dev/nvme0n1 /dev/nvme0n1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs) I can provide it if it will help. > > It also might be useful to find out exactly what that error reported by > smartctl means. Are you aware of whether it pre-existed the filesystem > issue or not? I believe I ran it before and did not notice the error. I deal with many disks, though, so it could have been that I just didn't notice it, or that I ran it on a different machine. > >> Error Information (NVMe Log 0x01, max 64 entries) >> Num ErrCount SQId CmdId Status PELoc LBA NSID VS >> 0 1 1 0x0000 0x0286 - 0 1 - >> >> If CmdId is the opcode, then it's a flush (matches the fact that LBA=0), but I'm guessing it's the tag. 0x0286 is NVME_SC_ACCESS_DENIED, which doesn't appear to match, though (if I picked the right enum). ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 7:46 ` Avi Kivity @ 2017-04-11 11:30 ` Emmanuel Florac 2017-04-11 11:40 ` Avi Kivity 0 siblings, 1 reply; 32+ messages in thread From: Emmanuel Florac @ 2017-04-11 11:30 UTC (permalink / raw) To: Avi Kivity; +Cc: Brian Foster, linux-xfs [-- Attachment #1: Type: text/plain, Size: 1003 bytes --] Le Tue, 11 Apr 2017 10:46:07 +0300 Avi Kivity <avi@scylladb.com> écrivait: > $ sudo xfs_repair /dev/nvme0n1 > Phase 1 - find and verify superblock... > couldn't verify primary superblock - not enough secondary superblocks > with matching geometry !!! Which version of xfs_repair is this? Try to export the FS structure with xfs_metadump, something like xfs_metadump /dev/nvme0n1 /some/file.dmp And check the errors it reports, they may be informative. In the case where metadump works out fine, you should then try to have a look at the FS structure using the dump (to avoid wrecking it more that it already is): xfs_db -c "sb 0" /some/file.dmp -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ [-- Attachment #2: Signature digitale OpenPGP --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 11:30 ` Emmanuel Florac @ 2017-04-11 11:40 ` Avi Kivity 2017-04-11 12:00 ` Emmanuel Florac 0 siblings, 1 reply; 32+ messages in thread From: Avi Kivity @ 2017-04-11 11:40 UTC (permalink / raw) To: Emmanuel Florac; +Cc: Brian Foster, linux-xfs On 04/11/2017 02:30 PM, Emmanuel Florac wrote: > Le Tue, 11 Apr 2017 10:46:07 +0300 > Avi Kivity <avi@scylladb.com> écrivait: > >> $ sudo xfs_repair /dev/nvme0n1 >> Phase 1 - find and verify superblock... >> couldn't verify primary superblock - not enough secondary superblocks >> with matching geometry !!! > Which version of xfs_repair is this? xfsprogs-4.9.0-1.fc25.x86_64 > Try to export the FS structure with xfs_metadump, something like > > xfs_metadump /dev/nvme0n1 /some/file.dmp > > And check the errors it reports, they may be informative. bad magic number xfs_metadump: cannot read superblock for ag 1 bad magic number xfs_metadump: cannot read superblock for ag 2 Metadata CRC error detected at xfs_agfl block 0x1dcf0963/0x200 bad magic number xfs_metadump: cannot read superblock for ag 3 Metadata CRC error detected at xfs_agfl block 0x2cb68e13/0x200 xfs_metadump: Filesystem log is dirty; image will contain unobfuscated metadata in log. cache_purge: shake on cache 0x55accee162b0 left 3 nodes!? > In the case where metadump works out fine, you should then try to have a > look at the FS structure using the dump (to avoid wrecking it more that > it already is): > > xfs_db -c "sb 0" /some/file.dmp > $ sudo xfs_db -c "sb 0" /tmp/fs xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic number 0x5846534d) Use -F to force a read attempt. $ sudo xfs_db -F -c "sb 0" /tmp/fs xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic number 0x5846534d) xfs_db: V1 inodes unsupported. Please try an older xfsprogs. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 11:40 ` Avi Kivity @ 2017-04-11 12:00 ` Emmanuel Florac 2017-04-11 12:03 ` Avi Kivity 0 siblings, 1 reply; 32+ messages in thread From: Emmanuel Florac @ 2017-04-11 12:00 UTC (permalink / raw) To: Avi Kivity; +Cc: Brian Foster, linux-xfs [-- Attachment #1: Type: text/plain, Size: 977 bytes --] Le Tue, 11 Apr 2017 14:40:15 +0300 Avi Kivity <avi@scylladb.com> écrivait: > $ sudo xfs_db -c "sb 0" /tmp/fs > xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic > number 0x5846534d) > Use -F to force a read attempt. > $ sudo xfs_db -F -c "sb 0" /tmp/fs > xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic > number 0x5846534d) > xfs_db: V1 inodes unsupported. Please try an older xfsprogs. Oops, I forgot one important part, sorry, you must restore the meta_dump to a file first: xfs_mdrestore /tmp/fs /tmp/fsimage then run xfs_db on the /tmp/fsimage: xfs_db -c 'sb 0' -c 'p' /tmp/fsimage -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ [-- Attachment #2: Signature digitale OpenPGP --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 12:00 ` Emmanuel Florac @ 2017-04-11 12:03 ` Avi Kivity 2017-04-11 12:49 ` Emmanuel Florac 0 siblings, 1 reply; 32+ messages in thread From: Avi Kivity @ 2017-04-11 12:03 UTC (permalink / raw) To: Emmanuel Florac; +Cc: Brian Foster, linux-xfs On 04/11/2017 03:00 PM, Emmanuel Florac wrote: > Le Tue, 11 Apr 2017 14:40:15 +0300 > Avi Kivity <avi@scylladb.com> écrivait: > >> $ sudo xfs_db -c "sb 0" /tmp/fs >> xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic >> number 0x5846534d) >> Use -F to force a read attempt. >> $ sudo xfs_db -F -c "sb 0" /tmp/fs >> xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic >> number 0x5846534d) >> xfs_db: V1 inodes unsupported. Please try an older xfsprogs. > Oops, I forgot one important part, sorry, you must restore the > meta_dump to a file first: > > xfs_mdrestore /tmp/fs /tmp/fsimage > > then run xfs_db on the /tmp/fsimage: > > xfs_db -c 'sb 0' -c 'p' /tmp/fsimage > magicnum = 0x58465342 blocksize = 4096 dblocks = 125026902 rblocks = 0 rextents = 0 uuid = 50b25ad8-3eb9-4273-b7f2-d0a435b3a08f logstart = 67108869 rootino = 96 rbmino = 97 rsumino = 98 rextsize = 1 agblocks = 31256726 agcount = 4 rbmblocks = 0 logblocks = 61048 versionnum = 0xb4b5 sectsize = 512 inodesize = 512 inopblock = 8 fname = "\000\000\000\000\000\000\000\000\000\000\000\000" blocklog = 12 sectlog = 9 inodelog = 9 inopblog = 3 agblklog = 25 rextslog = 0 inprogress = 0 imax_pct = 25 icount = 1959744 ifree = 89 fdblocks = 91586587 frextents = 0 uquotino = null gquotino = null qflags = 0 flags = 0 shared_vn = 0 inoalignmt = 4 unit = 0 width = 0 dirblklog = 0 logsectlog = 0 logsectsize = 0 logsunit = 1 features2 = 0x18a bad_features2 = 0x18a features_compat = 0 features_ro_compat = 0x1 features_incompat = 0x1 features_log_incompat = 0 crc = 0x3ebf41de (correct) spino_align = 0 pquotino = null lsn = 0x70002828b meta_uuid = 00000000-0000-0000-0000-000000000000 ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 12:03 ` Avi Kivity @ 2017-04-11 12:49 ` Emmanuel Florac 2017-04-11 13:07 ` Avi Kivity 0 siblings, 1 reply; 32+ messages in thread From: Emmanuel Florac @ 2017-04-11 12:49 UTC (permalink / raw) To: Avi Kivity; +Cc: Brian Foster, linux-xfs [-- Attachment #1: Type: text/plain, Size: 2556 bytes --] Le Tue, 11 Apr 2017 15:03:12 +0300 Avi Kivity <avi@scylladb.com> écrivait: > On 04/11/2017 03:00 PM, Emmanuel Florac wrote: > > Le Tue, 11 Apr 2017 14:40:15 +0300 > > Avi Kivity <avi@scylladb.com> écrivait: > > > >> $ sudo xfs_db -c "sb 0" /tmp/fs > >> xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic > >> number 0x5846534d) > >> Use -F to force a read attempt. > >> $ sudo xfs_db -F -c "sb 0" /tmp/fs > >> xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic > >> number 0x5846534d) > >> xfs_db: V1 inodes unsupported. Please try an older xfsprogs. > > Oops, I forgot one important part, sorry, you must restore the > > meta_dump to a file first: > > > > xfs_mdrestore /tmp/fs /tmp/fsimage > > > > then run xfs_db on the /tmp/fsimage: > > > > xfs_db -c 'sb 0' -c 'p' /tmp/fsimage > > > > magicnum = 0x58465342 > blocksize = 4096 > dblocks = 125026902 > rblocks = 0 > rextents = 0 > uuid = 50b25ad8-3eb9-4273-b7f2-d0a435b3a08f > logstart = 67108869 > rootino = 96 > rbmino = 97 > rsumino = 98 > rextsize = 1 > agblocks = 31256726 > agcount = 4 > rbmblocks = 0 > logblocks = 61048 > versionnum = 0xb4b5 > sectsize = 512 > inodesize = 512 > inopblock = 8 > fname = "\000\000\000\000\000\000\000\000\000\000\000\000" > blocklog = 12 > sectlog = 9 > inodelog = 9 > inopblog = 3 > agblklog = 25 > rextslog = 0 > inprogress = 0 > imax_pct = 25 > icount = 1959744 > ifree = 89 > fdblocks = 91586587 > frextents = 0 > uquotino = null > gquotino = null > qflags = 0 > flags = 0 > shared_vn = 0 > inoalignmt = 4 > unit = 0 > width = 0 > dirblklog = 0 > logsectlog = 0 > logsectsize = 0 > logsunit = 1 > features2 = 0x18a > bad_features2 = 0x18a > features_compat = 0 > features_ro_compat = 0x1 > features_incompat = 0x1 > features_log_incompat = 0 > crc = 0x3ebf41de (correct) > spino_align = 0 > pquotino = null > lsn = 0x70002828b > meta_uuid = 00000000-0000-0000-0000-000000000000 > > Tha looks reasonable enough... Heck, what's happening? You could try to run an integrity check from xfs_db (still using the dump) to locate the error: xfs_db -c 'sb 0' -c 'check' /tmp/fsimage What does it report? -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ [-- Attachment #2: Signature digitale OpenPGP --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 12:49 ` Emmanuel Florac @ 2017-04-11 13:07 ` Avi Kivity 2017-04-11 16:13 ` Emmanuel Florac 0 siblings, 1 reply; 32+ messages in thread From: Avi Kivity @ 2017-04-11 13:07 UTC (permalink / raw) To: Emmanuel Florac; +Cc: Brian Foster, linux-xfs On 04/11/2017 03:49 PM, Emmanuel Florac wrote: > Le Tue, 11 Apr 2017 15:03:12 +0300 > Avi Kivity <avi@scylladb.com> écrivait: > >> On 04/11/2017 03:00 PM, Emmanuel Florac wrote: >>> Le Tue, 11 Apr 2017 14:40:15 +0300 >>> Avi Kivity <avi@scylladb.com> écrivait: >>> >>>> $ sudo xfs_db -c "sb 0" /tmp/fs >>>> xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic >>>> number 0x5846534d) >>>> Use -F to force a read attempt. >>>> $ sudo xfs_db -F -c "sb 0" /tmp/fs >>>> xfs_db: /tmp/fs is not a valid XFS filesystem (unexpected SB magic >>>> number 0x5846534d) >>>> xfs_db: V1 inodes unsupported. Please try an older xfsprogs. >>> Oops, I forgot one important part, sorry, you must restore the >>> meta_dump to a file first: >>> >>> xfs_mdrestore /tmp/fs /tmp/fsimage >>> >>> then run xfs_db on the /tmp/fsimage: >>> >>> xfs_db -c 'sb 0' -c 'p' /tmp/fsimage >>> >> magicnum = 0x58465342 >> blocksize = 4096 >> dblocks = 125026902 >> rblocks = 0 >> rextents = 0 >> uuid = 50b25ad8-3eb9-4273-b7f2-d0a435b3a08f >> logstart = 67108869 >> rootino = 96 >> rbmino = 97 >> rsumino = 98 >> rextsize = 1 >> agblocks = 31256726 >> agcount = 4 >> rbmblocks = 0 >> logblocks = 61048 >> versionnum = 0xb4b5 >> sectsize = 512 >> inodesize = 512 >> inopblock = 8 >> fname = "\000\000\000\000\000\000\000\000\000\000\000\000" >> blocklog = 12 >> sectlog = 9 >> inodelog = 9 >> inopblog = 3 >> agblklog = 25 >> rextslog = 0 >> inprogress = 0 >> imax_pct = 25 >> icount = 1959744 >> ifree = 89 >> fdblocks = 91586587 >> frextents = 0 >> uquotino = null >> gquotino = null >> qflags = 0 >> flags = 0 >> shared_vn = 0 >> inoalignmt = 4 >> unit = 0 >> width = 0 >> dirblklog = 0 >> logsectlog = 0 >> logsectsize = 0 >> logsunit = 1 >> features2 = 0x18a >> bad_features2 = 0x18a >> features_compat = 0 >> features_ro_compat = 0x1 >> features_incompat = 0x1 >> features_log_incompat = 0 >> crc = 0x3ebf41de (correct) >> spino_align = 0 >> pquotino = null >> lsn = 0x70002828b >> meta_uuid = 00000000-0000-0000-0000-000000000000 >> >> > Tha looks reasonable enough... Heck, what's happening? You could try > to run an integrity check from xfs_db (still using the dump) to locate > the error: > > xfs_db -c 'sb 0' -c 'check' /tmp/fsimage > > What does it report? > $ sudo xfs_db -c 'sb 0' -c 'check' /tmp/fsimage ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_db. If you are unable to mount the filesystem, then use the xfs_repair -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. xfs_repair did not recognize the superblock, and started hunting for the second one, emitting dots in the process. I stopped it, since it failed on the live disk. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 13:07 ` Avi Kivity @ 2017-04-11 16:13 ` Emmanuel Florac 2017-04-11 16:44 ` Avi Kivity 0 siblings, 1 reply; 32+ messages in thread From: Emmanuel Florac @ 2017-04-11 16:13 UTC (permalink / raw) To: Avi Kivity; +Cc: Brian Foster, linux-xfs [-- Attachment #1: Type: text/plain, Size: 1148 bytes --] Le Tue, 11 Apr 2017 16:07:56 +0300 Avi Kivity <avi@scylladb.com> écrivait: > $ sudo xfs_db -c 'sb 0' -c 'check' /tmp/fsimage > ERROR: The filesystem has valuable metadata changes in a log which > needs to be replayed. Mount the filesystem to replay the log, and > unmount it before re-running xfs_db. If you are unable to mount the > filesystem, then use the xfs_repair -L option to destroy the log and > attempt a repair. Note that destroying the log may cause corruption > -- please attempt a mount of the filesystem before doing this. > > > xfs_repair did not recognize the superblock, and started hunting for > the second one, emitting dots in the process. I stopped it, since it > failed on the live disk. > Can you mount the image, or does it fail immmediately because of the CRC error? -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ [-- Attachment #2: Signature digitale OpenPGP --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 16:13 ` Emmanuel Florac @ 2017-04-11 16:44 ` Avi Kivity 2017-04-11 16:48 ` Eric Sandeen 0 siblings, 1 reply; 32+ messages in thread From: Avi Kivity @ 2017-04-11 16:44 UTC (permalink / raw) To: Emmanuel Florac; +Cc: Brian Foster, linux-xfs On 04/11/2017 07:13 PM, Emmanuel Florac wrote: > Le Tue, 11 Apr 2017 16:07:56 +0300 > Avi Kivity <avi@scylladb.com> écrivait: > >> $ sudo xfs_db -c 'sb 0' -c 'check' /tmp/fsimage >> ERROR: The filesystem has valuable metadata changes in a log which >> needs to be replayed. Mount the filesystem to replay the log, and >> unmount it before re-running xfs_db. If you are unable to mount the >> filesystem, then use the xfs_repair -L option to destroy the log and >> attempt a repair. Note that destroying the log may cause corruption >> -- please attempt a mount of the filesystem before doing this. >> >> >> xfs_repair did not recognize the superblock, and started hunting for >> the second one, emitting dots in the process. I stopped it, since it >> failed on the live disk. >> > Can you mount the image, or does it fail immmediately because of the > CRC error? > Fails immediately. I'll probably format it with ext4, wait for the firmware update, and then reformat it with xfs. Since the firmware bug was acknowledged, I don't know what we can gain from it. My disk is mostly a git and imap mirror anyway, + a large ccache repository, + a throwaway database. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 16:44 ` Avi Kivity @ 2017-04-11 16:48 ` Eric Sandeen 2017-04-12 15:15 ` Christoph Hellwig 0 siblings, 1 reply; 32+ messages in thread From: Eric Sandeen @ 2017-04-11 16:48 UTC (permalink / raw) To: Avi Kivity, Emmanuel Florac; +Cc: Brian Foster, linux-xfs On 4/11/17 11:44 AM, Avi Kivity wrote: > > > On 04/11/2017 07:13 PM, Emmanuel Florac wrote: >> Le Tue, 11 Apr 2017 16:07:56 +0300 >> Avi Kivity <avi@scylladb.com> écrivait: >> >>> $ sudo xfs_db -c 'sb 0' -c 'check' /tmp/fsimage >>> ERROR: The filesystem has valuable metadata changes in a log which >>> needs to be replayed. Mount the filesystem to replay the log, and >>> unmount it before re-running xfs_db. If you are unable to mount the >>> filesystem, then use the xfs_repair -L option to destroy the log and >>> attempt a repair. Note that destroying the log may cause corruption >>> -- please attempt a mount of the filesystem before doing this. >>> >>> >>> xfs_repair did not recognize the superblock, and started hunting for >>> the second one, emitting dots in the process. I stopped it, since it >>> failed on the live disk. >>> >> Can you mount the image, or does it fail immmediately because of the >> CRC error? >> > > Fails immediately. > > I'll probably format it with ext4, wait for the firmware update, and > then reformat it with xfs. Since the firmware bug was acknowledged, I > don't know what we can gain from it. My disk is mostly a git and imap > mirror anyway, + a large ccache repository, + a throwaway database. Honestly, I'd be a little leary of ext4 too - I don't know what the underlying problem is, but it must be related to some IO pattern that is more common on xfs, but it's a leap to say that it's never present on any other fs... IOWS: a drive firmware bug that corrupts data probably can't be trusted with any filesystem. As an experiment, though, if you want to play, it might be interesting to mkfs.xfs it with a 4096 byte sector size, and see if that makes it happier. By default, xfs is doing metadata IO in 512 chunks, something other filesystems won't do by default. I guess you don't know how to provoke the corruption, though, to be able to run that test reliably... -Eric ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-11 16:48 ` Eric Sandeen @ 2017-04-12 15:15 ` Christoph Hellwig 2017-04-12 15:34 ` Eric Sandeen 0 siblings, 1 reply; 32+ messages in thread From: Christoph Hellwig @ 2017-04-12 15:15 UTC (permalink / raw) To: Eric Sandeen; +Cc: Avi Kivity, Emmanuel Florac, Brian Foster, linux-xfs On Tue, Apr 11, 2017 at 11:48:19AM -0500, Eric Sandeen wrote: > As an experiment, though, if you want to play, it might be interesting to > mkfs.xfs it with a 4096 byte sector size, and see if that makes it happier. > By default, xfs is doing metadata IO in 512 chunks, something other filesystems > won't do by default. > > I guess you don't know how to provoke the corruption, though, to be able > to run that test reliably... Btw, it might be a good idea to move to 4k sector size as the default, on pretty much any modern hardware sector sizes are 4k or larger internally, and 512 byte writes will always involve read-modify-write cycles. And unlike SATA or SCSI disks NVMe doesn't have a physical block size attribute, so we can't even look at that. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-12 15:15 ` Christoph Hellwig @ 2017-04-12 15:34 ` Eric Sandeen 2017-04-12 15:45 ` Christoph Hellwig 0 siblings, 1 reply; 32+ messages in thread From: Eric Sandeen @ 2017-04-12 15:34 UTC (permalink / raw) To: Christoph Hellwig; +Cc: Avi Kivity, Emmanuel Florac, Brian Foster, linux-xfs On 4/12/17 10:15 AM, Christoph Hellwig wrote: > On Tue, Apr 11, 2017 at 11:48:19AM -0500, Eric Sandeen wrote: >> As an experiment, though, if you want to play, it might be interesting to >> mkfs.xfs it with a 4096 byte sector size, and see if that makes it happier. >> By default, xfs is doing metadata IO in 512 chunks, something other filesystems >> won't do by default. >> >> I guess you don't know how to provoke the corruption, though, to be able >> to run that test reliably... > > Btw, it might be a good idea to move to 4k sector size as the default, > on pretty much any modern hardware sector sizes are 4k or larger > internally, and 512 byte writes will always involve read-modify-write > cycles. And unlike SATA or SCSI disks NVMe doesn't have a physical > block size attribute, so we can't even look at that. Is it safe to do that on a device that /actually/ has only 512 sectors? I /think/ Brian's tear detection helps, but </handwave> is it legit to do metadata IO larger than the fundamental IO size of the storage? -Eric ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-12 15:34 ` Eric Sandeen @ 2017-04-12 15:45 ` Christoph Hellwig 2017-04-12 16:15 ` Avi Kivity 0 siblings, 1 reply; 32+ messages in thread From: Christoph Hellwig @ 2017-04-12 15:45 UTC (permalink / raw) To: Eric Sandeen Cc: Christoph Hellwig, Avi Kivity, Emmanuel Florac, Brian Foster, linux-xfs On Wed, Apr 12, 2017 at 10:34:47AM -0500, Eric Sandeen wrote: > Is it safe to do that on a device that /actually/ has only 512 sectors? Except for NVMe none of the storage standards actually guarantees sector atomicy, although the whole storage stack traditionally relies on it.. Maybe we should claim a 4k physical block size for NVMe devices that hab 512 byte LBAs and a "Atomic Write Unit Power Fail" setting of at least 8 so that the mkfs sector size logic triggers.. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-12 15:45 ` Christoph Hellwig @ 2017-04-12 16:15 ` Avi Kivity 2017-04-12 16:20 ` Christoph Hellwig 0 siblings, 1 reply; 32+ messages in thread From: Avi Kivity @ 2017-04-12 16:15 UTC (permalink / raw) To: Christoph Hellwig, Eric Sandeen; +Cc: Emmanuel Florac, Brian Foster, linux-xfs On 04/12/2017 06:45 PM, Christoph Hellwig wrote: > On Wed, Apr 12, 2017 at 10:34:47AM -0500, Eric Sandeen wrote: >> Is it safe to do that on a device that /actually/ has only 512 sectors? > Except for NVMe none of the storage standards actually guarantees > sector atomicy, although the whole storage stack traditionally relies on > it.. > > Maybe we should claim a 4k physical block size for NVMe devices that > hab 512 byte LBAs and a "Atomic Write Unit Power Fail" setting of at > least 8 so that the mkfs sector size logic triggers.. This preserves the ability to do O_DIRECT reads at 512 byte granularity, yes? We make use of that (it's probably less important on NVMe; still why waste bandwidth needlessly). ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-12 16:15 ` Avi Kivity @ 2017-04-12 16:20 ` Christoph Hellwig 2017-04-12 16:22 ` Eric Sandeen 2017-04-12 16:22 ` Avi Kivity 0 siblings, 2 replies; 32+ messages in thread From: Christoph Hellwig @ 2017-04-12 16:20 UTC (permalink / raw) To: Avi Kivity Cc: Christoph Hellwig, Eric Sandeen, Emmanuel Florac, Brian Foster, linux-xfs On Wed, Apr 12, 2017 at 07:15:40PM +0300, Avi Kivity wrote: > This preserves the ability to do O_DIRECT reads at 512 byte granularity, > yes? No. > We make use of that (it's probably less important on NVMe; still why > waste bandwidth needlessly). In that case it would be the wrong thing for you. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-12 16:20 ` Christoph Hellwig @ 2017-04-12 16:22 ` Eric Sandeen 2017-04-12 16:24 ` Avi Kivity 2017-04-12 16:22 ` Avi Kivity 1 sibling, 1 reply; 32+ messages in thread From: Eric Sandeen @ 2017-04-12 16:22 UTC (permalink / raw) To: Christoph Hellwig, Avi Kivity; +Cc: Emmanuel Florac, Brian Foster, linux-xfs On 4/12/17 11:20 AM, Christoph Hellwig wrote: > On Wed, Apr 12, 2017 at 07:15:40PM +0300, Avi Kivity wrote: >> This preserves the ability to do O_DIRECT reads at 512 byte granularity, >> yes? > > No. > >> We make use of that (it's probably less important on NVMe; still why >> waste bandwidth needlessly). > > In that case it would be the wrong thing for you. And it would be really interesting to see if the 512-byte DIOs you issue under ext4 might trigger the same problem in the firmware. (This is all pure speculation, but that's all we've got) ;) -Eric ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-12 16:22 ` Eric Sandeen @ 2017-04-12 16:24 ` Avi Kivity 0 siblings, 0 replies; 32+ messages in thread From: Avi Kivity @ 2017-04-12 16:24 UTC (permalink / raw) To: Eric Sandeen, Christoph Hellwig; +Cc: Emmanuel Florac, Brian Foster, linux-xfs On 04/12/2017 07:22 PM, Eric Sandeen wrote: > On 4/12/17 11:20 AM, Christoph Hellwig wrote: >> On Wed, Apr 12, 2017 at 07:15:40PM +0300, Avi Kivity wrote: >>> This preserves the ability to do O_DIRECT reads at 512 byte granularity, >>> yes? >> No. >> >>> We make use of that (it's probably less important on NVMe; still why >>> waste bandwidth needlessly). >> In that case it would be the wrong thing for you. > And it would be really interesting to see if the 512-byte DIOs you > issue under ext4 might trigger the same problem in the firmware. We only issue 512-byte reads, writes are always 4096-byte aligned (and usually 128k). The disk that crashed was my /home; it did see some database loads, but not much. > > (This is all pure speculation, but that's all we've got) ;) > > -Eric ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-12 16:20 ` Christoph Hellwig 2017-04-12 16:22 ` Eric Sandeen @ 2017-04-12 16:22 ` Avi Kivity 2017-04-12 17:41 ` Christoph Hellwig 1 sibling, 1 reply; 32+ messages in thread From: Avi Kivity @ 2017-04-12 16:22 UTC (permalink / raw) To: Christoph Hellwig; +Cc: Eric Sandeen, Emmanuel Florac, Brian Foster, linux-xfs On 04/12/2017 07:20 PM, Christoph Hellwig wrote: > On Wed, Apr 12, 2017 at 07:15:40PM +0300, Avi Kivity wrote: >> This preserves the ability to do O_DIRECT reads at 512 byte granularity, >> yes? > No. :( > >> We make use of that (it's probably less important on NVMe; still why >> waste bandwidth needlessly). > In that case it would be the wrong thing for you. Would it be under our control? ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-12 16:22 ` Avi Kivity @ 2017-04-12 17:41 ` Christoph Hellwig 0 siblings, 0 replies; 32+ messages in thread From: Christoph Hellwig @ 2017-04-12 17:41 UTC (permalink / raw) To: Avi Kivity Cc: Christoph Hellwig, Eric Sandeen, Emmanuel Florac, Brian Foster, linux-xfs On Wed, Apr 12, 2017 at 07:22:20PM +0300, Avi Kivity wrote: > > > We make use of that (it's probably less important on NVMe; still why > > > waste bandwidth needlessly). > > In that case it would be the wrong thing for you. > > Would it be under our control? Yes, even if we switch a mkfs default you could still manually override it as long as the device supports a smaller logical block size, similar to how we treat 512 logical / 4k physical SAS and SATA drives today. ^ permalink raw reply [flat|nested] 32+ messages in thread
* allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) 2017-04-10 9:23 filesystem dead, xfs_repair won't help Avi Kivity 2017-04-10 9:42 ` Avi Kivity @ 2017-04-10 9:43 ` L A Walsh 2017-04-10 16:01 ` Eric Sandeen 2017-04-10 15:49 ` filesystem dead, xfs_repair won't help Eric Sandeen 2 siblings, 1 reply; 32+ messages in thread From: L A Walsh @ 2017-04-10 9:43 UTC (permalink / raw) To: linux-xfs Avi Kivity wrote: > Today my kernel complained that in memory metadata is corrupt and > asked that I run xfs_repair. But xfs_repair doesn't like the > superblock and isn't able to find a secondary superblock. > Why doesn't xfs have an option to mount with metadata checksumming disabled so people can recover their data? Seems like it should be easy to provide, no? Or rather, if a disk is created with the crc option, is it possible to later switch it off or mount it without with checking disabled? Yes, I know the mantra is that they should have had backups, but in practice it's seems not the case in a majority of uses outside of enterprise usage. It sure seems that disabling a particular file or directory (if necessary) affected by a bad-crc, would be preferable to losing the whole disk. That said, how many crc errors would be likely to make things unreadable or inaccessible? Given that the default before crc-checking was that the disks were still usable (often with no error being flagged or noticed), I'd suspect that the crc-checking is causing many errors to be be flagged that before wouldn't have even been noticed. Overall I'm wondering if the crc option won't cause more disk-losses than would occur without the option. Or, in other words, it seems that since crc-checking seems to cause the disk to be lost, turning on crc checking is almost guaranteed to cause a higher incidence of data loss if it can't be disable. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) 2017-04-10 9:43 ` allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) L A Walsh @ 2017-04-10 16:01 ` Eric Sandeen 2017-04-10 18:05 ` L A Walsh 0 siblings, 1 reply; 32+ messages in thread From: Eric Sandeen @ 2017-04-10 16:01 UTC (permalink / raw) To: L A Walsh, linux-xfs On 4/10/17 4:43 AM, L A Walsh wrote: > Avi Kivity wrote: >> Today my kernel complained that in memory metadata is corrupt and >> asked that I run xfs_repair. But xfs_repair doesn't like the >> superblock and isn't able to find a secondary superblock. >> > Why doesn't xfs have an option to mount with metadata checksumming > disabled so people can recover their data? Because if checksums are bad, your metadata is almost certainly bad, and with bad metadata, you're not going to be recovering data either. (and FWIW, CRCs are only the first line of defense: structure verifiers come after that. The chance of a CRC being bad and everything else checking out is extremely small.) > Seems like it should be easy to provide, no? > > Or rather, if a disk is created with the crc option, is it possible > to later switch it off or mount it without with checking disabled? It is not possible. > Yes, I know the mantra is that they should have had backups, but > in practice it's seems not the case in a majority of uses outside > of enterprise usage. It sure seems that disabling a particular file > or directory (if necessary) affected by a bad-crc, would be > preferable to losing the whole disk. That said, how many crc > errors would be likely to make things unreadable or inaccessible? How log is a piece of string? ;) Totally depends on the details. > Given that the default before crc-checking was that the disks > were still usable (often with no error being flagged or noticed), Before, we had a lot of ad-hoc checks (or not.) Many of those checks, and/or IO errors when trying to read garbage metadata, would also shut down the filesystem. Proceeding with mutilated metadata is almost never a good thing. You'll wander off into garbage and shut down the fs at best, and OOPS at worst. (Losing a filesystem is preferable to losing a system!) > I'd suspect that the crc-checking is causing many errors to be > be flagged that before wouldn't have even been noticed. Yes, that's precisely the point of CRCs. :) > Overall I'm wondering if the crc option won't cause more disk-losses > than would occur without the option. Or, in other words, it seems > that since crc-checking seems to cause the disk to be lost, turning > on crc checking is almost guaranteed to cause a higher incidence of > data loss if it can't be disable. When CRCs detect metadata corruption, the next step is to run xfs_repair to salvage what can be salvaged, and retrieve what's left of your data after that. Disabling CRCs and proceeding in kernelspace with known metadata corruption would be a dangerous total crapshoot. -Eric ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) 2017-04-10 16:01 ` Eric Sandeen @ 2017-04-10 18:05 ` L A Walsh 2017-04-11 12:57 ` Emmanuel Florac 0 siblings, 1 reply; 32+ messages in thread From: L A Walsh @ 2017-04-10 18:05 UTC (permalink / raw) To: Eric Sandeen; +Cc: linux-xfs Eric Sandeen wrote: > On 4/10/17 4:43 AM, L A Walsh wrote: >> Avi Kivity wrote: >>> Today my kernel complained that in memory metadata is corrupt and >>> asked that I run xfs_repair. But xfs_repair doesn't like the >>> superblock and isn't able to find a secondary superblock. >>> >> Why doesn't xfs have an option to mount with metadata checksumming >> disabled so people can recover their data? > > Because if checksums are bad, your metadata is almost certainly bad, > and with bad metadata, you're not going to be recovering data either. ---- Sorry, but I really don't by that a 1 bit error in metadata will automatically cause problems with data recovery. If the date on the file shows it was created at some time with nanoseconds=1 and that gets bumped to 3 (or virtually any number < an equivalent of 1 second) it will trigger a crc error. But I don't care. > > (and FWIW, CRCs are only the first line of defense: structure > verifiers come after that. The chance of a CRC being bad and > everything else checking out is extremely small.) ---- If the crc error has caught a bit rot, that wouldn't be true. Only if the crc error catches a bug in the XFS SW, would that be likely. Since I was told that it was protecting me against bit-rot and not a lower stability or quality of XFS overall, then it's more likely that data could be recovered. Though, again, this is one of those things like allowing use of the free-space extent that you could *allow* users to use at their own risk -- but something, likely, that you won't. This is another case where your logic is flawed. Permitting mounting w/o enforcement is not a guarantee of data recovery, BUT allowing the user to make the decision of whether or not they can recover anything useful should be up to the owner of the computer. Yet it seems clear you aren't using sound engineering practice to justify your actions. Any bit rot metadata corruption is unlikely to wipe 10 TeraBytes of data. Understand your position. You are claiming the crc option is detecting errors that were previously undetected. People have operated huge filesystems (I'm certain that my 10TB partition is tiny compared to enterprise usage) for years without experiencing noticeable problems. Yet when crc is turned on, suddenly they are expected to buy into crc detecting corruption so severe that nothing can be recovered (when such has not been the case since XFS's inception). >> Seems like it should be easy to provide, no? >> >> Or rather, if a disk is created with the crc option, is it possible >> to later switch it off or mount it without with checking disabled? > > It is not possible. ----- Not possible eh? In the SW world? The only way it would not be possible is if it were *administratively prohibited*. Working around detected bugs or flaws isn't known to be "not possible" by a long shot. Take ZFS, which , I'm told, can not only recover corrupted data from other sectors, but doesn't require shutting down the file system due to the problem detection. That certainly doesn't sound like "impossible". If the crc option is only a canary, and not a cipher then recovery of most data should be possible. Are you saying that the crc option doesn't simply do an integrity check but is converting what was "plaintext" into some encoded form? That isn't what it is documented to do. >> Yes, I know the mantra is that they should have had backups, but >> in practice it's seems not the case in a majority of uses outside >> of enterprise usage. It sure seems that disabling a particular file >> or directory (if necessary) affected by a bad-crc, would be >> preferable to losing the whole disk. That said, how many crc >> errors would be likely to make things unreadable or inaccessible? > > How log is a piece of string? ;) Totally depends on the details. -- That depends on whether or not it is a software error caused by a typo or by 1 or more bit-flips in a given sector. > >> Given that the default before crc-checking was that the disks >> were still usable (often with no error being flagged or noticed), > > Before, we had a lot of ad-hoc checks (or not.) Many of those checks, > and/or IO errors when trying to read garbage metadata, would also > shut down the filesystem. --- But those checks were rarely triggered. It was often the case (you claim) that they went undiscovered for some time -- thus a "need"[sic] for crc to detect a 1 bit-rot-flip in a 100TB file system and mark the entire file system as bad. Sorry, that's bull. You need to compartmentalize damage or its worthless. Noticing a error in 1 sector shouldn't shutdown or prevent 100TB of other daya from being accessed (or usable). > Proceeding with mutilated metadata is almost never a good thing. > You'll wander off into garbage and shut down the fs at best, and OOPS > at worst. (Losing a filesystem is preferable to losing a system!) ---- > >> I'd suspect that the crc-checking is causing many errors to be >> be flagged that before wouldn't have even been noticed. > > Yes, that's precisely the point of CRCs. :) ---- If they wouldn't have been noticed -- then they wouldn't have caused problems. crc is creating problems where before they didn't -- by definition -- because they catch "many errors... that before, WOULDN'T HAVE BEEN NOTICED". That's my point. >> Overall I'm wondering if the crc option won't cause more >> disk-losses than would occur without the option. Or, in other >> words, it seems that since crc-checking seems to cause the disk >> to be lost, turning on crc checking is almost guaranteed to cause >> a higher incidence of data loss if it can't be disable. > > When CRCs detect metadata corruption, the next step is to run > xfs_repair to salvage what can be salvaged, and retrieve what's > left of your data after that. Disabling CRCs and proceeding in > kernelspace with known metadata corruption would be a dangerous > total crapshoot. --- Right..xfsrepair -- like the base-note poster tried and and had fail. The crc errors I'm seen complaints about are ones were xfsrepair don't work. At that point, disabling the volume is not helpful. I'm sure it wouldn't be trivial, but creating a separate file system, "XFS2" from the original XFS sources that responded to data or metadata corruption by returning empty data where it was impossible to return anything useful instead of flagging the disk as "bad", would be a way to allow data recovery to the extent that it made sense (assuming the original sources couldn't do the same toggling off a config-flag). I'm sure you can out-type and come up with various reasons as to why XFS or crc can't auto-correct. Maybe instead of a crc, you should be using a well established check that allows recovery from multiple data bit failure. Supposedly the 4K block size had more error-resistance and *recover* than the 512-byte format. Certainly, with crc's on all the metadata, a more robust algorithm could automatically recover from such errors. If it is that fragile, then perhaps you should consider enabling the independant use of the free-inode, which would certainly raise performance on mature filesystems. I did get that it's been tested on virgin and fresh file systems and showed no benefit with such, but it would be nice if such tests were done on 7-10+ year-old filesystems that "often" exceeded 75% disk space usage -- even going over 80-90% usage at times for a short period. It may not be a normal state, but it does happen. Certainly it would be something worthy of testing with real-life data. :) *cheers* -linda ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) 2017-04-10 18:05 ` L A Walsh @ 2017-04-11 12:57 ` Emmanuel Florac 2017-04-11 13:34 ` Eric Sandeen 0 siblings, 1 reply; 32+ messages in thread From: Emmanuel Florac @ 2017-04-11 12:57 UTC (permalink / raw) To: L A Walsh; +Cc: Eric Sandeen, linux-xfs [-- Attachment #1: Type: text/plain, Size: 1278 bytes --] Le Mon, 10 Apr 2017 11:05:38 -0700 L A Walsh <xfs@tlinx.org> écrivait: > I'm sure it wouldn't be trivial, but creating a separate > file system, "XFS2" from the original XFS sources that responded > to data or metadata corruption by returning empty data where > it was impossible to return anything useful instead of flagging > the disk as "bad", would be a way to allow data recovery to > the extent that it made sense (assuming the original sources > couldn't do the same toggling off a config-flag). It would probably much easier to add an option to mount the filesystem without crc, similar to "norecovery", that doesn't replay the journal. It would be of course read-only, but in a similar case it would be much easier and practical for everyone. So far I believed that metadata CRCs were a promise of safer filesystems; now that I've setup several multi-hundred terabytes volumes with CRC enabled, I'm getting nervous... -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ [-- Attachment #2: Signature digitale OpenPGP --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) 2017-04-11 12:57 ` Emmanuel Florac @ 2017-04-11 13:34 ` Eric Sandeen 2017-04-11 16:18 ` Emmanuel Florac 0 siblings, 1 reply; 32+ messages in thread From: Eric Sandeen @ 2017-04-11 13:34 UTC (permalink / raw) To: Emmanuel Florac, L A Walsh; +Cc: linux-xfs [-- Attachment #1.1: Type: text/plain, Size: 2366 bytes --] On 4/11/17 7:57 AM, Emmanuel Florac wrote: > Le Mon, 10 Apr 2017 11:05:38 -0700 > L A Walsh <xfs@tlinx.org> écrivait: > >> I'm sure it wouldn't be trivial, but creating a separate >> file system, "XFS2" from the original XFS sources that responded >> to data or metadata corruption by returning empty data where >> it was impossible to return anything useful instead of flagging >> the disk as "bad", would be a way to allow data recovery to >> the extent that it made sense (assuming the original sources >> couldn't do the same toggling off a config-flag). > > It would probably much easier to add an option to mount the filesystem > without crc, similar to "norecovery", that doesn't replay the journal. > It would be of course read-only, but in a similar case it would be much > easier and practical for everyone. Yes, I actually whipped up a patch to do just that, because I was curious. Although I don't think it would fly, I may send it just to have a record out on the list. > So far I believed that metadata CRCs were a promise of safer > filesystems; now that I've setup several multi-hundred terabytes > volumes with CRC enabled, I'm getting nervous... Why? So far there's a lot of fear & speculation from some quarters, but no reports of any actual real-world significant downside to CRC integrity checking. A few amendments to my possibly too-quick reply yesterday, though... One, not every CRC error will shut down your filesystem - far from it. As a quick test of Linda's first scenario, you can corrupt a timestamp without changing the CRC, using xfs_db's expert mode. That inode will be inaccessible until it's fixed with xfs_repair, but the filesystem continues on happily. Two, after talking with Darrick I realized that I misrepresented things a bit; we checksum the entire sector of metadata, so yes, even a bitflip in an unused portion of that location could cause a crc mismatch and therefore a metadata read error. But again, this would render that data structure inaccessible until repair, but it would not take the entire filesystem offline. Three, none of this has anything to do with the email that started this thread. Bad firmware turned Avi's SSD into a vat of goo, and CRCs are not in any way related to his inability to recover his filesystem. Thanks, -Eric [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 867 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) 2017-04-11 13:34 ` Eric Sandeen @ 2017-04-11 16:18 ` Emmanuel Florac 2017-04-11 16:34 ` Eric Sandeen 0 siblings, 1 reply; 32+ messages in thread From: Emmanuel Florac @ 2017-04-11 16:18 UTC (permalink / raw) To: Eric Sandeen; +Cc: L A Walsh, linux-xfs [-- Attachment #1: Type: text/plain, Size: 813 bytes --] Le Tue, 11 Apr 2017 08:34:44 -0500 Eric Sandeen <sandeen@sandeen.net> écrivait: > Three, none of this has anything to do with the email that started > this thread. Bad firmware turned Avi's SSD into a vat of goo, and > CRCs are not in any way related to his inability to recover his > filesystem. OK, but xfs_db finds and reads the sb OK, and it looks fine at first look; why does xfs_repair fail completely? I'm not actually certain that Avi's SSD is a "vat of goo"... -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ [-- Attachment #2: Signature digitale OpenPGP --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) 2017-04-11 16:18 ` Emmanuel Florac @ 2017-04-11 16:34 ` Eric Sandeen 0 siblings, 0 replies; 32+ messages in thread From: Eric Sandeen @ 2017-04-11 16:34 UTC (permalink / raw) To: Emmanuel Florac; +Cc: L A Walsh, linux-xfs [-- Attachment #1.1: Type: text/plain, Size: 2410 bytes --] On 4/11/17 11:18 AM, Emmanuel Florac wrote: > Le Tue, 11 Apr 2017 08:34:44 -0500 > Eric Sandeen <sandeen@sandeen.net> écrivait: > >> Three, none of this has anything to do with the email that started >> this thread. Bad firmware turned Avi's SSD into a vat of goo, and >> CRCs are not in any way related to his inability to recover his >> filesystem. > > OK, but xfs_db finds and reads the sb OK, and it looks fine at first > look; why does xfs_repair fail completely? I'm not actually certain > that Avi's SSD is a "vat of goo"... Well, the AGF printed out by the kernel on the mount attempt was nowhere close to a valid AGF structure. Ok, "vat of goo" may have been too strong, but there is at least one core filesystem structure which are completely scrambled. That's just the one that was obvious from the mount attempt; I was assuming there were likely more areas of extreme damage, but that was an assumption on my part. Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Metadata CRC error detected at xfs_agfl_read_verify+0xcd/0x100 [xfs], xfs_agfl block 0x2cb68e13 Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): Unmount and run xfs_repair Apr 10 11:41:20 avi.cloudius-systems.com kernel: XFS (nvme0n1): First 64 bytes of corrupted metadata buffer: Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75400: 23 40 8f 28 5b 50 3a b4 f8 54 1e 31 97 f4 fe ed #@.([P:..T.1.... Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75410: 62 87 57 51 ee 9d 31 02 ec 2c 10 46 6c 93 db 09 b.WQ..1..,.Fl... Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75420: ae 7a ea b3 91 49 7e d3 99 a4 25 49 11 c5 8b be .z...I~...%I.... Apr 10 11:41:20 avi.cloudius-systems.com kernel: ffff9004a5b75430: e4 2e 14 d4 8a f8 5f 98 66 d8 67 72 ec c9 1a d5 ......_.f.gr.... Again, don't fixate on the "CRC" error. The above is /not/ an AGFL for this filesystem. typedef struct xfs_agfl { __be32 agfl_magicnum; __be32 agfl_seqno; uuid_t agfl_uuid; __be64 agfl_lsn; __be32 agfl_crc; __be32 agfl_bno[]; /* actually XFS_AGFL_SIZE(mp) */ } __attribute__((packed)) xfs_agfl_t; The magicnum is wrong. The seqno is invalid. The UUID data in agfl_uuid does not match this filesystem. etc... -Eric [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 867 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-10 9:23 filesystem dead, xfs_repair won't help Avi Kivity 2017-04-10 9:42 ` Avi Kivity 2017-04-10 9:43 ` allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) L A Walsh @ 2017-04-10 15:49 ` Eric Sandeen 2017-04-10 16:23 ` Christoph Hellwig 2017-04-11 7:48 ` Avi Kivity 2 siblings, 2 replies; 32+ messages in thread From: Eric Sandeen @ 2017-04-10 15:49 UTC (permalink / raw) To: Avi Kivity, linux-xfs There is a known firmware bug on Intel 600P drives which causes corruption with XFS, FWIW. https://bugzilla.redhat.com/show_bug.cgi?id=1402533 On 4/10/17 4:23 AM, Avi Kivity wrote: > Today my kernel complained that in memory metadata is corrupt and > asked that I run xfs_repair. But xfs_repair doesn't like the > superblock and isn't able to find a secondary superblock. > > Latest Fedora 25 kernel, new Intel NVMe drive (worked for a few weeks > without issue). > > Anything I can do to recover the data? > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-10 15:49 ` filesystem dead, xfs_repair won't help Eric Sandeen @ 2017-04-10 16:23 ` Christoph Hellwig 2017-04-11 7:48 ` Avi Kivity 1 sibling, 0 replies; 32+ messages in thread From: Christoph Hellwig @ 2017-04-10 16:23 UTC (permalink / raw) To: Eric Sandeen; +Cc: Avi Kivity, linux-xfs On Mon, Apr 10, 2017 at 10:49:55AM -0500, Eric Sandeen wrote: > There is a known firmware bug on Intel 600P drives which > causes corruption with XFS, FWIW. Which apparently affects the whole range of controllers, e.g. also the Pro 6000p and E6000p at least, no idea if there are any more. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: filesystem dead, xfs_repair won't help 2017-04-10 15:49 ` filesystem dead, xfs_repair won't help Eric Sandeen 2017-04-10 16:23 ` Christoph Hellwig @ 2017-04-11 7:48 ` Avi Kivity 1 sibling, 0 replies; 32+ messages in thread From: Avi Kivity @ 2017-04-11 7:48 UTC (permalink / raw) To: Eric Sandeen, linux-xfs That seems to be it. On 04/10/2017 06:49 PM, Eric Sandeen wrote: > There is a known firmware bug on Intel 600P drives which > causes corruption with XFS, FWIW. > > https://bugzilla.redhat.com/show_bug.cgi?id=1402533 > > On 4/10/17 4:23 AM, Avi Kivity wrote: >> Today my kernel complained that in memory metadata is corrupt and >> asked that I run xfs_repair. But xfs_repair doesn't like the >> superblock and isn't able to find a secondary superblock. >> >> Latest Fedora 25 kernel, new Intel NVMe drive (worked for a few weeks >> without issue). >> >> Anything I can do to recover the data? >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> ^ permalink raw reply [flat|nested] 32+ messages in thread
end of thread, other threads:[~2017-04-12 17:41 UTC | newest] Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-04-10 9:23 filesystem dead, xfs_repair won't help Avi Kivity 2017-04-10 9:42 ` Avi Kivity 2017-04-10 15:35 ` Brian Foster 2017-04-11 7:46 ` Avi Kivity 2017-04-11 11:30 ` Emmanuel Florac 2017-04-11 11:40 ` Avi Kivity 2017-04-11 12:00 ` Emmanuel Florac 2017-04-11 12:03 ` Avi Kivity 2017-04-11 12:49 ` Emmanuel Florac 2017-04-11 13:07 ` Avi Kivity 2017-04-11 16:13 ` Emmanuel Florac 2017-04-11 16:44 ` Avi Kivity 2017-04-11 16:48 ` Eric Sandeen 2017-04-12 15:15 ` Christoph Hellwig 2017-04-12 15:34 ` Eric Sandeen 2017-04-12 15:45 ` Christoph Hellwig 2017-04-12 16:15 ` Avi Kivity 2017-04-12 16:20 ` Christoph Hellwig 2017-04-12 16:22 ` Eric Sandeen 2017-04-12 16:24 ` Avi Kivity 2017-04-12 16:22 ` Avi Kivity 2017-04-12 17:41 ` Christoph Hellwig 2017-04-10 9:43 ` allow mounting w/crc-checking disabled? (was Re: filesystem dead, xfs_repair won't help) L A Walsh 2017-04-10 16:01 ` Eric Sandeen 2017-04-10 18:05 ` L A Walsh 2017-04-11 12:57 ` Emmanuel Florac 2017-04-11 13:34 ` Eric Sandeen 2017-04-11 16:18 ` Emmanuel Florac 2017-04-11 16:34 ` Eric Sandeen 2017-04-10 15:49 ` filesystem dead, xfs_repair won't help Eric Sandeen 2017-04-10 16:23 ` Christoph Hellwig 2017-04-11 7:48 ` Avi Kivity
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.