From: John Jore <john@jore.no>
To: "linux-xfs@vger.kernel.org" <linux-xfs@vger.kernel.org>
Subject: Bug in xfs_repair 5..4.0 / Unable to repair metadata corruption
Date: Sun, 9 Feb 2020 06:19:34 +0000 [thread overview]
Message-ID: <b2babb761ed24dc986abc3073c5c47fc@jore.no> (raw)
In-Reply-To: <186d30f217e645728ad1f34724cbe3e7@jore.no>
[-- Attachment #1: Type: text/plain, Size: 2590 bytes --]
Hi all,
Not sure if this is the appropriate forum to reports xfs_repair bugs? If wrong, please point me in the appropriate direction?
I have a corrupted XFS volume which mounts fine, but xfs_repair is unable to repair it and volume eventually shuts down due to metadata corruption if writes are performed.
Originally I used xfs_repair from CentOS 8.1.1911, but cloned latest xfs_repair from git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git (Today, Feb 9th, reports as version 5.4.0)
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- 16:08:04: scanning agi unlinked lists - 64 of 64 allocation groups done
- process known inodes and perform inode discovery...
- agno = 45
- agno = 15
- agno = 0
- agno = 30
- agno = 60
- agno = 46
- agno = 16
Metadata corruption detected at 0x4330e3, xfs_inode block 0x17312a3f0/0x2000
- agno = 61
- agno = 31
- agno = 47
- agno = 62
- agno = 48
- agno = 49
- agno = 32
- agno = 33
- agno = 17
- agno = 1
bad magic number 0x0 on inode 18253615584
bad version number 0x0 on inode 18253615584
bad magic number 0x0 on inode 18253615585
bad version number 0x0 on inode 18253615585
bad magic number 0x0 on inode 18253615586
.....
bad magic number 0x0 on inode 18253615584, resetting magic number
bad version number 0x0 on inode 18253615584, resetting version number
bad magic number 0x0 on inode 18253615585, resetting magic number
bad version number 0x0 on inode 18253615585, resetting version number
bad magic number 0x0 on inode 18253615586, resetting magic number
bad version number 0x0 on inode 18253615586, resetting version number
....
- agno = 16
- agno = 17
Metadata corruption detected at 0x4330e3, xfs_inode block 0x17312a3f0/0x2000
- agno = 18
- agno = 19
...
Phase 7 - verify and correct link counts...
- 16:10:41: verify and correct link counts - 64 of 64 allocation groups done
Metadata corruption detected at 0x433385, xfs_inode block 0x17312a3f0/0x2000
libxfs_writebufr: write verifier failed on xfs_inode bno 0x17312a3f0/0x2000
releasing dirty buffer (bulk) to free list!
Does not matter how many times, I've lost count, I re-run xfs_repair, with, or without -d, it never does repair the volume.
Volume is a ~12GB LV build using 4x 4TB disks in RAID 5 using a 3Ware 9690SA controller.
Any suggestions or additional data I can provide?
John
[-- Attachment #2: output.log --]
[-- Type: application/octet-stream, Size: 14757 bytes --]
Phase 1 - find and verify superblock...
- reporting progress in intervals of 15 minutes
- block cache size set to 556792 entries
Phase 2 - using internal log
- zero log...
zero_log: head block 213781 tail block 213781
- scan filesystem freespace and inode maps...
- 16:08:04: scanning filesystem freespace - 64 of 64 allocation groups done
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- 16:08:04: scanning agi unlinked lists - 64 of 64 allocation groups done
- process known inodes and perform inode discovery...
- agno = 45
- agno = 15
- agno = 0
- agno = 30
- agno = 60
- agno = 46
- agno = 16
Metadata corruption detected at 0x4330e3, xfs_inode block 0x17312a3f0/0x2000
- agno = 61
- agno = 31
- agno = 47
- agno = 62
- agno = 48
- agno = 49
- agno = 32
- agno = 33
- agno = 17
- agno = 1
bad magic number 0x0 on inode 18253615584
bad version number 0x0 on inode 18253615584
bad magic number 0x0 on inode 18253615585
bad version number 0x0 on inode 18253615585
bad magic number 0x0 on inode 18253615586
bad version number 0x0 on inode 18253615586
bad magic number 0x0 on inode 18253615587
bad version number 0x0 on inode 18253615587
bad magic number 0x0 on inode 18253615588
bad version number 0x0 on inode 18253615588
bad magic number 0x0 on inode 18253615589
bad version number 0x0 on inode 18253615589
bad magic number 0x0 on inode 18253615590
bad version number 0x0 on inode 18253615590
bad magic number 0x0 on inode 18253615591
bad version number 0x0 on inode 18253615591
bad magic number 0x0 on inode 18253615592
bad version number 0x0 on inode 18253615592
bad magic number 0x0 on inode 18253615593
bad version number 0x0 on inode 18253615593
bad magic number 0x0 on inode 18253615594
bad version number 0x0 on inode 18253615594
bad magic number 0x0 on inode 18253615595
bad version number 0x0 on inode 18253615595
bad magic number 0x0 on inode 18253615596
bad version number 0x0 on inode 18253615596
bad magic number 0x0 on inode 18253615597
bad version number 0x0 on inode 18253615597
bad magic number 0x0 on inode 18253615598
bad version number 0x0 on inode 18253615598
bad magic number 0x0 on inode 18253615599
bad version number 0x0 on inode 18253615599
bad magic number 0x0 on inode 18253615600
bad version number 0x0 on inode 18253615600
bad magic number 0x0 on inode 18253615601
bad version number 0x0 on inode 18253615601
bad magic number 0x0 on inode 18253615602
bad version number 0x0 on inode 18253615602
bad magic number 0x0 on inode 18253615603
bad version number 0x0 on inode 18253615603
bad magic number 0x0 on inode 18253615604
bad version number 0x0 on inode 18253615604
bad magic number 0x0 on inode 18253615605
bad version number 0x0 on inode 18253615605
bad magic number 0x0 on inode 18253615606
bad version number 0x0 on inode 18253615606
bad magic number 0x0 on inode 18253615607
bad version number 0x0 on inode 18253615607
bad magic number 0x0 on inode 18253615608
bad version number 0x0 on inode 18253615608
bad magic number 0x0 on inode 18253615609
bad version number 0x0 on inode 18253615609
bad magic number 0x0 on inode 18253615610
bad version number 0x0 on inode 18253615610
bad magic number 0x0 on inode 18253615611
bad version number 0x0 on inode 18253615611
bad magic number 0x0 on inode 18253615612
bad version number 0x0 on inode 18253615612
bad magic number 0x0 on inode 18253615613
bad version number 0x0 on inode 18253615613
bad magic number 0x0 on inode 18253615614
bad version number 0x0 on inode 18253615614
bad magic number 0x0 on inode 18253615615
bad version number 0x0 on inode 18253615615
bad magic number 0x0 on inode 18253615584, resetting magic number
bad version number 0x0 on inode 18253615584, resetting version number
bad magic number 0x0 on inode 18253615585, resetting magic number
bad version number 0x0 on inode 18253615585, resetting version number
bad magic number 0x0 on inode 18253615586, resetting magic number
bad version number 0x0 on inode 18253615586, resetting version number
bad magic number 0x0 on inode 18253615587, resetting magic number
bad version number 0x0 on inode 18253615587, resetting version number
bad magic number 0x0 on inode 18253615588, resetting magic number
bad version number 0x0 on inode 18253615588, resetting version number
bad magic number 0x0 on inode 18253615589, resetting magic number
bad version number 0x0 on inode 18253615589, resetting version number
bad magic number 0x0 on inode 18253615590, resetting magic number
bad version number 0x0 on inode 18253615590, resetting version number
bad magic number 0x0 on inode 18253615591, resetting magic number
bad version number 0x0 on inode 18253615591, resetting version number
bad magic number 0x0 on inode 18253615592, resetting magic number
bad version number 0x0 on inode 18253615592, resetting version number
bad magic number 0x0 on inode 18253615593, resetting magic number
bad version number 0x0 on inode 18253615593, resetting version number
bad magic number 0x0 on inode 18253615594, resetting magic number
bad version number 0x0 on inode 18253615594, resetting version number
bad magic number 0x0 on inode 18253615595, resetting magic number
bad version number 0x0 on inode 18253615595, resetting version number
bad magic number 0x0 on inode 18253615596, resetting magic number
bad version number 0x0 on inode 18253615596, resetting version number
bad magic number 0x0 on inode 18253615597, resetting magic number
bad version number 0x0 on inode 18253615597, resetting version number
bad magic number 0x0 on inode 18253615598, resetting magic number
bad version number 0x0 on inode 18253615598, resetting version number
bad magic number 0x0 on inode 18253615599, resetting magic number
bad version number 0x0 on inode 18253615599, resetting version number
bad magic number 0x0 on inode 18253615600, resetting magic number
bad version number 0x0 on inode 18253615600, resetting version number
bad magic number 0x0 on inode 18253615601, resetting magic number
bad version number 0x0 on inode 18253615601, resetting version number
bad magic number 0x0 on inode 18253615602, resetting magic number
bad version number 0x0 on inode 18253615602, resetting version number
bad magic number 0x0 on inode 18253615603, resetting magic number
bad version number 0x0 on inode 18253615603, resetting version number
bad magic number 0x0 on inode 18253615604, resetting magic number
bad version number 0x0 on inode 18253615604, resetting version number
bad magic number 0x0 on inode 18253615605, resetting magic number
bad version number 0x0 on inode 18253615605, resetting version number
bad magic number 0x0 on inode 18253615606, resetting magic number
bad version number 0x0 on inode 18253615606, resetting version number
bad magic number 0x0 on inode 18253615607, resetting magic number
bad version number 0x0 on inode 18253615607, resetting version number
bad magic number 0x0 on inode 18253615608, resetting magic number
bad version number 0x0 on inode 18253615608, resetting version number
bad magic number 0x0 on inode 18253615609, resetting magic number
bad version number 0x0 on inode 18253615609, resetting version number
bad magic number 0x0 on inode 18253615610, resetting magic number
bad version number 0x0 on inode 18253615610, resetting version number
bad magic number 0x0 on inode 18253615611, resetting magic number
bad version number 0x0 on inode 18253615611, resetting version number
bad magic number 0x0 on inode 18253615612, resetting magic number
bad version number 0x0 on inode 18253615612, resetting version number
bad magic number 0x0 on inode 18253615613, resetting magic number
bad version number 0x0 on inode 18253615613, resetting version number
bad magic number 0x0 on inode 18253615614, resetting magic number
bad version number 0x0 on inode 18253615614, resetting version number
bad magic number 0x0 on inode 18253615615, resetting magic number
bad version number 0x0 on inode 18253615615, resetting version number
- agno = 34
- agno = 63
- agno = 18
- agno = 19
- agno = 35
- agno = 2
- agno = 50
- agno = 51
- agno = 52
- agno = 36
- agno = 37
- agno = 3
- agno = 53
- agno = 54
- agno = 4
- agno = 55
- agno = 5
- agno = 38
- agno = 39
- agno = 40
- agno = 6
- agno = 7
- agno = 41
- agno = 42
- agno = 8
- agno = 9
- agno = 56
- agno = 43
- agno = 10
- agno = 44
- agno = 57
- agno = 11
- agno = 58
- agno = 12
- agno = 59
- agno = 13
- agno = 14
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- 16:10:36: process known inodes and inode discovery - 11776 of 11776 inodes done
- process newly discovered inodes...
- 16:10:36: process newly discovered inodes - 64 of 64 allocation groups done
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- 16:10:37: setting up duplicate extent list - 64 of 64 allocation groups done
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 2
- agno = 3
- agno = 1
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
- agno = 40
- agno = 41
- agno = 42
- agno = 43
- agno = 44
- agno = 45
- agno = 46
- agno = 47
- agno = 48
- agno = 49
- agno = 50
- agno = 51
- agno = 52
- agno = 53
- agno = 54
- agno = 55
- agno = 56
- agno = 57
- agno = 58
- agno = 59
- agno = 60
- agno = 61
- agno = 62
- agno = 63
- 16:10:38: check for inodes claiming duplicate blocks - 11776 of 11776 inodes done
Phase 5 - rebuild AG headers and trees...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
- agno = 40
- agno = 41
- agno = 42
- agno = 43
- agno = 44
- agno = 45
- agno = 46
- agno = 47
- agno = 48
- agno = 49
- agno = 50
- agno = 51
- agno = 52
- agno = 53
- agno = 54
- agno = 55
- agno = 56
- agno = 57
- agno = 58
- agno = 59
- agno = 60
- agno = 61
- agno = 62
- agno = 63
- 16:10:39: rebuild AG headers and trees - 64 of 64 allocation groups done
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
Metadata corruption detected at 0x4330e3, xfs_inode block 0x17312a3f0/0x2000
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
- agno = 40
- agno = 41
- agno = 42
- agno = 43
- agno = 44
- agno = 45
- agno = 46
- agno = 47
- agno = 48
- agno = 49
- agno = 50
- agno = 51
- agno = 52
- agno = 53
- agno = 54
- agno = 55
- agno = 56
- agno = 57
- agno = 58
- agno = 59
- agno = 60
- agno = 61
- agno = 62
- agno = 63
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
- 16:10:41: verify and correct link counts - 64 of 64 allocation groups done
Metadata corruption detected at 0x433385, xfs_inode block 0x17312a3f0/0x2000
libxfs_writebufr: write verifier failed on xfs_inode bno 0x17312a3f0/0x2000
releasing dirty buffer (bulk) to free list!
XFS_REPAIR Summary Sun Feb 9 16:11:57 2020
Phase Start End Duration
Phase 1: 02/09 16:07:56 02/09 16:07:57 1 second
Phase 2: 02/09 16:07:57 02/09 16:08:04 7 seconds
Phase 3: 02/09 16:08:04 02/09 16:10:36 2 minutes, 32 seconds
Phase 4: 02/09 16:10:36 02/09 16:10:38 2 seconds
Phase 5: 02/09 16:10:38 02/09 16:10:39 1 second
Phase 6: 02/09 16:10:39 02/09 16:10:41 2 seconds
Phase 7: 02/09 16:10:41 02/09 16:10:41
Total run time: 2 minutes, 45 seconds
done
Repair of readonly mount complete. Immediate reboot encouraged.
next parent reply other threads:[~2020-02-09 6:19 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <186d30f217e645728ad1f34724cbe3e7@jore.no>
2020-02-09 6:19 ` John Jore [this message]
2020-02-10 3:47 ` Bug in xfs_repair 5..4.0 / Unable to repair metadata corruption Eric Sandeen
2020-02-10 3:49 ` Eric Sandeen
[not found] ` <60f32c031f4345a2b680fbc8531f7bd3@jore.no>
2020-02-10 10:33 ` John Jore
2020-02-10 14:36 ` Eric Sandeen
2020-02-10 14:43 ` Eric Sandeen
2020-02-10 15:35 ` Eric Sandeen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b2babb761ed24dc986abc3073c5c47fc@jore.no \
--to=john@jore.no \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).