xfs_repair couldn't verify primary superblock

xfs_repair couldn't verify primary superblock

[Stéphane Larose]

[-- Attachment #1.1: Type: text/plain, Size: 4968 bytes --]

Hello,

When I try to mount a cleany unmounted XFS filesystem, I get those errors in the log :

[1650856.121229] XFS (dm-24): Mounting V4 Filesystem
[1650856.135455] XFS (dm-24): Log inconsistent or not a log (last==0, first!=1)
[1650856.135461] XFS (dm-24): empty log check failed
[1650856.135463] XFS (dm-24): log mount/recovery failed: error 22
[1650856.135495] XFS (dm-24): log mount failed

I have tried xfs_repair but...

manitou:~ # xfs_repair /dev/dm-24
Phase 1 - find and verify superblock...
couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!

attempting to find secondary superblock...
........................................

After a lot of dots, xfs_repair can't find secondary superblocks.

This is where my knowledge of XFS filesystem stops. Looking at the superblock information, I have a feeling that the primary superblock is fine but not the secondary superblocks (strange blocksize and dblocks numbers). Is there any way to copy the primary superblock to the secondary superblocks? Maybe this is not a good idea?

Thank you for any help!

manitou:~ # xfs_db /dev/dm-24
xfs_db> sb 0
xfs_db> p
magicnum = 0x58465342
blocksize = 4096
dblocks = 805306368
rblocks = 0
rextents = 0
uuid = 1da22ae2-a572-4db7-b177-b90021a20863
logstart = 536870916
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 201326592
agcount = 4
rbmblocks = 0
logblocks = 393216
versionnum = 0xb4a4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 28
rextslog = 0
inprogress = 0
imax_pct = 5
icount = 15424
ifree = 147
fdblocks = 269387727
frextents = 0
uquotino = 0
gquotino = null
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 1
features2 = 0x8a
bad_features2 = 0x8a
features_compat = 0
features_ro_compat = 0
features_incompat = 0
features_log_incompat = 0
crc = 0 (unchecked)
pquotino = 0
lsn = 0
xfs_db> sb 1
xfs_db> p
magicnum = 0xf6b89fbf
blocksize = 1483932129
dblocks = 15694009933101722137
rblocks = 11068626756016902811
rextents = 1593063622148300128
uuid = 6d1ec280-7f96-c817-0f72-d68433061de8
logstart = 17414611744493230066
rootino = 9197560587205818314
rbmino = 1554642860151675405
rsumino = 377387355824336886
rextsize = 1433889823
agblocks = 3940333176
agcount = 2930968516
rbmblocks = 422920841
logblocks = 4017127169
versionnum = 0xd520
sectsize = 19409
inodesize = 16393
inopblock = 2470
fname = "\323R|\0066\274\271\252\240\370-\231"
blocklog = 164
sectlog = 42
inodelog = 53
inopblog = 10
agblklog = 72
rextslog = 24
inprogress = 224
imax_pct = 46
icount = 11911544493988237103
ifree = 3385761089021187134
fdblocks = 9395999607433847230
frextents = 7946546560364525221
uquotino = 15084273978162174417
gquotino = 478665847475212977
qflags = 0x1301
flags = 0x37
shared_vn = 3
inoalignmt = 3222551805
unit = 3368784749
width = 3167603851
dirblklog = 130
logsectlog = 143
logsectsize = 2101
logsunit = 2030308911
features2 = 0x5d5cbbfe
bad_features2 = 0x2afcd122
features_compat = 0x2a3d35d5
features_ro_compat = 0xa4e58ba9
features_incompat = 0x13dccb60
features_log_incompat = 0xe79d437a
crc = 0xbd38d745 (unchecked)
pquotino = 7778233919291903441
lsn = 0xa8ffb559acd96aea
xfs_db> sb 2
xfs_db> p
magicnum = 0x41474341
blocksize = 1195590471
dblocks = 4847921951085253716
rblocks = 4846789398308864835
rextents = 4702132125296707412
uuid = 41544354-5441-4141-4754-544743544743
logstart = 4703821056660554049
rootino = 4703802364996815473
rbmino = 7814906735281653061
rsumino = 4991471925827290437
rextsize = 1162167621
agblocks = 1162167621
agcount = 1162167621
rbmblocks = 1162167621
logblocks = 1162167621
versionnum = 0x4545
sectsize = 17733
inodesize = 17733
inopblock = 17733
fname = "EEEEEEEEEEEE"
blocklog = 69
sectlog = 69
inodelog = 69
inopblog = 69
agblklog = 69
rextslog = 69
inprogress = 69
imax_pct = 69
icount = 4991471925827290437
ifree = 4991471925827290437
fdblocks = 4991471925827290437
frextents = 4991471925827290437
uquotino = 4991471925827290437
gquotino = 4991471925827290437
qflags = 0x4545
flags = 0x45
shared_vn = 69
inoalignmt = 1162167621
unit = 1162167621
width = 1162167621
dirblklog = 69
logsectlog = 69
logsectsize = 17733
logsunit = 1162167621
features2 = 0x45454545
bad_features2 = 0x45454545
features_compat = 0x45454545
features_ro_compat = 0x45454545
features_incompat = 0x450a6461
features_log_incompat = 0x74612e75
crc = 0x6e697469 (unchecked)
pquotino = 8531350831795363699
lsn = 0x74617420302e3032
xfs_db>

---
Stéphane Larose
Analyste de l'informatique
Institut de Biologie Intégrative et des Systèmes (IBIS)
Pavillon Charles-Eugène-Marchand
Université Laval


[-- Attachment #1.2: Type: text/html, Size: 19188 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair couldn't verify primary superblock

[Eric Sandeen]

On 4/20/16 1:50 PM, Stéphane Larose wrote:
> Hello,
> 
>  
> 
> When I try to mount a cleany unmounted XFS filesystem, I get those errors in the log :
> 
> 
> [1650856.121229] XFS (dm-24): Mounting V4 Filesystem
> [1650856.135455] XFS (dm-24): Log inconsistent or not a log (last==0, first!=1)
> [1650856.135461] XFS (dm-24): empty log check failed
> [1650856.135463] XFS (dm-24): log mount/recovery failed: error 22
> [1650856.135495] XFS (dm-24): log mount failed

You could maybe use xfs_logdump to see what's there, but...

> I have tried xfs_repair but…
> 
> manitou:~ # xfs_repair /dev/dm-24
> 
> Phase 1 - find and verify superblock...
> couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!
> 
> attempting to find secondary superblock...
> ........................................

> After a lot of dots, xfs_repair can’t find secondary superblocks.
> 
> This is where my knowledge of XFS filesystem stops. Looking at the superblock information, I have a feeling that the primary superblock is fine but not the secondary superblocks (strange blocksize and dblocks numbers). Is there any way to copy the primary superblock to the secondary superblocks? Maybe this is not a good idea?
> 
> Thank you for any help!
> 

... the stated locations of the backup supers certainly don't contain good data:

> xfs_db> sb 1
> xfs_db> p
> magicnum = 0xf6b89fbf
> blocksize = 1483932129
> dblocks = 15694009933101722137
> rblocks = 11068626756016902811
> rextents = 1593063622148300128
...
> xfs_db> sb 2
> xfs_db> p
> magicnum = 0x41474341
> blocksize = 1195590471
> dblocks = 4847921951085253716
> rblocks = 4846789398308864835
> rextents = 4702132125296707412
...

which is why it couldn't find any valid/matching backups.  The stated AG size
does look correct, though, for the overall size of the filesystem as stated
in SB 0.

Is there more to this story?  Did something "interesting" happen with the
underlying storage?  Or with the filesystem prior to this problem?

does "blkid /dev/dm-24" say anything other than "xfs filesystem?"

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_repair couldn't verify primary superblock

[Stéphane Larose]

Hi Eric,

Nothing more interesting. From the log:

2016-04-04T14:48:48.735738-04:00 manitou kernel: [271211.548878] XFS (dm-24): Mounting V4 Filesystem
2016-04-04T14:48:48.882738-04:00 manitou kernel: [271211.694242] XFS (dm-24): Ending clean mount

Then no more logs about dm-24.

Also no errors from the underlying storage (verified in SANtricity) which is an IS5000. The filesystem was new, the first mount was on 2016-04-04.

manitou:~ # blkid /dev/dm-24
/dev/dm-24: UUID="1da22ae2-a572-4db7-b177-b90021a20863" TYPE="xfs"

Thank you for your help,

Stéphane

-----Message d'origine-----
De : Eric Sandeen [mailto:sandeen@sandeen.net] 
Envoyé : 20 avril 2016 16:14
À : Stéphane Larose <Stephane.Larose@ibis.ulaval.ca>
Objet : Re: xfs_repair couldn't verify primary superblock

On 4/20/16 1:50 PM, Stéphane Larose wrote:
> Hello,
> 
>  
> 
> When I try to mount a cleany unmounted XFS filesystem, I get those errors in the log :
> 
> 
> [1650856.121229] XFS (dm-24): Mounting V4 Filesystem [1650856.135455] 
> XFS (dm-24): Log inconsistent or not a log (last==0, first!=1) 
> [1650856.135461] XFS (dm-24): empty log check failed [1650856.135463] 
> XFS (dm-24): log mount/recovery failed: error 22 [1650856.135495] XFS 
> (dm-24): log mount failed

You could maybe use xfs_logdump to see what's there, but...

> I have tried xfs_repair but…
> 
> manitou:~ # xfs_repair /dev/dm-24
> 
> Phase 1 - find and verify superblock...
> couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!
> 
> attempting to find secondary superblock...
> ........................................

> After a lot of dots, xfs_repair can’t find secondary superblocks.
> 
> This is where my knowledge of XFS filesystem stops. Looking at the superblock information, I have a feeling that the primary superblock is fine but not the secondary superblocks (strange blocksize and dblocks numbers). Is there any way to copy the primary superblock to the secondary superblocks? Maybe this is not a good idea?
> 
> Thank you for any help!
> 

... the stated locations of the backup supers certainly don't contain good data:

> xfs_db> sb 1
> xfs_db> p
> magicnum = 0xf6b89fbf
> blocksize = 1483932129
> dblocks = 15694009933101722137
> rblocks = 11068626756016902811
> rextents = 1593063622148300128
...
> xfs_db> sb 2
> xfs_db> p
> magicnum = 0x41474341
> blocksize = 1195590471
> dblocks = 4847921951085253716
> rblocks = 4846789398308864835
> rextents = 4702132125296707412
...

which is why it couldn't find any valid/matching backups.  The stated AG size does look correct, though, for the overall size of the filesystem as stated in SB 0.

Is there more to this story?  Did something "interesting" happen with the underlying storage?  Or with the filesystem prior to this problem?

does "blkid /dev/dm-24" say anything other than "xfs filesystem?"

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair couldn't verify primary superblock

[Eric Sandeen]

On 4/21/16 10:42 AM, Stéphane Larose wrote:
> Hi Eric,
> 
> Nothing more interesting. From the log:
> 
> 2016-04-04T14:48:48.735738-04:00 manitou kernel: [271211.548878] XFS (dm-24): Mounting V4 Filesystem
> 2016-04-04T14:48:48.882738-04:00 manitou kernel: [271211.694242] XFS (dm-24): Ending clean mount
> 
> Then no more logs about dm-24.
> 
> Also no errors from the underlying storage (verified in SANtricity)
> which is an IS5000. The filesystem was new, the first mount was on
> 2016-04-04.
> 
> manitou:~ # blkid /dev/dm-24
> /dev/dm-24: UUID="1da22ae2-a572-4db7-b177-b90021a20863" TYPE="xfs"
> 
> Thank you for your help,

Any chance that some other host is accessing the same LUN on the SAN?

This really looks like something external corrupted the filesystem
by writing over it...

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_repair couldn't verify primary superblock

[Stéphane Larose]

Hi Eric,

Yes, that could be the problem. I have another host that have access to the same LUN. On the other host, I had a problem doing a live extend of an ext3 filesystem in the same LVM Volume Group. Maybe this corrupted the XFS filesystem. Anyway, we decided this morning to delete the partition and recreate the filesystem. We will be able to regenerate the data that was on it.

Thanks,

Stéphane

-----Message d'origine-----
De : xfs-bounces@oss.sgi.com [mailto:xfs-bounces@oss.sgi.com] De la part de Eric Sandeen
Envoyé : 21 avril 2016 14:34
À : xfs@oss.sgi.com
Objet : Re: xfs_repair couldn't verify primary superblock

On 4/21/16 10:42 AM, Stéphane Larose wrote:
> Hi Eric,
> 
> Nothing more interesting. From the log:
> 
> 2016-04-04T14:48:48.735738-04:00 manitou kernel: [271211.548878] XFS 
> (dm-24): Mounting V4 Filesystem
> 2016-04-04T14:48:48.882738-04:00 manitou kernel: [271211.694242] XFS 
> (dm-24): Ending clean mount
> 
> Then no more logs about dm-24.
> 
> Also no errors from the underlying storage (verified in SANtricity) 
> which is an IS5000. The filesystem was new, the first mount was on 
> 2016-04-04.
> 
> manitou:~ # blkid /dev/dm-24
> /dev/dm-24: UUID="1da22ae2-a572-4db7-b177-b90021a20863" TYPE="xfs"
> 
> Thank you for your help,

Any chance that some other host is accessing the same LUN on the SAN?

This really looks like something external corrupted the filesystem by writing over it...

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs