mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Steffen D.]

Hello list!

I'll try to explain my problem:

mounting a xfs file system gives the message:
"failure code -3003" and "structure needs cleaning"

xfs_repair /dev/sda4 does not work
"inux-ibbi:/home/steffen # xfs_repair /dev/sda4
Phase 1 - Superblock finden und =C3=BCberpr=C3=BCfen...
Phase 2 - ein internes Protokoll benutzen
        - Null-Protokoll...
FEHLER: Das Dateisystem hat wertvolle Metadaten-=C3=84nderungen in einem
Protokoll, das wiederholt werden sollte. H=C3=A4ngen Sie das Dateisystem ei=
n,
um das Protokoll zu wiederholen und h=C3=A4ngen Sie es wieder aus um
xfs_repair erneut auszuf=C3=BChren. Wenn Sie au=C3=9Fer Stande sind, das
Dateisystem einzuh=C3=A4ngen, benutzen Sie die -L-Option um das Protokoll z=
u
zerst=C3=B6ren und versuchen Sie eine Reparatur.
Beachten Sie, dass die Zerst=C3=B6rung des Protokolls Schaden verursachen
kann -- bitte versuchen sie das Dateisystem einzuh=C3=A4ngen ehe Sie dies t=
un."

xfs_repair -n /dev/sda4 said:
"linux-ibbi:/home/steffen # xfs_repair -n /dev/sda4
Phase 1 - Superblock finden und =C3=BCberpr=C3=BCfen...
Phase 2 - ein internes Protokoll benutzen
        - freier Speicher und Inode-Karten des Dateisystems werden
gescannt...
Metadata corruption detected at block 0x1a1d6001/0x200
flfirst 118 in agf 2 too large (max =3D 118)
agf 118 freelist blocks bad, skipping freelist scan
Metadata corruption detected at block 0x272c1001/0x200
flfirst 118 in agf 3 too large (max =3D 118)
agf 118 freelist blocks bad, skipping freelist scan
agi unlinked bucket 51 is 61133235 in ag 2 (inode=3D598004147)
sb_icount 349952, counted 357504
sb_ifree 9426, counted 7433
sb_fdblocks 9895143, counted 9426793
        - Wurzel-Inode-St=C3=BCck gefunden
Phase 3 - f=C3=BCr jedes AG...
        - agi unverkn=C3=BCpfte Listen werden gescannt (aber nicht bereinig=
t...
        - bekannte Inodes werden behandelt und Inode-Entdeckung wird
durchgef=C3=BChrt...
        - agno =3D 0
        - agno =3D 1
        - agno =3D 2
        - agno =3D 3
        - neu entdeckte Inodes werden behandelt...
Phase 4 - auf doppelte Bl=C3=B6cke =C3=BCberpr=C3=BCfen...
        - Liste mit doppeltem Ausma=C3=9F wird eingerichtet...
        - es wird gepr=C3=BCft ob Inodes Blocks doppelt beanspruchen...
        - agno =3D 1
        - agno =3D 2
        - agno =3D 3
        - agno =3D 0
Kein Ver=C3=A4nderungskennzeichen gesetzt, Phase 5 wird =C3=BCbersprungen
Phase 6 - Inode-Verbindbarkeit wird gepr=C3=BCft...
        - Dateisystem wird durchquert ...
        - durchqueren beendet ...
        - nicht verbundene Inodes werden nach lost+found verschoben ...
disconnected inode 598004147, k=C3=B6nnte zu lost+found gehen
Phase 7 - Verweisanzahl wird gepr=C3=BCft
would have reset inode 598004147 nlinks from 0 to 1
Kein =C3=84nderungskennzeichen gesetzt, Leeren des Dateisystems wird
=C3=BCbersprungen und es wird beendet"

Do I really need to use xfs_repair-L ?

Thank you and best regards

Steffen



-- 
Signatur ausser Betrieb.

Re: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Carlos E. R.]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

On 2016-10-30 21:36, Steffen D. wrote:
> Hello list!
> 
> I'll try to explain my problem:
> 

> "inux-ibbi:/home/steffen # xfs_repair /dev/sda4
> Phase 1 - Superblock finden und =C3=BCberpr=C3=BCfen...
> Phase 2 - ein internes Protokoll benutzen

I don't know if it is too late, but to make things easier for the
experts here, it is better to run the command in English. Like this:

LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 xfs_repair /dev/sda4


I have a script for convenience:


cer@Telcontar:~> cat /usr/local/bin/ingles
#!/bin/sh

LANG=en_US.UTF-8 \
  LC_ALL=en_US.UTF-8 \
  DICTIONARY=english \
  KDE_LANG=en_US.UTF-8 \
  exec "$@"

cer@Telcontar:~>



-- 
Cheers / Saludos,

		Carlos E. R.
		(from 13.1 x86_64 "Bottle" at Telcontar)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Steffen D.]

[-- Attachment #1: Type: text/plain, Size: 3108 bytes --]

Hello again
thanks to Carlos for the hint about the language.
Here are the messages in english:
----------
linux-ibbi:/home/steffen # LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
xfs_repair /dev/sda4
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
linux-ibbi:/home/steffen #
----------

and

----------
linux-ibbi:/home/steffen # LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
xfs_repair -n /dev/sda4
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
Metadata corruption detected at block 0x1a1d6001/0x200
flfirst 118 in agf 2 too large (max = 118)
agf 118 freelist blocks bad, skipping freelist scan
Metadata corruption detected at block 0x272c1001/0x200
flfirst 118 in agf 3 too large (max = 118)
agf 118 freelist blocks bad, skipping freelist scan
agi unlinked bucket 51 is 61133235 in ag 2 (inode=598004147)
sb_icount 349952, counted 357504
sb_ifree 9426, counted 7433
sb_fdblocks 9895143, counted 9426793
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 0
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 598004147, would move to lost+found
Phase 7 - verify link counts...
would have reset inode 598004147 nlinks from 0 to 1
No modify flag set, skipping filesystem flush and exiting.
linux-ibbi:/home/steffen #
----------

Any idea what to do? Or do have to use xfs_repair -L

Thanks a lot

Steffen


---------- Forwarded message ----------
From: Carlos E. R. <robin.listas@telefonica.net>
Date: 2016-10-30 21:45 GMT+01:00
Subject: Re: mounting a xfs file system gives the message: "failure
code -3003" and "structure needs cleaning"
To: linux-xfs@vger.kernel.org


On 2016-10-30 21:36, Steffen D. wrote:
> Hello list!
>
> I'll try to explain my problem:
>

> "inux-ibbi:/home/steffen # xfs_repair /dev/sda4
> Phase 1 - Superblock finden und =C3=BCberpr=C3=BCfen...
> Phase 2 - ein internes Protokoll benutzen

I don't know if it is too late, but to make things easier for the
experts here, it is better to run the command in English. Like this:




-- 
Signatur ausser Betrieb.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Dave Chinner]

On Mon, Oct 31, 2016 at 08:29:37PM +0100, Steffen D. wrote:
> Hello again
> thanks to Carlos for the hint about the language.
> Here are the messages in english:
> ----------
> linux-ibbi:/home/steffen # LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
> xfs_repair /dev/sda4
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
> ERROR: The filesystem has valuable metadata changes in a log which needs to
> be replayed.  Mount the filesystem to replay the log, and unmount it before
> re-running xfs_repair.  If you are unable to mount the filesystem, then use
> the -L option to destroy the log and attempt a repair.
> Note that destroying the log may cause corruption -- please attempt a mount
> of the filesystem before doing this.
> linux-ibbi:/home/steffen #
> ----------
> 
> and
> 
> ----------
> linux-ibbi:/home/steffen # LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
> xfs_repair -n /dev/sda4
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - scan filesystem freespace and inode maps...
> Metadata corruption detected at block 0x1a1d6001/0x200
> flfirst 118 in agf 2 too large (max = 118)
> agf 118 freelist blocks bad, skipping freelist scan
> Metadata corruption detected at block 0x272c1001/0x200
> flfirst 118 in agf 3 too large (max = 118)
> agf 118 freelist blocks bad, skipping freelist scan
> agi unlinked bucket 51 is 61133235 in ag 2 (inode=598004147)
> sb_icount 349952, counted 357504
> sb_ifree 9426, counted 7433
> sb_fdblocks 9895143, counted 9426793

hmmm - AGFL sizing issue?

What is the error in dmesg when you try to mount the filesystem?
What hardware are you running on (cpu, ram, etc) and where/how was
this filesystem created? Did it come from another machine?

xfs_reapir -L should fix it, but I'd like to understand what the
problem was first and how it happened....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Steffen D.]

Hi Dave,

2016-10-31 20:42 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> On Mon, Oct 31, 2016 at 08:29:37PM +0100, Steffen D. wrote:
>> Hello again
>> thanks to Carlos for the hint about the language.
>> Here are the messages in english:
>> ----------
>> linux-ibbi:/home/steffen # LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
>> ...
>
> hmmm - AGFL sizing issue?
>
> What is the error in dmesg when you try to mount the filesystem?

----------
[46389.083808] XFS (sda4): Mounting V5 Filesystem
[46389.193790] XFS (sda4): Starting recovery (logdev: internal)
[46390.697811] XFS (sda4): Metadata corruption detected at
xfs_agf_read_verify+0x5a/0x100 [xfs], block 0x1a1d6001
[46390.697814] XFS (sda4): Unmount and run xfs_repair
[46390.697815] XFS (sda4): First 64 bytes of corrupted metadata buffer:
[46390.697817] ffff880213e2d600: 58 41 47 46 00 00 00 01 00 00 00 02
01 a1 d6 00  XAGF............
[46390.697818] ffff880213e2d610: 00 72 fb d6 00 73 ee 3e 00 00 00 00
00 00 00 02  .r...s.>........
[46390.697819] ffff880213e2d620: 00 00 00 02 00 00 00 00 00 00 00 76
00 00 00 06  ...........v....
[46390.697820] ffff880213e2d630: 00 00 00 07 00 83 6d 93 00 28 47 02
00 00 00 29  ......m..(G....)
[46390.697853] XFS (sda4): metadata I/O error: block 0x1a1d6001
("xfs_trans_read_buf_map") error 117 numblks 1
----------

> What hardware are you running on (cpu, ram, etc) and

Intel i5-4460 4 x 3.20 GHz
8 GB Ram
HDD:
sda: SCSI-SATA ST500DM002-1BD14
sdd: Sandisk SDSSDP128G
Linux OpenSuse 42.1



> where/how was
> this filesystem created? Did it come from another machine?

Was created while installing the previous system (OpenSuse 13.2)

The problem starts after a system crash while I was copying some files.

>
> xfs_reapir -L should fix it, but I'd like to understand what the
> problem was first and how it happened....
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com



-- 
Signatur ausser Betrieb.

Re: Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Dave Chinner]

On Mon, Oct 31, 2016 at 09:20:56PM +0100, Steffen D. wrote:
> Hi Dave,
> 
> 2016-10-31 20:42 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> > On Mon, Oct 31, 2016 at 08:29:37PM +0100, Steffen D. wrote:
> >> Hello again
> >> thanks to Carlos for the hint about the language.
> >> Here are the messages in english:
> >> ----------
> >> linux-ibbi:/home/steffen # LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
> >> ...
> >
> > hmmm - AGFL sizing issue?
> >
> > What is the error in dmesg when you try to mount the filesystem?
> 
> ----------
> [46389.083808] XFS (sda4): Mounting V5 Filesystem
> [46389.193790] XFS (sda4): Starting recovery (logdev: internal)
> [46390.697811] XFS (sda4): Metadata corruption detected at
> xfs_agf_read_verify+0x5a/0x100 [xfs], block 0x1a1d6001
> [46390.697814] XFS (sda4): Unmount and run xfs_repair
> [46390.697815] XFS (sda4): First 64 bytes of corrupted metadata buffer:
> [46390.697817] ffff880213e2d600: 58 41 47 46 00 00 00 01 00 00 00 02 01 a1 d6 00  XAGF............
> [46390.697818] ffff880213e2d610: 00 72 fb d6 00 73 ee 3e 00 00 00 00 00 00 00 02  .r...s.>........
> [46390.697819] ffff880213e2d620: 00 00 00 02 00 00 00 00 00 00 00 76 00 00 00 06  ...........v....
> [46390.697820] ffff880213e2d630: 00 00 00 07 00 83 6d 93 00 28 47 02 00 00 00 29  ......m..(G....)
> [46390.697853] XFS (sda4): metadata I/O error: block 0x1a1d6001
> ("xfs_trans_read_buf_map") error 117 numblks 1

flfirst = 0x76 = 118
fllast = 0x6 = 6
flcount = 0x7 = 7

That implies you've some variant of this problem:

commit 96f859d52bcb1c6ea6f3388d39862bf7143e2f30
Author: Darrick J. Wong <darrick.wong@oracle.com>
Date:   Mon Jan 4 16:13:21 2016 +1100

    libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct
    
    Because struct xfs_agfl is 36 bytes long and has a 64-bit integer
    inside it, gcc will quietly round the structure size up to the nearest
    64 bits -- in this case, 40 bytes.  This results in the XFS_AGFL_SIZE
    macro returning incorrect results for v5 filesystems on 64-bit
    machines (118 items instead of 119).  As a result, a 32-bit xfs_repair
    will see garbage in AGFL item 119 and complain.
    
    Therefore, tell gcc not to pad the structure so that the AGFL size
    calculation is correct.
    
    cc: <stable@vger.kernel.org> # 3.10 - 4.4
    Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
    Reviewed-by: Dave Chinner <dchinner@redhat.com>
    Signed-off-by: Dave Chinner <david@fromorbit.com>


> ----------
> 
> > What hardware are you running on (cpu, ram, etc) and
> 
> Intel i5-4460 4 x 3.20 GHz
> 8 GB Ram
> HDD:
> sda: SCSI-SATA ST500DM002-1BD14
> sdd: Sandisk SDSSDP128G
> Linux OpenSuse 42.1

What kernel version? 32 bit or 64 bit?

> > where/how was
> > this filesystem created? Did it come from another machine?
> 
> Was created while installing the previous system (OpenSuse 13.2)

What kernel version was that?  32 bit or 64 bit?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Steffen D.]

Hi Dave

2016-10-31 21:37 GMT+01:00 Dave Chinner <david@fromorbit.com>:

> What kernel version? 32 bit or 64 bit?

booth kernels where 64 bit

-- 
Signatur ausser Betrieb.

Re: Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Dave Chinner]

On Mon, Oct 31, 2016 at 09:53:49PM +0100, Steffen D. wrote:
> Hi Dave
> 
> 2016-10-31 21:37 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> 
> > What kernel version? 32 bit or 64 bit?
> 
> booth kernels where 64 bit

What kernel versions? Both old and new versions are important. Also
how long ago you did you upgrade?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Steffen D.]

2016-10-31 21:58 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> On Mon, Oct 31, 2016 at 09:53:49PM +0100, Steffen D. wrote:
>> Hi Dave
>>
>> 2016-10-31 21:37 GMT+01:00 Dave Chinner <david@fromorbit.com>:
>>
>> > What kernel version? 32 bit or 64 bit?
>>
>> booth kernels where 64 bit
>
> What kernel versions? Both old and new versions are important.

old kernel: 3.16
new kernel: 4.1

sorry, I have not more detailed informations


> Also
> how long ago you did you upgrade?

The upgrade was just 2 days ago

-- 
Signatur ausser Betrieb.

Re: Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Dave Chinner]

On Mon, Oct 31, 2016 at 10:09:09PM +0100, Steffen D. wrote:
> 2016-10-31 21:58 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> > On Mon, Oct 31, 2016 at 09:53:49PM +0100, Steffen D. wrote:
> >> Hi Dave
> >>
> >> 2016-10-31 21:37 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> >>
> >> > What kernel version? 32 bit or 64 bit?
> >>
> >> booth kernels where 64 bit
> >
> > What kernel versions? Both old and new versions are important.
> 
> old kernel: 3.16
> new kernel: 4.1
> 
> sorry, I have not more detailed informations

Do youhave the minor numbers for the kernel versions? major numbers
are insufficient to tell me what I need to know, because the commit
I quoted got backported to various stable kernels....

(uname -a output is usually best when someone asks for kernel
versions).

> > Also
> > how long ago you did you upgrade?
> 
> The upgrade was just 2 days ago

Ok, so the first iteration across the AGFLs could trigger this,
depending on what version of 4.1 is being run...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Carlos E. R.]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

On 2016-10-31 22:09, Steffen D. wrote:
> 2016-10-31 21:58 GMT+01:00 Dave Chinner <david@fromorbit.com>:


>> What kernel versions? Both old and new versions are important.
> 
> old kernel: 3.16
> new kernel: 4.1
> 
> sorry, I have not more detailed informations

If you upgraded in place (example, zypper dup), the /var/log/zypp/
directory contains all the history of updates and installations.

If you installed new on top of the old install, then no.


-- 
Cheers / Saludos,

		Carlos E. R.
		(from 13.1 x86_64 "Bottle" at Telcontar)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Fwd: mounting a xfs file system gives the message: "failure code -3003" and "structure needs cleaning"

[Steffen D.]

Thank You! worked perfect!

2016-11-02 21:48 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> On Wed, Nov 02, 2016 at 08:33:48PM +0100, Steffen D. wrote:
>> Hi Dave,
>>
>> 2016-10-31 20:42 GMT+01:00 Dave Chinner <david@fromorbit.com>:
>>
>> >>
>> >
>> > hmmm - AGFL sizing issue?
>> >
>> > xfs_reapir -L should fix it, but I'd like to understand what the
>>
>> Do you think it works? Or is it very risky?
>
> xfs_repair rebuilds the AGFL from scratch regardless of whether it
> is ok to begin with or not. As for the log being discarded, there's
> not a of inconsistency - mainly unlinked inodes that get cleaned up,
> which log recovery would do anyway...
>
> So there's isn't any obvious significant risks to do this.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com



-- 
Signatur ausser Betrieb.