xfs_repair on filesystem stuck in "rebuild AG headers and trees"?

xfs_repair on filesystem stuck in "rebuild AG headers and trees"?

[Libor Klepáč]

Hi,
we have virtual machine with 8TB data disk with around 5TB of data in few large files (backup repository of nakivo backup solution - it contains snapshots of vmware machines - one file per snapshot).

We have recently upgraded VM from ubuntu 20.04 to ubuntu 22.04 and after reboot, mount of this filesystem took ages.

I started xfs_repair on it and now it spits line
rebuild AG headers and trees - 16417 of 16417 allocation groups done

in 15 minutes interval for last two days.
Is it in loop?
Can i break it? 
Data on disk is just remote copy of backup, so it can be lost, it will just take some time to transfer it again.

Kernel is probably from package 5.15.0-58.64  - sorry i don't know real version, cannot get it from ubuntu package.
xfsprogs should be 5.13.0-1ubuntu2

Thanks for any info,

with regards,
Libor

Re: xfs_repair on filesystem stuck in "rebuild AG headers and trees"?

[Libor Klepáč]

Hi,
i breaked xfs_repair and lauched it again.
On beginning of phase 6 i have
failed to create prefetch thread: Resource temporarily unavailable
https://download.bcom.cz/xfs/Screenshot_20230130_114707.jpeg

Also, there is dmesg, when it was trying to mount it and it was stuck
https://download.bcom.cz/xfs/dmesg.txt

Libor


From: Libor Klepáč <libor.klepac@bcom.cz>
Sent: Thursday, January 26, 2023 11:46
To: linux-xfs@vger.kernel.org <linux-xfs@vger.kernel.org>
Subject: xfs_repair on filesystem stuck in "rebuild AG headers and trees"? 
 
Hi,
we have virtual machine with 8TB data disk with around 5TB of data in few large files (backup repository of nakivo backup solution - it contains snapshots of vmware machines - one file per snapshot).

We have recently upgraded VM from ubuntu 20.04 to ubuntu 22.04 and after reboot, mount of this filesystem took ages.

I started xfs_repair on it and now it spits line
rebuild AG headers and trees - 16417 of 16417 allocation groups done

in 15 minutes interval for last two days.
Is it in loop?
Can i break it? 
Data on disk is just remote copy of backup, so it can be lost, it will just take some time to transfer it again.

Kernel is probably from package 5.15.0-58.64  - sorry i don't know real version, cannot get it from ubuntu package.
xfsprogs should be 5.13.0-1ubuntu2

Thanks for any info,

with regards,
Libor

Re: xfs_repair on filesystem stuck in "rebuild AG headers and trees"?

[Eric Sandeen]

Both your kernel and xfsprogs are older versions from your distro.

I would report this bug to your distro, or try upstream versions to see if
the problem has been resolved.

-Eric

On 1/30/23 4:57 AM, Libor Klepáč wrote:
> Hi,
> i breaked xfs_repair and lauched it again.
> On beginning of phase 6 i have
> failed to create prefetch thread: Resource temporarily unavailable
> https://download.bcom.cz/xfs/Screenshot_20230130_114707.jpeg
> 
> Also, there is dmesg, when it was trying to mount it and it was stuck
> https://download.bcom.cz/xfs/dmesg.txt
> 
> Libor
> 
> 
> From: Libor Klepáč <libor.klepac@bcom.cz>
> Sent: Thursday, January 26, 2023 11:46
> To: linux-xfs@vger.kernel.org <linux-xfs@vger.kernel.org>
> Subject: xfs_repair on filesystem stuck in "rebuild AG headers and trees"? 
>  
> Hi,
> we have virtual machine with 8TB data disk with around 5TB of data in few large files (backup repository of nakivo backup solution - it contains snapshots of vmware machines - one file per snapshot).
> 
> We have recently upgraded VM from ubuntu 20.04 to ubuntu 22.04 and after reboot, mount of this filesystem took ages.
> 
> I started xfs_repair on it and now it spits line
> rebuild AG headers and trees - 16417 of 16417 allocation groups done
> 
> in 15 minutes interval for last two days.
> Is it in loop?
> Can i break it? 
> Data on disk is just remote copy of backup, so it can be lost, it will just take some time to transfer it again.
> 
> Kernel is probably from package 5.15.0-58.64  - sorry i don't know real version, cannot get it from ubuntu package.
> xfsprogs should be 5.13.0-1ubuntu2
> 
> Thanks for any info,
> 
> with regards,
> Libor
> 
> 

Re: xfs_repair on filesystem stuck in "rebuild AG headers and trees"?

[Libor Klepáč]

Hi,
thanks for the tip.
I tried xfsprogs version  6.0.0 from newer ubuntu and it seems it fixed the filesystem (or at least did not spiral into loop :)
Meantime, i have created new filesystem, so i will keep this one until data do replicate to new filesystem

Thanks,
Libor


From: Eric Sandeen <sandeen@redhat.com>
Sent: Monday, January 30, 2023 16:16
To: Libor Klepáč <libor.klepac@bcom.cz>; linux-xfs@vger.kernel.org <linux-xfs@vger.kernel.org>
Subject: Re: xfs_repair on filesystem stuck in "rebuild AG headers and trees"? 
 
Both your kernel and xfsprogs are older versions from your distro.

I would report this bug to your distro, or try upstream versions to see if
the problem has been resolved.

-Eric

On 1/30/23 4:57 AM, Libor Klepáč wrote:
> Hi,
> i breaked xfs_repair and lauched it again.
> On beginning of phase 6 i have
> failed to create prefetch thread: Resource temporarily unavailable
> https://download.bcom.cz/xfs/Screenshot_20230130_114707.jpeg
> 
> Also, there is dmesg, when it was trying to mount it and it was stuck
> https://download.bcom.cz/xfs/dmesg.txt
> 
> Libor
> 
> 
> From: Libor Klepáč <libor.klepac@bcom.cz>
> Sent: Thursday, January 26, 2023 11:46
> To: linux-xfs@vger.kernel.org <linux-xfs@vger.kernel.org>
> Subject: xfs_repair on filesystem stuck in "rebuild AG headers and trees"? 
>  
> Hi,
> we have virtual machine with 8TB data disk with around 5TB of data in few large files (backup repository of nakivo backup solution - it contains snapshots of vmware machines - one file per snapshot).
> 
> We have recently upgraded VM from ubuntu 20.04 to ubuntu 22.04 and after reboot, mount of this filesystem took ages.
> 
> I started xfs_repair on it and now it spits line
> rebuild AG headers and trees - 16417 of 16417 allocation groups done
> 
> in 15 minutes interval for last two days.
> Is it in loop?
> Can i break it? 
> Data on disk is just remote copy of backup, so it can be lost, it will just take some time to transfer it again.
> 
> Kernel is probably from package 5.15.0-58.64  - sorry i don't know real version, cannot get it from ubuntu package.
> xfsprogs should be 5.13.0-1ubuntu2
> 
> Thanks for any info,
> 
> with regards,
> Libor
> 
>