XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

[Stefan Priebe]

Hello list,

one of my servers is crashing every night with the following errors:

XFS (sda3): xfs_iunlink_remove: xfs_imap returned error -22.
XFS (sda3): xfs_inactive_ifree: xfs_ifree returned error -22
XFS (sda3): xfs_do_force_shutdown(0x1) called from line 1872 of file 
fs/xfs/xfs_inode.c. Return address = 0xffffffff9e33cf53
XFS (sda3): I/O Error Detected. Shutting down filesystem

I ran xfs_repair v4.9.0 but it seems unable to recover / fix this error. 
I'm running an OpenSUSE 42.2 kernel based von v4.4. Is there anything i 
can do to fix this?

Thanks!

Greets,
Stefan

Re: XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

[Eric Sandeen]

On 4/17/17 2:54 AM, Stefan Priebe wrote:
> Hello list,
> 
> one of my servers is crashing every night with the following errors:
> 
> XFS (sda3): xfs_iunlink_remove: xfs_imap returned error -22.
> XFS (sda3): xfs_inactive_ifree: xfs_ifree returned error -22 XFS


-EINVAL

> (sda3): xfs_do_force_shutdown(0x1) called from line 1872 of file
> fs/xfs/xfs_inode.c. Return address = 0xffffffff9e33cf53 XFS (sda3):
> I/O Error Detected. Shutting down filesystem
> 
> I ran xfs_repair v4.9.0 but it seems unable to recover / fix this
> error. I'm running an OpenSUSE 42.2 kernel based von v4.4. Is there
> anything i can do to fix this?

# echo 11 > /proc/sys/fs/xfs/error_level

to get a full backtrace from the shutdown (if you didn't already)
so we can at least see what process is encountering the errors.

I have a hunch, though - are you running xfs_fsr overnight?

-Eric


> Thanks!
> 

Re: XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

[Eric Sandeen]

On 4/17/17 7:19 AM, Eric Sandeen wrote:
> 
> 
> On 4/17/17 2:54 AM, Stefan Priebe wrote:
>> Hello list,
>>
>> one of my servers is crashing every night with the following errors:
>>
>> XFS (sda3): xfs_iunlink_remove: xfs_imap returned error -22.
>> XFS (sda3): xfs_inactive_ifree: xfs_ifree returned error -22 XFS
> 
> 
> -EINVAL
> 
>> (sda3): xfs_do_force_shutdown(0x1) called from line 1872 of file
>> fs/xfs/xfs_inode.c. Return address = 0xffffffff9e33cf53 XFS (sda3):
>> I/O Error Detected. Shutting down filesystem
>>
>> I ran xfs_repair v4.9.0 but it seems unable to recover / fix this
>> error. I'm running an OpenSUSE 42.2 kernel based von v4.4. Is there
>> anything i can do to fix this?
> 
> # echo 11 > /proc/sys/fs/xfs/error_level
> 
> to get a full backtrace from the shutdown (if you didn't already)
> so we can at least see what process is encountering the errors.
> 
> I have a hunch, though - are you running xfs_fsr overnight?

(or maybe more likely, xfsdump?)

-Eric

Re: XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

[Stefan Priebe - Profihost AG]

Hi Eric,

> On 4/17/17 2:54 AM, Stefan Priebe wrote:
>> Hello list,
>>
>> one of my servers is crashing every night with the following errors:
>>
>> XFS (sda3): xfs_iunlink_remove: xfs_imap returned error -22.
>> XFS (sda3): xfs_inactive_ifree: xfs_ifree returned error -22 XFS
> 
> 
> -EINVAL
> 
>> (sda3): xfs_do_force_shutdown(0x1) called from line 1872 of file
>> fs/xfs/xfs_inode.c. Return address = 0xffffffff9e33cf53 XFS (sda3):
>> I/O Error Detected. Shutting down filesystem
>>
>> I ran xfs_repair v4.9.0 but it seems unable to recover / fix this
>> error. I'm running an OpenSUSE 42.2 kernel based von v4.4. Is there
>> anything i can do to fix this?
> 
> # echo 11 > /proc/sys/fs/xfs/error_level
> 
> to get a full backtrace from the shutdown (if you didn't already)
> so we can at least see what process is encountering the errors.

done.

> I have a hunch, though - are you running xfs_fsr overnight?

No only rsync from dev/a to dev/b. But the reported device is a (source).

I also have two empty folders in /lost+found which i cannot delete. Even
after running xfs_repair.

# rm -r /lost+found/
rm: cannot remove ‘/lost+found/536874081’: Directory not empty
rm: cannot remove ‘/lost+found/536874085’: Directory not empty

Greets,
Stefan

Re: XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

[Stefan Priebe - Profihost AG]

Am 17.04.2017 um 17:04 schrieb Stefan Priebe - Profihost AG:
> Hi Eric,
> 
>> On 4/17/17 2:54 AM, Stefan Priebe wrote:
>>> Hello list,
>>>
>>> one of my servers is crashing every night with the following errors:
>>>
>>> XFS (sda3): xfs_iunlink_remove: xfs_imap returned error -22.
>>> XFS (sda3): xfs_inactive_ifree: xfs_ifree returned error -22 XFS
>>
>>
>> -EINVAL
>>
>>> (sda3): xfs_do_force_shutdown(0x1) called from line 1872 of file
>>> fs/xfs/xfs_inode.c. Return address = 0xffffffff9e33cf53 XFS (sda3):
>>> I/O Error Detected. Shutting down filesystem
>>>
>>> I ran xfs_repair v4.9.0 but it seems unable to recover / fix this
>>> error. I'm running an OpenSUSE 42.2 kernel based von v4.4. Is there
>>> anything i can do to fix this?
>>
>> # echo 11 > /proc/sys/fs/xfs/error_level
>>
>> to get a full backtrace from the shutdown (if you didn't already)
>> so we can at least see what process is encountering the errors.
> 
> done.
> 
>> I have a hunch, though - are you running xfs_fsr overnight?
> 
> No only rsync from dev/a to dev/b. But the reported device is a (source).

verified: but that job is done 3-4 hours before crashing.

> 
> I also have two empty folders in /lost+found which i cannot delete. Even
> after running xfs_repair.
> 
> # rm -r /lost+found/
> rm: cannot remove ‘/lost+found/536874081’: Directory not empty
> rm: cannot remove ‘/lost+found/536874085’: Directory not empty
> 
> Greets,
> Stefan
> 

Re: XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

[Stefan Priebe - Profihost AG]

System crashed again but output was the same.

2017-04-18 07:30:18     XFS (sda3): xfs_iunlink_remove: xfs_imap
returned error -22.
2017-04-18 07:30:18     XFS (sda3): xfs_inactive_ifree: xfs_ifree
returned error -22
2017-04-18 07:30:18     XFS (sda3): xfs_do_force_shutdown(0x1) called
from line 1872 of file fs/xfs/xfs_inode.c. Return address =
0xffffffff9b33cf93
2017-04-18 07:30:18     XFS (sda3): I/O Error Detected. Shutting down
filesystem
2017-04-18 07:30:18     XFS (sda3): Please umount the filesystem and
rectify the problem(s)
2017-04-18 07:30:18     XFS (sda3): xfs_log_force: error -5 returned.
2017-04-18 07:30:28     XFS (sda3): xfs_log_force: error -5 returned.
2017-04-18 07:30:58     XFS (sda3): xfs_log_force: error -5 returned.
2017-04-18 07:31:28     XFS (sda3): xfs_log_force: error -5 returned.

Stefan
Am 17.04.2017 um 17:06 schrieb Stefan Priebe - Profihost AG:
> 
> Am 17.04.2017 um 17:04 schrieb Stefan Priebe - Profihost AG:
>> Hi Eric,
>>
>>> On 4/17/17 2:54 AM, Stefan Priebe wrote:
>>>> Hello list,
>>>>
>>>> one of my servers is crashing every night with the following errors:
>>>>
>>>> XFS (sda3): xfs_iunlink_remove: xfs_imap returned error -22.
>>>> XFS (sda3): xfs_inactive_ifree: xfs_ifree returned error -22 XFS
>>>
>>>
>>> -EINVAL
>>>
>>>> (sda3): xfs_do_force_shutdown(0x1) called from line 1872 of file
>>>> fs/xfs/xfs_inode.c. Return address = 0xffffffff9e33cf53 XFS (sda3):
>>>> I/O Error Detected. Shutting down filesystem
>>>>
>>>> I ran xfs_repair v4.9.0 but it seems unable to recover / fix this
>>>> error. I'm running an OpenSUSE 42.2 kernel based von v4.4. Is there
>>>> anything i can do to fix this?
>>>
>>> # echo 11 > /proc/sys/fs/xfs/error_level
>>>
>>> to get a full backtrace from the shutdown (if you didn't already)
>>> so we can at least see what process is encountering the errors.
>>
>> done.
>>
>>> I have a hunch, though - are you running xfs_fsr overnight?
>>
>> No only rsync from dev/a to dev/b. But the reported device is a (source).
> 
> verified: but that job is done 3-4 hours before crashing.
> 
>>
>> I also have two empty folders in /lost+found which i cannot delete. Even
>> after running xfs_repair.
>>
>> # rm -r /lost+found/
>> rm: cannot remove ‘/lost+found/536874081’: Directory not empty
>> rm: cannot remove ‘/lost+found/536874085’: Directory not empty
>>
>> Greets,
>> Stefan
>>

Re: XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

[Darrick J. Wong]

On Tue, Apr 18, 2017 at 09:01:17AM +0200, Stefan Priebe - Profihost AG wrote:
> System crashed again but output was the same.
> 
> 2017-04-18 07:30:18     XFS (sda3): xfs_iunlink_remove: xfs_imap
> returned error -22.
> 2017-04-18 07:30:18     XFS (sda3): xfs_inactive_ifree: xfs_ifree
> returned error -22
> 2017-04-18 07:30:18     XFS (sda3): xfs_do_force_shutdown(0x1) called
> from line 1872 of file fs/xfs/xfs_inode.c. Return address =
> 0xffffffff9b33cf93
> 2017-04-18 07:30:18     XFS (sda3): I/O Error Detected. Shutting down
> filesystem
> 2017-04-18 07:30:18     XFS (sda3): Please umount the filesystem and
> rectify the problem(s)
> 2017-04-18 07:30:18     XFS (sda3): xfs_log_force: error -5 returned.
> 2017-04-18 07:30:28     XFS (sda3): xfs_log_force: error -5 returned.
> 2017-04-18 07:30:58     XFS (sda3): xfs_log_force: error -5 returned.
> 2017-04-18 07:31:28     XFS (sda3): xfs_log_force: error -5 returned.

That looks like some sort of problem with a garbage value in
di_next_unlinked that we hit when removing inodes from the AGI unlinked
inode list.

Could you take a metadump and get it to us for further analysis?
Eric might be interested. ;)

--D

> 
> Stefan
> Am 17.04.2017 um 17:06 schrieb Stefan Priebe - Profihost AG:
> > 
> > Am 17.04.2017 um 17:04 schrieb Stefan Priebe - Profihost AG:
> >> Hi Eric,
> >>
> >>> On 4/17/17 2:54 AM, Stefan Priebe wrote:
> >>>> Hello list,
> >>>>
> >>>> one of my servers is crashing every night with the following errors:
> >>>>
> >>>> XFS (sda3): xfs_iunlink_remove: xfs_imap returned error -22.
> >>>> XFS (sda3): xfs_inactive_ifree: xfs_ifree returned error -22 XFS
> >>>
> >>>
> >>> -EINVAL
> >>>
> >>>> (sda3): xfs_do_force_shutdown(0x1) called from line 1872 of file
> >>>> fs/xfs/xfs_inode.c. Return address = 0xffffffff9e33cf53 XFS (sda3):
> >>>> I/O Error Detected. Shutting down filesystem
> >>>>
> >>>> I ran xfs_repair v4.9.0 but it seems unable to recover / fix this
> >>>> error. I'm running an OpenSUSE 42.2 kernel based von v4.4. Is there
> >>>> anything i can do to fix this?
> >>>
> >>> # echo 11 > /proc/sys/fs/xfs/error_level
> >>>
> >>> to get a full backtrace from the shutdown (if you didn't already)
> >>> so we can at least see what process is encountering the errors.
> >>
> >> done.
> >>
> >>> I have a hunch, though - are you running xfs_fsr overnight?
> >>
> >> No only rsync from dev/a to dev/b. But the reported device is a (source).
> > 
> > verified: but that job is done 3-4 hours before crashing.
> > 
> >>
> >> I also have two empty folders in /lost+found which i cannot delete. Even
> >> after running xfs_repair.
> >>
> >> # rm -r /lost+found/
> >> rm: cannot remove ‘/lost+found/536874081’: Directory not empty
> >> rm: cannot remove ‘/lost+found/536874085’: Directory not empty
> >>
> >> Greets,
> >> Stefan
> >>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

[Stefan Priebe - Profihost AG]

Hi,

the FS is mounted and in production. I didn't know how to dump a running FS.

Greets,
Stefan

Am 18.04.2017 um 18:59 schrieb Darrick J. Wong:
> On Tue, Apr 18, 2017 at 09:01:17AM +0200, Stefan Priebe - Profihost AG wrote:
>> System crashed again but output was the same.
>>
>> 2017-04-18 07:30:18     XFS (sda3): xfs_iunlink_remove: xfs_imap
>> returned error -22.
>> 2017-04-18 07:30:18     XFS (sda3): xfs_inactive_ifree: xfs_ifree
>> returned error -22
>> 2017-04-18 07:30:18     XFS (sda3): xfs_do_force_shutdown(0x1) called
>> from line 1872 of file fs/xfs/xfs_inode.c. Return address =
>> 0xffffffff9b33cf93
>> 2017-04-18 07:30:18     XFS (sda3): I/O Error Detected. Shutting down
>> filesystem
>> 2017-04-18 07:30:18     XFS (sda3): Please umount the filesystem and
>> rectify the problem(s)
>> 2017-04-18 07:30:18     XFS (sda3): xfs_log_force: error -5 returned.
>> 2017-04-18 07:30:28     XFS (sda3): xfs_log_force: error -5 returned.
>> 2017-04-18 07:30:58     XFS (sda3): xfs_log_force: error -5 returned.
>> 2017-04-18 07:31:28     XFS (sda3): xfs_log_force: error -5 returned.
> 
> That looks like some sort of problem with a garbage value in
> di_next_unlinked that we hit when removing inodes from the AGI unlinked
> inode list.
> 
> Could you take a metadump and get it to us for further analysis?
> Eric might be interested. ;)
> 
> --D
> 
>>
>> Stefan
>> Am 17.04.2017 um 17:06 schrieb Stefan Priebe - Profihost AG:
>>>
>>> Am 17.04.2017 um 17:04 schrieb Stefan Priebe - Profihost AG:
>>>> Hi Eric,
>>>>
>>>>> On 4/17/17 2:54 AM, Stefan Priebe wrote:
>>>>>> Hello list,
>>>>>>
>>>>>> one of my servers is crashing every night with the following errors:
>>>>>>
>>>>>> XFS (sda3): xfs_iunlink_remove: xfs_imap returned error -22.
>>>>>> XFS (sda3): xfs_inactive_ifree: xfs_ifree returned error -22 XFS
>>>>>
>>>>>
>>>>> -EINVAL
>>>>>
>>>>>> (sda3): xfs_do_force_shutdown(0x1) called from line 1872 of file
>>>>>> fs/xfs/xfs_inode.c. Return address = 0xffffffff9e33cf53 XFS (sda3):
>>>>>> I/O Error Detected. Shutting down filesystem
>>>>>>
>>>>>> I ran xfs_repair v4.9.0 but it seems unable to recover / fix this
>>>>>> error. I'm running an OpenSUSE 42.2 kernel based von v4.4. Is there
>>>>>> anything i can do to fix this?
>>>>>
>>>>> # echo 11 > /proc/sys/fs/xfs/error_level
>>>>>
>>>>> to get a full backtrace from the shutdown (if you didn't already)
>>>>> so we can at least see what process is encountering the errors.
>>>>
>>>> done.
>>>>
>>>>> I have a hunch, though - are you running xfs_fsr overnight?
>>>>
>>>> No only rsync from dev/a to dev/b. But the reported device is a (source).
>>>
>>> verified: but that job is done 3-4 hours before crashing.
>>>
>>>>
>>>> I also have two empty folders in /lost+found which i cannot delete. Even
>>>> after running xfs_repair.
>>>>
>>>> # rm -r /lost+found/
>>>> rm: cannot remove ‘/lost+found/536874081’: Directory not empty
>>>> rm: cannot remove ‘/lost+found/536874085’: Directory not empty
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: XFS crashes with xfs_iunlink_remove: xfs_imap returned error -22

[Darrick J. Wong]

On Tue, Apr 18, 2017 at 07:32:04PM +0200, Stefan Priebe - Profihost AG wrote:
> Hi,
> 
> the FS is mounted and in production. I didn't know how to dump a running FS.

Since there's no lvm here, unmounting, metadumping, and mounting is
probably the only reasonable option.

--D

> 
> Greets,
> Stefan
> 
> Am 18.04.2017 um 18:59 schrieb Darrick J. Wong:
> > On Tue, Apr 18, 2017 at 09:01:17AM +0200, Stefan Priebe - Profihost AG wrote:
> >> System crashed again but output was the same.
> >>
> >> 2017-04-18 07:30:18     XFS (sda3): xfs_iunlink_remove: xfs_imap
> >> returned error -22.
> >> 2017-04-18 07:30:18     XFS (sda3): xfs_inactive_ifree: xfs_ifree
> >> returned error -22
> >> 2017-04-18 07:30:18     XFS (sda3): xfs_do_force_shutdown(0x1) called
> >> from line 1872 of file fs/xfs/xfs_inode.c. Return address =
> >> 0xffffffff9b33cf93
> >> 2017-04-18 07:30:18     XFS (sda3): I/O Error Detected. Shutting down
> >> filesystem
> >> 2017-04-18 07:30:18     XFS (sda3): Please umount the filesystem and
> >> rectify the problem(s)
> >> 2017-04-18 07:30:18     XFS (sda3): xfs_log_force: error -5 returned.
> >> 2017-04-18 07:30:28     XFS (sda3): xfs_log_force: error -5 returned.
> >> 2017-04-18 07:30:58     XFS (sda3): xfs_log_force: error -5 returned.
> >> 2017-04-18 07:31:28     XFS (sda3): xfs_log_force: error -5 returned.
> > 
> > That looks like some sort of problem with a garbage value in
> > di_next_unlinked that we hit when removing inodes from the AGI unlinked
> > inode list.
> > 
> > Could you take a metadump and get it to us for further analysis?
> > Eric might be interested. ;)
> > 
> > --D
> > 
> >>
> >> Stefan
> >> Am 17.04.2017 um 17:06 schrieb Stefan Priebe - Profihost AG:
> >>>
> >>> Am 17.04.2017 um 17:04 schrieb Stefan Priebe - Profihost AG:
> >>>> Hi Eric,
> >>>>
> >>>>> On 4/17/17 2:54 AM, Stefan Priebe wrote:
> >>>>>> Hello list,
> >>>>>>
> >>>>>> one of my servers is crashing every night with the following errors:
> >>>>>>
> >>>>>> XFS (sda3): xfs_iunlink_remove: xfs_imap returned error -22.
> >>>>>> XFS (sda3): xfs_inactive_ifree: xfs_ifree returned error -22 XFS
> >>>>>
> >>>>>
> >>>>> -EINVAL
> >>>>>
> >>>>>> (sda3): xfs_do_force_shutdown(0x1) called from line 1872 of file
> >>>>>> fs/xfs/xfs_inode.c. Return address = 0xffffffff9e33cf53 XFS (sda3):
> >>>>>> I/O Error Detected. Shutting down filesystem
> >>>>>>
> >>>>>> I ran xfs_repair v4.9.0 but it seems unable to recover / fix this
> >>>>>> error. I'm running an OpenSUSE 42.2 kernel based von v4.4. Is there
> >>>>>> anything i can do to fix this?
> >>>>>
> >>>>> # echo 11 > /proc/sys/fs/xfs/error_level
> >>>>>
> >>>>> to get a full backtrace from the shutdown (if you didn't already)
> >>>>> so we can at least see what process is encountering the errors.
> >>>>
> >>>> done.
> >>>>
> >>>>> I have a hunch, though - are you running xfs_fsr overnight?
> >>>>
> >>>> No only rsync from dev/a to dev/b. But the reported device is a (source).
> >>>
> >>> verified: but that job is done 3-4 hours before crashing.
> >>>
> >>>>
> >>>> I also have two empty folders in /lost+found which i cannot delete. Even
> >>>> after running xfs_repair.
> >>>>
> >>>> # rm -r /lost+found/
> >>>> rm: cannot remove ‘/lost+found/536874081’: Directory not empty
> >>>> rm: cannot remove ‘/lost+found/536874085’: Directory not empty
> >>>>
> >>>> Greets,
> >>>> Stefan
> >>>>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html