Disconnected inodes after test xfs/261

Disconnected inodes after test xfs/261

[Jan Kara]

  Hello,

  in my test KVM with today's Linus' kernel I'm getting xfs_repair
complaint about disconnected inodes after the test xfs/261 finishes
(with success). xfs_repair output is like:
xfs_repair -n /dev/vdb2
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 132, would move to lost+found
disconnected inode 133, would move to lost+found
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
---
Given how trivial test xfs/261 is, it seems like created private mtab files
that also get unlinked don't get added to AGI unlinked list before umount.
I didn't have a detailed look whether that's possible or not and probably
won't get to it before Christmas. So I'm sending this just in case someone
more knowledgeable has ideas earlier...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Dave Chinner]

On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
>   Hello,
> 
>   in my test KVM with today's Linus' kernel I'm getting xfs_repair
> complaint about disconnected inodes after the test xfs/261 finishes
> (with success). xfs_repair output is like:
> xfs_repair -n /dev/vdb2
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan (but don't clear) agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
> No modify flag set, skipping phase 5
> Phase 6 - check inode connectivity...
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> disconnected inode 132, would move to lost+found
> disconnected inode 133, would move to lost+found
> Phase 7 - verify link counts...
> No modify flag set, skipping filesystem flush and exiting.
> ---
> Given how trivial test xfs/261 is, it seems like created private mtab files
> that also get unlinked don't get added to AGI unlinked list before umount.
> I didn't have a detailed look whether that's possible or not and probably
> won't get to it before Christmas. So I'm sending this just in case someone
> more knowledgeable has ideas earlier...

I don't see that here. If you mount/unmount the filesystem, does the
warning go away? i.e. xfs_repair -n ignores the contents of
the log, so if the unlinked list transactions are in the log then
log recovery will make everything good again.

That said, if unmount is not leaving the log clean, then we've still
got an issue we need to get to the bottom of.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Michael L. Semon]

On 12/17/14 16:02, Dave Chinner wrote:
> On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
>>   Hello,
>>
>>   in my test KVM with today's Linus' kernel I'm getting xfs_repair
>> complaint about disconnected inodes after the test xfs/261 finishes
>> (with success). xfs_repair output is like:
>> xfs_repair -n /dev/vdb2
>> Phase 1 - find and verify superblock...
>> Phase 2 - using internal log
>>         - scan filesystem freespace and inode maps...
>>         - found root inode chunk
>> Phase 3 - for each AG...
>>         - scan (but don't clear) agi unlinked lists...
>>         - process known inodes and perform inode discovery...
>>         - agno = 0
>>         - agno = 1
>>         - agno = 2
>>         - agno = 3
>>         - process newly discovered inodes...
>> Phase 4 - check for duplicate blocks...
>>         - setting up duplicate extent list...
>>         - check for inodes claiming duplicate blocks...
>>         - agno = 0
>>         - agno = 1
>>         - agno = 2
>>         - agno = 3
>> No modify flag set, skipping phase 5
>> Phase 6 - check inode connectivity...
>>         - traversing filesystem ...
>>         - traversal finished ...
>>         - moving disconnected inodes to lost+found ...
>> disconnected inode 132, would move to lost+found
>> disconnected inode 133, would move to lost+found
>> Phase 7 - verify link counts...
>> No modify flag set, skipping filesystem flush and exiting.
>> ---
>> Given how trivial test xfs/261 is, it seems like created private mtab files
>> that also get unlinked don't get added to AGI unlinked list before umount.
>> I didn't have a detailed look whether that's possible or not and probably
>> won't get to it before Christmas. So I'm sending this just in case someone
>> more knowledgeable has ideas earlier...
> 
> I don't see that here. If you mount/unmount the filesystem, does the
> warning go away? i.e. xfs_repair -n ignores the contents of
> the log, so if the unlinked list transactions are in the log then
> log recovery will make everything good again.
> 
> That said, if unmount is not leaving the log clean, then we've still
> got an issue we need to get to the bottom of.
> 
> Cheers,
> 
> Dave.

I've seen this but seemingly only on v4-superblock XFS and only with Dave's 
xfsprogs RFC libxfs patches applied.  [Nice patchset, BTW.]  Will `git pull` 
everything again, then take it home to the x86 dungeon and test it again.

Good luck!

Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Jan Kara]

[-- Attachment #1: Type: text/plain, Size: 3533 bytes --]

On Thu 18-12-14 08:02:26, Dave Chinner wrote:
> On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
> >   Hello,
> > 
> >   in my test KVM with today's Linus' kernel I'm getting xfs_repair
> > complaint about disconnected inodes after the test xfs/261 finishes
> > (with success). xfs_repair output is like:
> > xfs_repair -n /dev/vdb2
> > Phase 1 - find and verify superblock...
> > Phase 2 - using internal log
> >         - scan filesystem freespace and inode maps...
> >         - found root inode chunk
> > Phase 3 - for each AG...
> >         - scan (but don't clear) agi unlinked lists...
> >         - process known inodes and perform inode discovery...
> >         - agno = 0
> >         - agno = 1
> >         - agno = 2
> >         - agno = 3
> >         - process newly discovered inodes...
> > Phase 4 - check for duplicate blocks...
> >         - setting up duplicate extent list...
> >         - check for inodes claiming duplicate blocks...
> >         - agno = 0
> >         - agno = 1
> >         - agno = 2
> >         - agno = 3
> > No modify flag set, skipping phase 5
> > Phase 6 - check inode connectivity...
> >         - traversing filesystem ...
> >         - traversal finished ...
> >         - moving disconnected inodes to lost+found ...
> > disconnected inode 132, would move to lost+found
> > disconnected inode 133, would move to lost+found
> > Phase 7 - verify link counts...
> > No modify flag set, skipping filesystem flush and exiting.
> > ---
> > Given how trivial test xfs/261 is, it seems like created private mtab files
> > that also get unlinked don't get added to AGI unlinked list before umount.
> > I didn't have a detailed look whether that's possible or not and probably
> > won't get to it before Christmas. So I'm sending this just in case someone
> > more knowledgeable has ideas earlier...
> 
> I don't see that here. If you mount/unmount the filesystem, does the
> warning go away? i.e. xfs_repair -n ignores the contents of
> the log, so if the unlinked list transactions are in the log then
> log recovery will make everything good again.
  No, the problem is still there after mounting and unmounting the
filesystem.

Given what Michael wrote: I'm running xfs_repair version 3.2.1, filesystem
is V4.

When I look via xfs_db at the inode I can see nlink is 1 which looks
strange. So maybe the problem is somewhere else than I thought:
xfs_db> inode 132
xfs_db> p
core.magic = 0x494e
core.mode = 0100000
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 1
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 0
core.gid = 0
core.flushiter = 1
core.atime.sec = Thu Dec 18 11:08:55 2014
core.atime.nsec = 510013169
core.mtime.sec = Thu Dec 18 11:08:55 2014
core.mtime.nsec = 510013169
core.ctime.sec = Thu Dec 18 11:08:55 2014
core.ctime.nsec = 510013169
core.size = 0
core.nblocks = 1
core.extsize = 0
core.nextents = 1
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 0
next_unlinked = null
u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,13,1,0]

I have taken xfs_metadump just after test xfs/261 completed and xfs_repair
reported error. It is attached.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

[-- Attachment #2: xfs-inode.img.xz --]
[-- Type: application/x-xz, Size: 21672 bytes --]

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Jan Kara]

On Thu 18-12-14 11:36:42, Jan Kara wrote:
> On Thu 18-12-14 08:02:26, Dave Chinner wrote:
> > On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
> > >   Hello,
> > > 
> > >   in my test KVM with today's Linus' kernel I'm getting xfs_repair
> > > complaint about disconnected inodes after the test xfs/261 finishes
> > > (with success). xfs_repair output is like:
> > > xfs_repair -n /dev/vdb2
> > > Phase 1 - find and verify superblock...
> > > Phase 2 - using internal log
> > >         - scan filesystem freespace and inode maps...
> > >         - found root inode chunk
> > > Phase 3 - for each AG...
> > >         - scan (but don't clear) agi unlinked lists...
> > >         - process known inodes and perform inode discovery...
> > >         - agno = 0
> > >         - agno = 1
> > >         - agno = 2
> > >         - agno = 3
> > >         - process newly discovered inodes...
> > > Phase 4 - check for duplicate blocks...
> > >         - setting up duplicate extent list...
> > >         - check for inodes claiming duplicate blocks...
> > >         - agno = 0
> > >         - agno = 1
> > >         - agno = 2
> > >         - agno = 3
> > > No modify flag set, skipping phase 5
> > > Phase 6 - check inode connectivity...
> > >         - traversing filesystem ...
> > >         - traversal finished ...
> > >         - moving disconnected inodes to lost+found ...
> > > disconnected inode 132, would move to lost+found
> > > disconnected inode 133, would move to lost+found
> > > Phase 7 - verify link counts...
> > > No modify flag set, skipping filesystem flush and exiting.
> > > ---
> > > Given how trivial test xfs/261 is, it seems like created private mtab files
> > > that also get unlinked don't get added to AGI unlinked list before umount.
> > > I didn't have a detailed look whether that's possible or not and probably
> > > won't get to it before Christmas. So I'm sending this just in case someone
> > > more knowledgeable has ideas earlier...
> > 
> > I don't see that here. If you mount/unmount the filesystem, does the
> > warning go away? i.e. xfs_repair -n ignores the contents of
> > the log, so if the unlinked list transactions are in the log then
> > log recovery will make everything good again.
>   No, the problem is still there after mounting and unmounting the
> filesystem.
> 
> Given what Michael wrote: I'm running xfs_repair version 3.2.1, filesystem
> is V4.
  Oh, and what might be related: Test xfs/071 passes but xfs_repair
complains like:
*** xfs_repair -n output ***
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
inode 131 - extent offset too large - start 14, count 1, offset 2251799813685247
correcting nextents for inode 131
bad data fork in inode 131
would have cleared inode 131
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
entry "071" in shortform directory 128 references free inode 131
would have junked entry "071" in directory inode 128
inode 131 - extent offset too large - start 14, count 1, offset 2251799813685247
correcting nextents for inode 131
bad data fork in inode 131
would have cleared inode 131
        - agno = 1
        - agno = 2
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
entry "071" in shortform directory inode 128 points to free inode 131
would junk entry
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Jan Kara]

On Thu 18-12-14 11:36:42, Jan Kara wrote:
> On Thu 18-12-14 08:02:26, Dave Chinner wrote:
> > On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
> > >   Hello,
> > > 
> > >   in my test KVM with today's Linus' kernel I'm getting xfs_repair
> > > complaint about disconnected inodes after the test xfs/261 finishes
> > > (with success). xfs_repair output is like:
> > > xfs_repair -n /dev/vdb2
> > > Phase 1 - find and verify superblock...
> > > Phase 2 - using internal log
> > >         - scan filesystem freespace and inode maps...
> > >         - found root inode chunk
> > > Phase 3 - for each AG...
> > >         - scan (but don't clear) agi unlinked lists...
> > >         - process known inodes and perform inode discovery...
> > >         - agno = 0
> > >         - agno = 1
> > >         - agno = 2
> > >         - agno = 3
> > >         - process newly discovered inodes...
> > > Phase 4 - check for duplicate blocks...
> > >         - setting up duplicate extent list...
> > >         - check for inodes claiming duplicate blocks...
> > >         - agno = 0
> > >         - agno = 1
> > >         - agno = 2
> > >         - agno = 3
> > > No modify flag set, skipping phase 5
> > > Phase 6 - check inode connectivity...
> > >         - traversing filesystem ...
> > >         - traversal finished ...
> > >         - moving disconnected inodes to lost+found ...
> > > disconnected inode 132, would move to lost+found
> > > disconnected inode 133, would move to lost+found
> > > Phase 7 - verify link counts...
> > > No modify flag set, skipping filesystem flush and exiting.
> > > ---
> > > Given how trivial test xfs/261 is, it seems like created private mtab files
> > > that also get unlinked don't get added to AGI unlinked list before umount.
> > > I didn't have a detailed look whether that's possible or not and probably
> > > won't get to it before Christmas. So I'm sending this just in case someone
> > > more knowledgeable has ideas earlier...
> > 
> > I don't see that here. If you mount/unmount the filesystem, does the
> > warning go away? i.e. xfs_repair -n ignores the contents of
> > the log, so if the unlinked list transactions are in the log then
> > log recovery will make everything good again.
>   No, the problem is still there after mounting and unmounting the
> filesystem.
> 
> Given what Michael wrote: I'm running xfs_repair version 3.2.1, filesystem
> is V4.
> 
> When I look via xfs_db at the inode I can see nlink is 1 which looks
> strange. So maybe the problem is somewhere else than I thought:
> xfs_db> inode 132
> xfs_db> p
> core.magic = 0x494e
> core.mode = 0100000
> core.version = 2
> core.format = 2 (extents)
> core.nlinkv2 = 1
> core.onlink = 0
> core.projid_lo = 0
> core.projid_hi = 0
> core.uid = 0
> core.gid = 0
> core.flushiter = 1
> core.atime.sec = Thu Dec 18 11:08:55 2014
> core.atime.nsec = 510013169
> core.mtime.sec = Thu Dec 18 11:08:55 2014
> core.mtime.nsec = 510013169
> core.ctime.sec = Thu Dec 18 11:08:55 2014
> core.ctime.nsec = 510013169
> core.size = 0
> core.nblocks = 1
> core.extsize = 0
> core.nextents = 1
> core.naextents = 0
> core.forkoff = 0
> core.aformat = 2 (extents)
> core.dmevmask = 0
> core.dmstate = 0
> core.newrtbm = 0
> core.prealloc = 0
> core.realtime = 0
> core.immutable = 0
> core.append = 0
> core.sync = 0
> core.noatime = 0
> core.nodump = 0
> core.rtinherit = 0
> core.projinherit = 0
> core.nosymlinks = 0
> core.extsz = 0
> core.extszinherit = 0
> core.nodefrag = 0
> core.filestream = 0
> core.gen = 0
> next_unlinked = null
> u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,13,1,0]
> 
> I have taken xfs_metadump just after test xfs/261 completed and xfs_repair
> reported error. It is attached.
  OK, so I understand better what's going on. The detached inodes are
actually inodes from quota files being created by quotacheck on mount. Test
xfs/261 first mounts with uquota - that adds user quota ino and quota
feature just fine. But then it mounts with gquota - now we go through
different path since xfs_sb_version_hasquota() is already true. We find
that group quota isn't initialized though and thus we still need to
allocate inode for group quota file. However the change to sb_gquotaino
isn't written out because the following test in xfs_sb_quota_to_disk()
fails:

	if ((*fields & XFS_SB_GQUOTINO) &&
                                (from->sb_qflags & XFS_GQUOTA_ACCT))
                to->sb_gquotino = cpu_to_be64(from->sb_gquotino);

because from->sb_qflags doesn't have XFS_GQUOTA_ACCT set (only
mp->m_qflags has XFS_GQUOTA_ACCT set). Now I'm somewhat lost in which flags
should be modified where so I'm not sure whether the test is just wrong or
whether sb_flags should be updated earlier (they get set only later in
xfs_qm_mount_quotas()).

							Honza


-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Michael L. Semon]

On 12/17/14 16:02, Dave Chinner wrote:
> On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
>>   Hello,
>>
>>   in my test KVM with today's Linus' kernel I'm getting xfs_repair
>> complaint about disconnected inodes after the test xfs/261 finishes
>> (with success). xfs_repair output is like:
>> xfs_repair -n /dev/vdb2
>> Phase 1 - find and verify superblock...
>> Phase 2 - using internal log
>>         - scan filesystem freespace and inode maps...
>>         - found root inode chunk
>> Phase 3 - for each AG...
>>         - scan (but don't clear) agi unlinked lists...
>>         - process known inodes and perform inode discovery...
>>         - agno = 0
>>         - agno = 1
>>         - agno = 2
>>         - agno = 3
>>         - process newly discovered inodes...
>> Phase 4 - check for duplicate blocks...
>>         - setting up duplicate extent list...
>>         - check for inodes claiming duplicate blocks...
>>         - agno = 0
>>         - agno = 1
>>         - agno = 2
>>         - agno = 3
>> No modify flag set, skipping phase 5
>> Phase 6 - check inode connectivity...
>>         - traversing filesystem ...
>>         - traversal finished ...
>>         - moving disconnected inodes to lost+found ...
>> disconnected inode 132, would move to lost+found
>> disconnected inode 133, would move to lost+found
>> Phase 7 - verify link counts...
>> No modify flag set, skipping filesystem flush and exiting.
>> ---
>> Given how trivial test xfs/261 is, it seems like created private mtab files
>> that also get unlinked don't get added to AGI unlinked list before umount.
>> I didn't have a detailed look whether that's possible or not and probably
>> won't get to it before Christmas. So I'm sending this just in case someone
>> more knowledgeable has ideas earlier...
> 
> I don't see that here. If you mount/unmount the filesystem, does the
> warning go away? i.e. xfs_repair -n ignores the contents of
> the log, so if the unlinked list transactions are in the log then
> log recovery will make everything good again.
> 
> That said, if unmount is not leaving the log clean, then we've still
> got an issue we need to get to the bottom of.
> 
> Cheers,
> 
> Dave.

OK, I played with this a bit, though in all but one case, my XFS stack 
is new.  A bisect will be done to find when this started.  kernel 
3.10.63 here at work is OK, where new xfsprogs is installed.

In trying out new kernels, I hit the "kernel too old" wall on boot.  
Therefore, a new system was brewed from slackware-current, and xfstests 
was installed on top of that.  This result is from the "huge-smp" 
vendor kernel, as it came from the original binary packages:

root@kyhorse:/var/lib/xfstests# uname -a
Linux kyhorse 3.14.24-smp #2 SMP Fri Nov 14 18:34:21 CST 2014 i686 Intel(R) Pentium(R) 4 CPU 1.80GHz GenuineIntel GNU/Linux

root@kyhorse:/var/lib/xfstests# ./check xfs/261
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/i686 kyhorse 3.14.24-smp
MKFS_OPTIONS  -- -f -bsize=4096 /dev/sda7
MOUNT_OPTIONS -- /dev/sda7 /mnt/scratch

xfs/261 2s ... 3s
_check_xfs_filesystem: filesystem on /dev/sda7 is inconsistent (c) (see /var/lib/xfstests/results//xfs/261.full)
_check_xfs_filesystem: filesystem on /dev/sda7 is inconsistent (r) (see /var/lib/xfstests/results//xfs/261.full)
Ran: xfs/261
Passed all 1 tests

root@kyhorse:/var/lib/xfstests# xfs_repair $SCRQ\b \bATCH_DEV
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 132, moving to lost+found
disconnected inode 133, moving to lost+found
Phase 7 - verify and correct link counts...
Note - quota info will be regenerated on next quota mount.
done

A mount/umount sequence did not change matters.  xfsprogs appears to 
be version 3.1.11, glancing from the tarball names in the a/ directory 
here:

ftp://ftp.slackware.com/pub/slackware/slackware-current/slackware/

The new setup was just made bootable before I came in to work, and the 
bisect will start tonight and will hopefully be posted by Monday night.

Thanks!

Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Eric Sandeen]

On 12/18/14 4:56 AM, Jan Kara wrote:
> On Thu 18-12-14 11:36:42, Jan Kara wrote:
>> On Thu 18-12-14 08:02:26, Dave Chinner wrote:
>>> On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
>>>>   Hello,
>>>>
>>>>   in my test KVM with today's Linus' kernel I'm getting xfs_repair
>>>> complaint about disconnected inodes after the test xfs/261 finishes
>>>> (with success). xfs_repair output is like:
>>>> xfs_repair -n /dev/vdb2
>>>> Phase 1 - find and verify superblock...
>>>> Phase 2 - using internal log
>>>>         - scan filesystem freespace and inode maps...
>>>>         - found root inode chunk
>>>> Phase 3 - for each AG...
>>>>         - scan (but don't clear) agi unlinked lists...
>>>>         - process known inodes and perform inode discovery...
>>>>         - agno = 0
>>>>         - agno = 1
>>>>         - agno = 2
>>>>         - agno = 3
>>>>         - process newly discovered inodes...
>>>> Phase 4 - check for duplicate blocks...
>>>>         - setting up duplicate extent list...
>>>>         - check for inodes claiming duplicate blocks...
>>>>         - agno = 0
>>>>         - agno = 1
>>>>         - agno = 2
>>>>         - agno = 3
>>>> No modify flag set, skipping phase 5
>>>> Phase 6 - check inode connectivity...
>>>>         - traversing filesystem ...
>>>>         - traversal finished ...
>>>>         - moving disconnected inodes to lost+found ...
>>>> disconnected inode 132, would move to lost+found
>>>> disconnected inode 133, would move to lost+found
>>>> Phase 7 - verify link counts...
>>>> No modify flag set, skipping filesystem flush and exiting.
>>>> ---
>>>> Given how trivial test xfs/261 is, it seems like created private mtab files
>>>> that also get unlinked don't get added to AGI unlinked list before umount.
>>>> I didn't have a detailed look whether that's possible or not and probably
>>>> won't get to it before Christmas. So I'm sending this just in case someone
>>>> more knowledgeable has ideas earlier...
>>>
>>> I don't see that here. If you mount/unmount the filesystem, does the
>>> warning go away? i.e. xfs_repair -n ignores the contents of
>>> the log, so if the unlinked list transactions are in the log then
>>> log recovery will make everything good again.
>>   No, the problem is still there after mounting and unmounting the
>> filesystem.
>>
>> Given what Michael wrote: I'm running xfs_repair version 3.2.1, filesystem
>> is V4.
>   Oh, and what might be related: Test xfs/071 passes but xfs_repair
> complains like:
> *** xfs_repair -n output ***
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan (but don't clear) agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
> inode 131 - extent offset too large - start 14, count 1, offset 2251799813685247
> correcting nextents for inode 131
> bad data fork in inode 131
> would have cleared inode 131

That's addressed by either 
http://oss.sgi.com/archives/xfs/2014-09/msg00524.html
or
http://oss.sgi.com/archives/xfs/2014-12/msg00106.html

FWIW...

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Jan Kara]

On Thu 18-12-14 16:21:58, Michael L. Semon wrote:
> On 12/17/14 16:02, Dave Chinner wrote:
> > On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
> >>   Hello,
> >>
> >>   in my test KVM with today's Linus' kernel I'm getting xfs_repair
> >> complaint about disconnected inodes after the test xfs/261 finishes
> >> (with success). xfs_repair output is like:
> >> xfs_repair -n /dev/vdb2
> >> Phase 1 - find and verify superblock...
> >> Phase 2 - using internal log
> >>         - scan filesystem freespace and inode maps...
> >>         - found root inode chunk
> >> Phase 3 - for each AG...
> >>         - scan (but don't clear) agi unlinked lists...
> >>         - process known inodes and perform inode discovery...
> >>         - agno = 0
> >>         - agno = 1
> >>         - agno = 2
> >>         - agno = 3
> >>         - process newly discovered inodes...
> >> Phase 4 - check for duplicate blocks...
> >>         - setting up duplicate extent list...
> >>         - check for inodes claiming duplicate blocks...
> >>         - agno = 0
> >>         - agno = 1
> >>         - agno = 2
> >>         - agno = 3
> >> No modify flag set, skipping phase 5
> >> Phase 6 - check inode connectivity...
> >>         - traversing filesystem ...
> >>         - traversal finished ...
> >>         - moving disconnected inodes to lost+found ...
> >> disconnected inode 132, would move to lost+found
> >> disconnected inode 133, would move to lost+found
> >> Phase 7 - verify link counts...
> >> No modify flag set, skipping filesystem flush and exiting.
> >> ---
> >> Given how trivial test xfs/261 is, it seems like created private mtab files
> >> that also get unlinked don't get added to AGI unlinked list before umount.
> >> I didn't have a detailed look whether that's possible or not and probably
> >> won't get to it before Christmas. So I'm sending this just in case someone
> >> more knowledgeable has ideas earlier...
> > 
> > I don't see that here. If you mount/unmount the filesystem, does the
> > warning go away? i.e. xfs_repair -n ignores the contents of
> > the log, so if the unlinked list transactions are in the log then
> > log recovery will make everything good again.
> > 
> > That said, if unmount is not leaving the log clean, then we've still
> > got an issue we need to get to the bottom of.
> > 
> > Cheers,
> > 
> > Dave.
> 
> OK, I played with this a bit, though in all but one case, my XFS stack 
> is new.  A bisect will be done to find when this started.  kernel 
> 3.10.63 here at work is OK, where new xfsprogs is installed.
> 
> In trying out new kernels, I hit the "kernel too old" wall on boot.  
> Therefore, a new system was brewed from slackware-current, and xfstests 
> was installed on top of that.  This result is from the "huge-smp" 
> vendor kernel, as it came from the original binary packages:
> 
> root@kyhorse:/var/lib/xfstests# uname -a
> Linux kyhorse 3.14.24-smp #2 SMP Fri Nov 14 18:34:21 CST 2014 i686 Intel(R) Pentium(R) 4 CPU 1.80GHz GenuineIntel GNU/Linux
> 
> root@kyhorse:/var/lib/xfstests# ./check xfs/261
> FSTYP         -- xfs (non-debug)
> PLATFORM      -- Linux/i686 kyhorse 3.14.24-smp
> MKFS_OPTIONS  -- -f -bsize=4096 /dev/sda7
> MOUNT_OPTIONS -- /dev/sda7 /mnt/scratch
> 
> xfs/261 2s ... 3s
> _check_xfs_filesystem: filesystem on /dev/sda7 is inconsistent (c) (see /var/lib/xfstests/results//xfs/261.full)
> _check_xfs_filesystem: filesystem on /dev/sda7 is inconsistent (r) (see /var/lib/xfstests/results//xfs/261.full)
> Ran: xfs/261
> Passed all 1 tests
> 
> root@kyhorse:/var/lib/xfstests# xfs_repair $SCRQ\b \bATCH_DEV
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan and clear agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
> Phase 5 - rebuild AG headers and trees...
>         - reset superblock...
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> disconnected inode 132, moving to lost+found
> disconnected inode 133, moving to lost+found
> Phase 7 - verify and correct link counts...
> Note - quota info will be regenerated on next quota mount.
> done
> 
> A mount/umount sequence did not change matters.  xfsprogs appears to 
> be version 3.1.11, glancing from the tarball names in the a/ directory 
> here:
> 
> ftp://ftp.slackware.com/pub/slackware/slackware-current/slackware/
> 
> The new setup was just made bootable before I came in to work, and the 
> bisect will start tonight and will hopefully be posted by Monday night.
  I have found the culprit of the bug so no bisection is needed. See my
other email in this thread...
								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Dave Chinner]

On Thu, Dec 18, 2014 at 02:55:55PM +0100, Jan Kara wrote:
> On Thu 18-12-14 11:36:42, Jan Kara wrote:
> > On Thu 18-12-14 08:02:26, Dave Chinner wrote:
> > > On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
> > > >   Hello,
> > > > 
> > > >   in my test KVM with today's Linus' kernel I'm getting xfs_repair
> > > > complaint about disconnected inodes after the test xfs/261 finishes
> > > > (with success). xfs_repair output is like:
> > > > xfs_repair -n /dev/vdb2
> > > > Phase 1 - find and verify superblock...
> > > > Phase 2 - using internal log
> > > >         - scan filesystem freespace and inode maps...
> > > >         - found root inode chunk
> > > > Phase 3 - for each AG...
> > > >         - scan (but don't clear) agi unlinked lists...
> > > >         - process known inodes and perform inode discovery...
> > > >         - agno = 0
> > > >         - agno = 1
> > > >         - agno = 2
> > > >         - agno = 3
> > > >         - process newly discovered inodes...
> > > > Phase 4 - check for duplicate blocks...
> > > >         - setting up duplicate extent list...
> > > >         - check for inodes claiming duplicate blocks...
> > > >         - agno = 0
> > > >         - agno = 1
> > > >         - agno = 2
> > > >         - agno = 3
> > > > No modify flag set, skipping phase 5
> > > > Phase 6 - check inode connectivity...
> > > >         - traversing filesystem ...
> > > >         - traversal finished ...
> > > >         - moving disconnected inodes to lost+found ...
> > > > disconnected inode 132, would move to lost+found
> > > > disconnected inode 133, would move to lost+found
> > > > Phase 7 - verify link counts...
> > > > No modify flag set, skipping filesystem flush and exiting.
> > > > ---
> > > > Given how trivial test xfs/261 is, it seems like created private mtab files
> > > > that also get unlinked don't get added to AGI unlinked list before umount.
> > > > I didn't have a detailed look whether that's possible or not and probably
> > > > won't get to it before Christmas. So I'm sending this just in case someone
> > > > more knowledgeable has ideas earlier...
> > > 
> > > I don't see that here. If you mount/unmount the filesystem, does the
> > > warning go away? i.e. xfs_repair -n ignores the contents of
> > > the log, so if the unlinked list transactions are in the log then
> > > log recovery will make everything good again.
> >   No, the problem is still there after mounting and unmounting the
> > filesystem.
> > 
> > Given what Michael wrote: I'm running xfs_repair version 3.2.1, filesystem
> > is V4.
> > 
> > When I look via xfs_db at the inode I can see nlink is 1 which looks
> > strange. So maybe the problem is somewhere else than I thought:
> > xfs_db> inode 132
> > xfs_db> p
> > core.magic = 0x494e
> > core.mode = 0100000
> > core.version = 2
> > core.format = 2 (extents)
> > core.nlinkv2 = 1
> > core.onlink = 0
> > core.projid_lo = 0
> > core.projid_hi = 0
> > core.uid = 0
> > core.gid = 0
> > core.flushiter = 1
> > core.atime.sec = Thu Dec 18 11:08:55 2014
> > core.atime.nsec = 510013169
> > core.mtime.sec = Thu Dec 18 11:08:55 2014
> > core.mtime.nsec = 510013169
> > core.ctime.sec = Thu Dec 18 11:08:55 2014
> > core.ctime.nsec = 510013169
> > core.size = 0
> > core.nblocks = 1
> > core.extsize = 0
> > core.nextents = 1
> > core.naextents = 0
> > core.forkoff = 0
> > core.aformat = 2 (extents)
> > core.dmevmask = 0
> > core.dmstate = 0
> > core.newrtbm = 0
> > core.prealloc = 0
> > core.realtime = 0
> > core.immutable = 0
> > core.append = 0
> > core.sync = 0
> > core.noatime = 0
> > core.nodump = 0
> > core.rtinherit = 0
> > core.projinherit = 0
> > core.nosymlinks = 0
> > core.extsz = 0
> > core.extszinherit = 0
> > core.nodefrag = 0
> > core.filestream = 0
> > core.gen = 0
> > next_unlinked = null
> > u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,13,1,0]
> > 
> > I have taken xfs_metadump just after test xfs/261 completed and xfs_repair
> > reported error. It is attached.
>   OK, so I understand better what's going on. The detached inodes are
> actually inodes from quota files being created by quotacheck on mount. Test
> xfs/261 first mounts with uquota - that adds user quota ino and quota
> feature just fine. But then it mounts with gquota - now we go through
> different path since xfs_sb_version_hasquota() is already true. We find
> that group quota isn't initialized though and thus we still need to
> allocate inode for group quota file. However the change to sb_gquotaino
> isn't written out because the following test in xfs_sb_quota_to_disk()
> fails:
> 
> 	if ((*fields & XFS_SB_GQUOTINO) &&
>                                 (from->sb_qflags & XFS_GQUOTA_ACCT))
>                 to->sb_gquotino = cpu_to_be64(from->sb_gquotino);
> 
> because from->sb_qflags doesn't have XFS_GQUOTA_ACCT set (only
> mp->m_qflags has XFS_GQUOTA_ACCT set).

I think you've got that wrong, but at the same time got it right. ;)

That is, we get there through xfs_qm_mount_quotas(), which does this
in the the write_changes section:

        spin_lock(&mp->m_sb_lock);
        sbf = mp->m_sb.sb_qflags;
        mp->m_sb.sb_qflags = mp->m_qflags & XFS_MOUNT_QUOTA_ALL;
        spin_unlock(&mp->m_sb_lock);

        if (sbf != (mp->m_qflags & XFS_MOUNT_QUOTA_ALL)) {
                if (xfs_qm_write_sb_changes(mp, XFS_SB_QFLAGS)) {

Hence (from->sb_qflags & XFS_GQUOTA_ACCT) evaluates are true because
we've copied mp->m_qflags into from->sb_qflags before logging the
change. that's the bit you got wrong.

However, the code there only passes XFS_SB_QFLAGS to xfs_log_sb(),
and that means (*fileds & XFS_SB_GQUOTINO) evaluates as false, and
we fail to write the new group quota inode. SO you are right in that
this is the check that is failing. :)

And, again, the reason I wasn't able to reproduce it is that I
tested on a kernel running that "get rid of field based superblock
logging", which meant the above check was simply:

	if (from->sb_qflags & XFS_GQUOTA_ACCT)
		to->sb_gquotino = cpu_to_be64(from->sb_gquotino);

and, from above, we know that this is set appropriately. Hence the
group quota inode was updated and the test did not fail.

FYI, I noticed a bunch of these sb logging field bugs a while back, which
is why I wrote the patch to simply get rid of it. You've found two
of those bugs in the past few days....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Disconnected inodes after test xfs/261

[Michael L. Semon]

On 12/18/14 21:03, Dave Chinner wrote:
> On Thu, Dec 18, 2014 at 02:55:55PM +0100, Jan Kara wrote:
>> On Thu 18-12-14 11:36:42, Jan Kara wrote:
>>> On Thu 18-12-14 08:02:26, Dave Chinner wrote:
>>>> On Wed, Dec 17, 2014 at 08:35:35PM +0100, Jan Kara wrote:
>>>>>   Hello,
>>>>>
>>>>>   in my test KVM with today's Linus' kernel I'm getting xfs_repair
>>>>> complaint about disconnected inodes after the test xfs/261 finishes
>>>>> (with success). xfs_repair output is like:
>>>>> xfs_repair -n /dev/vdb2
>>>>> Phase 1 - find and verify superblock...
>>>>> Phase 2 - using internal log
>>>>>         - scan filesystem freespace and inode maps...
>>>>>         - found root inode chunk
>>>>> Phase 3 - for each AG...
>>>>>         - scan (but don't clear) agi unlinked lists...
>>>>>         - process known inodes and perform inode discovery...
>>>>>         - agno = 0
>>>>>         - agno = 1
>>>>>         - agno = 2
>>>>>         - agno = 3
>>>>>         - process newly discovered inodes...
>>>>> Phase 4 - check for duplicate blocks...
>>>>>         - setting up duplicate extent list...
>>>>>         - check for inodes claiming duplicate blocks...
>>>>>         - agno = 0
>>>>>         - agno = 1
>>>>>         - agno = 2
>>>>>         - agno = 3
>>>>> No modify flag set, skipping phase 5
>>>>> Phase 6 - check inode connectivity...
>>>>>         - traversing filesystem ...
>>>>>         - traversal finished ...
>>>>>         - moving disconnected inodes to lost+found ...
>>>>> disconnected inode 132, would move to lost+found
>>>>> disconnected inode 133, would move to lost+found
>>>>> Phase 7 - verify link counts...
>>>>> No modify flag set, skipping filesystem flush and exiting.
>>>>> ---
>>>>> Given how trivial test xfs/261 is, it seems like created private mtab files
>>>>> that also get unlinked don't get added to AGI unlinked list before umount.
>>>>> I didn't have a detailed look whether that's possible or not and probably
>>>>> won't get to it before Christmas. So I'm sending this just in case someone
>>>>> more knowledgeable has ideas earlier...
>>>>
>>>> I don't see that here. If you mount/unmount the filesystem, does the
>>>> warning go away? i.e. xfs_repair -n ignores the contents of
>>>> the log, so if the unlinked list transactions are in the log then
>>>> log recovery will make everything good again.
>>>   No, the problem is still there after mounting and unmounting the
>>> filesystem.
>>>
>>> Given what Michael wrote: I'm running xfs_repair version 3.2.1, filesystem
>>> is V4.
>>>
>>> When I look via xfs_db at the inode I can see nlink is 1 which looks
>>> strange. So maybe the problem is somewhere else than I thought:
>>> xfs_db> inode 132
>>> xfs_db> p
>>> core.magic = 0x494e
>>> core.mode = 0100000
>>> core.version = 2
>>> core.format = 2 (extents)
>>> core.nlinkv2 = 1
>>> core.onlink = 0
>>> core.projid_lo = 0
>>> core.projid_hi = 0
>>> core.uid = 0
>>> core.gid = 0
>>> core.flushiter = 1
>>> core.atime.sec = Thu Dec 18 11:08:55 2014
>>> core.atime.nsec = 510013169
>>> core.mtime.sec = Thu Dec 18 11:08:55 2014
>>> core.mtime.nsec = 510013169
>>> core.ctime.sec = Thu Dec 18 11:08:55 2014
>>> core.ctime.nsec = 510013169
>>> core.size = 0
>>> core.nblocks = 1
>>> core.extsize = 0
>>> core.nextents = 1
>>> core.naextents = 0
>>> core.forkoff = 0
>>> core.aformat = 2 (extents)
>>> core.dmevmask = 0
>>> core.dmstate = 0
>>> core.newrtbm = 0
>>> core.prealloc = 0
>>> core.realtime = 0
>>> core.immutable = 0
>>> core.append = 0
>>> core.sync = 0
>>> core.noatime = 0
>>> core.nodump = 0
>>> core.rtinherit = 0
>>> core.projinherit = 0
>>> core.nosymlinks = 0
>>> core.extsz = 0
>>> core.extszinherit = 0
>>> core.nodefrag = 0
>>> core.filestream = 0
>>> core.gen = 0
>>> next_unlinked = null
>>> u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,13,1,0]
>>>
>>> I have taken xfs_metadump just after test xfs/261 completed and xfs_repair
>>> reported error. It is attached.
>>   OK, so I understand better what's going on. The detached inodes are
>> actually inodes from quota files being created by quotacheck on mount. Test
>> xfs/261 first mounts with uquota - that adds user quota ino and quota
>> feature just fine. But then it mounts with gquota - now we go through
>> different path since xfs_sb_version_hasquota() is already true. We find
>> that group quota isn't initialized though and thus we still need to
>> allocate inode for group quota file. However the change to sb_gquotaino
>> isn't written out because the following test in xfs_sb_quota_to_disk()
>> fails:
>>
>> 	if ((*fields & XFS_SB_GQUOTINO) &&
>>                                 (from->sb_qflags & XFS_GQUOTA_ACCT))
>>                 to->sb_gquotino = cpu_to_be64(from->sb_gquotino);
>>
>> because from->sb_qflags doesn't have XFS_GQUOTA_ACCT set (only
>> mp->m_qflags has XFS_GQUOTA_ACCT set).
> 
> I think you've got that wrong, but at the same time got it right. ;)
> 
> That is, we get there through xfs_qm_mount_quotas(), which does this
> in the the write_changes section:
> 
>         spin_lock(&mp->m_sb_lock);
>         sbf = mp->m_sb.sb_qflags;
>         mp->m_sb.sb_qflags = mp->m_qflags & XFS_MOUNT_QUOTA_ALL;
>         spin_unlock(&mp->m_sb_lock);
> 
>         if (sbf != (mp->m_qflags & XFS_MOUNT_QUOTA_ALL)) {
>                 if (xfs_qm_write_sb_changes(mp, XFS_SB_QFLAGS)) {
> 
> Hence (from->sb_qflags & XFS_GQUOTA_ACCT) evaluates are true because
> we've copied mp->m_qflags into from->sb_qflags before logging the
> change. that's the bit you got wrong.
> 
> However, the code there only passes XFS_SB_QFLAGS to xfs_log_sb(),
> and that means (*fileds & XFS_SB_GQUOTINO) evaluates as false, and
> we fail to write the new group quota inode. SO you are right in that
> this is the check that is failing. :)
> 
> And, again, the reason I wasn't able to reproduce it is that I
> tested on a kernel running that "get rid of field based superblock
> logging", which meant the above check was simply:
> 
> 	if (from->sb_qflags & XFS_GQUOTA_ACCT)
> 		to->sb_gquotino = cpu_to_be64(from->sb_gquotino);
> 
> and, from above, we know that this is set appropriately. Hence the
> group quota inode was updated and the test did not fail.
> 
> FYI, I noticed a bunch of these sb logging field bugs a while back, which
> is why I wrote the patch to simply get rid of it. You've found two
> of those bugs in the past few days....

And I didn't do very well in hunting things down.  Bisect took me 
back to "xfs: Start using pquotaino from the superblock", but that 
first bisect step took me to 3.14 and probably over the recent 
fixes.  Just chiming in to note that even before then, the new 
xfsprogs is needed for xfs/261 to pass outright.  Also, the patches 
cited elsewhere in this thread didn't seem to solve matters...but 
the second patch had to be placed in by hand.

Thanks!

Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs