xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning

[Andreas Klauer]

Hi,

having some problems growing XFS, using Linux 3.10.1 x86_64, xfsprogs 3.1.11.

The filesystem is on LVM, and I made the volume larger. I did this several 
times before and it never causes any issue.

This time, trying to grow the filesystem failed and it seemed corrupt. 
xfs_repair restored its original size somehow (even though shrinking is 
not supposed to be possible?). Unfortunately I did not save the output 
of the commands the first time around, but the issue is reproducable 
in its current state.

If anyone could tell me what I'm doing wrong?

# mount
> /dev/mapper/lvm-xfs on /mnt/xfs type xfs (rw)

# df -h
> /dev/mapper/lvm-xfs          2.0T  1.9T  102G  95% /mnt/xfs

# blockdev --getsize64 /dev/lvm/xfs
> 2306397437952

# xfs_info /mnt/xfs
> meta-data=/dev/mapper/lvm-xfs    isize=256    agcount=16, agsize=32768000 blks
>          =                       sectsz=4096  attr=2
> data     =                       bsize=4096   blocks=524288000, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal               bsize=4096   blocks=32768, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0

# xfs_growfs /mnt/xfs
> meta-data=/dev/mapper/lvm-xfs    isize=256    agcount=16, agsize=32768000 blks
>          =                       sectsz=4096  attr=2
> data     =                       bsize=4096   blocks=524288000, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal               bsize=4096   blocks=32768, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
> data blocks changed from 524288000 to 563085312

# umount /mnt/xfs
# mount /dev/lvm/xfs /mnt/xfs

# df -h
> /dev/mapper/lvm-xfs          2.1T  1.9T  250G  89% /mnt/xfs

# xfs_info /mnt/xfs
> meta-data=/dev/mapper/lvm-xfs    isize=256    agcount=18, agsize=32768000 blks
>          =                       sectsz=4096  attr=2
> data     =                       bsize=4096   blocks=563085312, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal               bsize=4096   blocks=32768, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0

# umount /mnt/xfs

# xfs_repair -v /dev/lvm/xfs
> Phase 1 - find and verify superblock...
> writing modified primary superblock
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
> primary/secondary superblock 4 conflict - AG superblock geometry info conflicts with filesystem geometry
> reset bad sb for ag 4
> primary/secondary superblock 2 conflict - AG superblock geometry info conflicts with filesystem geometry
> reset bad sb for ag 2
> primary/secondary superblock 1 conflict - AG superblock geometry info conflicts with filesystem geometry
> reset bad sb for ag 1
> primary/secondary superblock 3 conflict - AG superblock geometry info conflicts with filesystem geometry
> reset bad sb for ag 3
> sb_icount 0, counted 8704
> sb_ifree 0, counted 854
> sb_fdblocks 0, counted 26540703
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan and clear agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - agno = 15
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - agno = 15
> Phase 5 - rebuild AG headers and trees...
>         - reset superblock...
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify and correct link counts...
> done

# mount /dev/lvm/xfs /mnt/xfs

# df -h
> /dev/mapper/lvm-xfs          2.0T  1.9T  102G  95% /mnt/xfs

# xfs_growfs /mnt/xfs
> meta-data=/dev/mapper/lvm-xfs    isize=256    agcount=16, agsize=32768000 blks
>          =                       sectsz=4096  attr=2
> data     =                       bsize=4096   blocks=524288000, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal               bsize=4096   blocks=32768, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
> data blocks changed from 524288000 to 563085312
 
Back to square one and rinse and repeat.

1) What does "XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning" mean?

2) Should I reformat or any way to repair/grow properly?   

Thanks
Andreas Klauer

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning

[Dave Chinner]

On Thu, Jul 18, 2013 at 01:04:37PM +0200, Andreas Klauer wrote:
> Hi,
> 
> having some problems growing XFS, using Linux 3.10.1 x86_64, xfsprogs 3.1.11.
> 
> The filesystem is on LVM, and I made the volume larger. I did this several 
> times before and it never causes any issue.
> 
> This time, trying to grow the filesystem failed and it seemed corrupt. 
> xfs_repair restored its original size somehow (even though shrinking is 
> not supposed to be possible?). Unfortunately I did not save the output 
> of the commands the first time around, but the issue is reproducable 
> in its current state.
> 
> If anyone could tell me what I'm doing wrong?
> 
> # mount
> > /dev/mapper/lvm-xfs on /mnt/xfs type xfs (rw)
> 
> # df -h
> > /dev/mapper/lvm-xfs          2.0T  1.9T  102G  95% /mnt/xfs
> 
> # blockdev --getsize64 /dev/lvm/xfs
> > 2306397437952
> 
> # xfs_info /mnt/xfs
> > meta-data=/dev/mapper/lvm-xfs    isize=256    agcount=16, agsize=32768000 blks
> >          =                       sectsz=4096  attr=2
> > data     =                       bsize=4096   blocks=524288000, imaxpct=25
> >          =                       sunit=0      swidth=0 blks
> > naming   =version 2              bsize=4096   ascii-ci=0
> > log      =internal               bsize=4096   blocks=32768, version=2
> >          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> > realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> # xfs_growfs /mnt/xfs
> > meta-data=/dev/mapper/lvm-xfs    isize=256    agcount=16, agsize=32768000 blks
> >          =                       sectsz=4096  attr=2
> > data     =                       bsize=4096   blocks=524288000, imaxpct=25
> >          =                       sunit=0      swidth=0 blks
> > naming   =version 2              bsize=4096   ascii-ci=0
> > log      =internal               bsize=4096   blocks=32768, version=2
> >          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> > realtime =none                   extsz=4096   blocks=0, rtextents=0
> > xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
> > data blocks changed from 524288000 to 563085312

What's in dmesg?

> > Phase 2 - using internal log
> >         - zero log...
> >         - scan filesystem freespace and inode maps...
> > primary/secondary superblock 4 conflict - AG superblock geometry info conflicts with filesystem geometry
> > reset bad sb for ag 4
> > primary/secondary superblock 2 conflict - AG superblock geometry info conflicts with filesystem geometry
> > reset bad sb for ag 2
> > primary/secondary superblock 1 conflict - AG superblock geometry info conflicts with filesystem geometry
> > reset bad sb for ag 1
> > primary/secondary superblock 3 conflict - AG superblock geometry info conflicts with filesystem geometry
> > reset bad sb for ag 3
> > sb_icount 0, counted 8704
> > sb_ifree 0, counted 854
> > sb_fdblocks 0, counted 26540703

So it looks like it got to AG 5 and failed for some reason....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning

[Andreas Klauer]

On Thu, Jul 18, 2013 at 09:13:06PM +1000, Dave Chinner wrote:
> What's in dmesg?

I forgot to check. *blush*

[ 8004.578647] ffff8801d16f5000: 58 46 53 42 00 00 10 00 00 00 00 00 1f 40 00 00  XFSB.........@..
[ 8004.578652] ffff8801d16f5010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 8004.578654] ffff8801d16f5020: cb fe 0d 27 44 d9 43 67 85 17 0a 28 35 68 0e f2  ...'D.Cg...(5h..
[ 8004.578656] ffff8801d16f5030: 00 00 00 00 04 00 00 07 00 00 00 00 00 00 00 c0  ................
[ 8004.578660] XFS (dm-19): Internal error xfs_sb_read_verify at line 730 of file fs/xfs/xfs_mount.c.  Caller 0xffffffff811e99bd

[ 8004.578663] CPU: 1 PID: 80 Comm: kworker/1:1H Not tainted 3.10.1 #1
[ 8004.578665] Hardware name:                  /DP35DP, BIOS DPP3510J.86A.0572.2009.0715.2346 07/15/2009
[ 8004.578671] Workqueue: xfslogd xfs_buf_iodone_work
[ 8004.578674]  ffffffff81655f86 0000000000000072 ffffffff811eb542 ffffffff811e99bd
[ 8004.578677]  ffff8802000002da ffff8802312be5fd ffff8801c67f4a80 0000000000000075
[ 8004.578680]  ffff88021c04f800 0000000000001000 ffffffff8123764c ffffffff811e99bd
[ 8004.578683] Call Trace:
[ 8004.578688]  [<ffffffff81655f86>] ? dump_stack+0xd/0x17
[ 8004.578692]  [<ffffffff811eb542>] ? xfs_corruption_error+0x62/0x90
[ 8004.578700]  [<ffffffff811e99bd>] ? xfs_buf_iodone_work+0x8d/0xb0
[ 8004.578702]  [<ffffffff8123764c>] ? xfs_sb_read_verify+0x11c/0x130
[ 8004.578704]  [<ffffffff811e99bd>] ? xfs_buf_iodone_work+0x8d/0xb0
[ 8004.578706]  [<ffffffff811e99bd>] ? xfs_buf_iodone_work+0x8d/0xb0
[ 8004.578709]  [<ffffffff81087e2a>] ? process_one_work+0x13a/0x3b0
[ 8004.578711]  [<ffffffff81088b96>] ? worker_thread+0x116/0x370
[ 8004.578713]  [<ffffffff81088a80>] ? manage_workers.isra.29+0x290/0x290
[ 8004.578715]  [<ffffffff8108e5c3>] ? kthread+0xb3/0xc0
[ 8004.578718]  [<ffffffff81090000>] ? posix_cpu_timer_set+0xf0/0x300
[ 8004.578719]  [<ffffffff8108e510>] ? kthread_create_on_node+0x120/0x120
[ 8004.578722]  [<ffffffff8165ce2c>] ? ret_from_fork+0x7c/0xb0
[ 8004.578724]  [<ffffffff8108e510>] ? kthread_create_on_node+0x120/0x120
[ 8004.578725] XFS (dm-19): Corruption detected. Unmount and run xfs_repair
[ 8004.578731] XFS (dm-19): metadata I/O error: block 0x4e200000 ("xfs_trans_read_buf_map") error 117 numblks 8
[ 8004.578734] XFS (dm-19): error 117 reading secondary superblock for ag 5

> So it looks like it got to AG 5 and failed for some reason....

Thanks for your quick reply!

I'm also getting panics for other XFS filesystems which I didn't even grow 
nor touch in any other way:

[ 8920.597875] XFS (dm-16): xfs_iread: validation failed for inode 275419712 failed
[ 8920.597880] ffff88014e46a000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[ 8920.597881] ffff88014e46a010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[ 8920.597883] ffff88014e46a020: ff ff ff ff 00 00 00 00 45 30 45 07 00 00 00 00  ........E0E.....
[ 8920.597884] ffff88014e46a030: 4d d5 25 2c 32 a7 01 56 00 00 00 00 00 00 21 01  M.%,2..V......!.
[ 8920.597886] XFS (dm-16): Internal error xfs_iread at line 1062 of file fs/xfs/xfs_inode.c.  Caller 0xffffffff811f0b1e

[ 8920.597889] CPU: 0 PID: 3913 Comm: dar Not tainted 3.10.1 #1
[ 8920.597890] Hardware name:                  /DP35DP, BIOS DPP3510J.86A.0572.2009.0715.2346 07/15/2009
[ 8920.597892]  ffffffff81655f86 000000000000006a ffffffff811eb542 ffffffff811f0b1e
[ 8920.597894]  0000000000000426 ffffffff81699314 0000000000000075 ffff880190bd30c0
[ 8920.597896]  ffff88021ccd9000 0000000000000000 ffffffff8122fe92 ffffffff811f0b1e
[ 8920.597898] Call Trace:
[ 8920.597903]  [<ffffffff81655f86>] ? dump_stack+0xd/0x17
[ 8920.597907]  [<ffffffff811eb542>] ? xfs_corruption_error+0x62/0x90
[ 8920.597910]  [<ffffffff811f0b1e>] ? xfs_iget+0x2de/0x5b0
[ 8920.597913]  [<ffffffff8122fe92>] ? xfs_iread+0xf2/0x2c0
[ 8920.597915]  [<ffffffff811f0b1e>] ? xfs_iget+0x2de/0x5b0
[ 8920.597917]  [<ffffffff811fcd2e>] ? kmem_zone_alloc+0x5e/0xe0
[ 8920.597919]  [<ffffffff811f0b1e>] ? xfs_iget+0x2de/0x5b0
[ 8920.597921]  [<ffffffff811fb557>] ? xfs_lookup+0xb7/0xe0
[ 8920.597923]  [<ffffffff811f5455>] ? xfs_vn_lookup+0x45/0x90
[ 8920.597926]  [<ffffffff81129c83>] ? lookup_dcache+0xa3/0xd0
[ 8920.597928]  [<ffffffff81129ae4>] ? lookup_real+0x14/0x50
[ 8920.597930]  [<ffffffff81129ce2>] ? __lookup_hash+0x32/0x50
[ 8920.597933]  [<ffffffff81654670>] ? lookup_slow+0x3c/0xa2
[ 8920.597935]  [<ffffffff8112c7d1>] ? path_lookupat+0x6c1/0x730
[ 8920.597938]  [<ffffffff810f2385>] ? pagevec_lru_move_fn+0xc5/0xe0
[ 8920.597940]  [<ffffffff810f1b30>] ? pagevec_lookup+0x20/0x20
[ 8920.597942]  [<ffffffff8112c86f>] ? filename_lookup+0x2f/0xd0
[ 8920.597944]  [<ffffffff8112b1b4>] ? getname_flags.part.50+0x84/0x130
[ 8920.597946]  [<ffffffff8112f812>] ? user_path_at_empty+0x92/0x120
[ 8920.597948]  [<ffffffff81124f07>] ? cp_new_stat+0x117/0x130
[ 8920.597950]  [<ffffffff81125110>] ? vfs_fstatat+0x40/0x90
[ 8920.597952]  [<ffffffff811251c2>] ? SYSC_newlstat+0x12/0x30
[ 8920.597955]  [<ffffffff8104462a>] ? syscall_trace_enter+0x1a/0x1f0
[ 8920.597958]  [<ffffffff8165d06c>] ? tracesys+0x7e/0xe2
[ 8920.597959]  [<ffffffff8165d0cb>] ? tracesys+0xdd/0xe2
[ 8920.597961] XFS (dm-16): Corruption detected. Unmount and run xfs_repair
[ 8941.762510] ffff880170014000: 58 46 53 42 00 00 10 00 00 00 00 00 1f 40 00 00  XFSB.........@..
[ 8941.762515] ffff880170014010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 8941.762517] ffff880170014020: cb fe 0d 27 44 d9 43 67 85 17 0a 28 35 68 0e f2  ...'D.Cg...(5h..
[ 8941.762519] ffff880170014030: 00 00 00 00 04 00 00 07 00 00 00 00 00 00 00 c0  ................


That's odd since before 3.10.1 kernel I was using 3.10 and nothing 
like this ever happened. Should I downgrade the kernel?

Regards
Andreas Klauer

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning

[Dave Chinner]

On Thu, Jul 18, 2013 at 01:29:39PM +0200, Andreas Klauer wrote:
> On Thu, Jul 18, 2013 at 09:13:06PM +1000, Dave Chinner wrote:
> > What's in dmesg?
> 
> I forgot to check. *blush*
> 
> [ 8004.578647] ffff8801d16f5000: 58 46 53 42 00 00 10 00 00 00 00 00 1f 40 00 00  XFSB.........@..
> [ 8004.578652] ffff8801d16f5010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
> [ 8004.578654] ffff8801d16f5020: cb fe 0d 27 44 d9 43 67 85 17 0a 28 35 68 0e f2  ...'D.Cg...(5h..
> [ 8004.578656] ffff8801d16f5030: 00 00 00 00 04 00 00 07 00 00 00 00 00 00 00 c0  ................
> [ 8004.578660] XFS (dm-19): Internal error xfs_sb_read_verify at line 730 of file fs/xfs/xfs_mount.c.  Caller 0xffffffff811e99bd
> 
> [ 8004.578663] CPU: 1 PID: 80 Comm: kworker/1:1H Not tainted 3.10.1 #1
> [ 8004.578665] Hardware name:                  /DP35DP, BIOS DPP3510J.86A.0572.2009.0715.2346 07/15/2009
> [ 8004.578671] Workqueue: xfslogd xfs_buf_iodone_work
> [ 8004.578674]  ffffffff81655f86 0000000000000072 ffffffff811eb542 ffffffff811e99bd
> [ 8004.578677]  ffff8802000002da ffff8802312be5fd ffff8801c67f4a80 0000000000000075
> [ 8004.578680]  ffff88021c04f800 0000000000001000 ffffffff8123764c ffffffff811e99bd
> [ 8004.578683] Call Trace:
> [ 8004.578688]  [<ffffffff81655f86>] ? dump_stack+0xd/0x17
> [ 8004.578692]  [<ffffffff811eb542>] ? xfs_corruption_error+0x62/0x90
> [ 8004.578700]  [<ffffffff811e99bd>] ? xfs_buf_iodone_work+0x8d/0xb0
> [ 8004.578702]  [<ffffffff8123764c>] ? xfs_sb_read_verify+0x11c/0x130
> [ 8004.578704]  [<ffffffff811e99bd>] ? xfs_buf_iodone_work+0x8d/0xb0
> [ 8004.578706]  [<ffffffff811e99bd>] ? xfs_buf_iodone_work+0x8d/0xb0
> [ 8004.578709]  [<ffffffff81087e2a>] ? process_one_work+0x13a/0x3b0
> [ 8004.578711]  [<ffffffff81088b96>] ? worker_thread+0x116/0x370
> [ 8004.578713]  [<ffffffff81088a80>] ? manage_workers.isra.29+0x290/0x290
> [ 8004.578715]  [<ffffffff8108e5c3>] ? kthread+0xb3/0xc0
> [ 8004.578718]  [<ffffffff81090000>] ? posix_cpu_timer_set+0xf0/0x300
> [ 8004.578719]  [<ffffffff8108e510>] ? kthread_create_on_node+0x120/0x120
> [ 8004.578722]  [<ffffffff8165ce2c>] ? ret_from_fork+0x7c/0xb0
> [ 8004.578724]  [<ffffffff8108e510>] ? kthread_create_on_node+0x120/0x120
> [ 8004.578725] XFS (dm-19): Corruption detected. Unmount and run xfs_repair
> [ 8004.578731] XFS (dm-19): metadata I/O error: block 0x4e200000 ("xfs_trans_read_buf_map") error 117 numblks 8
> [ 8004.578734] XFS (dm-19): error 117 reading secondary superblock for ag 5
> 
> > So it looks like it got to AG 5 and failed for some reason....

Ok, so the problem is as expected - the secondary superblock in AG 5
is not verifying correctly. Can you run:

# xfs_db -r -c "sb 0" -c p -c "sb 5" -c p <dev>

And post the output?

> Thanks for your quick reply!
> 
> I'm also getting panics for other XFS filesystems which I didn't even grow 
> nor touch in any other way:
> 
> [ 8920.597875] XFS (dm-16): xfs_iread: validation failed for inode 275419712 failed
> [ 8920.597880] ffff88014e46a000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
> [ 8920.597881] ffff88014e46a010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
> [ 8920.597883] ffff88014e46a020: ff ff ff ff 00 00 00 00 45 30 45 07 00 00 00 00  ........E0E.....
> [ 8920.597884] ffff88014e46a030: 4d d5 25 2c 32 a7 01 56 00 00 00 00 00 00 21 01  M.%,2..V......!.
> [ 8920.597886] XFS (dm-16): Internal error xfs_iread at line 1062 of file fs/xfs/xfs_inode.c.  Caller 0xffffffff811f0b1e

Yup, that's a real corruption. Something has trashed a location
where inodes should be on disk.

> That's odd since before 3.10.1 kernel I was using 3.10 and nothing 
> like this ever happened. Should I downgrade the kernel?

There shouldn't be any XFS changes between 3.10.0 and 3.10.1, so I'm
not sure that's your problem. It looks to me like there's
pre-existing corruption on disk, and 3.10 is simply finding it. Have
you recently upgraded from an older kernel (i.e. older than 3.9)?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning

[Andreas.Klauer]

Quoting Dave Chinner <david@fromorbit.com>:
> There shouldn't be any XFS changes between 3.10.0 and 3.10.1, so I'm
> not sure that's your problem. It looks to me like there's
> pre-existing corruption on disk, and 3.10 is simply finding it. Have
> you recently upgraded from an older kernel (i.e. older than 3.9)?

Hey Dave,

thanks again for your help. You may be right about it being some other  
corruption.
Another kernel panic (this time, reiserfs!) caused the box to hang  
entirely, and
I'm in a Ubuntu 13.04 LiveCD now (sending this mail via webmail  
interface). I will link
to a photo of the reiserfs panic at the bottom of the mail in case  
it's of interest.

I did a memtest and SMART selftest (only short so far, will do a long  
one later on),
no errors there, and everything looks fine from the rescue system as  
well, so I'm
not sure what is going on.

The LVM is on LUKS which in turn is on RAID-5 (mdadm) so even a single  
HDD failure
should be covered by the RAID, although the panic/reset caused it to resync.
There was nothing in dmesg that'd point to a HDD or any other IO  
failure though.

> Ok, so the problem is as expected - the secondary superblock in AG 5
> is not verifying correctly. Can you run:
>
> # xfs_db -r -c "sb 0" -c p -c "sb 5" -c p <dev>
>
> And post the output?

These values are from the Ubuntu live / rescue system now.
While I did not modify it - working with snapshots for now -
I'm not sure how useful they are anymore after the reboot.
The growing of the filesystem worked fine for a snapshot,
no errors at all.

magicnum = 0x58465342
blocksize = 4096
dblocks = 524288000
rblocks = 0
rextents = 0
uuid = cbfe0d27-44d9-4367-8517-0a2835680ef2
logstart = 67108871
rootino = 192
rbmino = 193
rsumino = 194
rextsize = 1
agblocks = 32768000
agcount = 16
rbmblocks = 0
logblocks = 32768
versionnum = 0xbca4
sectsize = 4096
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 12
inodelog = 8
inopblog = 4
agblklog = 25
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 8704
ifree = 854
fdblocks = 26540703
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 12
logsectsize = 4096
logsunit = 4096
features2 = 0xa
bad_features2 = 0xa
magicnum = 0x58465342
blocksize = 4096
dblocks = 524288000
rblocks = 0
rextents = 0
uuid = cbfe0d27-44d9-4367-8517-0a2835680ef2
logstart = 67108871
rootino = 192
rbmino = 193
rsumino = 194
rextsize = 1
agblocks = 32768000
agcount = 16
rbmblocks = 0
logblocks = 32768
versionnum = 0xbca4
sectsize = 4096
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 12
inodelog = 8
inopblog = 4
agblklog = 25
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 6272
ifree = 575
fdblocks = 19359822
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 12
logsectsize = 4096
logsunit = 4096
features2 = 0xa
bad_features2 = 0xa


Regards
Andreas Klauer

PS: the reiserfs panic - sorry it's a photo, if there's an easy way to  
make Linux dump such somewhere digitally, I haven't learned of it yet

http://666kb.com/i/cfw9k11348jh7hlmw.png

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning

[Andreas.Klauer]

Quoting Dave Chinner <david@fromorbit.com>:
> Have you recently upgraded from an older kernel (i.e. older than 3.9)?

Sorry, I think I missed this question.

I update about once a month to latest stable from kernel.org.

3.8.5 (April) -> 3.9.0 (May) -> 3.9.4 (June) -> 3.10.0 (July 1st) ->  
3.10.1 (July 15th)

So no "older" kernels really. In case something really goes south I  
have good backups...

I wish it was a HDD failure, then at least I'd know what to do about it. ;)

Regards
Andreas Klauer

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning

[Dave Chinner]

On Thu, Jul 18, 2013 at 04:51:03PM +0200, Andreas.Klauer@metamorpher.de wrote:
> Quoting Dave Chinner <david@fromorbit.com>:
> >Have you recently upgraded from an older kernel (i.e. older than 3.9)?
> 
> Sorry, I think I missed this question.
> 
> I update about once a month to latest stable from kernel.org.

Ok, so the verifiers have been there for a while. Next question
-when was the last time you tried to grow the filesystem?

> 3.8.5 (April) -> 3.9.0 (May) -> 3.9.4 (June) -> 3.10.0 (July 1st) ->
> 3.10.1 (July 15th)
> 
> So no "older" kernels really. In case something really goes south I
> have good backups...

I happy to hear someone say that ;)

As it is, can you do an XFS event trace trace when you attempt to
grow the filesystem and send the output of the trace to me? See
here:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

for where to get trace-cmd and how to run it to gather the event
trace.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs