* Failure growing xfs with linux 3.10.5
@ 2013-08-11 7:11 Michael Maier
2013-08-11 18:36 ` Eric Sandeen
0 siblings, 1 reply; 26+ messages in thread
From: Michael Maier @ 2013-08-11 7:11 UTC (permalink / raw)
To: xfs
Hello!
I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
I tried to grow an existing xfs file system on a backup device and got
the following error:
kernel: [ 3702.275590] ffff88004f308c00: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00 XFSB............
kernel: [ 3702.275597] ffff88004f308c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
kernel: [ 3702.275601] ffff88004f308c20: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1 F.....M.......f.
kernel: [ 3702.275604] ffff88004f308c30: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80 ................
kernel: [ 3702.275610] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file /tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller
0xffffffffa08bd2fd
kernel: [ 3702.275610]
kernel: [ 3702.275617] CPU: 1 PID: 368 Comm: kworker/1:1H Tainted: P O 3.10.5-1.1.g4e0ffc2-desktop #1
kernel: [ 3702.275620] Hardware name: Gigabyte Technology Co., Ltd. GA-990XA-UD3/GA-990XA-UD3, BIOS F13 10/26/2012
kernel: [ 3702.275667] Workqueue: xfslogd xfs_buf_iodone_work [xfs]
kernel: [ 3702.275671] ffffffff815205a5 ffff88022ec52ec0 ffffffffa08bfb82 ffffffffa08bd2fd
kernel: [ 3702.275678] ffff8801000002da 0000000000000000 ffff8801ed868b00 ffff880221fa5000
kernel: [ 3702.275684] 0000000000000075 ffff88004f308c00 ffffffffa0916d77 ffffffffa08bd2fd
kernel: [ 3702.275690] Call Trace:
kernel: [ 3702.275707] [<ffffffff81005957>] dump_trace+0x87/0x380
kernel: [ 3702.275716] [<ffffffff81005d2d>] show_stack_log_lvl+0xdd/0x1e0
kernel: [ 3702.275723] [<ffffffff8100728c>] show_stack+0x1c/0x50
kernel: [ 3702.275753] [<ffffffffa08bfb82>] xfs_corruption_error+0x62/0x90 [xfs]
kernel: [ 3702.275838] [<ffffffffa0916d77>] xfs_sb_read_verify+0x117/0x130 [xfs]
kernel: [ 3702.276020] [<ffffffffa08bd2fd>] xfs_buf_iodone_work+0x8d/0xb0 [xfs]
kernel: [ 3702.276059] [<ffffffff8105c673>] process_one_work+0x153/0x460
kernel: [ 3702.276068] [<ffffffff8105d729>] worker_thread+0x119/0x340
kernel: [ 3702.276076] [<ffffffff810640c6>] kthread+0xc6/0xd0
kernel: [ 3702.276086] [<ffffffff8152b42c>] ret_from_fork+0x7c/0xb0
kernel: [ 3702.276093] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
kernel: [ 3702.276168] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
kernel: [ 3702.276177] XFS (dm-33): error 117 reading secondary superblock for ag 16
I tried to repair (xfsprogs-3.1.11) the FS as suggested, but this didn't help.
The FS relies on LVM, which itself relies on a LUKS partition.
It has been grown a few times before with different kernels < 3.10.
Growing it with 3.9.8 afterwards worked as expected.
Is there meanwhile a solution for this problem?
Some more information about the filesystem after growing
it with 3.9.8 but now running again 3.10.5:
Version of LVM: lvm2-2.02.96
gdisk -l /dev/sdh
GPT fdisk (gdisk) version 0.8.7
Partition table scan:
MBR: hybrid
BSD: not present
APM: not present
GPT: present
Found valid GPT with hybrid MBR; using GPT.
Disk /dev/sdh: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): ....
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2925 sectors (1.4 MiB)
Number Start (sector) End (sector) Size Code Name
1 2048 5860532223 2.7 TiB 8E00 primary
lvdisplay --units k /dev/mapper/backupMy-daten3
--- Logical volume ---
LV Path /dev/backupMy/daten3
LV Name daten3
VG Name backupMy
LV UUID uuid
LV Write Access read/write
LV Creation host, time ,
LV Status available
# open 1
LV Size 1384120320.00 KiB
Current LE 337920
Segments 5
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:33
xfs_info /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=45, agsize=7700480 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=346030080, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=60160, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/backupMy-daten3
1.3T 1.2T 93G 94% /mnt
Now, I tried a xfs_repair (on linux 3.10.5) and got the following:
xfs_repair /dev/mapper/backupMy-daten3
Phase 1 - find and verify superblock...
writing modified primary superblock
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
primary/secondary superblock 11 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 11
primary/secondary superblock 12 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 12
primary/secondary superblock 14 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 14
primary/secondary superblock 10 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 10
primary/secondary superblock 8 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 8
primary/secondary superblock 9 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 9
primary/secondary superblock 5 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 5
primary/secondary superblock 1 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 1
primary/secondary superblock 2 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 2
primary/secondary superblock 3 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 3
primary/secondary superblock 4 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 4
primary/secondary superblock 15 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 15
primary/secondary superblock 13 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 13
primary/secondary superblock 6 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 6
primary/secondary superblock 7 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 7
invalid start block 4471539 in record 1 of bno btree block 41/1
invalid start block 5139463 in record 2 of bno btree block 41/1
invalid start block 6389489 in record 3 of bno btree block 41/1
invalid start block 5139463 in record 1 of cnt btree block 41/2
invalid start block 4471539 in record 2 of cnt btree block 41/2
invalid start block 6389489 in record 3 of cnt btree block 41/2
agf_freeblks 1464854, counted 1 in ag 41
agf_longest 1310991, counted 1 in ag 41
sb_icount 0, counted 6528
sb_ifree 0, counted 665
sb_fdblocks 0, counted 80515
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
inode 1963848080 - bad extent starting block number 348028928, offset 0
correcting nextents for inode 1963848080
bad data fork in inode 1963848080
cleared inode 1963848080
inode 1963848084 - bad extent starting block number 348553216, offset 0
correcting nextents for inode 1963848084
bad data fork in inode 1963848084
cleared inode 1963848084
inode 1963848085 - bad extent starting block number 349077504, offset 0
correcting nextents for inode 1963848085
bad data fork in inode 1963848085
cleared inode 1963848085
inode 1963848087 - bad extent starting block number 349932241, offset 0
correcting nextents for inode 1963848087
bad data fork in inode 1963848087
cleared inode 1963848087
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
- agno = 40
- agno = 41
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 10
- agno = 8
- agno = 9
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 37
- agno = 38
- agno = 39
- agno = 41
- agno = 40
- agno = 36
- agno = 21
entry "file1" at block 3 offset 1936 in directory inode 1486526508 references free inode 1963848080
clearing inode number in entry at offset 1936...
entry "file2" at block 3 offset 2128 in directory inode 1486526508 references free inode 1963848084
clearing inode number in entry at offset 2128...
entry "file3" at block 3 offset 2168 in directory inode 1486526508 references free inode 1963848085
clearing inode number in entry at offset 2168...
entry "file4" at block 3 offset 2240 in directory inode 1486526508 references free inode 1963848087
clearing inode number in entry at offset 2240...
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
bad hash table for directory inode 1486526508 (no data entry): rebuilding
rebuilding directory inode 1486526508
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
result:
All my data copied after the growing under 3.9.8 is lost
and the FS has the original size again before growing it:
df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/backupMy-daten3
1.2T 1.2T 282M 100% /mnt
Doing xfs_repair again gives:
xfs_repair /dev/mapper/backupMy-daten3
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
- agno = 40
- agno = 41
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 19
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
- agno = 40
- agno = 41
- agno = 20
- agno = 5
- agno = 18
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
xfs_info /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=319815680, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=60160, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Doing xfs_growfs again gives:
xfs_growfs /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=319815680, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=60160, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
data blocks changed from 319815680 to 346030080
xfs_info /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=45, agsize=7700480 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=346030080, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=60160, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
df -k /mnt
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/backupMy-daten3
1383879680 1278733572 105146108 93% /mnt
-> The growing of the FS seems to be done anyway :-).
Doing xfs_repair again gives:
xfs_repair /dev/mapper/backupMy-daten3
Phase 1 - find and verify superblock...
writing modified primary superblock
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
primary/secondary superblock 10 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 10
primary/secondary superblock 9 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 9
primary/secondary superblock 11 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 11
primary/secondary superblock 5 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 5
primary/secondary superblock 6 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 6
primary/secondary superblock 14 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 14
primary/secondary superblock 13 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 13
primary/secondary superblock 7 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 7
primary/secondary superblock 8 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 8
primary/secondary superblock 15 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 15
primary/secondary superblock 2 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 2
primary/secondary superblock 4 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 4
primary/secondary superblock 12 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 12
primary/secondary superblock 1 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 1
primary/secondary superblock 3 conflict - AG superblock geometry info conflicts with filesystem geometry
reset bad sb for ag 3
invalid start block 4096000 in record 1 of bno btree block 41/1
invalid start block 4096000 in record 1 of cnt btree block 41/2
agf_freeblks 3604481, counted 1 in ag 41
agf_longest 3604480, counted 1 in ag 41
sb_icount 0, counted 6528
sb_ifree 0, counted 669
sb_fdblocks 0, counted 80515
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
- agno = 40
- agno = 41
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 12
- agno = 6
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
- agno = 40
- agno = 41
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 11
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
-> xfs_growfs has been reverted again :-( because
"AG superblock geometry info conflicts with filesystem geometry"
df -k /mnt
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/backupMy-daten3
1279022080 1278733476 288604 100% /mnt
What should I do now? Waht's wrong?
Thanks,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-11 7:11 Failure growing xfs with linux 3.10.5 Michael Maier
@ 2013-08-11 18:36 ` Eric Sandeen
2013-08-12 16:50 ` Michael Maier
[not found] ` <52090C6C.6060604@allmail.net>
0 siblings, 2 replies; 26+ messages in thread
From: Eric Sandeen @ 2013-08-11 18:36 UTC (permalink / raw)
To: Michael Maier; +Cc: xfs
On 8/11/13 2:11 AM, Michael Maier wrote:
> Hello!
>
> I think I'm facing the same problem as already described here:
> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
Maybe you can try the tracing Dave suggested in that thread?
It certainly does look similar.
<snip>
> xfs_info /mnt
> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
> = sectsz=512 attr=2
> data = bsize=4096 blocks=319815680, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0
> log =internal bsize=4096 blocks=60160, version=2
> = sectsz=512 sunit=0 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
>
>
FWIW, f I make a filesystem with that geometry on 3.10.5, and grow it to the same
size as below, using xfsprogs-3.1.11,
> Doing xfs_growfs again gives:
>
> xfs_growfs /mnt
> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
> = sectsz=512 attr=2
> data = bsize=4096 blocks=319815680, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0
> log =internal bsize=4096 blocks=60160, version=2
> = sectsz=512 sunit=0 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
> data blocks changed from 319815680 to 346030080
it works fine here. I don't know if luks could have something to do with it.
Still, reread the thread you pointed to ,and see if you can gather some
of the info Dave asked for then (which was never provided on that thread,
so it died.)
Thanks,
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-11 18:36 ` Eric Sandeen
@ 2013-08-12 16:50 ` Michael Maier
2013-08-13 0:54 ` Dave Chinner
[not found] ` <52090C6C.6060604@allmail.net>
1 sibling, 1 reply; 26+ messages in thread
From: Michael Maier @ 2013-08-12 16:50 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs
Eric Sandeen wrote:
> On 8/11/13 2:11 AM, Michael Maier wrote:
>> Hello!
>>
>> I think I'm facing the same problem as already described here:
>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>
> Maybe you can try the tracing Dave suggested in that thread?
I sent you a trace.
Meanwhile, I faced another problem on another xfs-file system with linux
3.10.5 which I never saw before. During writing a few bytes to disc, I
got "disc full" and the writing failed.
At the same time, df reported 69G of free space! I ran xfs_repair -n and
got:
xfs_repair -n /dev/mapper/raid0-daten2
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
sb_ifree 591, counted 492
^^^^^^^^^^^^^^^^^^^^^^^^^
What does this mean? How can I get rid of it w/o loosing data? This file
system was created a few days ago and never resized.
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 5
- agno = 1
- agno = 4
- agno = 2
- agno = 7
- agno = 3
- agno = 6
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
Thanks,
regards,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
[not found] ` <52090C6C.6060604@allmail.net>
@ 2013-08-13 0:04 ` Dave Chinner
2013-08-13 15:30 ` Michael Maier
0 siblings, 1 reply; 26+ messages in thread
From: Dave Chinner @ 2013-08-13 0:04 UTC (permalink / raw)
To: Michael Maier; +Cc: Eric Sandeen, xfs
[ re-ccing the list, because finding this is in everyone's interest ]
On Mon, Aug 12, 2013 at 06:25:16PM +0200, Michael Maier wrote:
> Eric Sandeen wrote:
> > On 8/11/13 2:11 AM, Michael Maier wrote:
> >> Hello!
> >>
> >> I think I'm facing the same problem as already described here:
> >> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
> >
> > Maybe you can try the tracing Dave suggested in that thread?
> > It certainly does look similar.
>
> I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).
>
> xfs_growfs /mnt
> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
> = sectsz=512 attr=2
> data = bsize=4096 blocks=319815680, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0
> log =internal bsize=4096 blocks=60160, version=2
> = sectsz=512 sunit=0 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
> data blocks changed from 319815680 to 346030080
>
> The entry in messages was:
>
> Aug 12 18:09:50 dualc kernel: [ 257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00 XFSB............
> Aug 12 18:09:50 dualc kernel: [ 257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> Aug 12 18:09:50 dualc kernel: [ 257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1 F.....M.......f.
> Aug 12 18:09:50 dualc kernel: [ 257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80 ................
> Aug 12 18:09:50 dualc kernel: [ 257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file
> /daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd
.....
> Aug 12 18:09:50 dualc kernel: [ 257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
> Aug 12 18:09:50 dualc kernel: [ 257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
> Aug 12 18:09:50 dualc kernel: [ 257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16
Ok, so that's reading the secondary superblock for AG 16. You're
growing the filesystem from 42 to 45 AGs, so this problem is not
related to the actual grow operation - it's tripping over a problem
that already exists on disk before the grow operation is started.
i.e. this is likely to be a real corruption being seen, and it
happened some time in the distant past and so we probably won't ever
be able to pinpoint the cause of the problem.
That said, let's have a look at the broken superblock. Can you post
the output of the commands:
# xfs_db -r -c "sb 16" -c p <dev>
and
# xfs_db -r -c "sb 16" -c "type data" -c p <dev>
so we can see the exact contents of that superblock?
FWIW, how many times has this filesystem ben grown? Did it start
with only 32 AGs (i.e. 10TB in size)?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-12 16:50 ` Michael Maier
@ 2013-08-13 0:54 ` Dave Chinner
2013-08-13 14:55 ` Michael Maier
0 siblings, 1 reply; 26+ messages in thread
From: Dave Chinner @ 2013-08-13 0:54 UTC (permalink / raw)
To: Michael Maier; +Cc: Eric Sandeen, xfs
On Mon, Aug 12, 2013 at 06:50:55PM +0200, Michael Maier wrote:
> Eric Sandeen wrote:
> > On 8/11/13 2:11 AM, Michael Maier wrote:
> >> Hello!
> >>
> >> I think I'm facing the same problem as already described here:
> >> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
> >
> > Maybe you can try the tracing Dave suggested in that thread?
>
> I sent you a trace.
>
>
>
>
> Meanwhile, I faced another problem on another xfs-file system with linux
> 3.10.5 which I never saw before. During writing a few bytes to disc, I
> got "disc full" and the writing failed.
>
> At the same time, df reported 69G of free space! I ran xfs_repair -n and
> got:
>
>
> xfs_repair -n /dev/mapper/raid0-daten2
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
> - scan filesystem freespace and inode maps...
> sb_ifree 591, counted 492
> ^^^^^^^^^^^^^^^^^^^^^^^^^
> What does this mean? How can I get rid of it w/o loosing data? This file
> system was created a few days ago and never resized.
Superblock inode counting is lazy - it can get out of sync in after
an unclean shutdown, but generally mounting a dirty filesystem will
result in it being recalculated rather than trusted to be correct.
So there's nothing to worry about here.
> Phase 7 - verify link counts...
> No modify flag set, skipping filesystem flush and exiting.
Ok, - it was a no-modify run, so it wouldn't have complained about a
dirty log needing replay. Hence the counters are probably out
because the filesystem has a dirty log.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-13 0:54 ` Dave Chinner
@ 2013-08-13 14:55 ` Michael Maier
2013-08-14 5:43 ` Dave Chinner
0 siblings, 1 reply; 26+ messages in thread
From: Michael Maier @ 2013-08-13 14:55 UTC (permalink / raw)
To: Dave Chinner; +Cc: Eric Sandeen, xfs
Dave Chinner wrote:
> On Mon, Aug 12, 2013 at 06:50:55PM +0200, Michael Maier wrote:
>> Meanwhile, I faced another problem on another xfs-file system with linux
>> 3.10.5 which I never saw before. During writing a few bytes to disc, I
>> got "disc full" and the writing failed.
>>
>> At the same time, df reported 69G of free space! I ran xfs_repair -n and
>> got:
>>
>>
>> xfs_repair -n /dev/mapper/raid0-daten2
>> Phase 1 - find and verify superblock...
>> Phase 2 - using internal log
>> - scan filesystem freespace and inode maps...
>> sb_ifree 591, counted 492
>> ^^^^^^^^^^^^^^^^^^^^^^^^^
>> What does this mean? How can I get rid of it w/o loosing data? This file
>> system was created a few days ago and never resized.
>
> Superblock inode counting is lazy - it can get out of sync in after
> an unclean shutdown, but generally mounting a dirty filesystem will
> result in it being recalculated rather than trusted to be correct.
> So there's nothing to worry about here.
When will it be self healed? I still can see it today after 4 remounts!
This is strange and I can't use the free space, which I need! How can it
be forced to be repaired w/o data loss?
Thanks,
kind regards,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-13 0:04 ` Dave Chinner
@ 2013-08-13 15:30 ` Michael Maier
2013-08-14 5:53 ` Stan Hoeppner
2013-08-14 6:20 ` Dave Chinner
0 siblings, 2 replies; 26+ messages in thread
From: Michael Maier @ 2013-08-13 15:30 UTC (permalink / raw)
To: Dave Chinner; +Cc: Eric Sandeen, xfs
Dave Chinner wrote:
> [ re-ccing the list, because finding this is in everyone's interest ]
>
> On Mon, Aug 12, 2013 at 06:25:16PM +0200, Michael Maier wrote:
>> Eric Sandeen wrote:
>>> On 8/11/13 2:11 AM, Michael Maier wrote:
>>>> Hello!
>>>>
>>>> I think I'm facing the same problem as already described here:
>>>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>>>
>>> Maybe you can try the tracing Dave suggested in that thread?
>>> It certainly does look similar.
>>
>> I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).
>>
>> xfs_growfs /mnt
>> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
>> = sectsz=512 attr=2
>> data = bsize=4096 blocks=319815680, imaxpct=25
>> = sunit=0 swidth=0 blks
>> naming =version 2 bsize=4096 ascii-ci=0
>> log =internal bsize=4096 blocks=60160, version=2
>> = sectsz=512 sunit=0 blks, lazy-count=1
>> realtime =none extsz=4096 blocks=0, rtextents=0
>> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
>> data blocks changed from 319815680 to 346030080
>>
>> The entry in messages was:
>>
>> Aug 12 18:09:50 dualc kernel: [ 257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00 XFSB............
>> Aug 12 18:09:50 dualc kernel: [ 257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> Aug 12 18:09:50 dualc kernel: [ 257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1 F.....M.......f.
>> Aug 12 18:09:50 dualc kernel: [ 257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80 ................
>> Aug 12 18:09:50 dualc kernel: [ 257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file
>> /daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd
> .....
>> Aug 12 18:09:50 dualc kernel: [ 257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
>> Aug 12 18:09:50 dualc kernel: [ 257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
>> Aug 12 18:09:50 dualc kernel: [ 257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16
>
> Ok, so that's reading the secondary superblock for AG 16. You're
> growing the filesystem from 42 to 45 AGs, so this problem is not
> related to the actual grow operation - it's tripping over a problem
> that already exists on disk before the grow operation is started.
> i.e. this is likely to be a real corruption being seen, and it
> happened some time in the distant past and so we probably won't ever
> be able to pinpoint the cause of the problem.
>
> That said, let's have a look at the broken superblock. Can you post
> the output of the commands:
>
> # xfs_db -r -c "sb 16" -c p <dev>
done after the failed growfs mentioned above:
magicnum = 0x58465342
blocksize = 4096
dblocks = 319815680
rblocks = 0
rextents = 0
uuid = 4691c680-a9a9-4d8c-8fe2-18fde87f66e1
logstart = 67108868
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 7700480
agcount = 42
rbmblocks = 0
logblocks = 60160
versionnum = 0xb4a4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 23
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 6464
ifree = 631
fdblocks = 1124026
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 1
features2 = 0xa
bad_features2 = 0xa
>
> and
>
> # xfs_db -r -c "sb 16" -c "type data" -c p <dev>
000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000
020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080
040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000
060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019
080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000
0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000
0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e
0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>
> so we can see the exact contents of that superblock?
>
> FWIW, how many times has this filesystem ben grown?
I can't say for sure, about 4 or 5 times?
> Did it start
> with only 32 AGs (i.e. 10TB in size)?
10TB? No. The device just has 3 TB. You most probably meant 10GB?
I'm not sure, but it definitely started with > 100GB.
Thanks,
kind regards,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-13 14:55 ` Michael Maier
@ 2013-08-14 5:43 ` Dave Chinner
2013-08-14 15:16 ` Michael Maier
0 siblings, 1 reply; 26+ messages in thread
From: Dave Chinner @ 2013-08-14 5:43 UTC (permalink / raw)
To: Michael Maier; +Cc: Eric Sandeen, xfs
On Tue, Aug 13, 2013 at 04:55:00PM +0200, Michael Maier wrote:
> Dave Chinner wrote:
> > On Mon, Aug 12, 2013 at 06:50:55PM +0200, Michael Maier wrote:
> >> Meanwhile, I faced another problem on another xfs-file system with linux
> >> 3.10.5 which I never saw before. During writing a few bytes to disc, I
> >> got "disc full" and the writing failed.
> >>
> >> At the same time, df reported 69G of free space! I ran xfs_repair -n and
> >> got:
> >>
> >>
> >> xfs_repair -n /dev/mapper/raid0-daten2
> >> Phase 1 - find and verify superblock...
> >> Phase 2 - using internal log
> >> - scan filesystem freespace and inode maps...
> >> sb_ifree 591, counted 492
> >> ^^^^^^^^^^^^^^^^^^^^^^^^^
> >> What does this mean? How can I get rid of it w/o loosing data? This file
> >> system was created a few days ago and never resized.
> >
> > Superblock inode counting is lazy - it can get out of sync in after
> > an unclean shutdown, but generally mounting a dirty filesystem will
> > result in it being recalculated rather than trusted to be correct.
> > So there's nothing to worry about here.
>
> When will it be self healed?
that depends on whether there's actually a problem. Like I said in
the part you snipped off - if you ran xfs_repair -n on filesystem
that needs log recovery that accounting difference is expected.
> I still can see it today after 4 remounts!
See what?
> This is strange and I can't use the free space, which I need! How can it
> be forced to be repaired w/o data loss?
The above is complaining about a free inode count mismatch, not a
problem about free space being wrong. What problem are you actually
having?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-13 15:30 ` Michael Maier
@ 2013-08-14 5:53 ` Stan Hoeppner
2013-08-14 15:05 ` Michael Maier
2013-08-14 6:20 ` Dave Chinner
1 sibling, 1 reply; 26+ messages in thread
From: Stan Hoeppner @ 2013-08-14 5:53 UTC (permalink / raw)
To: Michael Maier; +Cc: Eric Sandeen, xfs
On 8/13/2013 10:30 AM, Michael Maier wrote:
> Dave Chinner wrote:
>> [ re-ccing the list, because finding this is in everyone's interest ]
>>
>> On Mon, Aug 12, 2013 at 06:25:16PM +0200, Michael Maier wrote:
>>> Eric Sandeen wrote:
>>>> On 8/11/13 2:11 AM, Michael Maier wrote:
>>>>> Hello!
>>>>>
>>>>> I think I'm facing the same problem as already described here:
>>>>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>>>>
>>>> Maybe you can try the tracing Dave suggested in that thread?
>>>> It certainly does look similar.
>>>
>>> I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).
>>>
>>> xfs_growfs /mnt
>>>>>>>>> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
>>> = sectsz=512 attr=2
>>> data = bsize=4096 blocks=319815680, imaxpct=25
>>> = sunit=0 swidth=0 blks
>>> naming =version 2 bsize=4096 ascii-ci=0
>>> log =internal bsize=4096 blocks=60160, version=2
>>> = sectsz=512 sunit=0 blks, lazy-count=1
>>> realtime =none extsz=4096 blocks=0, rtextents=0
>>> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
>>> data blocks changed from 319815680 to 346030080
>>>
>>> The entry in messages was:
>>>
>>> Aug 12 18:09:50 dualc kernel: [ 257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00 XFSB............
>>> Aug 12 18:09:50 dualc kernel: [ 257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>> Aug 12 18:09:50 dualc kernel: [ 257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1 F.....M.......f.
>>> Aug 12 18:09:50 dualc kernel: [ 257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80 ................
>>> Aug 12 18:09:50 dualc kernel: [ 257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file
>>> /daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd
>> .....
>>> Aug 12 18:09:50 dualc kernel: [ 257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
>>> Aug 12 18:09:50 dualc kernel: [ 257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
>>> Aug 12 18:09:50 dualc kernel: [ 257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16
>>
>> Ok, so that's reading the secondary superblock for AG 16. You're
>> growing the filesystem from 42 to 45 AGs, so this problem is not
The AGs are ~30GB. Going from 42 to 45 grows the XFS by 90GB. Michael,
how much were you attempting to grow? Surely more than 90GB.
>> related to the actual grow operation - it's tripping over a problem
>> that already exists on disk before the grow operation is started.
>> i.e. this is likely to be a real corruption being seen, and it
>> happened some time in the distant past and so we probably won't ever
>> be able to pinpoint the cause of the problem.
>>
>> That said, let's have a look at the broken superblock. Can you post
>> the output of the commands:
>>
>> # xfs_db -r -c "sb 16" -c p <dev>
>
> done after the failed growfs mentioned above:
>
> magicnum = 0x58465342
> blocksize = 4096
> dblocks = 319815680
> rblocks = 0
> rextents = 0
> uuid = 4691c680-a9a9-4d8c-8fe2-18fde87f66e1
> logstart = 67108868
> rootino = 128
> rbmino = 129
> rsumino = 130
> rextsize = 1
> agblocks = 7700480
> agcount = 42
> rbmblocks = 0
> logblocks = 60160
> versionnum = 0xb4a4
> sectsize = 512
> inodesize = 256
> inopblock = 16
> fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
> blocklog = 12
> sectlog = 9
> inodelog = 8
> inopblog = 4
> agblklog = 23
> rextslog = 0
> inprogress = 0
> imax_pct = 25
> icount = 6464
> ifree = 631
> fdblocks = 1124026
> frextents = 0
> uquotino = 0
> gquotino = 0
> qflags = 0
> flags = 0
> shared_vn = 0
> inoalignmt = 2
> unit = 0
> width = 0
> dirblklog = 0
> logsectlog = 0
> logsectsize = 0
> logsunit = 1
> features2 = 0xa
> bad_features2 = 0xa
>
>>
>> and
>>
>> # xfs_db -r -c "sb 16" -c "type data" -c p <dev>
>
> 000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000
> 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080
> 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000
> 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019
> 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000
> 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000
> 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e
> 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>
>> so we can see the exact contents of that superblock?
>>
>> FWIW, how many times has this filesystem ben grown?
>
> I can't say for sure, about 4 or 5 times?
>
>> Did it start
>> with only 32 AGs (i.e. 10TB in size)?
>
> 10TB? No. The device just has 3 TB. You most probably meant 10GB?
> I'm not sure, but it definitely started with > 100GB.
According to your xfs_info output that I highlighted above, and assuming
my math here is correct,
(((7700480*4096)/1048576)*42)= 1,263,360 GB or ~1.23 TB
this filesystem was 1.23 TB w/42 AGs before the grow operation.
Assuming defaults were used during mkfs.xfs it would appear the initial
size of this filesystem was ~120GB. And it would appear it has been
grown to ~10x its original size, and from 4 AGs to 42 AGs. That seems
like a lot of growth, to me. And Dave states the latest grow operation
was to 45 AGs, which would yield a ~1.32TB filesystem, not 3TB.
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-13 15:30 ` Michael Maier
2013-08-14 5:53 ` Stan Hoeppner
@ 2013-08-14 6:20 ` Dave Chinner
2013-08-14 16:20 ` Michael Maier
2013-08-14 16:51 ` Eric Sandeen
1 sibling, 2 replies; 26+ messages in thread
From: Dave Chinner @ 2013-08-14 6:20 UTC (permalink / raw)
To: Michael Maier; +Cc: Eric Sandeen, xfs
On Tue, Aug 13, 2013 at 05:30:58PM +0200, Michael Maier wrote:
> Dave Chinner wrote:
> > [ re-ccing the list, because finding this is in everyone's interest ]
> >
> > On Mon, Aug 12, 2013 at 06:25:16PM +0200, Michael Maier wrote:
> >> Eric Sandeen wrote:
> >>> On 8/11/13 2:11 AM, Michael Maier wrote:
> >>>> Hello!
> >>>>
> >>>> I think I'm facing the same problem as already described here:
> >>>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
> >>>
> >>> Maybe you can try the tracing Dave suggested in that thread?
> >>> It certainly does look similar.
> >>
> >> I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).
> >>
> >> xfs_growfs /mnt
> >> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
> >> = sectsz=512 attr=2
> >> data = bsize=4096 blocks=319815680, imaxpct=25
> >> = sunit=0 swidth=0 blks
> >> naming =version 2 bsize=4096 ascii-ci=0
> >> log =internal bsize=4096 blocks=60160, version=2
> >> = sectsz=512 sunit=0 blks, lazy-count=1
> >> realtime =none extsz=4096 blocks=0, rtextents=0
> >> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
> >> data blocks changed from 319815680 to 346030080
> >>
> >> The entry in messages was:
> >>
> >> Aug 12 18:09:50 dualc kernel: [ 257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00 XFSB............
> >> Aug 12 18:09:50 dualc kernel: [ 257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> >> Aug 12 18:09:50 dualc kernel: [ 257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1 F.....M.......f.
> >> Aug 12 18:09:50 dualc kernel: [ 257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80 ................
> >> Aug 12 18:09:50 dualc kernel: [ 257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file
> >> /daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd
> > .....
> >> Aug 12 18:09:50 dualc kernel: [ 257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
> >> Aug 12 18:09:50 dualc kernel: [ 257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
> >> Aug 12 18:09:50 dualc kernel: [ 257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16
> >
> > Ok, so that's reading the secondary superblock for AG 16. You're
> > growing the filesystem from 42 to 45 AGs, so this problem is not
> > related to the actual grow operation - it's tripping over a problem
> > that already exists on disk before the grow operation is started.
> > i.e. this is likely to be a real corruption being seen, and it
> > happened some time in the distant past and so we probably won't ever
> > be able to pinpoint the cause of the problem.
> >
> > That said, let's have a look at the broken superblock. Can you post
> > the output of the commands:
> >
> > # xfs_db -r -c "sb 16" -c p <dev>
>
> done after the failed growfs mentioned above:
Looks fine....
> > and
> >
> > # xfs_db -r -c "sb 16" -c "type data" -c p <dev>
>
> 000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000
> 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080
> 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000
> 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019
> 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000
> 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000
> 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e
> 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
> 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
There's your problem - the empty space in the superblock is supposed
to be zero. mkfs zeros it and we rely on it being zero for various
reasons.
And one of those reasons is that we use the fact it shoul dbe zero
to determine if we should be checking the CRC of the superblock.
That is if there's a single bit error in the superblock and we are
missing the correct bit in the version numbers that say CRCs are
enabled, we use the fact that the superblock CRC field - which your
filesystem knowns nothing about - should be zero to validate that
the CRC feature bit is correctly set. The above superblock will
indicate that there is a CRC set on the superblock, find the
necessary version number is not correct, and so therefore we have a
corruption in that superblock that the kernel code cannot handle
without a user telling it what is correct.
So, the fact grwofs is failing is actually the correct behaviour for
the filesystem to have in this case - the superblock is corrupt,
just not obviously so.
> > so we can see the exact contents of that superblock?
> >
> > FWIW, how many times has this filesystem ben grown?
>
> I can't say for sure, about 4 or 5 times?
>
> > Did it start
> > with only 32 AGs (i.e. 10TB in size)?
>
> 10TB? No. The device just has 3 TB. You most probably meant 10GB?
> I'm not sure, but it definitely started with > 100GB.
I misplaced a digit A block size of 4096 bytes and:
agcount=42, agsize=7700480 blks
So the filesystem size is 42 * 7700480 * 4096 = 1.26TB.
The question I'm asking is how many AGs did the filesystem start
with, because this:
commit 1375cb65e87b327a8dd4f920c3e3d837fb40e9c2
Author: Dave Chinner <dchinner@redhat.com>
Date: Tue Oct 9 14:50:52 2012 +1100
xfs: growfs: don't read garbage for new secondary superblocks
When updating new secondary superblocks in a growfs operation, the
superblock buffer is read from the newly grown region of the
underlying device. This is not guaranteed to be zero, so violates
the underlying assumption that the unused parts of superblocks are
zero filled. Get a new buffer for these secondary superblocks to
ensure that the unused regions are zero filled correctly.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Carlos Maiolino <cmaiolino@redhat.com>
Signed-off-by: Ben Myers <bpm@sgi.com>
Is the only possible reason I can think of that would result in
non-zero empty space in a secondary superblock. And that implies
that the filesystem started with 16 AGs or less, and was grown with
an older kernel with this bug in it.
If it makes you feel any better, the bug that caused this had been
in the code for 15+ years and you are the first person I know of to
have ever hit it....
xfs_repair doesn't appear to have any checks in it to detect this
situation or repair it - there are some conditions for zeroing the
unused parts of a superblock, but they are focussed around detecting
and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15
years ago.
Hence looks like we'll need some new xfs_repair functionality to fix
this. It might take me a little while to get you a fix - perhaps
someone else with a little bit of spare time could get it done
sooner than I can. Anyone?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 5:53 ` Stan Hoeppner
@ 2013-08-14 15:05 ` Michael Maier
2013-08-14 17:31 ` Stan Hoeppner
0 siblings, 1 reply; 26+ messages in thread
From: Michael Maier @ 2013-08-14 15:05 UTC (permalink / raw)
To: stan; +Cc: Eric Sandeen, xfs
Stan Hoeppner schrieb:
> On 8/13/2013 10:30 AM, Michael Maier wrote:
>> Dave Chinner wrote:
>>> [ re-ccing the list, because finding this is in everyone's interest ]
>>>
>>> On Mon, Aug 12, 2013 at 06:25:16PM +0200, Michael Maier wrote:
>>>> Eric Sandeen wrote:
>>>>> On 8/11/13 2:11 AM, Michael Maier wrote:
>>>>>> Hello!
>>>>>>
>>>>>> I think I'm facing the same problem as already described here:
>>>>>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>>>>>
>>>>> Maybe you can try the tracing Dave suggested in that thread?
>>>>> It certainly does look similar.
>>>>
>>>> I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).
>>>>
>>>> xfs_growfs /mnt
>
>
>>>>>>>>>> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
>
>
>>>> = sectsz=512 attr=2
>>>> data = bsize=4096 blocks=319815680, imaxpct=25
>>>> = sunit=0 swidth=0 blks
>>>> naming =version 2 bsize=4096 ascii-ci=0
>>>> log =internal bsize=4096 blocks=60160, version=2
>>>> = sectsz=512 sunit=0 blks, lazy-count=1
>>>> realtime =none extsz=4096 blocks=0, rtextents=0
>>>> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
>>>> data blocks changed from 319815680 to 346030080
>>>>
>>>> The entry in messages was:
>>>>
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00 XFSB............
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1 F.....M.......f.
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80 ................
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file
>>>> /daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd
>>> .....
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16
>>>
>>> Ok, so that's reading the secondary superblock for AG 16. You're
>>> growing the filesystem from 42 to 45 AGs, so this problem is not
>
> The AGs are ~30GB. Going from 42 to 45 grows the XFS by 90GB. Michael,
> how much were you attempting to grow? Surely more than 90GB.
>
>>> related to the actual grow operation - it's tripping over a problem
>>> that already exists on disk before the grow operation is started.
>>> i.e. this is likely to be a real corruption being seen, and it
>>> happened some time in the distant past and so we probably won't ever
>>> be able to pinpoint the cause of the problem.
>>>
>>> That said, let's have a look at the broken superblock. Can you post
>>> the output of the commands:
>>>
>>> # xfs_db -r -c "sb 16" -c p <dev>
>>
>> done after the failed growfs mentioned above:
>>
>> magicnum = 0x58465342
>> blocksize = 4096
>> dblocks = 319815680
>> rblocks = 0
>> rextents = 0
>> uuid = 4691c680-a9a9-4d8c-8fe2-18fde87f66e1
>> logstart = 67108868
>> rootino = 128
>> rbmino = 129
>> rsumino = 130
>> rextsize = 1
>> agblocks = 7700480
>> agcount = 42
>> rbmblocks = 0
>> logblocks = 60160
>> versionnum = 0xb4a4
>> sectsize = 512
>> inodesize = 256
>> inopblock = 16
>> fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
>> blocklog = 12
>> sectlog = 9
>> inodelog = 8
>> inopblog = 4
>> agblklog = 23
>> rextslog = 0
>> inprogress = 0
>> imax_pct = 25
>> icount = 6464
>> ifree = 631
>> fdblocks = 1124026
>> frextents = 0
>> uquotino = 0
>> gquotino = 0
>> qflags = 0
>> flags = 0
>> shared_vn = 0
>> inoalignmt = 2
>> unit = 0
>> width = 0
>> dirblklog = 0
>> logsectlog = 0
>> logsectsize = 0
>> logsunit = 1
>> features2 = 0xa
>> bad_features2 = 0xa
>>
>>>
>>> and
>>>
>>> # xfs_db -r -c "sb 16" -c "type data" -c p <dev>
>>
>> 000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000
>> 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080
>> 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000
>> 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019
>> 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000
>> 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000
>> 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e
>> 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>>
>>> so we can see the exact contents of that superblock?
>>>
>>> FWIW, how many times has this filesystem ben grown?
>>
>> I can't say for sure, about 4 or 5 times?
>>
>>> Did it start
>>> with only 32 AGs (i.e. 10TB in size)?
>>
>> 10TB? No. The device just has 3 TB. You most probably meant 10GB?
>> I'm not sure, but it definitely started with > 100GB.
>
> According to your xfs_info output that I highlighted above, and assuming
> my math here is correct,
>
> (((7700480*4096)/1048576)*42)= 1,263,360 GB or ~1.23 TB
>
> this filesystem was 1.23 TB w/42 AGs before the grow operation.
> Assuming defaults were used during mkfs.xfs it would appear the initial
> size of this filesystem was ~120GB. And it would appear it has been
> grown to ~10x its original size, and from 4 AGs to 42 AGs. That seems
> like a lot of growth, to me. And Dave states the latest grow operation
> was to 45 AGs, which would yield a ~1.32TB filesystem, not 3TB.
He wrote about 10TB (=Terra(!) Byte). The entire HD only has 3 TB.
10TB is impossible!
The last growfs was about 100GB. The FS was grown lots of times
since the initial creation one or two years ago.
About the sizes before the last grow and after it: see
http://www.spinics.net/lists/xfs/msg21032.html
Thanks,
regards,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 5:43 ` Dave Chinner
@ 2013-08-14 15:16 ` Michael Maier
2013-08-15 0:58 ` Dave Chinner
0 siblings, 1 reply; 26+ messages in thread
From: Michael Maier @ 2013-08-14 15:16 UTC (permalink / raw)
To: Dave Chinner; +Cc: Eric Sandeen, xfs
Dave Chinner wrote:
> On Tue, Aug 13, 2013 at 04:55:00PM +0200, Michael Maier wrote:
>> Dave Chinner wrote:
>>> On Mon, Aug 12, 2013 at 06:50:55PM +0200, Michael Maier wrote:
>>>> Meanwhile, I faced another problem on another xfs-file system with linux
>>>> 3.10.5 which I never saw before. During writing a few bytes to disc, I
>>>> got "disc full" and the writing failed.
>>>>
>>>> At the same time, df reported 69G of free space! I ran xfs_repair -n and
>>>> got:
>>>>
>>>>
>>>> xfs_repair -n /dev/mapper/raid0-daten2
>>>> Phase 1 - find and verify superblock...
>>>> Phase 2 - using internal log
>>>> - scan filesystem freespace and inode maps...
>>>> sb_ifree 591, counted 492
>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^
>>>> What does this mean? How can I get rid of it w/o loosing data? This file
>>>> system was created a few days ago and never resized.
>>>
>>> Superblock inode counting is lazy - it can get out of sync in after
>>> an unclean shutdown, but generally mounting a dirty filesystem will
>>> result in it being recalculated rather than trusted to be correct.
>>> So there's nothing to worry about here.
>>
>> When will it be self healed?
>
> that depends on whether there's actually a problem. Like I said in
> the part you snipped off - if you ran xfs_repair -n on filesystem
> that needs log recovery that accounting difference is expected.
I know, that option -n doesn't do anything. It was intended, because
xfs_repair destroyed a lot of data when applied at the other problem I
have _and_ it repaired nothing at the same time! The other problem isn't
fixed at all although xfs_repair was used w/o -n.
That's why am I asking if a real xfs_repair will fix this problem in
this case _w/o_ loosing any data.
>
>> I still can see it today after 4 remounts!
>
> See what?
The problem
sb_ifree 591, counted 492
^^^^^^^^^^^^^^^^^^^^^^^^^
>> This is strange and I can't use the free space, which I need! How can it
>> be forced to be repaired w/o data loss?
>
> The above is complaining about a free inode count mismatch, not a
> problem about free space being wrong. What problem are you actually
> having?
The application, which wanted to write a few bytes gets a "disk full"
error although df -h reports 69GB of free space.
Thanks,
regards,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 6:20 ` Dave Chinner
@ 2013-08-14 16:20 ` Michael Maier
2013-08-14 16:37 ` Eric Sandeen
2013-08-15 17:18 ` Eric Sandeen
2013-08-14 16:51 ` Eric Sandeen
1 sibling, 2 replies; 26+ messages in thread
From: Michael Maier @ 2013-08-14 16:20 UTC (permalink / raw)
To: Dave Chinner; +Cc: Eric Sandeen, xfs
Dave Chinner wrote:
> On Tue, Aug 13, 2013 at 05:30:58PM +0200, Michael Maier wrote:
>> Dave Chinner wrote:
>>> [ re-ccing the list, because finding this is in everyone's interest ]
>>>
>>> On Mon, Aug 12, 2013 at 06:25:16PM +0200, Michael Maier wrote:
>>>> Eric Sandeen wrote:
>>>>> On 8/11/13 2:11 AM, Michael Maier wrote:
>>>>>> Hello!
>>>>>>
>>>>>> I think I'm facing the same problem as already described here:
>>>>>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>>>>>
>>>>> Maybe you can try the tracing Dave suggested in that thread?
>>>>> It certainly does look similar.
>>>>
>>>> I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).
>>>>
>>>> xfs_growfs /mnt
>>>> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
>>>> = sectsz=512 attr=2
>>>> data = bsize=4096 blocks=319815680, imaxpct=25
>>>> = sunit=0 swidth=0 blks
>>>> naming =version 2 bsize=4096 ascii-ci=0
>>>> log =internal bsize=4096 blocks=60160, version=2
>>>> = sectsz=512 sunit=0 blks, lazy-count=1
>>>> realtime =none extsz=4096 blocks=0, rtextents=0
>>>> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
>>>> data blocks changed from 319815680 to 346030080
>>>>
>>>> The entry in messages was:
>>>>
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00 XFSB............
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1 F.....M.......f.
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80 ................
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file
>>>> /daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd
>>> .....
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
>>>> Aug 12 18:09:50 dualc kernel: [ 257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16
>>>
>>> Ok, so that's reading the secondary superblock for AG 16. You're
>>> growing the filesystem from 42 to 45 AGs, so this problem is not
>>> related to the actual grow operation - it's tripping over a problem
>>> that already exists on disk before the grow operation is started.
>>> i.e. this is likely to be a real corruption being seen, and it
>>> happened some time in the distant past and so we probably won't ever
>>> be able to pinpoint the cause of the problem.
>>>
>>> That said, let's have a look at the broken superblock. Can you post
>>> the output of the commands:
>>>
>>> # xfs_db -r -c "sb 16" -c p <dev>
>>
>> done after the failed growfs mentioned above:
>
> Looks fine....
>
>>> and
>>>
>>> # xfs_db -r -c "sb 16" -c "type data" -c p <dev>
>>
>> 000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000
>> 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080
>> 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000
>> 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019
>> 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000
>> 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000
>> 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e
>> 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>> 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>
> There's your problem - the empty space in the superblock is supposed
> to be zero. mkfs zeros it and we rely on it being zero for various
> reasons.
>
> And one of those reasons is that we use the fact it shoul dbe zero
> to determine if we should be checking the CRC of the superblock.
> That is if there's a single bit error in the superblock and we are
> missing the correct bit in the version numbers that say CRCs are
> enabled, we use the fact that the superblock CRC field - which your
> filesystem knowns nothing about - should be zero to validate that
> the CRC feature bit is correctly set. The above superblock will
> indicate that there is a CRC set on the superblock, find the
> necessary version number is not correct, and so therefore we have a
> corruption in that superblock that the kernel code cannot handle
> without a user telling it what is correct.
>
> So, the fact grwofs is failing is actually the correct behaviour for
> the filesystem to have in this case - the superblock is corrupt,
> just not obviously so.
>
>>> so we can see the exact contents of that superblock?
>>>
>>> FWIW, how many times has this filesystem ben grown?
>>
>> I can't say for sure, about 4 or 5 times?
>>
>>> Did it start
>>> with only 32 AGs (i.e. 10TB in size)?
>>
>> 10TB? No. The device just has 3 TB. You most probably meant 10GB?
>> I'm not sure, but it definitely started with > 100GB.
>
> I misplaced a digit A block size of 4096 bytes and:
>
> agcount=42, agsize=7700480 blks
>
> So the filesystem size is 42 * 7700480 * 4096 = 1.26TB.
>
> The question I'm asking is how many AGs did the filesystem start
> with, because this:
>
> commit 1375cb65e87b327a8dd4f920c3e3d837fb40e9c2
> Author: Dave Chinner <dchinner@redhat.com>
> Date: Tue Oct 9 14:50:52 2012 +1100
>
> xfs: growfs: don't read garbage for new secondary superblocks
>
> When updating new secondary superblocks in a growfs operation, the
> superblock buffer is read from the newly grown region of the
> underlying device. This is not guaranteed to be zero, so violates
> the underlying assumption that the unused parts of superblocks are
> zero filled. Get a new buffer for these secondary superblocks to
> ensure that the unused regions are zero filled correctly.
>
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> Reviewed-by: Carlos Maiolino <cmaiolino@redhat.com>
> Signed-off-by: Ben Myers <bpm@sgi.com>
>
> Is the only possible reason I can think of that would result in
> non-zero empty space in a secondary superblock. And that implies
> that the filesystem started with 16 AGs or less,
yes
> and was grown with
> an older kernel with this bug in it.
yes.
> If it makes you feel any better, the bug that caused this had been
> in the code for 15+ years and you are the first person I know of to
> have ever hit it....
Probably the second one :-) See
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
> xfs_repair doesn't appear to have any checks in it to detect this
> situation or repair it - there are some conditions for zeroing the
> unused parts of a superblock, but they are focussed around detecting
> and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15
> years ago.
The _big problem_ is: xfs_repair not just doesn't repair it, but it
_causes data loss_ in some situations!
Given the following situation I ran in:
- xfs_growfs started running linux 3.10.5.
- Saw the error message on the konsole:
XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
data blocks changed from 319815680 to 346030080
- Checked with df -> The growing seems to be done. Decision: Analyse the
problem later when there is more time.
- Some days later, entry found in messages:
"Corruption detected. Unmount and run xfs_repair"
- I did it as suggested.
Result: FS has the original size again before growing the FS and
complete loss of all data written since this faulty growing. And: FS
isn't repaired.
If it is not a problem at all (that's how I understood you here), why is
there a error message and the suggest to run xfs_repair, which obviously
isn't able at all to repair this problem but leads directly to data loss?
Thanks for your clarification. I hope other people read this thread
before they are loosing data :-(.
What to do now?
- Don't use >= 3.10.x kernel. Or:
- Ignore it (how can I distinguish this case from other cases?) Or:
- Recreate the complete FS.
Thanks for clarification,
regards,
Michael.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 16:20 ` Michael Maier
@ 2013-08-14 16:37 ` Eric Sandeen
2013-08-15 17:18 ` Eric Sandeen
1 sibling, 0 replies; 26+ messages in thread
From: Eric Sandeen @ 2013-08-14 16:37 UTC (permalink / raw)
To: Michael Maier; +Cc: xfs
On 8/14/13 11:20 AM, Michael Maier wrote:
> Dave Chinner wrote:
>> On Tue, Aug 13, 2013 at 05:30:58PM +0200, Michael Maier wrote:
>>> Dave Chinner wrote:
>>>> [ re-ccing the list, because finding this is in everyone's interest ]
>>>>
>>>> On Mon, Aug 12, 2013 at 06:25:16PM +0200, Michael Maier wrote:
>>>>> Eric Sandeen wrote:
>>>>>> On 8/11/13 2:11 AM, Michael Maier wrote:
>>>>>>> Hello!
>>>>>>>
>>>>>>> I think I'm facing the same problem as already described here:
>>>>>>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>>>>>>
>>>>>> Maybe you can try the tracing Dave suggested in that thread?
>>>>>> It certainly does look similar.
>>>>>
>>>>> I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).
>>>>>
>>>>> xfs_growfs /mnt
>>>>> meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks
>>>>> = sectsz=512 attr=2
>>>>> data = bsize=4096 blocks=319815680, imaxpct=25
>>>>> = sunit=0 swidth=0 blks
>>>>> naming =version 2 bsize=4096 ascii-ci=0
>>>>> log =internal bsize=4096 blocks=60160, version=2
>>>>> = sectsz=512 sunit=0 blks, lazy-count=1
>>>>> realtime =none extsz=4096 blocks=0, rtextents=0
>>>>> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
>>>>> data blocks changed from 319815680 to 346030080
>>>>>
>>>>> The entry in messages was:
>>>>>
>>>>> Aug 12 18:09:50 dualc kernel: [ 257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00 XFSB............
>>>>> Aug 12 18:09:50 dualc kernel: [ 257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>>>> Aug 12 18:09:50 dualc kernel: [ 257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1 F.....M.......f.
>>>>> Aug 12 18:09:50 dualc kernel: [ 257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80 ................
>>>>> Aug 12 18:09:50 dualc kernel: [ 257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file
>>>>> /daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd
>>>> .....
>>>>> Aug 12 18:09:50 dualc kernel: [ 257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
>>>>> Aug 12 18:09:50 dualc kernel: [ 257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
>>>>> Aug 12 18:09:50 dualc kernel: [ 257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16
>>>>
>>>> Ok, so that's reading the secondary superblock for AG 16. You're
>>>> growing the filesystem from 42 to 45 AGs, so this problem is not
>>>> related to the actual grow operation - it's tripping over a problem
>>>> that already exists on disk before the grow operation is started.
>>>> i.e. this is likely to be a real corruption being seen, and it
>>>> happened some time in the distant past and so we probably won't ever
>>>> be able to pinpoint the cause of the problem.
>>>>
>>>> That said, let's have a look at the broken superblock. Can you post
>>>> the output of the commands:
>>>>
>>>> # xfs_db -r -c "sb 16" -c p <dev>
>>>
>>> done after the failed growfs mentioned above:
>>
>> Looks fine....
>>
>>>> and
>>>>
>>>> # xfs_db -r -c "sb 16" -c "type data" -c p <dev>
>>>
>>> 000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000
>>> 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080
>>> 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000
>>> 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019
>>> 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000
>>> 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000
>>> 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e
>>> 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>> 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>> 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>> 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>> 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>> 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>> 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>> 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>> 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e
>>
>> There's your problem - the empty space in the superblock is supposed
>> to be zero. mkfs zeros it and we rely on it being zero for various
>> reasons.
>>
>> And one of those reasons is that we use the fact it shoul dbe zero
>> to determine if we should be checking the CRC of the superblock.
>> That is if there's a single bit error in the superblock and we are
>> missing the correct bit in the version numbers that say CRCs are
>> enabled, we use the fact that the superblock CRC field - which your
>> filesystem knowns nothing about - should be zero to validate that
>> the CRC feature bit is correctly set. The above superblock will
>> indicate that there is a CRC set on the superblock, find the
>> necessary version number is not correct, and so therefore we have a
>> corruption in that superblock that the kernel code cannot handle
>> without a user telling it what is correct.
>>
>> So, the fact grwofs is failing is actually the correct behaviour for
>> the filesystem to have in this case - the superblock is corrupt,
>> just not obviously so.
>>
>>>> so we can see the exact contents of that superblock?
>>>>
>>>> FWIW, how many times has this filesystem ben grown?
>>>
>>> I can't say for sure, about 4 or 5 times?
>>>
>>>> Did it start
>>>> with only 32 AGs (i.e. 10TB in size)?
>>>
>>> 10TB? No. The device just has 3 TB. You most probably meant 10GB?
>>> I'm not sure, but it definitely started with > 100GB.
>>
>> I misplaced a digit A block size of 4096 bytes and:
>>
>> agcount=42, agsize=7700480 blks
>>
>> So the filesystem size is 42 * 7700480 * 4096 = 1.26TB.
>>
>> The question I'm asking is how many AGs did the filesystem start
>> with, because this:
>>
>> commit 1375cb65e87b327a8dd4f920c3e3d837fb40e9c2
>> Author: Dave Chinner <dchinner@redhat.com>
>> Date: Tue Oct 9 14:50:52 2012 +1100
>>
>> xfs: growfs: don't read garbage for new secondary superblocks
>>
>> When updating new secondary superblocks in a growfs operation, the
>> superblock buffer is read from the newly grown region of the
>> underlying device. This is not guaranteed to be zero, so violates
>> the underlying assumption that the unused parts of superblocks are
>> zero filled. Get a new buffer for these secondary superblocks to
>> ensure that the unused regions are zero filled correctly.
>>
>> Signed-off-by: Dave Chinner <dchinner@redhat.com>
>> Reviewed-by: Carlos Maiolino <cmaiolino@redhat.com>
>> Signed-off-by: Ben Myers <bpm@sgi.com>
>>
>> Is the only possible reason I can think of that would result in
>> non-zero empty space in a secondary superblock. And that implies
>> that the filesystem started with 16 AGs or less,
>
> yes
>
>> and was grown with
>> an older kernel with this bug in it.
>
> yes.
>
>> If it makes you feel any better, the bug that caused this had been
>> in the code for 15+ years and you are the first person I know of to
>> have ever hit it....
>
> Probably the second one :-) See
> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>
>> xfs_repair doesn't appear to have any checks in it to detect this
>> situation or repair it - there are some conditions for zeroing the
>> unused parts of a superblock, but they are focussed around detecting
>> and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15
>> years ago.
>
> The _big problem_ is: xfs_repair not just doesn't repair it, but it
> _causes data loss_ in some situations!
>
> Given the following situation I ran in:
> - xfs_growfs started running linux 3.10.5.
>
> - Saw the error message on the konsole:
> XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning
> data blocks changed from 319815680 to 346030080
>
> - Checked with df -> The growing seems to be done. Decision: Analyse the
> problem later when there is more time.
>
> - Some days later, entry found in messages:
> "Corruption detected. Unmount and run xfs_repair"
>
> - I did it as suggested.
> Result: FS has the original size again before growing the FS and
> complete loss of all data written since this faulty growing. And: FS
> isn't repaired.
> If it is not a problem at all (that's how I understood you here), why is
> there a error message and the suggest to run xfs_repair, which obviously
> isn't able at all to repair this problem but leads directly to data loss?
It seems that perhaps the error during growfs has left the filesystem
in a dangerous state - perhaps 45 AGs in memory but only 42 on disk,
I'm not certain. So you proceeded with the mounted fs thinking it had
more space, but When you did the subsequent repair, it only found 42
on disk, and "fixed" it by removing anything past that. </handwave>
> Thanks for your clarification. I hope other people read this thread
> before they are loosing data :-(.
>
>
> What to do now?
> - Don't use >= 3.10.x kernel. Or:
That's probably a decent workaround in the short term, at least for
xfs_growfs.
> - Ignore it (how can I distinguish this case from other cases?) Or:
> - Recreate the complete FS.
or:
- wait a bit 'til we get xfs_repair fixed to address the root cause.
I'll take a look at the image you provided me with, and see if I can
make some quick progress.
-Eric
>
> Thanks for clarification,
> regards,
> Michael.
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 6:20 ` Dave Chinner
2013-08-14 16:20 ` Michael Maier
@ 2013-08-14 16:51 ` Eric Sandeen
1 sibling, 0 replies; 26+ messages in thread
From: Eric Sandeen @ 2013-08-14 16:51 UTC (permalink / raw)
To: Dave Chinner; +Cc: Michael Maier, xfs
On 8/14/13 1:20 AM, Dave Chinner wrote:
> Hence looks like we'll need some new xfs_repair functionality to fix
> this. It might take me a little while to get you a fix - perhaps
> someone else with a little bit of spare time could get it done
> sooner than I can. Anyone?
Pff, not like I have the time, but it also looks pretty trivial.
secondary_sb_wack() only conditionally checks & zeroes unused space;
making it unconditional is probably all it takes.
I think it also needs to _not_ zero sb_bad_features2 as it does
today, because that is supposed to be kept in sync w/ sb_features2.
I have a patch I'll test; it'd be great to get a metadump from Michael
to test it on his original failure case.
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 15:05 ` Michael Maier
@ 2013-08-14 17:31 ` Stan Hoeppner
2013-08-14 18:13 ` Michael Maier
0 siblings, 1 reply; 26+ messages in thread
From: Stan Hoeppner @ 2013-08-14 17:31 UTC (permalink / raw)
To: Michael Maier; +Cc: Eric Sandeen, xfs
On 8/14/2013 10:05 AM, Michael Maier wrote:
> Stan Hoeppner schrieb:
>> On 8/13/2013 10:30 AM, Michael Maier wrote:
>>> Dave Chinner wrote:
>>>> [ re-ccing the list, because finding this is in everyone's interest ]
...
...
>>>> FWIW, how many times has this filesystem ben grown?
>>>
>>> I can't say for sure, about 4 or 5 times?
>>>
>>>> Did it start
>>>> with only 32 AGs (i.e. 10TB in size)?
>>>
>>>>>>> 10TB? No. The device just has 3 TB. You most probably meant 10GB?
>>> I'm not sure, but it definitely started with > 100GB.
>>
>> According to your xfs_info output that I highlighted above, and assuming
>> my math here is correct,
>>
>> (((7700480*4096)/1048576)*42)= 1,263,360 GB or ~1.23 TB
>>
>> this filesystem was 1.23 TB w/42 AGs before the grow operation.
>> Assuming defaults were used during mkfs.xfs it would appear the initial
>> size of this filesystem was ~120GB. And it would appear it has been
>> grown to ~10x its original size, and from 4 AGs to 42 AGs. That seems
>> like a lot of growth, to me. And Dave states the latest grow operation
>> was to 45 AGs, which would yield a ~1.32TB filesystem, not 3TB.
>
> He wrote about 10TB (=Terra(!) Byte). The entire HD only has 3 TB.
> 10TB is impossible!
I was referring to the 3TB you mention above. I misunderstood that to
mean the size of the filesystem, but you did clearly state "device".
Sorry about the oversight there.
> The last growfs was about 100GB. The FS was grown lots of times
> since the initial creation one or two years ago.
Yes, this is the critical information Dave was looking for which relates
to xfs_growfs failing.
> About the sizes before the last grow and after it: see
> http://www.spinics.net/lists/xfs/msg21032.html
Unrelated to the grow bug, I'm wondering why you started so small and
are growing an XFS in such small chunks, a great number of times, over a
period of years, on a single 3TB disk, instead of making a 3TB XFS out
of the gate.
If you keep growing until you consume the disk, you'll have ~100
allocation groups. Typically you'd want to have no more than 4 AGs per
spindle. You already have 42 (or 45) which will tend to seek the disk
to death with many workloads, driving latency through the roof and
decreasing throughput substantially. Do you notice any performance
problems yet? Or is this XFS strictly being used as a WORM like backup
silo?
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 17:31 ` Stan Hoeppner
@ 2013-08-14 18:13 ` Michael Maier
2013-08-14 22:20 ` Stan Hoeppner
0 siblings, 1 reply; 26+ messages in thread
From: Michael Maier @ 2013-08-14 18:13 UTC (permalink / raw)
To: stan; +Cc: Eric Sandeen, xfs
Stan Hoeppner wrote:
> If you keep growing until you consume the disk, you'll have ~100
> allocation groups. Typically you'd want to have no more than 4 AGs per
> spindle. You already have 42 (or 45) which will tend to seek the disk
> to death with many workloads, driving latency through the roof and
> decreasing throughput substantially. Do you notice any performance
> problems yet?
What are expected rates for copying e.g. a 10GB file? It's a Seagate
Barracuda 3000GB Model ST3000DM001 SATA connected to SATA 6 Gb/s chip.
The source and the destination FS is LUKS crypted. About 3 GB usable RAM
(cache), AMD FX-8350 processor @ max. 3800MHz.
It's getting slower as more as the free space on the fs is reduced
(beginning at about the last GB). Resizing it makes the problem
disappear again.
> Or is this XFS strictly being used as a WORM like backup
> silo?
yes
Thanks,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 18:13 ` Michael Maier
@ 2013-08-14 22:20 ` Stan Hoeppner
2013-08-15 17:05 ` Michael Maier
0 siblings, 1 reply; 26+ messages in thread
From: Stan Hoeppner @ 2013-08-14 22:20 UTC (permalink / raw)
To: Michael Maier; +Cc: Eric Sandeen, xfs
On 8/14/2013 1:13 PM, Michael Maier wrote:
> Stan Hoeppner wrote:
>> If you keep growing until you consume the disk, you'll have ~100
>> allocation groups. Typically you'd want to have no more than 4 AGs per
>> spindle. You already have 42 (or 45) which will tend to seek the disk
>> to death with many workloads, driving latency through the roof and
>> decreasing throughput substantially. Do you notice any performance
>> problems yet?
>
> What are expected rates for copying e.g. a 10GB file? It's a Seagate
> Barracuda 3000GB Model ST3000DM001 SATA connected to SATA 6 Gb/s chip.
> The source and the destination FS is LUKS crypted. About 3 GB usable RAM
> (cache), AMD FX-8350 processor @ max. 3800MHz.
Too many variables really to hazard a guess. If you put a gun to my
head, I'd say strictly looking at the ingest rate of the Seagate, at a
little less than half capacity, writing about 1/3rd of the way down the
platters, optimum throughput should be 80-100 MB/s or so in the last 3
AGs, if free space isn't too heavily fragmented.
> It's getting slower as more as the free space on the fs is reduced
> (beginning at about the last GB).
This is due to writing into fragmented free space in the 40+ AGs. This
occurs after the last large free space extents have been consumed, those
extents in the last 3 AGs created by the last xfs_growfs.
> Resizing it makes the problem
> disappear again.
After adding another ~90 GB of free space XFS will preferentially write
large files into the new large free extents, avoiding the existing
fragmented free space in the preexisting AGs.
>> Or is this XFS strictly being used as a WORM like backup
>> silo?
>
> yes
With so many smallish AGs, so many grow ops, and this backup workload,
I'm curious as to what your free space map looks like. Would you mind
posting the output of the following command, if you can?
$ xfs_db -r -c freesp <dev>
Thanks.
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 15:16 ` Michael Maier
@ 2013-08-15 0:58 ` Dave Chinner
2013-08-15 18:14 ` Michael Maier
0 siblings, 1 reply; 26+ messages in thread
From: Dave Chinner @ 2013-08-15 0:58 UTC (permalink / raw)
To: Michael Maier; +Cc: Eric Sandeen, xfs
On Wed, Aug 14, 2013 at 05:16:14PM +0200, Michael Maier wrote:
> Dave Chinner wrote:
> > On Tue, Aug 13, 2013 at 04:55:00PM +0200, Michael Maier wrote:
> >> Dave Chinner wrote:
> >>> On Mon, Aug 12, 2013 at 06:50:55PM +0200, Michael Maier wrote:
> >>>> Meanwhile, I faced another problem on another xfs-file system with linux
> >>>> 3.10.5 which I never saw before. During writing a few bytes to disc, I
> >>>> got "disc full" and the writing failed.
> >>>>
> >>>> At the same time, df reported 69G of free space! I ran xfs_repair -n and
> >>>> got:
> >>>>
> >>>>
> >>>> xfs_repair -n /dev/mapper/raid0-daten2
> >>>> Phase 1 - find and verify superblock...
> >>>> Phase 2 - using internal log
> >>>> - scan filesystem freespace and inode maps...
> >>>> sb_ifree 591, counted 492
> >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^
> >>>> What does this mean? How can I get rid of it w/o loosing data? This file
> >>>> system was created a few days ago and never resized.
> >>>
> >>> Superblock inode counting is lazy - it can get out of sync in after
> >>> an unclean shutdown, but generally mounting a dirty filesystem will
> >>> result in it being recalculated rather than trusted to be correct.
> >>> So there's nothing to worry about here.
> >>
> >> When will it be self healed?
> >
> > that depends on whether there's actually a problem. Like I said in
> > the part you snipped off - if you ran xfs_repair -n on filesystem
> > that needs log recovery that accounting difference is expected.
>
> I know, that option -n doesn't do anything. It was intended, because
> xfs_repair destroyed a lot of data when applied at the other problem I
> have _and_ it repaired nothing at the same time!
xfs_repair will remove files it cannot repair because their metadata
is are too corrupted to repair or cannot be repaired safely. That's
always been the case for any filesystem repair tool - all they
guarantee is that the filesystem will be consistent after they are
run. Repairing a corrupted filesystem almost always results in some
form of data loss occurring....
If there is nothing wrong with the filesystem except the accouting
is wrong, then it will fix the accounting problem in phase 5 when
run without the -n parameter.
> >> This is strange and I can't use the free space, which I need! How can it
> >> be forced to be repaired w/o data loss?
> >
> > The above is complaining about a free inode count mismatch, not a
> > problem about free space being wrong. What problem are you actually
> > having?
>
> The application, which wanted to write a few bytes gets a "disk full"
> error although df -h reports 69GB of free space.
That's not necessarily a corruption, though, and most likely isn't
related to the accounting issue xfs_repair is reporting. Indeed,
this is typically a sign of being unable to allocate an inode
because there is insufficient contiguous free space in the
filesystem to allocate a new inode chunk. What does your free space
histogram look like?
# xfs_db -r -c "freesp -s" <dev>
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 22:20 ` Stan Hoeppner
@ 2013-08-15 17:05 ` Michael Maier
0 siblings, 0 replies; 26+ messages in thread
From: Michael Maier @ 2013-08-15 17:05 UTC (permalink / raw)
To: stan; +Cc: Eric Sandeen, xfs
Stan Hoeppner wrote:
> On 8/14/2013 1:13 PM, Michael Maier wrote:
>> Stan Hoeppner wrote:
>>> If you keep growing until you consume the disk, you'll have ~100
>>> allocation groups. Typically you'd want to have no more than 4 AGs per
>>> spindle. You already have 42 (or 45) which will tend to seek the disk
>>> to death with many workloads, driving latency through the roof and
>>> decreasing throughput substantially. Do you notice any performance
>>> problems yet?
>>
>> What are expected rates for copying e.g. a 10GB file? It's a Seagate
>> Barracuda 3000GB Model ST3000DM001 SATA connected to SATA 6 Gb/s chip.
>> The source and the destination FS is LUKS crypted. About 3 GB usable RAM
>> (cache), AMD FX-8350 processor @ max. 3800MHz.
>
> Too many variables really to hazard a guess. If you put a gun to my
> head, I'd say strictly looking at the ingest rate of the Seagate, at a
> little less than half capacity, writing about 1/3rd of the way down the
> platters, optimum throughput should be 80-100 MB/s or so in the last 3
> AGs, if free space isn't too heavily fragmented.
96MB/s / 40GB-file ~ 130GB free space. -> good guess!
[...]
> With so many smallish AGs, so many grow ops, and this backup workload,
> I'm curious as to what your free space map looks like. Would you mind
> posting the output of the following command, if you can?
>
> $ xfs_db -r -c freesp <dev>
after copying:
from to extents blocks pct
1 1 193 193 0.00
2 3 31 72 0.00
4 7 10 60 0.00
8 15 8 94 0.00
16 31 9 210 0.00
32 63 14 631 0.00
64 127 20 1974 0.01
128 255 16 2838 0.01
256 511 12 4923 0.02
512 1023 13 10065 0.04
1024 2047 14 23417 0.10
2048 4095 22 58439 0.25
4096 8191 8 44159 0.19
8192 16383 1 8192 0.04
16384 32767 7 165001 0.71
32768 65535 6 228278 0.99
262144 524287 1 398328 1.72
1048576 2097151 2 3560198 15.39
2097152 4194303 1 3709535 16.04
4194304 7700480 2 14909432 64.47
Regards,
Michael.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-14 16:20 ` Michael Maier
2013-08-14 16:37 ` Eric Sandeen
@ 2013-08-15 17:18 ` Eric Sandeen
2013-08-15 17:55 ` Michael Maier
1 sibling, 1 reply; 26+ messages in thread
From: Eric Sandeen @ 2013-08-15 17:18 UTC (permalink / raw)
To: Michael Maier; +Cc: xfs
On 8/14/13 11:20 AM, Michael Maier wrote:
> Dave Chinner wrote:
...
>> If it makes you feel any better, the bug that caused this had been
>> in the code for 15+ years and you are the first person I know of to
>> have ever hit it....
>
> Probably the second one :-) See
> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>
>> xfs_repair doesn't appear to have any checks in it to detect this
>> situation or repair it - there are some conditions for zeroing the
>> unused parts of a superblock, but they are focussed around detecting
>> and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15
>> years ago.
>
> The _big problem_ is: xfs_repair not just doesn't repair it, but it
> _causes data loss_ in some situations!
>
So as far as I can tell at this point, a few things have happened to
result in this unfortunate situation. Congratulations, you hit a
perfect storm. :(
1) prior resize operations populated unused portions of backup sbs w/ junk
2) newer kernels fail to verify superblocks in this state
3) during your growfs under 3.10, that verification failure aborted
backup superblock updates, leaving many unmodified
4a) xfs_repair doesn't find or fix the junk in the backup sbs, and
4b) when running, it looks for the superblocks which are "most matching"
other superblocks on the disk, and takes that version as correct.
So you had 16 superblocks (0-15) which were correct after the growfs.
But 16 didn't verify and was aborted, so nothing was updated after that.
This means that 16 onward have the wrong number of AGs and disk blocks;
i.e. they are the pre-growfs size, and there are 26 of them.
Today, xfs_repair sees this 26-to-16 vote, and decides that the 26
matching superblocks "win," rewrites the first superblock with this
geometry, and uses that to verify the rest of the filesytem. Hence
anything post-growfs looks out of bounds, and gets nuked.
So right now, I'm thinking that the "proper geometry" heuristic should
be adjusted, but how to do that in general, I'm not sure. Weighting
sb 0 heavily, especially if it matches many subsequent superblocks,
seems somewhat reasonable.
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-15 17:18 ` Eric Sandeen
@ 2013-08-15 17:55 ` Michael Maier
2013-08-15 18:14 ` Eric Sandeen
0 siblings, 1 reply; 26+ messages in thread
From: Michael Maier @ 2013-08-15 17:55 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs
Eric Sandeen wrote:
> On 8/14/13 11:20 AM, Michael Maier wrote:
>> Dave Chinner wrote:
>
> ...
>
>>> If it makes you feel any better, the bug that caused this had been
>>> in the code for 15+ years and you are the first person I know of to
>>> have ever hit it....
>>
>> Probably the second one :-) See
>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>>
>>> xfs_repair doesn't appear to have any checks in it to detect this
>>> situation or repair it - there are some conditions for zeroing the
>>> unused parts of a superblock, but they are focussed around detecting
>>> and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15
>>> years ago.
>>
>> The _big problem_ is: xfs_repair not just doesn't repair it, but it
>> _causes data loss_ in some situations!
>>
>
> So as far as I can tell at this point, a few things have happened to
> result in this unfortunate situation. Congratulations, you hit a
> perfect storm. :(
I can appease you - as it "only" hit my backup device and because I
noticed the problem before I really needed it: I didn't hit any data
loss over all, because the original data is ok and I repeated the backup
w/ the fixed FS now!
> 1) prior resize operations populated unused portions of backup sbs w/ junk
> 2) newer kernels fail to verify superblocks in this state
> 3) during your growfs under 3.10, that verification failure aborted
> backup superblock updates, leaving many unmodified
> 4a) xfs_repair doesn't find or fix the junk in the backup sbs, and
> 4b) when running, it looks for the superblocks which are "most matching"
> other superblocks on the disk, and takes that version as correct.
>
> So you had 16 superblocks (0-15) which were correct after the growfs.
> But 16 didn't verify and was aborted, so nothing was updated after that.
> This means that 16 onward have the wrong number of AGs and disk blocks;
> i.e. they are the pre-growfs size, and there are 26 of them.
>
> Today, xfs_repair sees this 26-to-16 vote, and decides that the 26
> matching superblocks "win," rewrites the first superblock with this
> geometry, and uses that to verify the rest of the filesytem. Hence
> anything post-growfs looks out of bounds, and gets nuked.
>
> So right now, I'm thinking that the "proper geometry" heuristic should
> be adjusted, but how to do that in general, I'm not sure. Weighting
> sb 0 heavily, especially if it matches many subsequent superblocks,
> seems somewhat reasonable.
This would have been my next question! I repaired it w/ the git
xfs_repair on the already reduced to original size FS. I think, if I
would have done the same w/ the grown FS, the FS most probably would be
reduced to the size before the growing.
Wouldn't it be better to not grow at all if there are problems detected?
Means: Don't do the check after the growing, but before? Ok, I could
have done it myself ... . From now on, I will do it like this!
Thanks,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-15 17:55 ` Michael Maier
@ 2013-08-15 18:14 ` Eric Sandeen
2013-08-15 18:35 ` Michael Maier
0 siblings, 1 reply; 26+ messages in thread
From: Eric Sandeen @ 2013-08-15 18:14 UTC (permalink / raw)
To: Michael Maier; +Cc: xfs
On 8/15/13 12:55 PM, Michael Maier wrote:
> Eric Sandeen wrote:
>> On 8/14/13 11:20 AM, Michael Maier wrote:
>>> Dave Chinner wrote:
>>
>> ...
>>
>>>> If it makes you feel any better, the bug that caused this had been
>>>> in the code for 15+ years and you are the first person I know of to
>>>> have ever hit it....
>>>
>>> Probably the second one :-) See
>>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>>>
>>>> xfs_repair doesn't appear to have any checks in it to detect this
>>>> situation or repair it - there are some conditions for zeroing the
>>>> unused parts of a superblock, but they are focussed around detecting
>>>> and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15
>>>> years ago.
>>>
>>> The _big problem_ is: xfs_repair not just doesn't repair it, but it
>>> _causes data loss_ in some situations!
>>>
>>
>> So as far as I can tell at this point, a few things have happened to
>> result in this unfortunate situation. Congratulations, you hit a
>> perfect storm. :(
>
> I can appease you - as it "only" hit my backup device and because I
> noticed the problem before I really needed it: I didn't hit any data
> loss over all, because the original data is ok and I repeated the backup
> w/ the fixed FS now!
>
>> 1) prior resize operations populated unused portions of backup sbs w/ junk
>> 2) newer kernels fail to verify superblocks in this state
>> 3) during your growfs under 3.10, that verification failure aborted
>> backup superblock updates, leaving many unmodified
>> 4a) xfs_repair doesn't find or fix the junk in the backup sbs, and
>> 4b) when running, it looks for the superblocks which are "most matching"
>> other superblocks on the disk, and takes that version as correct.
>>
>> So you had 16 superblocks (0-15) which were correct after the growfs.
>> But 16 didn't verify and was aborted, so nothing was updated after that.
>> This means that 16 onward have the wrong number of AGs and disk blocks;
>> i.e. they are the pre-growfs size, and there are 26 of them.
>>
>> Today, xfs_repair sees this 26-to-16 vote, and decides that the 26
>> matching superblocks "win," rewrites the first superblock with this
>> geometry, and uses that to verify the rest of the filesytem. Hence
>> anything post-growfs looks out of bounds, and gets nuked.
>>
>> So right now, I'm thinking that the "proper geometry" heuristic should
>> be adjusted, but how to do that in general, I'm not sure. Weighting
>> sb 0 heavily, especially if it matches many subsequent superblocks,
>> seems somewhat reasonable.
>
> This would have been my next question! I repaired it w/ the git
> xfs_repair on the already reduced to original size FS. I think, if I
> would have done the same w/ the grown FS, the FS most probably would be
> reduced to the size before the growing.
>
> Wouldn't it be better to not grow at all if there are problems detected?
> Means: Don't do the check after the growing, but before? Ok, I could
> have done it myself ... . From now on, I will do it like this!
well, see the next couple patches I'm about to send to the list ... ;)
but a check prior wouldn't have helped you, because repair didn't detect
the problem that growfs choked on.
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-15 0:58 ` Dave Chinner
@ 2013-08-15 18:14 ` Michael Maier
0 siblings, 0 replies; 26+ messages in thread
From: Michael Maier @ 2013-08-15 18:14 UTC (permalink / raw)
To: Dave Chinner; +Cc: Eric Sandeen, xfs
Dave Chinner wrote:
> On Wed, Aug 14, 2013 at 05:16:14PM +0200, Michael Maier wrote:
>> Dave Chinner wrote:
>>> On Tue, Aug 13, 2013 at 04:55:00PM +0200, Michael Maier wrote:
>>>> Dave Chinner wrote:
>>>>> On Mon, Aug 12, 2013 at 06:50:55PM +0200, Michael Maier wrote:
>>>>>> Meanwhile, I faced another problem on another xfs-file system with linux
>>>>>> 3.10.5 which I never saw before. During writing a few bytes to disc, I
>>>>>> got "disc full" and the writing failed.
>>>>>>
>>>>>> At the same time, df reported 69G of free space! I ran xfs_repair -n and
>>>>>> got:
>>>>>>
>>>>>>
>>>>>> xfs_repair -n /dev/mapper/raid0-daten2
>>>>>> Phase 1 - find and verify superblock...
>>>>>> Phase 2 - using internal log
>>>>>> - scan filesystem freespace and inode maps...
>>>>>> sb_ifree 591, counted 492
>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>>> What does this mean? How can I get rid of it w/o loosing data? This file
>>>>>> system was created a few days ago and never resized.
>>>>>
>>>>> Superblock inode counting is lazy - it can get out of sync in after
>>>>> an unclean shutdown, but generally mounting a dirty filesystem will
>>>>> result in it being recalculated rather than trusted to be correct.
>>>>> So there's nothing to worry about here.
>>>>
>>>> When will it be self healed?
>>>
>>> that depends on whether there's actually a problem. Like I said in
>>> the part you snipped off - if you ran xfs_repair -n on filesystem
>>> that needs log recovery that accounting difference is expected.
>>
>> I know, that option -n doesn't do anything. It was intended, because
>> xfs_repair destroyed a lot of data when applied at the other problem I
>> have _and_ it repaired nothing at the same time!
>
> xfs_repair will remove files it cannot repair because their metadata
> is are too corrupted to repair or cannot be repaired safely. That's
> always been the case for any filesystem repair tool - all they
> guarantee is that the filesystem will be consistent after they are
> run. Repairing a corrupted filesystem almost always results in some
> form of data loss occurring....
>
> If there is nothing wrong with the filesystem except the accouting
> is wrong, then it will fix the accounting problem in phase 5 when
> run without the -n parameter.
Ok, it's fixed now (w/ the git xfs_repair). Thanks for clarification.
I'm sorry, but I was a little bit scared because of the other problem
:-( I faced.
>>>> This is strange and I can't use the free space, which I need! How can it
>>>> be forced to be repaired w/o data loss?
>>>
>>> The above is complaining about a free inode count mismatch, not a
>>> problem about free space being wrong. What problem are you actually
>>> having?
>>
>> The application, which wanted to write a few bytes gets a "disk full"
>> error although df -h reports 69GB of free space.
>
> That's not necessarily a corruption, though, and most likely isn't
> related to the accounting issue xfs_repair is reporting. Indeed,
> this is typically a sign of being unable to allocate an inode
> because there is insufficient contiguous free space in the
> filesystem to allocate a new inode chunk. What does your free space
> histogram look like?
>
> # xfs_db -r -c "freesp -s" <dev>
Unfortunately, this isn't possible any more, because meanwhile I removed
a lot of data, therefore the actual data doesn't hit the situation I
faced a few days ago. Sorry. Should it happen again, I will for sure
remember your mail!
Thanks,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-15 18:14 ` Eric Sandeen
@ 2013-08-15 18:35 ` Michael Maier
2013-08-15 18:42 ` Eric Sandeen
0 siblings, 1 reply; 26+ messages in thread
From: Michael Maier @ 2013-08-15 18:35 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs
Eric Sandeen wrote:
> On 8/15/13 12:55 PM, Michael Maier wrote:
>> Eric Sandeen wrote:
>>> On 8/14/13 11:20 AM, Michael Maier wrote:
>>>> Dave Chinner wrote:
>>>
>>> ...
>>>
>>>>> If it makes you feel any better, the bug that caused this had been
>>>>> in the code for 15+ years and you are the first person I know of to
>>>>> have ever hit it....
>>>>
>>>> Probably the second one :-) See
>>>> http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428
>>>>
>>>>> xfs_repair doesn't appear to have any checks in it to detect this
>>>>> situation or repair it - there are some conditions for zeroing the
>>>>> unused parts of a superblock, but they are focussed around detecting
>>>>> and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15
>>>>> years ago.
>>>>
>>>> The _big problem_ is: xfs_repair not just doesn't repair it, but it
>>>> _causes data loss_ in some situations!
>>>>
>>>
>>> So as far as I can tell at this point, a few things have happened to
>>> result in this unfortunate situation. Congratulations, you hit a
>>> perfect storm. :(
>>
>> I can appease you - as it "only" hit my backup device and because I
>> noticed the problem before I really needed it: I didn't hit any data
>> loss over all, because the original data is ok and I repeated the backup
>> w/ the fixed FS now!
>>
>>> 1) prior resize operations populated unused portions of backup sbs w/ junk
>>> 2) newer kernels fail to verify superblocks in this state
>>> 3) during your growfs under 3.10, that verification failure aborted
>>> backup superblock updates, leaving many unmodified
>>> 4a) xfs_repair doesn't find or fix the junk in the backup sbs, and
>>> 4b) when running, it looks for the superblocks which are "most matching"
>>> other superblocks on the disk, and takes that version as correct.
>>>
>>> So you had 16 superblocks (0-15) which were correct after the growfs.
>>> But 16 didn't verify and was aborted, so nothing was updated after that.
>>> This means that 16 onward have the wrong number of AGs and disk blocks;
>>> i.e. they are the pre-growfs size, and there are 26 of them.
>>>
>>> Today, xfs_repair sees this 26-to-16 vote, and decides that the 26
>>> matching superblocks "win," rewrites the first superblock with this
>>> geometry, and uses that to verify the rest of the filesytem. Hence
>>> anything post-growfs looks out of bounds, and gets nuked.
>>>
>>> So right now, I'm thinking that the "proper geometry" heuristic should
>>> be adjusted, but how to do that in general, I'm not sure. Weighting
>>> sb 0 heavily, especially if it matches many subsequent superblocks,
>>> seems somewhat reasonable.
>>
>> This would have been my next question! I repaired it w/ the git
>> xfs_repair on the already reduced to original size FS. I think, if I
>> would have done the same w/ the grown FS, the FS most probably would be
>> reduced to the size before the growing.
>>
>> Wouldn't it be better to not grow at all if there are problems detected?
>> Means: Don't do the check after the growing, but before? Ok, I could
>> have done it myself ... . From now on, I will do it like this!
>
> well, see the next couple patches I'm about to send to the list ... ;)
Cool!
> but a check prior wouldn't have helped you, because repair didn't detect
> the problem that growfs choked on.
The old xfs_repair! Your patched one would have detected the problem if
I got it right.
But globally speaking: you're right - it's impossible to get 100%
security. But couldn't xfs_repair -n find other problems which therefore
could be repaired before growing the FS?
Thanks,
regards,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Failure growing xfs with linux 3.10.5
2013-08-15 18:35 ` Michael Maier
@ 2013-08-15 18:42 ` Eric Sandeen
0 siblings, 0 replies; 26+ messages in thread
From: Eric Sandeen @ 2013-08-15 18:42 UTC (permalink / raw)
To: Michael Maier; +Cc: xfs
On 8/15/13 1:35 PM, Michael Maier wrote:
>> but a check prior wouldn't have helped you, because repair didn't detect
>> > the problem that growfs choked on.
> The old xfs_repair! Your patched one would have detected the problem if
> I got it right.
>
> But globally speaking: you're right - it's impossible to get 100%
> security. But couldn't xfs_repair -n find other problems which therefore
> could be repaired before growing the FS?
yep, sure.
extN keeps track of the last fsck instance, and sometimes enforces it
prior to resize. XFS doesn't do that, but it's probably not bad
practice.
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2013-08-15 18:42 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-08-11 7:11 Failure growing xfs with linux 3.10.5 Michael Maier
2013-08-11 18:36 ` Eric Sandeen
2013-08-12 16:50 ` Michael Maier
2013-08-13 0:54 ` Dave Chinner
2013-08-13 14:55 ` Michael Maier
2013-08-14 5:43 ` Dave Chinner
2013-08-14 15:16 ` Michael Maier
2013-08-15 0:58 ` Dave Chinner
2013-08-15 18:14 ` Michael Maier
[not found] ` <52090C6C.6060604@allmail.net>
2013-08-13 0:04 ` Dave Chinner
2013-08-13 15:30 ` Michael Maier
2013-08-14 5:53 ` Stan Hoeppner
2013-08-14 15:05 ` Michael Maier
2013-08-14 17:31 ` Stan Hoeppner
2013-08-14 18:13 ` Michael Maier
2013-08-14 22:20 ` Stan Hoeppner
2013-08-15 17:05 ` Michael Maier
2013-08-14 6:20 ` Dave Chinner
2013-08-14 16:20 ` Michael Maier
2013-08-14 16:37 ` Eric Sandeen
2013-08-15 17:18 ` Eric Sandeen
2013-08-15 17:55 ` Michael Maier
2013-08-15 18:14 ` Eric Sandeen
2013-08-15 18:35 ` Michael Maier
2013-08-15 18:42 ` Eric Sandeen
2013-08-14 16:51 ` Eric Sandeen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.