xfs_growfs failure....

xfs_growfs failure....

[Joe Allen]

[-- Attachment #1.1: Type: text/plain, Size: 3248 bytes --]

I am in some difficulty here over a 100TB filesystem that
Is now unusable after a xfs_growfs command.

Is there someone that might help assist?

#mount: /dev/logfs-sessions/sessions: can't read superblock

Filesystem "dm-61": Disabling barriers, not supported with external log device
attempt to access beyond end of device
dm-61: rw=0, want=238995038208, limit=215943192576

I/O error in filesystem ("dm-61") meta-data dev dm-61 block 0x37a536dfff       ("xfs_read_buf") error 5 buf count 512
XFS: size check 2 failed
Filesystem "dm-61": Disabling barriers, not supported with external log device
attempt to access beyond end of device

xfs_repair -n <device> basically looks for superblocks (phase 1 I guess ) for a long time. I'm letting it run, but not much hope.

I'm hesitant to run xfs_repair -L  or without the -n flag for fear of making it worse.

I've run the following command to try to inspect

please let me know if there's anything else that might help determine if there's anything I can do to recover...

-bash-3.1# xfs_check -l /dev/logfs-sessions/logdev /dev/logfs-sessions/sessions
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_check.  If you are unable to mount the filesystem, then use
the xfs_repair -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

-bash-3.1# xfs_db -r -c 'sb 0' -c p /dev/logfs-sessions/sessions
magicnum = 0x58465342
blocksize = 4096
dblocks = 29874379776
rblocks = 0
rextents = 0
uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
logstart = 0
rootino = 2048
rbmino = 2049
rsumino = 2050
rextsize = 384
agblocks = 268435328
agcount = 112
rbmblocks = 0
logblocks = 32000
versionnum = 0x3184
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 28
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 7291520
ifree = 8514
fdblocks = 6623185597
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 128
width = 384
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0

-bash-3.1# xfs_db -r -c 'sb 2' -c p /dev/logfs-sessions/sessions
magicnum = 0x58465342
blocksize = 4096
dblocks = 24111418368
rblocks = 0
rextents = 0
uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
logstart = 0
rootino = 2048
rbmino = 2049
rsumino = 2050
rextsize = 384
agblocks = 268435328
agcount = 90
rbmblocks = 0
logblocks = 32000
versionnum = 0x3184
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 28
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 7291008
ifree = 8765
fdblocks = 862328436
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 128
width = 384
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0





[-- Attachment #1.2: Type: text/html, Size: 9740 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs failure....

[Dave Chinner]

On Wed, Feb 24, 2010 at 02:44:37AM -0800, Joe Allen wrote:
> I am in some difficulty here over a 100TB filesystem that
> Is now unusable after a xfs_growfs command.
> 
> Is there someone that might help assist?
> 
> #mount: /dev/logfs-sessions/sessions: can't read superblock
> 
> Filesystem "dm-61": Disabling barriers, not supported with external log device
> attempt to access beyond end of device
> dm-61: rw=0, want=238995038208, limit=215943192576

You've grown the filesystem to 238995038208 ѕectors (111.3TiB),
but the underlying device is only 215943192576 sectors (100.5TiB)
in size.

I'm assuming that you're trying to mount the filesystem after a
reboot? I make this assumption as growfs is an online operation and
won't grow if the underlying block device has not already been
grown. For a subsequent mount to fail with the underlying device
being too small, something about the underlying block
device had to change....

What kernel version and xfsprogs version are you using?

> xfs_repair -n <device> basically looks for superblocks (phase 1 I
> guess ) for a long time. I'm letting it run, but not much hope.

Don't repair the filesystem - there is nothing wrong with it
unless you start modifying stuff. What you need to do is fix the
underlying device to bring it back to the size it was supposed to
be at when the grow operation was run.

What does /proc/partitions tell you about the size of dm-61? does
that report the correct size, and if it does, what is it?

> I'm hesitant to run xfs_repair -L  or without the -n flag for fear of making it worse.

Good - don't run anything like that until you sort out whether the
underlying device is correctly sized or not.

> -bash-3.1# xfs_db -r -c 'sb 0' -c p /dev/logfs-sessions/sessions
> magicnum = 0x58465342
> blocksize = 4096
> dblocks = 29874379776

XFS definitely thinks it is 111.3TiB in size.

> rblocks = 0
> rextents = 0
> uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
> logstart = 0
> rootino = 2048
> rbmino = 2049
> rsumino = 2050
> rextsize = 384
> agblocks = 268435328
> agcount = 112

112 AGs of 1TiB each - that confirms the grow succeeded and it was
able to write metadata to disk between 100 and 111 TiB without
errors being reported. That implies the block device must have been
that big at some point...

> rbmblocks = 0
> logblocks = 32000
> versionnum = 0x3184
> sectsize = 512
> inodesize = 256
> inopblock = 16
> fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
> blocklog = 12
> sectlog = 9
> inodelog = 8
> inopblog = 4
> agblklog = 28
> rextslog = 0
> inprogress = 0
> imax_pct = 25
> icount = 7291520
> ifree = 8514
> fdblocks = 6623185597

With 24.5TiB of free space

> -bash-3.1# xfs_db -r -c 'sb 2' -c p /dev/logfs-sessions/sessions
> magicnum = 0x58465342
> blocksize = 4096
> dblocks = 24111418368

That's 89.9TiB...

> rblocks = 0
> rextents = 0
> uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
> logstart = 0
> rootino = 2048
> rbmino = 2049
> rsumino = 2050
> rextsize = 384
> agblocks = 268435328
> agcount = 90

And 90 AGs. That tells me the filesystem was created as a 90TiB
filesystem. Can you tell me if you attempted to grow from 90TiB to
100TiB or from 100TiB to 110TiB?  There were bugs at one point in
both the userspace grow code and the kernel code that resulted in
bad grows (hence the need to know the versions this occurred on
and what you were actually attempting to do), but these problems
can usually be fixed up with some xfs_db magic.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs failure....

[Peter Grandi]

> I am in some difficulty here over a 100TB filesystem

Shrewd idea! Because 'fsck' takes no time and memory, so the
bigger the filesystem the better! ;-).

> that Is now unusable after a xfs_growfs command. [ ... ] 

Wondering how long it took to backup 100TB; but of course doing
a 'grow' is guaranteed to be error free, so there :-).

> attempt to access beyond end of device dm-61: rw=0,
> want=238995038208, limit=215943192576

It looks like the underlying DM logical volume is smaller than
the new size of the filesystem, which is strange as 'xfs_growfs'
is supposed to fetch the size of the underlying block device if
none is specified explicitly on the command line. The different
is about 10% or 10TB, so it is far from trivial.

Looking at the superblock dumps there are some pretty huge
discrepancies, 

  -bash-3.1# xfs_db -r -c 'sb 0' -c p /dev/logfs-sessions/sessions
  magicnum = 0x58465342
  blocksize = 4096
  dblocks = 29874379776
  rblocks = 0
  rextents = 0
  uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
  logstart = 0
  rootino = 2048
  rbmino = 2049
  rsumino = 2050
  rextsize = 384
  agblocks = 268435328
  agcount = 112
  [ ... ]
  -bash-3.1# xfs_db -r -c 'sb 2' -c p /dev/logfs-sessions/sessions
  magicnum = 0x58465342
  blocksize = 4096
  dblocks = 24111418368
  rblocks = 0
  rextents = 0
  uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
  logstart = 0
  rootino = 2048
  rbmino = 2049
  rsumino = 2050
  rextsize = 384
  agblocks = 268435328
  agcount = 90
  [ ... ]

The 'dblocks' field is rather different, even if the 'uuid' and
'agblocks' is the same, and 'agcount' is also rather different.
In SB 0 'dblocks' 29874379776 means size 238995038208, which is
value of 'want' above. The products of 'agcount' and 'agblocks'
fit with the sizes.

It looks like that the filesystem was "grown" from ~92TiB to
~114TiB on a storage device that is reported as ~103TiB
long. Again, very strange.

My impression is that not enough history/context has been
provided to enable a good guess at what has happened and how to
undo the consequent damage.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs failure....

[Jason Vagalatos]

[-- Attachment #1.1: Type: text/plain, Size: 5505 bytes --]


Hi David,
I’m picking this up from Joe.  I’ll attempt to answer your questions.

The underlying device was grown from 89TB to 100TB.  The xfs filesystem utilizes an external logdev.  After the underlying device was grown by approx 11TB, we ran xfs_growfs –d <filesystem_mount_point>.  This command completed without errors, but the filesystem immediately went into a bad state.

We are running xfsprogs-2.8.20-1.el5.centos on RHEL Kernel Linux 2.6.18-53.1.19.el5 #1 SMP Tue Apr 22 03:01:10 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

We killed the xfs_repair before it was able to find a secondary superblock and make things worse.

Currently the underlying block device is:

--- Logical volume ---
  LV Name                /dev/logfs-sessions/sessions
  VG Name                logfs-sessions
  LV UUID                32TRbe-OIDw-u4aH-fUmD-FLmU-5jRv-MaVYDg
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                100.56 TB
  Current LE             26360253
  Segments               18
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:61

At this point what are our options to recover this filesystem?

Thank you for any help you may be able to provide.

Jason Vagalatos

From: Joe Allen
Sent: Wednesday, February 24, 2010 9:15 AM
To: Jason Vagalatos
Subject: Fwd: xfs_growfs failure....


wierd. he seems to be implying we were at 110tb and stuff is written there. I guess we need to be 100% sure of the space allocated. were the other Luns ever attached ?




Begin forwarded message:
From: Dave Chinner <david@fromorbit.com<mailto:david@fromorbit.com>>ng
Date: February 24, 2010 3:54:20 AM PST
To: Joe Allen <Joe.Allen@citrix.com<mailto:Joe.Allen@citrix.com>>
Cc: "xfs@oss.sgi.com<mailto:xfs@oss.sgi.com>" <xfs@oss.sgi.com<mailto:xfs@oss.sgi.com>>
Subject: Re: xfs_growfs failure....
On Wed, Feb 24, 2010 at 02:44:37AM -0800, Joe Allen wrote:

I am in some difficulty here over a 100TB filesystem that
Is now unusable after a xfs_growfs command.

Is there someone that might help assist?

#mount: /dev/logfs-sessions/sessions: can't read superblock

Filesystem "dm-61": Disabling barriers, not supported with external log device
attempt to access beyond end of device
dm-61: rw=0, want=238995038208, limit=215943192576

You've grown the filesystem to 238995038208 ѕectors (111.3TiB),
but the underlying device is only 215943192576 sectors (100.5TiB)
in size.

I'm assuming that you're trying to mount the filesystem after a
reboot? I make this assumption as growfs is an online operation and
won't grow if the underlying block device has not already been
grown. For a subsequent mount to fail with the underlying device
being too small, something about the underlying block
device had to change....

What kernel version and xfsprogs version are you using?


xfs_repair -n <device> basically looks for superblocks (phase 1 I
guess ) for a long time. I'm letting it run, but not much hope.

Don't repair the filesystem - there is nothing wrong with it
unless you start modifying stuff. What you need to do is fix the
underlying device to bring it back to the size it was supposed to
be at when the grow operation was run.

What does /proc/partitions tell you about the size of dm-61? does
that report the correct size, and if it does, what is it?


I'm hesitant to run xfs_repair -L  or without the -n flag for fear of making it worse.

Good - don't run anything like that until you sort out whether the
underlying device is correctly sized or not.


-bash-3.1# xfs_db -r -c 'sb 0' -c p /dev/logfs-sessions/sessions
magicnum = 0x58465342
blocksize = 4096
dblocks = 29874379776

XFS definitely thinks it is 111.3TiB in size.


rblocks = 0
rextents = 0
uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
logstart = 0
rootino = 2048
rbmino = 2049
rsumino = 2050
rextsize = 384
agblocks = 268435328
agcount = 112

112 AGs of 1TiB each - that confirms the grow succeeded and it was
able to write metadata to disk between 100 and 111 TiB without
errors being reported. That implies the block device must have been
that big at some point...


rbmblocks = 0
logblocks = 32000
versionnum = 0x3184
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 28
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 7291520
ifree = 8514
fdblocks = 6623185597

With 24.5TiB of free space


-bash-3.1# xfs_db -r -c 'sb 2' -c p /dev/logfs-sessions/sessions
magicnum = 0x58465342
blocksize = 4096
dblocks = 24111418368

That's 89.9TiB...


rblocks = 0
rextents = 0
uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
logstart = 0
rootino = 2048
rbmino = 2049
rsumino = 2050
rextsize = 384
agblocks = 268435328
agcount = 90

And 90 AGs. That tells me the filesystem was created as a 90TiB
filesystem. Can you tell me if you attempted to grow from 90TiB to
100TiB or from 100TiB to 110TiB?  There were bugs at one point in
both the userspace grow code and the kernel code that resulted in
bad grows (hence the need to know the versions this occurred on
and what you were actually attempting to do), but these problems
can usually be fixed up with some xfs_db magic.

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com<mailto:david@fromorbit.com>

[-- Attachment #1.2: Type: text/html, Size: 22021 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs failure....

[Jason Vagalatos]

[-- Attachment #1.1: Type: text/plain, Size: 6422 bytes --]

David,
This might provide some useful insight too..  I just remembered that the xfs_growfs command was run twice.  The first time it errored because I omitted the –d option.  I reran it with the –d option and it completed successfully.

Is it possible the xfs_growfs grew the filesystem 2x intended size by running it twice?  It seems that the command would fail the second time because all of the remaining space on the underlying device was used up by the first run?

Thanks,

Jason Vagalatos
Storage Administrator
Citrix|Online
7408 Hollister Avenue
Goleta California 93117
T:  805.690.2943 | M:  805.403.9433
jason.vagalatos@citrixonline.com<mailto:jason.vagalatos@citrixonline.com>
http://www.citrixonline.com<http://www.citrixonline.com/>

From: Jason Vagalatos
Sent: Wednesday, February 24, 2010 9:57 AM
To: 'david@fromorbit.com'
Cc: 'xfs@oss.sgi.com'; Joe Allen
Subject: RE: xfs_growfs failure....


Hi David,
I’m picking this up from Joe.  I’ll attempt to answer your questions.

The underlying device was grown from 89TB to 100TB.  The xfs filesystem utilizes an external logdev.  After the underlying device was grown by approx 11TB, we ran xfs_growfs –d <filesystem_mount_point>.  This command completed without errors, but the filesystem immediately went into a bad state.

We are running xfsprogs-2.8.20-1.el5.centos on RHEL Kernel Linux 2.6.18-53.1.19.el5 #1 SMP Tue Apr 22 03:01:10 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

We killed the xfs_repair before it was able to find a secondary superblock and make things worse.

Currently the underlying block device is:

--- Logical volume ---
  LV Name                /dev/logfs-sessions/sessions
  VG Name                logfs-sessions
  LV UUID                32TRbe-OIDw-u4aH-fUmD-FLmU-5jRv-MaVYDg
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                100.56 TB
  Current LE             26360253
  Segments               18
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:61

At this point what are our options to recover this filesystem?

Thank you for any help you may be able to provide.

Jason Vagalatos

From: Joe Allen
Sent: Wednesday, February 24, 2010 9:15 AM
To: Jason Vagalatos
Subject: Fwd: xfs_growfs failure....


wierd. he seems to be implying we were at 110tb and stuff is written there. I guess we need to be 100% sure of the space allocated. were the other Luns ever attached ?




Begin forwarded message:
From: Dave Chinner <david@fromorbit.com<mailto:david@fromorbit.com>>ng
Date: February 24, 2010 3:54:20 AM PST
To: Joe Allen <Joe.Allen@citrix.com<mailto:Joe.Allen@citrix.com>>
Cc: "xfs@oss.sgi.com<mailto:xfs@oss.sgi.com>" <xfs@oss.sgi.com<mailto:xfs@oss.sgi.com>>
Subject: Re: xfs_growfs failure....
On Wed, Feb 24, 2010 at 02:44:37AM -0800, Joe Allen wrote:
I am in some difficulty here over a 100TB filesystem that
Is now unusable after a xfs_growfs command.

Is there someone that might help assist?

#mount: /dev/logfs-sessions/sessions: can't read superblock

Filesystem "dm-61": Disabling barriers, not supported with external log device
attempt to access beyond end of device
dm-61: rw=0, want=238995038208, limit=215943192576

You've grown the filesystem to 238995038208 ѕectors (111.3TiB),
but the underlying device is only 215943192576 sectors (100.5TiB)
in size.

I'm assuming that you're trying to mount the filesystem after a
reboot? I make this assumption as growfs is an online operation and
won't grow if the underlying block device has not already been
grown. For a subsequent mount to fail with the underlying device
being too small, something about the underlying block
device had to change....

What kernel version and xfsprogs version are you using?

xfs_repair -n <device> basically looks for superblocks (phase 1 I
guess ) for a long time. I'm letting it run, but not much hope.

Don't repair the filesystem - there is nothing wrong with it
unless you start modifying stuff. What you need to do is fix the
underlying device to bring it back to the size it was supposed to
be at when the grow operation was run.

What does /proc/partitions tell you about the size of dm-61? does
that report the correct size, and if it does, what is it?

I'm hesitant to run xfs_repair -L  or without the -n flag for fear of making it worse.

Good - don't run anything like that until you sort out whether the
underlying device is correctly sized or not.

-bash-3.1# xfs_db -r -c 'sb 0' -c p /dev/logfs-sessions/sessions
magicnum = 0x58465342
blocksize = 4096
dblocks = 29874379776

XFS definitely thinks it is 111.3TiB in size.

rblocks = 0
rextents = 0
uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
logstart = 0
rootino = 2048
rbmino = 2049
rsumino = 2050
rextsize = 384
agblocks = 268435328
agcount = 112

112 AGs of 1TiB each - that confirms the grow succeeded and it was
able to write metadata to disk between 100 and 111 TiB without
errors being reported. That implies the block device must have been
that big at some point...

rbmblocks = 0
logblocks = 32000
versionnum = 0x3184
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 28
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 7291520
ifree = 8514
fdblocks = 6623185597

With 24.5TiB of free space

-bash-3.1# xfs_db -r -c 'sb 2' -c p /dev/logfs-sessions/sessions
magicnum = 0x58465342
blocksize = 4096
dblocks = 24111418368

That's 89.9TiB...

rblocks = 0
rextents = 0
uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
logstart = 0
rootino = 2048
rbmino = 2049
rsumino = 2050
rextsize = 384
agblocks = 268435328
agcount = 90

And 90 AGs. That tells me the filesystem was created as a 90TiB
filesystem. Can you tell me if you attempted to grow from 90TiB to
100TiB or from 100TiB to 110TiB?  There were bugs at one point in
both the userspace grow code and the kernel code that resulted in
bad grows (hence the need to know the versions this occurred on
and what you were actually attempting to do), but these problems
can usually be fixed up with some xfs_db magic.

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com<mailto:david@fromorbit.com>

[-- Attachment #1.2: Type: text/html, Size: 26164 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs failure....

[Joe Allen]

Thanks so much for the help interpreting. 
We are extremely grateful for your help. 
I have tried to include some more information you all suggest might help:


>> It looks like that the filesystem was "grown" from ~92TiB to
~114TiB on a storage device that is reported as ~103TiB
long. Again, very strange.

cat /proc/partitions
[snip]
253    61 107971596288 dm-61

= ~100TB. 

-bash-3.1# lvdisplay
--- Logical volume ---
  LV Name                /dev/logfs-sessions/sessions
  VG Name                logfs-sessions
  LV UUID                32TRbe-OIDw-u4aH-fUmD-FLmU-5jRv-MaVYDg
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                100.56 TB
  Current LE             26360253
  Segments               18
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:61

--- Logical volume ---
  LV Name                /dev/logfs-sessions/logdev
  VG Name                logfs-sessions
  LV UUID                cRDvJx-3QMS-VI7X-Oqj1-bGDA-ACki-quXpve
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                5.00 GB
  Current LE             1281
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:60


>>112 AGs of 1TiB each - that confirms the grow succeeded and it was able to write metadata to disk 
>>between 100 and 111 TiB without errors being reported. That implies the block device must have been that big at some point...

There were never 110 TB; only 100 were ever there...so I am not clear on this point. 

>>My impression is that not enough history/context has been
>>provided to enable a good guess at what has happened and how to
>>undo the consequent damage.You suggested more context might help: 


These were the commands run: 
 
pvcreate /dev/dm-50 /dev/dm-51 /dev/dm-52 /dev/dm-53 /dev/dm-54 /dev/dm-55
vgextend logfs-sessions /dev/dm-50 /dev/dm-51 /dev/dm-52 /dev/dm-53 /dev/dm-54 /dev/dm-55
lvextend -i 3 -I 512 -l +1406973 /dev/logfs-sessions/sessions /dev/dm-50 /dev/dm-51 /dev/dm-52
lvextend -i 3 -I 512 -l +1406973 /dev/logfs-sessions/sessions /dev/dm-53 /dev/dm-54 /dev/dm-55
xfs_growfs  /u01 (which failed)
xfs_growfs -d /u01 (which did not error out)
touch /u01/a


I am sorry I don't have the output of the xfs_growfs command any longer. 
Very shortly after someone noticed the filesystem was essentially offline -- input output error. 
Tried unmounting but couldn't... got out of memory errors even when doing ls.  
Tried rebooting and now FS is off line. 

The FS was 90TB, the purpose of the exercise was to grow it to 100TB. 

This is:

-bash-3.1# uname -a
Linux xx.com 2.6.18-53.1.19.el5 #1 SMP Tue Apr 22 03:01:10 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

rpm -qa | grep xfs
kmod-xfs-0.4-1.2.6.18_8.1.1.el5.centos.plus.1
xfsprogs-2.8.20-1.el5.centos

I've read about a case where Mr Chinner used xfd_db to set agcount and in some cases fix things up.
Don't know if I am a candidate for that approach... 
If there is any information I can gather I am more than ready to do that. 




-----Original Message-----
From: xfs-bounces@oss.sgi.com [mailto:xfs-bounces@oss.sgi.com] On Behalf Of Peter Grandi
Sent: Wednesday, February 24, 2010 9:10 AM
To: Linux XFS
Subject: Re: xfs_growfs failure....

> I am in some difficulty here over a 100TB filesystem

Shrewd idea! Because 'fsck' takes no time and memory, so the
bigger the filesystem the better! ;-).

> that Is now unusable after a xfs_growfs command. [ ... ] 

Wondering how long it took to backup 100TB; but of course doing
a 'grow' is guaranteed to be error free, so there :-).

> attempt to access beyond end of device dm-61: rw=0,
> want=238995038208, limit=215943192576

It looks like the underlying DM logical volume is smaller than
the new size of the filesystem, which is strange as 'xfs_growfs'
is supposed to fetch the size of the underlying block device if
none is specified explicitly on the command line. The different
is about 10% or 10TB, so it is far from trivial.

Looking at the superblock dumps there are some pretty huge
discrepancies, 

  -bash-3.1# xfs_db -r -c 'sb 0' -c p /dev/logfs-sessions/sessions
  magicnum = 0x58465342
  blocksize = 4096
  dblocks = 29874379776
  rblocks = 0
  rextents = 0
  uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
  logstart = 0
  rootino = 2048
  rbmino = 2049
  rsumino = 2050
  rextsize = 384
  agblocks = 268435328
  agcount = 112
  [ ... ]
  -bash-3.1# xfs_db -r -c 'sb 2' -c p /dev/logfs-sessions/sessions
  magicnum = 0x58465342
  blocksize = 4096
  dblocks = 24111418368
  rblocks = 0
  rextents = 0
  uuid = fc8bdf76-d962-43c1-ae60-b85f378978a6
  logstart = 0
  rootino = 2048
  rbmino = 2049
  rsumino = 2050
  rextsize = 384
  agblocks = 268435328
  agcount = 90
  [ ... ]

The 'dblocks' field is rather different, even if the 'uuid' and
'agblocks' is the same, and 'agcount' is also rather different.
In SB 0 'dblocks' 29874379776 means size 238995038208, which is
value of 'want' above. The products of 'agcount' and 'agblocks'
fit with the sizes.

It looks like that the filesystem was "grown" from ~92TiB to
~114TiB on a storage device that is reported as ~103TiB
long. Again, very strange.

My impression is that not enough history/context has been
provided to enable a good guess at what has happened and how to
undo the consequent damage.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs failure....

[Peter Grandi]

[ ... ]

>> It looks like that the filesystem was "grown" from ~92TiB to
>> ~114TiB on a storage device that is reported as ~103TiB
>> long. Again, very strange.

> cat /proc/partitions
> [snip]
> 253    61 107971596288 dm-61
> = ~100TB. 

That's the same as "limit=215943192576".

> -bash-3.1# lvdisplay
> --- Logical volume ---
>   LV Name                /dev/logfs-sessions/sessions
>   VG Name                logfs-sessions
>   LV UUID                32TRbe-OIDw-u4aH-fUmD-FLmU-5jRv-MaVYDg
>   LV Write Access        read/write
>   LV Status              available
>   # open                 0
>   LV Size                100.56 TB
>   Current LE             26360253
>   Segments               18
>   Allocation             inherit
>   Read ahead sectors     0
>   Block device           253:61

> --- Logical volume ---
>   LV Name                /dev/logfs-sessions/logdev
>   VG Name                logfs-sessions
>   LV UUID                cRDvJx-3QMS-VI7X-Oqj1-bGDA-ACki-quXpve
>   LV Write Access        read/write
>   LV Status              available
>   # open                 0
>   LV Size                5.00 GB
>   Current LE             1281
>   Segments               1
>   Allocation             inherit
>   Read ahead sectors     0
>   Block device           253:60


>> 112 AGs of 1TiB each - that confirms the grow succeeded and
>> it was able to write metadata to disk between 100 and 111 TiB
>> without errors being reported. That implies the block device
>> must have been that big at some point...

> There were never 110 TB; only 100 were ever there...so I am
> not clear on this point.

Not clear here too; because if the fs was grown to 110TB that
number must have come from somewhere.

[ ... ]

> These were the commands run: 
 
> pvcreate /dev/dm-50 /dev/dm-51 /dev/dm-52 /dev/dm-53 /dev/dm-54 /dev/dm-55
> vgextend logfs-sessions /dev/dm-50 /dev/dm-51 /dev/dm-52 /dev/dm-53 /dev/dm-54 /dev/dm-55
> lvextend -i 3 -I 512 -l +1406973 /dev/logfs-sessions/sessions /dev/dm-50 /dev/dm-51 /dev/dm-52
> lvextend -i 3 -I 512 -l +1406973 /dev/logfs-sessions/sessions /dev/dm-53 /dev/dm-54 /dev/dm-55

So you extended the LV twice, starting from 23,546,307 4MiB LEs
('dblocks=24111418368'), by 1,406,973 LEs each time.

That is consistent with the whole story, not at all obvious how
comes SB 0 got 'dblocks=29874379776' when 'dm-61' has 26360253
4MiB LEs equivalent to 26992899072 dblocks, for a difference of
11,255,784MiB. Especially as the size in SB0 tallies with the
number of AGs in it.

So far everything fine.

> xfs_growfs /u01 (which failed)
> xfs_growfs -d /u01 (which did not error out)
> touch /u01/a

> -bash-3.1# uname -a
> Linux xx.com 2.6.18-53.1.19.el5 #1 SMP Tue Apr 22 03:01:10 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

That seems to imply RH51, which is somewhat old.

> rpm -qa | grep xfs
> kmod-xfs-0.4-1.2.6.18_8.1.1.el5.centos.plus.1
> xfsprogs-2.8.20-1.el5.centos

BTW, I would use nowadays the 'elrepo' 'kmod' packages rather than
the CentOSPlus ones (even if the 'kmod-xfs' main version is the
same). And fortunately you are using 64b arch or else you'd never
be able able to 'fsck' the filesystem (it could still take weeks).

Also on my system I have 'xfsprogs-2.9.4-1.el5.centos' which is
rather newer. I wonder if there was some bug fix for 'growfs'
between 2.8.0 and 2.9.4.

> I've read about a case where Mr Chinner used xfd_db to set
> agcount and in some cases fix things up. Don't know if I am a
> candidate for that approach...

The good news from the SB dumps is that SB 2 is basically the
old one for the 90TB filesystem, and that growing the filesystem
does not actually change anything much in it, just the top level
metadata saying how big the filesystem is. It is likely that if
you revert to the SB 2 you should get back the 90TB version of
your filesystem untouched. Then you got to repair it anyhow, to
ensure that it is consistent, and that could take weeks.

The bad news is that for some reason 'growfs' thought the block
device was 110TB, and that's a problem, because perhaps that
strange number will surface again.

One wild guess is that something happened here:

  > xfs_growfs /u01 (which failed)
  > xfs_growfs -d /u01 (which did not error out)

during the first operation, given that you have an external log,
which *may* have caused trouble. Or else some 'growfs' bug in
2.8.0.

  BTW there is also that a 90-110TB single filesystem seems to me
  rather unwise, never mind one that spans linearly several block
  devices, which seems to me extremely avoidable (euphemism)
  unless the contents are entirely disposable.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs failure....

[Dave Chinner]

On Wed, Feb 24, 2010 at 10:37:30AM -0800, Joe Allen wrote:
> Thanks so much for the help interpreting. 
> We are extremely grateful for your help. 
> I have tried to include some more information you all suggest might help:
> 
> 
> >> It looks like that the filesystem was "grown" from ~92TiB to
> ~114TiB on a storage device that is reported as ~103TiB
> long. Again, very strange.
> 
> cat /proc/partitions
> [snip]
> 253    61 107971596288 dm-61
> 
> = ~100TB. 

OK.

> >>112 AGs of 1TiB each - that confirms the grow succeeded and it was able to write metadata to disk 
> >>between 100 and 111 TiB without errors being reported. That
> >>implies the block device must have been that big at some
> >>point...
> 
> There were never 110 TB; only 100 were ever there...so I am not clear on this point. 

Well, XFS writes the new AG header metadata synchronously to the
expanded region before making the superblock changes. If they didn't
error out, either:

	a. the block device was large enough
	b. the writes were silently ignored by the block device
	   (buggy block device)
	c. the block device wrapped them back around to the front of
	   the device (buggy block device) and overwrote stuff => fs
	   corruption.

SO regardless of what actually happened, you're going to need to
run xfs_repair after the main superblock is fixed up.

> >>My impression is that not enough history/context has been
> >>provided to enable a good guess at what has happened and how to
> >>undo the consequent damage.You suggested more context might help: 
> 
> 
> These were the commands run: 
>  
> pvcreate /dev/dm-50 /dev/dm-51 /dev/dm-52 /dev/dm-53 /dev/dm-54 /dev/dm-55
> vgextend logfs-sessions /dev/dm-50 /dev/dm-51 /dev/dm-52 /dev/dm-53 /dev/dm-54 /dev/dm-55
> lvextend -i 3 -I 512 -l +1406973 /dev/logfs-sessions/sessions /dev/dm-50 /dev/dm-51 /dev/dm-52
> lvextend -i 3 -I 512 -l +1406973 /dev/logfs-sessions/sessions /dev/dm-53 /dev/dm-54 /dev/dm-55
> xfs_growfs  /u01 (which failed)
> xfs_growfs -d /u01 (which did not error out)
> touch /u01/a

Ok, 2.8.20 is old enough to have the userspace growfs bugs, and I'd
say the kernel is old enough to have the growfs bugs as well. It is
entirely possible that executing it twice was the cause of this.

> I am sorry I don't have the output of the xfs_growfs command any longer. 
> Very shortly after someone noticed the filesystem was essentially offline -- input output error. 
> Tried unmounting but couldn't... got out of memory errors even when doing ls.  
> Tried rebooting and now FS is off line. 
> 
> The FS was 90TB, the purpose of the exercise was to grow it to 100TB. 
> 
> This is:
> 
> -bash-3.1# uname -a
> Linux xx.com 2.6.18-53.1.19.el5 #1 SMP Tue Apr 22 03:01:10 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
> 
> rpm -qa | grep xfs
> kmod-xfs-0.4-1.2.6.18_8.1.1.el5.centos.plus.1
> xfsprogs-2.8.20-1.el5.centos

> I've read about a case where Mr Chinner used xfd_db to set agcount
> and in some cases fix things up.
> Don't know if I am a candidate for that approach... 

That's exactly what is has to be done to get the superblock back
into the correct shape to enable a repair run to occur.

First, upgrade your userspace tools to the latest so you pick up all
the xfs_repair speedups and memory usage reductions. Then
calculate the correct block count for 100AGs (dblocks = agcount *
agblocks), modify the agcount and dblocks fields using xfs_db, then 
xfs_repair -n to confirm that it finds the superblock valid.

If the superblock is valid, you can then probably mount the
filesystem to replay the log. Unmount immediately afterwards. Note
that this may replay the bad growfs transaction, so you might need
to use xfs_db to reset the superblock again. Once this is done you
can run repair for real after which you should have a usable
filesytem.

This won't have restored the filesystem to the exact size of the
underlying block device - a subsequent grow would be needed to
extend the fs to include a partial AG at the end to use the
remaining space.

Good luck!

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs failure....

[Joe Allen]

Dave & Peter 

We were successfully able to get the FS back on line using the approach you outlined below. 
Thank you very much for your invaluable assistance. As long as we are in DR, 
We'll upgrade along the lines suggested. 

Thanks again!



On Wed, Feb 24, 2010 at 10:37:30AM -0800, Joe Allen wrote:
> Thanks so much for the help interpreting. 
> We are extremely grateful for your help. 
> I have tried to include some more information you all suggest might help:
> 
> 
> >> It looks like that the filesystem was "grown" from ~92TiB to
> ~114TiB on a storage device that is reported as ~103TiB
> long. Again, very strange.
> 
> cat /proc/partitions
> [snip]
> 253    61 107971596288 dm-61
> 
> = ~100TB. 

OK.

> >>112 AGs of 1TiB each - that confirms the grow succeeded and it was able to write metadata to disk 
> >>between 100 and 111 TiB without errors being reported. That
> >>implies the block device must have been that big at some
> >>point...
> 
> There were never 110 TB; only 100 were ever there...so I am not clear on this point. 

Well, XFS writes the new AG header metadata synchronously to the
expanded region before making the superblock changes. If they didn't
error out, either:

	a. the block device was large enough
	b. the writes were silently ignored by the block device
	   (buggy block device)
	c. the block device wrapped them back around to the front of
	   the device (buggy block device) and overwrote stuff => fs
	   corruption.

SO regardless of what actually happened, you're going to need to
run xfs_repair after the main superblock is fixed up.

> >>My impression is that not enough history/context has been
> >>provided to enable a good guess at what has happened and how to
> >>undo the consequent damage.You suggested more context might help: 
> 
> 
> These were the commands run: 
>  
> pvcreate /dev/dm-50 /dev/dm-51 /dev/dm-52 /dev/dm-53 /dev/dm-54 /dev/dm-55
> vgextend logfs-sessions /dev/dm-50 /dev/dm-51 /dev/dm-52 /dev/dm-53 /dev/dm-54 /dev/dm-55
> lvextend -i 3 -I 512 -l +1406973 /dev/logfs-sessions/sessions /dev/dm-50 /dev/dm-51 /dev/dm-52
> lvextend -i 3 -I 512 -l +1406973 /dev/logfs-sessions/sessions /dev/dm-53 /dev/dm-54 /dev/dm-55
> xfs_growfs  /u01 (which failed)
> xfs_growfs -d /u01 (which did not error out)
> touch /u01/a

Ok, 2.8.20 is old enough to have the userspace growfs bugs, and I'd
say the kernel is old enough to have the growfs bugs as well. It is
entirely possible that executing it twice was the cause of this.

> I am sorry I don't have the output of the xfs_growfs command any longer. 
> Very shortly after someone noticed the filesystem was essentially offline -- input output error. 
> Tried unmounting but couldn't... got out of memory errors even when doing ls.  
> Tried rebooting and now FS is off line. 
> 
> The FS was 90TB, the purpose of the exercise was to grow it to 100TB. 
> 
> This is:
> 
> -bash-3.1# uname -a
> Linux xx.com 2.6.18-53.1.19.el5 #1 SMP Tue Apr 22 03:01:10 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
> 
> rpm -qa | grep xfs
> kmod-xfs-0.4-1.2.6.18_8.1.1.el5.centos.plus.1
> xfsprogs-2.8.20-1.el5.centos

> I've read about a case where Mr Chinner used xfd_db to set agcount
> and in some cases fix things up.
> Don't know if I am a candidate for that approach... 

That's exactly what is has to be done to get the superblock back
into the correct shape to enable a repair run to occur.

First, upgrade your userspace tools to the latest so you pick up all
the xfs_repair speedups and memory usage reductions. Then
calculate the correct block count for 100AGs (dblocks = agcount *
agblocks), modify the agcount and dblocks fields using xfs_db, then 
xfs_repair -n to confirm that it finds the superblock valid.

If the superblock is valid, you can then probably mount the
filesystem to replay the log. Unmount immediately afterwards. Note
that this may replay the bad growfs transaction, so you might need
to use xfs_db to reset the superblock again. Once this is done you
can run repair for real after which you should have a usable
filesytem.

This won't have restored the filesystem to the exact size of the
underlying block device - a subsequent grow would be needed to
extend the fs to include a partial AG at the end to use the
remaining space.

Good luck!

Cheers,

Dave.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs