All of lore.kernel.org
 help / color / mirror / Atom feed
* Recover file after truncate
@ 2017-07-26  0:48 Andy Bennett
  2017-07-26 11:15 ` Brian Foster
  0 siblings, 1 reply; 7+ messages in thread
From: Andy Bennett @ 2017-07-26  0:48 UTC (permalink / raw)
  To: linux-xfs

Hi,

I have a file (~6GiB) that I was doing some analysis on. In the course of 
this I managed to do 'xxd something-else my-file' and overwrote it with < 
4.0K of data.


Running xfs_bmap on it gives me:

-----
my-file:
        0: [0..7]: 236935320..236935327
-----

Running xfs_bmap on another, similar big, file that was created around the 
same time gives me:

-----
other-file:
        0: [0..4194175]: 335806496..340000671
        1: [4194176..13828055]: 342870632..352504511
-----


A quick experiment with xxd suggests that the inode of the file remains the 
same if it exists before xxd writes to it. So my new file still has the 
same inode as the one I want to recover.

There hasn't been any write activity on this file system since I made this 
mistake.


Is there any hope of recovering any part of the original data?

I have an xxd dump of the first 624 bytes of the original file and there 
are some recurring features in it.

Grepping through the partition for that signature gives me this:

-----
$ sudo grep -obUaP "\x66\x66\xa2" /dev/nvme0n1p8 |tee TRACE
grep: exceeded PCRE's line length limit
519554640:ff¢
4377654787:ff¢
7961215381:ff¢
10165641473:ff¢
10849981825:ff¢
17851384491:ff¢
23231901998:ff¢
33898050969:ff¢
41781142596:ff¢
51651699392:ff¢
56040569029:ff¢
56277711167:ff¢
56814897544:ff¢
61037797435:ff¢
61269592210:ff¢
73946170693:ff¢
75199462354:ff¢
76071192135:ff¢
-----


The partition is on an NVMe SSD.


I'm not sure how to use xfs_logprint but converting the inode number to hex 
and grepping through doesn't seem to give me any matches.



Is there a way to find out how this file was allocated and restore it or 
fish the data out some how?

My mount options are

-----
/dev/nvme0n1p8 on /var/spool type xfs (rw,relatime,attr2,inode64,noquota)
-----

...and I'm running under Debian Jessie.

I've not (knowingly) got any manual TRIM or DISCARD jobs that will run.



What's the best way of working out what current files the offsets from grep 
correspond to? I guess the ones that don't correspond to a current file 
might be my data?



Thanks for anything you can do to help.




Regards,
@ndy

-- 
andyjpb@ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: Recover file after truncate
  2017-07-26  0:48 Recover file after truncate Andy Bennett
@ 2017-07-26 11:15 ` Brian Foster
  2017-07-26 22:12   ` Andy Bennett
  0 siblings, 1 reply; 7+ messages in thread
From: Brian Foster @ 2017-07-26 11:15 UTC (permalink / raw)
  To: Andy Bennett; +Cc: linux-xfs

On Wed, Jul 26, 2017 at 01:48:59AM +0100, Andy Bennett wrote:
> Hi,
> 
> I have a file (~6GiB) that I was doing some analysis on. In the course of
> this I managed to do 'xxd something-else my-file' and overwrote it with <
> 4.0K of data.
> 

This basically truncated the file and allocated new blocks for the xxd
output.

> 
> Running xfs_bmap on it gives me:
> 
> -----
> my-file:
>        0: [0..7]: 236935320..236935327
> -----
> 
> Running xfs_bmap on another, similar big, file that was created around the
> same time gives me:
> 
> -----
> other-file:
>        0: [0..4194175]: 335806496..340000671
>        1: [4194176..13828055]: 342870632..352504511

A ~2GB extent followed by a ~4.5GB extent.

> -----
> 
> 
> A quick experiment with xxd suggests that the inode of the file remains the
> same if it exists before xxd writes to it. So my new file still has the same
> inode as the one I want to recover.
> 
> There hasn't been any write activity on this file system since I made this
> mistake.
> 
> 
> Is there any hope of recovering any part of the original data?
> 

I suspect that most of your file data currently sits on free blocks in
the filesystem. If you had a bmap from before this incident, you could
probably point to exactly where they are. Since I assume that is
unlikely, you'd have to run something like you have below to try and
identify where that data is in the block device and try to piece
together the starting block and length of the associated extents.

I suppose that if you can distinguish the file data in any particular 4k
block from other random data then it might be possible to identify the
exact start/length values of the extents directly from the block device.
If you are luckier still and the original file was allocated with only
one or two extents similar to the above example, then you may be able to
piece together the file by putting the extents in the appropriate order.
FWIW, there are tools such as photorec that supposedly are able to do
this kind of thing with known file formats.

Note that I don't think there's any way to reassign the blocks to the
inode without doing some form of manual surgery on the filesystem, which
I would not recommend. If you actually were able to locate the file data
on the raw block device, the best course of action is to copy the data
from the block device to a new file on a separate filesystem and then
copy that file back into the original fs. Also note that once you start
writing anything to the original fs, the original data is at risk of
being overwritten. You may want to decommission the original storage or
make a copy to preserve the original state.

> I have an xxd dump of the first 624 bytes of the original file and there are
> some recurring features in it.
> 
> Grepping through the partition for that signature gives me this:
> 
> -----
> $ sudo grep -obUaP "\x66\x66\xa2" /dev/nvme0n1p8 |tee TRACE
> grep: exceeded PCRE's line length limit
> 519554640:ff¢
> 4377654787:ff¢
> 7961215381:ff¢
> 10165641473:ff¢
> 10849981825:ff¢
> 17851384491:ff¢
> 23231901998:ff¢
> 33898050969:ff¢
> 41781142596:ff¢
> 51651699392:ff¢
> 56040569029:ff¢
> 56277711167:ff¢
> 56814897544:ff¢
> 61037797435:ff¢
> 61269592210:ff¢
> 73946170693:ff¢
> 75199462354:ff¢
> 76071192135:ff¢
> -----
> 
> 
> The partition is on an NVMe SSD.
> 
> 
> I'm not sure how to use xfs_logprint but converting the inode number to hex
> and grepping through doesn't seem to give me any matches.
> 

What does the xfs_logprint output look like? Also, what is the inode
number of the original file? It is possible to get a hint on the
startblock and length of the original extent(s) if the log still has
EFI/EFD items present that describe the extent free operations, and that
information can be corroborated against either the inode information
and/or with the regions where file data is found in the blockdev scan.

Do note that the simple act of mounting the filesystem runs the risk of
overwriting previous log data. An 'xfs_metadump -go' might be a good way
to preserve current log content.

Brian

> 
> 
> Is there a way to find out how this file was allocated and restore it or
> fish the data out some how?
> 
> My mount options are
> 
> -----
> /dev/nvme0n1p8 on /var/spool type xfs (rw,relatime,attr2,inode64,noquota)
> -----
> 
> ...and I'm running under Debian Jessie.
> 
> I've not (knowingly) got any manual TRIM or DISCARD jobs that will run.
> 
> 
> 
> What's the best way of working out what current files the offsets from grep
> correspond to? I guess the ones that don't correspond to a current file
> might be my data?
> 
> 
> 
> Thanks for anything you can do to help.
> 
> 
> 
> 
> Regards,
> @ndy
> 
> -- 
> andyjpb@ashurst.eu.org
> http://www.ashurst.eu.org/
> 0x7EBA75FF
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: Recover file after truncate
  2017-07-26 11:15 ` Brian Foster
@ 2017-07-26 22:12   ` Andy Bennett
  2017-07-27 12:43     ` Brian Foster
  0 siblings, 1 reply; 7+ messages in thread
From: Andy Bennett @ 2017-07-26 22:12 UTC (permalink / raw)
  To: Brian Foster; +Cc: linux-xfs

Hi,


>> A quick experiment with xxd suggests that the inode of the 
>> file remains the
>> same if it exists before xxd writes to it. So my new file 
>> still has the same
>> inode as the one I want to recover.
>> 
>> There hasn't been any write activity on this file system since 
>> I made this ...
>
> I suspect that most of your file data currently sits on free blocks in
> the filesystem. If you had a bmap from before this incident, you could
> probably point to exactly where they are. Since I assume that is
> unlikely,

Yeah... I captured the file and subsequently managed to delete it before I 
had a chance to run a backup! :-(


> you'd have to run something like you have below to try and
> identify where that data is in the block device and try to piece
> together the starting block and length of the associated extents.
>
> I suppose that if you can distinguish the file data in any particular 4k
> block from other random data then it might be possible to identify the
> exact start/length values of the extents directly from the block device.

The actual data is IQ samples that I've captured from a device I hired but 
have now sent back. ...so it's essentially random binary data and I 
wouldn't like to try to write a classifier for it but I can easily pipe it 
through my toolchain without header or framing woes and see if it produces 
output that is pleasing.

So, to that extent, if I can get in the general vicinity of the file I can 
just chop the edges around until I'm happy. Hey, my 6 minute sample was a 
little over 6GiB so I'm sure I could just listen to the whole block device 
in a few hours. :-)


> If you are luckier still and the original file was allocated with only
> one or two extents similar to the above example, then you may be able to
> piece together the file by putting the extents in the appropriate order.
> FWIW, there are tools such as photorec that supposedly are able to do
> this kind of thing with known file formats.

I'm hoping it's in a couple of extents. How do I identify extents on disk?


> Note that I don't think there's any way to reassign the blocks to the
> inode without doing some form of manual surgery on the filesystem, which
> I would not recommend. If you actually were able to locate the file data
> on the raw block device, the best course of action is to copy the data
> from the block device to a new file on a separate filesystem and then
> copy that file back into the original fs.

That sounds good, thanks.


> Also note that once you start
> writing anything to the original fs, the original data is at risk of
> being overwritten. You may want to decommission the original storage or
> make a copy to preserve the original state.


> What does the xfs_logprint output look like?

Piping `xfs_logprint /dev/nvme0n1p8 ` thru' tail gives me this:


-----
Oper (13): tid: 4d4486ed  len: 16  clientid: TRANS  flags: none
EXTENTS inode data
----------------------------------------------------------------------------
Oper (14): tid: 4d4486ed  len: 0  clientid: TRANS  flags: COMMIT 

============================================================================
cycle: 1        version: 2              lsn: 1,211073   tail_lsn: 1,211069
length of Log Record: 512       prev offset: 211069             num ops: 5
uuid: 7274488e-64ab-4d2a-8f4d-cedec4db7e6f   format: little endian linux
h_size: 32768
----------------------------------------------------------------------------
Oper (0): tid: 94e15a8b  len: 0  clientid: TRANS  flags: START 
----------------------------------------------------------------------------
Oper (1): tid: 94e15a8b  len: 16  clientid: TRANS  flags: none
TRAN:    type: SWAPEXT       tid: 94e15a8b       num_items: 2
----------------------------------------------------------------------------
Oper (2): tid: 94e15a8b  len: 24  clientid: TRANS  flags: none
BUF:  #regs: 2   start blkno: 0 (0x0)  len: 1  bmap size: 1  flags: 0x9000
Oper (3): tid: 94e15a8b  len: 384  clientid: TRANS  flags: none
SUPER BLOCK Buffer: 
icount: 6360863066640355328  ifree: 94813521  fdblks: 0  frext: 0
----------------------------------------------------------------------------
Oper (4): tid: 94e15a8b  len: 0  clientid: TRANS  flags: COMMIT 

============================================================================
cycle: 1        version: 2              lsn: 1,211075   tail_lsn: 1,211073
length of Log Record: 512       prev offset: 211073             num ops: 5
uuid: 7274488e-64ab-4d2a-8f4d-cedec4db7e6f   format: little endian linux
h_size: 32768
----------------------------------------------------------------------------
Oper (0): tid: 9f4540d0  len: 0  clientid: TRANS  flags: START 
----------------------------------------------------------------------------
Oper (1): tid: 9f4540d0  len: 16  clientid: TRANS  flags: none
TRAN:    type: SWAPEXT       tid: 9f4540d0       num_items: 2
----------------------------------------------------------------------------
Oper (2): tid: 9f4540d0  len: 24  clientid: TRANS  flags: none
BUF:  #regs: 2   start blkno: 0 (0x0)  len: 1  bmap size: 1  flags: 0x9000
Oper (3): tid: 9f4540d0  len: 384  clientid: TRANS  flags: none
SUPER BLOCK Buffer: 
icount: 6360863066640355328  ifree: 94813521  fdblks: 0  frext: 0
----------------------------------------------------------------------------
Oper (4): tid: 9f4540d0  len: 0  clientid: TRANS  flags: COMMIT 

============================================================================
xfs_logprint: skipped 159283 zeroed blocks in range: 211077 - 370359
xfs_logprint: physical end of log
============================================================================
xfs_logprint: logical end of log
============================================================================
-----


The whole log is currently 4,044,259 lines long.



> Also, what is the inode
> number of the original file?

314798163



> It is possible to get a hint on the
> startblock and length of the original extent(s) if the log still has
> EFI/EFD items present that describe the extent free operations, and that
> information can be corroborated against either the inode information
> and/or with the regions where file data is found in the blockdev scan.

Piping `xfs_logprint /dev/nvme0n1p8 ` thru' less, seeking to the end and 
searching backwards for 'EFI' gives me this:

-----
============================================================================
cycle: 1        version: 2              lsn: 1,211069   tail_lsn: 1,211067
length of Log Record: 1536      prev offset: 211067             num ops: 15
uuid: 7274488e-64ab-4d2a-8f4d-cedec4db7e6f   format: little endian linux
h_size: 32768
----------------------------------------------------------------------------
Oper (0): tid: 4d4486ed  len: 0  clientid: TRANS  flags: START 
----------------------------------------------------------------------------
Oper (1): tid: 4d4486ed  len: 16  clientid: TRANS  flags: none
TRAN:    type: SWAPEXT       tid: 4d4486ed       num_items: 12
----------------------------------------------------------------------------
Oper (2): tid: 4d4486ed  len: 32  clientid: TRANS  flags: none
EFI:  #regs: 1    num_extents: 1  id: 0xffff8bff07e21e58
(s: 0x114112b, l: 1363) 
----------------------------------------------------------------------------
Oper (3): tid: 4d4486ed  len: 32  clientid: TRANS  flags: none
EFD:  #regs: 1    num_extents: 1  id: 0xffff8bff07e21e58
----------------------------------------------------------------------------
Oper (4): tid: 4d4486ed  len: 24  clientid: TRANS  flags: none
BUF:  #regs: 2   start blkno: 1 (0x1)  len: 1  bmap size: 1  flags: 0x2800
Oper (5): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
AGF Buffer: XAGF  Out of space
----------------------------------------------------------------------------
Oper (6): tid: 4d4486ed  len: 24  clientid: TRANS  flags: none
BUF:  #regs: 2   start blkno: 8 (0x8)  len: 8  bmap size: 1  flags: 0x2000
Oper (7): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
BUF DATA
----------------------------------------------------------------------------
Oper (8): tid: 4d4486ed  len: 24  clientid: TRANS  flags: none
BUF:  #regs: 3   start blkno: 16 (0x10)  len: 8  bmap size: 1  flags: 
0x2000
Oper (9): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
BUF DATA
Oper (10): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
BUF DATA
----------------------------------------------------------------------------
Oper (11): tid: 4d4486ed  len: 56  clientid: TRANS  flags: none
INODE: #regs: 3   ino: 0x89c95dd  flags: 0x5   dsize: 16
        blkno: 144479680  len: 32  boff: 14848
Oper (12): tid: 4d4486ed  len: 176  clientid: TRANS  flags: none
INODE CORE
magic 0x494e mode 0100600 version 3 format 2
nlink 1 uid 501 gid 100
atime 0x59790f5b mtime 0x59790f6d ctime 0x59790f6d
size 0x3a9c161 nblocks 0x3a9d extsize 0x0 nextents 0x1
naextents 0x0 forkoff 0 dmevmask 0x0 dmstate 0x0
flags 0x0 gen 0x51eeced9
Oper (13): tid: 4d4486ed  len: 16  clientid: TRANS  flags: none
EXTENTS inode data
----------------------------------------------------------------------------
Oper (14): tid: 4d4486ed  len: 0  clientid: TRANS  flags: COMMIT 
-----


> Do note that the simple act of mounting the filesystem runs the risk of
> overwriting previous log data. An 'xfs_metadump -go' might be a good way
> to preserve current log content.

I've been hibernating the machine and haven't tried remounting it 
read-only: I'm not expecting anything to do any serious writes on that 
partition and I don't think there will be anything much at all that tries 
to write there.


> Brian

Thanks for your help!




Regards,
@ndy

-- 
andyjpb@ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: Recover file after truncate
  2017-07-26 22:12   ` Andy Bennett
@ 2017-07-27 12:43     ` Brian Foster
  2017-07-27 15:04       ` Brian Foster
  2017-07-30 14:41       ` Andy Bennett
  0 siblings, 2 replies; 7+ messages in thread
From: Brian Foster @ 2017-07-27 12:43 UTC (permalink / raw)
  To: Andy Bennett; +Cc: linux-xfs

On Wed, Jul 26, 2017 at 11:12:19PM +0100, Andy Bennett wrote:
> Hi,
> 
> 
...
> 
> 
> > If you are luckier still and the original file was allocated with only
> > one or two extents similar to the above example, then you may be able to
> > piece together the file by putting the extents in the appropriate order.
> > FWIW, there are tools such as photorec that supposedly are able to do
> > this kind of thing with known file formats.
> 
> I'm hoping it's in a couple of extents. How do I identify extents on disk?
> 

File extents can be listed via userspace tools like xfs_bmap, as you've
shown previously. Once blocks are freed from a file, they are returned
to per allocation group btree data structures on disk. Note that the
extents in the btree may no longer represent the exact extents allocated
to your file because other, contiguous free space could have been merged
to form larger free extents.

The free space btree can be traversed offline via the xfs_db utility.
Something like the following could give you the free extent information
for each AG in the fs.

for i in $(seq 0 <maxag>); do
	echo agno $i
	xfs_db -c "agf $i" -c "addr bnoroot" -c "btdump"  <dev>
done

You can obtain the AG count of the fs via 'xfs_info <mnt>' or:

xfs_db -c "sb 0" -c "p agcount" <dev>

If I were you, I would probably take the following high level approach
to attempt a recovery:

- grep/search through the block device for blocks that appear to be
  associated with your file.

- For each identified offset in the raw blockdev, convert the 4k block
  address to an AG number and AG relative block:

xfs_db -c "convert daddr <block> agno" -c "convert daddr <block> agbno" <dev>

- Attempt to locate a free extent that covers that block in the free
  extent output for the associated AG number. Note that all of the btump
  output collected above is in the form of AG relative block offsets, so
  each AG has a block 0, 1, etc.

- If found, convert the starting agblock of the free extent back to a
  disk address:

xfs_db -c "convert agno N agbno B daddr" <dev>

- Based on the size of the free extent, dd that number of blocks off the
  raw block device to a file on a separate device:

dd if=<dev> bs=4k skip=<freeblockaddr> count=<count> of=extentfile.N

  Verify the file contains the data you expect based on the initial
  search.

- Repeat until you've copied each candidate extent to its own file on
  the separate fs. Then try to piece the individual extent files
  together into a new file that resembles the original file, combining
  them however necessary and/or trimming the extents if the boundaries
  seem to include unrelated data, etc.

> 
...
> > What does the xfs_logprint output look like?
> 
...
> Piping `xfs_logprint /dev/nvme0n1p8 ` thru' less, seeking to the end and
> searching backwards for 'EFI' gives me this:
> 
> -----
> ============================================================================
> cycle: 1        version: 2              lsn: 1,211069   tail_lsn: 1,211067
> length of Log Record: 1536      prev offset: 211067             num ops: 15
> uuid: 7274488e-64ab-4d2a-8f4d-cedec4db7e6f   format: little endian linux
> h_size: 32768
> ----------------------------------------------------------------------------
...
> ----------------------------------------------------------------------------
> Oper (2): tid: 4d4486ed  len: 32  clientid: TRANS  flags: none
> EFI:  #regs: 1    num_extents: 1  id: 0xffff8bff07e21e58
> (s: 0x114112b, l: 1363)

A 1363 block extent == ~5MB, so probably not a candidate unless your
file was more fragmented than anticipated.

Are you filtering the xfs_logprint output here? If so, please attach the
full output from xfs_logprint.

Brian

> ----------------------------------------------------------------------------
> Oper (3): tid: 4d4486ed  len: 32  clientid: TRANS  flags: none
> EFD:  #regs: 1    num_extents: 1  id: 0xffff8bff07e21e58
> ----------------------------------------------------------------------------
> Oper (4): tid: 4d4486ed  len: 24  clientid: TRANS  flags: none
> BUF:  #regs: 2   start blkno: 1 (0x1)  len: 1  bmap size: 1  flags: 0x2800
> Oper (5): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
> AGF Buffer: XAGF  Out of space
> ----------------------------------------------------------------------------
> Oper (6): tid: 4d4486ed  len: 24  clientid: TRANS  flags: none
> BUF:  #regs: 2   start blkno: 8 (0x8)  len: 8  bmap size: 1  flags: 0x2000
> Oper (7): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
> BUF DATA
> ----------------------------------------------------------------------------
> Oper (8): tid: 4d4486ed  len: 24  clientid: TRANS  flags: none
> BUF:  #regs: 3   start blkno: 16 (0x10)  len: 8  bmap size: 1  flags: 0x2000
> Oper (9): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
> BUF DATA
> Oper (10): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
> BUF DATA
> ----------------------------------------------------------------------------
> Oper (11): tid: 4d4486ed  len: 56  clientid: TRANS  flags: none
> INODE: #regs: 3   ino: 0x89c95dd  flags: 0x5   dsize: 16
>        blkno: 144479680  len: 32  boff: 14848
> Oper (12): tid: 4d4486ed  len: 176  clientid: TRANS  flags: none
> INODE CORE
> magic 0x494e mode 0100600 version 3 format 2
> nlink 1 uid 501 gid 100
> atime 0x59790f5b mtime 0x59790f6d ctime 0x59790f6d
> size 0x3a9c161 nblocks 0x3a9d extsize 0x0 nextents 0x1
> naextents 0x0 forkoff 0 dmevmask 0x0 dmstate 0x0
> flags 0x0 gen 0x51eeced9
> Oper (13): tid: 4d4486ed  len: 16  clientid: TRANS  flags: none
> EXTENTS inode data
> ----------------------------------------------------------------------------
> Oper (14): tid: 4d4486ed  len: 0  clientid: TRANS  flags: COMMIT -----
> 
> 
> > Do note that the simple act of mounting the filesystem runs the risk of
> > overwriting previous log data. An 'xfs_metadump -go' might be a good way
> > to preserve current log content.
> 
> I've been hibernating the machine and haven't tried remounting it read-only:
> I'm not expecting anything to do any serious writes on that partition and I
> don't think there will be anything much at all that tries to write there.
> 
> 
> > Brian
> 
> Thanks for your help!
> 
> 
> 
> 
> Regards,
> @ndy
> 
> -- 
> andyjpb@ashurst.eu.org
> http://www.ashurst.eu.org/
> 0x7EBA75FF
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: Recover file after truncate
  2017-07-27 12:43     ` Brian Foster
@ 2017-07-27 15:04       ` Brian Foster
  2017-07-30 14:36         ` Andy Bennett
  2017-07-30 14:41       ` Andy Bennett
  1 sibling, 1 reply; 7+ messages in thread
From: Brian Foster @ 2017-07-27 15:04 UTC (permalink / raw)
  To: Andy Bennett; +Cc: linux-xfs

On Thu, Jul 27, 2017 at 08:43:50AM -0400, Brian Foster wrote:
> On Wed, Jul 26, 2017 at 11:12:19PM +0100, Andy Bennett wrote:
> > Hi,
> > 
> > 
...
> 
> Are you filtering the xfs_logprint output here? If so, please attach the
> full output from xfs_logprint.
> 

Andy sent the logprint output privately due to the size of the
attachment.

Andy,

The last reference I see of your inode in the log before the size goes
to zero is this:

----------------------------------------------------------------------------                                           
Oper (2): tid: c6720129  len: 56  clientid: TRANS  flags: none                                                         
INODE: #regs: 2   ino: 0x12c37053  flags: 0x1   dsize: 0   
        blkno: 235989736  len: 32  boff: 9728              
Oper (3): tid: c6720129  len: 176  clientid: TRANS  flags: none                                                        
INODE CORE                                                 
magic 0x494e mode 0100644 version 3 format 2               
nlink 1 uid 501 gid 100                                    
atime 0x5977c96e mtime 0x5973ac16 ctime 0x5973ac16         
size 0x1a81f2218 nblocks 0x1a81f3 extsize 0x0 nextents 0x2 
naextents 0x0 forkoff 0 dmevmask 0x0 dmstate 0x0           
flags 0x0 gen 0xb8791e20                                   
----------------------------------------------------------------------------

This suggests the file size was 0x1a81f2218 (7115579928) and it had
0x1a81f3 (1737203) blocks across two extents (roughly 6.6GB). Note that
the file size is not exactly aligned to the (blockcount * 4096).

Further along, the following EFI exists:

----------------------------------------------------------------------------
Oper (19): tid: 6a85c7fc  len: 48  clientid: TRANS  flags: none
EFI:  #regs: 1    num_extents: 2  id: 0xffff8bfcb4f38990
(s: 0x12a9233, l: 688643) (s: 0x34ae0d5, l: 1048560)
----------------------------------------------------------------------------

... which frees two extents that add up to 688643 + 1048560 = 1737203
(0x1a81f3) blocks. The starting filesystem blocks for these extents are
0x12a9233 and 0x34ae0d5, respectively.

If you can convert those starting fsb values to agno/agbno combinations
as shown in my previous mail, you could confirm whether they reside in
the free space btrees for the associated allocation group of each
extent. If so, you could convert each start fsb to a raw daddr (xfs_db
-c "convert fsb 0x12a9233 daddr" <dev>) and use that to try and read
each extent off the raw device as previously discussed.

I have no idea whether those extents contain the file data you're after
(note that I'm not sure we've actually confirmed your file data would
have made it to disk), but if the file size values look sane that might
be worth a try before you go fishing the device for raw data.

Brian

> Brian
> 
> > ----------------------------------------------------------------------------
> > Oper (3): tid: 4d4486ed  len: 32  clientid: TRANS  flags: none
> > EFD:  #regs: 1    num_extents: 1  id: 0xffff8bff07e21e58
> > ----------------------------------------------------------------------------
> > Oper (4): tid: 4d4486ed  len: 24  clientid: TRANS  flags: none
> > BUF:  #regs: 2   start blkno: 1 (0x1)  len: 1  bmap size: 1  flags: 0x2800
> > Oper (5): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
> > AGF Buffer: XAGF  Out of space
> > ----------------------------------------------------------------------------
> > Oper (6): tid: 4d4486ed  len: 24  clientid: TRANS  flags: none
> > BUF:  #regs: 2   start blkno: 8 (0x8)  len: 8  bmap size: 1  flags: 0x2000
> > Oper (7): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
> > BUF DATA
> > ----------------------------------------------------------------------------
> > Oper (8): tid: 4d4486ed  len: 24  clientid: TRANS  flags: none
> > BUF:  #regs: 3   start blkno: 16 (0x10)  len: 8  bmap size: 1  flags: 0x2000
> > Oper (9): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
> > BUF DATA
> > Oper (10): tid: 4d4486ed  len: 128  clientid: TRANS  flags: none
> > BUF DATA
> > ----------------------------------------------------------------------------
> > Oper (11): tid: 4d4486ed  len: 56  clientid: TRANS  flags: none
> > INODE: #regs: 3   ino: 0x89c95dd  flags: 0x5   dsize: 16
> >        blkno: 144479680  len: 32  boff: 14848
> > Oper (12): tid: 4d4486ed  len: 176  clientid: TRANS  flags: none
> > INODE CORE
> > magic 0x494e mode 0100600 version 3 format 2
> > nlink 1 uid 501 gid 100
> > atime 0x59790f5b mtime 0x59790f6d ctime 0x59790f6d
> > size 0x3a9c161 nblocks 0x3a9d extsize 0x0 nextents 0x1
> > naextents 0x0 forkoff 0 dmevmask 0x0 dmstate 0x0
> > flags 0x0 gen 0x51eeced9
> > Oper (13): tid: 4d4486ed  len: 16  clientid: TRANS  flags: none
> > EXTENTS inode data
> > ----------------------------------------------------------------------------
> > Oper (14): tid: 4d4486ed  len: 0  clientid: TRANS  flags: COMMIT -----
> > 
> > 
> > > Do note that the simple act of mounting the filesystem runs the risk of
> > > overwriting previous log data. An 'xfs_metadump -go' might be a good way
> > > to preserve current log content.
> > 
> > I've been hibernating the machine and haven't tried remounting it read-only:
> > I'm not expecting anything to do any serious writes on that partition and I
> > don't think there will be anything much at all that tries to write there.
> > 
> > 
> > > Brian
> > 
> > Thanks for your help!
> > 
> > 
> > 
> > 
> > Regards,
> > @ndy
> > 
> > -- 
> > andyjpb@ashurst.eu.org
> > http://www.ashurst.eu.org/
> > 0x7EBA75FF
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: Recover file after truncate
  2017-07-27 15:04       ` Brian Foster
@ 2017-07-30 14:36         ` Andy Bennett
  0 siblings, 0 replies; 7+ messages in thread
From: Andy Bennett @ 2017-07-30 14:36 UTC (permalink / raw)
  To: Brian Foster; +Cc: linux-xfs

Hi,

> The last reference I see of your inode in the log before the size goes
> to zero is this:
>
> ---------------------------------------------------------------------------- 
>                                           
> Oper (2): tid: c6720129  len: 56  clientid: TRANS  flags: none  
>                                                        
> INODE: #regs: 2   ino: 0x12c37053  flags: 0x1   dsize: 0   
>         blkno: 235989736  len: 32  boff: 9728              
> Oper (3): tid: c6720129  len: 176  clientid: TRANS  flags: none 
>                                                        
> INODE CORE                                                 
> magic 0x494e mode 0100644 version 3 format 2               
> nlink 1 uid 501 gid 100                                    
> atime 0x5977c96e mtime 0x5973ac16 ctime 0x5973ac16         
> size 0x1a81f2218 nblocks 0x1a81f3 extsize 0x0 nextents 0x2 
> naextents 0x0 forkoff 0 dmevmask 0x0 dmstate 0x0           
> flags 0x0 gen 0xb8791e20                                   
> ----------------------------------------------------------------------------
>
> This suggests the file size was 0x1a81f2218 (7115579928) and it had
> 0x1a81f3 (1737203) blocks across two extents (roughly 6.6GB). Note that
> the file size is not exactly aligned to the (blockcount * 4096).
>
> Further along, the following EFI exists:
>
> ----------------------------------------------------------------------------
> Oper (19): tid: 6a85c7fc  len: 48  clientid: TRANS  flags: none
> EFI:  #regs: 1    num_extents: 2  id: 0xffff8bfcb4f38990
> (s: 0x12a9233, l: 688643) (s: 0x34ae0d5, l: 1048560)
> ----------------------------------------------------------------------------
>
> ... which frees two extents that add up to 688643 + 1048560 = 1737203
> (0x1a81f3) blocks. The starting filesystem blocks for these extents are
> 0x12a9233 and 0x34ae0d5, respectively.
>
> If you can convert those starting fsb values to agno/agbno combinations
> as shown in my previous mail, you could confirm whether they reside in
> the free space btrees for the associated allocation group of each
> extent.

Here are the translations for those extents:

fsb 0x12a9233
	agno 0
	agbno 0x12a9233 (19567155)
	daddr 0x9549198 (156537240)

fsb 0x34ae0d5
	agno 1
	agno 0x14ae0d5 (21684437)
	daddr 0x15a48150 (363102544)


xfs_db's 'freesp -d' output contains:

-----
...
       0 19566515      639
       0 19567155  1396803
       0 20963959   408468
...
       1 20621811   106501
       1 21684437  1104130
       1 22788568   421464
...
-----

Both chunks of free space appear to have more blocks that the original 
extents.


> If so, you could convert each start fsb to a raw daddr (xfs_db
> -c "convert fsb 0x12a9233 daddr" <dev>) and use that to try and read
> each extent off the raw device as previously discussed.

156537240 is in 512 byte blocks which is 19567155 in 4096 byte blocks.

dd if=/dev/nvme0n1p8 bs=4096 skip=19567155 count=688643 of=extentfile.0


363102544 is in 512 byte blocks which is 45387818 in 4096 byte blocks.

dd if=/dev/nvme0n1p8 bs=4096 skip=45387818 count=1048560 of=extentfile.1

These files both look good and 'cat extentfile.0 >> extentfile.1' gives me 
back the original order of the file.

-----
$ wc -c extentfile.1
7115583488 extentfile.1

$ du -hs extentfile.1
6.7G    extentfile.1
-----

...which seems slightly larger than the original file as reported in the 
'INODE CORE' message that you found in the log.

The recovered file ends up being some 53 extents long; probably because the 
filesystem I recovered it too was rather full.

However, when I run it through my tool chain, I seem to get exactly what I 
expect.


> I have no idea whether those extents contain the file data you're after
> (note that I'm not sure we've actually confirmed your file data would
> have made it to disk), but if the file size values look sane that might
> be worth a try before you go fishing the device for raw data.

It seems that it did and we have been able to retrieve it!


Thanks so much for your help. I really appreiciate it, especially the time 
you took to poor through my logs.

Let me know if you're ever near London and I'll shout you dinner and 
drinks.





Regards,
@ndy

-- 
andyjpb@ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF


^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: Recover file after truncate
  2017-07-27 12:43     ` Brian Foster
  2017-07-27 15:04       ` Brian Foster
@ 2017-07-30 14:41       ` Andy Bennett
  1 sibling, 0 replies; 7+ messages in thread
From: Andy Bennett @ 2017-07-30 14:41 UTC (permalink / raw)
  To: Brian Foster; +Cc: linux-xfs

Hi,

> The free space btree can be traversed offline via the xfs_db utility.
> Something like the following could give you the free extent information
> for each AG in the fs.
>
> for i in $(seq 0 <maxag>); do
> 	echo agno $i
> 	xfs_db -c "agf $i" -c "addr bnoroot" -c "btdump"  <dev>
> done

My xfs_db didn't have 'btdump' but I discovered 'freesp -d' which 
summarised the free regions for all AGs at once. I also needed to use 
'xfs_db -r ...' to attach it to the filesystem as it was still mounted 
read-write. (I wasn't sure if remounting it or unmounting it and mounting 
it again would destroy the logs.)


Thanks again for your help.



Regards,
@ndy

-- 
andyjpb@ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF

^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-07-30 14:41 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-26  0:48 Recover file after truncate Andy Bennett
2017-07-26 11:15 ` Brian Foster
2017-07-26 22:12   ` Andy Bennett
2017-07-27 12:43     ` Brian Foster
2017-07-27 15:04       ` Brian Foster
2017-07-30 14:36         ` Andy Bennett
2017-07-30 14:41       ` Andy Bennett

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.