* How to get back a deleted sub-volume.
@ 2016-12-11 17:40 Tomasz Kusmierz
2016-12-11 19:00 ` Chris Murphy
0 siblings, 1 reply; 6+ messages in thread
From: Tomasz Kusmierz @ 2016-12-11 17:40 UTC (permalink / raw)
To: linux-btrfs
Hi,
So, I've found my self in a pickle after following this steps:
1. trying to migrate an array to different system, it became apparent
that importing array there was not possible to import it because I've
had a very large amount of snapshots (every 15 minutes during office
hours amounting to few K) so I've had to remove snapshots for main
data storage.
2. while playing with live array, it become apparent that some bright
spark implemented a "delete all sub-volumes while removing array from
system" ... needles to say that this behaviour is unexpected to say al
least ... and I wanted to punch somebody in face.
3. the backup off-site server that was making backups every 30 minutes
was located in CEO house and his wife decide that it's not necessary
to have it connected
(laughs can start roughly here)
So I've got array with all the data there (theoretically COW, right ?)
with additional of plethora of snapshots (important data was snapped
every 15 minutes during a office hours to capture all the changes,
other sub-volumes were snapshoted daily)
This occurred roughly on 4-12-2016.
Since then I was trying to rescue as much data as I can, luckily I
managed to get a lot of data from snapshots for "other than share"
volumes (because those were not deleted :/) but the most important
volume "share" prove difficult. This subvolume comes out with a lot of
errors on readout with "btrfs restore /dev/sda /mnt2/temp2/ -x -m -S
-s -i -t".
Also for some reason I can't use a lot of root blocks that I find with
btrfs-find-root ..
To put some detail here:
btrfs-find-root -a /dev/sda
Superblock thinks the generation is 184540
Superblock thinks the level is 1
Well block 919363862528(gen: 184540 level: 1) seems good, and it
matches superblock
Well block 919356325888(gen: 184539 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919343529984(gen: 184538 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920041308160(gen: 184537 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919941955584(gen: 184536 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919670538240(gen: 184535 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920045371392(gen: 184532 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920070209536(gen: 184531 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920117510144(gen: 184530 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1 <<<---- here
stuff is gone
Well block 920139055104(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920139022336(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920138989568(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920138973184(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920137596928(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920137531392(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920137515008(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135991296(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135958528(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135925760(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135827456(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920135811072(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920133697536(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920133664768(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920133337088(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920133206016(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920132976640(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920132878336(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920132845568(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920128045056(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920127946752(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920127897600(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920127881216(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920127864832(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920127848448(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 920127799296(gen: 184511 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919815143424(gen: 184508 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919814569984(gen: 184508 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919811866624(gen: 184508 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919803232256(gen: 184496 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919712989184(gen: 184455 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919711891456(gen: 184455 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919706271744(gen: 184455 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919703568384(gen: 184455 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919699865600(gen: 184455 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 919647092736(gen: 184455 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417026879488(gen: 184334 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1 <<--- I can
only use that with btrfs restore and it's month old
Well block 1417025896448(gen: 184333 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417029484544(gen: 184332 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417029320704(gen: 184332 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417029173248(gen: 184332 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417027616768(gen: 184332 level: 0) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417024831488(gen: 184331 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417021079552(gen: 184330 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417014001664(gen: 184329 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417011970048(gen: 184328 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
Well block 1417009758208(gen: 184327 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level:
... rest of lines omitted to not trash the list to much.
this line:
Well block 920117510144(gen: 184530 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
is where stuff is missing,
and this line:
Well block 1417026879488(gen: 184334 level: 1) seems good, but
generation/level doesn't match, want gen: 184540 level: 1
is the only line that physically work with
btrfs restore /dev/sda /mnt2/temp2/ -x -m -S -s -i -t
as you can see there is a vast "generation hole" between 184530 and
184334 ... more 200 generations ... and this results in only being
able to access data that is month old :( and some of it is strangelly
corrupted :(
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to get back a deleted sub-volume.
2016-12-11 17:40 How to get back a deleted sub-volume Tomasz Kusmierz
@ 2016-12-11 19:00 ` Chris Murphy
2016-12-12 0:56 ` Tomasz Kusmierz
0 siblings, 1 reply; 6+ messages in thread
From: Chris Murphy @ 2016-12-11 19:00 UTC (permalink / raw)
To: Tomasz Kusmierz; +Cc: linux-btrfs
On Sun, Dec 11, 2016 at 10:40 AM, Tomasz Kusmierz
<tom.kusmierz@gmail.com> wrote:
> Hi,
>
> So, I've found my self in a pickle after following this steps:
> 1. trying to migrate an array to different system, it became apparent
> that importing array there was not possible to import it because I've
> had a very large amount of snapshots (every 15 minutes during office
> hours amounting to few K) so I've had to remove snapshots for main
> data storage.
True, there is no recursive incremental send.
> 2. while playing with live array, it become apparent that some bright
> spark implemented a "delete all sub-volumes while removing array from
> system" ... needles to say that this behaviour is unexpected to say al
> least ... and I wanted to punch somebody in face.
The technical part of this is vague. I'm guessing you used 'btrfs
device remove' butt it works no differently than lvremove - when a
device is removed from an array, it wipes the signature from that
device. You probably can restore that signature and use that device
again, depending on what the profile is for metadata and data, it may
be usable stand alone.
Proposing assault is probably not the best way to ask for advice
though. Just a guess.
>
> Since then I was trying to rescue as much data as I can, luckily I
> managed to get a lot of data from snapshots for "other than share"
> volumes (because those were not deleted :/) but the most important
> volume "share" prove difficult. This subvolume comes out with a lot of
> errors on readout with "btrfs restore /dev/sda /mnt2/temp2/ -x -m -S
> -s -i -t".
>
> Also for some reason I can't use a lot of root blocks that I find with
> btrfs-find-root ..
>
> To put some detail here:
> btrfs-find-root -a /dev/sda
> Superblock thinks the generation is 184540
> Superblock thinks the level is 1
> Well block 919363862528(gen: 184540 level: 1) seems good, and it
> matches superblock
> Well block 919356325888(gen: 184539 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919343529984(gen: 184538 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920041308160(gen: 184537 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919941955584(gen: 184536 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919670538240(gen: 184535 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920045371392(gen: 184532 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920070209536(gen: 184531 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920117510144(gen: 184530 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1 <<<---- here
> stuff is gone
> Well block 920139055104(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920139022336(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920138989568(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920138973184(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920137596928(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920137531392(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920137515008(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135991296(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135958528(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135925760(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135827456(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920135811072(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920133697536(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920133664768(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920133337088(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920133206016(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920132976640(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920132878336(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920132845568(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920128045056(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920127946752(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920127897600(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920127881216(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920127864832(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920127848448(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 920127799296(gen: 184511 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919815143424(gen: 184508 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919814569984(gen: 184508 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919811866624(gen: 184508 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919803232256(gen: 184496 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919712989184(gen: 184455 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919711891456(gen: 184455 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919706271744(gen: 184455 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919703568384(gen: 184455 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919699865600(gen: 184455 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 919647092736(gen: 184455 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417026879488(gen: 184334 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1 <<--- I can
> only use that with btrfs restore and it's month old
> Well block 1417025896448(gen: 184333 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417029484544(gen: 184332 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417029320704(gen: 184332 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417029173248(gen: 184332 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417027616768(gen: 184332 level: 0) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417024831488(gen: 184331 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417021079552(gen: 184330 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417014001664(gen: 184329 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417011970048(gen: 184328 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> Well block 1417009758208(gen: 184327 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level:
> ... rest of lines omitted to not trash the list to much.
>
> this line:
> Well block 920117510144(gen: 184530 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> is where stuff is missing,
A root tree is only kept around for about 2 minutes, so it's going to
be freed up and subject to being overwritten if there's no snapshot
to pin it in place.
>
> and this line:
> Well block 1417026879488(gen: 184334 level: 1) seems good, but
> generation/level doesn't match, want gen: 184540 level: 1
> is the only line that physically work with
> btrfs restore /dev/sda /mnt2/temp2/ -x -m -S -s -i -t
That particular
>
>
> as you can see there is a vast "generation hole" between 184530 and
> 184334 ... more 200 generations ... and this results in only being
> able to access data that is month old :( and some of it is strangelly
> corrupted :(
You're leaving out a lot of information. If you have thousands of
snapshots, and one of those snapshots has the data in it you want,
then you shouldn't need to use btrfs restore. Btrfs restore is a
scraping tool to try to find data, with no assurance of success at
all, from freed up metadata and data blocks. That data certainly could
be corrupt because all or parts of it may have been overwritten
already.
If the data is in a stable snapshot that has not been deleted, then
you should be able to find it and extract it with a normal mount. If
the file system can't be mounted normally then that's what you need to
look at first - so presumably it's not mounting normally; but does it
mount with usebackuproot? Or ro,usebackuproot? Or
ro,usebackuproot,degraded?
--
Chris Murphy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to get back a deleted sub-volume.
2016-12-11 19:00 ` Chris Murphy
@ 2016-12-12 0:56 ` Tomasz Kusmierz
2016-12-12 1:41 ` Chris Murphy
2016-12-12 4:14 ` Chris Murphy
0 siblings, 2 replies; 6+ messages in thread
From: Tomasz Kusmierz @ 2016-12-12 0:56 UTC (permalink / raw)
To: Chris Murphy; +Cc: linux-btrfs
Chris, for all the time you helped so far I have to really appologize
I've led you a stray ... so, reson the subvolumes were deleted is
nothing to do with btrfs it self, I'm using "Rockstor" to ease
managment tasks. This tool / environment / distribution treats a
singular btrfs FS as a "pool" ( something in line of zfs :/ ) and when
one removes a pool from the system it will actually go and delete
subvolumes from FS before unmounting it and removing reference of it
from it's DB (yes a bit shiet I know). so I'm not blaming anybody here
for disapearing subvolumes, it's just me commig back to belive in man
kind to get kicked in the gonads by mankind stupidity.
ALSO by importing the fs to their "solution" is just actually mounting
and walking the tree of subvolumes to to create all the references in
local DB (for rockstor of course, still nothing to do with btrfs
functionality). To be able to ïmport" I've had to remove before
mentioned snpshots becouse imports script was timing out.
So for a single subvolume (called physically "share") I was left with
no snapshots (removed by me to make import not time out) and then this
subvolume was removed when I was trying to remove a fs (pool) from a
running system.
I've polled both disks (2 disk fs raid1) and I'm trying to rescue as
much data as I can.
The question is, why suddenly when I removed the snapshots and
(someone else removed) the subvolume, there is such a great gap in
generations of FS (over 200 generations missing) and the most recent
generation that actually can me touched by btrfs restore is over a
month old.
How to over come that ?
On 11 December 2016 at 19:00, Chris Murphy <lists@colorremedies.com> wrote:
> On Sun, Dec 11, 2016 at 10:40 AM, Tomasz Kusmierz
> <tom.kusmierz@gmail.com> wrote:
>> Hi,
>>
>> So, I've found my self in a pickle after following this steps:
>> 1. trying to migrate an array to different system, it became apparent
>> that importing array there was not possible to import it because I've
>> had a very large amount of snapshots (every 15 minutes during office
>> hours amounting to few K) so I've had to remove snapshots for main
>> data storage.
>
> True, there is no recursive incremental send.
>
>> 2. while playing with live array, it become apparent that some bright
>> spark implemented a "delete all sub-volumes while removing array from
>> system" ... needles to say that this behaviour is unexpected to say al
>> least ... and I wanted to punch somebody in face.
>
> The technical part of this is vague. I'm guessing you used 'btrfs
> device remove' butt it works no differently than lvremove - when a
> device is removed from an array, it wipes the signature from that
> device. You probably can restore that signature and use that device
> again, depending on what the profile is for metadata and data, it may
> be usable stand alone.
>
> Proposing assault is probably not the best way to ask for advice
> though. Just a guess.
>
>
>
>
>>
>> Since then I was trying to rescue as much data as I can, luckily I
>> managed to get a lot of data from snapshots for "other than share"
>> volumes (because those were not deleted :/) but the most important
>> volume "share" prove difficult. This subvolume comes out with a lot of
>> errors on readout with "btrfs restore /dev/sda /mnt2/temp2/ -x -m -S
>> -s -i -t".
>>
>> Also for some reason I can't use a lot of root blocks that I find with
>> btrfs-find-root ..
>>
>> To put some detail here:
>> btrfs-find-root -a /dev/sda
>> Superblock thinks the generation is 184540
>> Superblock thinks the level is 1
>> Well block 919363862528(gen: 184540 level: 1) seems good, and it
>> matches superblock
>> Well block 919356325888(gen: 184539 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919343529984(gen: 184538 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920041308160(gen: 184537 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919941955584(gen: 184536 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919670538240(gen: 184535 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920045371392(gen: 184532 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920070209536(gen: 184531 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920117510144(gen: 184530 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1 <<<---- here
>> stuff is gone
>> Well block 920139055104(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920139022336(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920138989568(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920138973184(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920137596928(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920137531392(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920137515008(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920135991296(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920135958528(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920135925760(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920135827456(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920135811072(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920133697536(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920133664768(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920133337088(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920133206016(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920132976640(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920132878336(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920132845568(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920128045056(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920127946752(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920127897600(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920127881216(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920127864832(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920127848448(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 920127799296(gen: 184511 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919815143424(gen: 184508 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919814569984(gen: 184508 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919811866624(gen: 184508 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919803232256(gen: 184496 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919712989184(gen: 184455 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919711891456(gen: 184455 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919706271744(gen: 184455 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919703568384(gen: 184455 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919699865600(gen: 184455 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 919647092736(gen: 184455 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417026879488(gen: 184334 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1 <<--- I can
>> only use that with btrfs restore and it's month old
>> Well block 1417025896448(gen: 184333 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417029484544(gen: 184332 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417029320704(gen: 184332 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417029173248(gen: 184332 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417027616768(gen: 184332 level: 0) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417024831488(gen: 184331 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417021079552(gen: 184330 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417014001664(gen: 184329 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417011970048(gen: 184328 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> Well block 1417009758208(gen: 184327 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level:
>> ... rest of lines omitted to not trash the list to much.
>>
>> this line:
>> Well block 920117510144(gen: 184530 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> is where stuff is missing,
>
> A root tree is only kept around for about 2 minutes, so it's going to
> be freed up and subject to being overwritten if there's no snapshot
> to pin it in place.
>
>>
>> and this line:
>> Well block 1417026879488(gen: 184334 level: 1) seems good, but
>> generation/level doesn't match, want gen: 184540 level: 1
>> is the only line that physically work with
>> btrfs restore /dev/sda /mnt2/temp2/ -x -m -S -s -i -t
>
> That particular
>
>>
>>
>> as you can see there is a vast "generation hole" between 184530 and
>> 184334 ... more 200 generations ... and this results in only being
>> able to access data that is month old :( and some of it is strangelly
>> corrupted :(
>
> You're leaving out a lot of information. If you have thousands of
> snapshots, and one of those snapshots has the data in it you want,
> then you shouldn't need to use btrfs restore. Btrfs restore is a
> scraping tool to try to find data, with no assurance of success at
> all, from freed up metadata and data blocks. That data certainly could
> be corrupt because all or parts of it may have been overwritten
> already.
>
> If the data is in a stable snapshot that has not been deleted, then
> you should be able to find it and extract it with a normal mount. If
> the file system can't be mounted normally then that's what you need to
> look at first - so presumably it's not mounting normally; but does it
> mount with usebackuproot? Or ro,usebackuproot? Or
> ro,usebackuproot,degraded?
>
>
> --
> Chris Murphy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to get back a deleted sub-volume.
2016-12-12 0:56 ` Tomasz Kusmierz
@ 2016-12-12 1:41 ` Chris Murphy
2016-12-12 4:14 ` Chris Murphy
1 sibling, 0 replies; 6+ messages in thread
From: Chris Murphy @ 2016-12-12 1:41 UTC (permalink / raw)
To: Tomasz Kusmierz; +Cc: Chris Murphy, linux-btrfs
On Sun, Dec 11, 2016 at 5:56 PM, Tomasz Kusmierz <tom.kusmierz@gmail.com> wrote:
> Chris, for all the time you helped so far I have to really appologize
> I've led you a stray ... so, reson the subvolumes were deleted is
> nothing to do with btrfs it self, I'm using "Rockstor" to ease
> managment tasks. This tool / environment / distribution treats a
> singular btrfs FS as a "pool" ( something in line of zfs :/ ) and when
> one removes a pool from the system it will actually go and delete
> subvolumes from FS before unmounting it and removing reference of it
> from it's DB (yes a bit shiet I know). so I'm not blaming anybody here
> for disapearing subvolumes, it's just me commig back to belive in man
> kind to get kicked in the gonads by mankind stupidity.
>
> ALSO by importing the fs to their "solution" is just actually mounting
> and walking the tree of subvolumes to to create all the references in
> local DB (for rockstor of course, still nothing to do with btrfs
> functionality). To be able to ïmport" I've had to remove before
> mentioned snpshots becouse imports script was timing out.
>
> So for a single subvolume (called physically "share") I was left with
> no snapshots (removed by me to make import not time out) and then this
> subvolume was removed when I was trying to remove a fs (pool) from a
> running system.
>
> I've polled both disks (2 disk fs raid1) and I'm trying to rescue as
> much data as I can.
>
> The question is, why suddenly when I removed the snapshots and
> (someone else removed) the subvolume, there is such a great gap in
> generations of FS (over 200 generations missing) and the most recent
> generation that actually can me touched by btrfs restore is over a
> month old.
>
> How to over come that ?
Well it depends on how long it was from the time of the snapshots
being deleted to the time the file system was unmounted. If it wasn't
that long it might be possible to find a root from 'btrfs insp-in
dump-s -fa <dev>' (or btrfs-show-super with older progs) to see if you
can use any of the backup roots. Once so much time goes by, the no
longer used metadata for generations has their roots deleted and all
of the blocks used for both metadata and data are subject to being
overwritten. So my expectation is that there's too much time between
delete and umount, so there's nothing in the current file system that
points to the old generations.
It might be the metadata and data still exist. It's not entirely a
shot in the dark to find it.
First, try to find the oldest chunk tree you can with btrfs-show-super
-fa (btrfs insp dump-s -fa) and look in the backup roots list for the
chunk tree:
backup_roots[4]:
backup 0:
backup_tree_root: 21037056 gen: 7 level: 0
backup_chunk_root: 147456 gen: 6 level: 0
backup_extent_root: 21053440 gen: 7 level: 0
backup_fs_root: 4194304 gen: 4 level: 0
backup_dev_root: 20987904 gen: 6 level: 0
backup_csum_root: 21069824 gen: 7 level: 0
backup_total_bytes: 268435456000
backup_bytes_used: 393216
backup_num_devices: 1
backup 1:
backup_tree_root: 21086208 gen: 8 level: 0
backup_chunk_root: 147456 gen: 6 level: 0
backup_extent_root: 21102592 gen: 8 level: 0
backup_fs_root: 4194304 gen: 4 level: 0
backup_dev_root: 20987904 gen: 6 level: 0
backup_csum_root: 21118976 gen: 8 level: 0
backup_total_bytes: 268435456000
backup_bytes_used: 393216
backup_num_devices: 1
backup 2:
backup_tree_root: 21069824 gen: 9 level: 0
backup_chunk_root: 147456 gen: 6 level: 0
backup_extent_root: 21053440 gen: 9 level: 0
backup_fs_root: 21004288 gen: 9 level: 0
backup_dev_root: 21135360 gen: 9 level: 0
backup_csum_root: 21037056 gen: 9 level: 0
backup_total_bytes: 268435456000
backup_bytes_used: 487424
backup_num_devices: 1
backup 3:
backup_tree_root: 21151744 gen: 10 level: 0
backup_chunk_root: 147456 gen: 6 level: 0
backup_extent_root: 21168128 gen: 10 level: 0
backup_fs_root: 21004288 gen: 9 level: 0
backup_dev_root: 21135360 gen: 9 level: 0
backup_csum_root: 21184512 gen: 10 level: 0
backup_total_bytes: 268435456000
backup_bytes_used: 487424
backup_num_devices: 1
IN this case, all of the chunk roots are the same generation - so it's
no help really.
Second, list the chunk tree using either -t 3 for the current one, or
you can plug in the bytenr for an older backup chunk root if
available.
[root@f25h ~]# btrfs-debug-tree -t 3 /dev/mapper/vg-test2
btrfs-progs v4.8.5
chunk tree
leaf 147456 items 5 free space 15740 generation 6 owner 3
fs uuid 6cfaa40b-92f0-44e8-a06c-7e39d38e6446
chunk uuid a971b4ef-64fe-496d-adff-9f96734a2949
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
devid 1 total_bytes 268435456000 bytes_used 1094713344
io_align 4096 io_width 4096 sector_size 4096 type 0
generation 0 start_offset 0 dev_group 0
seek_speed 0 bandwidth 0
uuid 75573b53-f192-404f-b5f4-21f61feed32d
fsid 6cfaa40b-92f0-44e8-a06c-7e39d38e6446
item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 0) itemoff 16105 itemsize 80
length 4194304 owner 2 stripe_len 65536 type SYSTEM
io_align 4096 io_width 4096 sector_size 4096
num_stripes 1 sub_stripes 0
stripe 0 devid 1 offset 0
dev_uuid 75573b53-f192-404f-b5f4-21f61feed32d
item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 4194304) itemoff 16025 itemsize 80
length 8388608 owner 2 stripe_len 65536 type METADATA
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 0
stripe 0 devid 1 offset 4194304
dev_uuid 75573b53-f192-404f-b5f4-21f61feed32d
item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 12582912) itemoff 15945 itemsize 80
length 8388608 owner 2 stripe_len 65536 type DATA
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 0
stripe 0 devid 1 offset 12582912
dev_uuid 75573b53-f192-404f-b5f4-21f61feed32d
item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) itemoff 15865 itemsize 80
length 1073741824 owner 2 stripe_len 65536 type METADATA
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 1 offset 20971520
dev_uuid 75573b53-f192-404f-b5f4-21f61feed32d
total bytes 268435456000
bytes used 487424
uuid 6cfaa40b-92f0-44e8-a06c-7e39d38e6446
So in this example, it's item 2 and item 4 (type METADATA). What you
care about is the offset and the length for each of them. Any
reference for your data is most likely in these areas (unless so many
snapshots were deleted that it also resulted in a metadata chunk being
deleted in which case it's going to take a lot longer to find it).
You'll need to search those blocks (these are bytes not sectors so you
have to convert them) and you'll have to search for leaves that
contains the generation you're after. This is really tedious to do
manually, you'll want to automate it by writing some code but honestly
I can't help with that, maybe someone else on the list has an idea.
But basically what you're looking for is something that does what
btrfs-debug-tree does, but will just read the entire metadata chunks
you point it to, rather than it following only the valid active tree.
You want every leaf, even old deleted ones, to be turned into some
kind of human readable output like what btrfs-debug-tree gives.
Ideally you'd find an old tree root that has the generation you want.
This is a tree root
[root@f25h ~]# btrfs-debug-tree -b 21151744 /dev/mapper/vg-test2
btrfs-progs v4.8.5
leaf 21151744 items 13 free space 12844 generation 10 owner 1
fs uuid 6cfaa40b-92f0-44e8-a06c-7e39d38e6446
chunk uuid a971b4ef-64fe-496d-adff-9f96734a2949
item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
generation 10 root_dirid 0 bytenr 21168128 level 0 refs 1
lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
uuid 00000000-0000-0000-0000-000000000000
drop key (0 UNKNOWN.0 0) level 0
item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439
generation 9 root_dirid 0 bytenr 21135360 level 0 refs 1
lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
uuid 00000000-0000-0000-0000-000000000000
drop key (0 UNKNOWN.0 0) level 0
item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17
inode ref index 0 namelen 7 name: default
item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439
generation 9 root_dirid 256 bytenr 21004288 level 0 refs 1
lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
uuid 00000000-0000-0000-0000-000000000000
ctransid 9 otransid 0 stransid 0 rtransid 0
drop key (0 UNKNOWN.0 0) level 0
item 4 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14789 itemsize 160
inode generation 3 transid 0 size 0 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1481503752.0 (2016-12-11 17:49:12)
ctime 1481503752.0 (2016-12-11 17:49:12)
mtime 1481503752.0 (2016-12-11 17:49:12)
otime 1481503752.0 (2016-12-11 17:49:12)
item 5 key (ROOT_TREE_DIR INODE_REF 6) itemoff 14777 itemsize 12
inode ref index 0 namelen 2 name: ..
item 6 key (ROOT_TREE_DIR DIR_ITEM 2378154706) itemoff 14740 itemsize 37
location key (FS_TREE ROOT_ITEM -1) type DIR
transid 0 data_len 0 name_len 7
name: default
item 7 key (CSUM_TREE ROOT_ITEM 0) itemoff 14301 itemsize 439
generation 10 root_dirid 0 bytenr 21184512 level 0 refs 1
lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
uuid 00000000-0000-0000-0000-000000000000
drop key (0 UNKNOWN.0 0) level 0
item 8 key (UUID_TREE ROOT_ITEM 0) itemoff 13862 itemsize 439
generation 7 root_dirid 0 bytenr 21020672 level 0 refs 1
lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
uuid 86c465ab-e661-d34e-a3e0-bc7534db8b92
drop key (0 UNKNOWN.0 0) level 0
item 9 key (256 INODE_ITEM 0) itemoff 13702 itemsize 160
inode generation 10 transid 10 size 262144 nbytes 1310720
block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
sequence 27 flags 0x5(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
atime 0.0 (1969-12-31 17:00:00)
ctime 1481504170.772155917 (2016-12-11 17:56:10)
mtime 0.0 (1969-12-31 17:00:00)
otime 0.0 (1969-12-31 17:00:00)
item 10 key (256 EXTENT_DATA 0) itemoff 13649 itemsize 53
generation 10 type 1 (regular)
extent data disk byte 13369344 nr 262144
extent data offset 0 nr 262144 ram 262144
extent compression 0 (none)
item 11 key (FREE_SPACE UNTYPED 20971520) itemoff 13608 itemsize 41
location key (256 INODE_ITEM 0)
cache generation 10 entries 5 bitmaps 0
item 12 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 13169 itemsize 439
generation 4 root_dirid 256 bytenr 4358144 level 0 refs 1
lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none)
uuid 00000000-0000-0000-0000-000000000000
drop key (0 UNKNOWN.0 0) level 0
IF the whole thing is intact like this one, then you can use btrfs
restore -t to point to this tree root and it'll use it even though
it's deleted.
Anyway, it's a lot of work. But it's no more work than what Btrfs
developers are doing every day.
--
Chris Murphy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to get back a deleted sub-volume.
2016-12-12 0:56 ` Tomasz Kusmierz
2016-12-12 1:41 ` Chris Murphy
@ 2016-12-12 4:14 ` Chris Murphy
2016-12-12 19:47 ` Tomasz Kusmierz
1 sibling, 1 reply; 6+ messages in thread
From: Chris Murphy @ 2016-12-12 4:14 UTC (permalink / raw)
To: Tomasz Kusmierz; +Cc: Chris Murphy, linux-btrfs
Tomasz - try using 'btrfs-find-root -a <dev>' I totally forgot about
this option. It goes through the extent tree and might have a chance
of finding additional generations that aren't otherwise being found.
You can then plug those tree roots into 'btrfs restore -t <bytenr>'
and do it with the -D and -v options so it's a verbose dry run, and
see if the file listing it spits out is at all useful - if it has any
of the data you're looking for.
Chris Murphy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to get back a deleted sub-volume.
2016-12-12 4:14 ` Chris Murphy
@ 2016-12-12 19:47 ` Tomasz Kusmierz
0 siblings, 0 replies; 6+ messages in thread
From: Tomasz Kusmierz @ 2016-12-12 19:47 UTC (permalink / raw)
To: Chris Murphy; +Cc: linux-btrfs
Chris,
the "btrfs-show-super -fa" gives me nothing useful to work with.
the "btrfs-find-root -a <dev>" is actually something that I was
already using (see original post), but the list of roots given had a
rather LARGE hole of 200 generations that is located between right
after I've had everything removed and 1 month before the whole
situation.
On 12 December 2016 at 04:14, Chris Murphy <lists@colorremedies.com> wrote:
> Tomasz - try using 'btrfs-find-root -a <dev>' I totally forgot about
> this option. It goes through the extent tree and might have a chance
> of finding additional generations that aren't otherwise being found.
> You can then plug those tree roots into 'btrfs restore -t <bytenr>'
> and do it with the -D and -v options so it's a verbose dry run, and
> see if the file listing it spits out is at all useful - if it has any
> of the data you're looking for.
>
>
> Chris Murphy
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-12-12 19:47 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-12-11 17:40 How to get back a deleted sub-volume Tomasz Kusmierz
2016-12-11 19:00 ` Chris Murphy
2016-12-12 0:56 ` Tomasz Kusmierz
2016-12-12 1:41 ` Chris Murphy
2016-12-12 4:14 ` Chris Murphy
2016-12-12 19:47 ` Tomasz Kusmierz
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.