destroyed disk in btrfs raid

destroyed disk in btrfs raid

[laie]

Hello!

I've some trouble with my btrf filesystem. I've lost one backing raid 
device, it's luks header is overwritten and not restoreable.

The lost disk was recently added. 'btrfs filesystem balace' was running 
for some time, but the new device is the smallest in the set.

Data is stored with Raid0, Metadata with Raid1. Degraded mounting works 
fine.

Now I'm looking for a way to tell btrfs to provide me with a list of the 
corrupted files and delete them afterwards. This would be great, because 
otherwise it would take very long to get the data back from slow 
backups.

Thanks in advance
Max


btrfs --version
Btrfs v3.12

btrfs fi show
Label: userspace  uuid: there is one ;)
         Total devices 3 FS bytes used 27.21TiB
         devid    1 size 21.83TiB used 13.03TiB path /dev/dm-3
         devid    2 size 16.37TiB used 13.01TiB path /dev/dm-2
         devid    3 size 8.19TiB used 4.27TiB path

btrfs fi df /home/
Data, RAID0: total=30.24TiB, used=27.18TiB
System, RAID1: total=32.00MiB, used=1.99MiB
System, single: total=4.00MiB, used=0.00
Metadata, RAID1: total=32.00GiB, used=31.42GiB

Re: destroyed disk in btrfs raid

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1667 bytes --]

On Fri, May 09, 2014 at 08:02:45PM +0200, laie wrote:
> Hello!
> 
> I've some trouble with my btrf filesystem. I've lost one backing raid
> device, it's luks header is overwritten and not restoreable.
> 
> The lost disk was recently added. 'btrfs filesystem balace' was running for
> some time, but the new device is the smallest in the set.
> 
> Data is stored with Raid0, Metadata with Raid1. Degraded mounting works
> fine.
> 
> Now I'm looking for a way to tell btrfs to provide me with a list of the
> corrupted files and delete them afterwards. This would be great, because
> otherwise it would take very long to get the data back from slow backups.

   Simple solution: cat every file to /dev/null, and see which ones
fail with an I/O error. With RAID-0 data, losing a device is going to
damage most files, though, so don't necessarily expect much to survive.

   Hugo.

> Thanks in advance
> Max
> 
> 
> btrfs --version
> Btrfs v3.12
> 
> btrfs fi show
> Label: userspace  uuid: there is one ;)
>         Total devices 3 FS bytes used 27.21TiB
>         devid    1 size 21.83TiB used 13.03TiB path /dev/dm-3
>         devid    2 size 16.37TiB used 13.01TiB path /dev/dm-2
>         devid    3 size 8.19TiB used 4.27TiB path
> 
> btrfs fi df /home/
> Data, RAID0: total=30.24TiB, used=27.18TiB
> System, RAID1: total=32.00MiB, used=1.99MiB
> System, single: total=4.00MiB, used=0.00
> Metadata, RAID1: total=32.00GiB, used=31.42GiB
> 

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
             --- UDP jokes: It's OK if no-one gets them. ---             

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: destroyed disk in btrfs raid

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1332 bytes --]

On Fri, May 09, 2014 at 06:58:27PM +0100, Hugo Mills wrote:
> On Fri, May 09, 2014 at 08:02:45PM +0200, laie wrote:
> > Now I'm looking for a way to tell btrfs to provide me with a list of the
> > corrupted files and delete them afterwards. This would be great, because
> > otherwise it would take very long to get the data back from slow backups.
> 
>    Simple solution: cat every file to /dev/null, and see which ones
> fail with an I/O error. With RAID-0 data, losing a device is going to
> damage most files, though, so don't necessarily expect much to survive.

   Actually, you could cat just the first 256 KiB of each file to
/dev/null -- that should be sufficient, because with RAID-0, the
stripe size is 64 KiB, and 256 KiB is therefore 4 stripes, and so
should cover every device... :)

   Hugo.

> > btrfs fi show
> > Label: userspace  uuid: there is one ;)
> >         Total devices 3 FS bytes used 27.21TiB
> >         devid    1 size 21.83TiB used 13.03TiB path /dev/dm-3
> >         devid    2 size 16.37TiB used 13.01TiB path /dev/dm-2
> >         devid    3 size 8.19TiB used 4.27TiB path

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
             --- UDP jokes: It's OK if no-one gets them. ---             

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: destroyed disk in btrfs raid

[laie]

On 2014-05-09 20:01, Hugo Mills wrote:
> On Fri, May 09, 2014 at 06:58:27PM +0100, Hugo Mills wrote:
>> On Fri, May 09, 2014 at 08:02:45PM +0200, laie wrote:
>> > Now I'm looking for a way to tell btrfs to provide me with a list of the
>> > corrupted files and delete them afterwards. This would be great, because
>> > otherwise it would take very long to get the data back from slow backups.
>> 
>>    Simple solution: cat every file to /dev/null, and see which ones
>> fail with an I/O error. With RAID-0 data, losing a device is going to
>> damage most files, though, so don't necessarily expect much to 
>> survive.
> 
>    Actually, you could cat just the first 256 KiB of each file to
> /dev/null -- that should be sufficient, because with RAID-0, the
> stripe size is 64 KiB, and 256 KiB is therefore 4 stripes, and so
> should cover every device... :)

I was hoping for an internal tool, but I guess the matter is to 
specific. Thank you for the fast and easy solution.

Are you sure that using only 256KiB covers also huge files, even if only 
the last Part is missing?

Re: destroyed disk in btrfs raid

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

On Tue, May 13, 2014 at 10:16:59AM +0200, laie wrote:
> On 2014-05-09 20:01, Hugo Mills wrote:
> >On Fri, May 09, 2014 at 06:58:27PM +0100, Hugo Mills wrote:
> >>On Fri, May 09, 2014 at 08:02:45PM +0200, laie wrote:
> >>> Now I'm looking for a way to tell btrfs to provide me with a list of the
> >>> corrupted files and delete them afterwards. This would be great, because
> >>> otherwise it would take very long to get the data back from slow backups.
> >>
> >>   Simple solution: cat every file to /dev/null, and see which ones
> >>fail with an I/O error. With RAID-0 data, losing a device is going to
> >>damage most files, though, so don't necessarily expect much to survive.
> >
> >   Actually, you could cat just the first 256 KiB of each file to
> >/dev/null -- that should be sufficient, because with RAID-0, the
> >stripe size is 64 KiB, and 256 KiB is therefore 4 stripes, and so
> >should cover every device... :)
> 
> I was hoping for an internal tool, but I guess the matter is to specific.
> Thank you for the fast and easy solution.
> 
> Are you sure that using only 256KiB covers also huge files, even if only the
> last Part is missing?

   Aah, thinking about it harder, since you had a partial balance,
there will be some block groups that are on n-1 devices, and some that
are on n devices. This means that you could have some files with parts
on unbalanced block groups (so, probably safe) and parts on balanced
block groups (probably damaged).

   So go back to my earlier suggestion: cat every file completely.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- The trouble with you, Ibid, is you think you know everything. ---  
                                                                         

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: destroyed disk in btrfs raid

[laie]

On 2014-05-11 16:19, Hugo Mills wrote:
> On Tue, May 13, 2014 at 10:16:59AM +0200, laie wrote:
>> On 2014-05-09 20:01, Hugo Mills wrote:
>> >On Fri, May 09, 2014 at 06:58:27PM +0100, Hugo Mills wrote:
>> >>On Fri, May 09, 2014 at 08:02:45PM +0200, laie wrote:
>> >>> Now I'm looking for a way to tell btrfs to provide me with a list of the
>> >>> corrupted files and delete them afterwards. This would be great, because
>> >>> otherwise it would take very long to get the data back from slow backups.
>> >>
>> >>   Simple solution: cat every file to /dev/null, and see which ones
>> >>fail with an I/O error. With RAID-0 data, losing a device is going to
>> >>damage most files, though, so don't necessarily expect much to survive.

I finished building the List, about 40% of the Data is gone. So far so 
good.

As next step I planned to delete these files. This is not possible 
because I'm not able to mount the fs r/w.

btrfs: allowing degraded mounts
btrfs: bdev /dev/mapper/luks-0 errs: wr 37519, rd 32783, flush 0, 
corrupt 0, gen 0
Btrfs: too many missing devices, writeable mount is not allowed
btrfs: open_ctree failed

Is it correct remove the missing device now:

btrfs device delete missing /mnt

Or do I have to add the replacement first?

Re: destroyed disk in btrfs raid

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1793 bytes --]

On Wed, May 14, 2014 at 08:43:41PM +0200, laie wrote:
> On 2014-05-11 16:19, Hugo Mills wrote:
> >On Tue, May 13, 2014 at 10:16:59AM +0200, laie wrote:
> >>On 2014-05-09 20:01, Hugo Mills wrote:
> >>>On Fri, May 09, 2014 at 06:58:27PM +0100, Hugo Mills wrote:
> >>>>On Fri, May 09, 2014 at 08:02:45PM +0200, laie wrote:
> >>>>> Now I'm looking for a way to tell btrfs to provide me with a list of the
> >>>>> corrupted files and delete them afterwards. This would be great, because
> >>>>> otherwise it would take very long to get the data back from slow backups.
> >>>>
> >>>>   Simple solution: cat every file to /dev/null, and see which ones
> >>>>fail with an I/O error. With RAID-0 data, losing a device is going to
> >>>>damage most files, though, so don't necessarily expect much to survive.
> 
> I finished building the List, about 40% of the Data is gone. So far so good.
> 
> As next step I planned to delete these files. This is not possible because
> I'm not able to mount the fs r/w.
> 
> btrfs: allowing degraded mounts
> btrfs: bdev /dev/mapper/luks-0 errs: wr 37519, rd 32783, flush 0, corrupt 0,
> gen 0
> Btrfs: too many missing devices, writeable mount is not allowed
> btrfs: open_ctree failed
> 
> Is it correct remove the missing device now:
> 
> btrfs device delete missing /mnt
> 
> Or do I have to add the replacement first?

   You'd have to mount r/w before you can add a new disk. :)

   You should be able to mount r/w using the -o degraded mount option.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- Attempted murder, now honestly, what is that?  Do they give a ---  
                  Nobel Prize for attempted chemistry?                   

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: destroyed disk in btrfs raid

[laie]

On 2014-05-14 20:44, Hugo Mills wrote:
> On Wed, May 14, 2014 at 08:43:41PM +0200, laie wrote:
>> On 2014-05-11 16:19, Hugo Mills wrote:
>> >On Tue, May 13, 2014 at 10:16:59AM +0200, laie wrote:
>> >>On 2014-05-09 20:01, Hugo Mills wrote:
>> >>>On Fri, May 09, 2014 at 06:58:27PM +0100, Hugo Mills wrote:
>> >>>>On Fri, May 09, 2014 at 08:02:45PM +0200, laie wrote:
>> >>>>> Now I'm looking for a way to tell btrfs to provide me with a list of the
>> >>>>> corrupted files and delete them afterwards. This would be great, because
>> >>>>> otherwise it would take very long to get the data back from slow backups.
>> >>>>
>> >>>>   Simple solution: cat every file to /dev/null, and see which ones
>> >>>>fail with an I/O error. With RAID-0 data, losing a device is going to
>> >>>>damage most files, though, so don't necessarily expect much to survive.
>> 
>> I finished building the List, about 40% of the Data is gone. So far so 
>> good.
>> 
>> As next step I planned to delete these files. This is not possible 
>> because
>> I'm not able to mount the fs r/w.
>> 
>> btrfs: allowing degraded mounts
>> btrfs: bdev /dev/mapper/luks-0 errs: wr 37519, rd 32783, flush 0, 
>> corrupt 0,
>> gen 0
>> Btrfs: too many missing devices, writeable mount is not allowed
>> btrfs: open_ctree failed
>> 
>> Is it correct remove the missing device now:
>> 
>> btrfs device delete missing /mnt
>> 
>> Or do I have to add the replacement first?
> 
>    You'd have to mount r/w before you can add a new disk. :)
> 
>    You should be able to mount r/w using the -o degraded mount option.

The error above is occurs when I try to mount with

mount -o degraded source target

only

mount -o degraded,ro source target

works.

Re: destroyed disk in btrfs raid

[laie]

On 2014-05-14 23:44, laie wrote:
> On 2014-05-14 20:44, Hugo Mills wrote:
>> On Wed, May 14, 2014 at 08:43:41PM +0200, laie wrote:
>>> On 2014-05-11 16:19, Hugo Mills wrote:
>>> >On Tue, May 13, 2014 at 10:16:59AM +0200, laie wrote:
>>> >>On 2014-05-09 20:01, Hugo Mills wrote:
>>> >>>On Fri, May 09, 2014 at 06:58:27PM +0100, Hugo Mills wrote:
>>> >>>>On Fri, May 09, 2014 at 08:02:45PM +0200, laie wrote:
>>> >>>>> Now I'm looking for a way to tell btrfs to provide me with a list of the
>>> >>>>> corrupted files and delete them afterwards. This would be great, because
>>> >>>>> otherwise it would take very long to get the data back from slow backups.
>>> >>>>
>>> >>>>   Simple solution: cat every file to /dev/null, and see which ones
>>> >>>>fail with an I/O error. With RAID-0 data, losing a device is going to
>>> >>>>damage most files, though, so don't necessarily expect much to survive.
>>> 
>>> I finished building the List, about 40% of the Data is gone. So far 
>>> so good.
>>> 
>>> As next step I planned to delete these files. This is not possible 
>>> because
>>> I'm not able to mount the fs r/w.
>>> 
>>> btrfs: allowing degraded mounts
>>> btrfs: bdev /dev/mapper/luks-0 errs: wr 37519, rd 32783, flush 0, 
>>> corrupt 0,
>>> gen 0
>>> Btrfs: too many missing devices, writeable mount is not allowed
>>> btrfs: open_ctree failed
>>> 
>>> Is it correct remove the missing device now:
>>> 
>>> btrfs device delete missing /mnt
>>> 
>>> Or do I have to add the replacement first?
>> 
>>    You'd have to mount r/w before you can add a new disk. :)
>> 
>>    You should be able to mount r/w using the -o degraded mount option.
> 
> The error above is occurs when I try to mount with
> 
> mount -o degraded source target
> 
> only
> 
> mount -o degraded,ro source target
> 
> works.

I'm still stuck here, does anybody have an idea how its possible to get 
this filesystem to r/w mode?

The only solution I can think of is to copy the undamaged Data to a new 
filesystem. I don't like to do that.