delete missing /dev/sdd which is now added as /dev/sdd1

delete missing /dev/sdd which is now added as /dev/sdd1

[Rasmus Abrahamsen]

I have a BTRFS mirrored raid on three disks: 500 GB, 1000 GB, 1500 GB.
The 1500 GB is going bad.

I bought a 4000 GB and wanted to make a 1500 GB and also a 2500 GB
partition. By mistake I made a 1.5 GB and did not realize it until much
later.

I added the 1.5 GB to my raid and tried removing the 1500 GB, I
obviously coudn't. I ended up turning off the machine and unplugging the
1500 GB and tried the btrfs delete missing, but that failed too because
delete missing needs new space available otherwise it will fail.

So I reformatted my 1500 GB and added it to the raid again, it showed
up... but twice... I had one empty /dev/sdd and one non-empty /dev/sdd.

I figured I could fix it by removing and formatting it again, this time
creating a partition using fdisk, it would change the device to
/dev/sdd1.

I added the /dev/sdd1 to my raid and deleted the missing. Now the
/dev/sdd does not show up anymore and I have a /dev/sdd1. But I still
have the **** Some devices missing and the command btrfs delete missing
/mnt does not actually do anything. I tried btrfs delete /dev/sdd /mnt
but it said that /dev/sdd was busy.

How can I tell btrfs that it is NOT the same drive?

Here is a screenshot from the machine:
https://www.dropbox.com/s/f33l4ncsni7wd0s/2014-08-13%2007.31.42.jpg

I figured I could somehow change the /dev/ name, but I am not sure how.

Rasmus

Re: delete missing /dev/sdd which is now added as /dev/sdd1

[Marc MERLIN]

On Wed, Aug 13, 2014 at 07:35:49AM +0200, Rasmus Abrahamsen wrote:
> I added the /dev/sdd1 to my raid and deleted the missing. Now the
> /dev/sdd does not show up anymore and I have a /dev/sdd1. But I still
> have the **** Some devices missing and the command btrfs delete missing
> /mnt does not actually do anything. I tried btrfs delete /dev/sdd /mnt
> but it said that /dev/sdd was busy.

Have you tried
btrfs device delete missing /mnt/mountpoint ?
 
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: delete missing /dev/sdd which is now added as /dev/sdd1

[Rasmus Abrahamsen]

/mnt is the mount point. I now did the following

dd if=/dev/zero of=/dev/sdd

Finally, btrfs does not recognize the partition as part of my raid
anymore, but what's worse is that it now tells me that two missing
devices is a no go, so I cannot mount it as anything but read-only. All
the data is there, /dev/sdd1 was never used and /dev/sdd was the
originally missing one. Mounting as read-only prevents you from adding
devices.

I guess I have to start over... At least the old pool is still readable.
Sucks that I have to rebuild it though. I guess there is no way to keep
my snapshots without having them fill their full size.

On Wed, Aug 13, 2014, at 04:51 PM, Marc MERLIN wrote:
> On Wed, Aug 13, 2014 at 07:35:49AM +0200, Rasmus Abrahamsen wrote:
> > I added the /dev/sdd1 to my raid and deleted the missing. Now the
> > /dev/sdd does not show up anymore and I have a /dev/sdd1. But I still
> > have the **** Some devices missing and the command btrfs delete missing
> > /mnt does not actually do anything. I tried btrfs delete /dev/sdd /mnt
> > but it said that /dev/sdd was busy.
> 
> Have you tried
> btrfs device delete missing /mnt/mountpoint ?
>  
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" -
> A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet
>                                       cooking
> Home page: http://marc.merlins.org/  

Re: delete missing /dev/sdd which is now added as /dev/sdd1

[Duncan]

Rasmus Abrahamsen posted on Wed, 13 Aug 2014 17:36:06 +0200 as excerpted:

>  I now did the following
> 
> dd if=/dev/zero of=/dev/sdd
> 
> Finally, btrfs does not recognize the partition as part of my raid
> anymore, but what's worse is that it now tells me that two missing
> devices is a no go, so I cannot mount it as anything but read-only. All
> the data is there, /dev/sdd1 was never used and /dev/sdd was the
> originally missing one. Mounting as read-only prevents you from adding
> devices.

There's a patch in-flight that should, I believe, help with this.  I
haven't followed the specific details, but IIRC, it has something to do
with a missing sysfs entry.  Because of that missing entry, btrfs
(userspace) doesn't have the same view of filesystem device status that
the kernel has, and device add/replace/delete can go haywire.
Additionally, in some cases it triggers too many missing devices errors,
forcing the mount read-only, which in turn won't let you add devices to
fix the problem even if due to the circumstances you know all files are
actually there and intact.

So at this point you have three choices:

1) Give up on the filesystem, do a mkfs and go from there.

2) Wait for the patch to make it into kernel 3.17 and go from there.

3) Go read the list and find and cherrypick that patch, then rebuild with
it included, and go from there.


Meanwhile, an observation on the btrfs behavior that contributed to
getting you into this situation in the first place, and how to avoid it in
the future:

Btrfs filesystem and filesystem device detection relies on a "magic
string" found in the superblock to identify btrfs filesystems, and other
metadata within the superblock to determine what specific btrfs filesystem
it belongs to.  When events such as btrfs device add/delete/ replace that
change the devices composing a filesystem don't complete properly, this
superblock information can remain to confuse btrfs when it tries working
with the remaining device components in the filesystem.

That's what was happening to you, tho the missing sysfs entries mentioned
above make the situation rather worse.  Btrfs was still detecting these
remaining bits of superblock information saying that the device in
question belonged to the btrfs it originated with, and was getting
confused.

One way to avoid this problem is to use a tool called wipefs, part of
util-linux, to erase the magic-string btrfs detection uses, so that
particular device will no longer be detected as part of a btrfs, and thus
can't confuse btrfs.  Alternatively, you can simply zero-out the correct 8
bytes on the raw device manually, using for instance dd to write directly
to the device.

See this link from the btrfs wiki problem-FAQ for more (including how to
restore the 8-byte magic string using dd if you decide you made a mistake,
something not really possible if you zero out the entire device as you did
above):

https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#How_to_clean_up_old_superblock_.3F

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: delete missing /dev/sdd which is now added as /dev/sdd1

[Chris Murphy]

On Aug 13, 2014, at 5:19 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> 
> 2) Wait for the patch to make it into kernel 3.17 and go from there.

Based on integration right now it looks like a lot of stuff.


> 3) Go read the list and find and cherrypick that patch, then rebuild with
> it included, and go from there.

Could be this one:
http://git.kernel.org/cgit/linux/kernel/git/mason/linux-btrfs.git/patch/?id=0c414ebcd5ba515b58249099619a5d9dbcc1a3b8


Chris Murphy

Re: delete missing /dev/sdd which is now added as /dev/sdd1

[Rasmus Abrahamsen]

Wow, such a long and detailed message. Thank you so much for taking the
time to do this.

Well, to be honest, I am starting to like the idea of starting over,
although it will be a pain.

Besides, I would have to wait for any distro to make a live disk with
the new btrfs version because my linux partition is also on that raid.

I also learned that "btrfs delete missing" should only be used if the
drive is gone forever, not to force remove it because of a mistake I did
myself (tried to replace it with the 1,5 GB partition).

On Thu, Aug 14, 2014, at 04:06 AM, Chris Murphy wrote:
> 
> On Aug 13, 2014, at 5:19 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> > 
> > 2) Wait for the patch to make it into kernel 3.17 and go from there.
> 
> Based on integration right now it looks like a lot of stuff.
> 
> 
> > 3) Go read the list and find and cherrypick that patch, then rebuild with
> > it included, and go from there.
> 
> Could be this one:
> http://git.kernel.org/cgit/linux/kernel/git/mason/linux-btrfs.git/patch/?id=0c414ebcd5ba515b58249099619a5d9dbcc1a3b8
> 
> 
> Chris Murphy
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html