Add big device, remove small device, read-only

Add big device, remove small device, read-only

[Rasmus Abrahamsen]

Happy New Year!

I have a raid with a 1TB, .5TB, 1.5TB and recently added a 4TB and want to remove the 1.5TB. When saying btrfs dev delete it turned into readonly. I am on 4.2.5-1-ARCH and btrfs-progs v4.3.1 what can I do?
On top of that, my linux is on this same raid, so perhaps btrfs is writing some temp files in the filesystem but cannot?
. /dev/sdc1 on / type btrfs (ro,relatime,space_cache,subvolid=1187,subvol=/linux)

<zetok> Ralle: did you do balance before removing?

I did not, but I have experience with it balancing itself upon doing so.
Upon removing a device, that is.
I am just not sure how to proceed now that everything is read-only.

<zetok> I dunno, but generally with things that can crumble you should make it step-by-step
crumble?
<zetok> which, in combination arch + newest bleeding edge kernel should be mandatory
<zetok> I hope that you have backups?

I do have backups, but it's on Crashplan, so I would prefer not to have to go there.

<zetok> and do you have any logs?

Where would those be?
I never understood journalctl

<zetok> journalctl --since=today

Hmm, it was actually yesterday that I started the remove, so I did --since=yesterday
I am looking at the log now, please stnad by.
This is my log
http://pastebin.com/mCPi3y9r
But I fear that it became read-only before actually writing the error to the filesystem

Re: Add big device, remove small device, read-only

[Duncan]

Rasmus Abrahamsen posted on Fri, 01 Jan 2016 12:47:08 +0100 as excerpted:

> Happy New Year!
> 
> I have a raid with a 1TB, .5TB, 1.5TB and recently added a 4TB and want
> to remove the 1.5TB. When saying btrfs dev delete it turned into
> readonly. I am on 4.2.5-1-ARCH and btrfs-progs v4.3.1 what can I do?

This isn't going to help with the specific problem, and doesn't apply to 
your case now anyway as the 4 TB device has already been added so all 
you're doing now is deleting the old one, but FWIW...

There's a fairly new command, btrfs replace, that can be used to directly 
replace an old device with a new one, instead of doing btrfs device add, 
followed by btrfs device delete/remove.

> On top of that, my linux is on this same raid, so perhaps btrfs is
> writing some temp files in the filesystem but cannot?
> . /dev/sdc1 on / type btrfs
> (ro,relatime,space_cache,subvolid=1187,subvol=/linux)

Your wording leaves me somewhat confused.  You say your Linux, presumably 
your root filesystem, is on the same raid as the filesystem that is 
having problems.  That would imply that it's a different filesystem, 
which in turn would apply that the raid is below the filesystem level, 
say mdraid, dmraid, or hardware raid, with both your btrfs root 
filesystem, and the separate btrfs with the problems, on the same raid-
based device, presumably partitioned so you can put multiple filesystems 
on the same device.

Which of course would generally mean the two btrfs themselves aren't 
raid, unless of course you are using at least one non-btrfs raid as one 
device under a btrfs raid.  But while implied, that's not really 
supported by what you said, which suggests a single btrfs raid 
filesystem, instead.  In which case, perhaps you meant that this 
filesystem contains your root filesystem as well, not just that the raid 
contains it.

Of course, if your post had included the usual btrfs fi show and btrfs fi 
df (and btrfs fi usage would be good as well) that the wiki recommends be 
posted with such reports, that might make things clearer, but it doesn't, 
so we're left guessing...

But I'm assuming you meant a single multi-device btrfs, not multiple 
btrfs that happen to be on the same non-btrfs raid.

Another question the show and df would answer is what btrfs raid mode 
you're running.  The default for multiple device btrfs is of course raid1 
metadata and single mode data, but you might well have set it up with 
data and metadata in the same mode, and/or with raid0/5/6/10 for one or 
both data and metadata.  You didn't say and didn't provide the btrfs 
command output that would show it, so...

> <zetok> Ralle: did you do balance before removing?
> 
> I did not, but I have experience with it balancing itself upon doing so.
> Upon removing a device, that is.
> I am just not sure how to proceed now that everything is read-only.

You were correct in that regard.  btrfs device remove (or btrfs replace) 
trigger balance as part of the process, and balancing after adding a 
device, only to have balance trigger again with a delete/remove, is 
needless.

Actually, I suspect the remove-triggered balance ran across a problem it 
didn't know how to handle when attempting to move one of the chunks from 
the existing device, and that's what put the filesystem in read-only 
mode.  That's usually what happens when btrfs device remove triggers 
problems and people report it, anyway.  A balance before the remove would 
have simply triggered it then, anyway.

But what the specific problem is, and what to do about it, remains to be 
seen.  Having that btrfs fi show and btrfs fi df would be a good start, 
letting us know at least what raid type we're dealing with, etc.

> <zetok> I hope that you have backups?
> 
> I do have backups, but it's on Crashplan, so I would prefer not to have
> to go there.

That's wise, both him asking and you replying you already have them, but 
just want to avoid using them if possible.  Waaayyy too many folks 
posting here find out the hard way about the admin's first rule of 
backups, in simplified form, that if you don't have backups, you are 
declaring by your actions that the data not backed up is worth less to 
you than the time, resources and hassle required to do those backups, 
despite any after-the-fact protests to the contrary.  Not being in that 
group already puts you well ahead of the game! =:^)

> <zetok> and do you have any logs?
> 
> Where would those be?
> I never understood journalctl
> 
> <zetok> journalctl --since=today
> 
> Hmm, it was actually yesterday that I started the remove, so I did
> --since=yesterday I am looking at the log now, please stnad by.
> This is my log http://pastebin.com/mCPi3y9r But I fear that it became
> read-only before actually writing the error to the filesystem

Hmm...  Looks like my strategy of having both systemd's journald, and 
syslog-ng, might pay off.  I have journald configured to only do 
temporary files, which it keeps in /run/log/journal, with /run of course 
tmpfs.  That way I get systemd's journal enhancements like the ability to 
do systemctl status <some-service> and have it give me the latest few log 
entries associated with that service as well.  But it doesn't journal to 
anything but tmpfs so it's current boot only.  Meanwhile, syslog-ng is 
configured to take messages from journald and to sort and filter them as 
it normally would, before saving them to various text-based logs.  That 
way I get only the text-based and easily grepped, etc, logs in permanent 
storage.  That lets me filter out "log noise" before it even hits storage 
(journald can apparently only filter on the output side, it writes 
everything it sees to the journal).  Also, because text-based logs are 
append-only, they don't heavily fragment like journald's binary logs do 
on btrfs due to the more random write-pattern of the journal files and 
btrfs being cow-based so rewrites to existing parts of the file copy them 
elsewhere, triggering heavy fragmentation.  I figure that's the best of 
both worlds. =:^)

But a benefit I hadn't considered until now is that when storage goes 
read-only, the journal, being tmpfs-only in my case, continues to 
journal, even when syslog-ng can no longer log to permanent storage 
because it's now read-only. =:^)


Back to your situation, however.  These will be kernel messages and thus 
appear in dmesg.  While that's size-restricted and old messages are 
thrown away once the ring-buffer fills up, with luck your buffer is large 
enough that the trigger for that read-only is still in dmesg.

And while waiting to see if dmesg returns anything interesting, another 
set of questions.  How old is your btrfs and with what kernel and btrfs-
progs version was it created, if you know, and was it originally created 
with mkfs.btrfs, or converted from ext* using btrfs-convert?  I'll guess 
it was created with mkfs.btrfs, but I'm asking, since ext* conversions 
have their own set of problems that are rare or don't happen at all on 
native-created btrfs, and it's often balance that exposes these 
problems.  If you created with mkfs.btrfs, at least we don't have to 
worry about the whole set of conversion-related problems.

Meanwhile, depending on the problem, a reboot will likely get you back to 
read-write mode, assuming the filesystem still mounts, but there's a risk 
you won't be able to mount again, as well, again depending on the problem.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Add big device, remove small device, read-only

[Chris Murphy]

On Fri, Jan 1, 2016 at 4:47 AM, Rasmus Abrahamsen <btrfs@rasmusa.net> wrote:
> Happy New Year!
>
> I have a raid with a 1TB, .5TB, 1.5TB and recently added a 4TB and want to remove the 1.5TB. When saying btrfs dev delete it turned into readonly. I am on 4.2.5-1-ARCH and btrfs-progs v4.3.1 what can I do?

btrfs fi show
lsblk -f
btrfs fi usage <pathtomountpointforbtrfsvolumeunderdiscussion>


he remove, so I did --since=yesterday
> I am looking at the log now, please stnad by.
> This is my log
> http://pastebin.com/mCPi3y9r

What's this?

Dec 31 15:03:56 rasmusahome systemd-udevd[6340]: inotify_add_watch(9,
/dev/sdd1, 10) failed: No space left on device
Dec 31 15:04:01 rasmusahome kernel:  sdd: sdd1
Dec 31 15:04:01 rasmusahome systemd-udevd[6341]: inotify_add_watch(9,
/dev/sdd1, 10) failed: No space left on device
Dec 31 15:05:43 rasmusahome kernel: BTRFS info (device sdb1): disk
added /dev/sdd1

Why is udev first complaining about no space left on sdd1, but then
it's being added to the btrfs volume?



-- 
Chris Murphy

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

I accidentically sent my messages directly to Duncan, I am copying them
in here.

Hello Duncan,

Thank you for the amazing response. Wow, you are awesome.

Here is the output of fi show, fi df and mount, sorry for not providing
them to begin with:

http://pastebin.com/DpiuDvRy


> On 01 Jan 2016, at 17:39, Duncan <1i5t5.duncan@cox.net> wrote:
> 
> Rasmus Abrahamsen posted on Fri, 01 Jan 2016 12:47:08 +0100 as excerpted:
> 
>> Happy New Year!
>> 
>> I have a raid with a 1TB, .5TB, 1.5TB and recently added a 4TB and want
>> to remove the 1.5TB. When saying btrfs dev delete it turned into
>> readonly. I am on 4.2.5-1-ARCH and btrfs-progs v4.3.1 what can I do?
> 
> This isn't going to help with the specific problem, and doesn't apply to 
> your case now anyway as the 4 TB device has already been added so all 
> you're doing now is deleting the old one, but FWIW...
> 
> There's a fairly new command, btrfs replace, that can be used to directly 
> replace an old device with a new one, instead of doing btrfs device add, 
> followed by btrfs device delete/remove.
> 
>> On top of that, my linux is on this same raid, so perhaps btrfs is
>> writing some temp files in the filesystem but cannot?
>> . /dev/sdc1 on / type btrfs
>> (ro,relatime,space_cache,subvolid=1187,subvol=/linux)
> 
> Your wording leaves me somewhat confused.  You say your Linux, presumably 
> your root filesystem, is on the same raid as the filesystem that is 
> having problems.  That would imply that it's a different filesystem, 
> which in turn would apply that the raid is below the filesystem level, 
> say mdraid, dmraid, or hardware raid, with both your btrfs root 
> filesystem, and the separate btrfs with the problems, on the same raid-
> based device, presumably partitioned so you can put multiple filesystems 
> on the same device.
> 
> Which of course would generally mean the two btrfs themselves aren't 
> raid, unless of course you are using at least one non-btrfs raid as one 
> device under a btrfs raid.  But while implied, that's not really 
> supported by what you said, which suggests a single btrfs raid 
> filesystem, instead.  In which case, perhaps you meant that this 
> filesystem contains your root filesystem as well, not just that the raid 
> contains it.
> 
> Of course, if your post had included the usual btrfs fi show and btrfs fi 
> df (and btrfs fi usage would be good as well) that the wiki recommends be 
> posted with such reports, that might make things clearer, but it doesn't, 
> so we're left guessing...
> 
> But I'm assuming you meant a single multi-device btrfs, not multiple 
> btrfs that happen to be on the same non-btrfs raid.

My root / is a sub volume of my filesystem. I will only be talking about
the filesystem named “Fortune”, the one named “Glassbox” is not relevant
to this problem.

> 
> Another question the show and df would answer is what btrfs raid mode 
> you're running.  The default for multiple device btrfs is of course raid1 
> metadata and single mode data, but you might well have set it up with 
> data and metadata in the same mode, and/or with raid0/5/6/10 for one or 
> both data and metadata.  You didn't say and didn't provide the btrfs 
> command output that would show it, so...
> 
>> <zetok> Ralle: did you do balance before removing?
>> 
>> I did not, but I have experience with it balancing itself upon doing so.
>> Upon removing a device, that is.
>> I am just not sure how to proceed now that everything is read-only.
> 
> You were correct in that regard.  btrfs device remove (or btrfs replace) 
> trigger balance as part of the process, and balancing after adding a 
> device, only to have balance trigger again with a delete/remove, is 
> needless.
> 
> Actually, I suspect the remove-triggered balance ran across a problem it 
> didn't know how to handle when attempting to move one of the chunks from 
> the existing device, and that's what put the filesystem in read-only 
> mode.  That's usually what happens when btrfs device remove triggers 
> problems and people report it, anyway.  A balance before the remove would 
> have simply triggered it then, anyway.
> 
> But what the specific problem is, and what to do about it, remains to be 
> seen.  Having that btrfs fi show and btrfs fi df would be a good start, 
> letting us know at least what raid type we're dealing with, etc.
> 
>> <zetok> I hope that you have backups?
>> 
>> I do have backups, but it's on Crashplan, so I would prefer not to have
>> to go there.
> 
> That's wise, both him asking and you replying you already have them, but 
> just want to avoid using them if possible.  Waaayyy too many folks 
> posting here find out the hard way about the admin's first rule of 
> backups, in simplified form, that if you don't have backups, you are 
> declaring by your actions that the data not backed up is worth less to 
> you than the time, resources and hassle required to do those backups, 
> despite any after-the-fact protests to the contrary.  Not being in that 
> group already puts you well ahead of the game! =:^)
> 
>> <zetok> and do you have any logs?
>> 
>> Where would those be?
>> I never understood journalctl
>> 
>> <zetok> journalctl --since=today
>> 
>> Hmm, it was actually yesterday that I started the remove, so I did
>> --since=yesterday I am looking at the log now, please stnad by.
>> This is my log http://pastebin.com/mCPi3y9r But I fear that it became
>> read-only before actually writing the error to the filesystem
> 
> Hmm...  Looks like my strategy of having both systemd's journald, and 
> syslog-ng, might pay off.  I have journald configured to only do 
> temporary files, which it keeps in /run/log/journal, with /run of course 
> tmpfs.  That way I get systemd's journal enhancements like the ability to 
> do systemctl status <some-service> and have it give me the latest few log 
> entries associated with that service as well.  But it doesn't journal to 
> anything but tmpfs so it's current boot only.  Meanwhile, syslog-ng is 
> configured to take messages from journald and to sort and filter them as 
> it normally would, before saving them to various text-based logs.  That 
> way I get only the text-based and easily grepped, etc, logs in permanent 
> storage.  That lets me filter out "log noise" before it even hits storage 
> (journald can apparently only filter on the output side, it writes 
> everything it sees to the journal).  Also, because text-based logs are 
> append-only, they don't heavily fragment like journald's binary logs do 
> on btrfs due to the more random write-pattern of the journal files and 
> btrfs being cow-based so rewrites to existing parts of the file copy them 
> elsewhere, triggering heavy fragmentation.  I figure that's the best of 
> both worlds. =:^)
> 
> But a benefit I hadn't considered until now is that when storage goes 
> read-only, the journal, being tmpfs-only in my case, continues to 
> journal, even when syslog-ng can no longer log to permanent storage 
> because it's now read-only. =:^)
> 
> 
> Back to your situation, however.  These will be kernel messages and thus 
> appear in dmesg.  While that's size-restricted and old messages are 
> thrown away once the ring-buffer fills up, with luck your buffer is large 
> enough that the trigger for that read-only is still in dmesg.

Dmesg contains nothing interesting unfortunately. About 12 hours passed
and lots of cronjobs have attempted to run since then. I cannot disable
these because it is read-only. I can however stop the service. I will do
that now… Failed:

Failed to add /run/systemd/ask-password to directory watch: No space
left on device

> 
> And while waiting to see if dmesg returns anything interesting, another 
> set of questions.  How old is your btrfs and with what kernel and btrfs-
> progs version was it created, if you know, and was it originally created 
> with mkfs.btrfs, or converted from ext* using btrfs-convert?  I'll guess 
> it was created with mkfs.btrfs, but I'm asking, since ext* conversions 
> have their own set of problems that are rare or don't happen at all on 
> native-created btrfs, and it's often balance that exposes these 
> problems.  If you created with mkfs.btrfs, at least we don't have to 
> worry about the whole set of conversion-related problems.

I cannot remember how this filesystem was created, I believe it was
mkfs.btrfs. I created it last year (actually 2014) on the latest Arch
Linux. I created it as a btrfs from the beginning.

> 
> Meanwhile, depending on the problem, a reboot will likely get you back to 
> read-write mode, assuming the filesystem still mounts, but there's a risk 
> you won't be able to mount again, as well, again depending on the problem.

I am not physically next to my machine right now, so I was afraid of
rebooting in case I could do something remotely. I will be close to my
machine in a few hours. I will try rebooting it now, however.

> 
> -- 
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Thank you very much for your extremely thorough response despite my lack
of providing information.

Rasmus


UPDATE:

The reboot succeeded and it is not read-only anymore. I tried the
command:

btrfs dev delete /dev/sdb1 /mnt/fortune

again and it's running again. This time I am keeping a window with:

cat /proc/kmsg | grep BTRFS

running. We'll see how it goes.

Rasmus

On Fri, Jan 1, 2016, at 09:07 PM, Chris Murphy wrote:
> On Fri, Jan 1, 2016 at 4:47 AM, Rasmus Abrahamsen <btrfs@rasmusa.net>
> wrote:
> > Happy New Year!
> >
> > I have a raid with a 1TB, .5TB, 1.5TB and recently added a 4TB and want to remove the 1.5TB. When saying btrfs dev delete it turned into readonly. I am on 4.2.5-1-ARCH and btrfs-progs v4.3.1 what can I do?
> 
> btrfs fi show
> lsblk -f
> btrfs fi usage <pathtomountpointforbtrfsvolumeunderdiscussion>
> 
> 
> he remove, so I did --since=yesterday
> > I am looking at the log now, please stnad by.
> > This is my log
> > http://pastebin.com/mCPi3y9r
> 
> What's this?
> 
> Dec 31 15:03:56 rasmusahome systemd-udevd[6340]: inotify_add_watch(9,
> /dev/sdd1, 10) failed: No space left on device
> Dec 31 15:04:01 rasmusahome kernel:  sdd: sdd1
> Dec 31 15:04:01 rasmusahome systemd-udevd[6341]: inotify_add_watch(9,
> /dev/sdd1, 10) failed: No space left on device
> Dec 31 15:05:43 rasmusahome kernel: BTRFS info (device sdb1): disk
> added /dev/sdd1
> 
> Why is udev first complaining about no space left on sdd1, but then
> it's being added to the btrfs volume?
> 
> 
> 
> -- 
> Chris Murphy

I have no idea.

I am adding the commands you asked for:

[ralle@rasmusahome ~]$ sudo lsblk -f
NAME   FSTYPE LABEL    UUID                                 MOUNTPOINT
sda
`-sda1 btrfs  Fortune  f87c173c-c873-424f-9c79-3be2c4d37d88
sdb
|-sdb1 btrfs  Fortune  f87c173c-c873-424f-9c79-3be2c4d37d88
`-sdb2 btrfs  Glassbox ca44c8e4-1a26-4b4a-ba52-38a4b0d3f104
/mnt/glassbox
sdc
`-sdc1 btrfs  Fortune  f87c173c-c873-424f-9c79-3be2c4d37d88 /mnt/fortune
sdd
`-sdd1 btrfs  Fortune  f87c173c-c873-424f-9c79-3be2c4d37d88
sde
`-sde1

[ralle@rasmusahome ~]$ sudo btrfs fi usage /mnt/fortune/
Overall:
    Device size:                   5.00TiB
    Device allocated:              2.20TiB
    Device unallocated:            2.81TiB
    Device missing:                  0.00B
    Used:                          2.19TiB
    Free (estimated):              1.41TiB      (min: 1.41TiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 114.56MiB)

Data,RAID1: Size:1.09TiB, Used:1.08TiB
   /dev/sda1     787.00GiB
   /dev/sdb1     613.00GiB
   /dev/sdc1     326.00GiB
   /dev/sdd1     500.00GiB

Metadata,RAID1: Size:11.00GiB, Used:9.12GiB
   /dev/sda1       7.00GiB
   /dev/sdb1       4.00GiB
   /dev/sdc1       4.00GiB
   /dev/sdd1       7.00GiB

System,RAID1: Size:32.00MiB, Used:208.00KiB
   /dev/sda1      32.00MiB
   /dev/sdb1      32.00MiB

Unallocated:
   /dev/sda1     137.48GiB
   /dev/sdb1      16.00EiB
   /dev/sdc1     135.76GiB
   /dev/sdd1       3.14TiB

I am sorry if this message is very confusing...

Rasmus

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

This morning I wanted to check on progress of removing the device but
when trying to get the screen to wake, I was met with nothing. I tried
SSH, nothing. The machine had crashed or something of the like. After a
reboot, the device is still part of the filesystem.
I have now tried booting from an Arch Linux flash drive and try to
remove the device when booted from that.

Let's see how it goes.

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

Oops, sent this message from the wrong email, then tried attaching a file. I am really bad at this.

This morning I wanted to check on progress of removing the device but
when trying to get the screen to wake, I was met with nothing. I tried
SSH, nothing. The machine had crashed or something of the like. After a
reboot, the device is still part of the filesystem.
I have now tried booting from an Arch Linux flash drive and try to
remove the device when booted from that.

Let's see how it goes.

UPDATE: Seems like the kernel crashed. I have attached a picture link.
https://dl.dropboxusercontent.com/u/323865/2016-01-02%20IMG_8478.jpg

Can anyone give me any advice on what to do?

I think I will reboot and try a btrfs scrub before trying again.

Rasmus

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

Okay, the scrub seemingly did nothing, I am now running balance. This is what I am getting so far:

https://www.dropbox.com/s/02rpab0zcc6er2u/2016-01-02%20IMG_8483.jpg?dl=0

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

The scrub was frozen. I tried to cancel it but it had become read-only.
I rebooted and am now looking into what btrfsck can do for me

On Sat, Jan 2, 2016, at 05:45 PM, Rasmus Abrahamsen wrote:
> Okay, the scrub seemingly did nothing, I am now running balance. This is
> what I am getting so far:
> 
> https://www.dropbox.com/s/02rpab0zcc6er2u/2016-01-02%20IMG_8483.jpg?dl=0--
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Add big device, remove small device, read-only

[Chris Murphy]

On Fri, Jan 1, 2016 at 1:20 PM, Rasmus Abrahamsen <me@rasmusa.net> wrote:
> [ralle@rasmusahome ~]$ sudo btrfs fi usage /mnt/fortune/
[snip]
> Unallocated:
>    /dev/sda1     137.48GiB
>    /dev/sdb1      16.00EiB
>    /dev/sdc1     135.76GiB
>    /dev/sdd1       3.14TiB

Another phantom 16.00EiB, this time it appears as unallocated space on
a devices, rather than free space as it is in a bug I recently filed.

https://bugzilla.kernel.org/show_bug.cgi?id=110111


-- 
Chris Murphy

Re: Add big device, remove small device, read-only

[Chris Murphy]

On Sat, Jan 2, 2016 at 12:47 PM, Rasmus Abrahamsen <btrfs@rasmusa.net> wrote:
> The scrub was frozen. I tried to cancel it but it had become read-only.
> I rebooted and am now looking into what btrfsck can do for me

Stay away from btrfs check --repair until it's a last restort. As in,
you have to have all data scraped off the volume before you decide to
use it. It's gotten a lot better at not making things worse, but I
still don't trust it.


-- 
Chris Murphy

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

What do you recommend I do? Everything is redundant across disks.
Perhaps I can disconnect the one you mentioned and delete missing.

On Sat, Jan 2, 2016, at 08:56 PM, Chris Murphy wrote:
> On Fri, Jan 1, 2016 at 1:20 PM, Rasmus Abrahamsen <me@rasmusa.net> wrote:
> > [ralle@rasmusahome ~]$ sudo btrfs fi usage /mnt/fortune/
> [snip]
> > Unallocated:
> >    /dev/sda1     137.48GiB
> >    /dev/sdb1      16.00EiB
> >    /dev/sdc1     135.76GiB
> >    /dev/sdd1       3.14TiB
> 
> Another phantom 16.00EiB, this time it appears as unallocated space on
> a devices, rather than free space as it is in a bug I recently filed.
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=110111
> 
> 
> -- 
> Chris Murphy

Re: Add big device, remove small device, read-only

[Chris Murphy]

On Sat, Jan 2, 2016 at 12:58 PM, Rasmus Abrahamsen <btrfs@rasmusa.net> wrote:
> What do you recommend I do? Everything is redundant across disks.
> Perhaps I can disconnect the one you mentioned and delete missing.

No, please stop trying new things like throwing spaghetti at a wall.
The file system is not healthy or it wouldn't behave the way it is. If
you have a current backup, and want to keep playing to see how bad
it'll get, then keep proceeding.

But if you have any important data on this volume that you don't have
a current backup for, then you need to make as few changes as
possible. Don't delete devices. Don't run btrfsck. Don't scrub. Don't
balance. Just try to get it mounted first, and then btrfs send, rsync,
or cp the data you need off the volume onto a new volume.




-- 
Chris Murphy

Re: Add big device, remove small device, read-only

[Chris Murphy]

On Sat, Jan 2, 2016 at 1:15 PM, Chris Murphy <lists@colorremedies.com> wrote:
> On Sat, Jan 2, 2016 at 12:58 PM, Rasmus Abrahamsen <btrfs@rasmusa.net> wrote:
>> What do you recommend I do? Everything is redundant across disks.
>> Perhaps I can disconnect the one you mentioned and delete missing.
>
> No, please stop trying new things like throwing spaghetti at a wall.
> The file system is not healthy or it wouldn't behave the way it is. If
> you have a current backup, and want to keep playing to see how bad
> it'll get, then keep proceeding.
>
> But if you have any important data on this volume that you don't have
> a current backup for, then you need to make as few changes as
> possible. Don't delete devices. Don't run btrfsck. Don't scrub. Don't
> balance. Just try to get it mounted first, and then btrfs send, rsync,
> or cp the data you need off the volume onto a new volume.

That means try a normal mount with no mount options first. If that
doesn't work then try:

-o recovery
-o ro,recovery
-o ro,recovery,degraded

If you can get it rw mounted, then you maybe can make ro snapshots
that you can send/receive to a new Btrfs file system. Or rsync -a or
cp -a to a new other file system.

If you can't get it rw mounted, and don't already have ro snapshots of
the things you want, then you'll have to use rsync or cp.


-- 
Chris Murphy

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

Sounds like you think my best bet is to re-roll my filesystem instead of
attempt to repair it. Is that right?

I have snapshots which are based on each other as a backup of a remote
machine by date. By sending these snapshots to a new filesystem, will I
be able to have them still be incremental and save space?

On Sat, Jan 2, 2016, at 09:32 PM, Chris Murphy wrote:
> On Sat, Jan 2, 2016 at 1:15 PM, Chris Murphy <lists@colorremedies.com>
> wrote:
> > On Sat, Jan 2, 2016 at 12:58 PM, Rasmus Abrahamsen <btrfs@rasmusa.net> wrote:
> >> What do you recommend I do? Everything is redundant across disks.
> >> Perhaps I can disconnect the one you mentioned and delete missing.
> >
> > No, please stop trying new things like throwing spaghetti at a wall.
> > The file system is not healthy or it wouldn't behave the way it is. If
> > you have a current backup, and want to keep playing to see how bad
> > it'll get, then keep proceeding.
> >
> > But if you have any important data on this volume that you don't have
> > a current backup for, then you need to make as few changes as
> > possible. Don't delete devices. Don't run btrfsck. Don't scrub. Don't
> > balance. Just try to get it mounted first, and then btrfs send, rsync,
> > or cp the data you need off the volume onto a new volume.
> 
> That means try a normal mount with no mount options first. If that
> doesn't work then try:
> 
> -o recovery
> -o ro,recovery
> -o ro,recovery,degraded
> 
> If you can get it rw mounted, then you maybe can make ro snapshots
> that you can send/receive to a new Btrfs file system. Or rsync -a or
> cp -a to a new other file system.
> 
> If you can't get it rw mounted, and don't already have ro snapshots of
> the things you want, then you'll have to use rsync or cp.
> 
> 
> -- 
> Chris Murphy

Re: Add big device, remove small device, read-only

[Chris Murphy]

On Sat, Jan 2, 2016 at 1:39 PM, Rasmus Abrahamsen <btrfs@rasmusa.net> wrote:
> Sounds like you think my best bet is to re-roll my filesystem instead of
> attempt to repair it. Is that right?

If you want to get back to a working fs as soon as possible, yes. You
could take a btrfs-image of it first; if that fails you can use
btrfs-debug-tree. Both of those should direct the output to a file on
another file system. Maybe a dev will find it useful?

And then if you want you can keep trying to repair it, but my
experience is that in such cases the fs quickly is in the weeds and is
beyond repair even when it's possible to ro mount or use restore to
get all data out.


> I have snapshots which are based on each other as a backup of a remote
> machine by date. By sending these snapshots to a new filesystem, will I
> be able to have them still be incremental and save space?

Yes, the first one without -p is the big send. Then each additional
one uses the most immediate previous snapshot with -p. I haven't tried
the -e option with multiple increments, so I don't know if that works
or if it's expected to work.

I'm having to do this right now myself with an imploded raid1 that
won't mount so I might find out in the next hour or two if -e works.



-- 
Chris Murphy

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

Okay, I will rebuild it then. Thank you.

Where do I add -p?

On Sat, Jan 2, 2016, at 10:28 PM, Chris Murphy wrote:
> On Sat, Jan 2, 2016 at 1:39 PM, Rasmus Abrahamsen <btrfs@rasmusa.net>
> wrote:
> > Sounds like you think my best bet is to re-roll my filesystem instead of
> > attempt to repair it. Is that right?
> 
> If you want to get back to a working fs as soon as possible, yes. You
> could take a btrfs-image of it first; if that fails you can use
> btrfs-debug-tree. Both of those should direct the output to a file on
> another file system. Maybe a dev will find it useful?
> 
> And then if you want you can keep trying to repair it, but my
> experience is that in such cases the fs quickly is in the weeds and is
> beyond repair even when it's possible to ro mount or use restore to
> get all data out.
> 
> 
> > I have snapshots which are based on each other as a backup of a remote
> > machine by date. By sending these snapshots to a new filesystem, will I
> > be able to have them still be incremental and save space?
> 
> Yes, the first one without -p is the big send. Then each additional
> one uses the most immediate previous snapshot with -p. I haven't tried
> the -e option with multiple increments, so I don't know if that works
> or if it's expected to work.
> 
> I'm having to do this right now myself with an imploded raid1 that
> won't mount so I might find out in the next hour or two if -e works.
> 
> 
> 
> -- 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Add big device, remove small device, read-only

[Chris Murphy]

On Sat, Jan 2, 2016 at 2:30 PM, Rasmus Abrahamsen <btrfs@rasmusa.net> wrote:
> Okay, I will rebuild it then. Thank you.
>
> Where do I add -p?

OK for me, I have these ro snapshots:
drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150403
drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150404
drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150412
drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150922

# btrfs send everything-20150403 | btrfs receive /mnt/new
# btrfs send -p everything-20150403 everything-20150404 | btrfs receive /mnt/new
# btrfs send -p everything-20150404 everything-20150412 | btrfs receive /mnt/new
# btrfs send -p everything-20150412 everything-20150922 | btrfs receive /mnt/new

I have not tried to use && in between those, instead of just
separately issuing the commands.

It's valid to always use the same first subvolume as -p, but the
increments are then larger so even though it succeeds, it's also
inefficient.

-- 
Chris Murphy

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

Do you mean that the pipe is not a bash pipe or that you executed the
commands as written? Sounds like you had two terminals and executed the
left side of the pipe and then the right side.

On Sat, Jan 2, 2016, at 10:37 PM, Chris Murphy wrote:
> On Sat, Jan 2, 2016 at 2:30 PM, Rasmus Abrahamsen <btrfs@rasmusa.net>
> wrote:
> > Okay, I will rebuild it then. Thank you.
> >
> > Where do I add -p?
> 
> OK for me, I have these ro snapshots:
> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150403
> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150404
> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150412
> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150922
> 
> # btrfs send everything-20150403 | btrfs receive /mnt/new
> # btrfs send -p everything-20150403 everything-20150404 | btrfs receive
> /mnt/new
> # btrfs send -p everything-20150404 everything-20150412 | btrfs receive
> /mnt/new
> # btrfs send -p everything-20150412 everything-20150922 | btrfs receive
> /mnt/new
> 
> I have not tried to use && in between those, instead of just
> separately issuing the commands.
> 
> It's valid to always use the same first subvolume as -p, but the
> increments are then larger so even though it succeeds, it's also
> inefficient.
> 
> -- 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Add big device, remove small device, read-only

[Chris Murphy]

On Sat, Jan 2, 2016 at 2:42 PM, Rasmus Abrahamsen <btrfs@rasmusa.net> wrote:
> On Sat, Jan 2, 2016, at 10:37 PM, Chris Murphy wrote:
>> On Sat, Jan 2, 2016 at 2:30 PM, Rasmus Abrahamsen <btrfs@rasmusa.net>
>> wrote:
>> > Okay, I will rebuild it then. Thank you.
>> >
>> > Where do I add -p?
>>
>> OK for me, I have these ro snapshots:
>> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150403
>> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150404
>> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150412
>> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150922
>>
>> # btrfs send everything-20150403 | btrfs receive /mnt/new
>> # btrfs send -p everything-20150403 everything-20150404 | btrfs receive
>> /mnt/new
>> # btrfs send -p everything-20150404 everything-20150412 | btrfs receive
>> /mnt/new
>> # btrfs send -p everything-20150412 everything-20150922 | btrfs receive
>> /mnt/new
>>

> Do you mean that the pipe is not a bash pipe or that you executed the
> commands as written? Sounds like you had two terminals and executed the
> left side of the pipe and then the right side.

I execute those commands exactly as written, in a single shell. Each
command completes before I can execute the next one.

The part I wrote about && is about my lack of bash experience; I don't
know of a way to execute the above commands at once such that they
happen in exactly that sequence so I could walk away.






-- 
Chris Murphy

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

Okay thank you.

I don't know the ordering of && and |. I think you can use parentheses
in bash perhaps? In any case, line breaks work to separate the tasks.

I am now doing the following:

I detached my new 4 TB drive and am mounting -o degraded. Then I am
doing

# find . -type f -exec cat {} > /dev/null \;

In each folder of data I care about. If all of it is readable without my
TB, then I can safely create a new BTRFS on the 4 TB and migrate my data
over.

On Sat, Jan 2, 2016, at 11:46 PM, Chris Murphy wrote:
> On Sat, Jan 2, 2016 at 2:42 PM, Rasmus Abrahamsen <btrfs@rasmusa.net>
> wrote:
> > On Sat, Jan 2, 2016, at 10:37 PM, Chris Murphy wrote:
> >> On Sat, Jan 2, 2016 at 2:30 PM, Rasmus Abrahamsen <btrfs@rasmusa.net>
> >> wrote:
> >> > Okay, I will rebuild it then. Thank you.
> >> >
> >> > Where do I add -p?
> >>
> >> OK for me, I have these ro snapshots:
> >> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150403
> >> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150404
> >> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150412
> >> drwxrwx---. 1 chris chris  218 Apr  3  2015 everything-20150922
> >>
> >> # btrfs send everything-20150403 | btrfs receive /mnt/new
> >> # btrfs send -p everything-20150403 everything-20150404 | btrfs receive
> >> /mnt/new
> >> # btrfs send -p everything-20150404 everything-20150412 | btrfs receive
> >> /mnt/new
> >> # btrfs send -p everything-20150412 everything-20150922 | btrfs receive
> >> /mnt/new
> >>
> 
> > Do you mean that the pipe is not a bash pipe or that you executed the
> > commands as written? Sounds like you had two terminals and executed the
> > left side of the pipe and then the right side.
> 
> I execute those commands exactly as written, in a single shell. Each
> command completes before I can execute the next one.
> 
> The part I wrote about && is about my lack of bash experience; I don't
> know of a way to execute the above commands at once such that they
> happen in exactly that sequence so I could walk away.
> 
> 
> 
> 
> 
> 
> -- 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Add big device, remove small device, read-only

[Duncan]

Rasmus Abrahamsen posted on Fri, 01 Jan 2016 21:20:13 +0100 as excerpted:

> I accidentically sent my messages directly to Duncan, I am copying them
> in here.
> 
> Hello Duncan,
> 
> Thank you for the amazing response. Wow, you are awesome.

Just a note to mention that real life (TM) got in the way, and I'm a few 
days and a couple hundred posts behind on the list, now.  Sounds like you 
have a backup tho, and if worse comes to worse, you can simply blow away 
the filesystem and start over.  Between that and Chris Murphy helping you 
now, I read the thread to date but am simply marking it read without 
further replies as it exists ATM, but might reply to new posts to the 
thread from now, if I think I can be helpful.

(Which is why I try to discourage direct replies, too.  With direct 
replies to a single person, if that person doesn't get back...  While if 
it's to the list, there's more that can take up the thread, it's not on 
just one person.  Of course the just to me was an accident, but...)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Add big device, remove small device, read-only

[Rasmus Abrahamsen]

On Wed, Jan 6, 2016, at 07:45 AM, Duncan wrote:
> Rasmus Abrahamsen posted on Fri, 01 Jan 2016 21:20:13 +0100 as excerpted:
> 
> > I accidentically sent my messages directly to Duncan, I am copying them
> > in here.
> > 
> > Hello Duncan,
> > 
> > Thank you for the amazing response. Wow, you are awesome.
> 
> Just a note to mention that real life (TM) got in the way, and I'm a few 
> days and a couple hundred posts behind on the list, now.  Sounds like you 
> have a backup tho, and if worse comes to worse, you can simply blow away 
> the filesystem and start over.  Between that and Chris Murphy helping you 
> now, I read the thread to date but am simply marking it read without 
> further replies as it exists ATM, but might reply to new posts to the 
> thread from now, if I think I can be helpful.
> 
> (Which is why I try to discourage direct replies, too.  With direct 
> replies to a single person, if that person doesn't get back...  While if 
> it's to the list, there's more that can take up the thread, it's not on 
> just one person.  Of course the just to me was an accident, but...)
> 
> -- 
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Yeah, I have made many mistakes as this is my first time participating
on a mailing list :)

I decided to blow away the filesystem and start over. I did so by
physically removing the new 4 TB drive, mounting -o degraded and then
running:

find . -type f -exec cat {} >> /dev/null \;

In all folders that matter to me to make sure everything is readable
without the new drive. I then formatted the new drive as BTRFS and moved
over all my data. Then I added my old 4 TB drive to the new filesystem
and did balance -dconvert=raid1 -mconvert=raid1. I am now online with
the new filesystem and things are good.

I was expecting btrfs to just work (tm) when adding the new drive and
removing the old, so that made me a little sad. However, it's awesome
that no data was lost and my method actually worked.

Thanks for a great place and product.