Virtual Device Support

Virtual Device Support

[George Mitchell]

One the things that is frustrating me the most at this point from a user 
perspective regarding btrfs is the current lack of virtual devices to 
describe volumes and subvolumes.  The current method of simply using a 
random member device or a LABEL or a UUID is just not working well for 
me.  Having a well thought out virtual device infrastructure would 
greatly simplify use and also relieve problems with util-linux tools.  I 
REALLY DO understand that most if not all btrfs devs at this point are 
working overtime on things much more serious than this issue and I 
REALLY DO have the patience to wait on this one.  But I also REALLY DO 
hope there is some plan for this in the future.  Certainly a lot of 
design for it should already exist within software RAID md code and LVM 
code.

Re: Virtual Device Support

[Martin]

On 10/05/13 15:03, George Mitchell wrote:
> One the things that is frustrating me the most at this point from a user
> perspective ...  The current method of simply using a
> random member device or a LABEL or a UUID is just not working well for
> me.  Having a well thought out virtual device infrastructure would...

Sorry, I'm a bit lost for your comments...

What is your use case and what are you hoping/expecting to see?


I've been following development of btrfs for a while and I'm looking
forward to use it to efficiently replace some of the very useful
features of LVM2, drbd, and md-raid that I'm using at present...

OK, so the way of managing all that is going to be a little different.

How would you want that?


Regards,
Martin

Re: Virtual Device Support

[George Mitchell]

In reply to both of these comments in one message, let me give you an 
example.

I use shell scripts to mount and unmount btrfs volumes for backup 
purposes.  Most of these volumes are not listed in fstab simply because 
I do not want to have to clutter my fstab with volumes that are used 
only for backup.  So the only way I can mount them is either by LABEL or 
by UUID.  But I can't unmount them by either LABEL or UUID because that 
is not supported by util-linux and they have no intention of supporting 
it in the future.  So I have to resort to unmounting by directory and it 
becomes back and forth between LABEL and directory which becomes very 
confusing when you are dealing with complex shell scripts.  This is 
intolerable for me so I use a kludge that allows me to first translate 
from LABEL to device and then unmount by device.  To me it just seems 
klutzy that one has to resort to these sorts of games to use a file 
system that is supposed to be an improvement on what we already have.  A 
simple virtual volume identifier would resolve that. Doing the same for 
subvolumes would be nice, but I could live without it with no problem.  
I have worked with nixes for 30 years beginning with AT&T pre-SRV on 
DEC-1170s and have seen a lot of changes since those days, most of them 
for the better.  But, while functionality is mandatory, convenience is 
always appreciated and can help avoid costly mistakes and save time.  As 
I stated in my original post, I KNOW and appreciate that all of you are 
working hard on things that matter far more than this trivial item.  But 
it is a major convenience and clarity issue for me and I am sure it will 
be for others as well.  It is only rational that one should be able to 
expect to mount by LABEL and unmount by LABEL, but that doesn't work, 
and a major part of the reason that doesn't work is that btrfs does not 
conform to the pattern of just about every other file system on the 
planet in regards to how it treats mount points.  And this is not even 
to mention all the other issues involved like a large number of 
utilities that have no way of knowing that a given partition is mounted, 
which would also be resolved by virtual mount points since many if not 
most of those utilities understand and process virtual volume identifiers.

Please just do me a favor and think about this a bit before you just 
write it off.

- George



On 05/19/2013 04:04 AM, Martin wrote:
> On 10/05/13 15:03, George Mitchell wrote:
>> One the things that is frustrating me the most at this point from a user
>> perspective ...  The current method of simply using a
>> random member device or a LABEL or a UUID is just not working well for
>> me.  Having a well thought out virtual device infrastructure would...
> Sorry, I'm a bit lost for your comments...
>
> What is your use case and what are you hoping/expecting to see?
>
>
> I've been following development of btrfs for a while and I'm looking
> forward to use it to efficiently replace some of the very useful
> features of LVM2, drbd, and md-raid that I'm using at present...
>
> OK, so the way of managing all that is going to be a little different.
>
> How would you want that?
>
>
> Regards,
> Martin
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

On 05/19/2013 04:15 AM, Roman Mamedov wrote:
> On Fri, 10 May 2013 07:03:38 -0700
> George Mitchell <george@chinilu.com> wrote:
>
>> One the things that is frustrating me the most at this point from a user
>> perspective regarding btrfs is the current lack of virtual devices to
>> describe volumes and subvolumes.
>  From a user perspective btrfs subvolumes have a lot in common with just
> regular directories aka folders, and nothing in common with (block)devices.
> "Describing them with virtual devices" does not seem to make a whole lot of
> sense.
>

Re: Virtual Device Support

[Martin]

OK, so to summarise:


On 19/05/13 15:49, George Mitchell wrote:
> In reply to both of these comments in one message, let me give you an
> example.
> 
> I use shell scripts to mount and unmount btrfs volumes for backup
> purposes.  Most of these volumes are not listed in fstab simply because
> I do not want to have to clutter my fstab with volumes that are used
> only for backup.  So the only way I can mount them is either by LABEL or
> by UUID.  But I can't unmount them by either LABEL or UUID because that
> is not supported by util-linux and they have no intention of supporting
> it in the future.  So I have to resort to unmounting by directory ...

Which all comes to a way of working...

Likewise, I have some old and long used backups scripts that mount a
one-of-many backups disk pack. My solution is to use filesystem labels
and to use 'sed' to update just the one line in /etc/fstab for the
backups mount point label so that the backups are then mounted/unmounted
by the mount point.

I've never been able to use the /dev/sdXX numbering because the multiple
physical drives can be detected in a different order.

Agreed, that for the sake of good consistency, being able to unmount by
filesystem label is a nice/good idea. But is there any interest for that
to be picked up? Put in a bug/feature request onto bugzilla?


I would guess that most developers focus on mount point and let
fstab/mtab sort out the detail...

Regards,
Martin

Re: Virtual Device Support

[George Mitchell]

In my case, I am backing up a system spanning five drives formatted 
btrfs, on a separate drive containing a separate backup volume and 
multiple complete backups, each from a different point in time.  This 
gives me protection from filesystem corruption, since the backups are on 
a separate volume, also protection from accidental deletion and other 
such issues by having backups spread over time going back as far as 
three months.  It also makes things very simple since I can just mount 
one of these backup subvolumes in place of the original and immediately 
be up and running.  Of course, all btrfs volumes could die at once as a 
result of some obscure problem such as a poison update or something like 
that, and that is why I keep a constantly updated JFS (and period 
backups to bluray) copy on hand.  I realize this is not foolproof, and 
actually plan to extend it further.  But huge drives are terribly 
inexpensive right now and this is one way I can take advantage of that.  
Of course I could have done this using multiple partitions, and I may 
one day regret not doing it that way for the very reasons you point out. 
However, I believe that I am sufficiently protected at this point to 
take the risk.  I really figure that if something were to corrupt both 
my main system AND the backup volume at the same time, it would probably 
knock out separate partitions as well. But ... perhaps not.  But I am 
indeed aware that one filesystem corruption could knock out all of those 
backups in one sweep.  As for the umount issue, it is ridiculously easy 
to script around, it just seems like somebody, either on the util-linux 
side, or on the btrfs side, could provide a more elegant solution, but 
it seems to fly in the face of entrenched ideologies on both sides. 
Fortunately, my only real problem that I can't script around is the boot 
issue and that is hopefully, just a matter of time before it gets 
fixed.  Thanks for your thoughts,  George

On 05/21/2013 01:16 AM, Michael Johnson - MJ wrote:
> I realize I am a bit late to the party on this, and but I would like 
> to understand the details of the workflow you are describing as I am 
> not seeing the benefit to creating backups on different subvolumes 
> with btrfs (but that's not to say there aren't reasons).
>
> The way I've gone about things is to have one btrfs volume mounted at 
> say /mnt/brtfs, backup to it, and then creates a read-only snapshots 
> in /mnt/btrfs/.snapshots.  I think this gives me all the benefits of 
> what you are describing without any of the hassle.
>
> Now, if my btrfs were to become corrupted, I would loose all of my 
> backups, but I believe you would be in the same boat using 
> completely separate subvolumes as they still part of the same 
> underlying data structure.  You would have a similar issue with zfs I 
> believe.
>
> With more tradition filesystems where the volume management is 
> separate from the filesystem, having 2 separate instances of say xfs, 
> on lvm volumes or even different partitions, your filesystem 
> corruption would not spread from volume A to volume B so that level of 
> separation makes sense.  But with btrfs, treating the subvolumes like 
> this does not provide such protection, If this is the reason for your 
> workflow, it may be simply that it used to provide some benefit, but 
> that benefit is gone.  As such I think you can actually simplify your 
> workflow and utilize btrfs more fully.
>
> That being said, I don't really know your workflow and the reasons for 
> it, so it is quite possibly a reasonable thing to keep doing.  I 
> simply don't have enough information and know that I have often time 
> found my self annoyed with a change, spent a lot of time to working 
> around the change, and then later realized, but I just was stuck in an 
> old way of thinking and that the new way allowed a more elegant workflow.
>
> But that is all just food for thought.  I do agree that if I can say 
> "mount LABEL=foo" it would be expected that I could also say "umount 
> LABEL=foo".  Perhaps the right thing to do would be to have modify the 
> btrfs command to allow 'btrfs mount' and 'btrfs umount' similar to the 
> way zfs works as this would allow some fun magic to happen outside of 
> util-linux.
>
> In any case, I hope my thoughts are at least a little useful.  Cheers!
>
>
> On Sun, May 19, 2013 at 7:49 AM, George Mitchell <george@chinilu.com 
> <mailto:george@chinilu.com>> wrote:
>
>     In reply to both of these comments in one message, let me give you
>     an example.
>
>     I use shell scripts to mount and unmount btrfs volumes for backup
>     purposes.  Most of these volumes are not listed in fstab simply
>     because I do not want to have to clutter my fstab with volumes
>     that are used only for backup.  So the only way I can mount them
>     is either by LABEL or by UUID.  But I can't unmount them by either
>     LABEL or UUID because that is not supported by util-linux and they
>     have no intention of supporting it in the future.  So I have to
>     resort to unmounting by directory and it becomes back and forth
>     between LABEL and directory which becomes very confusing when you
>     are dealing with complex shell scripts.  This is intolerable for
>     me so I use a kludge that allows me to first translate from LABEL
>     to device and then unmount by device.  To me it just seems klutzy
>     that one has to resort to these sorts of games to use a file
>     system that is supposed to be an improvement on what we already
>     have.  A simple virtual volume identifier would resolve that.
>     Doing the same for subvolumes would be nice, but I could live
>     without it with no problem.  I have worked with nixes for 30 years
>     beginning with AT&T pre-SRV on DEC-1170s and have seen a lot of
>     changes since those days, most of them for the better.  But, while
>     functionality is mandatory, convenience is always appreciated and
>     can help avoid costly mistakes and save time.  As I stated in my
>     original post, I KNOW and appreciate that all of you are working
>     hard on things that matter far more than this trivial item.  But
>     it is a major convenience and clarity issue for me and I am sure
>     it will be for others as well.  It is only rational that one
>     should be able to expect to mount by LABEL and unmount by LABEL,
>     but that doesn't work, and a major part of the reason that doesn't
>     work is that btrfs does not conform to the pattern of just about
>     every other file system on the planet in regards to how it treats
>     mount points.  And this is not even to mention all the other
>     issues involved like a large number of utilities that have no way
>     of knowing that a given partition is mounted, which would also be
>     resolved by virtual mount points since many if not most of those
>     utilities understand and process virtual volume identifiers.
>
>     Please just do me a favor and think about this a bit before you
>     just write it off.
>
>     - George
>
>
>
>
>     On 05/19/2013 04:04 AM, Martin wrote:
>
>         On 10/05/13 15:03, George Mitchell wrote:
>
>             One the things that is frustrating me the most at this
>             point from a user
>             perspective ...  The current method of simply using a
>             random member device or a LABEL or a UUID is just not
>             working well for
>             me.  Having a well thought out virtual device
>             infrastructure would...
>
>         Sorry, I'm a bit lost for your comments...
>
>         What is your use case and what are you hoping/expecting to see?
>
>
>         I've been following development of btrfs for a while and I'm
>         looking
>         forward to use it to efficiently replace some of the very useful
>         features of LVM2, drbd, and md-raid that I'm using at present...
>
>         OK, so the way of managing all that is going to be a little
>         different.
>
>         How would you want that?
>
>
>         Regards,
>         Martin
>
>         --
>         To unsubscribe from this list: send the line "unsubscribe
>         linux-btrfs" in
>         the body of a message to majordomo@vger.kernel.org
>         <mailto:majordomo@vger.kernel.org>
>         More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
>     On 05/19/2013 04:15 AM, Roman Mamedov wrote:
>
>         On Fri, 10 May 2013 07:03:38 -0700
>         George Mitchell <george@chinilu.com
>         <mailto:george@chinilu.com>> wrote:
>
>             One the things that is frustrating me the most at this
>             point from a user
>             perspective regarding btrfs is the current lack of virtual
>             devices to
>             describe volumes and subvolumes.
>
>          From a user perspective btrfs subvolumes have a lot in common
>         with just
>
>         regular directories aka folders, and nothing in common with
>         (block)devices.
>         "Describing them with virtual devices" does not seem to make a
>         whole lot of
>         sense.
>
>     --
>     To unsubscribe from this list: send the line "unsubscribe
>     linux-btrfs" in
>     the body of a message to majordomo@vger.kernel.org
>     <mailto:majordomo@vger.kernel.org>
>     More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
>
> -- 
> Michael Johnson - MJ

Re: Virtual Device Support

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]

On Fri, 10 May 2013 07:03:38 -0700
George Mitchell <george@chinilu.com> wrote:

> One the things that is frustrating me the most at this point from a user 
> perspective regarding btrfs is the current lack of virtual devices to 
> describe volumes and subvolumes.

From a user perspective btrfs subvolumes have a lot in common with just
regular directories aka folders, and nothing in common with (block)devices.
"Describing them with virtual devices" does not seem to make a whole lot of
sense.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Virtual Device Support

[Chris Murphy]

On May 19, 2013, at 5:15 AM, Roman Mamedov <rm@romanrm.ru> wrote:

> On Fri, 10 May 2013 07:03:38 -0700
> George Mitchell <george@chinilu.com> wrote:
> 
>> One the things that is frustrating me the most at this point from a user 
>> perspective regarding btrfs is the current lack of virtual devices to 
>> describe volumes and subvolumes.
> 
> From a user perspective btrfs subvolumes have a lot in common with just
> regular directories aka folders, and nothing in common with (block)devices.
> "Describing them with virtual devices" does not seem to make a whole lot of
> sense.

It's not possible to mount regular directories with other file systems. In some ways the btrfs subvolume behaves like a folder. In other ways it acts like a device. If you stat the mount point for btrfs subvolumes, you get a unique device ID for each.

It seems inconsistent that mount and unmount allows a /dev/ designation, but only mount honors label and UUID.

I can mount with mount /dev/disk/by-uuid/xxxxxxx, but if I use umount /dev/disk/by-uuid/xxxxxxxx I get a bogus error:

umount: /dev/disk/by-uuid/xxxxxxxxxx: not mounted

mount /dev/disk/by-uuid will autocomplete/list uuids in that directory, umount will not. So just from a consistency standpoint that seems broken.


Chris Murphy

Re: Virtual Device Support

[Chris Murphy]

On May 19, 2013, at 12:18 PM, Chris Murphy <lists@colorremedies.com> wrote:

> 
> It's not possible to mount regular directories with other file systems. In some ways the btrfs subvolume behaves like a folder. In other ways it acts like a device. If you stat the mount point for btrfs subvolumes, you get a unique device ID for each.

Also a possible use case for btrfs subvolumes as virtual devices, if this isn't already possible or reliable, is in place of LV's for virtual machines. It's very convenient to point a VM to local LVM storage, and have it use an LV which can easily be created, located, and destroyed, and presented to the VM as a single block device. A btrfs subvolume isn't a block device of course, but if it can be pointed to in this fashion it would make it as useful in this scenario as LVs.

Otherwise I have to create another btrfs file system on a qcow2 image which then resides on btrfs. So it's double btrfs which performance wise I think is an undesirable hit.

Chris Murphy

Re: Virtual Device Support

[Duncan]

Chris Murphy posted on Sun, 19 May 2013 12:18:19 -0600 as excerpted:


> On May 19, 2013, at 5:15 AM, Roman Mamedov <rm@romanrm.ru> wrote:
> 
>> From a user perspective btrfs subvolumes have a lot in common with just
>> regular directories aka folders, and nothing in common with
>> (block)devices.
>> "Describing them with virtual devices" does not seem to make a whole
>> lot of sense.
> 
> It's not possible to mount regular directories with other file systems.

Actually, it /is/ possible, using bind-mounts, etc.  These even work at 
the individual file level, and I use a few that way here, for mounting 
usable device files over an otherwise nodev mounted filesystem (used for 
a named/bind chroot, bind-mounted and then remounted nodev,noexec, etc.).

But yes, bind-mounts are an exception to the general rule.  However, 
they're an exception that does make your above claim questionable, at 
least.  btrfs subvolumes are another such exception.

> In some ways the btrfs subvolume behaves like a folder. In other ways it
> acts like a device. If you stat the mount point for btrfs subvolumes,
> you get a unique device ID for each.

Agreed.

> It seems inconsistent that mount and unmount allows a /dev/ designation,
> but only mount honors label and UUID.

Yes.  I had tested btrfs a year ago and decided to wait so haven't been 
active here for 8 months or so, but am now getting back into btrfs as my 
requirements are different now, and as I'm reading the list, I've seen 
this frustrating inconsistency complained about more than once.  I'm 
about to setup a new btrfs system here once again, so don't yet know if 
it'll affect me personally, but given that I routinely use labels in 
fstab, it certainly could, depending on how the umounts are handled.  But 
at least I have a heads-up on the issue and can thus work around it 
should I need to.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Virtual Device Support

[George Mitchell]

Duncan,  The problem affects btrfs volumes that span multiple drive.  If 
you are using btrfs on a single drive that works just fine.  But in a 
multidrive situation, sometimes it works (when umount guesses the right 
device name) and sometimes it fails (when umount guesses the wrong 
device name). Have fun!   - George

On 05/20/2013 06:08 PM, Duncan wrote:
> Chris Murphy posted on Sun, 19 May 2013 12:18:19 -0600 as excerpted:
>
>
>> On May 19, 2013, at 5:15 AM, Roman Mamedov <rm@romanrm.ru> wrote:
>>
>>>  From a user perspective btrfs subvolumes have a lot in common with just
>>> regular directories aka folders, and nothing in common with
>>> (block)devices.
>>> "Describing them with virtual devices" does not seem to make a whole
>>> lot of sense.
>> It's not possible to mount regular directories with other file systems.
> Actually, it /is/ possible, using bind-mounts, etc.  These even work at
> the individual file level, and I use a few that way here, for mounting
> usable device files over an otherwise nodev mounted filesystem (used for
> a named/bind chroot, bind-mounted and then remounted nodev,noexec, etc.).
>
> But yes, bind-mounts are an exception to the general rule.  However,
> they're an exception that does make your above claim questionable, at
> least.  btrfs subvolumes are another such exception.
>
>> In some ways the btrfs subvolume behaves like a folder. In other ways it
>> acts like a device. If you stat the mount point for btrfs subvolumes,
>> you get a unique device ID for each.
> Agreed.
>
>> It seems inconsistent that mount and unmount allows a /dev/ designation,
>> but only mount honors label and UUID.
> Yes.  I had tested btrfs a year ago and decided to wait so haven't been
> active here for 8 months or so, but am now getting back into btrfs as my
> requirements are different now, and as I'm reading the list, I've seen
> this frustrating inconsistency complained about more than once.  I'm
> about to setup a new btrfs system here once again, so don't yet know if
> it'll affect me personally, but given that I routinely use labels in
> fstab, it certainly could, depending on how the umounts are handled.  But
> at least I have a heads-up on the issue and can thus work around it
> should I need to.
>

Re: Virtual Device Support

[Duncan]

George Mitchell posted on Mon, 20 May 2013 19:17:39 -0700 as excerpted:

> Duncan,  The problem affects btrfs volumes that span multiple drive.  If
> you are using btrfs on a single drive that works just fine.  But in a
> multidrive situation, sometimes it works (when umount guesses the right
> device name) and sometimes it fails (when umount guesses the wrong
> device name). Have fun!   - George

Thanks.  I had inferred that but glad to have it confirmed.  My planned 
usage will indeed be multi-device as I'm going to be using btrfs raid1, 
but now that you mention the multi-device trigger explicitly, I 
understand why my tests a year ago didn't run into the problem, as those 
tests were single-device.  (I had inferred the multi-device connection, 
but hadn't made the additional connection between that and my earlier 
tests being single-device until you explicitly mentioned the multi-dev 
trigger, explaining...)


My personal btrfs history is a bit complicated.  Until about a year ago, 
I was running md-raid-1 with four aging 300g "spinning rust" drives
(having earlier experimented with raid-6 and lvm, then ultimately 
deciding raid1 was better for me, but the raid6 experiment was the reason 
for the four).  Because they /were/ aging, I wasn't particularly 
comfortable with the thought of reducing redundancy down to two, and was 
rather dismayed to find out that btrfs' so-called raid1 support wasn't 
raid-1 in the traditional sense of N-way mirroring at all, but was 
limited to two-way-mirroring regardless of the number of physical 
devices.  So I researched but didn't deploy at that time, waiting for the 
raid-6 support (followed by n-way-mirroring) that was then planned for 
the next kernel cycle or two, but as we know now, took several kernel 
cycles to hit, with N-way-mirroring still not available.

Then I ran into hardware issues that turned out to be bad caps on my 8-
year-old mobo (tho it was dual-socket first-gen opteron, which I had 
upgraded to top-of-its-line dual-core Opteron 290s, thus four cores @ 2.8 
GHz, with 8 gigs RAM, so it wasn't as performance-dated as its age might 
otherwise imply).

However, those issues first appeared as storage errors, so knowing the 
drives were aging, I thought it was them, and I replaced, upgrading for 
the first time to 2.5" from the older 3.5" drives.  However, that was 
still the middle of the recession and I had severe budget issues, so I 
decided to try my luck with a single drive (temporarily) in place of the 
four I had been running, fortunately enough, since it didn't turn out to 
be the drive at all.  But not knowing that at the time and having the 
opportunity now with a single drive that was new and thus should be 
reliable, I decided to try btrfs on it, which was where my actual 
deployment and testing time came from.

But meanwhile, the hardware problems continued, and I found the old 
reiserfs was MUCH more stable under those conditions than btrfs, which 
would often be missing entire directory trees after a crash, where 
reiserfs would be missing maybe the last copied file or two.  (I was 
still trying to copy from the old raid1 to the new single device hard 
drive, so was copying entire trees over... all on severely unstable 
motherboard hardware that was frequently timing out SATA commands... 
sometimes to resume after a couple minutes, sometimes not.  Of course at 
the time I was still blaming it on the old drives since that was what I 
was copying from.  It only became apparent that they weren't the issue 
once I had enough on the new drive to try running from it with the others 
detached.)

So under those conditions I decided btrfs was definitely *NOT* 
appropriate, and returned to the long stable reiserfs, which I've 
continued using until now.

Meanwhile, after getting enough on the new drive to run from it, I 
realized it wasn't the old drives that were the problem, and eventually 
realized that I had bulging and even a few popped caps.  So mobo upgrade 
time it was.

Fortunately, bad financial situation tho I was in, I still had good 
enough credit to get an account approved at the local Fry's Electronics, 
and I was able to purchase mobo/cpu/memory/graphics, all upgraded at 
once, mostly even on year-same-as-cash terms.  And fortunately, my 
financial situation has improved dramatically since then, so that's long 
ago paid off and I'm on to new things.

One of those new things is a pair of SSDs, hence my renewed interest in 
btrfs, since reiserfs' journaling is definitely NOT SSD friendly.

Meanwhile, btrfs having a year to mature since my initial tests, and now 
being on new and actually WORKING (!!) hardware, and as I said, needing 
an alternative to the reiserfs I've been using for over a decade now, I'm 
again interested in btrfs.

This time around, I'm still interested in the checksumming and data 
integrity features and in particular the multi-device angle, but the N-
way-mirroring isn't as important now since I'm looking at only two 
devices anyway, and even that's a step up from my current single device.  
In addition, new this time, I'm now interested in the SSD features.

Meanwhile, given the SSDs and the impressive benchmarks I've seen for it, 
I've also looked into f2fs, but decided it's WAAAYYY too immature to 
seriously consider at this point.

I considered ext4 as well, but the ext* filesystems and I have just never 
gotten along very well, one of the reasons I've stuck with reiserfs for 
so long.  Here, time and time again reiserfs has been proven impressively 
stable, especially since ordered-journaling became the default back in 
2.6.16 or whatever, even thru hardware issues like the above that should 
challenge ANY filesystem.  Personally, I think part of the problem with 
ext* is that too many kernel devs think they know it well enough to mess 
with it, when they don't.  I know people who suffered data loss during 
the ext3 defaulting to writeback (as opposed to ordered) journaling 
period, for instance, while I was safe on reiserfs, which most kernel 
devs are scared to touch, so it just continues working as it always has.  
(Personally I'd had enough of a demonstration of the evils of writeback 
journaling with pre-ordered reiserfs to seriously doubt the sanity of the 
ext3 switch to writeback journaling by default from the moment I heard 
about it, and of course had I been running ext3, I'd have switched back 
to ordered immediately, but the person I know that lost data due to that 
wasn't following kernel development closely enough to know why, he just 
knew it was happening.  When I saw he was running ext3 and his kernel 
version, I asked about the journaling option, and sure enough, when he 
switched to ordered, the problem disappeared!)

Also, having run reiserfs for years, I see no reason for a modern fs to 
not have tail-packing (tho f2fs arguably has an excuse due to the 
technology it's targeting).  ext* has never had that, while btrfs has 
equivalent.  I guess it's possible with ext4 and kernel 3.6+ now, but 
there's still serious limitations to it on ext4.

So I'll try btrfs raid1 mode on the SSDs, and get the checksumming and 
data integrity features that the raid1 mode provides. =:^)  Plus I'll 
have btrfs' tail-packing, and the comfort of knowing that the same Chris 
Mason who worked on reiserfs for so long and introduced reiserfs ordered 
journal mode to mainline (IDR whether he originally authored it or not) 
is lead on btrfs now. =:^)

And hopefully, now that btrfs raid5/6 is in, in a few cycles the N-way 
mirrored code will make it as well, and I can add a third SSD and 
rebalance to it.  That time gap should ensure the third device is 
sufficiently separate in lot and possibly model number as well, so I 
don't have all my raid-1 data eggs in the same device lot basket. =:^)

Meanwhile, the old "spinning rust" drive, still with demonstrated 
reliable reiserfs, can continue to serve as a backup, both to the new SSD 
technology and same-lot raid-1 devices, and to the still not fully stable 
btrfs I'll be trying to run on them in "production" mode.

So yes, it's likely I'll have to devise workarounds to the multi-device 
btrfs label= umount problem myself.  I guess I'll find out in a day or 
two, as I actually deploy and my experiment progresses. =:^)

And yes, have fun I expect I will. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Virtual Device Support

[George Mitchell]

On 05/20/2013 08:59 PM, Duncan wrote:
> Then I ran into hardware issues that turned out to be bad caps on my 
> 8- year-old mobo (tho it was dual-socket first-gen opteron, which I 
> had upgraded to top-of-its-line dual-core Opteron 290s, thus four 
> cores @ 2.8 GHz, with 8 gigs RAM, so it wasn't as performance-dated as 
> its age might otherwise imply). However, those issues first appeared 
> as storage errors, so knowing the drives were aging, I thought it was 
> them, and I replaced, upgrading for the first time to 2.5" from the 
> older 3.5" drives.
And that is actually an advantage of using btrfs.  Because ... btrfs, 
unlike conventional RAID is very compatible and simple to use with 
SMART.  That way, by watching your system logs or journal, you are 
immediately aware of any hard drive issues.
> But meanwhile, the hardware problems continued, and I found the old 
> reiserfs was MUCH more stable under those conditions than btrfs, which 
> would often be missing entire directory trees after a crash, where 
> reiserfs would be missing maybe the last copied file or two. (I was 
> still trying to copy from the old raid1 to the new single device hard 
> drive, so was copying entire trees over... all on severely unstable 
> motherboard hardware that was frequently timing out SATA commands... 
> sometimes to resume after a couple minutes, sometimes not. Of course 
> at the time I was still blaming it on the old drives since that was 
> what I was copying from. It only became apparent that they weren't the 
> issue once I had enough on the new drive to try running from it with 
> the others detached.)
What can I say?  Hans Reiser, aside from his horrendous character flaws, 
is a software genius.  After a terrifying experience of having a hard 
drive fail and no backups, Hans Reiser and his helpers came to my aid 
and in short order I had all my data back intact.  I found that pretty 
impressive.  After that for a number of years I continued to use 
Reiserfs in a software RAID 1 configuration.  I never ever had any 
complaints about Reiserfs.  I really liked it and still do.  I really 
wanted to see Reiser4 see the light of day, but after Mr Reiser's 
incarceration, that has become more and more unlikely.  So, in 2009 I 
switch to hardware RAID 1 on a pair of old 3ware cards.  But file system 
RAID as offered by btrfs and zfs have the distinct advantage of not 
having to face those terrifying syncs after loss of a drive.  So now, as 
of April 2013, I am 100% on btrfs (well, almost).  I use 5 500GB Seagate 
2½" drives with all but boot filesystem (boot filesystem is spread 
across two Seagate 2½" 80GB drives formatted btrfs)  spread across them 
in RAID1 configuration. Additionally 100% of the content of those drives 
get backed up to another 500GB Seagate drive (formatted JFS) via cron 
every 3hrs AND to a 4TB Seagate drive (formatted btrfs raw, sans 
partitioning) via anacron daily, weekly, monthly, etc.  I also have a 
stripped down maintenance OS running on ext4 that I use to backup the 
main OS itself daily as a manual operation.  I am running this on a 
SuperMicro board with a dual core Pentium processor.  Not a particularly 
muscular system, but a VERY stable one.  I also use a small UPS unit 
which I think is a very good idea if you are doing write caching with 
btrfs, or any other filesystem for that matter. I use a CyberPower unit 
and CyberPower has a very nifty UPS tool for Linux which does 
auto-shutdown on low battery without intervention.

All in all I am very happy with this arrangement so far.  It has worked 
flawlessly in most respects and I really, really like btrfs. The two 
bugs in the ointment for me right now are 1) the infamous boot bug on 
kernel 3.8 whereby one gets repeated boot failures do to "open_ctree 
failure" and can only boot the system successfully after multiple 
attempts.  And 2) the btrfs incompatibilities like the umount issue 
which I script my way around by doing `mount -l` which provides both the 
LABEL and the mount point on the same line then `grep` out the label, 
extract the mount point with a `cut` and feed the verified mount point 
to `umount`.  That all works very sweet even if it is a bit clutzy.  And 
I am well enough aware of the dark side of all of this to steer clear of 
fatal moves with partitioning tools etc that don't have a clue that a 
mounted btrfs partition is ... a mounted btrfs partition even if it is 
NOT the mount point.  My only real concern at this point is the boot 
issue.  Overall, I have a lot less problems now than I did with hardware 
RAID and have not the least desire to go back.  btrfs could be better, 
but its still head and shoulders over any other approach I have tried, 
but that does not include zfs, which I have also heard very good things 
about.

Re: Virtual Device Support ("N-way mirror code")

[Martin]

Duncan,

Thanks for quiet a historical summary.

Yep, ReiserFS has stood the test of time very well and I'm still using
and abusing it still on various servers all the way from something like
a decade ago!

More recently I've been putting newer systems on ext4 mainly to take
advantage of extents for large files on all disk types, and also
deferred allocation to hopefully reduce wear on SSDs.

Meanwhile, I've seen no need to change the ReiserFS on the existing
systems, even for the multi-Terabyte backups. The near unlimited file
linking is beautiful for creating in effect incremental backups spanning
years!

All on raid1 or raid5, and all remarkably robust.

Enough waffle! :-)


On 21/05/13 04:59, Duncan wrote:
> And hopefully, now that btrfs raid5/6 is in, in a few cycles the N-way 
> mirrored code will make it as well

I too am waiting for the "N-way mirrored code" for example to have 3
copies of data/metadata across 4 physical disks.

When might that hit? Or is there a stable patch that can be added into
kernel 3.8.13?


Regards,
Martin

Re: Virtual Device Support ("N-way mirror code")

[Martin Steigerwald]

Am Dienstag, 21. Mai 2013, 13:19:31 schrieb Martin:
> Yep, ReiserFS has stood the test of time very well and I'm still using
> and abusing it still on various servers all the way from something like
> a decade ago!

Very interesting. I only used it for a short time and it worked.

But co-workers lost several ReiserFS filesystems completely.

Well, if you search for the terms corrupt and your favorite filesystem, you 
will always find hits.

Anyway, I won´t use ReiserFS 3 today for several reasons:

1) It is not yet actively developed anymore, but more in a maintenance. I know 
for some that might be a reason to use it, but I think this basically 
increases the risk of breakages instead of reducing it. That said, I didn´t 
hear of any, and also JFS is in maintenance, but appears to work as well.

2) As to my knowledge a fsck.reiserfs cannot tell the filesystem I check and 
possible ReiserFS3 filesystems in virtual machine image files on it appart, 
happily mixing them together in a huge big mess.

3) As to my knowledge mount times of large partitions can be quite long with 
ReiserFS 3.

That said, I am using BTRFS on my main laptop even for /home now after having 
used it on several other machines for more than a year. Despite from that 
wierd scrub issue that I "fixed" by redoing the filesystem, rsync backup 
appeared t be okay, I am ready to trust my data to BTRFS. Also my backup 
harddisks are BTRFS.

I like BTRFS for some reasons, two that immediately come to my mind:

1) It can prove to me that the data is intact. I find this rather valuable.

2) Due to snapshots I know have well snapshots for my backup. And even on SSD 
for my /home. I am not yet creating those in an automated way, but well I do 
use them.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Virtual Device Support ("N-way mirror code")

[George Mitchell]

On 05/23/2013 09:08 AM, Martin Steigerwald wrote:
> 3) As to my knowledge mount times of large partitions can be quite 
> long with ReiserFS 3.

That may well be, but I certainly wouldn't consider btrfs mount times 
"fast" in such cases.

[root@localhost ghmitch]# time mount LABEL=BACKUP /backup

real    0m18.133s
user    0m0.000s
sys     0m0.190s
[root@localhost ghmitch]#

Re: Virtual Device Support ("N-way mirror code")

[Martin Steigerwald]

Am Donnerstag, 23. Mai 2013, 18:41:11 schrieb George Mitchell:
> On 05/23/2013 09:08 AM, Martin Steigerwald wrote:
> > 3) As to my knowledge mount times of large partitions can be quite
> > long with ReiserFS 3.
> 
> That may well be, but I certainly wouldn't consider btrfs mount times
> "fast" in such cases.
> 
> [root@localhost ghmitch]# time mount LABEL=BACKUP /backup
> 
> real    0m18.133s
> user    0m0.000s
> sys     0m0.190s
> [root@localhost ghmitch]#

Well yes, I saw some a bit longer mount times for my 2 TB backup disk with 
quite some snapshots as well already.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Virtual Device Support ("N-way mirror code")

[Duncan]

Martin Steigerwald posted on Thu, 23 May 2013 18:08:35 +0200 as excerpted:

> Am Dienstag, 21. Mai 2013, 13:19:31 schrieb Martin:
>> Yep, ReiserFS has stood the test of time very well and I'm still using
>> and abusing it still on various servers all the way from something like
>> a decade ago!
> 
> Very interesting. I only used it for a short time and it worked.
> 
> But co-workers lost several ReiserFS filesystems completely.

Do you know if that was before (Chris's) ordered-mode patches?  I never 
lost complete FSs (even once when my AC went out and the disks overheated 
resulting in a head-crash... the disks worked again once temps returned 
to normal, tho I did lose some data where the platters were very likely 
physically damaged due to the head-crash), but before the ordered-mode 
patches, I did lose data a number of times due to simple loss of power or 
system lockup.

So I learned to keep tested backups, tho they weren't always current.  
But a few hours of repeated work on a not-current backup copy, sure beats 
days of recreation from scratch, and when that head-crash happened, I was 
glad I had 'em!

But after data=ordered, the only data loss was due to that physical head 
crash, and even then it was whatever files were in the physically damaged 
area, not the entire filesystem.  And with that and various other 
hardware problems I've had including wonky memory in various forms and a 
mobo that popped a few capacitors in the sata bus area that I was still 
able to run if I kept it cold enough (when it started timing out 
operations I'd know it was too warm), that notably, btrfs could NOT 
handle... yes, I have some pretty deep respect for reiserfs, now.  It 
survived hardware issues that nobody could /sanely/ expect /any/ 
filesystem to survive, yet reiserfs did.

> Well, if you search for the terms corrupt and your favorite filesystem,
> you will always find hits.

True.

> Anyway, I won´t use ReiserFS 3 today for several reasons:
> 
> 1) It is not yet actively developed anymore, but more in a maintenance.
> I know for some that might be a reason to use it, but I think this
> basically increases the risk of breakages instead of reducing it. That
> said, I didn´t hear of any, and also JFS is in maintenance, but appears
> to work as well.

Well, there's a difference between being left to rot, which I'm beginning 
to be concerned might be where reiserfs is likely to be headed at this 
point, and simply mature and feature complete, so the only real 
maintenance needed is to keep up with the ever-changing kernel api, which 
people changing that api must do for anything in-kernel.  That's where 
reiserfs has been for some time, now.

And as I believe I mentioned earlier, being simply mature is definitely 
better than the ext4, and for some time altho not so much any longer, 
ext3, were, where every kernel hacker and their brother seems to consider 
it worth changing, including even Linus himself when he took the ext3 
writeback-by-default commit, which lasted for several kernel cycles, that 
proved a bad decision for data safety in a number of cases I know about 
personally.

You mentioned jfs is in a similar position, what I'd call mature but 
maintained.

FWIW, I'd consider XFS to be a pretty good example of a somewhat more 
assertive middle ground, still being actively developed, new features 
being added, but generally by a core xfs group, not everybody and their 
brother, and arguably with a cautious enough approach that (as reiserfs 
with ordered mode, extended attributes and quotas being added after it 
was declared feature complete and more or less abandoned by its previous 
upstream developer), it's actually much more stable and broadly usable 
these days than it was in its heyday, when it had a reputation of being 
great for UPS-backed enterprise systems, but for eating data on joe-user 
line-only-powered systems should that line power disappear.

> 2) As to my knowledge a fsck.reiserfs cannot tell the filesystem I check
> and possible ReiserFS3 filesystems in virtual machine image files on it
> appart, happily mixing them together in a huge big mess.

AFAIK that's limited to the --rebuild-tree option, which comes with 
pretty scary warnings and requires not just a y/n, but actually a fully 
typed out yes, to proceed.  So it's not something that people should 
normally run -- they should be going to the backups if they have them 
before they run --rebuild-tree.  But it's there for those who didn't HAVE 
backups, and who are prepared to gamble with SOME data loss in ordered to 
have the chance at SOME recovery.  And even then, the instructions in the 
warning say to ddrescue or the like to create a backup image before 
trying to recover, just in case.

However, yes, with those caveats AFAIK that's still an issue.  Had such 
usage been foreseen all those years ago, I'm sure the implementation 
would have been rather different.

> 3) As to my knowledge mount times of large partitions can be quite long
> with ReiserFS 3.

Hmmm... Never seen that except in the case where it's replaying a 
journal, and in that case the mount time is limited by the size of the 
journal to replay.

However, it's quite likely that I simply don't run large enough 
filesystems for it to be an issue here, since I tend to use partitions 
far more heavily than most, so I've never actually run a reiserfs of more 
than a few hundred GB, let alone the multiple TBs common on unpartitioned 
multi-TB disks today.

And multiple partitions will continue to be the case here with btrfs, as 
well.  I'll use snapshots for the convenience of rollback, but probably 
won't be using the general case subvolume support, preferring entirely 
separate partitions instead.

Because a strong multi-partition policy as saved my *** more than once!  
The case that really iron-clad it for me was actually before I switched 
to Linux, when I was still running MS servantware (see my sig for 
context).  I was beta-testing the IE4 previews, and MS changed the way it 
handled the IE cache index file for performance reasons, maintaining its 
absolute disk addresses in memory instead of grabbing the info from the 
disk each time.

Then some of the testers started having horrible cross-linked files 
problems with a number of them losing valuable data.  Turns out that (MS' 
own) defrag was moving the files out from under IE, which in IE4 was now 
the desktop shell so it was now running all the time, including when the 
defrag was running, and when IE later wrote back the index to the 
absolute disk addresses that the file had been at before, it was 
overwriting other files that the defragger had moved into that spot in 
the mean time.

Eventually, MS fixed the problem by simply marking the cache files as 
system, read-only, so the defragger wouldn't touch them.

But in the mean time, a bunch of people running the affected IE4 pre-
releases lost data!

However, all I lost was a few unimportant temp files, because I had 
Internet Temporary Files located on my TEMP partition, and the only files 
on that partition besides IE's cache were other temporary files -- no big 
deal if they got overwritten with IE-cache-index!

And actually, I never even bothered reconfiguring defrag to avoid the 
problem, even after the problem was known and before it was fixed.  The 
only files possibly affected were temporary anyway.  No big deal.

But that reinforced what before that had been simply gut instinct into a 
hard-clad rule that I continue to observe to this day, of course on 
freedomware Linux now -- THOU SHALT KEEP THY FILE DATA TYPES SEPARATE!  
That means separate partitions if not separate physical drives, not some 
weird new-fangled subvolume thing where if the filesystem metadata gets 
screwed up it's a potential loss of everything on all the subvolumes 
despite the subvolume separation.

And it has saved some trouble at least once and I believe at least twice 
on Linux as well (tho unlike that first MS experience that left such an 
impression and created that hard-clad rule, I don't remember much about 
the details of these, as they just weren't that big a deal anyway, since 
my existing partitioning policy prevented them from becoming one)... and 
that's just the stuff I *KNOW* about!

So other than for the convenience of snapshots (which as the wiki says do 
NOT replace backups), I have no plans for btrfs subvolumes at all.  From 
my perspective, either it's the same general type of data and simply 
keeping it in ordinary directory trees is separation enough, or it NEEDS 
its own separate partition; there's no namby-pamby subvolume middle 
ground to be had.

(FWIW, I have similar philosophical issues with LVM, tho I realize 
there's a LOT of people using it by default, since that's what a lot of 
distros install by default, basically everything on LVM.)

> That said, I am using BTRFS on my main laptop even for /home now after
> having used it on several other machines for more than a year. Despite
> from that wierd scrub issue that I "fixed" by redoing the filesystem,
> rsync backup appeared t be okay, I am ready to trust my data to BTRFS.
> Also my backup harddisks are BTRFS.
> 
> I like BTRFS for some reasons, two that immediately come to my mind:
> 
> 1) It can prove to me that the data is intact. I find this rather
> valuable.

Indeed.

> 2) Due to snapshots I know have well snapshots for my backup. And even
> on SSD for my /home. I am not yet creating those in an automated way,
> but well I do use them.

As I already mentioned the warning on the wiki, do be aware of the 
limitations of snapshots.  They're NOT the same as separate backups.  I 
believe you know that already and just didn't mention it, but I'm worried 
about others who might come across your comment.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Virtual Device Support ("N-way mirror code")

[Martin Steigerwald]

Am Freitag, 24. Mai 2013, 06:13:04 schrieb Duncan:
> > 2) Due to snapshots I know have well snapshots for my backup. And even
> > on SSD for my /home. I am not yet creating those in an automated way,
> > but well I do use them.
> 
> As I already mentioned the warning on the wiki, do be aware of the 
> limitations of snapshots.  They're NOT the same as separate backups.  I 
> believe you know that already and just didn't mention it, but I'm worried 
> about others who might come across your comment.

Well, a snapshot is not a backup. Just like a RAID is also not a backup.

:)

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Virtual Device Support

[Chris Murphy]

On May 20, 2013, at 7:08 PM, Duncan <1i5t5.duncan@cox.net> wrote:

> Chris Murphy posted on Sun, 19 May 2013 12:18:19 -0600 as excerpted:
> 
>> It seems inconsistent that mount and unmount allows a /dev/ designation,
>> but only mount honors label and UUID.
> 
> Yes.  

I'm going to contradict myself and point out that mount with label or UUID is made unambiguous via either the default subvolume being mounted, or the -o subvol= option being specified. The volume label and UUID doesn't apply to umount because it's an ambiguous command. You'd have to umount a mountpoint, or possibly a subvolume specific UUID.


Chris Murphy

Re: Virtual Device Support

[Martin]

On 21/05/13 04:37, Chris Murphy wrote:
> 
> On May 20, 2013, at 7:08 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> 
>> Chris Murphy posted on Sun, 19 May 2013 12:18:19 -0600 as
>> excerpted:
>> 
>>> It seems inconsistent that mount and unmount allows a /dev/
>>> designation, but only mount honors label and UUID.
>> 
>> Yes.
> 
> I'm going to contradict myself and point out that mount with label or
> UUID is made unambiguous via either the default subvolume being
> mounted, or the -o subvol= option being specified. The volume label
> and UUID doesn't apply to umount because it's an ambiguous command.
> You'd have to umount a mountpoint, or possibly a subvolume specific
> UUID.


I'll admit that I prefer working with filesystem labels.


This is getting rather semantic... From "man umount", this is what
umount intends:

#####
umount [-dflnrv] {dir|device}...

The  umount  command  detaches the file system(s) mentioned from the
file hierarchy.  A file system is specified by giving the directory
where it has been mounted.  Giving the special device on which the file
system lives may also work, but is obsolete, mainly because it will fail
in case this device was mounted on more than one directory.
#####


I guess the ideas of labels and UUID and multiple devices came out a few
years later?... For btrfs, umount needs to operate on the default subvol
but with the means for also specifying a specific subvol if needed.

One hook for btrfs to extend what/how 'umount' operates might be to
perhaps extend what can be done with a "/sbin/(u?)mount.btrfs" 'helper'?


Regards,
Martin

Re: Virtual Device Support

[Chris Murphy]

On May 21, 2013, at 8:06 AM, Martin <m_btrfs@ml1.co.uk> wrote:

> On 21/05/13 04:37, Chris Murphy wrote:
>> 
>> I'm going to contradict myself and point out that mount with label or
>> UUID is made unambiguous via either the default subvolume being
>> mounted, or the -o subvol= option being specified. The volume label
>> and UUID doesn't apply to umount because it's an ambiguous command.
>> You'd have to umount a mountpoint, or possibly a subvolume specific
>> UUID.
> 
> I guess the ideas of labels and UUID and multiple devices came out a few
> years later?... For btrfs, umount needs to operate on the default subvol
> but with the means for also specifying a specific subvol if needed.

Yeah and I think specifying -o for umount isn't what devs are interested in doing, and I think that's understandable. But I'm pretty sure there are btrfs subvolume UUIDs now? So even if umount doesn't support -o, it should still support /dev/disk/by-uuid/xxxxxxxxxxx just as it does /dev/sda1 or whatever.


Chris Murphy