Ideas for btrfs-convert fix(or rework)

Ideas for btrfs-convert fix(or rework)

[Qu Wenruo]

Hi all,

Someone may already knows, I'm recently trying to fix(or rework) the old 
btrfs-convert, to allow it to support separate data and meta chunks.

[[FIX IDEA]]
The overall idea is quite simple and straight forward:
Separate meta/sys chunk at very *beginning*, then insert data chunks to 
covert ext* used space.
Then use normal btrfs routines to do remaining work.

No custom chunk/extent allocation nor wired system chunk mapping.

1. Create sb,sys and meta chunks at make_btrfs() time
    Created meta and sys chunk will be ensured not covering ext* used
    space
    (Done)

2. Open the temp fs, and insert data chunks to cover all ext* used space
    We will only need to ensure sys/meta chunks is large enough to
    containing created data chunk and block group items.
    Data chunks are 1:1 mapped on disk.
    (Done)

3. Create ext* subvolume and image
    Now do what ever we want, as all data chunks are allocated, we can
    do normal chunk allocation and won't overwrite ext* data.
    Much like old codes. But need some modification to co-operate with
    above codes
    (WIP)

4. Copy inodes
    Should be much like old codes, but it's done after ext2 image, it
    should be easier than old one.
    (Not started yet)

5. Relocate extents covered by first 1M and backup super blocks
    Hard part, maybe I can do it at step 1 and 2 to make things easy.
    (Not started yet)

6. Change may_rollback() check
    As now ext2 image is only mapped for used space, old may_rollback()
    check is not suitable here.
    (Not started yet)

Or ascii art version:
1. Original Ext2/3/4 used space:
0		256M		512M		768M		1G
|///|  |//|	|//|	|/////////////|

2. New sb + meta + sys chunks at make_btrfs() time:
					|SB|<-Sys->|<-Meta->|

3. Insert data chunks
|<-Data--->|	|<-----Data------------>|SB|<-Sys->|<-Meta->|

4. Create subvolume and ext* image
|<-Data--->|NewD|<-----Data------------>|SB|<-Sys->|<-Meta->|NewM|
NewD = new data chunk, NewM = new meta chunk


5. Relocate SB and 1M leading reserve
Not sure yet.
Maybe to relocate the whole chunk like:
|SB| |SB|  |NewD|<-----Data------------>|  |<-Sys->|<-Meta->|NewM|RecD|
RecD = relocated data chunk

Or at Step 2 we already reserved some holes for SB
|SB|D|SB|DD|NewD|<-----Data------------>|  |<-Sys->|<-Meta->|NewM|
D = data chunk


Existing codes can be checked at my github:
https://github.com/adam900710/btrfs-progs/tree/convert_chunk_profile

[[PROBLEMS]]
1. Too many patches
    Even after step 2, I have already 20 patches and near 1500 lines.
    Although most of them are codes to make old make_btrfs() more
    modularized to fit the need of flex of new make_btrfs_v2().

    I'm not sure when the number of patches will stop growing, and what
    makes thing worse is, maybe until the last few patches all the codes
    will be really executed, so I can't submit them half way...

2. superblock reserve is d*mning hard, not matter at what timing.
    Until I need to codes for superblock reserve, the codes are quite
    easy to write.
    But when it comes to sb reserve, I'm going to get crazy...

    I'll either add tons of codes to the simple chunk allocation to
    avoid using SB space and keep all chunks are large enough.
    Or I need to implement user-space whole chunk relocation codes.

    I miss the old days when sb_bytenr can be specified by user...
    Then my work should be quite close to end...

3. Too aggressive rework
    I suppose this will not be a fix, but a total rework of the whole
    btrfs-convert program.
    Maybe only less than 30% old codes will stay, most of them will be
    rewritten.

    No matter how I think the new code will be superior than old one,
    I'm pretty sure there will be tons of regression or new bugs for
    such huge rework.

    But without such work, btrfs-convert will always be a mess and no
    real support for balance.

So any ideas or advice is welcomed for this rework.

Thanks,
Qu

Re: Ideas for btrfs-convert fix(or rework)

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

On Tue, 10 Nov 2015 14:27:41 +0800
Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:

>     But without such work, btrfs-convert will always be a mess and no
>     real support for balance.

I wonder, what happened to the current btrfs-convert?

Perhaps a couple of years ago I converted a 7TB and ~70% full Ext4 filesystem
into Btrfs. At first the result showed up to have about 3TB of Metadata
chunks, but several iterations of balance reclassified this as Data (and thus
made it practical to also enable Metadata DUP). Everything went flawlessly and
with no data loss whatsoever.

But as we know recently there were reports that it now causes corruption or
does not work. So I wonder what happened that broke it, and is there really no
simpler fix? Even if requiring the user to balance to get rid of the bloated
Metadata like I had to.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ideas for btrfs-convert fix(or rework)

[Qu Wenruo]

Roman Mamedov wrote on 2015/11/10 12:55 +0500:
> On Tue, 10 Nov 2015 14:27:41 +0800
> Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>>      But without such work, btrfs-convert will always be a mess and no
>>      real support for balance.
>
> I wonder, what happened to the current btrfs-convert?
>
> Perhaps a couple of years ago I converted a 7TB and ~70% full Ext4 filesystem
> into Btrfs. At first the result showed up to have about 3TB of Metadata
> chunks, but several iterations of balance reclassified this as Data (and thus
> made it practical to also enable Metadata DUP). Everything went flawlessly and
> with no data loss whatsoever.

Old btrfs-convert works well, since it's using mixed block group and 
using it correctly.
It's quite hard to mess chunk types up if they can be both data and 
metadata.

But in that case, kernel doesn't support to convert mixed block group 
into separate ones.

Which means you will only get mixed block groups, and no method to get 
normal separate data/meta block groups.

So if things are correct, the btrfs you converted should still be in 
mixed-bg mode.
A recent 'btrfs fi df' command should show things like:
Data+Metadata, DUP: total=512.00MiB, used=68.23MiB

>
> But as we know recently there were reports that it now causes corruption or
> does not work. So I wonder what happened that broke it, and is there really no
> simpler fix? Even if requiring the user to balance to get rid of the bloated
> Metadata like I had to.
>

Yes, current btrfs-convert has problem on marking chunk meta or data.
Resulting some data extents may lay in metadata chunk, and vice verse.
At least it won't pass latest btrfsck.

In that incorrect chunk layout, I'm not surprised any thing go wrong.


For alternative fix, David mentioned one, but that's going to land in 
kernel, to support convert mixed block group.
And then we can still use the old mixed-bg method for btrfs-convert.
But I'm not sure if it

Thanks,
Qu

Re: Ideas for btrfs-convert fix(or rework)

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 3202 bytes --]

On Tue, 10 Nov 2015 16:16:26 +0800
Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:

> So if things are correct, the btrfs you converted should still be in 
> mixed-bg mode.
> A recent 'btrfs fi df' command should show things like:
> Data+Metadata, DUP: total=512.00MiB, used=68.23MiB

Actually not quite. After all the rebalancing, I currently have:

Data, single: total=8.37TiB, used=7.50TiB
System, DUP: total=32.00MiB, used=592.00KiB
Metadata, DUP: total=8.00GiB, used=6.42GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

> > But as we know recently there were reports that it now causes corruption or
> > does not work. So I wonder what happened that broke it, and is there really no
> > simpler fix? Even if requiring the user to balance to get rid of the bloated
> > Metadata like I had to.
> >
> 
> Yes, current btrfs-convert has problem on marking chunk meta or data.
> Resulting some data extents may lay in metadata chunk, and vice verse.
> At least it won't pass latest btrfsck.
> 
> In that incorrect chunk layout, I'm not surprised any thing go wrong.

Made an experiment:

# dd if=/dev/zero bs=1M count=1 seek=100000 of=100G
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0023896 s, 439 MB/s

# losetup /dev/loop7 100G

# mkfs.ext4 /dev/loop7 

// Success

# mount /dev/loop7 /mnt/tmp1/

// Copy some files there

# df -h /mnt/tmp1/
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop7       96G  2.6G   89G   3% /mnt/tmp1

# umount /mnt/tmp1/

# btrfs-convert -d /dev/loop7 
creating btrfs metadata.
creating ext2fs image file.
cleaning up system chunk.
conversion complete.

# mount /dev/loop7 /mnt/tmp1/

# btrfs fi df /mnt/tmp1/
Data, single: total=65.00GiB, used=1.32GiB
System, single: total=32.00MiB, used=12.00KiB
Metadata, single: total=32.63GiB, used=2.91GiB
GlobalReserve, single: total=92.00MiB, used=0.00B

# btrfs sub del /mnt/tmp1/ext2_saved
Transaction commit: none (default)
Delete subvolume '/mnt/tmp1/ext2_saved'

# btrfs fi df /mnt/tmp1/
Data, single: total=10.00GiB, used=120.12MiB
System, single: total=32.00MiB, used=4.00KiB
Metadata, single: total=4.00GiB, used=2.37GiB
GlobalReserve, single: total=52.00MiB, used=0.00B

// It has 2.6 GB of files, and most of them are now accounted as "Metadata".

# btrfs fi balance start -m /mnt/tmp1/
Done, had to relocate 5 out of 7 chunks

// They are now reclassified as data:

# btrfs fi df /mnt/tmp1/
Data, single: total=10.00GiB, used=2.49GiB
System, single: total=32.00MiB, used=4.00KiB
Metadata, single: total=1.00GiB, used=468.00KiB
GlobalReserve, single: total=4.00MiB, used=0.00B

// Unmount, mount, checked all files integrity with 'cfv', all files are
present and there is no corruption.

So this behaves just like I remember it to, after conversion at first a large
part of the data will show up as Metadata, but a balance fixes that. And most
importantly no corruption issues. Maybe it's just because this is only a small
test FS, but my production one earlier went through the same process without
problems as well.

Using btrfs-tools v3.17 and kernel 3.18.21

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ideas for btrfs-convert fix(or rework)

[Qu Wenruo]

Roman Mamedov wrote on 2015/11/10 14:08 +0500:
> On Tue, 10 Nov 2015 16:16:26 +0800
> Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>> So if things are correct, the btrfs you converted should still be in
>> mixed-bg mode.
>> A recent 'btrfs fi df' command should show things like:
>> Data+Metadata, DUP: total=512.00MiB, used=68.23MiB
>
> Actually not quite. After all the rebalancing, I currently have:
>
> Data, single: total=8.37TiB, used=7.50TiB
> System, DUP: total=32.00MiB, used=592.00KiB
> Metadata, DUP: total=8.00GiB, used=6.42GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
>>> But as we know recently there were reports that it now causes corruption or
>>> does not work. So I wonder what happened that broke it, and is there really no
>>> simpler fix? Even if requiring the user to balance to get rid of the bloated
>>> Metadata like I had to.
>>>
>>
>> Yes, current btrfs-convert has problem on marking chunk meta or data.
>> Resulting some data extents may lay in metadata chunk, and vice verse.
>> At least it won't pass latest btrfsck.
>>
>> In that incorrect chunk layout, I'm not surprised any thing go wrong.
>
> Made an experiment:
>
> # dd if=/dev/zero bs=1M count=1 seek=100000 of=100G
> 1+0 records in
> 1+0 records out
> 1048576 bytes (1.0 MB) copied, 0.0023896 s, 439 MB/s
>
> # losetup /dev/loop7 100G
>
> # mkfs.ext4 /dev/loop7
>
> // Success
>
> # mount /dev/loop7 /mnt/tmp1/
>
> // Copy some files there
>
> # df -h /mnt/tmp1/
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/loop7       96G  2.6G   89G   3% /mnt/tmp1
>
> # umount /mnt/tmp1/
>
> # btrfs-convert -d /dev/loop7
> creating btrfs metadata.
> creating ext2fs image file.
> cleaning up system chunk.
> conversion complete.

Here call 'btrfsck' from *latest* btrfs-progs.
You'll see tons of error, showing problems including:
1) stripes cross map boundary
    This will lead to kernel BUG_ON() when scrubbing.

2) Data extents in metadata chunks
    The problem causing the metadata accounting goes crazy.

>
> # mount /dev/loop7 /mnt/tmp1/
>
> # btrfs fi df /mnt/tmp1/
> Data, single: total=65.00GiB, used=1.32GiB
> System, single: total=32.00MiB, used=12.00KiB
> Metadata, single: total=32.63GiB, used=2.91GiB
> GlobalReserve, single: total=92.00MiB, used=0.00B
>

And your btrfs-convert is already broken, with wrong chunk allocation.

> # btrfs sub del /mnt/tmp1/ext2_saved
> Transaction commit: none (default)
> Delete subvolume '/mnt/tmp1/ext2_saved'
>
> # btrfs fi df /mnt/tmp1/
> Data, single: total=10.00GiB, used=120.12MiB
> System, single: total=32.00MiB, used=4.00KiB
> Metadata, single: total=4.00GiB, used=2.37GiB
> GlobalReserve, single: total=52.00MiB, used=0.00B
>
> // It has 2.6 GB of files, and most of them are now accounted as "Metadata".
>
> # btrfs fi balance start -m /mnt/tmp1/
> Done, had to relocate 5 out of 7 chunks
>
> // They are now reclassified as data:
>
> # btrfs fi df /mnt/tmp1/
> Data, single: total=10.00GiB, used=2.49GiB
> System, single: total=32.00MiB, used=4.00KiB
> Metadata, single: total=1.00GiB, used=468.00KiB
> GlobalReserve, single: total=4.00MiB, used=0.00B
>
> // Unmount, mount, checked all files integrity with 'cfv', all files are
> present and there is no corruption.
>
> So this behaves just like I remember it to, after conversion at first a large
> part of the data will show up as Metadata, but a balance fixes that. And most
> importantly no corruption issues. Maybe it's just because this is only a small
> test FS, but my production one earlier went through the same process without
> problems as well.

Yes, some problem can be fixed by such balance, as after balance, data 
and metadata will be relocated to correct new chunks.

But there may be a lot of hidden bugs here.
And we can't ignore such malfunction just because it's OK under some 
cases, or btrfs won't really become a production ready filesystem.

Thanks,
Qu

>
> Using btrfs-tools v3.17 and kernel 3.18.21
>

Re: Ideas for btrfs-convert fix(or rework)

[Duncan]

Qu Wenruo posted on Tue, 10 Nov 2015 17:18:02 +0800 as excerpted:

> Yes, some problem can be fixed by such balance, as after balance, data
> and metadata will be relocated to correct new chunks.
> 
> But there may be a lot of hidden bugs here.
> And we can't ignore such malfunction just because it's OK under some
> cases, or btrfs won't really become a production ready filesystem.

Very good point in general, but remember the context we're talking about 
here, btrfs-convert.

Convert is used once, if then, and a lot of generally considered stable 
filesystems get along perfectly fine without direct convert-from-ext* 
tools at all, telling people to either use their ext* as backups and 
restore/copy to their newly created filesystems of whatever new type, or 
backup the data they want to save from the old filesystem, then mkfs them 
directly to the new filesystem type, again, restoring/copying the backed 
up data back from the backups.

And after all, people using anything /other/ than ext* are going to have 
to do that anyway, unless even /more/ work is invested to deal with all 
the /other/ filesystems... all for something that's optionally used once 
and must work well if used, then forgotten about.

So an ext* specific convert tool, while definitely nice to have, remains 
entirely optional, and as such, unlike more critical actual filesystem 
features that will continue to be used as long as the filesystem exists, 
isn't really critical to btrfs becoming a production ready filesystem at 
all, because at some point, if it's still buggy it can simply be thrown 
away.

So a buggy convert tool, being optional, doesn't really affect the 
production readiness of the filesystem as a whole, at all.

Meanwhile, people really considering production readiness, at least in 
the enterprise setting, tend to be a pretty conservative bunch, and I'd 
argue that conservative admins will be unlikely to really trust such a 
conversion tool in any case, preferring to restore from existing 
backups.  I certainly know that /I'd/ have my doubts about trusting my 
data to a convert tool, and would /much/ rather do the copy over to the 
new filesystems thing, since at least that way, if anything goes wrong I 
know I still have the unchanged old copy that was perfectly fine to use 
the day/hour before still there and just as perfectly fine to use again.  
That's not something I'd be confident saying about the if-anything-goes-
wrong behavior of a convert-in-place tool!

So again, yes, convert is nice to have even if I'd never fully trust them 
myself, but it really doesn't affect the production readiness of the 
filesystem itself.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Ideas for btrfs-convert fix(or rework)

[Vytautas D]

[ resending as it didnt get through. ]

I got different opinion. btrfs-convert is something that
brought me to btrfs. While there are other bugs to fix, someone
dedicating time to fix btrfs-convert is of high interest to me.
Sending right message to community, might make some rolling distros to
trust it and experiment with it.

Thanks Qu and everyone for great work,
-Vytas

On Tue, Nov 10, 2015 at 10:31 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> Qu Wenruo posted on Tue, 10 Nov 2015 17:18:02 +0800 as excerpted:
>
>> Yes, some problem can be fixed by such balance, as after balance, data
>> and metadata will be relocated to correct new chunks.
>>
>> But there may be a lot of hidden bugs here.
>> And we can't ignore such malfunction just because it's OK under some
>> cases, or btrfs won't really become a production ready filesystem.
>
> Very good point in general, but remember the context we're talking about
> here, btrfs-convert.
>
> Convert is used once, if then, and a lot of generally considered stable
> filesystems get along perfectly fine without direct convert-from-ext*
> tools at all, telling people to either use their ext* as backups and
> restore/copy to their newly created filesystems of whatever new type, or
> backup the data they want to save from the old filesystem, then mkfs them
> directly to the new filesystem type, again, restoring/copying the backed
> up data back from the backups.
>
> And after all, people using anything /other/ than ext* are going to have
> to do that anyway, unless even /more/ work is invested to deal with all
> the /other/ filesystems... all for something that's optionally used once
> and must work well if used, then forgotten about.
>
> So an ext* specific convert tool, while definitely nice to have, remains
> entirely optional, and as such, unlike more critical actual filesystem
> features that will continue to be used as long as the filesystem exists,
> isn't really critical to btrfs becoming a production ready filesystem at
> all, because at some point, if it's still buggy it can simply be thrown
> away.
>
> So a buggy convert tool, being optional, doesn't really affect the
> production readiness of the filesystem as a whole, at all.
>
> Meanwhile, people really considering production readiness, at least in
> the enterprise setting, tend to be a pretty conservative bunch, and I'd
> argue that conservative admins will be unlikely to really trust such a
> conversion tool in any case, preferring to restore from existing
> backups.  I certainly know that /I'd/ have my doubts about trusting my
> data to a convert tool, and would /much/ rather do the copy over to the
> new filesystems thing, since at least that way, if anything goes wrong I
> know I still have the unchanged old copy that was perfectly fine to use
> the day/hour before still there and just as perfectly fine to use again.
> That's not something I'd be confident saying about the if-anything-goes-
> wrong behavior of a convert-in-place tool!
>
> So again, yes, convert is nice to have even if I'd never fully trust them
> myself, but it really doesn't affect the production readiness of the
> filesystem itself.
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ideas for btrfs-convert fix(or rework)

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

On 2015-11-12 05:23, Vytautas D wrote:
> [ resending as it didnt get through. ]
>
> I got different opinion. btrfs-convert is something that
> brought me to btrfs. While there are other bugs to fix, someone
> dedicating time to fix btrfs-convert is of high interest to me.
> Sending right message to community, might make some rolling distros to
> trust it and experiment with it.
While I can kind of understand this, what has been said about it being 
something you run exactly once per-system per-filesystem is not untrue 
for almost all users.  It has almost no value for people setting up new 
systems (because they can just use BTRFS directly), and I personally see 
very limited value for the two biggest rolling-release distros that I 
know of (Arch and Gentoo), because the very fact that you installed a 
system with either one means that you are fully capable of backing up 
your data, and reprovisioning the system using BTRFS instead of whatever 
filesystem you are already using (and that will _always_ be safer than 
in-place conversion).  I also see little value for non rolling-release 
systems other than Ubuntu (and possibly Mint, I don't know if they have 
reliably working in-place release upgrades or not) because system 
upgrades with those are usually full re-installs (and by the time that 
btrfs-convert has been around long enough that they consider it safe to 
package, BTRFS will be a lot more prevalent and will likely have been 
the default FS for them for a long time).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Ideas for btrfs-convert fix(or rework)

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1245 bytes --]

On Thu, 12 Nov 2015 08:27:49 -0500
Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:

> know of (Arch and Gentoo), because the very fact that you installed a 
> system with either one means that you are fully capable of backing up 
> your data, and reprovisioning the system using BTRFS instead of whatever 
> filesystem you are already using

Uhm, what?... How does the fact that I use one particular distro or another,
reflect on my ability to (easily) back up and restore my 10 TB storage array?
"You should maintain backups anyway so you could easily restore" -- yeah I do,
and they are in a remote location over a relatively slow link.

I chose Ext4 instead of the other options which seemed more attractive
(primarily XFS, but also JFS) for this array back when creating it a long time
ago, when Btrfs was still in its much earlier days than now, **SPECIFICALLY**
with the intent to use btrfs-convert "as soon as Btrfs somewhat matures". So
as time went, it did, I acted on this plan, and everything went just perfectly.

Why are people so quick to project "If I don't need this, nobody likely does".
Btrfs-convert is an amazing feature and it will be extremely sad if it ever
goes away.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ideas for btrfs-convert fix(or rework)

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 3256 bytes --]

On 2015-11-12 09:09, Roman Mamedov wrote:
> On Thu, 12 Nov 2015 08:27:49 -0500
> Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
>
>> know of (Arch and Gentoo), because the very fact that you installed a
>> system with either one means that you are fully capable of backing up
>> your data, and reprovisioning the system using BTRFS instead of whatever
>> filesystem you are already using
>
> Uhm, what?... How does the fact that I use one particular distro or another,
> reflect on my ability to (easily) back up and restore my 10 TB storage array?
> "You should maintain backups anyway so you could easily restore" -- yeah I do,
> and they are in a remote location over a relatively slow link.
I did not mean to imply that you had the ability to do it quickly, just 
the ability to do it.  I specifically stated 'very little' instead of 
'none' to account for cases like yours.  However, the statement is still 
accurate because Arch and Gentoo both have very low-level user 
involvement in the install process (no GUI, user manually provisions the 
storage and handles installing the base system).  If you are capable of 
installing such a system (and as such, capable of correctly using stuff 
like parted or fdisk without breaking anything), you are absolutely 
capable of reprovisioning such a system (how long that would take is a 
completely different matter).
> I chose Ext4 instead of the other options which seemed more attractive
> (primarily XFS, but also JFS) for this array back when creating it a long time
> ago, when Btrfs was still in its much earlier days than now, **SPECIFICALLY**
> with the intent to use btrfs-convert "as soon as Btrfs somewhat matures". So
> as time went, it did, I acted on this plan, and everything went just perfectly.
And you got somewhat lucky in that case (and actually did proper 
research, sadly most people don't seem to do anywhere near enough 
research when it comes to stuff like this).  Doing an in-place 
conversion of a filesystem (or an OS, or anything else for that matter) 
is _always_ going to be inherently more risk than just properly 
reprovisioning (take a look at the absolute insanity that is the 'free' 
Windows 10 upgrade for an example of this on the OS side).  Admittedly, 
compared to other software that does this (the only other program that I 
know of to do this is fstransform), btrfs-convert is relatively safe, 
but it is not possible for it to be any safer than just reprovisioning.
> Why are people so quick to project "If I don't need this, nobody likely does".
> Btrfs-convert is an amazing feature and it will be extremely sad if it ever
> goes away.
And as time goes on, it will be used less and less, and it's _always_ 
going to be something that 99% of users who actually use it (if you're 
doing any kind of RAID, it makes more sense from a data safety 
perspective to just reprovision so you can use BTRFS's native raid 
functionality) run exactly once per filesystem per system (once you've 
converted existing systems, you don't need it unless you were using dump 
for backups).  I'm not arguing that it should just go away, I'm trying 
to argue that it shouldn't be a development priority if it works correctly.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Ideas for btrfs-convert fix(or rework)

[Duncan]

Austin S Hemmelgarn posted on Thu, 12 Nov 2015 09:38:03 -0500 as
excerpted:

> I'm not arguing that [btrfs-convert] should just go away, I'm trying to
> argue that it shouldn't be a development priority if it works correctly.

Agreed.

If you go back to my reply that started this subthread, the statement I 
took issue with was that btrfs couldn't be accepted as stable unless 
btrfs-convert was stable.

I simply argued that wasn't the case, since a convert tool is entirely 
optional, and many fully stable filesystems do entirely without one.  
Btrfs as a filesystem doesn't /have/ to have this tool, it's optional, so 
btrfs as a filesystem should be able to stabilize just fine, just as have 
all the filesystems without a convert tool, no matter the condition of 
this tool.

Of course having a convert tool and having it working is indeed nice to 
have, and I said that, but the condition of a convert tool, or even 
having one at all, really shouldn't hold up stabilization of the 
filesystem as a whole, just as it hasn't held up stabilization of all the 
other filesystems that don't have such a tool.

/Because/ btrfs-convert is a very nice to have tool, yes, time spent 
working on it is worthwhile time.  But nobody, least of all me, was 
arguing otherwise.  I was just saying that the overall stability of btrfs 
as a filesystem shouldn't be affected by the stability of this tool, as 
it's optional, and many other accepted as stable filesystems don't have 
such tools, so just because btrfs does, if the tool doesn't happen to be 
stable, should have nothing to do with whether the filesystem itself is 
stable or not.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Ideas for btrfs-convert fix(or rework)

[David Sterba]

On Tue, Nov 10, 2015 at 02:27:41PM +0800, Qu Wenruo wrote:
> [[FIX IDEA]]
> 1. Too many patches

Not percieved as a problem as long as the patches are well separated and
reviewable. If you prefer reasonably many preparatory and trivial
patches, then be it.

> 2. superblock reserve is d*mning hard, not matter at what timing.
>     Until I need to codes for superblock reserve, the codes are quite
>     easy to write.
>     But when it comes to sb reserve, I'm going to get crazy...

What do you mean by the 'sb reserve'?

>     I'll either add tons of codes to the simple chunk allocation to
>     avoid using SB space and keep all chunks are large enough.
>     Or I need to implement user-space whole chunk relocation codes.
> 
>     I miss the old days when sb_bytenr can be specified by user...
>     Then my work should be quite close to end...
> 
> 3. Too aggressive rework
>     I suppose this will not be a fix, but a total rework of the whole
>     btrfs-convert program.
>     Maybe only less than 30% old codes will stay, most of them will be
>     rewritten.

I suppose that the rewrite would also touch generic code, so as far as
it's cleanups and other refactoring I won't object investing the time to
that.

>     No matter how I think the new code will be superior than old one,
>     I'm pretty sure there will be tons of regression or new bugs for
>     such huge rework.
> 
>     But without such work, btrfs-convert will always be a mess and no
>     real support for balance.
> 
> So any ideas or advice is welcomed for this rework.

In general, I welcome the efforts to make convert better. The in-place
conversion is a unique feature and a nice to have. Several people
mentioned that.  Next, we want to add support more filesystems, eg.
reiserfs code is ready and needs configure checks and minor updates to
convert.c.  Basically any filesystem that provides a library API similar
to what ext2 does can be added. Convert code in a good shape would save
us trouble.

Expecting the regressions, the test set for the convertors needs to be
enhanced, the current state is less then ideal. Possibly we can use the
images from e2fsprogs testsuite.

Re: Ideas for btrfs-convert fix(or rework)

[Qu Wenruo]

David Sterba wrote on 2015/11/16 18:46 +0100:
> On Tue, Nov 10, 2015 at 02:27:41PM +0800, Qu Wenruo wrote:
>> [[FIX IDEA]]
>> 1. Too many patches
>
> Not percieved as a problem as long as the patches are well separated and
> reviewable. If you prefer reasonably many preparatory and trivial
> patches, then be it.
>
>> 2. superblock reserve is d*mning hard, not matter at what timing.
>>      Until I need to codes for superblock reserve, the codes are quite
>>      easy to write.
>>      But when it comes to sb reserve, I'm going to get crazy...
>
> What do you mean by the 'sb reserve'?

Superblock reserved space.
And the first 1Mb reserved space.

IIRC, normal btrfs and convert btrfs doesn't reserve super block space well.
For normal btrfs, it's still possible for device extent covering 
btrfs_sb_offset(), not to mention the converted one, which only uses one sb.

But the good news is, although it needs extra codes(about 5 patches and 
over 500 lines), it shouldn't be a big problem as I'll reserve these 
space at early time.

>
>>      I'll either add tons of codes to the simple chunk allocation to
>>      avoid using SB space and keep all chunks are large enough.
>>      Or I need to implement user-space whole chunk relocation codes.
>>
>>      I miss the old days when sb_bytenr can be specified by user...
>>      Then my work should be quite close to end...
>>
>> 3. Too aggressive rework
>>      I suppose this will not be a fix, but a total rework of the whole
>>      btrfs-convert program.
>>      Maybe only less than 30% old codes will stay, most of them will be
>>      rewritten.
>
> I suppose that the rewrite would also touch generic code, so as far as
> it's cleanups and other refactoring I won't object investing the time to
> that.

Not that much, most of the codes are rework of make_btrfs() to allow it 
to alloc metadata and system chunk at early time.
It's almost 10 patches and over 1000 lines to rework the huge 
make_btrfs() into structuralized make_btrfs_v2().

And will remove some btrfs-convert only hooks like custom_alloc_extent().

As now, btrfs-convert will has a meta and system chunk(not covering 
existing ext* data) after make_btrfs_v2().
And then data chunks will be inserted to cover all ext* data.
So no extra extent allocate routine is needed.

>
>>      No matter how I think the new code will be superior than old one,
>>      I'm pretty sure there will be tons of regression or new bugs for
>>      such huge rework.
>>
>>      But without such work, btrfs-convert will always be a mess and no
>>      real support for balance.
>>
>> So any ideas or advice is welcomed for this rework.
>
> In general, I welcome the efforts to make convert better. The in-place
> conversion is a unique feature and a nice to have. Several people
> mentioned that.  Next, we want to add support more filesystems, eg.
> reiserfs code is ready and needs configure checks and minor updates to
> convert.c.  Basically any filesystem that provides a library API similar
> to what ext2 does can be added. Convert code in a good shape would save
> us trouble.
>
> Expecting the regressions, the test set for the convertors needs to be
> enhanced, the current state is less then ideal. Possibly we can use the
> images from e2fsprogs testsuite.

Nice idea.

I'll try to enhance self test after btrfs-convert rework.

Thanks,
Qu

> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>