* Ideas for btrfs-convert fix(or rework) @ 2015-11-10 6:27 Qu Wenruo 2015-11-10 7:55 ` Roman Mamedov 2015-11-16 17:46 ` David Sterba 0 siblings, 2 replies; 13+ messages in thread From: Qu Wenruo @ 2015-11-10 6:27 UTC (permalink / raw) To: btrfs; +Cc: David Sterba Hi all, Someone may already knows, I'm recently trying to fix(or rework) the old btrfs-convert, to allow it to support separate data and meta chunks. [[FIX IDEA]] The overall idea is quite simple and straight forward: Separate meta/sys chunk at very *beginning*, then insert data chunks to covert ext* used space. Then use normal btrfs routines to do remaining work. No custom chunk/extent allocation nor wired system chunk mapping. 1. Create sb,sys and meta chunks at make_btrfs() time Created meta and sys chunk will be ensured not covering ext* used space (Done) 2. Open the temp fs, and insert data chunks to cover all ext* used space We will only need to ensure sys/meta chunks is large enough to containing created data chunk and block group items. Data chunks are 1:1 mapped on disk. (Done) 3. Create ext* subvolume and image Now do what ever we want, as all data chunks are allocated, we can do normal chunk allocation and won't overwrite ext* data. Much like old codes. But need some modification to co-operate with above codes (WIP) 4. Copy inodes Should be much like old codes, but it's done after ext2 image, it should be easier than old one. (Not started yet) 5. Relocate extents covered by first 1M and backup super blocks Hard part, maybe I can do it at step 1 and 2 to make things easy. (Not started yet) 6. Change may_rollback() check As now ext2 image is only mapped for used space, old may_rollback() check is not suitable here. (Not started yet) Or ascii art version: 1. Original Ext2/3/4 used space: 0 256M 512M 768M 1G |///| |//| |//| |/////////////| 2. New sb + meta + sys chunks at make_btrfs() time: |SB|<-Sys->|<-Meta->| 3. Insert data chunks |<-Data--->| |<-----Data------------>|SB|<-Sys->|<-Meta->| 4. Create subvolume and ext* image |<-Data--->|NewD|<-----Data------------>|SB|<-Sys->|<-Meta->|NewM| NewD = new data chunk, NewM = new meta chunk 5. Relocate SB and 1M leading reserve Not sure yet. Maybe to relocate the whole chunk like: |SB| |SB| |NewD|<-----Data------------>| |<-Sys->|<-Meta->|NewM|RecD| RecD = relocated data chunk Or at Step 2 we already reserved some holes for SB |SB|D|SB|DD|NewD|<-----Data------------>| |<-Sys->|<-Meta->|NewM| D = data chunk Existing codes can be checked at my github: https://github.com/adam900710/btrfs-progs/tree/convert_chunk_profile [[PROBLEMS]] 1. Too many patches Even after step 2, I have already 20 patches and near 1500 lines. Although most of them are codes to make old make_btrfs() more modularized to fit the need of flex of new make_btrfs_v2(). I'm not sure when the number of patches will stop growing, and what makes thing worse is, maybe until the last few patches all the codes will be really executed, so I can't submit them half way... 2. superblock reserve is d*mning hard, not matter at what timing. Until I need to codes for superblock reserve, the codes are quite easy to write. But when it comes to sb reserve, I'm going to get crazy... I'll either add tons of codes to the simple chunk allocation to avoid using SB space and keep all chunks are large enough. Or I need to implement user-space whole chunk relocation codes. I miss the old days when sb_bytenr can be specified by user... Then my work should be quite close to end... 3. Too aggressive rework I suppose this will not be a fix, but a total rework of the whole btrfs-convert program. Maybe only less than 30% old codes will stay, most of them will be rewritten. No matter how I think the new code will be superior than old one, I'm pretty sure there will be tons of regression or new bugs for such huge rework. But without such work, btrfs-convert will always be a mess and no real support for balance. So any ideas or advice is welcomed for this rework. Thanks, Qu ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-10 6:27 Ideas for btrfs-convert fix(or rework) Qu Wenruo @ 2015-11-10 7:55 ` Roman Mamedov 2015-11-10 8:16 ` Qu Wenruo 2015-11-16 17:46 ` David Sterba 1 sibling, 1 reply; 13+ messages in thread From: Roman Mamedov @ 2015-11-10 7:55 UTC (permalink / raw) To: Qu Wenruo; +Cc: btrfs, David Sterba [-- Attachment #1: Type: text/plain, Size: 879 bytes --] On Tue, 10 Nov 2015 14:27:41 +0800 Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > But without such work, btrfs-convert will always be a mess and no > real support for balance. I wonder, what happened to the current btrfs-convert? Perhaps a couple of years ago I converted a 7TB and ~70% full Ext4 filesystem into Btrfs. At first the result showed up to have about 3TB of Metadata chunks, but several iterations of balance reclassified this as Data (and thus made it practical to also enable Metadata DUP). Everything went flawlessly and with no data loss whatsoever. But as we know recently there were reports that it now causes corruption or does not work. So I wonder what happened that broke it, and is there really no simpler fix? Even if requiring the user to balance to get rid of the bloated Metadata like I had to. -- With respect, Roman [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-10 7:55 ` Roman Mamedov @ 2015-11-10 8:16 ` Qu Wenruo 2015-11-10 9:08 ` Roman Mamedov 0 siblings, 1 reply; 13+ messages in thread From: Qu Wenruo @ 2015-11-10 8:16 UTC (permalink / raw) To: Roman Mamedov; +Cc: btrfs, David Sterba Roman Mamedov wrote on 2015/11/10 12:55 +0500: > On Tue, 10 Nov 2015 14:27:41 +0800 > Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > >> But without such work, btrfs-convert will always be a mess and no >> real support for balance. > > I wonder, what happened to the current btrfs-convert? > > Perhaps a couple of years ago I converted a 7TB and ~70% full Ext4 filesystem > into Btrfs. At first the result showed up to have about 3TB of Metadata > chunks, but several iterations of balance reclassified this as Data (and thus > made it practical to also enable Metadata DUP). Everything went flawlessly and > with no data loss whatsoever. Old btrfs-convert works well, since it's using mixed block group and using it correctly. It's quite hard to mess chunk types up if they can be both data and metadata. But in that case, kernel doesn't support to convert mixed block group into separate ones. Which means you will only get mixed block groups, and no method to get normal separate data/meta block groups. So if things are correct, the btrfs you converted should still be in mixed-bg mode. A recent 'btrfs fi df' command should show things like: Data+Metadata, DUP: total=512.00MiB, used=68.23MiB > > But as we know recently there were reports that it now causes corruption or > does not work. So I wonder what happened that broke it, and is there really no > simpler fix? Even if requiring the user to balance to get rid of the bloated > Metadata like I had to. > Yes, current btrfs-convert has problem on marking chunk meta or data. Resulting some data extents may lay in metadata chunk, and vice verse. At least it won't pass latest btrfsck. In that incorrect chunk layout, I'm not surprised any thing go wrong. For alternative fix, David mentioned one, but that's going to land in kernel, to support convert mixed block group. And then we can still use the old mixed-bg method for btrfs-convert. But I'm not sure if it Thanks, Qu ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-10 8:16 ` Qu Wenruo @ 2015-11-10 9:08 ` Roman Mamedov 2015-11-10 9:18 ` Qu Wenruo 0 siblings, 1 reply; 13+ messages in thread From: Roman Mamedov @ 2015-11-10 9:08 UTC (permalink / raw) To: Qu Wenruo; +Cc: btrfs, David Sterba [-- Attachment #1: Type: text/plain, Size: 3202 bytes --] On Tue, 10 Nov 2015 16:16:26 +0800 Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > So if things are correct, the btrfs you converted should still be in > mixed-bg mode. > A recent 'btrfs fi df' command should show things like: > Data+Metadata, DUP: total=512.00MiB, used=68.23MiB Actually not quite. After all the rebalancing, I currently have: Data, single: total=8.37TiB, used=7.50TiB System, DUP: total=32.00MiB, used=592.00KiB Metadata, DUP: total=8.00GiB, used=6.42GiB GlobalReserve, single: total=512.00MiB, used=0.00B > > But as we know recently there were reports that it now causes corruption or > > does not work. So I wonder what happened that broke it, and is there really no > > simpler fix? Even if requiring the user to balance to get rid of the bloated > > Metadata like I had to. > > > > Yes, current btrfs-convert has problem on marking chunk meta or data. > Resulting some data extents may lay in metadata chunk, and vice verse. > At least it won't pass latest btrfsck. > > In that incorrect chunk layout, I'm not surprised any thing go wrong. Made an experiment: # dd if=/dev/zero bs=1M count=1 seek=100000 of=100G 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0023896 s, 439 MB/s # losetup /dev/loop7 100G # mkfs.ext4 /dev/loop7 // Success # mount /dev/loop7 /mnt/tmp1/ // Copy some files there # df -h /mnt/tmp1/ Filesystem Size Used Avail Use% Mounted on /dev/loop7 96G 2.6G 89G 3% /mnt/tmp1 # umount /mnt/tmp1/ # btrfs-convert -d /dev/loop7 creating btrfs metadata. creating ext2fs image file. cleaning up system chunk. conversion complete. # mount /dev/loop7 /mnt/tmp1/ # btrfs fi df /mnt/tmp1/ Data, single: total=65.00GiB, used=1.32GiB System, single: total=32.00MiB, used=12.00KiB Metadata, single: total=32.63GiB, used=2.91GiB GlobalReserve, single: total=92.00MiB, used=0.00B # btrfs sub del /mnt/tmp1/ext2_saved Transaction commit: none (default) Delete subvolume '/mnt/tmp1/ext2_saved' # btrfs fi df /mnt/tmp1/ Data, single: total=10.00GiB, used=120.12MiB System, single: total=32.00MiB, used=4.00KiB Metadata, single: total=4.00GiB, used=2.37GiB GlobalReserve, single: total=52.00MiB, used=0.00B // It has 2.6 GB of files, and most of them are now accounted as "Metadata". # btrfs fi balance start -m /mnt/tmp1/ Done, had to relocate 5 out of 7 chunks // They are now reclassified as data: # btrfs fi df /mnt/tmp1/ Data, single: total=10.00GiB, used=2.49GiB System, single: total=32.00MiB, used=4.00KiB Metadata, single: total=1.00GiB, used=468.00KiB GlobalReserve, single: total=4.00MiB, used=0.00B // Unmount, mount, checked all files integrity with 'cfv', all files are present and there is no corruption. So this behaves just like I remember it to, after conversion at first a large part of the data will show up as Metadata, but a balance fixes that. And most importantly no corruption issues. Maybe it's just because this is only a small test FS, but my production one earlier went through the same process without problems as well. Using btrfs-tools v3.17 and kernel 3.18.21 -- With respect, Roman [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-10 9:08 ` Roman Mamedov @ 2015-11-10 9:18 ` Qu Wenruo 2015-11-10 10:31 ` Duncan 0 siblings, 1 reply; 13+ messages in thread From: Qu Wenruo @ 2015-11-10 9:18 UTC (permalink / raw) To: Roman Mamedov; +Cc: btrfs, David Sterba Roman Mamedov wrote on 2015/11/10 14:08 +0500: > On Tue, 10 Nov 2015 16:16:26 +0800 > Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > >> So if things are correct, the btrfs you converted should still be in >> mixed-bg mode. >> A recent 'btrfs fi df' command should show things like: >> Data+Metadata, DUP: total=512.00MiB, used=68.23MiB > > Actually not quite. After all the rebalancing, I currently have: > > Data, single: total=8.37TiB, used=7.50TiB > System, DUP: total=32.00MiB, used=592.00KiB > Metadata, DUP: total=8.00GiB, used=6.42GiB > GlobalReserve, single: total=512.00MiB, used=0.00B > >>> But as we know recently there were reports that it now causes corruption or >>> does not work. So I wonder what happened that broke it, and is there really no >>> simpler fix? Even if requiring the user to balance to get rid of the bloated >>> Metadata like I had to. >>> >> >> Yes, current btrfs-convert has problem on marking chunk meta or data. >> Resulting some data extents may lay in metadata chunk, and vice verse. >> At least it won't pass latest btrfsck. >> >> In that incorrect chunk layout, I'm not surprised any thing go wrong. > > Made an experiment: > > # dd if=/dev/zero bs=1M count=1 seek=100000 of=100G > 1+0 records in > 1+0 records out > 1048576 bytes (1.0 MB) copied, 0.0023896 s, 439 MB/s > > # losetup /dev/loop7 100G > > # mkfs.ext4 /dev/loop7 > > // Success > > # mount /dev/loop7 /mnt/tmp1/ > > // Copy some files there > > # df -h /mnt/tmp1/ > Filesystem Size Used Avail Use% Mounted on > /dev/loop7 96G 2.6G 89G 3% /mnt/tmp1 > > # umount /mnt/tmp1/ > > # btrfs-convert -d /dev/loop7 > creating btrfs metadata. > creating ext2fs image file. > cleaning up system chunk. > conversion complete. Here call 'btrfsck' from *latest* btrfs-progs. You'll see tons of error, showing problems including: 1) stripes cross map boundary This will lead to kernel BUG_ON() when scrubbing. 2) Data extents in metadata chunks The problem causing the metadata accounting goes crazy. > > # mount /dev/loop7 /mnt/tmp1/ > > # btrfs fi df /mnt/tmp1/ > Data, single: total=65.00GiB, used=1.32GiB > System, single: total=32.00MiB, used=12.00KiB > Metadata, single: total=32.63GiB, used=2.91GiB > GlobalReserve, single: total=92.00MiB, used=0.00B > And your btrfs-convert is already broken, with wrong chunk allocation. > # btrfs sub del /mnt/tmp1/ext2_saved > Transaction commit: none (default) > Delete subvolume '/mnt/tmp1/ext2_saved' > > # btrfs fi df /mnt/tmp1/ > Data, single: total=10.00GiB, used=120.12MiB > System, single: total=32.00MiB, used=4.00KiB > Metadata, single: total=4.00GiB, used=2.37GiB > GlobalReserve, single: total=52.00MiB, used=0.00B > > // It has 2.6 GB of files, and most of them are now accounted as "Metadata". > > # btrfs fi balance start -m /mnt/tmp1/ > Done, had to relocate 5 out of 7 chunks > > // They are now reclassified as data: > > # btrfs fi df /mnt/tmp1/ > Data, single: total=10.00GiB, used=2.49GiB > System, single: total=32.00MiB, used=4.00KiB > Metadata, single: total=1.00GiB, used=468.00KiB > GlobalReserve, single: total=4.00MiB, used=0.00B > > // Unmount, mount, checked all files integrity with 'cfv', all files are > present and there is no corruption. > > So this behaves just like I remember it to, after conversion at first a large > part of the data will show up as Metadata, but a balance fixes that. And most > importantly no corruption issues. Maybe it's just because this is only a small > test FS, but my production one earlier went through the same process without > problems as well. Yes, some problem can be fixed by such balance, as after balance, data and metadata will be relocated to correct new chunks. But there may be a lot of hidden bugs here. And we can't ignore such malfunction just because it's OK under some cases, or btrfs won't really become a production ready filesystem. Thanks, Qu > > Using btrfs-tools v3.17 and kernel 3.18.21 > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-10 9:18 ` Qu Wenruo @ 2015-11-10 10:31 ` Duncan 2015-11-12 10:23 ` Vytautas D 0 siblings, 1 reply; 13+ messages in thread From: Duncan @ 2015-11-10 10:31 UTC (permalink / raw) To: linux-btrfs Qu Wenruo posted on Tue, 10 Nov 2015 17:18:02 +0800 as excerpted: > Yes, some problem can be fixed by such balance, as after balance, data > and metadata will be relocated to correct new chunks. > > But there may be a lot of hidden bugs here. > And we can't ignore such malfunction just because it's OK under some > cases, or btrfs won't really become a production ready filesystem. Very good point in general, but remember the context we're talking about here, btrfs-convert. Convert is used once, if then, and a lot of generally considered stable filesystems get along perfectly fine without direct convert-from-ext* tools at all, telling people to either use their ext* as backups and restore/copy to their newly created filesystems of whatever new type, or backup the data they want to save from the old filesystem, then mkfs them directly to the new filesystem type, again, restoring/copying the backed up data back from the backups. And after all, people using anything /other/ than ext* are going to have to do that anyway, unless even /more/ work is invested to deal with all the /other/ filesystems... all for something that's optionally used once and must work well if used, then forgotten about. So an ext* specific convert tool, while definitely nice to have, remains entirely optional, and as such, unlike more critical actual filesystem features that will continue to be used as long as the filesystem exists, isn't really critical to btrfs becoming a production ready filesystem at all, because at some point, if it's still buggy it can simply be thrown away. So a buggy convert tool, being optional, doesn't really affect the production readiness of the filesystem as a whole, at all. Meanwhile, people really considering production readiness, at least in the enterprise setting, tend to be a pretty conservative bunch, and I'd argue that conservative admins will be unlikely to really trust such a conversion tool in any case, preferring to restore from existing backups. I certainly know that /I'd/ have my doubts about trusting my data to a convert tool, and would /much/ rather do the copy over to the new filesystems thing, since at least that way, if anything goes wrong I know I still have the unchanged old copy that was perfectly fine to use the day/hour before still there and just as perfectly fine to use again. That's not something I'd be confident saying about the if-anything-goes- wrong behavior of a convert-in-place tool! So again, yes, convert is nice to have even if I'd never fully trust them myself, but it really doesn't affect the production readiness of the filesystem itself. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-10 10:31 ` Duncan @ 2015-11-12 10:23 ` Vytautas D 2015-11-12 13:27 ` Austin S Hemmelgarn 0 siblings, 1 reply; 13+ messages in thread From: Vytautas D @ 2015-11-12 10:23 UTC (permalink / raw) Cc: Btrfs BTRFS [ resending as it didnt get through. ] I got different opinion. btrfs-convert is something that brought me to btrfs. While there are other bugs to fix, someone dedicating time to fix btrfs-convert is of high interest to me. Sending right message to community, might make some rolling distros to trust it and experiment with it. Thanks Qu and everyone for great work, -Vytas On Tue, Nov 10, 2015 at 10:31 AM, Duncan <1i5t5.duncan@cox.net> wrote: > Qu Wenruo posted on Tue, 10 Nov 2015 17:18:02 +0800 as excerpted: > >> Yes, some problem can be fixed by such balance, as after balance, data >> and metadata will be relocated to correct new chunks. >> >> But there may be a lot of hidden bugs here. >> And we can't ignore such malfunction just because it's OK under some >> cases, or btrfs won't really become a production ready filesystem. > > Very good point in general, but remember the context we're talking about > here, btrfs-convert. > > Convert is used once, if then, and a lot of generally considered stable > filesystems get along perfectly fine without direct convert-from-ext* > tools at all, telling people to either use their ext* as backups and > restore/copy to their newly created filesystems of whatever new type, or > backup the data they want to save from the old filesystem, then mkfs them > directly to the new filesystem type, again, restoring/copying the backed > up data back from the backups. > > And after all, people using anything /other/ than ext* are going to have > to do that anyway, unless even /more/ work is invested to deal with all > the /other/ filesystems... all for something that's optionally used once > and must work well if used, then forgotten about. > > So an ext* specific convert tool, while definitely nice to have, remains > entirely optional, and as such, unlike more critical actual filesystem > features that will continue to be used as long as the filesystem exists, > isn't really critical to btrfs becoming a production ready filesystem at > all, because at some point, if it's still buggy it can simply be thrown > away. > > So a buggy convert tool, being optional, doesn't really affect the > production readiness of the filesystem as a whole, at all. > > Meanwhile, people really considering production readiness, at least in > the enterprise setting, tend to be a pretty conservative bunch, and I'd > argue that conservative admins will be unlikely to really trust such a > conversion tool in any case, preferring to restore from existing > backups. I certainly know that /I'd/ have my doubts about trusting my > data to a convert tool, and would /much/ rather do the copy over to the > new filesystems thing, since at least that way, if anything goes wrong I > know I still have the unchanged old copy that was perfectly fine to use > the day/hour before still there and just as perfectly fine to use again. > That's not something I'd be confident saying about the if-anything-goes- > wrong behavior of a convert-in-place tool! > > So again, yes, convert is nice to have even if I'd never fully trust them > myself, but it really doesn't affect the production readiness of the > filesystem itself. > > -- > Duncan - List replies preferred. No HTML msgs. > "Every nonfree program has a lord, a master -- > and if you use the program, he is your master." Richard Stallman > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-12 10:23 ` Vytautas D @ 2015-11-12 13:27 ` Austin S Hemmelgarn 2015-11-12 14:09 ` Roman Mamedov 0 siblings, 1 reply; 13+ messages in thread From: Austin S Hemmelgarn @ 2015-11-12 13:27 UTC (permalink / raw) To: Vytautas D, Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 1527 bytes --] On 2015-11-12 05:23, Vytautas D wrote: > [ resending as it didnt get through. ] > > I got different opinion. btrfs-convert is something that > brought me to btrfs. While there are other bugs to fix, someone > dedicating time to fix btrfs-convert is of high interest to me. > Sending right message to community, might make some rolling distros to > trust it and experiment with it. While I can kind of understand this, what has been said about it being something you run exactly once per-system per-filesystem is not untrue for almost all users. It has almost no value for people setting up new systems (because they can just use BTRFS directly), and I personally see very limited value for the two biggest rolling-release distros that I know of (Arch and Gentoo), because the very fact that you installed a system with either one means that you are fully capable of backing up your data, and reprovisioning the system using BTRFS instead of whatever filesystem you are already using (and that will _always_ be safer than in-place conversion). I also see little value for non rolling-release systems other than Ubuntu (and possibly Mint, I don't know if they have reliably working in-place release upgrades or not) because system upgrades with those are usually full re-installs (and by the time that btrfs-convert has been around long enough that they consider it safe to package, BTRFS will be a lot more prevalent and will likely have been the default FS for them for a long time). [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 3019 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-12 13:27 ` Austin S Hemmelgarn @ 2015-11-12 14:09 ` Roman Mamedov 2015-11-12 14:38 ` Austin S Hemmelgarn 0 siblings, 1 reply; 13+ messages in thread From: Roman Mamedov @ 2015-11-12 14:09 UTC (permalink / raw) To: Austin S Hemmelgarn; +Cc: Vytautas D, Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 1245 bytes --] On Thu, 12 Nov 2015 08:27:49 -0500 Austin S Hemmelgarn <ahferroin7@gmail.com> wrote: > know of (Arch and Gentoo), because the very fact that you installed a > system with either one means that you are fully capable of backing up > your data, and reprovisioning the system using BTRFS instead of whatever > filesystem you are already using Uhm, what?... How does the fact that I use one particular distro or another, reflect on my ability to (easily) back up and restore my 10 TB storage array? "You should maintain backups anyway so you could easily restore" -- yeah I do, and they are in a remote location over a relatively slow link. I chose Ext4 instead of the other options which seemed more attractive (primarily XFS, but also JFS) for this array back when creating it a long time ago, when Btrfs was still in its much earlier days than now, **SPECIFICALLY** with the intent to use btrfs-convert "as soon as Btrfs somewhat matures". So as time went, it did, I acted on this plan, and everything went just perfectly. Why are people so quick to project "If I don't need this, nobody likely does". Btrfs-convert is an amazing feature and it will be extremely sad if it ever goes away. -- With respect, Roman [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-12 14:09 ` Roman Mamedov @ 2015-11-12 14:38 ` Austin S Hemmelgarn 2015-11-13 6:41 ` Duncan 0 siblings, 1 reply; 13+ messages in thread From: Austin S Hemmelgarn @ 2015-11-12 14:38 UTC (permalink / raw) To: Roman Mamedov; +Cc: Vytautas D, Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 3256 bytes --] On 2015-11-12 09:09, Roman Mamedov wrote: > On Thu, 12 Nov 2015 08:27:49 -0500 > Austin S Hemmelgarn <ahferroin7@gmail.com> wrote: > >> know of (Arch and Gentoo), because the very fact that you installed a >> system with either one means that you are fully capable of backing up >> your data, and reprovisioning the system using BTRFS instead of whatever >> filesystem you are already using > > Uhm, what?... How does the fact that I use one particular distro or another, > reflect on my ability to (easily) back up and restore my 10 TB storage array? > "You should maintain backups anyway so you could easily restore" -- yeah I do, > and they are in a remote location over a relatively slow link. I did not mean to imply that you had the ability to do it quickly, just the ability to do it. I specifically stated 'very little' instead of 'none' to account for cases like yours. However, the statement is still accurate because Arch and Gentoo both have very low-level user involvement in the install process (no GUI, user manually provisions the storage and handles installing the base system). If you are capable of installing such a system (and as such, capable of correctly using stuff like parted or fdisk without breaking anything), you are absolutely capable of reprovisioning such a system (how long that would take is a completely different matter). > I chose Ext4 instead of the other options which seemed more attractive > (primarily XFS, but also JFS) for this array back when creating it a long time > ago, when Btrfs was still in its much earlier days than now, **SPECIFICALLY** > with the intent to use btrfs-convert "as soon as Btrfs somewhat matures". So > as time went, it did, I acted on this plan, and everything went just perfectly. And you got somewhat lucky in that case (and actually did proper research, sadly most people don't seem to do anywhere near enough research when it comes to stuff like this). Doing an in-place conversion of a filesystem (or an OS, or anything else for that matter) is _always_ going to be inherently more risk than just properly reprovisioning (take a look at the absolute insanity that is the 'free' Windows 10 upgrade for an example of this on the OS side). Admittedly, compared to other software that does this (the only other program that I know of to do this is fstransform), btrfs-convert is relatively safe, but it is not possible for it to be any safer than just reprovisioning. > Why are people so quick to project "If I don't need this, nobody likely does". > Btrfs-convert is an amazing feature and it will be extremely sad if it ever > goes away. And as time goes on, it will be used less and less, and it's _always_ going to be something that 99% of users who actually use it (if you're doing any kind of RAID, it makes more sense from a data safety perspective to just reprovision so you can use BTRFS's native raid functionality) run exactly once per filesystem per system (once you've converted existing systems, you don't need it unless you were using dump for backups). I'm not arguing that it should just go away, I'm trying to argue that it shouldn't be a development priority if it works correctly. [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 3019 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-12 14:38 ` Austin S Hemmelgarn @ 2015-11-13 6:41 ` Duncan 0 siblings, 0 replies; 13+ messages in thread From: Duncan @ 2015-11-13 6:41 UTC (permalink / raw) To: linux-btrfs Austin S Hemmelgarn posted on Thu, 12 Nov 2015 09:38:03 -0500 as excerpted: > I'm not arguing that [btrfs-convert] should just go away, I'm trying to > argue that it shouldn't be a development priority if it works correctly. Agreed. If you go back to my reply that started this subthread, the statement I took issue with was that btrfs couldn't be accepted as stable unless btrfs-convert was stable. I simply argued that wasn't the case, since a convert tool is entirely optional, and many fully stable filesystems do entirely without one. Btrfs as a filesystem doesn't /have/ to have this tool, it's optional, so btrfs as a filesystem should be able to stabilize just fine, just as have all the filesystems without a convert tool, no matter the condition of this tool. Of course having a convert tool and having it working is indeed nice to have, and I said that, but the condition of a convert tool, or even having one at all, really shouldn't hold up stabilization of the filesystem as a whole, just as it hasn't held up stabilization of all the other filesystems that don't have such a tool. /Because/ btrfs-convert is a very nice to have tool, yes, time spent working on it is worthwhile time. But nobody, least of all me, was arguing otherwise. I was just saying that the overall stability of btrfs as a filesystem shouldn't be affected by the stability of this tool, as it's optional, and many other accepted as stable filesystems don't have such tools, so just because btrfs does, if the tool doesn't happen to be stable, should have nothing to do with whether the filesystem itself is stable or not. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-10 6:27 Ideas for btrfs-convert fix(or rework) Qu Wenruo 2015-11-10 7:55 ` Roman Mamedov @ 2015-11-16 17:46 ` David Sterba 2015-11-17 0:42 ` Qu Wenruo 1 sibling, 1 reply; 13+ messages in thread From: David Sterba @ 2015-11-16 17:46 UTC (permalink / raw) To: Qu Wenruo; +Cc: btrfs, David Sterba On Tue, Nov 10, 2015 at 02:27:41PM +0800, Qu Wenruo wrote: > [[FIX IDEA]] > 1. Too many patches Not percieved as a problem as long as the patches are well separated and reviewable. If you prefer reasonably many preparatory and trivial patches, then be it. > 2. superblock reserve is d*mning hard, not matter at what timing. > Until I need to codes for superblock reserve, the codes are quite > easy to write. > But when it comes to sb reserve, I'm going to get crazy... What do you mean by the 'sb reserve'? > I'll either add tons of codes to the simple chunk allocation to > avoid using SB space and keep all chunks are large enough. > Or I need to implement user-space whole chunk relocation codes. > > I miss the old days when sb_bytenr can be specified by user... > Then my work should be quite close to end... > > 3. Too aggressive rework > I suppose this will not be a fix, but a total rework of the whole > btrfs-convert program. > Maybe only less than 30% old codes will stay, most of them will be > rewritten. I suppose that the rewrite would also touch generic code, so as far as it's cleanups and other refactoring I won't object investing the time to that. > No matter how I think the new code will be superior than old one, > I'm pretty sure there will be tons of regression or new bugs for > such huge rework. > > But without such work, btrfs-convert will always be a mess and no > real support for balance. > > So any ideas or advice is welcomed for this rework. In general, I welcome the efforts to make convert better. The in-place conversion is a unique feature and a nice to have. Several people mentioned that. Next, we want to add support more filesystems, eg. reiserfs code is ready and needs configure checks and minor updates to convert.c. Basically any filesystem that provides a library API similar to what ext2 does can be added. Convert code in a good shape would save us trouble. Expecting the regressions, the test set for the convertors needs to be enhanced, the current state is less then ideal. Possibly we can use the images from e2fsprogs testsuite. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Ideas for btrfs-convert fix(or rework) 2015-11-16 17:46 ` David Sterba @ 2015-11-17 0:42 ` Qu Wenruo 0 siblings, 0 replies; 13+ messages in thread From: Qu Wenruo @ 2015-11-17 0:42 UTC (permalink / raw) To: dsterba, Qu Wenruo, btrfs, David Sterba David Sterba wrote on 2015/11/16 18:46 +0100: > On Tue, Nov 10, 2015 at 02:27:41PM +0800, Qu Wenruo wrote: >> [[FIX IDEA]] >> 1. Too many patches > > Not percieved as a problem as long as the patches are well separated and > reviewable. If you prefer reasonably many preparatory and trivial > patches, then be it. > >> 2. superblock reserve is d*mning hard, not matter at what timing. >> Until I need to codes for superblock reserve, the codes are quite >> easy to write. >> But when it comes to sb reserve, I'm going to get crazy... > > What do you mean by the 'sb reserve'? Superblock reserved space. And the first 1Mb reserved space. IIRC, normal btrfs and convert btrfs doesn't reserve super block space well. For normal btrfs, it's still possible for device extent covering btrfs_sb_offset(), not to mention the converted one, which only uses one sb. But the good news is, although it needs extra codes(about 5 patches and over 500 lines), it shouldn't be a big problem as I'll reserve these space at early time. > >> I'll either add tons of codes to the simple chunk allocation to >> avoid using SB space and keep all chunks are large enough. >> Or I need to implement user-space whole chunk relocation codes. >> >> I miss the old days when sb_bytenr can be specified by user... >> Then my work should be quite close to end... >> >> 3. Too aggressive rework >> I suppose this will not be a fix, but a total rework of the whole >> btrfs-convert program. >> Maybe only less than 30% old codes will stay, most of them will be >> rewritten. > > I suppose that the rewrite would also touch generic code, so as far as > it's cleanups and other refactoring I won't object investing the time to > that. Not that much, most of the codes are rework of make_btrfs() to allow it to alloc metadata and system chunk at early time. It's almost 10 patches and over 1000 lines to rework the huge make_btrfs() into structuralized make_btrfs_v2(). And will remove some btrfs-convert only hooks like custom_alloc_extent(). As now, btrfs-convert will has a meta and system chunk(not covering existing ext* data) after make_btrfs_v2(). And then data chunks will be inserted to cover all ext* data. So no extra extent allocate routine is needed. > >> No matter how I think the new code will be superior than old one, >> I'm pretty sure there will be tons of regression or new bugs for >> such huge rework. >> >> But without such work, btrfs-convert will always be a mess and no >> real support for balance. >> >> So any ideas or advice is welcomed for this rework. > > In general, I welcome the efforts to make convert better. The in-place > conversion is a unique feature and a nice to have. Several people > mentioned that. Next, we want to add support more filesystems, eg. > reiserfs code is ready and needs configure checks and minor updates to > convert.c. Basically any filesystem that provides a library API similar > to what ext2 does can be added. Convert code in a good shape would save > us trouble. > > Expecting the regressions, the test set for the convertors needs to be > enhanced, the current state is less then ideal. Possibly we can use the > images from e2fsprogs testsuite. Nice idea. I'll try to enhance self test after btrfs-convert rework. Thanks, Qu > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2015-11-17 0:42 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-11-10 6:27 Ideas for btrfs-convert fix(or rework) Qu Wenruo 2015-11-10 7:55 ` Roman Mamedov 2015-11-10 8:16 ` Qu Wenruo 2015-11-10 9:08 ` Roman Mamedov 2015-11-10 9:18 ` Qu Wenruo 2015-11-10 10:31 ` Duncan 2015-11-12 10:23 ` Vytautas D 2015-11-12 13:27 ` Austin S Hemmelgarn 2015-11-12 14:09 ` Roman Mamedov 2015-11-12 14:38 ` Austin S Hemmelgarn 2015-11-13 6:41 ` Duncan 2015-11-16 17:46 ` David Sterba 2015-11-17 0:42 ` Qu Wenruo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.