Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

[Kai Herlemann]

Hi,
I have on my ext4 filesystem some sparse files, mostly images from
ext4 filesystems.
Is btrfs-convert (4.9.1) able to deal with sparse files or can that
cause any problems?

Thanks in advance,
Kai

Re: Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

[Sean Greenslade]

On Sat, Apr 01, 2017 at 11:48:50AM +0200, Kai Herlemann wrote:
> Hi,
> I have on my ext4 filesystem some sparse files, mostly images from
> ext4 filesystems.
> Is btrfs-convert (4.9.1) able to deal with sparse files or can that
> cause any problems?

>From personal experience, I would recommend not using btrfs-convert on
ext4 partitions. I attempted it on a /home partition on one of my
machines, and while it did succeed in converting, the fs it produced had
weird issues that caused transation failures and thus semi-frequent
remount-ro. Btrfs-check, scrub, and balance were all unable to repair
the damage. I ended up recreating the parition from a backup.

As far as I know, there were no sparse files on this partition, either.

Just my one data point, for whatever it's worth.

--Sean

Re: Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

[Duncan]

Sean Greenslade posted on Sat, 01 Apr 2017 12:13:57 -0700 as excerpted:

> On Sat, Apr 01, 2017 at 11:48:50AM +0200, Kai Herlemann wrote:

>> I have on my ext4 filesystem some sparse files, mostly images from ext4
>> filesystems.
>> Is btrfs-convert (4.9.1) able to deal with sparse files or can that
>> cause any problems?
> 
> From personal experience, I would recommend not using btrfs-convert on
> ext4 partitions.

While I'd be extremely surprised if btrfs-convert didn't work on sparse 
files, since if it didn't it wouldn't be a general-purpose converter and 
thus wouldn't be suited to the purpose...

I must agree, tho on general principles, with Sean here, btrfs-convert 
isn't something I'd either use myself or recommend to others.  Consider:

1) Btrfs is considered on this list to be stabilizing, not fully stable 
and mature.  While in general (that is, even on stable and mature 
filesystems) the real value of your data can be defined by whether you 
care enough about it to have backups of that data -- if you don't, you 
self-evidently care less about that data than the time, resources and 
hassle you're saving by NOT doing the backup[1] -- on a still stabilizing 
filesystem such as btrfs, that applies even more strongly.  If you don't 
have a backup and aren't ready to use it if necessary, you really ARE 
declaring that data to be of less value than the time/hassle/resource 
cost of doing it.

2) It follows from #1 that (assuming you consider the data of reasonable 
value) you have backups, and are prepared to restore from them.  Which 
means you have the /space/ for that backup.

3) Which means there's very little reason to use a converter such as 
btrfs-convert, because you can just do a straightforward blow away the 
filesystem and restore from backup (or from the primaries or a secondary 
backup if it /is/ your backup).

4) In fact, since an in-place convert is almost certainly going to take 
more time than a blow-away and restore from backup, and the end result is 
pretty well guaranteed to be less optimally arranged in the new native 
format than a freshly created filesystem with data equally freshly copied 
over from backups or primary sources, there are pretty big reasons *NOT* 
to do an in-place convert.

5) And if you don't have current backups, then by creating a brand new 
btrfs in new space and copying over from your existing ext4, you 
"magically" create that recommended backup, since that ext4 can then be 
used as a backup for your new btrfs.  Of course you'll eventually need to 
update that backup, but meanwhile, it'll be a useful backup, should it be 
needed, while you're settling in on the new btrfs.  =:^)


Meanwhile, it can be noted that plain old cp has the -a/--archive option 
that makes using it for making and restoring backups easier, and it also 
has a --sparse option.  Back on reiserfs, I used to use the 
--sparse=always option for my backups here, without issue, tho on btrfs I 
use the compress (actually compress=lzo] mount option, which should 
compress sparse areas of files even if the files don't get created 
specifically as sparse files, so I don't worry about it on btrfs.

Tho if those ext4 images are to be actively used by VMs or are otherwise 
actively written to, on btrfs I'd consider using the nocow attribute for 
them, and it disables btrfs compression, so I'd consider sparse copying 
for them.  But that's an entirely different topic worthy of its own 
thread if your use-case requires it and you still have questions on it 
after doing your own research...

---
[1] Backup:  Note that a backup that hasn't been tested to be actually 
restorable isn't yet a backup, only a potential backup, as the job of 
making a backup isn't complete until that backup has been tested to be 
restorable.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

[Andrei Borzenkov]

02.04.2017 03:59, Duncan пишет:
> 
> 4) In fact, since an in-place convert is almost certainly going to take 
> more time than a blow-away and restore from backup,

This caught my eyes. Why? In-place convert just needs to recreate
metadata. If you have multi-terabyte worth of data copying them twice
hardly can be faster.

Re: Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

[Qu Wenruo]

At 04/01/2017 05:48 PM, Kai Herlemann wrote:
> Hi,
> I have on my ext4 filesystem some sparse files, mostly images from
> ext4 filesystems.
> Is btrfs-convert (4.9.1) able to deal with sparse files or can that
> cause any problems?

AFAIK, btrfs convert can handle it well, not matter if it's sparse or not.

Secondly, it's strongly recommended to use latest btrfs-convert, which 
is 4.10.2.

Although the convert part is not changed in recent release, but the 
rollback part (which rolls converted btrfs back to ext*) under goes a 
large modification to co-operate with convert.

Without that modification, rollback is not functional, which is fine if 
you don't want to rollback to ext*, but just in case.

Thanks,
Qu

>
> Thanks in advance,
> Kai
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

[Roman Mamedov]

On Sun, 2 Apr 2017 09:30:46 +0300
Andrei Borzenkov <arvidjaar@gmail.com> wrote:

> 02.04.2017 03:59, Duncan пишет:
> > 
> > 4) In fact, since an in-place convert is almost certainly going to take 
> > more time than a blow-away and restore from backup,
> 
> This caught my eyes. Why? In-place convert just needs to recreate
> metadata. If you have multi-terabyte worth of data copying them twice
> hardly can be faster.

In-place convert is most certainly faster than copy-away and restore, in fact
it can be very fast if you use the option to not calculate checksums for the
entire filesystem's data (btrfs-convert -d).

-- 
With respect,
Roman

Re: Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

[Austin S. Hemmelgarn]

On 2017-04-01 05:48, Kai Herlemann wrote:
> Hi,
> I have on my ext4 filesystem some sparse files, mostly images from
> ext4 filesystems.
> Is btrfs-convert (4.9.1) able to deal with sparse files or can that
> cause any problems?
I would tend to agree with some of the other people who have commented 
here, but with a slight twist:
If you just want to test BTRFS short-term, then in-place conversion is 
fine.  The moment you decide to keep using BTRFS for production systems, 
you should be creating new filesystems instead of doing in-place conversion.

There are a couple of reasons for this:
1. In-place conversion will give you an inherently sub-optimal on-disk 
data layout.  You'll need some extra work to remedy this, and because of 
that extra work, it will generally not be any quicker to convert 
in-place than to create a new filesystem.  You can of course choose not 
to fix this, but performance will likely suffer. (This can also be 
slightly mitigated by making sure to defragment the filesystem before 
conversion).
2. Slightly related to 1, btrfs-convert can leave behind artifacts from 
the conversion itself that may cause issues later.
3. Tying in with both 1 and 2, there's very little testing done on 
converted filesystems.  You are therefore by definition more likely to 
run into bugs or other issues.
4. In-place conversion is inherently risky.  If something goes wrong, 
you're going to have to restore from a backup anyway.

Re: Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

[Duncan]

Andrei Borzenkov posted on Sun, 02 Apr 2017 09:30:46 +0300 as excerpted:

> 02.04.2017 03:59, Duncan пишет:
>> 
>> 4) In fact, since an in-place convert is almost certainly going to take
>> more time than a blow-away and restore from backup,
> 
> This caught my eyes. Why? In-place convert just needs to recreate
> metadata. If you have multi-terabyte worth of data copying them twice
> hardly can be faster.

Why twice?  If you care about the data by definition you already have 
backups, so it's only copying back from those backups to the newly 
created filesystem, right?

And if you don't have backups, then by definition, you don't care about 
the data, at least not enough to be worth the hassle of a backup and thus 
arguably not enough to be worth the hassle of a convert, so just blow it 
away with a new mkfs and start from scratch since you self-evidently 
didn't care enough about the data for it to be worth a backup anyway, no 
problem.

And actually, it's not even a single extra copy that you won't have to do 
anyway, if you schedule your new filesystem creation as part of your 
normal backup regime in place of what would otherwise be a full backup 
that you now don't have to make, so copying the data from the old 
filesystem to the new one is simply replacing the full backup that you'd 
otherwise be doing at the same point in time.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Is btrfs-convert able to deal with sparse files in a ext4 filesystem?

[Kai Herlemann]

Hi @all who answered,
thank for your help and please excuse my late answer. I didn't see
your answers because of misconfiguration of my GMail filter for that
list.
The filesystem contains backups of some other filesystems (it's on a
external storage which is mirrored by RAID 1). So, if the filesystem
get lost, there are still the source partitions of the backups. But
you're right, I have to have actually more space purposes like using
btrfs-convert, which needs a backup, but can't/want afford that at the
moment because I'm student.

As far as I remember, I used in the past btrfs-convert to convert the
ext4 root partition of my laptop. It didn't really work with old
versions of btrfs-progs (I rollbacked it then or imported a backup),
but it worked with 4.4 or so.
Nevertheless, I won't use btrfs-convert after your warnings, and will
create a new filesystem and copy the data from ext4 to btrfs when I
have enough space.

Thank you all,
Kai