Re: How about adding an ioctl to convert a directory to a subvolume?

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 5529 bytes --]



On 2017年11月28日 16:29, Lu Fengqi wrote:
> On Mon, Nov 27, 2017 at 06:13:10PM +0800, Qu Wenruo wrote:
>>
>>
>> On 2017年11月27日 17:41, Lu Fengqi wrote:
>>> Hi all,
>>>
>>> As we all know, under certain circumstances, it is more appropriate to
>>> create some subvolumes rather than keep everything in the same
>>> subvolume. As the condition of demand change, the user may need to
>>> convert a previous directory to a subvolume. For this reason，how about
>>> adding an ioctl to convert a directory to a subvolume?
>>
>> The idea seems interesting.
>>
>> However in my opinion, this can be done quite easily in (mostly) user
>> space, thanks to btrfs support of relink.
>>
>> The method from Hugo or Chris is quite good, maybe it can be enhanced a
>> little.
>>
>> Use the following layout as an example:
>>
>> root_subv
>> |- subvolume_1
>> |  |- dir_1
>> |  |  |- file_1
>> |  |  |- file_2
>> |  |- dir_2
>> |     |- file_3
>> |- subvolume_2
>>
>> If we want to convert dir_1 into subvolume, we can do it like:
>>
>> 1) Create a temporary readonly snapshot of parent subvolume containing
>>   the desired dir
>>   # btrfs sub snapshot -r root_subv/subvolume_1 \
>>     root_subv/tmp_snapshot_1
>>
>> 2) Create a new subvolume, as destination.
>>   # btrfs sub create root_subv/tmp_dest/
>>
>> 3) Copy the content and sync the fs
>>   Use of reflink is necessary.
>>   # cp -r --reflink=always root_subv/tmp_snapshot_1/dir_1 \
>>     root_subv/tmp_dest
>>   # btrfs sync root_subv/tmp_dest
>>
>> 4) Delete temporary readonly snapshot
>>   # btrfs subvolume delete root_subv/tmp_snapshot_1
>>
>> 5) Remove the source dir
>>   # rm -rf root_subv/subvolume_1/dir_1
>>
>> 5) Create a final destination snapshot of "root_subv/temporary_dest"
>>   # btrfs subvolume snapshot root_subv/tmp_dest \
>>     root_subv/subvolume_1/dir_1
>>
>> 6) Remove the temporary destination
>>   # btrfs subvolume delete root_subv/tmp_dest
>>
>>
>> The main challenge is in step 3).
>> In fact above method can only handle normal dir/files.
>> If there is another subvolume inside the desired dir, current "cp -r" is
>> a bad idea.
>> We need to skip subvolume dir, and create snapshot for it.
>>
>> But it's quite easy to write a user space program to handle it.
>> Maybe using "find" command can already handle it well.
>>
>> Anyway, doing it in user space onsidering btrfs snapshot creation won't flush data, which already means some buffered data will not occur in snapshot.
is already possible and much easier than
>> doing it in kernel.
>>
>>>
>>> Users can convert by the scripts mentioned in this
>>> thread(https://www.spinics.net/lists/linux-btrfs/msg33252.html), but is
>>> it easier to use the off-the-shelf btrfs subcommand?
>>
>> If you just want to integrate the functionality into btrfs-progs, maybe
>> it's possible.
>>
>> But if you insist in providing a new ioctl for this, I highly doubt if
>> the extra hassle is worthy.
>>
> 
> Thanks for getting back to me.
> 
> The enhanced approach you provide is pretty good, and I agree that it
> meets the needs of some users.
> 
>>>
>>> After an initial consideration, our implementation is broadly divided
>>> into the following steps:
>>> 1. Freeze the filesystem or set the subvolume above the source directory
>>> to read-only;
>>
>> Not really need to freeze the whole fs.
>> Just create a readonly snapshot of the parent subvolume which contains
>> the dir.
>> That's how snapshot is designed for.
>>
> 
> I still worry about the following problem. Although tmp_snapshot_1 is
> read-only, the source directory dir_1 is still writable. Also, step 5)
> will delete dir_1, which may result in the loss of newly written data
> during the conversion.

Just a problem of timing.

Output message after or before creation of read-only snapshot to info
user will be good enough.

Especially considering btrfs snapshot creation won't flush data, which
already means some buffered data will not occur in snapshot.

Thanks,
Qu

> 
> I can not think of any method else for now, except I set the subvolume_1
> read-only(obviously this is a bad idea). Could you provide some
> suggestions?
> 
>>> 2. Perform a pre-check, for example, check if a cross-device link
>>> creation during the conversion;
>>
>> This can be done in-the-fly.
>> As the check is so easy (only needs to check if the inode number is 256).
>> We only need a mid-order iteration of the source dir (in temporary
>> snapshot), and for normal file, use reflink.
>> For subvolume dir, create a snapshot for it.
>>
>> And for such iteration, a python script less than 100 lines would be
>> sufficient.
> 
> Just to clarify, the check is to confirm if there is a hard link across
> dir_1.> The dir_1 can't be converted to subvolume if there is such a
> cross-device link that can not be created.
> 
>>
>> Thanks,
>> Qu
>>
>>> 3. Perform conversion, such as creating a new subvolume and moving the
>>> contents of the source directory;
>>> 4. Thaw the filesystem or restore the subvolume writable property.
>>>
>>> In fact, I am not so sure whether this use of freeze is appropriate
>>> because the source directory the user needs to convert may be located
>>> at / or /home and this pre-check and conversion process may take a long
>>> time, which can lead to some shell and graphical application suspended.
>>>
>>> Please give your comments if any.
>>>
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]