Re: [PATCH 0/5] block: loop: add file format subsystem and QCOW2 file format driver

[Manuel Bentele <development@manuel-bentele.de>]

On 8/25/19 2:15 PM, Manuel Bentele wrote:
> On 8/24/19 6:04 PM, Bart Van Assche wrote:
>> On 8/24/19 2:14 AM, Manuel Bentele wrote:
>>> On 8/24/19 5:37 AM, Bart Van Assche wrote:
>>>> On 8/23/19 3:56 PM, development@manuel-bentele.de wrote:
>>>>> During the discussion, it turned out that the implementation as device
>>>>> mapper target is not applicable. The device mapper stacks different
>>>>> functionality such as compression or encryption on multiple block
>>>>> device
>>>>> layers whereas an implementation for the QCOW2 container format
>>>>> provides
>>>>> these functionalities on one block device layer.
>>>> Is there a more detailed discussion available of this subject?
>>> No, the only discussion is the referenced one [1]. But there was a
>>> similar discussion in the master's thesis of Francesc Zacarias Ribot
>>> [2]. Unfortunately, I found no attempt on the mailing list that proposes
>>> his solution.
>>>
>>>> Are you familiar with the dm-crypt driver?
>>> I don't know the specific implementation details, but I use this driver
>>> personally and I like it. Do you want to propose that only the storage
>>> aspect of the QCOW2 container format should be used and all other
>>> functionality inside the container should be provided by available
>>> device mapper targets?
>> (+Mike Snitzer)
>>
>> Hmm, I haven't found any reference to the device mapper in the
>> document written by Francesc. Maybe that means that I overlooked
>> something?
> Oh sorry, you're right. I meant this in general for the topic 'QCOW2 in
> the kernel space'.
>
>> I referred to the dm-crypt driver because I think that's an example
>> that shows that QCOW2 file format support could be implemented using
>> the device mapper framework.
> Okay, now I get it :)
>
>> Mike, do you perhaps want to comment on what the most appropriate way
>> is to implement such functionality?
> To implement the QCOW2 format or other sparse container formats
> correctly, the implementation must be able to ...
>   - extend the capacity of the mapped block device
>   - shrink the capacity of the mapped block device
>   - rescan the paritions of the mapped block device
>
> Are all three functionalities feasible using the device mapper framework?
Because there was no answer, I have analyzed the device mapper in more
detail. I found out, that one can get access to the virtual and
"underlying" devices. The virtual device (mapped_device) is created and
managed by the device mapper. The mapped_device can be obtained in the
constructor of a device mapper target by calling dm_table_get_md(). The
function call needs the table of the dm_target as parameter and returns
a pointer to the mapped_device structure. The structure contains
pointers to the gendisk and the block_device of the mapped_device. The
"underlying" devices of the table can be obtained or added by calling
dm_get_device() in the constructor, too. The call returns a pointer to a
dm_dev structure. Then, the dm_dev structure contains a pointer to its
referenced block_device. Now there is direct access to the block_device
or gendisk structures. This means that one can implement the three
functionalities to support sparse container formats and implement my
file format subsystem and file format drivers as device mapper targets.
But one should take care of the direct access to the block_device and
gendisk structures in a device mapper target because sometimes there is
the risk of bypassing the device mapper framework. Please be careful and
read the comments and descriptions of the exported functions in the
device mapper framework.

Compared to the proposed loop device module integration, this approach
seems harder to achieve for me. Furthermore, the device mapper target
needs an additional user space utility to simplify the control of the
file format subsystem and drivers and help people who are afraid of the
dmsetup utility ;)

Would you accept the proposed file format subsystem and drivers
implemented as device mapper targets?

>> The entire patch series is available at
>> https://lore.kernel.org/linux-block/86279379-32ac-15e9-2f91-68ce9c94cfbf@manuel-bentele.de/T/#t.
> Note that PATCH [1/5] is missing in this series, although I've submitted
> it twice. I asked already in [1] for the reason but haven't received any
> answer, yet. Therefore, I temporarily insert a link to my repository
> showing the missing PATCH [1/5]:
> https://github.com/bahnwaerter/linux/commit/7a78da744b4c84809ad6aa20673a2b686bafb201
>
> Regards,
> Manuel
>
> [1] https://www.spinics.net/lists/linux-block/msg44255.html

Regards,
Manuel