Adding QCOW2 reading/writing support

Adding QCOW2 reading/writing support

[Manuel Bentele]

Hi everyone

I'm going to implement an in-kernel reading of QCOW2 images.
In the project, I only need the reading of QCOW2 images, but it's
essential to make thoughts for the implementation of the writing, too.
One of the difficulties seems to be the support of making an image
sparse (resizing the disk image).

Now, I want to ask you for advice: What is the best approach to achieve
this?

  * Implement the reading/writing in the device mapper?
  * Extend the loop device?
  * Create a new subsystem for the processing of sparse devices or images?

Or do you have any other idea?

Regards,
Manuel Bentele

Re: Adding QCOW2 reading/writing support

[Ming Lei]

Hi,

On Wed, Apr 17, 2019 at 5:33 AM Manuel Bentele <manuel-bentele@web.de> wrote:
>
> Hi everyone
>
> I'm going to implement an in-kernel reading of QCOW2 images.
> In the project, I only need the reading of QCOW2 images, but it's
> essential to make thoughts for the implementation of the writing, too.
> One of the difficulties seems to be the support of making an image
> sparse (resizing the disk image).

Could you describe this requirement in a bit more detail? Especially why
do you want to read/write QCOW2 in kernel?

>
> Now, I want to ask you for advice: What is the best approach to achieve
> this?
>
>   * Implement the reading/writing in the device mapper?
>   * Extend the loop device?
>   * Create a new subsystem for the processing of sparse devices or images?
>
> Or do you have any other idea?

The 1st idea is that if this feature is really good requirement.

Thanks,
Ming Lei

Re: Adding QCOW2 reading/writing support

[Manuel Bentele]

Hi,

On 17.04.19 03:35, Ming Lei wrote:
> Hi,
>
> On Wed, Apr 17, 2019 at 5:33 AM Manuel Bentele <manuel-bentele@web.de> wrote:
>> Hi everyone
>>
>> I'm going to implement an in-kernel reading of QCOW2 images.
>> In the project, I only need the reading of QCOW2 images, but it's
>> essential to make thoughts for the implementation of the writing, too.
>> One of the difficulties seems to be the support of making an image
>> sparse (resizing the disk image).
> Could you describe this requirement in a bit more detail? Especially why
> do you want to read/write QCOW2 in kernel?

Yes, of course. The implementation of reading a QCOW2 disk image
in-kernel is required for an already existing system in the university
environment.
At the moment, the Linux kernel, initramfs, etc. for each client in the
system is loaded via PXE boot and then the block device with the default
file system is included with the help of a modified nbd version, called
dnbd (distributed nbd).
Due to the fact that the data on the default file system is only for non
persistent one-time provision of a client, read access is sufficient.
The user related data is stored on a network storage, as mostly done in
large scale infrastructures.

Now, the goal is to minimize the network usage and avoid nbd.
Furthermore, fixed configured and packed boot images should be avoided.
Therefore, the advantage of the sparse and compression functionality of
QCOW2 should be used.
A workaround for that problem could be the local usage of nbd to include
the QCOW2 disk image as block device, but it involves a lot of
interaction between user and kernel space and thus an decreasing
performance. That leads to the motivation to implement the reading of
QCOW2 disk images directly in the kernel and aim for an merge into the
mainline kernel source to avoid out-of-kernel-tree maintenance.

If you have any questions related to the described use case or if you
require more information, please let me know.
Thanks for your help.

>> Now, I want to ask you for advice: What is the best approach to achieve
>> this?
>>
>>   * Implement the reading/writing in the device mapper?
>>   * Extend the loop device?
>>   * Create a new subsystem for the processing of sparse devices or images?
>>
>> Or do you have any other idea?
> The 1st idea is that if this feature is really good requirement.
>
> Thanks,
> Ming Lei
Regards,
Manuel Bentele

Re: Adding QCOW2 reading/writing support

[Hannes Reinecke]

On 4/16/19 11:30 PM, Manuel Bentele wrote:
> Hi everyone
> 
> I'm going to implement an in-kernel reading of QCOW2 images.
> In the project, I only need the reading of QCOW2 images, but it's
> essential to make thoughts for the implementation of the writing, too.
> One of the difficulties seems to be the support of making an image
> sparse (resizing the disk image).
> 
> Now, I want to ask you for advice: What is the best approach to achieve
> this?
> 
>    * Implement the reading/writing in the device mapper?
>    * Extend the loop device?
>    * Create a new subsystem for the processing of sparse devices or images?
> 
> Or do you have any other idea?
> 
I'm not sure if that would be met with universal acclaim; file formats 
are regarded as userland policies, and shouldn't be handled in the 
kernel at all.
Have you looked at fuse?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                              +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)

Re: Adding QCOW2 reading/writing support

[Hannes Reinecke]

On 4/17/19 1:32 PM, Manuel Bentele wrote:
> Hi,
> 
> On 17.04.19 03:35, Ming Lei wrote:
>> Hi,
>>
>> On Wed, Apr 17, 2019 at 5:33 AM Manuel Bentele <manuel-bentele@web.de> wrote:
>>> Hi everyone
>>>
>>> I'm going to implement an in-kernel reading of QCOW2 images.
>>> In the project, I only need the reading of QCOW2 images, but it's
>>> essential to make thoughts for the implementation of the writing, too.
>>> One of the difficulties seems to be the support of making an image
>>> sparse (resizing the disk image).
>> Could you describe this requirement in a bit more detail? Especially why
>> do you want to read/write QCOW2 in kernel?
> 
> Yes, of course. The implementation of reading a QCOW2 disk image
> in-kernel is required for an already existing system in the university
> environment.
> At the moment, the Linux kernel, initramfs, etc. for each client in the
> system is loaded via PXE boot and then the block device with the default
> file system is included with the help of a modified nbd version, called
> dnbd (distributed nbd).
> Due to the fact that the data on the default file system is only for non
> persistent one-time provision of a client, read access is sufficient.
> The user related data is stored on a network storage, as mostly done in
> large scale infrastructures.
> 
> Now, the goal is to minimize the network usage and avoid nbd.
> Furthermore, fixed configured and packed boot images should be avoided.
> Therefore, the advantage of the sparse and compression functionality of
> QCOW2 should be used.
> A workaround for that problem could be the local usage of nbd to include
> the QCOW2 disk image as block device, but it involves a lot of
> interaction between user and kernel space and thus an decreasing
> performance. That leads to the motivation to implement the reading of
> QCOW2 disk images directly in the kernel and aim for an merge into the
> mainline kernel source to avoid out-of-kernel-tree maintenance.
> 
> If you have any questions related to the described use case or if you
> require more information, please let me know.
> Thanks for your help.
> 
cramfs?
Or btrfs with compression enabled?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                              +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)

Re: Adding QCOW2 reading/writing support

[Manuel Bentele]

On 17.04.19 14:16, Hannes Reinecke wrote:
> On 4/17/19 1:32 PM, Manuel Bentele wrote:
>> Hi,
>>
>> On 17.04.19 03:35, Ming Lei wrote:
>>> Hi,
>>>
>>> On Wed, Apr 17, 2019 at 5:33 AM Manuel Bentele
>>> <manuel-bentele@web.de> wrote:
>>>> Hi everyone
>>>>
>>>> I'm going to implement an in-kernel reading of QCOW2 images.
>>>> In the project, I only need the reading of QCOW2 images, but it's
>>>> essential to make thoughts for the implementation of the writing, too.
>>>> One of the difficulties seems to be the support of making an image
>>>> sparse (resizing the disk image).
>>> Could you describe this requirement in a bit more detail? Especially
>>> why
>>> do you want to read/write QCOW2 in kernel?
>>
>> Yes, of course. The implementation of reading a QCOW2 disk image
>> in-kernel is required for an already existing system in the university
>> environment.
>> At the moment, the Linux kernel, initramfs, etc. for each client in the
>> system is loaded via PXE boot and then the block device with the default
>> file system is included with the help of a modified nbd version, called
>> dnbd (distributed nbd).
>> Due to the fact that the data on the default file system is only for non
>> persistent one-time provision of a client, read access is sufficient.
>> The user related data is stored on a network storage, as mostly done in
>> large scale infrastructures.
>>
>> Now, the goal is to minimize the network usage and avoid nbd.
>> Furthermore, fixed configured and packed boot images should be avoided.
>> Therefore, the advantage of the sparse and compression functionality of
>> QCOW2 should be used.
>> A workaround for that problem could be the local usage of nbd to include
>> the QCOW2 disk image as block device, but it involves a lot of
>> interaction between user and kernel space and thus an decreasing
>> performance. That leads to the motivation to implement the reading of
>> QCOW2 disk images directly in the kernel and aim for an merge into the
>> mainline kernel source to avoid out-of-kernel-tree maintenance.
>>
>> If you have any questions related to the described use case or if you
>> require more information, please let me know.
>> Thanks for your help.
>>
> cramfs?
> Or btrfs with compression enabled?
>
> Cheers,
>
> Hannes

Thanks for your simple idea to choose cramfs or btrfs with compression
enabled.
I will suggest that as alternative at the next project meeting.

Regards,
Manuel

Re: Adding QCOW2 reading/writing support

[Manuel Bentele]

On 17.04.19 13:58, Hannes Reinecke wrote:
> On 4/16/19 11:30 PM, Manuel Bentele wrote:
>> Hi everyone
>>
>> I'm going to implement an in-kernel reading of QCOW2 images.
>> In the project, I only need the reading of QCOW2 images, but it's
>> essential to make thoughts for the implementation of the writing, too.
>> One of the difficulties seems to be the support of making an image
>> sparse (resizing the disk image).
>>
>> Now, I want to ask you for advice: What is the best approach to achieve
>> this?
>>
>>    * Implement the reading/writing in the device mapper?
>>    * Extend the loop device?
>>    * Create a new subsystem for the processing of sparse devices or
>> images?
>>
>> Or do you have any other idea?
>>
> I'm not sure if that would be met with universal acclaim; file formats
> are regarded as userland policies, and shouldn't be handled in the
> kernel at all.
> Have you looked at fuse?
>
> Cheers,
>
> Hannes
I agree with your notice that file formats should stay outside the
kernel space.
Fuse would be a very good approach to reach the goal, but it's only
intended for file systems.
But QCOW2 in general can be seen as a container or virtual disk node
containing various data.
Therefore, functionality like blkdev_reread_part for partition scanning
or fsck is needed, but it is not available in the user space. That
means, code must be translated and duplicated which is again not an
appropriate solution.

I have seen, that the tool vmware-mount for the VMDK file format is
based on fuse and supports all the mentioned block device functionality.
I was wondering how they achieved that but I don't want to know how much
code was duplicated.

If you have an idea to solve the opposition for file formats that are
acting as a container or virtual disk node for various data, please let
me know.

Regards,
Manuel

Re: Adding QCOW2 reading/writing support

[Ming Lei]

On Thu, Apr 18, 2019 at 5:04 AM Manuel Bentele <manuel-bentele@web.de> wrote:
>
> On 17.04.19 14:16, Hannes Reinecke wrote:
> > On 4/17/19 1:32 PM, Manuel Bentele wrote:
> >> Hi,
> >>
> >> On 17.04.19 03:35, Ming Lei wrote:
> >>> Hi,
> >>>
> >>> On Wed, Apr 17, 2019 at 5:33 AM Manuel Bentele
> >>> <manuel-bentele@web.de> wrote:
> >>>> Hi everyone
> >>>>
> >>>> I'm going to implement an in-kernel reading of QCOW2 images.
> >>>> In the project, I only need the reading of QCOW2 images, but it's
> >>>> essential to make thoughts for the implementation of the writing, too.
> >>>> One of the difficulties seems to be the support of making an image
> >>>> sparse (resizing the disk image).
> >>> Could you describe this requirement in a bit more detail? Especially
> >>> why
> >>> do you want to read/write QCOW2 in kernel?
> >>
> >> Yes, of course. The implementation of reading a QCOW2 disk image
> >> in-kernel is required for an already existing system in the university
> >> environment.
> >> At the moment, the Linux kernel, initramfs, etc. for each client in the
> >> system is loaded via PXE boot and then the block device with the default
> >> file system is included with the help of a modified nbd version, called
> >> dnbd (distributed nbd).
> >> Due to the fact that the data on the default file system is only for non
> >> persistent one-time provision of a client, read access is sufficient.
> >> The user related data is stored on a network storage, as mostly done in
> >> large scale infrastructures.
> >>
> >> Now, the goal is to minimize the network usage and avoid nbd.
> >> Furthermore, fixed configured and packed boot images should be avoided.
> >> Therefore, the advantage of the sparse and compression functionality of
> >> QCOW2 should be used.
> >> A workaround for that problem could be the local usage of nbd to include
> >> the QCOW2 disk image as block device, but it involves a lot of
> >> interaction between user and kernel space and thus an decreasing
> >> performance. That leads to the motivation to implement the reading of
> >> QCOW2 disk images directly in the kernel and aim for an merge into the
> >> mainline kernel source to avoid out-of-kernel-tree maintenance.
> >>
> >> If you have any questions related to the described use case or if you
> >> require more information, please let me know.
> >> Thanks for your help.
> >>
> > cramfs?
> > Or btrfs with compression enabled?
> >
> > Cheers,
> >
> > Hannes
>
> Thanks for your simple idea to choose cramfs or btrfs with compression
> enabled.
> I will suggest that as alternative at the next project meeting.

Or vdo which provides compression in block device level:

https://github.com/dm-vdo/vdo


Thanks,
Ming Lei

Re: Adding QCOW2 reading/writing support

[Manuel Bentele]

On 18.04.19 03:05, Ming Lei wrote:
> On Thu, Apr 18, 2019 at 5:04 AM Manuel Bentele <manuel-bentele@web.de> wrote:
>> On 17.04.19 14:16, Hannes Reinecke wrote:
>>> On 4/17/19 1:32 PM, Manuel Bentele wrote:
>>>> Hi,
>>>>
>>>> On 17.04.19 03:35, Ming Lei wrote:
>>>>> Hi,
>>>>>
>>>>> On Wed, Apr 17, 2019 at 5:33 AM Manuel Bentele
>>>>> <manuel-bentele@web.de> wrote:
>>>>>> Hi everyone
>>>>>>
>>>>>> I'm going to implement an in-kernel reading of QCOW2 images.
>>>>>> In the project, I only need the reading of QCOW2 images, but it's
>>>>>> essential to make thoughts for the implementation of the writing, too.
>>>>>> One of the difficulties seems to be the support of making an image
>>>>>> sparse (resizing the disk image).
>>>>> Could you describe this requirement in a bit more detail? Especially
>>>>> why
>>>>> do you want to read/write QCOW2 in kernel?
>>>> Yes, of course. The implementation of reading a QCOW2 disk image
>>>> in-kernel is required for an already existing system in the university
>>>> environment.
>>>> At the moment, the Linux kernel, initramfs, etc. for each client in the
>>>> system is loaded via PXE boot and then the block device with the default
>>>> file system is included with the help of a modified nbd version, called
>>>> dnbd (distributed nbd).
>>>> Due to the fact that the data on the default file system is only for non
>>>> persistent one-time provision of a client, read access is sufficient.
>>>> The user related data is stored on a network storage, as mostly done in
>>>> large scale infrastructures.
>>>>
>>>> Now, the goal is to minimize the network usage and avoid nbd.
>>>> Furthermore, fixed configured and packed boot images should be avoided.
>>>> Therefore, the advantage of the sparse and compression functionality of
>>>> QCOW2 should be used.
>>>> A workaround for that problem could be the local usage of nbd to include
>>>> the QCOW2 disk image as block device, but it involves a lot of
>>>> interaction between user and kernel space and thus an decreasing
>>>> performance. That leads to the motivation to implement the reading of
>>>> QCOW2 disk images directly in the kernel and aim for an merge into the
>>>> mainline kernel source to avoid out-of-kernel-tree maintenance.
>>>>
>>>> If you have any questions related to the described use case or if you
>>>> require more information, please let me know.
>>>> Thanks for your help.
>>>>
>>> cramfs?
>>> Or btrfs with compression enabled?
>>>
>>> Cheers,
>>>
>>> Hannes
>> Thanks for your simple idea to choose cramfs or btrfs with compression
>> enabled.
>> I will suggest that as alternative at the next project meeting.
> Or vdo which provides compression in block device level:
>
> https://github.com/dm-vdo/vdo

Thanks for your hint. I will take a look at it.

Regards,
Manuel

Re: Adding QCOW2 reading/writing support

[Manuel Bentele]

Hi

On 4/18/19 12:02 PM, Manuel Bentele wrote:
> On 18.04.19 03:05, Ming Lei wrote:
>> On Thu, Apr 18, 2019 at 5:04 AM Manuel Bentele <manuel-bentele@web.de> wrote:
>>> On 17.04.19 14:16, Hannes Reinecke wrote:
>>>> On 4/17/19 1:32 PM, Manuel Bentele wrote:
>>>>> Hi,
>>>>>
>>>>> On 17.04.19 03:35, Ming Lei wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On Wed, Apr 17, 2019 at 5:33 AM Manuel Bentele
>>>>>> <manuel-bentele@web.de> wrote:
>>>>>>> Hi everyone
>>>>>>>
>>>>>>> I'm going to implement an in-kernel reading of QCOW2 images.
>>>>>>> In the project, I only need the reading of QCOW2 images, but it's
>>>>>>> essential to make thoughts for the implementation of the writing, too.
>>>>>>> One of the difficulties seems to be the support of making an image
>>>>>>> sparse (resizing the disk image).
>>>>>> Could you describe this requirement in a bit more detail? Especially
>>>>>> why
>>>>>> do you want to read/write QCOW2 in kernel?
>>>>> Yes, of course. The implementation of reading a QCOW2 disk image
>>>>> in-kernel is required for an already existing system in the university
>>>>> environment.
>>>>> At the moment, the Linux kernel, initramfs, etc. for each client in the
>>>>> system is loaded via PXE boot and then the block device with the default
>>>>> file system is included with the help of a modified nbd version, called
>>>>> dnbd (distributed nbd).
>>>>> Due to the fact that the data on the default file system is only for non
>>>>> persistent one-time provision of a client, read access is sufficient.
>>>>> The user related data is stored on a network storage, as mostly done in
>>>>> large scale infrastructures.
>>>>>
>>>>> Now, the goal is to minimize the network usage and avoid nbd.
>>>>> Furthermore, fixed configured and packed boot images should be avoided.
>>>>> Therefore, the advantage of the sparse and compression functionality of
>>>>> QCOW2 should be used.
>>>>> A workaround for that problem could be the local usage of nbd to include
>>>>> the QCOW2 disk image as block device, but it involves a lot of
>>>>> interaction between user and kernel space and thus an decreasing
>>>>> performance. That leads to the motivation to implement the reading of
>>>>> QCOW2 disk images directly in the kernel and aim for an merge into the
>>>>> mainline kernel source to avoid out-of-kernel-tree maintenance.
>>>>>
>>>>> If you have any questions related to the described use case or if you
>>>>> require more information, please let me know.
>>>>> Thanks for your help.
>>>>>
>>>> cramfs?
>>>> Or btrfs with compression enabled?
>>>>
>>>> Cheers,
>>>>
>>>> Hannes
>>> Thanks for your simple idea to choose cramfs or btrfs with compression
>>> enabled.
>>> I will suggest that as alternative at the next project meeting.
>> Or vdo which provides compression in block device level:
>>
>> https://github.com/dm-vdo/vdo
> Thanks for your hint. I will take a look at it.

I suggested all your ideas at the project meeting. The meeting revealed
that most of the people want to hold on the QCOW2 container format and
want to avoid the 'stacking' of other solutions.
In addition to that, FUSE should not be used due to performance reasons.

All in all, that leads to the plan to extend the existing loop device
driver to support not only raw binary files, but also the QCOW2
container format. If there exists a usable and clean solution at the
end, I will inform you and propose it for a review.

Thanks for your help.

Regards,
Manuel
<mailto:manuel-bentele@web.de>

Re: Adding QCOW2 reading/writing support

[Roman Penyaev]

On 2019-04-17 23:53, Manuel Bentele wrote:

[cut]

> If you have an idea to solve the opposition for file formats that are
> acting as a container or virtual disk node for various data, please let
> me know.

If you need a raw block device you can export qcow2 image as a network 
block
device, something as the following:

# modprobe nbd
# qemu-nbd --connect=/dev/nbd0 image.qcow2
# fdisk -l /dev/nbd0

I suppose simplest variant.

--
Roman

Re: Adding QCOW2 reading/writing support

[Manuel Bentele]

Hi,

On 5/14/19 4:28 PM, Roman Penyaev wrote:
> On 2019-04-17 23:53, Manuel Bentele wrote:
>
> [cut]
>
>> If you have an idea to solve the opposition for file formats that are
>> acting as a container or virtual disk node for various data, please let
>> me know.
>
> If you need a raw block device you can export qcow2 image as a network
> block
> device, something as the following:
>
> # modprobe nbd
> # qemu-nbd --connect=/dev/nbd0 image.qcow2
> # fdisk -l /dev/nbd0
>
> I suppose simplest variant.
>
> --
> Roman

Thanks for your suggestion. We already discussed this during a project
meeting. Some time ago, I have summarized that already (sorry for the
diverging conversation of this topic on the mailinglist).
All in all, the crucial point is the performance:

> [...]
>
> A workaround for that problem could be the local usage of nbd to include
> the QCOW2 disk image as block device, but it involves a lot of
> interaction between user and kernel space and thus an decreasing
> performance. That leads to the motivation to implement the reading of
> QCOW2 disk images directly in the kernel and aim for an merge into the
> mainline kernel source to avoid out-of-kernel-tree maintenance.
>
> [...]

--
Manuel