[Qemu-devel] [Bug 601946] [NEW] [Feature request] qemu-img multi-threaded compressed image conversion

[Qemu-devel] [Bug 601946] [NEW] [Feature request] qemu-img multi-threaded compressed image conversion

[Коренберг Марк]

Public bug reported:

Feature request:
qemu-img multi-threaded compressed image conversion

Suppose I want to convert raw image to compressed qcow2. Multi-threaded
conversion will be much faster, because bottleneck is compressing data.

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
[Feature request] qemu-img multi-threaded compressed image conversion
https://bugs.launchpad.net/bugs/601946
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: New

Bug description:
Feature request:
qemu-img multi-threaded compressed image conversion

Suppose I want to convert raw image to compressed qcow2. Multi-threaded conversion will be much faster, because bottleneck is compressing data.

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Jes Sorensen]

Hi,

The problem is that it is more than just the compression that is the
problem, with modern cpus disk speed is a problem, and compression is
often stream based. For now there isn't enough valid data that this
qualifies as a bug/rfe.

If you decide to try and implement it, and provide data showing that
this is actually a win, please reopen this.

Regards,
Jes

-- 
[Feature request] qemu-img multi-threaded compressed image conversion
https://bugs.launchpad.net/bugs/601946
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: New

Bug description:
Feature request:
qemu-img multi-threaded compressed image conversion

Suppose I want to convert raw image to compressed qcow2. Multi-threaded conversion will be much faster, because bottleneck is compressing data.

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Коренберг Марк]

1. during benchmark I used iotop and just top. qemu-img is eating all my cpu (3.07 Ghz) and disk streaming was at low speeds.
2. Writing on disk in ext4 is cached very strongly, so writing in 4 streams is not the problem.
3. For example, 7z give huge speed increase in when compressing in multiple threads.
4. Yes, i understand, that compressing is stream-based. So we can split input stream by chinks and compress each chunk individually.

You can use time qemu-img convert .... and see user/system/real
timings. In my cases, user time is nearly equal real time, so CPU work
in my case is the bottleneck.

-- 
[Feature request] qemu-img multi-threaded compressed image conversion
https://bugs.launchpad.net/bugs/601946
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: New

Bug description:
Feature request:
qemu-img multi-threaded compressed image conversion

Suppose I want to convert raw image to compressed qcow2. Multi-threaded conversion will be much faster, because bottleneck is compressing data.

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Jan-Simon Möller]

There're also projects like http://compression.ca/pbzip2/  .  We'll be
facing more and more cores per cpu, so we should use these techniques.

-- 
[Feature request] qemu-img multi-threaded compressed image conversion
https://bugs.launchpad.net/bugs/601946
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: New

Bug description:
Feature request:
qemu-img multi-threaded compressed image conversion

Suppose I want to convert raw image to compressed qcow2. Multi-threaded conversion will be much faster, because bottleneck is compressing data.

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Mike Ashton]

The compression in this case is certainly chunked already, otherwise you
couldn't implement a pseudo block device without reading the entire
stream to read the last block!  As the data in the new disk is
necessarily chunk compressed, parallelisation is perfect feasible, it's
just a question of the algorithm you use to arbitrate the work between
the threads, which may need some thought as you'd likely be navigating a
tree structure.

There's no question that Jes' suggestion would create a 12x speed up for
me, and there's pretty standard off the shelf server hardware with 48
cores.  As Jan-Simon Möller points out, being single-threaded and
single-process isn't much of an option any more.  If one is trying to
compress, say, a 4TB virtual disk image then using a little over 2% of
the available CPU time meaning you have to wait a week is going to be...
frustrating :)

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[oernii]

I'd like to note, that I use qemu-img to backup snapshots of images.
This works fine, it's just so slow. Of my 24 cores only 1 is used to
compress the image.

It could be so much faster.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Bernhard M. Wiedemann]

qcow2_write_compressed in block/qcow2.c would need to be changed. 
Currently it seems to need bigger changes as it always does compress+write for one block.
Not sure, how well it would handle multiple writes in parallel, so the safest would be to avoid that and just wait for the previous writer to finish before starting to write.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Thomas Huth]

** Changed in: qemu
   Importance: Undecided => Wishlist

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Quentin Casasnovas]

It looks like qcow2_write_compressed() has been removed and turned into
a qemu co-routine in qemu 2.8.0 (released in December 2017) to support
live compressed back-ups.  Any pointers to start working on this?  We
have servers with 128 CPUs and it's very sad to see them compress on a
single CPU and take tens of minutes instead of a few seconds.. :)

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Paolo Bonzini]

The fact that it's now a coroutine_fn doesn't change much, if anything
it makes it simpler to handle multiple writes in parallel.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Quentin Casasnovas]

That was also my feeling, so nice to get a confirmation!

Another related thing would be to allow qemu-nbd to write compressed
blocks its backing image - today if you use a qcow2 with compression,
any block which is written to gets uncompressed in the resulting image
and you need to recompress the image offline with qemu-img.

Would you have any pointers/documentation on how best to implement this
so both qemu-img and qemu-nbd can use multithreaded compressed writes ?
I'm totally new to qemu block subsystem.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Коренберг Марк]

@~quentin.casasnovas please report this as new feature request, instead
of adding comment to this one.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Jinank Jain]

@~quentin.casasnovas Are you still working on this? If not then I would
like to give this a shot?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Qemu-devel] [Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Quentin Casasnovas]

@Jinank I have not started working on this at all, so please go ahead!
Let me know if I can help with testing or anything, we make quite
extensive use of nbd and qcow2 images internally.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Thomas Huth]

The QEMU project is currently considering to move its bug tracking to
another system. For this we need to know which bugs are still valid
and which could be closed already. Thus we are setting older bugs to
"Incomplete" now.

If you still think this bug report here is valid, then please switch
the state back to "New" within the next 60 days, otherwise this report
will be marked as "Expired". Or please mark it as "Fix Released" if
the problem has been solved with a newer version of QEMU already.

Thank you and sorry for the inconvenience.


** Changed in: qemu
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  Incomplete

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Коренберг Марк]

** Changed in: qemu
       Status: Incomplete => New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  New

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions

[Bug 601946] Re: [Feature request] qemu-img multi-threaded compressed image conversion

[Thomas Huth]

This is an automated cleanup. This bug report has been moved to QEMU's
new bug tracker on gitlab.com and thus gets marked as 'expired' now.
Please continue with the discussion here:

 https://gitlab.com/qemu-project/qemu/-/issues/80


** Changed in: qemu
       Status: New => Expired

** Bug watch added: gitlab.com/qemu-project/qemu/-/issues #80
   https://gitlab.com/qemu-project/qemu/-/issues/80

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/601946

Title:
  [Feature request] qemu-img multi-threaded compressed image conversion

Status in QEMU:
  Expired

Bug description:
  Feature request:
  qemu-img multi-threaded compressed image conversion

  Suppose I want to convert raw image to compressed qcow2. Multi-
  threaded conversion will be much faster, because bottleneck is
  compressing data.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/601946/+subscriptions