[Qemu-devel] [Bug 1808928] [NEW] Bitmap Extra data is not supported

[Qemu-devel] [Bug 1808928] [NEW] Bitmap Extra data is not supported

[Ali Sag]

Public bug reported:

i am using dirty bitmaps and drive-backup. It works as aspected.

Lately, i encounter a disastrous error. There is not any information
about that situation. I cannot reach/open/attach/info or anything with a
qcow2 file.

virsh version
Compiled against library: libvirt 4.10.0
Using library: libvirt 4.10.0
Using API: QEMU 4.10.0
Running hypervisor: QEMU 2.12.0

"qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
extra data is not supported"

what is that mean? what should i do?
i cannot remove bitmap. i cannot open image or query.

** Affects: qemu
     Importance: Undecided
         Status: New


** Tags: bitmaps block dirt qcow2

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  New

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions

[Qemu-devel] [Bug 1808928] Re: Bitmap Extra data is not supported

[John Snow]

Hi, bitmap extensions have a field that allows us to attach
extra/arbitrary data to them. It is not currently used by QEMU. If this
field is set, it means something corrupted your qcow2.

Please make a backup of your qcow2 file first (because attempting to
repair a broken qcow2 can sometimes make it worse), and then please try
running qemu-img check to try and diagnose the image.

What happened to cause the "disastrous error", do you recall?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  New

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions

[Qemu-devel] [Bug 1808928] Re: Bitmap Extra data is not supported

[Ali Sag]

as far as i know nothing happened. it had worked normally while instance
was running. For a reason, instance is shutdown, then it never open
again. i have some backups, i tried to return previous backups. But they
also gave same error. Thanks to replication i could get it back. i
copied image from replication site...

as i said "qemu-img" command completely unusable on that image. 
There should be another mechanism. It blocks everything. 
I try to edit header to remove extra data. But i could not find header information on website. 

thanks

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  New

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions

[Qemu-devel] [Bug 1808928] Re: Bitmap Extra data is not supported

[John Snow]

OK, I think it is a bug that you cannot at least repair the image,
though the cause of corruption is still unknown to me. Do you have
libvirt/QEMU logs from when then VM was shut down, before it wouldn't
boot properly again?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  New

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions

[Qemu-devel] [Bug 1808928] Re: Bitmap Extra data is not supported

[Ali Sag]

Sorry, because of log time, VM logs were removed. Journalctl for libvirtd is remain.. It was like that.
018-12-17 17:01:50.990+0000: 43198: error : qemuMonitorIORead:606 : Unable to read from monitor: Connection reset by peer
Dec 17 20:01:50 vm-kvm09 libvirtd[43198]: 2018-12-17 17:01:50.991+0000: 43198: error : qemuProcessReportLogError:1935 : internal error: qemu unexpectedly closed the monitor: .guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio
Dec 17 20:01:51 vm-kvm09 libvirtd[43198]: 2018-12-17T17:01:50.984315Z qemu-kvm: -drive file=/var/lib/libvirt/images/exo1-jb-h01-2.qcow2,format=qcow2,if=none,id=drive-virtio-disk1,cache=none: Bitmap extra data is not supported
Dec 17 20:02:02 vm-kvm09 libvirtd[43198]: 2018-12-17 17:02:02.652+0000: 43203: warning : virQEMUCapsInit:923 : Failed to get host CPU cache info
Dec 17 20:02:02 vm-kvm09 libvirtd[43198]: 2018-12-17 17:02:02.668+0000: 68990: warning : virQEMUCapsInit:923 : Failed to get host CPU cache info
Dec 17 20:02:02 vm-kvm09 libvirtd[43198]: 2018-12-17 17:02:02.683+0000: 17099: warning : virQEMUCapsInit:923 : Failed to get host CPU cache info
Dec 17 20:09:58 vm-kvm09 libvirtd[43198]: 2018-12-17 17:09:58.700+0000: 68889: warning : virQEMUCapsInit:923 : Failed to get host CPU cache info
Dec 17 20:09:58 vm-kvm09 libvirtd[43198]: 2018-12-17 17:09:58.717+0000: 43200: warning : virQEMUCapsInit:923 : Failed to get host CPU cache info
Dec 17 20:09:58 vm-kvm09 libvirtd[43198]: 2018-12-17 17:09:58.732+0000: 43202: warning : virQEMUCapsInit:923 : Failed to get host CPU cache info
Dec 17 20:09:59 vm-kvm09 libvirtd[43198]: 2018-12-17 17:09:59.144+0000: 43198: error : qemuMonitorIORead:606 : Unable to read from monitor: Connection reset by peer
Dec 17 20:09:59 vm-kvm09 libvirtd[43198]: 2018-12-17 17:09:59.145+0000: 43198: error : qemuProcessReportLogError:1935 : internal error: qemu unexpectedly closed the monitor: 2018-12-17T17:09:59.132340Z qemu-kvm: -drive file=/var/lib/libvirt/images/exo1-jb-h01-2.qcow2,f...

I keep the broken image file. I run some qemu-img commands:

> qemu-img check /var/lib/libvirt/images/exo1-jb-h01-2.qcow2.1712
qemu-img: Could not open '/var/lib/libvirt/images/exo1-jb-h01-2.qcow2.1712': Bitmap extra data is not supported

> qemu-img info /var/lib/libvirt/images/exo1-jb-h01-2.qcow2.1712
qemu-img: Could not open '/var/lib/libvirt/images/exo1-jb-h01-2.qcow2.1712': Bitmap extra data is not supported

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  New

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions

[Bug 1808928] Re: Bitmap Extra data is not supported

[John Snow]

For now, QEMU cannot accept images with extra bitmap data, because QEMU
isn't aware of the semantics of those fields, so we cannot even allow
the image to load, just in case.

However, we SHOULD allow qemu-img check --repair to detect such bitmaps
as corruption so that images can be scrubbed and recovered. I will add
that to my TODO list.

Meanwhile, I believe the root cause of your problem here is a data
corruption event, but it's hard to tell exactly what it might be because
of the extra_data flag ... I can try to have some patches ready for you
in January that try to "ignore" the data and analyze the rest of the
image as best as possible, which might help us see what else went wrong.

** Changed in: qemu
     Assignee: (unassigned) => John Snow (jnsnow)

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  New

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions

[Bug 1808928] Re: Bitmap Extra data is not supported

[Thomas Huth]

John, did you find some spare time to work on those patches that you've
mentioned? I.e. has this been addressed already?

** Changed in: qemu
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  Incomplete

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions

[Bug 1808928] Re: Bitmap Extra data is not supported

[John Snow]

my patches went in, ultimately, and my focus was since shifted
elsewhere. I just tried this by *manually* adding some extra data to a
bitmap by hand.


qemu-img create -f qcow2 foo.qcow2 64m
qemu-img bitmap --add foo.qcow2 mybitmap

This creates a bitmap extension header like this (starting at 0x1f8)

000001f0  00 00 00 00 00 00 00 00  23 85 28 75 00 00 00 18  |........#.(u....|
00000200  00 00 00 01 00 00 00 00  00 00 00 00 00 00 00 20  |............... |
00000210  00 00 00 00 00 05 00 00                           |........        |

And a bitmap table that looks like this:

00050000  00 00 00 00 00 04 00 00  00 00 00 01 00 00 00 02  |................|
00050010  01 10 00 08 00 00 00 00  6d 79 62 69 74 6d 61 70  |........mybitmap|

I modified the bitmap table to add eight bytes of bad data:

00050000  00 00 00 00 00 04 00 00  00 00 00 01 00 00 00 02  |................|
00050010  01 10 00 08 00 00 00 08  62 61 64 64 61 74 61 21  |........baddata!|
00050020  6d 79 62 69 74 6d 61 70                           |mybitmap|

And modified the header accordingly to add eight bytes to the table
(0x20f := 0x28):

000001f0  00 00 00 00 00 00 00 00  23 85 28 75 00 00 00 18  |........#.(u....|
00000200  00 00 00 01 00 00 00 00  00 00 00 00 00 00 00 28  |...............(|
00000210  00 00 00 00 00 05 00 00                           |........        |

And in these cases, QEMU refuses to load or work with the image even
slightly, rendering you unable to remove it:

> ./qemu-img bitmap --remove foo.qcow2 mybitmap
qemu-img: Could not open 'foo.qcow2': Bitmap extra data is not supported

So, it's still an open issue.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  Incomplete

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions

[Bug 1808928] Re: Bitmap Extra data is not supported

[John Snow]

Sorry, that formatted *terribly*... please see the attachment for a raw
text version with arbitrarily long columns that looks nicer. :(

** Attachment added: "Raw text for comment #7 with better formatting (?)"
   https://bugs.launchpad.net/qemu/+bug/1808928/+attachment/5490362/+files/bztemp

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  Incomplete

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions

[Bug 1808928] Re: Bitmap Extra data is not supported

[Thomas Huth]

This is an automated cleanup. This bug report has been moved
to QEMU's new bug tracker on gitlab.com and thus gets marked
as 'expired' now. Please continue with the discussion here:

 https://gitlab.com/qemu-project/qemu/-/issues/58


** Changed in: qemu
       Status: Incomplete => Expired

** Changed in: qemu
     Assignee: John Snow (jnsnow) => (unassigned)

** Bug watch added: gitlab.com/qemu-project/qemu/-/issues #58
   https://gitlab.com/qemu-project/qemu/-/issues/58

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1808928

Title:
  Bitmap Extra data is not supported

Status in QEMU:
  Expired

Bug description:
  i am using dirty bitmaps and drive-backup. It works as aspected.

  Lately, i encounter a disastrous error. There is not any information
  about that situation. I cannot reach/open/attach/info or anything with
  a qcow2 file.

  virsh version
  Compiled against library: libvirt 4.10.0
  Using library: libvirt 4.10.0
  Using API: QEMU 4.10.0
  Running hypervisor: QEMU 2.12.0

  "qemu-img: Could not open '/var/lib/libvirt/images/test.qcow2': Bitmap
  extra data is not supported"

  what is that mean? what should i do?
  i cannot remove bitmap. i cannot open image or query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1808928/+subscriptions