qcow2 becomes 37P in size while qemu crashes

qcow2 becomes 37P in size while qemu crashes

[Chris Murphy]

Here is the bug write up so far, which contains most of the relevant details.
https://bugzilla.redhat.com/show_bug.cgi?id=1359325

Here are three teasers to get you to look at the bug:

1.
[root@f24m ~]# ls -lsh /var/lib/libvirt/images
total 57G
1.5G -rw-r-----. 1 qemu qemu 1.5G Jul 21 10:54
Fedora-Workstation-Live-x86_64-24-1.2.iso
1.4G -rw-r--r--. 1 qemu qemu 1.4G Jul 20 13:28
Fedora-Workstation-Live-x86_64-Rawhide-20160718.n.0.iso
4.4G -rw-r-----. 1 qemu qemu 4.4G Jul 22 10:43
openSUSE-Leap-42.2-DVD-x86_64-Build0109-Media.iso
 50G -rw-r--r--. 1 root root  37P Jul 22 13:23 uefi_opensuseleap42.2a3-1.qcow2
196K -rw-r--r--. 1 root root 193K Jul 22 08:46 uefi_opensuseleap42.2a3-2.qcow2
[root@f24m ~]#

Yes, it's using 50G worth of sectors on the drive. But then it's 37
Petabytes?! That's really weird.

[root@f24m ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       104G   67G   36G  65% /

2.
Btrfs mounts, reads, writes, just fine, no messages in dmesg other
than the usual mount messages; all before, during, and after the qemu
crash, and rebooting. I rebooted to do an offline btrfs check, which
has no complaints. Scrub has no complaints. Yes the qcow2 has +C xattr
set so there's no independent way to determine if/hoe it's corrupt.
But qemu-img does say it's corrupt and libvirt will not start the VM
anymore with this qcow2 attached.

3.
I've attached to the bug a filefrag -v output from the 37 P file,
which has ~900 extents. There's only one thing that's a bit out of the
ordinary, which is mentioned in the bug.

Pretty weird. To try to reproduce this I kinda need to delete that
qcow2 file. So if anyone has suggestions on what other information to
put in that bug report before I change the state of the system, lemme
know.

-- 
Chris Murphy

Re: qcow2 becomes 37P in size while qemu crashes

[Chris Murphy]

Using btrfs-debug-tree, I'm finding something a bit odd about some of
the items in this 39P file.

Seems normal

    item 71 key (994163 EXTENT_DATA 43689967616) itemoff 12467 itemsize 53
        extent data disk byte 617349906432 nr 80805888
        extent data offset 0 nr 80805888 ram 80805888
        extent compression(none)

Seems not normal

    item 58 key (994163 EXTENT_DATA 38345535488) itemoff 13156 itemsize 53
        extent data disk byte 0 nr 0
        extent data offset 394752000 nr 61440 ram 34626881742770176
        extent compression(none)

There are quite a large number of items that take the 2nd form, and in
each case the ram value is the same as above.


-- 
Chris Murphy

Re: qcow2 becomes 37P in size while qemu crashes

[Chris Murphy]

Kindof a rudimentary way of making a debug tree smaller, by filtering
by inode...

[root@f24m images]# btrfs-debug-tree -t 583 /dev/sda5 | grep -A 50 -B
50 994163 > inode994163_39PBfile.txt

381K gzip file
https://drive.google.com/open?id=0B_2Asp8DGjJ9Rk1qMG54MUNDWkE

But I'm going to rm this file so I can see if the problem is reproducible.

---

Chris Murphy

Re: qcow2 becomes 37P in size while qemu crashes

[Chris Mason]

On 07/23/2016 04:05 PM, Chris Murphy wrote:
> Using btrfs-debug-tree, I'm finding something a bit odd about some of
> the items in this 39P file.
>
> Seems normal
>
>     item 71 key (994163 EXTENT_DATA 43689967616) itemoff 12467 itemsize 53
>         extent data disk byte 617349906432 nr 80805888
>         extent data offset 0 nr 80805888 ram 80805888
>         extent compression(none)
>
> Seems not normal
>
>     item 58 key (994163 EXTENT_DATA 38345535488) itemoff 13156 itemsize 53
>         extent data disk byte 0 nr 0
>         extent data offset 394752000 nr 61440 ram 34626881742770176
>         extent compression(none)
>
> There are quite a large number of items that take the 2nd form, and in
> each case the ram value is the same as above.
>
>

Can't really blame a bit flip for that one.  Looks like our hole 
truncation math has gone crazy there.  I'll see what I can find, but 
please yell if you can reproduce.

Thanks!

-chris

Re: qcow2 becomes 37P in size while qemu crashes

[Chris Murphy]

On Mon, Aug 1, 2016 at 9:26 AM, Chris Mason <clm@fb.com> wrote:
>
>
> On 07/23/2016 04:05 PM, Chris Murphy wrote:
>>
>> Using btrfs-debug-tree, I'm finding something a bit odd about some of
>> the items in this 39P file.
>>
>> Seems normal
>>
>>     item 71 key (994163 EXTENT_DATA 43689967616) itemoff 12467 itemsize 53
>>         extent data disk byte 617349906432 nr 80805888
>>         extent data offset 0 nr 80805888 ram 80805888
>>         extent compression(none)
>>
>> Seems not normal
>>
>>     item 58 key (994163 EXTENT_DATA 38345535488) itemoff 13156 itemsize 53
>>         extent data disk byte 0 nr 0
>>         extent data offset 394752000 nr 61440 ram 34626881742770176
>>         extent compression(none)
>>
>> There are quite a large number of items that take the 2nd form, and in
>> each case the ram value is the same as above.
>>
>>
>
> Can't really blame a bit flip for that one.  Looks like our hole truncation
> math has gone crazy there.  I'll see what I can find, but please yell if you
> can reproduce.

I've been trying, but no joy. Several bugs are required before hitting this.

User bug: I inadvertently edited the machine using virsh to use the
same backing qcow2 file for vda and vdb.
virsh bug: No warning for this double usage.
virt-manager bug: Starts the VM without warning when two virtio
devices are pointed to the same backing file.

Corruption of the file is inevitable. But exactly when it grew to 37P
I'm not sure, so far reproducing this results in a file limited to the
create time specified file size.


-- 
Chris Murphy