Re: [Qemu-devel] [Bug 1662050] Re: qemu-img convert a overlay qcow2 image into a entire image

[Stefan Hajnoczi <stefanha@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4862 bytes --]

On Tue, Feb 14, 2017 at 02:17:16AM -0000, wayen wrote:
> ** Attachment added: "qemu-img map new.qcow2 output"
>    https://bugs.launchpad.net/qemu/+bug/1662050/+attachment/4818564/+files/qemu_img_map_new_qcow2.txt

Thanks for posting the attachments.

I ran a script to find unallocated clusters in the delta.qcow2 host
file.  Most are actually qcow2 metadata (L1/L2 tables, refcount blocks).

This output shows that any image file size reduction you are hoping to
achieve can only come from zero clusters.

There are no holes in the files that would result in significant image
file size reduction if a new image were written out.

Just wanted to share this info in case anyone else is thinking about how
to optimize qcow2 files.  I still think rewriting images sequentially
can be useful - if internal snapshots were used and deleted then COW can
result in holes.

Hole at 0 size 5.0 clusters
Hole at 393216 size 1.0 clusters
Hole at 589824 size 1.0 clusters
Hole at 1114112 size 1.0 clusters
Hole at 1310720 size 1.0 clusters
Hole at 1507328 size 1.0 clusters
Hole at 1703936 size 1.0 clusters
Hole at 2293760 size 1.0 clusters
Hole at 2621440 size 1.0 clusters
Hole at 3080192 size 1.0 clusters
Hole at 5111808 size 1.0 clusters
Hole at 6291456 size 1.0 clusters
Hole at 30408704 size 1.0 clusters
Hole at 47906816 size 1.0 clusters
Hole at 142671872 size 1.0 clusters
Hole at 219545600 size 1.0 clusters
Hole at 667090944 size 1.0 clusters
Hole at 853868544 size 1.0 clusters
Hole at 1562640384 size 1.0 clusters
Hole at 2147483648 size 1.0 clusters
Hole at 2617180160 size 1.0 clusters
Hole at 3411148800 size 1.0 clusters
Hole at 4107075584 size 1.0 clusters
Hole at 4294967296 size 1.0 clusters
Hole at 4452646912 size 1.0 clusters
Hole at 4792057856 size 1.0 clusters
Hole at 5494865920 size 1.0 clusters
Hole at 5645271040 size 1.0 clusters
Hole at 5702483968 size 1.0 clusters
Hole at 6187188224 size 1.0 clusters
Hole at 6442450944 size 1.0 clusters
Hole at 6862995456 size 1.0 clusters
Hole at 6987317248 size 1.0 clusters
Hole at 7567245312 size 1.0 clusters
Hole at 8135245824 size 1.0 clusters
Hole at 8590589952 size 1.0 clusters
Hole at 8613462016 size 1.0 clusters
Hole at 9055436800 size 1.0 clusters
Hole at 9703522304 size 1.0 clusters
Hole at 10279321600 size 1.0 clusters
Hole at 10737418240 size 3.0 clusters
Hole at 10844372992 size 1.0 clusters
Hole at 11167858688 size 1.0 clusters
Hole at 11209605120 size 1.0 clusters
Hole at 11209801728 size 1.0 clusters
Hole at 11730944000 size 1.0 clusters
Hole at 12183207936 size 1.0 clusters
Hole at 12705464320 size 1.0 clusters
Hole at 12884901888 size 1.0 clusters
Hole at 13444120576 size 1.0 clusters
Hole at 13910016000 size 1.0 clusters
Hole at 14182711296 size 1.0 clusters
Hole at 15025635328 size 1.0 clusters
Hole at 15032385536 size 1.0 clusters

The following script draws the allocated clusters and holes in the image
file.  I took your qemu-img map output, filtered out any lines with
base.qcow2, and sorted using sort -k3 -g to sort on the "Mapped to"
field.  Then I ran ./qcow2-map-svg.py <filtered.txt >output.svg.

#!/usr/bin/python3
import sys
import io

COLOR_ALLOCATED = '#ffaaaa'
COLOR_HOLE = '#999999'

def svg_percentage(value, total):
    return '{0}%'.format(100.0 * value / total)

def svg_rect(x, width, color):
    print('<rect x="{0}" y="0" width="{1}" height="40" fill="{2}" stroke="none" />'.format(x, width, color), file=out)

def svg_text(x, y, text):
    print('<text x="{0}" y="{1}">{2}</text>'.format(x, y, text), file=out)

out = io.StringIO()

print('''<?xml version="1.0" encoding="UTF-8" ?>
<svg xmlns="http://www.w3.org/2000/svg" version="1.1">''', file=out)

file_map = []
header = True
for line in sys.stdin:
    if header:
        header = False
        continue

    offset, length, mapped, filename = line.split()
    offset = int(offset, 16)
    length = int(length, 16)
    mapped = int(mapped, 16)

    file_map.append((offset, length, mapped, filename))

file_size = file_map[-1][2] + file_map[-1][1]
last_mapped = 0
for _, length, mapped, _ in file_map:
    if mapped > last_mapped:
#        if mapped - last_mapped:
#            print('Hole at {0} size {1} clusters'.format(last_mapped, (mapped - last_mapped) / 65536))
        svg_rect(svg_percentage(last_mapped, file_size),
                 svg_percentage(mapped - last_mapped, file_size),
                 COLOR_HOLE)

    svg_rect(svg_percentage(mapped, file_size),
             svg_percentage(length, file_size),
             COLOR_ALLOCATED)
    last_mapped = mapped + length
if last_mapped < file_size:
    svg_rect(svg_percentage(last_mapped, file_size),
             svg_percentage(file_size - last_mapped, file_size),
             COLOR_HOLE)

for i in range(10):
    svg_text(svg_percentage(i, 10), 60, svg_percentage(i, 10))

print('</svg>', file=out)

print(out.getvalue())

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]