Re: [Qemu-devel] [libvirt] Looking for project ideas and mentors for Google Summer of Code 2014

From: Fam Zheng <famz@redhat.com>
To: Stefan Hajnoczi <stefanha@gmail.com>
Cc: "Michal Privoznik" <mprivozn@redhat.com>,
	"Kevin Wolf" <kwolf@redhat.com>,
	"Peter Maydell" <peter.maydell@linaro.org>,
	"Peter Crosthwaite" <peter.crosthwaite@xilinx.com>,
	kvm <kvm@vger.kernel.org>,
	"libvir-list@redhat.com" <libvir-list@redhat.com>,
	qemu-devel <qemu-devel@nongnu.org>,
	"Max Reitz" <mreitz@redhat.com>,
	"Anthony Liguori" <aliguori@amazon.com>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"Andreas Färber" <afaerber@suse.de>,
	"Richard Henderson" <rth@twiddle.net>
Subject: Re: [Qemu-devel] [libvirt] Looking for project ideas and mentors for Google Summer of Code 2014
Date: Fri, 7 Feb 2014 15:01:27 +0800	[thread overview]
Message-ID: <20140207070127.GA17846@T430.nay.redhat.com> (raw)
In-Reply-To: <CAJSP0QWpuJ7QY1A7DnZJ3M8pQ65zUmpngxC6v37SAJMX4LTxSQ@mail.gmail.com>

On Thu, 02/06 13:27, Stefan Hajnoczi wrote:
> On Tue, Feb 4, 2014 at 1:38 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
> > On 03.02.2014 08:45, Stefan Hajnoczi wrote:
> >>
> >> KVM & libvirt: you are welcome to join the QEMU umbrella organization
> >> like last year.
> >>
> >
> > I've updated wiki with a libvirt idea. But I can sense more to come later as
> > I have some time to think about it :)
> 
> Great, thanks!
> 
> I have added my QEMU block layer ideas too.
> 
> Deadline for the organization application is 14th of February.  As
> part of the application form we need to show our list of ideas.
> Thanks for posting your project ideas so that our list is ready.
> 

I'd like to add persistent dirty bitmap as an idea but I seem to have no
account on wiki, so I'll just reply here, please help with review and update
the page if it makes sense. (Who could create an account for me, BTW?)

Project Idea Below
==================

Incremental backup of block images
----------------------------------

Summary: Implement persistent incremental image backup.

Users want to do regular backup of VM image data to protect data from
unexpected loss.  Incremental backup is a backup strategy that only copies out
the "new data" that is changed since previous backup, to reduce the overhead of
backup and improve the storage utilization. To track which part of guest data
is changed, QEMU needs to store image's "dirty bitmap" on the disk as well as
the image data itself.

The task is to implement a new block driver (a filter) to load/store this
persistent dirty bitmap file, and maintain the dirty bits while the guest
writes to the data image. As a prerequisite, you also need to make the design
of this bitmap file format. Then, design test cases and write scripts to test
the driver.

The persistent bitmap file must contain:

0. Magic bits to identify the format of this file.
1. Bitmap granularity (e.g. 64 KB)
2. The actual bitmap (1 TB disk @ 64 KB granularity = 2 MB bitmap)
3. Flags including a "clean" flag. The "clean" flag is used to tell whether the
   persistent bitmap file is safe to use again.  When QEMU opens the persistent
   dirty bitmap, it clears the "clean" flag.  When QEMU deactivates and
   finishes writing out the dirty bitmap, it sets the "clean" flag.  If the
   QEMU process crashes it is not safe to trust the dirty bitmap; a full backup
   must be performed. Make use of this flag in the driver to limit the
   performance overhead.

Links:
[http://en.wikipedia.org/wiki/Incremental_backup Incremental backup]
[http://lists.nongnu.org/archive/html/qemu-devel/2014-01/msg02156.html QMP:
Introduce incremental drive-backup with in-memory dirty bitmap]

Details:
Skill level: intermediate
Language: C
Mentors:  Fam Zheng <famz@redhat.com> (fam on IRC),
          Stefan Hajnoczi <stefanha@redhat.com> (stefanha on IRC)

Thanks,
Fam