From mboxrd@z Thu Jan 1 00:00:00 1970 From: Fam Zheng Subject: Re: [Qemu-devel] [libvirt] Looking for project ideas and mentors for Google Summer of Code 2014 Date: Fri, 7 Feb 2014 15:01:27 +0800 Message-ID: <20140207070127.GA17846@T430.nay.redhat.com> References: <52F0DF5C.9000500@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Michal Privoznik , Kevin Wolf , Peter Maydell , Peter Crosthwaite , kvm , "libvir-list@redhat.com" , qemu-devel , Max Reitz , Anthony Liguori , Paolo Bonzini , Andreas =?iso-8859-1?Q?F=E4rber?= , Richard Henderson To: Stefan Hajnoczi Return-path: Received: from mx1.redhat.com ([209.132.183.28]:45831 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750708AbaBGHBc (ORCPT ); Fri, 7 Feb 2014 02:01:32 -0500 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Thu, 02/06 13:27, Stefan Hajnoczi wrote: > On Tue, Feb 4, 2014 at 1:38 PM, Michal Privoznik wrote: > > On 03.02.2014 08:45, Stefan Hajnoczi wrote: > >> > >> KVM & libvirt: you are welcome to join the QEMU umbrella organization > >> like last year. > >> > > > > I've updated wiki with a libvirt idea. But I can sense more to come later as > > I have some time to think about it :) > > Great, thanks! > > I have added my QEMU block layer ideas too. > > Deadline for the organization application is 14th of February. As > part of the application form we need to show our list of ideas. > Thanks for posting your project ideas so that our list is ready. > I'd like to add persistent dirty bitmap as an idea but I seem to have no account on wiki, so I'll just reply here, please help with review and update the page if it makes sense. (Who could create an account for me, BTW?) Project Idea Below ================== Incremental backup of block images ---------------------------------- Summary: Implement persistent incremental image backup. Users want to do regular backup of VM image data to protect data from unexpected loss. Incremental backup is a backup strategy that only copies out the "new data" that is changed since previous backup, to reduce the overhead of backup and improve the storage utilization. To track which part of guest data is changed, QEMU needs to store image's "dirty bitmap" on the disk as well as the image data itself. The task is to implement a new block driver (a filter) to load/store this persistent dirty bitmap file, and maintain the dirty bits while the guest writes to the data image. As a prerequisite, you also need to make the design of this bitmap file format. Then, design test cases and write scripts to test the driver. The persistent bitmap file must contain: 0. Magic bits to identify the format of this file. 1. Bitmap granularity (e.g. 64 KB) 2. The actual bitmap (1 TB disk @ 64 KB granularity = 2 MB bitmap) 3. Flags including a "clean" flag. The "clean" flag is used to tell whether the persistent bitmap file is safe to use again. When QEMU opens the persistent dirty bitmap, it clears the "clean" flag. When QEMU deactivates and finishes writing out the dirty bitmap, it sets the "clean" flag. If the QEMU process crashes it is not safe to trust the dirty bitmap; a full backup must be performed. Make use of this flag in the driver to limit the performance overhead. Links: [http://en.wikipedia.org/wiki/Incremental_backup Incremental backup] [http://lists.nongnu.org/archive/html/qemu-devel/2014-01/msg02156.html QMP: Introduce incremental drive-backup with in-memory dirty bitmap] Details: Skill level: intermediate Language: C Mentors: Fam Zheng (fam on IRC), Stefan Hajnoczi (stefanha on IRC) Thanks, Fam From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59609) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WBfRP-0002DY-2t for qemu-devel@nongnu.org; Fri, 07 Feb 2014 02:01:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WBfRH-0002Px-QT for qemu-devel@nongnu.org; Fri, 07 Feb 2014 02:01:35 -0500 Received: from mx1.redhat.com ([209.132.183.28]:56374) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WBfRH-0002Po-JD for qemu-devel@nongnu.org; Fri, 07 Feb 2014 02:01:27 -0500 Date: Fri, 7 Feb 2014 15:01:27 +0800 From: Fam Zheng Message-ID: <20140207070127.GA17846@T430.nay.redhat.com> References: <52F0DF5C.9000500@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Subject: Re: [Qemu-devel] [libvirt] Looking for project ideas and mentors for Google Summer of Code 2014 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: Kevin Wolf , Peter Maydell , Peter Crosthwaite , kvm , "libvir-list@redhat.com" , Michal Privoznik , qemu-devel , Max Reitz , Anthony Liguori , Paolo Bonzini , Andreas =?iso-8859-1?Q?F=E4rber?= , Richard Henderson On Thu, 02/06 13:27, Stefan Hajnoczi wrote: > On Tue, Feb 4, 2014 at 1:38 PM, Michal Privoznik wrote: > > On 03.02.2014 08:45, Stefan Hajnoczi wrote: > >> > >> KVM & libvirt: you are welcome to join the QEMU umbrella organization > >> like last year. > >> > > > > I've updated wiki with a libvirt idea. But I can sense more to come later as > > I have some time to think about it :) > > Great, thanks! > > I have added my QEMU block layer ideas too. > > Deadline for the organization application is 14th of February. As > part of the application form we need to show our list of ideas. > Thanks for posting your project ideas so that our list is ready. > I'd like to add persistent dirty bitmap as an idea but I seem to have no account on wiki, so I'll just reply here, please help with review and update the page if it makes sense. (Who could create an account for me, BTW?) Project Idea Below ================== Incremental backup of block images ---------------------------------- Summary: Implement persistent incremental image backup. Users want to do regular backup of VM image data to protect data from unexpected loss. Incremental backup is a backup strategy that only copies out the "new data" that is changed since previous backup, to reduce the overhead of backup and improve the storage utilization. To track which part of guest data is changed, QEMU needs to store image's "dirty bitmap" on the disk as well as the image data itself. The task is to implement a new block driver (a filter) to load/store this persistent dirty bitmap file, and maintain the dirty bits while the guest writes to the data image. As a prerequisite, you also need to make the design of this bitmap file format. Then, design test cases and write scripts to test the driver. The persistent bitmap file must contain: 0. Magic bits to identify the format of this file. 1. Bitmap granularity (e.g. 64 KB) 2. The actual bitmap (1 TB disk @ 64 KB granularity = 2 MB bitmap) 3. Flags including a "clean" flag. The "clean" flag is used to tell whether the persistent bitmap file is safe to use again. When QEMU opens the persistent dirty bitmap, it clears the "clean" flag. When QEMU deactivates and finishes writing out the dirty bitmap, it sets the "clean" flag. If the QEMU process crashes it is not safe to trust the dirty bitmap; a full backup must be performed. Make use of this flag in the driver to limit the performance overhead. Links: [http://en.wikipedia.org/wiki/Incremental_backup Incremental backup] [http://lists.nongnu.org/archive/html/qemu-devel/2014-01/msg02156.html QMP: Introduce incremental drive-backup with in-memory dirty bitmap] Details: Skill level: intermediate Language: C Mentors: Fam Zheng (fam on IRC), Stefan Hajnoczi (stefanha on IRC) Thanks, Fam