From: Sergei Shtepa <sergei.shtepa@veeam.com>
To: <axboe@kernel.dk>, <linux-block@vger.kernel.org>,
<linux-kernel@vger.kernel.org>
Cc: Sergei Shtepa <sergei.shtepa@veeam.com>
Subject: [PATCH 00/20] blksnap - creating non-persistent snapshots for backup
Date: Mon, 13 Jun 2022 18:52:53 +0300 [thread overview]
Message-ID: <1655135593-1900-1-git-send-email-sergei.shtepa@veeam.com> (raw)
Hi all.
I suggest the blksnap kernel module for consideration. It allows to create
non-persistent snapshots of any block devices. The main purpose of such
snapshots is to create a backup of block devices.
A snapshot is created simultaneously for several block devices, ensuring
their mutual consistency in the backup.
A change tracker is implemented in the module. It allows to determine
which blocks were changed during the time between the last snapshot
created and any of the previous snapshots of the same generation.
This allows to implement incremental and differential backups.
An arbitrary range of sectors on any block device can be used to store
snapshot changes. The size of the storage for changes can be increased after the
snapshot is created by adding new sector ranges. This allows to create a
storage of differences in individual files on a file system that can occupy
the entire space of a block device and increase the storage of differences
as needed.
To create images of snapshots of block devices, the module stores blocks
of the original block device that have been changed since the snapshot was
taken. To do this, the module intercepts write requests and reads blocks
that need to be overwritten. This algorithm guarantees the safety of the
data of the original block device in case of overflow of the snapshot and
even in case of unpredictable critical errors.
To connect and disconnect the module to the block layer, the concept of a
block device filter is introduced. Functions for connecting filters are
added to the block layer and the ability to intercept I/O requests is
provided.
The blksnap module was created specifically for upstream based on the
experience of operating the out-of-tree veeamsnap module, which is part of
the Veeam Agent for Linux product. I am sure that the module will be in
demand by other creators of backup tools and will save them from having to
use their out-of-tree kernel modules.
A tool, a library for working with blksnap, tests and some documentations
can be found at www.github.com/veeam/blksnap.
Sergei Shtepa (20):
block, blk_filter: enable block device filters
block, blksnap: header file of the module interface
block, blksnap: module management interface functions
block, blksnap: init() and exit() functions
block, blksnap: interaction with sysfs
block, blksnap: attaching and detaching the filter and handling a bios
block, blksnap: map of change block tracking
block, blksnap: big buffer in the form of an array of pages
block, blksnap: minimum data storage unit of the original block device
block, blksnap: buffer in memory for the minimum data storage unit
block, blksnap: functions and structures for performing block I/O
operations
block, blksnap: storage for storing difference blocks
block, blksnap: event queue from the difference storage
block, blksnap: owner of information about overwritten blocks of the
original block device
block, blksnap: snapshot image block device
block, blksnap: snapshot
block, blksnap: debugging mechanism for monitoring memory consumption
block, blksnap: Kconfig
block, blksnap: Makefile
block, blksnap: adds a blksnap to the kernel tree
block/Kconfig | 8 +
block/bdev.c | 129 +++++
block/blk-core.c | 88 ++++
drivers/block/Kconfig | 2 +
drivers/block/Makefile | 1 +
drivers/block/blksnap/Kconfig | 101 ++++
drivers/block/blksnap/Makefile | 20 +
drivers/block/blksnap/big_buffer.c | 218 ++++++++
drivers/block/blksnap/big_buffer.h | 27 +
drivers/block/blksnap/cbt_map.c | 280 ++++++++++
drivers/block/blksnap/cbt_map.h | 112 ++++
drivers/block/blksnap/chunk.c | 352 +++++++++++++
drivers/block/blksnap/chunk.h | 129 +++++
drivers/block/blksnap/ctrl.c | 445 ++++++++++++++++
drivers/block/blksnap/ctrl.h | 7 +
drivers/block/blksnap/diff_area.c | 602 +++++++++++++++++++++
drivers/block/blksnap/diff_area.h | 179 +++++++
drivers/block/blksnap/diff_buffer.c | 146 ++++++
drivers/block/blksnap/diff_buffer.h | 78 +++
drivers/block/blksnap/diff_io.c | 204 ++++++++
drivers/block/blksnap/diff_io.h | 122 +++++
drivers/block/blksnap/diff_storage.c | 316 +++++++++++
drivers/block/blksnap/diff_storage.h | 94 ++++
drivers/block/blksnap/event_queue.c | 90 ++++
drivers/block/blksnap/event_queue.h | 64 +++
drivers/block/blksnap/main.c | 109 ++++
drivers/block/blksnap/memory_checker.c | 100 ++++
drivers/block/blksnap/memory_checker.h | 41 ++
drivers/block/blksnap/params.h | 10 +
drivers/block/blksnap/snapimage.c | 345 ++++++++++++
drivers/block/blksnap/snapimage.h | 65 +++
drivers/block/blksnap/snapshot.c | 671 ++++++++++++++++++++++++
drivers/block/blksnap/snapshot.h | 76 +++
drivers/block/blksnap/sysfs.c | 81 +++
drivers/block/blksnap/sysfs.h | 5 +
drivers/block/blksnap/tracker.c | 693 +++++++++++++++++++++++++
drivers/block/blksnap/tracker.h | 71 +++
drivers/block/blksnap/version.h | 8 +
include/linux/blk_snap.h | 460 ++++++++++++++++
include/linux/blk_types.h | 22 +
include/linux/blkdev.h | 81 +++
41 files changed, 6652 insertions(+)
create mode 100644 drivers/block/blksnap/Kconfig
create mode 100644 drivers/block/blksnap/Makefile
create mode 100644 drivers/block/blksnap/big_buffer.c
create mode 100644 drivers/block/blksnap/big_buffer.h
create mode 100644 drivers/block/blksnap/cbt_map.c
create mode 100644 drivers/block/blksnap/cbt_map.h
create mode 100644 drivers/block/blksnap/chunk.c
create mode 100644 drivers/block/blksnap/chunk.h
create mode 100644 drivers/block/blksnap/ctrl.c
create mode 100644 drivers/block/blksnap/ctrl.h
create mode 100644 drivers/block/blksnap/diff_area.c
create mode 100644 drivers/block/blksnap/diff_area.h
create mode 100644 drivers/block/blksnap/diff_buffer.c
create mode 100644 drivers/block/blksnap/diff_buffer.h
create mode 100644 drivers/block/blksnap/diff_io.c
create mode 100644 drivers/block/blksnap/diff_io.h
create mode 100644 drivers/block/blksnap/diff_storage.c
create mode 100644 drivers/block/blksnap/diff_storage.h
create mode 100644 drivers/block/blksnap/event_queue.c
create mode 100644 drivers/block/blksnap/event_queue.h
create mode 100644 drivers/block/blksnap/main.c
create mode 100644 drivers/block/blksnap/memory_checker.c
create mode 100644 drivers/block/blksnap/memory_checker.h
create mode 100644 drivers/block/blksnap/params.h
create mode 100644 drivers/block/blksnap/snapimage.c
create mode 100644 drivers/block/blksnap/snapimage.h
create mode 100644 drivers/block/blksnap/snapshot.c
create mode 100644 drivers/block/blksnap/snapshot.h
create mode 100644 drivers/block/blksnap/sysfs.c
create mode 100644 drivers/block/blksnap/sysfs.h
create mode 100644 drivers/block/blksnap/tracker.c
create mode 100644 drivers/block/blksnap/tracker.h
create mode 100644 drivers/block/blksnap/version.h
create mode 100644 include/linux/blk_snap.h
--
2.20.1
next reply other threads:[~2022-06-13 18:55 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-13 15:52 Sergei Shtepa [this message]
2022-06-13 15:52 ` [PATCH 01/20] block, blk_filter: enable block device filters Sergei Shtepa
2022-06-13 21:50 ` Randy Dunlap
2022-06-14 9:19 ` Sergei Shtepa
2022-06-14 9:21 ` Sergei Shtepa
2022-06-14 9:01 ` kernel test robot
2022-07-06 12:59 ` Christoph Hellwig
2022-07-07 8:26 ` Sergei Shtepa
2022-07-07 17:26 ` Christoph Hellwig
2022-07-08 10:45 ` Sergei Shtepa
2022-07-13 11:56 ` Christoph Hellwig
2022-07-13 13:47 ` Sergei Shtepa
2022-07-14 5:12 ` Christoph Hellwig
2022-07-14 9:22 ` Sergei Shtepa
2022-06-13 15:52 ` [PATCH 02/20] block, blksnap: header file of the module interface Sergei Shtepa
2022-07-06 13:03 ` Christoph Hellwig
2022-06-13 15:52 ` [PATCH 03/20] block, blksnap: module management interface functions Sergei Shtepa
2022-06-13 22:44 ` Chaitanya Kulkarni
2022-06-13 15:52 ` [PATCH 04/20] block, blksnap: init() and exit() functions Sergei Shtepa
2022-06-13 15:52 ` [PATCH 05/20] block, blksnap: interaction with sysfs Sergei Shtepa
2022-06-13 15:52 ` [PATCH 06/20] block, blksnap: attaching and detaching the filter and handling a bios Sergei Shtepa
2022-06-13 15:53 ` [PATCH 07/20] block, blksnap: map of change block tracking Sergei Shtepa
2022-06-13 15:53 ` [PATCH 08/20] block, blksnap: big buffer in the form of an array of pages Sergei Shtepa
2022-07-06 13:09 ` Christoph Hellwig
2022-06-13 15:53 ` [PATCH 09/20] block, blksnap: minimum data storage unit of the original block device Sergei Shtepa
2022-06-13 15:53 ` [PATCH 10/20] block, blksnap: buffer in memory for the minimum data storage unit Sergei Shtepa
2022-06-13 15:53 ` [PATCH 11/20] block, blksnap: functions and structures for performing block I/O operations Sergei Shtepa
2022-07-07 17:33 ` Christoph Hellwig
2022-06-13 15:53 ` [PATCH 12/20] block, blksnap: storage for storing difference blocks Sergei Shtepa
2022-06-13 15:53 ` [PATCH 13/20] block, blksnap: event queue from the difference storage Sergei Shtepa
2022-06-13 15:53 ` [PATCH 14/20] block, blksnap: owner of information about overwritten blocks of the original block device Sergei Shtepa
2022-06-13 15:53 ` [PATCH 15/20] block, blksnap: snapshot image " Sergei Shtepa
2022-07-06 13:13 ` Christoph Hellwig
2022-07-07 9:16 ` Sergei Shtepa
2022-07-07 17:24 ` Christoph Hellwig
2022-07-08 7:58 ` Sergei Shtepa
2022-07-08 8:04 ` Christoph Hellwig
2022-06-13 15:53 ` [PATCH 16/20] block, blksnap: snapshot Sergei Shtepa
2022-06-13 15:53 ` [PATCH 17/20] block, blksnap: debugging mechanism for monitoring memory consumption Sergei Shtepa
2022-06-13 15:53 ` [PATCH 18/20] block, blksnap: Kconfig Sergei Shtepa
2022-06-13 21:38 ` Randy Dunlap
2022-07-06 13:06 ` Christoph Hellwig
2022-06-13 15:53 ` [PATCH 19/20] block, blksnap: Makefile Sergei Shtepa
2022-07-06 13:06 ` Christoph Hellwig
2022-06-13 15:53 ` [PATCH 20/20] block, blksnap: adds a blksnap to the kernel tree Sergei Shtepa
2022-06-18 2:11 ` kernel test robot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1655135593-1900-1-git-send-email-sergei.shtepa@veeam.com \
--to=sergei.shtepa@veeam.com \
--cc=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.