* [PATCH 0/1] Block snapshot module and block layer filter API
@ 2020-10-02 12:56 Sergei Shtepa
[not found] ` <1601643362-7370-2-git-send-email-sergei.shtepa@veeam.com>
0 siblings, 1 reply; 2+ messages in thread
From: Sergei Shtepa @ 2020-10-02 12:56 UTC (permalink / raw)
To: axboe, mchehab+huawei, davem, robh, koct9i, damien.lemoal, jack,
ming.lei, steve, linux-kernel, linux-kbuild, linux-block
Cc: Sergei Shtepa
Hello everyone! Requesting for your comments and suggestions.
We propose a new kernel module - blk-snap.
This module implements snapshot and changed block tracking functionality.
It is intended to create backup copies of any block devices without usage
of device-mapper.
Snapshots are temporary and are destroyed after the backup process has
finished. Changed block tracking allows for incremental and differential
backup copies.
blk-snap uses block layer filter API.
Block layer filter API provides a callback to intercept bio-requests.
If a block device disappears for whatever reason, send a synchronous
request to remove the device from filtering.
Previously, we have already offered a patch with the implementation of the
block layer filter api (https://patchwork.kernel.org/patch/11741447/).
Link to that discussion:
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2290127.html.
Now we also offer the in-tree module.
blk-snap kernel module is a product of a deep refactoring of the
out-of-tree kernel veeamsnap (https://github.com/veeam/veeamsnap/) module:
* all conditional compilation branches that served for the purpose of
compatibility with older kernels have been removed;
* linux kernel code style has been applied;
* blk-snap mostly takes advantage of the existing kernel code instead of
reinventing the wheel;
* all redundant code (such as persistent cbt and snapstore collector) has
been removed.
We would appreciate your feedback!
Several important things are still have to be done:
* refactoring the module interface for interaction with a user-space code,
it is already clear that the implementation of some calls can be improved;
* haven't yet tested the build on architectures other than x86_64;
* the user-space library is not ready yet and tool to control the module
is to be created;
* autotests need to be created;
* regression testing has been conducted in a cursory manner, so there is
a chance for mistakes.
Sergei Shtepa (1):
blk-snap - Block snapshot module This module implements snapshot and
changed block tracking functionality. It is intended to create
backup copies of any block devices without usage of device-mapper.
MAINTAINERS | 14 +
block/Kconfig | 11 +
block/Makefile | 1 +
block/blk-core.c | 11 +-
block/blk-filter-internal.h | 34 +
block/blk-filter.c | 288 ++++++
block/genhd.c | 24 +
drivers/block/Kconfig | 6 +-
drivers/block/Makefile | 3 +-
drivers/block/blk-snap/Kconfig | 24 +
drivers/block/blk-snap/Makefile | 29 +
drivers/block/blk-snap/big_buffer.c | 193 ++++
drivers/block/blk-snap/big_buffer.h | 24 +
drivers/block/blk-snap/blk-snap-ctl.h | 190 ++++
drivers/block/blk-snap/blk_deferred.c | 566 +++++++++++
drivers/block/blk-snap/blk_deferred.h | 67 ++
drivers/block/blk-snap/blk_descr_file.c | 82 ++
drivers/block/blk-snap/blk_descr_file.h | 26 +
drivers/block/blk-snap/blk_descr_mem.c | 66 ++
drivers/block/blk-snap/blk_descr_mem.h | 14 +
drivers/block/blk-snap/blk_descr_multidev.c | 86 ++
drivers/block/blk-snap/blk_descr_multidev.h | 25 +
drivers/block/blk-snap/blk_descr_pool.c | 190 ++++
drivers/block/blk-snap/blk_descr_pool.h | 38 +
drivers/block/blk-snap/blk_redirect.c | 507 ++++++++++
drivers/block/blk-snap/blk_redirect.h | 73 ++
drivers/block/blk-snap/blk_util.c | 33 +
drivers/block/blk-snap/blk_util.h | 10 +
drivers/block/blk-snap/cbt_map.c | 210 +++++
drivers/block/blk-snap/cbt_map.h | 62 ++
drivers/block/blk-snap/common.h | 29 +
drivers/block/blk-snap/ctrl_fops.c | 693 ++++++++++++++
drivers/block/blk-snap/ctrl_fops.h | 19 +
drivers/block/blk-snap/ctrl_pipe.c | 562 +++++++++++
drivers/block/blk-snap/ctrl_pipe.h | 34 +
drivers/block/blk-snap/ctrl_sysfs.c | 73 ++
drivers/block/blk-snap/ctrl_sysfs.h | 5 +
drivers/block/blk-snap/defer_io.c | 393 ++++++++
drivers/block/blk-snap/defer_io.h | 39 +
drivers/block/blk-snap/filter.c | 72 ++
drivers/block/blk-snap/filter.h | 7 +
drivers/block/blk-snap/main.c | 83 ++
drivers/block/blk-snap/params.c | 57 ++
drivers/block/blk-snap/params.h | 29 +
drivers/block/blk-snap/rangevector.c | 85 ++
drivers/block/blk-snap/rangevector.h | 31 +
drivers/block/blk-snap/snapimage.c | 982 ++++++++++++++++++++
drivers/block/blk-snap/snapimage.h | 16 +
drivers/block/blk-snap/snapshot.c | 228 +++++
drivers/block/blk-snap/snapshot.h | 17 +
drivers/block/blk-snap/snapstore.c | 929 ++++++++++++++++++
drivers/block/blk-snap/snapstore.h | 68 ++
drivers/block/blk-snap/snapstore_device.c | 532 +++++++++++
drivers/block/blk-snap/snapstore_device.h | 63 ++
drivers/block/blk-snap/snapstore_file.c | 52 ++
drivers/block/blk-snap/snapstore_file.h | 15 +
drivers/block/blk-snap/snapstore_mem.c | 89 ++
drivers/block/blk-snap/snapstore_mem.h | 20 +
drivers/block/blk-snap/snapstore_multidev.c | 118 +++
drivers/block/blk-snap/snapstore_multidev.h | 22 +
drivers/block/blk-snap/tracker.c | 521 +++++++++++
drivers/block/blk-snap/tracker.h | 50 +
drivers/block/blk-snap/tracking.c | 260 ++++++
drivers/block/blk-snap/tracking.h | 12 +
drivers/block/blk-snap/version.h | 7 +
include/linux/blk-filter.h | 41 +
include/linux/genhd.h | 3 +-
67 files changed, 9157 insertions(+), 6 deletions(-)
create mode 100644 block/blk-filter-internal.h
create mode 100644 block/blk-filter.c
create mode 100644 drivers/block/blk-snap/Kconfig
create mode 100644 drivers/block/blk-snap/Makefile
create mode 100644 drivers/block/blk-snap/big_buffer.c
create mode 100644 drivers/block/blk-snap/big_buffer.h
create mode 100644 drivers/block/blk-snap/blk-snap-ctl.h
create mode 100644 drivers/block/blk-snap/blk_deferred.c
create mode 100644 drivers/block/blk-snap/blk_deferred.h
create mode 100644 drivers/block/blk-snap/blk_descr_file.c
create mode 100644 drivers/block/blk-snap/blk_descr_file.h
create mode 100644 drivers/block/blk-snap/blk_descr_mem.c
create mode 100644 drivers/block/blk-snap/blk_descr_mem.h
create mode 100644 drivers/block/blk-snap/blk_descr_multidev.c
create mode 100644 drivers/block/blk-snap/blk_descr_multidev.h
create mode 100644 drivers/block/blk-snap/blk_descr_pool.c
create mode 100644 drivers/block/blk-snap/blk_descr_pool.h
create mode 100644 drivers/block/blk-snap/blk_redirect.c
create mode 100644 drivers/block/blk-snap/blk_redirect.h
create mode 100644 drivers/block/blk-snap/blk_util.c
create mode 100644 drivers/block/blk-snap/blk_util.h
create mode 100644 drivers/block/blk-snap/cbt_map.c
create mode 100644 drivers/block/blk-snap/cbt_map.h
create mode 100644 drivers/block/blk-snap/common.h
create mode 100644 drivers/block/blk-snap/ctrl_fops.c
create mode 100644 drivers/block/blk-snap/ctrl_fops.h
create mode 100644 drivers/block/blk-snap/ctrl_pipe.c
create mode 100644 drivers/block/blk-snap/ctrl_pipe.h
create mode 100644 drivers/block/blk-snap/ctrl_sysfs.c
create mode 100644 drivers/block/blk-snap/ctrl_sysfs.h
create mode 100644 drivers/block/blk-snap/defer_io.c
create mode 100644 drivers/block/blk-snap/defer_io.h
create mode 100644 drivers/block/blk-snap/filter.c
create mode 100644 drivers/block/blk-snap/filter.h
create mode 100644 drivers/block/blk-snap/main.c
create mode 100644 drivers/block/blk-snap/params.c
create mode 100644 drivers/block/blk-snap/params.h
create mode 100644 drivers/block/blk-snap/rangevector.c
create mode 100644 drivers/block/blk-snap/rangevector.h
create mode 100644 drivers/block/blk-snap/snapimage.c
create mode 100644 drivers/block/blk-snap/snapimage.h
create mode 100644 drivers/block/blk-snap/snapshot.c
create mode 100644 drivers/block/blk-snap/snapshot.h
create mode 100644 drivers/block/blk-snap/snapstore.c
create mode 100644 drivers/block/blk-snap/snapstore.h
create mode 100644 drivers/block/blk-snap/snapstore_device.c
create mode 100644 drivers/block/blk-snap/snapstore_device.h
create mode 100644 drivers/block/blk-snap/snapstore_file.c
create mode 100644 drivers/block/blk-snap/snapstore_file.h
create mode 100644 drivers/block/blk-snap/snapstore_mem.c
create mode 100644 drivers/block/blk-snap/snapstore_mem.h
create mode 100644 drivers/block/blk-snap/snapstore_multidev.c
create mode 100644 drivers/block/blk-snap/snapstore_multidev.h
create mode 100644 drivers/block/blk-snap/tracker.c
create mode 100644 drivers/block/blk-snap/tracker.h
create mode 100644 drivers/block/blk-snap/tracking.c
create mode 100644 drivers/block/blk-snap/tracking.h
create mode 100644 drivers/block/blk-snap/version.h
create mode 100644 include/linux/blk-filter.h
--
2.28.0
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [PATCH 1/1] blk-snap - Block snapshot module This module implements snapshot and changed block tracking functionality. It is intended to create backup copies of any block devices without usage of device-mapper.
[not found] ` <CALYGNiORw=DrKxoaoQPJP8TNM-S_W0Zdtpvra_eMs+Ri5f2P-g@mail.gmail.com>
@ 2020-10-06 8:43 ` Sergei Shtepa
0 siblings, 0 replies; 2+ messages in thread
From: Sergei Shtepa @ 2020-10-06 8:43 UTC (permalink / raw)
To: Konstantin Khlebnikov
Cc: Jens Axboe, mchehab+huawei, David Miller, robh, damien.lemoal,
Jan Kara, ming.lei, steve, Linux Kernel Mailing List,
linux-kbuild, linux-block
Thanks for the answer.
Unfortunately, blk-rq-qos cannot be used efficiently for this purpose.
blk-rq-qos is good for collecting request queue processing metrics.
The level at which the interception is performed is too low - it happens
after the device driver has already received the request for processing.
For the filter to work efficiently, we need to ensure that the interception
is performed on a higher level. It is required to put processing of
multiple BIOs on hold while COW algorithm is being executed for them.
We must not be blocking the request processing queue, and also we would
like to avoid impacting the IO scheduler operations.
--
Sergei Shtepa
Veeam Software developer.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2020-10-06 8:50 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-02 12:56 [PATCH 0/1] Block snapshot module and block layer filter API Sergei Shtepa
[not found] ` <1601643362-7370-2-git-send-email-sergei.shtepa@veeam.com>
[not found] ` <CALYGNiORw=DrKxoaoQPJP8TNM-S_W0Zdtpvra_eMs+Ri5f2P-g@mail.gmail.com>
2020-10-06 8:43 ` [PATCH 1/1] blk-snap - Block snapshot module This module implements snapshot and changed block tracking functionality. It is intended to create backup copies of any block devices without usage of device-mapper Sergei Shtepa
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).