From: Nikos Tsironis <ntsironis@arrikto.com>
To: Chaitanya Kulkarni <chaitanyak@nvidia.com>
Cc: "djwong@kernel.org" <djwong@kernel.org>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
"clm@fb.com" <clm@fb.com>,
"dm-devel@redhat.com" <dm-devel@redhat.com>,
"osandov@fb.com" <osandov@fb.com>,
"msnitzer@redhat.com >> msnitzer@redhat.com"
<msnitzer@redhat.com>, Bart Van Assche <bvanassche@acm.org>,
"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
Christoph Hellwig <hch@lst.de>,
"roland@purestorage.com" <roland@purestorage.com>,
"zach.brown@ni.com" <zach.brown@ni.com>,
"dsterba@suse.com" <dsterba@suse.com>,
"josef@toxicpanda.com" <josef@toxicpanda.com>,
"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"mpatocka@redhat.com" <mpatocka@redhat.com>,
"kbus >> Keith Busch" <kbusch@kernel.org>,
"Frederick.Knight@netapp.com" <Frederick.Knight@netapp.com>,
Jens Axboe <axboe@kernel.dk>, "tytso@mit.edu" <tytso@mit.edu>,
"martin.petersen@oracle.com >> Martin K. Petersen"
<martin.petersen@oracle.com>, "jack@suse.com" <jack@suse.com>,
linux-fsdevel <linux-fsdevel@vger.kernel.org>,
"lsf-pc@lists.linux-foundation.org"
<lsf-pc@lists.linux-foundation.org>
Subject: Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload
Date: Tue, 1 Mar 2022 19:34:01 +0200 [thread overview]
Message-ID: <012723a9-2e9c-c638-4944-fa560e1b0df0@arrikto.com> (raw)
In-Reply-To: <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com>
On 1/27/22 09:14, Chaitanya Kulkarni wrote:
> Hi,
>
> * Background :-
> -----------------------------------------------------------------------
>
> Copy offload is a feature that allows file-systems or storage devices
> to be instructed to copy files/logical blocks without requiring
> involvement of the local CPU.
>
> With reference to the RISC-V summit keynote [1] single threaded
> performance is limiting due to Denard scaling and multi-threaded
> performance is slowing down due Moore's law limitations. With the rise
> of SNIA Computation Technical Storage Working Group (TWG) [2],
> offloading computations to the device or over the fabrics is becoming
> popular as there are several solutions available [2]. One of the common
> operation which is popular in the kernel and is not merged yet is Copy
> offload over the fabrics or on to the device.
>
> * Problem :-
> -----------------------------------------------------------------------
>
> The original work which is done by Martin is present here [3]. The
> latest work which is posted by Mikulas [4] is not merged yet. These two
> approaches are totally different from each other. Several storage
> vendors discourage mixing copy offload requests with regular READ/WRITE
> I/O. Also, the fact that the operation fails if a copy request ever
> needs to be split as it traverses the stack it has the unfortunate
> side-effect of preventing copy offload from working in pretty much
> every common deployment configuration out there.
>
> * Current state of the work :-
> -----------------------------------------------------------------------
>
> With [3] being hard to handle arbitrary DM/MD stacking without
> splitting the command in two, one for copying IN and one for copying
> OUT. Which is then demonstrated by the [4] why [3] it is not a suitable
> candidate. Also, with [4] there is an unresolved problem with the
> two-command approach about how to handle changes to the DM layout
> between an IN and OUT operations.
>
> We have conducted a call with interested people late last year since
> lack of LSFMMM and we would like to share the details with broader
> community members.
>
> * Why Linux Kernel Storage System needs Copy Offload support now ?
> -----------------------------------------------------------------------
>
> With the rise of the SNIA Computational Storage TWG and solutions [2],
> existing SCSI XCopy support in the protocol, recent advancement in the
> Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer
> DMA support in the Linux Kernel mainly for NVMe devices [7] and
> eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit
> from Copy offload operation.
>
> With this background we have significant number of use-cases which are
> strong candidates waiting for outstanding Linux Kernel Block Layer Copy
> Offload support, so that Linux Kernel Storage subsystem can to address
> previously mentioned problems [1] and allow efficient offloading of the
> data related operations. (Such as move/copy etc.)
>
> For reference following is the list of the use-cases/candidates waiting
> for Copy Offload support :-
>
> 1. SCSI-attached storage arrays.
> 2. Stacking drivers supporting XCopy DM/MD.
> 3. Computational Storage solutions.
> 7. File systems :- Local, NFS and Zonefs.
> 4. Block devices :- Distributed, local, and Zoned devices.
> 5. Peer to Peer DMA support solutions.
> 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF.
>
> * What we will discuss in the proposed session ?
> -----------------------------------------------------------------------
>
> I'd like to propose a session to go over this topic to understand :-
>
> 1. What are the blockers for Copy Offload implementation ?
> 2. Discussion about having a file system interface.
> 3. Discussion about having right system call for user-space.
> 4. What is the right way to move this work forward ?
> 5. How can we help to contribute and move this work forward ?
>
> * Required Participants :-
> -----------------------------------------------------------------------
>
> I'd like to invite file system, block layer, and device drivers
> developers to:-
>
> 1. Share their opinion on the topic.
> 2. Share their experience and any other issues with [4].
> 3. Uncover additional details that are missing from this proposal.
>
> Required attendees :-
>
> Martin K. Petersen
> Jens Axboe
> Christoph Hellwig
> Bart Van Assche
> Zach Brown
> Roland Dreier
> Ric Wheeler
> Trond Myklebust
> Mike Snitzer
> Keith Busch
> Sagi Grimberg
> Hannes Reinecke
> Frederick Knight
> Mikulas Patocka
> Keith Busch
>
> -ck
>
> [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf
> [2] https://www.snia.org/computational
> https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/
> https://www.eideticom.com/products.html
> https://www.xilinx.com/applications/data-center/computational-storage.html
> [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy
> [4] https://www.spinics.net/lists/linux-block/msg00599.html
> [5] https://lwn.net/Articles/793585/
> [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned-
> namespaces-zns-as-go-to-industry-technology/
> [7] https://github.com/sbates130272/linux-p2pmem
> [8] https://kernel.dk/io_uring.pdf
I would like to participate in the discussion too.
The dm-clone target would also benefit from copy offload, as it heavily
employs dm-kcopyd. I have been exploring redesigning kcopyd in order to
achieve increased IOPS in dm-clone and dm-snapshot for small copies over
NVMe devices, but copy offload sounds even more promising, especially
for larger copies happening in the background (as is the case with
dm-clone's background hydration).
Thanks,
Nikos
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
next prev parent reply other threads:[~2022-03-01 17:34 UTC|newest]
Thread overview: 113+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CGME20220127071544uscas1p2f70f4d2509f3ebd574b7ed746d3fa551@uscas1p2.samsung.com>
[not found] ` <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com>
2022-01-28 19:59 ` [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Adam Manzanares
2022-01-31 11:49 ` Johannes Thumshirn
2022-01-31 19:03 ` Bart Van Assche
2022-02-01 1:54 ` Luis Chamberlain
2022-02-02 5:57 ` Kanchan Joshi
[not found] ` <20220201102122.4okwj2gipjbvuyux@mpHalley-2>
2022-02-01 18:31 ` [dm-devel] [RFC PATCH 0/3] NVMe copy offload patches Mikulas Patocka
2022-02-01 18:32 ` [dm-devel] [RFC PATCH 1/3] block: add copy offload support Mikulas Patocka
2022-02-01 19:18 ` Bart Van Assche
2022-02-03 18:50 ` Mikulas Patocka
2022-02-03 20:11 ` Keith Busch
2022-02-03 22:49 ` Bart Van Assche
2022-02-04 12:09 ` Mikulas Patocka
2022-02-04 13:34 ` Jens Axboe
2022-02-02 16:21 ` Keith Busch
2022-02-02 16:40 ` Mikulas Patocka
2022-02-02 18:40 ` Knight, Frederick
2022-02-01 18:33 ` [dm-devel] [RFC PATCH 2/3] nvme: " Mikulas Patocka
2022-02-01 19:18 ` Bart Van Assche
2022-02-01 19:25 ` Mikulas Patocka
2022-02-01 18:33 ` [dm-devel] [RFC PATCH 3/3] nvme: add the "debug" host driver Mikulas Patocka
2022-02-02 6:01 ` Adam Manzanares
2022-02-03 16:06 ` Luis Chamberlain
2022-02-03 16:15 ` Christoph Hellwig
2022-02-03 19:34 ` Luis Chamberlain
2022-02-03 19:46 ` Adam Manzanares
2022-02-03 20:57 ` Mikulas Patocka
2022-02-03 22:52 ` Adam Manzanares
2022-02-04 3:00 ` Chaitanya Kulkarni
2022-02-04 3:05 ` Chaitanya Kulkarni
2022-02-02 8:00 ` Chaitanya Kulkarni
2022-02-02 12:38 ` Klaus Jensen
2022-02-03 15:38 ` Luis Chamberlain
2022-02-03 16:52 ` Keith Busch
2022-02-03 19:50 ` Adam Manzanares
2022-02-04 3:12 ` Chaitanya Kulkarni
2022-02-04 6:28 ` Damien Le Moal
2022-02-04 7:58 ` Chaitanya Kulkarni
2022-02-04 8:24 ` Javier González
2022-02-04 9:58 ` Chaitanya Kulkarni
2022-02-04 11:34 ` Javier González
2022-02-04 14:15 ` Hannes Reinecke
2022-02-04 14:24 ` Keith Busch
2022-02-04 16:01 ` Christoph Hellwig
2022-02-04 19:41 ` [dm-devel] [RFC PATCH 0/3] NVMe copy offload patches Nitesh Shetty
[not found] ` <CGME20220207141901epcas5p162ec2387815be7a1fd67ce0ab7082119@epcas5p1.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 00/10] Add Copy offload support Nitesh Shetty
[not found] ` <CGME20220207141908epcas5p4f270c89fc32434ea8b525fa973098231@epcas5p4.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 01/10] block: make bio_map_kern() non static Nitesh Shetty
[not found] ` <CGME20220207141913epcas5p4d41cb549b7cca1ede5c7a66bbd110da0@epcas5p4.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 02/10] block: Introduce queue limits for copy-offload support Nitesh Shetty
2022-02-08 7:01 ` Damien Le Moal
2022-02-08 18:43 ` Nitesh Shetty
[not found] ` <CGME20220207141918epcas5p4f9badc0c3f3f0913f091c850d2d3bd81@epcas5p4.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 03/10] block: Add copy offload support infrastructure Nitesh Shetty
2022-02-07 22:45 ` kernel test robot
2022-02-07 23:26 ` kernel test robot
2022-02-08 7:21 ` Damien Le Moal
2022-02-09 10:22 ` Nitesh Shetty
2022-02-09 7:48 ` Dan Carpenter
2022-02-09 10:32 ` Nitesh Shetty
[not found] ` <CGME20220207141924epcas5p26ad9cf5de732224f408aded12ed0a577@epcas5p2.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 04/10] block: Introduce a new ioctl for copy Nitesh Shetty
2022-02-09 3:39 ` kernel test robot
[not found] ` <CGME20220207141930epcas5p2bcbff65f78ad1dede64648d73ddb3770@epcas5p2.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 05/10] block: add emulation " Nitesh Shetty
2022-02-08 3:20 ` kernel test robot
2022-02-16 13:32 ` Mikulas Patocka
2022-02-17 13:18 ` Nitesh Shetty
[not found] ` <CGME20220207141937epcas5p2bd57ae35056c69b3e2f9ee2348d6af19@epcas5p2.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 06/10] nvme: add copy support Nitesh Shetty
2022-02-10 7:08 ` kernel test robot
[not found] ` <CGME20220207141942epcas5p4bda894a5833513c9211dcecc7928a951@epcas5p4.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 07/10] nvmet: add copy command support for bdev and file ns Nitesh Shetty
2022-02-07 18:10 ` kernel test robot
2022-02-07 20:12 ` kernel test robot
2022-02-10 8:31 ` kernel test robot
2022-02-11 7:52 ` Dan Carpenter
[not found] ` <CGME20220207141948epcas5p4534f6bdc5a1e2e676d7d09c04f8b4a5b@epcas5p4.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 08/10] dm: Add support for copy offload Nitesh Shetty
2022-02-16 13:51 ` Mikulas Patocka
2022-02-24 12:42 ` Nitesh Shetty
2022-02-25 9:12 ` Mikulas Patocka
[not found] ` <CGME20220207141953epcas5p32ccc3c0bbe642cea074edefcc32302a5@epcas5p3.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 09/10] dm: Enable copy offload for dm-linear target Nitesh Shetty
[not found] ` <CGME20220207141958epcas5p25f1cd06726217696d13c2dfbea010565@epcas5p2.samsung.com>
2022-02-07 14:13 ` [dm-devel] [PATCH v2 10/10] dm kcopyd: use copy offload support Nitesh Shetty
2022-02-07 9:57 ` [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Nitesh Shetty
2022-02-07 10:45 ` David Disseldorp
2022-03-01 17:34 ` Nikos Tsironis [this message]
[not found] ` <c4124f39-1ee9-8f34-e731-42315fee15f9@nvidia.com>
2022-03-03 18:36 ` Nikos Tsironis
2022-03-08 20:48 ` Nikos Tsironis
2022-03-09 8:51 ` Mikulas Patocka
2022-03-09 15:49 ` Nikos Tsironis
2021-05-11 0:15 Chaitanya Kulkarni
2021-05-11 21:15 ` Knight, Frederick
2021-05-12 2:21 ` Bart Van Assche
[not found] ` <CGME20210512071321eucas1p2ca2253e90449108b9f3e4689bf8e0512@eucas1p2.samsung.com>
2021-05-12 7:13 ` Javier González
2021-05-12 7:30 ` Johannes Thumshirn
[not found] ` <CGME20210928191342eucas1p23448dcd51b23495fa67cdc017e77435c@eucas1p2.samsung.com>
2021-09-28 19:13 ` Javier González
2021-09-29 6:44 ` Johannes Thumshirn
2021-09-30 9:43 ` Chaitanya Kulkarni
2021-09-30 9:53 ` Javier González
2021-10-06 10:01 ` Javier González
2021-10-13 8:35 ` Javier González
2021-09-30 16:20 ` Bart Van Assche
2021-10-06 10:05 ` Javier González
2021-10-06 17:33 ` Bart Van Assche
[not found] ` <20211008064925.oyjxbmngghr2yovr@mpHalley.local>
2021-10-29 0:21 ` Chaitanya Kulkarni
2021-10-29 5:51 ` Hannes Reinecke
2021-10-29 8:16 ` Javier González
2021-10-29 16:15 ` Bart Van Assche
2021-11-01 17:54 ` Keith Busch
2021-10-29 8:14 ` Javier González
2021-11-03 19:27 ` Javier González
[not found] ` <20211116134324.hbs3tp5proxootd7@ArmHalley.localdomain>
2021-11-16 17:59 ` Bart Van Assche
[not found] ` <20211117125224.z36hp2crpj4fwngc@ArmHalley.local>
2021-11-17 15:52 ` Bart Van Assche
[not found] ` <CA+1E3rJRT+89OCyqRtb5BFbez0BfkKvCGijd=nObMEB3_v6MyA@mail.gmail.com>
2021-11-19 16:21 ` Bart Van Assche
2021-05-12 7:36 ` Erwin van Londen
2021-05-12 15:23 ` Hannes Reinecke
2021-05-12 15:45 ` Himanshu Madhani
2021-05-17 16:39 ` Kanchan Joshi
2021-05-18 0:15 ` Bart Van Assche
2021-06-11 6:03 ` Chaitanya Kulkarni
2021-06-11 15:35 ` Nikos Tsironis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=012723a9-2e9c-c638-4944-fa560e1b0df0@arrikto.com \
--to=ntsironis@arrikto.com \
--cc=Frederick.Knight@netapp.com \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=chaitanyak@nvidia.com \
--cc=clm@fb.com \
--cc=djwong@kernel.org \
--cc=dm-devel@redhat.com \
--cc=dsterba@suse.com \
--cc=hch@lst.de \
--cc=jack@suse.com \
--cc=josef@toxicpanda.com \
--cc=kbusch@kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-scsi@vger.kernel.org \
--cc=lsf-pc@lists.linux-foundation.org \
--cc=martin.petersen@oracle.com \
--cc=mpatocka@redhat.com \
--cc=msnitzer@redhat.com \
--cc=osandov@fb.com \
--cc=roland@purestorage.com \
--cc=tytso@mit.edu \
--cc=zach.brown@ni.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).