* [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload @ 2021-05-11 0:15 Chaitanya Kulkarni 2021-05-11 21:15 ` Knight, Frederick ` (10 more replies) 0 siblings, 11 replies; 43+ messages in thread From: Chaitanya Kulkarni @ 2021-05-11 0:15 UTC (permalink / raw) To: linux-block, linux-scsi, linux-nvme, dm-devel, lsf-pc Cc: axboe, msnitzer, bvanassche, martin.petersen, osandov, roland, mpatocka, kbusch, rwheeler, hch, Frederick.Knight, zach.brown Hi, * Background :- ----------------------------------------------------------------------- Copy offload is a feature that allows file-systems or storage devices to be instructed to copy files/logical blocks without requiring involvement of the local CPU. With reference to the RISC-V summit keynote [1] single threaded performance is limiting due to Denard scaling and multi-threaded performance is slowing down due Moore's law limitations. With the rise of SNIA Computation Technical Storage Working Group (TWG) [2], offloading computations to the device or over the fabrics is becoming popular as there are several solutions available [2]. One of the common operation which is popular in the kernel and is not merged yet is Copy offload over the fabrics or on to the device. * Problem :- ----------------------------------------------------------------------- The original work which is done by Martin is present here [3]. The latest work which is posted by Mikulas [4] is not merged yet. These two approaches are totally different from each other. Several storage vendors discourage mixing copy offload requests with regular READ/WRITE I/O. Also, the fact that the operation fails if a copy request ever needs to be split as it traverses the stack it has the unfortunate side-effect of preventing copy offload from working in pretty much every common deployment configuration out there. * Current state of the work :- ----------------------------------------------------------------------- With [3] being hard to handle arbitrary DM/MD stacking without splitting the command in two, one for copying IN and one for copying OUT. Which is then demonstrated by the [4] why [3] it is not a suitable candidate. Also, with [4] there is an unresolved problem with the two-command approach about how to handle changes to the DM layout between an IN and OUT operations. * Why Linux Kernel Storage System needs Copy Offload support now ? ----------------------------------------------------------------------- With the rise of the SNIA Computational Storage TWG and solutions [2], existing SCSI XCopy support in the protocol, recent advancement in the Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer DMA support in the Linux Kernel mainly for NVMe devices [7] and eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit from Copy offload operation. With this background we have significant number of use-cases which are strong candidates waiting for outstanding Linux Kernel Block Layer Copy Offload support, so that Linux Kernel Storage subsystem can to address previously mentioned problems [1] and allow efficient offloading of the data related operations. (Such as move/copy etc.) For reference following is the list of the use-cases/candidates waiting for Copy Offload support :- 1. SCSI-attached storage arrays. 2. Stacking drivers supporting XCopy DM/MD. 3. Computational Storage solutions. 7. File systems :- Local, NFS and Zonefs. 4. Block devices :- Distributed, local, and Zoned devices. 5. Peer to Peer DMA support solutions. 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. * What we will discuss in the proposed session ? ----------------------------------------------------------------------- I'd like to propose a session to go over this topic to understand :- 1. What are the blockers for Copy Offload implementation ? 2. Discussion about having a file system interface. 3. Discussion about having right system call for user-space. 4. What is the right way to move this work forward ? 5. How can we help to contribute and move this work forward ? * Required Participants :- ----------------------------------------------------------------------- I'd like to invite file system, block layer, and device drivers developers to:- 1. Share their opinion on the topic. 2. Share their experience and any other issues with [4]. 3. Uncover additional details that are missing from this proposal. Required attendees :- Martin K. Petersen Jens Axboe Christoph Hellwig Bart Van Assche Zach Brown Roland Dreier Ric Wheeler Trond Myklebust Mike Snitzer Keith Busch Sagi Grimberg Hannes Reinecke Frederick Knight Mikulas Patocka Keith Busch Regards, Chaitanya [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf [2] https://www.snia.org/computational https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/ https://www.eideticom.com/products.html https://www.xilinx.com/applications/data-center/computational-storage.html [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy [4] https://www.spinics.net/lists/linux-block/msg00599.html [5] https://lwn.net/Articles/793585/ [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- namespaces-zns-as-go-to-industry-technology/ [7] https://github.com/sbates130272/linux-p2pmem [8] https://kernel.dk/io_uring.pdf -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni @ 2021-05-11 21:15 ` Knight, Frederick 2021-05-12 2:21 ` Bart Van Assche ` (9 subsequent siblings) 10 siblings, 0 replies; 43+ messages in thread From: Knight, Frederick @ 2021-05-11 21:15 UTC (permalink / raw) To: Chaitanya Kulkarni, linux-block, linux-scsi, linux-nvme, dm-devel, lsf-pc Cc: axboe, msnitzer, bvanassche, martin.petersen, osandov, roland, mpatocka, kbusch, rwheeler, hch, zach.brown I'd love to participate in this discussion. You mention the 2 different models (single command vs. multi-command). Just as a reminder, there are specific reasons for those 2 different models. Some applications know both the source and the destination, so can use the single command model (the application is aware it is doing a copy). But, there is a group of applications that do NOT know both pieces of information at the same time, in the same thread, in the same context (the application is NOT aware it is doing a copy - the application thinks it is doing reads and writes). That is why there are 2 different models - because the application engineers didn't want to change their application. So, the author of the CP application (the shell copy command) wanted to use the existing READ / WRITE model (2 commands). Just replace the READ with "get the data ready" and replace the WRITE with "use the data you got ready". It was easier for that application to use the existing model, rather than totally redesigning the application. But, other application engineers had a code base that already knew a copy was happening, and their code already knew both the source and destination in the same code path. A BACKUP application is one that generally fits into this camp. So, it was easier for that application to replace that function with a single copy request. Another application was a VM mastering/replicating application that could spin up new VM images very quickly - the source and destination are known to be able to use a single request. When this offload journey began, both interfaces were needed and used. But yes, it did bifurcate the space, creating 2 camps of engineers - each with their favorite method (based on the application where they planned to use it). Each camp of engineers often sees no reason that the other camp can't just switch to do it the way they do - if they'd only see the light. But, originally, there were 2 different sets of requirements that each drove a specific design of a copy offload model. Even NVMe has recently joined the copy offload camp with a new COPY command (single namespace, multiple source ranges, single destination range - works well for defrag, and other use cases). I'm confident its capabilities will grow over time. SO, I think this will be a great discussion to have!!! Fred Knight -----Original Message----- From: Chaitanya Kulkarni <Chaitanya.Kulkarni@wdc.com> Sent: Monday, May 10, 2021 8:16 PM To: linux-block@vger.kernel.org; linux-scsi@vger.kernel.org; linux-nvme@lists.infradead.org; dm-devel@redhat.com; lsf-pc@lists.linux-foundation.org Cc: axboe@kernel.dk; msnitzer@redhat.com; bvanassche@acm.org; martin.petersen@oracle.com; roland@purestorage.com; mpatocka@redhat.com; Hannes Reinecke <hare@suse.de>; kbusch@kernel.org; rwheeler@redhat.com; hch@lst.de; Knight, Frederick <Frederick.Knight@netapp.com>; zach.brown@ni.com; osandov@fb.com Subject: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi, * Background :- ----------------------------------------------------------------------- Copy offload is a feature that allows file-systems or storage devices to be instructed to copy files/logical blocks without requiring involvement of the local CPU. With reference to the RISC-V summit keynote [1] single threaded performance is limiting due to Denard scaling and multi-threaded performance is slowing down due Moore's law limitations. With the rise of SNIA Computation Technical Storage Working Group (TWG) [2], offloading computations to the device or over the fabrics is becoming popular as there are several solutions available [2]. One of the common operation which is popular in the kernel and is not merged yet is Copy offload over the fabrics or on to the device. * Problem :- ----------------------------------------------------------------------- The original work which is done by Martin is present here [3]. The latest work which is posted by Mikulas [4] is not merged yet. These two approaches are totally different from each other. Several storage vendors discourage mixing copy offload requests with regular READ/WRITE I/O. Also, the fact that the operation fails if a copy request ever needs to be split as it traverses the stack it has the unfortunate side-effect of preventing copy offload from working in pretty much every common deployment configuration out there. * Current state of the work :- ----------------------------------------------------------------------- With [3] being hard to handle arbitrary DM/MD stacking without splitting the command in two, one for copying IN and one for copying OUT. Which is then demonstrated by the [4] why [3] it is not a suitable candidate. Also, with [4] there is an unresolved problem with the two-command approach about how to handle changes to the DM layout between an IN and OUT operations. * Why Linux Kernel Storage System needs Copy Offload support now ? ----------------------------------------------------------------------- With the rise of the SNIA Computational Storage TWG and solutions [2], existing SCSI XCopy support in the protocol, recent advancement in the Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer DMA support in the Linux Kernel mainly for NVMe devices [7] and eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit from Copy offload operation. With this background we have significant number of use-cases which are strong candidates waiting for outstanding Linux Kernel Block Layer Copy Offload support, so that Linux Kernel Storage subsystem can to address previously mentioned problems [1] and allow efficient offloading of the data related operations. (Such as move/copy etc.) For reference following is the list of the use-cases/candidates waiting for Copy Offload support :- 1. SCSI-attached storage arrays. 2. Stacking drivers supporting XCopy DM/MD. 3. Computational Storage solutions. 7. File systems :- Local, NFS and Zonefs. 4. Block devices :- Distributed, local, and Zoned devices. 5. Peer to Peer DMA support solutions. 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. * What we will discuss in the proposed session ? ----------------------------------------------------------------------- I'd like to propose a session to go over this topic to understand :- 1. What are the blockers for Copy Offload implementation ? 2. Discussion about having a file system interface. 3. Discussion about having right system call for user-space. 4. What is the right way to move this work forward ? 5. How can we help to contribute and move this work forward ? * Required Participants :- ----------------------------------------------------------------------- I'd like to invite file system, block layer, and device drivers developers to:- 1. Share their opinion on the topic. 2. Share their experience and any other issues with [4]. 3. Uncover additional details that are missing from this proposal. Required attendees :- Martin K. Petersen Jens Axboe Christoph Hellwig Bart Van Assche Zach Brown Roland Dreier Ric Wheeler Trond Myklebust Mike Snitzer Keith Busch Sagi Grimberg Hannes Reinecke Frederick Knight Mikulas Patocka Keith Busch Regards, Chaitanya [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf [2] https://www.snia.org/computational https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/ https://www.eideticom.com/products.html https://www.xilinx.com/applications/data-center/computational-storage.html [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy [4] https://www.spinics.net/lists/linux-block/msg00599.html [5] https://lwn.net/Articles/793585/ [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- namespaces-zns-as-go-to-industry-technology/ [7] https://github.com/sbates130272/linux-p2pmem [8] https://kernel.dk/io_uring.pdf -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni 2021-05-11 21:15 ` Knight, Frederick @ 2021-05-12 2:21 ` Bart Van Assche [not found] ` <CGME20210512071321eucas1p2ca2253e90449108b9f3e4689bf8e0512@eucas1p2.samsung.com> ` (8 subsequent siblings) 10 siblings, 0 replies; 43+ messages in thread From: Bart Van Assche @ 2021-05-12 2:21 UTC (permalink / raw) To: Chaitanya Kulkarni, linux-block, linux-scsi, linux-nvme, dm-devel, lsf-pc Cc: axboe, msnitzer, martin.petersen, osandov, roland, mpatocka, kbusch, rwheeler, hch, Frederick.Knight, zach.brown On 5/10/21 5:15 PM, Chaitanya Kulkarni wrote: > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? Are there any blockers left? My understanding is that what is needed is to implement what has been proposed recently (https://lore.kernel.org/linux-nvme/yq1blf3smcl.fsf@ca-mkp.ca.oracle.com/). Anyway, I'm interested to attend the conversation about this topic. Thanks, Bart. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
[parent not found: <CGME20210512071321eucas1p2ca2253e90449108b9f3e4689bf8e0512@eucas1p2.samsung.com>]
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <CGME20210512071321eucas1p2ca2253e90449108b9f3e4689bf8e0512@eucas1p2.samsung.com> @ 2021-05-12 7:13 ` Javier González 0 siblings, 0 replies; 43+ messages in thread From: Javier González @ 2021-05-12 7:13 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: axboe, roland, msnitzer, SelvaKumar S, bvanassche, linux-scsi, linux-nvme, osandov, linux-block, dm-devel, mpatocka, Kanchan Joshi, martin.petersen, kbusch, lsf-pc, rwheeler, hch, Frederick.Knight, zach.brown On 11.05.2021 00:15, Chaitanya Kulkarni wrote: >Hi, > >* Background :- >----------------------------------------------------------------------- > >Copy offload is a feature that allows file-systems or storage devices >to be instructed to copy files/logical blocks without requiring >involvement of the local CPU. > >With reference to the RISC-V summit keynote [1] single threaded >performance is limiting due to Denard scaling and multi-threaded >performance is slowing down due Moore's law limitations. With the rise >of SNIA Computation Technical Storage Working Group (TWG) [2], >offloading computations to the device or over the fabrics is becoming >popular as there are several solutions available [2]. One of the common >operation which is popular in the kernel and is not merged yet is Copy >offload over the fabrics or on to the device. > >* Problem :- >----------------------------------------------------------------------- > >The original work which is done by Martin is present here [3]. The >latest work which is posted by Mikulas [4] is not merged yet. These two >approaches are totally different from each other. Several storage >vendors discourage mixing copy offload requests with regular READ/WRITE >I/O. Also, the fact that the operation fails if a copy request ever >needs to be split as it traverses the stack it has the unfortunate >side-effect of preventing copy offload from working in pretty much >every common deployment configuration out there. > >* Current state of the work :- >----------------------------------------------------------------------- > >With [3] being hard to handle arbitrary DM/MD stacking without >splitting the command in two, one for copying IN and one for copying >OUT. Which is then demonstrated by the [4] why [3] it is not a suitable >candidate. Also, with [4] there is an unresolved problem with the >two-command approach about how to handle changes to the DM layout >between an IN and OUT operations. > >* Why Linux Kernel Storage System needs Copy Offload support now ? >----------------------------------------------------------------------- > >With the rise of the SNIA Computational Storage TWG and solutions [2], >existing SCSI XCopy support in the protocol, recent advancement in the >Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer >DMA support in the Linux Kernel mainly for NVMe devices [7] and >eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit >from Copy offload operation. > >With this background we have significant number of use-cases which are >strong candidates waiting for outstanding Linux Kernel Block Layer Copy >Offload support, so that Linux Kernel Storage subsystem can to address >previously mentioned problems [1] and allow efficient offloading of the >data related operations. (Such as move/copy etc.) > >For reference following is the list of the use-cases/candidates waiting >for Copy Offload support :- > >1. SCSI-attached storage arrays. >2. Stacking drivers supporting XCopy DM/MD. >3. Computational Storage solutions. >7. File systems :- Local, NFS and Zonefs. >4. Block devices :- Distributed, local, and Zoned devices. >5. Peer to Peer DMA support solutions. >6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. > >* What we will discuss in the proposed session ? >----------------------------------------------------------------------- > >I'd like to propose a session to go over this topic to understand :- > >1. What are the blockers for Copy Offload implementation ? >2. Discussion about having a file system interface. >3. Discussion about having right system call for user-space. >4. What is the right way to move this work forward ? >5. How can we help to contribute and move this work forward ? > >* Required Participants :- >----------------------------------------------------------------------- > >I'd like to invite file system, block layer, and device drivers >developers to:- > >1. Share their opinion on the topic. >2. Share their experience and any other issues with [4]. >3. Uncover additional details that are missing from this proposal. > >Required attendees :- > >Martin K. Petersen >Jens Axboe >Christoph Hellwig >Bart Van Assche >Zach Brown >Roland Dreier >Ric Wheeler >Trond Myklebust >Mike Snitzer >Keith Busch >Sagi Grimberg >Hannes Reinecke >Frederick Knight >Mikulas Patocka >Keith Busch > >Regards, >Chaitanya > >[1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf >[2] https://www.snia.org/computational >https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/ > https://www.eideticom.com/products.html >https://www.xilinx.com/applications/data-center/computational-storage.html >[3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy >[4] https://www.spinics.net/lists/linux-block/msg00599.html >[5] https://lwn.net/Articles/793585/ >[6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- >namespaces-zns-as-go-to-industry-technology/ >[7] https://github.com/sbates130272/linux-p2pmem >[8] https://kernel.dk/io_uring.pdf I would like to participate in this discussion too. Cc'in Selva and Kanchan, who have been posting several series for NVMe Simple Copy (SCC). Even though SCC is a very narrow use-case of copy-offload, it seems like a good start to start getting generic code in the block layer. Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni ` (2 preceding siblings ...) [not found] ` <CGME20210512071321eucas1p2ca2253e90449108b9f3e4689bf8e0512@eucas1p2.samsung.com> @ 2021-05-12 7:30 ` Johannes Thumshirn [not found] ` <CGME20210928191342eucas1p23448dcd51b23495fa67cdc017e77435c@eucas1p2.samsung.com> 2021-05-12 7:36 ` Erwin van Londen ` (6 subsequent siblings) 10 siblings, 1 reply; 43+ messages in thread From: Johannes Thumshirn @ 2021-05-12 7:30 UTC (permalink / raw) To: Chaitanya Kulkarni, linux-block, linux-scsi, linux-nvme, dm-devel, lsf-pc Cc: axboe, msnitzer, bvanassche, martin.petersen, osandov, roland, mpatocka, kbusch, rwheeler, hch, Frederick.Knight, zach.brown On 11/05/2021 02:15, Chaitanya Kulkarni wrote: > Hi, > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > > With reference to the RISC-V summit keynote [1] single threaded > performance is limiting due to Denard scaling and multi-threaded > performance is slowing down due Moore's law limitations. With the rise > of SNIA Computation Technical Storage Working Group (TWG) [2], > offloading computations to the device or over the fabrics is becoming > popular as there are several solutions available [2]. One of the common > operation which is popular in the kernel and is not merged yet is Copy > offload over the fabrics or on to the device. > > * Problem :- > ----------------------------------------------------------------------- > > The original work which is done by Martin is present here [3]. The > latest work which is posted by Mikulas [4] is not merged yet. These two > approaches are totally different from each other. Several storage > vendors discourage mixing copy offload requests with regular READ/WRITE > I/O. Also, the fact that the operation fails if a copy request ever > needs to be split as it traverses the stack it has the unfortunate > side-effect of preventing copy offload from working in pretty much > every common deployment configuration out there. > > * Current state of the work :- > ----------------------------------------------------------------------- > > With [3] being hard to handle arbitrary DM/MD stacking without > splitting the command in two, one for copying IN and one for copying > OUT. Which is then demonstrated by the [4] why [3] it is not a suitable > candidate. Also, with [4] there is an unresolved problem with the > two-command approach about how to handle changes to the DM layout > between an IN and OUT operations. > > * Why Linux Kernel Storage System needs Copy Offload support now ? > ----------------------------------------------------------------------- > > With the rise of the SNIA Computational Storage TWG and solutions [2], > existing SCSI XCopy support in the protocol, recent advancement in the > Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer > DMA support in the Linux Kernel mainly for NVMe devices [7] and > eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit > from Copy offload operation. > > With this background we have significant number of use-cases which are > strong candidates waiting for outstanding Linux Kernel Block Layer Copy > Offload support, so that Linux Kernel Storage subsystem can to address > previously mentioned problems [1] and allow efficient offloading of the > data related operations. (Such as move/copy etc.) > > For reference following is the list of the use-cases/candidates waiting > for Copy Offload support :- > > 1. SCSI-attached storage arrays. > 2. Stacking drivers supporting XCopy DM/MD. > 3. Computational Storage solutions. > 7. File systems :- Local, NFS and Zonefs. > 4. Block devices :- Distributed, local, and Zoned devices. > 5. Peer to Peer DMA support solutions. > 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. > > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > ----------------------------------------------------------------------- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. > > Required attendees :- > > Martin K. Petersen > Jens Axboe > Christoph Hellwig > Bart Van Assche > Zach Brown > Roland Dreier > Ric Wheeler > Trond Myklebust > Mike Snitzer > Keith Busch > Sagi Grimberg > Hannes Reinecke > Frederick Knight > Mikulas Patocka > Keith Busch > I would like to participate in this discussion as well. A generic block layer copy API is extremely helpful for filesystem garbage collection and copy operations like copy_file_range(). -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
[parent not found: <CGME20210928191342eucas1p23448dcd51b23495fa67cdc017e77435c@eucas1p2.samsung.com>]
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <CGME20210928191342eucas1p23448dcd51b23495fa67cdc017e77435c@eucas1p2.samsung.com> @ 2021-09-28 19:13 ` Javier González 2021-09-29 6:44 ` Johannes Thumshirn ` (2 more replies) 0 siblings, 3 replies; 43+ messages in thread From: Javier González @ 2021-09-28 19:13 UTC (permalink / raw) To: Johannes Thumshirn Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, bvanassche, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, lsf-pc, rwheeler On 12.05.2021 07:30, Johannes Thumshirn wrote: >On 11/05/2021 02:15, Chaitanya Kulkarni wrote: >> Hi, >> >> * Background :- >> ----------------------------------------------------------------------- >> >> Copy offload is a feature that allows file-systems or storage devices >> to be instructed to copy files/logical blocks without requiring >> involvement of the local CPU. >> >> With reference to the RISC-V summit keynote [1] single threaded >> performance is limiting due to Denard scaling and multi-threaded >> performance is slowing down due Moore's law limitations. With the rise >> of SNIA Computation Technical Storage Working Group (TWG) [2], >> offloading computations to the device or over the fabrics is becoming >> popular as there are several solutions available [2]. One of the common >> operation which is popular in the kernel and is not merged yet is Copy >> offload over the fabrics or on to the device. >> >> * Problem :- >> ----------------------------------------------------------------------- >> >> The original work which is done by Martin is present here [3]. The >> latest work which is posted by Mikulas [4] is not merged yet. These two >> approaches are totally different from each other. Several storage >> vendors discourage mixing copy offload requests with regular READ/WRITE >> I/O. Also, the fact that the operation fails if a copy request ever >> needs to be split as it traverses the stack it has the unfortunate >> side-effect of preventing copy offload from working in pretty much >> every common deployment configuration out there. >> >> * Current state of the work :- >> ----------------------------------------------------------------------- >> >> With [3] being hard to handle arbitrary DM/MD stacking without >> splitting the command in two, one for copying IN and one for copying >> OUT. Which is then demonstrated by the [4] why [3] it is not a suitable >> candidate. Also, with [4] there is an unresolved problem with the >> two-command approach about how to handle changes to the DM layout >> between an IN and OUT operations. >> >> * Why Linux Kernel Storage System needs Copy Offload support now ? >> ----------------------------------------------------------------------- >> >> With the rise of the SNIA Computational Storage TWG and solutions [2], >> existing SCSI XCopy support in the protocol, recent advancement in the >> Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer >> DMA support in the Linux Kernel mainly for NVMe devices [7] and >> eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit >> from Copy offload operation. >> >> With this background we have significant number of use-cases which are >> strong candidates waiting for outstanding Linux Kernel Block Layer Copy >> Offload support, so that Linux Kernel Storage subsystem can to address >> previously mentioned problems [1] and allow efficient offloading of the >> data related operations. (Such as move/copy etc.) >> >> For reference following is the list of the use-cases/candidates waiting >> for Copy Offload support :- >> >> 1. SCSI-attached storage arrays. >> 2. Stacking drivers supporting XCopy DM/MD. >> 3. Computational Storage solutions. >> 7. File systems :- Local, NFS and Zonefs. >> 4. Block devices :- Distributed, local, and Zoned devices. >> 5. Peer to Peer DMA support solutions. >> 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. >> >> * What we will discuss in the proposed session ? >> ----------------------------------------------------------------------- >> >> I'd like to propose a session to go over this topic to understand :- >> >> 1. What are the blockers for Copy Offload implementation ? >> 2. Discussion about having a file system interface. >> 3. Discussion about having right system call for user-space. >> 4. What is the right way to move this work forward ? >> 5. How can we help to contribute and move this work forward ? >> >> * Required Participants :- >> ----------------------------------------------------------------------- >> >> I'd like to invite file system, block layer, and device drivers >> developers to:- >> >> 1. Share their opinion on the topic. >> 2. Share their experience and any other issues with [4]. >> 3. Uncover additional details that are missing from this proposal. >> >> Required attendees :- >> >> Martin K. Petersen >> Jens Axboe >> Christoph Hellwig >> Bart Van Assche >> Zach Brown >> Roland Dreier >> Ric Wheeler >> Trond Myklebust >> Mike Snitzer >> Keith Busch >> Sagi Grimberg >> Hannes Reinecke >> Frederick Knight >> Mikulas Patocka >> Keith Busch >> > >I would like to participate in this discussion as well. A generic block layer >copy API is extremely helpful for filesystem garbage collection and copy operations >like copy_file_range(). Hi all, Since we are not going to be able to talk about this at LSF/MM, a few of us thought about holding a dedicated virtual discussion about Copy Offload. I believe we can use Chaitanya's thread as a start. Given the current state of the current patches, I would propose that we focus on the next step to get the minimal patchset that can go upstream so that we can build from there. Before we try to find a date and a time that fits most of us, who would be interested in participating? Thanks, Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-09-28 19:13 ` Javier González @ 2021-09-29 6:44 ` Johannes Thumshirn 2021-09-30 9:43 ` Chaitanya Kulkarni 2021-09-30 16:20 ` Bart Van Assche 2 siblings, 0 replies; 43+ messages in thread From: Johannes Thumshirn @ 2021-09-29 6:44 UTC (permalink / raw) To: Javier González Cc: Nitesh, Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, bvanassche, linux-scsi, hch, roland, Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Vincent, lsf-pc, rwheeler On 28/09/2021 21:13, Javier González wrote: > Since we are not going to be able to talk about this at LSF/MM, a few of > us thought about holding a dedicated virtual discussion about Copy > Offload. I believe we can use Chaitanya's thread as a start. Given the > current state of the current patches, I would propose that we focus on > the next step to get the minimal patchset that can go upstream so that > we can build from there. > > Before we try to find a date and a time that fits most of us, who would > be interested in participating? I'd definitively be interested in participating. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-09-28 19:13 ` Javier González 2021-09-29 6:44 ` Johannes Thumshirn @ 2021-09-30 9:43 ` Chaitanya Kulkarni 2021-09-30 9:53 ` Javier González 2021-10-06 10:01 ` Javier González 2021-09-30 16:20 ` Bart Van Assche 2 siblings, 2 replies; 43+ messages in thread From: Chaitanya Kulkarni @ 2021-09-30 9:43 UTC (permalink / raw) To: Javier González, Johannes Thumshirn Cc: Nitesh, Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, bvanassche, linux-scsi, hch, roland, Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Vincent, lsf-pc, rwheeler Javier, > > Hi all, > > Since we are not going to be able to talk about this at LSF/MM, a few of > us thought about holding a dedicated virtual discussion about Copy > Offload. I believe we can use Chaitanya's thread as a start. Given the > current state of the current patches, I would propose that we focus on > the next step to get the minimal patchset that can go upstream so that > we can build from there. > I agree with having a call as it has been two years I'm trying to have this discussion. Before we setup a call, please summarize following here :- 1. Exactly what work has been done so far. 2. What kind of feedback you got. 3. What are the exact blockers/objections. 4. Potential ways of moving forward. Although this all information is present in the mailing archives it is scattered all over the places, looking at the long CC list above we need to get the everyone on the same page in order to have a productive call. Once we have above discussion we can setup a precise agenda and assign slots. > Before we try to find a date and a time that fits most of us, who would > be interested in participating? > > Thanks, > Javier -ck -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-09-30 9:43 ` Chaitanya Kulkarni @ 2021-09-30 9:53 ` Javier González 2021-10-06 10:01 ` Javier González 1 sibling, 0 replies; 43+ messages in thread From: Javier González @ 2021-09-30 9:53 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, bvanassche, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 30.09.2021 09:43, Chaitanya Kulkarni wrote: >Javier, > >> >> Hi all, >> >> Since we are not going to be able to talk about this at LSF/MM, a few of >> us thought about holding a dedicated virtual discussion about Copy >> Offload. I believe we can use Chaitanya's thread as a start. Given the >> current state of the current patches, I would propose that we focus on >> the next step to get the minimal patchset that can go upstream so that >> we can build from there. >> > >I agree with having a call as it has been two years I'm trying to have >this discussion. > >Before we setup a call, please summarize following here :- > >1. Exactly what work has been done so far. >2. What kind of feedback you got. >3. What are the exact blockers/objections. >4. Potential ways of moving forward. > >Although this all information is present in the mailing archives it is >scattered all over the places, looking at the long CC list above we need >to get the everyone on the same page in order to have a productive call. > >Once we have above discussion we can setup a precise agenda and assign >slots. Sounds reasonable. Let me collect all this information and post it here. I will maintain a list of people that has showed interest on joining. For now: - Martin - Johannes - Fred - Chaitanya - Adam - Kanchan - Selva - Nitesh - Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-09-30 9:43 ` Chaitanya Kulkarni 2021-09-30 9:53 ` Javier González @ 2021-10-06 10:01 ` Javier González 2021-10-13 8:35 ` Javier González 1 sibling, 1 reply; 43+ messages in thread From: Javier González @ 2021-10-06 10:01 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, bvanassche, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 30.09.2021 09:43, Chaitanya Kulkarni wrote: >Javier, > >> >> Hi all, >> >> Since we are not going to be able to talk about this at LSF/MM, a few of >> us thought about holding a dedicated virtual discussion about Copy >> Offload. I believe we can use Chaitanya's thread as a start. Given the >> current state of the current patches, I would propose that we focus on >> the next step to get the minimal patchset that can go upstream so that >> we can build from there. >> > >I agree with having a call as it has been two years I'm trying to have >this discussion. > >Before we setup a call, please summarize following here :- > >1. Exactly what work has been done so far. We can categorize that into two sets. First one for XCopy (2014), and second one for NVMe Copy (2021). XCOPY set ********* - block-generic copy command (single range, between one source/destination device) - ioctl interface for the above - SCSI plumbing (block-generic to XCOPY conversion) - device-mapper support: offload copy whenever possible (if IO is not split while traveling layers of virtual devices) NVMe-Copy set ************* - block-generic copy command (multiple ranges, between one source/destination device) - ioctl interface for the above - NVMe plumbing (block-generic to NVMe Copy conversion) - copy-emulation (read + write) in block-layer - device-mapper support: no offload, rather fall back to copy-emulation >2. What kind of feedback you got. For NVMe Copy, the major points are - a) add copy-emulation in block-layer and use that if copy-offload is not natively supported by device b) user-interface (ioctl) should be extendable for copy across two devices (one source, one destination) c) device-mapper targets should support copy-offload, whenever possible "whenever possible" cases get reduced compared to XCOPY because NVMe Copy is wit >3. What are the exact blockers/objections. I think it was device-mapper for XCOPY and remains the same for NVMe Copy as well. Device-mapper support requires decomposing copy operation to read and write. While that is not great for efficiency PoV, bigger concern is to check if we are taking the same route as XCOPY. From Martin's document (http://mkp.net/pubs/xcopy.pdf), if I got it right, one the major blocker is having more failure cases than successful ones. And that did not justify the effort/code to wire up device mapper. Is that a factor to consider for NVMe Copy (which is narrower in scope than XCOPY). >4. Potential ways of moving forward. a) we defer attempt device-mapper support (until NVMe has support/usecase), and address everything else (reusable user-interface etc.) b) we attempt device-mapper support (by moving to composite read+write communication between block-layer and nvme) Is this enough in your mind to move forward with a specific agenda? If we can, I would like to target the meetup in the next 2 weeks. Thanks, Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-10-06 10:01 ` Javier González @ 2021-10-13 8:35 ` Javier González 0 siblings, 0 replies; 43+ messages in thread From: Javier González @ 2021-10-13 8:35 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, bvanassche, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler Chaitanya, Did you have a chance to look at the answers below? I would like to start finding candidate dates throughout the next couple of weeks. Thanks, Javier On 06.10.2021 12:01, Javier González wrote: >On 30.09.2021 09:43, Chaitanya Kulkarni wrote: >>Javier, >> >>> >>>Hi all, >>> >>>Since we are not going to be able to talk about this at LSF/MM, a few of >>>us thought about holding a dedicated virtual discussion about Copy >>>Offload. I believe we can use Chaitanya's thread as a start. Given the >>>current state of the current patches, I would propose that we focus on >>>the next step to get the minimal patchset that can go upstream so that >>>we can build from there. >>> >> >>I agree with having a call as it has been two years I'm trying to have >>this discussion. >> >>Before we setup a call, please summarize following here :- >> >>1. Exactly what work has been done so far. > > >We can categorize that into two sets. First one for XCopy (2014), and >second one for NVMe Copy (2021). > >XCOPY set ********* >- block-generic copy command (single range, between one > source/destination device) >- ioctl interface for the above >- SCSI plumbing (block-generic to XCOPY conversion) >- device-mapper support: offload copy whenever possible (if IO is not > split while traveling layers of virtual devices) > >NVMe-Copy set ************* >- block-generic copy command (multiple ranges, between one > source/destination device) >- ioctl interface for the above >- NVMe plumbing (block-generic to NVMe Copy conversion) >- copy-emulation (read + write) in block-layer >- device-mapper support: no offload, rather fall back to copy-emulation > > >>2. What kind of feedback you got. > >For NVMe Copy, the major points are - a) add copy-emulation in >block-layer and use that if copy-offload is not natively supported by >device b) user-interface (ioctl) should be extendable for copy across >two devices (one source, one destination) c) device-mapper targets >should support copy-offload, whenever possible > >"whenever possible" cases get reduced compared to XCOPY because NVMe >Copy is wit > >>3. What are the exact blockers/objections. > >I think it was device-mapper for XCOPY and remains the same for NVMe >Copy as well. Device-mapper support requires decomposing copy operation >to read and write. While that is not great for efficiency PoV, bigger >concern is to check if we are taking the same route as XCOPY. > >From Martin's document (http://mkp.net/pubs/xcopy.pdf), if I got it >right, one the major blocker is having more failure cases than >successful ones. And that did not justify the effort/code to wire up >device mapper. Is that a factor to consider for NVMe Copy (which is >narrower in scope than XCOPY). > >>4. Potential ways of moving forward. > >a) we defer attempt device-mapper support (until NVMe has >support/usecase), and address everything else (reusable user-interface >etc.) > >b) we attempt device-mapper support (by moving to composite read+write >communication between block-layer and nvme) > > >Is this enough in your mind to move forward with a specific agenda? If >we can, I would like to target the meetup in the next 2 weeks. > >Thanks, >Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-09-28 19:13 ` Javier González 2021-09-29 6:44 ` Johannes Thumshirn 2021-09-30 9:43 ` Chaitanya Kulkarni @ 2021-09-30 16:20 ` Bart Van Assche 2021-10-06 10:05 ` Javier González 2 siblings, 1 reply; 43+ messages in thread From: Bart Van Assche @ 2021-09-30 16:20 UTC (permalink / raw) To: Javier González, Johannes Thumshirn Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, martin.petersen, Kanchan Joshi, lsf-pc, rwheeler On 9/28/21 12:13 PM, Javier González wrote: > Since we are not going to be able to talk about this at LSF/MM, a few of > us thought about holding a dedicated virtual discussion about Copy > Offload. I believe we can use Chaitanya's thread as a start. Given the > current state of the current patches, I would propose that we focus on > the next step to get the minimal patchset that can go upstream so that > we can build from there. > > Before we try to find a date and a time that fits most of us, who would > be interested in participating? Given the technical complexity of this topic and also that the people who are interested live in multiple time zones, I prefer email to discuss the technical aspects of this work. My attempt to summarize how to implement copy offloading is available here: https://github.com/bvanassche/linux-kernel-copy-offload. Feedback on this text is welcome. Thanks, Bart. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-09-30 16:20 ` Bart Van Assche @ 2021-10-06 10:05 ` Javier González 2021-10-06 17:33 ` Bart Van Assche 0 siblings, 1 reply; 43+ messages in thread From: Javier González @ 2021-10-06 10:05 UTC (permalink / raw) To: Bart Van Assche Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 30.09.2021 09:20, Bart Van Assche wrote: >On 9/28/21 12:13 PM, Javier González wrote: >>Since we are not going to be able to talk about this at LSF/MM, a few of >>us thought about holding a dedicated virtual discussion about Copy >>Offload. I believe we can use Chaitanya's thread as a start. Given the >>current state of the current patches, I would propose that we focus on >>the next step to get the minimal patchset that can go upstream so that >>we can build from there. >> >>Before we try to find a date and a time that fits most of us, who would >>be interested in participating? > >Given the technical complexity of this topic and also that the people who are >interested live in multiple time zones, I prefer email to discuss the technical >aspects of this work. My attempt to summarize how to implement copy offloading >is available here: https://protect2.fireeye.com/v1/url?k=ba7e5d9a-e5e564d5-ba7fd6d5-0cc47a30d446-07a47f3f53cbfe53&q=1&e=c3973bdc-b6fd-43fb-80e6-0c86cb6b4d5f&u=https%3A%2F%2Fgithub.com%2Fbvanassche%2Flinux-kernel-copy-offload. >Feedback on this text is welcome. Thanks for sharing this Bart. I agree that the topic is complex. However, we have not been able to find a clear path forward in the mailing list. What do you think about joining the call to talk very specific next steps to get a patchset that we can start reviewing in detail. I think that your presence in the call will help us all. What do you think? -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-10-06 10:05 ` Javier González @ 2021-10-06 17:33 ` Bart Van Assche [not found] ` <20211008064925.oyjxbmngghr2yovr@mpHalley.local> 0 siblings, 1 reply; 43+ messages in thread From: Bart Van Assche @ 2021-10-06 17:33 UTC (permalink / raw) To: Javier González Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 10/6/21 3:05 AM, Javier González wrote: > I agree that the topic is complex. However, we have not been able to > find a clear path forward in the mailing list. Hmm ... really? At least Martin Petersen and I consider device mapper support essential. How about starting from Mikulas' patch series that supports the device mapper? See also https://lore.kernel.org/all/alpine.LRH.2.02.2108171630120.30363@file01.intranet.prod.int.rdu2.redhat.com/ > What do you think about joining the call to talk very specific next > steps to get a patchset that we can start reviewing in detail. I can do that. Thanks, Bart. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
[parent not found: <20211008064925.oyjxbmngghr2yovr@mpHalley.local>]
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <20211008064925.oyjxbmngghr2yovr@mpHalley.local> @ 2021-10-29 0:21 ` Chaitanya Kulkarni 2021-10-29 5:51 ` Hannes Reinecke 2021-10-29 8:14 ` Javier González 0 siblings, 2 replies; 43+ messages in thread From: Chaitanya Kulkarni @ 2021-10-29 0:21 UTC (permalink / raw) To: Javier González Cc: Vincent Fu, Bart, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, Assche, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, Kanchan, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 10/7/21 11:49 PM, Javier González wrote: > External email: Use caution opening links or attachments > > > On 06.10.2021 10:33, Bart Van Assche wrote: >> On 10/6/21 3:05 AM, Javier González wrote: >>> I agree that the topic is complex. However, we have not been able to >>> find a clear path forward in the mailing list. >> >> Hmm ... really? At least Martin Petersen and I consider device mapper >> support essential. How about starting from Mikulas' patch series that >> supports the device mapper? See also >> https://lore.kernel.org/all/alpine.LRH.2.02.2108171630120.30363@file01.intranet.prod.int.rdu2.redhat.com/ >> When we add a new REQ_OP_XXX we need to make sure it will work with device mapper, so I agree with Bart and Martin. Starting with Mikulas patches is a right direction as of now.. > > Thanks for the pointers. We are looking into Mikulas' patch - I agree > that it is a good start. > >>> What do you think about joining the call to talk very specific next >>> steps to get a patchset that we can start reviewing in detail. >> >> I can do that. > > Thanks. I will wait until Chaitanya's reply on his questions. We will > start suggesting some dates then. > I think at this point we need to at least decide on having a first call focused on how to proceed forward with Mikulas approach ... Javier, can you please organize a call with people you listed in this thread earlier ? > Thanks, > Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-10-29 0:21 ` Chaitanya Kulkarni @ 2021-10-29 5:51 ` Hannes Reinecke 2021-10-29 8:16 ` Javier González 2021-10-29 16:15 ` Bart Van Assche 2021-10-29 8:14 ` Javier González 1 sibling, 2 replies; 43+ messages in thread From: Hannes Reinecke @ 2021-10-29 5:51 UTC (permalink / raw) To: Chaitanya Kulkarni, Javier González Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, Bart Van Assche, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, martin.petersen, Kanchan Joshi, Johannes Thumshirn, lsf-pc, rwheeler On 10/29/21 2:21 AM, Chaitanya Kulkarni wrote: > On 10/7/21 11:49 PM, Javier González wrote: >> External email: Use caution opening links or attachments >> >> >> On 06.10.2021 10:33, Bart Van Assche wrote: >>> On 10/6/21 3:05 AM, Javier González wrote: >>>> I agree that the topic is complex. However, we have not been able to >>>> find a clear path forward in the mailing list. >>> >>> Hmm ... really? At least Martin Petersen and I consider device mapper >>> support essential. How about starting from Mikulas' patch series that >>> supports the device mapper? See also >>> https://lore.kernel.org/all/alpine.LRH.2.02.2108171630120.30363@file01.intranet.prod.int.rdu2.redhat.com/ >>> > > When we add a new REQ_OP_XXX we need to make sure it will work with > device mapper, so I agree with Bart and Martin. > > Starting with Mikulas patches is a right direction as of now.. > >> >> Thanks for the pointers. We are looking into Mikulas' patch - I agree >> that it is a good start. >> >>>> What do you think about joining the call to talk very specific next >>>> steps to get a patchset that we can start reviewing in detail. >>> >>> I can do that. >> >> Thanks. I will wait until Chaitanya's reply on his questions. We will >> start suggesting some dates then. >> > > I think at this point we need to at least decide on having a first call > focused on how to proceed forward with Mikulas approach ... > > Javier, can you please organize a call with people you listed in this > thread earlier ? > Also Keith presented his work on a simple zone-based remapping block device, which included an in-kernel copy offload facility. Idea is to lift that as a standalone patch such that we can use it a fallback (ie software) implementation if no other copy offload mechanism is available. Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-10-29 5:51 ` Hannes Reinecke @ 2021-10-29 8:16 ` Javier González 2021-10-29 16:15 ` Bart Van Assche 1 sibling, 0 replies; 43+ messages in thread From: Javier González @ 2021-10-29 8:16 UTC (permalink / raw) To: Hannes Reinecke Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, Bart Van Assche, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, Chaitanya Kulkarni, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 29.10.2021 07:51, Hannes Reinecke wrote: >On 10/29/21 2:21 AM, Chaitanya Kulkarni wrote: >>On 10/7/21 11:49 PM, Javier González wrote: >>>External email: Use caution opening links or attachments >>> >>> >>>On 06.10.2021 10:33, Bart Van Assche wrote: >>>>On 10/6/21 3:05 AM, Javier González wrote: >>>>>I agree that the topic is complex. However, we have not been able to >>>>>find a clear path forward in the mailing list. >>>> >>>>Hmm ... really? At least Martin Petersen and I consider device mapper >>>>support essential. How about starting from Mikulas' patch series that >>>>supports the device mapper? See also >>>>https://lore.kernel.org/all/alpine.LRH.2.02.2108171630120.30363@file01.intranet.prod.int.rdu2.redhat.com/ >>>> >> >>When we add a new REQ_OP_XXX we need to make sure it will work with >>device mapper, so I agree with Bart and Martin. >> >>Starting with Mikulas patches is a right direction as of now.. >> >>> >>>Thanks for the pointers. We are looking into Mikulas' patch - I agree >>>that it is a good start. >>> >>>>>What do you think about joining the call to talk very specific next >>>>>steps to get a patchset that we can start reviewing in detail. >>>> >>>>I can do that. >>> >>>Thanks. I will wait until Chaitanya's reply on his questions. We will >>>start suggesting some dates then. >>> >> >>I think at this point we need to at least decide on having a first call >>focused on how to proceed forward with Mikulas approach ... >> >>Javier, can you please organize a call with people you listed in this >>thread earlier ? >> >Also Keith presented his work on a simple zone-based remapping block >device, which included an in-kernel copy offload facility. >Idea is to lift that as a standalone patch such that we can use it a >fallback (ie software) implementation if no other copy offload >mechanism is available. > I believe this is in essence what we are trying to convey here: a minimal patchset that enables Simple Copy and the infra around to extend copy-offload use-cases. I look forward to hear Keith's ideas around this! Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-10-29 5:51 ` Hannes Reinecke 2021-10-29 8:16 ` Javier González @ 2021-10-29 16:15 ` Bart Van Assche 2021-11-01 17:54 ` Keith Busch 1 sibling, 1 reply; 43+ messages in thread From: Bart Van Assche @ 2021-10-29 16:15 UTC (permalink / raw) To: Hannes Reinecke; +Cc: linux-block, dm-devel, lsf-pc, linux-nvme, linux-scsi On 10/28/21 10:51 PM, Hannes Reinecke wrote: > Also Keith presented his work on a simple zone-based remapping block device, which included an in-kernel copy offload facility. > Idea is to lift that as a standalone patch such that we can use it a fallback (ie software) implementation if no other copy offload mechanism is available. Is a link to the presentation available? Thanks, Bart. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-10-29 16:15 ` Bart Van Assche @ 2021-11-01 17:54 ` Keith Busch 0 siblings, 0 replies; 43+ messages in thread From: Keith Busch @ 2021-11-01 17:54 UTC (permalink / raw) To: Bart Van Assche; +Cc: linux-scsi, linux-nvme, linux-block, dm-devel, lsf-pc On Fri, Oct 29, 2021 at 09:15:43AM -0700, Bart Van Assche wrote: > On 10/28/21 10:51 PM, Hannes Reinecke wrote: > > Also Keith presented his work on a simple zone-based remapping block device, which included an in-kernel copy offload facility. > > Idea is to lift that as a standalone patch such that we can use it a fallback (ie software) implementation if no other copy offload mechanism is available. > > Is a link to the presentation available? Thanks for the interest. I didn't post them online as the conference didn't provide it, and I don't think the slides would be particularly interesting without the prepared speech anyway. The presentation described a simple prototype implementing a redirection table on zone block devices. There was one bullet point explaining how a generic kernel implementation would be an improvement. For zoned block devices, an "append" like copy offload would be an even better option. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-10-29 0:21 ` Chaitanya Kulkarni 2021-10-29 5:51 ` Hannes Reinecke @ 2021-10-29 8:14 ` Javier González 2021-11-03 19:27 ` Javier González 1 sibling, 1 reply; 43+ messages in thread From: Javier González @ 2021-10-29 8:14 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, Bart Van Assche, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 29.10.2021 00:21, Chaitanya Kulkarni wrote: >On 10/7/21 11:49 PM, Javier González wrote: >> External email: Use caution opening links or attachments >> >> >> On 06.10.2021 10:33, Bart Van Assche wrote: >>> On 10/6/21 3:05 AM, Javier González wrote: >>>> I agree that the topic is complex. However, we have not been able to >>>> find a clear path forward in the mailing list. >>> >>> Hmm ... really? At least Martin Petersen and I consider device mapper >>> support essential. How about starting from Mikulas' patch series that >>> supports the device mapper? See also >>> https://lore.kernel.org/all/alpine.LRH.2.02.2108171630120.30363@file01.intranet.prod.int.rdu2.redhat.com/ >>> > >When we add a new REQ_OP_XXX we need to make sure it will work with >device mapper, so I agree with Bart and Martin. > >Starting with Mikulas patches is a right direction as of now.. > >> >> Thanks for the pointers. We are looking into Mikulas' patch - I agree >> that it is a good start. >> >>>> What do you think about joining the call to talk very specific next >>>> steps to get a patchset that we can start reviewing in detail. >>> >>> I can do that. >> >> Thanks. I will wait until Chaitanya's reply on his questions. We will >> start suggesting some dates then. >> > >I think at this point we need to at least decide on having a first call >focused on how to proceed forward with Mikulas approach ... > >Javier, can you please organize a call with people you listed in this >thread earlier ? Here you have a Doogle for end of next week and the week after OCP. Please fill it out until Wednesday. I will set up a call with the selected slot: https://doodle.com/poll/r2c8duy3r8g88v8q?utm_source=poll&utm_medium=link Thanks, Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-10-29 8:14 ` Javier González @ 2021-11-03 19:27 ` Javier González [not found] ` <20211116134324.hbs3tp5proxootd7@ArmHalley.localdomain> 0 siblings, 1 reply; 43+ messages in thread From: Javier González @ 2021-11-03 19:27 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, Bart Van Assche, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 29.10.2021 10:14, Javier González wrote: >On 29.10.2021 00:21, Chaitanya Kulkarni wrote: >>On 10/7/21 11:49 PM, Javier González wrote: >>>External email: Use caution opening links or attachments >>> >>> >>>On 06.10.2021 10:33, Bart Van Assche wrote: >>>>On 10/6/21 3:05 AM, Javier González wrote: >>>>>I agree that the topic is complex. However, we have not been able to >>>>>find a clear path forward in the mailing list. >>>> >>>>Hmm ... really? At least Martin Petersen and I consider device mapper >>>>support essential. How about starting from Mikulas' patch series that >>>>supports the device mapper? See also >>>>https://lore.kernel.org/all/alpine.LRH.2.02.2108171630120.30363@file01.intranet.prod.int.rdu2.redhat.com/ >>>> >> >>When we add a new REQ_OP_XXX we need to make sure it will work with >>device mapper, so I agree with Bart and Martin. >> >>Starting with Mikulas patches is a right direction as of now.. >> >>> >>>Thanks for the pointers. We are looking into Mikulas' patch - I agree >>>that it is a good start. >>> >>>>>What do you think about joining the call to talk very specific next >>>>>steps to get a patchset that we can start reviewing in detail. >>>> >>>>I can do that. >>> >>>Thanks. I will wait until Chaitanya's reply on his questions. We will >>>start suggesting some dates then. >>> >> >>I think at this point we need to at least decide on having a first call >>focused on how to proceed forward with Mikulas approach ... >> >>Javier, can you please organize a call with people you listed in this >>thread earlier ? > >Here you have a Doogle for end of next week and the week after OCP. >Please fill it out until Wednesday. I will set up a call with the >selected slot: > > https://doodle.com/poll/r2c8duy3r8g88v8q?utm_source=poll&utm_medium=link > >Thanks, >Javier I sent the invite for the people that signed up into the Doodle. The call will take place on Monday November 15th, 17.00-19.00 CET. See the list of current participants below. If anyone else wants to participate, please send me a note and I will extend the invite. Johannes.Thumshirn@wdc.com Vincent.fu@samsung.com a.dawn@samsung.com a.manzanares@samsung.com bvanassche@acm.org himanshu.madhani@oracle.com joshi.k@samsung.com kch@nvidia.com martin.petersen@oracle.com nj.shetty@samsung.com selvakuma.s1@samsung Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
[parent not found: <20211116134324.hbs3tp5proxootd7@ArmHalley.localdomain>]
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <20211116134324.hbs3tp5proxootd7@ArmHalley.localdomain> @ 2021-11-16 17:59 ` Bart Van Assche [not found] ` <20211117125224.z36hp2crpj4fwngc@ArmHalley.local> [not found] ` <CA+1E3rJRT+89OCyqRtb5BFbez0BfkKvCGijd=nObMEB3_v6MyA@mail.gmail.com> 1 sibling, 1 reply; 43+ messages in thread From: Bart Van Assche @ 2021-11-16 17:59 UTC (permalink / raw) To: Javier González, Chaitanya Kulkarni Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 11/16/21 05:43, Javier González wrote: > - Here, we need copy emulation to support encryption > without dealing with HW issues and garbage Hi Javier, Thanks very much for having taken notes and also for having shared these. Regarding the above comment, after the meeting I learned that the above is not correct. Encryption in Android is LBA independent and hence it should be possible to offload F2FS garbage collection in Android once the (UFS) storage controller supports this. For the general case, I propose to let the dm-crypt driver decide whether or not to offload data copying since that driver knows whether or not data copying can be offloaded. Thanks, Bart. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
[parent not found: <20211117125224.z36hp2crpj4fwngc@ArmHalley.local>]
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <20211117125224.z36hp2crpj4fwngc@ArmHalley.local> @ 2021-11-17 15:52 ` Bart Van Assche 0 siblings, 0 replies; 43+ messages in thread From: Bart Van Assche @ 2021-11-17 15:52 UTC (permalink / raw) To: Javier González Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, Chaitanya Kulkarni, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 11/17/21 04:53, Javier González wrote: > Thanks for sharing this. We will make sure that DM / MD are supported > and then we can cover examples. Hopefully, you guys can help with the > bits for dm-crypt to make the decision to offload when it make sense. Will ask around to learn who should work on this. > I will update the notes to keep them alive. Maybe we can have them open > in your github page? Feel free to submit a pull request. Thanks, Bart. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
[parent not found: <CA+1E3rJRT+89OCyqRtb5BFbez0BfkKvCGijd=nObMEB3_v6MyA@mail.gmail.com>]
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <CA+1E3rJRT+89OCyqRtb5BFbez0BfkKvCGijd=nObMEB3_v6MyA@mail.gmail.com> @ 2021-11-19 16:21 ` Bart Van Assche 0 siblings, 0 replies; 43+ messages in thread From: Bart Van Assche @ 2021-11-19 16:21 UTC (permalink / raw) To: Kanchan Joshi, Javier González Cc: Vincent Fu, linux-nvme, dm-devel, Adam Manzanares, osandov, msnitzer, linux-scsi, hch, roland, Nitesh Shetty, zach.brown, Chaitanya Kulkarni, SelvaKumar S, Chaitanya Kulkarni, linux-block, mpatocka, kbusch, Frederick.Knight, axboe, Kanchan Joshi, martin.petersen, Johannes Thumshirn, lsf-pc, rwheeler On 11/19/21 02:47, Kanchan Joshi wrote: > Given the multitude of things accumulated on this topic, Martin > suggested to have a table/matrix. > Some of those should go in the initial patchset, and the remaining are > to be staged for subsequent work. > Here is the attempt to split the stuff into two buckets. Please change > if something needs to be changed below. > > 1. Driver > ********* > Initial: NVMe Copy command (single NS) > Subsequent: Multi NS copy, XCopy/Token-based Copy > > 2. Block layer > ************** > Initial: > - Block-generic copy (REQ_OP_COPY), with interface accommodating two block-devs > - Emulation, when offload is natively absent > - DM support (at least dm-linear) > > 3. User-interface > ***************** > Initial: new ioctl or io_uring opcode > > 4. In-kernel user > ****************** > Initial: at least one user > - dm-kcopyd user (e.g. dm-clone), or FS requiring GC (F2FS/Btrfs) > > Subsequent: > - copy_file_range Integrity support and inline encryption support are missing from the above overview. Both are supported by the block layer. See also block/blk-integrity.c and include/linux/blk-crypto.h. I'm not claiming that these should be supported in the first version but I think it would be good to add these to the above overview. Thanks, Bart. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni ` (3 preceding siblings ...) 2021-05-12 7:30 ` Johannes Thumshirn @ 2021-05-12 7:36 ` Erwin van Londen 2021-05-12 15:23 ` Hannes Reinecke ` (5 subsequent siblings) 10 siblings, 0 replies; 43+ messages in thread From: Erwin van Londen @ 2021-05-12 7:36 UTC (permalink / raw) To: Chaitanya Kulkarni, linux-block, linux-scsi, linux-nvme, dm-devel, lsf-pc Cc: axboe, msnitzer, bvanassche, martin.petersen, hch, roland, mpatocka, kbusch, rwheeler, osandov, Frederick.Knight, zach.brown [-- Attachment #1.1: Type: text/plain, Size: 5795 bytes --] On Tue, 2021-05-11 at 00:15 +0000, Chaitanya Kulkarni wrote: > Hi, > > * Background :- > --------------------------------------------------------------------- > -- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > > With reference to the RISC-V summit keynote [1] single threaded > performance is limiting due to Denard scaling and multi-threaded > performance is slowing down due Moore's law limitations. With the > rise > of SNIA Computation Technical Storage Working Group (TWG) [2], > offloading computations to the device or over the fabrics is becoming > popular as there are several solutions available [2]. One of the > common > operation which is popular in the kernel and is not merged yet is > Copy > offload over the fabrics or on to the device. > > * Problem :- > --------------------------------------------------------------------- > -- > > The original work which is done by Martin is present here [3]. The > latest work which is posted by Mikulas [4] is not merged yet. These > two > approaches are totally different from each other. Several storage > vendors discourage mixing copy offload requests with regular > READ/WRITE > I/O. Also, the fact that the operation fails if a copy request ever > needs to be split as it traverses the stack it has the unfortunate > side-effect of preventing copy offload from working in pretty much > every common deployment configuration out there. > > * Current state of the work :- > --------------------------------------------------------------------- > -- > > With [3] being hard to handle arbitrary DM/MD stacking without > splitting the command in two, one for copying IN and one for copying > OUT. Which is then demonstrated by the [4] why [3] it is not a > suitable > candidate. Also, with [4] there is an unresolved problem with the > two-command approach about how to handle changes to the DM layout > between an IN and OUT operations. > > * Why Linux Kernel Storage System needs Copy Offload support now ? > --------------------------------------------------------------------- > -- > > With the rise of the SNIA Computational Storage TWG and solutions > [2], > existing SCSI XCopy support in the protocol, recent advancement in > the > Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer > DMA support in the Linux Kernel mainly for NVMe devices [7] and > eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit > from Copy offload operation. > > With this background we have significant number of use-cases which > are > strong candidates waiting for outstanding Linux Kernel Block Layer > Copy > Offload support, so that Linux Kernel Storage subsystem can to > address > previously mentioned problems [1] and allow efficient offloading of > the > data related operations. (Such as move/copy etc.) > > For reference following is the list of the use-cases/candidates > waiting > for Copy Offload support :- > > 1. SCSI-attached storage arrays. > 2. Stacking drivers supporting XCopy DM/MD. > 3. Computational Storage solutions. > 7. File systems :- Local, NFS and Zonefs. > 4. Block devices :- Distributed, local, and Zoned devices. > 5. Peer to Peer DMA support solutions. > 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. > > * What we will discuss in the proposed session ? > --------------------------------------------------------------------- > -- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > --------------------------------------------------------------------- > -- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. > > Required attendees :- > > Martin K. Petersen > Jens Axboe > Christoph Hellwig > Bart Van Assche > Zach Brown > Roland Dreier > Ric Wheeler > Trond Myklebust > Mike Snitzer > Keith Busch > Sagi Grimberg > Hannes Reinecke > Frederick Knight > Mikulas Patocka > Keith Busch > > Regards, > Chaitanya > +1 here. I would like to see how this pans out as many differences may be observed from a standards, implementation and operations point of view. > [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf > [2] https://www.snia.org/computational > https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/ > https://www.eideticom.com/products.html > https://www.xilinx.com/applications/data-center/computational-storage.html > [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy > [4] https://www.spinics.net/lists/linux-block/msg00599.html > [5] https://lwn.net/Articles/793585/ > [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- > namespaces-zns-as-go-to-industry-technology/ > [7] https://github.com/sbates130272/linux-p2pmem > [8] https://kernel.dk/io_uring.pdf > > > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://listman.redhat.com/mailman/listinfo/dm-devel > [-- Attachment #1.2: Type: text/html, Size: 8430 bytes --] [-- Attachment #2: Type: text/plain, Size: 97 bytes --] -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni ` (4 preceding siblings ...) 2021-05-12 7:36 ` Erwin van Londen @ 2021-05-12 15:23 ` Hannes Reinecke 2021-05-12 15:45 ` Himanshu Madhani ` (4 subsequent siblings) 10 siblings, 0 replies; 43+ messages in thread From: Hannes Reinecke @ 2021-05-12 15:23 UTC (permalink / raw) To: Chaitanya Kulkarni, linux-block, linux-scsi, linux-nvme, dm-devel, lsf-pc Cc: axboe, msnitzer, bvanassche, martin.petersen, osandov, roland, mpatocka, kbusch, rwheeler, hch, Frederick.Knight, zach.brown On 5/11/21 2:15 AM, Chaitanya Kulkarni wrote: > Hi, > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > The neverending topic. Count me in. Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni ` (5 preceding siblings ...) 2021-05-12 15:23 ` Hannes Reinecke @ 2021-05-12 15:45 ` Himanshu Madhani 2021-05-17 16:39 ` Kanchan Joshi ` (3 subsequent siblings) 10 siblings, 0 replies; 43+ messages in thread From: Himanshu Madhani @ 2021-05-12 15:45 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: axboe, roland, msnitzer, bvanassche, linux-scsi, linux-nvme, Petersen, linux-block, dm-devel, mpatocka, Martin, kbusch, osandov, lsf-pc, rwheeler, hch, Frederick.Knight, zach.brown > On May 10, 2021, at 7:15 PM, Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com> wrote: > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. I would like to participate in this discussion as well. -- Himanshu Madhani Oracle Linux Engineering -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni ` (6 preceding siblings ...) 2021-05-12 15:45 ` Himanshu Madhani @ 2021-05-17 16:39 ` Kanchan Joshi 2021-05-18 0:15 ` Bart Van Assche ` (2 subsequent siblings) 10 siblings, 0 replies; 43+ messages in thread From: Kanchan Joshi @ 2021-05-17 16:39 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: axboe, roland, msnitzer, bvanassche, linux-scsi, linux-nvme, osandov, linux-block, dm-devel, mpatocka, martin.petersen, kbusch, lsf-pc, rwheeler, hch, Frederick.Knight, zach.brown > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > ----------------------------------------------------------------------- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. > I'd like to participate in discussion. Hopefully we can get consensus on some elements (or discover new issues) before Dec. An async-interface (via io_uring) would be good to be discussed while we are at it. -- Kanchan -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni ` (7 preceding siblings ...) 2021-05-17 16:39 ` Kanchan Joshi @ 2021-05-18 0:15 ` Bart Van Assche 2021-06-11 6:03 ` Chaitanya Kulkarni 2021-06-11 15:35 ` Nikos Tsironis 10 siblings, 0 replies; 43+ messages in thread From: Bart Van Assche @ 2021-05-18 0:15 UTC (permalink / raw) To: Chaitanya Kulkarni, linux-block, linux-scsi, linux-nvme, dm-devel, lsf-pc Cc: axboe, msnitzer, martin.petersen, osandov, roland, mpatocka, kbusch, rwheeler, hch, Frederick.Knight, zach.brown On 5/10/21 5:15 PM, Chaitanya Kulkarni wrote: > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? We need to achieve agreement about an approach. The text below is my attempt at guiding the discussion. A HTML version is available at https://github.com/bvanassche/linux-kernel-copy-offload. As usual, feedback is welcome. Bart. # Implementing Copy Offloading in the Linux Kernel ## Introduction Efforts to add copy offloading support in the Linux kernel started considerable time ago. Despite this copy offloading support is not yet upstream and there is no detailed plan yet of how to implement copy offloading. This document outlines a possible implementation. The purpose of this document is to help guiding the conversations around copy offloading. ## Block Layer We need an interface to pass copy offload requests from user space or file systems to block drivers. Although the first implementation of copy offloading added a single operation to the block layer for copy offloading, there seems to be agreement today to implement copy offloading as two operations, namely `REQ_COPY_IN` and `REQ_COPY_OUT`. A possible approach is as follows: * Fall back to a non-offloaded copy operation if necessary, e.g. if copy offloading is not supported or if data is encrypted and the ciphertext depends on the LBA. The following code may be a good starting point: `drivers/md/dm-kcopyd.c`. * If the block driver supports copy offloading, submit the `REQ_COPY_IN` operation first. The block driver stores the data ranges associated with the `REQ_COPY_IN` operation. * Wait for completion of the `REQ_COPY_IN` operation. * After the `REQ_COPY_IN` operation has completed, submit the `REQ_COPY_OUT` operation and include a reference to the `REQ_COPY_IN` operation. If the block driver that receives the `REQ_COPY_OUT` operation receives a matching `REQ_COPY_IN` operation, offload the copy operation. Otherwise report that no data has been copied and let the block layer perform a non-offloaded copy operation. The operation type is stored in the top bits of the `bi_opf` member of struct bio. With each bio a single data buffer and a single contiguous byte range on the storage medium are associated. Pointers to the data buffer occur in `bi_io_vec[]`. The affected byte range is represented by `bi_iter.bi_sector` and `bi_iter.bi_size`. While the NVMe and SCSI copy offload commands both support multiple source ranges, XCOPY supports multiple destination ranges while the NVMe simple copy command supports a single destination range. Possible approaches for passing the data ranges involved in a copy operation from the block layer to block drivers are as follows: * Attach a bio to each copy offload request and encode all relevant copy offload parameters in that data buffer. These parameters include source device and source ranges for `REQ_COPY_IN` and destination device and destination ranges for `REQ_COPY_OUT`. Let the block drivers translate these parameters into something the storage device understands (NVMe simple copy parameters or SCSI XCOPY parameters). Fill in the parameter structure size in `bi_iter.bi_size`. Set `bi_vcnt` to 1 and fill in `bio->bi_io_vec[0]`. * Map each source range and each destination range onto a different bio. Link all the bios with the `bi_next` pointer and attach these bios to the copy offload requests. Leave `bi_vcnt` zero. This is related but not identical to the approach followed by `__blkdev_issue_discard()`. I think that the first approach would require more changes in the device mapper than the second approach since the device mapper code knows how to split bios but not how to split a buffer with LBA range descriptors. The following code needs to be modified no matter how copy offloading is implemented: * Request cloning. The code for checking the limits before request are cloned compares `blk_rq_sectors()` with `max_sectors`. This is inappropriate for `REQ_COPY_*` requests. * Request splitting. `bio_split()` assumes that `bi_iter.bi_size` represents the number of bytes affected on the medium. * Code related to retrying the original requests of a merged request with mixed failfast attributes, e.g. `blk_rq_err_bytes()`. * Code related to partially completing a request, e.g. `blk_update_request()`. * The code for merging block layer requests. * `blk_mq_end_request()` since it calls `blk_update_request()` and `blk_rq_bytes()`. * The plugging code because of the following test in the plugging code: `blk_rq_bytes(last) >= BLK_PLUG_FLUSH_SIZE`. * The I/O accounting code (task_io_account_read()) since that code uses bio_has_data() and hence skips discard, secure erase and write zeroes requests: ``` static inline bool bio_has_data(struct bio *bio) { return bio && bio->bi_iter.bi_size && bio_op(bio) != REQ_OP_DISCARD && bio_op(bio) != REQ_OP_SECURE_ERASE && bio_op(bio) != REQ_OP_WRITE_ZEROESy; } ``` Block drivers will need to use the `special_vec` member of struct request to pass the copy offload parameters to the storage device. That member is used e.g. when a REQ_OP_DISCARD operation is submitted to an NVMe driver. The SCSI sd driver uses `special_vec` while processing an UNMAP or WRITE SAME command. ## Device Mapper The device mapper may have to split a request. As an example, LVM is based on the dm-linear driver. A request that is submitted to an LVM volume has to be split if it affects multiple block devices. Copy offload requests that affect multiple block devices should be split or should be onloaded. The call chain for bio-based dm drivers is as follows: ``` dm_submit_bio(bio) -> __split_and_process_bio(md, map, bio) -> __split_and_process_non_flush(clone_info) -> __clone_and_map_data_bio(clone_info, target_info, sector, len) -> clone_bio(dm_target_io, bio, sector, len) -> __map_bio(dm_target_io) -> ti->type->map(dm_target_io, clone) ``` ## NVMe Process copy offload commands by translating REQ_COPY_OUT requests into simple copy commands. ## SCSI >From inside `sd_revalidate_disk()`, query the third-party copy VPD page. Extract the following parameters (see also SPC-6): * MAXIMUM CSCD DESCRIPTOR COUNT * MAXIMUM SEGMENT DESCRIPTOR COUNT * MAXIMUM DESCRIPTOR LIST LENGTH * Supported third-party copy commands. * SUPPORTED CSCD DESCRIPTOR ID (0 or more) * ROD type descriptor (0 or more) * TOTAL CONCURRENT COPIES * MAXIMUM IDENTIFIED CONCURRENT COPIES * MAXIMUM SEGMENT LENGTH >From inside `sd_init_command()`, translate REQ_COPY_OUT into either EXTENDED COPY or POPULATE TOKEN + WRITE USING TOKEN. Set the parameters in the copy offload commands as follows: * We may have to set the STR bit. From SPC-6: "A sequential striped (STR) bit set to one specifies to the copy manager that the majority of the block device references in the parameter list represent sequential access of several block devices that are striped. This may be used by the copy manager to perform reads from a copy source block device at any time and in any order during processing of an EXTENDED COPY command as described in 6.6.5.3. A STR bit set to zero specifies to the copy manager that disk references, if any, may not be sequential." * Set the LIST ID USAGE field to 3 and the LIST ID to 0. This means that neither "held data" nor the RECEIVE COPY STATUS command are supported. This improves security because the data that is being copied cannot be accessed via the LIST ID. * We may have to set the G_SENSE (good with sense data) bit. From SPC-6: " If the G _SENSE bit is set to one and the copy manager completes the EXTENDED COPY command with GOOD status, then the copy manager shall include sense data with the GOOD status in which the sense key is set to COMPLETED, the additional sense code is set to EXTENDED COPY INFORMATION AVAILABLE, and the COMMAND-SPECIFIC INFORMATION field is set to the number of segment descriptors the copy manager has processed." * Clear the IMMED bit. ## System Call Interface To submit copy offload requests from user space, we need: * A system call for passing these requests, e.g. copy_file_range() or io_uring. * Add a copy offload parameter format description to the user space ABI. The parameters include source device, source ranges, destination device and destination ranges. * A flag that indicates whether or not it is acceptable to fall back to onloading the copy operation. ## Sysfs Interface To do: define which aspects of copy offloading should be configurable through new sysfs parameters under /sys/block/*/queue/. ## See Also * Martin Petersen, [Copy Offload](https://www.mail-archive.com/linux-scsi@vger.kernel.org/msg28998.html), linux-scsi, 28 May 2014. * Mikulas Patocka, [ANNOUNCE: SCSI XCOPY support for the kernel and device mapper](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg686111.html), 15 July 2014. * [kcopyd documentation](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/kcopyd.html), kernel.org. * Martin K. Petersen, [Copy Offload - Here Be Dragons](http://mkp.net/pubs/xcopy.pdf), 2019-08-21. * Martin K. Petersen, [Re: [dm-devel] [RFC PATCH v2 1/2] block: add simple copy support](https://lore.kernel.org/linux-nvme/yq1blf3smcl.fsf@ca-mkp.ca.oracle.com/), linux-nvme mailing list, 2020-12-08. * NVM Express Organization, [NVMe - TP 4065b Simple Copy Command 2021.01.25 - Ratified.pdf](https://workspace.nvmexpress.org/apps/org/workgroup/allmembers/download.php/4773/NVMe%20-%20TP%204065b%20Simple%20Copy%20Command%202021.01.25%20-%20Ratified.pdf), 2021-01-25. * Selvakumar S, [[RFC PATCH v5 0/4] add simple copy support](https://lore.kernel.org/linux-nvme/20210219124517.79359-1-selvakuma.s1@samsung.com/), linux-nvme, 2021-02-19. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni ` (8 preceding siblings ...) 2021-05-18 0:15 ` Bart Van Assche @ 2021-06-11 6:03 ` Chaitanya Kulkarni 2021-06-11 15:35 ` Nikos Tsironis 10 siblings, 0 replies; 43+ messages in thread From: Chaitanya Kulkarni @ 2021-06-11 6:03 UTC (permalink / raw) To: linux-block, linux-scsi, linux-nvme, dm-devel, lsf-pc; +Cc: ckulkarnilinux On 5/10/21 17:15, Chaitanya Kulkarni wrote: > Hi, > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > > With reference to the RISC-V summit keynote [1] single threaded > performance is limiting due to Denard scaling and multi-threaded > performance is slowing down due Moore's law limitations. With the rise > of SNIA Computation Technical Storage Working Group (TWG) [2], > offloading computations to the device or over the fabrics is becoming > popular as there are several solutions available [2]. One of the common > operation which is popular in the kernel and is not merged yet is Copy > offload over the fabrics or on to the device. > > * Problem :- > ----------------------------------------------------------------------- > > The original work which is done by Martin is present here [3]. The > latest work which is posted by Mikulas [4] is not merged yet. These two > approaches are totally different from each other. Several storage > vendors discourage mixing copy offload requests with regular READ/WRITE > I/O. Also, the fact that the operation fails if a copy request ever > needs to be split as it traverses the stack it has the unfortunate > side-effect of preventing copy offload from working in pretty much > every common deployment configuration out there. > > * Current state of the work :- > ----------------------------------------------------------------------- > > With [3] being hard to handle arbitrary DM/MD stacking without > splitting the command in two, one for copying IN and one for copying > OUT. Which is then demonstrated by the [4] why [3] it is not a suitable > candidate. Also, with [4] there is an unresolved problem with the > two-command approach about how to handle changes to the DM layout > between an IN and OUT operations. > > * Why Linux Kernel Storage System needs Copy Offload support now ? > ----------------------------------------------------------------------- > > With the rise of the SNIA Computational Storage TWG and solutions [2], > existing SCSI XCopy support in the protocol, recent advancement in the > Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer > DMA support in the Linux Kernel mainly for NVMe devices [7] and > eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit > from Copy offload operation. > > With this background we have significant number of use-cases which are > strong candidates waiting for outstanding Linux Kernel Block Layer Copy > Offload support, so that Linux Kernel Storage subsystem can to address > previously mentioned problems [1] and allow efficient offloading of the > data related operations. (Such as move/copy etc.) > > For reference following is the list of the use-cases/candidates waiting > for Copy Offload support :- > > 1. SCSI-attached storage arrays. > 2. Stacking drivers supporting XCopy DM/MD. > 3. Computational Storage solutions. > 7. File systems :- Local, NFS and Zonefs. > 4. Block devices :- Distributed, local, and Zoned devices. > 5. Peer to Peer DMA support solutions. > 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. > > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > ----------------------------------------------------------------------- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. > > Required attendees :- > > Martin K. Petersen > Jens Axboe > Christoph Hellwig > Bart Van Assche > Zach Brown > Roland Dreier > Ric Wheeler > Trond Myklebust > Mike Snitzer > Keith Busch > Sagi Grimberg > Hannes Reinecke > Frederick Knight > Mikulas Patocka > Keith Busch > > Regards, > Chaitanya > > [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf > [2] https://www.snia.org/computational > https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/ > https://www.eideticom.com/products.html > https://www.xilinx.com/applications/data-center/computational-storage.html > [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy > [4] https://www.spinics.net/lists/linux-block/msg00599.html > [5] https://lwn.net/Articles/793585/ > [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- > namespaces-zns-as-go-to-industry-technology/ > [7] https://github.com/sbates130272/linux-p2pmem > [8] https://kernel.dk/io_uring.pdf > > Mail server is dropping emails from the mailing list, adding personal email address. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni ` (9 preceding siblings ...) 2021-06-11 6:03 ` Chaitanya Kulkarni @ 2021-06-11 15:35 ` Nikos Tsironis 10 siblings, 0 replies; 43+ messages in thread From: Nikos Tsironis @ 2021-06-11 15:35 UTC (permalink / raw) To: Chaitanya Kulkarni, linux-block, linux-scsi, linux-nvme, dm-devel, lsf-pc Cc: axboe, msnitzer, bvanassche, martin.petersen, osandov, roland, mpatocka, kbusch, rwheeler, hch, Frederick.Knight, zach.brown On 5/11/21 3:15 AM, Chaitanya Kulkarni wrote: > Hi, > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > > With reference to the RISC-V summit keynote [1] single threaded > performance is limiting due to Denard scaling and multi-threaded > performance is slowing down due Moore's law limitations. With the rise > of SNIA Computation Technical Storage Working Group (TWG) [2], > offloading computations to the device or over the fabrics is becoming > popular as there are several solutions available [2]. One of the common > operation which is popular in the kernel and is not merged yet is Copy > offload over the fabrics or on to the device. > > * Problem :- > ----------------------------------------------------------------------- > > The original work which is done by Martin is present here [3]. The > latest work which is posted by Mikulas [4] is not merged yet. These two > approaches are totally different from each other. Several storage > vendors discourage mixing copy offload requests with regular READ/WRITE > I/O. Also, the fact that the operation fails if a copy request ever > needs to be split as it traverses the stack it has the unfortunate > side-effect of preventing copy offload from working in pretty much > every common deployment configuration out there. > > * Current state of the work :- > ----------------------------------------------------------------------- > > With [3] being hard to handle arbitrary DM/MD stacking without > splitting the command in two, one for copying IN and one for copying > OUT. Which is then demonstrated by the [4] why [3] it is not a suitable > candidate. Also, with [4] there is an unresolved problem with the > two-command approach about how to handle changes to the DM layout > between an IN and OUT operations. > > * Why Linux Kernel Storage System needs Copy Offload support now ? > ----------------------------------------------------------------------- > > With the rise of the SNIA Computational Storage TWG and solutions [2], > existing SCSI XCopy support in the protocol, recent advancement in the > Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer > DMA support in the Linux Kernel mainly for NVMe devices [7] and > eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit > from Copy offload operation. > > With this background we have significant number of use-cases which are > strong candidates waiting for outstanding Linux Kernel Block Layer Copy > Offload support, so that Linux Kernel Storage subsystem can to address > previously mentioned problems [1] and allow efficient offloading of the > data related operations. (Such as move/copy etc.) > > For reference following is the list of the use-cases/candidates waiting > for Copy Offload support :- > > 1. SCSI-attached storage arrays. > 2. Stacking drivers supporting XCopy DM/MD. > 3. Computational Storage solutions. > 7. File systems :- Local, NFS and Zonefs. > 4. Block devices :- Distributed, local, and Zoned devices. > 5. Peer to Peer DMA support solutions. > 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. > > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > ----------------------------------------------------------------------- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. > > Required attendees :- > > Martin K. Petersen > Jens Axboe > Christoph Hellwig > Bart Van Assche > Zach Brown > Roland Dreier > Ric Wheeler > Trond Myklebust > Mike Snitzer > Keith Busch > Sagi Grimberg > Hannes Reinecke > Frederick Knight > Mikulas Patocka > Keith Busch > > Regards, > Chaitanya > > [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf > [2] https://www.snia.org/computational > https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/ > https://www.eideticom.com/products.html > https://www.xilinx.com/applications/data-center/computational-storage.html > [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy > [4] https://www.spinics.net/lists/linux-block/msg00599.html > [5] https://lwn.net/Articles/793585/ > [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- > namespaces-zns-as-go-to-industry-technology/ > [7] https://github.com/sbates130272/linux-p2pmem > [8] https://kernel.dk/io_uring.pdf > I would like to participate in this discussion too. Thanks, Nikos -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
[parent not found: <CGME20220127071544uscas1p2f70f4d2509f3ebd574b7ed746d3fa551@uscas1p2.samsung.com>]
[parent not found: <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com>]
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com> @ 2022-01-28 19:59 ` Adam Manzanares 2022-01-31 11:49 ` Johannes Thumshirn 2022-01-31 19:03 ` Bart Van Assche ` (5 subsequent siblings) 6 siblings, 1 reply; 43+ messages in thread From: Adam Manzanares @ 2022-01-28 19:59 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: djwong, linux-nvme, clm, dm-devel, osandov, msnitzer@redhat.com >> msnitzer@redhat.com, Bart Van Assche, linux-scsi, Christoph Hellwig, roland, zach.brown, dsterba, josef, linux-block, mpatocka, kbus >> Keith Busch, Frederick.Knight, Jens Axboe, tytso, martin.petersen@oracle.com >> Martin K. Petersen, jack, linux-fsdevel, lsf-pc On Thu, Jan 27, 2022 at 07:14:13AM +0000, Chaitanya Kulkarni wrote: > Hi, > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > > With reference to the RISC-V summit keynote [1] single threaded > performance is limiting due to Denard scaling and multi-threaded > performance is slowing down due Moore's law limitations. With the rise > of SNIA Computation Technical Storage Working Group (TWG) [2], > offloading computations to the device or over the fabrics is becoming > popular as there are several solutions available [2]. One of the common > operation which is popular in the kernel and is not merged yet is Copy > offload over the fabrics or on to the device. > > * Problem :- > ----------------------------------------------------------------------- > > The original work which is done by Martin is present here [3]. The > latest work which is posted by Mikulas [4] is not merged yet. These two > approaches are totally different from each other. Several storage > vendors discourage mixing copy offload requests with regular READ/WRITE > I/O. Also, the fact that the operation fails if a copy request ever > needs to be split as it traverses the stack it has the unfortunate > side-effect of preventing copy offload from working in pretty much > every common deployment configuration out there. > > * Current state of the work :- > ----------------------------------------------------------------------- > > With [3] being hard to handle arbitrary DM/MD stacking without > splitting the command in two, one for copying IN and one for copying > OUT. Which is then demonstrated by the [4] why [3] it is not a suitable > candidate. Also, with [4] there is an unresolved problem with the > two-command approach about how to handle changes to the DM layout > between an IN and OUT operations. > > We have conducted a call with interested people late last year since > lack of LSFMMM and we would like to share the details with broader > community members. Was on that call and I am interested in joining this discussion. > > * Why Linux Kernel Storage System needs Copy Offload support now ? > ----------------------------------------------------------------------- > > With the rise of the SNIA Computational Storage TWG and solutions [2], > existing SCSI XCopy support in the protocol, recent advancement in the > Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer > DMA support in the Linux Kernel mainly for NVMe devices [7] and > eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit > from Copy offload operation. > > With this background we have significant number of use-cases which are > strong candidates waiting for outstanding Linux Kernel Block Layer Copy > Offload support, so that Linux Kernel Storage subsystem can to address > previously mentioned problems [1] and allow efficient offloading of the > data related operations. (Such as move/copy etc.) > > For reference following is the list of the use-cases/candidates waiting > for Copy Offload support :- > > 1. SCSI-attached storage arrays. > 2. Stacking drivers supporting XCopy DM/MD. > 3. Computational Storage solutions. > 7. File systems :- Local, NFS and Zonefs. > 4. Block devices :- Distributed, local, and Zoned devices. > 5. Peer to Peer DMA support solutions. > 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. > > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > ----------------------------------------------------------------------- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. > > Required attendees :- > > Martin K. Petersen > Jens Axboe > Christoph Hellwig > Bart Van Assche > Zach Brown > Roland Dreier > Ric Wheeler > Trond Myklebust > Mike Snitzer > Keith Busch > Sagi Grimberg > Hannes Reinecke > Frederick Knight > Mikulas Patocka > Keith Busch > > -ck > > [1]https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=3933d1bc-66a8e8f3-39325af3-0cc47a30d446-55df181e6aabd8e8&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Fcontent.riscv.org*2Fwp-content*2Fuploads*2F2018*2F12*2FA-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggCI71vV3$ > [2] https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=e9dc0639-b6473f76-e9dd8d76-0cc47a30d446-03d65bc9ad20d215&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Fwww.snia.org*2Fcomputational__;JSUlJQ!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggLInnHhS$ > https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=13eb47ed-4c707ea2-13eacca2-0cc47a30d446-3d06014a33154497&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Fwww.napatech.com*2Fsupport*2Fresources*2Fsolution-descriptions*2Fnapatech-smartnic-solution-for-hardware-offload*2F__;JSUlJSUlJSU!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggJJSlhVh$ > https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=8ba72fbf-d43c16f0-8ba6a4f0-0cc47a30d446-359457fd63a1a13d&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Fwww.eideticom.com*2Fproducts.html__;JSUlJQ!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggGCerEbv$ > https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=75b96fa9-2a2256e6-75b8e4e6-0cc47a30d446-0403b00d6ff1bab8&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Fwww.xilinx.com*2Fapplications*2Fdata-center*2Fcomputational-storage.html__;JSUlJSUl!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggK0Hp6vG$ > [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy > [4] https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=3a49563e-65d26f71-3a48dd71-0cc47a30d446-3cecc3d55115742b&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Fwww.spinics.net*2Flists*2Flinux-block*2Fmsg00599.html__;JSUlJSUl!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggPvo936U$ > [5] https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=910e6991-ce9550de-910fe2de-0cc47a30d446-c412c0c3c4c51c2b&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Flwn.net*2FArticles*2F793585*2F__;JSUlJSUl!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggIprHJMJ$ > [6] https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=0ab886e2-5523bfad-0ab90dad-0cc47a30d446-df0ae4acca6d59f2&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Fnvmexpress.org*2Fnew-nvmetm-specification-defines-zoned-__;JSUlJQ!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggB4MUwfa$ > namespaces-zns-as-go-to-industry-technology/ > [7] https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=44a1a51b-1b3a9c54-44a02e54-0cc47a30d446-8577b144c92493eb&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Fgithub.com*2Fsbates130272*2Flinux-p2pmem__;JSUlJSU!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggEa99Fso$ > [8] https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=0745845d-58debd12-07440f12-0cc47a30d446-53178030a251a9d8&q=1&e=c880f1d4-0275-4c86-ba38-205de0f24f69&u=https*3A*2F*2Fkernel.dk*2Fio_uring.pdf__;JSUlJQ!!EwVzqGoTKBqv-0DWAJBm!BhtIUewpIpaTRbAVe6VvjiRs-431N4ehiLybkoGuMxLiIvcuYlijJGJWlXVggJUxR2B3$ -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2022-01-28 19:59 ` Adam Manzanares @ 2022-01-31 11:49 ` Johannes Thumshirn 0 siblings, 0 replies; 43+ messages in thread From: Johannes Thumshirn @ 2022-01-31 11:49 UTC (permalink / raw) To: chaitanyak, a.manzanares Cc: djwong, linux-nvme, clm, dm-devel, hch, msnitzer, bvanassche, linux-scsi, osandov, roland, zach.brown, josef, linux-block, mpatocka, dsterba, Frederick.Knight, axboe, tytso, martin.petersen, kbusch, jack, linux-fsdevel, lsf-pc On Fri, 2022-01-28 at 19:59 +0000, Adam Manzanares wrote: > On Thu, Jan 27, 2022 at 07:14:13AM +0000, Chaitanya Kulkarni wrote: > > > > * Current state of the work :- > > ------------------------------------------------------------------- > > ---- > > > > With [3] being hard to handle arbitrary DM/MD stacking without > > splitting the command in two, one for copying IN and one for > > copying > > OUT. Which is then demonstrated by the [4] why [3] it is not a > > suitable > > candidate. Also, with [4] there is an unresolved problem with the > > two-command approach about how to handle changes to the DM layout > > between an IN and OUT operations. > > > > We have conducted a call with interested people late last year > > since > > lack of LSFMMM and we would like to share the details with broader > > community members. > > Was on that call and I am interested in joining this discussion. Same for me :) -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com> 2022-01-28 19:59 ` Adam Manzanares @ 2022-01-31 19:03 ` Bart Van Assche 2022-02-01 1:54 ` Luis Chamberlain ` (4 subsequent siblings) 6 siblings, 0 replies; 43+ messages in thread From: Bart Van Assche @ 2022-01-31 19:03 UTC (permalink / raw) To: Chaitanya Kulkarni, linux-block, linux-scsi, dm-devel, linux-nvme, linux-fsdevel, Jens Axboe, msnitzer@redhat.com >> msnitzer@redhat.com, martin.petersen@oracle.com >> Martin K. Petersen, roland, mpatocka, Hannes Reinecke, kbus >> Keith Busch, Christoph Hellwig, Frederick.Knight, zach.brown, osandov, lsf-pc, djwong, josef, clm, dsterba, tytso, jack On 1/26/22 23:14, Chaitanya Kulkarni wrote: > [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf > [2] https://www.snia.org/computational > https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/ > https://www.eideticom.com/products.html > https://www.xilinx.com/applications/data-center/computational-storage.html > [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy > [4] https://www.spinics.net/lists/linux-block/msg00599.html > [5] https://lwn.net/Articles/793585/ > [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- > namespaces-zns-as-go-to-industry-technology/ > [7] https://github.com/sbates130272/linux-p2pmem > [8] https://kernel.dk/io_uring.pdf Please consider adding the following link to the above list: https://github.com/bvanassche/linux-kernel-copy-offload Thanks, Bart. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com> 2022-01-28 19:59 ` Adam Manzanares 2022-01-31 19:03 ` Bart Van Assche @ 2022-02-01 1:54 ` Luis Chamberlain 2022-02-02 5:57 ` Kanchan Joshi ` (3 subsequent siblings) 6 siblings, 0 replies; 43+ messages in thread From: Luis Chamberlain @ 2022-02-01 1:54 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: djwong, linux-nvme, clm, dm-devel, osandov, msnitzer@redhat.com >> msnitzer@redhat.com, Bart Van Assche, linux-scsi, Christoph Hellwig, roland, zach.brown, dsterba, josef, linux-block, mpatocka, kbus >> Keith Busch, Frederick.Knight, Jens Axboe, tytso, martin.petersen@oracle.com >> Martin K. Petersen, jack, linux-fsdevel, lsf-pc > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > ----------------------------------------------------------------------- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. Consider me intersted in this topic. Luis -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com> ` (2 preceding siblings ...) 2022-02-01 1:54 ` Luis Chamberlain @ 2022-02-02 5:57 ` Kanchan Joshi [not found] ` <20220201102122.4okwj2gipjbvuyux@mpHalley-2> ` (2 subsequent siblings) 6 siblings, 0 replies; 43+ messages in thread From: Kanchan Joshi @ 2022-02-02 5:57 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: djwong, linux-nvme, clm, dm-devel, osandov, msnitzer@redhat.com >> msnitzer@redhat.com, Bart Van Assche, linux-scsi, Christoph Hellwig, roland, zach.brown, dsterba, josef, linux-block, mpatocka, kbus >> Keith Busch, Frederick.Knight, Jens Axboe, tytso, martin.petersen@oracle.com >> Martin K. Petersen, jack, linux-fsdevel, lsf-pc On Thu, Jan 27, 2022 at 12:51 PM Chaitanya Kulkarni <chaitanyak@nvidia.com> wrote: > > Hi, > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > > With reference to the RISC-V summit keynote [1] single threaded > performance is limiting due to Denard scaling and multi-threaded > performance is slowing down due Moore's law limitations. With the rise > of SNIA Computation Technical Storage Working Group (TWG) [2], > offloading computations to the device or over the fabrics is becoming > popular as there are several solutions available [2]. One of the common > operation which is popular in the kernel and is not merged yet is Copy > offload over the fabrics or on to the device. > > * Problem :- > ----------------------------------------------------------------------- > > The original work which is done by Martin is present here [3]. The > latest work which is posted by Mikulas [4] is not merged yet. These two > approaches are totally different from each other. Several storage > vendors discourage mixing copy offload requests with regular READ/WRITE > I/O. Also, the fact that the operation fails if a copy request ever > needs to be split as it traverses the stack it has the unfortunate > side-effect of preventing copy offload from working in pretty much > every common deployment configuration out there. > > * Current state of the work :- > ----------------------------------------------------------------------- > > With [3] being hard to handle arbitrary DM/MD stacking without > splitting the command in two, one for copying IN and one for copying > OUT. Which is then demonstrated by the [4] why [3] it is not a suitable > candidate. Also, with [4] there is an unresolved problem with the > two-command approach about how to handle changes to the DM layout > between an IN and OUT operations. > > We have conducted a call with interested people late last year since > lack of LSFMMM and we would like to share the details with broader > community members. I'm keen on this topic and would like to join the F2F discussion. The Novmber call did establish some consensus on requirements. Planning to have a round or two of code-discussions soon. Thanks, -- Kanchan -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
[parent not found: <20220201102122.4okwj2gipjbvuyux@mpHalley-2>]
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <20220201102122.4okwj2gipjbvuyux@mpHalley-2> @ 2022-02-07 9:57 ` Nitesh Shetty 0 siblings, 0 replies; 43+ messages in thread From: Nitesh Shetty @ 2022-02-07 9:57 UTC (permalink / raw) To: Javier González Cc: djwong, linux-nvme, clm, dm-devel, osandov, msnitzer@redhat.com >> msnitzer@redhat.com, Bart Van Assche, linux-scsi, Christoph Hellwig, roland, zach.brown, dsterba, Chaitanya Kulkarni, josef, linux-block, mpatocka, kbus >> Keith Busch, Frederick.Knight, Jens Axboe, tytso, Kanchan Joshi, martin.petersen@oracle.com >> Martin K. Petersen, jack, linux-fsdevel, lsf-pc Chaitanya, I would like to join the conversation. Thanks, Nitesh On Sun, Feb 6, 2022 at 7:29 PM Javier González <javier@javigon.com> wrote: > > On 27.01.2022 07:14, Chaitanya Kulkarni wrote: > >Hi, > > > >* Background :- > >----------------------------------------------------------------------- > > > >Copy offload is a feature that allows file-systems or storage devices > >to be instructed to copy files/logical blocks without requiring > >involvement of the local CPU. > > > >With reference to the RISC-V summit keynote [1] single threaded > >performance is limiting due to Denard scaling and multi-threaded > >performance is slowing down due Moore's law limitations. With the rise > >of SNIA Computation Technical Storage Working Group (TWG) [2], > >offloading computations to the device or over the fabrics is becoming > >popular as there are several solutions available [2]. One of the common > >operation which is popular in the kernel and is not merged yet is Copy > >offload over the fabrics or on to the device. > > > >* Problem :- > >----------------------------------------------------------------------- > > > >The original work which is done by Martin is present here [3]. The > >latest work which is posted by Mikulas [4] is not merged yet. These two > >approaches are totally different from each other. Several storage > >vendors discourage mixing copy offload requests with regular READ/WRITE > >I/O. Also, the fact that the operation fails if a copy request ever > >needs to be split as it traverses the stack it has the unfortunate > >side-effect of preventing copy offload from working in pretty much > >every common deployment configuration out there. > > > >* Current state of the work :- > >----------------------------------------------------------------------- > > > >With [3] being hard to handle arbitrary DM/MD stacking without > >splitting the command in two, one for copying IN and one for copying > >OUT. Which is then demonstrated by the [4] why [3] it is not a suitable > >candidate. Also, with [4] there is an unresolved problem with the > >two-command approach about how to handle changes to the DM layout > >between an IN and OUT operations. > > > >We have conducted a call with interested people late last year since > >lack of LSFMMM and we would like to share the details with broader > >community members. > > Chaitanya, > > I would also like to join the F2F conversation as a follow up of the > virtual one last year. We will have a first version of the patches > posted in the next few weeks. This will hopefully serve as a good first > step. > > Adding Kanchan to thread too. > > Javier -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com> ` (4 preceding siblings ...) [not found] ` <20220201102122.4okwj2gipjbvuyux@mpHalley-2> @ 2022-02-07 10:45 ` David Disseldorp 2022-03-01 17:34 ` Nikos Tsironis 6 siblings, 0 replies; 43+ messages in thread From: David Disseldorp @ 2022-02-07 10:45 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: Jens Axboe, msnitzer@redhat.com >> msnitzer@redhat.com, tytso, linux-scsi, djwong, josef, linux-nvme, linux-block, clm, dm-devel, dsterba, jack, linux-fsdevel, lsf-pc On Thu, 27 Jan 2022 07:14:13 +0000, Chaitanya Kulkarni wrote: > Hi, > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > > With reference to the RISC-V summit keynote [1] single threaded > performance is limiting due to Denard scaling and multi-threaded > performance is slowing down due Moore's law limitations. With the rise > of SNIA Computation Technical Storage Working Group (TWG) [2], > offloading computations to the device or over the fabrics is becoming > popular as there are several solutions available [2]. One of the common > operation which is popular in the kernel and is not merged yet is Copy > offload over the fabrics or on to the device. > > * Problem :- > ----------------------------------------------------------------------- > > The original work which is done by Martin is present here [3]. The > latest work which is posted by Mikulas [4] is not merged yet. These two > approaches are totally different from each other. Several storage > vendors discourage mixing copy offload requests with regular READ/WRITE > I/O. Also, the fact that the operation fails if a copy request ever > needs to be split as it traverses the stack it has the unfortunate > side-effect of preventing copy offload from working in pretty much > every common deployment configuration out there. > > * Current state of the work :- > ----------------------------------------------------------------------- > > With [3] being hard to handle arbitrary DM/MD stacking without > splitting the command in two, one for copying IN and one for copying > OUT. Which is then demonstrated by the [4] why [3] it is not a suitable > candidate. Also, with [4] there is an unresolved problem with the > two-command approach about how to handle changes to the DM layout > between an IN and OUT operations. > > We have conducted a call with interested people late last year since > lack of LSFMMM and we would like to share the details with broader > community members. > > * Why Linux Kernel Storage System needs Copy Offload support now ? > ----------------------------------------------------------------------- > > With the rise of the SNIA Computational Storage TWG and solutions [2], > existing SCSI XCopy support in the protocol, recent advancement in the > Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer > DMA support in the Linux Kernel mainly for NVMe devices [7] and > eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit > from Copy offload operation. > > With this background we have significant number of use-cases which are > strong candidates waiting for outstanding Linux Kernel Block Layer Copy > Offload support, so that Linux Kernel Storage subsystem can to address > previously mentioned problems [1] and allow efficient offloading of the > data related operations. (Such as move/copy etc.) > > For reference following is the list of the use-cases/candidates waiting > for Copy Offload support :- > > 1. SCSI-attached storage arrays. > 2. Stacking drivers supporting XCopy DM/MD. > 3. Computational Storage solutions. > 7. File systems :- Local, NFS and Zonefs. > 4. Block devices :- Distributed, local, and Zoned devices. > 5. Peer to Peer DMA support solutions. > 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. > > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > ----------------------------------------------------------------------- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. I'd like to attend this discussion. I've worked on the LIO XCOPY implementation in drivers/target/target_core_xcopy.c and added Samba's FSCTL_SRV_COPYCHUNK/FSCTL_DUPLICATE_EXTENTS_TO_FILE support. Cheers, David -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com> ` (5 preceding siblings ...) 2022-02-07 10:45 ` David Disseldorp @ 2022-03-01 17:34 ` Nikos Tsironis [not found] ` <c4124f39-1ee9-8f34-e731-42315fee15f9@nvidia.com> 6 siblings, 1 reply; 43+ messages in thread From: Nikos Tsironis @ 2022-03-01 17:34 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: djwong, linux-nvme, clm, dm-devel, osandov, msnitzer@redhat.com >> msnitzer@redhat.com, Bart Van Assche, linux-scsi, Christoph Hellwig, roland, zach.brown, dsterba, josef, linux-block, mpatocka, kbus >> Keith Busch, Frederick.Knight, Jens Axboe, tytso, martin.petersen@oracle.com >> Martin K. Petersen, jack, linux-fsdevel, lsf-pc On 1/27/22 09:14, Chaitanya Kulkarni wrote: > Hi, > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > > With reference to the RISC-V summit keynote [1] single threaded > performance is limiting due to Denard scaling and multi-threaded > performance is slowing down due Moore's law limitations. With the rise > of SNIA Computation Technical Storage Working Group (TWG) [2], > offloading computations to the device or over the fabrics is becoming > popular as there are several solutions available [2]. One of the common > operation which is popular in the kernel and is not merged yet is Copy > offload over the fabrics or on to the device. > > * Problem :- > ----------------------------------------------------------------------- > > The original work which is done by Martin is present here [3]. The > latest work which is posted by Mikulas [4] is not merged yet. These two > approaches are totally different from each other. Several storage > vendors discourage mixing copy offload requests with regular READ/WRITE > I/O. Also, the fact that the operation fails if a copy request ever > needs to be split as it traverses the stack it has the unfortunate > side-effect of preventing copy offload from working in pretty much > every common deployment configuration out there. > > * Current state of the work :- > ----------------------------------------------------------------------- > > With [3] being hard to handle arbitrary DM/MD stacking without > splitting the command in two, one for copying IN and one for copying > OUT. Which is then demonstrated by the [4] why [3] it is not a suitable > candidate. Also, with [4] there is an unresolved problem with the > two-command approach about how to handle changes to the DM layout > between an IN and OUT operations. > > We have conducted a call with interested people late last year since > lack of LSFMMM and we would like to share the details with broader > community members. > > * Why Linux Kernel Storage System needs Copy Offload support now ? > ----------------------------------------------------------------------- > > With the rise of the SNIA Computational Storage TWG and solutions [2], > existing SCSI XCopy support in the protocol, recent advancement in the > Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer > DMA support in the Linux Kernel mainly for NVMe devices [7] and > eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit > from Copy offload operation. > > With this background we have significant number of use-cases which are > strong candidates waiting for outstanding Linux Kernel Block Layer Copy > Offload support, so that Linux Kernel Storage subsystem can to address > previously mentioned problems [1] and allow efficient offloading of the > data related operations. (Such as move/copy etc.) > > For reference following is the list of the use-cases/candidates waiting > for Copy Offload support :- > > 1. SCSI-attached storage arrays. > 2. Stacking drivers supporting XCopy DM/MD. > 3. Computational Storage solutions. > 7. File systems :- Local, NFS and Zonefs. > 4. Block devices :- Distributed, local, and Zoned devices. > 5. Peer to Peer DMA support solutions. > 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. > > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > ----------------------------------------------------------------------- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. > > Required attendees :- > > Martin K. Petersen > Jens Axboe > Christoph Hellwig > Bart Van Assche > Zach Brown > Roland Dreier > Ric Wheeler > Trond Myklebust > Mike Snitzer > Keith Busch > Sagi Grimberg > Hannes Reinecke > Frederick Knight > Mikulas Patocka > Keith Busch > > -ck > > [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf > [2] https://www.snia.org/computational > https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/ > https://www.eideticom.com/products.html > https://www.xilinx.com/applications/data-center/computational-storage.html > [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy > [4] https://www.spinics.net/lists/linux-block/msg00599.html > [5] https://lwn.net/Articles/793585/ > [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- > namespaces-zns-as-go-to-industry-technology/ > [7] https://github.com/sbates130272/linux-p2pmem > [8] https://kernel.dk/io_uring.pdf I would like to participate in the discussion too. The dm-clone target would also benefit from copy offload, as it heavily employs dm-kcopyd. I have been exploring redesigning kcopyd in order to achieve increased IOPS in dm-clone and dm-snapshot for small copies over NVMe devices, but copy offload sounds even more promising, especially for larger copies happening in the background (as is the case with dm-clone's background hydration). Thanks, Nikos -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
[parent not found: <c4124f39-1ee9-8f34-e731-42315fee15f9@nvidia.com>]
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <c4124f39-1ee9-8f34-e731-42315fee15f9@nvidia.com> @ 2022-03-03 18:36 ` Nikos Tsironis 2022-03-08 20:48 ` Nikos Tsironis 1 sibling, 0 replies; 43+ messages in thread From: Nikos Tsironis @ 2022-03-03 18:36 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: djwong, linux-nvme, clm, dm-devel, osandov, msnitzer@redhat.com >> msnitzer@redhat.com, Bart Van Assche, linux-scsi, Christoph Hellwig, roland, zach.brown, dsterba, josef, linux-block, mpatocka, kbus >> Keith Busch, Frederick.Knight, Jens Axboe, tytso, martin.petersen@oracle.com >> Martin K. Petersen, jack, linux-fsdevel, lsf-pc On 3/1/22 23:32, Chaitanya Kulkarni wrote: > Nikos, > >>> [8] https://kernel.dk/io_uring.pdf >> >> I would like to participate in the discussion too. >> >> The dm-clone target would also benefit from copy offload, as it heavily >> employs dm-kcopyd. I have been exploring redesigning kcopyd in order to >> achieve increased IOPS in dm-clone and dm-snapshot for small copies over >> NVMe devices, but copy offload sounds even more promising, especially >> for larger copies happening in the background (as is the case with >> dm-clone's background hydration). >> >> Thanks, >> Nikos > > If you can document your findings here it will be great for me to > add it to the agenda. > Hi, Give me a few days to gather my notes, because it's been a while since the last time I worked on this, and I will come back with a summary of my findings. Nikos -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload [not found] ` <c4124f39-1ee9-8f34-e731-42315fee15f9@nvidia.com> 2022-03-03 18:36 ` Nikos Tsironis @ 2022-03-08 20:48 ` Nikos Tsironis 2022-03-09 8:51 ` Mikulas Patocka 1 sibling, 1 reply; 43+ messages in thread From: Nikos Tsironis @ 2022-03-08 20:48 UTC (permalink / raw) To: Chaitanya Kulkarni Cc: djwong, linux-nvme, clm, dm-devel, osandov, msnitzer@redhat.com >> msnitzer@redhat.com, Bart Van Assche, linux-scsi, Christoph Hellwig, roland, zach.brown, dsterba, josef, linux-block, mpatocka, kbus >> Keith Busch, Frederick.Knight, Jens Axboe, tytso, martin.petersen@oracle.com >> Martin K. Petersen, jack, linux-fsdevel, lsf-pc On 3/1/22 23:32, Chaitanya Kulkarni wrote: > Nikos, > >>> [8] https://kernel.dk/io_uring.pdf >> >> I would like to participate in the discussion too. >> >> The dm-clone target would also benefit from copy offload, as it heavily >> employs dm-kcopyd. I have been exploring redesigning kcopyd in order to >> achieve increased IOPS in dm-clone and dm-snapshot for small copies over >> NVMe devices, but copy offload sounds even more promising, especially >> for larger copies happening in the background (as is the case with >> dm-clone's background hydration). >> >> Thanks, >> Nikos > > If you can document your findings here it will be great for me to > add it to the agenda. > My work focuses mainly on improving the IOPs and latency of the dm-snapshot target, in order to bring the performance of short-lived snapshots as close as possible to bare-metal performance. My initial performance evaluation of dm-snapshot had revealed a big performance drop, while the snapshot is active; a drop which is not justified by COW alone. Using fio with blktrace I had noticed that the per-CPU I/O distribution was uneven. Although many threads were doing I/O, only a couple of the CPUs ended up submitting I/O requests to the underlying device. The same issue also affects dm-clone, when doing I/O with sizes smaller than the target's region size, where kcopyd is used for COW. The bottleneck here is kcopyd serializing all I/O. Users of kcopyd, such as dm-snapshot and dm-clone, cannot take advantage of the increased I/O parallelism that comes with using blk-mq in modern multi-core systems, because I/Os are issued only by a single CPU at a time, the one on which kcopyd’s thread happens to be running. So, I experimented redesigning kcopyd to prevent I/O serialization by respecting thread locality for I/Os and their completions. This made the distribution of I/O processing uniform across CPUs. My measurements had shown that scaling kcopyd, in combination with scaling dm-snapshot itself [1] [2], can lead to an eventual performance improvement of ~300% increase in sustained throughput and ~80% decrease in I/O latency for transient snapshots, over the null_blk device. The work for scaling dm-snapshot has been merged [1], but, unfortunately, I haven't been able to send upstream my work on kcopyd yet, because I have been really busy with other things the last couple of years. I haven't looked into the details of copy offload yet, but it would be really interesting to see how it affects the performance of random and sequential workloads, and to check how, and if, scaling kcopyd affects the performance, in combination with copy offload. Nikos [1] https://lore.kernel.org/dm-devel/20190317122258.21760-1-ntsironis@arrikto.com/ [2] https://lore.kernel.org/dm-devel/425d7efe-ab3f-67be-264e-9c3b6db229bc@arrikto.com/ -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2022-03-08 20:48 ` Nikos Tsironis @ 2022-03-09 8:51 ` Mikulas Patocka 2022-03-09 15:49 ` Nikos Tsironis 0 siblings, 1 reply; 43+ messages in thread From: Mikulas Patocka @ 2022-03-09 8:51 UTC (permalink / raw) To: Nikos Tsironis Cc: djwong, linux-nvme, clm, dm-devel, osandov, msnitzer@redhat.com >> msnitzer@redhat.com, Bart Van Assche, linux-scsi, Christoph Hellwig, roland, zach.brown, Chaitanya Kulkarni, josef, linux-block, dsterba, kbus @imap.gmail.com>> Keith Busch, Frederick.Knight, Jens Axboe, tytso, martin.petersen@oracle.com >> Martin K. Petersen, jack, linux-fsdevel, lsf-pc [-- Attachment #1: Type: TEXT/PLAIN, Size: 2404 bytes --] On Tue, 8 Mar 2022, Nikos Tsironis wrote: > My work focuses mainly on improving the IOPs and latency of the > dm-snapshot target, in order to bring the performance of short-lived > snapshots as close as possible to bare-metal performance. > > My initial performance evaluation of dm-snapshot had revealed a big > performance drop, while the snapshot is active; a drop which is not > justified by COW alone. > > Using fio with blktrace I had noticed that the per-CPU I/O distribution > was uneven. Although many threads were doing I/O, only a couple of the > CPUs ended up submitting I/O requests to the underlying device. > > The same issue also affects dm-clone, when doing I/O with sizes smaller > than the target's region size, where kcopyd is used for COW. > > The bottleneck here is kcopyd serializing all I/O. Users of kcopyd, such > as dm-snapshot and dm-clone, cannot take advantage of the increased I/O > parallelism that comes with using blk-mq in modern multi-core systems, > because I/Os are issued only by a single CPU at a time, the one on which > kcopyd’s thread happens to be running. > > So, I experimented redesigning kcopyd to prevent I/O serialization by > respecting thread locality for I/Os and their completions. This made the > distribution of I/O processing uniform across CPUs. > > My measurements had shown that scaling kcopyd, in combination with > scaling dm-snapshot itself [1] [2], can lead to an eventual performance > improvement of ~300% increase in sustained throughput and ~80% decrease > in I/O latency for transient snapshots, over the null_blk device. > > The work for scaling dm-snapshot has been merged [1], but, > unfortunately, I haven't been able to send upstream my work on kcopyd > yet, because I have been really busy with other things the last couple > of years. > > I haven't looked into the details of copy offload yet, but it would be > really interesting to see how it affects the performance of random and > sequential workloads, and to check how, and if, scaling kcopyd affects > the performance, in combination with copy offload. > > Nikos Hi Note that you must submit kcopyd callbacks from a single thread, otherwise there's a race condition in snapshot. The snapshot code doesn't take locks in the copy_callback and it expects that the callbacks are serialized. Maybe, adding the locks to copy_callback would solve it. Mikulas [-- Attachment #2: Type: text/plain, Size: 98 bytes --] -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload 2022-03-09 8:51 ` Mikulas Patocka @ 2022-03-09 15:49 ` Nikos Tsironis 0 siblings, 0 replies; 43+ messages in thread From: Nikos Tsironis @ 2022-03-09 15:49 UTC (permalink / raw) To: Mikulas Patocka Cc: djwong, linux-nvme, clm, dm-devel, osandov, msnitzer@redhat.com >> msnitzer@redhat.com, Bart Van Assche, linux-scsi, Christoph Hellwig, roland, zach.brown, Chaitanya Kulkarni, josef, linux-block, dsterba, kbus @imap.gmail.com>> Keith Busch, Frederick.Knight, Jens Axboe, tytso, martin.petersen@oracle.com >> Martin K. Petersen, jack, linux-fsdevel, lsf-pc On 3/9/22 10:51, Mikulas Patocka wrote: > > Hi > > Note that you must submit kcopyd callbacks from a single thread, otherwise > there's a race condition in snapshot. > Hi, Thanks for the feedback. Yes, I'm aware of that. > The snapshot code doesn't take locks in the copy_callback and it expects > that the callbacks are serialized. > > Maybe, adding the locks to copy_callback would solve it. > That's what I did. I used a lock to ensure that kcopyd callbacks are serialized for persistent snapshots. For transient snapshots we can lift this limitation, and complete pending exceptions out-of-oder and in "parallel", i.e., without explicitly serializing kcopyd callbacks. The locks in pending_complete() are enough in this case. Nikos -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 43+ messages in thread
end of thread, other threads:[~2022-03-09 16:12 UTC | newest] Thread overview: 43+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-11 0:15 [dm-devel] [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni 2021-05-11 21:15 ` Knight, Frederick 2021-05-12 2:21 ` Bart Van Assche [not found] ` <CGME20210512071321eucas1p2ca2253e90449108b9f3e4689bf8e0512@eucas1p2.samsung.com> 2021-05-12 7:13 ` Javier González 2021-05-12 7:30 ` Johannes Thumshirn [not found] ` <CGME20210928191342eucas1p23448dcd51b23495fa67cdc017e77435c@eucas1p2.samsung.com> 2021-09-28 19:13 ` Javier González 2021-09-29 6:44 ` Johannes Thumshirn 2021-09-30 9:43 ` Chaitanya Kulkarni 2021-09-30 9:53 ` Javier González 2021-10-06 10:01 ` Javier González 2021-10-13 8:35 ` Javier González 2021-09-30 16:20 ` Bart Van Assche 2021-10-06 10:05 ` Javier González 2021-10-06 17:33 ` Bart Van Assche [not found] ` <20211008064925.oyjxbmngghr2yovr@mpHalley.local> 2021-10-29 0:21 ` Chaitanya Kulkarni 2021-10-29 5:51 ` Hannes Reinecke 2021-10-29 8:16 ` Javier González 2021-10-29 16:15 ` Bart Van Assche 2021-11-01 17:54 ` Keith Busch 2021-10-29 8:14 ` Javier González 2021-11-03 19:27 ` Javier González [not found] ` <20211116134324.hbs3tp5proxootd7@ArmHalley.localdomain> 2021-11-16 17:59 ` Bart Van Assche [not found] ` <20211117125224.z36hp2crpj4fwngc@ArmHalley.local> 2021-11-17 15:52 ` Bart Van Assche [not found] ` <CA+1E3rJRT+89OCyqRtb5BFbez0BfkKvCGijd=nObMEB3_v6MyA@mail.gmail.com> 2021-11-19 16:21 ` Bart Van Assche 2021-05-12 7:36 ` Erwin van Londen 2021-05-12 15:23 ` Hannes Reinecke 2021-05-12 15:45 ` Himanshu Madhani 2021-05-17 16:39 ` Kanchan Joshi 2021-05-18 0:15 ` Bart Van Assche 2021-06-11 6:03 ` Chaitanya Kulkarni 2021-06-11 15:35 ` Nikos Tsironis [not found] <CGME20220127071544uscas1p2f70f4d2509f3ebd574b7ed746d3fa551@uscas1p2.samsung.com> [not found] ` <f0e19ae4-b37a-e9a3-2be7-a5afb334a5c3@nvidia.com> 2022-01-28 19:59 ` Adam Manzanares 2022-01-31 11:49 ` Johannes Thumshirn 2022-01-31 19:03 ` Bart Van Assche 2022-02-01 1:54 ` Luis Chamberlain 2022-02-02 5:57 ` Kanchan Joshi [not found] ` <20220201102122.4okwj2gipjbvuyux@mpHalley-2> 2022-02-07 9:57 ` Nitesh Shetty 2022-02-07 10:45 ` David Disseldorp 2022-03-01 17:34 ` Nikos Tsironis [not found] ` <c4124f39-1ee9-8f34-e731-42315fee15f9@nvidia.com> 2022-03-03 18:36 ` Nikos Tsironis 2022-03-08 20:48 ` Nikos Tsironis 2022-03-09 8:51 ` Mikulas Patocka 2022-03-09 15:49 ` Nikos Tsironis
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).