The de-facto way of copying data in I/O stack has been pulling it from one location followed by pushing to another. The farther the application, requiring copy, is from storage, the longer it takes for the trip to be over. With copy-offload the trips get shorter as the storage device presents an interface to do internal data-copying. This enables the host to optimise the pull-and-push method, freeing up the host CPU, RAM and the fabric elements. The copy-offload interface has existed in SCSI storage for at least a decade through XCOPY but faced insurmountable challenges in getting into the Linux I/O stack. As for NVMe storage, copy-offload made its way into the main specification with a new Simple Copy Command(SCC) recently. This has stimulated a renewed interest and efforts towards copy-offload in the Linux community. In this talk, we speak of the upstream efforts that we are doing around SCC - https://lore.kernel.org/linux-nvme/20210219124517.79359-1-selvakuma.s1@samsung.com/#r We'd extensively cover the design-decisions and seek the feedback on the plumbing aspects such as - 1. User-interface. Should it be a new ioctl/syscall, io_uring based opcode or must it fit into existing syscalls such as copy_file_range. 2. The transport mode between block-layer and NVMe. A chain of empty bios (like discard) vs bio with payload. 3. Must SCSI XCOPY compatibility be considered while we go about building interfaces around NVMe SCC? 4. Feasibility and challenges for in-kernel use cases, including the file-systems and device-mappers Thanks, -------------- Selva & Nitesh
On 6/25/21 7:47 AM, Selva Jove wrote: > The de-facto way of copying data in I/O stack has been pulling it from > one location followed by pushing to another. The farther the > application, requiring copy, is from storage, the longer it takes for > the trip to be over. With copy-offload the trips get shorter as the > storage device presents an interface to do internal data-copying. This > enables the host to optimise the pull-and-push method, freeing up the > host CPU, RAM and the fabric elements. > > The copy-offload interface has existed in SCSI storage for at least a > decade through XCOPY but faced insurmountable challenges in getting > into the Linux I/O stack. As for NVMe storage, copy-offload made its > way into the main specification with a new Simple Copy Command(SCC) > recently. This has stimulated a renewed interest and efforts towards > copy-offload in the Linux community. > > In this talk, we speak of the upstream efforts that we are doing around SCC - > https://lore.kernel.org/linux-nvme/20210219124517.79359-1-selvakuma.s1@samsung.com/#r > > We'd extensively cover the design-decisions and seek the feedback on > the plumbing aspects such as - > > 1. User-interface. Should it be a new ioctl/syscall, io_uring based > opcode or must it fit into existing syscalls such as copy_file_range. > 2. The transport mode between block-layer and NVMe. A chain of empty > bios (like discard) vs bio with payload. > 3. Must SCSI XCOPY compatibility be considered while we go about > building interfaces around NVMe SCC? > 4. Feasibility and challenges for in-kernel use cases, including the > file-systems and device-mappers This topic seems closely related to "Storage: Copy Offload" (https://lore.kernel.org/linux-block/BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com/). Notes about implementing copy offloading are available at https://github.com/bvanassche/linux-kernel-copy-offload. Thanks, Bart.
On Fri, 2021-06-25 at 09:02 -0700, Bart Van Assche wrote: > On 6/25/21 7:47 AM, Selva Jove wrote: > > The de-facto way of copying data in I/O stack has been pulling it > > from one location followed by pushing to another. The farther the > > application, requiring copy, is from storage, the longer it takes > > for the trip to be over. With copy-offload the trips get shorter as > > the storage device presents an interface to do internal data- > > copying. This enables the host to optimise the pull-and-push > > method, freeing up the host CPU, RAM and the fabric elements. > > > > The copy-offload interface has existed in SCSI storage for at least > > a decade through XCOPY but faced insurmountable challenges in > > getting into the Linux I/O stack. As for NVMe storage, copy-offload > > made its way into the main specification with a new Simple Copy > > Command(SCC) recently. This has stimulated a renewed interest and > > efforts towards copy-offload in the Linux community. > > > > In this talk, we speak of the upstream efforts that we are doing > > around SCC - > > https://lore.kernel.org/linux-nvme/20210219124517.79359-1-selvakuma.s1@samsung.com/#r > > > > We'd extensively cover the design-decisions and seek the feedback > > on the plumbing aspects such as - > > > > 1. User-interface. Should it be a new ioctl/syscall, io_uring based > > opcode or must it fit into existing syscalls such as > > copy_file_range. > > 2. The transport mode between block-layer and NVMe. A chain of > > empty bios (like discard) vs bio with payload. > > 3. Must SCSI XCOPY compatibility be considered while we go about > > building interfaces around NVMe SCC? > > 4. Feasibility and challenges for in-kernel use cases, including > > the file-systems and device-mappers > > This topic seems closely related to "Storage: Copy Offload" > ( > https://lore.kernel.org/linux-block/BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com/ > ). Not only that but it would seem to be better suited for LSF/MM on the IO track than the kernel summit track of Plumbers: https://lore.kernel.org/bpf/60ca126b.1c69fb81.90af4.93a7@mx.google.com/ Since the plumbing aspects are more how would current practitioners use the API and how might we plumb it into filesystems. James > Notes about implementing copy offloading are available at > https://github.com/bvanassche/linux-kernel-copy-offload. > > Thanks, > > Bart. >
Hi Bart and James,
Thanks for the heads up.
LSF/MM is a fine idea for the topic. It's just that we had a bit of uncertainty
about in-person event and December felt somewhat distant. So we thought
of making some progress by posting this to LPC.
Thanks,
Selva & Nitesh
On Fri, Jun 25, 2021 at 9:38 PM James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
>
> On Fri, 2021-06-25 at 09:02 -0700, Bart Van Assche wrote:
> > On 6/25/21 7:47 AM, Selva Jove wrote:
> > > The de-facto way of copying data in I/O stack has been pulling it
> > > from one location followed by pushing to another. The farther the
> > > application, requiring copy, is from storage, the longer it takes
> > > for the trip to be over. With copy-offload the trips get shorter as
> > > the storage device presents an interface to do internal data-
> > > copying. This enables the host to optimise the pull-and-push
> > > method, freeing up the host CPU, RAM and the fabric elements.
> > >
> > > The copy-offload interface has existed in SCSI storage for at least
> > > a decade through XCOPY but faced insurmountable challenges in
> > > getting into the Linux I/O stack. As for NVMe storage, copy-offload
> > > made its way into the main specification with a new Simple Copy
> > > Command(SCC) recently. This has stimulated a renewed interest and
> > > efforts towards copy-offload in the Linux community.
> > >
> > > In this talk, we speak of the upstream efforts that we are doing
> > > around SCC -
> > > https://lore.kernel.org/linux-nvme/20210219124517.79359-1-selvakuma.s1@samsung.com/#r
> > >
> > > We'd extensively cover the design-decisions and seek the feedback
> > > on the plumbing aspects such as -
> > >
> > > 1. User-interface. Should it be a new ioctl/syscall, io_uring based
> > > opcode or must it fit into existing syscalls such as
> > > copy_file_range.
> > > 2. The transport mode between block-layer and NVMe. A chain of
> > > empty bios (like discard) vs bio with payload.
> > > 3. Must SCSI XCOPY compatibility be considered while we go about
> > > building interfaces around NVMe SCC?
> > > 4. Feasibility and challenges for in-kernel use cases, including
> > > the file-systems and device-mappers
> >
> > This topic seems closely related to "Storage: Copy Offload"
> > (
> > https://lore.kernel.org/linux-block/BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com/
> > ).
>
> Not only that but it would seem to be better suited for LSF/MM on the
> IO track than the kernel summit track of Plumbers:
>
> https://lore.kernel.org/bpf/60ca126b.1c69fb81.90af4.93a7@mx.google.com/
>
> Since the plumbing aspects are more how would current practitioners use
> the API and how might we plumb it into filesystems.
>
> James
>
>
> > Notes about implementing copy offloading are available at
> > https://github.com/bvanassche/linux-kernel-copy-offload.
> >
> > Thanks,
> >
> > Bart.
> >
>
>