[TECH TOPIC] Settling Copy Offload via NVMe SCC

[TECH TOPIC] Settling Copy Offload via NVMe SCC

[Selva Jove]

The de-facto way of copying data in I/O stack has been pulling it from
one location followed by pushing to another. The farther the
application, requiring copy, is from storage, the longer it takes for
the trip to be over. With copy-offload the trips get shorter as the
storage device presents an interface to do internal data-copying. This
enables the host to optimise the pull-and-push method, freeing up the
host CPU, RAM and the fabric elements.

The copy-offload interface has existed in SCSI storage for at least a
decade through XCOPY but faced insurmountable challenges in getting
into the Linux I/O stack. As for NVMe storage, copy-offload made its
way into the main specification with a new Simple Copy Command(SCC)
recently. This has stimulated a renewed interest and efforts towards
copy-offload in the Linux community.

In this talk, we speak of the upstream efforts that we are doing around SCC -
https://lore.kernel.org/linux-nvme/20210219124517.79359-1-selvakuma.s1@samsung.com/#r

We'd extensively cover the design-decisions and seek the feedback on
the plumbing aspects such as -

1. User-interface. Should it be a new ioctl/syscall, io_uring based
opcode or must it fit into existing syscalls such as copy_file_range.
2. The transport mode between block-layer and NVMe. A chain of empty
bios (like discard) vs bio with payload.
3. Must SCSI XCOPY compatibility be considered while we go about
building interfaces around NVMe SCC?
4. Feasibility and challenges for in-kernel use cases, including the
file-systems and device-mappers

Thanks,
--------------
Selva & Nitesh

Re: [TECH TOPIC] Settling Copy Offload via NVMe SCC

[Bart Van Assche]

On 6/25/21 7:47 AM, Selva Jove wrote:
> The de-facto way of copying data in I/O stack has been pulling it from
> one location followed by pushing to another. The farther the
> application, requiring copy, is from storage, the longer it takes for
> the trip to be over. With copy-offload the trips get shorter as the
> storage device presents an interface to do internal data-copying. This
> enables the host to optimise the pull-and-push method, freeing up the
> host CPU, RAM and the fabric elements.
> 
> The copy-offload interface has existed in SCSI storage for at least a
> decade through XCOPY but faced insurmountable challenges in getting
> into the Linux I/O stack. As for NVMe storage, copy-offload made its
> way into the main specification with a new Simple Copy Command(SCC)
> recently. This has stimulated a renewed interest and efforts towards
> copy-offload in the Linux community.
> 
> In this talk, we speak of the upstream efforts that we are doing around SCC -
> https://lore.kernel.org/linux-nvme/20210219124517.79359-1-selvakuma.s1@samsung.com/#r
> 
> We'd extensively cover the design-decisions and seek the feedback on
> the plumbing aspects such as -
> 
> 1. User-interface. Should it be a new ioctl/syscall, io_uring based
> opcode or must it fit into existing syscalls such as copy_file_range.
> 2. The transport mode between block-layer and NVMe. A chain of empty
> bios (like discard) vs bio with payload.
> 3. Must SCSI XCOPY compatibility be considered while we go about
> building interfaces around NVMe SCC?
> 4. Feasibility and challenges for in-kernel use cases, including the
> file-systems and device-mappers

This topic seems closely related to "Storage: Copy Offload"
(https://lore.kernel.org/linux-block/BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com/).

Notes about implementing copy offloading are available at
https://github.com/bvanassche/linux-kernel-copy-offload.

Thanks,

Bart.

Re: [TECH TOPIC] Settling Copy Offload via NVMe SCC

[James Bottomley]

On Fri, 2021-06-25 at 09:02 -0700, Bart Van Assche wrote:
> On 6/25/21 7:47 AM, Selva Jove wrote:
> > The de-facto way of copying data in I/O stack has been pulling it
> > from one location followed by pushing to another. The farther the
> > application, requiring copy, is from storage, the longer it takes
> > for the trip to be over. With copy-offload the trips get shorter as
> > the storage device presents an interface to do internal data-
> > copying.  This enables the host to optimise the pull-and-push
> > method, freeing up the host CPU, RAM and the fabric elements.
> > 
> > The copy-offload interface has existed in SCSI storage for at least
> > a decade through XCOPY but faced insurmountable challenges in
> > getting into the Linux I/O stack. As for NVMe storage, copy-offload 
> > made its way into the main specification with a new Simple Copy
> > Command(SCC) recently. This has stimulated a renewed interest and
> > efforts towards copy-offload in the Linux community.
> > 
> > In this talk, we speak of the upstream efforts that we are doing
> > around SCC -
> > https://lore.kernel.org/linux-nvme/20210219124517.79359-1-selvakuma.s1@samsung.com/#r
> > 
> > We'd extensively cover the design-decisions and seek the feedback
> > on the plumbing aspects such as -
> > 
> > 1. User-interface. Should it be a new ioctl/syscall, io_uring based
> > opcode or must it fit into existing syscalls such as
> > copy_file_range. 
> > 2. The transport mode between block-layer and NVMe. A chain of
> > empty bios (like discard) vs bio with payload.
> > 3. Must SCSI XCOPY compatibility be considered while we go about
> > building interfaces around NVMe SCC?
> > 4. Feasibility and challenges for in-kernel use cases, including
> > the file-systems and device-mappers
> 
> This topic seems closely related to "Storage: Copy Offload"
> (
> https://lore.kernel.org/linux-block/BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com/
> ).

Not only that but it would seem to be better suited for LSF/MM on the
IO track than the kernel summit track of Plumbers:

https://lore.kernel.org/bpf/60ca126b.1c69fb81.90af4.93a7@mx.google.com/

Since the plumbing aspects are more how would current practitioners use
the API and how might we plumb it into filesystems.

James


> Notes about implementing copy offloading are available at
> https://github.com/bvanassche/linux-kernel-copy-offload.
> 
> Thanks,
> 
> Bart.
> 

Re: [TECH TOPIC] Settling Copy Offload via NVMe SCC

[Selva Jove]

Hi Bart and James,

Thanks for the heads up.
LSF/MM is a fine idea for the topic. It's just that we had a bit of uncertainty
about in-person event and December felt somewhat distant. So we thought
of making some progress by posting this to LPC.

Thanks,
Selva & Nitesh

On Fri, Jun 25, 2021 at 9:38 PM James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
>
> On Fri, 2021-06-25 at 09:02 -0700, Bart Van Assche wrote:
> > On 6/25/21 7:47 AM, Selva Jove wrote:
> > > The de-facto way of copying data in I/O stack has been pulling it
> > > from one location followed by pushing to another. The farther the
> > > application, requiring copy, is from storage, the longer it takes
> > > for the trip to be over. With copy-offload the trips get shorter as
> > > the storage device presents an interface to do internal data-
> > > copying.  This enables the host to optimise the pull-and-push
> > > method, freeing up the host CPU, RAM and the fabric elements.
> > >
> > > The copy-offload interface has existed in SCSI storage for at least
> > > a decade through XCOPY but faced insurmountable challenges in
> > > getting into the Linux I/O stack. As for NVMe storage, copy-offload
> > > made its way into the main specification with a new Simple Copy
> > > Command(SCC) recently. This has stimulated a renewed interest and
> > > efforts towards copy-offload in the Linux community.
> > >
> > > In this talk, we speak of the upstream efforts that we are doing
> > > around SCC -
> > > https://lore.kernel.org/linux-nvme/20210219124517.79359-1-selvakuma.s1@samsung.com/#r
> > >
> > > We'd extensively cover the design-decisions and seek the feedback
> > > on the plumbing aspects such as -
> > >
> > > 1. User-interface. Should it be a new ioctl/syscall, io_uring based
> > > opcode or must it fit into existing syscalls such as
> > > copy_file_range.
> > > 2. The transport mode between block-layer and NVMe. A chain of
> > > empty bios (like discard) vs bio with payload.
> > > 3. Must SCSI XCOPY compatibility be considered while we go about
> > > building interfaces around NVMe SCC?
> > > 4. Feasibility and challenges for in-kernel use cases, including
> > > the file-systems and device-mappers
> >
> > This topic seems closely related to "Storage: Copy Offload"
> > (
> > https://lore.kernel.org/linux-block/BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com/
> > ).
>
> Not only that but it would seem to be better suited for LSF/MM on the
> IO track than the kernel summit track of Plumbers:
>
> https://lore.kernel.org/bpf/60ca126b.1c69fb81.90af4.93a7@mx.google.com/
>
> Since the plumbing aspects are more how would current practitioners use
> the API and how might we plumb it into filesystems.
>
> James
>
>
> > Notes about implementing copy offloading are available at
> > https://github.com/bvanassche/linux-kernel-copy-offload.
> >
> > Thanks,
> >
> > Bart.
> >
>
>