From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58A3CC48BE0 for ; Fri, 11 Jun 2021 15:36:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 417BD613CA for ; Fri, 11 Jun 2021 15:36:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231815AbhFKPic (ORCPT ); Fri, 11 Jun 2021 11:38:32 -0400 Received: from mail-wr1-f54.google.com ([209.85.221.54]:35492 "EHLO mail-wr1-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231726AbhFKPic (ORCPT ); Fri, 11 Jun 2021 11:38:32 -0400 Received: by mail-wr1-f54.google.com with SMTP id m18so6549697wrv.2 for ; Fri, 11 Jun 2021 08:36:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arrikto-com.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=1gJ6dCUk7MGx3nyeY4tceuRDx0IqD6WQep3tU+Nsnig=; b=ayZ7Th+FUL1IdLuQUsQ649lG93yAnJCp6a/JVmOIYMk9IHNlGhAMerPxPbePxK5pa/ oGwuDiNOkT6QqS4imL2V3EeuYTu7d8FHZDLWKaVY6YrZpKtsfd24C+swlygsxWMSFqhb vNqjrzoCcPdyFswAK5t2cvADMUhdUH4cU0jz1G+9yj2nQJTJ0VJrq3YRAU5f+R+3fgx8 MI6+MaHVpVwG+DLM7NU4fAmhN6q4Vzcy7uefib5mH/SwIwYGU0Ha3ImexgPjxxK41ivo 13M71gi+J7gsNL7V/IQrid+r1r7XfA+QVPT+UfWK4Rt5LFLakGK04iUNq6pCUuzGkyvu oNLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=1gJ6dCUk7MGx3nyeY4tceuRDx0IqD6WQep3tU+Nsnig=; b=sU8AQfs7kPgCPU3jv2bx5DBnu7dZCP5fdNBcmdhZA5oYaUvohOlOH1bW1SynXH0xih 796c1sxEZhFck7Z1ToHuUAMj3DFli7DT1X4BTh2zX4NGi6cqxfJLVpPZzHNQKJa0Tbvt u5QT97fjo7+RVfiQBtdQVj0PbqAMyDDTugKbZHnRDPmSP6cf75hcrDsuckmy/lfH44sz l2KBUFKzrPI03cL/1q+4pfYrnrCo+bcSAbMLKltPvY/BaAaJ/Rht4GOzR+ecFf05iBkx x2xUeUeYQ0voaXAkuxkHHuBhNLLu9+vR/hKxEWmnTXL8k3f6xGMMR61OjXFYOiqmlWF1 1Sqg== X-Gm-Message-State: AOAM530Ju0n1tGTxuadj/9rQzdP7Zc/aSgjmilfq8ne4ibNN4Wg6kpbp dGCb0ndt5fOQLgS6vx2etssLmg== X-Google-Smtp-Source: ABdhPJxhmnoXYJW4akSHMZifbHrI4LLz7dfOgw5vNpc3GS3e+8FnSwwd2BKVjKrCyC2YabFWAwla3w== X-Received: by 2002:a5d:5189:: with SMTP id k9mr4619420wrv.319.1623425733350; Fri, 11 Jun 2021 08:35:33 -0700 (PDT) Received: from [172.16.10.100] (130.43.52.65.dsl.dyn.forthnet.gr. [130.43.52.65]) by smtp.gmail.com with ESMTPSA id m132sm6449128wmf.10.2021.06.11.08.35.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 11 Jun 2021 08:35:32 -0700 (PDT) Subject: Re: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload To: Chaitanya Kulkarni , "linux-block@vger.kernel.org" , "linux-scsi@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "dm-devel@redhat.com" , "lsf-pc@lists.linux-foundation.org" Cc: "axboe@kernel.dk" , "msnitzer@redhat.com" , "bvanassche@acm.org" , "martin.petersen@oracle.com" , "roland@purestorage.com" , "mpatocka@redhat.com" , "hare@suse.de" , "kbusch@kernel.org" , "rwheeler@redhat.com" , "hch@lst.de" , "Frederick.Knight@netapp.com" , "zach.brown@ni.com" , "osandov@fb.com" References: From: Nikos Tsironis Message-ID: <9b14e6c3-8880-4694-1b7e-c82cacd8c9c0@arrikto.com> Date: Fri, 11 Jun 2021 18:35:30 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org On 5/11/21 3:15 AM, Chaitanya Kulkarni wrote: > Hi, > > * Background :- > ----------------------------------------------------------------------- > > Copy offload is a feature that allows file-systems or storage devices > to be instructed to copy files/logical blocks without requiring > involvement of the local CPU. > > With reference to the RISC-V summit keynote [1] single threaded > performance is limiting due to Denard scaling and multi-threaded > performance is slowing down due Moore's law limitations. With the rise > of SNIA Computation Technical Storage Working Group (TWG) [2], > offloading computations to the device or over the fabrics is becoming > popular as there are several solutions available [2]. One of the common > operation which is popular in the kernel and is not merged yet is Copy > offload over the fabrics or on to the device. > > * Problem :- > ----------------------------------------------------------------------- > > The original work which is done by Martin is present here [3]. The > latest work which is posted by Mikulas [4] is not merged yet. These two > approaches are totally different from each other. Several storage > vendors discourage mixing copy offload requests with regular READ/WRITE > I/O. Also, the fact that the operation fails if a copy request ever > needs to be split as it traverses the stack it has the unfortunate > side-effect of preventing copy offload from working in pretty much > every common deployment configuration out there. > > * Current state of the work :- > ----------------------------------------------------------------------- > > With [3] being hard to handle arbitrary DM/MD stacking without > splitting the command in two, one for copying IN and one for copying > OUT. Which is then demonstrated by the [4] why [3] it is not a suitable > candidate. Also, with [4] there is an unresolved problem with the > two-command approach about how to handle changes to the DM layout > between an IN and OUT operations. > > * Why Linux Kernel Storage System needs Copy Offload support now ? > ----------------------------------------------------------------------- > > With the rise of the SNIA Computational Storage TWG and solutions [2], > existing SCSI XCopy support in the protocol, recent advancement in the > Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer > DMA support in the Linux Kernel mainly for NVMe devices [7] and > eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit > from Copy offload operation. > > With this background we have significant number of use-cases which are > strong candidates waiting for outstanding Linux Kernel Block Layer Copy > Offload support, so that Linux Kernel Storage subsystem can to address > previously mentioned problems [1] and allow efficient offloading of the > data related operations. (Such as move/copy etc.) > > For reference following is the list of the use-cases/candidates waiting > for Copy Offload support :- > > 1. SCSI-attached storage arrays. > 2. Stacking drivers supporting XCopy DM/MD. > 3. Computational Storage solutions. > 7. File systems :- Local, NFS and Zonefs. > 4. Block devices :- Distributed, local, and Zoned devices. > 5. Peer to Peer DMA support solutions. > 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. > > * What we will discuss in the proposed session ? > ----------------------------------------------------------------------- > > I'd like to propose a session to go over this topic to understand :- > > 1. What are the blockers for Copy Offload implementation ? > 2. Discussion about having a file system interface. > 3. Discussion about having right system call for user-space. > 4. What is the right way to move this work forward ? > 5. How can we help to contribute and move this work forward ? > > * Required Participants :- > ----------------------------------------------------------------------- > > I'd like to invite file system, block layer, and device drivers > developers to:- > > 1. Share their opinion on the topic. > 2. Share their experience and any other issues with [4]. > 3. Uncover additional details that are missing from this proposal. > > Required attendees :- > > Martin K. Petersen > Jens Axboe > Christoph Hellwig > Bart Van Assche > Zach Brown > Roland Dreier > Ric Wheeler > Trond Myklebust > Mike Snitzer > Keith Busch > Sagi Grimberg > Hannes Reinecke > Frederick Knight > Mikulas Patocka > Keith Busch > > Regards, > Chaitanya > > [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf > [2] https://www.snia.org/computational > https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/ > https://www.eideticom.com/products.html > https://www.xilinx.com/applications/data-center/computational-storage.html > [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy > [4] https://www.spinics.net/lists/linux-block/msg00599.html > [5] https://lwn.net/Articles/793585/ > [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- > namespaces-zns-as-go-to-industry-technology/ > [7] https://github.com/sbates130272/linux-p2pmem > [8] https://kernel.dk/io_uring.pdf > I would like to participate in this discussion too. Thanks, Nikos