From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE659C33C9B for ; Tue, 7 Jan 2020 18:14:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A25B32146E for ; Tue, 7 Jan 2020 18:14:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=wdc.com header.i=@wdc.com header.b="IcHqNOg+"; dkim=pass (1024-bit key) header.d=sharedspace.onmicrosoft.com header.i=@sharedspace.onmicrosoft.com header.b="OuRxqlmq" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728384AbgAGSOY (ORCPT ); Tue, 7 Jan 2020 13:14:24 -0500 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:34387 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727925AbgAGSOY (ORCPT ); Tue, 7 Jan 2020 13:14:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1578420863; x=1609956863; h=from:to:cc:subject:date:message-id: content-transfer-encoding:mime-version; bh=j2tN6+V1pVxcaWER8t6KvObGStMa9m3VVcoDZo3HOsU=; b=IcHqNOg+FY4t/L82GJi0BtiyfGt4cevxrKRPOILjGG+ucEeyKMMBgxVq qEXZtS0E73ycpAHyHrPDkCpYvnaN2WrvMmj4QJlTBrX3uBYpPnty4oEED rJDFe4ejeyjoGu0Bwvvrbmq9GLggthDGijSJ816Jq8AOIISXwXwF+KX02 xoMjbfnd2Q98k26TYkb8ydXXX9F0ShUwJIGv2rsyGglpd62nSKhWbO66Y +mWUvTrywyH2AdrvKYQ2DJGGnuuUfybWrB7mpsmzHJNaNseR3dBjIGj0w bPu+vcxR+2/DpCFKPoFenxxCek+TFUkcqBU+oB8lMQ5uaZ/CK9qZFXWc3 g==; IronPort-SDR: 6NqzSdyPZEDLBekS4PclRh/hvz7Yn5TRJ9wx2TvAoohFquK5SK/fFi+2EoXrGWcq0hzL1IU28F vpgp/qSezSGp8ayjOJYlDwNFSEd5T1Qj8B+81mK1IXYWrPlT06oFyip1tiDBD6Yy+oQkVtdePZ m3W6vCd/6c/uYhAgzlBj/BYUFhBZeEiQ3/Lto+lHF9IH2Yo3BjwCjyXIywz1efaTO8SNmYrmUU kpSlXQ8VjdOXaNSZBiwQBycJ4+sXqQFSQ42gH+rI+8LWaw2jLf6T94VBDmf9O8MfAAlpI4HjrK kA4= X-IronPort-AV: E=Sophos;i="5.69,406,1571673600"; d="scan'208";a="234671188" Received: from mail-bl2nam02lp2056.outbound.protection.outlook.com (HELO NAM02-BL2-obe.outbound.protection.outlook.com) ([104.47.38.56]) by ob1.hgst.iphmx.com with ESMTP; 08 Jan 2020 02:14:18 +0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TNLVkqx2l2BLai+1l5Qh3uQOlhXIqvN5zbho10zBhGfMlkTVB7NUIa5D8LCainIZ7tQtatYxDSYFpFy6UWE5hHyb1NGV3TmYMCLxtIMneQU0GH+um2+AyrPZIH4d/jYCbaAeGyz43yBrnOt82YLnsWAUl4dHY6+SWI0PbNfqKp0IOmWvIeRZl8vbhAihDzOOOrAUOtNFZiF+V7iPIO3WKu/HD0wMYMGQZeBps2gWgnazuloVB5p8SOTKJt0MrRdAUpapkIGC/pEWQ2kCrcsZjOXZ3CTtincv50X1+QqcJzg0jGpg2ndoIsJPoWTe1St+BpJTHEjEcZ5b673VKQdFag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mm+UKAdhfdscAtLxwHrWPWYX+y7j7grRWRWz+3jKFEY=; b=JPWFrFlkO1agLrX/rtUdnIEA8ruNcc5i0W8350xt3xRskWXBxCZyzF6Uk+GWE+NLV3BS8fqGFrMjgCk0zS1dKyKP22GkKDoUbJKGM7udt4RPiyQCigbgELK2KtceCnYxpRnlFACu2icKxY7G+tWRAnIRks1tjMGCugxdTitWIoLh3AMto6Kv9Sv9hCdImIqZyt865NYzedyKDdrUkYNK6MkEK417Q1YBJ4ozcR14OJMhSZiB/EX0QJzCE0H0a8A8BJlcEWCjsbw0Kj3Hd0eoPvQ8ckJDoTwSR48dZ/+V9AZhDYl/psGSOa1E2h1mhJPFlnWieAIJZHlF6z1pknFwMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=wdc.com; dmarc=pass action=none header.from=wdc.com; dkim=pass header.d=wdc.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sharedspace.onmicrosoft.com; s=selector2-sharedspace-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mm+UKAdhfdscAtLxwHrWPWYX+y7j7grRWRWz+3jKFEY=; b=OuRxqlmqL+bKIFRGQB1txxoKewoUGRDUGdhmQ75o1H92ThDCjJ32jCvDlWvHmEBUPOx+uiNKOFcKZB9hj821eknEqv9gbD03XUVPyJ5/61gZkqcQ0Nuey/Y2y/YJqQ7XTtLQ5LYXnYMYkvCAN8DoDi88AvEVCr2y4xaZ3U+KXWw= Received: from BYAPR04MB5749.namprd04.prod.outlook.com (20.179.57.21) by BYAPR04MB4470.namprd04.prod.outlook.com (52.135.237.79) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2602.15; Tue, 7 Jan 2020 18:14:17 +0000 Received: from BYAPR04MB5749.namprd04.prod.outlook.com ([fe80::a8ea:4ba9:cb57:e90f]) by BYAPR04MB5749.namprd04.prod.outlook.com ([fe80::a8ea:4ba9:cb57:e90f%5]) with mapi id 15.20.2602.016; Tue, 7 Jan 2020 18:14:17 +0000 From: Chaitanya Kulkarni To: "linux-block@vger.kernel.org" , "linux-scsi@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "dm-devel@redhat.com" , "lsf-pc@lists.linux-foundation.org" CC: "axboe@kernel.dk" , "bvanassche@acm.org" , "hare@suse.de" , "Martin K. Petersen" , Keith Busch , Christoph Hellwig , Stephen Bates , "msnitzer@redhat.com" , "mpatocka@redhat.com" , "zach.brown@ni.com" , "roland@purestorage.com" , "rwheeler@redhat.com" , "frederick.knight@netapp.com" , Matias Bjorling Subject: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Thread-Topic: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Thread-Index: AQHVxYZFlOEpmvYia0KWnm3aEM0giQ== Date: Tue, 7 Jan 2020 18:14:17 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Chaitanya.Kulkarni@wdc.com; x-originating-ip: [199.255.45.62] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: e26e0278-6748-4345-c3ec-08d7939d6903 x-ms-traffictypediagnostic: BYAPR04MB4470: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: wdcipoutbound: EOP-TRUE x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 027578BB13 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(4636009)(376002)(346002)(396003)(39860400002)(136003)(366004)(189003)(199004)(53754006)(33656002)(2906002)(4326008)(478600001)(55016002)(9686003)(186003)(66476007)(7696005)(66446008)(66946007)(66556008)(6506007)(76116006)(64756008)(54906003)(316002)(110136005)(26005)(8676002)(66574012)(8936002)(81156014)(71200400001)(7416002)(81166006)(966005)(5660300002)(86362001)(52536014)(15398625002);DIR:OUT;SFP:1102;SCL:1;SRVR:BYAPR04MB4470;H:BYAPR04MB5749.namprd04.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: kDiD37JNy1JZmPtUS/37Sot6KGO73Hr4rc05OR1VzDSubpJwBFTBrLy2vPoeuGAMUDb4oy5pRobKRBOlvIJHO3PDlOIaUwFDwBRNDwx6E0rHT77ih8YhfWSca8Ew3nJTXYnRtK2HCwvL8uucm4+J9kOYyPtrNZM8e/yHIDa+91/8cA3n2zKOWAKEzoVydKcOFy+C+d2McSr5/Vb7EG7fpwg0Zf4tggfjt0NAoZ2S75ZmPYrhVNnFs2sWnu8e3Dzn6Xjb8vHKwkQ78uEg4mnajXZ8DaDq9kqf3T6sx7H4muj+ig1W/2ObezykOBLiMCW22m9oYGQl4Cwq+QqsdX3GqBi62qzuW7z6iW1ri8ZlfEN6CslbzPwJ11T6GI44O3M3IxNg6JTwd4V6uxskDZiK5UFra0oPZXZK5Xzdp9bTpnBBqkI4XDjMoe2/uKqB/fRbfk2KPJb4L6AfrkuqJH5geTlZphIyKCyN9CoaxrhFtdYYxe6Q1rDArZ49Mh/1O6twzdsCNl1D3CYPR311WIJF6nPdXpFWRTg1XsOe1YmojErl1R9aq7dzI0uqlsuInACjt6upEoY3J1kWn4Y7WbpUNVmbmALqJdbqzHuQai2FUlE= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: wdc.com X-MS-Exchange-CrossTenant-Network-Message-Id: e26e0278-6748-4345-c3ec-08d7939d6903 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Jan 2020 18:14:17.5528 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: uV11/h+HQVGH+b5DmUGntMKLo6BW2Q3YISL02PtEEeMRyp5pY6axhwkDnCH4dH8iJ4f4lPAG5T0mZXteMpMAYj8MU3qAbjzdfNp/Y0hhgoE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR04MB4470 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Hi all,=0A= =0A= * Background :-=0A= -----------------------------------------------------------------------=0A= =0A= Copy offload is a feature that allows file-systems or storage devices=0A= to be instructed to copy files/logical blocks without requiring=0A= involvement of the local CPU.=0A= =0A= With reference to the RISC-V summit keynote [1] single threaded=0A= performance is limiting due to Denard scaling and multi-threaded=0A= performance is slowing down due Moore's law limitations. With the rise=0A= of SNIA Computation Technical Storage Working Group (TWG) [2],=0A= offloading computations to the device or over the fabrics is becoming=0A= popular as there are several solutions available [2]. One of the common=0A= operation which is popular in the kernel and is not merged yet is Copy=0A= offload over the fabrics or on to the device.=0A= =0A= * Problem :-=0A= -----------------------------------------------------------------------=0A= =0A= The original work which is done by Martin is present here [3]. The=0A= latest work which is posted by Mikulas [4] is not merged yet. These two=0A= approaches are totally different from each other. Several storage=0A= vendors discourage mixing copy offload requests with regular READ/WRITE=0A= I/O. Also, the fact that the operation fails if a copy request ever=0A= needs to be split as it traverses the stack it has the unfortunate=0A= side-effect of preventing copy offload from working in pretty much=0A= every common deployment configuration out there.=0A= =0A= * Current state of the work :-=0A= -----------------------------------------------------------------------=0A= =0A= With [3] being hard to handle arbitrary DM/MD stacking without=0A= splitting the command in two, one for copying IN and one for copying=0A= OUT. Which is then demonstrated by the [4] why [3] it is not a suitable=0A= candidate. Also, with [4] there is an unresolved problem with the=0A= two-command approach about how to handle changes to the DM layout=0A= between an IN and OUT operations.=0A= =0A= * Why Linux Kernel Storage System needs Copy Offload support now ?=0A= -----------------------------------------------------------------------=0A= =0A= With the rise of the SNIA Computational Storage TWG and solutions [2],=0A= existing SCSI XCopy support in the protocol, recent advancement in the=0A= Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer=0A= DMA support in the Linux Kernel mainly for NVMe devices [7] and=0A= eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit=0A= from Copy offload operation.=0A= =0A= With this background we have significant number of use-cases which are=0A= strong candidates waiting for outstanding Linux Kernel Block Layer Copy=0A= Offload support, so that Linux Kernel Storage subsystem can to address=0A= previously mentioned problems [1] and allow efficient offloading of the=0A= data related operations. (Such as move/copy etc.)=0A= =0A= For reference following is the list of the use-cases/candidates waiting=0A= for Copy Offload support :-=0A= =0A= 1. SCSI-attached storage arrays.=0A= 2. Stacking drivers supporting XCopy DM/MD.=0A= 3. Computational Storage solutions.=0A= 7. File systems :- Local, NFS and Zonefs.=0A= 4. Block devices :- Distributed, local, and Zoned devices.=0A= 5. Peer to Peer DMA support solutions.=0A= 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF.=0A= =0A= * What we will discuss in the proposed session ?=0A= -----------------------------------------------------------------------=0A= =0A= I'd like to propose a session to go over this topic to understand :-=0A= =0A= 1. What are the blockers for Copy Offload implementation ?=0A= 2. Discussion about having a file system interface.=0A= 3. Discussion about having right system call for user-space.=0A= 4. What is the right way to move this work forward ?=0A= 5. How can we help to contribute and move this work forward ?=0A= =0A= * Required Participants :-=0A= -----------------------------------------------------------------------=0A= =0A= I'd like to invite block layer, device drivers and file system=0A= developers to:-=0A= =0A= 1. Share their opinion on the topic.=0A= 2. Share their experience and any other issues with [4].=0A= 3. Uncover additional details that are missing from this proposal.=0A= =0A= Required attendees :-=0A= =0A= Martin K. Petersen=0A= Jens Axboe=0A= Christoph Hellwig=0A= Bart Van Assche=0A= Stephen Bates=0A= Zach Brown=0A= Roland Dreier=0A= Ric Wheeler=0A= Trond Myklebust=0A= Mike Snitzer=0A= Keith Busch=0A= Sagi Grimberg=0A= Hannes Reinecke=0A= Frederick Knight=0A= Mikulas Patocka=0A= Matias Bj=F8rling=0A= =0A= [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-fo= r-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterso= n-.pdf=0A= [2] https://www.snia.org/computational=0A= https://www.napatech.com/support/resources/solution-descriptions/napatech-s= martnic-solution-for-hardware-offload/=0A= https://www.eideticom.com/products.html=0A= https://www.xilinx.com/applications/data-center/computational-storage.html= =0A= [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy=0A= [4] https://www.spinics.net/lists/linux-block/msg00599.html=0A= [5] https://lwn.net/Articles/793585/=0A= [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned-=0A= namespaces-zns-as-go-to-industry-technology/=0A= [7] https://github.com/sbates130272/linux-p2pmem=0A= [8] https://kernel.dk/io_uring.pdf=0A= =0A= Regards,=0A= Chaitanya=0A= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E37AEC282DD for ; Tue, 7 Jan 2020 18:15:21 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8E278214D8 for ; Tue, 7 Jan 2020 18:15:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Ucr+mQY0"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=wdc.com header.i=@wdc.com header.b="TZxj0pUv"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sharedspace.onmicrosoft.com header.i=@sharedspace.onmicrosoft.com header.b="OuRxqlmq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E278214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=wdc.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=InZ8xNIWJEJfN5ynimMacedxwiQCkkYVggfZGTF19TU=; b=Ucr+mQY0KMhDGj fsAjQUyoKuoMQ0LOvob7NJjLHIaeDUGYG8pzXhrhTaetbKuVNC1RtNdGmqr95Z6fFy7zx122ysX1N HhL9hRlnOmvdpsGC48Ist0B7PkLxvOXa8KRh8+E27g7G0Wu+WHwMYRBybJLz0MBb9QHtlkCaJjDaA Zh2R+dYJkfcSmvjuSXBpzVyXuUCWXEWHvaFAZcRYa6n9cM44ieQZlZ98LQOtwLXFW7xJAcdQJ3HWz Q2JYlr74HppmmBeao2ANEC1951jMEh57cXvQFfX98o5gyxFoegZi4U4U5G5qeacJ6Vcv6JhxB+W3I wL0bqmsNn7vo3xJGRasQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iotNt-0003TP-Dm; Tue, 07 Jan 2020 18:15:17 +0000 Received: from esa1.hgst.iphmx.com ([68.232.141.245]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iotN5-0001PC-7n for linux-nvme@lists.infradead.org; Tue, 07 Jan 2020 18:14:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1578420867; x=1609956867; h=from:to:cc:subject:date:message-id: content-transfer-encoding:mime-version; bh=j2tN6+V1pVxcaWER8t6KvObGStMa9m3VVcoDZo3HOsU=; b=TZxj0pUvJJaSsxDdfFm+T7QpW8MA7cDTFoMtvUwSJMH5OU5ILhAKr9eU VskBKCsS1q9T4Ui14zUROpvd7zCv1XXBVWefD5L0xLuZptgcc/S3fDflU xm02iizWfkcNTzH1qWYVHdFrYT8Ex8UrMmKniLvCSZU8eSfKCh5kIjRie eJKo5XlJTYQ87DNo/LCS8AFXZR5GwSv6GgpgV/qiM8b+WE0K/NBRDI5VB 0neDcZoxbvrdTwB8/2nw9oKtr2iO5foFRSPuQgRivewOUq1xqTae1Prqc yuwftvsIoGgL23tRFdldCS6nMDwwBJaSlMQ9UiItsoN2xP4lBVAaky8G9 g==; IronPort-SDR: 6NqzSdyPZEDLBekS4PclRh/hvz7Yn5TRJ9wx2TvAoohFquK5SK/fFi+2EoXrGWcq0hzL1IU28F vpgp/qSezSGp8ayjOJYlDwNFSEd5T1Qj8B+81mK1IXYWrPlT06oFyip1tiDBD6Yy+oQkVtdePZ m3W6vCd/6c/uYhAgzlBj/BYUFhBZeEiQ3/Lto+lHF9IH2Yo3BjwCjyXIywz1efaTO8SNmYrmUU kpSlXQ8VjdOXaNSZBiwQBycJ4+sXqQFSQ42gH+rI+8LWaw2jLf6T94VBDmf9O8MfAAlpI4HjrK kA4= X-IronPort-AV: E=Sophos;i="5.69,406,1571673600"; d="scan'208";a="234671188" Received: from mail-bl2nam02lp2056.outbound.protection.outlook.com (HELO NAM02-BL2-obe.outbound.protection.outlook.com) ([104.47.38.56]) by ob1.hgst.iphmx.com with ESMTP; 08 Jan 2020 02:14:18 +0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TNLVkqx2l2BLai+1l5Qh3uQOlhXIqvN5zbho10zBhGfMlkTVB7NUIa5D8LCainIZ7tQtatYxDSYFpFy6UWE5hHyb1NGV3TmYMCLxtIMneQU0GH+um2+AyrPZIH4d/jYCbaAeGyz43yBrnOt82YLnsWAUl4dHY6+SWI0PbNfqKp0IOmWvIeRZl8vbhAihDzOOOrAUOtNFZiF+V7iPIO3WKu/HD0wMYMGQZeBps2gWgnazuloVB5p8SOTKJt0MrRdAUpapkIGC/pEWQ2kCrcsZjOXZ3CTtincv50X1+QqcJzg0jGpg2ndoIsJPoWTe1St+BpJTHEjEcZ5b673VKQdFag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mm+UKAdhfdscAtLxwHrWPWYX+y7j7grRWRWz+3jKFEY=; b=JPWFrFlkO1agLrX/rtUdnIEA8ruNcc5i0W8350xt3xRskWXBxCZyzF6Uk+GWE+NLV3BS8fqGFrMjgCk0zS1dKyKP22GkKDoUbJKGM7udt4RPiyQCigbgELK2KtceCnYxpRnlFACu2icKxY7G+tWRAnIRks1tjMGCugxdTitWIoLh3AMto6Kv9Sv9hCdImIqZyt865NYzedyKDdrUkYNK6MkEK417Q1YBJ4ozcR14OJMhSZiB/EX0QJzCE0H0a8A8BJlcEWCjsbw0Kj3Hd0eoPvQ8ckJDoTwSR48dZ/+V9AZhDYl/psGSOa1E2h1mhJPFlnWieAIJZHlF6z1pknFwMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=wdc.com; dmarc=pass action=none header.from=wdc.com; dkim=pass header.d=wdc.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sharedspace.onmicrosoft.com; s=selector2-sharedspace-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mm+UKAdhfdscAtLxwHrWPWYX+y7j7grRWRWz+3jKFEY=; b=OuRxqlmqL+bKIFRGQB1txxoKewoUGRDUGdhmQ75o1H92ThDCjJ32jCvDlWvHmEBUPOx+uiNKOFcKZB9hj821eknEqv9gbD03XUVPyJ5/61gZkqcQ0Nuey/Y2y/YJqQ7XTtLQ5LYXnYMYkvCAN8DoDi88AvEVCr2y4xaZ3U+KXWw= Received: from BYAPR04MB5749.namprd04.prod.outlook.com (20.179.57.21) by BYAPR04MB4470.namprd04.prod.outlook.com (52.135.237.79) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2602.15; Tue, 7 Jan 2020 18:14:17 +0000 Received: from BYAPR04MB5749.namprd04.prod.outlook.com ([fe80::a8ea:4ba9:cb57:e90f]) by BYAPR04MB5749.namprd04.prod.outlook.com ([fe80::a8ea:4ba9:cb57:e90f%5]) with mapi id 15.20.2602.016; Tue, 7 Jan 2020 18:14:17 +0000 From: Chaitanya Kulkarni To: "linux-block@vger.kernel.org" , "linux-scsi@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "dm-devel@redhat.com" , "lsf-pc@lists.linux-foundation.org" Subject: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Thread-Topic: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Thread-Index: AQHVxYZFlOEpmvYia0KWnm3aEM0giQ== Date: Tue, 7 Jan 2020 18:14:17 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Chaitanya.Kulkarni@wdc.com; x-originating-ip: [199.255.45.62] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: e26e0278-6748-4345-c3ec-08d7939d6903 x-ms-traffictypediagnostic: BYAPR04MB4470: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: wdcipoutbound: EOP-TRUE x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 027578BB13 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(4636009)(376002)(346002)(396003)(39860400002)(136003)(366004)(189003)(199004)(53754006)(33656002)(2906002)(4326008)(478600001)(55016002)(9686003)(186003)(66476007)(7696005)(66446008)(66946007)(66556008)(6506007)(76116006)(64756008)(54906003)(316002)(110136005)(26005)(8676002)(66574012)(8936002)(81156014)(71200400001)(7416002)(81166006)(966005)(5660300002)(86362001)(52536014)(15398625002); DIR:OUT; SFP:1102; SCL:1; SRVR:BYAPR04MB4470; H:BYAPR04MB5749.namprd04.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: kDiD37JNy1JZmPtUS/37Sot6KGO73Hr4rc05OR1VzDSubpJwBFTBrLy2vPoeuGAMUDb4oy5pRobKRBOlvIJHO3PDlOIaUwFDwBRNDwx6E0rHT77ih8YhfWSca8Ew3nJTXYnRtK2HCwvL8uucm4+J9kOYyPtrNZM8e/yHIDa+91/8cA3n2zKOWAKEzoVydKcOFy+C+d2McSr5/Vb7EG7fpwg0Zf4tggfjt0NAoZ2S75ZmPYrhVNnFs2sWnu8e3Dzn6Xjb8vHKwkQ78uEg4mnajXZ8DaDq9kqf3T6sx7H4muj+ig1W/2ObezykOBLiMCW22m9oYGQl4Cwq+QqsdX3GqBi62qzuW7z6iW1ri8ZlfEN6CslbzPwJ11T6GI44O3M3IxNg6JTwd4V6uxskDZiK5UFra0oPZXZK5Xzdp9bTpnBBqkI4XDjMoe2/uKqB/fRbfk2KPJb4L6AfrkuqJH5geTlZphIyKCyN9CoaxrhFtdYYxe6Q1rDArZ49Mh/1O6twzdsCNl1D3CYPR311WIJF6nPdXpFWRTg1XsOe1YmojErl1R9aq7dzI0uqlsuInACjt6upEoY3J1kWn4Y7WbpUNVmbmALqJdbqzHuQai2FUlE= MIME-Version: 1.0 X-OriginatorOrg: wdc.com X-MS-Exchange-CrossTenant-Network-Message-Id: e26e0278-6748-4345-c3ec-08d7939d6903 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Jan 2020 18:14:17.5528 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: uV11/h+HQVGH+b5DmUGntMKLo6BW2Q3YISL02PtEEeMRyp5pY6axhwkDnCH4dH8iJ4f4lPAG5T0mZXteMpMAYj8MU3qAbjzdfNp/Y0hhgoE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR04MB4470 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200107_101427_314048_A1F560A8 X-CRM114-Status: GOOD ( 12.95 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "axboe@kernel.dk" , "msnitzer@redhat.com" , "bvanassche@acm.org" , "Martin K. Petersen" , Matias Bjorling , Stephen Bates , "roland@purestorage.com" , "mpatocka@redhat.com" , "hare@suse.de" , Keith Busch , "rwheeler@redhat.com" , Christoph Hellwig , "frederick.knight@netapp.com" , "zach.brown@ni.com" Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org Hi all, * Background :- ----------------------------------------------------------------------- Copy offload is a feature that allows file-systems or storage devices to be instructed to copy files/logical blocks without requiring involvement of the local CPU. With reference to the RISC-V summit keynote [1] single threaded performance is limiting due to Denard scaling and multi-threaded performance is slowing down due Moore's law limitations. With the rise of SNIA Computation Technical Storage Working Group (TWG) [2], offloading computations to the device or over the fabrics is becoming popular as there are several solutions available [2]. One of the common operation which is popular in the kernel and is not merged yet is Copy offload over the fabrics or on to the device. * Problem :- ----------------------------------------------------------------------- The original work which is done by Martin is present here [3]. The latest work which is posted by Mikulas [4] is not merged yet. These two approaches are totally different from each other. Several storage vendors discourage mixing copy offload requests with regular READ/WRITE I/O. Also, the fact that the operation fails if a copy request ever needs to be split as it traverses the stack it has the unfortunate side-effect of preventing copy offload from working in pretty much every common deployment configuration out there. * Current state of the work :- ----------------------------------------------------------------------- With [3] being hard to handle arbitrary DM/MD stacking without splitting the command in two, one for copying IN and one for copying OUT. Which is then demonstrated by the [4] why [3] it is not a suitable candidate. Also, with [4] there is an unresolved problem with the two-command approach about how to handle changes to the DM layout between an IN and OUT operations. * Why Linux Kernel Storage System needs Copy Offload support now ? ----------------------------------------------------------------------- With the rise of the SNIA Computational Storage TWG and solutions [2], existing SCSI XCopy support in the protocol, recent advancement in the Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer DMA support in the Linux Kernel mainly for NVMe devices [7] and eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit from Copy offload operation. With this background we have significant number of use-cases which are strong candidates waiting for outstanding Linux Kernel Block Layer Copy Offload support, so that Linux Kernel Storage subsystem can to address previously mentioned problems [1] and allow efficient offloading of the data related operations. (Such as move/copy etc.) For reference following is the list of the use-cases/candidates waiting for Copy Offload support :- 1. SCSI-attached storage arrays. 2. Stacking drivers supporting XCopy DM/MD. 3. Computational Storage solutions. 7. File systems :- Local, NFS and Zonefs. 4. Block devices :- Distributed, local, and Zoned devices. 5. Peer to Peer DMA support solutions. 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. * What we will discuss in the proposed session ? ----------------------------------------------------------------------- I'd like to propose a session to go over this topic to understand :- 1. What are the blockers for Copy Offload implementation ? 2. Discussion about having a file system interface. 3. Discussion about having right system call for user-space. 4. What is the right way to move this work forward ? 5. How can we help to contribute and move this work forward ? * Required Participants :- ----------------------------------------------------------------------- I'd like to invite block layer, device drivers and file system developers to:- 1. Share their opinion on the topic. 2. Share their experience and any other issues with [4]. 3. Uncover additional details that are missing from this proposal. Required attendees :- Martin K. Petersen Jens Axboe Christoph Hellwig Bart Van Assche Stephen Bates Zach Brown Roland Dreier Ric Wheeler Trond Myklebust Mike Snitzer Keith Busch Sagi Grimberg Hannes Reinecke Frederick Knight Mikulas Patocka Matias Bj=F8rling [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-fo= r-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterso= n-.pdf [2] https://www.snia.org/computational https://www.napatech.com/support/resources/solution-descriptions/napatech-s= martnic-solution-for-hardware-offload/ https://www.eideticom.com/products.html https://www.xilinx.com/applications/data-center/computational-storage.html [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy [4] https://www.spinics.net/lists/linux-block/msg00599.html [5] https://lwn.net/Articles/793585/ [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- namespaces-zns-as-go-to-industry-technology/ [7] https://github.com/sbates130272/linux-p2pmem [8] https://kernel.dk/io_uring.pdf Regards, Chaitanya _______________________________________________ linux-nvme mailing list linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chaitanya Kulkarni Subject: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Date: Tue, 7 Jan 2020 18:14:17 +0000 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: "linux-block@vger.kernel.org" , "linux-scsi@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "dm-devel@redhat.com" , "lsf-pc@lists.linux-foundation.org" Cc: "axboe@kernel.dk" , "msnitzer@redhat.com" , "bvanassche@acm.org" , "Martin K. Petersen" , Christoph, Matias Bjorling , Stephen Bates , "roland@purestorage.com" , "mpatocka@redhat.com" , Keith Busch , "rwheeler@redhat.com" , Hellwig , "frederick.knight@netapp.com" , "zach.brown@ni.com" List-Id: dm-devel.ids Hi all, * Background :- ----------------------------------------------------------------------- Copy offload is a feature that allows file-systems or storage devices to be instructed to copy files/logical blocks without requiring involvement of the local CPU. With reference to the RISC-V summit keynote [1] single threaded performance is limiting due to Denard scaling and multi-threaded performance is slowing down due Moore's law limitations. With the rise of SNIA Computation Technical Storage Working Group (TWG) [2], offloading computations to the device or over the fabrics is becoming popular as there are several solutions available [2]. One of the common operation which is popular in the kernel and is not merged yet is Copy offload over the fabrics or on to the device. * Problem :- ----------------------------------------------------------------------- The original work which is done by Martin is present here [3]. The latest work which is posted by Mikulas [4] is not merged yet. These two approaches are totally different from each other. Several storage vendors discourage mixing copy offload requests with regular READ/WRITE I/O. Also, the fact that the operation fails if a copy request ever needs to be split as it traverses the stack it has the unfortunate side-effect of preventing copy offload from working in pretty much every common deployment configuration out there. * Current state of the work :- ----------------------------------------------------------------------- With [3] being hard to handle arbitrary DM/MD stacking without splitting the command in two, one for copying IN and one for copying OUT. Which is then demonstrated by the [4] why [3] it is not a suitable candidate. Also, with [4] there is an unresolved problem with the two-command approach about how to handle changes to the DM layout between an IN and OUT operations. * Why Linux Kernel Storage System needs Copy Offload support now ? ----------------------------------------------------------------------- With the rise of the SNIA Computational Storage TWG and solutions [2], existing SCSI XCopy support in the protocol, recent advancement in the Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer DMA support in the Linux Kernel mainly for NVMe devices [7] and eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit from Copy offload operation. With this background we have significant number of use-cases which are strong candidates waiting for outstanding Linux Kernel Block Layer Copy Offload support, so that Linux Kernel Storage subsystem can to address previously mentioned problems [1] and allow efficient offloading of the data related operations. (Such as move/copy etc.) For reference following is the list of the use-cases/candidates waiting for Copy Offload support :- 1. SCSI-attached storage arrays. 2. Stacking drivers supporting XCopy DM/MD. 3. Computational Storage solutions. 7. File systems :- Local, NFS and Zonefs. 4. Block devices :- Distributed, local, and Zoned devices. 5. Peer to Peer DMA support solutions. 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF. * What we will discuss in the proposed session ? ----------------------------------------------------------------------- I'd like to propose a session to go over this topic to understand :- 1. What are the blockers for Copy Offload implementation ? 2. Discussion about having a file system interface. 3. Discussion about having right system call for user-space. 4. What is the right way to move this work forward ? 5. How can we help to contribute and move this work forward ? * Required Participants :- ----------------------------------------------------------------------- I'd like to invite block layer, device drivers and file system developers to:- 1. Share their opinion on the topic. 2. Share their experience and any other issues with [4]. 3. Uncover additional details that are missing from this proposal. Required attendees :- Martin K. Petersen Jens Axboe Christoph Hellwig Bart Van Assche Stephen Bates Zach Brown Roland Dreier Ric Wheeler Trond Myklebust Mike Snitzer Keith Busch Sagi Grimberg Hannes Reinecke Frederick Knight Mikulas Patocka Matias Bj=F8rling [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-fo= r-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterso= n-.pdf [2] https://www.snia.org/computational https://www.napatech.com/support/resources/solution-descriptions/napatech-s= martnic-solution-for-hardware-offload/ https://www.eideticom.com/products.html https://www.xilinx.com/applications/data-center/computational-storage.html [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy [4] https://www.spinics.net/lists/linux-block/msg00599.html [5] https://lwn.net/Articles/793585/ [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned- namespaces-zns-as-go-to-industry-technology/ [7] https://github.com/sbates130272/linux-p2pmem [8] https://kernel.dk/io_uring.pdf Regards, Chaitanya