From: Nitesh Shetty <nj.shetty@samsung.com>
To: lsf-pc@lists.linux-foundation.org
Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-nvme@lists.infradead.org,
Nitesh Shetty <nj.shetty@samsung.com>
Subject: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload
Date: Fri, 13 Jan 2023 15:16:48 +0530 [thread overview]
Message-ID: <20230113094648.15614-1-nj.shetty@samsung.com> (raw)
In-Reply-To: CGME20230113094723epcas5p2f6f81ca1ad85f4b26829b87f8ec301ce@epcas5p2.samsung.com
Hi all,
Background:
===========
Copy offload is a feature that allows file-systems or storage devices
to be instructed to copy files/logical blocks without requiring
involvement of the local CPU.
We are working on copy offload[1], mainly focused on NVMe Copy command
(single NS, TP 4065) and NVMe over fabrics.
Previously Chaitanya Kulkarni presented talk on copy offload[2].
Current state of work:
======================
Our latest patch series[1] covers
1. Driver
- NVMe Copy command (single NS, TP 4065), including support
in nvme-target (for block and file back-end).
2. Block layer
- Block-generic copy (REQ_COPY flag), with interface
accommodating two block-devs, and multi-source/destination
interface
- Emulation, when offload is natively absent
- dm-linear support (for cases not requiring split)
3. User-interface
- new ioctl
4. In-kernel user
- dm-kcopyd
5. Tools:
- fio
- blktests
Performance:
With the async design of copy-emulation/offload using fio[3],
we were able to see the following improvements as
compared to userspace read and write on a NVMeOF TCP setup:
Setup1: Network Speed: 1000Mb/s
Host PC: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
Target PC: AMD Ryzen 9 5900X 12-Core Processor
block size 8k, range 1:
635% improvement in IO BW (107 MiB/s to 787 MiB/s).
Network utilisation drops from 97% to 14%.
block-size 2M, range 16:
2555% improvement in IO BW (100 MiB/s to 2655 MiB/s).
Network utilisation drops from 89% to 0.62%.
Setup2: Network Speed: 100Gb/s
Server: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz, 72 cores
(host and target have the same configuration)
block-size 8k, range 1:
6.5% improvement in IO BW (791 MiB/s to 843 MiB/s).
Network utilisation drops from 6.75% to 0.14%.
block-size 2M, range 16:
15% improvement in IO BW (1027 MiB/s to 1183 MiB/s).
Network utilisation drops from 8.42% to ~0%.
Overall, in these tests, kernel copy emulation reduces network utilisation
drastically and improves IO bandwidth.
What we will discuss in the proposed session ?
==============================================
Through this session I would like to discuss and get opinion of community
on minimum set of requirement for copy offload for this phase. Also share
some of the blockers we are facing and would get opinion on how we can
proceed further.
Required attendees:
===================
Martin K. Petersen
Jens Axboe
Christoph Hellwig
Mike Snitzer
Hannes Reinecke
Chaitanya Kulkarni
Bart Van Assche
Damien Le Moal
Mikulas Patocka
Keith Busch
Sagi Grimberg
Javier Gonzalez
Kanchan Joshi
Links:
======
[1] https://lore.kernel.org/lkml/20230112115908.23662-1-nj.shetty@samsung.com/T/#m91a2f506eaa214035a8596fa9aa8d2b9f46654cc
[2] https://lore.kernel.org/all/BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com/
[3] https://github.com/vincentkfu/fio/tree/copyoffload
next parent reply other threads:[~2023-01-13 9:53 UTC|newest]
Thread overview: 104+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CGME20230113094723epcas5p2f6f81ca1ad85f4b26829b87f8ec301ce@epcas5p2.samsung.com>
2023-01-13 9:46 ` Nitesh Shetty [this message]
[not found] <CGME20220127071544uscas1p2f70f4d2509f3ebd574b7ed746d3fa551@uscas1p2.samsung.com>
2022-01-27 7:14 ` [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload Chaitanya Kulkarni
2022-01-28 19:59 ` Adam Manzanares
2022-01-31 11:49 ` Johannes Thumshirn
2022-01-31 19:03 ` Bart Van Assche
2022-02-01 1:54 ` Luis Chamberlain
2022-02-01 10:21 ` Javier González
2022-02-07 9:57 ` Nitesh Shetty
2022-02-02 5:57 ` Kanchan Joshi
2022-02-07 10:45 ` David Disseldorp
2022-03-01 17:34 ` Nikos Tsironis
2022-03-01 21:32 ` Chaitanya Kulkarni
2022-03-03 18:36 ` Nikos Tsironis
2022-03-08 20:48 ` Nikos Tsironis
2022-03-09 8:51 ` Mikulas Patocka
2022-03-09 15:49 ` Nikos Tsironis
2021-05-11 0:15 Chaitanya Kulkarni
2021-05-11 0:15 ` Chaitanya Kulkarni
2021-05-11 21:15 ` Knight, Frederick
2021-05-11 21:15 ` Knight, Frederick
2021-05-12 2:21 ` Bart Van Assche
2021-05-12 2:21 ` Bart Van Assche
[not found] ` <CGME20210512071321eucas1p2ca2253e90449108b9f3e4689bf8e0512@eucas1p2.samsung.com>
2021-05-12 7:13 ` Javier González
2021-05-12 7:13 ` Javier González
2021-05-12 7:30 ` Johannes Thumshirn
2021-05-12 7:30 ` Johannes Thumshirn
[not found] ` <CGME20210928191342eucas1p23448dcd51b23495fa67cdc017e77435c@eucas1p2.samsung.com>
2021-09-28 19:13 ` Javier González
2021-09-28 19:13 ` Javier González
2021-09-29 6:44 ` Johannes Thumshirn
2021-09-29 6:44 ` Johannes Thumshirn
2021-09-30 9:43 ` Chaitanya Kulkarni
2021-09-30 9:43 ` Chaitanya Kulkarni
2021-09-30 9:53 ` Javier González
2021-09-30 9:53 ` Javier González
2021-10-06 10:01 ` Javier González
2021-10-06 10:01 ` Javier González
2021-10-13 8:35 ` Javier González
2021-10-13 8:35 ` Javier González
2021-09-30 16:20 ` Bart Van Assche
2021-09-30 16:20 ` Bart Van Assche
2021-10-06 10:05 ` Javier González
2021-10-06 10:05 ` Javier González
2021-10-06 17:33 ` Bart Van Assche
2021-10-06 17:33 ` Bart Van Assche
2021-10-08 6:49 ` Javier González
2021-10-08 6:49 ` Javier González
2021-10-29 0:21 ` Chaitanya Kulkarni
2021-10-29 5:51 ` Hannes Reinecke
2021-10-29 8:16 ` Javier González
2021-10-29 16:15 ` Bart Van Assche
2021-11-01 17:54 ` Keith Busch
2021-10-29 8:14 ` Javier González
2021-11-03 19:27 ` Javier González
2021-11-16 13:43 ` Javier González
2021-11-16 17:59 ` Bart Van Assche
2021-11-17 12:53 ` Javier González
2021-11-17 15:52 ` Bart Van Assche
2021-11-19 7:38 ` Javier González
2021-11-19 10:47 ` Kanchan Joshi
2021-11-19 15:51 ` Keith Busch
2021-11-19 16:21 ` Bart Van Assche
2021-11-22 7:39 ` Kanchan Joshi
2021-05-12 15:23 ` Hannes Reinecke
2021-05-12 15:23 ` Hannes Reinecke
2021-05-12 15:45 ` Himanshu Madhani
2021-05-12 15:45 ` Himanshu Madhani
2021-05-17 16:39 ` Kanchan Joshi
2021-05-17 16:39 ` Kanchan Joshi
2021-05-18 0:15 ` Bart Van Assche
2021-05-18 0:15 ` Bart Van Assche
2021-06-11 6:03 ` Chaitanya Kulkarni
2021-06-11 6:03 ` Chaitanya Kulkarni
2021-06-11 15:35 ` Nikos Tsironis
2021-06-11 15:35 ` Nikos Tsironis
[not found] <CGME20200107181551epcas5p4f47eeafd807c28a26b4024245c4e00ab@epcas5p4.samsung.com>
2020-01-07 18:14 ` Chaitanya Kulkarni
2020-01-07 18:14 ` Chaitanya Kulkarni
2020-01-07 18:14 ` Chaitanya Kulkarni
2020-01-08 10:17 ` Javier González
2020-01-08 10:17 ` Javier González
2020-01-08 10:17 ` Javier González
2020-01-09 0:51 ` Logan Gunthorpe
2020-01-09 0:51 ` Logan Gunthorpe
2020-01-09 0:51 ` Logan Gunthorpe
2020-01-09 3:18 ` Bart Van Assche
2020-01-09 3:18 ` Bart Van Assche
2020-01-09 3:18 ` Bart Van Assche
2020-01-09 4:01 ` Chaitanya Kulkarni
2020-01-09 4:01 ` Chaitanya Kulkarni
2020-01-09 4:01 ` Chaitanya Kulkarni
2020-01-09 5:56 ` Damien Le Moal
2020-01-09 5:56 ` Damien Le Moal
2020-01-09 5:56 ` Damien Le Moal
2020-01-10 5:33 ` Martin K. Petersen
2020-01-10 5:33 ` Martin K. Petersen
2020-01-10 5:33 ` Martin K. Petersen
2020-01-24 14:23 ` Nikos Tsironis
2020-01-24 14:23 ` Nikos Tsironis
2020-01-24 14:23 ` Nikos Tsironis
2020-02-13 5:11 ` joshi.k
2020-02-13 5:11 ` joshi.k
2020-02-13 5:11 ` joshi.k
2020-02-13 13:09 ` Knight, Frederick
2020-02-13 13:09 ` Knight, Frederick
2020-02-13 13:09 ` Knight, Frederick
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230113094648.15614-1-nj.shetty@samsung.com \
--to=nj.shetty@samsung.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-scsi@vger.kernel.org \
--cc=lsf-pc@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.