RE: [PATCH 5/5] nvme: support for zoned namespaces

[Matias Bjorling <Matias.Bjorling@wdc.com>]

> -----Original Message-----
> From: Javier González <javier@javigon.com>
> Sent: Tuesday, 16 June 2020 18.03
> To: Matias Bjørling <mb@lightnvm.io>
> Cc: Damien Le Moal <Damien.LeMoal@wdc.com>; Jens Axboe
> <axboe@kernel.dk>; Niklas Cassel <Niklas.Cassel@wdc.com>; Ajay Joshi
> <Ajay.Joshi@wdc.com>; Sagi Grimberg <sagi@grimberg.me>; Keith Busch
> <Keith.Busch@wdc.com>; Dmitry Fomichev <Dmitry.Fomichev@wdc.com>;
> Aravind Ramesh <Aravind.Ramesh@wdc.com>; linux-
> nvme@lists.infradead.org; linux-block@vger.kernel.org; Hans Holmberg
> <Hans.Holmberg@wdc.com>; Christoph Hellwig <hch@lst.de>; Matias Bjorling
> <Matias.Bjorling@wdc.com>
> Subject: Re: [PATCH 5/5] nvme: support for zoned namespaces
> 
> On 16.06.2020 17:20, Matias Bjørling wrote:
> >On 16/06/2020 17.02, Javier González wrote:
> >>On 16.06.2020 14:42, Damien Le Moal wrote:
> >>>On 2020/06/16 23:16, Javier González wrote:
> >>>>On 16.06.2020 12:35, Damien Le Moal wrote:
> >>>>>On 2020/06/16 21:24, Javier González wrote:
> >>>>>>On 16.06.2020 14:06, Matias Bjørling wrote:
> >>>>>>>On 16/06/2020 14.00, Javier González wrote:
> >>>>>>>>On 16.06.2020 13:18, Matias Bjørling wrote:
> >>>>>>>>>On 16/06/2020 12.41, Javier González wrote:
> >>>>>>>>>>On 16.06.2020 08:34, Keith Busch wrote:
> >>>>>>>>>>>Add support for NVM Express Zoned Namespaces (ZNS)
> Command
> >>>>>>>>>>>Set defined in NVM Express TP4053. Zoned namespaces are
> >>>>>>>>>>>discovered based on their Command Set Identifier reported in
> >>>>>>>>>>>the namespaces Namespace Identification Descriptor list. A
> >>>>>>>>>>>successfully discovered Zoned Namespace will be registered
> >>>>>>>>>>>with the block layer as a host managed zoned block device
> >>>>>>>>>>>with Zone Append command support. A namespace that does not
> >>>>>>>>>>>support append is not supported by the driver.
> >>>>>>>>>>
> >>>>>>>>>>Why are we enforcing the append command? Append is optional
> on
> >>>>>>>>>>the current ZNS specification, so we should not make this
> >>>>>>>>>>mandatory in the implementation. See specifics below.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>There is already general support in the kernel for the zone
> >>>>>>>>>append command. Feel free to submit patches to emulate the
> >>>>>>>>>support. It is outside the scope of this patchset.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>It is fine that the kernel supports append, but the ZNS
> >>>>>>>>specification does not impose the implementation for append, so
> >>>>>>>>the driver should not do that either.
> >>>>>>>>
> >>>>>>>>ZNS SSDs that choose to leave append as a non-implemented
> >>>>>>>>optional command should not rely on emulated SW support,
> >>>>>>>>specially when traditional writes work very fine for a large
> >>>>>>>>part of current ZNS use cases.
> >>>>>>>>
> >>>>>>>>Please, remove this virtual constraint.
> >>>>>>>
> >>>>>>>The Zone Append command is mandatory for zoned block devices.
> >>>>>>>Please see https://lwn.net/Articles/818709/ for the background.
> >>>>>>
> >>>>>>I do not see anywhere in the block layer that append is mandatory
> >>>>>>for zoned devices. Append is emulated on ZBC, but beyond that
> >>>>>>there is no mandatory bits. Please explain.
> >>>>>
> >>>>>This is to allow a single write IO path for all types of zoned
> >>>>>block device for higher layers, e.g file systems. The on-going
> >>>>>re-work of btrfs zone support for instance now relies 100% on zone
> >>>>>append being supported. That significantly simplifies the file
> >>>>>system support and more importantly remove the need for locking
> >>>>>around block allocation and BIO issuing, allowing to preserve a
> >>>>>fully asynchronous write path that can include workqueues for
> >>>>>efficient CPU usage of things like encryption and compression.
> >>>>>Without zone append, file system would either (1) have to reject
> >>>>>these drives that do not support zone append, or (2) implement 2
> >>>>>different write IO path (slower regular write and zone append).
> >>>>>None of these options are ideal, to say the least.
> >>>>>
> >>>>>So the approach is: mandate zone append support for ZNS devices. To
> >>>>>allow other ZNS drives, an emulation similar to SCSI can be
> >>>>>implemented, with that emulation ideally combined to work for both
> >>>>>types of drives if possible.
> >>>>
> >>>>Enforcing QD=1 becomes a problem on devices with large zones. In a
> >>>>ZNS device that has smaller zones this should not be a problem.
> >>>
> >>>Let's be precise: this is not running the drive at QD=1, it is "at
> >>>most one write *request* per zone". If the FS is simultaneously using
> >>>multiple block groups mapped to different zones, you will get a total
> >>>write QD > 1, and as many reads as you want.
> >>>
> >>>>Would you agree that it is possible to have a write path that relies
> >>>>on QD=1, where the FS / application has the responsibility for
> >>>>enforcing this? Down the road this QD can be increased if the device
> >>>>is able to buffer the writes.
> >>>
> >>>Doing QD=1 per zone for writes at the FS layer, that is, at the BIO
> >>>layer does not work. This is because BIOs can be as large as the FS
> >>>wants them to be. Such large BIO will be split into multiple requests
> >>>in the block layer, resulting in more than one write per zone. That
> >>>is why the zone write locking is at the scheduler level, between BIO
> >>>split and request dispatch. That avoids the multiple requests
> >>>fragments of a large BIO to be reordered and fail. That is mandatory
> >>>as the block layer itself can occasionally reorder requests and lower
> >>>levels such as AHCI HW is also notoriously good at reversing
> >>>sequential requests. For NVMe with multi-queue, the IO issuing
> >>>process getting rescheduled on a different CPU can result in
> >>>sequential IOs being in different queues, with the likely result of
> >>>an out-of-order execution. All cases are avoided with zone write
> >>>locking and at most one write request dispatch per zone as
> >>>recommended by the ZNS specifications (ZBC and ZAC standards for SMR
> >>>HDDs are silent on this).
> >>>
> >>
> >>I understand. I agree that the current FSs supporting ZNS follow this
> >>approach and it makes sense that there is a common interface that
> >>simplifies the FS implementation. See the comment below on the part I
> >>believe we see things differently.
> >>
> >>
> >>>>I would be OK with some FS implementations to rely on append and
> >>>>impose the constraint that append has to be supported (and it would
> >>>>be our job to change that), but I would like to avoid the driver
> >>>>rejecting initializing the device because current FS implementations
> >>>>have implemented this logic.
> >>>
> >>>What is the difference between the driver rejecting drives and the FS
> >>>rejecting the same drives ? That has the same end result to me: an
> >>>entire class of devices cannot be used as desired by the user.
> >>>Implementing zone append emulation avoids the rejection entirely
> >>>while still allowing the FS to have a single write IO path, thus
> >>>simplifying the code.
> >>
> >>The difference is that users that use a raw ZNS device submitting I/O
> >>through the kernel would still be able to use these devices. The
> >>result would be that the ZNS SSD is recognized and initialized, but
> >>the FS format fails.
> >>
> >>>
> >>>>We can agree that a number of initial customers will use these
> >>>>devices raw, using the in-kernel I/O path, but without a FS on top.
> >>>>
> >>>>Thoughts?
> >>>>
> >>>>>and note that
> >>>>>this emulation would require the drive to be operated with
> >>>>>mq-deadline to enable zone write locking for preserving write
> >>>>>command order. While on a HDD the performance penalty is minimal,
> >>>>>it will likely be significant on a SSD.
> >>>>
> >>>>Exactly my concern. I do not want ZNS SSDs to be impacted by this
> >>>>type of design decision at the driver level.
> >>>
> >>>But your proposed FS level approach would end up doing the exact same
> >>>thing with the same limitation and so the same potential performance
> >>>impact.
> >>>The block
> >>>layer generic approach has the advantage that we do not bother the
> >>>higher levels with the implementation of in-order request dispatch
> >>>guarantees.
> >>>File systems
> >>>are complex enough. The less complexity is required for zone support,
> >>>the better.
> >>
> >>This depends very much on how the FS / application is managing
> >>stripping. At the moment our main use case is enabling user-space
> >>applications submitting I/Os to raw ZNS devices through the kernel.
> >>
> >>Can we enable this use case to start with?
> >
> >It is free for everyone to load kernel modules into the kernel. Those
> >modules may not have the appropriate checks or may rely on the zone
> >append functionality. Having per use-case limit is a no-go and at best
> >a game of whack-a-mole.
> 
> Let's focus on mainline support. We are leaving append as not enabled based
> on customer requests for some ZNS products and would like this devices to be
> supported. This is not at all a corner use-case but a very general one.
> 
> >
> >You already agreed to create a set of patches to add the appropriate
> >support for emulating zone append. As these would fix your specific
> >issue, please go ahead and submit those.
> 
> I agreed to solve the use case that some of our customers are enabling and this
> is what I am doing.
> 
> Again, to start with I would like to have a path where ZNS namespaces are
> identified independently of append support. Then specific users can require
> append if they please to do so. We will of course take care of sending patches
> for this.

As was previously said, there are users in the kernel that depends on zone append. As a result, it is not an option not to have this. Please go ahead and send the patches and you'll have the behavior you are seeking. 

Best, Matias
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme