Re: [LSF/MM/BPF BoF] BoF for Zoned Storage

[Adam Manzanares <a.manzanares@samsung.com>]

On Thu, Mar 03, 2022 at 09:49:06AM +0000, Damien Le Moal wrote:
> On 2022/03/03 8:29, Javier González wrote:
> > On 03.03.2022 06:32, Javier González wrote:
> >>
> >>> On 3 Mar 2022, at 04.24, Luis Chamberlain <mcgrof@kernel.org> wrote:
> >>>
> >>> Thinking proactively about LSFMM, regarding just Zone storage..
> >>>
> >>> I'd like to propose a BoF for Zoned Storage. The point of it is
> >>> to address the existing point points we have and take advantage of
> >>> having folks in the room we can likely settle on things faster which
> >>> otherwise would take years.
> >>>
> >>> I'll throw at least one topic out:
> >>>
> >>>  * Raw access for zone append for microbenchmarks:
> >>>      - are we really happy with the status quo?
> >>>    - if not what outlets do we have?
> >>>
> >>> I think the nvme passthrogh stuff deserves it's own shared
> >>> discussion though and should not make it part of the BoF.
> >>>
> >>>  Luis
> >>
> >> Thanks for proposing this, Luis.
> >>
> >> I’d like to join this discussion too.
> >>
> >> Thanks,
> >> Javier
> > 
> > Let me expand a bit on this. There is one topic that I would like to
> > cover in this session:
> > 
> >    - PO2 zone sizes
> >        In the past weeks we have been talking to Damien and Matias around
> >        the constraint that we currently have for PO2 zone sizes. While
> >        this has not been an issue for SMR HDDs, the gap that ZNS
> >        introduces between zone capacity and zone size causes holes in the
> >        address space. This unmapped LBA space has been the topic of
> >        discussion with several ZNS adopters.
> > 
> >        One of the things to note here is that even if the zone size is a
> >        PO2, the zone capacity is typically not. This means that even when
> >        we can use shifts to move around zones, the actual data placement
> >        algorithms need to deal with arbitrary sizes. So at the end of the
> >        day applications that use a contiguous address space - like in a
> >        conventional block device -, will have to deal with this.
> 
> "the actual data placement algorithms need to deal with arbitrary sizes"
> 
> ???
> 
> No it does not. With zone cap < zone size, the amount of sectors that can be
> used within a zone may be smaller than the zone size, but:
> 1) Writes still must be issued at the WP location so choosing a zone for writing
> data has the same constraint regardless of the zone capacity: Do I have enough
> usable sectors left in the zone ?

Are you saying holes are irrelevant because an application has to know the 
status of a zone by querying the device for the zone status before using a zone
and at that point it should know a start LBA? I see your point here but we have
to assume things to arrive at this conclusion.

Let's think of another scenario where the drive is managed by a user space 
application that knows the status of zones and picks a zone because it knows 
it is free. To calculate the start offset in terms of LBAs the application has 
to use the difference in zone_size and zone_cap to calculate the write offset
in terms of LBAs. 

My argument is that the zone_size is a construct conceived to make a ZNS zone
a power of 2 that creates a hole in the LBA space. Applications don't want
to deal with the power of 2 constraint and neither do devices. It seems like
the existing zoned kernel infrastructure, which made sense for SMR, pushed 
this constraint onto devices and onto users. Arguments can be made for where 
complexity should lie, but I don't think this decision made things easier for
someone to use a ZNS SSD as a block device. 

> 2) Reading after the WP is not useful (if not outright stupid), regardless of
> where the last usable sector in the zone is (at zone start + zone size or at
> zone start + zone cap).

Of course but the with po2 you force useless LBA space even if you fill a zone.


> 
> And talking about "use a contiguous address space" is in my opinion nonsense in
> the context of zoned storage since by definition, everything has to be managed
> using zones as units. The only sensible range for a "contiguous address space"
> is "zone start + min(zone cap, zone size)".

Definitely disagree with this given previous arguments. This is a construct 
forced upon us because of zoned storage legacy.

> 
> >        Since chunk_sectors is no longer required to be a PO2, we have
> >        started the work in removing this constraint. We are working in 2
> >        phases:
> > 
> >          1. Add an emulation layer in NVMe driver to simulate PO2 devices
> > 	when the HW presents a zone_capacity = zone_size. This is a
> > 	product of one of Damien's early concerns about supporting
> > 	existing applications and FSs that work under the PO2
> > 	assumption. We will post these patches in the next few days.
> > 
> >          2. Remove the PO2 constraint from the block layer and add
> > 	support for arbitrary zone support in btrfs. This will allow the
> > 	raw block device to be present for arbitrary zone sizes (and
> > 	capacities) and btrfs will be able to use it natively.
> 
> Zone sizes cannot be arbitrary in btrfs since block groups must be a multiple of
> 64K. So constraints remain and should be enforced, at least by btrfs.

I don't think we should base a lot of decisions on the work that has gone into 
btrfs. I think it is very promising, but I don't think it is settled that it 
is the only way people will consume ZNS SSDs.

> 
> > 
> > 	For completeness, F2FS works natively in PO2 zone sizes, so we
> > 	will not do work here for now, as the changes will not bring any
> > 	benefit. For F2FS, the emulation layer will help use devices
> > 	that do not have PO2 zone sizes.
> > 
> >       We are working towards having at least a RFC of (2) before LSF/MM.
> >       Since this is a topic that involves several parties across the
> >       stack, I believe that a F2F conversation will help laying the path
> >       forward.
> > 
> > Thanks,
> > Javier
> > 
> 
> 
> -- 
> Damien Le Moal
> Western Digital Research