Re: [PATCH 2/2] block: Split and submit bios in LBA order

From: Bart Van Assche <bvanassche@acm.org>
To: Christoph Hellwig <hch@lst.de>
Cc: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>,
	linux-block@vger.kernel.org, Jaegeuk Kim <jaegeuk@kernel.org>,
	Jan Kara <jack@suse.cz>,
	Damien Le Moal <damien.lemoal@opensource.wdc.com>,
	Johannes Thumshirn <johannes.thumshirn@wdc.com>
Subject: Re: [PATCH 2/2] block: Split and submit bios in LBA order
Date: Tue, 21 Mar 2023 07:36:12 -0700	[thread overview]
Message-ID: <100dfc73-d8f3-f08f-e091-3c08707e95f5@acm.org> (raw)
In-Reply-To: <20230321055537.GA18035@lst.de>

On 3/20/23 22:55, Christoph Hellwig wrote:
> On Mon, Mar 20, 2023 at 04:32:57PM -0700, Bart Van Assche wrote:
>> The use case I'm looking at is Android devices with UFS storage. UFS is
>> based on SCSI and hence only REQ_OP_WRITE is supported natively. There is a
>> REQ_OP_ZONE_APPEND emulation in drivers/scsi/sd_zbc.c but it restricts the
>> queue depth to one.
> 
> The queue depth (per zone) is limited for regular writes to, for the
> same reason that the zone append emulations limits them.  You seem
> to be very aware of that too as you've tried various methods to lift
> that limit, none of which seems to ultimatively work.

Hi Christoph,

The UFSHCI specification is very clear about the requirement that UFS 
host controllers must process SCSI commands in order if host software 
sets one bit at a time in the UFSHCI 3.0 doorbell register: "For Task 
Management Requests and Transfer Requests, software may issue multiple 
commands at a time, and may issue new commands before previous commands 
have completed. When software sets the corresponding doorbell register, 
the Task Management Requests and Transfer Requests automatically get a 
time stamp with their issue time. The commands within a command list 
(Task Management List or Transfer Request List) shall be processed in
the order of their time stamps, starting from the oldest time stamp. In 
the case multiple commands from the same list have the same time stamp, 
they shall be processed in the order of their command list index,
starting from the lowest index."

Damien and Jens agree about introducing an additional hardware queue for 
preserving the order of zoned writes as one can see here: 
https://lore.kernel.org/linux-block/ed255a4a-a0da-a962-2da4-13321d0a75c5@kernel.dk/

In our tests pipelining zoned writes (REQ_OP_WRITE) works fine as long 
as the UFS error handler is not activated. After the UFS error handler 
has been scheduled and before the SCSI host state is changed into 
SHOST_RECOVERY, the UFS host controller driver responds with 
SCSI_MLQUEUE_HOST_BUSY. I'm still working on a solution for the 
reordering caused by this mechanism.

Thanks,

Bart.