[PATCH 1/2] block: return errors from blk_execute_rq()

[PATCH 1/2] block: return errors from blk_execute_rq()

[Keith Busch]

The synchronous blk_execute_rq() had not provided a way for its callers
to know if its request was successful or not. Return the errno from the
completion status.

Reported-by: Yuanyuan Zhong <yzhong@purestorage.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
 block/blk-exec.c       | 6 ++++--
 include/linux/blkdev.h | 2 +-
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/block/blk-exec.c b/block/blk-exec.c
index beae70a0e5e5..3877a2677dd4 100644
--- a/block/blk-exec.c
+++ b/block/blk-exec.c
@@ -21,7 +21,7 @@ static void blk_end_sync_rq(struct request *rq, blk_status_t error)
 {
 	struct completion *waiting = rq->end_io_data;
 
-	rq->end_io_data = NULL;
+	rq->end_io_data = ERR_PTR(blk_status_to_errno(error));
 
 	/*
 	 * complete last, if this is a stack request the process (and thus
@@ -72,8 +72,9 @@ EXPORT_SYMBOL_GPL(blk_execute_rq_nowait);
  * Description:
  *    Insert a fully prepared request at the back of the I/O scheduler queue
  *    for execution and wait for completion.
+ * Return: The errno value of the blk_status_t provided to blk_mq_end_request().
  */
-void blk_execute_rq(struct gendisk *bd_disk, struct request *rq, int at_head)
+int blk_execute_rq(struct gendisk *bd_disk, struct request *rq, int at_head)
 {
 	DECLARE_COMPLETION_ONSTACK(wait);
 	unsigned long hang_check;
@@ -87,5 +88,6 @@ void blk_execute_rq(struct gendisk *bd_disk, struct request *rq, int at_head)
 		while (!wait_for_completion_io_timeout(&wait, hang_check * (HZ/2)));
 	else
 		wait_for_completion_io(&wait);
+	return PTR_ERR_OR_ZERO(rq->end_io_data);
 }
 EXPORT_SYMBOL(blk_execute_rq);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index b91ba6207365..15e4ffac33af 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -938,7 +938,7 @@ extern int blk_rq_map_kern(struct request_queue *, struct request *, void *, uns
 extern int blk_rq_map_user_iov(struct request_queue *, struct request *,
 			       struct rq_map_data *, const struct iov_iter *,
 			       gfp_t);
-extern void blk_execute_rq(struct gendisk *, struct request *, int);
+extern int blk_execute_rq(struct gendisk *, struct request *, int);
 extern void blk_execute_rq_nowait(struct gendisk *,
 				  struct request *, int, rq_end_io_fn *);
 
-- 
2.25.4

[PATCH 2/2] nvme: use return value from blk_execute_rq()

[Keith Busch]

We don't have an nvme status to report if the driver's .queue_rq()
returns an error without dispatching the requested nvme command. Use the
return value from blk_execute_rq() so the caller may know their command
was not successful.

Reported-by: Yuanyuan Zhong <yzhong@purestorage.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
 drivers/nvme/host/core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 85538c38aae9..b57157106cac 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1000,12 +1000,12 @@ int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 	if (poll)
 		nvme_execute_rq_polled(req->q, NULL, req, at_head);
 	else
-		blk_execute_rq(NULL, req, at_head);
+		ret = blk_execute_rq(NULL, req, at_head);
 	if (result)
 		*result = nvme_req(req)->result;
 	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
 		ret = -EINTR;
-	else
+	else if (nvme_req(req)->status)
 		ret = nvme_req(req)->status;
  out:
 	blk_mq_free_request(req);
-- 
2.25.4

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Yuanyuan Zhong]

>         if (poll)
>                 nvme_execute_rq_polled(req->q, NULL, req, at_head);
You may need to audit other completion handlers for blk_execute_rq_nowait().
How to get error ret from polled rq?
>         else
> -               blk_execute_rq(NULL, req, at_head);
> +               ret = blk_execute_rq(NULL, req, at_head);
>         if (result)
>                 *result = nvme_req(req)->result;
>         if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
>                 ret = -EINTR;
> -       else
> +       else if (nvme_req(req)->status)
>                 ret = nvme_req(req)->status;


-- 
Regards,
Yuanyuan Zhong

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Keith Busch]

On Fri, Apr 16, 2021 at 10:12:11AM -0700, Yuanyuan Zhong wrote:
> >         if (poll)
> >                 nvme_execute_rq_polled(req->q, NULL, req, at_head);
> You may need to audit other completion handlers for blk_execute_rq_nowait().

Why? Those callers already provide their own callback that directly get
the error.

> How to get error ret from polled rq?

Please see nvme_end_sync_rq() for that driver's polled handler callback.
It already has the error.

Re: [PATCH 1/2] block: return errors from blk_execute_rq()

[Chaitanya Kulkarni]

On 4/16/21 09:54, Keith Busch wrote:
> +extern int blk_execute_rq(struct gendisk *, struct request *, int);

I've not checked if it will work or not, but we have been dropping
the extern keywords for the new declarations and I'm not sure if that
applies here or not, so looks good to me.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Chaitanya Kulkarni]

On 4/16/21 09:54, Keith Busch wrote:
> We don't have an nvme status to report if the driver's .queue_rq()
> returns an error without dispatching the requested nvme command. Use the
> return value from blk_execute_rq() so the caller may know their command
> was not successful.
>
> Reported-by: Yuanyuan Zhong <yzhong@purestorage.com>
> Signed-off-by: Keith Busch <kbusch@kernel.org>


Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Christoph Hellwig]

On Sat, Apr 17, 2021 at 02:17:35AM +0900, Keith Busch wrote:
> On Fri, Apr 16, 2021 at 10:12:11AM -0700, Yuanyuan Zhong wrote:
> > >         if (poll)
> > >                 nvme_execute_rq_polled(req->q, NULL, req, at_head);
> > You may need to audit other completion handlers for blk_execute_rq_nowait().
> 
> Why? Those callers already provide their own callback that directly get
> the error.
> 
> > How to get error ret from polled rq?
> 
> Please see nvme_end_sync_rq() for that driver's polled handler callback.
> It already has the error.

But it never looks at it..

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Keith Busch]

On Mon, Apr 19, 2021 at 09:16:05AM +0200, Christoph Hellwig wrote:
> On Sat, Apr 17, 2021 at 02:17:35AM +0900, Keith Busch wrote:
> > On Fri, Apr 16, 2021 at 10:12:11AM -0700, Yuanyuan Zhong wrote:
> > > >         if (poll)
> > > >                 nvme_execute_rq_polled(req->q, NULL, req, at_head);
> > > You may need to audit other completion handlers for blk_execute_rq_nowait().
> > 
> > Why? Those callers already provide their own callback that directly get
> > the error.
> > 
> > > How to get error ret from polled rq?
> > 
> > Please see nvme_end_sync_rq() for that driver's polled handler callback.
> > It already has the error.
> 
> But it never looks at it..

The question was how to get ret. I didn't mean to imply the example was
actually using it. :)

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Yuanyuan Zhong]

On Mon, Apr 19, 2021 at 8:14 AM Keith Busch <kbusch@kernel.org> wrote:
>
> On Mon, Apr 19, 2021 at 09:16:05AM +0200, Christoph Hellwig wrote:
> > On Sat, Apr 17, 2021 at 02:17:35AM +0900, Keith Busch wrote:
> > > On Fri, Apr 16, 2021 at 10:12:11AM -0700, Yuanyuan Zhong wrote:
> > > > >         if (poll)
> > > > >                 nvme_execute_rq_polled(req->q, NULL, req, at_head);
> > > > You may need to audit other completion handlers for blk_execute_rq_nowait().
> > >
> > > Why? Those callers already provide their own callback that directly get
> > > the error.

See below clarification. I was wondering whether the way you were going to
propose for nvme_end_sync_rq() would establish new convention for other
blk_execute_rq_nowait() completion handlers implementation.

> > >
> > > > How to get error ret from polled rq?
> > >
> > > Please see nvme_end_sync_rq() for that driver's polled handler callback.
> > > It already has the error.
> >
> > But it never looks at it..
>
> The question was how to get ret. I didn't mean to imply the example was
> actually using it. :)

The question was how to let nvme_end_sync_rq() propagate the blk_status_t
error to the ret for __nvme_submit_sync_cmd(). That's part of the problem
here: __nvme_submit_sync_cmd() may return success for a command that
failed submission.

-- 
Regards,
Yuanyuan Zhong

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Keith Busch]

On Mon, Apr 19, 2021 at 10:27:42AM -0700, Yuanyuan Zhong wrote:
> On Mon, Apr 19, 2021 at 8:14 AM Keith Busch <kbusch@kernel.org> wrote:
> >
> > On Mon, Apr 19, 2021 at 09:16:05AM +0200, Christoph Hellwig wrote:
> > > On Sat, Apr 17, 2021 at 02:17:35AM +0900, Keith Busch wrote:
> > > > On Fri, Apr 16, 2021 at 10:12:11AM -0700, Yuanyuan Zhong wrote:
> > > > > >         if (poll)
> > > > > >                 nvme_execute_rq_polled(req->q, NULL, req, at_head);
> > > > > You may need to audit other completion handlers for blk_execute_rq_nowait().
> > > >
> > > > Why? Those callers already provide their own callback that directly get
> > > > the error.
> 
> See below clarification. I was wondering whether the way you were going to
> propose for nvme_end_sync_rq() would establish new convention for other
> blk_execute_rq_nowait() completion handlers implementation.

I'm not sure it can be turned into a common pattern. The callbacks are
implementation specific.
 
> > > >
> > > > > How to get error ret from polled rq?
> > > >
> > > > Please see nvme_end_sync_rq() for that driver's polled handler callback.
> > > > It already has the error.
> > >
> > > But it never looks at it..
> >
> > The question was how to get ret. I didn't mean to imply the example was
> > actually using it. :)
> 
> The question was how to let nvme_end_sync_rq() propagate the blk_status_t
> error to the ret for __nvme_submit_sync_cmd(). That's part of the problem
> here: __nvme_submit_sync_cmd() may return success for a command that
> failed submission.

---
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index b57157106cac..a0fb9ad132af 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -949,6 +949,9 @@ static void nvme_end_sync_rq(struct request *rq, blk_status_t error)
 	struct completion *waiting = rq->end_io_data;
 
 	rq->end_io_data = NULL;
+	if (error && !nvme_req(rq)->status)
+		nvme_req(rq)->status = blk_status_to_errno(error);
 	complete(waiting);
 }
 
--

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Yuanyuan Zhong]

On Mon, Apr 19, 2021 at 10:48 AM Keith Busch <kbusch@kernel.org> wrote:
>

> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index b57157106cac..a0fb9ad132af 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -949,6 +949,9 @@ static void nvme_end_sync_rq(struct request *rq, blk_status_t error)
>         struct completion *waiting = rq->end_io_data;
>
>         rq->end_io_data = NULL;
> +       if (error && !nvme_req(rq)->status)
Is setting nvme_req(rq)->status for each error return in ->queue_rq()
going to gradually roll out, and eventually skipping the fallback here?

> +               nvme_req(rq)->status = blk_status_to_errno(error);
Casting int negative errno to u16 will get 0xfff., meaning NVME_SC_DNR is set.
Is that always right?

--
Regards,
Yuanyuan Zhong

[PATCH 2/2] nvme: use return value from blk_execute_rq()

[Keith Busch]

We don't have an nvme status to report if the driver's .queue_rq()
returns an error without dispatching the requested nvme command. Use the
return value from blk_execute_rq() so the caller may know their command
was not successful.

Reported-by: Yuanyuan Zhong <yzhong@purestorage.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
 drivers/nvme/host/core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 85538c38aae9..b57157106cac 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1000,12 +1000,12 @@ int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 	if (poll)
 		nvme_execute_rq_polled(req->q, NULL, req, at_head);
 	else
-		blk_execute_rq(NULL, req, at_head);
+		ret = blk_execute_rq(NULL, req, at_head);
 	if (result)
 		*result = nvme_req(req)->result;
 	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
 		ret = -EINTR;
-	else
+	else if (nvme_req(req)->status)
 		ret = nvme_req(req)->status;
  out:
 	blk_mq_free_request(req);
-- 
2.25.4

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Keith Busch]

On Fri, Apr 16, 2021 at 07:04:34PM -0700, Casey Chen wrote:
> > On Fri, Apr 16, 2021 at 10:12:11AM -0700, Yuanyuan Zhong wrote:
> > > >         if (poll)
> > > >                 nvme_execute_rq_polled(req->q, NULL, req, at_head);
> > > You may need to audit other completion handlers for blk_execute_rq_nowait().
> >
> > Why? Those callers already provide their own callback that directly get
> > the error.
> 
> We should make sure all callbacks provided to blk_execute_rq_nowait()
> carry error back. i.e. by reusing rq->end_io_data.
>
> > > How to get error ret from polled rq?
> >
> > Please see nvme_end_sync_rq() for that driver's polled handler callback.
> > It already has the error.
> 
> nvme_end_sync_rq() currently doesn't store error in rq->end_io_data as
> you proposed in patch 1.

The question was how the error gets back to the caller, and they already
have it.

Patch 1 is specific to the sync execution. All the async users' handling
of the provided error are implementation specific. If they're not using
it correctly, then they can be fixed too.

Re: [PATCH 2/2] nvme: use return value from blk_execute_rq()

[Casey Chen]

> On Fri, Apr 16, 2021 at 10:12:11AM -0700, Yuanyuan Zhong wrote:
> > >         if (poll)
> > >                 nvme_execute_rq_polled(req->q, NULL, req, at_head);
> > You may need to audit other completion handlers for blk_execute_rq_nowait().
>
> Why? Those callers already provide their own callback that directly get
> the error.

We should make sure all callbacks provided to blk_execute_rq_nowait()
carry error back. i.e. by reusing rq->end_io_data.

>
> > How to get error ret from polled rq?
>
> Please see nvme_end_sync_rq() for that driver's polled handler callback.
> It already has the error.

nvme_end_sync_rq() currently doesn't store error in rq->end_io_data as
you proposed in patch 1.

- Casey