[PATCH v2] block/io_uring: resubmit when result is -EAGAIN

[PATCH v2] block/io_uring: resubmit when result is -EAGAIN

[Fabian Ebner]

Linux SCSI can throw spurious -EAGAIN in some corner cases in its
completion path, which will end up being the result in the completed
io_uring request.

Resubmitting such requests should allow block jobs to complete, even
if such spurious errors are encountered.

Co-authored-by: Stefan Hajnoczi <stefanha@gmail.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---

Changes from v1:
    * Focus on what's relevant for the patch itself in the commit
      message.
    * Add Stefan's comment.
    * Add Stefano's R-b tag (I hope that's fine, since there was no
      change code-wise).

 block/io_uring.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/block/io_uring.c b/block/io_uring.c
index 00a3ee9fb8..dfa475cc87 100644
--- a/block/io_uring.c
+++ b/block/io_uring.c
@@ -165,7 +165,21 @@ static void luring_process_completions(LuringState *s)
         total_bytes = ret + luringcb->total_read;
 
         if (ret < 0) {
-            if (ret == -EINTR) {
+            /*
+             * Only writev/readv/fsync requests on regular files or host block
+             * devices are submitted. Therefore -EAGAIN is not expected but it's
+             * known to happen sometimes with Linux SCSI. Submit again and hope
+             * the request completes successfully.
+             *
+             * For more information, see:
+             * https://lore.kernel.org/io-uring/20210727165811.284510-3-axboe@kernel.dk/T/#u
+             *
+             * If the code is changed to submit other types of requests in the
+             * future, then this workaround may need to be extended to deal with
+             * genuine -EAGAIN results that should not be resubmitted
+             * immediately.
+             */
+            if (ret == -EINTR || ret == -EAGAIN) {
                 luring_resubmit(s, luringcb);
                 continue;
             }
-- 
2.30.2

Re: [PATCH v2] block/io_uring: resubmit when result is -EAGAIN

[Stefano Garzarella]

On Thu, Jul 29, 2021 at 11:10:29AM +0200, Fabian Ebner wrote:
>Linux SCSI can throw spurious -EAGAIN in some corner cases in its
>completion path, which will end up being the result in the completed
>io_uring request.
>
>Resubmitting such requests should allow block jobs to complete, even
>if such spurious errors are encountered.
>
>Co-authored-by: Stefan Hajnoczi <stefanha@gmail.com>
>Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
>Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
>---
>
>Changes from v1:
>    * Focus on what's relevant for the patch itself in the commit
>      message.
>    * Add Stefan's comment.
>    * Add Stefano's R-b tag (I hope that's fine, since there was no
>      change code-wise).

Yep, it's fine :-)

Thanks,
Stefano

Re: [PATCH v2] block/io_uring: resubmit when result is -EAGAIN

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

On Thu, Jul 29, 2021 at 11:10:29AM +0200, Fabian Ebner wrote:
> Linux SCSI can throw spurious -EAGAIN in some corner cases in its
> completion path, which will end up being the result in the completed
> io_uring request.
> 
> Resubmitting such requests should allow block jobs to complete, even
> if such spurious errors are encountered.
> 
> Co-authored-by: Stefan Hajnoczi <stefanha@gmail.com>
> Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
> Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
> ---
> 
> Changes from v1:
>     * Focus on what's relevant for the patch itself in the commit
>       message.
>     * Add Stefan's comment.
>     * Add Stefano's R-b tag (I hope that's fine, since there was no
>       change code-wise).
> 
>  block/io_uring.c | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)

Thanks, applied to my block tree:
https://gitlab.com/stefanha/qemu/commits/block

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2] block/io_uring: resubmit when result is -EAGAIN

[Kevin Wolf]

Am 29.07.2021 um 11:10 hat Fabian Ebner geschrieben:
> Linux SCSI can throw spurious -EAGAIN in some corner cases in its
> completion path, which will end up being the result in the completed
> io_uring request.
> 
> Resubmitting such requests should allow block jobs to complete, even
> if such spurious errors are encountered.
> 
> Co-authored-by: Stefan Hajnoczi <stefanha@gmail.com>
> Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
> Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
> ---
> 
> Changes from v1:
>     * Focus on what's relevant for the patch itself in the commit
>       message.
>     * Add Stefan's comment.
>     * Add Stefano's R-b tag (I hope that's fine, since there was no
>       change code-wise).
> 
>  block/io_uring.c | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/block/io_uring.c b/block/io_uring.c
> index 00a3ee9fb8..dfa475cc87 100644
> --- a/block/io_uring.c
> +++ b/block/io_uring.c
> @@ -165,7 +165,21 @@ static void luring_process_completions(LuringState *s)
>          total_bytes = ret + luringcb->total_read;
>  
>          if (ret < 0) {
> -            if (ret == -EINTR) {
> +            /*
> +             * Only writev/readv/fsync requests on regular files or host block
> +             * devices are submitted. Therefore -EAGAIN is not expected but it's
> +             * known to happen sometimes with Linux SCSI. Submit again and hope
> +             * the request completes successfully.
> +             *
> +             * For more information, see:
> +             * https://lore.kernel.org/io-uring/20210727165811.284510-3-axboe@kernel.dk/T/#u
> +             *
> +             * If the code is changed to submit other types of requests in the
> +             * future, then this workaround may need to be extended to deal with
> +             * genuine -EAGAIN results that should not be resubmitted
> +             * immediately.
> +             */
> +            if (ret == -EINTR || ret == -EAGAIN) {
>                  luring_resubmit(s, luringcb);
>                  continue;
>              }

Reviewed-by: Kevin Wolf <kwolf@redhat.com>

Question about the preexisting code, though: luring_resubmit() requires
that the caller calls ioq_submit() later so that the request doesn't
just sit in a queue without getting any attention, but actually gets
submitted to the kernel.

In the call chain ioq_submit() -> luring_process_completions() ->
luring_resubmit(), who takes care of that?

Kevin

Re: [PATCH v2] block/io_uring: resubmit when result is -EAGAIN

[Stefano Garzarella]

On Mon, Aug 02, 2021 at 02:40:36PM +0200, Kevin Wolf wrote:
>Am 29.07.2021 um 11:10 hat Fabian Ebner geschrieben:
>> Linux SCSI can throw spurious -EAGAIN in some corner cases in its
>> completion path, which will end up being the result in the completed
>> io_uring request.
>>
>> Resubmitting such requests should allow block jobs to complete, even
>> if such spurious errors are encountered.
>>
>> Co-authored-by: Stefan Hajnoczi <stefanha@gmail.com>
>> Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
>> Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
>> ---
>>
>> Changes from v1:
>>     * Focus on what's relevant for the patch itself in the commit
>>       message.
>>     * Add Stefan's comment.
>>     * Add Stefano's R-b tag (I hope that's fine, since there was no
>>       change code-wise).
>>
>>  block/io_uring.c | 16 +++++++++++++++-
>>  1 file changed, 15 insertions(+), 1 deletion(-)
>>
>> diff --git a/block/io_uring.c b/block/io_uring.c
>> index 00a3ee9fb8..dfa475cc87 100644
>> --- a/block/io_uring.c
>> +++ b/block/io_uring.c
>> @@ -165,7 +165,21 @@ static void luring_process_completions(LuringState *s)
>>          total_bytes = ret + luringcb->total_read;
>>
>>          if (ret < 0) {
>> -            if (ret == -EINTR) {
>> +            /*
>> +             * Only writev/readv/fsync requests on regular files or host block
>> +             * devices are submitted. Therefore -EAGAIN is not expected but it's
>> +             * known to happen sometimes with Linux SCSI. Submit again and hope
>> +             * the request completes successfully.
>> +             *
>> +             * For more information, see:
>> +             * https://lore.kernel.org/io-uring/20210727165811.284510-3-axboe@kernel.dk/T/#u
>> +             *
>> +             * If the code is changed to submit other types of requests in the
>> +             * future, then this workaround may need to be extended to deal with
>> +             * genuine -EAGAIN results that should not be resubmitted
>> +             * immediately.
>> +             */
>> +            if (ret == -EINTR || ret == -EAGAIN) {
>>                  luring_resubmit(s, luringcb);
>>                  continue;
>>              }
>
>Reviewed-by: Kevin Wolf <kwolf@redhat.com>
>
>Question about the preexisting code, though: luring_resubmit() requires
>that the caller calls ioq_submit() later so that the request doesn't
>just sit in a queue without getting any attention, but actually gets
>submitted to the kernel.
>
>In the call chain ioq_submit() -> luring_process_completions() ->
>luring_resubmit(), who takes care of that?

Mmm, good point.
There should be the same problem with ioq_submit() -> 
luring_process_completions() -> luring_resubmit_short_read() -> 
luring_resubmit().

Should we schedule a BH for example in luring_resubmit() to make sure 
that ioq_submit() is invoked after a resubmission?

Thanks,
Stefano

Re: [PATCH v2] block/io_uring: resubmit when result is -EAGAIN

[Kevin Wolf]

Am 04.08.2021 um 16:50 hat Stefano Garzarella geschrieben:
> On Mon, Aug 02, 2021 at 02:40:36PM +0200, Kevin Wolf wrote:
> > Am 29.07.2021 um 11:10 hat Fabian Ebner geschrieben:
> > > Linux SCSI can throw spurious -EAGAIN in some corner cases in its
> > > completion path, which will end up being the result in the completed
> > > io_uring request.
> > > 
> > > Resubmitting such requests should allow block jobs to complete, even
> > > if such spurious errors are encountered.
> > > 
> > > Co-authored-by: Stefan Hajnoczi <stefanha@gmail.com>
> > > Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
> > > Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
> > > ---
> > > 
> > > Changes from v1:
> > >     * Focus on what's relevant for the patch itself in the commit
> > >       message.
> > >     * Add Stefan's comment.
> > >     * Add Stefano's R-b tag (I hope that's fine, since there was no
> > >       change code-wise).
> > > 
> > >  block/io_uring.c | 16 +++++++++++++++-
> > >  1 file changed, 15 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/block/io_uring.c b/block/io_uring.c
> > > index 00a3ee9fb8..dfa475cc87 100644
> > > --- a/block/io_uring.c
> > > +++ b/block/io_uring.c
> > > @@ -165,7 +165,21 @@ static void luring_process_completions(LuringState *s)
> > >          total_bytes = ret + luringcb->total_read;
> > > 
> > >          if (ret < 0) {
> > > -            if (ret == -EINTR) {
> > > +            /*
> > > +             * Only writev/readv/fsync requests on regular files or host block
> > > +             * devices are submitted. Therefore -EAGAIN is not expected but it's
> > > +             * known to happen sometimes with Linux SCSI. Submit again and hope
> > > +             * the request completes successfully.
> > > +             *
> > > +             * For more information, see:
> > > +             * https://lore.kernel.org/io-uring/20210727165811.284510-3-axboe@kernel.dk/T/#u
> > > +             *
> > > +             * If the code is changed to submit other types of requests in the
> > > +             * future, then this workaround may need to be extended to deal with
> > > +             * genuine -EAGAIN results that should not be resubmitted
> > > +             * immediately.
> > > +             */
> > > +            if (ret == -EINTR || ret == -EAGAIN) {
> > >                  luring_resubmit(s, luringcb);
> > >                  continue;
> > >              }
> > 
> > Reviewed-by: Kevin Wolf <kwolf@redhat.com>
> > 
> > Question about the preexisting code, though: luring_resubmit() requires
> > that the caller calls ioq_submit() later so that the request doesn't
> > just sit in a queue without getting any attention, but actually gets
> > submitted to the kernel.
> > 
> > In the call chain ioq_submit() -> luring_process_completions() ->
> > luring_resubmit(), who takes care of that?
> 
> Mmm, good point.
> There should be the same problem with ioq_submit() ->
> luring_process_completions() -> luring_resubmit_short_read() ->
> luring_resubmit().
> 
> Should we schedule a BH for example in luring_resubmit() to make sure that
> ioq_submit() is invoked after a resubmission?

Or just loop in ioq_submit() after calling luring_process_completions()
if new requests were added to the queue?

Kevin

Re: [PATCH v2] block/io_uring: resubmit when result is -EAGAIN

[Stefano Garzarella]

On Wed, Aug 04, 2021 at 06:52:15PM +0200, Kevin Wolf wrote:
>Am 04.08.2021 um 16:50 hat Stefano Garzarella geschrieben:
>> On Mon, Aug 02, 2021 at 02:40:36PM +0200, Kevin Wolf wrote:
>> > Am 29.07.2021 um 11:10 hat Fabian Ebner geschrieben:
>> > > Linux SCSI can throw spurious -EAGAIN in some corner cases in its
>> > > completion path, which will end up being the result in the completed
>> > > io_uring request.
>> > >
>> > > Resubmitting such requests should allow block jobs to complete, even
>> > > if such spurious errors are encountered.
>> > >
>> > > Co-authored-by: Stefan Hajnoczi <stefanha@gmail.com>
>> > > Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
>> > > Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
>> > > ---
>> > >
>> > > Changes from v1:
>> > >     * Focus on what's relevant for the patch itself in the commit
>> > >       message.
>> > >     * Add Stefan's comment.
>> > >     * Add Stefano's R-b tag (I hope that's fine, since there was no
>> > >       change code-wise).
>> > >
>> > >  block/io_uring.c | 16 +++++++++++++++-
>> > >  1 file changed, 15 insertions(+), 1 deletion(-)
>> > >
>> > > diff --git a/block/io_uring.c b/block/io_uring.c
>> > > index 00a3ee9fb8..dfa475cc87 100644
>> > > --- a/block/io_uring.c
>> > > +++ b/block/io_uring.c
>> > > @@ -165,7 +165,21 @@ static void luring_process_completions(LuringState *s)
>> > >          total_bytes = ret + luringcb->total_read;
>> > >
>> > >          if (ret < 0) {
>> > > -            if (ret == -EINTR) {
>> > > +            /*
>> > > +             * Only writev/readv/fsync requests on regular files or host block
>> > > +             * devices are submitted. Therefore -EAGAIN is not expected but it's
>> > > +             * known to happen sometimes with Linux SCSI. Submit again and hope
>> > > +             * the request completes successfully.
>> > > +             *
>> > > +             * For more information, see:
>> > > +             * https://lore.kernel.org/io-uring/20210727165811.284510-3-axboe@kernel.dk/T/#u
>> > > +             *
>> > > +             * If the code is changed to submit other types of requests in the
>> > > +             * future, then this workaround may need to be extended to deal with
>> > > +             * genuine -EAGAIN results that should not be resubmitted
>> > > +             * immediately.
>> > > +             */
>> > > +            if (ret == -EINTR || ret == -EAGAIN) {
>> > >                  luring_resubmit(s, luringcb);
>> > >                  continue;
>> > >              }
>> >
>> > Reviewed-by: Kevin Wolf <kwolf@redhat.com>
>> >
>> > Question about the preexisting code, though: luring_resubmit() requires
>> > that the caller calls ioq_submit() later so that the request doesn't
>> > just sit in a queue without getting any attention, but actually gets
>> > submitted to the kernel.
>> >
>> > In the call chain ioq_submit() -> luring_process_completions() ->
>> > luring_resubmit(), who takes care of that?
>>
>> Mmm, good point.
>> There should be the same problem with ioq_submit() ->
>> luring_process_completions() -> luring_resubmit_short_read() ->
>> luring_resubmit().
>>
>> Should we schedule a BH for example in luring_resubmit() to make sure that
>> ioq_submit() is invoked after a resubmission?
>
>Or just loop in ioq_submit() after calling luring_process_completions()
>if new requests were added to the queue?
>

I was just concerned that we might cycle a bit if a request always 
returns -EAGAIN, while scheduling a task might give room for other 
devices to queue other requests.

But maybe this happens so occasionally that we might not worry about 
it...

Stefano