* Spurious EIO on AIO+DIO+RWF_NOWAIT
@ 2018-12-09 11:19 Avi Kivity
2018-12-10 12:48 ` Goldwyn Rodrigues
0 siblings, 1 reply; 5+ messages in thread
From: Avi Kivity @ 2018-12-09 11:19 UTC (permalink / raw)
To: linux-kernel, linux-aio, Goldwyn Rodrigues
I have an application that receives spurious EIO when running with
RWF_NOWAIT enabled. Removing RWF_NOWAIT causes those EIOs to disappear.
The application uses AIO+DIO, and errors were seen on both xfs and ext4.
I suspect the following code:
/*
* Process one completed BIO. No locks are held.
*/
static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio)
{
struct bio_vec *bvec;
unsigned i;
blk_status_t err = bio->bi_status;
if (err) {
if (err == BLK_STS_AGAIN && (bio->bi_opf & REQ_NOWAIT))
dio->io_error = -EAGAIN;
else
dio->io_error = -EIO;
}
Could it be that REQ_NOWAIT was dropped from bio->bi_opf? or that
bio->bi_status got changed along the way?
Reducing the test case may be a lot of work, but I can package a binary
using docker if a reproducer is needed. Seen on 4.18.19 and 4.19.something.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Spurious EIO on AIO+DIO+RWF_NOWAIT
2018-12-09 11:19 Spurious EIO on AIO+DIO+RWF_NOWAIT Avi Kivity
@ 2018-12-10 12:48 ` Goldwyn Rodrigues
2018-12-10 12:49 ` Avi Kivity
2018-12-12 12:05 ` Avi Kivity
0 siblings, 2 replies; 5+ messages in thread
From: Goldwyn Rodrigues @ 2018-12-10 12:48 UTC (permalink / raw)
To: Avi Kivity; +Cc: linux-kernel, linux-aio
On 13:19 09/12, Avi Kivity wrote:
> I have an application that receives spurious EIO when running with
> RWF_NOWAIT enabled. Removing RWF_NOWAIT causes those EIOs to disappear. The
> application uses AIO+DIO, and errors were seen on both xfs and ext4.
>
>
> I suspect the following code:
>
>
> /*
> * Process one completed BIO. No locks are held.
> */
> static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio)
> {
> struct bio_vec *bvec;
> unsigned i;
> blk_status_t err = bio->bi_status;
>
> if (err) {
> if (err == BLK_STS_AGAIN && (bio->bi_opf & REQ_NOWAIT))
> dio->io_error = -EAGAIN;
> else
> dio->io_error = -EIO;
> }
>
> Could it be that REQ_NOWAIT was dropped from bio->bi_opf? or that
> bio->bi_status got changed along the way?
>
I don't think REQ_NOWAIT is dropped. I am assuming bio->bi_status error
is set differently. Is the blk queue being stopped? Is it possible to
instrument the kernel in your testcase?
--
Goldwyn
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Spurious EIO on AIO+DIO+RWF_NOWAIT
2018-12-10 12:48 ` Goldwyn Rodrigues
@ 2018-12-10 12:49 ` Avi Kivity
2018-12-12 12:05 ` Avi Kivity
1 sibling, 0 replies; 5+ messages in thread
From: Avi Kivity @ 2018-12-10 12:49 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: linux-kernel, linux-aio
On 10/12/2018 14.48, Goldwyn Rodrigues wrote:
> On 13:19 09/12, Avi Kivity wrote:
>> I have an application that receives spurious EIO when running with
>> RWF_NOWAIT enabled. Removing RWF_NOWAIT causes those EIOs to disappear. The
>> application uses AIO+DIO, and errors were seen on both xfs and ext4.
>>
>>
>> I suspect the following code:
>>
>>
>> /*
>> * Process one completed BIO. No locks are held.
>> */
>> static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio)
>> {
>> struct bio_vec *bvec;
>> unsigned i;
>> blk_status_t err = bio->bi_status;
>>
>> if (err) {
>> if (err == BLK_STS_AGAIN && (bio->bi_opf & REQ_NOWAIT))
>> dio->io_error = -EAGAIN;
>> else
>> dio->io_error = -EIO;
>> }
>>
>> Could it be that REQ_NOWAIT was dropped from bio->bi_opf? or that
>> bio->bi_status got changed along the way?
>>
> I don't think REQ_NOWAIT is dropped. I am assuming bio->bi_status error
> is set differently. Is the blk queue being stopped?
Not that I know of.
> Is it possible to
> instrument the kernel in your testcase?
>
I'm happy to apply patches, or run systemtap (or similar) scripts.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Spurious EIO on AIO+DIO+RWF_NOWAIT
2018-12-10 12:48 ` Goldwyn Rodrigues
2018-12-10 12:49 ` Avi Kivity
@ 2018-12-12 12:05 ` Avi Kivity
2018-12-12 14:41 ` Goldwyn Rodrigues
1 sibling, 1 reply; 5+ messages in thread
From: Avi Kivity @ 2018-12-12 12:05 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: linux-kernel, linux-aio
On 12/10/18 2:48 PM, Goldwyn Rodrigues wrote:
> On 13:19 09/12, Avi Kivity wrote:
>> I have an application that receives spurious EIO when running with
>> RWF_NOWAIT enabled. Removing RWF_NOWAIT causes those EIOs to disappear. The
>> application uses AIO+DIO, and errors were seen on both xfs and ext4.
>>
>>
>> I suspect the following code:
>>
>>
>> /*
>> * Process one completed BIO. No locks are held.
>> */
>> static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio)
>> {
>> struct bio_vec *bvec;
>> unsigned i;
>> blk_status_t err = bio->bi_status;
>>
>> if (err) {
>> if (err == BLK_STS_AGAIN && (bio->bi_opf & REQ_NOWAIT))
>> dio->io_error = -EAGAIN;
>> else
>> dio->io_error = -EIO;
>> }
>>
>> Could it be that REQ_NOWAIT was dropped from bio->bi_opf? or that
>> bio->bi_status got changed along the way?
>>
> I don't think REQ_NOWAIT is dropped. I am assuming bio->bi_status error
> is set differently. Is the blk queue being stopped? Is it possible to
> instrument the kernel in your testcase?
>
I traced the function, and I see bio->bi_status == BLK_STS_NOTSUPP and
bio->bi_opf == REQ_OP_WRITE|REQ_SYNC|REQ_NOMERGE|REQ_FUA|REQ_NOWAIT.
Presumably the NOTSUPP is the result of NOWAIT not being supported down
the stack, but shouldn't it be detected earlier? And not converted to EIO?
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Spurious EIO on AIO+DIO+RWF_NOWAIT
2018-12-12 12:05 ` Avi Kivity
@ 2018-12-12 14:41 ` Goldwyn Rodrigues
0 siblings, 0 replies; 5+ messages in thread
From: Goldwyn Rodrigues @ 2018-12-12 14:41 UTC (permalink / raw)
To: Avi Kivity; +Cc: linux-kernel, linux-aio
On 14:05 12/12, Avi Kivity wrote:
> On 12/10/18 2:48 PM, Goldwyn Rodrigues wrote:
> > On 13:19 09/12, Avi Kivity wrote:
> > > I have an application that receives spurious EIO when running with
> > > RWF_NOWAIT enabled. Removing RWF_NOWAIT causes those EIOs to disappear. The
> > > application uses AIO+DIO, and errors were seen on both xfs and ext4.
> > >
> > >
> > > I suspect the following code:
> > >
> > >
> > > /*
> > > * Process one completed BIO. No locks are held.
> > > */
> > > static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio)
> > > {
> > > struct bio_vec *bvec;
> > > unsigned i;
> > > blk_status_t err = bio->bi_status;
> > >
> > > if (err) {
> > > if (err == BLK_STS_AGAIN && (bio->bi_opf & REQ_NOWAIT))
> > > dio->io_error = -EAGAIN;
> > > else
> > > dio->io_error = -EIO;
> > > }
> > >
> > > Could it be that REQ_NOWAIT was dropped from bio->bi_opf? or that
> > > bio->bi_status got changed along the way?
> > >
> > I don't think REQ_NOWAIT is dropped. I am assuming bio->bi_status error
> > is set differently. Is the blk queue being stopped? Is it possible to
> > instrument the kernel in your testcase?
> >
>
> I traced the function, and I see bio->bi_status == BLK_STS_NOTSUPP and
> bio->bi_opf == REQ_OP_WRITE|REQ_SYNC|REQ_NOMERGE|REQ_FUA|REQ_NOWAIT.
> Presumably the NOTSUPP is the result of NOWAIT not being supported down the
> stack, but shouldn't it be detected earlier? And not converted to EIO?
>
I don't think there is a way to detect it earlier. However, I think we should
return -EOPNOTSUPP if the lower layers do not support REQ_NOWAIT. I will write
a patch to modify this.
--
Goldwyn
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-12-12 14:41 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-09 11:19 Spurious EIO on AIO+DIO+RWF_NOWAIT Avi Kivity
2018-12-10 12:48 ` Goldwyn Rodrigues
2018-12-10 12:49 ` Avi Kivity
2018-12-12 12:05 ` Avi Kivity
2018-12-12 14:41 ` Goldwyn Rodrigues
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).