Error handling in fio_libaio_event()

Error handling in fio_libaio_event()

[Scott Forman (scforman)]

Hi Andrey,

In looking further at fio_libaio_event(), I am wondering if it is possible to get the errno value if one of the events have an error (i.e. if ev->res != io_u->xfer_buflen).

I see that io_u->error is set to –ev->res if ev->res is greater than io_u->xfer_buflen.  But since my io_u->xfer_buflen is a large value, I am guessing that this is not the errno value that caused the error.  Is that true?
	
Also, if ev->res is less than io_u->xfer_buflen, then io_u->resid is set to (io_u->xfer_buflen - ev->res).  Does that mean that there will be another attempt to write out the io_u->resid bytes?

Again, thanks to you and the others on the fio mailing list for any guidance you can give.

Sincerely,
Scott


From: Andrey Kuzmin <andrey.v.kuzmin@gmail.com>
Date: Tuesday, March 28, 2017 at 11:22 AM
To: "Scott Forman (scforman)" <scforman@cisco.com>, "fio@vger.kernel.org" <fio@vger.kernel.org>
Subject: Re: How does FIO detect write errors when ioengine is libaio


On Tue, Mar 28, 2017, 21:13 Scott Forman (scforman) <scforman@cisco.com> wrote:
Hi,

I am trying to determine how fio detects write errors when the ioengine is set to libaio.

From a code example on stack overflow, linux-kernel-aio-functionality, it appears that they can be detected by examining the events returned from io_getevents().

But I do not see any code in libaio.c that examines the events returned from io_getevents().

static struct io_u *fio_libaio_event(struct thread_data *td, int event)
{
struct libaio_data *ld = td->io_ops_data;
struct io_event *ev;
struct io_u *io_u;

ev = ld->aio_events + event;
io_u = container_of(ev->obj, struct io_u, iocb);

if (ev->res != io_u->xfer_buflen) {
if (ev->res > io_u->xfer_buflen)
io_u->error = -ev->res;
else
io_u->resid = io_u->xfer_buflen - ev->res;
} else
io_u->error = 0;
Regards,
Andrey

Can you verify that fio does check for write errors for engine libaio, and if so, where that error checking is done?

Thanks in advance!

Sincerely,

Scott Forman


-- 
Regards,
Andrey

Re: Error handling in fio_libaio_event()

[Sitsofe Wheeler]

Hi Scott,

You might find https://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt useful...

On 4 April 2017 at 01:48, Scott Forman (scforman) <scforman@cisco.com> wrote:
>
> In looking further at fio_libaio_event(), I am wondering if it is possible to get the errno value if one of the events have an error (i.e. if ev->res != io_u->xfer_buflen).

Note this handles one type of error - the case where the I/O was
accepted at submission time but for some reason when it completed it
was short or too long(?). As alluded to, errors can also be returned
at submission time. It's also sensible to check that io_getevents
hasn't errored too.

> I see that io_u->error is set to –ev->res if ev->res is greater than io_u->xfer_buflen.  But since my io_u->xfer_buflen is a large value, I am guessing that this is not the errno value that caused the error.  Is that true?

This isn't a true errno but rather is a synthesized fio error - I
guess this is fio checking that it hasn't asked to store more than it
allocated space for and trying to indicate that it's an error if this
were to happen.

> Also, if ev->res is less than io_u->xfer_buflen, then io_u->resid is set to (io_u->xfer_buflen - ev->res).  Does that mean that there will be another attempt to write out the io_u->resid bytes?

Not necessarily. For example what happens if you perform a write that
goes past the end of the file - would you want to retry that even
though it returned back as being short? Is the fact it came back short
a definite error?

-- 
Sitsofe | http://sucs.org/~sits/

Re: Error handling in fio_libaio_event()

[Scott Forman (scforman)]

Hi Sitsofe, Andrey and fio mailing list,

Thanks again for your past help!

I am running into a new problem when I check for write errors after calling io_getevents() that I am hoping you can help me out with.

Comparing it to your function fio_libaio_event() in engines/libaio.c, I am finding that  (ev->res != io_u->xfer_buflen) and that (ev->res > io_u->xfer_buflen).   The value of ev->res, which I believe is the result of the write, is equal to the value of the count parameter passed to io_prep_pwrite(), so it seems like the write succeeded.  

So I was surprised to find the io_u->xfer_buflen value (or in my case, the struct iocb.u.c.nbytes value) less than what it was set to in io_prep_pwrite().  Do you know what might be causing this?

Thanks,
Scott



On 4/3/17, 11:17 PM, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:

    Hi Scott,
    
    You might find https://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt useful...
    
    On 4 April 2017 at 01:48, Scott Forman (scforman) <scforman@cisco.com> wrote:
    >
    > In looking further at fio_libaio_event(), I am wondering if it is possible to get the errno value if one of the events have an error (i.e. if ev->res != io_u->xfer_buflen).
    
    Note this handles one type of error - the case where the I/O was
    accepted at submission time but for some reason when it completed it
    was short or too long(?). As alluded to, errors can also be returned
    at submission time. It's also sensible to check that io_getevents
    hasn't errored too.
    
    > I see that io_u->error is set to –ev->res if ev->res is greater than io_u->xfer_buflen.  But since my io_u->xfer_buflen is a large value, I am guessing that this is not the errno value that caused the error.  Is that true?
    
    This isn't a true errno but rather is a synthesized fio error - I
    guess this is fio checking that it hasn't asked to store more than it
    allocated space for and trying to indicate that it's an error if this
    were to happen.
    
    > Also, if ev->res is less than io_u->xfer_buflen, then io_u->resid is set to (io_u->xfer_buflen - ev->res).  Does that mean that there will be another attempt to write out the io_u->resid bytes?
    
    Not necessarily. For example what happens if you perform a write that
    goes past the end of the file - would you want to retry that even
    though it returned back as being short? Is the fact it came back short
    a definite error?
    
    -- 
    Sitsofe | http://sucs.org/~sits/
    

Re: Error handling in fio_libaio_event()

[Sitsofe Wheeler]

Hi,

On 17 October 2017 at 00:44, Scott Forman (scforman) <scforman@cisco.com> wrote:
>
> I am running into a new problem when I check for write errors after calling io_getevents() that I am hoping you can help me out with.
>
> Comparing it to your function fio_libaio_event() in engines/libaio.c, I am finding that  (ev->res != io_u->xfer_buflen) and that (ev->res > io_u->xfer_buflen).   The value of ev->res, which I believe is the result of the write, is equal to the value of the count parameter passed to io_prep_pwrite(), so it seems like the write succeeded.

This sounds strange - a long write? I'd expect ev->res to be less than
or equal to the length of the buffer you passed in for that I/O but
you're saying that it's greater? Are you sure you've got the ev->res
you're looking at matches to the I/O you're checking against? Or was
that a typo?

> So I was surprised to find the io_u->xfer_buflen value (or in my case, the struct iocb.u.c.nbytes value) less than what it was set to in io_prep_pwrite().  Do you know what might be causing this?

If ev->res is LESS than io_u->xfer_buflen then you have a short I/O
which happens when less I/O than you wanted is serviced. I actually
gave an example of when this could happen originally:

> On 4/3/17, 11:17 PM, "Sitsofe Wheeler" <sitsofe@gmail.com> wrote:
>
>     On 4 April 2017 at 01:48, Scott Forman (scforman) <scforman@cisco.com> wrote:
>     >
>     > In looking further at fio_libaio_event(), I am wondering if it is possible to get the errno value if one of the events have an error (i.e. if ev->res != io_u->xfer_buflen).
>
[...]
>
>     > Also, if ev->res is less than io_u->xfer_buflen, then io_u->resid is set to (io_u->xfer_buflen - ev->res).  Does that mean that there will be another attempt to write out the io_u->resid bytes?
>
>     Not necessarily. For example what happens if you *perform a write that
>     goes past the end of the file* [emphasis added] - would you want to retry that even
>     though it returned back as being short? Is the fact it came back short
>     a definite error?

You may also find https://github.com/littledan/linux-aio handy if very
C++ oriented.

-- 
Sitsofe | http://sucs.org/~sits/