* Re: AIO support
@ 2021-09-20 10:22 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 10:22 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]
Hi Marcel,
No, because I don't believe it will work. I think the underlying problem
is the difference between streams and files. When a process creates a
stream (e.g. socket), the kernel creates a buffer and can start
receiving data from an external source. This buffer can be accessed
non-blocking from the process and the kernel can signal to the process
when data was received. If the process opens a file, the kernel doesn't
know what the process will do next (read or write, offset of the file,
transfer size). So the kernel doesn't read/write any bytes unless the
process sends a command to the kernel. And the POSIX read/write
functions on files are always blocking. That leaves us with using
multi-threading (POSIX AIO) or other linux specific syscalls (libaio,
preadv2).
Regards
Simon
On 9/20/21 9:20 AM, Marcel Holtmann wrote:
> Hi Simon,
>
>> I'm looking now into libaio. POSIX AIO is implemented in libc with user threads, while libaio uses linux native syscalls. Maybe I can also get rid of signal usage. IMHO non-blocking file access / async IO is quite important for embedded applications and it would be nice to have this feature in ell.
>
> have you looked into using l_idle for providing async read/write callbacks that way. I have the feeling that AIO is total overkill for embedded applications.
>
> Regards
>
> Marcel
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-25 17:37 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-25 17:37 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]
Hi Denis
On 9/24/21 4:21 PM, Denis Kenzior wrote:
>> I started testing the aio interface for ell and it seems to work,
>> except it's not asynchronous. Turns out libaio works only under
>> certain circumstances:
>> http://lse.sourceforge.net/io/aio.html
>
> yes, this was a widely reported criticism. I assume O_DIRECT is not
> applicable to your use case?
I tried to open the file with O_DIRECT, but still got no asynchronous
operations. I don't know what the problem was, NVMe, btrfs, ... also I
tried it on an usb stick with FAT32. Anyway, if it's only working under
certain conditions it's useless and dangerous for real time
applications. I can't expect the user of my aio interface to be an
expert in Linux filesystems and knowing all the restrictions of aio.
>
> Honestly, for ell, io_uring would be much more interesting to support.
> With maybe a fallback to aio or a threaded/blocking implementation for
> older kernels.
I might write an io_uring implementation in the future. But since
io_uring is not available on my target system, I have to stick with
POSIX AIO. A lot of Linux BSPs are only available for outdated kernel.
Regards,
Simon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-24 14:21 Denis Kenzior
0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-24 14:21 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 898 bytes --]
Hi Simon,
On 9/24/21 3:36 AM, Simon Maurer wrote:
> I started testing the aio interface for ell and it seems to work, except it's
> not asynchronous. Turns out libaio works only under certain circumstances:
> http://lse.sourceforge.net/io/aio.html
yes, this was a widely reported criticism. I assume O_DIRECT is not applicable
to your use case?
> So it's useless. There is a newer alternative called io_uring, but only for
> newer kernels (5.1) and not available to my target system. I will go back to the
Honestly, for ell, io_uring would be much more interesting to support. With
maybe a fallback to aio or a threaded/blocking implementation for older kernels.
> POSIX AIO and use SIGEV_THREAD (writing an eventfd in the notify function)
> instead of SIGEV_SIGNAL.
>
You still have to contend with librt use, which I think Marcel was against.
Regards,
-Denis
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-24 8:36 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-24 8:36 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 483 bytes --]
I started testing the aio interface for ell and it seems to work, except
it's not asynchronous. Turns out libaio works only under certain
circumstances:
http://lse.sourceforge.net/io/aio.html
So it's useless. There is a newer alternative called io_uring, but only
for newer kernels (5.1) and not available to my target system. I will go
back to the POSIX AIO and use SIGEV_THREAD (writing an eventfd in the
notify function) instead of SIGEV_SIGNAL.
Regards,
Simon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-22 11:46 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-22 11:46 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 164 bytes --]
I have a first draft of the complete aio API. l_aio_cancel doesn't work
as expected and also timeout adjustment has not yet been implemented.
Regards,
Simon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-21 21:43 Denis Kenzior
0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-21 21:43 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 302 bytes --]
Hi Simon,
On 9/20/21 3:05 PM, Simon Maurer wrote:
> I started a new version of aio.c:
> https://github.com/mausys/ell/blob/master/ell/aio.c
> but I have yet to implement l_io_destroy and l_io_cancel
I took a _very_ quick peek and what you have here so far seems reasonable.
Regards,
-Denis
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-20 20:05 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 20:05 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 144 bytes --]
I started a new version of aio.c:
https://github.com/mausys/ell/blob/master/ell/aio.c
but I have yet to implement l_io_destroy and l_io_cancel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-20 16:52 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 16:52 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 0 bytes --]
[-- Attachment #2: attachment.htm --]
[-- Type: text/html, Size: 1673 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-20 16:31 Denis Kenzior
0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-20 16:31 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 424 bytes --]
Hi Simon,
Please no top-posting.
On 9/20/21 11:00 AM, Simon Maurer wrote:
> Hi Denis,
>
> Yes I agree, but I still need the the syscall wrappers, because glibc doesn't
Can't we syscall() in ell, like we do with keyctl in key.c?
> provide them. So bundling it with ell and make libaio a private library of ell
> seams to be a good solution for me.
That would not be our first choice.
Regards,
-Denis
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-20 16:00 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 16:00 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 751 bytes --]
Hi Denis,
Yes I agree, but I still need the the syscall wrappers, because glibc
doesn't provide them. So bundling it with ell and make libaio a private
library of ell seams to be a good solution for me.
Regards,
Simon
On 9/20/21 3:29 PM, Denis Kenzior wrote:
> Hi Simon,
>
> On 9/20/21 10:23 AM, Simon Maurer wrote:
>> I wrote a little "proof of concept" for using ell with libaio:
>> https://github.com/mausys/ellaio/blob/main/aio.c
>> No more signals and threads. Also libaio is LGPL too, so it could be
>> bundled with ell.
>
> libaio seems to be a small convenience wrapper around linux system calls
> (~500 lines). Why not use io_submit, io_setup, etc directly for the
> parts you need?
>
> Regards,
> -Denis
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-20 15:29 Denis Kenzior
0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-20 15:29 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 448 bytes --]
Hi Simon,
On 9/20/21 10:23 AM, Simon Maurer wrote:
> I wrote a little "proof of concept" for using ell with libaio:
> https://github.com/mausys/ellaio/blob/main/aio.c
> No more signals and threads. Also libaio is LGPL too, so it could be bundled
> with ell.
libaio seems to be a small convenience wrapper around linux system calls (~500
lines). Why not use io_submit, io_setup, etc directly for the parts you need?
Regards,
-Denis
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-20 15:23 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 15:23 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 220 bytes --]
I wrote a little "proof of concept" for using ell with libaio:
https://github.com/mausys/ellaio/blob/main/aio.c
No more signals and threads. Also libaio is LGPL too, so it could be
bundled with ell.
Regards
Simon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-20 9:20 Marcel Holtmann
0 siblings, 0 replies; 20+ messages in thread
From: Marcel Holtmann @ 2021-09-20 9:20 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 498 bytes --]
Hi Simon,
> I'm looking now into libaio. POSIX AIO is implemented in libc with user threads, while libaio uses linux native syscalls. Maybe I can also get rid of signal usage. IMHO non-blocking file access / async IO is quite important for embedded applications and it would be nice to have this feature in ell.
have you looked into using l_idle for providing async read/write callbacks that way. I have the feeling that AIO is total overkill for embedded applications.
Regards
Marcel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-20 9:14 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 9:14 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 325 bytes --]
I'm looking now into libaio. POSIX AIO is implemented in libc with user
threads, while libaio uses linux native syscalls. Maybe I can also get
rid of signal usage. IMHO non-blocking file access / async IO is quite
important for embedded applications and it would be nice to have this
feature in ell.
Regards
Simon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-20 8:15 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 8:15 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 3178 bytes --]
Hi Marcel
On 9/19/21 7:59 PM, Marcel Holtmann wrote:
> Hi Simon,
>
>>>> Awesome library! Thank you very much. I'm using it for an embedded linux
>>> Glad you like it!
>>>> application with not so strict real time constraints. Problem is, I need non blocking file access, because the memory card on my system is quite slow. It turns out that this whole non-blocking access doesn't work for files, so now I'm
>>> Can you elaborate? I always thought
>>> fcntl(fd, F_SETFL, O_NONBLOCK);
>>> would enable non-blocking behavior on files just fine? Or is this an issue with a particular driver?
>>
>> I tested it and never got EAGAIN. I found this in the man page of open for the flag O_NONBLOCK or O_NDELAY:
>>
>> "Note that this flag has no effect for regular files and block devices; that is, I/O operations will (briefly) block when device activity is required, regardless of whether O_NONBLOCK is set. Since O_NONBLOCK semantics might eventually be implemented, applications should not depend upon blocking behavior when specifying this flag for regular files and block devices."
>>
>> An alternative could be preadv2 with the flag RWF_NOWAIT. But it is only available since linux 4.14 and we are using an older kernel. Also epoll doesn't wait for preadv2.
>
> I am a bit cautious about aio_write etc. since it requires you to link with -lrt according to the manpage. We actually wanted to keep it at just -lc since that helps in case of other C libraries besides glibc.
I don't like that either. If I am informed correctly librt is part of
glibc and in verison 2.34, librt is not needed anymore. MUSL supports
POSIX AIO, but neither android bionic nor ulibc implemented it. I
thought of making aio an optional part of ell, and handle the different
cases in the configure script.
>
> That you require signal handling inside the API is something that worries me additionally. Even with signalfd, doing any complex signal handling besides SIGTERM is kinda messy. I think that AIO support might be better done outside of ELL.
Signal handling in my aio implementation is optional and the signal just
triggers l_aio_poll, which checks all outstanding requests. The problem
I had with the signalhandling was when multiple requests were completed
at the same time (between epoll_wait calls), the signal handler was
executed only once. My first approach was to send request information
with the signal but that failed for multiple requests. My patch earlier
made this information available to the user, but independet of AIO
handling, I still think it might be a good idea to make siginfo
available to the user.
>
> What kind of file-reading or file-writing are you doing that you need AIO support anyway? You can actually poll() a file fd. You just have to do it carefully so it doesn’t become a busy loop. Have you tried just using l_idle to only read/write when we having nothing else going on in our mainloop?
>
> Regards
>
> Marcel
> _______________________________________________
> ell mailing list -- ell(a)lists.01.org
> To unsubscribe send an email to ell-leave(a)lists.01.org
>
Regards
Simon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-19 21:11 Denis Kenzior
0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-19 21:11 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]
Hi Simon,
>> Can you elaborate? I always thought
>> fcntl(fd, F_SETFL, O_NONBLOCK);
>>
>> would enable non-blocking behavior on files just fine? Or is this an issue
>> with a particular driver?
>>
>
> I tested it and never got EAGAIN. I found this in the man page of open for the
> flag O_NONBLOCK or O_NDELAY:
>
> "Note that this flag has no effect for regular files and block devices; that
> is, I/O operations will (briefly) block when device activity is required,
> regardless of whether O_NONBLOCK is set. Since O_NONBLOCK semantics might
> eventually be implemented, applications should not depend upon blocking
> behavior when specifying this flag for regular files and block devices."
>
Ah, indeed. I think I even knew that, sorry for the noise.
> An alternative could be preadv2 with the flag RWF_NOWAIT. But it is only
> available since linux 4.14 and we are using an older kernel. Also epoll doesn't
> wait for preadv2.
Right, epoll will just simply return immediately on regular files. Have you
tried using a timer or l_idle to drive the reading? Is there ioctl(fd,
FIONREAD, ...) or equivalent that could be used not to block?
Regards,
-Denis
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-19 19:59 Marcel Holtmann
0 siblings, 0 replies; 20+ messages in thread
From: Marcel Holtmann @ 2021-09-19 19:59 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 1993 bytes --]
Hi Simon,
>>> Awesome library! Thank you very much. I'm using it for an embedded linux
>> Glad you like it!
>>> application with not so strict real time constraints. Problem is, I need non blocking file access, because the memory card on my system is quite slow. It turns out that this whole non-blocking access doesn't work for files, so now I'm
>> Can you elaborate? I always thought
>> fcntl(fd, F_SETFL, O_NONBLOCK);
>> would enable non-blocking behavior on files just fine? Or is this an issue with a particular driver?
>
> I tested it and never got EAGAIN. I found this in the man page of open for the flag O_NONBLOCK or O_NDELAY:
>
> "Note that this flag has no effect for regular files and block devices; that is, I/O operations will (briefly) block when device activity is required, regardless of whether O_NONBLOCK is set. Since O_NONBLOCK semantics might eventually be implemented, applications should not depend upon blocking behavior when specifying this flag for regular files and block devices."
>
> An alternative could be preadv2 with the flag RWF_NOWAIT. But it is only available since linux 4.14 and we are using an older kernel. Also epoll doesn't wait for preadv2.
I am a bit cautious about aio_write etc. since it requires you to link with -lrt according to the manpage. We actually wanted to keep it at just -lc since that helps in case of other C libraries besides glibc.
That you require signal handling inside the API is something that worries me additionally. Even with signalfd, doing any complex signal handling besides SIGTERM is kinda messy. I think that AIO support might be better done outside of ELL.
What kind of file-reading or file-writing are you doing that you need AIO support anyway? You can actually poll() a file fd. You just have to do it carefully so it doesn’t become a busy loop. Have you tried just using l_idle to only read/write when we having nothing else going on in our mainloop?
Regards
Marcel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-19 19:43 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-19 19:43 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 1746 bytes --]
Hi Denis,
On 9/19/21 6:00 PM, Denis Kenzior wrote:
> Hi Simon,
>
> On 9/19/21 12:12 PM, Simon Maurer wrote:
>> Hi
>> Awesome library! Thank you very much. I'm using it for an embedded linux
>
> Glad you like it!
>
>> application with not so strict real time constraints. Problem is, I
>> need non blocking file access, because the memory card on my system is
>> quite slow. It turns out that this whole non-blocking access doesn't
>> work for files, so now I'm
>
> Can you elaborate? I always thought
> fcntl(fd, F_SETFL, O_NONBLOCK);
>
> would enable non-blocking behavior on files just fine? Or is this an
> issue with a particular driver?
>
I tested it and never got EAGAIN. I found this in the man page of open
for the flag O_NONBLOCK or O_NDELAY:
"Note that this flag has no effect for regular files and block devices;
that is, I/O operations will (briefly) block when device activity is
required, regardless of whether O_NONBLOCK is set. Since O_NONBLOCK
semantics might eventually be implemented, applications should not
depend upon blocking behavior when specifying this flag for regular
files and block devices."
An alternative could be preadv2 with the flag RWF_NOWAIT. But it is only
available since linux 4.14 and we are using an older kernel. Also epoll
doesn't wait for preadv2.
>> using the POSIX AIO interface and I've integrated it into ELL:
>> https://github.com/mausys/ell/blob/master/ell/aio.c
>> (But needs to be linked against rt)
>> This is just a draft and it also works on top of ELL, but maybe other
>> applications need this too.
>>
>
> Thanks, I'll take a peek later during the week.
>
> Regards,
> -Denis
Best regards,
Simon
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: AIO support
@ 2021-09-19 18:00 Denis Kenzior
0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-19 18:00 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 928 bytes --]
Hi Simon,
On 9/19/21 12:12 PM, Simon Maurer wrote:
> Hi
> Awesome library! Thank you very much. I'm using it for an embedded linux
Glad you like it!
> application with not so strict real time constraints. Problem is, I need non
> blocking file access, because the memory card on my system is quite slow. It
> turns out that this whole non-blocking access doesn't work for files, so now I'm
Can you elaborate? I always thought
fcntl(fd, F_SETFL, O_NONBLOCK);
would enable non-blocking behavior on files just fine? Or is this an issue with
a particular driver?
> using the POSIX AIO interface and I've integrated it into ELL:
> https://github.com/mausys/ell/blob/master/ell/aio.c
> (But needs to be linked against rt)
> This is just a draft and it also works on top of ELL, but maybe other
> applications need this too.
>
Thanks, I'll take a peek later during the week.
Regards,
-Denis
^ permalink raw reply [flat|nested] 20+ messages in thread
* AIO support
@ 2021-09-19 17:12 Simon Maurer
0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-19 17:12 UTC (permalink / raw)
To: ell
[-- Attachment #1: Type: text/plain, Size: 606 bytes --]
Hi
Awesome library! Thank you very much. I'm using it for an embedded linux
application with not so strict real time constraints. Problem is, I need
non blocking file access, because the memory card on my system is quite
slow. It turns out that this whole non-blocking access doesn't work for
files, so now I'm using the POSIX AIO interface and I've integrated it
into ELL:
https://github.com/mausys/ell/blob/master/ell/aio.c
(But needs to be linked against rt)
This is just a draft and it also works on top of ELL, but maybe other
applications need this too.
Best regards,
Simon Maurer
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2021-09-25 17:37 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-20 10:22 AIO support Simon Maurer
-- strict thread matches above, loose matches on Subject: below --
2021-09-25 17:37 Simon Maurer
2021-09-24 14:21 Denis Kenzior
2021-09-24 8:36 Simon Maurer
2021-09-22 11:46 Simon Maurer
2021-09-21 21:43 Denis Kenzior
2021-09-20 20:05 Simon Maurer
2021-09-20 16:52 Simon Maurer
2021-09-20 16:31 Denis Kenzior
2021-09-20 16:00 Simon Maurer
2021-09-20 15:29 Denis Kenzior
2021-09-20 15:23 Simon Maurer
2021-09-20 9:20 Marcel Holtmann
2021-09-20 9:14 Simon Maurer
2021-09-20 8:15 Simon Maurer
2021-09-19 21:11 Denis Kenzior
2021-09-19 19:59 Marcel Holtmann
2021-09-19 19:43 Simon Maurer
2021-09-19 18:00 Denis Kenzior
2021-09-19 17:12 Simon Maurer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.