All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: AIO support
@ 2021-09-20 10:22 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 10:22 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]

Hi Marcel,

No, because I don't believe it will work. I think the underlying problem 
is the difference between streams and files. When a process creates a 
stream (e.g. socket), the kernel creates a buffer and can start 
receiving data from an external source. This buffer can be accessed 
non-blocking from the process and the kernel can signal to the process 
when data was received. If the process opens a file, the kernel doesn't 
know what the process will do next (read or write, offset of the file, 
transfer size). So the kernel doesn't read/write any bytes unless the 
process sends a command to the kernel. And the POSIX read/write 
functions on files are always blocking. That leaves us with using 
multi-threading (POSIX AIO) or other linux specific syscalls (libaio, 
preadv2).

Regards
Simon

On 9/20/21 9:20 AM, Marcel Holtmann wrote:
> Hi Simon,
> 
>> I'm looking now into libaio. POSIX AIO is implemented in libc with user threads, while libaio uses linux native syscalls. Maybe I can also get rid of signal usage. IMHO non-blocking file access / async IO is quite important for embedded applications and it would be nice to have this feature in ell.
> 
> have you looked into using l_idle for providing async read/write callbacks that way. I have the feeling that AIO is total overkill for embedded applications.
> 
> Regards
> 
> Marcel
> 

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-25 17:37 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-25 17:37 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

Hi Denis

On 9/24/21 4:21 PM, Denis Kenzior wrote:

>> I started testing the aio interface for ell and it seems to work, 
>> except it's not asynchronous. Turns out libaio works only under 
>> certain circumstances:
>> http://lse.sourceforge.net/io/aio.html
> 
> yes, this was a widely reported criticism.  I assume O_DIRECT is not 
> applicable to your use case?

I tried to open the file with O_DIRECT, but still got no asynchronous 
operations. I don't know what the problem was, NVMe, btrfs, ... also I 
tried it on an usb stick with FAT32. Anyway, if it's only working under 
certain conditions it's useless and dangerous for real time 
applications. I can't expect the user of my aio interface to be an 
expert in Linux filesystems and knowing all the restrictions of aio.

> 
> Honestly, for ell, io_uring would be much more interesting to support.  
> With maybe a fallback to aio or a threaded/blocking implementation for 
> older kernels.

I might write an io_uring implementation in the future. But since 
io_uring is not available on my target system, I have to stick with 
POSIX AIO. A lot of Linux BSPs are only available for outdated kernel.

Regards,
Simon

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-24 14:21 Denis Kenzior
  0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-24 14:21 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 898 bytes --]

Hi Simon,

On 9/24/21 3:36 AM, Simon Maurer wrote:
> I started testing the aio interface for ell and it seems to work, except it's 
> not asynchronous. Turns out libaio works only under certain circumstances:
> http://lse.sourceforge.net/io/aio.html

yes, this was a widely reported criticism.  I assume O_DIRECT is not applicable 
to your use case?

> So it's useless. There is a newer alternative called io_uring, but only for 
> newer kernels (5.1) and not available to my target system. I will go back to the

Honestly, for ell, io_uring would be much more interesting to support.  With 
maybe a fallback to aio or a threaded/blocking implementation for older kernels.

> POSIX AIO and use SIGEV_THREAD (writing an eventfd in the notify function) 
> instead of SIGEV_SIGNAL.
> 

You still have to contend with librt use, which I think Marcel was against.

Regards,
-Denis

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-24  8:36 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-24  8:36 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 483 bytes --]

I started testing the aio interface for ell and it seems to work, except 
it's not asynchronous. Turns out libaio works only under certain 
circumstances:
http://lse.sourceforge.net/io/aio.html
So it's useless. There is a newer alternative called io_uring, but only 
for newer kernels (5.1) and not available to my target system. I will go 
back to the POSIX AIO and use SIGEV_THREAD (writing an eventfd in the 
notify function) instead of SIGEV_SIGNAL.

Regards,
Simon


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-22 11:46 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-22 11:46 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 164 bytes --]

I have a first draft of the complete aio API. l_aio_cancel doesn't work 
as expected and also timeout adjustment has not yet been implemented.

Regards,
Simon

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-21 21:43 Denis Kenzior
  0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-21 21:43 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 302 bytes --]

Hi Simon,

On 9/20/21 3:05 PM, Simon Maurer wrote:
> I started a new version of aio.c:
> https://github.com/mausys/ell/blob/master/ell/aio.c
> but I have yet to implement l_io_destroy and l_io_cancel

I took a _very_ quick peek and what you have here so far seems reasonable.

Regards,
-Denis

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-20 20:05 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 20:05 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 144 bytes --]

I started a new version of aio.c:
https://github.com/mausys/ell/blob/master/ell/aio.c
but I have yet to implement l_io_destroy and l_io_cancel

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-20 16:52 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 16:52 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 0 bytes --]



[-- Attachment #2: attachment.htm --]
[-- Type: text/html, Size: 1673 bytes --]

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-20 16:31 Denis Kenzior
  0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-20 16:31 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

Hi Simon,

Please no top-posting.

On 9/20/21 11:00 AM, Simon Maurer wrote:
> Hi Denis,
> 
> Yes I agree, but I still need the the syscall wrappers, because glibc doesn't 

Can't we syscall() in ell, like we do with keyctl in key.c?

> provide them. So bundling it with ell and make libaio a private library of ell 
> seams to be a good solution for me.

That would not be our first choice.

Regards,
-Denis

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-20 16:00 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 16:00 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

Hi Denis,

Yes I agree, but I still need the the syscall wrappers, because glibc 
doesn't provide them. So bundling it with ell and make libaio a private 
library of ell seams to be a good solution for me.

Regards,
Simon

On 9/20/21 3:29 PM, Denis Kenzior wrote:
> Hi Simon,
> 
> On 9/20/21 10:23 AM, Simon Maurer wrote:
>> I wrote a little "proof of concept" for using ell with libaio:
>> https://github.com/mausys/ellaio/blob/main/aio.c
>> No more signals and threads. Also libaio is LGPL too, so it could be 
>> bundled with ell.
> 
> libaio seems to be a small convenience wrapper around linux system calls 
> (~500 lines).  Why not use io_submit, io_setup, etc directly for the 
> parts you need?
> 
> Regards,
> -Denis

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-20 15:29 Denis Kenzior
  0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-20 15:29 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

Hi Simon,

On 9/20/21 10:23 AM, Simon Maurer wrote:
> I wrote a little "proof of concept" for using ell with libaio:
> https://github.com/mausys/ellaio/blob/main/aio.c
> No more signals and threads. Also libaio is LGPL too, so it could be bundled 
> with ell.

libaio seems to be a small convenience wrapper around linux system calls (~500 
lines).  Why not use io_submit, io_setup, etc directly for the parts you need?

Regards,
-Denis

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-20 15:23 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20 15:23 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 220 bytes --]

I wrote a little "proof of concept" for using ell with libaio:
https://github.com/mausys/ellaio/blob/main/aio.c
No more signals and threads. Also libaio is LGPL too, so it could be 
bundled with ell.

Regards
Simon

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-20  9:20 Marcel Holtmann
  0 siblings, 0 replies; 20+ messages in thread
From: Marcel Holtmann @ 2021-09-20  9:20 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 498 bytes --]

Hi Simon,

> I'm looking now into libaio. POSIX AIO is implemented in libc with user threads, while libaio uses linux native syscalls. Maybe I can also get rid of signal usage. IMHO non-blocking file access / async IO is quite important for embedded applications and it would be nice to have this feature in ell.

have you looked into using l_idle for providing async read/write callbacks that way. I have the feeling that AIO is total overkill for embedded applications.

Regards

Marcel

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-20  9:14 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20  9:14 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

I'm looking now into libaio. POSIX AIO is implemented in libc with user 
threads, while libaio uses linux native syscalls. Maybe I can also get 
rid of signal usage. IMHO non-blocking file access / async IO is quite 
important for embedded applications and it would be nice to have this 
feature in ell.

Regards
Simon

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-20  8:15 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-20  8:15 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 3178 bytes --]

Hi Marcel

On 9/19/21 7:59 PM, Marcel Holtmann wrote:
> Hi Simon,
> 
>>>> Awesome library! Thank you very much. I'm using it for an embedded linux
>>> Glad you like it!
>>>> application with not so strict real time constraints. Problem is, I need non blocking file access, because the memory card on my system is quite slow. It turns out that this whole non-blocking access doesn't work for files, so now I'm
>>> Can you elaborate?  I always thought
>>> fcntl(fd, F_SETFL, O_NONBLOCK);
>>> would enable non-blocking behavior on files just fine?  Or is this an issue with a particular driver?
>>
>> I tested it and never got EAGAIN. I found this in the man page of open for the flag O_NONBLOCK or O_NDELAY:
>>
>> "Note  that this flag has no effect for regular files and block devices; that is, I/O operations will (briefly) block when device activity is required, regardless of whether O_NONBLOCK is set.  Since O_NONBLOCK semantics might eventually be implemented, applications should not depend  upon  blocking  behavior  when specifying this flag for regular files and block devices."
>>
>> An alternative could be preadv2 with the flag RWF_NOWAIT. But it is only available since linux 4.14 and we are using an older kernel. Also epoll doesn't wait for preadv2.
> 
> I am a bit cautious about aio_write etc. since it requires you to link with -lrt according to the manpage. We actually wanted to keep it at just -lc since that helps in case of other C libraries besides glibc.

I don't like that either. If I am informed correctly librt is part of 
glibc and in verison 2.34, librt is not needed anymore. MUSL supports 
POSIX AIO, but neither android bionic nor ulibc implemented it. I 
thought of making aio an optional part of ell, and handle the different 
cases in the configure script.

> 
> That you require signal handling inside the API is something that worries me additionally. Even with signalfd, doing any complex signal handling besides SIGTERM is kinda messy. I think that AIO support might be better done outside of ELL.

Signal handling in my aio implementation is optional and the signal just 
triggers l_aio_poll, which checks all outstanding requests. The problem 
I had with the signalhandling was when multiple requests were completed 
at the same time (between epoll_wait calls), the signal handler was 
executed only once. My first approach was to send request information 
with the signal but that failed for multiple requests. My patch earlier 
made this information available to the user, but independet of AIO 
handling, I still think it might be a good idea to make siginfo 
available to the user.

> 
> What kind of file-reading or file-writing are you doing that you need AIO support anyway? You can actually poll() a file fd. You just have to do it carefully so it doesn’t become a busy loop. Have you tried just using l_idle to only read/write when we having nothing else going on in our mainloop?
> 
> Regards
> 
> Marcel
> _______________________________________________
> ell mailing list -- ell(a)lists.01.org
> To unsubscribe send an email to ell-leave(a)lists.01.org
> 

Regards
Simon

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-19 21:11 Denis Kenzior
  0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-19 21:11 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

Hi Simon,

>> Can you elaborate?  I always thought
>> fcntl(fd, F_SETFL, O_NONBLOCK);
>>
>> would enable non-blocking behavior on files just fine?  Or is this an issue 
>> with a particular driver?
>>
> 
> I tested it and never got EAGAIN. I found this in the man page of open for the 
> flag O_NONBLOCK or O_NDELAY:
> 
> "Note  that this flag has no effect for regular files and block devices; that 
> is, I/O operations will (briefly) block when device activity is required, 
> regardless of whether O_NONBLOCK is set.  Since O_NONBLOCK semantics might 
> eventually be implemented, applications should not depend  upon  blocking  
> behavior  when specifying this flag for regular files and block devices."
> 

Ah, indeed.  I think I even knew that, sorry for the noise.

> An alternative could be preadv2 with the flag RWF_NOWAIT. But it is only 
> available since linux 4.14 and we are using an older kernel. Also epoll doesn't 
> wait for preadv2.

Right, epoll will just simply return immediately on regular files.  Have you 
tried using a timer or l_idle to drive the reading?  Is there ioctl(fd, 
FIONREAD, ...) or equivalent that could be used not to block?

Regards,
-Denis

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-19 19:59 Marcel Holtmann
  0 siblings, 0 replies; 20+ messages in thread
From: Marcel Holtmann @ 2021-09-19 19:59 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 1993 bytes --]

Hi Simon,

>>> Awesome library! Thank you very much. I'm using it for an embedded linux 
>> Glad you like it!
>>> application with not so strict real time constraints. Problem is, I need non blocking file access, because the memory card on my system is quite slow. It turns out that this whole non-blocking access doesn't work for files, so now I'm 
>> Can you elaborate?  I always thought
>> fcntl(fd, F_SETFL, O_NONBLOCK);
>> would enable non-blocking behavior on files just fine?  Or is this an issue with a particular driver?
> 
> I tested it and never got EAGAIN. I found this in the man page of open for the flag O_NONBLOCK or O_NDELAY:
> 
> "Note  that this flag has no effect for regular files and block devices; that is, I/O operations will (briefly) block when device activity is required, regardless of whether O_NONBLOCK is set.  Since O_NONBLOCK semantics might eventually be implemented, applications should not depend  upon  blocking  behavior  when specifying this flag for regular files and block devices."
> 
> An alternative could be preadv2 with the flag RWF_NOWAIT. But it is only available since linux 4.14 and we are using an older kernel. Also epoll doesn't wait for preadv2.

I am a bit cautious about aio_write etc. since it requires you to link with -lrt according to the manpage. We actually wanted to keep it at just -lc since that helps in case of other C libraries besides glibc.

That you require signal handling inside the API is something that worries me additionally. Even with signalfd, doing any complex signal handling besides SIGTERM is kinda messy. I think that AIO support might be better done outside of ELL.

What kind of file-reading or file-writing are you doing that you need AIO support anyway? You can actually poll() a file fd. You just have to do it carefully so it doesn’t become a busy loop. Have you tried just using l_idle to only read/write when we having nothing else going on in our mainloop?

Regards

Marcel

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-19 19:43 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-19 19:43 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 1746 bytes --]

Hi Denis,


On 9/19/21 6:00 PM, Denis Kenzior wrote:
> Hi Simon,
> 
> On 9/19/21 12:12 PM, Simon Maurer wrote:
>> Hi
>> Awesome library! Thank you very much. I'm using it for an embedded linux 
> 
> Glad you like it!
> 
>> application with not so strict real time constraints. Problem is, I 
>> need non blocking file access, because the memory card on my system is 
>> quite slow. It turns out that this whole non-blocking access doesn't 
>> work for files, so now I'm 
> 
> Can you elaborate?  I always thought
> fcntl(fd, F_SETFL, O_NONBLOCK);
> 
> would enable non-blocking behavior on files just fine?  Or is this an 
> issue with a particular driver?
> 

I tested it and never got EAGAIN. I found this in the man page of open 
for the flag O_NONBLOCK or O_NDELAY:

"Note  that this flag has no effect for regular files and block devices; 
that is, I/O operations will (briefly) block when device activity is 
required, regardless of whether O_NONBLOCK is set.  Since O_NONBLOCK 
semantics might eventually be implemented, applications should not 
depend  upon  blocking  behavior  when specifying this flag for regular 
files and block devices."

An alternative could be preadv2 with the flag RWF_NOWAIT. But it is only 
available since linux 4.14 and we are using an older kernel. Also epoll 
doesn't wait for preadv2.


>> using the POSIX AIO interface and I've integrated it into ELL:
>> https://github.com/mausys/ell/blob/master/ell/aio.c
>> (But needs to be linked against rt)
>> This is just a draft and it also works on top of ELL, but maybe other 
>> applications need this too.
>>
> 
> Thanks, I'll take a peek later during the week.
> 
> Regards,
> -Denis

Best regards,
Simon

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: AIO support
@ 2021-09-19 18:00 Denis Kenzior
  0 siblings, 0 replies; 20+ messages in thread
From: Denis Kenzior @ 2021-09-19 18:00 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

Hi Simon,

On 9/19/21 12:12 PM, Simon Maurer wrote:
> Hi
> Awesome library! Thank you very much. I'm using it for an embedded linux 

Glad you like it!

> application with not so strict real time constraints. Problem is, I need non 
> blocking file access, because the memory card on my system is quite slow. It 
> turns out that this whole non-blocking access doesn't work for files, so now I'm 

Can you elaborate?  I always thought
fcntl(fd, F_SETFL, O_NONBLOCK);

would enable non-blocking behavior on files just fine?  Or is this an issue with 
a particular driver?

> using the POSIX AIO interface and I've integrated it into ELL:
> https://github.com/mausys/ell/blob/master/ell/aio.c
> (But needs to be linked against rt)
> This is just a draft and it also works on top of ELL, but maybe other 
> applications need this too.
> 

Thanks, I'll take a peek later during the week.

Regards,
-Denis

^ permalink raw reply	[flat|nested] 20+ messages in thread

* AIO support
@ 2021-09-19 17:12 Simon Maurer
  0 siblings, 0 replies; 20+ messages in thread
From: Simon Maurer @ 2021-09-19 17:12 UTC (permalink / raw)
  To: ell

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

Hi
Awesome library! Thank you very much. I'm using it for an embedded linux 
application with not so strict real time constraints. Problem is, I need 
non blocking file access, because the memory card on my system is quite 
slow. It turns out that this whole non-blocking access doesn't work for 
files, so now I'm using the POSIX AIO interface and I've integrated it 
into ELL:
https://github.com/mausys/ell/blob/master/ell/aio.c
(But needs to be linked against rt)
This is just a draft and it also works on top of ELL, but maybe other 
applications need this too.

Best regards,
Simon Maurer

^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2021-09-25 17:37 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-20 10:22 AIO support Simon Maurer
  -- strict thread matches above, loose matches on Subject: below --
2021-09-25 17:37 Simon Maurer
2021-09-24 14:21 Denis Kenzior
2021-09-24  8:36 Simon Maurer
2021-09-22 11:46 Simon Maurer
2021-09-21 21:43 Denis Kenzior
2021-09-20 20:05 Simon Maurer
2021-09-20 16:52 Simon Maurer
2021-09-20 16:31 Denis Kenzior
2021-09-20 16:00 Simon Maurer
2021-09-20 15:29 Denis Kenzior
2021-09-20 15:23 Simon Maurer
2021-09-20  9:20 Marcel Holtmann
2021-09-20  9:14 Simon Maurer
2021-09-20  8:15 Simon Maurer
2021-09-19 21:11 Denis Kenzior
2021-09-19 19:59 Marcel Holtmann
2021-09-19 19:43 Simon Maurer
2021-09-19 18:00 Denis Kenzior
2021-09-19 17:12 Simon Maurer

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.