Re: Linux has timerfd, signalfd and eventfd. Is it time for a "unionfd"?

Re: Linux has timerfd, signalfd and eventfd. Is it time for a "unionfd"?

[Joe Doe]

I just realized that epoll can do exactly what I asked for, so please ignore!

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Linux has timerfd, signalfd and eventfd. Is it time for a "unionfd"?

[Yann Droneaud]

Hi,

Le mardi 05 janvier 2021 à 17:13 -0800, Joe Doe a écrit :
> One of the architectures that Linux applications use, is to run in a
> loop that watches a number of FDs (e.g. epoll()) and react to the
> events from those FDs.
> 
> Now let's say I am writing a communication library that users can use
> to send and receive messages to other processes over the network. And
> let's assume that for reasons, my library is using multiple
> connections internally.
> 
> I have no way to construct my library in such a way that it will only
> export a single FD to the user, and hide the rest of the internal
> connections. I will either have to expose all the FDs, making it more
> difficult for the user, or I would have to run the library code in a
> different thread, and use eventfd to make the interface simpler.
> 
> Having a "UnionFD" that would get triggered when one of my other FDs
> is ready for I/O would solve the problem.
> 

It's already here, epollfd_create1() returns a file descriptor that can
be polled.


> For example an API of the hypothetical library with the tools we have
> today might look something like this:
> 
> int comm_lib_init(void); // Returns the "master" FD
> int comm_lib_accept(int master_fd); // Returns an FD when a new
> connection is accepted
> int comm_lib_send_message(int master_fd, const char *dest, void
> *data,
> size_t data_len); // If a new connection was required to send this
> message, then it will return the FD
> int comm_lib_recv_message(int master_fd, ...);
> 
> The user of the library would have to do the bookkeeping for all the
> FDs that are returned by the above APIs themself.
> 
> Instead with a "UnionFD", the API would only ever return a single FD
> and the user would call `comm_lib_send_message` and
> `comm_lib_recv_message` with this FD.
> 
> E.g. the application's code would look something like that:
> 
> comm_fd = comm_lib_init();
> epollfd = poll_create1(0);
> 
> ev.events = EPOLLIN;
> ev.data.fd = comm_fd;
> epoll_ctl(epollfd, EPOLL_CTL_ADD, comm_fd, &ev);
> for (;;) {
>      fds = epoll_wait(epollfd, events, MAX_EVENTS, -1);
> }
> 
> Once in the loop, the user doesn't have to worry about FD bookkeeping
> anymore.
> 
> Another example would be a much simpler HTTP library. The library
> would just return a single "master_fd" that the user would plug in
> their epoll() and just call something like `get_next_request()` when
> the FD gets triggered. That way we could easily add this HTTP server
> to an epoll() loop that monitors other FDs as well. No more worrying
> about accept(), and per-connection FD bookkeeping.
> 
> I believe a construct like this would help some popular existing
> libraries as well, like zeromq and the grpc library.
> 
> Does the above make sense? Or is there already a way to achieve that
> in Linux that I am not aware of?
> 
> Thanks,
> 

Regards.

-- 
Yann Droneaud
OPTEYA



_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Linux has timerfd, signalfd and eventfd. Is it time for a "unionfd"?

[Joe Doe]

One of the architectures that Linux applications use, is to run in a
loop that watches a number of FDs (e.g. epoll()) and react to the
events from those FDs.

Now let's say I am writing a communication library that users can use
to send and receive messages to other processes over the network. And
let's assume that for reasons, my library is using multiple
connections internally.

I have no way to construct my library in such a way that it will only
export a single FD to the user, and hide the rest of the internal
connections. I will either have to expose all the FDs, making it more
difficult for the user, or I would have to run the library code in a
different thread, and use eventfd to make the interface simpler.

Having a "UnionFD" that would get triggered when one of my other FDs
is ready for I/O would solve the problem.

For example an API of the hypothetical library with the tools we have
today might look something like this:

int comm_lib_init(void); // Returns the "master" FD
int comm_lib_accept(int master_fd); // Returns an FD when a new
connection is accepted
int comm_lib_send_message(int master_fd, const char *dest, void *data,
size_t data_len); // If a new connection was required to send this
message, then it will return the FD
int comm_lib_recv_message(int master_fd, ...);

The user of the library would have to do the bookkeeping for all the
FDs that are returned by the above APIs themself.

Instead with a "UnionFD", the API would only ever return a single FD
and the user would call `comm_lib_send_message` and
`comm_lib_recv_message` with this FD.

E.g. the application's code would look something like that:

comm_fd = comm_lib_init();
epollfd = poll_create1(0);

ev.events = EPOLLIN;
ev.data.fd = comm_fd;
epoll_ctl(epollfd, EPOLL_CTL_ADD, comm_fd, &ev);
for (;;) {
     fds = epoll_wait(epollfd, events, MAX_EVENTS, -1);
}

Once in the loop, the user doesn't have to worry about FD bookkeeping anymore.

Another example would be a much simpler HTTP library. The library
would just return a single "master_fd" that the user would plug in
their epoll() and just call something like `get_next_request()` when
the FD gets triggered. That way we could easily add this HTTP server
to an epoll() loop that monitors other FDs as well. No more worrying
about accept(), and per-connection FD bookkeeping.

I believe a construct like this would help some popular existing
libraries as well, like zeromq and the grpc library.

Does the above make sense? Or is there already a way to achieve that
in Linux that I am not aware of?

Thanks,

Joe

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies