WaitForMultipleObjects/etc. In Kernel

WaitForMultipleObjects/etc. In Kernel

[Network Nut]

Hi All,

This is my first post to anything Linux, so if there is a better mailing
list, please let me know.

I think that the facility by which a thread can block while waiting for any
of several synchronization primitives (*mutex*, *semaphore*, *event*,
*waitable
timer*)...is not only "nice to have", but fundamental to complex (clean)
multi-threaded programming. Of course, this facility is available on
Windows as *WaitForMultipleObjects *with all the associated functions.

That said, I have a C++ project on Windows that is nearly 100% portable.
The part that is not portable is mostly the synchronization elements. Every
year or so I take a cursory look at Linux's (Unix's) synchronization model,
and the process thread model, and...I guess you have heard it before - it's
simply not the same.

I have also seen various attempts on the web to create wrapper functions
that provide a Windows synchro API on Linux, but again, the result is never
quite the same. IBM, in their attempt and exposition, admits this. It is
clear, at least to me, that if one wants true ability to wait for any of
the various synchronization objects simultaneously, there needs to be
deliberate, explicit kernel support.

I was wondering:

   1. What is the likelihood that the guardians of the kernel would even
   allow something so fundamentally different inside?
   2. My gut feeling is that adding such support would fundamentally change


   other aspects of Linux. For example, I am vaguely familiar with asio, but
   to do it the way it was done in Windows...

Finally, this is not a situation where the end-game is having a kernel of
my own that has such support. Support would need to be pre-existing for the
users of my project.

Please note that I am not a Windows bigot trying to impose a
Microsoft-centric world-view on Linux. I think this facility transcends any
particular OS, and I feel that it is merely fortunate coincidence that the
Microsoft (DEC) engineers had the inclination to do it.

-Nut

Re: WaitForMultipleObjects/etc. In Kernel

[Clemens Ladisch]

Network Nut wrote:
> I think that the facility by which a thread can block while waiting for any
> of several synchronization primitives (*mutex*, *semaphore*, *event*, *waitable
> timer*)...is not only "nice to have", but fundamental to complex (clean)
> multi-threaded programming.

You mean a facility like (e)poll, which can wait for things like timerfd,
signalfd, or eventfd?


Regards,
Clemens

RE: WaitForMultipleObjects/etc. In Kernel

[Network Nut]

> -----Original Message-----
> From: Clemens Ladisch [mailto:clemens@ladisch.de]
> Sent: Sunday, January 26, 2014 12:33 PM
> To: Network Nut
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: WaitForMultipleObjects/etc. In Kernel
> 
> Network Nut wrote:
> > I think that the facility by which a thread can block while waiting
> > for any of several synchronization primitives (*mutex*, *semaphore*,
> > *event*, *waitable timer*)...is not only "nice to have", but
> > fundamental to complex (clean) multi-threaded programming.
> 
> You mean a facility like (e)poll, which can wait for things like timerfd,
signalfd,
> or eventfd?
> 
> 
> Regards,
> Clemens

Yes, that seems to be what I am looking for.

I have a concern:

As you know, under Windows, synchronization objects such as {event | mutex |
semaphore | timer}; all have names that are computer-global. Process B can
open, and use, any {event | mutex | semaphore | timer} that was created by
process A, as long as Process B knows the name that was used by Process A to
create the {event | mutex | semaphore | timer}. These synchronization
objects may also be nameless if the programmer so chooses, in which case,
process B would have to use a circuitous method to determine how to find the
kernel-object created by Process A.

Is there a method I can use where Process A and Process B can both refer to
the same eventfd by name?

BTW, the man page for epoll_wait seems to be incorrect. It says:

"The timeout argument specifies the minimum number of milliseconds
       that epoll_wait() will block."

I think the word "minimum" should be "maximum".

Regards,

-Nut

Re: WaitForMultipleObjects/etc. In Kernel

[Clemens Ladisch]

Network Nut wrote:
> As you know, under Windows, synchronization objects such as {event | mutex |
> semaphore | timer}; all have names that are computer-global. Process B can
> open, and use, any {event | mutex | semaphore | timer} that was created by
> process A, as long as Process B knows the name that was used by Process A to
> create the {event | mutex | semaphore | timer}. [...]
> Is there a method I can use where Process A and Process B can both refer to
> the same eventfd by name?

In Unix, the most common method of sharing file descriptors is by
letting child processes inherit them.  (This also works in Windows, but
is not as efficient due to the process creation overhead.)

Unrelated processes cannot directly open objects created by another
process (with the exception of sockets and pipes, which can be created
in the file system).  However, sharing of any file descriptor is
possible by sending it in a control message through a Unix domain
socket.

> BTW, the man page for epoll_wait seems to be incorrect.

<https://www.kernel.org/doc/man-pages/reporting_bugs.html>

> "The timeout argument specifies the minimum number of milliseconds
>        that epoll_wait() will block."
>
> I think the word "minimum" should be "maximum".

This sentence was copied from the poll(2) man page, where the previous
sentence says "poll() blocks until one of the events occurs".  So the
word "block" implies that no event has occured.


Regards,
Clemens

RE: WaitForMultipleObjects/etc. In Kernel

[Network Nut]

> Unrelated processes cannot directly open objects created by another
> process (with the exception of sockets and pipes, which can be created in
> the file system).  However, sharing of any file descriptor is possible by
> sending it in a control message through a Unix domain socket.

I just spent a bit more time snooping around the Internet to see what is
possible and what is not.

My gut feeling, before trying to port my application from Windows to Linux,
was that I have been using (enjoying) synchronization primitives on Windows
that are fundamental to generalized-synchronization. Most importantly, as I
mentioned initially, I do not believe that I am victim of tunnel-vision,
where my familiarity with a particular OS has biased my perception of how
synchronization should be done. On the contrary, knowing what I know about
synchronization now, if it were left to me to design a Generalized
Synchronization Model for an OS kernel from scratch, without regard for
pre-existing OS's, the model that I would devise would probably look very
similar to what the Windows engineers  did. I feel a bit bad saying this, as
my intent here is not to take a stab at Linux's synchronization model. It's
to say...I feel that I am struggling to synthesize what I regard as a
"regular model" from bits and pieces on Linux. I am convinced that, if I
were going in the other direction, from Linux to Windows, I would not have
any problem recreating, under Windows, whatever model I had been using under
Linux.

That said, I really do need to have multiple threads, each within their
respective processes, P1:T1, P2:T2, ...Pn:Tn; all blocking on what are
effectively potentially globally-named synchronization primitives
(semaphore, mutex event)...and...ideally...as each thread blocks against
multiple, potentially-named primitives, there is an option for the master
blocking function [epoll_wait, in this case], to time-out, with the
expiration for time-out being specified as an argument to the function. And
to make matters worse, sometimes, while blocking on {semaphore + event +
event +....}, I need to block, simultaneously, on asynchronous I/O.

So what I know so far is that, if my problem is to be solved:

1. epoll/epoll_wait/etc. will definitely be part of it (Thanks!)
2. I am OK with timerfd and eventfd (Thanks!)
3. Other synchronization primitives will have to be accessible vial
file-descriptor.
4. If epoll_wait is to block on asynchronous I/O, use io_set_eventfd  to
bind completion of asynchronous I/O to signaling of an eventfd
5. I can simulate system-global named mutex using shared-memory for
underlying state of mutex (POCO NamedMutex)
6. I can get named semaphore using POSIX sem_create 

It seems that the remaining problem is to get named mutex and named
semaphore to be accessible by file-descriptor.

> > BTW, the man page for epoll_wait seems to be incorrect.
> 
> <https://www.kernel.org/doc/man-pages/reporting_bugs.html>
> 
> > "The timeout argument specifies the minimum number of milliseconds
> >        that epoll_wait() will block."
> >
> > I think the word "minimum" should be "maximum".
> 
> This sentence was copied from the poll(2) man page, where the previous
> sentence says "poll() blocks until one of the events occurs".  So the word
> "block" implies that no event has occured.

So that means that it is a bug, right? Or?

Regards,

-Nut

Re: WaitForMultipleObjects/etc. In Kernel

[Clemens Ladisch]

Network Nut wrote:
> 5. I can simulate system-global named mutex using shared-memory for
> underlying state of mutex (POCO NamedMutex)
> 6. I can get named semaphore using POSIX sem_create
>
> It seems that the remaining problem is to get named mutex and named
> semaphore to be accessible by file-descriptor.

Forget about the POSIX stuff.  You can implement a mutex by using
an eventfd that has the value 1 in the unlocked state; a read locks,
a write of 1 unlocks.  You can implement a semaphore by using an eventfd
with EFD_SEMAPHORE.


And I should note that it is a common pattern to start a bunch of other
processes from a helper process; this not only allows inheriting file
descriptors, but also makes monitoring these processes easier from the
parent.  Do all of your processes really need to be started
independently?


>>> "The timeout argument specifies the minimum number of milliseconds
>>>        that epoll_wait() will block."
>>>
>>> I think the word "minimum" should be "maximum".
>>
>> This sentence was copied from the poll(2) man page, where the previous
>> sentence says "poll() blocks until one of the events occurs".  So the word
>> "block" implies that no event has occured.
>
> So that means that it is a bug, right? Or?

The purpose of a man page is certainly not to confuse its readers.  I'd
guess the maintainer would be happy about an improved wording.


Regards,
Clemens

RE: WaitForMultipleObjects/etc. In Kernel

[Network Nut]

> -----Original Message-----
> From: Clemens Ladisch [mailto:clemens@ladisch.de]
> Sent: Tuesday, January 28, 2014 3:04 AM
> To: Network Nut
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: WaitForMultipleObjects/etc. In Kernel
> 
> Network Nut wrote:
> > 5. I can simulate system-global named mutex using shared-memory for
> > underlying state of mutex (POCO NamedMutex) 6. I can get named
> > semaphore using POSIX sem_create
> >
> > It seems that the remaining problem is to get named mutex and named
> > semaphore to be accessible by file-descriptor.
> 
> Forget about the POSIX stuff.  You can implement a mutex by using an
> eventfd that has the value 1 in the unlocked state; a read locks, a write
of 1
> unlocks.  You can implement a semaphore by using an eventfd with
> EFD_SEMAPHORE.

I was looking at POSIX because it allows naming of the primitives.

I need to reference inter-process {mutex, event, semaphore}, each identified
by string, if feasible.
I need to epoll_wait on inter-process {mutex, event, semaphore}.

> And I should note that it is a common pattern to start a bunch of other
> processes from a helper process; this not only allows inheriting file
> descriptors, but also makes monitoring these processes easier from the
> parent.  Do all of your processes really need to be started independently?

Yes, all my processes need to be started independently. 

Regards,

-Net

RE: WaitForMultipleObjects/etc. In Kernel

[Clemens Ladisch]

Network Nut wrote:
>I was looking at POSIX because it allows naming of the primitives.

Linux uses two orthogonal mechanisms for synchronization
primitives and for naming/sharing.

>I need to epoll_wait on inter-process {mutex, event, semaphore}.

Use eventfd.

>I need to reference inter-process {mutex, event, semaphore}, each
>identified
>by string, if feasible.

Send the fd through a Unix domain socket.


Regards,
Clemens

RE: WaitForMultipleObjects/etc. In Kernel

[Network Nut]

> -----Original Message-----
> From: Clemens Ladisch [mailto:clemens@ladisch.de]
> Sent: Wednesday, January 29, 2014 2:31 AM
> To: Network Nut
> Cc: linux-kernel@vger.kernel.org
> Subject: RE: WaitForMultipleObjects/etc. In Kernel
> 
> Network Nut wrote:
> >I was looking at POSIX because it allows naming of the primitives.
> 
> Linux uses two orthogonal mechanisms for synchronization primitives and for
> naming/sharing.
> 
> >I need to epoll_wait on inter-process {mutex, event, semaphore}.
> 
> Use eventfd.
> 
> >I need to reference inter-process {mutex, event, semaphore}, each
> >identified by string, if feasible.
> 
> Send the fd through a Unix domain socket.

Hi Again,

I was thinking that, rather than as for specifics, I should present my general problem, and ask how long-time Linux experts would solve it.

I have a master process M, that executes continually, from the birth to death of user-session.

I have many (distinct) processes that will be launched, and these processes, P1, P2, ...Pn, expect to see that M is executing. These processes:

1. expect to have access to a shared-memory section that already exists because M created it
2. expect to use a semaphore that already exists because M created it
3. expect to use a mutex that exists because M created it

P1, P2, ...Pn all know the path of image on disk of M. They are also permitted to maintain a fixed string that can be used to "get at" the mutex and semaphore.

How would P1, P2, ...Pn get at the semaphore that M created?

Please note that M cannot have any prior knowledge at all of P1, P2, ...Pn. P1...etc. must initiate  communication with M.

[I don't want to misuse/abuse linux-kernel with my personal questions, so if there is a more appropriate group, please let me know.]

Regards,

-Net

Re: WaitForMultipleObjects/etc. In Kernel

[Austin S. Hemmelgarn]

On 01/30/2014 06:49 PM, Network Nut wrote:
>> -----Original Message-----
>> From: Clemens Ladisch [mailto:clemens@ladisch.de]
>> Sent: Wednesday, January 29, 2014 2:31 AM
>> To: Network Nut
>> Cc: linux-kernel@vger.kernel.org
>> Subject: RE: WaitForMultipleObjects/etc. In Kernel
>>
>> Network Nut wrote:
>>> I was looking at POSIX because it allows naming of the primitives.
>>
>> Linux uses two orthogonal mechanisms for synchronization primitives and for
>> naming/sharing.
>>
>>> I need to epoll_wait on inter-process {mutex, event, semaphore}.
>>
>> Use eventfd.
>>
>>> I need to reference inter-process {mutex, event, semaphore}, each
>>> identified by string, if feasible.
>>
>> Send the fd through a Unix domain socket.
> 
> Hi Again,
> 
> I was thinking that, rather than as for specifics, I should present my general problem, and ask how long-time Linux experts would solve it.
> 
> I have a master process M, that executes continually, from the birth to death of user-session.
> 
> I have many (distinct) processes that will be launched, and these processes, P1, P2, ...Pn, expect to see that M is executing. These processes:
> 
> 1. expect to have access to a shared-memory section that already exists because M created it
> 2. expect to use a semaphore that already exists because M created it
> 3. expect to use a mutex that exists because M created it
> 
> P1, P2, ...Pn all know the path of image on disk of M. They are also permitted to maintain a fixed string that can be used to "get at" the mutex and semaphore.
> 
> How would P1, P2, ...Pn get at the semaphore that M created?
> 
> Please note that M cannot have any prior knowledge at all of P1, P2, ...Pn. P1...etc. must initiate  communication with M.
> 
> [I don't want to misuse/abuse linux-kernel with my personal questions, so if there is a more appropriate group, please let me know.]
> 
> Regards,
> 
> -Net
Assuming that you're porting to mainline distributions (and not embedded
devices), named SHM segments are accessible (providing the accessing
process has correct permissions) under /dev/shm.  You just need to make
sure that you create the segment with the right permissions for the
other processes to access it.

RE: WaitForMultipleObjects/etc. In Kernel

[Network Nut]

> -----Original Message-----
> From: Austin S. Hemmelgarn [mailto:ahferroin7@gmail.com]
> Sent: Friday, January 31, 2014 11:05 AM
> To: Network Nut; 'Clemens Ladisch'
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: WaitForMultipleObjects/etc. In Kernel

> >> Network Nut wrote:
> > I was thinking that, rather than as for specifics, I should present my general
> problem, and ask how long-time Linux experts would solve it.
> >
> > I have a master process M, that executes continually, from the birth to
> death of user-session.
> >
> > I have many (distinct) processes that will be launched, and these
> processes, P1, P2, ...Pn, expect to see that M is executing. These processes:
> >
> > 1. expect to have access to a shared-memory section that already exists
> because M created it
> > 2. expect to use a semaphore that already exists because M created it
> > 3. expect to use a mutex that exists because M created it
> >
> > P1, P2, ...Pn all know the path of image on disk of M. They are also
> permitted to maintain a fixed string that can be used to "get at" the mutex
> and semaphore.
> >
> > How would P1, P2, ...Pn get at the semaphore that M created?
> >
> > Please note that M cannot have any prior knowledge at all of P1, P2, ...Pn.
> P1...etc. must initiate  communication with M.
> >
> > [I don't want to misuse/abuse linux-kernel with my personal questions, so
> if there is a more appropriate group, please let me know.]
> >
> > Regards,
> >
> > -Net
> Assuming that you're porting to mainline distributions (and not embedded
> devices), named SHM segments are accessible (providing the accessing
> process has correct permissions) under /dev/shm.  You just need to make
> sure that you create the segment with the right permissions for the
> other processes to access it.

Thanks, Austin.

I already know how to do named shared memory between two processes. I only included that to describe my overall problem.

The problem that I am having is how I can make three totally-independent processes interact:

1. M is a master process that creates a semaphore.
2. P1 is a process that operates against the semaphore.
3. P2 is a process that operates against the semaphore.
4. It is not permissible that M be responsible for launching P1 or P2.
5.  The semaphore, one way or another, must allow itself to be specified as one of the synchronization primitives in epoll_wait()

How do I do this?

-Nut

Re: WaitForMultipleObjects/etc. In Kernel

[Clemens Ladisch]

Network Nut wrote:
>> Assuming that you're porting to mainline distributions (and not embedded
>> devices), named SHM segments are accessible (providing the accessing
>> process has correct permissions) under /dev/shm.  You just need to make
>> sure that you create the segment with the right permissions for the
>> other processes to access it.
>
> I already know how to do named shared memory between two processes. I only included that to describe my overall problem.
>
> The problem that I am having is how I can make three totally-independent processes interact:
>
> 1. M is a master process that creates a semaphore.
> 2. P1 is a process that operates against the semaphore.
> 3. P2 is a process that operates against the semaphore.
> 4. It is not permissible that M be responsible for launching P1 or P2.
> 5.  The semaphore, one way or another, must allow itself to be specified as one of the synchronization primitives in epoll_wait()

This general problem descripton does not say anything more than your first
mail.

Use eventfd.  To share it, use a Unix domain socket created by M.  (This
socket must be created at a well-known path.  shm_open() works similarly,
but that it creates a file in a RAM disk and mmap()s it is just an
implementation detail.)


Regards,
Clemens

RE: WaitForMultipleObjects/etc. In Kernel

[Network Nut]

> -----Original Message-----
> From: Clemens Ladisch [mailto:clemens@ladisch.de]
> Sent: Friday, January 31, 2014 4:54 PM
> To: Network Nut
> Cc: 'Austin S. Hemmelgarn'; linux-kernel@vger.kernel.org
> Subject: Re: WaitForMultipleObjects/etc. In Kernel
> 
> Network Nut wrote:
> >> Assuming that you're porting to mainline distributions (and not
> >> embedded devices), named SHM segments are accessible (providing the
> >> accessing process has correct permissions) under /dev/shm.  You just
> >> need to make sure that you create the segment with the right
> >> permissions for the other processes to access it.
> >
> > I already know how to do named shared memory between two processes.
> I only included that to describe my overall problem.
> >
> > The problem that I am having is how I can make three totally-independent
> processes interact:
> >
> > 1. M is a master process that creates a semaphore.
> > 2. P1 is a process that operates against the semaphore.
> > 3. P2 is a process that operates against the semaphore.
> > 4. It is not permissible that M be responsible for launching P1 or P2.
> > 5.  The semaphore, one way or another, must allow itself to be
> > specified as one of the synchronization primitives in epoll_wait()
> 
> This general problem descripton does not say anything more than your first
> mail.
> 
> Use eventfd.  To share it, use a Unix domain socket created by M.  (This
> socket must be created at a well-known path.  shm_open() works similarly,
> but that it creates a file in a RAM disk and mmap()s it is just an
> implementation detail.)

How do I create the socket at "well-known path"?

Regards,

-Nut

RE: WaitForMultipleObjects/etc. In Kernel

[Network Nut]

> -----Original Message-----
> From: Network Nut [mailto:sillystack@gmail.com]
> Sent: Friday, January 31, 2014 5:00 PM
> To: 'Clemens Ladisch'
> Cc: 'Austin S. Hemmelgarn'; linux-kernel@vger.kernel.org
> Subject: RE: WaitForMultipleObjects/etc. In Kernel
> > -----Original Message-----
> > From: Clemens Ladisch [mailto:clemens@ladisch.de]
> > Sent: Friday, January 31, 2014 4:54 PM
> > To: Network Nut
> > Cc: 'Austin S. Hemmelgarn'; linux-kernel@vger.kernel.org
> > Subject: Re: WaitForMultipleObjects/etc. In Kernel
> >
> > Network Nut wrote:
> > >> Assuming that you're porting to mainline distributions (and not
> > >> embedded devices), named SHM segments are accessible (providing
> the
> > >> accessing process has correct permissions) under /dev/shm.  You
> > >> just need to make sure that you create the segment with the right
> > >> permissions for the other processes to access it.
> > >
> > > I already know how to do named shared memory between two
> processes.
> > I only included that to describe my overall problem.
> > >
> > > The problem that I am having is how I can make three
> > > totally-independent
> > processes interact:
> > >
> > > 1. M is a master process that creates a semaphore.
> > > 2. P1 is a process that operates against the semaphore.
> > > 3. P2 is a process that operates against the semaphore.
> > > 4. It is not permissible that M be responsible for launching P1 or P2.
> > > 5.  The semaphore, one way or another, must allow itself to be
> > > specified as one of the synchronization primitives in epoll_wait()
> >
> > This general problem descripton does not say anything more than your
> > first mail.
> >
> > Use eventfd.  To share it, use a Unix domain socket created by M.
> > (This socket must be created at a well-known path.  shm_open() works
> > similarly, but that it creates a file in a RAM disk and mmap()s it is
> > just an implementation detail.)
> 
> How do I create the socket at "well-known path"?

Nevermind. I see it. 

It is the sockaddr_un.sun_path.

Thanks!

-Nut