having problems with daemonizing

having problems with daemonizing

[Lowell Gilbert]

Hi.

I have an application working successfully with Xenomai 3.0.8 on a 4.14
kernel. I use Yocto to build the system; when I tried to move to a newer
version of Yocto, my application hung on trying to become a daemon. This
is happening with the daemon() call (which is what I've used up to now)
and with fork().

I built a test application so that I could confirm that this problem
only occurs when I link (and wrap) with Xenomai. However, Xenomai
doesn't seem to do anything significant with fork, so I'm puzzled about
why this might be happening. I am not using libdaemon.

Here are the changes that I thought might be significant:
| newer (nonworking setup)  | older (working) |
| gcc-cross-arm-8.2.0       |           7.3.0 |
| glibc-2.28                |            2.26 |
| glib-2.0-1_2.58.0         |     1_2.52.3-r0 |
| binutils-cross-arm-2.31.1 |          2.29.1 |
| coreutils-8.30            |            8.27 |

Does anything jump out as a candidate for causing problems with a fork()
call? Is there anything else I should be considering?

Thanks.

Be well.

Re: having problems with daemonizing

[Steve Freyder]

On 4/26/2019 4:18 PM, Lowell Gilbert via Xenomai wrote:
> Hi.
>
> I have an application working successfully with Xenomai 3.0.8 on a 4.14
> kernel. I use Yocto to build the system; when I tried to move to a newer
> version of Yocto, my application hung on trying to become a daemon. This
> is happening with the daemon() call (which is what I've used up to now)
> and with fork().
>
> I built a test application so that I could confirm that this problem
> only occurs when I link (and wrap) with Xenomai. However, Xenomai
> doesn't seem to do anything significant with fork, so I'm puzzled about
> why this might be happening. I am not using libdaemon.
>
> Here are the changes that I thought might be significant:
> | newer (nonworking setup)  | older (working) |
> | gcc-cross-arm-8.2.0       |           7.3.0 |
> | glibc-2.28                |            2.26 |
> | glib-2.0-1_2.58.0         |     1_2.52.3-r0 |
> | binutils-cross-arm-2.31.1 |          2.29.1 |
> | coreutils-8.30            |            8.27 |
>
> Does anything jump out as a candidate for causing problems with a fork()
> call? Is there anything else I should be considering?
>
> Thanks.
>
> Be well.
>
I can tell you that I have a hang issue due to fork() in a Xenomai
program if, after the fork(), I don't do an exec().  I believe
the hang is related to registry access, and the fact that the
Unix domain socket connecting to sysregd that is inherited by
the forked process (which has FD_CLOEXEC set) hasn't yet gotten
closed (no exec() yet so no action on FD_CLOEXEC flags yet).

If you are running into the same problem, and you don't require
registry access, you should see the problem go away if you throw
the --no-registry switch on the command line that invokes your
program.  That's not a real fix, but it's perhaps a way to know
if you're seeing a related problem.

In my case, the way I see the "hang" is via an attempt to list
the contents of /run/xenomai using find:

root:~ # find /run/xenomai

If I run a program XX that uses the registry, that does a fork() call
and then does not exec(), and while that program is running, I
execute the above find command, it will hang part way through the
listing.  If I kill program XX, the listing continues (un-hangs).

If I run a program that uses the registry, that does a fork() and
then an exec(), no such hang occurs during the find command.

Phillipe made the change to fix this originally by adding SOCK_CLOEXEC
to the socket() call in sysreg.c, and it did fix it but I realized
much later it fixes it only if you actually call exec(), which in my
code I always do, but more recently one of our developers had some
code that didn't exec(), which was the first time I saw this hang.

Phillipe, I had it on my list to ask you about this but it hasn't
bitten me lately and I forgot until I saw this msg about fork().

I think deamonizing in its canonical form of: fork(), let the forked
process take over, and then exit() in the parent, is problematic when
you have a wrapped main() where the wrappers already initialized the
sysreg mechanism but the process that was done for is now gone, and
the fork()'ed process has no idea it has a sysreg socket in hand.

Perhaps the better answer when daemonizing is to use --no-init and then
have the forked() process do manual xenomai_init() call?

HTH,

Regards,
Steve

Re: having problems with daemonizing

[Philippe Gerum]

On 4/27/19 12:20 AM, Steve Freyder wrote:
> On 4/26/2019 4:18 PM, Lowell Gilbert via Xenomai wrote:
>> Hi.
>>
>> I have an application working successfully with Xenomai 3.0.8 on a 4.14
>> kernel. I use Yocto to build the system; when I tried to move to a newer
>> version of Yocto, my application hung on trying to become a daemon. This
>> is happening with the daemon() call (which is what I've used up to now)
>> and with fork().
>>
>> I built a test application so that I could confirm that this problem
>> only occurs when I link (and wrap) with Xenomai. However, Xenomai
>> doesn't seem to do anything significant with fork, so I'm puzzled about
>> why this might be happening. I am not using libdaemon.
>>
>> Here are the changes that I thought might be significant:
>> | newer (nonworking setup)  | older (working) |
>> | gcc-cross-arm-8.2.0       |           7.3.0 |
>> | glibc-2.28                |            2.26 |
>> | glib-2.0-1_2.58.0         |     1_2.52.3-r0 |
>> | binutils-cross-arm-2.31.1 |          2.29.1 |
>> | coreutils-8.30            |            8.27 |
>>
>> Does anything jump out as a candidate for causing problems with a fork()
>> call? Is there anything else I should be considering?
>>
>> Thanks.
>>
>> Be well.
>>
> I can tell you that I have a hang issue due to fork() in a Xenomai
> program if, after the fork(), I don't do an exec().  I believe
> the hang is related to registry access, and the fact that the
> Unix domain socket connecting to sysregd that is inherited by
> the forked process (which has FD_CLOEXEC set) hasn't yet gotten
> closed (no exec() yet so no action on FD_CLOEXEC flags yet).
> 
> If you are running into the same problem, and you don't require
> registry access, you should see the problem go away if you throw
> the --no-registry switch on the command line that invokes your
> program.  That's not a real fix, but it's perhaps a way to know
> if you're seeing a related problem.
> 
> In my case, the way I see the "hang" is via an attempt to list
> the contents of /run/xenomai using find:
> 
> root:~ # find /run/xenomai
> 
> If I run a program XX that uses the registry, that does a fork() call
> and then does not exec(), and while that program is running, I
> execute the above find command, it will hang part way through the
> listing.  If I kill program XX, the listing continues (un-hangs).
> 
> If I run a program that uses the registry, that does a fork() and
> then an exec(), no such hang occurs during the find command.
> 
> Phillipe made the change to fix this originally by adding SOCK_CLOEXEC
> to the socket() call in sysreg.c, and it did fix it but I realized
> much later it fixes it only if you actually call exec(), which in my
> code I always do, but more recently one of our developers had some
> code that didn't exec(), which was the first time I saw this hang.
> 
> Phillipe, I had it on my list to ask you about this but it hasn't
> bitten me lately and I forgot until I saw this msg about fork().
> 
> I think deamonizing in its canonical form of: fork(), let the forked
> process take over, and then exit() in the parent, is problematic when
> you have a wrapped main() where the wrappers already initialized the
> sysreg mechanism but the process that was done for is now gone, and
> the fork()'ed process has no idea it has a sysreg socket in hand.
> 
> Perhaps the better answer when daemonizing is to use --no-init and then
> have the forked() process do manual xenomai_init() call?
> 

I don't know yet, I'll follow up on this.

-- 
Philippe.

Re: having problems with daemonizing

[Julien Blanc]

Le vendredi 26 avril 2019 à 17:20 -0500, Steve Freyder via Xenomai a
écrit :
> 
> I can tell you that I have a hang issue due to fork() in a Xenomai
> program if, after the fork(), I don't do an exec().  I believe
> the hang is related to registry access, and the fact that the
> Unix domain socket connecting to sysregd that is inherited by
> the forked process (which has FD_CLOEXEC set) hasn't yet gotten
> closed (no exec() yet so no action on FD_CLOEXEC flags yet).

We ran into a similar issue (still using 3.0.5, which seems not to have
the FD_CLOEXEC flag set).

If you don’t share any file descriptors with the child process, the
following (after the fork and before the exec) did help in our case :

DIR* dirs = opendir("/proc/self/fd");
int fd = dirfd(dirs);
dirent* d = readdir(dirs);
while (d != nullptr)
{
	int a = 0;
	a = std::strtol(d->d_name, nullptr, 10);
	if (errno == 0 && a > 2 && a != fd)
	{
		close(a);
	}
	d = readdir(dirs);
}
closedir(dirs);

Basically it closes any open file descriptor by the child process,
except stdout/stderr.

Note that we are forking from a linux domain thread hosted in a xenomai
application, creating a pure linux application. In that case the
registry stuff is complete nonsense in the child process. But this
should also be ok for other use cases.

Regards,

Julien

RE: having problems with daemonizing

[Lange Norbert]

> -----Original Message-----
> From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Lowell
> Gilbert via Xenomai
> Sent: Freitag, 26. April 2019 23:19
> To: xenomai@xenomai.org
> Subject: having problems with daemonizing
>
> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
>
>
> Hi.
>
> I have an application working successfully with Xenomai 3.0.8 on a 4.14
> kernel. I use Yocto to build the system; when I tried to move to a newer
> version of Yocto, my application hung on trying to become a daemon. This is
> happening with the daemon() call (which is what I've used up to now) and
> with fork().
>
> I built a test application so that I could confirm that this problem only occurs
> when I link (and wrap) with Xenomai. However, Xenomai doesn't seem to do
> anything significant with fork, so I'm puzzled about why this might be
> happening. I am not using libdaemon.
>
> Here are the changes that I thought might be significant:
> | newer (nonworking setup)  | older (working) |
> | gcc-cross-arm-8.2.0       |           7.3.0 |
> | glibc-2.28                |            2.26 |
> | glib-2.0-1_2.58.0         |     1_2.52.3-r0 |
> | binutils-cross-arm-2.31.1 |          2.29.1 |
> | coreutils-8.30            |            8.27 |
>
> Does anything jump out as a candidate for causing problems with a fork()
> call? Is there anything else I should be considering?

There is a general issue with glibc 2.28 (and newer), exec'ing a Xenomai executable after a fork:
https://www.xenomai.org/pipermail/xenomai/2019-March/040519.html

Norbert
________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________

Re: having problems with daemonizing

[Lowell Gilbert]

Philippe Gerum via Xenomai <xenomai@xenomai.org> writes:

> On 4/27/19 12:20 AM, Steve Freyder wrote:
>> I think deamonizing in its canonical form of: fork(), let the forked
>> process take over, and then exit() in the parent, is problematic when
>> you have a wrapped main() where the wrappers already initialized the
>> sysreg mechanism but the process that was done for is now gone, and
>> the fork()'ed process has no idea it has a sysreg socket in hand.
>> 
>> Perhaps the better answer when daemonizing is to use --no-init and then
>> have the forked() process do manual xenomai_init() call?

I think that's reasonable. With the caveat that it turns xenomai_init()
into an API function.

> I don't know yet, I'll follow up on this.

Re: having problems with daemonizing

[Philippe Gerum]

On 4/28/19 5:26 PM, Philippe Gerum via Xenomai wrote:
> On 4/27/19 12:20 AM, Steve Freyder wrote:
>> On 4/26/2019 4:18 PM, Lowell Gilbert via Xenomai wrote:
>>> Hi.
>>>
>>> I have an application working successfully with Xenomai 3.0.8 on a 4.14
>>> kernel. I use Yocto to build the system; when I tried to move to a newer
>>> version of Yocto, my application hung on trying to become a daemon. This
>>> is happening with the daemon() call (which is what I've used up to now)
>>> and with fork().
>>>
>>> I built a test application so that I could confirm that this problem
>>> only occurs when I link (and wrap) with Xenomai. However, Xenomai
>>> doesn't seem to do anything significant with fork, so I'm puzzled about
>>> why this might be happening. I am not using libdaemon.
>>>
>>> Here are the changes that I thought might be significant:
>>> | newer (nonworking setup)  | older (working) |
>>> | gcc-cross-arm-8.2.0       |           7.3.0 |
>>> | glibc-2.28                |            2.26 |
>>> | glib-2.0-1_2.58.0         |     1_2.52.3-r0 |
>>> | binutils-cross-arm-2.31.1 |          2.29.1 |
>>> | coreutils-8.30            |            8.27 |
>>>
>>> Does anything jump out as a candidate for causing problems with a fork()
>>> call? Is there anything else I should be considering?
>>>
>>> Thanks.
>>>
>>> Be well.
>>>
>> I can tell you that I have a hang issue due to fork() in a Xenomai
>> program if, after the fork(), I don't do an exec().  I believe
>> the hang is related to registry access, and the fact that the
>> Unix domain socket connecting to sysregd that is inherited by
>> the forked process (which has FD_CLOEXEC set) hasn't yet gotten
>> closed (no exec() yet so no action on FD_CLOEXEC flags yet).
>>
>> If you are running into the same problem, and you don't require
>> registry access, you should see the problem go away if you throw
>> the --no-registry switch on the command line that invokes your
>> program.  That's not a real fix, but it's perhaps a way to know
>> if you're seeing a related problem.
>>
>> In my case, the way I see the "hang" is via an attempt to list
>> the contents of /run/xenomai using find:
>>
>> root:~ # find /run/xenomai
>>
>> If I run a program XX that uses the registry, that does a fork() call
>> and then does not exec(), and while that program is running, I
>> execute the above find command, it will hang part way through the
>> listing.  If I kill program XX, the listing continues (un-hangs).
>>
>> If I run a program that uses the registry, that does a fork() and
>> then an exec(), no such hang occurs during the find command.
>>
>> Phillipe made the change to fix this originally by adding SOCK_CLOEXEC
>> to the socket() call in sysreg.c, and it did fix it but I realized
>> much later it fixes it only if you actually call exec(), which in my
>> code I always do, but more recently one of our developers had some
>> code that didn't exec(), which was the first time I saw this hang.
>>
>> Phillipe, I had it on my list to ask you about this but it hasn't
>> bitten me lately and I forgot until I saw this msg about fork().
>>
>> I think deamonizing in its canonical form of: fork(), let the forked
>> process take over, and then exit() in the parent, is problematic when
>> you have a wrapped main() where the wrappers already initialized the
>> sysreg mechanism but the process that was done for is now gone, and
>> the fork()'ed process has no idea it has a sysreg socket in hand.
>>
>> Perhaps the better answer when daemonizing is to use --no-init and then
>> have the forked() process do manual xenomai_init() call?
>>
> 
> I don't know yet, I'll follow up on this.
> 

Could you try the patch below? Ideally, we should have this in 3.0.9 if this improves the situation.

Thanks,

diff --git a/lib/cobalt/init.c b/lib/cobalt/init.c
index abd990692..02a99c569 100644
--- a/lib/cobalt/init.c
+++ b/lib/cobalt/init.c
@@ -184,20 +184,26 @@ static void low_init(void)
 	cobalt_ticks_init(f->clock_freq);
 }
 
+static int cobalt_init_2(void);
+
 static void cobalt_fork_handler(void)
 {
 	cobalt_unmap_umm();
 	cobalt_clear_tsd();
 	cobalt_print_init_atfork();
-	if (cobalt_init())
+	if (cobalt_init_2())
 		exit(EXIT_FAILURE);
 }
 
-static void __cobalt_init(void)
+static inline void commit_stack_memory(void)
 {
-	struct sigaction sa;
+	char stk[PTHREAD_STACK_MIN / 2];
+	cobalt_commit_memory(stk);
+}
 
-	low_init();
+static void cobalt_init_1(void)
+{
+	struct sigaction sa;
 
 	sa.sa_sigaction = cobalt_sigdebug_handler;
 	sigemptyset(&sa.sa_mask);
@@ -228,20 +234,9 @@ static void __cobalt_init(void)
 			    " sizeof(cobalt_sem_shadow): %Zd!",
 			    sizeof(sem_t),
 			    sizeof(struct cobalt_sem_shadow));
-
-	cobalt_mutex_init();
-	cobalt_sched_init();
-	cobalt_thread_init();
-	cobalt_print_init();
 }
 
-static inline void commit_stack_memory(void)
-{
-	char stk[PTHREAD_STACK_MIN / 2];
-	cobalt_commit_memory(stk);
-}
-
-int cobalt_init(void)
+static int cobalt_init_2(void)
 {
 	pthread_t ptid = pthread_self();
 	struct sched_param parm;
@@ -249,7 +244,12 @@ int cobalt_init(void)
 
 	commit_stack_memory();	/* We only need this for the main thread */
 	cobalt_default_condattr_init();
-	__cobalt_init();
+
+	low_init();
+	cobalt_mutex_init();
+	cobalt_sched_init();
+	cobalt_thread_init();
+	cobalt_print_init();
 
 	if (__cobalt_control_bind)
 		return 0;
@@ -288,12 +288,19 @@ int cobalt_init(void)
 	return 0;
 }
 
+int cobalt_init(void)
+{
+	cobalt_init_1();
+
+	return cobalt_init_2();
+}
+
 static int get_int_arg(const char *name, const char *arg,
 		       int *valp, int min)
 {
 	int value, ret;
 	char *p;
-	
+
 	errno = 0;
 	value = (int)strtol(arg, &p, 10);
 	if (errno || *p || value < min) {


-- 
Philippe.

Re: having problems with daemonizing

[Lowell Gilbert]

Philippe Gerum via Xenomai <xenomai@xenomai.org> writes:

> Could you try the patch below? Ideally, we should have this in 3.0.9 if this improves the situation.

It works for me with auto-init enabled, and solves my file-open problem as well.

Re: having problems with daemonizing

[Philippe Gerum]

On 5/3/19 4:26 PM, Lowell Gilbert wrote:
> Philippe Gerum via Xenomai <xenomai@xenomai.org> writes:
> 
>> Could you try the patch below? Ideally, we should have this in 3.0.9 if this improves the situation.
> 
> It works for me with auto-init enabled, and solves my file-open problem as well.
> 

Ok, thanks for the feedback.

-- 
Philippe.