* asterisk hangs with RT priority
@ 2008-05-18 16:10 Patrick McHardy
2008-05-20 9:59 ` Patrick McHardy
2008-12-24 11:36 ` Herbert Xu
0 siblings, 2 replies; 8+ messages in thread
From: Patrick McHardy @ 2008-05-18 16:10 UTC (permalink / raw)
To: Linux Kernel Mailinglist
I'm seeing hanging asterisk processes on startup when using
RT priority with current -git. Unfortunately I'm not sure about
the last working version, but it was one of the final 2.6.25 rcs.
The hang stops when executing "schedtool -N $(pidof asterisk)".
The last thing seen in strace (tried multiple times, output is
identical each time) is:
...
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1
ENOENT (No such file or directory)
close(3) = 0
socket(PF_FILE, SOCK_STREAM, 0) = 3
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1
ENOENT (No such file or directory)
close(3) = 0
open("/etc/group", O_RDONLY|0x80000 /* O_??? */) = 3
_llseek(3, 0, [0], SEEK_CUR) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=504, ...}) = 0
mmap2(NULL, 504, PROT_READ, MAP_SHARED, 3, 0) = 0xb7f2e000
_llseek(3, 504, [504], SEEK_SET) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=504, ...}) = 0
munmap(0xb7f2e000, 504) = 0
close(3) = 0
setgroups32(3, [103, 20, 29]) = 0
setuid32(102) = 0
when executing the schedtool command above at this point, it
continues with
capset(0x19980330, 0, {CAP_NET_ADMIN, CAP_NET_ADMIN, 0}) = 0
geteuid32() = 102
stat64("/etc/asterisk/extconfig.conf", {st_mode=S_IFREG|0640,
st_size=1506, ...}) = 0
open("/etc/asterisk/extconfig.conf", O_RDONLY|O_LARGEFILE) = 3
...
sysrq-t shows:
...
[ 32.032337] =======================
[ 32.032337] S21asterisk S 00000000 0 1220 1148
[ 32.032337] c7d7cf00 00000282 c0103158 00000000 c10e1800
c7cbbc00 c7cbbd6c ffffffea
[ 32.032337] 00000004 c7cbbbf8 c7d7cf78 c012097f c7cbaa80
070c0065 070c0065 c7d7cf3c
[ 32.032337] 00000000 c7cbbd00 0000000c 00000000 00000003
c7cbbc00 00000001 c7cbbc00
[ 32.032337] Call Trace:
[ 32.032337] [<c0103158>] ? do_notify_resume+0x51/0x78e
[ 32.032337] [<c012097f>] do_wait+0x5b1/0xb8c
[ 32.032337] [<c0119ce2>] ? default_wake_function+0x0/0xd
[ 32.032337] [<c0120fbf>] sys_wait4+0x65/0xa2
[ 32.032337] [<c0121023>] sys_waitpid+0x27/0x29
[ 32.032337] [<c0103c4a>] syscall_call+0x7/0xb
[ 32.032337] [<c0320000>] ? serial_pci_guess_board+0x170/0x1ce
[ 32.032337] =======================
[ 32.032337] asterisk R running 0 1229 1220
[ 32.032337] c7d7afb0 00000286 00000000 00001000 00000000
c7cbaa80 c7cbabf0 b7fa9ff4
[ 32.032337] b7faa668 b7f68b40 c7d7a000 c0103cee b7fa9ff4
00000007 0000035f b7faa668
[ 32.032337] b7f68b40 bfca8b04 0804abc8 0000007b 0000007b
c0100000 ffffffff b7f97ff6
[ 32.032337] Call Trace:
[ 32.032337] [<c0103cee>] work_resched+0x5/0x28
[ 32.032337] =======================
...
[ 32.032337] Sched Debug Version: v0.07, 2.6.26-rc1 #28
[ 32.032337] now at 31553.299102 msecs
[ 32.032337] .sysctl_sched_latency : 20.000000
[ 32.032337] .sysctl_sched_min_granularity : 4.000000
[ 32.032337] .sysctl_sched_wakeup_granularity : 10.000000
[ 32.032337] .sysctl_sched_child_runs_first : 0.000001
[ 32.032337] .sysctl_sched_features : 895
[ 32.032337]
[ 32.032337] cpu#0, 2667.480 MHz
[ 32.032337] .nr_running : 3
[ 32.032337] .load : 179570
[ 32.032337] .nr_switches : 7594
[ 32.032337] .nr_load_updates : 6015
[ 32.032337] .nr_uninterruptible : 0
[ 32.032337] .jiffies : 4294698635
[ 32.032337] .next_balance : 0.000000
[ 32.032337] .curr->pid : 1235
[ 32.032337] .clock : 31339.548336
[ 32.032337] .cpu_load[0] : 179570
[ 32.032337] .cpu_load[1] : 178546
[ 32.032337] .cpu_load[2] : 178043
[ 32.032337] .cpu_load[3] : 177875
[ 32.032337] .cpu_load[4] : 178008
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 0.000001
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 0.000001
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 0
[ 32.032337] .load : 0
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 0.000001
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 0.000001
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 0
[ 32.032337] .load : 0
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 0.000001
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 0.000001
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 0
[ 32.032337] .load : 0
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 0.000001
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 0.000001
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 0
[ 32.032337] .load : 0
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] cfs_rq[0]:
[ 32.032337] .exec_clock : 0.000000
[ 32.032337] .MIN_vruntime : 26351.612585
[ 32.032337] .min_vruntime : 6015.548989
[ 32.032337] .max_vruntime : 26351.612585
[ 32.032337] .spread : 0.000000
[ 32.032337] .spread0 : 0.000000
[ 32.032337] .nr_running : 2
[ 32.032337] .load : 2048
[ 32.032337] .nr_spread_over : 0
[ 32.032337]
[ 32.032337] runnable tasks:
[ 32.032337] task PID tree-key switches prio
exec-runtime
sum-exec sum-sleep
[ 32.032337]
--------------------------------------------------------------------------------
--------------------------
[ 32.032337] klogd 1168 26351.612585 14 120
0
0 0.000000 0.000000
0.000000
[ 32.032337] asterisk 1229 25780.061231 22 89
0
0 0.000000 0.000000
0.000000
[ 32.032337] R bash 1235 26362.344107 89 120
0
0 0.000000 0.000000
0.000000
[ 32.032337]
The above is from inside lguest, but the problem happens on
a real system as well. If more information is needed, please
ask.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: asterisk hangs with RT priority
2008-05-18 16:10 asterisk hangs with RT priority Patrick McHardy
@ 2008-05-20 9:59 ` Patrick McHardy
2008-12-24 11:36 ` Herbert Xu
1 sibling, 0 replies; 8+ messages in thread
From: Patrick McHardy @ 2008-05-20 9:59 UTC (permalink / raw)
To: Linux Kernel Mailinglist
Patrick McHardy wrote:
> I'm seeing hanging asterisk processes on startup when using
> RT priority with current -git. Unfortunately I'm not sure about
> the last working version, but it was one of the final 2.6.25 rcs.
> The hang stops when executing "schedtool -N $(pidof asterisk)".
The problem is gone in the current -git tree.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: asterisk hangs with RT priority
2008-05-18 16:10 asterisk hangs with RT priority Patrick McHardy
2008-05-20 9:59 ` Patrick McHardy
@ 2008-12-24 11:36 ` Herbert Xu
2008-12-24 13:42 ` Dhaval Giani
1 sibling, 1 reply; 8+ messages in thread
From: Herbert Xu @ 2008-12-24 11:36 UTC (permalink / raw)
To: Patrick McHardy, Ingo Molnar; +Cc: Linux Kernel Mailing List
On Sun, May 18, 2008 at 06:10:03PM +0200, Patrick McHardy wrote:
> I'm seeing hanging asterisk processes on startup when using
> RT priority with current -git. Unfortunately I'm not sure about
> the last working version, but it was one of the final 2.6.25 rcs.
> The hang stops when executing "schedtool -N $(pidof asterisk)".
I'm now hitting exactly the same problem with 2.6.28-rc9 (I had
it with 2.6.27 too but thought it might go away on its own :)
I've made a small program to attempt to reproduce this based on
the asterisk code. Unfortunately it doesn't work on its own.
However, if you run it under strace then it does hang in the
same away as asterisk (which seems to happen after the setuid
call, as strace shows it before hanging and the printk after
it in asterisk gets executed).
Note that the hang goes away if either setscheduler or setuid
is removed.
#include <sched.h>
int main(void)
{
struct sched_param sched = {.sched_priority = 10};
sched_setscheduler(0, SCHED_RR, &sched);
setuid(106);
return 0;
}
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: asterisk hangs with RT priority
2008-12-24 11:36 ` Herbert Xu
@ 2008-12-24 13:42 ` Dhaval Giani
2008-12-24 20:37 ` Herbert Xu
0 siblings, 1 reply; 8+ messages in thread
From: Dhaval Giani @ 2008-12-24 13:42 UTC (permalink / raw)
To: Herbert Xu
Cc: Patrick McHardy, Ingo Molnar, Linux Kernel Mailing List,
Bharata B Rao, Srivatsa Vaddagiri, Peter Zijlstra, Balbir Singh
On Wed, Dec 24, 2008 at 10:36:17PM +1100, Herbert Xu wrote:
> On Sun, May 18, 2008 at 06:10:03PM +0200, Patrick McHardy wrote:
> > I'm seeing hanging asterisk processes on startup when using
> > RT priority with current -git. Unfortunately I'm not sure about
> > the last working version, but it was one of the final 2.6.25 rcs.
> > The hang stops when executing "schedtool -N $(pidof asterisk)".
>
> I'm now hitting exactly the same problem with 2.6.28-rc9 (I had
> it with 2.6.27 too but thought it might go away on its own :)
>
> I've made a small program to attempt to reproduce this based on
> the asterisk code. Unfortunately it doesn't work on its own.
> However, if you run it under strace then it does hang in the
> same away as asterisk (which seems to happen after the setuid
> call, as strace shows it before hanging and the printk after
> it in asterisk gets executed).
>
> Note that the hang goes away if either setscheduler or setuid
> is removed.
>
Hmm. Do you have CONFIG_FAIR_USER_SCHED set on?
Thanks,
--
regards,
Dhaval
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: asterisk hangs with RT priority
2008-12-24 13:42 ` Dhaval Giani
@ 2008-12-24 20:37 ` Herbert Xu
2008-12-24 21:32 ` Peter Zijlstra
0 siblings, 1 reply; 8+ messages in thread
From: Herbert Xu @ 2008-12-24 20:37 UTC (permalink / raw)
To: Dhaval Giani
Cc: Patrick McHardy, Ingo Molnar, Linux Kernel Mailing List,
Bharata B Rao, Srivatsa Vaddagiri, Peter Zijlstra, Balbir Singh
On Wed, Dec 24, 2008 at 07:12:25PM +0530, Dhaval Giani wrote:
>
> Hmm. Do you have CONFIG_FAIR_USER_SCHED set on?
I can't find such a config option, but
$ grep SCHED .config
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=m
CONFIG_IOSCHED_DEADLINE=m
CONFIG_IOSCHED_CFQ=m
CONFIG_DEFAULT_IOSCHED="noop"
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
CONFIG_SCHED_HRTICK=y
CONFIG_NET_SCHED=y
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
$
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: asterisk hangs with RT priority
2008-12-24 20:37 ` Herbert Xu
@ 2008-12-24 21:32 ` Peter Zijlstra
2008-12-24 22:07 ` Herbert Xu
0 siblings, 1 reply; 8+ messages in thread
From: Peter Zijlstra @ 2008-12-24 21:32 UTC (permalink / raw)
To: Herbert Xu
Cc: Dhaval Giani, Patrick McHardy, Ingo Molnar,
Linux Kernel Mailing List, Bharata B Rao, Srivatsa Vaddagiri,
Balbir Singh
On Thu, 2008-12-25 at 07:37 +1100, Herbert Xu wrote:
> On Wed, Dec 24, 2008 at 07:12:25PM +0530, Dhaval Giani wrote:
> >
> > Hmm. Do you have CONFIG_FAIR_USER_SCHED set on?
>
> I can't find such a config option, but
>
> $ grep SCHED .config
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> CONFIG_GROUP_SCHED=y
> CONFIG_FAIR_GROUP_SCHED=y
> CONFIG_RT_GROUP_SCHED=y
> CONFIG_USER_SCHED=y
> # CONFIG_CGROUP_SCHED is not set
> CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
> CONFIG_SCHED_HRTICK=y
So you have uid-group scheduling and RT-group scheduling enabled (a
feature that's experimental for real and has never been enabled by
default), looking at the sys_setuid() code, the real uid change is done
by switch_uid() and that doesn't have a failable scheduler hook.
The thing is, I suspect the uid you switch to doesn't have a RT runtime
quota configured, therefore the RT task that gets placed in it by
switch_uid() doesn't get to run.
[ Please read Documentation/scheduler/sched-rt-group.txt
when you enable RT group scheduling ]
The correct thing would be for switch_uid() (or set_user) to fail with
-EINVAL, much like cpu_cgroup_can_attach() currently does for cgroup
grouping.
After that it demonstrates a bug in your test program, which fails to
check errors ;-)
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: asterisk hangs with RT priority
2008-12-24 21:32 ` Peter Zijlstra
@ 2008-12-24 22:07 ` Herbert Xu
2008-12-25 7:52 ` Peter Zijlstra
0 siblings, 1 reply; 8+ messages in thread
From: Herbert Xu @ 2008-12-24 22:07 UTC (permalink / raw)
To: Peter Zijlstra
Cc: herbert, dhaval, kaber, mingo, linux-kernel, bharata, vatsa, balbir
Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
>
> So you have uid-group scheduling and RT-group scheduling enabled (a
> feature that's experimental for real and has never been enabled by
> default), looking at the sys_setuid() code, the real uid change is done
> by switch_uid() and that doesn't have a failable scheduler hook.
An experimental marking is no excuse for being broken.
> The thing is, I suspect the uid you switch to doesn't have a RT runtime
> quota configured, therefore the RT task that gets placed in it by
> switch_uid() doesn't get to run.
>
> [ Please read Documentation/scheduler/sched-rt-group.txt
> when you enable RT group scheduling ]
This seems broken to me. The only way for a process to get into
RR mode is if it had been set by someone with the right privileges
or if it was inherited from its parent.
So having a default where such a process stops executing altogether
after performing a setuid is wrong.
The default should be to give each user a non-zero allotment.
> The correct thing would be for switch_uid() (or set_user) to fail with
> -EINVAL, much like cpu_cgroup_can_attach() currently does for cgroup
> grouping.
>
> After that it demonstrates a bug in your test program, which fails to
> check errors ;-)
Well the program which this was based on, asterisk does check for
errors on setuid. However, even if setuid did return an error this
isn't much better than the status quo since the user will be left
with the question "why on earth is asterisk failing on setuid?".
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: asterisk hangs with RT priority
2008-12-24 22:07 ` Herbert Xu
@ 2008-12-25 7:52 ` Peter Zijlstra
0 siblings, 0 replies; 8+ messages in thread
From: Peter Zijlstra @ 2008-12-25 7:52 UTC (permalink / raw)
To: Herbert Xu; +Cc: dhaval, kaber, mingo, linux-kernel, bharata, vatsa, balbir
On Thu, 2008-12-25 at 09:07 +1100, Herbert Xu wrote:
> Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> >
> > So you have uid-group scheduling and RT-group scheduling enabled (a
> > feature that's experimental for real and has never been enabled by
> > default), looking at the sys_setuid() code, the real uid change is done
> > by switch_uid() and that doesn't have a failable scheduler hook.
>
> An experimental marking is no excuse for being broken.
True, and we strive to fix them -- so thanks for finding this one.
> > The thing is, I suspect the uid you switch to doesn't have a RT runtime
> > quota configured, therefore the RT task that gets placed in it by
> > switch_uid() doesn't get to run.
> >
> > [ Please read Documentation/scheduler/sched-rt-group.txt
> > when you enable RT group scheduling ]
>
> This seems broken to me. The only way for a process to get into
> RR mode is if it had been set by someone with the right privileges
> or if it was inherited from its parent.
>
> So having a default where such a process stops executing altogether
> after performing a setuid is wrong.
True, therefore the setuid must fail.
> The default should be to give each user a non-zero allotment.
That's sadly impossible. The thing is, you cannot over-commit this time
-- nor change an active configuration. So the only possibility left is a
default of 0.
> > The correct thing would be for switch_uid() (or set_user) to fail with
> > -EINVAL, much like cpu_cgroup_can_attach() currently does for cgroup
> > grouping.
> >
> > After that it demonstrates a bug in your test program, which fails to
> > check errors ;-)
>
> Well the program which this was based on, asterisk does check for
> errors on setuid. However, even if setuid did return an error this
> isn't much better than the status quo since the user will be left
> with the question "why on earth is asterisk failing on setuid?".
Maybe a more descriptive error like -ENOTIME ?
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2008-12-25 7:52 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-05-18 16:10 asterisk hangs with RT priority Patrick McHardy
2008-05-20 9:59 ` Patrick McHardy
2008-12-24 11:36 ` Herbert Xu
2008-12-24 13:42 ` Dhaval Giani
2008-12-24 20:37 ` Herbert Xu
2008-12-24 21:32 ` Peter Zijlstra
2008-12-24 22:07 ` Herbert Xu
2008-12-25 7:52 ` Peter Zijlstra
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.